Test infrastructure of a webapp

Testing large web applications can be really challenging. For this effort we focused on testing the site in 4 different fronts:

1. Unit testing
2. Functional testing
3. UI testing
4. Performance testing

Unit Testing

We invested quite a bit of time to cover our code with as many unit tests as possible. Of course the decision on how to develop unittests was pretty easy, given that we are working on django. We used django's testing framework and standard library unittest from django utils. Currently we have an overall coverage of ~85%, and here is an example of how it changes over time.

Functional Testing

Selenium is the standard way of performing functional testing. It simulates a user's browsing behavior on a site. This is what is used also for oDesk's functional testing. Since ver1.4, django supports natively selenium testing.

[...] LiveServerTestCase allows the use of automated test clients other than the Django dummy client such as, for example, the Selenium client, to execute a series of functional tests inside a browser and simulate a real user's actions.

For every basic group of pages oDesk's visitor site supports, extensive functional tests are written to identify and report broken pieces of the site.

UI Testing

This is where things got really interesting. When we talk about UI testing we mainly mean testing the actual visual result that the user sees in their browser, and also make sure that the structure of the page is as expected. To give some examples of what our expectations were, we wanted to detect:

• broken images in the site
• broken layout on a page
• missing text

There are various tools that can do this, but integration is not always as easy. The tools that we ended up researching were:

1. Quality Bots
2. Fighting layout bugs
3. validator.nu

All of those looked pretty promising and are open source.

Quality Bots

This tool is really promising. It is developed by Google and its primary goal is to reduce the regression test suite and provide free web testing at scale with minimal human intervention. Usually UI testing that happens by different frameworks is via image comparison, but even if it sounds promising it is not an industry de facto quality assurance methodology. As described in Quality bots site:

[it] will crawl the website on a given platform and browser, while crawling it will record the HTML elements rendered at each pixel of the page. Later this data will be used to compare and calculate layout score.

The approach Quality Bots is following sounded really promising, but integration of such a tool in our infrastructure turned out to be more time consuming than we wanted, so we decided to defer this for a later time. However, I strongly recommend anyone working on testing to read though Quality Bots wiki/code, to understand how it works. Even if you don't end up using the tool, you can definitely get ideas out of Google's testing procedure.

Fighting layout bugs (FLB)

Fighting layout bugs is an automatic library for the detection of layout bugs. It currently supports detection for the following scenarios:

• invalid image URLs
• text near or overlapping horizontal edge
• text near or overlapping vertical edge
• text with too low contrast
• elements with invisible focus

All these scenarios are commonly found in software and instead of catching them manually, we integrated FLB with our framework and detection happens automatically. FLB is written in java and we integrated it in django with py4j. Py4j gateway server is run automatically by a fabric script executing tests. FLB is used with Firefox via the WebDriver implementation provided by Selenium. FLB test cases are invoked each time selenium.get method is executed. Here is how this is implemented:

validator.nu

As a sanity/lint check, we also validate the structure of our HTML. Invalid HTML can usually lead to ugly layout bugs. validator.nu is used by w3c for HTML5 validation. It validates HTML5, SVG, MathML, RDF, IRI and more. It also runs as standalone service. So for us it was a no-brainer to use it. We integrated it by implementing a middleware. This middleware sends content to a local instance of validator.nu on process_response. An HtmlValidationError is thrown when the html is invalid. In this case, we add a list of html errors in the response and output this list of errors at the bottom of the page; here is an example of how it looks:

Performance Testing

We use various tools to test our site's performance. A well known tool we use is apache's ab tool. ab is a tool for benchmarking apache's HTTP server. It shows how many requests per second (RPS) an apache installation is capable of serving.

We also use apache's JMeter and bash scripts to produce heavy load on our servers to test their strength on different load types. With those tests:

• we check response codes for various groups of pages
• we measure the min, max, average response time for accessing these links
• we display the success rate for accessing all of the links
• we issue random requests to our servers with various concurrency levels

Last but not least, something that we are currently looking into is a log replay mechanism to measure our performance. In general with performance testing, we can test with various loads and for some specific URLs, though the traffic we produce is not realistic. With log replay functionality we have the ability to "replay" requests based on apache's access log. This way, we have the ability to measure our performance under traffic that is produced by real users.

I would strongly recommend, if you are interested in reading about performance testing, to go through this resource: http://www.igvita.com/2008/09/30/load-testing-with-log-replay/, its really useful.

Testing Results Presentation

All our tests are run with a single fabric command, to which we can pass arguments to disable specific stages if we want to. This command is invoked in every build we run via jenkins and if our tests fail, the build also fails. Code coverage, counts of failing tests, screenshots of broken layouts (found via UI testing) and soon performance results are all presented with graphs in jenkins. Here's a few example screenshots:

Screenshots of broken URLs in UI

Counts of successful vs broken tests

Example of text overlapping with edge

Building a structured and distributed CMS

As a first step of oDesk's rebranding project, we had to move our home-grown CMS into a more robust and full featured one. Having in mind that we'll eventually need to develop the rest of the visitor site in the same infrastructure and after benchmarking drupal vs django-cms (see here analysis) we picked django-CMS. So currently all our static pages are build via our django-CMS installation.

Building static pages with reusable SCSS components

One of our primary goals was to be able to use reusable CSS components in our static pages, so that we don't end up with high level of customized CSS in our static pages. We were using Compass to define CSS components but we wanted to expose them in our CMS also. So we had to create a custom plugin for django-CMS. The plugin includes 5 basic fields which you can see here:

And the way this works is the following:

class SassStylesheetPlugin(CMSPlugin):
    name = models.CharField(...)
    css_media = models.CharField(...)
    scss_body = models.TextField(...)
    compiled_css_body = models.TextField(...)
    upload = models.BooleanField(...)

    def _get_compiled_css(self):
        compiled_body = scss_compiler.compile_scss(self.scss_body)
        content = '<style type="text/css" media="%s">%s</style>' % (
            self.css_media, compiled_body)
        if not self.upload:
            return content
        return compressor.output()

    def save(self, commit=False):
        self.compiled_css_body = self._get_compiled_css()
        return super(SassStylesheetPlugin, self).save(commit=commit)

The plugin reads the input SCSS and compiles it to a CSS output. The compressor we use is django-compressor and the SCSS compiler we use is pyScss.

This way writing static pages in our CMS becomes easier and less messy. CMS administrators can @import reusable SCSS components that are globally defined in our stylesheets and use them in their pages. As a result we can have static pages that are using less CSS code and actually look and feel similar to the rest of the site.

Using S3 as a shared resource

One of the things we always have in mind while developing the visitor site is that we are operating in a distributed environment in the cloud—our services are replicated on multiple EC2 instances in multiple availability zones. As a result, a single server instance failure or even a whole availability zone failure should not impact our reliability.

When admins create static pages or upload content such as images, these files need to be stored in a central location so that they can be accessed by all servers. S3 provides an ideal solution for this, as it is a storage service that can also serve the content directly to the visitors.

Every time a page on the CMS is modified we automatically compile the SCSS to CSS, we create a file and upload it to S3 (just as we do with images and other files) and then just append in the HTML of the page a link to the stylesheet.

New oDesk brand launched!

After a long period of silence, I'm back! About a year ago I transitioned to oDesk, where I got the chance to learn and implement many new technologies.

The new oDesk visitors site (ie whatever the user sees before they signup) is built on a django stack, using apache and running on multiple EC2 instances, with an RDS backend. We also use continuous integration with jenkins and have an automated deployment process which allows us to easily push our code changes in multiple servers seamlessly. In our FE we are using HTML5, Compass and OOCSS to make our code cleaner and more reusable and django compressor to minify it.

The road was long, rough and exciting. I learned way too many things and still learning during this project...since its not over yet. :)

I'll try to share my experiences in the following months, but for now please browse through the odesk site as a visitor and enjoy the experience.

Stay tuned for more updates.