Wednesday, September 5, 2012

Test infrastructure of a webapp

Testing large web applications can be really challenging. For this effort we focused on testing the site in 4 different fronts:
  1. Unit testing
  2. Functional testing
  3. UI testing
  4. Performance testing

Unit Testing

We invested quite a bit of time to cover our code with as many unit tests as possible. Of course the decision on how to develop unittests was pretty easy, given that we are working on django. We used django's testing framework and standard library unittest from django utils. Currently we have an overall coverage of ~85%, and here is an example of how it changes over time.

Functional Testing

Selenium is the standard way of performing functional testing. It simulates a user's browsing behavior on a site. This is what is used also for oDesk's functional testing. Since ver1.4, django supports natively selenium testing.
[...] LiveServerTestCase allows the use of automated test clients other than the Django dummy client such as, for example, the Selenium client, to execute a series of functional tests inside a browser and simulate a real user's actions.
For every basic group of pages oDesk's visitor site supports, extensive functional tests are written to identify and report broken pieces of the site.

UI Testing

This is where things got really interesting. When we talk about UI testing we mainly mean testing the actual visual result that the user sees in their browser, and also make sure that the structure of the page is as expected. To give some examples of what our expectations were, we wanted to detect:
  • broken images in the site
  • broken layout on a page
  • missing text
There are various tools that can do this, but integration is not always as easy. The tools that we ended up researching were:
  1. Quality Bots
  2. Fighting layout bugs
  3. validator.nu
All of those looked pretty promising and are open source.

Quality Bots

This tool is really promising. It is developed by Google and its primary goal is to reduce the regression test suite and provide free web testing at scale with minimal human intervention. Usually UI testing that happens by different frameworks is via image comparison, but even if it sounds promising it is not an industry de facto quality assurance methodology. As described in Quality bots site:
[it] will crawl the website on a given platform and browser, while crawling it will record the HTML elements rendered at each pixel of the page. Later this data will be used to compare and calculate layout score.

The approach Quality Bots is following sounded really promising, but integration of such a tool in our infrastructure turned out to be more time consuming than we wanted, so we decided to defer this for a later time. However, I strongly recommend anyone working on testing to read though Quality Bots wiki/code, to understand how it works. Even if you don't end up using the tool, you can definitely get ideas out of Google's testing procedure.

Fighting layout bugs (FLB)

Fighting layout bugs is an automatic library for the detection of layout bugs. It currently supports detection for the following scenarios:
  • invalid image URLs
  • text near or overlapping horizontal edge
  • text near or overlapping vertical edge
  • text with too low contrast
  • elements with invisible focus
All these scenarios are commonly found in software and instead of catching them manually, we integrated FLB with our framework and detection happens automatically. FLB is written in java and we integrated it in django with py4j. Py4j gateway server is run automatically by a fabric script executing tests. FLB is used with Firefox via the WebDriver implementation provided by Selenium. FLB test cases are invoked each time selenium.get method is executed. Here is how this is implemented:

validator.nu

As a sanity/lint check, we also validate the structure of our HTML. Invalid HTML can usually lead to ugly layout bugs. validator.nu is used by w3c for HTML5 validation. It validates HTML5, SVG, MathML, RDF, IRI and more. It also runs as standalone service. So for us it was a no-brainer to use it. We integrated it by implementing a middleware. This middleware sends content to a local instance of validator.nu on process_response. An HtmlValidationError is thrown when the html is invalid. In this case, we add a list of html errors in the response and output this list of errors at the bottom of the page; here is an example of how it looks:

Performance Testing

We use various tools to test our site's performance. A well known tool we use is apache's ab tool. ab is a tool for benchmarking apache's HTTP server. It shows how many requests per second (RPS) an apache installation is capable of serving.

We also use apache's JMeter and bash scripts to produce heavy load on our servers to test their strength on different load types. With those tests:
  • we check response codes for various groups of pages
  • we measure the min, max, average response time for accessing these links
  • we display the success rate for accessing all of the links
  • we issue random requests to our servers with various concurrency levels

Last but not least, something that we are currently looking into is a log replay mechanism to measure our performance. In general with performance testing, we can test with various loads and for some specific URLs, though the traffic we produce is not realistic. With log replay functionality we have the ability to "replay" requests based on apache's access log. This way, we have the ability to measure our performance under traffic that is produced by real users.

I would strongly recommend, if you are interested in reading about performance testing, to go through this resource: http://www.igvita.com/2008/09/30/load-testing-with-log-replay/, its really useful.

Testing Results Presentation

All our tests are run with a single fabric command, to which we can pass arguments to disable specific stages if we want to. This command is invoked in every build we run via jenkins and if our tests fail, the build also fails. Code coverage, counts of failing tests, screenshots of broken layouts (found via UI testing) and soon performance results are all presented with graphs in jenkins. Here's a few example screenshots:




150 comments:

  1. Excellent post!!! Selenium automation testing tool makes your software validation process lot simpler. Keep on updating your blog with such awesome information.
    Best Selenium Training Institute in Chennai|Selenium Training in Chennai

    ReplyDelete
    Replies
    1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training India . Nowadays Java has tons of job opportunities on various vertical industry.

      Delete
  2. It's an precious information, happy to see your valuable post.
    Regards,
    Loadrunner Training in Chennai|Qtp training in Chennai

    ReplyDelete
  3. The future of software testing is on positive note. It offers huge career prospects for talented professionals to be skilled software testers. Software testing training|Software training|Software testing training in chennai

    ReplyDelete
  4. Really nice information you had posted. Its very informative and definitely it will be useful for many people

    SEO Company in Chennai

    ReplyDelete
  5. Thanks for Sharing the valuable information and thanks for sharing the wonderful article..We are glad to see such a wonderful article..
    QTP Training in Chennai | QTP Training Institute in Chennai | QTP Training

    ReplyDelete
  6. Great article..!!About testing.Thanks for sharing your informative blog of post.
    software testing course in chennai | software testing training in chennai

    ReplyDelete
  7. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    Digital Marketing Company in India

    ReplyDelete
  8. Excellent sharing. Testing is a wonderful field for IT peoples. Want to learn Selenium Training reach GangBoard. Hadoop Online Training

    ReplyDelete
  9. Nice sharing. R is a language and environment for statistical computing and graphics. Want to make a career in R Programming. Learn R Programming Online Training course @ GangBoard. We are the best provider of online training on evergreen technologies.

    ReplyDelete
  10. You have done really great job. Your blog is very unique and informative. Thanks. Devops Online Training | Data Science Online Training

    ReplyDelete
  11. Thank you for the useful post.Selenium Training It helps a lot in my training. I share you with my students. Keep posting more.
    Selenium Training in Chennai | Selenium Training in Anna Nagar

    ReplyDelete
  12. I have read your blog its very attractive and impressive. I like it your blog.

    Digital Marketing Company in Chennai Digital Marketing Agency

    ReplyDelete
  13. Nice post. I used to be checking continuously this blog and I am inspired! Very helpful info specially the last part. I take care of such information much. Load Test Website

    ReplyDelete
  14. Thank you for taking the time to provide us with your valuable information. Website Monitoring Tools

    ReplyDelete
  15. Thanks for the useful post. Keep posting more like this.
    Webdesign Lüdenscheid

    ReplyDelete
  16. Your website content nice nice and interesting to observe.
    regionale jobbörse

    ReplyDelete

  17. I found a lot of interesting information here. A really good post
    office 2010 professional plus key deutsch

    ReplyDelete
  18. Your website content nice nice and interesting to observe.
    seo lüdenscheid

    ReplyDelete
  19. Interesting post! This is really helpful for me. I like it! Thanks for sharing!
    office 2016 professional plus deutsch download

    ReplyDelete
  20. Your website content nice nice and interesting to observe.
    Microsoft Office online kaufen

    ReplyDelete
  21. Interesting post! This is really helpful for me. I like it! Thanks for sharing!
    online marketing lüdenscheid

    ReplyDelete
  22. Good post and I like it very much. By the way, anybody try this increase app downloads? I do not how to use.

    ReplyDelete
  23. Informative and impressive. Keep Updating
    Jobvermittlung

    ReplyDelete
  24. Thanks for sharing this unique and informative content which provided me the required information..
    Android Project Center in Chennai | Android Project Center in Velachery

    ReplyDelete
  25. The information shared was very much useful My sincere thanks for sharing this post Please Continue to share this post...Mat Lab Projects Center in Chennai | Mat Lab Projects Center in Velachery

    ReplyDelete
  26. Those guidelines additionally worked to become a good way to recognize that other people online have the identical fervor like mine to grasp great deal more around this condition.


    Selenium Training in Chennai

    ReplyDelete
  27. Thanks for your informative article. Your post helped me to understand the future and career prospects. Keep on updating your blog with such awesome article.
    Final Year Project Center in Chennai | Final Year Project Center in Velachery

    ReplyDelete
  28. The course in general was very useful and interesting ,tutors worked hard to introduce their best experience that lead trainees to professionalism.
    SAP Training Institute in Chennai | SAP Training Institute in Velachery | Best SAP Training Institute in Chennai

    ReplyDelete
  29. Those guidelines additionally worked to become a good way to
    recognize that other people online have the identical fervor like mine
    to grasp great deal more around this condition.


    java training in chennai

    ReplyDelete
  30. Awesome Blog with Smart Content, Especially the concept of Testing Infrastructure was excellent. Thanks for sharing such a nice blog. Automation Anywhere Training in Chennai | RPA Training Institute in Chennai | UI Path Training Institute in Chennai | Blue Prism Training Institute in Chennai

    ReplyDelete
  31. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care....Java Training in Chennai | Web Designing Training Institute in Chennai | DotNet Training Institute in Chennai

    ReplyDelete
  32. Good post. I stumble upon a daily basis. It is always interesting to read articles from other authors and practice something from their websites...
    Best Adobe Photoshop Training Institute in Chennai | No.1 CorelDraw Training Institute in Chennai

    ReplyDelete
  33. Nice post! It is really interesting to read from the beginning & I would like to share your blog to my circles, keep your blog updated. Excellent Graphics Designing Training Institute in Chennai | Perfect CorelDraw Training Institute in Chennai

    ReplyDelete
  34. Thanks for posting this useful content, Good to know about new things here, Let me share this... Excellent Photoshop Training Institute in Chennai | Perfect CorelDraw Training Institute in Velachery

    ReplyDelete
  35. Excellent blog, Allen Thomas
    Wow!! Really great lists.. Thanks Preston for sharing.. This post through get quickly ideas and write... Perfect CorelDraw Training Institute in Chennai | Best Graphics Designing Training Institute in Velachery

    ReplyDelete
  36. Great article of content is King. Helps to gain knowledge of Digital Marketing learning new things. Thank You for sharing This Valuable Information.
    Vacation Classes in Chennai | Best Technical Boot Camp in Chennai

    ReplyDelete
  37. article has a complete data with nice info.good work keep it up and collect much more data.
    Summer Courses for Business Administration in Chennai | Best Summer Courses in Porur

    ReplyDelete
  38. Article has a complete data with nice info.good work keep it up and collect much more data.
    Good Vacation Classes in Chennai | Best Summer Course in Porur

    ReplyDelete
  39. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.

    software testing course in chennai

    ReplyDelete
  40. Thanks for your informative post!!! After completing my graduation, i am confused whether to choose web design as my career..Summer Courses in Chennai | Summer Courses in Pallavaram

    ReplyDelete
  41. Your post has everything that is obtained complete and really useful. It's a beautiful presentation. The pictures are much the better. This post is very good. Thank you for presenting a very good post.
    Summer Camp Training in Chennai | Summer Camp Training in Velachery

    ReplyDelete
  42. Great blog, Really nice post..Thanks for sharing such an informative post to our vision.
    Summer Courses for Business Administration | Best Summer Course in Porur

    ReplyDelete
  43. Thanks for sharing information to our knowledge, it helps me plenty keep sharing…
    Good Summer Courses in Porur | Perfect Summer Course in Guindy

    ReplyDelete
  44. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here.
    google-cloud-platform-training-in-chennai

    ReplyDelete
  45. Great tips and its so easy to understand. really You explain everything in detailed manner and it was very interesting to read. Thank you..Good Technical Boot Camp in Chennai | Summer Course in Porur

    ReplyDelete
  46. Thanks for posting your useful and interesting Article ...Its really helpful to us,keep updating..
    Best Summer Courses for Hardware Networking in Chennai | No.1 Summer Course in Guindy

    ReplyDelete
  47. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information.
    No.1 Summer Training Courses in Porur | Best Summer Courses for Hardware Networking in Chennai

    ReplyDelete
  48. Very Nice Blog I like the way you explained these things. I’ve been looking for ways to improve my website and overall rankings.I hope your future article will help me further
    Excellent Graphics Designing Technical Boot Camp in Chennai | Perfect Vacation Courses in Chennai

    ReplyDelete
  49. Wow superb...Thank you for sharing the fantastic article,keep updating such a interesting post... Best Summer Courses for Hardware and Networking in Chennai | No.1 Vacation Classes in Guindy

    ReplyDelete
  50. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information.
    Best Photoshop Summer Courses in Guindy | No.1 Technical Boot Camp in Chennai

    ReplyDelete
  51. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information, this is useful to me…
    Software Testing Summer Courses in Adyar | Summer Courses in Velachery | Java Summer Courses in Perungudi

    ReplyDelete
  52. Impressive blog with lovely information. really very useful article for us thanks for sharing such a wonderful blog...
    Photoshop Training Institute in Chennai | Best Multimedia Training Institute in Velachery

    ReplyDelete
  53. Excellent Blog very imperative good content, this article is useful to beginners and real time Employees..
    Graphics Designing Training Institute in chennai | Best Multimedia courses in Velachery

    ReplyDelete
  54. Very interesting blog which helps me to get the in depth knowledge about the technology, Thanks for sharing such a nice blog... Multimedia Training Institute in Chennai | MatLab Training Institute in Chennai | PCB Training Institute in Chennai | Hardware & Networking Training in Chennai

    ReplyDelete
  55. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    CorelDraw Training Institute in Chennai | CorelDraw Courses in Velachery

    ReplyDelete
  56. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    Good AWS Training in Tambaram | AWS Courses in Adyar | AWS Institute in Besant Nagar

    ReplyDelete
  57. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas. Linux Exam Center in Chennai | CCNA Exam Center in Chennai | CCNP Exam Center in Chennai | Tally ERP9 Exam Center in Chennai

    ReplyDelete
  58. Mmm.. good to be here in your article or post, whatever, I think I should also work hard for my own website like I see some good and updated working in your site.

    AWS Certification Exam Centers in Taramani | AWS Training in Shozhinganallur

    ReplyDelete
  59. Its an excellent blog to help beginners, am using it, was trying to find something like this from so many days, finally got it...
    CorelDraw Certification Training in Chennai | Advance CorelDraw Training in Madipakkam

    ReplyDelete
  60. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website...
    Android Exam Certification Training in Chennai | No.1 Android Exam in Velachery

    ReplyDelete
  61. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog..
    Hardware and Networking Certification Center in Chennai | No.1 Hardware Exam in Medavakkam

    ReplyDelete
  62. Nice post. After reading your post am really very happy to found such an informative post. Keep sharing.. Certified Ethical Hacking Training in Chennai | No.1 Ethical Hacking in Porur

    ReplyDelete
  63. I like your blog format as you create user engagement in the complete article. It seems round up of all published posts. Thanks for sharing, such a nice article. UIPath Exam Center in Chennai | Automation Anywhere Exam Center in Chennai | Blue Prism Exam Center in Chennai

    ReplyDelete
  64. Brilliant piece of information, I had come to know about your web-page from my friend hardtack, Chennai,i have read atleast 9 posts of yours by now, and let me tell you, your webpage gives the best and the most interesting information.
    CompTIA Security+ Certifications Training in Chennai | Best S+ Exams in Taramani

    ReplyDelete
  65. Very nice post and right to the point. I am not sure if this is actually the best place to ask but do you folks have any idea where to hire some professional writers? Thanks in advance.... Hardware and Networking Certifications Training in Chennai | No.1 Networking Exams in Besant Nagar

    ReplyDelete
  66. Thanks for posting this useful content, Good to know about new things here, Let me share this,
    CompTIA Security Plus Certifications Course in Chennai | Best CompTIA Security+ Training in Mandaveli

    ReplyDelete
  67. your post really made my day good, i have gathered more information from your blog. Thanks for sharing such an informative post.
    CompTIA A Plus Certifications Coaching Center in Chennai | No.1 CompTIA Security+ Training in Pallavaram

    ReplyDelete
  68. Thank you for your post. This was really an appreciating one. You done a good job. Keep on blogging like this unique information with us. Comptia Network Plus Certifications Coaching in Chennai | Best N+ Training in Pallavaram

    ReplyDelete
  69. Great post. Articles that have meaningful and insightful comments are more enjoyable, at least to me.
    ISTQB Certifications Exam Cost in Chennai | QA Testing in Thiruvanmiyur

    ReplyDelete

  70. After going over a handful of the articles on your web page, I truly appreciate your
    technique of writing a blog. I added it to my bookmark site
    list and will be checking back in the near future. Please visit my website too and tell
    me your opinion.
    AQ

    kanchipuram sarees

    ReplyDelete
  71. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
    Android Certifications Exam Coaching in Chennai | No.1 Android Classes in Besant Nagar

    ReplyDelete
  72. The Information which you provided is very much useful for me. Thank You for Sharing Valuable Information.I like this blog and this is very informative. Keep sharing.
    ISTQB Certifications Exam institute in Chennai | QA Testing in Sholinganallur

    ReplyDelete
  73. Superb! I found some useful information in your blog, it was awesome to read.Thank you for sharing.
    Hardware and Networking Certifications Exam Center in Chennai | No.1 Hardware Training in Adambakkam

    ReplyDelete
  74. Nice post.Thanks for sharing this post .I really appreciate the kind of topics you post here.
    Infrastructure erp software in chennai

    ReplyDelete
  75. Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..
    ISTQB Certification Exam Course in Chennai | No.1 QA Testing in Nanganallur

    ReplyDelete
  76. There are many ways to do the development, but this is the best one to work with all the technologies
    Cisco CCNA Certification Exams Center in Chennai | No.1 Cisco CCNA Course in Pallavaram

    ReplyDelete
  77. I strongly believe that there will be good opportunities for those who looked at your site.
    Android Certifications Center in Chennai | No.1 Android Exam in Saidapet

    ReplyDelete
  78. Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals.
    AWS training in Chennai
    selenium training in Chennai

    ReplyDelete
  79. Thanks for the good words! Really appreciated. Great post. I’ve been commenting a lot on a few blogs recently, but I hadn’t thought about my approach until you brought it up. 
    Data Science Training in Chennai
    Data science training in bangalore
    Data science online training
    Data science training in pune
    Data science training in kalyan nagar

    ReplyDelete
  80. Thanks for sharing such a great information..Its really nice and informative.The style of writing is excellent and also the content is top-notch. Thanks for that shrewdness you provide the readers!

    PHP Training

    ReplyDelete
  81. I feel that i have read very nice information Today by this article. Happy to read this blog. Kindly Visit Us @ Ramanathapuram Hospital List Hospitals in Ramanathapuram

    ReplyDelete
  82. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Embeeded Project Center in Chennai | Embedded Project Training in Saidapet

    ReplyDelete
  83. Nice information about this valuable article,which helps to learn more
    VLSI Project Center in Chennai | VLSI Project in Velachery

    ReplyDelete
  84. great info, thanks a lot.

    Leading Mobile Service Centre in Chennai, We repair all models of smart phones like Samsung, Apple, Motorola, Lenovo, Redmi, Honor, Oneplus, Oppo, Coolpad, Htc and many other models.

    Mobile Service Center in Chennai | Mobile Service Centre | Mobile Service Center | Mobile Service Center near me | Mobile Service Center in velachery | Mobile Service Chennai

    ReplyDelete
  85. Thanks for sharing this information and keep updating us. This information is really very useful to me. keep on sharing....
    Real Time Project Center in Chennai | Real Time Project in Kanchipuram

    ReplyDelete
  86. Nice post.Thanks for sharing this post .I really appreciate the kind of topics you post here.
    Real Time Project Center in Chennai | Real Time Project in Kanchipuram

    ReplyDelete
  87. Articles that you have provided contains very interesting useful information. Thanks for sharing this valuable info and keep updating.
    Big Data Project Center in Chennai | No.1 Big Data Training in Guindy

    ReplyDelete
  88. This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
    python training in chennai
    python training in Bangalore
    Python training institute in chennai

    ReplyDelete
  89. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
    DevOps online Training
    Best Devops Training institute in Chennai

    ReplyDelete
  90. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us. Do check Six Sigma Training in Bangalore | Six Sigma Training in Dubai & Get trained by an expert who will enrich you with the latest trends.

    ReplyDelete
  91. Amazing article Such a nice source of information i got from you. It's really interesting and informative article. Thanks for sharing ! Kindly Visit Us @ Conversation warehouses Builders in Tamilnadu Colour Roofing Sheet manufacturer in Tamilnadu

    ReplyDelete