Category: web testing

Total 32 Posts

Video Notes – Top Tips for Writing Better Selenium Tests

Top Tips for Writing Better Selenium Tests
Santiago Suarez Ordoñez
Sauce Ninja & Selenium Committer
51 min, 24 sec

My Notes:

  • Video quality is not great. Essentially a display of a code editor with too small of a font.
  • When build break due to flaky tests, developers stop trusting the build
  • Selenium 1 Tips
    • Implicit waits
      • Part of Selenium 2
      • Have to code yourself in Selenium 1. Hook into doCommand.
    • Reasonable timeouts (over 90s)
      • 30 seconds may be low when driving lots of browsers
    • Generate application states with test handles
      • Tendency is to write long test sequences
      • Better to have short sets of user actions that can be plugged together.
      • Even better if have some method to setup the application to meet the pre-requisites without having go through the steps a user must got through to get there.
      • Make sure these short cuts are turned off in production.
    • Wait for the right events (never ever use static pauses)
      • Do not use sleep
      • Implicit waits help
      • Even better is understanding how the elements appear within the application and take that into account with custom wait functions.
    • Don’t use verifications
      • Referring to the Selenium 1 “wait…” vs. “verify…” commands
      • Use the “wait…” variants.
      • Use asserts from your test frameworks instead.
    • Ignore Open and WaitForPageToLoad Features
      • Ignore the selenium failures for these commands
      • Behavior is inconsistent across browsers
      • Should behave like a user – of they see what they need to do, they do it vs waiting for the browser to say it is done. If the needed item is not there, the test will fail at that point.
  • Selenium 2 Tips
    • Selenium 2 is more robust than Selenium 1
    • Implicit Waits
      • Easier than Selenium 1
      • Use driver.implicitlyWait
    • Use findElement instead of a complex locator
      • Selenium 1 xpath and css locators may confuse people reading the code
      • Selenium 2 allows you to be explicit with how things are made, able to retrieve a list of items, interact with that list, etc. rather than using a complex selector
  • Sauce Tips
    • Report pass/fail status automatically
    • Use Sauce Connect 2


Test Heuristics Cheat Sheet

One of my team members was talking about test heuristics after attending the CAST 2011 conference. I went looking for Elisabeth Hendrickson’s excellent 2 page PDF entitled "Test Heuristics Cheat Sheet" and discovered the link I had not longer worked. A quick search revealed the current location is

This sheet is divided into the following sections:

  • Data Type Attacks
    • Paths/Files
    • Time and Date
    • Numbers
    • Strings
    • General
  • Web Tests
    • Navigation
    • Input
    • Syntax
    • Preferences
  • Testing Wisdom
  • Heuristics
    • Variable Analysis
    • Touch Points
    • Boundaries
    • Goldilocks
    • … and many more
  • Frameworks
    • Judgment
    • Observations
    • Flow
    • Requirements
    • Nouns & Verbs
    • Deming’s Cycle

I used to keep this pinned to wall above my desk. I should do it again. Recommended.


Velocity 2010: Nicole Sullivan, “The Top 5 Mistakes of Massive CSS”

The following are my notes from this presentation at the Velocity 2010 conference:

The Top 5 Mistakes of Massive CSS
Nicole Sullivan, Consultant
Stoyan Stefanov, Google
2010 O’REILLY Velocity – Web Performance and Operations Conference
June 22 – 24, 2010

(Length 37 minutes, 54 seconds)

(Some demo isssues the first couple of minutes of the video…)

Why optimize CSS?

  • It blocks progressive rendering
  • Does not minimize as well as javascript

Top 1000 Alexa Sites Study

  • 42% don’t gzip CSS
  • 44% more than 1 CSS file
  • 56% serve CSS with cookie
  • 62% don’t minify
  • 21% have > 100K of CSS

Object oriented CSS is a way to make CSS smaller.

  • Concentrate on the selector
  • Example: Grids projects, 574 butes, 14 lines

Granularity Fail + Stale Rules + Unpredicatbility + Duplication + Specificity Wars = Massive CSS

#5 Granularity Fail

  • should not match underlying programming object model
  • compare to legos – use CSS similarly
  • How to determine if correct
  • 1) Perform a visual inventory
    • Take a site object
    • Break it down into component CSS objects
    • Look for repeating patterns, for example headings
  • 2) Develop library of component parts

#4 State Fails

2 types of stale:

  • Truly stale – no longer in use
  • Rules used after user action or on subsequent pages

  • Try dust-me selectors
  • Track over time

#3 Unpredictability

  • Elements should be consistent across site (like lego bricks that don’t change behavior depending on where they are used…)
  • How to test? Search for repeating definitions of same item (for example grep h[1-6]

#2 Specificity Wars

  • Programmers making items more specific in order to prevent others from overriding their settings

How to simplify:

  • Use hacks sparingly
  • Avoid styling ids
  • Avoid !important

What to do

  • Style classes rather than elements
  • Give rules the same strength

#1 Duplication

  • Grep is your friend
  • grep for margin set to zero – too many, may need a reset stylesheet
  • grep for float – too many, may need grid system
  • grep for font-size – may be heading disguised as something else
  • Example from Facebook



Check Lists vs Scripted Testing

I have written before about lessons software development professionals can learn from Atul Gawande.

Cem Kaner read an article about how doctors can use checklists to radically improve patient care and reflected on his own background in both law and testing. The result is the presentation The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers. I found the second half of the presentation most fascinating with the many examples of how lawyers use checklists and how this can apply to testers.

Kaner’s point about learning – scripted testing does not make the person running the test a better tester. Checklists encourages the tester to think. I couldn’t agree more.


Accidental Correctness – Date Sorting…

I have written about accidental correctness before. I recently released a fix for an item that was "accidentally correct" for nearly a year.

Early last year, we released a new feature which used Lucene to index the entries. As part of the UI, the entries were displayed – newest entries first. Things worked great. Until the new year.

It turns out we were sorting not on the date but on the date string stored in Lucene. The string was in MM/DD/YYYY format. So, the entries that started with 12 were before 11, etc. All appeared correct. The newest entries were at the top. Then January came. All of a sudden, the newest entries were at the bottom of the list. Sigh.

Lesson re-learned: make sure your test dates span years.


Flow Chart of HTTP Response Codes

Some time ago, I was looking up an HTTP response code and came across a handy flow chart of HTTP response codes on Alan Dean’s ( site. Unfortunately, the link I had written down was down. I was able to find the graphic on Flickr. A useful reference diagram.

Alan’ Deans HTTP Response Code Flow Chart


2010 CWE/SANS Top 25 Most Dangerous Programming Errors

The 2010 CWE (Common Weakness Enumeration) / SANS Top 25 Most Dangerous Programming Errors has been released. The full report should be required reading for all web programmers and testers. A pdf version is also available.

Here are the 25 items:

  1. Failure to Preserve Web Page Structure (‘Cross-site Scripting’)
  2. Improper Sanitization of Special Elements used in an SQL Command (‘SQL Injection’)
  3. Buffer Copy without Checking Size of Input (‘Classic Buffer Overflow’)
  4. Cross-Site Request Forgery (CSRF)
  5. Improper Access Control (Authorization)
  6. Reliance on Untrusted Inputs in a Security Decision
  7. Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’)
  8. Unrestricted Upload of File with Dangerous Type
  9. Improper Sanitization of Special Elements used in an OS Command (‘OS Command Injection’)
  10. Missing Encryption of Sensitive Data
  11. Use of Hard-coded Credentials
  12. Buffer Access with Incorrect Length Value
  13. Improper Control of Filename for Include/Require Statement in PHP Program (‘PHP File Inclusion’)
  14. Improper Validation of Array Index
  15. Improper Check for Unusual or Exceptional Conditions
  16. Information Exposure Through an Error Message
  17. Integer Overflow or Wraparound
  18. Incorrect Calculation of Buffer Size
  19. Missing Authentication for Critical Function
  20. Download of Code Without Integrity Check
  21. Incorrect Permission Assignment for Critical Resource
  22. Allocation of Resources Without Limits or Throttling
  23. URL Redirection to Untrusted Site (‘Open Redirect’)
  24. Use of a Broken or Risky Cryptographic Algorithm
  25. Race Condition


GTAC 2008 – Taming the Beast – How to Test an AJAX Application

I am in the process of watching the videos from the GTAC (Google Test Automation Conference) held in Seattle in October 2003. This post contains my notes for the presentation Taming the Beast – How to Test an AJAX Application (1 hour 1 minute) by Markus Clermont & John Thomas held on October 23, 2008.

I am not recommending spending time watching this video. The presentation is not bad. However, it does not translate well to video. There is a lot of interaction with the audience – which is great when you are present at the presentation. However, in the video the audience cannot be heard and the presenters do not repeat many of the questions / responses. In addition, the talk is misnamed. While an AJAX application is used as the example throughout the application, most of the talk is not about how to test an AJAX application. The talk is a good discussion of how to tame automation – automation of any application.

Here are the slide titles:

  • AJAX: A Different Beast
  • An Example GWT Application – (GWT = Taming the Beast – How to Test an AJAX Application)
  • Some Statistics – (At this point there are questions to the audience regarding their automated testing experience.)
  • System Architecture
  • Small Medium Large – (Essentially how much of the application is under test)
  • Testing Layer Pairs
  • Are we done?
  • Comparison
  • Conclusion



Selenium, Firefox and Self Signed Certificates

A little while ago, I needed to run some Selenium processes against a different server which was https only with a self signed certificate. When Selenium fired up, I saw the SSL certificate warnings followed by the test failing. Even when I inserted a pause into my script so I would have time to add the security exception, the test still failed.

A search on the web led to Elliot Smith’s post entitled Dealing with self-signed SSL certificates when running Selenium server with Firefox. At the time, the post was only a few days old. It was exactly what I needed and worked like a champ. Thank you Elliot!

I would add to Elliot’s procedure that the cert_override.txt and cert8.db files can be added to source control. Even if the person executing the tests has never followed the procedure regarding creating a Firefox profile, the process will still work.

Finally, a minor warning. If you find that you need to add additional self-signed certificates, you will need to do that as part of a full-fledged Firefox profile – not the limited version created here. In that case, just like the first time, you will need to delete all of the Firefox created files and directories except cert_override.txt and cert8.db just like the first time it was setup. (I suppose you could also copy the 2 cert*.* files from a different full fledged profile – as long as you were confident regarding the contents of those files.)