Over the weekend I've realized that the script that checks for 404s started spitting out failures. Historically, these errors pop up from time to time when some rogue third party script loads for more than ninety seconds. Before friday, this script had a very low failure rate that is under a fraction of a percent.
Let me step back a bit before I go through the solution of this problem. My context has a very interesting and complex caching architecture. Because of this, we decided to wrap the selenium open function with a hard assert so that if the url we intend to test returns a 404, the entire phpunit test class will return a failure.
At around 3pm last friday we started getting 100% failures. At first, I thought this was caused by a third party script. I even thought that maybe the problem is caused by the newly released selenium 2 codebase. But that wasn't really the problem since switching out the selenium standalone server jar to an older version returns the same error.
Maybe you figured out the problem by now ....
The issue turned out to be caused by our custom selenium->open function. The problem was that since I am checking for the 404 condition, any requests to open a bad URL will return a failure because the hard assert in the open function, will always force a failure.
Lesson Learned: Perform code reviews in your automation code. Know which function uses what and acknowledge your exceptions.
No comments:
Post a Comment