of 404s, automation and other things

Over the weekend I've realized that the script that checks for 404s started spitting out failures. Historically, these errors pop up from time to time when some rogue third party script loads for more than ninety seconds. Before friday, this script had a very low failure rate that is under a fraction of a percent.

Let me step back a bit before I go through the solution of this problem. My context has a very interesting and complex caching architecture. Because of this, we decided to wrap the selenium open function with a hard assert so that if the url we intend to test returns a 404, the entire phpunit test class will return a failure.

At around 3pm last friday we started getting 100% failures. At first, I thought this was caused by a third party script. I even thought that maybe the problem is caused by the newly released selenium 2 codebase. But that wasn't really the problem since switching out the selenium standalone server jar to an older version returns the same error.

Maybe you figured out the problem by now ....

The issue turned out to be caused by our custom selenium->open function. The problem was that since I am checking for the 404 condition, any requests to open a bad URL will return a failure because the hard assert in the open function, will always force a failure.

Lesson Learned: Perform code reviews in your automation code. Know which function uses what and acknowledge your exceptions.

So you think you know automation? Part One

I stepped into a briar patch today.

The Michael Bolton (@michaelbolton) kind of patch. No you sillies, not the singer but a philosopher disguised as a canadian tester. Early this afternoon, I've unwittingly butted into his conversation with Adam Yuret (@AdamYuret) regarding Automated Checks, aka Automated tests to the uninitiated. I'll leave it up to you to find out why things of this nature should be called checks and not tests.

Here are some of Michael's points regarding the value Automated Checks;

1. at least THESE examples appear to work to some degree.
2. We found a bunch of interesting problems when we developed these automated checks.
3. These automated checks, on first principles, will probably make it easier to perform other automated checks.
4. These automated checks, on first principles, will probably make it easier to explore the application with automation's help.
5. The automated checks helped us to identify problems with load, performance, and stress that would have been hard otherwise.
6. These automated checks will help us to identify at least some unexpected and unwelcome changes.
7. These automated checks (since they come with logging) may assist with debugging and with retesting, should they expose bugs.
8. These checks provide coverage of stuff that with my Big Brain and Clumsy Fingers I would consider tedious, trivial, and awful.
9. These automated checks, like all forms of testing, provide partial answers that might be useful.

CHALLENGE: The skillful tester NOW presents a counterargument for every single one of those heuristics. Over to you. :)

The "skillful tester" in me agreed to his 8th point and argued that;

PA:  The moment a person starts considering something as tedious, trivial and awful, that person has least slowed down learning.
MB: What if that person uses boredom as a heuristic trigger to do something more valuable?
PA: But the I'm stuck heuristic is different from boredom. Boredom comes because you just can't think of anything better to do.
MB: I would argue that boredom is an important variety of "I'm stuck."
PA: Besides, to me, boredom is an effect and not a cause.

And much to my surprise, Michael issued to me a secondary challenge.

I'd recommend that you practice thinking more expansively (and critically) on this subject. You're right; what might *also* be right?

So looking into that earlier statement, "These checks provide coverage of stuff that with my Big Brain and Clumsy Fingers I would consider tedious, trivial, and awful.".

So what is this stuff that Michael is talking about? And what of coverage? Why does my brain have to be big? Do I understand why fingers can be clumsy? And what does the brain and fingers have to do with being the tasks I deem tedious, trivial or awful?

Now, for more serious questions, as a tester, how does automated checks bring value to my work? will it make testing my products more efficient, even make it better? or is it just throw away work? How do these checks bring value to my team? how about to my organization? What is it really that I have that require automation? What is it that I need that call for automation? Is it a far stretch to employ it? Will it require a culture change within the entire organization that I'm working for? Do I even understand it's purpose?

I dare say that if you are automating checks for the sake of your laziness and misunderstanding of your product, then don't. Automation is not just a programmatic conversion of your manual tests. Your manual tests that you deem tedious at this point in time will probably give new insight of the product under test the next time around. However you approach your product at this point in time will definitely be different as time progresses since you would have better understanding of the product (I hope) and can find other ways or paths to take which could lead you further in your exploration.

That is enough for now. Time to get some feedback.

A Conversation About Page Objects

Yesterday, I had the chance to meet the Aussie Uber Tester herself, Trish Khoo aka @hogfish who is stateside as of the writing of this post. Among the things that we discussed were automation patterns and how we use and implement them within the context of what we do. I started blabbing about how we started using the page object pattern even before the first selenium conference made it cool.

So, what is this Page Objects buzz and what is it good for?

Page Objects is essentially a coding pattern where you represent the pages under test as objects that you can interact with. In it's simplest form, a page object based test would contain two things, an element in the page under test and the actions that you can perform with that element. This pattern has proven to be very useful especially in the context that I'm in where I'm dealing with a hundred or so page types but the automated checks usually begin and end on the same page under test.

The question Trish posed before me was, how practical is the Page Object pattern if you are dealing with a complicated scenario that spans a LOT of pages. I was a little dumb founded. I didn't really think that far ahead. The most that I've dealt with in my case are 4 pages tops and even then, at that level, maintenance gets a little tricky, more annoying than tricky really.

Trish mentioned a pattern that she employs which to my understanding is behavioral based which I chose to dub as Persona Objects. I think she called it something else but I'm sticking with my name for the sake of this post. The focus of this particular coding pattern is no longer on the page under test but on the user or the persona the user takes when implementing a series of actions that spans multiple pages.

This question came at an opportune time because in the next few months, we are getting ready to look into how automation can help us test our CMS. If you know Drupal and familiar with Panels, Views and the Features module, you'll know that navigating through this system is almost a weaving art form.

A specific example I can think of is curating a specialized content-type that can be used for a never before seen page. From a CMS perspective you'd need certain types of data from multiple data sources, a list of pages in the CMS that you need to go through in order to create the content-type and finally, be able to validate the output data from CMS that our front end folks can consume in order to build the final page. To finish things off, we can also include a check for the resulting page.

I tried to layout a solution to the above problem and felt like the Page Objects pattern can handle the situation pretty well. Then taking a closer look at my scribblings, it seemed to me that after I'm done writing my script, someone would have to test it.

Yuck.

Good thing I have a couple more months to think about this and would love your input on how you do things on your end.