Monday, April 2, 2012

Of Oracles and Heuristics

You might have heard testers use the term Oracles and/or Heuristics when discussing how we test our projects. I'd like to provide an explanation of the context in which we as testers use these terms. 
By definition, an oracle is a principle or mechanism that can tell us if the software under test is working according to someone's criteria. "Someone" in most cases is usually the person who knows how the application is supposed to work or look. This person can be the product owner, the project manager, the developer or even the tester. In most companies, this is usually the Product Owner that works hand in hand with a Project Manager. Simply put, Oracles, help us decide whether a product's behavior is inappropriate. Ultimately, oracles are usually ideas or behaviors that can confirm or tell us if a test passes or fails.
Aside from requirements and known expected user behavior, subject matter experts can be used as oracles too. So don't be surprised if members of the testing team bombard you with questions about a project if you haven't clarified the requirements and scenarios within that requirement extensively to your tester.  
Heuristics on the other hand, are fallible ideas or methods that can help you investigate or solve a problem. In our context, Heuristics are application behavioral patterns that we can use to design our tests around a given project. We as a test team, also use heuristics to connect some missing requirement dots by way of discovery through exploration of the application. Heuristics are fallible due to the fact that what could be assumed as correct at this point in time can be false after a certain time has passed. Case and point, requirements change all the time. In it's entirety, Heuristics helps us design tests, investigate bugs and report information.
A classic example of a heuristic are what we call as consistency heuristics or the HICCUPPS Heuristic. This usually makes us ask the following questions as a launch point in discovering how to test any given project;
  • Is the software under test consistent with the History the other version of this software?
  • Is the software under test consistent with the Image that your company wants to project to it's consumers?
  • Is the software under test consistent with Comparable products or competitor websites?
  • Is the software under test consistent with the stakeholders Claims?
  • Is the software under test consistent with User Expectations?
  • Is the software under test consistent with how other Products in your company behave?
  • Is the software under test consistent with the Purpose of the application?
  • Is the software under test consistent with certain Standards and Statutes, such as Legal, Accessibility etc?
Due to the nature of heuristics, inconsistencies within the application may or may not be a bug. That is just the nature of a fallible heuristic. For example, if for some reason a particular functionality in the most recent version doesn't quite behave the same way as one or two versions ago, it could be that the behavior had a bug and has been fixed. 

Nevertheless, the usefulness of oracles and heuristics will ultimately be on the tester's judgement and maybe his/her experience and understanding of the software under test.

[Update: See Michael Bolton's comment below regarding some details that need to be ironed out, i'll be working on updating this post with his clarifications in a separate post]

  1. Testing without a Map by Michael Bolton
  2. Oracles by Michael Bolton
Note: This was an internal blog post that's initially shared to the rest of our technology group and has been sanitized to remove particular details.

Post a Comment