By definition, an oracle is a principle or mechanism that can tell us if the software under test is working according to someone's criteria. "Someone" in most cases is usually the person who knows how the application is supposed to work or look. This person can be the product owner, the project manager, the developer or even the tester. In most companies, this is usually the Product Owner that works hand in hand with a Project Manager. Simply put, Oracles, help us decide whether a product's behavior is inappropriate. Ultimately, oracles are usually ideas or behaviors that can confirm or tell us if a test passes or fails.
Aside from requirements and known expected user behavior, subject matter experts can be used as oracles too. So don't be surprised if members of the testing team bombard you with questions about a project if you haven't clarified the requirements and scenarios within that requirement extensively to your tester.
Heuristics on the other hand, are fallible ideas or methods that can help you investigate or solve a problem. In our context, Heuristics are application behavioral patterns that we can use to design our tests around a given project. We as a test team, also use heuristics to connect some missing requirement dots by way of discovery through exploration of the application. Heuristics are fallible due to the fact that what could be assumed as correct at this point in time can be false after a certain time has passed. Case and point, requirements change all the time. In it's entirety, Heuristics helps us design tests, investigate bugs and report information.
A classic example of a heuristic are what we call as consistency heuristics or the HICCUPPS Heuristic. This usually makes us ask the following questions as a launch point in discovering how to test any given project;
- Is the software under test consistent with the History the other version of this software?
- Is the software under test consistent with the Image that your company wants to project to it's consumers?
- Is the software under test consistent with Comparable products or competitor websites?
- Is the software under test consistent with the stakeholders Claims?
- Is the software under test consistent with User Expectations?
- Is the software under test consistent with how other Products in your company behave?
- Is the software under test consistent with the Purpose of the application?
- Is the software under test consistent with certain Standards and Statutes, such as Legal, Accessibility etc?
Due to the nature of heuristics, inconsistencies within the application may or may not be a bug. That is just the nature of a fallible heuristic. For example, if for some reason a particular functionality in the most recent version doesn't quite behave the same way as one or two versions ago, it could be that the behavior had a bug and has been fixed.
Nevertheless, the usefulness of oracles and heuristics will ultimately be on the tester's judgement and maybe his/her experience and understanding of the software under test.
[Update: See Michael Bolton's comment below regarding some details that need to be ironed out, i'll be working on updating this post with his clarifications in a separate post]
[Update: See Michael Bolton's comment below regarding some details that need to be ironed out, i'll be working on updating this post with his clarifications in a separate post]
Sources:
- Testing without a Map by Michael Bolton
- Oracles by Michael Bolton
Note: This was an internal blog post that's initially shared to the rest of our technology group and has been sanitized to remove particular details.
4 comments:
Perze, Nice post and well explained to the audience who are not aware of Heuristics and Oracles.
I'm sharing this with the testers in my network and also newbie(s) in software testing which can help them to understand about "Heuristics & Oracles" in nice way.
-- Santhosh Tuppad
http://tuppad.com/blog/
Hi, Perze...
On the whole, pretty good, but there are a couple of details that I believe are important to get clear.
"An oracle is a principle or mechanism that can tell us if the software under test is working according to someone's criteria."
That's not what we say about what an oracle is. An oracle, to us, is a principle or mechanism by which we recognize a problem. There is no principle or mechanism that can reliably tell you whether software is working, whether by one person's criteria or not. The product returns a value four for the input 2 + 2. Yay... but it's silently deleting files in the background. Or the answer was in a white font in a white output field. Or the four changes to a five a few seconds after the test is deemed finished. Or that the product works perfectly as long as it's running on our network. When something appears to be working, it's not that it is actually working; it's that you don't have an oracle to tell you that it isn't working. Oracles are fallible. As Doug Hoffman is fond of saying, tests don't pass or fail. Tests reveal information. When we infer that there's a problem in the product, that inference comes from an oracle. When we infer that there's no problem, we have to be careful to be explicit: there's no problem of which we're aware, no problem that is being pointed to by an oracle.
All this is to say that oracles are heuristic. A heuristic, as you point out, is a fallible method for solving a problem or making a decision. An oracle is a specific kind of heuristic. An oracle (fallibly) helps us to make the decision "Is there a problem here?"
Oracles are all heuristic, but not all heuristics are oracles. For example, there's another kind of heuristic: a mnemonic. A mnemonic helps to solve the problem of forgetting something. HICCUPPS is a mnemonic for remembering several classes of oracles; those oracles are (heuristically) based on the idea of consistency.
---Michael B.
Michael have said:
"Oracles are heuristics"
After read this sentence, I was thinking about it, reading about it even tweeting about it, and this was my conclusions:
First I read some text of Hoffman where he points some clasification of oracles.
He points True Oracles and Heuristics Oracles (among others).
I thought that:
Heuristic Oracles are based in heuristic sources so obviously are fallibles.
True Oracles as algorithm are not based in heuristic sources.
Then considering an oracle has 3 parts, Generator, Comparator and Evaluator.
Analysing an example of "True Oracle":
- Generator (ideally infallible algorithm)
- Comparator (SUT' function)
- Evaluator (human interpretation)
Even thinking (ideally) the algorithm is infallible, the evaluator or evaluation process is based in human interpretation. This makes the oracle fallible.
"True Oracles"
Generator of expected results: Infallible source.
Evaluator: Human (makes it fallible)
"Heuristics Oracles"
Generator of expected results: Sources heuristic
Evaluator: Human (makes it more fallible)
In conclusion, all the oracles are heuristic because all of them relies on human interpretation.
David García Romero
David, thanks for your comment.
I think this is where the philosophy behind oracles and how we as testers use the concept when it comes to testing diverges. Oracles in it's purest sense supposedly, never tells lies. But, it's up to the person who received information from the oracle on whether to believe it or not. The notion of truth whether someone believes it or not doesn't make it any less truth.
In our case, since we focus on the context of the application under test, as you said, any oracle is still bound by human interpretation and that indeed makes things fallible. But that doesn't stop there. Remember Jerry Weinberg's definition of quality? Quality is value to some person. The usefulness of oracles also boils down to the context in which we are trying to employ that oracle.
As Michael pointed out in his comments above, an oracle is a principle or mechanism by which we recognize a problem, and because we are able to recognize a problem, we might have a solution.
Post a Comment