Saturday, March 16, 2013

Test Management is more about testing than management.


This started as a Test Manager survey from Mike Lyles (https://t.co/LSvlWobcdW) and as I was answering question 45, I just kept on writing.

Any test manager needs to understand the three basic things any tester is or should be doing on any given day. For a couple of years now, I've been using what the Bach brothers refer to as the TBS Metrics of Session Based Test Management fame. T stands for "Test Design and Execution" related activities, B stands for "Bug Investigation and Reporting" and S stands for "Session Setup". On any given day, the amount spent on all three should add up to 100% of their time for a particular session or day depending on how you want to implement this.

Test design and execution means evaluating the product and looking for problems. Bug investigation and reporting is what happens once the tester stumbles into behavior that looks like it might be a problem. Session setup is anything else testers do that makes the first two tasks possible, including tasks such as configuring equipment, locating materials, reading manuals, or writing a session report.[1]

Based on personal experience, TBS is a really good way to look at the Work Breakdown of any given tester/ test group. You can see where the focus is from a management perspective. If a higher percentage of time is spent on Test Design and Execution, then you know that the testers are doing what they are hired for, which is ... ummm ... Testing. If a majority of time is spent on Bug investigation and reporting, it could be that the product is not quite ready for release, or that the focus of the testers are on regression duties due to the number of bugs and fixes that need validating. In this case, testing can't continue. If a majority of time is focused on Session Setup, your tester/team might be in too much meetings (which might be good or not) or is trying to solve a technology problem that your operations or support team can help with, for example setting up a LAMP stack for your web app so the tester can test in a local environment. Either way, testing has been limited.

There is no ideal percentage combination between all three since this is just an indication of what your context has to deal with. Or this is a way for you to initially evaluate and come up with questions before you have a one on one with a team member. You can even look at trends of any given tester or project based activities but trust me when I tell you that all this will give you is a graph and nothing more. As a manager, I really like what Jon Bach mentioned in his STP Crew talk about delivering Value with Test Metrics (paywall alert).

"Less BS, more T".

This is just a birds-eye view or guide for understanding what your team is up to. One on one debriefings or retrospectives will help and are more effective in finding out what is really going on with your organization. If you really want to be more effective as a manager, get in the trenches. Try to be at par with your testers' knowledge about the product. As a manager, your mission is two fold; serve your organization in making sure your testers are finding the necessary information about the quality of your software and you serve your testers in making sure that you enable them to do what they were hired for and that is to test. Bugs are just the bonus that comes as a result of your team testing properly and intelligently.

The one main lesson that I can take with me from this is that, Test Management is indeed more about testing than management.

Sources:

  1. SBTM by Jon Bach - http://people.mozilla.com/~nhirata/SBT/sbtm.pdf

8 comments:

Unknown said...

Excellent writing Perze. This is great information and thanks for it. You're right on point and I especially like how you state that if the team is spending more time on bug reviews and resolution, then we are working in the wrong space. Great words and advice!!!

Phil Kirkham said...

I hadn't come across TBS before so thanks for fluent me n in this - and thanks to Mike for the survey that sparked he post. Gotta love the testing community for spreading ideas

Jyothi said...

Excellent writeup Perze.
This is a great perspective that test teams need to imbibe.
Good moral here.

Pradeep Soundararajan said...

I must tell you how much I tried to work on the TBS within a session and I failed to capture it well or was confused about it. It distracted me from the mission in hand and ever since I have not been doing it.

The sad part is - the people who have learned SBTM from me (whose work you may already be exposed to) don't do TBS as well. However, we have ensured that we do less BS (pun intended, just like how you might have) and more Testing and Test Coverage.

The way I can recollect running SBTM as a manager on one of the early projects was to dedicate sessions to investigate bugs than doing it within a session so I understand how many sessions or focus has been given on repro or investigating issues.

Unknown said...

Pradeep,

Thanks for your feedback. As I mentioned in the post, it really depends on what you intend the reporting information is for. I was able to implement TBS in lieu of the daily status reports and without the implementation of SBTM. It gave me the kind of information that I needed and the format is understandable enough that I can forward it to the other non-involved development leads that they can understand the information without the costly debriefing.

There were challenges of course especially in the beginning and once we had momentum, challenges of sustaining the quality of information cropped up but these issues can be realized and corrected with the one-on-one sessions.

I do know 5 people in your organization that has done this with me and maybe that would be an interesting lunch and learn for the rest of your organization to hear about.

William Echlin said...

TBS is an approach for monitoring and tracking that I hadn't heard of before. Love the simplicity of it. My only concern is that time tracking (which is what this boils down to) is notoriously difficult to do.

In my experience people hate having to record how much time they've spent against activities. Maybe if you're lucky people will record data accurately for the first few weeks. But a month or so in you can guarantee that they'll be guessing the figures last thing on a Friday, for the whole week, before they leave for the weekend.

So whilst I totally agree that having TBS figures is hugely beneficial you have to question how accurate those figures are. Nothing worse than making decisions based on wrong data.

Maybe there is an opportunity to capture this data in a test management tool so that there is less overhead on the tester for recording it? Then again most test management tools record the 'T' part of it so it's just a question of picking up the B and S parts.

Like the idea I just think you have to be careful with the implementation in order to get accurate figures.

Unknown said...

Bill,

I'd like to clarify that TBS is not about Time Tracking nor Control. If anything, it's a reminder akin to the daily scrums.

As you said, it is notoriously difficult to do and personally, we don't have very effective oracles in finding out the truth if what was recorded is really accurate or not. As a tester, I really appreciate my former managers who trusted me with what I can do with my time which in turn made me more productive rather than being on my toes and worrying about how many minutes i've spent on something, etc.

One thing I didn't get to include in the post is that the reporting part of TBS should not even take more than 5 - 10 minutes of your time since a bulk of what you are doing has already been recorded in the sessions of SBTM itself.

Nicola said...

Great write-up. Thanks Perze.