Thursday, November 19, 2015

Drupal 8 is finally out ... let us pray

Drupal is my CMS, I shall always be in want.
Drupal makes me lie down in PHPastures;
Drupal leads me to murky MySQL waters;
Drupal restores my code on every reroll.
Drupal leads me not in the right path module for his argument's sake.
Even though I walk through the taxonomy 
of the views of death,
I fear no update;
for I am a tester;
your logs and your community of commiters--they comfort me.
You prepare a cache table before me
in the presence of my stakeholders;
you anoint my HEAD with l10n;
contrib module integration bugs overflows.
Surely Drupal legacy goodness and mercy shall follow me
all the days of my life,
and I shall feed my family
forever.

https://www.drupal.org/news/drupal-8.0.0-released

Monday, November 9, 2015

Saturday, August 8, 2015

Value and Waste as byproducts of Software Development

The act of Software Development essentially produces two things, value and waste. Value can be rather vague when you don't understand the context. According to +Gerald Weinberg, Quality is value to some person. +James Bach then extended that definition to, Quality is value to some person that matters. That is the value that this talk was referring to.
For any software development group to be successful, value should be maximized and waste should be minimized.

According to Lean manufacturing concepts, there are seven types of waste:
types of waste

Overproduction is essentially creating features that has no users. This is waste because no one is  willing to exchange something valuable for something that you've built. 

As an example of overproduction as waste: In the past few months, I've assumed a role as a product owner of an in-house automation tool that my team uses. When we started, we had limited information on how this tool will be used so we built in "generic" features. As time progressed, we noticed that out of a total of 200 or so functionalities that we've baked into this tool, only a handful of them are being used. 

The next type of waste, Waiting, is technically the amount of time that is spent by tickets/issues/requests while waiting for someone work on them. Another type of waste is unnecessary context-switching. In this case, time is also being spent when a developer is forced to switch between contexts when trying to fix a feature that the developer was working on in the past. Testing and Programming needs to be tightly coupled in order to minimize this particular waste.

Transport is waste when it comes to movement. This is usually defined as your delivery flow. How long does it take you to deliver an idea into a stable and usable feature in the hands of your customers? How long is your build-measure-learn loop?

Inappropriate Processing is waste as a result of bad design which forces extra activity. An example for this would be investing in a record-playback tool as a long-term investment. In the long run, maintenance can be a nightmare. Just because something is cool and new, it doesn't mean we should adopt it. Another example would be when you are developing a metrics program where nobody ever uses the information from those metrics in order to make a decision.

Excessive Inventory is waste when we produce more than the demand. In software, this can be reflected in over subscribing third party services without effectively measuring what you need. Most of this waste is usually shown as a knee jerk reaction to scaling. Bloated servers that are only used at 10% capacity even at maximum peak, or over subscribing to maintenance support costs. The one example that Thomas pointed out were Unreleased Features. You have to understand that your context is changing, your consumers preferences are changing. Every day a feature is not in production, devalues the feature to a point of uselessness.

Unnecessary Motion is when you are doing more than what you need to. In software development, this waste happens when you are misusing your resources and people. Is your company asking people to do the things that they are not good at? i.e. asking testers to focus on documentation rather than test design. What are your testers and developers doing? is there unnecessary friction? are there repetitive manual steps to get a build to an environment?

Personally, I think Defects are the mother of all wastes. If you don't fix an issue right away where multiple modules are dependent on, and you release that to your customers, fixing all downstream dependencies tend to take longer because you are not only fixing one thing, but everything else that's dependent on it. Imagine that waste in an iOS app submission workflow.

When software developers have appropriate detection tools, some defects can be instantly fixed provided you don't ignore the warning. For example, your IDE can give you instantaneous feedback if you missed a semi-colon when writing javascript code. Of course, static analysis or these sophisticated IDE's can't catch everything. There's a need for redundant ways to check for problems brought about by new changes and these ways should all include properly thought of test design.

As a part of the leadership team in my software development group we've been looking at how to minimize waste in our current processes. The good thing is that my company doesn't think of testing as waste. I would readily admit that there are things that our testing team is doing that seem wasteful. Writing acceptance criteria and automation of testing scenarios that somehow doesn't make it to production is waste, or atomically testing feature details that doesn't necessarily bring value back to the customer.

Not everything in the development process has return value. There are also wasteful activities done by programmers. Code comments don't inherently provide value to the customer, or maybe even documentation since not all documentation are created equal. Creating a tag that doesn't get released can also be considered as waste.

The fact of the matter is that waste is an inherent byproduct of creation. Our goal is not to eliminate, but to minimize.

Do you look at testing is part of your waste reduction efforts or is your approach to testing producing more waste?

Note: This post is based on one of the sessions that I attended during CAST by Thomas Vaniotis about The Context-Driven Tester in a Lean Startup.

Thursday, August 6, 2015

Remembering CAST2015 -- Part 1

This is my second CAST that I've attended in person and technically the fourth that I've participated in by watching the live webcast.

I flew into GRR on sunday and had the chance to catch up with +Dhanasekar Subramaniam over dinner. The details of that conversation requires a blog post in itself and will (probably) write that separately.

On the first day of CAST, I attended +Robert Sabourin's tutorial on Testing Fundamentals for Expert Testers. What really stood out for me on that talk has been the way +Robert Sabourin organized his presentation. He started talking about the History and Principles surrounding Quality, Economics and Management that can be applied on how we test software effectively.

The participants went through a series of exercises that further explained concepts on semantics, interpretation, pivoting based on gathered relevant data, ambiguity, etc. I also went through some practical applications in using logic, decision tables, as well as logic reduction techniques that can simplify the number of rules you will need to test by employing equivalence class partitioning.

+Robert Sabourin also focused on applying a Heuristic Model in solving problems. In order for you to show your understanding of the problem at hand, do you know;

  • What are you looking for?
  • Can you restate the problem?
  • Can you make a visual model?
  • Do you have enough information to find a solution?
    • Can you identify the variables and the relationships between these variables?
  • Do you understand the concepts used in stating the problem?
  • Do you need to ask questions?
From there, the tutorial focused on more details on equivalence partitioning, story boarding, control flow testing as well as state models.

In all honesty, that session had too many concepts to be understood and absorbed in a single day. As an RST and BBST alumni, I've been able to pick and choose which concepts I've encountered before and which ones that are completely new to me. 

There's more to be said about the first day but my brain is completely mush right now. Until the next post.

Wednesday, February 4, 2015

how 'controllability' is causing me some spam

[Edit: As of Feb. 5, 2015, the issue was fixed. I am no longer spammed]

Controllability and Observability are two basic pillars of testability. In order for me to test a given system, I should be able to observe data or a given state. For me to make my tests even more comprehensive, I need to be able to control how that data behaves as well. 

Three years ago, I was working on a project that sends various content through email. one of the challenges we had was setting up email accounts where we can send these emails to. creating new accounts is extremely painful considering how we didn't have the necessary infrastructure to create fake emails on the fly.

The simplest solution I can think of at that time is allow the "+" character in email addresses so I can use (and reuse) gmail's alias feature. After that feature was added, I proceeded in automating creation of these emails by appending epoch time to my email. As far as I can remember, these accounts were added to a non-production user database, and scripts that use these email addresses run once a day.

Recently, I've been receiving newsletter emails from said company but when I try to unsubscribe, I get flagged with an invalid email error. Looks like they fixed/removed that testability feature I requested which stops me from unsubscribing. What puzzles me even more is that when I check the epoch time equivalent, these were users that were automagically created 3 years ago and must have been pushed to a production environment. 

So there you go, everyday I get reminded of a past I've left behind. I should just use Tamper Data, override the unsubscribe POST and get it over with.

oh well. 

Thursday, April 10, 2014

lessons learned from a two dollar load test

Recently, I had the chance to work with one of our engineering groups and lead load testing efforts for one of our web redesign projects. The objective was simple, introduce stress into the system and observe how the application and the infrastructure around that application behaves and reacts under various degrees of stress.

The first logical step in planning for a load test is picking a tool understanding what information the engineering team deems as important. The objective usually gives you a good overview of what they need, but you shouldn't stop there. Part of the information that you need to know is the data story. What path does the data take when it moves from curation until it gets served to the user. You do need to understand what happens in between.

The next step would be to understand the architecture behind the application. These are some of the questions that I usually ask. Are you using CDN's?  What are your caching solutions? What are these cache's respective TTLs? Do we have multiple servers behind a load balancer? What is the load balancer algorithm?  Is there a way for me to hit an origin server directly? What are the declared settings in httpd.conf? etc. These questions will most likely provide you an insight of how complicated the data flow scenarios will be.

Talking about the data flow scenarios, this usually follows a rather standard set of scenarios. How does the system behave during a greenfield request? (A greenfield request is when none of the caches are set, this usually takes the longest) How does the system behave when the request is fully cached? and finally, every boolean combination in between. A bonus case would depend on the caching mechanism that's used. memcached, for example, has a behavior where if the cache is completely full, instead of writing to memory, it starts writing to disk. The response times in this particular case is almost as slow as a greenfield request and most of the time this is a culprit that's a bit tricky to troubleshoot if you don't have the proper server instrumentation.

I usually correlate most data flow diagrams to a series of rivers, waterfalls and dams. The end user is at the end of line and content creation is at the source. The network is the path where the water flows, the size of the path defines the bandwidth, the dams represent the caches and of course the water is the data that eventually a person consumes.

Finally we deal with the testing approaches which is really where the good stuff happens. In the years that I've performed load testing, no one has summed the vantage points where you can perform load testing better than Scott Barber. The vantage points Scott pointed out in his Web Load Testing for Dummies book (I can't seem to find a link to buy this anywhere, help?), fall under three categories, Behind the firewall; Within the cloud; and from the User's Perspective.

Each of the above vantage points provide different types of information. I will not list all of them, but I will provide what I think were my major takeaways. Behind the firewall tests or Load Testing 1.0, enables you to test component performance. Within the Cloud, aka Outside the firewall or Load Testing 1.5, provides information about the performance when there are multiple components and how these components relate to each other.  Lastly, from the user's perspective aka Load Testing 2.0. This approach is rather new and was not an option until 2009 or 2010. This gives you the ability to load real browsers from different areas of the world and swarm a particular URL. Think DDOS but used for good (not evil). The information you get out of this will be very close to what your user's response times are and will also give you a sense of how your third party content really affects your load times from that perspective.


Early this week, we had our first dry run using the User's Perspective approach using Neustar's Web Performance Tool (aka Browsermob). Within 10 minutes of the test, we found an issue with the load balancer algorithm brought about by what seemingly is a VIP misconfiguration. Personally, this is proof that it's never too early to perform load testing since it would have been disastrous if this was opened up to the world. The other cool thing about this was that we didn't have to pay a hefty licensing sum if we decided to use loadrunner, or go through a complicated setup process for our own jmeter based load generator infrastructure (which we are considering).

Based on the number of concurrent users that we used and the length of the test, it cost us (literally) $2.25 to identify problems that would have caused problems that would cost money, lost personnel time and above all, a tarnished reputation (yeah, its possible).

How do you load test?

Wednesday, March 26, 2014

testing lessons learned from a dum biryani

One of the better analogies I've heard about testing starts with the notion of preparing a dish that you will be serving to someone. As a chef you would probably ask what the customer wants and you would perform the due diligence of getting the ingredients you need, prep the ingredients, make sure you have all the necessary utensils, cookware and equipment. Then you can cook and serve the food to the customer.

There are subtle nuances in-between that whole cooking process that will most likely spell the difference between the quality of the food that's being served. A chef normally tastes the food as he cooks it. Carefully adds the ingredients and tweaks the taste by adding salt, pepper and other spices while tasting after a new change is introduced. This is heuristically true with testing. Whenever there there is a change, a risk might have been introduced and the effects of that change needs to be evaluated. That change is usually evaluated through testing, whether you are confirming the known effects of that change (bug fixes or new features), or performing "regression" testing of features that you know and care about, or performing exploratory testing to find out more things about the product  or system under test.

Test as you go, taste as you go.

For a while, this testing analogy was closest to how my understanding of testing has evolved over the years. Until March 25, 2014 (last night, as of this writing) ... when I was introduced to the Dum Biryani.


The Dum Biryani is an Indian dish that contains rice, marinated meat, spices among other ingredients. What's really fascinating with this dish is how it's cooked. The chef, layers (basmati?) rice, vegetables and meat in combination with spices in a clay, ceramic or copper pot then cover the top of the container with dough then cooked in low flame for a long, long, time. The dough cover mimic the effects of being in a pressure cooker. The result is a really good rice dish with an amazingly tender meat that goes really well with the rice.

Did you notice the gap and how this fails the tasting analogy with regards to testing? This is what really blew my mind. After the dough is closed, there is no way for the chef to correct the dish through tasting. I was also told that a really good chef will know how the Dum Biryani will taste like based on the aroma that the food releases while the ingredients are pressure cooked. The smell doesn't help the analogy either because the cook still can't effect change while the dough is closed.

What I learned was that these chefs don't need to taste the dish despite the myriad of variables that may affect the taste. To name a few, these variables are the type of rice, the water used to cook the rice, the elevation where the dish was cooked, the material of the vessel where the dish is cooked, the type of saffron being used, etc, etc, etc.

After further discussion, I realized that these chefs have learned how to cook this dish to the point where tasting can be done by feeling the ingredients through their fingers. Tasting was indeed done but not through "regular" means. The chef has attained a new level of expertise by leaning towards learning through experience and has most likely continued to work on his craft through experimentation. As testers, we should strive to do just that.

To date, no one has heard of anyone that became a respected dum biryani cook after taking a test. 

I digress.