The Problem with Requirement Documents

Lucas Dargis outlines several reasons why he believes testers should use requirement documents for checking purposes, but should not rely heavily on them for testing. Lucas Dargis is a software testing consultant. He has led the testing efforts of mission critical and flagship projects for several global companies. He specializes in the development and implementation of testing strategies.

The prevalent idea that testers are dependent on a requirements document to do their job is a dangerous one. Requirements are not always needed to test. In fact, in many situations, they may actually reduce a tester’s effectiveness.

The process of deriving tests directly from the requirements has several names. The ISTQB uses the term “specification-based testing”, sometimes it’s referred to as “Happy Path” testing, but I think the most appropriate name is “checking”. Michael Bolton wrote a well-known post about this topic. Checking is confirming that what we believe is actually true. Products are built in accordance with the requirements, so the requirements are what we believe to be true. When we verify that our product meets the requirements, we are “checking” the product. When a tester relies on a requirement document to test, he isn’t testing, he’s checking.

When we test, we are exploring, investigating and learning. Our actions are influenced by new questions and ideas that haven’t yet been explored. The use of requirement documents while testing can cause problems because it can give a false sense of test completeness, it can steer testers in the wrong direction, and it can reduce the independent thinking of the tester.

Gives a False Sense of Test Completeness

If we have verified that our product meets all of the requirements, does that mean that the product has been well tested? True, you have verified that the product behaves the way we expect it to (in specific situations) but you still don’t know how the product will behave in situations not specified in the requirements.

“That wasn’t in the requirements” I’ve heard testers use this excuse many times and it drives me crazy. As a tester it is your job to investigate the product to learn about areas and behaviors outside of the requirements. A product is much more than its conformance to the requirements. It’s up to you to cover that gap between what is expected of a product and what actually is.

Checking that a product meets the requirements is necessary of course, but checking alone does not indicate test completeness.

Steers Testers in the Wrong Direction

When you first start testing a new product or feature, what should be the first thing you test? Some might say you should verify the requirements. I challenge that view. I think the requirements should be one of the last things you test. In my opinion, a developer who writes code that didn’t meet the requirements failed to do his job. Before the test team sees the product, the developer has already spent hours working and verifying that his work is correct. Although possible, a strong developer rarely produces work that doesn’t meet the requirements. With that in mind, there is little value in retesting his work, especially when you consider the other aspects of the product that haven’t been tested yet.

Requirement documents can stifle the creativity of exploratory testing. When testers have a requirements document in front of them, they may be more likely to verify the requirements first and focus on areas where the likelihood of learning new information is the lowest. Instead, they should focus their efforts on new tests and unexplored areas where the opportunity for learning is the highest. When testing without requirements, you eliminate its influence on your testing decisions; you have to rely on your own abilities, your knowledge, and your curiosity. You test.

All testers have heard the old saying “Trust but verify”. I’m not saying that we shouldn’t verify developer’s work, only that there’s more value in performing a test for the first time, then performing the same test multiple times. Focus on the activities that produce the most value first.

Reduces Independence

The idea of “independence” (See ISTQB Foundation) refers to how close a person is to a product. A developer who wrote the code would have the least amount of independence, while a person in a different company who has never seen the product would have the most independence. Independence can often be a great quality for testers. Testers with no preconceived notions of how the product is supposed to work are able to view the product with more objectivity.

Consider the common situation where a product was built according to incorrect requirements. The tester was able to verify that the product met the requirements, so was there an issue? In this case the requirements served as a false crutch to the tester. Just because the product met the requirements doesn’t mean it was working correctly. A tester with no knowledge of the requirements would have been better positioned to identify any errors because he wouldn’t have been comparing it to an incorrect “truth”.

We have now seen some reasons why testers should avoid relying on requirement documents. While they may be a necessity for different “checking” activities, testers who wish to provide value thorough “testing” must understand that their value is best realized when they test without requirements.

Source: uTest Software Testing Blog

Advertisements

2 responses to “The Problem with Requirement Documents

  1. Hi, Lucas…

    I appreciate the reference. We’re a little more nuanced in our description of testing and checking these days. You can see our current thinking on the subject at http://www.satisfice.com/blog/archives/856, and at http://www.developsense.com/blog/2013/03/testing-and-checking-redefined/.

    Personally, I wouldn’t say ” [a tester’s] value is best realized when they test without requirements.” I would say that the tester’s value is best realized when she provides valuable (and perhaps surprising) new information to her client about the product.

    Also, I would be very careful to distinguish between requirements and requirements documents. It’s a common problem in our world to confuse an idea and the artifact that represents it. So, one kind of information that testing might reveal is that the requirements document is inconsistent with the requirements.

    Cheers,

    —Michael B.

  2. Hi Michael,

    Thanks for the comments! Just FYI, this is not my website. It’s referencing the original post found here: http://blog.utest.com/guest-post-the-problem-with-requirement-documents/2013/04/

    To me, saying a tester’s value is best realized when they provide value is redundant. You’re not really saying anything. I should have used the word “better” instead of “best” and my point would have been clearer.

    Thanks to your writing, I understand the differences between requirements and requirements documents. I tried to make sure I was consistent in that article, but maybe I slipped up a bit.

    Lucas

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s