Agile in the Trenches

First let me clarify the two main types of test perspectives:

• Unit Tests: A Test Performed at the DEVELOPER level that doesn’t include the latest code in a system build. This is where Test Driven Development is so critical
• System Testing: The testing of the entire system, with al of the latest code from each developer. This is where we need to do macro-level functionality testing and scalability testing (although developers can certainly do scalability testing at the unit level as well)

Many managers actually believe they are delivering production quality iterations by addressing only a few of the following needs:

1) Unit Test Level Integrity: Test Driven Development is Typically in Place
2) Build (Compilation) Level Integrity: Continuous Integration is Typically in Place
3) System Level Function Testing with Regression: Often not done for a variety of reasons, including the lack of knowledge that it would need to be done.
4) System Level Stress and Scalability Testing: Often not done in the iteration, but done towards the end, a critical mistake that often causes delays in rolling out at application

To put it another way, an iteration must address all of the requirements that are being built in that iteration successfully by having stakeholders verify that indeed ‘it works’. In addition we need the regression testing and script creation to ensure we have not broken previously working functionality. We also need to ensure that the iteration can scale to the non-functional requirement as specified to use by the business, such as:

“The application must scale to 1,000 concurrent users with response time on average of one second or less”

I believe many Agile managers who do (1) and (2) above, feel they are doing enough. Unfortunately, if you agree with my definition of a production system, using only (1) and (2) will not verify the system level functionality, the regression on older functionality, or the scalability of your iteration.

As I mentioned, many Agile organizations will correctly execute ‘Test Driven Development’ (TDD) and ‘Continuous Integration’ (CI) which validates the system at the ‘Unit’ Level per developer and at the System level only in the sense that a build can succeed. Just because a build can occur does not mean the functionality will pass the scrutiny of the business stakeholders.

If you’re not familiar with ‘Continuous Integration’ it is one of the most important practices for the Agile Team. A ‘Continuous Integration’ (CI) process is very simple: A Source Control System must exist in both Agile and Non-Agile programming so I will not spend any time on it (other then to say I do not advocate ‘merge’ source control where multiple users are working on the same file at the same time, and then try to ‘merge their results’).

When a file is checked in to the Source Control System, an automated System Level Build occurs, verifying that the file that was just checked in (or series of files) has not just broken the compilation. If the build fails to compile, usually the entire team is notified by a small icon in their system tray. This is incredibly important as the sooner you can catch a defect and fix it; statistically the better your odds are of a successful release. If the build is broken everything stops and an emergency intervention is performed to fix the defect. This is another reason Agile is so successful. Most processes leave integration until the end, and integration is one of the largest source of defects in most systems. This is the ultimate in catching defect early and fixing them in real-time. This is at the core of the Agile philosophy.

Many do integration builds nightly (you may have heard of the ‘Daily Build’). When you have technology like C.I. available, the daily build is not enough, as we need to know immediately when a file is checked in and a build has just been damaged.

Checking in a file is a very serious statement by a developer. He is stating to the team:

“All my TDD Tests have passed and I can compile and run the project on my computer successfully. I also believe this code I have checked in will not cause scalability problems and is of production quality”

It is unfortunately all too common for a developer to check in a file that meets these conditions but when compiled using all of the other changes that occurred, the build will not compile (remember the developer may not have files that were checked in by others recently on his computer). For example, another developer could change an interface and modify his code so that it functions correctly. However another developer also made changes that now, when taken as a whole, will not compile. Luckily there is a free and easy solution for this in a tool like CruiseControl.NET from ThoughtWorks.

Nunit provides a very east to use environment for creating your test-fixtures. To learn more there are now books being released that describe Nunit such as:

Pragmatic Unit Testing in C# with NUnit (Pragmatic Programmers) by Andy Hunt, Dave Thomas
Publisher: The Pragmatic Programmers; 1 edition (May, 2004)
ISBN: 0974514020

And for a great book on the practice of Test-Driven Development:

Test Driven Development: By Example by Kent Beck
Publisher: Addison-Wesley Professional; 1st edition (November 8, 2002)
ISBN: 0321146530

I would also like to give honorable mention to TestDrive.NET, an application that is fantastic in improving the productivity for .NET Developers using a number of TDD Frameworks (Nunit included):

Remember, TDD is hard work, and often is one of the hardest cultural changes that Agile brings about. I have offered you some resources, however TDD is far more complex and difficult to do correctly. It takes far more then just reading a few books and installing some software.

1) System Level Functionality Testing (with Regression Testing)
2) System Level Scalability Testing (with Regression Testing)

Both of these must occur if we are to proclaim that we have ‘Production Quality’ in our iteration. Therefore the role of QA is dramatically changed. In our process we dedicate one QA expert to our project using a tool like WinRunner © from Mercury Interactive or QuickTest Professional © also from Mercury. A ‘Functional Testing’ tool with ‘recordable scripts’ to record our interactions with the system, and a library of regression tests are added to (almost daily) is required to provide effective Iteration validation at the system level, with evolving Regression tests as we grow the system through our iterations.

The way System Level Functional testing is added to our life cycle is as follows:

1) A pair of developers (or a single developer if your group is not using paired programming) successfully completes a testable until of work (be it a User Story or Use Case, whichever your organization calls it).

2) The developer(s) alert the QA team (which can be one person) immediately so they can get the latest build with the new changes and test (often with the business experts) that indeed the work that was performed meets the requirements.

3) Once success is validated the QA tester can turn on the ‘scripting’ mode and record their interaction for this particular part of the system. This new ‘script’ is then added to the growing list of previous scripts (much like the developer does in ‘Test Driven Development’).

In this way the QA group is always one step behind the developers. As developers complete their work, QA is there to verify it at the system level and build the regression script. We then on a nightly basis run our regression scripts. Imagine the case described above was to occur, and a new regression test was created. At the end of the day, the QA group will start running all the automated regression tests. The computer quickly simulates the keystrokes of a real user and tests all of the functionality we have defined. It can take hours to run so imagine everyone goes home.

The next morning the first thing many people will focus on is the computer (or computers in some cases to test concurrency) for any failures. Sometimes there will be a functional that worked just 24 hours ago, yet is now broken. In spite of the Continuous Integration and TDD, a feature that was build in say, Iteration 5, is now not working. Usually because the defect was caught so early, and we know that it was, it is easily fixed. This defect is cause for a ‘red alert’ that I liken to an assembly line worker stopping the assembly line. Typically this defect is fixed, the specific test that failed is re-run to verify and we continue the day (however we will not know perhaps until tomorrow if THAT fix ended up breaking something else).

The key here (and one of the most important aspects to Agile) is the ‘catching of defects as early as possible’. When we do this, it makes fixing them usually an order of magnitude faster and easier as we ‘just wrote the code’. The problem is usually obviously and quickly solved. Compare that with defects you don’t learn about for weeks or even months after you wrote the code.

Recently I discussed the QA issues of this article with Mr. Hans Buwalda, a well-known automated testing specialist and his Action Based Testing (ABT) approach. This approach, while not agile in nature, seems to be quite applicable to agile when applied to the iteration. It takes the involvement of the QA team a step further actually. Instead of having QA one step behind development, QA is now one or more steps ahead of the development team in a quite interesting way.

Under ABT, the QA team develops groups of high-level tests well in advance of development (but only for the features in the iteration, not looking beyond the iteration) through discussions with and approval by business stakeholders and subject matter experts. These groups of test cases, known as “test modules”, can be at the unit level, functional level or even business level.

ABT tests are described as a sequence of "actions". Since ABT focuses on the automation of individual actions, rather than test cases, full automation of these tests is usually achieved at about the same time as the software under test in the iteration. Compared to the recording strategy described above, Action Based Testing, with its emphasis on automating actions instead of test cases, is much less dependant on specific UI details or behaviors of the application under test.

I am experimenting now with ABT and other firms have been successful with it. I plan on integrating it into my agile framework if it fits well and improvements on my current techniques as described.

There are many tools to perform this testing depending on if it is a ‘Web’ application or a ‘Window Forms’ application (or a Windows Service or Web Service for example).

We like to leverage a free tool as much as we can that is shipped with the more powerful editions of Visual Studio 2002 and above called the ‘Microsoft Application Center Test’ or ACT for short. However this tool is only for Web Applications. If we are building a non-web application we must look to other stress test tools such as LoadRunner © from Mercury (and no we don’t get anything out of recommending Mercury. I just really like their products).

In Agile we don’t optimize our code until the last moment, otherwise you might be optimizing code that is not a bottleneck, or is for a feature that has been cut. Due to this, I tend to not start daily scalability testing until the second week of iterations (assuming they are two weeks in duration), Developers will know when this starts and will only optimize code that is found to be a bottleneck. They have already checked in code that is production quality and is optimized to a reasonable level. This is typically on a Tuesday (the second day of the second week). This scalability testing occurs every day from this day forward until the end of the iteration and problems are caught and solved immediately as ‘red alert’ items.

For example, if we find that we have a feature that is taking on average, in our stress testing, to take ten second to perform when we have a requirement that it be five seconds, we would create a new task for the developers to optimize this area at a level beyond the optimization when it was checked in. Perhaps it is determined that an expensive series of out of process database calls can be replaced by retrieving the information from a cache ‘in process’. They key is we don’t want to spend 100% more time on a feature to try to optimize it until it becomes a problem and the need arises. Otherwise we are ‘predicting the future’ anther practice Agile avoids.

A developer writes a test, then code to make the test pass. The TDD Framework will show that test passing but other tests now failing. The developer immediately understands why; makes the change and now all tests pass. The code is further enhanced through refactoring and checked in. Unfortunately that has broken the build, and CI has alerted us immediately. By looking at the code that was checked in, the reason is immediately found, and the fix made. This time all of the unit tests pass, the code is checked in and the build does not break.

The QA team is alerted that there is new functionality in the latest build, so they get the latest code and manually verify with the business that all is well. Indeed it is so they create the regression script. All scripts are left to run over night. In the morning the team learns that a previously working feature is now broken. As it has to be code written in the last 24 hours there is a strong likelihood someone will know why this has occurred. Indeed in this case a developer knows what happened, makes the change to the code, ensures all the TDD Tests are still green and checks in the code. The CI is fine, so the regression test that was failing is re-run. Indeed this is now fixed. Everyone goes about the task or continuing their work on the iteration.

When stress testing begins we find a problem area. The programmer(s) optimize the code, re-run their TDD Tests and they pass, check in the code (and the CI is OK) and the Functional regression testing is fine. In addition the performance is now great.

So you can see how these are very powerful techniques that add tremendous power to an Agile development organization. Our ‘checks and balances’ are very empowering as they empower us to make significant changes to a system, where if that change caused a problem, we would know about it very soon. This is how Agile can ‘turn on a dime’ and embrace changes (sometimes quite significant changes) without any problem. If something was broken in the process of implementing the changes we will know and we will react very quickly. This is a key reason why Agile is the right process for environments with changing needs, indecisive business analysts and generally a ‘moving target’ as to what the system needs to do.

Damon Carr is the CEO and Chief Technologist of agilefactor, based in NY, New York. The company continues its growth via Microsoft.NET Leadership, Software Engineering best practices Agile Process Evolution and the new book by Mr. Carr covering "Agile Software Factories" to be published in early 2006 by Addison-Wesley. Prior to joining agilefactor, Damon was the CTO and Co-Founder of Monetaire, a global company with offices in Europe and the US, where he spearheaded all technical innovation, including their flagship product which was named a 'top three solution' in Europe by Forrester Research. Damon continues to speak at the software industry's most prestigious conferences including the upcoming 'Better Software Conference' (http://www.sqe.com/bettersoftwareconf/) where he is a featured speaker on 'Agile Processes and Software Factory Evolution', he also leads various high profile global user groups.

Contact the author.