Past the Point of Simplicity: Has the Agile Movement Gone Too Far?

After all, working software is the only real thing that your business stakeholders will judge your efforts as successful or not (they could care less about the internal structure – a huge problem in that it creates a negative psychological environment for the development team and an issue I deal directly with in this article). What about maintainability, scalability, code reuse, internal robustness, component use, etc. etc. etc.? I will address this head on in this article. Has the Agile movement gone to far in its near obsessive love of simplicity and reduction of ceremony at the expense of unwittingly encouraging teams to throw out common sense good object oriented design in an effort to 'do the easiest thing that works'?

I am, first and foremost, a practitioner, trainer and coach on Agile Development (I happen to implement in the .NET/C# environment mostly in Financial Services). However in my 6 years of 'seeing the Agile light' I have evolved a sense of wanting in the past few years for some of the necessary and missing elements of the Software Development Lifecycle (SDLC) that Agile removed. Most of it I was glad to see go (like up front Analysis at the micro level – A sure disaster in almost all situations). I have found myself 'adding back in' some new elements into my company's Agile processes and I will share one of the most important here in this article.

These additions are a combination of some of the old and well known items, but mostly these are new concepts that we have invented and found add dramatically to the effectiveness of Agile Development. These are all covered in our upcoming book so if this article was useful to you, the book contains many more great add-ins to Agile (a process we call 'agilefactor'). For example, Metrics in Software have gone though period of being 'in style' and not. I have always been a big fan of metrics if they can be measured with a minimum amount of disruption and can add significantly to the managers (and the developers) understanding of the status of their project (or of a collection of projects).

What they are doing is NOT the easiest thing that works in the long run (perhaps it might be in the short run) because they are often creating havoc inside the system in by throwing design principles out the window. Even when you have collective code ownership and a pair knows that there is another similar class that could be leveraged, and code reused, and that they really should create an abstract parent containing all that shared code, it all seems like too much work so they create another concrete as a stand alone class with no inheritance or polymorphism (when they should of moved the common code into that new abstract parent, assuming it was a semantically correct thing to do, and inherit these two classes from that abstract parent, overriding functionality where necessary).

Polymorphism's magic will take care of the rest so when you call the method 'GetDiscount' it will know (for example) that a class called 'Corporations' (a concrete class) returns a different result then the GetDiscount present on 'Individuals', another concrete class inheriting from the same abstract parent 'Customer'. Perhaps 'Customer' has a method called GetDiscount marked abstract. This is a terribly simply example but usually what happens is the developers are lazy and execute something to the effect of:

“Lets just create another class called Individual and we can add Switch/Case statements in our code by checking the type of the object to get the functionality we need.”

This is so wrong, I don't know where to begin. This problem is disastrous and it is the exception, not the rule, where an Agile team 'does the right thing' and bites the bullet, refactoring their code in Design Patterns as soon as the need arises.

In my book, this is how I determine a team can reach the holy grail of the promised Agile 'Flat Cost Curve' where the cost of change over time is a near constant. They achieve it by a near obsessive focus on 'Refactoring to Design Patterns' and 'Refactoring' in general to good Object Oriented Design. The extra work now pays off massively later as I will show. The reason almost no Agile teams can even come close to the flat cost curve is the fact that they:

• Have not instituted a mandatory 'Design Pattern' mastery in each team member


• Do not have a systemic way to 'Refactor to Patterns' when the opportunity presents itself in each iteration in a near obsessive manner

Is this harder? Perhaps at first. Is this more intrusive on your system and does it have the chance to break more code? Absolutely. But that is why you have TDD and Continuous Integration. These practices are precisely why Agile works! They allow you to make these kind of sweeping but necessary changes to your technical design. If you break something you will know immediately due to red TDD tests and you will be able to fix them immediately.

With Continuous Integration (see Agile in the Trenches for a discussion of this) you will know if there are problems at the System level with the build. This is why Agile beats the other Software Development Processes in almost all cases (as measure by Scope, Cost, Time and Quality) because you can 'embrace change' and make the hard changes, immediately knowing where you need to fix problems. Other processes offer you no such 'safety net'.

The problem is that so many Agile team are not engaging in the very thing Agile is meant to facilitate because they don't have the proper knowledge and if they do, they don't put in the effort to do the right thing. I can say this with confidence as I have seen it in countless Agile projects all over the world. Radical and sometimes dramatic change to your system without hesitation should be the rule, not the exception when it is required. You have your safety nets to allow you to bite the bullet and do the right thing so why are so many teams creating horrible technical designs that degrade dramatically over time?

This is a simple example but it illustrates the problem very well. Instead of coding proper object oriented code, teams are 'banging out' code as fast as they can with no regard for proper design and no regard for code reuse, in many cases copying and pasting code between classes (always a sure sign of a problem). They feel justified by the 'Do the simplest thing that works' mantra of Agile, not understanding that this is not a free pass to destroy the integrity of your architecture. In fact this is NOT the easiest thing that works, as you will pay dearly down the line (and this is why the flat cost curve is a myth for almost every Agile team I have not integrated these new metrics for which I will describe shortly).

This can be solved also (and perhaps more elegantly) by the Decorator pattern. The problem is, in my experience it is very rare to find teams that have a collective knowledge of design patterns where any member can even explain to you the difference between these patterns, so how can they be expected to 'do the right thing' and do the refactoring into the pattern when the need arises? The answer: Require a mastery of Design Patterns for all of your team members.

It is inexcusable to allow a team in an object-oriented environment to exist without requiring all members to master these patterns, now over a decade old. You are throwing money away if you ignore this and there is no other way but to bite the bullet and buy them books, send them to training, hire a consultant, or whatever it takes. In my book I determine this is the #1 most important critical success factor to Agile development.

Once Design Patterns are learned, require your team to master the excellent book 'Refactoring to Patterns' by Joshua Kerievsky (Addison-Wesley Professional (August 5, 2004) ISBN: 0321213351). If this is too much for your team, appoint a 'Lead Architecture Sub-Team' to master these concepts which is a subset containing your best people. In my research I show that this is the only way I am aware of for your team to routinely reach a flat cost curve: the obsessive refactoring to patterns as a core part of your Agile process, iteration by iteration. If you are unwilling to put in the learning then you might as well resign yourself to throwing money away and building systems of far less quality, as statistically that is what will happen with 90% probability.

OK so let's say you have mastered patterns, you are refactoring to patterns and you want to see how your system is evolving over time (or devolving). How can you measure your progress, iteration to iteration, and know you are on the right track?

Now that I have explained why you need it, let me explain how you can calculate it and use it. The CAR breaks things down (from a UML perspective) into 'bad' collaborations between items (in UML these are relationships and dependencies) and 'good' collaborations. A good collaboration is when a concrete class has a relationship or association to an abstract class or an Interface. It is very much in line with the concept of 'design by contract'.

When you 'Refactor to Patterns' (and you should not perform refactorings until they are necessary – a key point to keep in mind – don't try to predict the future). For example, if you knew that Individuals were coming later on in your project, you should NOT create the abstract class immediately. You should only refactor into this proper Object Oriented Design on the first occurrence that would necessitate it. Anything else would be predicting the future. How do you know that Individuals will later on be removed from this system based on an executive decision down the line? You don't, so don't try to predict the future.

There has been some lively debate between myself and some industry luminaries around 'should you perform a refactoring on the first occurrence that would require it, the second occurrence, or which? Ron Jeffries for example has been an incredible sounding board for me on this topic and I am in his debt for his insight and amazing wisdom around these issues.

In my experience, in over 80% of the cases, when a first occurrence emerges which would necessitate a 'hard' redesign', there will be a second occurrence later on. If you are sharp you might be saying 'hold on a minute... Aren't YOU predicting the future?' In a way, yes and in a way no.

The refactoring will benefit the system NOW. Even if that third occurrence never comes, we have still benefited. However the odds say it WILL occur. So I am doing something NOW that improves the system and has a statistical likelihood to bring major benefits in the future. If they never come, we have not lost anything. We have simply spent time improving our architecture.

The debate is 'what if that refactoring was overkill'. This is up to the people 'in the trenches' to decide if the refactoring is worth it. Perhaps you know 99.9% that there will never be that third occurrence. It might be OK to not perform the refactoring. Like most things in life 'it depends', however as a rule of thumb you SHOULD perform the refactoring if you are unsure as the statistics speak for themselves.

Therefore I firmly recommend you perform the refactoring on the first occurrence that would require it. This is important because adding a participant to an existing pattern, or properly implement object oriented design, can be as much as ¼ the effort of creating the item from scratch. This is due to the code reuse you are leveraging (i.e. PROPER OBJECT ORIENTED DESIGN). This is the secret to the flat cost curve. I believe I am the first to crack this puzzle and provide a solution that works more times then it doesn't in getting you to that flat cost curve.

Let's take a worst case scenario using a totally different design. Imagine the following class structure in UML for a system that generates Invoices and notifications based on information from a ledger. All of the classes know about the other classes (a design that appears to be abysmal, but you never know when a design like this is mandatory for reasons we are not aware of).

Here we have a design where any change to any class has the potential of breaking ALL of the other three classes. This is hardly a robust design capable of withstanding the rigors of production and on-going maintenance without a lot of heartache. We have a combination of one bidirectional UML relationship and five bidirectional dependencies. All of these classes are concrete and clearly appears to not be an acceptable design. What can we do?

Design Patterns to the rescue! The Mediator pattern is documented and common knowledge among those that have kept pace with Software Engineering improvements. It addresses just this sort of problem. The Mediator will eliminate these dependencies and relationships and create a single 'Mediator' class that oversees the communication between all of these classes so they do not need to be aware of each other (information hiding is good).

Before I move on let me define the Core Agility Ratio (CAR) and what it would calculate as in the example above. Keep in mind, our goal with the CAR is to keep it below 1 or as close to 1 as possible (or as low as possible).

• SUM the total number of UML dependencies and UML relationships between concrete classes. Bidirectional counts as 2. There are “Bad Relations”. We do not count 'concrete to Abstract Class or Interface'. We also do not count composition relationships, aggregations, or generalizations; this is the numerator of our ratio.
• SUM the total number of participants in our system that have non-trivial behavior (for example, an enumeration would not be included). This is the denominator. We include Abstract Classes and Interfaces as well as concrete classes (and other elements like structures).

1) Total Number of 'Bad' Collaborations = 12 (Remember bidirectional counts as 2) 2) Total Number of Participants = 4

Given our stated goal of having a CAR below one (or close to it) we have failed miserably. And that is good, as it shows the ratio is working as we can look at this design and know it is bad.

So how does this look when we refactor it using a design pattern?

Here is the new and improved design using a Mediator:

All information that might be necessary from an early method call in a later call is persisted in the mediator. For example, if the 'Total Account Balance' is needed by the 'Notification Engine' that value could be returned in the call that is made from the 'Transaction Mediator' to the 'Ledger'. Then, when required, the value could be passed on as a method parameter to the Notification Engine.

NOTE: You are cheating if you pass 'this' or an instance of an object as a parameter. You then create a bidirectional dependency as you would not pass it if you were not going to perform a call back, creating a dependency between the called class making the callback. The collaboration just became bidirectional.

This is a simplified example but try it yourself. Run though some refactoring to pattern examples and see for yourself how the CAR is lowered. It is really nothing incredibly new as it simply encourages a 'design and code by contract approach' or the introduction of classes to dramatically reduce coupling and increase cohesion (remember those two key concepts)?

I almost forgot. Let's see what our new CAR ratio is with our 'Refactoring to a Design Pattern' (in this case a Mediator).

I purposefully used an example that did not involve a refactoring to an abstract class or an interface as I wanted to illustrate how you can lower your CAR by adding MORE concrete classes in some cases if they reduce or eliminate your bad collaborations.

In closing I will illustrate our UML example discussed previously with Customers.

Here is the concrete class that lacks the proper refactoring:

NOTE: I added UML Cardinality to this model. The 0..* shows that any of the Customers can have zero, one or many addresses and zero, one or many Invoices.

So why is the numerator zero? There are no 'Bad' relationships. Remember, we do NOT count relationships or dependencies between concrete classes and Abstract Classes or Interfaces. The other three relations are 'Generalize' relations (for the three children of Customer) and are not counted in the CAR.

For some reason Agile Development has caused many to abandon common sense object oriented design principles. Don't let that happen to you, and use the CAR ratio to help you achieve your project goals and ensure your system's architecture is not degrading over time. Also require your team (or a subset) to master Design Patterns and 'Refactoring to Patterns' using the resources listed here. As always feel free to contact me directly for more information.

Damon Carr is the CEO and Chief Technologist of agilefactor, based in NY, New York. The company continues its growth via Microsoft.NET Leadership, Software Engineering best practices Agile Process Evolution and the new book by Mr. Carr covering "Agile Software Factories" to be published in early 2006 by Addison-Wesley. Prior to joining agilefactor, Damon was the CTO and Co-Founder of Monetaire, a global company with offices in Europe and the US, where he spearheaded all technical innovation, including their flagship product which was named a 'top three solution' in Europe by Forrester Research. Damon continues to speak at the software industry's most prestigious conferences including the upcoming 'Better Software Conference' (http://www.sqe.com/bettersoftwareconf/) where he is a featured speaker on 'Agile Processes and Software Factory Evolution', he also leads various high profile global user groups.

Contact the author.