Driving Cost Out of Engineering, Part II

So are the BoD members duped by the unscrupulous outsourcer who makes promises they cannot keep or hides costs until it is too late, or do they truly understand what they are asking? Some unscrupulous offshore outsource firms will make claims of cost savings that are 70% to 80% over in-house software development team costs and savings of 300% over traditional US-based contractor rates. While it is true that engineers in Asia are paid much less then US-based engineers, more scrupulous outsource firms willingly point out that the total cost to outsource for offshore software engineering capability includes things like travel costs and typically significant onshore management effort of the communication processes necessary between the US domain experts and the offshore – non-US cultured, non-native English-speaking engineers. These scrupulous firms will verify that when adding together all associated costs, the real savings potential from outsourcing is more likely somewhere around 20% over in-house development teams and about half of the cost of US contractors. But of course, cost is only a small part of the larger picture.

What the good BoD member is likely to understand is that the value of outsourcing is that it gives the firm a variable cost model. Engineering is typically a fixed cost – involving long-term costs associated with salaries, facilities, equipment and infrastructure. This is a problem because revenue is often cyclical, with some quarters doing well and others doing poorly. High fixed costs during or after a poor quarter is a problem. Outsourcing software engineering can turn engineering into a variable cost that can ramp up and down as revenue cycles up and down.

The way to make outsourcing work in a variable cost model is to establish a small and carefully selected core team of employee engineers. These are the keepers of the keys, the people with a deep sense of understanding of the business domain, a strong sense of ownership, integrity, innovation and drive to keep the product moving forward. These permanent employees represent the minimal level of staffing at which the engineering department cannot cut below or the company loses their ability to create and maintain products.

As demand and funds are available, the core team can be augmented with a variable cost outsourced team of whatever size is needed and fundable. As funding increases (either through revenue growth or through investment) the outsource team can rapidly increase in number to meet the demand. And as funding decreases, the outsourced team can just as rapidly be decreased in size, preserving needed moneys (rather then spending the money on empty buildings and severance packages). If this concept is clear, now couple it with a real achievable 20% savings and the picture becomes clear.

Most scrupulous outsourcing firms will agree that a smallish US-based engineering team of ten engineers or less can be as cost-effective or better then the best offshore team. The reason for this goes straight to the heart of why software projects so often fail. Most large software projects fail because of poor communications. The formal communications needs of a ten-person or smaller team are very minimal. However, when companies need to grow rapidly beyond this level – perhaps with a second or third round of investment, or as a result of a significant market opportunity – these minimal communication processes break down swiftly. Suddenly, the team that was once a highly effective engineering team is now a much larger mob of talented, but out of control individuals.

To rapidly double the size of an in-house engineering team is extremely disruptive to productivity, or requires a long slow growth curve that allows new team members the time they need to get established (and time for the existing team to assimilate the new members). For a large scrupulous outsource firm (the kind with a thousand engineers in bench strength), assembling teams is an integral part of their business. Because these teams are typically assembled from other teams that are ramping down, the members come together with full understanding of processes, tools and techniques.

To grow an in-house team by 10 to 20 or more additional permanent hires or contractors, supplied with the same tools and techniques, and all fully operating under the same well-established engineering processes can take 6 months or more (if you want more people, add more time). A major advantage that an outsourcing firm who is specialized in developing high-quality software products can provide is the ability to assemble a team – ready to go to work, with well-established processes, trained in the tools and technologies needed for the project – in a matter of a couple of weeks.

If product quality suffers when new team members are added too quickly, then what about the quality of an outsourced engineering firm? The “product” is the responsibility of the engineering executive. If he/she agrees with or conforms to the cost containment model pushed by the BoD through outsourcing, how then does he/she ensure the quality of the product the outsource firm produces? The obvious suspects are worth mentioning, but not dwelling upon – strong credentials (such as certified as CMM Level 5), reference able clients, known referrals, and deep technical capabilities.

There are additional attributes that characterize an outsourcing software engineering firm that consistently produces high-quality products – the most important of these is their ability to successfully manage the communication process. According to multiple sources, 70% to 80% of all large software projects fail, mostly due to poor communication (see the article by Caper-Jones: Jones_April_4_2004). Communication with an engineering team 10,000 miles away – in a time zone 12+ hours different, in a different culture, and with a different native language– requires very special skills of the entire team, as well as well-established processes wrapped around those special skills. The onshore members of the team are key to successful communication. An engineering team without onshore membership to provide close customer collaboration (particularly necessary in gathering requirements, or defining architecture, as well as managing the project throughout the full lifecycle of the project) is a disaster waiting to happen.

Secondly, it is important that the outsourcing team align well with the local team – adaptable processes, compatible personalities, and development styles. Beware of too close alignment; I once knew a VP of Engineering who was such an introvert that he hardly ever spoke. He found an outsourcing team that was just like him, in that they never said a thing until the very end of the project. What a surprise he had when he found out that the result was not quite what he had expected.

Some engineering executives consider their team the best in the business, and perhaps they are. However, out of the only 180 software engineering organizations worldwide rated at Level 5 on the Capability Maturity Model (CMM) scale, only 5 exist within the U.S. Therefore, it is possible to outsource to an engineering team that is actually better at consistently producing high-quality software products on a predictable cost and schedule than the typical in-house engineering team.

And lastly, it is important to meet the team in person. Anyone representing an outsource firm who says it is a waste of time and money to visit the offshore team at their location is hiding something. Satisfy yourself in person that the team works in a modern facility, in a well-equipped computing environment, with proper security, with managerial controls, etc. Check out the team in person; experience the esprit de corps, the local community support, the cultural influences for and against developing good software products. Witness firsthand the team working through peer reviews, conducting a code walk-through, analyzing errors, gathering metrics, and watch how they manage their projects internally.

Secondly, and more directly to the point, even though the order to outsource may come from above, if the outsourcing effort fails the person most likely to lose their job is the engineering executive, not the BoD member. Consequently, most engineering executives are very careful about who they partner with to outsource engineering work. Not all firms offering outsourced services can develop software with equal quality, on a predictable schedule, for an affordable fee, or with flexible processes that ensure the customer receives the system functionality they really need. Basically, it comes down to trust. The engineering executive must be able to trust that the outsourcing firm he/she has partnered with will actually deliver what they promise.

Ed has been building software products for more than 20 years, with particular expertise in automating economic analyses, decision support and supply-chain management processes. He is currently a sales executive with Agilis Solutions and has provided strategic technology leadership in roles such as the vice president of engineering for Egghead.com, director of technology at Nike and director of software engineering at Boeing.

Contact the author.