Justin Vincent

Why It's a Bad Idea to Warranty Custom Code (Bugs vs Mistakes)

A few months ago I had an interesting discussion with a client about “Bugs vs Mistakes”. During the discussion my client  said  that he didn’t like the idea that he was first paying a developer to ‘make a mistake’ and then second paying the developer to ‘fix the mistake’.

He didn’t mind so much about ‘Bugs’. ‘Mistakes’ were the issue.

My Client’s Classification of a Bug:

On one page we had a search box, but rather than use std HTML we wanted to make it look nice by adding a background graphic with CSS. If you typed more than 30 characters the text overshot the end of the background image that would be considered a bug (an oversight).

My Client’s Classification of a Mistake:

The site had a section where you could store credit card information and a section where you could store bank information. A mistake would be if the drop down in the bank account section had the text “Credit Card” when it should  read “Bank Account” (a copy paste error).

During the discussion he made the interesting assertion that software developers ‘mistakes’ should be warranted and fixed free of charge. That’s an interesting concept and worthy of discussion. In this article I hope to show that neither the customer, nor developer can benefit from a “Warranted Code” code agreement when developing 100% hand coded custom software.

ALL Bugs Are Mistakes

It’s difficult for a non programmer to appreciate just how many mistakes it’s possible to make when coding. It’s even difficult for a new programmer to appreciate it. It’s only after you’ve been coding for a few years that you can truly grasp the magnitude of what can go wrong.

It’s not the same as, say, the number of mistakes you might make when writing an essay. After all once you have finished the essay it stays the same and the words don’t change…

But dynmic code is a different story. Each line consists of variables that rely on other variables that rely on functions that are chained together. One logic chain may have a depth of hundreds of library functions that rely on other library functions and so on. An infinite number of mistakes and bugs just waiting to happen.

If a developer were to try to think of ALL the possible things that could go wrong during their daily coding sessions – their output would probably slow down by a huge order of magnitude.

It’s not unusual for custom projects to have more than 10,000 lines of custom code, all interacting with each other. There’s an old saying “Only one company in the world makes bug free code. It’s NASA. It costs them a $1000 a line”. I bet that most startup’s would not be willing to pay $10m for creating 10,000 lines of custom code 😉

So, as a result, coders have no choice but to choose a comparatively small amount of tests to check any new code is “good enough” before handing it off.

The typical tests a web developer performs after building web functionality might be:

• Does a record get inserted in the database when I click the “Save” button?
• Does the error popup if I don’t enter a value? Correct error message?
• Is the inserted record correctly associated with the logged in user?
• etc.

In other words the developer is testing the ‘meat and potatoes’ of the application logic. As you can imagine when the developer has their head stuck in this level of ‘logic complexity’ it’s vey easy for them to miss something aesthetic such as… does the drop-down say ‘Credit Card’ or ‘Bank Account’.

Yet, from the clients perspective it seems astounding that the developer could pay so little attention to the front end and miss the obvious mistake staring them in the face – however from the developers perspective the front end is only the very tip of the iceberg.

For example, a web page on the front end might consist of two drop-downs and a button – but the backend interactions can involve thousands of lines of code with infinite error possibilities – and it is these that a good developer must focus on. After all, the wrong text on the front end is a 2 second fix, but badly tested code on the backend can bring the entire site down.

Don’t Eat Your Own Dogfood

In the world of software development it is widely acknowledged that coders should NEVER test their own output. This is for good reason. A dog can’t smell its own breath. It’s impossible for a developer to guess how other people might interact with the product they’ve made. There are other good reasons discussed here. If a developer was to FULLY test their own work it would decrease their output dramatically.

I know this because after the discussion with my client I tested the theory. I painstakingly tested all my new code to the Nth degree before releasing it. I was slowed down by about 80% due to the massive amount of permutations I needed to test for (rather than the core tests). My output became very low and my cost to ship each functional unit went up by a factor of 5. I also became bored and un-inspired with the work.

In my opinion, the very idea of warranted code is bad for the customer. It would force ANY sensible developer to test way more permutations than actually required – ‘on the clock’ – during billed time.

Yes the customer may end up with “better code” but at a huge price increase in cost and productivity.

A better idea is for the programmer to output work at their usual rate with a sensible amount of tests (i.e. database tests etc.) – but not test every possible outcome. Then as the project gets closer to launch, and bugs are picked up, they are can be cycled back to the developer who polishes them at that point.

With this method you only pay for “bugs that really matter” – vs – paying for “all possible code permutations” that a developer might think of.

You could go a step further and say that the appropriate time to get picky about how the software works and what it looks like is in the Q&A cycle (just before launch). Large software companies have an entire Q&A team that does nothing other than test software and report bugs to developers. Smaller software houses may not have dedicated teams, instead they usually have a dedicated Q&A cycle, in which everyone on the team focuses on testing the software and then cycles back bugs for developers to fix (once again, just before launch).

Development Cycle

A good testing/development cycle might look like:

1. Brainstorm features
2. Wireframe front-end
3. Developers code new custom features
4. Business owners check new custom features (loop back to step 3 as needed)
5. Final Q&A
6. Developers fix issues (loop back to 5 as needed)
7. Launch
8. Fix bugs that customers find

It’s important to note that at stage 4 we are not looking for 100% perfection. We should be looking for minimum viable product acceptable to the business owners. Then Q&A stage 5 will throw up plenty of finer detail bugs missed at stage 4 (and rightly so).

You will also note that at step 8 even more bugs are found by customers. This is why entrepreneurs should take a chill pill when it comes to perfectionist tendencies. In 20 years of software development I’ve never seen software that was bug free.

Conclusion

By using Q&A teams and Q&A cycles correctly entrepreneurs empower their developers to output code as quickly as possible without having to get bogged down in the minutiae of testing all the possible permutations as they go along. Even though it may be painful to see an imperfect product emerge, this is most likely a cheaper and faster way to develop software.

The likely alternative is to slow down developers by 80% and grind productivity to a near stand-still. It also risks stifling developers creative output, which can kill the spark and make a developers job less satisfying. Which in turn might give the developer itchy feet.

Update

There are some exceptions to this rule. For example if your software house has created a framework that you use as a cookie cutter system for all of your clients and you only create iphone apps  – then I can imagine you could warranty your work without too much of an issue (because presumably you would have tested your cookie cutter framework to the Nth degree).

The above article is targeted at developers creating one off custom code for a new start-up or business.

If you found this article interesting check out the weekly tech podcast I co-host (TechZing) which you can subscribe to on itunes by clicking this link or listen to here. You can also follow me on twitter @justinvincent