Nov 22, 2010

Sometimes Rewriting is the Only Option

Some time ago, I had the dubious fortune of having to work with a huge amount of legacy C# code. Mind you, the system was no more than a few years, yet it consisted of some one million lines of code. It was written by some outsourcing company, who (I later learned) got paid per line(!). Yes, that is true.

I did some statistics on the code base and found that 90% of the code lines were duplicates, copy-pasted from other parts of the system. In other words, only 10% were pure, pristine code. Not only that, but the code itself was a complete mess with no internal structure of any kind. Almost every method was static. Even the constants were defined in several places, with different values sometimes. The largest C# file consisted of twenty thousands lines, and on top of that, the system contained thousands of SQL stored procedures with an average of a thousand lines of SQL code each. The database structure held thousands of de-normalized tables and hundreds of "project" databases, each with identical schema. I'm not kidding.

It was the definition of big-ball-of-mud.

Needless to say, the system was extremely fragile, buggy and incredible slow. It could only handle a couple of simultaneous web users. The slightest change required editing maybe 20 files. It had already amassed some three thousand (known) bugs, although that was only the tip of the iceberg. It was a never ending minefield littered with barbed wires, failed attempts, wrong turns and skulking, raving mad, overworked programmers on the edge of giving up.

I have seen the system evil programmers have to maintain when they die, and it's not pretty...

I did my best explaining about technical debt and the importance of maintainable code. In my view, the system would eventually consume the entire company due to it being so costly to maintain. I argued with management about this and claimed we should rewrite rather then refactor. My standpoint was that it would be easier to re-create 10% of the code from scratch rather then spending the rest of our lives trying to eradicate 90%. It was no use. My pleas fell on deaf ears.

So we tried working around it using wrappers, strangling application, refactoring and adapters. The problem was that as soon as we tried to integrate with it, we failed because were not one single point of interaction, but thousands. We did pretty well transforming the company to something akin to agile thinking, but it was in vain since the system itself continued to fight us on every change.

I know there are lots of smart people who are arguing that it is less costly to user modern refactoring techniques rather than trying a big bang rewrite. I'm not saying these people are wrong, but there may come a time when one must accept the fact that some systems costs more to change than to rewrite. It's either that or desperately holding on until the entire business is lost in spiral of missed deadlines and lost customers.

On the other hand, I have been part of a very successful rewrite. But it did not happen until the company were on the very brink of giving up completely. Management simply had no other option.
Lets try a rewrite, we don't have much more to lose anyway and if things continue like this we are out of business anyway.
Any maybe that's what it takes. It's a bit sad it has to go that far, but there are some nice upsides to a rewrite, and in this specific case, the rewriting effort turned out to be a huge success for this company!

So rewriting might be a catalyst for something very good. You have a chance to rethink the systems positioning, a chance to add new features and new approaches which can help you target entirely new markets and customers. For instance, to change focus from feature-complete to ease-of-use or something like that.

After all, it's not the system that defines the company, it's the people and the sum of their experiences. Out of this mess something new and powerful might emerge, but it takes lots of courage and commitment to see it through.

10 comments:

  1. I couldn't agree more. Anyways first step to refactoring is to write tests and when you have tests it's relatively easy to rewrite the application when cleaning would take too much time, especially when existing code is tightly coupled spaghetti.

    ReplyDelete
  2. I have never understood the huge bias against rewriting. In your case, it seems a perfectly clear option. I think the bias comes from underestimating the cost of figuring out the requirements to get the system where it is now, and overestimating the cost of rewriting it. It seen as a complete "do over" when in reality is is far less than that.

    In your case, you have a system that meets the requirements, but is non-functional on the back end. It takes time to get there, and none of that work has to be re-done if you rewrite. You just make the same system again with a better back end.

    I have personally rewritten my own systems on a number of occasions and they always come out much better -- cleaner design, more efficient, more elegant, more scalable, etc., because I am able to leverage all that I learned the first time. You will never get that clean break with refactoring.

    ReplyDelete
  3. Very interesting post, to me this seems like more of a project management screw up than technical (to start with), signing a contract where you pay per line just encourages poor reuse! Do you know much about the conditions around signing off on this project? I do wonder if this whole mess could have been avoided with more knowledgeable stack holders who drove the initial project...

    ReplyDelete
  4. It is all fine and dandy until the rewrite does not work the same as the original. Then you are SOL.

    ReplyDelete
  5. @Anthony: No, not any details, except everything was outsourced completely to a bunch of programmers who did everything paragraph-by-paragraph from the spec. Always adding more code... copy paste, never refactoring. They didn't even use arrays in for() loops, they unrolled all loops and added switch cases instead!

    But you are certainly right, the code quality should have been reviewed regularly instead of... well, never. :)

    ReplyDelete
  6. I was in a similar position with an embedded system and a home-grown "real-time" OS that was cooperative (i.e. not preemptive) with the stupidest message-passing algorithm that computer science has ever seen. The workaround to that became to directly call entry points in other processes (euphemistically called "quick message"). The embedded system had been in integration test for a year and showed no signs of slowing down in its ability to generate bugs. I recommended replacing the OS with our standard (also home-grown but sane) real time OS and showed how this could be done fairly quickly (2-3 months). Didn't manage to convince *anyone*. Another year later the whole project got canned. Not saying that following my recommendation would have save it, but it was a mess just plain beyond repair.

    ReplyDelete
  7. Huh... "paid per line",
    "unrolled all loops and added switch cases instead"

    I think that the buyer can sue them for robbery....

    ReplyDelete
  8. Can you please let us know the consulting company so that we never would hire them for anything? Also did you fire the manager who drew up the contract to pay them per line of code? Seriously, who does that these days. I run a consulting firm and I have never encountered anybody who wanted to pay me by the number of lines of code that we wrote.

    ReplyDelete
  9. Yeah i had a lot of legacy code in C# and other technologies. My only option was rewrite each part of code.

    ReplyDelete

What are your comments? I'd love to hear 'em, so don't be shy now :-)  If you want to get in touch with me directly just drop me a mail.