Premature Flexibilization Is The Root of Whatever Evil Is Left

Posted by Dan Milstein on Fri, Jul 03, 2009 @ 10:24 AM

digg it |

reddit |

delicious |

Facebook |

Twitter |

Okay, so you know how they teach you, in Computer Science school, that lovely quote: "premature optimization is the root of all evil"? (variously attributed to Donald Knuth, or Tony Hoare, or other CS luminaries). Well, in my more recent experience, there's another programmer habit which is in hot competition for the evil-rooting title. I don't think it has as catchy a name, so I'm going to propose one here: Premature Flexibilization.

And here's what I mean: the practice of adding complexity to your code to make it more "flexible", in anticipation of future change.

Why would this be so bad? I hear you saying. I mean, we all know that software requirements change, so isn't it good to make your code future-proof, by making it flexible, able to handle those changes that are inevitably going to come? Most of the classic expositions of the glories of Object-Oriented programming are rife with this -- look, we're using abstract classes, so now the subclasses can vary, thus our code will be flexible and reusable, isn't that great?

Here's why this fails so badly in real life: it's not just that requirements change, it's that they almost always change in ways which the initial programmers didn't expect. E.g. you've got your nice abstract base classes, all set to add more variants, but it turns out you just need the one variant... but you need to add a whole bunch of features to it. Or you have to make it thread-safe. Or connect it via RPC to Javascript code running in a browser. Or auto-generate it via a script. But, because you've made it flexible in one direction (adding more variants), you've also made it harder to vary in other directions (features, thread-safety, connecting-to-a-browser-ness, auto-generability). This is a near iron-clad rule: flexibility comes with attendant complexity, and complexity makes things harder to change in unexpected ways.

Or, to put it another way: if you make something as absolutely damn simple as possible, then extending it along any axis is at least doable. But that's the only way.

Take a careful look at Design Patterns, that classic text of OO design. Notice how so many of the patterns are about making your software flexible, about making it adaptable to change. Then notice that all their lovely real-world examples are for frameworks which no one uses anymore. They've been replaced by other "flexible" frameworks (actually, they've usually been replaced by several generations of "flexible" frameworks, each of which turned out to be the wrong kind of flexible)

In contrast, consider the C standard library. It wasn't designed to be flexible, it was designed to be as simple as possible. And, lo, these 35 years later, it's still being used. Sure, it's been wrapped by other layers, improved this way and that, but it's still a live thing.

Also, note that "simple" is a very subtle idea. It doesn't necessarily mean "crude". Sometimes, the simplest way to do something is to, e.g. write a Domain-Specific Language--which is in no way trivial. But, and here's the key distinction, it's a solution to the problems you have right now. It's not a solution to problems you think are coming later. If you can train yourself into the deep habit of only solving today's problems, you'll be far better prepared for tomorrow than if you tried to look ahead.

What do you think? Agree? Disagree? Any examples that loom large in your experience?

Comments

Other issues related to "premature flexibilization": not only do you lose the time you spend making it more flexible, you also lose time documenting the more complex usage, and other programmers (or a months-later-you) lose time getting up to speed on the more complex usage.

Posted @ Friday, July 03, 2009 12:25 PM by thorny_sun

I think you're taking your examples to the extreme. Yes sometimes the requirements that we can get are a different animal, something that your design approach didn't anticipated at all, but in my experience those change are not the rule but the exception (again i`m talking in the context of the type of changes you expressed in your post).

Posted @ Friday, July 03, 2009 2:23 PM by Fernando

I think of it as speculative programming, I think it's a waste of time. Not because it makes the code base more complex, but because it just means effort and hours are going into work that is certainly not presently useful and will in all likelihood never be used.

It is pretty glorious when the speculative programming you did 2 years ago comes into play and gloriously enables a new features with the flick of a switch. Just like it's also glorious to win at roulette.

Posted @ Friday, July 03, 2009 2:49 PM by Julius Davies

I think you're oversimplifying.

First: there are certain rules of flexibility that generally result in easier evolution. For example, minimizing global variables is generally just a good habit to get into. We do it so unconsciously (now!) that we don't even notice how often it bought us some crucial flexibility. Even if you only need "one" of something, you might as well build as if you may one day want two. It isn't much additional work (since it is second nature) and it often pays off. Global variables may be superficially "simpler" than parameters, but that is seldom the case in the long run.

Second: in general it is more noticeable when flexibility goes unused (because we mourn the waste) than when it goes used (because things just worked out "as they should").

Third: flexibility is a bet that you'll need to do X rather than Y. It's oversimple to say: "you always bet wrong." You do not. Maybe you often bet wrong but not always. Or maybe you usually bet right because you've been down this path before. It depends a lot on the situation.

So I have no simple rule of thumb on this issue. Sometimes it's a no-brainer like "avoid global variables" and other times it's a complex bet about an uncertain future.

Posted @ Friday, July 03, 2009 3:14 PM by Paul

Very true.

This is expressed in the Python community as YAGNI: Ya Ain't Gonna Need It.

Posted @ Friday, July 03, 2009 3:16 PM by Chris Ryland

Interesting post. Somewhere in between there is this happy medium between "boxing yourself into a corner" vs. ultimate flexibility.

The problem you state above is not just restricted to software code. It is a well known issue on the requirements side as well.

As a product manager, I have always loathed the concept of options or configurations. When done right, it has its place (iGoogle, Google reader etc. are good examples). But I have always been against adding options with the justification that users may want to do something. To me such statements basically is the first symptom of the product manager saying that he does not know what the user actually wants or that the product manager not wanting to make a hard decision of saying that the requested feature is not for everyone.

I used to be called the "option" cop in my previous jobs. When I was reviewing or sitting through discussions on future stuff, I questioned every user option we wanted to add.

The analogy I always use (you can call it imperfect or say that it does not apply to software) is Bose Wave Radio. I am sure people who knew how to use graphic equalizers on their music systems hated it when Bose hid all of these options. But the vast majority including me who had no idea what to do with all those dials were quite satisfied that we did not have anything to do to listen to good music.

Simplicity you state is hard for people to comprehend, especially software developers (sorry) but to me "less always tends to be more" for all the reasons you have stated.

Posted @ Friday, July 03, 2009 3:30 PM by Gopal Shenoy

I noticed this, too, a year or two back. One place I worked, the boss's top priority was flexibility. As much as possible should be coded in up front, so he wouldn't have to put it in later. It led to exactly the sorts of problems you list here: you can't change the requirements without changing code anyway, so we spent too much time thrashing to come up with a flexible design that wasn't flexible in some key manner.

Most of the flexibility went unused, and time-to-market suffered, often to the point of the whole project being canceled. The flexibility that did get used seemed to be more derived from "good" coding practices like small functions acting on local data than on whether something-or-other was a Factory or not.

Posted @ Friday, July 03, 2009 4:18 PM by sapphirecat

Wow, no way dude that is WAY cool!

RT
www.anonymize.tk

Posted @ Friday, July 03, 2009 4:50 PM by Johnny MAck

Putting in *the right* level of flexibility is the art of programming.

Posted @ Friday, July 03, 2009 5:13 PM by Antonin Hildebrand

My answer to this problem is the same as to premature optimization. To avoid premature anything make it mature.
If you know your project road map - you know what new features to expect. If you have already added features A1, A2 and A3 it is reasonable to expect A4, so it makes sense to be flexible enough to add feature An easy.

Posted @ Friday, July 03, 2009 5:54 PM by Macloud

Hey Cool post!
I happen to agree with you in regards to premature flexibility. Though I feel it is better expressed
as "You’re NOT gonna need it!".
However I am not with you on the design patterns comments. I have found GoF design patterns to be most helpful in writing more efficient and ultimately more simple code, identifying the appropriate pattern to match your intentions is the key and seems to me to be more about elegant efficient design than flexibility.
Another technique I believe helps to "keep code to only what you need today" is test/behaviour driven development. Creating a test or specifying behaviour 1st assists in staying focused on what you need now.
Oh and implementing a DSL may be tough work in C but its a snap in Ruby.
See here

Posted @ Friday, July 03, 2009 6:12 PM by impurist

I call this extensibilitis.

http://www.google.ca/search?hl=en&safe=off&client=firefox-a&rls=com.ubuntu:en-US:unofficial&hs=CfX&ei=npBOStOjOomnlAergqWUDQ&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=%22*+is+an+extensible+framework+for+*%22&spell=1

Posted @ Friday, July 03, 2009 6:15 PM by Adrian Thurston

I’m glad you brought this point up. On the project I’ve been working on there has been a voice nagging me to plan for all kinds of potential changes and I’ve been having to fight it off. When I worked at Netscape/AOL and SimplyHired, we had to deal with very large amounts of traffic, which required all kinds of interesting scaling methodologies, had engineering teams of 5-10 people, and did have some insight into where our products were going to be going. In those situations I think speculative flexibilization (nice word btw) was well worth the extra time and complexity that it cost.

These days I’m a lone programmer working for a small company (novuslifesolutions.com) that maybe in 5 years will have half the traffic that simplyhired.com has now ;-) In this case I write mostly procedural code with objects only where it makes sense and build pretty much just what we need right now. In working with my growing code base I’ve found it very simple and direct to work with. It doesn’t have layers and layers that I have to know and wade through. It is also seems easier to make somewhat fundamental changes to because there’s not endless layers of dependencies. I’m sure that as we grow, have to scale broadly, and hire other engineers that this will all change, but for now I like it :-)

Posted @ Friday, July 03, 2009 6:21 PM by Daniel Billotte

Totally agree. :-)

I actually wrote about this when I wrote my principles of software:

Don't optimize for generality: don't make your procedures more general than strictly needed, in order to prepare for an eventual future when this generality may be needed. Also, if some particular requirement in your software changed and some of the generality in a given procedure or module is not required any longer, simplify the code by removing the generalizations. If they are needed in the future, they can be restored from the older versions.

I included this inside my Premature optimization anti-principle (here the optimization is in the sense of making the code flexible). I think I will actually split this into a separate section and probably put a link to this post. :-)

Posted @ Friday, July 03, 2009 6:45 PM by Alejo

Forgot to include the link to my principles of software article (in case someone reading this also wants to check it out: as I said, the section about premature optimization is related): http://azul.freaks-unidos.net/principles-of-software.

Posted @ Friday, July 03, 2009 6:47 PM by Alejo

I've been programming for 20+ years now, and I agree 100% with this article. Say what you want "in theory" -- IN PRACTICE, based on actual, real-world observation, it is clear to me that it is, in fact, impossible to anticipate future changes in a piece of code, and that "planning ahead," while it sometimes works a little bit for the short term, generally turns around and bites you you-know-where in the long term.

Posted @ Friday, July 03, 2009 9:19 PM by Chris Chiesa

Great insights! I agree completely. My most horrifying experiences with premature generalization, however, have been with database design. Where my user base wants a program to create a series of single rows of data directly linked to a parent row, the db designers put in a massive hierarchy of dozens of tables that split up that single data-row into a highly complex series of dozens of related tables, because of the potential for future business changes, and because it "more correctly models the real world". Every call to this item requires a nighmare forest of sql. Sadly, every screen that views this data displays it as a single row, and all the reports connected to the system output the data as a single row. The system has been in place for years and the extra features have never been used, but every developer writing code against it suffers needlessly. It's intellectual masturbation. Yes, I work in a government shop.

Posted @ Friday, July 03, 2009 9:36 PM by Corey

The name I've seen for this in the past is speculative generality, which I probably heard for the first time around 10 years ago. It's definitely a source of plenty of "evil" in the same sense as for premature optimization.

Posted @ Friday, July 03, 2009 9:41 PM by Dan Tull

"Flexibilization" ? What's wrong with the word "abstraction"? When we write flexible code, we do it through abstractions: additional function parameters, additional types, additional paths through the code, and so forth. Sometimes it makes sense to do because you are certain or near certain that you will need the functionality eventually and it is easier to do as you write the code then at a later date.

Usually, however, premature abstraction is evil. It makes your code more difficult to reason about and adds places for bugs to hide. It can be particularly confusing to someone who is maintaining your code and sees this extra abstraction that isn't (apparently) used; it can greatly hinder there ability to modify the code correctly with confidence.

Posted @ Friday, July 03, 2009 10:23 PM by Nick

Our best practice is to only create an Interface when you add your second implementation. Otherwise we might use a class prefix of the word "Base" to indicate this is the first implementation. For Example: BaseImageManager is a good class name to use until you create an RpcImageManager and at that time you might refactor the code and use a common Interface.

Posted @ Friday, July 03, 2009 10:48 PM by Christopher Burkey

I like the concept. "Flexibility" is really another form of optimization - you're going to optimize your productivity later by doing extra work (design and implementation) now. As you point out, it's almost always a bad investment.

Posted @ Saturday, July 04, 2009 1:56 AM by Peter

@Nick

Abstraction is not the same as flexibilization. Abstraction can for example be that all the features _you currently need_ can be seen as a generalized problem, that can be solved with a single algorithm instead of 10 different ones. While this may or may not make the code more flexible for future changes, what you win is less code and reduced complexity now, and also code that is conseptually easier to understand. I would not call that flexibilization.

Flexibilization, as presented, is that you add extra complexity in order to meet future requrements. That is nearly the opposite of the above kind of abstraction. Most code that present itself as a framework is likely to suffer from the decease of premature flexibilization.

Posted @ Saturday, July 04, 2009 4:16 AM by GS

I couldn't agree more. I have a progammer who couldn't find the shortest distance between two points if it were drawn for him.

Provided a simple interface he will make a very "robust" implementation with scores of useless features that have no mapping to requirements, no examples, no javadoc and no tests. Simply by forcing him to have 100% code coverage on unit tests cuts down on the madness a bit.

Posted @ Saturday, July 04, 2009 9:25 AM by txherper

I think that as you gain experience -- ie you've made many mistakes already -- you become more adept at detecting when something needs to be 'flexibilized' (in the parlance of our times), despite the fact that it is not immediately necessary.

Another thing to consider is that if you abide by the DRY principle, it is sometimes necessary to add a layer of flexibilization to remove redundancy.

Posted @ Saturday, July 04, 2009 3:52 PM by bruno c.

I agree completely. I think the Java environments (JEE), frameworks and application servers are almost unusable due to their 'flexibility' - there is 200 ways to do one simple thing.

Wanting to B64 a binary stream recently I found there were about 13 different b64 en/decoders in the environment, all with their own gotchas.

Flexible? Yeah! Like COBOL is flexible - an enormous obese immovable elephant that has stuffed everything that the zoo visitors have thrown at it down its throat for years.

As the author said: better like the C standard library - a simple clear set of primitives that you can *combine* flexibly.

And also: *IN*flexibility is good.

The walls in this office are *not* flexible and that is what makes them nice solid, stable usable walls.

Flexible things fall over.

Posted @ Monday, July 06, 2009 7:40 AM by Joe Mellon

Another good example of f***ed flexibility is the Java Date().

What you actually want to use (date.GetYear() etc) is deprecated: someone who sits on a commitee obviously found such functions too useful, and shoved the whole Calendar thing in (there is a choice of them too of course) as a sort of job creation program.

Posted @ Monday, July 06, 2009 8:37 AM by Joe Mellon

Good post, I agree with you.
I wrote a blog entry that is related to this: http://www.bestbrains.dk/Blog/2007/11/04/LetsMakeItGeneric.aspx

The bottom line is that genericity is one dimension amond several, and you should make the cost/benefit analysis before moving on that just as well as you should on, say, the features dimension.

Posted @ Friday, July 10, 2009 3:17 AM by Kristian Dupont

I've noticed the "art" side of proper flexibility thing here, too. A large number of programmers completely fail to properly generalize code, and instead add in "flexibility" that is completely pointless. (And yes, Java Date() is a wonderful illustration of this.)

Another example of extreme-flexibility run amok - I've seen this twice now with software I've taken over. Business-process app backed by an RMDBS evolves for a few years, coder gets annoyed making table changes for the nth time, and writes a generic bucket-of-bits KVP implementation to pull random data out of a huge unstructured table.

That makes me insane.

Posted @ Wednesday, August 05, 2009 10:59 AM by fishbane

@Peter: "Flexibility" is really another form of optimization

I think that this is the key. As with premature optimization, premature "flexibilization" may do a lot of harm. But sometimes you really need to think about performance early, an example of this is when you're choosing the right algorithm. Same thing with flexibility. Sometimes it is better to think about it earlier then later.

But in general you're right, premature flexibilization is evil too.

Posted @ Thursday, November 05, 2009 8:57 AM by vsg

I am currently reading about the MVVM pattern for use with Microsoft WPF. See link.

http://msdn.microsoft.com/en-us/magazine/dd419663.aspx

This describes a design pattern for separating the user-interface from the data model. Although the paper continually describes this as simple, it seems very complex to me.

I mention this because it is an upfront design decision that will affect the entire development process. If the pattern is truly good, then it should be beneficial. However, my gut reaction to it is that it way too abstract and complicated.

I am curious to know if anybody has experience the MVVM pattern and WPF. Is this useful or does it correspond with premature flexibilization?

Posted @ Tuesday, February 23, 2010 9:18 PM by JeffB

Posted @ Tuesday, February 23, 2010 9:20 PM by JeffB

Post Comment
Name *
Email *
Website (optional)
Comment * Allowed tags: <a> link, <b> bold, <i> italics
Receive email when someone replies.