Learning, understanding, and horizontal development

As developers feel the daily pressure to deliver, they tend to skip a crucial step in the process: learning and understanding the system. There’s a huge difference between just adding more lines of code to the codebase and making changes that maintain the conceptual integrity of the system.

Horizontal development is the result of not spending enough time learning and understanding the system you’re working on. By starting to churn out code too soon, you’ll inevitably be adding silos of functionality in parallel to existing functionality.

The outcome is legacy code. It doesn’t matter how well factored it is, and what test coverage it has!

I was discussing software development with a colleague today, and we arrived at an interesting conclusion that fits well with my previous experience. We concluded that software development is an activity that requires constant learning.

There is the obvious kind learning: a new language every year (pragmatic programmers), the web framework of the month, the application server of the project, and so on. This type of learning is implicit in our profession (but not always acknowledged by those who pay our salaries), and it’s what makes it interesting.

But, there’s another kind of learning, or let’s call it understanding. It’s the kind of understanding you need to gain to actually comprehend what you’re doing; the kind of understanding needed to maintain the conceptual integrity of a system.

Vague? Let’s look at some examples.

What happens when you tell a junior, fresh out of college, software developer to add or change a certain feature in a non-trivial system? Assuming that the developer is competent, but not experienced, you’ll see him happily churning out code after a few hours of browsing through the code and asking some questions.

Or what if you are to introduce framework X for your web application? How do you go about it? Most people will download and install the framework, google “framework X tutorial”, read the first chapters, and aim at learning the rest as they go along.

Or the most common situation: You’re a contractor fresh on a project and you want to prove that you’re worth the money. Isn’t banging on the keyboard as soon as you get your login credentials a good proof?

What is the outcome of these scenarios, and what lessons do we learn? Let’s start with the second scenario. While it would be very good to actually go through the entire tutorial for framework X and read a 400 page book on top of that, no customer or team would accept it.

Imagine if you said: “Ok guys. I need to actually learn this framework and all its features, so that I know how to use it properly. See you in a week”. Since this isn’t really an option, you go on googling tutorials. An average developer will get away with this. After all, we’re not talking rocket science, but one thing is inevitable. If that developer’s code is reviewed or maintained by someone who actually knows how to use the framework, all its features and idioms, the verdict will not be in favor of the stressed developer, who followed the tutorial.

There’s collateral damage as well. If nobody on the team has a deeper understanding of the framework the team is using, a negative attitude may sweep across the team; everybody will think that the framework is hard to work with and doesn’t actually solve any problem.

I don’t want to start defending old technologies with known short-comings, but honestly, how many developers, who actually understood what a Java enterprise bean is and how to use it, do you know? How many web applications have you seen where JSPs, servlets, filters and tag files are used in accordance with Sun’s recommendations and guidelines? These are old technologies, but Spring suffers from this as well when in hands of teams that have learned the 10% needed to get started.

Of course, I’m guilty of this too. There exist many tests based EasyMock with my name in the author tag from 2007 that do far too much verifying, mocking instead of stubbing, and are brittle and over-specified in general.

In summation: Learning just the bare minimum to get you started will produce sub-optimal results that will cost money in the long run due to complicated maintenance. Is this what people call technical debt?

Also keep in mind that use of frameworks tends to stagnate at a certain level. Idioms will be reused and solutions copied. It’s not a linear growth where the team learns 10% during the first month, 20% during the second, and 100% after a year. Expect a utilization of 40-50%… at best.

So, not developing a thorough understanding of a framework will produce sub-optimal results and give an experienced reviewer or maintainer some WTF moments. What of the other two scenarios? They are more dangerous!
Regardless of whether you are a junior developer or a seasoned contractor doing your fiftieth project, starting banging on the keyboard too soon is almost always a bad idea. Why? Because of…

• Architecture. Every system has one. It may be good or bad, but it’s there. Architecture is usually documented or apparent, but it takes a moment to get it.
• Patterns and idioms. Hopefully, similar problems are solved in similar ways, but these ways will most likely not be documented and need to be dug out. In more complicated cases there may be competing patterns and idioms in a system, and we need to find the right ones.
• Metaphors and language. Workflows and processes in the system use a language and some metaphors. Do they come from the business? Do we understand them?

In my opinion, these factors comprise a systems conceptual integrity. And they do take time to grasp!
Apart from gaining an understanding of how not to violate the conceptual integrity of a system, a developer must figure out some mechanical stuff as well, such as what classes to refactor, what to add, and what to delete (help!).

This takes time. Hours, maybe even days, during which the developer browses the source code without producing a single line of code. My experience is that in the majority of cases, gaining a deep understanding of the effect of your changes will produce a clean, idiomatic solution that will keep the system conceptually consistent.

The opposite of learning and understanding works like this: You feel pressure to produce and you don’t want to break something that works. What do you do? Horizontal development to the rescue! Create another silo of functionality, introduce it in parallel, duplicate some code, and ta-da! New feature implemented without touching the old system! This kind of development, be its outcome covered with tests or not, results in immediate legacy code! It’s horizontal because of the new silos and duplication.

Doesn’t an extensive test suite protect us from horizontal development? It should, but tests don’t verify the conceptual integrity of the system, only its behavior.

So what is my point? My point is that we should acknowledge the difficulties in extending and modifying software and spend some more time figuring out how to introduce the changes and extensions. Learn and understand. It pays off!