Death by design patterns, or On the cognitive load of the abstractions in the source code

After many years coding in different teams — big and small — I think I can now finally explain to myself why and when do we need, or not, “clever” code built with patterns and abstractions.

I have started my career in a rather big Java product (10k+ classes) and internalized (much too) well various design patterns: from all the clever abstractions to inversion of control and stuff. Later on, I worked on a bunch of smaller Python, Clojure and other projects and the common mantra in the teams was that you don’t need complex design patterns in small projects, but you do after some threshold. No one defined the threshold, though… With some experience I gained a good intuition when I can write clear code with or without abstractions, but throughout my career I always wanted to define a better criterion that I could share with others: what is exactly “small”, when exactly do we need to start hiding things behind the abstractions and making things generic? I have built a bunch of heuristics around it, but the answer eluded me.

At some point I worked on a Python product written by a person with some good business knowledge, but essentially not a trained developer. They handled our team a bunch of cron-scheduled Python scripts that somehow implemented a complex data processing pipeline and our job was to maintain those. I remember my horror of looking at the code which had zero abstractions and essentially written as a bunch of linear scripts with copy-pasted bits of logic all over the place. Of course, our team has decided to bring some order to that. We have rigorously created abstractions for individual business logic workflows, implementations of those abstractions, factories to instantiate them, and more, and even more on top of that. Fast-forward a couple of months, I wasn’t working on a product for some time and then came back on the team. I looked at the code and I came to realization: the older version better. I had no idea what new code did. Instead of linear logic I had to jump between interfaces, their implementations and a bunch of other abstractions to gather together a complete understanding of the overall implementation. The voice of an experienced developer in my head said: “chosen poorly abstractions are here”. But another one annoyingly replied: “what does it mean to be a poorly chosen abstraction?”.

The life moved on, different teams and different products, but that experience was still itching.

Recently I have been reading about various approaches to the team organization (“Team Topologies” in particular) and how teams can be organized to reduce the cognitive load. This made me think of cognitive load more in general.

Back to the coding practices, I have already realized long ago that the code and overall design need to be simple: there is enough inherent complexity in any domain by itself, while “clever” code, more often than not, only makes things worse. But importantly I have found the “why”. It always feels like “duh…” when you finally realize it, but yes — it is the cognitive load.

It is a bit more complex, though. Turns out, you can use the cognitive load argument both ways: one could argue that you need abstractions to hide the complexity, as well as another argument is to avoid the abstractions and keep things simple.

Here is my latest heuristic: a “cognitive load estimation”. The question to answer is which version of the code creates the least cognitive load for the future reader. Again, “duh…”, this joins the decades-known “the code must be easy to read” idea, but I guess everyone needs to find their own path to it. So, before introducing abstractions to a new project I now try to decide if future readers would be able to confidently make changes without jumping all over the source code tree, with or without any complex design, and choose the path that better supports the idea.

What do you think?

P.S. By the way, back to the original question: you might need good code design, but you never need “clever” code.