Introduction
I once worked on a project in 2023 that used embedded devices to capture images of building facades for defect analysis. Our architecture involved BalenaOS, which allows you to run your applications as Docker containers, giving you easy deployability and management. We ran into a situation where we needed to be able to shut down or restart the host when the user pressed a button on the user interface. My first solution was to use C TCP/IP socket programming to solve this problem, simply by having a server application running on the host that our application could connect to and invoke the commands. Long after I left the company, a former colleague figured out that we could use systemd/D-Bus to accomplish this much more cleanly, with fewer moving parts.
In hindsight, I had this idea because that’s all I had in my toolbox, and I didn’t sit on the problem long enough to explore other options. It felt so right at the time.
Many developers’ first solutions come from their lived experiences; we call this intuitive thinking (System 1). It can feel right, but it’s rarely the best. Deliberately exploring solutions gets you better results, and doing this repeatedly trains your intuition so that your first ideas are better next time.
The ideas we will explore come from Thinking, Fast and Slow by Daniel Kahneman and A Philosophy of Software Design by John Ousterhout. These books informed my thoughts on this topic.
Where these ideas originate
Thinking, Fast and Slow was written by Daniel Kahneman, an American psychologist best known for his groundbreaking work on the psychology of judgment and decision-making. His book invites us to think of the brain as operating in two modes: System 1 (intuitive, fast) and System 2 (deliberate, slow).
A Philosophy of Software Design was written by John Ousterhout, an American computer scientist who teaches at Stanford University.
He gives us a sound, grounded perspective on the philosophy of “designing it twice”. To him, designing twice is a virtue; however, he acknowledges that there are psychological forces that stand in the way, and Kahneman’s book gives us the psychological insights (such as System 1 thinking) to see them.
I’m just as guilty of sometimes operating in System 1.
Your first design is System 1’s answer
Kahneman says System 1 “operates automatically and quickly, with little or no effort and no sense of voluntary control”. System 2, on the other hand, “allocates attention to the effortful mental activities that demand it, including complex computations”. In simpler terms, System 1 is what comes to mind first, while System 2 is when you really have to think hard. In my BalenaOS project, C socket programming was what came to mind first to solve the shutdown/restart problem I was presented with, because it was all I knew.
Ball and bat thought experiment
Consider that a ball and a bat cost $1.10. You are then told the bat costs a dollar more than the ball. From there, you are asked to work out the cost of the ball. Work the answer out, and when you are ready, reveal the correct answer below.
So if you got the wrong answer, you’ve just witnessed System 1 in action. But don’t be harsh on yourself: more than half the students at Harvard, MIT and Princeton gave the wrong intuitive answer.
People who quickly default to the wrong intuitive answer are, as Kahneman puts it, “ardent followers of the law of least effort”. According to him, they don’t even bother to check their answer.
Kahneman explains that “System 1 excels at constructing the best possible story that incorporates ideas currently activated, but it does not (cannot) allow for information it does not have”. He goes on to say, “when information is scarce, which is a common occurrence, System 1 operates as a machine for jumping to conclusions”, leading to what he calls WYSIATI, which stands for “what you see is all there is”.
In my BalenaOS project, I worked with all there was in my head. I constructed a story that incorporated ideas I already knew (C socket programming) and convinced myself this is all there is.
Design it twice!
Ousterhout says, “Designing software is hard, so it’s unlikely that your first thoughts about how to structure a module or system will produce the best design”. Today, I admit my first design was naive; I didn’t consider the pros and cons or explore radically different options.
Even smart people are not immune to this mistake. He goes on to say that “[smart people] subconsciously believe that ‘smart people get it right the first time,’ so if they try multiple designs it would mean they are not smart after all”. Ego is a human problem, and we are human, so it’s only logical to think that some of us don’t want to be caught out looking like imposters. Nobody gets it right the first time, not even smart people.
Why polishing your first design isn’t enough
Once you’ve accepted your first design, no amount of polishing will improve things if it’s the wrong one. Instead, you’ll be forcing a design that should have been abandoned a long time ago for a better one.
The reason why polishing does not help is that once we’ve bought into a design that comes intuitively to us, we tend to linger within that same solution space. Kahneman appropriately calls this anchoring. He says it’s explained by an “automatic operation of System 1”. When you’re anchored, you’re less likely to change course and consider different options, as Ousterhout would recommend. He illustrates this by saying, “People adjust less (stay closer to the anchor) when their mental resources are depleted, either because their memory is loaded with digits or because they are slightly drunk”.
We work in an industry where constant context-switching leads to burnout and depletes our mental resources, which makes it harder not to think in the territory of System 1. Many of us can recall hours chasing elusive bugs, only to later find ourselves performing poorly on tasks that demand our attention.
My first design, using C socket programming for a TCP/IP solution, was both intuitive and an anchor. In fact, I was stuck within that same solution space. We noticed that the design introduced another program that had to be compiled and installed on the host, which added complexity.
Instead of re-evaluating whether this was the correct solution, I added automation to our build process to reduce how much a developer would have to do. Nonetheless, despite any improvements, I was still anchored to it.
Ousterhout’s point isn’t to improve on the current design but to consider radically different approaches, like systemd/D-Bus, which later turned out to be one of those. Therefore, a different design breaks the grip of the anchor.
Designing twice trains your intuition
Ousterhout says the design-twice approach not only improves your designs, but it also “improves your design skills. The process of devising and comparing multiple approaches will teach you about the factors that make designs better or worse”. This implies that over time, it makes you better at weeding out bad designs from good ones, with better intuitive (System 1) accuracy.
Kahneman asserts that “intuition cannot be trusted in the absence of stable regularities in the environment”. In some aspects of software development, feedback can be fast, but most design flaws are insidious and only pose a problem when much time has elapsed. Hence, it can be incredibly hard to build intuition when feedback is delayed.
Getting compiler feedback on a missing semicolon is fast, but that’s not what happens when your application now has to deal with unprecedented traffic that it was never designed for. Feedback on whether my design was the correct one came much later, when my colleagues discovered flaws in it on closer inspection.
Evaluating different design approaches can be cheap, and the design phase is where you have a lot of wiggle room to find the right solution. Once implementation is done, changing the design is a lot harder.
Kahneman warns that a “mind that follows WYSIATI will achieve high confidence much too easily by ignoring what it does not know”. The act of comparing options and thinking carefully and deliberately is the antidote to environments where feedback comes much later.
What I’ll do differently
We now understand that System 2’s deliberate and slow thinking is far more effective, but it is mentally demanding. However, AI agents have gotten better over the years, and they would have made it incredibly easy for me to explore different ways I could achieve communication with the host without expending too many mental resources. The ultimate decision of which design I’d choose would still rest with me, and that would involve understanding the context I’m solving the problem for and the constraints we have.
Apart from the systemd/D-Bus method, there are many other options available, such as the Supervisor API, which has existed since 2015. However, I feel like I might have found this option and skipped it because it didn’t feel familiar and had complexities that made me resort to the more intuitive, hacky socket approach I was already aware of. Again, WYSIATI.
System 1 is insidious, and to even claim you would have done otherwise is an example of hindsight bias.
Given my new-found perspective on this topic, I can say I would be more careful with my design approaches when the stakes are high. If I feel like reaching for what I’m already familiar with, that’s the time to abandon it when it serves to obscure a better approach.
Leaning into the philosophy of designing twice and considering radically different approaches is how we hone our intuition and become better problem solvers.