Despite it how sounds like, this is not a ranting piece. This is an examination of why despite ongoing effort of keeping up with practices and tools, software 'engineering' hasn't gotten easier.
It's a feeling that's been bugging me for awhile but can't put a finger on. I'm like a skillful sailor on a boat lost in the sea without a good map to navigate with. Everywhere I go looks the same, there's no destination because the point is to keep sailing.
If "software is eating the world" then intuitively developing software mustn't be that hard. "Look, just use this open source package, it's easy." Yes it's really just plumbing.
Yeah, see... there are things that they teach how to do, and they are things you're just expected to know how to do that are almost not teachable.
Let's talk about the comfortable kind first: the teachable things. Basically anything you can google with that's distillable into a page of tutorial is not difficult. "Create a blog in 10 minutes", those are true marvels. At the higher end we have books that spell out exactly what's going on with exquisite details. The pure act of coding itself isn't difficult. Do this and you get that result everytime, it's the stuff of pleasure.
Software design, from architecture to UX to games, is outside of scope here. Design difficult in their own way, is legitimately part of the development process but largely an activity that has limited reflexivity exposure with the development process. I know there's agile but designers don't make decision based on the requirement of keeping code clean. Let's see if you agree by the end of this.
With that out of the way, I have some idea of why making software gets difficult.
Anytime you do any serious coding, you are playing at least one of these three games: puzzle-solving, yak-shaving, and feedback loop.
Everytime you try to change behaviour in a system you need to know how things work within. This attempt is an act of puzzle-solving.
The larger the code base, the tougher the puzzle. The entire discipline of software engineering is trying to make this puzzle-solving easier. New programming languages are invented to end up with easy puzzles, allegedly. Functional, OOP, you-name-it all try to help us make sense of code better so the inevitable puzzles (bugs) are more bearable.
Yak-shaving is what makes up heist movie. In order to pull a heist you need access to the bank vault. To that you need a crew of diverse talent. To hire them you need to fly around the world to recruit. To that do you need to raise capital. And so on.
To construct your software you need a set of tools that need another set of tools to function. The dependency chain continue to cascade up. Some call this layers of abstractions, it's largely a good thing software but it's easy to take it too far. In different context it's called bureaucracy.
Yak-shaving is even more intense in an eatablished code base. Any prerequisite that doesn't yet exist needs to be done in house. The more layer of abstractions there are, the more work needs to go in to make sure they stay 'clean'.
Shaving a serious yak can be an act of faith. By choosing a solution you are counting on the requirements to that solution being feasible. You may not know what those are until you are ready to cross that bridge. Pray that it's not something you dread.
Lastly we have the quality of feedback loop. This isn't by itself the game but it's the predominant meta-game.
During the development process, slow feedback loop begets slower feedback loop. A rapid feedback loop means knowing you've made mistakes instantly. The faster the loop the more you stay in the flow zone.
A quick feedback loop is a result of good tooling. You can't have that when compilation takes 20 minutes.
These things never come free. Prices attached to them include automated tests, CI/CD, logging/monitoring, quality error reports, etc.
That's the three games being played. I think that's it but I can't be sure it's entirely comprehensive. When building mental maps I choose to be minimally complex than maximaly accurate. Feel free to point out loop holes here.
Moving forward. Notice the venture of creating a new app from a blank canvas isn't quite playing those three games. It doesn't involve taking systems apart. Sure there may be aspects of yak-shaving (trying to set Java path in Windows) but nowhere approaching a crisis of faith.
Now the pieces are in place but it's not yet the answer. These three games are in and of themselves quite solvable. Software tools advance all the time towards improving at least one of them.
But it doesn't matter. Software is still difficult to make and run because of this: software goes through ages.
Here's an easy analogy. A six year-old human is not twice a three year-old baby. A 12 y/o isn't twice a 6 y/o. They are fundamentally different. Similar for corporations. A mid-sized company runs completely different from a giant conglomerate.
Not only do they operate differently, have different capabilities, they need to be treated differently and given different expectations.
A running software goes through phases similarly. Development in each phase come with different amount of yak-shaving, feedback loop and puzzles.
I don't yet know what those phases are and what are the expected challenge-levels for each game. Like child-raising I expect different phases (characterized by size of code base and surface area of feature set) to require completely different combo of each game rather than progressively more intense uniformingly.
Software development is made difficult when all phases are treated the same.
In the leadership realm, there's an understanding of peace-time vs war-time leaders. Each age of software development require different mind games on the part of developers.
Nobody comes with every mindset it takes to tackle every phase. Yet domain knowledge are often tragically stuck in the same people. Having to transmitting these knowledge become a reluctant and poorly done side quests.
So they end up working a 12 y/o software the same way they treat a 2 y/o. But what got them here won't get them there.
Alright, so when do we do what? That is the map we lack. We don't even know where to draw the lines between age groups, let alone how to act with them.
As a software grow, the developers may or may not believe in it over time. Similar to a parent or a founder, you are not obligated to like the result of your offspring but you do have to adapt to keep it alive.