Given that an unexamined life is not worth living, a unexamined crisis is not worth experiencing.

There was a crisis at work 1.5 weeks ago I hadn't had to experience for quite some years. A new feature was rolled out to production; things were broken on multiple fronts I had to work overtime to put out the fire. It's now largely under control. Such things was not unprecendented but at the user scale of where we are, each move was all the more impactful than years ago.

This is a premature one-man post-mortem to process the shell shock.

Some engineering context: what used to be was that the separation of logic and data were clean. With this new feature data is now made stateful, logic has to run according the newly designed states within the data.

Concretely speaking, a leave-application object now is responsible for telling us who has the authority to operate on it.

Because meta-data come in such varied forms, their manifestation is largely unpredictable, the resultng behavior becomes all the more unpredictable.

The more behavior is data-driven, the more indeterministic it becomes. That wasn't something I anticipated, which I should have from similar projects in the past.

Assumptions about which object has what data is dangerous. Assumptions are tolerated when everything is controlled from code but they are now downright hazardous when data calls the shots.

Had I foreseen that, I would've had code logic produce plenty of default data rather than null/empty. Since that was not easy to do during emergency, I was forced to overcome that with special logics, with great reluctance.

We have to accept that large scale data manipulation is no longer a solution going forward (even if it used to be). We've unknowingly crossed into big data zone.

I'm glad the damage was contained and nobody got hurt (that I know of). If I've had no sense of responsibility, such crisis would've made a great vehicle for thrill-seeking adventure.