Richard Bucker

Streaming Operating Systems Suck

Posted at — Jan 28, 2022

From the devops story folder…

streaming operating systems suck

This month I had a number of very challenging production problems. Now that much of it is starting to recover I’m considering the root cause. Unfortunately the root cause is going to be a result of my intuition but after 40 years service I’m ok with that.

In 2021 I deployed 2 web applications(chickens) into my chicken-coop. (Cattle run in containers, Bison run in VMs, and chickens run in jails.) My chickens were running quite well. The stack is golang, a private DSL, sqlite, tcl, bash. One of the projects has been turned into a bison and it’s working well.

Under the covers there is a lot of data and it’s expected to get larger. There is probably a need to federate the data for the customers as well as devops efficiency. So the data is federated by the processing year and any data that’s needed for reporting year over year is either attached or summarized and copied from year to year. Unfortunately most of the data was historical and so I’d never experienced a rollover year. That happened a few weeks ago.

And it only got worse.

I desparately started converting one of the apps from tcl to my dsl and at almost the exact same time the tcl version started working again. Sure most devops houses have a more managed approached to this like holding back updates until something critical happens… but this is just ass backwards. I do not have a team dedicated to monitoring and pruning my production OS. I do not have the budget to buy production OS (if my customers knew what that cost they would never do it.)

So my advice…