EVENT: I have a single Rackspace instance that I’m using for a client of mine. The machine was intended to host multiple projects and clients but I never tried very hard and so my client is only paying me about 50% of the VPS cost alone.
LESSON: margins should never be less than 50-100% of cost.
EVENT: That same machine is in a zone that Rackspace has designated end of life. At some point they are going to turn it off or migrate it for me.
LESSON: projects or customers that do not generate enough raw revenue need better margins so there is incentive to maintain them. Regardless of the actual code complexity there is a basic devops cost per application and machine and user… etc.
EVENT: Moving bulk VMs from vendor to vendor or datacenter to datacenter is not without risk. I had 3 VMs in Google’s compute engine. While the service is nice and pretty reasonable it’s better than having to drive into a COLO in order to reboot a machine. The Risk is that these VMs are too easy to delete. I moved my development environment from GCE to an in-house server. Now that things are working nicely, in-house, it’s time to decommission the GCE device. I deleted the wrong machine. The machine names I selected were DEVBOX, PCORE1, PCORE3…. I deleted PCORE3 when I had meant to delete DEVBOX.
LESSON1: completely separate prod from dev. Name the machines better. find a way to put a token inside the filesystem of the target machine that can act like a LOCK such that a delete script might have to see the token before deleting the environment. I’m not saying what happens when you have 100’s of machines; just a few.
LESSON2: use a script to delete the VM, but the script should take a snapshot of the environment first. This way you can be assured that target is, in fact, the right one.
NOTE: containers might be the way to go, however, while this might protect the VM it’s not going to protect the container and especially the data volume container.