Richard Bucker

etcd and fleet not starting with CoreOS running on GCE

Posted at — Apr 4, 2015

Google Compute Engine (GCE) is a cool tool. If you do not know what it is you should check it out. Over the years they have made it more configurable that the limited selections that it started with. At one point you could load your own raw image and install that. Cool!

The best part is that I can deploy and destroy systems from a GUI, CLI, and API.

Now for the hard stuff.

I had a CoreOS installation, following the alpha channel, for almost a year now. In that year I have not had to do ANYTHING. In fact I lost my cloud-config file and was then able to recover it using both the CLI and the GUI.

One day, a few months ago, I lost the ability to connect to one of my subdomains.Between the naked domains on my nameserver and lighttpd acting as a reverse proxy… it just would not work.

What I discovered… (a) boot disk was full (b) no logs © etcd would not start (d) fleet would not start (e) my lighttpd container was deleted; it was a total mess.


The drive was full so I started cleaning up my images and instances. (not a great choice)

/var/log/btmp was HUGE and google results suggested I was under attack. So I deleted that file and it is already growing fast. There are a number of possible solutions from fail2ban and denyhost. I’ll make a choice soon enough.

Ran my docker build and run commands for my lighttpd container

Removed the etcd and fleet sections in my cloud-config because my CoreOS instance was a single and those config options were for multinode.

And then it worked.

NOTE:  Rocket is now installed by default. NICE!

UPDATE: Just a helpful link. More etcd info link and link.