Richard Bucker

painful etcd configuration

Posted at — Mar 20, 2016

I’m not sure what happened but my etcd configuration stopped working. I recently acquired two Intel NUC devices and while they were a pain in the ass to configure the BIOS (I found this link and everything is all better from the boot)

Next it was a matter of selecting the topology. This is normally an easy task but lately I have experienced a few bugs which I cannot explain. The one good news is that I have been able to try and try again.

$ sudo rm -rf /var/lib/etcd2/proxy
$ sudo systemctl stop etcd2
$ sudo coreos-cloudinit –from-file ./cloud-config.yml
$ sudo systemctl start etcd2
$ sudo journalctl -f -u etcd2

And then test the health of the system


$ etcdctl –debug cluster-health 

But for some strange reason my proxy node (worker) seems to lose a healthy connection to the etcd leader.

Mar 21 00:50:43 corenuc2 systemd[1]: Started etcd2.
Mar 21 00:50:43 corenuc2 etcd2[4036]: proxy: listening for client requests on http://0.0.0.0:4001
Mar 21 00:51:13 corenuc2 etcd2[4036]: could not get cluster response from http://0.0.0.0:2380: Get http://0.0.0.0:2380/members: dial tcp 0.0.0.0:2380: connection refused
Mar 21 00:51:13 corenuc2 etcd2[4036]: proxy: could not retrieve cluster information from the given urls
Mar 21 00:51:19 corenuc2 etcd2[4036]: proxy: zero endpoints currently available
Mar 21 00:51:19 corenuc2 etcd2[4036]: proxy: zero endpoints currently available
Mar 21 00:51:26 corenuc2 etcd2[4036]: proxy: zero endpoints currently available
Mar 21 00:51:26 corenuc2 etcd2[4036]: proxy: zero endpoints currently available


My SOLO etcd server:
coreos:
  etcd2:
    name: etcdserver
    initial-cluster: etcdserver=http://192.168.86.113:2380
    listen-client-urls: http://127.0.0.1:2379,http://192.168.86.113:2379,http://127.0.0.1:4001,http://192.168.86.113:4001
    #listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    advertise-client-urls: http://192.168.86.113:2379
    initial-advertise-peer-urls: http://192.168.86.113:2380 

    listen-peer-urls: http://192.168.86.113:2380,http://192.168.86.113:7001


My worker with etcd configured as a proxy
coreos:
  etcd2:
    proxy: on
    name: etcdworker
    initial-cluster: etcdserver=http://192.168.86.113:2380

    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001