Richard Bucker

Struggling with rkt, flannel, etcd

Posted at — Jul 19, 2016

I’ll say it again, I want to make Docker go away from my stack for all the reasons that the CoreOS team talks about. Granted I cannot eliminate it all but I should be able to get rid of a lot. And here are my challenges:etcd2Make sure that etcd is listening on 0.0.0.0. I wish this were not the case because it means that rogue apps could communicate with etcd directly just by connecting to the network and while not a terrible thing it does require more network security instead of secure by default.I had several problems with my cloud_config, aka user_data, and frankly I did not want to reinstall my IntelNuc as it is unpleasant to install.I manually updated: /var/lib/coreos-install/user_dataI also edited my local cloud_config.ymland I bootstrapped from my local cloud_config.yml with this:sudo systemctl stop etcd2sudo systemctl stop fleetsleep 2sudo rm -rf /var/lib/etcd2/proxy/clustersudo rm -rf /var/lib/etcd2/proxysudo coreos-cloudinit –from-file ./cloud-config.yml fleetI’m twisted on this because one CoreOS webpage says that fleet is included and another says it’s not. There are also countless docker examples and no fleet examples.Building fleet with rkt on the CoreOS host was yet another hardship but I managed to fix it.$ etcdctl set /coreos.com/network/config ‘{ “Network”: “172.16.0.0/16” }'and for the moment I’m running flannel in the foreground.sudo ./artifacts/flanneldI have read some docs that show how to configure fleet from the cloud_config file but it’s limited information and does not actually start flanneld. It’s merely configuration.I found the missing link. Starting flannel from the systemctl looks likeLsudo systemctl start flanneldLooking at the flanneld.service file I was disappointed to see that flannel used docker and that it was really very complicated to launch:/usr/lib64/systemd/system/flanneld.servicerktrkt lacks daemon mode and the ability to reattach to a daemon. These are mechanisms found in docker and were helpful when creating the unit files. rkt requires sidekicks.One trick is getting the networks correct. It’s going to be a particular challenge when I start clustering the machines.SkyDNSThe rkt-skydns project does a good job of tracking the pod’s IP address. The project was missing some basic support for SkyDNS but the developer made the corrections very quickly. Now I can start my brb, aka helloworld, with a simple command line and is easily converted to a fleet unit:ExecStart=/usr/bin/rkt run –net=default –insecure-options=image </span>        /home/rbucker/bin/brb-0.0.1-linux-amd64.aci </span>        /home/rbucker/bin/rkt-sidekick-v0.0.2-linux-amd64.aci </span>        – –cidr 172.16.0.0/16 -f ‘{“host”:"$ip", “port”:3000}’ /skydns/local/ncx/brbOne bug remains, multiple networks, which one is installed in DNS?the bottom lineThis is hard and the documentation make it harder still. The various teams may or may not make changes to the code or projects and that means toiling through this same exercise again and I do not look forward to it. I am likely at the point in time where I must learn everything I can about kubernetes and leave the details, like these, to someone else.I’m really surprised that someone does not have a project that makes this a no brainer. Something very opinionated and simply works.UPDATE: notice that I added some comments about launching flanneld… but since we are talking about the flannel subnet with no host ingress it feels like kubernetes is still the right answer as it has a mechanism for port forwarding.