Richard Bucker

vmware, docker-machine, docker swarm, traefik and gitlab

Posted at — Aug 7, 2018

I’ve been experimenting with my configuration for a while and I have not achieved 100% coverage. Taking lessons learned in this space including configuration as code and lights out DEVOPS and idempotent deploy I’m moving to the next step. One funny thing to note is that I spent 45 minutes looking for my code and notes in every git wiki I had only to recall it was a post and not in the code. RATTS!Part 1 - HardwareThe system I am running on is a 2017 Intel Skull NUC with 32GB ram and 1TB disk. In 2018 Intel upgraded the hardware and I want to upgrade but for the moment this is what I have. Currently live I’m running a Google WiFi router but that will be replaced with a Ubiquity EdgeRouter-X. The Ubiquity has a unified command center and that makes me happy.Part 2 - VMwareThere is something to be said for running a container OS on bare metal but then there is even more for running VMware on bare metal. Except for some of the overhead one can still allocate 100% of the system resources to a single active VM and you can still have alternates and backups. Also, you can take snapshots and so on. Lastly, with the right license you can do a lot of automation with APIs and in fact docker-machine is a handy tool. And you can mount a VMware filesystem.Part 3 - RancherOSThere are a number of operating systems that you can choose to operate. I really like CoreOS, however, now that it’s been acquired by Red Hat I just do not know the status of the product. Also it’s costs are still too high. One thing that CoreOS does, which I thought was a value, is the auto updates. Sure it’s only down for 9 seconds and the company is very reliable, the problem is that your system might be in the middle of some critical task or maybe a database write and that quick update causes dataloss or perceived malfunction from the customer perspective. RancherOS is no more or less capable. Since the main OS is actually running in side it’s own container almost everything can be updated with no downtime. It’s also easy to deploy with scripts and VMware. My only issue is that it requires 4GB to boot and get any sort of meaningful work done.Beside the docker swarm machines you are going to need a console machine. Since I like immutable and reproducible installations. The console is also going to be RancherOS. I also tend to use Alpine Linux in some cases and inside containers.Still looking into Docker Moby, Fedora Atomic, Alpine Linux. But I’m staying away from anything that might be too configurable. The console machine is meant to aid in setup and recovery.Part 4 - DockerInstalled by default. Might need to install docker-compose and docker-machine. RancherOS has some read-only partitions so you have to take care where you put stuff.Part 5 - Docker/VMware volumesI need to store or persist my services in my persistent storage. There are several challenges in this configuration. [1] cannot share RW volume with multiple containers on multiple swarm nodes. [2] cannot snapshot these volumes [3] there are essentially hosted volumes meaning that cannot be access on other hardware nodes.It’s undefined right now but it might be possible to put a network filesystem in front of the VMware storage but I’m not there yet.In this config I’m planning to deploy a traefik server and a gitlab instance.docker volume create –driver vsphere traefikdocker volume create –driver vsphere gitlabVerify the volumesdocker volume lsPart 6 - DNS and nameserverThis is a mixed bag here… register your domain, set your nameserver, get whatever DDNS credentials you need and then continue with the traefik setup.Part 7 - default traefik configTraefik configuration is pretty plain. The basics are here and here. Once you know what you’re basic configuration is going to look like then mount an alpine container and create the files.docker run –rm -it -v traefik:/traefik alpine /bin/shHead over to the traefik folder and create the 3 require files. Set the chmod correctly and then paste the file fragments.cd /traefiktouch acme.json docker-compose.yml traefik.tomlchmod 0600 acme.jsonAnd create the traefik networkdocker network create –scope swarm –attachable –driver overlay traefikPart 8 - Launch traefikdocker stack deploy -c docker-compose.yml traefika lot more to come….