Richard Bucker

Docker Gotchas

Posted at — Jun 4, 2015

I’ve watched as the memes have gravitated to Docker like bees to nectar or flies to poop. I guess that’s a half full half empty thing. As I’ve said many times in the past you gotta know your stack. And the more I use Docker and the more they advance the project the more I realize that it’s just too early for adoption… unless you have a person or three on the inside.By example my first CoreOS+docker server is continuously running out of disk space in part because btmp is filling up (which I’ve read is probably an attack). My second CoreOS+Docker server shows an explosion of files in the Docker overlay folder. I’m not certain why I need so many folders and if any of them are zombies or not.Another article made mention of my filesystem iNodes. While my filesystem is at %5 used and iNodes at 15%; which is not an unhealthy ratio, in my opinion, but since this is a single Docker container running on a 200GB HDD it suggests that getting any sort of density is going to be unpredictable and painful.I may need to investigate NixOS containers.UPDATE:  found this article echoing my problem.UPDATE: I made some discoveries that are not actually documented anywhere.  The first truth that I realized is that proper VMs are probably still a better proposition than containers of any kind. CoreOS toolbox and it’s analogs make sense but not for actually running a production service. This is yet another reason why the Phusion containers are a bad choice.Next, you simply do not need intermediate layers. The first reason is that many times the builder falsely uses an old container. For example I have a RUN go get some_go_package and docker does not rebuild it. SO I have to but my go get commands after my ADD commands in order to insure they are executed.Now that I’m deleting all of my images, intermediate layers, I’m getting ready to rebuild my containers with the –rm and –force-rm options. I do not want any intermediate files. The issue is that docker build is consuming too much disk space and any hope of reasonable density is impossible.delete all of your contianersdocker rm $(docker ps -a -q)delete all of your imagesdocker rmi $(docker images -q)delete eveythingdocker rmi $(docker images -q)** this last one takes a VERY long time. My 331 overlays has been deleting for nearly 30 minutes and white it was only 2.5GB it made up 15% of the inodes meaning there were a lot of files.UPDATE:  One more thing to keep in mind as my system is still deleting those intermediate layers… GCE (google compute engine) limits the iops for my host VM and so bulk deletes like this are going to take a while regardless whether it’s an SSD or HDD.UPDATE: In hindsight this sort of cleanup (a) would have been faster to rebuild the host (b) if there had been more than one container on this system I would never been able to clean it up because there are no tags. The also means that the build and deploy system should probably be different machines.UPDATE: compressing a container [link] (devbox and devboxrun are my image and container names)ID=$(docker run -d devbox /bin/bash)docker export $ID | docker import - devboxrunUPDATE: backup compressed snapshotsID=$(docker run -d image-name /bin/bash)(docker export $ID | gzip -c > image.tgz)gzip -dc image.tgz | docker import - flat-image-name UPDATE: I’m testing my last clean build configuration (bash functions)function cleanbox {        docker rm $(docker ps -q -a –filter=image=builddevbox:latest)        docker rmi builddevbox        docker rmi golang:cross}function buildbox {        docker build –force-rm –rm -t builddevbox .        ID=$(docker run -d builddevbox /bin/bash)        docker export $ID | docker import - devbox        cleanbox}