Richard Bucker

Updating My Chicken Coop

Posted at — Sep 1, 2022

I like having a single piece of massive hardware whether it’s a Corvette or a 256 core Ampere arm processor. But I also like going to the grocery store without requiring launch codes. In the last few years I’ve been down the “cattle vs pets” rabbit hole only to realize that cattle are as important to farmers as pets are to most people. So for me, “chickens” were a better metaphor and so I breed chickens in the coop.

About 7 years ago I built an orchestration system that competed with an internal development project. They failed twice with many 100s of developers, several calendar years, at a cost of $35M and I succeeded with 1 developer and 2 months. I’m not an expert I just knew how to GSD. (plenty of room for improvement). In recent years my chickens have been running on a single Intel NUC Skull and later a Skull plus– hosting VMware with my guest OS installation.

Now here we are in 2022 and VMware has been bought and the new master wants to increase the prices. Both Chicken coops and barns are dirty places so things like dust in the vents and so on will cause overheating and component failure. And in my second worst nightmare my vmware host refuses to let me login. I thought I mistyped the password and so I tried enough times that I was locked out permanently. This happened once before and I had to reinstall everything so I’m not happy about this… There are some settings for retry count, retry delay, and permanent lockout. I thought I had set those to something reasonable, however, a patch must have reverted it.

argh…

Over the last few months I have been relocating my chickens into the cloud and other slightly smaller NUCs. I have one project left in the VMware barn and I need to relocate it. I have a disaster recovery instance running elsewere but I’m not ready to relocate it yet. At this exact moment I have 4 new “GMKtec NucBoc5” which I installed OpenBSD 7.1.

OpenBSD is all love/hate

Linux is getting bigger abd bigger for no appreciable reason. I do not need all that crazy hardware support just to feed a few chickens. Even many thousands. The idea is to keep the chickens lean.

My current desktop is 2x 34in monitors on a Pop_OS installation. I have a backup machine in case the barn crashes but it’s a pain to keep them in sync. It’s the single reason that I moved to ChromeOS devices in the first place.

Chicken or the Egg

(consider softraid for encrypting the partition; and autoconf with tftp server)

I’m not sure. But these are the steps:

I recommend deploying a TFTP server with an OpenBSD autoconf file. In this case start with just the root account and then build out the user account from scratch… unless you want to start by allowing root-permit-ssh and then the admin accounts are created the proper way and you do not have to su with a password. There is some nuance here.

By default OpenBSD seems to install these packages:

$ pkg_info
intel-firmware-20220207v0 microcode update binaries for Intel CPUs
inteldrm-firmware-20211027 firmware binary images for inteldrm(4) driver
vmm-firmware-1.14.0p0 firmware binary images for vmm(4) driver

so installing my list is pretty simple to repeat:

I’ve omitted some because they are deps and I’ve added the ‘–’ because the versions might change. Some of the packages are part of my projects.

$ su -
<root password>
# pkg_add bash-- bzip2-- curl-- cvsps-- desktop-file-utils-- figlet-- freedt-- gettext-runtime-- gettext-tools-- git-- glib2-- gmake-- htop-- lz4-- lzo2-- ncdu-- neofetch-- nghttp2-- openvpn-- rlwrap-- rsync-- sqlite3-- tree-- wget-- xz-- pftop-- iftop-- sudo--gettext- vim--no_x11 coreutils

make common folders

The tmp folder in the user’s HOME is kinda uncertain. The side effect of /tmp is that it is volitile and that’s good because it will be recovered where HOME/tmp is not really.

mkdir -p ~/{src,tmp,bin}

login .profile

Depending on whether we are using bash or the default shell who really knows where to put this stuff. In this case I like .profile but for no particular reason than it works and does not collide with anything else. Just append this to the .profile. Some of the lines refer to the tools in the DSL So it’s more of a DEV than UAT/PRD update.

Logging into a remote system and setting up the environment for interactive and non-interactive sessions can be a bit challenging if you do not know the details. While inserting the following into the .profile works in a manner parts should actually be in a .kshrc or .bashrc.

export LD_LIBRARY_PATH=/home/rbucker/lib
export C_INCLUDE_PATH=/home/rbucker/include

pathadd() {
    if [ -d "$1" ] && [[ ":$PATH:" != *":$1:"* ]]; then
        PATH="${PATH:+"$PATH:"}$1"
    fi
}

pathadd $HOME/go/bin

if [ -f $HOME/.kshrc -a -r $HOME/.kshrc ]; then
	ENV=$HOME/.kshrc		# set ENV if there is an rc file
	export ENV
fi

#
#
# .kshrc or .bashrc could begin here
#
#

echo "++++++++++++++++++++++++++++++++"
sqlite3 --version
fossil version
go version
git version
alias tclsh="rlwrap -c tclsh"
alias tmux="TERM=xterm-256color tmux"
echo "++++++++++++++++++++++++++++++++"

nf=`which neofetch`
if [[ "$nf" != "" ]]; then
        neofetch
fi

parse_git_branch() {
	x=$(fossil branch current 2>/dev/null || (git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1/'))
	print -- ${x:+\(}${x}${x:+\)}
}
export _chroot=$(cat /etc/_chroot 2>/dev/null)

export PS1="\[\e]0;\u@\h: \w\a\]\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[01;31m\]"'$(parse_git_branch)'"\[\033[00m\]\$"

export HISTFILE="${HOME}/.ksh_history"
export HISTSIZE=5000

dot files

.tmux.conf

set -g default-terminal "screen-256color"
set -g mouse on
bind-key          S choose-window "join-pane -v -s "%%""
bind-key          V choose-window "join-pane -h -s "%%""

.vimrc

set t_ti=
set backspace=indent,eol,start
set mouse=a
set ttyfast
if has("mouse_sgr")
    set ttymouse=sgr
else
    set ttymouse=xterm
end
colorscheme murphy
syntax on
set background=dark

map <MiddleMouse> <Nop>
imap <MiddleMouse> <Nop>
map <2-MiddleMouse> <Nop>
imap <2-MiddleMouse> <Nop>
map <3-MiddleMouse> <Nop>
imap <3-MiddleMouse> <Nop>
map <4-MiddleMouse> <Nop>
imap <4-MiddleMouse> <Nop>

or copy them

scp .gitconfig host:.
scp .vimrc host:.
scp .tmux.conf host:.

vim color themes

This primer is starting to bother me. One thing I like about What Lucas has said about *BSD is that everything is included. It’s what makes *BSD a complete solution. All these tweaks are just a distraction.

git clone https://github.com/gosukiwi/vim-atom-dark /home/rbucker/.vim/pack/colors/start/atom 
git clone https://github.com/NLKNguyen/papercolor-theme.git ~/.vim/pack/colors/start/papercolor-theme

ssh-copy-id

I have no idea why OpenBSD left this off. I suppose it’s a bood idea to make copying your kets difficult…

Here is one variation: https://github.com/soko1/OpenBSD/blob/master/scripts/ssh-copy-id

Here is mine. I suppose I should try ksh instead. Happiness is both work.

#!/usr/bin/env ksh
#!/usr/bin/env bash

while getopts i: flag; do
	case "${flag}" in 
	i) idfile=${OPTARG};;
	esac
done
shift $((OPTIND -1))

if [ -z $idfile ]; then
	echo "missing -i identity file (or authorized keys)"
	exit
else
	if [[ $idfile = *.pub ]]; then
		echo "have a pub file"
	else
		if [[ $idfile = */authorized_keys ]]; then
			echo "have a keys file"
		else
			idfile="${idfile}.pub"
		fi
	fi
fi

if [ ! -e $idfile ]; then
	echo "id or key file ${idfile} does not exist"
	exit
fi

for arg do 
	echo '--> '"\`$arg'"  ; 
	cat ${idfile} | ssh -T $arg 'read KEY; umask 0077; mkdir -p ~/.ssh; grep -q -F "$KEY" ~/.ssh/authorized_keys 2>/dev/null || echo "$KEY" >> ~/.ssh/authorized_keys' ; 
done

echo DONE

doas and sudo

# cp /etc/examples/doas.conf /etc/doas.conf 
(edit the doas.conf and add 'nopass' to the only line)

doas visudo
(un-comment the wheel-nopass)

And repeat on all the servers.

(doas and sudo are effectively the same for muscle memory)

DNA Manipulation

Also, in recent days, I’ve watched some geeks talk about “command line tools for uber geeks” etc… and sure there is plenty of eye candy there but in my day to day job I work on many different flavors of OS… Clear Linux, CentOS, Ubuntu, OpenBSD, FreeBSD, ChromeOS-linux so developming “muscle memory” for the fancy tools even .profile tweaks is not plausable. In one UAT (user acceptance testing) system I get a fresh home directory every time I login and no ability to load tools which, of course, increases the attacker’s surface area.

For example OpenBSD is moving to doas where most are still using sudo.

daemons

use mkservice as installed by freedt.

use chroot when you can or should.

use pf on every chicken. It’s effectively the skin that keeps the bugs out.

Basic email forward-relay

This kinda sucks… my chickens are behind my ISPs residential gateway/firewall which means I cannot connect to port 25 outside my local network. I happen to have a synology with the capability to forward emails. And so it was a fire and forget. Almost.

todo:

Depending on the MX records and the setup of the relay this can be tricky and take time to test. My last email took 10 minutes once it left the lab.

Disaster recovery and high availability

As I recall the Axioms of DR and HA…

simply put: reliable systems lower operational costs and increase profits.

That includes software; network; NAS and SAN.

Mounting a shared drive for simple things

… like backups

# create the shared volume and export mount point (I used a synology)
# create the local mount point and add permissions
doas mkdir -p /mnt/drive
doas chmod a+rwx /mnt/drive
doas mount -t nfs host.domain.local:/volume1/drivedrive /mnt/drive

doas mkdir -p /mnt/fossil
doas sh -c "echo 'host.domain.local:/volume1/drive /mnt/drive nfs rw,nodev,nosuid 0 0' >>/etc/fstab"

You can move the mount to the /etc/fstab file so that it can automount. You can also configure it to prevent the mount from creating devices or exec’ing files from the mount. (low level sneaky stuff)

Enable Threading

Something happened some time back and the OpenBSD team decided to disable Intel CPU threads. The idea was that there is a CPU bug that one process could see data in another (read the CVE)… I thought this was limited to Intel. This morning I realized my AMD Ryzen 7 was partially disabled upon first boot. Now I do not completely understand why this switch is off by default when in most cases people to not timeshare their computers any more. But ok maybe there is a human nature thing going on. What’s worse is I have not heard Linux or FreeBSD doing the same. OK, maybe over cautious. But when was AMD lumped in? And why didn’t my GMKtec have the same symptoms.

Mitigated CVE-2019-1125, a potential exploit utilizing a variant of the Spectre side-channel vulnerability in Intel CPUs built since 2012.

From the command line

sysctl hw.smt=1

Or place hw.smt=1 in /etc/sysctl.conf, so it stays on across reboots.

Share your success

I used the info from neofetch in the subject to tell OpenBSD what’s working. I wish they could parse the results to determine whether the hardware was good… like a list of laptops.

(dmesg; sysctl hw.sensors) | \
   mail -s "Intel Corporation NUC6i7KYB, works OK" dmesg@openbsd.org

Conclusion

Now you have a baby chicken. If you’re building web apps I recommend using carp, relayd, httpd, letsencrypt, acme, building a simple DSL of you own that might look like TCL and PHP had a baby. Keep it simple and try not to implement an exec function. Use SQLite for federated storage. When implementing the DSL keep in mind that you should audit imported packages. (see nodejs)

UPDATE I had to rush out the door afew minutes ago to get the kids from the bus stop. At the same time Pop_OS warned me that an update was ready for deploy. Without a moments hesitation I pressed UPDATE and scooted off. The update might not auto rebot but then you never know. By contrast OpenBSD only patches bugs. The packages might get updated but unless you completely fail your precautions it might not be necessary to update on demand except for syspatch. It’s certainly a more controlled coop where you might not replace the bedding until you really need to or it’s safe. Whereas CoreOS, CentOS stream, Pop_OS, and a few others… make that decision for you.