Richard Bucker

Curated golang libs and things

Posted at — Oct 30, 2014

I need a goto place where I can track or at least link to my favorite libraries. Since golang is now my complete preference here is my list:

Good article on self referential functions. helpful article for http/REST. List of go tools. And here you can setup a project.

Instructions for setting up for go cross compiling.

gopass - console password prompt

gorename - the article, the source (the source comes from Google and attached to the go tools repo)

gogs - self hosted git service written in Go.

go-bindata - Useful for embedding binary data into a go program. go-bindata-assertfs - Serve embedded files (http handler type funcs)

msgpack seems to be a good and efficient message container. Of course there is JSON and many different implementations with different performance profiles. What is nice about MP is it’s support for so many different languages which could provide significant interop.

MQ - gnatsd or here - go implementation of a nats server (client libs are in adjacent projects). This project has a broker. Beanstalkd - is also a strong pub/sub broker although redis can also be used in this capacity. And of course ZeroMQ offers brokered and brokerless topologies. (I’m not a fan of RabbitMQ so no link is given). libchann is also pretty close to a MQ. go-nanomsg and mangos. messagePack code generator. MessagePack home. Kite seems to offer a combination of RPC and netchan/libchan features. (libchan is stalled; netchan is deprecated; new-netchan is in limbo) Que-go is a work queue based on the ruby analog. There is more than one way to do this but it’s worth a read.

CLI - there are some new 3rd party libs for processing the command line.  From what I’ve seen they must be deeply rooted in the psyche of the author. The stdlib does a good job and does not need replacement. Command lines do not need to be complicated and they should not be.

Logging - Lumberjack for it’s rolling logs. But remember that Docker like having the logs expressed through stdout. While I’m not a fan of aggregated log files this(log-shuttle) logger might be useful.

Database - this category is tough. It depends on the type of DB [SQL, NoSQL, other] that you might be deploying and then there are a multitude of wrappers to choose from. The latest that caught my eye was a mongo-like document API that used Postgres 9.4 as the backend. In this case if the underpinning or translated structure is “least surprising” then you might get the best of both worlds in one container. (Golang database drivers)

There are a number of ORM libraries that I have used and reused. My favorite is gorp. Since I only use the ORM methods that support proper marshaling I have been looking into other libraries. Ones that are smaller, easier to use, more thoughtful. This article was helpful. SQLX seems like a good option. One important feature is Server support. Many of the ORMs require specifying the dialect and than can be a challenge.

I prefer to write the proper optimized SQL than letting then letting the ORM layer make that decision for me. Early use if “hibernate” which required hand customization via the HQL syntax meant learning two SQL variations in order to code and debug.
The Hood project has gone to great lengths to incorporate the GoLang tags as complete metadata for the DDL. While it’s interesting it’s a lot of work. It means that teams will have to develop a meaningful style guide in order to use Hood in it’s opinionated way. I think the Go Authors did not really want that to happen as evidenced by gofmt and it’s motivation(s). And frankly if you need a DDL then implement a DDL rather than manually preprocessing the DDL by embedding DDL meta info in the structure tags. (see go generate)

The model project does not get me to the happy place either; even though it’s author is the one that forced me to shift more directly toward the cleaner SQL marshaling (SQLX) rather than the mega-orm.

Authentication

Dependencies - godep, glide and then there is gopkg.in for API versioning.

Load testing with vegeta or boom, written in go, look interesting because they offer a CLI as well as libs that can be integrated into your application. (yes; there is apache bench [ab], loadrunner, seige, and so many others)

Crypto - crypt storing config data in etc (Kelsey Hightower wrote this; I have not been able to find an authoritative link) What makes this interesting is that every application is sharing the same storage but that only the owner (holder of the keys) can retrieve the data. It would be interesting to combine crypt with go-bindata so that the keys might be auto-baked into the application as part of the pipeline. Additionally, one needs to publish the public keys in the etcd engine itself so that others might publish data for your applications consumption across the cluster. There is some interesting ssh APIs too.

I have been following YubiCo for a long time. They have a number of interesting projects and while I like their HSM is does have a few limitations by itself; that being TPS. So there are only two use-cases that make sense. (a) systems with low transaction rates, or (b) use the HSM to decrypt the persisted, encrypted, private keys and leave the transcoding of the cipher/plain-text to a proxy. This increases the TPS but creates a trust issue. This Yubi-KeyServer is worth watching with the only concern being that it’s authorship is outside the US.

Instrumentation - “instrumentation through composition” (article), lumbermill, instruments, metrics, go-metrics, Metrics (video) Monitoring from StackExchange (article, home, code)

Web - vulcand looks like a nice reverse proxy and/or HA with interesting ambassador features for docker

IDE - Wide(web based IDE), lightIDE, Atom, Brackets, and so many others.

As a service on Windows

DNS - GeoDNS. A very lightweight DNS server. SkyDNS2.

a go version of gitreceive so that you can trigger hooks.

templates - I really like the golang std library for templating but there is something to be said for mustache if cross platform or shared templates is your thing.

Package templates - requires go 1.4

Go package updater via browser instead of terminal.

libvirt

link to good concurrency slide deck and video

One of the Go Authors has been arguing against frameworks. I do not recall which one or how strongly but then there are some frameworks that me. These might not really be frameworks but templates…. but who knows where the line blurs. This year John Graham-Cumming presented his dotgo project. I cannot agree with him more. His project reads from stdin and write to stdout. The processing takes place in a goroutine so the results may be unordered but could be highly efficient.

Here is my Docker post with a few links to some docker related projects.

Jobber as a replacement for cron

Mailgun is a great toolkit for sending emails. I tried sendgrid but was not able to get it going and mailgun simply worked.