In conclusion… while Bosun(B) is still not the ideal monitoring system neither is Prometheus(P).
I am running Bosun in a Docker container hosted on CoreOS. Fleet service/unit files keep it running. However in once case I have experienced at least one severe crash as a result of a disk full condition. That it is implemented as part golang, java and python is an annoyance. The MIT license is about the only good thing.
I am trying to integrate Prometheus into my pipeline but losing steam fast. The Prometheus design seems to desire that you integrate your own cache inside your application and then allow the server to scrape the data, however, if the interval between scrapes is shorter than the longest transient session of your application then you need a gateway. A place to shuttle your data that will be a little more persistent.
(1) storing the data in my application might get me started more quickly
(2) getting the server to pull the data might be more secure
(3) using a push gateway may be a semi persistent way to collect data from short lived apps but it might also act as a proxy for WAN data collection.
UPDATE: I was wrong about one thing. I was reading the godoc and I was left with the general notion that the local handler was going to collect all of the telemetry but I could not locate the actual touch point between the Prometheus configuration and the data being collected in the main part of the application. As it happens the library can instrument the http handler in addition to it’s other metrics.