Captures Prometheus AlertManager alerts and writes them in a MySQL or Postgres database for future examination.
Because given a noisy enough alerting environment, offline querying capabilities of triggered alerts is extremely valuable.
- You stand up one of these however you like (multi-arch Docker images provided)
- You setup AlertManager to point at it and propagate your alerts in.
- Every alert that gets triggered reaches your database.
- Profit.
graph TD
A[alertmanager] -->|POST|B(AlertSnitch)
B --> |Save|C(MySQL/PG Database)
C -.-|Graph|G[Grafana]
C -.-|Query|D[MySQL/PG Client]
style B fill:#f9f,stroke:#333,stroke-width:1px
style C fill:#00A0A0,stroke:#333,stroke-width:1px
style D fill:#00C000
style G fill:#00C000
Simply install to your $GOPATH using your GO tools
$ go get gitlab.com/yakshaving.art/alertsnitch`To run AlertSnitch requires a MySQL or Postgres database to write to.
The database must be initialized with AlertSnitch model.
AlertSnitch will not become online until the model is up to date with the expected one. Bootstrapping scripts are provided in the [scripts][./script.d] folder.
For specifics about how to set up the MySQL DSN refer to Go MySQL client driver
This is a sample of a DSN that would connect to the local host over a Unix socket
export ALERTSNITCH_BACKEND="mysql"
export ALERTSNITCH_DSN="${MYSQL_USER}:${MYSQL_PASSWORD}@/${MYSQL_DATABASE}"export ALERTSNITCH_BACKEND="postgres"
export ALERTSNITCH_DSN="sslmode=disable user=${PGUSER} password=${PGPASSWORD} host=${PGHOST} database=${PGDATABASE}"Run using docker in this very registry, for ex.
$ docker run --rm \
-p 9567:9567 \
-e ALERTSNITCH_DSN \
-e ALERTSNITCH_BACKEND \
registry.gitlab.com/yakshaving.art/alertsnitch- Open a terminal and run the following
- Copy the AlertSnitch binary from your $GOPATH to
/usr/local/binwithsudo cp ~/go/bin/alertsnitch /usr/local/bin - Now run AlertSnitch as with just
alertsnitch- To just see the alerts that are being received, use the null backend with
ALERTSNITCH_BACKEND=null
- To just see the alerts that are being received, use the null backend with
Once AlertSnitch is up and running, configure the Prometheus Alert Manager to
forward every alert to it on the /webhooks path.
---
receivers:
- name: alertsnitch
webhook_configs:
- url: http://<alertsnitch-host-or-ip>:9567/webhookThen add the route
# We want to send all alerts to alertsnitch and then continue to the
# appropiate handler.
route:
routes:
- receiver: alertsnitch
continue: true- -database-backend sets the database backend to connect to, supported are
mysql,postgresandnull - -debug dumps the received WebHook payloads to the log so you can understand what is going on
- -listen.address string address in which to listen for HTTP requests (default ":9567")
- -version prints the version and exit
- ALERTSNITCH_DSN required database connection query string
- ALERTSNITCH_ADDR same as -listen.address
- ALERTSNITCH_BACKEND same as -database-backend
AlertSnitch offers a /-/ready endpoint which will return 200 if the
application is ready to accept WebHook posts.
During startup AlertSnitch will probe the MySQL database and the database model version. If everything is as expected it will set itself as ready.
In case of failure it will return a 500 and will write the error in the response payload.
AlertSnitch offers a /-/health endpoint which will return 200 as long as
the MySQL/Postgres database is reachable.
In case of error it will return a 500 and will write the error in the response payload.
AlertSnitch provides Prometheus metrics on /metrics as per Prometheus
convention.
There is no offering of security of any kind. AlertSnitch is not ment to be exposed to the internet but to be executed in an internal network reachable by the alert manager.
AlertSnitch writes alerts in such a way that they can be explored using Grafana's MySQL/Postgres Data Source plugin. Refer to Grafana documentation for further instructions.
We provide a couple of Makefile tasks to make it easy to run integration tests locally, to get a full coverage sample run:
make bootstrap_local_testing
make integration
go tool cover -html=coverage.out
make teardown_local_testing