This is the companion repository to a series of presentations over the three pillars of observability within Grafana.
It is presented as a self-enclosed Docker sandbox that includes all of the components required to run on a local machine and experiment with the services provided.
Since the original series, this repository has seen its use grow. Whilst we still hugely encourage everyone to sign up for a Grafana Cloud account, this repository assist an easy way to get started with Grafana's offerings in a non-cloud, local based setup and to experiment with configuration settings on those offerings.
The following demonstration environment requires:
The demos from this series were based on the application and code in this repository, which includes:
- Docker Compose manifest for easy setup.
- Four-service application:
- A service requesting data from a REST API server.
- A REST API server that receives requests and utilises a Database for storing/retrieving data for those requests.
- A recorder service for storing messages to an AQMP bus.
- A Postgres Database for storing/retrieving data from.
- Tempo instance for storing trace information.
- Loki instance for storing log information.
- Mimir instance for storing metric information.
- Phlare instance for storing profiling information.
- Grafana instance for visualising observability information.
- Grafana Agent instance for receiving traces and producing metrics and logs based on these traces.
- A Node Exporter instance to retrieve resource metrics from the local host.
Docker Compose will download the required Docker images, before starting the demonstration environment. Data will be emitted from the microservice application and be stored in Loki, Tempo and Prometheus. You can login to the Grafana instance to visualise this data. To execute the environment and login:
- Start a new command-line interface in your Operating System and run:
docker-compose up
- Login to the local Grafana instance at http://localhost:3000/. The default credentials for Grafana are
adminas both the username and password. Change the credentials if you wish, or skip. NOTE: This assumes that port 3000 is not already in use. If this port is not free, edit thedocker-compose.ymlfile and alter the lineto some other host port that is free, for example:- "3000:3000"- "3123:3000" - Navigate to the MLT dashboard.
- Explore the data sources using the Grafana Explorer.
The following is a brief explanation of each of the most important provided components.
Grafana is a visualisation tool that allows the creation of dashboards from various data sources. More information can be found here.
The Grafana instance is described in the grafana section of the docker-compose.yml manifest.
It:
- Mounts two repository directories to provide pre-provisioned data sources for data.
- A pre-provisioned dashboard for correlating metrics, logs and traces.
- This dashboard uses metrics from span traces to provide RED (Rate/Error/Duration) signals.
- Data links are built into Grafana panels to pre-populate TraceQL queries based on the data. These act as an initial guide for digging into more detailed trace queries by extending the TraceQL.
- Exposes port
3000for local login. - Enables two Tempo features, namely span search and service graph support.
No custom configuration is used.
Note that in this latest version, topnav navigation within Grafana is enabled. If you wish to default back to the old UI, remove the topnav feature flag in the GF_FEATURE_TOGGLES_ENABLE environment variable for the grafana service in the docker-compose manifest.
Mimir is a backend store for metrics data from various sources. More information can be found here.
The Mimir instance is described in the mimir section of the docker-compose.yml manifest.
The configuration file (mimir/mimir.yml):
- Configures a single instance container acting as all relevant microservices.
- Stores the metrics data in-container (this will be lost on container deletion).
Additionally to the scraped metrics, it also receives remotely written metrics from the Tempo service, which derives metrics from incoming trace spans.
An example of the Mimir data source showing a histogram with exemplars (links to relevant traces), is here, once the system is running and has collected enough data.
Loki is a backend store for longterm log retention. More information can be found here.
The Loki instance is described in the loki section of the docker-compose.yml manifest.
This instance is simply the latest Loki image available, and exposes its interface on port 3100.
The microservices application sends its logs directly to the Loki instance in this environment, via its REST API.
Loki's default configuration is used, and as such no custom configuration file is bound to the container (unlike Mimir and Tempo).
An example of the Loki data source using LogQL is here.
Tempo is a backend store for longterm trace retention. More information can be found here.
The Tempo instance is described in the tempo section of the docker-compose.yml manifest.
The Tempo service imports a configuration file (tempo/tempo.yaml) that initialises the service with some sensible defaults as well as allowing the receiving of traces in a variety of different formats.
Tempo is also configured to generate metrics from incoming trace spans as part of it's configuration. As such, this no longer occurs via Grafana Agent (although the original configuration for the Agent to carry this out has been left in the Agent configuration file as a guide).
For an example of a span search, look at the Explorer page using the Tempo data source, here.
For an example of the mini-APM table and Service Graphs, use the 'Service Graph' tab here.
Traces are instrumented using the OpenTelemetry SDK, more details on which can be found here.
Phlare is currently at version 0.1.x.
Phlare is a continuous profiling backend store, and is the latest of the database products to be announced by Grafana. See here for the latest documentation.
Phlare uses a configuration file (phlare/phlare.yaml) that is configured to scrape pprof profiles from the Mythical microservices.
pprof samples are scraped directly from the application on the /debug/pprof/profile endpoint (and supported using appropriate NPM modules).
You can see an example of profiling in action once the system is running by using the Explorer to visualise the profiles stored here.
Note: Currently it can take a few minutes before enough profiling information is available. You may see an error until a few minutes have passed on initial startup.
Grafana Agent is a locally installed agent that acts as:
- A Prometheus scraping service.
- A Tempo backend service receiver and trace span processor.
- A Promtail (Loki logs receiver) instance.
Grafana Agent has remote write capabilities that allows it to send metrics, logs and trace data to backend stores (such as Mimir, Loki and Tempo). More information on Grafana Agent can be found here.
Its main role in this environment is to receive trace spans from the microservice application and process them to extract log information before storing them in the final backend stores.
It also defines several metrics scrape jobs, including:
- Retrieving metrics from the Prometheus instance itself.
- Metrics from the microservices application.
- Metrics from the installed Node Exporter instance.
All of these metrics are remotely written to the Mimir instance.
Note that the configuration now includes a commented section where metrics used to be generated from incoming trace spans; this is now handled directly in Tempo via server-side metrics generation, although the original configuration block has been left commented out in the Agent config.
The configuration file for it can be found in agent/config.yaml.
The source for the microservice application can be found in source. It is a three service application that utilises a Postgres database and AQMP bus to store data.
The services are written in JavaScript and execute as NodeJS scripts inside Docker containers.
The requester services makes 'random' requests to the server, which then inserts, retrieves or deletes data from the Postgres database. The requester service also stores data to the AQMP queue via the recorder service.
All three services use common code to deal with the queue, logging and tracing requirements they have.
There is a common Dockerfile that is used to build all three services.
This demo can be run against Grafana Cloud using Docker Compose as follows:
-
Configure and source the Metrics, Logs and Traces environment variables in the file
cloud/envvars-grafana-cloud-unconfigured.shfrom a Grafana Cloud instance. Generally only the environment variables tagged with “ID” and “API_KEY” need to be updated, sometimes the _HOST variables will need to be modifyed.source cloud/envvars-grafana-cloud-unconfigured.sh -
Configure the environment
./ctl.sh cloud-configure
-
Run the demo using Docker Compose
./ctl.sh cloud-up
-
Load the example dashboard
cloud/dashboard-metrics-logs-traces-1.jsoninto the Grafana Cloud Instance to visualize the metrics, logs and traces being generated by this demo -
Stop the demo using
./ctl.sh cloud-down