This tool sets up a simple Kusto emulator with Fluentd to pull all Docker logs into the Kusto emulator.
The purpose of this repository is to provide a local development environment for aggregating Docker logs using a Kusto emulator. This setup allows developers to test and debug their logging configurations and scripts in a controlled environment before deploying them to a production system. Additionally, it aims to help developers search through logs and dashboard complex systems by providing a way to visualize and analyze log data effectively using Azure Kusto Explorer.
The log aggregation process involves several steps to ensure that Docker logs are collected and sent to the Kusto emulator. Here's how it works:
- Docker Logging Driver: Docker containers generate logs which are managed by the Docker logging driver. In this setup, the
json-filelogging driver is used to store logs in JSON format. - File Tailing: Fluentd, a log collector, is configured to tail these JSON log files. This is achieved by binding the Docker log directory into the Fluentd container, allowing Fluentd to access and read the log files.
- Log Processing: Fluentd processes the tailed log files using a custom
kusto.luascript. This script is responsible for formatting and preparing the logs for ingestion into the Kusto emulator. - Log Ingestion: The processed logs are then sent to the Kusto emulator through an HTTP request, where they can be queried and analyzed using Kusto Query Language (KQL).
This setup ensures that all Docker logs are efficiently collected and made available for analysis in a local development environment.
CAUTION: The Fluentd configuration is a bit hacky because Fluentd does not natively support the Kusto emulator, as the Kusto emulator does not support log streaming. We had to create a custom kusto.lua script to handle aggregating the logs. However, if more than 10,000 logs are sent in 2 seconds, you will likely encounter issues. It is very possible that you would be limited before that number. This is for local development only.
It is important to note that in this setup, logs are not sent to Azure Data Explorer. Instead, they are ingested into a local Kusto emulator. This distinction is crucial for understanding the scope, limitations, and security of this tool:
- Local Development: The Kusto emulator is intended for local development and testing purposes. It allows developers to experiment with log aggregation and querying without the need for an Azure subscription.
- Security: You do not have to worry about your personal docker containers being ingested into ADX
This approach provides a cost-effective and convenient way to develop and test logging configurations before deploying them to a production environment that uses Azure Data Explorer.
- Docker
- Docker Compose
- Web browser
To use this tool, follow these steps:
-
Clone the repository:
git clone <repository-url> cd kusto-docker-log-aggregator
-
Run Docker Compose to start the emulator and log collector:
docker compose up -d
The log collector will collect all JSON logs and send them to the Kusto emulator.
-
Open a browser and navigate to Azure Data Explorer:
- Login if needed with any Microsoft account.
- Alternative: If you are using a Windows machine, you can download the Azure Kusto Explorer desktop application for a more integrated experience.
-
Add a connection to the Kusto emulator:
- Under the query section, add a connection to
http://localhost:8080/.
- Under the query section, add a connection to
-
Run the following query to create the application logs table and ingestion mapping:
// Create the table with columns 'timestamp' and 'event' .create table application_logs (timestamp: datetime, event: dynamic) // Create the ingestion mapping for JSON data .create table application_logs ingestion json mapping 'json_mapping' '[{"column":"timestamp", "path":"$[\'time\']", "datatype":"datetime"}, {"column":"event", "path":"$", "datatype":"dynamic"}]'
Once the setup is complete, Docker logs will be collected and sent to the Kusto emulator. You can query the logs using the Azure Data Explorer interface.
-
Retrieve all logs:
application_logs | take 100
-
Filter logs by a specific time range:
application_logs | where timestamp between(datetime(2023-01-01) .. datetime(2023-01-02))
-
Search for specific events:
application_logs | where event contains "error"
To generate example logs, you can use the following docker-compose.yml file located in example_log_generator/docker-compose.yml:
version: '3.8'
services:
log-generator:
image: bash
command: |
/bin/sh -c "count=1; while true; do echo \"{\\\"
7FF1
counter\\\": $$count, \\\"date\\\": \\\"$$(date -u +'%Y-%m-%dT%H:%M:%SZ')\\\" }\"; sleep 1; count=$$((count+1)); done"
logging:
driver: json-file
options:
max-size: "512m"
max-file: "3"
labels: "serviceName,stackId"
tag: '{ "imageName": "{{.ImageName}}", "containerName": "{{.Name}}", "containerId": "{{.ID}}" }'
labels:
stackId: LOCAL-DEVELOPER-STACKTo start the log generator, navigate to the example_log_generator directory and run:
docker compose up -dTo see the counter count up in Azure Data Explorer, you can use the following query:
application_logs
| extend log=parse_json(tostring(parse_json(tostring(event.log))))
| extend counter=log.counter
| order by timestamp desc
| limit 10This query will parse the JSON logs, extract the counter field, and display the latest 10 logs ordered by timestamp.
To enhance the log output for your containers, add the following configuration to each service in your docker-compose.yml file:
logging:
driver: json-file
options:
max-size: "512m"
max-file: "3"
labels: "serviceName,stackId"
tag: '{ "imageName": "{{.ImageName}}", "containerName": "{{.Name}}", "containerId": "{{.ID}}" }'
labels:
stackId: LOCAL-DEVELOPER-STACKChanges made:
- Improved the section title for clarity.
- Provided a more specific file reference (
docker-compose.yml). - Enhanced the description for better readability.
-
Logs not appearing in Kusto:
- Ensure Docker containers are running:
docker ps - Check Fluentd logs for errors:
docker logs <fluentd-container-id>
- Ensure Docker containers are running:
-
Connection issues:
- Verify that the Kusto emulator is accessible at
http://localhost:8080/.
- Verify that the Kusto emulator is accessible at
Contributions are welcome! Please open an issue or submit a pull request.
This project is licensed under the MIT License. See the LICENSE file for details.