Smart and lightweight Log Parser and Log Shipper written in Node. It can ship logs to Elasticsearch and thus also to Logsene. See Documentation.
This project contains a library and patterns for log parsing and cli tools and installers to use logagent-js as log shipper with the following features:
- log format detection and intelligent pattern matching
- pattern library included
- easy to extend with custom patterns and JS transform functions
- recognition of Date and Number fields
- replace sensitive data with SHA-1 hash codes
- GeoIP lookup with automatic GeoIP db updates (maxmind geopip-lite files)
-
log format converter (e.g. text to JSON, line delimited JSON or YAML)
-
log shipper for Logsene
- including cli, launchd (Mac OS X), upstart and systemd (Linux) service installer
- disk buffer for failed inserts during network outage
- Standard input (stdin) that can read the output stream from any Linux cli tool
- patterns are applied to each incoming text line; includes support for multi-line patters, e.g. for Java Stack Traces and JSON input.
- Syslog Server (UDP) listener - logagent-js can also act as a syslog server and receive Syslog messages via UDP. The parser is applied to the message field.
- Heroku Log Drain makes it easy to ship Heroku logs to Elasticsearch or Logsene
- Cloud Foundry Log Drain
- logagent-js applies patterns defined in patterns.yml to all logs and creates structured logs from plain-text log lines
- GeoIP lookups for IP address fields, including automatic download and update of the GeoIP lite database from Maxmind
Logagent doesn't lose data. It stores parsed logs to a disk buffer if the network connection to the Elasticsearch API fails. Logagent retries shipping logs later, when the network or Elasticsearch is available again.
- bulk inserts to Logsene / Elasticsearch API
- JSON, line delimited JSON and YML to standard output
- Deployable as a system service: systemd, upstart (Linux), or launchd (Mac OS X)
- Docker Container to receive logs via syslog
- Deployment to Heroku as Heroku Log drain
- Deployment to Cloud Foundry as Cloud Foundry Log drain (thus usable with Pivotal, Bluemix, etc.)
- Node.js module to integrate parsers into Node.js programs
- logagent-js is a part of SPM for Docker to parse Container Logs
The documentation is available here.
Official Node.js downloads and instructions. E.g. for Debian/Ubuntu:
curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs
npm i logagent-js -g
# Be Evil: parse all logs
# stream logs to Logsene 1-Click ELK stack
logagent --index LOGSENE_TOKEN /var/log/*.log
# Act as syslog server on UDP and write YAML formatted messages to console
logagent -u 514 -y
# Act as syslog server on UDP and forward messages to Logsene
logagent -u 514 --index LOGSENE_TOKEN
# Install the service (Linux, Mac OS X)
sudo logagent-setup LOGSENE_TOKEN
# Adjust CLI parameters for your needs
vi /etc/sematext/logagent.conf
- Sematext Agent for Docker - collects metrics, events and logs from Docker API and CoreOS. Logagent-js is a component of sematext-agent-docker. More Information: Innovative Docker Log Management
- Logsene-CLI - Enables searching Logsene log entries from the command-line.
- SPM Agent for Node.js - collects performance metrics for Node and io.js applications
- Custom Metrics - Custom Metrics for SPM
- Winston-Logsene - Logging for Node.js - Winston transport layer for Logsene
- Twitter: @sematext
- Blog: sematext.com/blog
- Homepage: sematext.com