fluent-plugin-sort

Fluent-plugin-sort is a Fluentd plugin to sort messages in buffer. Fluentd doesn't guarantee message order but you may keep message order.

If you use large buffer, you will keep order of many messages but messages are delayed. If you use small buffer, you will keep order of some messages but messages are delayed a bit.

% gem install fluent-plugin-sort

Fluentd doesn't guarantee message order. It means that messages may be out of order. Fluent-plugin-sort plugin reduces the case.

Fluent-plugin-sort keeps received messages before routing the next output plugin. Fluent-plugin-sort sorts buffered messages when buffer is flushed. It will reduce out of order messages but it's not perfect. Some messages may be out of order.

Here are sample configuration:

<match message.**>
  type sort
  add_tag_prefix sorted.
  flush_interval 60
</match>

You can reduce out of order messages by increasing flush_interval parameter in type sort configuration. If its value is large, many messages will be sorted at once. It reduces out of order messages. But large flush interval delays message routing. Tune the parameter carefully.

You can use fluent-plugin-sort for log archive system. See Log archive at project-hatohol/hatohol Wiki. It describes architectures, how to set up and how to configure.

Fluent-plugin-sort is a buffered output plugin. You can use buffer related parameters.

• Fluent::HandleTagNameMixin: It adds the following parameters:
  • add_tag_prefix
  • add_tag_suffix
  • remove_tag_prefix
  • remove_tag_suffix

Default: time

It specifies key for sort. You can sort messages by time-stamp in a fluent message or a record value.

If you want to use time-stamp in a fluent message, specify time. time is the default value.

Example:

<match>
  type sort
  sort_key time
</match>

If you want to use a record value, specify record key with attribute: prefix.

Here is an example to use id value:

<match>
  type sort
  sort_key attribute:id
</match>

Buffered records:

[
  {"id": 2},
  {"id": 4},
  {"id": 1},
  {"id": 3}
]

Sorted result:

[
  {"id": 1},
  {"id": 2},
  {"id": 3},
  {"id": 4}
]

You can use deep record value by separating by .. Using . as delimiter is inspired by JSONPath.

Here is an example to use body.time-stamp value:

<match>
  type sort
  sort_key attribute:body.time-stamp
</match>

Buffered records:

[
  {"body": {"time-stamp": 1413272109}},
  {"body": {"time-stamp": 1413272107}},
  {"body": {"time-stamp": 1413272108}},
  {"body": {"time-stamp": 1413272106}}
]

Sorted result:

[
  {"body": {"time-stamp": 1413272106}},
  {"body": {"time-stamp": 1413272107}},
  {"body": {"time-stamp": 1413272108}},
  {"body": {"time-stamp": 1413272109}}
]

If attribute value is null or attribute doesn't exist, the message is treated as the most oldest message.

Buffered records:

[
  {"body": {"time-stamp":    1413272109}},
  {"body": {"time-stamp":    1413272107}},
  {"body": {"time-stamp":    1413272108}},
  {"body": {"no-time-stamp": true}}
]

Sorted result:

[
  {"body": {"no-time-stamp": true}},
  {"body": {"time-stamp":    1413272107}},
  {"body": {"time-stamp":    1413272108}},
  {"body": {"time-stamp":    1413272109}}
]

• Project Hatohol

LGPL 2.1 or later. See doc/text/lgpl-2.1.txt for details.

Use GitHub issues to submit a question and bug report. You can use English or Japanese on it.

The repository for fluent-plugin-sort is on GitHub.

• ...