Fluent-bit Splitting Large JSON Log into Partial Messages #10378

gursharan-bagha · 2025-05-22T09:04:45Z

gursharan-bagha
May 22, 2025

Stack Details

AWS Opensearch Service
Micros running in AWS ECS Fargate
Fluentbit running as a sidecar container
Ingesting logs to Opensearch.

Using AWS Managed Fluent-bit image: public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

Problem Statement

All JSON-formatted logs are being parsed correctly, except for one particular log from an application that generates an unusually large log entry. The size details of this log are:

Character Count: ~415,000
Byte Size (UTF-8): ~415 KB
Line Count (approx.): 41
File Size on Disk: ~415 KB

Fluent Bit appears to split this single log into multiple partial_message chunks and stores them under the log field.

In the ECS task logs, I see the following warning, which seems related:

[ warn] [engine] failed to flush chunk '1-1747838836.435799559.flb', retry in 8 seconds: task_id=1, input=forward.1 > output=es.2 (out_id=2)

I’d like to understand:

How does Fluent Bit handle such large log entries?

What are the best practices or possible solutions to reliably process and forward large logs like this without data loss or chunking issues?

patrick-stephens · 2025-05-22T10:09:17Z

patrick-stephens
May 22, 2025
Maintainer

Note if it's for the AWS image then they are using an unsupported 1.9 version of OSS so I would raise on their repo directly. You can test with latest OSS image to confirm.

How does the actual log entry look on disk in the file? Is it a single line or separate lines there?
What is your actual config? There are various options there as well to control things so it is important.

4 replies

gursharan-bagha May 22, 2025
Author

@patrick-stephens .. Thanks for your response.

Regarding log document.. Its consist of multipe json fields such as: (Problem with full_message field)

{
  "_index": "index_name-",
  "_id": "_9Cydfdfdfddfdfddffdum",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2025-05-22T11:13:30.527Z",
    "host": "ip-999-99-11-2.eu-west-1.compute.internal",
    "version": "1.1",
    "short_message": "There is a example of short message",
    "full_message": "There is a example of short message",  <<<< This fields contains large data.. Approx 415,000 characters. 
    "timestamp": 1747912410.527,
    "level": 6,
    "facility": "logstash-gelf",
    "some_field": "some-data",
    "some_field": 79dfddfff,
    "some_field": "00052661207820368341",
    "logLevel": "INFO",
    "logger": "m.dfdffdfddfd/.fddfdffr",
    "some_field": "dfdfddd",
    "some_field": "dfdfddfd",
    "source": "stdout",
    "container_id": "6565de65d6d8dfd8fd8ef8dfd",
    "container_name": "service-app",
    "ecs_cluster": "abcdemo",
    "ecs_task_arn": "arn:aws:ecs:eu-west-1:111111111:task/1111111/11106e7dc2d02111111111",
    "ecs_task_definition": "app-service:108"
  },
  "fields": {
    "@timestamp": [
      "2025-05-22T11:13:30.527Z"
    ]
  },
  "sort": [
    1747912410527
  ]
}

patrick-stephens May 22, 2025
Maintainer

I mean how does it look on disk? The actual file you're reading.

Also, config please

gursharan-bagha May 22, 2025
Author

How does it look on disk?

it's a single-line JSON string, just very long...

Config details:

We’re using the official AWS Fluent Bit image public.ecr.aws/aws-observability/aws-for-fluent-bit:latest in a sidecar (fluentbit) via FireLens on ECS Fargate.

"containerDefinitions": [
        {
            "name": "fluentbit",
            "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:latest",
            "cpu": 0,
            "memoryReservation": 1024,
            "portMappings": [],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-create-group": "true",
                    "max-buffer-size": "25m",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "firelens"
                }
            },
            "systemControls": [],
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "/fluent-bit/configs/parse-json.conf"
                }
            }
        },

The configuration is loaded from a static file /fluent-bit/configs/parse-json.conf, with no custom buffer or chunk size settings.

Application containers use the awsfirelens log driver to forward logs to OpenSearch using the es output plugin and parameters from AWS SSM.

"logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "AWS_Region": "us-east-1",
                    "Logstash_Format": "On",
                    "log-driver-buffer-limit": "526870912",
                    "Suppress_Type_Name": "On",
                    "Port": "443",
                    "Logstash_Prefix": "abc-app",
                    "tls": "On",
                    "Name": "es"
                },
                "secretOptions": [
                    {
                        "name": "HTTP_Passwd",
                        "valueFrom": "arn:aws:ssm:us-east-1:323232323232323:parameter/OPEN_PASS"
                    },
                    {
                        "name": "HTTP_User",
                        "valueFrom": "arn:aws:ssm:us-east-1:323232323232323:parameter/OPEN_US"
                    },
                    {
                        "name": "Host",
                        "valueFrom": "arn:aws:ssm:us-east-1:323232323232323:parameter/OPEN_HOST"
                    }
                ]
            },

gursharan-bagha May 23, 2025
Author

@patrick-stephens Did you get a chance to check it.?
Thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fluent-bit Splitting Large JSON Log into Partial Messages #10378

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Fluent-bit Splitting Large JSON Log into Partial Messages #10378

Uh oh!

Uh oh!

gursharan-bagha May 22, 2025

Stack Details

Problem Statement

I’d like to understand:

Replies: 1 comment · 4 replies

Uh oh!

patrick-stephens May 22, 2025 Maintainer

Uh oh!

Uh oh!

gursharan-bagha May 22, 2025 Author

Uh oh!

patrick-stephens May 22, 2025 Maintainer

Uh oh!

gursharan-bagha May 22, 2025 Author

How does it look on disk?

Config details:

Uh oh!

gursharan-bagha May 23, 2025 Author

gursharan-bagha
May 22, 2025

Replies: 1 comment 4 replies

patrick-stephens
May 22, 2025
Maintainer

gursharan-bagha May 22, 2025
Author

patrick-stephens May 22, 2025
Maintainer

gursharan-bagha May 22, 2025
Author

gursharan-bagha May 23, 2025
Author