How to modify JSON log line without Lua? #10468

alongwill · 2025-06-13T07:47:52Z

alongwill
Jun 13, 2025

Here is an example log entry I'm working on as displayed via stdout:

[[1749722196.000000000, {}], {
  "log"=>"{"msg":"ANDY TEST: Dummy log line from DEBUG Pod","cluster":"andy-test","foo":"bar"}", 
  "kubernetes"=>{
    "pod_name"=>"omni-deploy-749b956cb4-jtwsb", 
    "namespace_name"=>"omni", 
    ... 
    "container_name"=>"omni", 
    "container_image"=>"ghcr.io/siderolabs/omni:v0.50.1"
  },
  "cluster"=>"stage", 
  "hostname"=>"worker-1", 
  "source"=>"kube"
}]

The issue we're having is that when this log eventually gets into Victoria Logs, cluster=andy-test overwrites cluster=stage.

We are able to rename that cluster label using Lua

function rename_field(tag, timestamp, record)
    record["log"] = string.gsub(record["log"], "cluster", "app_cluster")
    return 1, timestamp, record
end

However I'm wondering if there is a better way to address this?

Full Fluent-bit pipeline

[INPUT]
      Name tail
      Alias kubernetes
      Path /var/log/containers/*.log
      Parser containerd
      Tag kube.*
      Mem_Buf_Limit 5MB
      Skip_Long_Lines On
      Skip_Empty_Lines On
[FILTER]
      Name kubernetes
      Match kube.*
      Merge_Log Off
      Keep_Log Off
      K8S-Logging.Parser On
      K8S-Logging.Exclude On
[FILTER]
      Name modify
      Match *
      Set cluster stage
      Set hostname ${HOSTNAME} 
[OUTPUT]
      name         loki
      match        audit.*
      host         vlogs-vlinsert.monitoring.svc
      uri          /insert/loki/api/v1/push
      port         9428
      Tenant_Id    0
      label_keys   $path,$time,
8000
$stream,$cluster,$source,$hostname,$kind,$apiVersion,$level,$stage,$verb,$user,$groups,$auditID,$userAgent
      header       VL-Msg-Field msg
      header       VL-Time-Field stageTimestamp
      header       VL-Stream-Fields cluster,source
      remove_keys  path,stream,time,stageTimestamp
      drop_single_key raw
      line_format  key_value
[OUTPUT]
      name         loki
      match        kube.*
      host         vlogs-vlinsert.monitoring.svc
      uri          /insert/loki/api/v1/push
      port         9428
      Tenant_Id    0
      label_keys   $path,$time,$stream,$cluster,$source,$hostname,$kubernetes['pod_name'],$kubernetes['container_name'],$kubernetes['namespace_name']
      header       VL-Msg-Field msg
      header       VL-Time-Field time
      header       VL-Stream-Fields cluster,source,kubernetes_namespace_name,kubernetes_container_name
      remove_keys  path,stream,time,kubernetes
      drop_single_key raw
      line_format  key_value
      Auto_Kubernetes_Labels true

alongwill · 2025-06-13T09:33:40Z

alongwill
Jun 13, 2025
Author

This almost works, but nest converts it into a map instead of raw JSON, which means when log gets to Victoria Logs, its processed as a single log line rather than being split into labels. 🤔

    [FILTER]
        Name parser
        Match kube.*
        Key_Name log
        Parser json
    [FILTER]
        Name modify
        Match kube.*
        Rename cluster app_cluster
    [FILTER]
        Name nest
        Match kube.*
        Operation nest
        Wildcard *
        Nest_under log

0 replies

patrick-stephens · 2025-06-16T17:02:58Z

patrick-stephens
Jun 16, 2025
Maintainer

Are you using the kubernetes filter with merge log set?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

How to modify JSON log line without Lua? #10468

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

How to modify JSON log line without Lua? #10468

Uh oh!

Uh oh!

alongwill Jun 13, 2025

Replies: 2 comments

Uh oh!

Uh oh!

alongwill Jun 13, 2025 Author

Uh oh!

patrick-stephens Jun 16, 2025 Maintainer

alongwill
Jun 13, 2025

alongwill
Jun 13, 2025
Author

patrick-stephens
Jun 16, 2025
Maintainer