fix(config): prohibit multiline interpolation [WIP] #22995

pront · 2025-05-05T20:53:48Z

Summary

Warning

This PR has a very wide surface area. I will think if it is possible to split it.
Also, it is not actively worked on.
YAML config loading seems to be working fine, but TOML configs have a few bugs.

The envinronment variable (and the secrets) resolution now happens after the config string is parsed into a TOML table. The interpolation now happens on the table keys and values. As a side effect, this fixes a bug where comments referring to env vars (or secrets) that don't exist caused a config build error.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Injecting whole blocks now results in error e.g.

export SOURCES_BLOCK="sources:\"
  demo:
    type: demo_logs
    format: json
    interval: 1

${SOURCES_BLOCK}

The config above will fail to load.

How did you test this PR?

Vector config:

secret:
  backend_1:
    type: file
    path: /Users/pavlos.rontidis/CLionProjects/vector/pront/data/top_secret.json

# ${IN_COMMENT_BUT_DOES_NOT_EXIST}

sources:
  demo_logs_1:
    type: demo_logs
    format: json
    interval: 60
    # noop: "SECRET[i_dont_exist]"

transforms:
  t0:
    inputs:
      - demo_logs_1
    type: "remap"
    source: |
      .host = "${HOSTNAME}"
      .environment = "${ENV:?emv must be supplied}"
      .tenant = "${TENANT:-undefined}"
      .day = "SECRET[backend_1.day]"


sinks:
  s0:
    type: "SECRET[backend_1.type]"
    inputs: [ "t0" ]
    encoding:
      codec: json
      json:
        pretty: true

Output:

{
  "day": "monday",
  "environment": "${ENV:?emv must be supplied}",
  "host": "${HOSTNAME}",
  "message": "{\"host\":...",
  "service": "vector",
  "source_type": "demo_logs",
  "tenant": "${TENANT:-undefined}",
  "timestamp": "2025-05-05T20:52:42.024064Z"
}

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
The CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
  - ./scripts/check_changelog_fragments.sh
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

References

closes: Commented environment variables are still attempted to be interpolated #797
closes: Secrets in commented-out lines in vector.toml cause config validation to fail #22145
Ref: Environment variable substitution and YAML special characters #17343 (need to double check if this is resolved by this PR)

datadog-vectordotdev · 2025-05-05T21:41:56Z

Datadog Report

Branch report: pront/intepolation-after-parsing
Commit report: 6204310
Test service: vector

✅ 0 Failed, 3 Passed, 0 Skipped, 2.13s Total Time

pront · 2025-05-06T16:35:56Z

src/config/loading/interpolation.rs

@@ -0,0 +1,297 @@
+use std::{collections::HashMap, sync::LazyLock};


Moved vars.rs to this location. But kept the same use/export.

pront · 2025-05-06T19:03:44Z

src/config/loading/loader.rs

+    deserialize_table(table)
+}
+
+pub fn resolve_environment_variables(table: Table) -> Result<Table, Vec<String>> {


This was extracted from prepare_input

changelog.d/multiline_interpolation_change.breaking.md

pront · 2025-05-07T14:41:08Z

src/config/loading/secret.rs

@@ -88,11 +89,10 @@ impl SecretBackendLoader {
 }



Note: Currently (on master) we don't support multiple types for secrets.

2025-05-07T14:38:00.776983Z ERROR vector::cli: Configuration error. error=Error while retrieving secret from backend "backend_1": invalid type: integer `1`, expected a string at line 1 column 10.

However, interpolate_secrets supports multiple types. I consider this out of scope for this PR but we could go further and allow e.g. integer secrets.

pront · 2025-05-07T14:42:09Z

With COUNT=1 gives:

2025-05-06T21:40:58.830829Z  INFO vector::app: Log level is enabled. level="info"
2025-05-06T21:40:58.831986Z  INFO vector::app: Loading configs. paths=["/tmp/tmp.yaml"]
2025-05-06T21:40:58.838663Z ERROR vector::cli: Configuration error. error=invalid type: string "1", expected usize

Fixed and added tests to cover this.

pront · 2025-05-13T20:37:09Z

So there are some tests failures:

existing e2e regression
this PR broke some tests (no_proxy is set to a string but schema specifies array of strings)

Will fix (1) in a separate PR
Will fix (2) in this PR (after 1 is fixed)

pront · 2025-06-18T18:31:37Z

Still testing this. I found at least one bug for TOML configs. YAML configs seems to be working fine.

…ore interpolation"

…onverted to array eventually)

pront changed the title ~~fix(config): disallow multiline interpolation~~ fix(config): prohibit multiline interpolation May 6, 2025

pront commented May 6, 2025

View reviewed changes

pront marked this pull request as ready for review May 6, 2025 21:19

pront requested a review from a team as a code owner May 6, 2025 21:19

pront requested a review from jszwedko May 6, 2025 21:20

This comment was marked as outdated.

Sign in to view

github-advanced-security bot found potential problems May 7, 2025

View reviewed changes

changelog.d/multiline_interpolation_change.breaking.md Fixed Show fixed Hide fixed

pront commented May 7, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

pront changed the title ~~fix(config): prohibit multiline interpolation~~ fix(config): prohibit multiline interpolation [WIP] May 12, 2025

pront added 14 commits June 19, 2025 10:05

fix(config): disallow env var block injection and ignore comments bef…

8e41e21

…ore interpolation"

add test for block in invalid ctx

ba367c0

improve changelog

43743ef

fix bug after bad refactoring, cleanup

22f6727

handle additional types (i64, f64, bool)

158b5ba

add workaround to changelog

b5b13d8

add test for multiple secret types

7147831

chore: update generated license files

33d52a5

markdown lint

22830fd

coerce

87d343a

chore(dev): vdev fmt

00fb53c

chore(dev): vdev fmt

8deffa7

chore(dev): cargo vdev build licenses

4d867a8

permissive conversion to array, for cases like proxy.no_config='x' (c…

a5ded21

…onverted to array eventually)

pront added 4 commits June 19, 2025 10:05

add guard to skip secret processing

aac8ff1

push the coercion logic deeper in the stack

007108d

remove dbg statement

7a8688e

wip

3ea6d19

pront marked this pull request as draft June 19, 2025 15:15

pront added 5 commits June 19, 2025 11:47

better coercion test

0f66b60

fix any_of and one_of hnadling and separate them

3994988

dbg commit

b6987bf

move resoltion in Process

97a5648

Debug

2cfe741

pront removed the request for review from a team June 20, 2025 13:49

pront force-pushed the pront/intepolation-after-parsing branch from 62d735b to 2cfe741 Compare June 20, 2025 14:00

pront mentioned this pull request Jun 20, 2025

chore(website): improve environment variable docs #23236

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(config): prohibit multiline interpolation [WIP] #22995

fix(config): prohibit multiline interpolation [WIP] #22995

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

		@@ -0,0 +1,297 @@
		use std::{collections::HashMap, sync::LazyLock};

fix(config): prohibit multiline interpolation [WIP] #22995

Are you sure you want to change the base?

fix(config): prohibit multiline interpolation [WIP] #22995

Uh oh!

Conversation

Uh oh!

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

Notes

References

Uh oh!

Uh oh!

Datadog Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!