Config-Change not reflected properly on failed events #23098

dev756 · 2025-05-23T23:16:24Z

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Intro

I am writing to an InfluxDB instance via vector sink from a http-source. The InfluxDB instance is running in a docker container orchestrated with nomad and registered as a service in consul. In case the instance is restarted the port, where InfluxDB can be reached, changes because the ports are dynamically assigned.
The vector config changes accordingly (however not instantly) because it is generated via a consul-template

Problem

I have problems with the port changes which are handled, at least to my understanding, strangely. I have recreated the behaviour locally on my machine both with vector 0.46.1 and 0.47.0.

Initially InfluxDB is exposed via port 8086 and vector is configured to write to that port. Everything works nice (as expected). All incoming messages are written to InfluxDB:

Now i change the port of InfluxDB to 8087. Vector still writes to 8086. That of course does not work and messages are not written anymore in InfluxDB:

Now i update the config of the InfluxDB-sink to write to port 8087 (and refresh the config) and write some new messages to the input. Still the messages are not written to InfluxDB and not only the InfluxDB-sink isn't getting any more messages but other transforms as well which are not only to InfluxDB but to follow-up transforms as well.

But the error number on right is increasing although vector should know the correct port to write to.
We get the following error in the log:

WARN sink{component_kind="sink" component_id=influxdb_request_stats component_type=influxdb_logs}:request{request_id=8}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111) internal_log_rate_limit=true

Now i switch back the port of InfluxDB to 8086 and we can see that vector now writes the messages that have been originally target to port 8086 to InfluxDB. And the transform steps also get their events:

And afterwards i set the port of InfluxDB back to port 8087 and the rest of the messages are written:

I have experienced the same behaviour with the loki sink as well (but didnt simulate that here)

Expectation

I would expect that vector understands that the endpoint of the sink has changed due to the config change and retries the already existing messages to the configured endpoint. However to me it appears that vector continues to try to send messages to the exact endpoint that has been configued when the messages have been first tried to be sent.

On top i do not understand why it affects other semi-dependent transform steps.

Configuration

sources:
  logs:
    type: "http"
    address: "0.0.0.0:8078"
    encoding: "json"
transforms:
  request_log_only:
    type: "filter"
    inputs: ["logs"]
    condition: 'exists(.type) && .type == "request-log" && exists(.service)'
sinks:
  influxdb_request_stats:
    type: "influxdb_logs"
    inputs: ["request_log_only"]
    database: "${INFLUXDB_DB}"
    measurement: "request_stats"
    endpoint: "http://172.17.0.1:8087"
    # Some config related to fields and tags which should not be relevant
    buffer:
      type: disk
      max_size: 1073741824
      when_full: drop_newest
    request:
      timeout_secs: 10

Version

vector 0.47.0 (aarch64-unknown-linux-musl 3d5af22 2025-05-20 13:53:41.638057046)

Debug Output

Example Data

No response

Additional Context

No response

References

No response

The text was updated successfully, but these errors were encountered:

dev756 added the type: bug A code related bug. label May 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Config-Change not reflected properly on failed events #23098

Config-Change not reflected properly on failed events #23098

Config-Change not reflected properly on failed events #23098

Config-Change not reflected properly on failed events #23098

Comments

A note for the community

Problem

Intro

Problem

Expectation

Configuration

Version

Debug Output

Example Data

Additional Context

References