Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
Intro
I am writing to an InfluxDB instance via vector sink from a http-source. The InfluxDB instance is running in a docker container orchestrated with nomad and registered as a service in consul. In case the instance is restarted the port, where InfluxDB can be reached, changes because the ports are dynamically assigned.
The vector config changes accordingly (however not instantly) because it is generated via a consul-template
Problem
I have problems with the port changes which are handled, at least to my understanding, strangely. I have recreated the behaviour locally on my machine both with vector 0.46.1 and 0.47.0.
Initially InfluxDB is exposed via port 8086 and vector is configured to write to that port. Everything works nice (as expected). All incoming messages are written to InfluxDB:
Now i change the port of InfluxDB to 8087. Vector still writes to 8086. That of course does not work and messages are not written anymore in InfluxDB:
Now i update the config of the InfluxDB-sink to write to port 8087 (and refresh the config) and write some new messages to the input. Still the messages are not written to InfluxDB and not only the InfluxDB-sink isn't getting any more messages but other transforms as well which are not only to InfluxDB but to follow-up transforms as well.
But the error number on right is increasing although vector should know the correct port to write to.
We get the following error in the log:
WARN sink{component_kind="sink" component_id=influxdb_request_stats component_type=influxdb_logs}:request{request_id=8}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111) internal_log_rate_limit=true
Now i switch back the port of InfluxDB to 8086 and we can see that vector now writes the messages that have been originally target to port 8086 to InfluxDB. And the transform steps also get their events:
And afterwards i set the port of InfluxDB back to port 8087 and the rest of the messages are written:
I have experienced the same behaviour with the loki sink as well (but didnt simulate that here)
Expectation
I would expect that vector understands that the endpoint of the sink has changed due to the config change and retries the already existing messages to the configured endpoint. However to me it appears that vector continues to try to send messages to the exact endpoint that has been configued when the messages have been first tried to be sent.
On top i do not understand why it affects other semi-dependent transform steps.
Configuration
sources:
logs:
type: "http"
address: "0.0.0.0:8078"
encoding: "json"
transforms:
request_log_only:
type: "filter"
inputs: ["logs"]
condition: 'exists(.type) && .type == "request-log" && exists(.service)'
sinks:
influxdb_request_stats:
type: "influxdb_logs"
inputs: ["request_log_only"]
database: "${INFLUXDB_DB}"
measurement: "request_stats"
endpoint: "http://172.17.0.1:8087"
# Some config related to fields and tags which should not be relevant
buffer:
type: disk
max_size: 1073741824
when_full: drop_newest
request:
timeout_secs: 10
Version
vector 0.47.0 (aarch64-unknown-linux-musl 3d5af22 2025-05-20 13:53:41.638057046)
Debug Output
Example Data
No response
Additional Context
No response
References
No response