8000 ClickHouse retries on not-ready servers because of `PublishNotReadyAddresses` · Issue #1687 · Altinity/clickhouse-operator · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

ClickHouse retries on not-ready servers because of PublishNotReadyAddresses #1687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
cangyin opened this issue Apr 17, 2025 · 1 comment · May be fixed by #1688
Open

ClickHouse retries on not-ready servers because of PublishNotReadyAddresses #1687

cangyin opened this issue Apr 17, 2025 · 1 comment · May be fixed by #1688

Comments

@cangyin
Copy link
cangyin commented Apr 17, 2025 •
edited
Loading

If a server failed to start, DNS should fail to resolve (server name does not correspond to any IP address), so as to tell the initiator ClickHouse server to avoid connecting to this server.

With PublishNotReadyAddresses=true, the initiator ClickHouse server believes a failed server should exist and makes several retries to connect it. Thus when there is any failed server, all subsequent queries will slow down for several seconds to wait for the connection retries to run out.

PublishNotReadyAddresses here

clickhouse-operator/pkg/model/chk/creator/service.go

Line 227 in 69d5e39

PublishNotReadyAddresses: true,

As k8s docs says:

The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery

But a ClickHouse cluster does not need peer discovery because it is designed as multi-master architecture, where peer addresses are resolved from their hostnames before query execution (perhaps with DNS cache). An IP address of failed server does not make it better.
@cangyin
Copy link
Author
cangyin commented Apr 17, 2025 •
edited
Loading

In environments where administrators/users does not have the access to a incorrectly configured /etc/resolv.conf on host machine, not-ready servers may cause queries on initiator server to hang on DNS timeouts. This can be a workaround, but may not be turned on by default for the reason above.

For example, if in an isolated network where /etc/resolv.conf on host machine is configured to the Google DNS server 8.8.8.8, but ClickHouse administrators/users can not modify it. PublishNotReadyAddresses=true becomes a choice.

@cangyin cangyin linked a pull request Apr 17, 2025 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Disable PublishNotReadyAddresses by default cangyin/clickhouse-operator
1 participant
@cangyin
0