Description
In what area(s)?
/area scheduler
Ask your question here
We’ve encountered a regression after upgrading from Dapr 1.14.1 to 1.15.5 in an environment using Istio with strict service registry policies.
Context
- We use Istio with a service registry that blocks all outbound traffic by default.
- We deploy dapr through the dapr operator which sets certain env variables automatically on the daprd sidecar
- We explicitly allow traffic to Dapr system services (e.g., placement, sentry, API, scheduler) using Istio
ServiceEntryandDestinationRuleresources. - In Dapr 1.14.1, the sidecar was configured to connect to the scheduler using:
- name: DAPR_SCHEDULER_HOST_ADDRESS
value: >
dapr-scheduler-server-0.dapr-scheduler-server.dapr-system.svc.cluster.local:50006,
dapr-scheduler-server-1.dapr-scheduler-server.dapr-system.svc.cluster.local:50006,
dapr-scheduler-server-2.dapr-scheduler-server.dapr-system.svc.cluster.local:50006This worked fine with our Istio setup. We have service/port combinations mentioned above added in our istio service registry so istio allows the traffic.
Issue
In Dapr 1.15.5, a new environment variable was introduced:
- name: DAPR_SCHEDULER_HOST_ADDRESS_DNS_A
value: dapr-scheduler-server-a.dapr-system.svc.cluster.local:443This causes the sidecar to resolve the A-record and connect directly to the Pod IPs of the scheduler servers (bypassing the services)
Problem: Pod IPs are ephemeral and cannot be whitelisted in Istio ServiceEntry resources. As a result, the Istio sidecar blocks these requests, breaking the scheduler communication.
Expected Behavior
The sidecar should continue to use the explicitly listed scheduler endpoints (via DAPR_SCHEDULER_HOST_ADDRESS) that are covered by our ServiceEntry definitions.
Actual Behavior
The sidecar attempts to connect to Pod IPs resolved via the DNS A-record, which are not whitelisted and thus blocked by Istio.
Request
- Can this behavior be made configurable or fallback to the previous behavior?
- Alternatively, can the scheduler client respect the
DAPR_SCHEDULER_HOST_ADDRESSsetting as primary? - Is there a solution to this problem I'm not aware of?