Configure sidecar installation to bypass calls to container image registry during pod startup #56058

JakubLedworowski · 2025-04-24T14:14:27Z

JakubLedworowski
Apr 24, 2025

Hello Everyone, I am looking for help in configuring istio sidecar mode.

In sidecar mode, in default configuration, we have 3 containers for a pod:

istio-init
app
istio-proxy

After the istio-init is started, it reconfigures the iptables so that all subsequent network requests will go through istio-proxy.
My use case is that the container images are always pulled directly to the pod by a containerd plugin, even the istio-proxy image itself. It fails, because of the chicken and egg problem: request to registry to pull of the istio-proxy image must go through istio-proxy itself.

In short, the scheme looks like this:

start pod
pull istio-init image
start istio-init, iptables reconfiguration
pull istio-proxy and app image - this fails, because istio-proxy is not yet started
start istio-proxy and app - this fails, because the images were not pulled

Question: how to configure istio installation to bypass calls to containers registries (preferebaly by DNS name, not CIDR)?

This is the install command I am using:

istioctl install --set values.global.imagePullPolicy=Always -f samples/bookinfo/demo-profile-no-gateways.yaml -y

Thanks in advance for all replies :)

Answered by rsalmond

Apr 28, 2025

Ahhh yes, this is the information we needed. Thank you for clarifying!

Okay so essentially what you are facing is a variation of the "how do I run an istio sidecar with hostNetwork: true" problem (point 2 in this doc). I am not aware of any standard practice for automatically extending the mesh to K8s managed microvm's. The closest thing I know of is this guide to extending the service mesh to include VM's which are external to the K8s cluster.

I can think of two avenues to explore for this use case.

Create a purpose built solution to add support for the microVM. This could be something like implementing containers running in a network namespace within the microVM, or building your VM i…

View full answer

sridhargaddam · 2025-04-24T14:35:22Z

sridhargaddam
Apr 24, 2025
Collaborator

Can you check if using native sidecars helps - https://istio.io/latest/blog/2023/native-sidecars/

1 reply

JakubLedworowski Apr 25, 2025
Author

Hi, this is using native sidecars:

kubectl get pod -o "custom-columns=""NAME:.metadata.name,""INIT:.spec.initContainers[*].name,""CONTAINERS:.spec.containers[*].name"
NAME       INIT         CONTAINERS
nginx   istio-init   nginx,istio-proxy

I updated the description with install command.

rsalmond · 2025-04-24T16:51:10Z

rsalmond
Apr 24, 2025
Collaborator

The container runtime which pulls the container images does not execute inside the network namespace of the pod, so the iptables rules set up by the init container do not apply to it.

What happens if you try to manually pull the image istio/proxyv2:1.24.5 on the worker node?

6 replies

JakubLedworowski Apr 25, 2025
Author

From the behavior I observe the image pull is indeed affected by the change, probably due to the design of the beforementioned containerd plugin.
I confirmed it by manually injecting the initContainer with explicitly excluded all IP ranges for docker.io:

First inject the default settings using istioctl:

istioctl kube-inject -f nginx.yaml > nginx-istio.yaml

Then modify the istio-init args to exclude CIDR for image registry, eg. docker.io:

(...)
initContainers:
  - args:
    - istio-iptables
    - -p
    - "15001"
    - -z
    - "15006"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - "$exclude_cidrs"  # <<-- exclude CIDRs here
    - -b
    - '*'
    - -d
    - 15090,15021,15020
    - --log_output_level=default:info

When applied, both the nginx and istio-proxy containers start properly.

rsalmond Apr 25, 2025
Collaborator

In that case something is very wrong. It should not be possible for a pod in an isolated network namespace to affect containerd running in the host network namespace. What else is unusual about this cluster / pod?

JakubLedworowski Apr 28, 2025
Author

Let me explain here. I don't want to go into too much details, but a diagram is necessary.
The pod is actually a small virtual machine, using a project called kata-containers.
So, the iptables reconfiguration is actually happening on the guest kernel side, not the host.

iptables reconfiguration

On the right there is a regular reconfiguration which happens on the host side. But for kata runtime (left side), the reconfiguration happens on the guest kernel.

image pull

Then, when the app and istio-proxy images are to be pulled, there is a call to a kata-agent's rust crate, called image-rs, that pulls the actual layers and unpacks them on the guest OS. The plugin for containerd, that collects image metadata beforehand, is called nydus-snapshotter.

So, having the above context, the call from image-rs to the registry for pulling app and istio-proxy images must be excluded from redirection via istio-proxy, because it is not yet started.

Do you think there is a mechanism in istio for doing so?

rsalmond Apr 28, 2025
Collaborator

Ahhh yes, this is the information we needed. Thank you for clarifying!

Okay so essentially what you are facing is a variation of the "how do I run an istio sidecar with hostNetwork: true" problem (point 2 in this doc). I am not aware of any standard practice for automatically extending the mesh to K8s managed microvm's. The closest thing I know of is this guide to extending the service mesh to include VM's which are external to the K8s cluster.

I can think of two avenues to explore for this use case.

Create a purpose built solution to add support for the microVM. This could be something like implementing containers running in a network namespace within the microVM, or building your VM images with the Istio VM package pre-installed (assuming that works, try doing it manually first), or something else. I'm not sure what "real" support for this use case should look like.
Work around the image pull problem. This could involve pre-pulling the container images inside the micro VM, or modifying the image pull process so that it bypasses iptables redirection. One way to do this could be to have the image pull process run as UID 1337 which is a special UID used by istio internally and which does not follow the iptables redirection rules.

2 is probably faster, but less robust. You may find other Istio features are still broken after trying this. AFAICT you're breaking new ground here. If there's prior art for this setup, I do not know of it.

Answer selected by JakubLedworowski

JakubLedworowski May 5, 2025
Author

@rsalmond Thank you for your response.

I found the pod annotation that allows to easily configure the bypass: traffic.sidecar.istio.io/excludeOutboundIPRanges
Source: https://istio.io/latest/docs/reference/config/annotations/#SidecarTrafficExcludeOutboundIPRanges

However, like you said, this is not robust, as it needs a DNS resolution by some external service or script at every pod startup time. Unfortunately, as a side-effect, it affects all those use cases, where the registry is used for app functionality in runtime. Pre-pulling all container images before starting istio-init might be a more clear alternative without side effects.

rsalmond May 5, 2025
Collaborator

Yeah those annotations are somewhat blunt instruments. If you have some degree of control over your container registry, one option might be to put it on an oddball port and use https://istio.io/latest/docs/reference/config/annotations/#SidecarTrafficExcludeOutboundPorts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure sidecar installation to bypass calls to container image registry during pod startup #56058

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Configure sidecar installation to bypass calls to container image registry during pod startup #56058

JakubLedworowski Apr 24, 2025

Replies: 2 comments · 7 replies

sridhargaddam Apr 24, 2025 Collaborator

JakubLedworowski Apr 25, 2025 Author

rsalmond Apr 24, 2025 Collaborator

JakubLedworowski Apr 25, 2025 Author

rsalmond Apr 25, 2025 Collaborator

JakubLedworowski Apr 28, 2025 Author

iptables reconfiguration

image pull

rsalmond Apr 28, 2025 Collaborator

JakubLedworowski May 5, 2025 Author

rsalmond May 5, 2025 Collaborator

JakubLedworowski
Apr 24, 2025

Replies: 2 comments 7 replies

sridhargaddam
Apr 24, 2025
Collaborator

JakubLedworowski Apr 25, 2025
Author

rsalmond
Apr 24, 2025
Collaborator

JakubLedworowski Apr 25, 2025
Author

rsalmond Apr 25, 2025
Collaborator

JakubLedworowski Apr 28, 2025
Author

rsalmond Apr 28, 2025
Collaborator

JakubLedworowski May 5, 2025
Author

rsalmond May 5, 2025
Collaborator