v1.16 Backports 2025-04-15 #38949

tklauser · 2025-04-15T08:40:03Z

k8s/resource: Don't Add to WaitGroup asynchronously #38692 (@joamaki)
cilium: Fix device controller's dependency on netfilter #38777 (@borkmann)
docs: Document L7 policy IPv6 bug #38591 (@gentoo-root)
k8s: fix missing service endpoints delete events #38779 (@marseel) ⚠️ resolved conflicts
docs: how to update existing pool in multi-pool ipam #38539 (@liyihuang) ⚠️ resolved conflicts
operator/networkpolicy: consider all rules when validating policy #38801 (@tklauser)
iptables/ipset: Fix ipset ops panic #38890 (@pippolo84)

Once this PR is merged, a GitHub action will update the labels of these PRs:

 38692 38777 38591 38779 38539 38801 38890

[ upstream commit 2e3c0bb ] The startWhenNeeded() runs in a forked goroutine and thus may run concurrently to a Stop() which calls r.wg.Wait(). Thus we cannot manipulate the WaitGroup anymore. Refactor startWhenNeeded() to call wg.Done() either when short-circuiting or when the informer is stopped so that we don't need to call Add(1). This should fix the following panic: PANIC package: github.com/cilium/cilium/operator/pkg/bgpv2 • TestClusterConfigConditions panic: sync: WaitGroup is reused before previous Wait has returned [recovered] panic: sync: WaitGroup is reused before previous Wait has returned ... sync.(*WaitGroup).Wait(0x416abf?) /root/sdk/go1.24.1/src/sync/waitgroup.go:120 +0x74 github.com/cilium/cilium/pkg/k8s/resource.(*resource[...]).Stop(0x47e40c, {0xc0009d3bc8?, 0x4f0053?}) /host/pkg/k8s/resource/resource.go:356 +0x2a github.com/cilium/hive/cell.(*DefaultLifecycle).Stop(0xc000c44ea0, 0xc000c7f600, {0x3b64078?, 0xc00037dab0?}) /host/vendor/github.com/cilium/hive/cell/lifecycle.go:167 +0x2e6 github.com/cilium/hive.(*Hive).Stop(0xc000c39a40, 0xc000c7f600, {0x3b64078, 0xc00037dab0}) /host/vendor/github.com/cilium/hive/hive.go:374 +0xd9 github.com/cilium/cilium/operator/pkg/bgpv2.TestClusterConfigConditions.func1(0xc000bfc700) /host/operator/pkg/bgpv2/cluster_test.go:872 +0x358 I validated that this fixes the issue by doing an early return in the test case to force Resource[T].Stop() to race with Resource[T].startWhenNeeded: > --- a/operator/pkg/bgpv2/cluster_test.go > +++ b/operator/pkg/bgpv2/cluster_test.go > @@ -842,6 +842,8 @@ func TestClusterConfigConditions(t *testing.T) { > watchersReady() > + return I ran this short-circuited test 1000 times with "stress". Signed-off-by: Jussi Maki <jussi@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

[ upstream commit 1dd7cea ] When trying to disable netfilter in the kernel and starting up Cilium with the below options, it fails as follows: kernel# cat .config | grep NETFILTER # CONFIG_NETFILTER is not set # ./daemon/cilium-agent --enable-ipv4=true --enable-ipv6=false \ --bpf-lb-algorithm=maglev --bpf-lb-maglev-table-size=2039 \ --bpf-lb-mode=dsr --bpf-lb-acceleration=native --devices=enp5s0 \ --bpf-lb-dsr-dispatch=ipip --disable-envoy-version-check=true \ --k8s-kubeconfig-path=$HOME/.kube/config \ --kube-proxy-replacement=true --routing-mode=native \ --enable-ipv4-masquerade=false --ipam=cluster-pool \ --enable-ipip-termination --install-iptables-rules=false \ --enable-l7-proxy=false [...] time=2025-04-07T10:40:05Z level=info msg="All Cilium CRDs have been found and are available" module=agent.infra.k8s-synced-crdsync time="2025-04-07T10:40:05.954428144Z" level=info msg="Local boot ID is \"5efaef07-4a21-4db1-82e7-05c73687cff8\"" subsys=node time=2025-04-07T10:40:05Z level=error msg="Start hook failed" function="*linux.devicesController.Start (agent.datapath.devices-controller)" error="creating netlink handle: protocol not supported" time=2025-04-07T10:40:05Z level=error msg="Failed to start hive" error="creating netlink handle: protocol not supported" duration=57.406398ms time=2025-04-07T10:40:05Z level=info msg="Stopping hive" time=2025-04-07T10:40:05Z level=info msg="agent.datapath.sysctl.job-reconcile (rev=8)" module=health time=2025-04-07T10:40:05Z level=info msg="agent.datapath.sysctl.job-refresh (rev=9)" module=health time=2025-04-07T10:40:05Z level=info msg="agent.infra.k8s-synced-crdsync.job-sync-crds (rev=7)" module=health time="2025-04-07T10:40:05.955728969Z" level=info msg="Stopped gops server" address="127.0.0.1:9890" subsys=gops time="2025-04-07T10:40:05.955744039Z" level=fatal msg="failed to start: creating netlink handle: protocol not supported" subsys=daemon The device controller itself doesn't need anything netfilter related, thus fix up the handle. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>

[ upstream commit bcbc27e ] Signed-off-by: Maxim Mikityanskiy <maxim@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

[ upstream commit d82ee40 ] In case of problem with k8s connectivity, Reflector performs relist operation and DeltaFIFO generates Sync events for already existing elements and Delete events for removed elements. However, we were not handling events DeletedFinalStateUnknown which resulted in stale cache for services. Signed-off-by: Marcel Zieba <marcel.zieba@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

[ upstream commit 181d574 ] Add a section to help users to know how to update the existing ip pool in multi-pool ipam mode since it's not editable. Signed-off-by: Liyi Huang <liyi.huang@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

[ upstream commit a92cbc7 ] Currently, only the validation result of the last policy rule is propagated to the CiliumNetworkPolicy/CiliumClusterwideNetworkPolicy status field. Fix this by joining all sanitization errors instead of just the last one in a list of rules. Fixes #38795 Signed-off-by: Tobias Klauser <tobias@cilium.io>

[ upstream commit 5b22071 ] The new test exercises the reconciler when a batched operation fails and a retry is attempted. When retrying, the cilium/statedb reconciler calls the sequential operations (e.g: `Update` instead of `UpdateBatch`) for each item in the failed batch. Since the sequential operations are not expected to be called in the current ipset implementation, the reconciler panics: === RUN TestOpsRetry panic: Unexpectedly Update() called for reconciliation goroutine 53 [running]: github.com/cilium/cilium/pkg/datapath/iptables/ipset.(*ops).Update(0xc000929000?, {0x7f2a50ec86d8?, 0x7f2a97b32a78?}, {0x70?, 0xc000780008?}, 0xc0006508c0?) /home/pippolo/cilium/cilium/pkg/datapath/iptables/ipset/ops.go:77 +0x25 github.com/cilium/statedb/reconciler.(*incrementalRound[...]).processSingle(0x37926c0, 0xc000929000, 0x2, 0x0) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/incremental.go:214 +0x170 github.com/cilium/statedb/reconciler.(*incrementalRound[...]).processRetries(0x37926c0) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/incremental.go:190 +0x58 github.com/cilium/statedb/reconciler.(*reconciler[...]).incremental(0x3778660, {0x37469f0, 0xc000952ab0}, {0x371ddb0, 0xc0006443c0}, 0xc0006303a0) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/incremental.go:73 +0x2de github.com/cilium/statedb/reconciler.(*reconciler[...]).reconcileLoop(0x3778660, {0x37469f0, 0xc000952ab0}, {0x374b780, 0xc000988000}) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/reconciler.go:92 +0x568 github.com/cilium/hive/job.(*jobOneShot).start(0xc00093cde0, {0x37469f0, 0xc000952ab0}, 0x0?, {0x374b780, 0xc00093cae0}, {{{0x0, 0x0, 0x0}}, 0xc0003493e0, ...}) /home/pippolo/cilium/cilium/vendor/github.com/cilium/hive/job/oneshot.go:138 +0x4fd created by github.com/cilium/hive/job.(*group).Start.func1 in goroutine 50 /home/pippolo/cilium/cilium/vendor/github.com/cilium/hive/job/job.go:167 +0x172 FAIL github.com/cilium/cilium/pkg/datapath/iptables/ipset 0.233s FAIL This will be fixed in a subsequent commit. Related: 7b80f94 ("iptables/ipset: Use batch operations") Signed-off-by: Fabio Falzoi <fabio.falzoi@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

tklauser · 2025-04-15T08:40:30Z

/test

pippolo84 · 2025-04-15T08:59:07Z

@tklauser caould you please apply the following diff to #38890 backport?

diff --git a/pkg/datapath/iptables/ipset/ipset_test.go b/pkg/datapath/iptables/ipset/ipset_test.go
index 9df1cc3728..f8fa7bec8f 100644
--- a/pkg/datapath/iptables/ipset/ipset_test.go
+++ b/pkg/datapath/iptables/ipset/ipset_test.go
@@ -11,6 +11,7 @@ import (
        "fmt"
        "html/template"
        "io"
+       "log/slog"
        "net/netip"
        "strings"
        "sync/atomic"
@@ -461,7 +462,7 @@ func TestOpsRetry(t *testing.T) {
                        newOps,
                        newReconciler,
                ),
-               cell.Provide(func(logger *slog.Logger) *ipset {
+               cell.Provide(func(logger logrus.FieldLogger) *ipset {
                        return &ipset{
                                executable: funcExecutable(func(ctx context.Context, command string, stdin string, arg ...string) ([]byte, error) {
                                        // fail the operation at the first attempt
@@ -486,7 +487,7 @@ func TestOpsRetry(t *testing.T) {
 
        log := hivetest.Logger(t, hivetest.LogLevel(slog.LevelError))
 
-       require.NoError(t, hive.Start(log, t.Context()))
+       require.NoError(t, hive.Start(log, context.Background()))
 
        obj := &tables.IPSetEntry{
                Name:   CiliumNodeIPSetV4,
@@ -514,7 +515,7 @@ func TestOpsRetry(t *testing.T) {
                <-watch
        }
 
-       require.NoError(t, hive.Stop(log, t.Context()))
+       require.NoError(t, hive.Stop(log, context.Background()))
 }
 
 func TestIPSetList(t *testing.T) {

This should make the test compatible with the stable branch.

[ upstream commit 0491518 ] When a cilium/statedb reconciler relies on batched operations and a failure happens, the subsequent retry calls the non-batched operations (e.g: `Update` instead of `UpdateBatch`). Therefore, to avoid a panic in case of a retry, we need to define the non-batched operations too for the ipset reconciler. Fixes: 7b80f94 ("iptables/ipset: Use batch operations") Signed-off-by: Fabio Falzoi <fabio.falzoi@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

tklauser · 2025-04-15T09:15:48Z

@tklauser caould you please apply the following diff to #38890 backport?
[...]

@pippolo84 done, thanks for the patch!

tklauser · 2025-04-15T09:15:57Z

/test

pippolo84

Thanks!

[ upstream commit 5b22071 ] [ backporter's note: applied #38949 (comment) ] The new test exercises the reconciler when a batched operation fails and a retry is attempted. When retrying, the cilium/statedb reconciler calls the sequential operations (e.g: `Update` instead of `UpdateBatch`) for each item in the failed batch. Since the sequential operations are not expected to be called in the current ipset implementation, the reconciler panics: === RUN TestOpsRetry panic: Unexpectedly Update() called for reconciliation goroutine 53 [running]: github.com/cilium/cilium/pkg/datapath/iptables/ipset.(*ops).Update(0xc000929000?, {0x7f2a50ec86d8?, 0x7f2a97b32a78?}, {0x70?, 0xc000780008?}, 0xc0006508c0?) /home/pippolo/cilium/cilium/pkg/datapath/iptables/ipset/ops.go:77 +0x25 github.com/cilium/statedb/reconciler.(*incrementalRound[...]).processSingle(0x37926c0, 0xc000929000, 0x2, 0x0) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/incremental.go:214 +0x170 github.com/cilium/statedb/reconciler.(*incrementalRound[...]).processRetries(0x37926c0) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/incremental.go:190 +0x58 github.com/cilium/statedb/reconciler.(*reconciler[...]).incremental(0x3778660, {0x37469f0, 0xc000952ab0}, {0x371ddb0, 0xc0006443c0}, 0xc0006303a0) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/incremental.go:73 +0x2de github.com/cilium/statedb/reconciler.(*reconciler[...]).reconcileLoop(0x3778660, {0x37469f0, 0xc000952ab0}, {0x374b780, 0xc000988000}) /home/pippolo/cilium/cilium/vendor/github.com/cilium/statedb/reconciler/reconciler.go:92 +0x568 github.com/cilium/hive/job.(*jobOneShot).start(0xc00093cde0, {0x37469f0, 0xc000952ab0}, 0x0?, {0x374b780, 0xc00093cae0}, {{{0x0, 0x0, 0x0}}, 0xc0003493e0, ...}) /home/pippolo/cilium/cilium/vendor/github.com/cilium/hive/job/oneshot.go:138 +0x4fd created by github.com/cilium/hive/job.(*group).Start.func1 in goroutine 50 /home/pippolo/cilium/cilium/vendor/github.com/cilium/hive/job/job.go:167 +0x172 FAIL github.com/cilium/cilium/pkg/datapath/iptables/ipset 0.233s FAIL This will be fixed in a subsequent commit. Related: 7b80f94 ("iptables/ipset: Use batch operations") Signed-off-by: Fabio Falzoi <fabio.falzoi@isovalent.com> Signed-off-by: gray <greyschwinger@gmail.com> s

joamaki and others added 7 commits April 15, 2025 10:35

docs: Document L7 policy IPv6 bug

587fb4d

[ upstream commit bcbc27e ] Signed-off-by: Maxim Mikityanskiy <maxim@isovalent.com> Signed-off-by: Tobias Klauser <tobias@cilium.io>

tklauser added kind/backports This PR provides functionality previously merged into master. backport/1.16 This PR represents a backport for Cilium 1.16.x of a PR that was merged to main. labels Apr 15, 2025

tklauser requested review from borkmann, gentoo-root, marseel, liyihuang, pippolo84 and joamaki April 15, 2025 08:40

tklauser marked this pull request as ready for review April 15, 2025 08:40

tklauser requested a review from a team as a code owner April 15, 2025 08:40

tklauser enabled auto-merge April 15, 2025 08:40

tklauser force-pushed the pr/v1.16-backport-2025-04-15-10-35 branch from 0e49224 to 1d5fdfc Compare April 15, 2025 09:15

pippolo84 approved these changes Apr 15, 2025

View reviewed changes

marseel approved these changes Apr 15, 2025

View reviewed changes

liyihuang approved these changes Apr 15, 2025

View reviewed changes

joamaki approved these changes Apr 17, 2025

View reviewed changes

tklauser added this pull request to the merge queue Apr 17, 2025

Merged via the queue into v1.16 with commit c98100f Apr 17, 2025
275 checks passed

tklauser deleted the pr/v1.16-backport-2025-04-15-10-35 branch April 17, 2025 09:44

pippolo84 mentioned this pull request Apr 22, 2025

v1.17 Backports 2025-04-22 #39075

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

v1.16 Backports 2025-04-15 #38949

v1.16 Backports 2025-04-15 #38949

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1.16 Backports 2025-04-15 #38949

v1.16 Backports 2025-04-15 #38949

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!