[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

argocd-redis secret-init initcontainer timeout #18356

Closed
ziouf opened this issue May 22, 2024 · 16 comments
Closed

argocd-redis secret-init initcontainer timeout #18356

ziouf opened this issue May 22, 2024 · 16 comments
Assignees
@leoluz @pasha-codefresh
Labels
bug/severity:major Malfunction in one of the core component, impacting a majority of users bug Something isn't working

Comments

@ziouf
Copy link
ziouf commented May 22, 2024

Describe the bug

secret-init initContainer is failing to check/init argocd/argocd-redis secret

To Reproduce

kubectl apply -k https://github.com/argoproj/argo-cd/raw/master/manifests/install.yaml

Expected behavior

InitContainer should succeed to check/init secret argocd/argocd-redis

Screenshots

Version

$ argocd version
argocd: v2.11.0+bc53266
  BuildDate: 2024-05-21T22:20:05Z
  GitCommit: bc53266591b632f1a1639ae458f31467446ffe48
  GitTreeState: clean
  GoVersion: go1.22.1
  Compiler: gc
  Platform: linux/amd64

Logs

$ kubectl describe pod/argocd-redis-565687fb7d-68xdd -n argocd 
Name:             argocd-redis-565687fb7d-68xdd
Namespace:        argocd
Priority:         0
Service Account:  argocd-redis
Node:             REDACTED
Start Time:       Wed, 22 May 2024 09:16:31 +0200
Labels:           app.kubernetes.io/name=argocd-redis
                  pod-template-hash=565687fb7d
Annotations:      cni.projectcalico.org/containerID: faff595beac8d993968609af65d1e133d01fa9b970a0e482301ef9b1b55e0b15
                  cni.projectcalico.org/podIP: 100.64.182.116/32
                  cni.projectcalico.org/podIPs: 100.64.182.116/32
                  kubectl.kubernetes.io/restartedAt: 2024-05-14T08:33:16+02:00
Status:           Pending
SeccompProfile:   RuntimeDefault
IP:               100.64.182.116
IPs:
  IP:           100.64.182.116
Controlled By:  ReplicaSet/argocd-redis-565687fb7d
Init Containers:
  secret-init:
    Container ID:    containerd://2e686a331f4f9a483160e70875f0057734a85a63065e48841b974ab5160957ee
    Image:           quay.io/argoproj/argocd:latest
    Image ID:        quay.io/argoproj/argocd@sha256:717a945c52f15cef5659b94bba3ab360f5a7b86685a978ace448f76f71063231
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      argocd
      admin
      redis-initial-password
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    20
      Started:      Wed, 22 May 2024 09:25:21 +0200
      Finished:     Wed, 22 May 2024 09:25:51 +0200
    Ready:          False
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nftnx (ro)
Containers:
  redis:
    Container ID:  
    Image:         docker.io/library/redis:7.0.15-alpine
    Image ID:      
    Port:          6379/TCP
    Host Port:     0/TCP
    Args:
      --save
      
      --appendonly
      no
      --requirepass $(REDIS_PASSWORD)
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      REDIS_PASSWORD:  <set to the key 'auth' in secret 'argocd-redis'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nftnx (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-nftnx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors: 
Tolerations:         node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned argocd/argocd-redis-565687fb7d-68xdd to REDACTED
  Normal   Pulled     9m23s (x5 over 12m)   kubelet            Container image "quay.io/argoproj/argocd:latest" already present on machine
  Normal   Created    9m23s (x5 over 12m)   kubelet            Created container secret-init
  Normal   Started    9m23s (x5 over 12m)   kubelet            Started container secret-init
  Warning  BackOff    2m45s (x31 over 11m)  kubelet            Back-off restarting failed container secret-init in pod argocd-redis-565687fb7d-68xdd_argocd(e00bcbf4-853f-44fe-8302-82f0997884a9)


$ kubectl logs pod/argocd-redis-565687fb7d-68xdd -c secret-init  -n argocd
Checking for initial Redis password in secret argocd/argocd-redis at key auth. 
time="2024-05-22T07:25:51Z" level=fatal msg="Post \"https://10.32.0.1:443/api/v1/namespaces/argocd/secrets\": dial tcp 10.32.0.1:443: i/o timeout"
@ziouf ziouf added the bug Something isn't working label May 22, 2024
@otakakot
Copy link
otakakot commented May 22, 2024 •
edited
Loading

I also encountered the same phenomenon by following stable, installed v2.11.0 and rolled back.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml

@otherguy
Copy link

Same here, also rolled back to 2.11.0

@NAVRockClimber
Copy link

Same here

@liron-telemessage
Copy link
liron-telemessage commented May 22, 2024 •
edited
Loading

Hi,
The issue is that Argocd tries to create a secret for Redis via Kubernetes API.
the current network policy does not allow Argocd Redis sever to even contact the Kubernetes API.
as a workaround we temporarily changed the network policy for redis: argocd-redis-network-policy:
and changed to pod selector:
podSelector:
matchLabels:
app.kubernetes.io/name: argocd-redis

change the selector to something like app.kubernetes.io/name: argocd-redis-tmp

after that delete the Redis pod, it will now be able to create the secret and the system will start running.
once everything is running, revert the network policy selector (remove "-tmp").

@Tomasz-Marciniak
Copy link
Tomasz-Marciniak commented May 22, 2024 •
edited
Loading

Download install.yaml and add Kubernetes API port (in my case 16443) to network policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-redis-network-policy
spec:
  egress:
    - ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
        - port: 16443
          protocol: TCP

@yyzxw yyzxw mentioned this issue May 22, 2024
Closed
14 tasks
@yyzxw
Copy link
Contributor
yyzxw commented May 22, 2024

same here, fixed by #18358

@RispyCZ
Copy link
RispyCZ commented May 22, 2024

I have same issue with argocd-redis-ha-haproxy

@ftmiro
Copy link
ftmiro commented May 22, 2024

We faced the same problem and also patched the argocd-redis-ha-proxy-network-policy NetworkPolicy as workaround:

  - ports:
    - port: 443
      protocol: TCP

@todaywasawesome todaywasawesome added the bug/severity:major Malfunction in one of the core component, impacting a majority of users label May 22, 2024
@yz89122 yz89122 mentioned this issue May 22, 2024
Closed
14 tasks
george-angel added a commit to utilitywarehouse/system-manifests that referenced this issue May 22, 2024
@george-angel
Allow access to APIServer to fetch Redis Secret
677f08b 
argoproj/argo-cd#18356
@george-angel george-angel mentioned this issue May 22, 2024
Merged
george-angel added a commit to utilitywarehouse/system-manifests that referenced this issue May 22, 2024
@george-angel
Allow access to APIServer to fetch Redis Secret (#321)
c384b45 
argoproj/argo-cd#18356
@ojasgo
Copy link
ojasgo commented May 23, 2024

In my case I had to use v2.11.0 and also update NetworkPolicy as above mentioned.

@ngaxavi
Copy link
ngaxavi commented May 23, 2024

i had the same issue and patched the argocd-redis-network-policy Networkpolicy as @Tomasz-Marciniak suggested.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-redis-network-policy
spec:
  egress:
    - ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
        - port: 6443 # your kubernetes api port
          protocol: TCP
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: argocd-server
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: argocd-repo-server
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: argocd-application-controller
      ports:
        - port: 6379
          protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-redis
  policyTypes:
    - Ingress
    - Egress

@gadgetmg gadgetmg mentioned this issue May 27, 2024
Merged
@sambonbonne
Copy link

I think I had the same issue on 2.11.1 (after an upgrade from 2.10) and the upgrade to 2.11.2 fixed it for me.

@pasha-codefresh
Copy link
Member

Fixed in 2.11.2

@sirTangale
Copy link

issue persist in helm install

@mkilchhofer
Copy link
Member

issue persist in helm install

@sirTangale Can you raise an issue in argo-helm, please? 😃

Also I'd like to mention that the helm chart is handling it slightly different (via a helm hook). Before Helm will deploy the core components of Argo CD (server / repo-server / ...), Helm will wait until the Secret is generated (the Job runs to completion without errors).

I highly appreciate detailed steps to reproduce.

@mmarisetty mmarisetty mentioned this issue Sep 19, 2024
Merged
6 tasks
@bestrocker221
Copy link

Issue still persists in 2.12.6

@DjarallahBrahim DjarallahBrahim mentioned this issue Nov 29, 2024
Open
3 tasks
@DjarallahBrahim
Copy link

hello

same isssue on GKE with argocd-server:

W1129 16:07:32.309506 7 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.11/tools/cache/reflector.go:169: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:argocd:argocd-server" cannot list resource "configmaps" in API group "" in the namespace "argocd" E1129 16:07:32.309849 7 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.11/tools/cache/reflector.go:169: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:argocd:argocd-server" cannot list resource "configmaps" in API group "" in the namespace "argocd"

i tried to add egresse rule for argocd-repo-server-network-policy but it did not work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leoluz leoluz

@pasha-codefresh pasha-codefresh

Labels
bug/severity:major Malfunction in one of the core component, impacting a majority of users bug Something isn't working
Projects
None yet
Milestone
No milestone