ArgoCD repo-server --parallelismlimit recommendations based on argocd repo-server metrics #22080
Unanswered
smashingraph
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello!
I am currently struggling with argocd repo server OOM kills that occurrs quite often on two ArgoCD instances for multi-tenant applications (> ~900 applications on one instance and > ~1600 applications on another)
Based on following documentation: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#scaling-up
It is said to configure parallelismlimit:
Today we have the default set to 0, so no limit, which is bad.
The thing is that I don't really know how much I would need to configure, and don't want to play with that without knowing the impact.
We have enabled HA, with a minimum of 3 replicas and max of 5, with memory requests set to 3Gi and memory limits now set to 10Gi to avoid as much as possible OOM kills, but this is really only a plaster.
We have monitoring enabled and have access to argocd repo server metrics.
So my question would be, which metrics would help to correlate the number of parallels manifest generation and the number that leads to OOM kill ? Would it be the Repository Server Goroutines Usage (argocd.repo_server.go.go_routines ?
Usually it is around 20-22, but we have seen spikes to 422! on one and 50 on the other.
So I am thinking it would be reasonable to set a parallelismlimit to 25 for example ? Is this a correct assumption ?
What are the impact of this limitation by the way, the reconciliation phase would then just be slower I presume right ?
I think I will also have to look at the argocd.argoproj.io/manifest-generate-paths annotation as we have repositories holding multiple applications.
Thanks in advance for some clues about this,
Best Regards,
Beta Was this translation helpful? Give feedback.
All reactions