[SURE-9419] Move agent registration into controller container #3416
Conversation
ff27a4b to
9caf17e
Compare
| @@ -125,6 +124,15 @@ func (a *FleetAgent) Run(cmd *cobra.Command, args []string) error { | |||
| RenewDeadline: *leaderOpts.RenewDeadline, | |||
| Callbacks: leaderelection.LeaderCallbacks{ | |||
| OnStartedLeading: func(ctx context.Context) { | |||
There was a problem hiding this comment.
Right, so if I properly understand the new process, the (single) agent container will seek to acquire a leader election lease and, once it has it, will register the cluster before proceeding to running the agent reconcilers. 👍
In my understanding (just checking if it is aligned with yours), the next step would involve launching a goroutine, in parallel to running the reconcilers, for reporting cluster status. Happy to discuss if I've misunderstood any of this ;)
There was a problem hiding this comment.
Right, so if I properly understand the new process, the (single) agent container will seek to acquire a leader election lease and, once it has it, will register the cluster before proceeding to running the agent reconcilers. 👍
Correct.
In my understanding (just checking if it is aligned with yours), the next step would involve launching a goroutine, in parallel to running the reconcilers, for reporting cluster status. Happy to discuss if I've misunderstood any of this ;)
As we've discussed, my initial idea was to make it an ordinary reconciler and use requeue to immitate a ticker, but now I'm going to put the functionality into a (controller-runtime) Runnable and have the manager execute it. Then I'll take care of migrating the other parts from wrangler/lasso to controller-runtime.
76e12dd to
63e038a
Compare
63e038a to
87714f9
Compare
Since we have changed the agent from using a StatefulSet to a Deployment, we also needed and have implemented leader election for every container of the agent pod. This ensures that
even if the replicas are temporarily (as in a rollout) or permanently (as in replica size) above one.
Now, if replicas are to be set to anything above one, this will currently lead to the agent of the second pod to perform the cluster registration, even though the agent cannot run after it completed the registration. The cluster registration will complete, but the controller will wait for the lease. This state potentially leads to confusion, as there are cluster registrations in the upstream cluster that have never and may never be used. Apparently this state is also considered inconsistent with the UI. Lastly, we use more leases than necessary.
While theoretically possible that the name for the lease is the same for the cluster registration container as well as for the agent controller, this may lead to extended waiting times, as the leases need to be acquired. Also, it is not guaranteed that the agent controller of the same pod will get the lease after the cluster registration container has let go of the lease , potentially leading to the same problem with unused cluster registrations (imagine replica size 3 from 1).
For those reasons the agent registration is to be moved into the controller container.
Part of #3377