Feature: Ordered Install/Upgrade #375
Comments
This statement sums the problem statement quite well |
|
Today, Application Distributors are more often than not composing applications from ready made charts from bitnami et. al or vendors using charts to distribute their applications. I do not have data to back this up, I'm mostly speaking from experience. With that, we'll see Application Distributors need to package charts that require to be distributed as a single installable entities. Some of these applications need to be composed from external dependencies (databases, caches, object stores etc) that might be separate distinct charts or templates in a chart. Today, having these applications start up in a particular order isn't always a simple thing to achieve. Here is a concrete, but simplified example. I have an application that will conditionally generate a self-singed certificate if one does not exist in a defined secret at start up. I would like to have There are two strategies I can use to allow my application to wait for cert-manager to generate a certificate before starting
With chart resource ordering, the problem I see isn't that getting resources ordered is an impossible task, its the difficulty of achieving it that is the issue. It becomes harder the further away an Application Distributor is to the applications being packaged, especially if they are composing different applications from different charts/projects. |
|
been thinking about this for the last few days (or weeks now). First thought, is I think we should certainly support something with much better UX than using annotations as #373 proposes. For example, representing resource sets (or "layers") in separate directories (though I'm sure there is a better way out there than this example). Where the user can easily identify / support different resource sets, and their order without hunting through annotations in templates. I think #373 proposes annotations as that is perhap the only option for Charts V2 API as we have today. Second thought, is that options 1. and two 2. ie. layers within a chart, or ordering sub-charts are not necessarily exclusive. But obviously they do occupy a large portion of the same solution space. The differences is perhaps whether Helm want to allow users to compose different charts into an "application". But that also might mean artificially breaking a chart into several pieces to support ordering (as describe above). There also might also be a middle group, Third thought, extending on the above. Helm must consider upgrades as much as installs. Maintaining and upgrading software Day 2 is almost always harder than installing on Day 1. IMHO, a single package should always be subject to a single operation (install and upgrade) ie. all layers would be upgraded to a new chart version in a single Helm operation. Whereas, and IMHO especially if orchestrated by higher level software, a chart with ordered sub-chart dependencies could be updated progressively/independently e.g. db the blue/green the app layer. |
|
https://kro.run/docs/overview#resourcegraphdefinition for inspiration |
Layers are used to order resources in one chart. These annotations should not affect ordering resources outside of the same chart/subchart. I call this out 11d56ea
This should not be a problem anymore. See comment above. |
|
I agree it makes sense to scope sequencing options for resources within a chart to only depend on other resources within the same chart. Along those lines, the ability to sequence dependencies for a chart should be only for each entire dependency, not dig into a specific resource within any of those (that can happen with the above annotations). In short, I agree with @mattfarina that orchestration for multiple charts—apart from functionality we already have, like dependencies of a chart—should be considered outside the scope of Helm 4. Other projects in the Helm ecosystem (like Matt said, Hypper, Helmfile, Flux Helm controller) can handle orchestrating multiple, independent charts. |
|
I really like the idea of this Feature request. I'll give this topic some more thought and see if I can have more useful feedback. But I'll give what I can now. One frustration I often have with "just throwing all the manifests at the server all at once" is clumsy scheduling. I'll use the camunda helm chart as an example, because my team and I maintain that. We have a set of pods that handle storage that could be layer 1, that would be postgresql and elasticsearch. then pods that depend on that would be our other apps zeebe, operate, tasklist, optimize ... When all the pods get thrown onto the server at once, I get It usually takes 5-ish rounds of helm-install , helm-uninstall, delete pvcs, helm-install to get all my affinity+antiAffinity rules perfect such that the scheduling puts the pods in this order: With a layered approach, then the database storage could be installed first, and the affinity rules put it on the right nodes, and the other pods wouldn't need antiaffinity rules to be scheduled. Another point is that often customers come to us with logs about why one component fails and they can't see the dependency tree that is apparent in my head. If I had this, Layer 1 would be postgresql, and keycloak . Layer 2 would be zeebe, Layer 3 would be anything that depends on zeebe, like operate. It would always be clear what would need to be debugged first. I definitely see the challenges in implementing this, though. |
My view is that installations could be layered, and upgrades could throw all the manifests at the server all at once (if it makes the problem any simpler to implement). The layered approach doesn't matter as much for upgrades because if I have a dependency (layer 2) that is out-of-date with the database version (layer 1), then pods in layer 2 are already going to be crashlooping. |
|
@gjenkins8 I wanted to reply to your three thoughts here #375 (comment)
|
This approach of only using |
There have been discussions about adding the ability for Helm to do an ordered installation where different parts of the order are applied to Kubernetes at different times.
Currently, Helm sends all the resources (except CRDs) at one time. When there are subcharts, which are embedded in the chart, those are pulled into a shared sandbox, rendered together, and sent at once. Helm does order the resources by type as we have experience that this has mattered in some cases. The reason Helm sends everything at once is that controllers should eventually reconcile and make everything consistent.
This does not always work for applications, in practice.
Configuration management tools can be used to install different parts of an application separately to deal with this problem. But, this does not work for general distribution of applications as charts where end users can
helm installan application.The proposal is to do this with multiple times Helm sends resource to Kubernetes.
There are currently two ideas being discussed:
There may be alternative solutions to this problem and we are happy to discuss them. Neither of these 2 proposals is a solution for all the things.
We are interested in feedback and questions on these two ideas.
The text was updated successfully, but these errors were encountered: