I recently read this great piece by Timur Tukaev discussing how to approach the growing complexity of Kubernetes. You can read it here. Basically as Kubernetes continues to expand to be the everything platform, the amount of functionality it contains is taking longer and longer to learn.
Right now every business that adopts Kubernetes is basically rolling their own bespoke infrastructure. Timur's idea is to try and solve this problem by following the Linux distro model. You'd have groups of people with similar needs work together to make an out-of-the-box Kubernetes setup geared towards their specific needs. I wouldn't start from a blank cluster, but a cluster already configured for my specific usecase (ML, web applications, batch job processing).
I understand the idea, but I think the root cause of all of this is simply a lack of a meaningful package management system for Kubernetes. Helm has done the best it can, but practically speaking it's really far from where we would need to be in order to have something even approaching a Linux package manager.
More specifically we need something between the very easy to use but easy to mess up Helm and the highly bespoke and complex to write Operator concept.
- Centralized State Management
- Maintain a robust, centralized state store for all deployed resources, akin to a package database.
- Provide consistency checks to detect and reconcile drifts between the desired and actual states.
- Advanced Dependency Resolution
- Implement dependency trees with conflict resolution.
- Ensure dependencies are satisfied dynamically, including handling version constraints and providing alternatives where possible.
- Granular Resource Lifecycle Control
- Include better support for orchestrating changes across interdependent Kubernetes objects.
- Secure Packaging Standards
- Enforce package signing and verification mechanisms with a centralized trust system.
- Include better support for orchestrating changes across interdependent Kubernetes objects.
- Native Support for Multi-Cluster Management
- Allow packages to target multiple clusters and namespaces with standardized overrides.
- Provide tools to synchronize package versions across clusters efficiently.
- Rollback Mechanisms
- Improve rollback functionality by snapshotting cluster states (beyond Helm’s existing rollback features) and ensuring consistent recovery even after partial failures.
- Declarative and Immutable Design
- Introduce a declarative approach where the desired state is managed directly (similar to GitOps) rather than relying on templates.
- Integration with Kubernetes APIs
- Directly leverage Kubernetes APIs like Custom Resource Definitions (CRDs) for managing installed packages and versions.
- Provide better integration with Kubernetes-native tooling (e.g., kubectl, kustomize).
Again a lot of this is Operators, but Operators are proving too complicated for normal people to write. I think we could reuse a lot of that work, keep that functionality and create something similar to what the Operator allows you to do with less maintenance complexity.
Still I'd love to see the Kubernetes folks do anything in this area. The current state of the world is so bespoke and frankly broken there is a ton of low hanging fruit in this space.