Understanding Kubernetes Controllers in 2026: A Complete Guide to Self-Healing Infrastructure
Hello everyone, welcome back to our Kubernetes deep dive series!
We've already explored etcd and the API server in our previous blogs. Now it's time to meet the real workers behind Kubernetes' magic - controllers.
These are the tireless pieces of code that never sleep. They're constantly watching the API server through a mechanism called "watch" and making sure your cluster stays exactly how you want it. Crashed pod? Controller fixes it. Need 5 replicas? Controller maintains them.
This is what gives Kubernetes its famous self-healing power. Let's dive in and see how it all works!
What Are Kubernetes Controllers?
Think of controllers as tireless workers who never sleep. Their job is simple but crucial: they constantly check if your cluster looks the way you want it to look. If something's off, they fix it.
Here's the basic idea. You tell Kubernetes "I want 3 copies of my application running." That's your desired state. Controllers then make sure the actual state matches this. If one pod crashes, the controller notices and spins up a new one. No human intervention needed.
This whole approach is what makes Kubernetes self-healing. You're not manually managing servers or restarting failed containers. Controllers handle all of that for you automatically.
How Controllers Actually Work
Controllers run what's called a reconciliation loop. Sounds fancy, but it's pretty straightforward.
The loop works like this:
The controller watches specific resources in your cluster
It compares the current state with what you've declared as desired
If there's a mismatch, it takes action to fix it
Then it goes back to step 1 and repeats forever
Let's say you've deployed an application with 5 replicas. The controller constantly checks: "Are there 5 pods running? Are they healthy? Are they using the right container image?" If a pod dies, the controller immediately notices and creates a replacement.
All these controllers run inside something called the controller manager. It's basically a supervisor process that runs multiple controllers at once. Each controller focuses on a specific type of resource. The Deployment controller handles Deployments, the Job controller handles Jobs, and so on.
The beauty of this system is that controllers don't just run once. They're always watching, always ready to respond. This continuous monitoring is what keeps your applications resilient.
Common Types of Controllers
Kubernetes comes with several built-in controllers, each designed for specific use cases.
Deployment is probably the one you'll use most. It manages your application pods and handles updates smoothly. Want to roll out a new version? The Deployment controller gradually replaces old pods with new ones, making sure your app stays available during the update.
ReplicaSet ensures you have the exact number of pod copies running. Deployments actually use ReplicaSets under the hood. You'll rarely create ReplicaSets directly, but they're doing important work behind the scenes.
StatefulSet is for applications that need stable identities and persistent storage. Think databases or apps that need to remember which pod is which. Unlike Deployments, StatefulSets give each pod a consistent name and storage that sticks around even if the pod gets recreated.
DaemonSet runs one pod on every node in your cluster. It's perfect for things like log collectors or monitoring agents that need to run everywhere.
Job creates pods that run to completion and then stop. Good for batch processing or one-time tasks. Once the work is done, the Job controller marks it as complete.
CronJob is like Job but runs on a schedule. Need to run a backup every night at 2 AM? CronJob has you covered.
Each controller has its own reconciliation logic tuned for its specific purpose. But they all follow the same basic pattern of watching and reconciling.
Custom Controllers and CRDs
Here's where things get really interesting. You're not limited to the built-in controllers. You can write your own.
Custom Resource Definitions (CRDs) let you extend Kubernetes with your own resource types. Once you've defined a CRD, you can write a custom controller to manage it.
Why would you do this? Let's say you're running a database as a service. You could create a custom "Database" resource. Users would create Database objects, and your custom controller would handle all the complexity: provisioning storage, configuring replication, setting up backups, everything.
This is the foundation of the Operator pattern in Kubernetes. Operators are essentially custom controllers that encode operational knowledge about running complex applications. Companies use them to automate everything from database management to machine learning pipelines.
The key insight is that Kubernetes gives you the same building blocks it uses internally. You can extend it to automate whatever workflows make sense for your organization.
How Controllers Communicate
Controllers don't have a direct connection to your pods or nodes. Everything goes through the Kubernetes API server.
The typical flow looks like this:
You create or update a resource (like a Deployment)
The API server stores it in etcd
Controllers watch the API for changes to resources they care about
When something changes, the API notifies the controller
The controller reads the current state and figures out what to do
It sends commands back through the API to make changes
This design is pretty clever. Controllers don't need to know about each other. They just watch the API and react to changes. The API server becomes a central coordination point.
The Watch API is particularly important here. Instead of controllers constantly polling "has anything changed?", they open a long-lived connection. The API server pushes notifications when relevant resources change. This is way more efficient and means controllers can react almost instantly.
When a controller needs to update something, it sends the new desired state to the API server. The API validates it, stores it, and then other controllers can see and react to that change. It's a clean, loosely coupled system.
Building Your Own Controller
Want to build a custom controller? It's more accessible than you might think.
The basic structure looks like this:
First, you define your Custom Resource Definition. This describes what your new resource looks like, what fields it has, and how to validate it.
Then you write the controller logic. The core is the reconciliation function. It takes a resource object and figures out what actions to take. Should it create some pods? Update a service? Delete old resources? That's all up to you.
You'll need to set up watches for the resources you care about. The controller framework handles most of the plumbing. You just register what you want to watch and provide your reconciliation function.
Tools like Kubebuilder and Operator SDK make this much easier. They generate project scaffolding, handle the boilerplate, and let you focus on your actual business logic. Kubebuilder in particular is great because it follows Kubernetes conventions and generates a lot of useful code for you.
A simple controller might be just a few hundred lines of code. You define your CRD, write a reconcile function that creates or updates child resources, add some error handling, and you're done.
The Operator SDK is another excellent option, especially if you're building operators to distribute to others. It includes tools for testing, packaging, and publishing your operator.
Best Practices and Common Pitfalls
Writing controllers requires some care. Here are things that'll save you headaches.
Handle errors properly. Your reconciliation might fail for all kinds of reasons. The API might be temporarily unavailable, resources might not be ready yet, or you might hit a genuine error. Don't panic and crash. Return an error and let the framework retry with backoff.
Make reconciliation idempotent. Your controller might run multiple times for the same change. That's normal. Make sure running reconciliation twice doesn't break things. Check what already exists before creating new resources.
Avoid infinite loops. If your controller updates a resource, and that update triggers another reconciliation, which updates the resource again, you've got a problem. Be careful about what fields you modify and use status subresources when appropriate.
Log everything important. When things go wrong (and they will), good logs are your best friend. Log when you're taking action, what you're doing, and why. But don't spam logs with every tiny check.
Use status subresources. Put the current state information in the status field of your resource. This keeps it separate from the spec (desired state) and prevents infinite reconciliation loops.
Set owner references. When your controller creates child resources, set owner references. This ensures they get cleaned up when the parent is deleted. Kubernetes handles garbage collection for you if you set this up correctly.
Test thoroughly. Write unit tests for your reconciliation logic. Use envtest from controller-runtime to test against a real API server. Controllers are tricky to debug in production, so catch issues early.
For troubleshooting, start with the controller logs. They'll show you what the controller is doing and why. Check the events on your resources too (kubectl describe will show them). Events often reveal why something isn't working.
If a controller seems stuck, check if it's running at all, if it's watching the right resources, and if its service account has the necessary permissions. RBAC issues are a common gotcha.
Wrapping Up
Controllers are the engine that makes Kubernetes tick. They're constantly working to keep your cluster in the state you've declared. This declarative approach is why Kubernetes is so powerful for automation.
The built-in controllers handle common patterns like deployments and jobs. But the real power comes from extensibility. You can write custom controllers to automate whatever operational logic matters for your applications.
Give it a try. Pick something repetitive you do in your cluster and automate it with a controller. You'll learn a ton, and you might just solve a real problem in the process.
