Why I Moved Most Of My Services To Kubernetes

I'll be the first to admit, I love having bare-bones servers (virtual or physical) that I can put my projects on, and in fact this blog used to run on one. The problem is, I also like to keep to a good update schedule, and I hate downtime. The other issue is, I have a tendency to turn a virtual machine into a multi-function system, rather than having it do one job.

"But what's that got to do with moving to Kubernetes?!" I hear you say. Well, downtime is the primary driver. I pride myself on running highly available services whether it's AstroKit, or this blog, and by all means I have the ability to create multiple virtual machines and I do. The problem is, virtual machines come at the cost of resources even with the use of ballooning devices, thin disks, and so on, as well as having to install an OS, integrate the new VM into the CINC estate, and hook it up to monitoring properly. When you have a habit of creating little side projects like I do, this ends up becoming tedious, and that leads to the second issue of turning a virtual machine destined to run a single project into a mess of projects and NGINX configurations, as well as confusing HAProxy routing (why is AstroKit pointing to the same server as my personal blog?!).

And that is where Kubernetes comes in. I already had a cluster from experimenting, and migrating most of my projects to containers was fairly straightforward, especially those built using Node.js or Java. PHP projects were a bit more involved, but once I had a rough base Dockerfile, even that became trivial work. So, I got to work putting each on a private Docker Hub organisation, configured my cluster to be able to fetch from Docker Hub, and began setting the projects up. That's where yet another benefit comes in: Namespaces, jobs, and deployments. Whilst we can organise VMs in various ways, especially with CINC by tagging them in specific ways, namespaces, jobs, and deployments make management insanely easy. Take AstroKit for example, it has 4 distinct components:

• The front-facing web API - Allows the app to access the various features
• Catalogue Service - Contains the logic and astronomical calculations for calculating deep sky object positions from a given location on Earth at any given time, as well as the database itself (over 13,000 objects)
• Weather Service - Handles weather models and outputting the latest forecast data
• Geocoding Service - Handles converting coordinates to actual named places with R-Tree lookups

These components all live under the astrokit namespace, and then each has its own deployment. This means that when I need to update the catalogue service for whatever reason, I can simply push a new container to Docker Hub, update the version in Kubernetes, and trigger a rolling update - With the right rollout configuration and readiness checks, Kubernetes can deploy the new version without downtime, keeping the existing instance available until its replacement is ready to receive traffic. Deploying that change to a virtual machine on the other hand would require either packaging a new bundle, or pulling directly from git and rebuilding, and then restarting the application on each server. And if something breaks? Without a proper rolling deployment mechanism in place, that server could be out of service until the change is rolled back.

The other advantage is the CronJob functionality. The weather service needs to pull new files every half hour, and thanks to CronJobs, Kubernetes handles scheduling that work, including choosing an appropriate node for the resulting Job. If 02 doesn't have much resource availability, it can simply schedule the job on another node - everything is linked together using an NFS mount. That scheduling also means that multiple instances can be spread across the cluster - one node goes down, the app stays alive (when configured correctly, at least), and that becomes a huge benefit when the cluster needs updating. I simply have to drain the node being updated, perform the update, then uncordon it and let Kubernetes schedule workloads onto it again. No having to take servers out of a pool, no having to check that HAProxy knows that a server is unreachable.

All of this also contributes to the final issue I have - multi-purpose servers. Kubernetes becomes the multi-purpose server itself... Which is exactly what it should be. As long as I maintain good hygiene with namespaces, resource limits, and deployment configurations, I can manage everything easily, and I can spin more pods up in seconds, or a completely new project in minutes at the most. Of course, Kubernetes isn't free of complexity. There's the cluster itself to maintain, networking to consider, persistent storage to manage, and an entirely different set of operational problems to learn. In my case, I already had a cluster running, so the additional complexity was minimal compared to the maintenance overhead I was eliminating.