Many of the posts I read here are about Docker. Is anybody using Kubernetes to manage their self hosted stuff? For those who’ve tried it and went back to Docker, why?
I’m doing my 3rd rebuild of a K8s cluster after learning things that I’ve done wrong and wanted to start fresh, but when enhancing my Docker setup and deciding between K8s and Docker Swarm, I decided on K8s for the learning opportunities and how it could help me at work.
What’s your story?
I was looking into converting my docker services into a cluster to get high availability and to learn it for work, but while investigating it, I read that kubernetes is actually meant for scalability and just a single service per cluster.
Also read that docker swarm is actually what is recommended for my homelab use case. So I’m right now on my way to convert everything to docker stacks. What do you think?
I’m not sure what you mean by that.
It provides high availability if you have multiple nodes and pods.
Also what do you mean by single service per cluster? Because that’s not the idea at all.
Of course high availability always requires multiple nodes.
Its just that while choosing how to set up my cluster I looked up several options (proxmox, swarm, kubernetes…) and I noticed that kubernetes is generally meant for bigger deployments.
I only need a single replica for each of my containers and they can all run on a single node, so kubernetes is overkill just to get high availability For my use case
Kubernetes is useful if you have gone full cattle over pets. And that is very uncommon in home setups. If you only own one or two small machines you cannot destroy infra easily in a “cattle” way, and the bloatware that comes with Kubernetes doesn’t help you neither.
In homelabs and home servers the pros of Kubernetes are not very useful: high availability, auto-scaling, gitops integrations, etc: Why would you need autoscaling and HA for a SFTP used only by you? Instead you write a docker-compose.yml and call it a day.
While you’re probably right overall, there are many good reasons to use k8s. The api provides all sorts of benefits. Kubectl, k9s, and other operational UIs . Good deployment models and tools like argo. Loads of helm charts that are (theoretically) ready to use.
No, those things aren’t free. There’s a lot of overhead to running k8s.
This mostly, I haven’t seen a compelling reason to leave my docker setup.
I think the biggest reasons for me have been growth and professional development. I started my home cluster 8 years ago as a single node of basically just running the hack/ scripts on my Linux desktop. I’ve been able to grow that same cluster to 6 hosts as I’ve replaced desktops and as I got a bit into the used enterprise server scene. I’ve replaced multiple routers and moved behind cloudflare, added a private CA a few times, added solid persistence with rook+ceph, and built my ideal telemetry stack, added velero backups into Backblaze b2, and probably a lot more I’m not thinking of.
That whole time, I’ve had to do almost zero maintenance or upgrades on the side projects I’ve built over the years, or on the self hosted services I’ve run. If you ignore the day or so a year I’ve spent cursing my propensity to upgrade a tad too early and hit snags, though I’ve just about always been able to resolve them pretty quickly and have learned even more from those times.
And on top of that, I get to take a lot of that expertise to work where it happens to pay quite well. And I’ve spent some time working towards building the knowledge into a side gig. Maybe someday that’ll pay the bills too.
One line from your comment struck a chord. The part about maintenance and upgrades. I feel like I get stuff set up and working and go about my life and then a failure happens at the most inopportune moment. Mostly, the failures are when I have a few hours free and decide to upgrade the OS and everything breaks and all the dependencies fall apart and some feature is no longer supported. That’s where I started looking to K8s to just roll back until I have time to manage it.
The one exception to this is if you’re using your homelab to learn kubernetes.
That was the only time I used K8s and k3s on my homelab.
And for anything that I do want to set up in a HA/cattle kind of way, I use Docker Swarm, as it feels like a more comfortable extension of docker compose.
I like the Kubes
Running an RKE cluster as VMs on my ceph+proxmox cluster. Using Rook and external ceph as my storage backend and loving it. I haven’t fully migrated all of my services, but thus far it’s working well enough for me!
I like the concept, but hate the configuration schema and tooling which is all needlessly obtuse (eg. helm)
Helm is one of the reasons I became interested in Kubernetes. I really like the idea of a package where all I have to do is provide my preferences in a values file. Before swarm was mature, I was managing my containers with complicated shell scripts to bring stuff up in the right order and it became fragile and unmaintainable.
