Moving from a single machine with Docker to a cluster of Pi’s

I decided to finally make use of my four Raspberry Pi model 3’s and take the challenge to move all of my home services to them. Previously, I ran a x86 Desktop as a server in my living room. The loud noises coming from the server made it uncomfortable to be in sometimes. The loud noisy box is home to this website and many other applications such as Plex, Transmission, OpenVPN, Jenkins, Samba, and various Node.js projects all running in Docker. Having all of those applications running on a single box is a single point of failure and makes system administration harder when reboots are required.

To make administration easier, I decided that one Pi should be a load balancer for as many applications as possible. Yes, I know that having a single Pi as a load balancer is also a single point of failure but it makes administrating the other Pi’s easier. I researched how to do HTTP and TCP load balancing with NGINX and made a Docker container for it which runs on one Pi.

Now I needed to think about where to run all of these containers and made a mental map of where to run them. I decided the best way for deploying containers would be through a private local registry so I created a Docker registry on one of the Pi’s and pushed all of the images. Let’s take a look at the application architecture to see what each Raspberry Pi is doing.

Now I have all of my applications from the loud server running on the quiet Pi’s and many of the containers are load balanced. The next task is to figure out a way to manage the containers and automate the image building process. I wrote a complex Bash script that accomplishes my requirements to manage the little cluster of Pi’s using many SSH and GIT commands. I found Git to be the easiest way for me to manage my HTML and configuration files. Since BitBucket offers private Git repositories for free, I used them. If I need to make any changes, my Bash script will do a simple pull from the repository and each node will be in sync. It was a tough journey but I learned a lot.