kubemux and Multi-Cluster Management

This past year I did less Kubernetes development and more usage — mainly lots of clusters migrating from Mesos to Kubernetes. After the migration finishes I should write a blog to record it. This post first introduces how I manage multiple clusters in my work, and what the kubemux tool I wrote actually does.

This is the implementation effect and the repository of the software.

https://github.com/corvofeng/kubemux

Installation

# MacOS
brew install corvofeng/tap/kubemux

# Linux -- using bin: https://github.com/marcosnils/bin
bin install https://github.com/corvofeng/kubemux ~/usr/bin
# bin ls
# Path                  Version  URL                                                       Status
# ~/usr/bin/kubemux     v1.1.2   https://github.com/corvofeng/kubemux/releases/tag/v1.1.2  OK

# Linux -- using binary
cd /tmp
rm kubemux_linux_amd64.tar.gz
wget https://github.com/corvofeng/kubemux/releases/latest/download/kubemux_linux_amd64.tar.gz
tar -zxvf kubemux_linux_amd64.tar.gz
sudo install -v kubemux /usr/local/bin

kubeconfig

ls ~/.kube
# pve-kube.config xxx

kubemux kube --kube pve-kube.config

# I suggest you add the completion support
#   source <(kubemux completion bash)
#   source <(kubemux completion zsh)
# or you can add the command into the .bashrc or .zshrc.
kubemux kube --kube <tab>

tmuxinator

mkdir ~/.tmuxinator

echo '
name: kubemux
root: "~/"
windows:
  - p1:
    - ls
    - pwd
  - p2:
    - pwd
    - echo "hello world"
  - p3: htop
' > ~/.tmuxinator/kubemux.yml

kubemux -p kubemux

Kubernetes Multi-Cluster Management Solution

The official Kubernetes website provides a solution for switching between clusters using the context in KUBECONFIG.

https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
  name: development
- cluster:
  name: test

users:
- name: developer
- name: experimenter

contexts:
- context:
  name: dev-frontend
- context:
  name: dev-storage
- context:
  name: exp-test

Many tools like kubecm are also based on this, which brings a few problems:

Every time you add or remove a cluster, you have to maintain one big KUBECONFIG file
All terminals share the same KUBECONFIG file, which means only one cluster can be operated at a time

In real production multi-cluster management this is very inconvenient and error-prone. Next I’ll introduce a scheme using tmux configuration file splitting: each Kubernetes cluster gets its own tmux session, and each session has its own KUBECONFIG file — so you can operate multiple clusters simultaneously and independently.

tmux Multiple Sessions

tmux is a powerful terminal multiplexer that allows you to create and manage multiple sessions in the same terminal window. This is very useful for managing multiple servers or clusters because you can easily switch between different sessions without opening multiple terminal windows.

To use tmux, you need to install it first. In most Linux distributions, you can install tmux using the following command:

1	sudo apt install tmux

After the installation is complete, you can start tmux using the following command:

tmux

This will create a new tmux session in your terminal window. You can use the following command to switch between different sessions:

1 2	tmux attach-session -t <session-name> tmux new-session -s <session-name>

tmux -L

In simple terms, the -L socket-name parameter allows you to specify the location of the tmux socket, and different sockets correspond to completely isolated sessions.

We can use different environment variables in different sessions to achieve environment separation.

For example, with the following two commands, you can create two completely independent terminals with their own environment variables:

1 2	KUBECONFIG=~/.kube/config-aa tmux -L aa KUBECONFIG=~/.kube/config-bb tmux -L bb

The script here can already achieve multi-cluster management. So why do we introduce tmuxinator and my new tool kubemux?

The configuration for the production environment needs to go through a jump host. How can we use KUBECONFIG locally? (I have attached it at the end)
I want the tmux session to have multiple windows with their own functionalities.

tmuxinator Usage and Limitations

https://github.com/tmuxinator/tmuxinator

It is a tool written in Ruby that allows you to define tmux terminals in YAML format. It also supports templating. Here is an example of a templated YAML file:

name: project
root: ~/<%= @settings["workspace"] %>
# tmuxinator start project workspace=~/workspace/todo

windows:
  - small_project:
      root: ~/projects/company/small_project
      panes:
        - start this
        - start that

During the months of managing cluster environment migration, some of the most frequently used commands were:

1
2
3

tmuxinator tpl project=ingame-pre-na
tmuxinator tpl project=ingame-pre-sg
tmuxinator tpl project=ingame-pre-fra

It can help me perfectly differentiate different environments, and because I use fzf, I can even fuzzy search for the environment I want to open.

Limitations: Since it is written in Ruby, it requires a relatively new version of Ruby installed on the machine. AWS requires logging into a jump host for operations, but the machines we use are very old, and I don’t want to compile and reinstall Ruby. After a quick look at the code, I found that it doesn’t use any advanced features. It would be very easy to completely rewrite it in Golang.

Implementation of kubemux

For this project, I heavily relied on ChatGPT, mainly for writing the logic and unit tests. Originally, I thought it would take 2-3 days of work, but the development cycle was shortened to 1 day. The final result is also very good. After rewriting, kubemux has no dependencies and is very lightweight to install.

The project is open source here, and it should be compatible with tmuxinator configurations. If you find any missing features, feel free to open an issue.

https://github.com/corvofeng/kubemux

Summary

Managing dozens of clusters from the command line is completely feasible and the results are quite good. Hopefully, it can also improve everyone’s efficiency.

Appendix: My Template Configuration

name: <%= @settings["project"] %>
root: ~/GitRepo

socket_name: <%= @settings["project"] %>
on_project_start:

  # Since a jump host is needed to connect to the API,
  # I used ssh to open a socks5 proxy on my local machine.
  # The following method is used to obtain a locally available port and the corresponding jump host:
  - export KUBECONFIG=~/.kube/config-<%= @settings["project"] %>
  - export TMUX_SSH_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1])')"
  - export TMUX_SSH_HOST="<%= @settings["host"] %>"
startup_window: kubectl

windows:
  - proxy:
      layout: main-vertical
      panes:
        - startup:
          - ls -alh
          - yq -i e '.current-context |= "tpl-<%= @settings["project"] %>"' $KUBECONFIG
          - yq -i e '.contexts[0].name |= "tpl-<%= @settings["project"] %>"' $KUBECONFIG
          - yq -i e ".clusters[0].cluster.proxy-url |=\"socks5://127.0.0.1:$TMUX_SSH_PORT\"" $KUBECONFIG
          - ssh -D $TMUX_SSH_PORT $TMUX_SSH_HOST
  - kubectl: ls # kubectl get pods

In addition to modifying the context, you can also add prompts to the terminal PS1, similar to this:

https://github.com/jonmosco/kube-ps1