Millet Porridge

English version of https://corvo.myseu.cn

0%

kubemux and Multi-Cluster Management

Posted on Edited on In Disqus:
In the past year, I have focused on utilizing Kubernetes more extensively, particularly in the context of cluster migrations. I have led the migration of numerous clusters from Mesos to Kubernetes, and I plan to document this process in a comprehensive blog post. In this blog post, I will first provide an overview of how I manage multiple clusters in my work. I will also introduce kubemux, a tool that I developed to facilitate this management. kubemux allows me to easily switch between clusters, view cluster information, and perform common tasks such as creating and deleting pods.

In the past year, I have focused on utilizing Kubernetes more extensively, particularly in the context of cluster migrations. I have led the migration of numerous clusters from Mesos to Kubernetes, and I plan to document this process in a comprehensive blog post. In this blog post, I will first provide an overview of how I manage multiple clusters in my work. I will also introduce kubemux, a tool that I developed to facilitate this management. kubemux allows me to easily switch between clusters, view cluster information, and perform common tasks such as creating and deleting pods.

This is the implementation and repository of the kubemux.

https://github.com/corvofeng/kubemux

Installation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# MacOS
brew install corvofeng/tap/kubemux

# Linux -- using bin: https://github.com/marcosnils/bin
bin install https://github.com/corvofeng/kubemux ~/usr/bin
# bin ls
# Path                  Version  URL                                                       Status
# ~/usr/bin/kubemux     v1.1.2   https://github.com/corvofeng/kubemux/releases/tag/v1.1.2  OK

# Linux -- using binary
cd /tmp
rm kubemux_linux_amd64.tar.gz
wget https://github.com/corvofeng/kubemux/releases/latest/download/kubemux_linux_amd64.tar.gz
tar -zxvf kubemux_linux_amd64.tar.gz
sudo install -v kubemux /usr/local/bin

kubeconfig

1
2
3
4
5
6
7
8
9
10
ls ~/.kube
# pve-kube.config xxx

kubemux kube --kube pve-kube.config

# I suggest you add the completion support
#   source <(kubemux completion bash)
#   source <(kubemux completion zsh)
# or you can add the command into the .bashrc or .zshrc.
kubemux kube --kube <tab>

tmuxinator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
mkdir ~/.tmuxinator

echo '
name: kubemux
root: "~/"
windows:
  - p1:
    - ls
    - pwd
  - p2:
    - pwd
    - echo "hello world"
  - p3: htop
' > ~/.tmuxinator/kubemux.yml

kubemux -p kubemux

Kubernetes Multi-Cluster Management Solution

The official Kubernetes website provides a solution for switching between clusters using the context in KUBECONFIG.

https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
  name: development
- cluster:
  name: test

users:
- name: developer
- name: experimenter

contexts:
- context:
  name: dev-frontend
- context:
  name: dev-storage
- context:
  name: exp-test

Many existing tools for managing multiple Kubernetes clusters, such as kubecm, rely on maintaining a single KUBECONFIG file. This approach has several drawbacks:

  1. KUBECONFIG file maintenance: Adding and deleting clusters requires manually editing the KUBECONFIG file, which can become tedious and error-prone, especially when managing a large number of clusters.
  2. ingle-cluster operation: Since all clusters are configured in the same KUBECONFIG file, only one cluster can be operated on at a time. This can be inconvenient and inefficient in a multi-cluster management environment.

Solution using tmux configuration file splitting:

To address these problems, I propose a solution that utilizes tmux configuration file splitting. This approach involves creating a separate tmux session for each Kubernetes cluster. Each tmux session has its own KUBECONFIG file, which allows you to operate on multiple clusters simultaneously and independently.

tmux Multiple Sessions

tmux is a powerful terminal multiplexer that allows you to create and manage multiple sessions in the same terminal window. This is very useful for managing multiple servers or clusters because you can easily switch between different sessions without opening multiple terminal windows.

To use tmux, you need to install it first. In most Linux distributions, you can install tmux using the following command:

1
sudo apt install tmux

After the installation is complete, you can start tmux using the following command:

1
tmux

This will create a new tmux session in your terminal window. You can use the following command to switch between different sessions:

1
2
tmux attach-session -t <session-name>
tmux new-session -s <session-name>

tmux -L

In simple terms, the -L socket-name parameter allows you to specify the location of the tmux socket, and different sockets correspond to completely isolated sessions.

We can use different environment variables in different sessions to achieve environment separation.

For example, with the following two commands, you can create two completely independent terminals with their own environment variables:

1
2
KUBECONFIG=~/.kube/config-aa tmux -L aa
KUBECONFIG=~/.kube/config-bb tmux -L bb

The script here can already achieve multi-cluster management. So why do we introduce tmuxinator and my new tool kubemux?

  1. The configuration for the production environment needs to go through a jump host. How can we use KUBECONFIG locally? (I have attached it at the end)
  2. I want the tmux session to have multiple windows with their own functionalities.

tmuxinator Usage and Limitations

https://github.com/tmuxinator/tmuxinator

It is a tool written in Ruby that allows you to define tmux terminals in YAML format. It also supports templating. Here is an example of a templated YAML file:

1
2
3
4
5
6
7
8
9
10
name: project
root: ~/<%= @settings["workspace"] %>
# tmuxinator start project workspace=~/workspace/todo

windows:
  - small_project:
      root: ~/projects/company/small_project
      panes:
        - start this
        - start that

During the months of managing cluster environment migration, some of the most frequently used commands were:

1
2
3
tmuxinator tpl project=ingame-pre-na
tmuxinator tpl project=ingame-pre-sg
tmuxinator tpl project=ingame-pre-fra

It can help me perfectly differentiate different environments, and because I use fzf, I can even fuzzy search for the environment I want to open.

image

Limitations: Since it is written in Ruby, it requires a relatively new version of Ruby installed on the machine. AWS requires logging into a jump host for operations, but the machines we use are very old, and I don’t want to compile and reinstall Ruby. After a quick look at the code, I found that it doesn’t use any advanced features. It would be very easy to completely rewrite it in Golang.

Implementation of kubemux

For this project, I heavily relied on ChatGPT, mainly for writing the logic and unit tests. Originally, I thought it would take 2-3 days of work, but the development cycle was shortened to 1 day. The final result is also very good. After rewriting, kubemux has no dependencies and is very lightweight to install.

The project is open source here, and it should be compatible with tmuxinator configurations. If you find any missing features, feel free to open an issue.

https://github.com/corvofeng/kubemux

Summary

Managing dozens of clusters from the command line is completely feasible and the results are quite good. Hopefully, it can also improve everyone’s efficiency.

Appendix: My Template Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
name: <%= @settings["project"] %>
root: ~/GitRepo

socket_name: <%= @settings["project"] %>
on_project_start:

  # Since a jump host is needed to connect to the API,
  # I used ssh to open a socks5 proxy on my local machine.
  # The following method is used to obtain a locally available port and the corresponding jump host:
  - export KUBECONFIG=~/.kube/config-<%= @settings["project"] %>
  - export TMUX_SSH_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1])')"
  - export TMUX_SSH_HOST="<%= @settings["host"] %>"
startup_window: kubectl

windows:
  - proxy:
      layout: main-vertical
      panes:
        - startup:
          - ls -alh
          - yq -i e '.current-context |= "tpl-<%= @settings["project"] %>"' $KUBECONFIG
          - yq -i e '.contexts[0].name |= "tpl-<%= @settings["project"] %>"' $KUBECONFIG
          - yq -i e ".clusters[0].cluster.proxy-url |=\"socks5://127.0.0.1:$TMUX_SSH_PORT\"" $KUBECONFIG
          - ssh -D $TMUX_SSH_PORT $TMUX_SSH_HOST
  - kubectl: ls # kubectl get pods

In addition to modifying the context, you can also add prompts to the terminal PS1, similar to this:

https://github.com/jonmosco/kube-ps1

image

Related Posts