Millet Porridge

English version of https://corvo.myseu.cn

0%

How a Kubernetes-based PaaS Platform Provides ssh Services (Research)

Posted on Edited on In Disqus:
I've been researching how to provide container shell login for users. This blog organizes my thoughts, exploring two approaches: one accessing pods in the Kubernetes cluster via a proxy; the other based on webshell while also allowing users to log in locally. It analyzes both approaches' security and functional usage.

Note: this article is reposted from my company intranet blog.

Introduction

I’ve been researching how to provide container shell login for users. This blog organizes my thoughts, exploring two approaches: one accessing pods in the Kubernetes cluster via a proxy; the other based on webshell but allowing users to log in locally. It analyzes both approaches’ security and functional usage.

User Requirements

Although we’re a PaaS platform — developers provide code and configuration and we deploy it — when the following problems arise, developers may want to log into their own containers and run certain commands:

  1. Environment debugging during development: viewing environment variables, test-connecting databases, printing logs and verifying code behavior
  2. Already-online services: developers expect to manually execute some scripts; using a real shell to run scripts is faster and more convenient

Solution - Adding a Shell Container

In the current environment, launching an application creates three kubernetes services: a Deployment, a Service, and an Ingress. Ingress forwards traffic to the corresponding Pod.

To avoid affecting already-running online services, the shell login function doesn’t happen in the online container; instead a new Pod is started separately to provide the shell service.

Since the Kubernetes cluster doesn’t use the host network but an internal IP allocated by flannel, after this Pod starts, how can it be accessed from outside the Kubernetes cluster:

  1. Add a jump server or proxy service providing an external egress, with the proxy server accessing Kubernetes’ internal resources
  2. Use WebShell: our platform assigns a domain for the service, plus authentication to improve security

Below I introduce the research results for both approaches.

ssh Proxy Usage - ProxyCommand

The ssh discussed in this part is OpenSSH, which must support the ProxyCommand parameter.

Service Proxying TCP Ports

Since we provide an external proxy, how do we distinguish each application internally? Here I plan to use the Service-proxying-TCP-ports feature. The whole service is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
---
apiVersion: v1
kind: Service
metadata:
  name: app01-ssh
spec:
  selector:
    app: app01-ssh
  ports:
  - protocol: TCP
    port: 22
    targetPort: 22
    name: proxied-tcp-22

For different applications, as long as they’re pods in k8s, we access the corresponding service in the form telnet appXX-ssh 22.

See the appendix for the Dockerfile and Kubernetes configuration files.

Using a Jump Server

The jump server’s ssh config is as follows

1
2
3
4
Host deb
    User root
    HostName app1.ssh
    ProxyCommand ssh -q -W %h:%p jumpserver
1
2
3
4
5
6
7
8
9
                          +---------------+
                +-------->+   app3.ssh    |
                |         +---------------+
+---------------+         +---------------+
|ssh jump server+-------->+   app2.ssh    |
+---------------+         +---------------+
                |         +---------------+
                +-------->+   app1.ssh    |
                          +---------------+

Using a Proxy – HTTPProxy

This is one approach I researched:

OpenSSH’s ProxyCommand works by using a child process to establish a TCP connection, then communicating with the parent process via pipes. After understanding this, we can actually rely on HTTP to establish the connection and then transfer TCP packets.

1
2
# note: the nc here must be netcat-openbsd (the openbsd version of nc; gnu's nc has no -X feature)
ssh -o "ProxyCommand=nc -X connect -x 127.0.0.1:8888 %h %p" [email protected]

Here 127.0.0.1:8888 is the http proxy opened on the local machine; for a fairly simple proxy server see HTTP proxy principles and implementation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var http = require('http');
var net = require('net');
var url = require('url');

function connect(cReq, cSock) {
    var u = url.parse('http://' + cReq.url);

    var pSock = net.connect(u.port, u.hostname, function() {
        cSock.write('HTTP/1.1 200 Connection Established\r\n\r\n');
        pSock.pipe(cSock);
    }).on('error', function(e) {
        cSock.end();
    });

    cSock.pipe(pSock);
}

http.createServer().on('connect', connect).listen(8888, '0.0.0.0');

For the concrete online service, I personally lean toward HTTPProxy. Two benefits:

  1. Establishing a Jump Server requires maintaining an OpenSSH server; an http proxy only requires maintaining a fairly small amount of code
  2. In an http proxy, you can conveniently control which services users may reach — a whitelist can be built into the program, making security higher than a Jump Server

WebShell - Web and Local Functionality

Basic WebShell

The WebShell research mainly revolves around gotty, mainly relying on gotty+tmux to support machine access.

1
2
# its function is creating a new window with tmux
gotty -w tmux new -A -s gotty

Visiting http://127.0.0.1:9000 shows:

The gotty Client

gotty’s README introduces how to access the GoTTY service from a terminal — gotty-client was released:

1
2
3
go get github.com/moul/gotty-client/cmd/gotty-client

gotty-client http://127.0.0.1:9000/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
                                                               ┌─────────────────┐
                                                      ┌──────➤│    /bin/bash    │
                                                      │        └─────────────────┘
                 ┌──────────────┐                ┌──────────┐
                 │              │                │  Gotty   │
┌───────┐   ┌──➤│   Browser    │───────┐        │          │
│       │   │    │              │       │        │          │
│       │   │    └──────────────┘       │        │          │   ┌─────────────────┐
│  Bob  │───┤                       websockets─➤│          │─➤│ emacs /var/www  │
│       │   │    ╔═ ══ ══ ══ ══ ╗       │        │          │   └─────────────────┘
│       │   │    ║              ║       │        │          │
└───────┘   └──➤║ gotty-client ║───────┘        │          │
                 ║              ║                │          │
                 ╚═ ══ ══ ══ ══ ╝                └──────────┘
                                                      │        ┌─────────────────┐
                                                      └──────➤│   tmux attach   │
                                                               └─────────────────┘

The basic principle is reusing GoTTY’s websocket. But trying it out, I found its tmux support is really poor.

For heavy tmux users this is almost completely unusable — the character printing is too outrageous.

Comparing the Two Approaches

With the two candidate approaches above — one based on native ssh, the other resembling WebShell — I’ll discuss both in terms of difficulty, security, and maintenance cost.

Functionality and Implementation Difficulty Discussion

  • Functionally: Both approaches can meet users’ requirement of connecting into containers.

    I normally use native ssh too much; personally I hope to leverage the various features OpenSSH provides. WebSSH gives a gaudy feeling; shortcut key support has problems — one Ctrl-W means the page gets closed.

  • Implementation difficulty:

    1. The former needs an added service as proxy server, plus good private key management
    2. The latter needs combining with WebShell tools like gotty, likewise needing good permission control; it also supports password login itself

Maintenance and Decommission Cost Discussion

The former requires maintaining the proxy server, plus the ProxyCommand for local connections.

The latter may require maintaining gotty and possibly gotty-client in future.

The latter has higher maintenance cost.

Security Discussion

  • Data transmission encryption:

    1. The former encrypts via ssh itself
    2. The latter can use websocket over https, encrypting transmission via https

  • Cluster security guarantee (not-logged-in state):

    1. The former provides a layer-4 proxy, so security needs special attention — otherwise service ports in the cluster that shouldn’t be exposed may be exposed (solution proposed below)
    2. The latter’s login happens entirely from the web page; its security matches the PaaS platform

  • Cluster security guarantee (logged-in state): In both systems user operations can be recorded, but after granting root permission users may change that themselves. In both systems users can access all other services in the cluster — hard to restrict for now.

Regarding the layer-4 proxy service’s security, I suggest guaranteeing it the following ways (taking the http proxy server as an example):

  1. This proxy server only allows internal network access
  2. This proxy server adds basic auth, requiring users to give username and password when connecting
  3. This proxy only allows certain domains — like ssh service domains resembling app01-ssh — to pass, disallowing access to other domains or ips besides ssh domains

If the above 3 points can be achieved, the proxy server’s security is basically equal to WebShell’s.

Summary

This article explored two approaches for externally connecting to Kubernetes pods: one accessing pods in the Kubernetes cluster via ssh proxy; the other based on WebShell but allowing users to log in locally. It analyzed both approaches’ functionality, implementation difficulty, and security.

From a personal perspective, I lean toward the proxy approach — after all, native ssh has many companion tools, e.g. scp, sshfs, and configurations like LocalForward can also be leveraged.

Known Problems and Solutions

While testing remote server connections, the following problem appeared:

1
2
3
~ ❤  ssh -o "ProxyCommand=nc -X connect -x 10.202.37.229:8081 %h %p" [email protected]
Bad packet length 1349676916.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: message authentication code incorrect

After capturing packets with wireshark, I saw netcat splitting the tcp packets, causing the remote server to parse incomplete ssh packets and be unable to respond.

Given the above problem, I decided to write a simple proxy tool myself to replace nc. Here’s the simple version of the tool:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
# -*- coding: utf-8 -*-

"""
Usage, python3:

ssh -o "ProxyCommand=python3 tun.py 10.202.37.229:8081 %h %p" [email protected]
"""

import sys
import socket
from threading import Thread

import http.client as httplib
from urllib.parse import urlparse, urlencode
from urllib.request import urlopen, Request
from urllib.error import HTTPError


class ProxyHTTPConnection(httplib.HTTPConnection, object):
    def __init__(self, *args, **kwargs):
        super(ProxyHTTPConnection, self).__init__(*args, **kwargs)
        self.stop = False

    def connect(self):
        httplib.HTTPConnection.connect(self)
        # send proxy CONNECT request
        self.send(b"CONNECT %s:%d HTTP/1.0\r\n\r\n" %
                  (self._real_host.encode('utf8'), self._real_port))
        # expect a HTTP/1.0 200 Connection established
        response = self.response_class(self.sock, method=self._method)
        (version, code, message) = response._read_status()
        if code != 200:
            # proxy returned and error, abort connection, and raise exception
            self.close()
            raise socket.error(
                "Proxy connection failed: %d %s" % (code, message.strip()))
        self.interpreterloop(self.sock)

    def interpreterloop(self, sock):
        Thread(target=self.readloop, args=(sock, )).start()
        self.writeloop(sock)

    def readloop(self, sock):
        while not self.stop:
            try:
                sock.send(sys.stdin.buffer.read(1))
            except socket.error as e:
                print(e)
                return

    def writeloop(self, sock):
        while not self.stop:
            try:
                c = sock.recv(1)
            except KeyboardInterrupt:
                self.stop = True
                return
            if not c:
                break
            sys.stdout.buffer.write(c)
            sys.stdout.flush()


if __name__ == '__main__':
    proxy = sys.argv[1]
    remote_host = sys.argv[2]
    remote_port = sys.argv[3]

    p = ProxyHTTPConnection(proxy)
    p._real_host = remote_host
    p._real_port = int(remote_port)
    p.connect()

Appendix

Enabling the sshd Service in Kubernetes

Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
# ssh-test
FROM python:3.7.4-stretch

RUN apt-get update && apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:hello' | chpasswd
RUN echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config

# RUN echo "export VISIBLE=now" >> /etc/profile

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]
  • Kubernetes configuration files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app01-ssh
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app01-ssh
  template:
    metadata:
      labels:
        app: app01-ssh
    spec:
      containers:
      - name: app01-ssh
        image: test
        imagePullPolicy: Never
        ports:
        - containerPort: 22
          name: app01-ssh
---
apiVersion: v1
kind: Service
metadata:
  name: app01-ssh
spec:
  selector:
    app: app01-ssh
  ports:
  - protocol: TCP
    port: 22
    targetPort: 22
    name: proxied-tcp-22
Related Posts