Millet Porridge

English version of https://corvo.myseu.cn

0%

A Solution for Providing Dashboard Support on a Kubernetes-based PaaS Platform

Posted on Edited on In Disqus:
The PaaS platform I maintain introduced Kubernetes as underlying support, enabling more things via the Kubernetes ecosystem. This blog mainly introduces how to provide Dashboard functionality for ordinary users, plus some extensible ideas.

The PaaS platform I’ve been maintaining introduced Kubernetes as underlying support; the Kubernetes ecosystem enables doing more things. This blog mainly introduces how to provide dashboard functionality to users, plus some extensible ideas. I hope readers have some kubernetes usage experience and understand rbac.

Dashboard Functionality

Kubernetes natively provides a Web interface — the Dashboard. For details see the official documentation:

After installation, we generally use it via tokens; different tokens have different permissions.

![][3]

The token mentioned above is a Bearer Token. Besides entering it in the interface, you can use it like this — just add the header:

1
curl -H "Authorization: Bearer ${TOKEN}" https://{dashboard}/api/myresource

Brief Discussion of Using Dashboard on a PaaS Platform

Requirements Analysis

Dashboard’s own functionality is very powerful, but giving everyone admin permission is obviously unrealistic. For an ordinary user, the PaaS platform deploys and runs his application (code); all he needs to care about is his own project. The platform also needs good permission control to prevent one user from operating another user’s applications.

Permission System Design

Based on the requirements discussion above, what the platform needs to do is create each user’s own permissions and limit accessible resources. Consider this situation:

We have a user A who owns an application group (G); the group deploys a series of applications (a1, a2…). In Kubernetes, we map this group concept to a namespace: group (G) <=> namespace (NS). The permission control strategy we need then becomes user A’s permission control within namespace NS.

Token Distribution Strategy

With permission control in place, all that’s needed is distributing tokens to users. Of course this is an extremely unsafe practice: tokens in Kubernetes generally don’t change after creation. Distributing such tokens carries big security risks, in two aspects:

  1. If user A saves the token, he can log into Dashboard without going through the platform — bad for auditing work.
  2. Once a token leaks, the PaaS platform can hardly react (because the token is outside the platform’s control — it can’t tell when the leak happened, nor immediately revoke the token); the security risk is relatively high.

Therefore the best practice is not giving the token to users at all. Each time a user wants to log into the dashboard, they jump from the platform carrying security information; at dashboard login, the platform’s own program requests the token, keeping it out of users’ hands.

If by here you haven’t understood the content above, I suggest re-reading the requirements. If you still don’t understand, don’t read on — below only introduces the concrete implementation plan.

Kubernetes Permission Restrictions

Kubernetes itself has a fairly complex permission control system; there’s no need to agonize too much when designing — just distinguish permissions that can be given to users from those that can’t. I’ll paste my permission control strategy directly; it doesn’t necessarily suit everyone — just use it as a reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: xxx:xxx-group-yyy
  namespace: xxx-group-yyy
rules:
  # can view services, pods, logs, events under the current NS
  - apiGroups: [""]
    resources: ["services", "pods", "pods/log", "events"]
    verbs: ["get", "list", "watch"]

  # can use the exec command to enter containers
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "list", "watch"]

  # can view deployments and replicasets
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]

  # can view jobs, cronjobs and ingresses
  - apiGroups: ["batch", "extensions"]
    resources: ["cronjobs", "jobs", "ingresses"]
    verbs: ["get", "list", "watch"]

As the comments above say, give users read-only permissions as much as possible. Maybe you’ve noticed: users don’t even need permission to view namespaces — this is also for security, to prevent users from learning other people’s namespaces.

Distributing Tokens and Security Guarantees

This is the core content of this blog: how to let users log into the dashboard imperceptibly (hiding the token from users).

The method used by this solution: add a layer of access-control gateway to handle the token-fetching operation. The concrete flow diagram is below.

![][4]

A few points to note:

  1. The secret_code given by the PaaS is time-limited; users are not allowed to keep accessing with the same secret_code
  2. Communication between the gateway and the PaaS platform should be encrypted; the gateway must be trusted by the PaaS platform
  3. The gateway should not store the token long-term
  4. Gateway access should best add OpenID verification, ensuring the gateway can precisely locate every access by every user

Experience Optimization

  1. First, from step 2 to step 3: after obtaining the secret_code, it can jump to the gateway entrance via a 302 redirect
  2. The gateway can temporarily store the mapping between secret_code and token — improving user experience and effectively reducing PaaS platform pressure
  3. The dashboard’s webshell feature is based on websocket support, so ensure your gateway can pass websocket requests; otherwise the terminal disconnects a few minutes after connecting. Websockets can last for hours
  4. When jumping to the gateway, more information can be carried — e.g. a pod’s id — so the gateway can jump directly to the corresponding pod, making it convenient for users to open the webshell

I won’t elaborate on the gateway implementation; just one suggestion: verify the secret_code immediately after jumping to the gateway. Due to the dashboard‘s frontend routing implementation, the secret_code should best be encrypted into a cookie after verification. For implementation questions you can email me to discuss.

Summary

This blog mainly introduced a feature allowing ordinary users to use the dashboard. In implementation strategy it uses kubernetes permission restrictions and a token-hiding scheme. I’ve already added this scheme to the PaaS platform I’m responsible for; stability meets work requirements, and security is as introduced in the blog — please weigh it yourselves.

I’m sorry the blog has many words and few pictures, and gives no concrete implementation. First, code implementation isn’t important; second, it’s company code and inconvenient to share — please forgive me.

[3]: https://rawforcorvofeng.cn/v2-0de7524250e9af31bdaa12eef16922f7_720w.jpg [4]: https://rawforcorvofeng.cn/dashboard.png

Related Posts