Millet Porridge

English version of https://corvo.myseu.cn

0%

Kubernetes Dashboard Customization Introduction

Posted on Edited on In Disqus:
We needed to add Webshell functionality to our PaaS; reusing the native Dashboard directly seemed to reduce much development burden, so I planned secondary development. But Dashboard's own permission control is too weak, and its frontend pages are many and complex. To integrate it into our platform, the frontend needs trimming and the backend's authentication changed. This blog mainly introduces how to run and debug the program, and the concrete trimming process.

My previous blog actually introduced another Dashboard provisioning scheme. That scheme didn’t involve Dashboard secondary development. The earlier scheme suits clusters where rbac is usable; the current scheme has broader applicability — you only need a kubeconfig file to provide users a Dashboard with complete permission control.

Dashboard‘s own permission control is too weak, and its frontend pages are many and complex. To integrate it into our platform, the frontend needs trimming and the backend’s authentication method changing. Modifications throughout should be as small as possible, so we can update along with Dashboard‘s updates later — otherwise, when a security issue someday requires updating, patching would be hard.

20220313174904

This modification doesn’t require prior Dashboard experience, but you should have some understanding of K8s APIs, then read some Dashboard code. Its frontend is Angular; its backend is Golang.

Well-written code generally has a similar style, mostly practicing table-driven methods and middleware thinking. Frontend page config files are certainly kept together; the backend certainly has middleware — we just add an authentication module.

First Get the Program Running

First you need a usable kubeconfig file, ensuring your kubectl operations work.

My previous blog introduced kubectl proxy usage; if interested, read it — I believe your understanding of Kubernetes API concepts will deepen.

Frontend

The frontend is fairly simple to run

1
> yarn start:frontend

Enabling websocket Support

One thing to note: the default config read, aio/proxy.conf.json, doesn’t explicitly enable websocket, so the shell page’s usage isn’t a long connection. If you want full-featured webshell while debugging locally too, change it to the following.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "/api": {
    "target": "http://localhost:9090",
    "secure": false,
    "ws": true,
    "changeOrigin": true
  },
  "/config": {
    "target": "http://localhost:9090",
    "secure": false,
    "ws": true,
    "changeOrigin": true
  }
}

Backend

I personally dislike that Golang must compile to a binary before running; I usually just go run main.go directly.

1
2
3
4
5
> cd src/app/backend/
> go run dashboard.go --kubeconfig /home/corvo/.kube/config \
    --metrics-provider none \
    --disable-settings-authorizer \
    --insecure-port 9090

Then open the frontend page — there should be a full-featured Dashboard.

Frontend Page Trimming

The concrete code is in src/app/frontend/chrome/nav/template.html; you can adjust the order and remove unneeded components

1
2
3
4
5
6
7
8
9
10
11
12
<kd-nav-item class="kd-nav-item"
             state="/cronjob"
             id="nav-cronjob"
             i18n>Cron Jobs
</kd-nav-item>
<!--
<kd-nav-item class="kd-nav-item"
             state="/daemonset"
             id="nav-daemonset"
             i18n>Daemon Sets
</kd-nav-item>
-->

Streamlining the search Feature

Because when the search box triggers, it searches all given resources — and a single NS may have very many pods, making the search box slow — we also hide certain components of the search logic. Of course you can also adjust the display order here.

The concrete code is in src/app/frontend/search/template.html

1
2
3
4
5
6
7
<kd-job-list (onchange)="onListUpdate($event)"
             [hideable]="true"></kd-job-list>
<!--
<kd-daemon-set-list (onchange)="onListUpdate($event)"
                    [hideable]="true"></kd-daemon-set-list>
<kd-pod-list (onchange)="onListUpdate($event)"
             [hideable]="true"></kd-pod-list>

Adding Backend Authentication

Authentication Location

The dashboard backend uses the go-restful library. Let’s find how it adds middleware — it has the concept of filter:

1
2
3
4
5
6
7
8
// code from https://github.com/emicklei/go-restful/blob/v3/examples/filters/restful-filters.go
restful.Filter(globalLogging)

// Global Filter
func globalLogging(req *restful.Request, resp *restful.Response, chain *restful.FilterChain) {
        log.Printf("[global-filter (logger)] %s,%s\n", req.Request.Method, req.Request.URL)
        chain.ProcessFilter(req, resp)
}

Our goal is adding several global filters to the dashboard backend for authentication and user event recording.

Now look at the dashboard code location — we just re-authenticate once for all APIs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// src/app/backend/handler/apihandler.go

// CreateHTTPAPIHandler creates a new HTTP handler that handles all requests to the API of the backend.
func CreateHTTPAPIHandler(iManager integration.IntegrationManager, cManager clientapi.ClientManager,
        authManager authApi.AuthManager, sManager settingsApi.SettingsManager,
        sbManager systembanner.SystemBannerManager) (http.Handler, error) {

        apiV1Ws := new(restful.WebService)
}

// src/app/backend/handler/filter.go
func InstallFilters(ws *restful.WebService, manager clientapi.ClientManager) {
        ws.Filter(requestAndResponseLogger)
        ws.Filter(metricsFilter)
        ws.Filter(validateXSRFFilter(manager.CSRFKey()))
        ws.Filter(restrictedResourcesFilter)
}

The Authentication Scheme

We use OpenID for login, but I didn’t want to develop this login module either, so I reused keycloak-gatekeeper — which, looking again recently, has stopped being maintained…

Its function is mainly filtering non-logged-in users while passing the logged-in user’s concrete information (e.g. email or username) to the backend. Based on email and username, judge whether the current request’s namespace and api are allowed for the user.

20210909211333

Automatic kubeconfig Refresh

In the Kubernetes cluster we use, the kubeconfig token expires and needs periodic refreshing. This involves another Dashboard modification:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// src/app/backend/dashboard.go
// here the manager is generated from the kubeconfig file we pass, but this manager is too weak
clientManager := client.NewClientManager(args.Holder.GetKubeConfigFile(), args.Holder.GetApiServerHost())

// app/backend/client/manager.go
// once initialized here, the configuration inside cannot change
func NewClientManager(kubeConfigPath, apiserverHost string) clientapi.ClientManager {
	result := &clientManager{
		kubeConfigPath: kubeConfigPath,
		apiserverHost:  apiserverHost,
	}

	result.init()
	return result
}

To be able to update kubeconfig according to my own wishes, I added a new generator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
func NewXXXClientManager(kubeConfigPath, apiserverHost string) clientapi.ClientManager {
    // you can consider this class inheriting the class above; then at initialization we add a timer to refresh the configuration periodically
	result := &xxxClientManager{
		cm: clientManager{
			kubeConfigPath: kubeConfigPath,
			apiserverHost:  apiserverHost,
		},
		rawConfigPath: kubeConfigPath,
	}
	result.init()
	return result
}

func (icm *xxxClientManager) init() {
	icm.refresh() // first refresh the configuration once
	tickRefresh := time.Tick(8 * time.Hour)
	go func() {

		for {
			select {
			case <-tickRefresh:
				icm.refresh()
			}
		}
	}()
}

Summary

I’m just sharing our scheme, hoping those maintaining PaaS platforms gain something. Personally I feel PaaS platforms with strong development capability embed WebShell directly; we really have no frontend people, so we patched this together. The scheme has run stably for half a month so far — stability should be no problem.

Related Posts