Millet Porridge

English version of https://corvo.myseu.cn

0%

The Monitoring Service of a Kubernetes-based PaaS Platform

Posted on Edited on In Disqus:
The PaaS platform I maintain introduced Kubernetes as underlying support, enabling more via the Kubernetes ecosystem. This blog introduces how to provide chart monitoring services to ordinary users (following the last post on dashboard support).

Overview

The PaaS platform I’ve been maintaining introduced Kubernetes as underlying support; the Kubernetes ecosystem enables doing more things. This blog mainly introduces how to provide chart monitoring services to ordinary users (following the last post, providing dashboard support). It assumes readers can build a Kubernetes cluster and a basic monitoring system themselves, and can use PromQL simply, because that’s not what this blog mainly wants to introduce.

I added this feature even before the dashboard, but explaining it is really complex. I kept putting it off; today I have time, so let’s share it.

Solution Topology

I’ll be lazy here — no desire to draw diagrams.

The image shows the left half of the whole solution: collection tools gather machine and container information, uploading it to Prometheus; the Grafana at bottom right queries and displays data via PromQL. The two parts below briefly introduce Prometheus and Grafana, but the focus comes later.

The Plain Collection and Display Tool - Prometheus

This is an Ingress CPU utilization chart:

“Plain” mainly because this tool’s job is collecting data; each query takes only one statement, there’s little history functionality, and permission control is nearly nonexistent. Such a basic tool certainly can’t be provided to users directly.

The Beautiful Display Tool - Grafana

Compared with the crude image above, Grafana‘s functionality is much more powerful: it can display multiple charts and has permission control — very suitable for building big-picture monitoring. For each user of a PaaS platform, being able to see their own application’s monitoring is truly a wonderful thing.

Dividing Line ———

If the introduction ended here, everyone could quickly get it working by tinkering in their own Kubernetes clusters. As PaaS platform developers, we naturally must consider the organization and management of multiple users and multiple applications: presenting users the data they need while ensuring users and applications don’t affect each other. Yes — like the last post’s dashboard support solution, permission design and control are also needed.

Permission Synchronization Strategy

The PaaS Platform’s Permission System

In the PaaS platform, suppose we have a user A who owns one or more application groups (G1, G2); each group deploys a series of applications (a1, a2…).

Grafana’s Permission System

In Grafana’s permission system there are some Teams; a user can belong to one or more Teams.

Each Folder can own one or more dashboards.

Mapping

Considering the relationships above, our approach becomes clear. My design is:

  1. Each of the user’s groups corresponds to one Team, and that Team has access permission to one Folder
  2. Each application has one dashboard

You can also use other designs (e.g. no Folder, each Team owning some dashboards). The scheme above is just what I find convenient to manage — each Team has its own Folder.

I use Python to periodically sync permissions and chart information. I considered Golang at the time, but honestly, this kind of code suits Python far too well. The parts below are quite niche; I suggest not agonizing over them — you most likely won’t use them and will definitely have to read the docs yourself.

1
2
grafana-api==1.0.3  # for communicating with the Grafana API
grafanalib==0.5.7   # creating Grafana charts

Creating Users, Teams, Folders

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# below is creating a team; the others are similar
def ensure_team(g_api, team_name):
    team_id = -1
    try:
        # {'message': 'Team created', 'teamId': 121}
        g_team = g_api.teams.add_team({'name': team_name})
        team_id = g_team['teamId']
    except GrafanaClientError as e:
        if e.status_code == 409:
            # 'Client Error 409: Team name taken'
            g_team = g_api.teams.get_team_by_name(team_name)[0]
            team_id = g_team['id']
    assert team_id != -1
    return team_id

Syncing Folder and Team Permissions

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# below syncs the team and its users; since users may increase or decrease, corresponding additions/removals are needed
def sync_user_team(g_api, team_id, user_ids):
    orig_members = g_api.teams.get_team_members(team_id)
    now_user_ids = set(user_ids)
    orig_user_ids = set([u['userId'] for u in orig_members])

    for user_id in now_user_ids - orig_user_ids:
        try:
            g_api.teams.add_team_member(team_id, user_id)
        except GrafanaBadInputError:
            pass

    for user_id in orig_user_ids - now_user_ids:
        g_api.teams.remove_team_member(team_id, user_id)

# below syncs folder and team permissions; all ordinary users have read-only permission
def sync_folder_permission(folder, team_id=None):
    # https://grafana.com/docs/grafana/latest/http_api/folder_permissions/
    g_api = get_grafana_api()  # type: GrafanaFace
    permissions = []
    if team_id != None:
        permissions.append({
            "teamId": team_id,
            "permission": 1  # view permission only
        })

    g_api.folder.update_folder_permissions(
        folder['uid'],
        {
            "items": permissions,
        }
    )

Dashboard Page Design and Syncing

This part is actually quite hard. I wanted to use a declarative style, like kubectl apply -f xxx.yaml, but Grafana‘s API is too hard to use. One solution: import dashboard chart data — you can see the apis here were added manually.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def sync_dashboard_data(dashboard_uid, folder_id, dashboard_json_data, inputs=[]):
    """Sync the dashboard's group info and the dashboard's panel data
    """
    g_api = get_grafana_api()  # type: GrafanaFace
    d = g_api.dashboard.get_dashboard(dashboard_uid)

    data = {
        'dashboard': dashboard_json_data,
        'folderId': folder_id,
        "inputs": inputs,
        'overwrite': True
    }
    put_dashboard_path = "/dashboards/import"
    r = g_api.api.POST(put_dashboard_path, json=data)

# For generating the data style, refer here:
# https://github.com/weaveworks/grafanalib/blob/master/grafanalib/tests/examples/example-elasticsearch.dashboard.py

Final Effect

For each application in our system, there’s basically a chart like this:

As administrator I normally don’t look at every application either — maybe just a rough overview.

ingress

https://github.com/kubernetes/ingress-nginx/blob/master/deploy/grafana/dashboards/nginx.json

node_exporter

https://github.com/rfrail3/grafana-dashboards/blob/master/prometheus/node-exporter-full.json

Summary

In this article I mainly wanted to introduce how our system is designed, hoping to provide some reference when you design system integration solutions. Comments and discussion are welcome.

By the way, a complaint about Grafana: it actually can’t set a URL avatar (I even went and read grafana’s code to change the avatar, and found it can’t be modified) — really infuriating!!!

Related Posts