Millet Porridge

English version of https://corvo.myseu.cn

0%

Simply Using Ansible to Manage Kubernetes Clusters

Posted on Edited on In Disqus:
I don't know whether readers of my blog normally use Ansible. As a full-featured ops tool, Ansible can handle most problems I meet at work. Recently my main focus has been Kubernetes. In this blog I describe the problems I solved with Ansible, hoping that when you use Kubernetes in the future you'll consider managing clusters with Ansible.

Updated 2020-12-05 I’ve already cancelled the digitalocean service — it’s really a bit expensive!!!

Original Content

I don’t know whether readers of my blog normally use Ansible. As a full-featured ops tool, Ansible can handle the vast majority of problems I meet at work. Recently my main work focus has been Kubernetes. In this blog I’ll briefly describe the problems Ansible solves, hoping that when you use Kubernetes in the future, you’ll consider using Ansible to manage clusters.

I hope readers have some basic Kubernetes operation experience and Ansible usage experience.

Building the Kubernetes Cluster

Building a highly available Kubernetes cluster includes two parts: the etcd cluster and the Kubernetes cluster.

This process won’t be covered in the blog — there are many blogs to read, and they’re fairly complete.

Let me recommend two Kubernetes services you can quickly build and use:

  1. okteto

    https://okteto.com/

    You can log in with just a Github account and get a simple Kubernetes cluster. Permissions are limited, but deploying applications is enough. One downside: free accounts shut containers down after 24 hours, but paid accounts are expensive at $20 a month. Fine for simple use; don’t consider it for services.

  2. digitalocean

    https://cloud.digitalocean.com/

    My referral link: https://m.do.co/c/e02a6de85b29

    I personally prefer digitalocean: creating a Kubernetes cluster with two nodes costs just $20 a month, and it’s a native Kubernetes environment. After much consideration (actually none at all), I directly bought a digitalocean cluster — the service is worry-free too.

Ingress Management

The last Ingress introduction is here: Ingress Implementation Strategies and Source Code Reading. When I use Ingress, I always choose the Kubernetes community edition. The community Ingress needs some customized configuration depending on the user’s situation:

For example, when my Ingress is not the egress but Nginx forwards into the Kubernetes cluster, the Ingress needs to obtain the real client IP. For the principle see getting the user’s real IP. In the Ingress, the following configuration is needed; my personal solution is:

1
2
3
4
5
6
7
8
9
10
11
12
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-forwarded-headers: "true"
  compute-full-forwarded-for: "true"
  proxy-real-ip-cidr: "10.0.0.0/8,{{kubernetes_sub_net}}

This file can be generated and applied by ansible:

1
2
3
4
5
6
7
8
9
10
11
12
---
- block:
  - name: upload kube-ingress.yml
    template:
      src: templates/ingress.yml.j2
      dest: /tmp/kube-ingress_{{item.name}}.yml
    with_items: '{{ cluster_ingress }}' # different private clusters have different Ingresses
  
  - name: install ingress
    shell: KUBECONFIG={{kubernetes_config}} kubectl apply -f /tmp/kube-ingress_{{item.name}}.yml
    with_items: '{{ cluster_ingress }}' # different private clusters have different Ingresses
  delegate_to: "{{ groups['kubernetes_masters'][0] }}"

Let me briefly introduce the key points above:

  1. KUBECONFIG={{kubernetes_config}} kubectl apply xxx

Different kubernetes config files can operate different clusters.

  1. with_items: '{{ cluster_ingress }}' # different private clusters have different Ingresses This is our business requirement — multiple Ingresses must be deployed for different businesses

  2. delegate_to: "{{ groups['kubernetes_masters'][0] }}"

Specifies this operation only runs on the first machine with the master role.

docker secret Management

Kubernetes official documentation: pulling images from a private registry.

When I first started using Kubernetes, for convenience I called kubectl to create secrets, like this:

1
2
3
4
5
6
## Create docker-registr
## Please don't use code like this anymore
- name: Create docker registry
  shell: KUBECONFIG={{kubernetes_config}} kubectl -n {{item}} create secret docker-registry "{{ docker_registry_secret_name }}" --docker-server="{{ docker_registry_server }}" --docker-username="{{ docker_registry_username }}" --docker-password="{{ docker_registry_pwd }}" --docker-email="{{ docker_registry_email }}"
  ignore_errors: true
  with_items: "{{need_secret_namespaces}}"

The kubectl create secret way is fine when creating secrets, but modifying secrets becomes troublesome. After the project gradually stabilized, I updated this part of the code: no more imperative commands like create secret — it’s now declarative. See the code for the specifics:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
- block:
  # create the dockerconfigjson variable
  - set_fact:
      dockerconfigjson: "{{ lookup('template', './secret-json.yml.j2')|to_json}}"

  # generate the yaml file from the dockerconfigjson variable
  - name: upload kube-docker-secrets.yml
    template:
      src: templates/docker-secrets.yml.j2
      dest: /tmp/kube-docker-secrets.yml

  # perform the apply operation
  - name: install ocker-secretes
    shell: KUBECONFIG={{kubernetes_config}} kubectl apply -f /tmp/kube-docker-secrets.yml
  delegate_to: "{{ groups['kubernetes_masters'][0] }}"

secret-json.yml.j2 generates the secret file

1
2
3
4
5
6
7
8
9
10
{
  "auths": {
      "{{docker_registry_server}}":{
        "username": "{{docker_registry_username}}",
        "password": "{{docker_registry_pwd}}",
        "email": "{{docker_registry_email}}",
        "auth": "{{(docker_registry_username + ':' + docker_registry_pwd)| b64encode}}"
      }
  }
}

docker-secrets.yml.j2 generates the yaml file for apply

1
2
3
4
5
6
7
8
9
10
11
12
{% for namespace in need_secret_namespaces %}
# secrets belong to different namespaces and must be handled per namespace
---
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: {{docker_registry_secret_name}}
  namespace: {{namespace}}
data:
  .dockerconfigjson: {{dockerconfigjson| b64encode }}
{% endfor %}

Generating secrets declaratively has one big advantage: as long as the configuration hasn’t changed, you can keep applying it.

Summary

This blog mainly introduced some of my strategies when using Kubernetes. What I really wanted to share is the part about Ansible operating Kubernetes secrets — there’s little material online. It was mainly written from my own exploration; I hope readers can use it when needed.

Although I use Ansible to manage the Kubernetes cluster, please understand: the operations above only configure the surrounding environment — e.g. configuring Ingress and storing docker secrets. Actually publishing applications doesn’t use Ansible either.

Appendix

For Dockerhub login, I’ve added a simple sample; readers can visit:

https://github.com/corvofeng/BlogCode/tree/master/ansible-kubernetes

Related Posts