Using Ingress in Kubernetes

I’ve been looking into Kubernetes-related issues lately. For a locally built cluster, we need Ingress to expose services. I encountered some problems during use, but searching with Google didn’t yield answers. Much of the content may come from my own exploration; please point out any incorrect usage. While debugging, I actually also read part of the Ingress implementation code — one article probably can’t cover it, so I’ll open another.

!! This article does NOT cover how to install a kubernetes cluster — please install one first!!

The Role of Ingress in K8s

The official Ingress introduction explains Ingress’s role in a K8s cluster.

Ingress exposes HTTP or HTTPS routes, allowing access to in-cluster services from outside the cluster. The routing policy for this part is defined by Ingress resources.

If you’ve configured multi-domain Nginx before, you know that for different domains, server_name can determine the backend. Ingress’s basic principle is also choosing the corresponding backend location via different server_names — though the concrete implementations of various Ingresses may differ somewhat.

server {
    listen       80;
    server_name  app1.example.org;
    ...
}
server {
    listen       80;
    server_name  app2.example.org;
    ...
}

Two Ingress Implementations I’ve Used

When we normally speak of Ingress, we should be speaking of the Ingress service or the Ingress API. Different implementations can follow this same API. I recently researched and used two implementations: one by the Nginx company (Nginx Inc), the other by the open source community (Kubernetes). When using them, be sure to understand the differences between the two Ingress implementations and choose the Ingress reasonably.

Nginx Inc’s Implementation

https://github.com/nginxinc/kubernetes-ingress

The Kubernetes Community’s Implementation

https://github.com/kubernetes/ingress-nginx

Comparison of the Ingresses

The original table comes from nginx-ingress-controllers.md.

This is the 2019-11-05 version; the future may differ — please follow the link for the latest version.

Aspect or Feature	kubernetes/ingress-nginx	nginxinc/kubernetes-ingress with NGINX	nginxinc/kubernetes-ingress with NGINX Plus
Fundamental
Authors	Kubernetes community	NGINX Inc and community	NGINX Inc and community
NGINX version	Custom NGINX build that includes several third-party modules	NGINX official mainline build	NGINX Plus
Commercial support	N/A	N/A	Included
Load balancing configuration via the Ingress resource
Merging Ingress rules with the same host	Supported	Supported via Mergeable Ingresses	Supported via Mergeable Ingresses
HTTP load balancing extensions - Annotations	See the supported annotations	See the supported annotations	See the supported annotations
HTTP load balancing extensions – ConfigMap	See the supported ConfigMap keys	See the supported ConfigMap keys	See the supported ConfigMap keys
TCP/UDP	Supported via a ConfigMap	Supported via a ConfigMap with native NGINX configuration	Supported via a ConfigMap with native NGINX configuration
Websocket	Supported	Supported via an annotation	Supported via an annotation
TCP SSL Passthrough	Supported via a ConfigMap	Not supported	Not supported
JWT validation	Not supported	Not supported	Supported
Session persistence	Supported via a third-party module	Not supported	Supported
Canary testing (by header, cookie, weight)	Supported via annotations	Supported via custom resources	Supported via custom resources
Configuration templates *1	See the template	See the templates	See the templates
Load balancing configuration via Custom Resources
HTTP load balancing	Not supported	See VirtualServer and VirtualServerRoute resources.	See VirtualServer and VirtualServerRoute resources.
Deployment
Command-line arguments *2	See the arguments	See the arguments	See the arguments
TLS certificate and key for the default server	Required as a command-line argument/ auto-generated	Required as a command-line argument	Required as a command-line argument
Helm chart	Supported	Supported	Supported
Operational
Reporting the IP address(es) of the Ingress controller into Ingress resources	Supported	Supported	Supported
Extended Status	Supported via a third-party module	Not supported	Supported
Prometheus Integration	Supported	Supported	Supported
Dynamic reconfiguration of endpoints (no configuration reloading)	Supported with a third-party Lua module	Not supported	Supported

Ingress Usage in Practice

The sections below introduce some of my Ingress practices. Although fairly simple, I think these are the most common usages.

Selecting Some Cluster Machines When Creating the Ingress

When creating the Ingress Pod, a cluster may only need 2-4 machines serving externally — there’s no need to create one on every node. Below is part of the configuration when creating this Pod; note that it’s deployed as a DaemonSet, and this deployment form allows us to select the corresponding machines.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress
  namespace: nginx-ingress

Node selection is done through the nodeSelector configuration; in the config below, the nodes with run_ingress=true are selected.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
       #prometheus.io/port: "9113"
    spec:
      nodeSelector:
        run_ingress: "true"
      serviceAccountName: nginx-ingress
      ...

Nodes can be labeled like this:

1	kubectl label node <node name> run_ingress="true"

Basic Ingress Configuration

In the structure above, a simple service generally consists of several pods + 1 service + 1 ingress:

# start a python3.7 container listening on port 6000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: arya-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: arya-app
  template:   
    metadata:
      labels:
        app: arya-app
    spec:
      containers:
      - name: myapp-container
        image: python:3.7.4-stretch
        command: ['python3', '-m', 'http.server', '6000']

# create the corresponding service
---
apiVersion: v1
kind: Service
metadata:
  name: arya-svc
spec:
  selector:
    app: arya-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 6000
    name: http

# create the service's corresponding Ingress configuration
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: arya-prod
spec:
  rules:
  - host: arya.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: arya-svc
          servicePort: 80

The result is as follows:

~ ❤  curl arya.example.com
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>

Canary Releases with Ingress

Our usual work also needs the canary release feature. After investigating I found that Nginx Inc’s Ingress implementation cannot do canary releases directly — it needs custom resource support. So I chose the Kubernetes community’s Ingress. Through gradual use I discovered the two Ingress implementations differ: Nginx Inc uses ordinary Nginx, while Kubernetes uses OpenResty. The specific differences will be introduced in the next blog post; this one only covers canary usage.

For detailed configuration see the official canary documentation

A canary release needs two versions of the program, one of which is the live version while the other is not yet online. The two versions correspond to different service and ingress configurations. Let’s look at the live version first.

# prod_arya.yaml — the live version corresponds to 753
---
apiVersion: v1
kind: Service
metadata:
  name: arya-prod
spec:
  selector:
    app: arya-v753
  ports:
  - protocol: TCP
    port: 80
    targetPort: 6000
    name: http

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: arya-prod
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: arya.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: arya-prod
          servicePort: 80

For the version needing canary treatment, please note: the two hosts — i.e. the externally served domain — must be consistent; what differs is the related configuration added in annotations.

# canary_arya.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: arya-develop-canary
spec:
  selector:
    app: arya-v754
  ports:
  - protocol: TCP
    port: 80
    targetPort: 6000
    name: http

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: arya-prod-canary
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/canary-by-header: canary
spec:
  rules:
  - host: arya.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: arya-develop-canary
          servicePort: 80

With this configuration, you can test with the following commands:

# direct requests — requests randomly reach one of the two version environments
~ ❤  for i in `seq 10`;do curl -s  arya.example.com/version;done
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}

# modify the http header
# 754 is the new version — all traffic points to the new environment
~ ❤  for i in `seq 10`;do curl -sL -H "canary: always"  arya.example.com/version;done
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}
{"result":{"version_id":"754"}}

# 753 is the currently online version — below, all traffic goes to the old environment
~ ❤  for i in `seq 10`;do curl -sL -H "canary: never"  arya.example.com/version;done
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}
{"result":{"version_id":"753"}}

Summary

This blog mainly introduced Ingress’s role in Kubernetes and some practical work. The next Ingress-related blog will introduce the differences between the two Ingress implementations (Nginx Inc and the Kubernetes community), and explore the use of OpenResty in finer detail.