Millet Porridge

English version of https://corvo.myseu.cn

0%

Ingress Implementation Strategies and Source Code Reading

Posted on Edited on In Disqus:
The last blog introduced simple Ingress usage but didn't dig into the two Ingress implementations. This post briefly analyzes the internal strategies of both — please point out any problems.

Following the last blog’s introduction to Ingress usage, this post explains the two Ingress implementations from Nginx Inc and the Kubernetes community, then focuses on the Kubernetes community’s Ingress implementation, giving the related code logic and introducing how to debug it.

Basic Ingress Implementation Principle

This was introduced at the start of the last blog: Ingress implementations based on Nginx or OpenResty use multi-domain configuration like the following, choosing the proper backend service via different server_names.

1
2
3
4
5
6
7
8
9
10
server {
    listen       80;
    server_name  app1.example.org;
    ...
}
server {
    listen       80;
    server_name  app2.example.org;
    ...
}

So far I’ve used Nginx Inc’s implementation:

https://github.com/nginxinc/kubernetes-ingress

and the Kubernetes community’s implementation:

https://github.com/kubernetes/ingress-nginx

Viewing the Configuration Inside the Ingress Container

Even if you won’t manually modify the Nginx config file in the container yourself, you should at least understand the config file’s structure for future debugging. This section introduces the Nginx configuration of both Ingress implementations.

After starting the Ingress container, please confirm for yourself which machine your Ingress container is on.

1
2
3
4
5
6
7
8
# Nginx Inc's Ingress container
07b350a2a1d5        nginx/nginx-ingress                                         "/nginx-ingress -ngi…"   16 seconds ago      Up 16 seconds                                 k8s_nginx-ingress_nginx-ingress-9l5sj_nginx-ingress_26f8244c-84eb-4ff9-8040-11b788439f8c_0

# the Kubernetes community's Ingress container
58af9c7aafd1        29024c9c6e70           "/usr/bin/dumb-init …"   17 hours ago        Up 17 hours                             k8s_nginx-ingress-controller_nginx-ingress-controller-5f96cfcb6-4fcxj_ingress-nginx_6498182d-755a-4ce6-b0d1-a3693b59bc11_0

# enter the container
docker exec -ti 07b350a2a1d5 /bin/bash

Nginx Inc’s Nginx Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# config file entry point:
#  /etc/nginx/nginx.conf
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    
    access_log  /var/log/nginx/access.log  main;

    # ...

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;

    # ...
}

The config file above is fairly simple: support for different domains is added via include /etc/nginx/conf.d/*.conf;.

Let’s randomly pick a config file to look at:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# configuration for default/arya-prod-canary

upstream default-arya-prod-canary-arya.example.com-arya-develop-canary-80 {
	random two least_conn;
	server 172.32.3.87:6000 max_fails=1 fail_timeout=10s;
	server 172.32.3.88:6000 max_fails=1 fail_timeout=10s;
}

server {
	listen 80;
	server_tokens on;

	server_name arya.example.com;

	location / {
		proxy_http_version 1.1;
		proxy_connect_timeout 60s;
		proxy_read_timeout 60s;
		client_max_body_size 1m;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Port $server_port;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_buffering on;
		
		proxy_pass http://default-arya-prod-canary-arya.example.com-arya-develop-canary-80;
	}
}

This config file is fairly conventional — just ordinary Nginx reverse proxy configuration: matching domains via different server_names and using proxy_pass to forward to the IP corresponding to the backend pod.

The Kubernetes Community’s Nginx Configuration

Now let’s look at the Kubernetes community’s Nginx configuration — or rather OpenResty configuration. Its /etc/nginx/nginx.conf file is quite long; I’ll excerpt a section to explain:

Below is the configuration for arya.example.com. You can see no pod information at all in the upstream file — all requests are handled by lua. But don’t worry; the following sections will introduce how the Lua scripts obtain pods and handle requests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
upstream upstream_balancer {
	### Attention!!!
	#
	# We no longer create "upstream" section for every backend.
	# Backends are handled dynamically using Lua. If you would like to debug
	# and see what backends ingress-nginx has in its memory you can
	# install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
	# Once you have the plugin you can use "kubectl ingress-nginx backends" command to
	# inspect current backends.
	#
	###
	
	server 0.0.0.1; # placeholder
	
	balancer_by_lua_block {
		balancer.balance()
	}
	
	keepalive 32;
	
	keepalive_timeout  60s;
	keepalive_requests 100;
	
}

## start server arya.example.com
server {
	server_name arya.example.com ;
	
	listen 80  ;
	listen 443  ssl http2 ;
	
	set $proxy_upstream_name "-";
	
	ssl_certificate_by_lua_block {
		certificate.call()
	}
	
	location / {
		
		set $namespace      "default";
		set $ingress_name   "arya-prod";
		set $service_name   "arya-prod";
		set $service_port   "80";
		set $location_path  "/";
		
		rewrite_by_lua_block {
			lua_ingress.rewrite({
				force_ssl_redirect = false,
				ssl_redirect = true,
				force_no_ssl_redirect = false,
				use_port_in_redirects = false,
			})
			balancer.rewrite()
			plugins.run()
		}
		
		header_filter_by_lua_block {
			plugins.run()
		}
		body_filter_by_lua_block {
			
		}
		
		log_by_lua_block {
			balancer.log()
			monitor.call()
			plugins.run()
		}
		
		port_in_redirect off;
		
		set $balancer_ewma_score -1;
		set $proxy_upstream_name "default-arya-prod-80";
		set $proxy_host          $proxy_upstream_name;
		set $pass_access_scheme  $scheme;
		set $pass_server_port    $server_port;
		set $best_http_host      $http_host;
		set $pass_port           $pass_server_port;
		
		set $proxy_alternative_upstream_name "";
		
		client_max_body_size                    1m;
		proxy_set_header Host                   $best_http_host;
		...
		# mitigate HTTPoxy Vulnerability
		...
		
		proxy_pass http://upstream_balancer;
		
		proxy_redirect                          off;
		
	}
	
}
## end server arya.example.com

How the K8s Community’s Ingress Implementation Reads and Writes Config Files

The official doc How it works briefly introduces its implementation: the Ingress controller’s main goal is to generate the config file (Nginx.conf); this behavior requires reloading Nginx after the config file changes. However, the controller can avoid reloading the Nginx config when only upstreams change — this feature is mainly implemented with lua-nginx-module.

Next I’ll analyze the Ingress code to see how it refreshes upstreams without reloading Nginx.

Reference code https://github.com/kubernetes/ingress-nginx/tree/nginx-0.26.1: version 0.26.1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// internal/ingress/controller/controller.go
// syncIngress collects all the pieces required to assemble the NGINX
// configuration file and passes the resulting data structures to the backend
// (OnUpdate) when a reload is deemed necessary.
func (n *NGINXController) syncIngress(interface{}) error {
	// ...
	// get each application's upstream information
	hosts, servers, pcfg := n.getConfiguration(ings)

	if !n.IsDynamicConfigurationEnough(pcfg) {
		//...
		// if upstreams cannot be modified directly via lua, Nginx.conf must be rewritten
		err := n.OnUpdate(*pcfg)
		// ...
	}

	err := wait.ExponentialBackoff(retry, func() (bool, error) {
		// dynamically refresh upstreams
		err := n.configureDynamically(pcfg)
		if err == nil {
			klog.V(2).Infof("Dynamic reconfiguration succeeded.")
			return true, nil
		}
		// ...
	})
	}
	// ...

	n.runningConfig = pcfg

	return nil
}

Let’s look at what operations the configureDynamically function performs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// internal/ingress/controller/nginx.go

// configureDynamically encodes new Backends in JSON format and POSTs the
// payload to an internal HTTP endpoint handled by Lua.
func (n *NGINXController) configureDynamically(pcfg *ingress.Configuration) error {
	backendsChanged := !reflect.DeepEqual(n.runningConfig.Backends, pcfg.Backends)
	if backendsChanged {
		// the backend changed — modify the backend
		// "backend" here means the ip and port behind the http upstream
		err := configureBackends(pcfg.Backends)
		if err != nil {
			return err
		}
	}

	streamConfigurationChanged := !reflect.DeepEqual(n.runningConfig.TCPEndpoints, pcfg.TCPEndpoints) || !reflect.DeepEqual(n.runningConfig.UDPEndpoints, pcfg.UDPEndpoints)
	if streamConfigurationChanged {
		// the stream changed
		// "stream" here means the TCP and UDP services directly reverse-proxied in Nginx
		err := updateStreamConfiguration(pcfg.TCPEndpoints, pcfg.UDPEndpoints)
		if err != nil {
			return err
		}
	}
	// ...

	return nil
}

Since we’ve come this far, let’s take backend updating as an example and check how it interacts with Nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
// internal/ingress/controller/nginx.go
func configureBackends(rawBackends []*ingress.Backend) error {
	backends := make([]*ingress.Backend, len(rawBackends))

	// initialize backend information
	for i, backend := range rawBackends {
		var service *apiv1.Service
		if backend.Service != nil {
			service = &apiv1.Service{Spec: backend.Service.Spec}
		}
		luaBackend := &ingress.Backend{
			Name:                 backend.Name,
			Port:                 backend.Port,
			SSLPassthrough:       backend.SSLPassthrough,
			SessionAffinity:      backend.SessionAffinity,
			UpstreamHashBy:       backend.UpstreamHashBy,
			LoadBalancing:        backend.LoadBalancing,
			Service:              service,
			NoServer:             backend.NoServer,
			TrafficShapingPolicy: backend.TrafficShapingPolicy,
			AlternativeBackends:  backend.AlternativeBackends,
		}

		var endpoints []ingress.Endpoint
		for _, endpoint := range backend.Endpoints {
			endpoints = append(endpoints, ingress.Endpoint{
				Address: endpoint.Address,
				Port:    endpoint.Port,
			})
		}

		luaBackend.Endpoints = endpoints
		backends[i] = luaBackend
	}

	// send a POST request to the Nginx server, asking Nginx to update the backends
	statusCode, _, err := nginx.NewPostStatusRequest("/configuration/backends", "application/json", backends)
	if err != nil {
		return err
	}

	if statusCode != http.StatusCreated {
		return fmt.Errorf("unexpected error code: %d", statusCode)
	}

	return nil
}

// The request-sending code is simple, so I paste it directly below
// internal/nginx/main.go
// NewPostStatusRequest creates a new POST request to the internal NGINX status server
func NewPostStatusRequest(path, contentType string, data interface{}) (int, []byte, error) {
	url := fmt.Sprintf("http://127.0.0.1:%v%v", StatusPort, path)

	buf, err := json.Marshal(data)
	if err != nil {
		return 0, nil, err
	}

	client := http.Client{}
	res, err := client.Post(url, contentType, bytes.NewReader(buf))
	if err != nil {
		return 0, nil, err
	}
	defer res.Body.Close()

	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		return 0, nil, err
	}

	return res.StatusCode, body, nil
}

And how does OpenResty handle this information?

Readers can check the Nginx.conf config file in the OpenResty container; it contains several interfaces for Ingress operations. I won’t list the Nginx config here — I’ll paste the related Lua file directly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
-- rootfs/etc/nginx/lua/configuration.lua
function _M.call()
  if ngx.var.request_method ~= "POST" and ngx.var.request_method ~= "GET" then
    ngx.status = ngx.HTTP_BAD_REQUEST
    ngx.print("Only POST and GET requests are allowed!")
    return
  end

  if ngx.var.request_uri == "/configuration/servers" then
    handle_servers()
    return
  end

  if ngx.var.request_uri == "/configuration/general" then
    handle_general()
    return
  end

  if ngx.var.uri == "/configuration/certs" then
    handle_certs()
    return
  end

  if ngx.var.request_uri ~= "/configuration/backends" then
    ngx.status = ngx.HTTP_NOT_FOUND
    ngx.print("Not found!")
    return
  end

  -- here, if sent as a GET request, you can obtain the current backend data
  if ngx.var.request_method == "GET" then
    ngx.status = ngx.HTTP_OK
    ngx.print(_M.get_backends_data())
    return
  end

  -- non-GET requests execute the statements below
  local backends = fetch_request_body()
  if not backends then
    ngx.log(ngx.ERR, "dynamic-configuration: unable to read valid request body")
    ngx.status = ngx.HTTP_BAD_REQUEST
    return
  end

  -- modify the backend here
  local success, err = configuration_data:set("backends", backends)
  if not success then
    ngx.log(ngx.ERR, "dynamic-configuration: error updating configuration: " .. tostring(err))
    ngx.status = ngx.HTTP_BAD_REQUEST
    return
  end

  ngx.status = ngx.HTTP_CREATED
end

From observing these code segments, you can see how Ingress flushes configuration into Lua variables. If you look even more carefully, you can also learn when nginx.conf gets updated. When using Ingress, try to avoid those operations, so Nginx updates stay as fast as possible.

One more point: the current section only covers how Ingress flushes upstream configuration, not how, when our request arrives, the related configuration is read and sent to the backend. Interested readers can check that part themselves. Through that code you can understand the principle of the canary release described in the last post, as well as some load balancing strategies.

How to Debug the K8s Community’s Ingress Implementation

The last section introduced part of the K8s community Ingress logic. Since this Ingress implementation doesn’t write upstreams directly into the config file, we can’t simply cat the file when debugging. Here I provide two ideas.

Enter the OpenResty Container to View Configuration

Readers may have forgotten: in the last section, requesting the backend via GET makes the lua script return the current configuration information, which definitely includes the backend IPs and ports.

1
curl 127.0.0.1:10246/configuration/backends

Using the kubectl Plugin

I found this plugin in a recent search; you can use it to read Ingress configuration — a fairly universal method.

https://kubernetes.github.io/ingress-nginx/kubectl-plugin/

1
kubectl ingress-nginx backends -n ingress-nginx

Summary

This blog mainly introduced the two Ingress implementations and explained the K8s community’s Ingress implementation in detail, basically presenting the backend-updating code to everyone. Of course no one can guarantee the future Ingress won’t change greatly. As things stand, the lua scripts already implement quite a lot; my usage direction should be adding some access policy restrictions on this basis, such as rate limiting or IP banning.

Related Posts