Millet Porridge

English version of https://corvo.myseu.cn

0%

K8s API Gateway And Functional Outlook

Posted on Edited on In Disqus:
In fact, we often consider the function expansion of API gateway for our PaaS platform. Recently, I start to look into the API gateway products. In this blog, I will briefly introduce the function and give some considerations for the future development. I hope to be able to attract more ideas. I would like readers to share their prospects for API gateways.

In fact, we often consider the function expansion of API gateway for our PaaS platform. Recently, I start to look into the API gateway products. In this blog, I will briefly introduce the function and give some considerations for the future development. I hope to be able to attract more ideas. I would like readers to share their prospects for API gateways.

Simple Function in API Gateway

The current API gateways in our PaaS system are Istio Inagress Gateway and Kubernetes Community Ingress. There are a few additional functions for these two implements, and we just use them to expose the interface. We only use the grayscale and traffic forwarding, and the authentication and rate limit do not rely on them at all.

Kubernetes Community Ingress

The Ingress NGINX Controller we use is created by Kubernetes Community which based on OpenResty.

https://kubernetes.github.io/ingress-nginx/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: arya-prod
spec:
  rules:
  - host: arya.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: arya-svc
          servicePort: 80

Istio Ingress Gateway

Due to some reasons, we need to use Istio Ingress Gateway. It works just like Ingress, but need to use Gateway and VirtualService to configure.

https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# An ingress Gateway describes a load balancer operating at the edge of the mesh that receives incoming HTTP/TCP connections
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: arya-gateway
  namespace:  xxx-group-2
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "arya-python3.istio.xxx.com"

# # Virtual Service configure the route rules
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: arya-virtual-service
  namespace:  xxx-group-2
spec:
  hosts:
  - "arya-python3.istio.xxx.com"
  gateways:
  - arya-gateway
  http:
  - match:
    - uri:
        prefix: /static
    route:
    - destination:
        host: prod-arya-python3-static
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: prod-arya-python3

Current API gateway products

Well, An API gateway could not just expose the service, but also have the ability to control the traffic, including authentication, rate limit, and so on. A good gateway also need to allow user add their own plugins.

Kong

Kong Doc

Kong use OpenResty as its backend, and it has a plugin system. also support Python, Golang plugins.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
---
apiVersion: configuration.konghq.com/v1
plugin: openid-connect
kind: KongPlugin
metadata:
  name: oidc-auth
config:
  issuer: https://accounts.google.com/.well-known/openid-configuration
  client_id:
  - <client-id>
  client_secret:
  - <client-secret>
  redirect_uri:
  - http://192.0.2.8.xip.io
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: productpage
  namespace: bookinfo
  annotations:
    konghq.com/plugins: oidc-auth  # could be an array to contain the plugins
spec:
  ingressClassName: kong
  rules:
  - http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: productpage
            port:
              number: 9080

I appreciate its interactive logic, when you have multiple domains, each domain can set its own enabled plugins and their configuration. It is a mature commercial product, so the usability is still very strong, but unfortunately it does not support Wasm yet.

Istio

Istio docs

Istio use Envoy as it’s backend, and support wasm plugins. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: openid-connect
  namespace: istio-ingress
spec:
  selector:
    labels:
      istio: ingressgateway
  url: oci://private-registry:5000/openid-connect/openid:latest
  imagePullPolicy: IfNotPresent
  imagePullSecret: private-registry-pull-secret
  phase: AUTHN
  pluginConfig:
    openid_server: authn
    openid_realm: ingress

Currently, you can’t add a plugin for a single VirtualService, and it only supports the IngressGateway selector. If you want many plugins, its logic is controlled by proiority, the interaction is very complex.

Several possible directions and technology pre-study

I have done some brief research on several possible solutions. It’s possible to have implementation strategies for all the solutions I posted here.

Optimizations for Kong

In terms of interaction logic and usability alone, I don’t think there’s much to optimize, the only thing I find difficult is the inability to use Wasm type plugins.

A few methods:

  1. Just like Kong’s Golang or Python plugin use rpc to call wasm plugin
  2. Use Lua to implement a wasm vm, run Wasm code directly
  3. Wasm embed into OpenResty, here is a solution: https://github.com/api7/wasm-nginx-module

IMHO, I assume this is the latter two that the Kong team will consider. I have used the Python plugin, it seems a little hard to use and introduce some maintenance issues. The latter two are more easy to maintain and may have better performance.

Optimizations for Istio

The interaction logic

If we want to use Istio as an API gateway, we need to use VirtualService to configure the route rules. Use annotations to configure the plugins like Kong is enough.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
---
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: openid-connect
  namespace: xxx-group-2 # In the same namespace with VirtualService
spec:
  imagePullPolicy: IfNotPresent
  imagePullSecret: private-registry-pull-secret
  phase: AUTHN
  pluginConfig:
    openid_server: authn
    openid_realm: ingress
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: arya-virtual-service
  namespace: xxx-group-2  
  annotations:
    istio.io/plugins: openid-connect
spec:
  hosts:
  - "arya-python3.istio.xxx.com"
  gateways:
  - arya-gateway
  http:
  - match:
    - uri:
        prefix: /static
    route:
    - destination:
        host: prod-arya-python3-static
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: prod-arya-python3

Envoy supports Lua

Why we want lua The plugin environment in kong is Almost all Lua plugins are certified for enterprise level, and if they can be used directly in Istio+Envoy, I believe many users will be willing to use Envoy as a gateway.

There are two methods to support Lua plugin:

  1. Convert Lua to wasm, according to the information I have gathered, there is no ready-made compiler, you need to write it from scratch

  2. Recently, there is a project named rider created by Netease team that supports running Lua code in Envoy. It means we can emulate the Kong’s PDK to use its plugins

https://xie.infoq.cn/article/1cb74a7512460b7d4dbc9f42c

Extended sidecar

In my blog, there are many articles about PaaS systems and I’m currently maintaining a PaaS system. I will give a solution from a PaaS user’s point of view. The two above solutions need us to update the ingress controller and the istio. What’s worse, we need to write some code to support the new features. A simple project could wait for the solution, but we can’t, we need to meet the user’s factual needs.

If we want to have a functional API gateway for every apps. The good solution is to add our own sidecar like Istio, but only for the the API traffic. This sidecar could import user config with kind of Nginx, Lua or Wasm, and these configs could be placed in the git repo.

And consider this method:

  1. Performance, sidecar with pod easy to exntend and theoretically it has the same performance as Istio’s
  2. Extensibility, we gave the power for use to controll their traffic more precisely.

20220417142319

Personal view and PaaS platform direction

The technology itself is not good or bad, only suitable or unsuitable.

For PaaS platform, scalability and simple interaction logic are important. There are thoundouds of apps in our PaaS system with kinds of demands. We cannot hope to extend the platform functionality by using Istio’s description files exclusively, but always provide a solution that works.

For my personal view, when I use K8s to hold a few small projects, I would like to have some functionality plugged in. I’m interested in Wasm plugin support.

In my spare time, I would consider improving Istio’s controller with a better interaction logic.

Conclusion

I’d like the reader consider this article as a research report about the API gateway. I hope you could share some ideas and thoughts with me.

Related Posts