Millet Porridge

English version of https://corvo.myseu.cn

0%

Diverse Uses of kubectl

Posted on Edited on In Disqus:
kubectl is the official K8s command-line tool for conveniently operating K8s clusters. This article introduces some unconventional kubectl usages — not that you must try them, just that when needed you can remember kubectl may have similar features, instead of writing a new tool on the client-api.

kubectl is the official K8s command-line tool for conveniently operating K8s clusters. This article mainly introduces some unconventional kubectl usages. I hope readers have basic K8s usage experience.

One article also introduces some tricks; I happened to find it while writing this blog, so I’ll share it too:

Ready-to-use commands and tips for kubectl

Printing the APIs Currently Used

1
2
3
# kubectl's main job is interacting with the ApiServer; we can print the interaction process like this.
# This command is especially suitable when debugging your own api interfaces.
kubectl get ns -v=9

20210816195205

Filtering Containers by Status and Deleting Them

This is a command I learned here: Force Delete Evicted / Terminated Pods in Kubernetes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl get pods --all-namespaces --field-selector status.phase=Pending -o json | \
  jq '.items[] | "kubectl delete pods \(.metadata.name) -n \(.metadata.namespace)"' | \
  xargs -n 1 bash -c


# This command should be taken apart:
# first, get all pods in Pending status across all namespaces, output as json
# this statement actually has many variants — e.g. I want to find Failed status, or a certain deployment
kubectl get pods --all-namespaces --field-selector status.phase=Pending -o json 

# process the json variable to generate usable scripts
# here's the point I want to introduce: using jq and kubectl's output to build usable commands
jq '.items[] | "kubectl delete pods \(.metadata.name) -n \(.metadata.namespace)"'

# execute each command
# note: such commands must be well debugged — deleting unexpected pods would be bad.
xargs -n 1 bash -c


# for example, the statement below finds all Pods and prints executable statements
kubectl get pods --all-namespaces --field-selector status.phase=Running -o json | \
  jq '.items[] | "kubectl get pods \(.metadata.name) -o wide -n \(.metadata.namespace)"'

"kubectl get pods metrics-server-6d684c7b5-gtd6q -o wide -n kube-system"
"kubectl get pods local-path-provisioner-58fb86bdfd-98frc -o wide -n kube-system"
"kubectl get pods nginx-deployment-574b87c764-xppmx -o wide -n default"

# of course, if only deleting some pods under a single NS, I'd choose the method below — but it's very inconvenient for multiple NSes
kubectl -n default get pods  | grep Completed | awk '{print $1}' | xargs kubectl -n default delete pods

Counting All Pods Running on a Specific Machine

kubectl can use two kinds of selectors: label and field. See the official site’s introductions:

1
2
3
# it's a kind of selector that can be combined with the awk or xargs above
# personally I don't like using it — directly getting all pods and grepping feels faster
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=pve-node1

Counting the Pod Distribution Across Machines

I don’t know whether readers have seen my article: Implementing fine-grained pod control on a kubernetes-based PaaS platform. The prerequisite for balanced distribution work is knowing the pod distribution across machines. The best way is simply counting after obtaining pod information — this work can be done with awk.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
kubectl -n default get pods -o wide -l app="nginx" | awk '{print $7}'|\
 awk '{ count[$0]++  } 
 END { 
   printf("%-35s: %s\n","Word","Count");
   for(ind in count){
    printf("%-35s: %d\n",ind,count[ind]);
   }
 }'

# execution result below
Word                               : Count
NODE                               : 1
pve-node1                          : 1
pve-node2                          : 1


# I haven't studied awk's syntax deeply; interested readers can research it — I won't dig further here.

Using kubectl proxy

You can understand this command as proxying K8s’s ApiServer; using this proxy you can call APIs directly without authentication. After starting it, you can even achieve kubectl matryoshka. Here’s an example:

1
2
3
4
5
6
7
8
9
10
11
12
# when you call kubectl directly without setting kubeconfig
kubectl get ns -v=9
# it can print an error like the following
curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.21.3 (linux/amd64) kubernetes/ca643a4" 'http://localhost:8080/api?timeout=32s'
skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused                     
# that is, when you don't specify a kubeconfig file, kubectl defaults to accessing the local 8080 port
# so let's first start a kubectl proxy listening on 8080, then access directly with kubectl — will it work?
# Facts prove it works as expected.
KUBECONFIG=~/.kube/config-symv3 kubectl proxy  -p 8080
kubectl get ns
NAME                           STATUS   AGE
default                        Active   127d

The default-started proxy blocks certain APIs and has some restrictions — e.g. you can’t use exec to enter pods. Use kubectl proxy --help to see, for example:

1
2
3
4
5
6
7
# only allow local access
--accept-hosts='^localhost$,^127\.0\.0\.1$,^\[::1\]$': Regular expression for hosts that the proxy should accept.
# don't allow access to the APIs below — i.e. by default you can't exec into containers
--reject-paths='^/api/.*/pods/.*/exec,^/api/.*/pods/.*/attach': Regular expression for paths that the proxy should reject. Paths specified here will be rejected even accepted by --accept-paths.

# skipping the exec restriction is also simple — just remove reject-paths
kubectl proxy -p 8080 --keepalive 3600s --reject-paths='' -v=9

Some say this kubectl proxy may be useless — that may just mean you don’t yet have an actual application scenario. For example, when I want to debug K8s dashboard code: using the kubeconfig file directly, I can’t see the concrete request process. If you add a proxy forwarding layer and set -v=9, you automatically get a logging tool — quite useful when debugging.

Summary

kubectl is a powerful command-line tool. Above I only introduced a bit of my exploration of its usage at work. I don’t encourage everyone to memorize these commands — I just hope that when readers need it, they can remember kubectl may have similar features, so there’s no need to study the client-api for a few temporary requirements.

Related Posts