This event happened in March 2021.
Recently one machine in our self-built Kubernetes cluster was intruded and used for mining; we later found the cause — fortunately it was only used for mining…
Network security is a serious matter: it always appears when you least expect it, and by the time you react it’s too late. I hope readers also gain inspiration to check and harden their own clusters.
Intrusion Phenomena
An abnormal process was detected on a machine:
1 | ./.system -o pool.supportxmr.com:3333 --donate-level=1 --coin=monero -u 46EPFzvnX5GH61ejkPpNcRNm8kVjs8oHS9VwCkKRCrJX27XEW2y1NPLfSa54DGHxqnKfzDUVW1jzBfekk3hrCVCm |
Simply put, our machine was being used for mining…
After the problem appeared, we immediately shut down docker. Actually we should have isolated the environment and dumped the mining program for later analysis.
Investigating the Specific Cause
Empty iptables
With an abnormal process present, there was definitely an intrusion. The first thing I checked was iptables.
Sure enough, the machine’s iptables rules were empty — meaning this machine was streaking.
kubelet Streaking
An internal colleague suggested kubelet might have been intruded. After checking other components, we began checking the kubelet component.
Finally we found anomalies in the kubelet logs:
Improper kubelet Settings
Confirming the intrusion problem: kubelet parameters were set wrongly, allowing direct access to kubelet’s api.
We found that in kubelet’s startup options, this position was commented out:
And then the config in the file forbidding anonymous access was not read:
This configuration was commented out by my own improper operation.
Since it was a newly added machine, the problem was found that same night. I manage the whole cluster and investigated along with everyone, so the cause was found quickly. That night I rescanned the configuration items on all other machines — if their firewalls had failed, similar intrusions would have occurred. Fortunately this event was contained to 1 machine.
Improvement Plan
Actually this problem could theoretically have been avoided; it took multiple layers of vulnerabilities for a malicious scanner to find it. I’ve organized possible improvement strategies from outside in.
- Machine firewall settings: the machine firewall is the outermost layer of the whole system. Even if the machine’s firewall sync fails, it must not default to opening all ports — everything should be closed, waiting for the administrator to connect to the tty terminal to check.
- When using machines: if a machine isn’t exposed for external use, and a public IP is optional, try not to have a public IP. Our machine was scanned for vulnerabilities after only 1 day online — imagine how dangerous the public network is.
- When using kubelet and other system services, shouldn’t port listening be considered?
Can we avoid listening on
0.0.0.0and only listen on the machine’s internal IP? - When using kubelet and other programs, designing or building systems: for anonymous access permission control, we need to consider what problems arise if a port is anonymous, whether anonymous access should be allowed, and if not, how to build an authentication system?
- When system administrators operate, is there a fairly standardized process? Should production environments only be operated via scripts? Problems brought by manual operations on production are hard to investigate and locate.
I’m not just throwing out questions here — I want to tell everyone that when considering system design, security deserves consideration.
Summary
After the intrusion event, a colleague joked: good thing there were no other economic losses, otherwise I might have been sent home. As the cluster administrator, only I know best how serious the problem was. Essentially, the problem was already quite serious: the intruder effectively had complete control of docker on the machine. If readers have read my docker series content, you understand the permissions clearly.
Because of this event, not only I but basically all the SA colleagues got chewed out. It still stings a bit. I hope everyone takes network security seriously: start with hardening firewalls and avoiding listening on unnecessary ports — these two are at least the easiest to implement.