Use iptables, not ipvs or userspace in kube-proxy

Introduction

Thank you for clicking through to my arcticle. I've been a DevOps engineer for 2 years in dev-team of 7 engineers.

My name is MINSEOK, LEE, but I use Unchaptered as an alias on the interenet. So, you can call me anythings "MINSEOK, LEE" or "Unchaptered" to ask something.

[References]

1. 김징어의 Devlog | The 3 modes that kube-proxy manage network(userspace, iptables, IPVS)
   [k8s] kube-proxy가 네트워크를 관리하는 3가지 모드(userspace, iptables, IPVS)

2. TIGREA | Comparing kube-proxy modes: iptables or IPVS?

Topic

1. What's the kube-proxy

2. The "userspace" mode of kube-proxy.

3. The "iptables" mode of kube-proxy.

4. The "IPVS" mode of kube-proxy.

What's the kube-proxy

The kube-proxy is the component that does the actual manipulation to allow access into ClusterIP and NodePort, in services of kubernetes.

Initially, userspace was the default management mode.
In Dec 2019, iptables the default management mode.
In the future, we expect the default management mode to change from iptables to IPVS.

The "userspace" mode of kube-proxy.

1. Client send a request into Cluster ID.

2. Kube-proxy receive the request, is passing through iptables.

3. Kube-proxy support load balancing using Round Robin.

4. All distributes request is processing by each pods.

When connection requests is failed?
The kube-proxy try to re-connect other pods.

The "iptables" mode of kube-proxy.

1. Client send a request.

2. Iptables forward request into each pods.

What's the difference between userspace and iptables?

The kube-proxy manage iptables, only.
All requests is directly forwarded into each pods.
So, iptables performs better than userspace.

When connection requests is failed?
The connection requests is failed.

The "IPVS(IP Virtual Server)" mode of kube-proxy.

The IPVS mode is an L4 load balancing technology in the Linux Kernel.

It's included in Netfilter, the networking framework within the Linux Kernel.
Therefore, the IPVS kernel module must be installed on the node.

IPVS mode is faster and performs better than iptables mode, because it operates in kernel space and stores data structures as hash tables. It also has more load balancing algorithms available to take advantage of.

Load Balancing Algorithm

1. rr(round robin)
   Allocates CPU in order and time, with no prioritization among processes.

2. lc(least connection)
   Selects the server with the least number of connections.

3. dh(destination hashing)

   Computes a hash value with the destination ip address to select the physical server to distribute to.

4. sh(source hashing)
   Compute a hash value with the source IP address to select the real server to distribute.

5. sed(shortest expected delay)

   Select the server with the fastest response time.

6. nq(never queue)

   Similar to sed, but chooses the first server with zero active connections.

Potential Risk of IPVS

The packet, processed by IPVS, is very different with basic packet, processing by iptables' filter hook

If you want to use IPVS with application, used iptables, you would check "Is IPVS compatible with other application used iptables".

Algorithm Complexity of iptables and IPVS

1. The iptables is B(n) complexity

2. The IPVS is B(1) complexity

Performance

1. Impact on round-trip response times. 
   When one microservice makes an API call to another microservice, how long does it take on average for the first microservice to send the request to and receive the response back from the second microservice?

2. Impact on total CPU usage. 
   What is the total CPU usage of a host when running your microservices, including userspace and kernel/system usage, across all the processes that are needed to support your microservices including kube-proxy?

Round-Trip Response Times

• The difference in average round-trip response times between iptables and IPVS is trivially insignificant until you get beyond 1,000 services (10,000 backend pods).

• The difference in average round-trip response times is only discernible when not using keepalive connections. i.e. when using a new connection for every request.

Total CPU

• The difference in CPU usage between iptables and IPVS is relatively insignificant until you get beyond 1,000 services (with 10,000 backend pods).

• At 10,000 services (with 100,000 backend pods), the increase in CPU with iptables is ~35% of a core, and with IPVS is ~8% of a core.

Conclusion

In general situation, the choice that use iptables is more stable and efficient than userspace and IPVS.

The only place where IPVS performs noticeably better than iptables is when the number of services exceeds 1,000.

Given the system requirements and complexity introduced by IPVS, iptables is safe in most cases.