From networking to CNI and L-LIPAM

Introduction

Thank you for clicking through to my arcticle. I've been a DevOps engineer for 2 years in dev-team of 7 engineers.

My name is MINSEOK, LEE, but I use Unchaptered as an alias on the interenet. So, you can call me anythings "MINSEOK, LEE" or "Unchaptered" to ask something.

[Notice]

This article was referenced in these posts.

1. Amazon VPC CNI (GitHub) | CNI Proposal

2. eljoelee.log (Blog) | AEWS STUDY - 2주차 Network

K8S Network Model

K8S (Blog) | Cluster Networking
K8S (Blog) | Kubernetes Network Model

Every pod, in cluster, gets its own unique cluster-wide IP address.
You almost never to deal with mapping pod(container) port to host ports.

It creates a clean, backwards-compatible model where each pods can be treated VMs or physical hosts from the perspectives:

1. port allocation

2. naming

3. search service

4. load balancing

5. application configuration

6. application migration

Fundamental Requirements of K8S Network Model

(Except barring any intentional network segmentation policies)
The k8s imposes the following fundamental requirements.

• pods can communicate with all other pods on any other node without NAT.

• agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node.

The k8s' networking these four issues:

• Containers within a Pod use networking to communicate via loopback.

• Cluster networking provides communication between different Pods.

• The Service API lets you expose an application running in Pods to be reachable from outside your cluster.

  • Ingress provides extra functionality specifically for exposing HTTP applications, websites and APIs.

  • Gateway API is an add-on that provides an expressive, extensible, and role-oriented family of API kinds for modeling service networking.

• You can also use Services to publish services only for consumption inside your cluster.

What's the K8S CNI?

The CNI(Container Network Interface) configure network environment of k8s.
For CNI function, K8S supports these addons.

What's the Amazon VPC CNI?

The Amazon VPC CNI allowcate IP Address to each pods.
The CNI makes direct communicationpossible between pods and node(worker), because pods' IP bandwidth is same with node(worker).

• kubelet ↔ VPC CNI | Append/remove CNI

• VPC CNI ↔ L-IPAM(Wa rm IP Pool) | Append/remove IP adress.

What's the difference between Calico CNI vs CNI

• Configure the same network bandwidth for nodes and pods to optimize network communication (performance, latency)

• Communication between pods

  • K8S CNI : Overlay(VXLAN, IP-IP etc) Communication

  • AWS VPC CNI : Direct Communication between each pods.

What is IPAM?

• Amazon IPAM is used for ip administartion & trace tool. For more information, see What's the IPAM.

What's the L-IPAM?

• The kubernetes uses L-IPAM to maintain a Warm Pool of secondary ip addresses available to each node.

• Whenever L-IPAM receives a request from the kubelet for adding a Pod...

  • It allocate secondary ip address to pod.

  • It pull secondary ip address from warm pool.

  • It build warm pool.

    • It determines the available ENI and secondary ip addresses from the node's metadata.

    • It pull running pod's data when DaemonSet restarted. For example, pod's name, namespace and ip address.

Prac : check basic info of network

1. Check CNI Information

   1. Entire name

      kubectl describe daemonset aws-node --namespace kube-system | grep Image

      

   2. Target name

      kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

      

2. Check config of kube-proxy with iptables.
   → references : unchaptered (blog) | Why don't you use ipvs?

   kubectl describe cm -n kube-system kube-proxy-config

   

3. Check "Do pods and nodes have the same bandwidth?"

   1. Get list of pods' ip

      kubectl get pod -n kube-system -o=custom-columns=NAME:.metadata.name,IP:.status.podIP,STATUS:.status.phase

   2. Get list of nodes' ip

      aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table

      

4. Other utility codes

   1. Get name-list of pods.

      kubectl get pod -A -o name

   2. Count pods.

      kubectl get pod -A -o name | wc -l

Prac : Check network information of node

1. Check CNI info

   for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i tree /var/log/aws-routed-eni; echo; done
   ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/plugin.log | jq
   ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/ipamd.log | jq
   ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/egress-v6-plugin.log | jq
   ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/ebpf-sdk.log | jq
   ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/network-policy-agent.log | jq

2. Check network info

   for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -br -c addr; echo; done
   for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done
   for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done
   ssh ec2-user@$N1 sudo iptables -t nat -S
   ssh ec2-user@$N1 sudo iptables -t nat -L -n -v

Conclusion

• Amazon CLI is more efficient than CalicoCNI.

• The kubernetes manages IP addresses efficiently through its internal L-IPAM.