CNI Performance Test Report¶

This test report covers the performance testing of CNIs including Cilium, Calico and Macvlan.

Test background¶

Test environment¶

Tested CNI types¶

This test covers three CNIs including Cilium, Calico and Macvlanis and is based on the following models:

• Cilium
  • cilium (vxlan mode)
• Calico
  • calico (vxlan mode)
  • calico (underlay mode)
  • callico (ipip mode)
• Macvlan-Standalone

Test cases¶

This test includes the following nine Test cases:

• Node to node
• Pod to node where the Pod resides
• Pod to node across nodes
• Pod to Pod on the same node
• Pod to Pod across nodes
• Node to Service, and the Endpoint corresponding to the Service is on this node
• Node to Service, and the Endpoint corresponding to the Service is on other nodes
• Pod to Service, the Endpoint corresponding to the Service is on the same node as this Pod
• Pod to Service, and Service corresponding Endpoint and this Pod are on different nodes

Test metrics¶

• Pod throughput (Pod to node)
• Service throughput (Service to node)
• Latency - long connection (Pod to node)
• Latency - short connection (Pod to node)

Testing tools and commands¶

This test uses the netperf tool for performance testing, and the test commands for different metrics are as follows:

• Throughput

  netperf -t TCP_STREAM -H <serverIP> -p <serverPort> -l 60 -- -m 1024
  iperf3 -c <serverIP> -i 1 -t 60
  

• Time delay - long connection

  netperf -t TCP_RR -H <serverIP> -p <serverPort> -l 60 -- -r 1024 -O "MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY. STDDEV_LATENCY"
  

• Time delay - short link

  netperf -t TCP_CRR -H <serverIP> -p <serverPort> -l 60 -- -r 1024 -O "MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY. STDDEV_LATENCY"
  

Test results¶

For the four test metrics, the test results for different CNI patterns in different test cases are shown below:

Chart Conclusion¶

• Pod throughput

  Test cases	Conclusions
  Pod to node（the same node）	Macvlan-standalone has the highest throughput, while cilium vxlan has the lowest
  Pod to node（different nodes）	Calico-ipip, calico-underlay and Macvlan-standalone outperform than calico-vxlan and cilium-vxlan
  Pod to Pod（the same node）	Macvlan outperforms cilium, and calico has similarly worse performance in all three modes
  Pod to Pod（different nodes）	Macvlan is the best, followed by cilium-vxlan and then calico

• Service throughput

  Test cases	Conclusions
  Service to node, both on the same node	Macvlan is the best and followed by cilium, and calico has similarly worse performance in three modes
  Service to node, both on different nodes	Macvlan, calico-ipip, calico-underlay, cilium-vxlan have similar performance, and calico-vxlan is the worst
  Service to Pod, both on the same node	Macvlan has the best performance and is followed by cilium. Calico has similarly worse performance in all three modes
  Service to Pod（different nodes）	Both macvlan and cilium perform better than calico

• Pod latency

  Test cases	Conclusions
  Pod to node（the same node）	The five CNI modes perform similarly, with the lowest macvlan latency
  Pod to node（different nodes）	Macvlan has the best performance, cilium-vxlan is the second best, and calico-vxlan has the highest latency
  Pod to Pod（the same node）	Macvlan outperforms cilium, and calico has similarly worse performance in all three modes
  Pod to Pod（different nodes）	Macvlan has the best performance, followed by calico-underlay and then calico-vxlan

• Pod short traces

  Test cases	Conclusions
  Pod to node（the same node）	The five cni modes has similar latencies, macvlan and calico-underlay has the smallest one
  Pod to node（different nodes）	Macvlan outperforms cilium-vxlan, and calico-vxlan has the highest latency
  Pod to Pod（the same node）	Macvlan outperforms cilium, and calico has similarly worse performance in all three modes
  Pod to Pod（different nodes）	Macvlan has the best performance, followed by calico-underlay and then calico-vxlan

Detailed data¶

• Pod-related throughput (Gbits/sec)

  Test cases	Cilium(vxlan)	Calico(vxlan)	Calico(ipip)	Calico(underlay)	Macvlan-standalone	Sriov-standalone
  Node to node	9.16	9.16	9.16	9.16	9.16	9.16
  Pod to node（the same node）	12.6	13.8	14.23	14.2	16.4	15.8
  Pod to node（different nodes）	3.02	1.57	8.24	9.2	9.15	9.09
  Pod to Pod（the same node）	15.3	10.2	10.33	10.27	22.2	14.1
  Pod to Pod（different nodes）	8.30	1.37	4.51	3.89	9.18	9.21

• Service related throughput (Gbits/sec)

  Test cases	Cilium(vxlan)	Calico(vxlan)	Calico(ipip)	Calico(underlay)	Macvlan-standalone	Sriov-standalone
  Service to node（the same node）	8.30	12	11.77	12.07	16.6	15.8
  Service to node（different nodes）	7.41	1.23	8.57	9.037	9.14	9.10
  Service to Pod（the same node）	15.8	10.2	10	10.17	20.2	15.6
  Service to Pod（different nodes）	8.27	1.3	4.81	3.49	9.15	16.3

• netpref latency (long connection) (Microseconds)

  Test cases	Cilium(vxlan)	Calico(vxlan)	Calico(ipip)	Calico(underlay)	Macvlan-standalone	Sriov-standalone
  Node to node	113.27	113.27	113.27	113.27	113.27	113.27
  Pod to node（the same node）	76.76	76.81	74.55	62.27	61.25	74.34
  Pod to node（different nodes）	150.26	241.46	225.68	170.23	95.76	109.77
  Pod to Pod（the same node）	65.62	104.82	91.44	103.71	45.47	66.74
  Pod to Pod（different nodes）	201.27	275.22	248.72	195.61	81.74	78.83

• netpref latency (short connection) (Microseconds)

  Test cases	Cilium(vxlan)	Calico(vxlan)	Calico(ipip)	Calico(underlay)	Macvlan-standalone	Sriov-standalone
  Node to node	439.04	439.04	439.04	439.04	439.04	439.04
  Pod to node（the same node）	259.92	230.65	226.26	203.63	187.41	225.66
  Pod to node（different nodes）	613.59	803.80	767.08	587.53	358.48	413.77
  Pod to Pod（the same node）	198.99	982.92	265.07	277.75	127.48	275.52
  Pod to Pod（different nodes）	789.32	940.65	857.6	628.21	290.59	284.16

Comments