16.kubernetes笔记|16.kubernetes笔记 CNI网络插件(二) Calico介绍
Calico简介
Calico 是一种容器之间互通的网络方案。在虚拟化平台中，比如 OpenStack、Docker 等都需要实现 workloads 之间互连，但同时也需要对容器做隔离控制，就像在 Internet 中的服务仅开放80端口、公有云的多租户一样，提供隔离和管控机制。而在多数的虚拟化平台实现中，通常都使用二层隔离技术来实现容器的网络，这些二层的技术有一些弊端，比如需要依赖 VLAN、bridge 和隧道等技术，其中 bridge 带来了复杂性，vlan 隔离和 tunnel 隧道则消耗更多的资源并对物理环境有要求，随着网络规模的增大，整体会变得越加复杂。我们尝试把 Host 当作 Internet 中的路由器，同样使用 BGP 同步路由，并使用 iptables 来做安全访问策略，最终设计出了 Calico 方案。
设计思想：Calico 不使用隧道或 NAT 来实现转发，而是巧妙的把所有二三层流量转换成三层流量，并通过 host 上路由配置完成跨 Host 转发
设计优势：
1.更优的资源利用
二层网络通讯需要依赖广播消息机制，广播消息的开销与 host 的数量呈指数级增长，Calico 使用的三层路由方法，则完全抑制了二层广播，减少了资源开销。
2.可扩展性
Calico 使用与 Internet 类似的方案，Internet 的网络比任何数据中心都大，Calico 同样天然具有可扩展性。
3.简单而更容易 debug
因为没有隧道，意味着 workloads 之间路径更短更简单，配置更少，在 host 上更容易进行 debug 调试。
4.更少的依赖
Calico 仅依赖三层路由可达。
5.可适配性
Calico 较少的依赖性使它能适配所有 VM、Container、白盒或者混合环境场景。
Calico 网络Node之间通信网络
IPIP(可跨网段通信)
从字面来理解，就是把一个IP数据包又套在一个IP包里，即把 IP 层封装到 IP 层的一个 tunnel。它的作用其实基本上就相当于一个基于IP层的网桥！一般来说，普通的网桥是基于mac层的，根本不需 IP，而这个 ipip 则是通过两端的路由做一个 tunnel，把两个本来不通的网络通过点对点连接起来。
类似vxlan 但封装开销比vxlan小 效率相对更高一些，但安全性也更差
Vxlan(可跨网段通信)
与Flannel Vxlan原理相同
BGP(二层网络通信)
边界网关协议（Border Gateway Protocol, BGP）是互联网上一个核心的去中心化自治路由协议。它通过维护IP路由表或‘前缀’表来实现自治系统（AS）之间的可达性，属于矢量路由协议。BGP不使用传统的内部网关协议（IGP）的指标，而使用基于路径、网络策略或规则集来决定路由。因此，它更适合被称为矢量性协议，而不是路由协议。BGP，通俗的讲就是讲接入到机房的多条线路（如电信、联通、移动等）融合为一体，实现多线单IP，BGP 机房的优点：服务器只需要设置一个IP地址，最佳访问路由是由网络上的骨干路由器根据路由跳数与其它技术指标来确定的，不会占用服务器的任何系统
实际上，Calico 项目提供的 BGP 网络解决方案，与 Flannel 的 host-gw 模式几乎一样。也就是说，Calico也是基于路由表实现容器数据包转发，但不同于Flannel使用flanneld进程来维护路由信息的做法，而Calico项目使用BGP协议来自动维护整个集群的路由信息。
部署推荐方案:
BGP+Vxlan
其中BGP在官方的推荐方案中 以50个节点为界区别了不同规模使用不同的部署方案
- 小规模网络:BGP peer 一对一网络：每个节点都是有N-1条路由，小型网络适用，当节点数N变多时，路由表更新及AIP-SERVER都需要承受很大的压力 类似网络拓扑结构中的 网状拓扑结构
- 大规模网络:BGP Reflector 路由反射器：选择一到多个节点做为Reflector，所有节点路由都汇总给Reflector，所有节点都路由都指向Reflector ，适合大型网络，类似网络拓扑结构中的星型网络
- Felix：运行在每一台 Host 的 agent 进程，主要负责网络接口管理和监听、路由、ARP 管理、ACL 管理和同步、状态上报等。
- etcd：分布式键值存储，主要负责网络元数据一致性，确保Calico网络状态的准确性，可以与kubernetes共用；
- BGP Client（BIRD）：Calico 为每一台 Host 部署一个 BGP Client，使用 BIRD 实现，BIRD 是一个单独的持续发展的项目，实现了众多动态路由协议比如 BGP、OSPF、RIP 等。在 Calico 的角色是监听 Host 上由 Felix 注入的路由信息，然后通过 BGP 协议广播告诉剩余 Host 节点，从而实现网络互通。
- BGP Route Reflector：在大型网络规模中，如果仅仅使用 BGP client 形成 mesh 全网互联的方案就会导致规模限制，因为所有节点之间俩俩互联，需要 N^2 个连接，为了解决这个规模问题，可以采用 BGP 的 Router Reflector 的方法，使所有 BGP Client 仅与特定 RR 节点互联并做路由同步，从而大大减少连接数。
- 是让calico/node独立运行于Kubernetes集群之外，但calico/kube-controllers依然需要以Pod资源运行中集群之上;
- 是以CNI插件方式配置Calico完全托管运行于Kubernetes集群之上，类似于我们前面曾经部署托管Flannel网络插件的方式。
对于后一种方式,Calico提供了在线的部署清单，它分别为50节点及以下规模和50节点以上规模的Kubernetes集群使用Kubernetes API作为Dabastore提供了不同的配置清单，也为使用独立的etcd集群提供了专用配置清单。但这3种类型的配置清单中,Calico默认启用的是基于IPIP隧道的叠加网络，因而它会在所有流量上使用IPIP隧道而不是BGP路由。以下配置定义在部署清单中DaemonSet/calico-node资源的Pod模板中的calico-node容器之上。
在IPv4类型的地址池上启用的IPIP及其类型，支持3种可用值
Always(全局流量)、Cross-SubNet(跨子网流量)和Never3种可用值
- name: CALICO_IPV4POOL_IPIP
value: "Always" - 是否在IPV4地址池上启用VXLAN隧道协议，取值及意义与Flannel的VXLAN后端相同; 但在全局流量启用VXLAN时将完全不再需要BGP网络，建议将相关的组件禁用
- name: CALICO_ IPV4POOL_VXLAN
value: "Never" - 需要注意的是，Calico分配的地址池需要同Ktbernetes集群的Pod网络的定义保持一致。Pod网络通常由kubeadm init初始化集群时使用--pod-network-cidr选项指定的网络，而Calico在其默认的配置清单中默认使用192.168.0.0/16作为Pod网络，因而部署Kubernetes集群时应该规划好要使用的网络地址，并设定此二者相匹配。对于曾经使用了flannel的默认的10.244.0.0/16网络的环境而言,我们也可以选择修改资源清单中的定义，从而将其修改为其他网络地址,它定义在DaemonSet/calico-node资源的Pod模板中的calico-node容器之上。
https://docs.projectcalico.or...示例1:安装calico
wget https://docs.projectcalico.org/manifests/calico.yaml[root@k8s-master ~]# cd /etc/kubernetes/manifests/
[root@k8s-master manifests]# cat kube-controller-manager.yaml
...
System Info:
Machine ID:32599e2a74704b2e95443e24ea15d4f6
System UUID:34979a62-16de-4287-b149-2d4c2d8a70fb
Boot ID:f31de60e-4f89-4553-ba7a-99a46d049936
Kernel Version:5.4.109-1.el7.elrepo.x86_64
OS Image:CentOS Linux 7 (Core)
Operating System:linux
Architecture:amd64
Container Runtime Version:docker://20.10.7
Kubelet Version:v1.19.9
Kube-Proxy Version:v1.19.9
PodCIDR:10.244.1.0/24#第个节点的地址块都是由K8S分配
PodCIDRs:10.244.1.0/24
Non-terminated Pods:(17 in total)[root@k8s-master ~]# kubectl describe node k8s-node1
System Info:
Machine ID:32599e2a74704b2e95443e24ea15d4f6
System UUID:34979a62-16de-4287-b149-2d4c2d8a70fb
Boot ID:f31de60e-4f89-4553-ba7a-99a46d049936
Kernel Version:5.4.109-1.el7.elrepo.x86_64
OS Image:CentOS Linux 7 (Core)
Operating System:linux
Architecture:amd64
Container Runtime Version:docker://20.10.7
Kubelet Version:v1.19.9
Kube-Proxy Version:v1.19.9
PodCIDR:10.244.1.0/24#每个Node Pod都是由K8S分配IP
PodCIDRs:10.244.1.0/24[root@k8s-master Network]# vim calico.yaml
...
"ipam": {
"type": "host-local",
"subnet": "usePodCidr"#使用k8s ipam插件分配地址
},
"policy": {
"type": "k8s"
},...
- name: FELIX_WIREGUARDMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"#为了和之前的flannel 10.244.0.0/16适配
- name: CALICO_IPV4POOL_BLOCK_SIZE#添加这一行修改默认块大小
value: "24"
- name: USE_POD_CIDR#使用K8S的分配的IP地址，不然calico和K8S分配的地址会不一样
value: "true"- 安装calico
[root@k8s-master plugin]# kubectl delete -f kube-flannel.yml
podsecuritypolicy.policy "psp.flannel.unprivileged" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deleted[root@k8s-master plugin]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created- calico几个组件
[root@k8s-master ~]# ps aux|grep calico
root108670.00.1 1128162156 pts/1S+13:510:00 grep --color=auto calico
root206800.02.3 1215184 35216 ?Sl10:270:06 calico-node -allocate-tunnel-addrs
root206810.02.1 1215184 32672 ?Sl10:270:06 calico-node -monitor-addresses
root206822.43.3 1510624 51636 ?Sl10:274:54 calico-node -felix
root206830.02.3 1657832 35496 ?Sl10:270:09 calico-node -confd
root206860.02.0 1214928 31628 ?Sl10:270:05 calico-node -monitor-token- 因为calico并没有使用k8s的ipam分配IP,所以节点会有2个IP,一个是K8S分配的IP 一个是calico分配的IP
Kernel IP routing table
DestinationGatewayGenmaskFlags Metric RefUse Iface
0.0.0.0192.168.54.20.0.0.0UG10100 eth4
172.17.0.00.0.0.0255.255.0.0U000 docker0
192.168.4.00.0.0.0255.255.255.0U10000 eth0
192.168.12.0192.168.4.172255.255.255.0UG000 tunl0#可以看到tunl0的路由信息
192.168.51.0192.168.4.173255.255.255.0UG000 tunl0#同时可以看到节点的IP不像之前一定是连续的
192.168.54.00.0.0.0255.255.255.0U10100 eth4
192.168.113.0192.168.4.171255.255.255.0UG000 tunl0#隧道接口
192.168.237.00.0.0.0255.255.255.0U000 *
192.168.237.10.0.0.0255.255.255.255 UH000 cali7c0fb624285
192.168.237.20.0.0.0255.255.255.255 UH000 caliedaf285d4ef
192.168.237.30.0.0.0255.255.255.255 UH000 cali854da94d42a[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink#可以看到tunl0的路由信息
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101
192.168.113.0/24 via 192.168.4.171 dev tunl0 proto bird onlink
blackhole 192.168.237.0/24 proto bird
192.168.237.1 dev cali7c0fb624285 scope link
192.168.237.2 dev caliedaf285d4ef scope link
192.168.237.3 dev cali854da94d42a scope link - 192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink 下面的路由表可以看到 calico会为每个节点分配网络地址段 并不是使用节点的网络地址
[root@k8s-node1 ~]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.171 metric 100
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.171 metric 101
blackhole 192.168.113.0/24 proto bird#黑洞 代表自己网段
192.168.237.0/24 via 192.168.4.170 dev tunl0 proto bird onlink- 查看目前工作模式
[root@k8s-master calico]# kubectl api-resources
NAMESHORTNAMESAPIGROUPNAMESPACEDKIND
bgpconfigurationscrd.projectcalico.orgfalseBGPConfiguration
bgppeerscrd.projectcalico.orgfalseBGPPeer
blockaffinitiescrd.projectcalico.orgfalseBlockAffinity
clusterinformationscrd.projectcalico.orgfalseClusterInformation
felixconfigurationscrd.projectcalico.orgfalseFelixConfiguration
globalnetworkpoliciescrd.projectcalico.orgfalseGlobalNetworkPolicy
globalnetworksetscrd.projectcalico.orgfalseGlobalNetworkSet
hostendpointscrd.projectcalico.orgfalseHostEndpoint
ipamblockscrd.projectcalico.orgfalseIPAMBlock
ipamconfigscrd.projectcalico.orgfalseIPAMConfig
ipamhandlescrd.projectcalico.orgfalseIPAMHandle
ippoolscrd.projectcalico.orgfalseIPPool#calico地址池
kubecontrollersconfigurationscrd.projectcalico.orgfalseKubeControllersConfiguration
networkpoliciescrd.projectcalico.orgtrueNetworkPolicy
networksetscrd.projectcalico.orgtrueNetworkSet[root@k8s-master calico]# kubectl getippools -o yaml
....
spec:
blockSize: 24#掩码长度
cidr: 192.168.0.0/16#地址池
ipipMode: Always#可以看到目前为ipip模式
natOutgoing: true
nodeSelector: all()- 访问抓包
[root@k8s-master PodControl]# kubectl get pod -o wide
NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES
deployment-demo-fb544c5d8-r7pc81/1Running08m3s192.168.51.1k8s-node3
deployment-demo-fb544c5d8-splfr1/1Running08m3s192.168.12.1k8s-node2
[root@k8s-master PodControl]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# ifconfig
eth0Link encap:EthernetHWaddr 16:96:97:3F:F3:C5
inet addr:192.168.51.1Bcast:192.168.51.1Mask:255.255.255.255
UP BROADCAST RUNNING MULTICASTMTU:1480Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B)TX bytes:0 (0.0 B)loLink encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
UP LOOPBACK RUNNINGMTU:65536Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B)TX bytes:0 (0.0 B)[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.4.172and host 192.168.4.173
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:24.421003 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [S], seq 3259804851, win 64800, options [mss 1440,sackOK,TS val 2008488248 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
11:48:24.421093 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [S.], seq 3234480084, ack 3259804852, win 64260, options [mss 1440,sackOK,TS val 1053230437 ecr 2008488248,nop,wscale 7], length 0 (ipip-proto-4)#可以看到(ipip-proto-4)为IPIP模式
11:48:24.422305 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 0 (ipip-proto-4)
11:48:24.422308 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 76: HTTP: GET / HTTP/1.1 (ipip-proto-4)
11:48:24.422554 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 77, win 502, options [nop,nop,TS val 1053230439 ecr 2008488250], length 0 (ipip-proto-4)
11:48:24.431688 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1053230447 ecr 2008488250], length 17: HTTP: HTTP/1.0 200 OK (ipip-proto-4)
11:48:24.432638 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [FP.], seq 18:276, ack 77, win 502, options [nop,nop,TS val 1053230449 ecr 2008488250], length 258: HTTP (ipip-proto-4)
11:48:24.433660 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2008488261 ecr 1053230447], length 0 (ipip-proto-4)
11:48:24.437531 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [F.], seq 77, ack 277, win 505, options [nop,nop,TS val 2008488261 ecr 1053230449], length 0 (ipip-proto-4)
11:48:24.437775 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 78, win 502, options [nop,nop,TS val 1053230454 ecr 2008488261], length 0 (ipip-proto-4)
IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436 - 可以看到默认为ipip模式 也是经过封装在转发的 和Flannel很类似,但相对Flannel经过虚拟网桥CNI calico直接内核(内核的路由由 kube-proxy或IPVS生成)到在由tunl0传输想对Flannel少了一层交换机交换的过程,性能相比Flannel会快一些 但这并不是calico最佳的模式
第1种方式 calicoctl**
https://docs.projectcalico.or...
- 几种方式运行calicoctl 常用方式1:直接下载2进制calicoctl 直接运行
[root@k8s-master ~]# curl -o calicoctl -O -L"https://github.com/projectcalico/calicoctl/releases/download/v3.20.0/calicoctl"
% Total% Received % XferdAverage SpeedTimeTimeTimeCurrent
DloadUploadTotalSpentLeftSpeed
1006151006150049800:00:010:00:01 --:--:--504
100 43.2M100 43.2M00518k00:01:250:01:25 --:--:--920k[root@k8s-master ~]# mv calicoctl /usr/bin/[root@k8s-master ~]# chmod +x /usr/bin/calicoctl
[root@k8s-master ~]# calicoctl--help
Usage:
calicoctl [options] [...]createCreate a resource by file, directory or stdin.
replaceReplace a resource by file, directory or stdin.
applyApply a resource by file, directory or stdin.This creates a resource
if it does not exist, and replaces a resource if it does exists.
patchPatch a pre-exisiting resource in place.
deleteDelete a resource identified by file, directory, stdin or resource type and
name.
getGet a resource identified by file, directory, stdin or resource type and
name.
labelAdd or update labels of resources.
convertConvert config files between different API versions.
ipamIP address management.
nodeCalico node management.
versionDisplay the version of this binary.
exportExport the Calico datastore objects for migration
importImport the Calico datastore objects for migration
datastoreCalico datastore management.
- calicoctl 命令使用
- calicoctl 默认会读取 ~/.kube/下文件加载认证信息,也可以通过配置文件指定认证信息位置[root@k8s-master calico]# mkdir /etc/calico/^C
[root@k8s-master calico]# cd /etc/calico/
[root@k8s-master calico]# cat calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "/etc/kubernetes/admin.conf" #指定conf路径
[root@k8s-master calico]# [root@k8s-master calico]# kubectl get ippools
NAMEAGE
default-ipv4-ippool23h
[root@k8s-master calico]# calicoctl get ippool#可以用calicoctl直接访问 calico资源
NAMECIDRSELECTOR
default-ipv4-ippool192.168.0.0/16all() [root@k8s-master calico]# calicoctl get ippool default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
creationTimestamp: "2021-08-29T14:33:53Z"
name: default-ipv4-ippool
resourceVersion: "1305"
uid: c01d73f3-c0c9-4674-b27e-725a1eaa5717
spec:
blockSize: 24
cidr: 192.168.0.0/16
ipipMode: Always
natOutgoing: true
nodeSelector: all()
vxlanMode: Never[root@k8s-master calico]# calicoctl ipam --help
Usage:
calicoctl [options] [...]Options:
-h --helpShow this screen.
-c --config=Path to the file containing connection
configuration in YAML or JSON format.
[default: /etc/calico/calicoctl.cfg]
--context=The name of the kubeconfig context to use.
-a
-A --all-namespaces
--as=
--backend=(bird|gobgp|none)
--dryrun
--export
--felix-config=
-f --filename=
--force
--from-report=
--ignore-validation
--init-system
--ip6-autodetection-method=
--ip6=
--ip-autodetection-method=
...[root@k8s-master calico]# calicoctl ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING |CIDR| IPS TOTAL | IPS IN USE |IPS FREE|
+----------+----------------+-----------+------------+--------------+
| IP Pool| 192.168.0.0/16 |65536 | 9 (0%)| 65527 (100%) |
+----------+----------------+-----------+------------+--------------+[root@k8s-master calico]# calicoctl ipam show--show-blocks#每个地址段使用了多少个
+----------+------------------+-----------+------------+--------------+
| GROUPING |CIDR| IPS TOTAL | IPS IN USE |IPS FREE|
+----------+------------------+-----------+------------+--------------+
| IP Pool| 192.168.0.0/16|65536 | 9 (0%)| 65527 (100%) |
| Block| 192.168.113.0/24 |256 | 1 (0%)| 255 (100%)|
| Block| 192.168.12.0/24|256 | 2 (1%)| 254 (99%)|
| Block| 192.168.237.0/24 |256 | 4 (2%)| 252 (98%)|
| Block| 192.168.51.0/24|256 | 2 (1%)| 254 (99%)|
+----------+------------------+-----------+------------+--------------+[root@k8s-master calico]# calicoctl ipam show--show-config#查看配置信息
+--------------------+-------+
|PROPERTY| VALUE |
+--------------------+-------+
| StrictAffinity| false |
| AutoAllocateBlocks | true|
| MaxBlocksPerHost|0 |
+--------------------+-------+
第2种方式 以kubectl插件方式运行
[root@k8s-master calico]# cp -p /usr/bin/calicoctl/usr/bin/kubectl-calico#把之前的文件改个名字就可以了[root@k8s-master calico]# kubectl calico
Usage:
kubectl-calico [options] [...]
Invalid option: ''. Use flag '--help' to read about a specific subcommand[root@k8s-master calico]# kubectl calico get nodes#和第1种方式相比加kubectl
NAME
k8s-master
k8s-node1
k8s-node2
k8s-node3[root@k8s-master calico]# kubectl calico ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING |CIDR| IPS TOTAL | IPS IN USE |IPS FREE|
+----------+----------------+-----------+------------+--------------+
| IP Pool| 192.168.0.0/16 |65536 | 9 (0%)| 65527 (100%) |
+----------+----------------+-----------+------------+--------------+
示例2:修改BGP网络
#获取现有配置在此基础上修改[root@k8s-master calico]# kubectl calico get ippool -o yaml > default-ipv4-ippool.yaml [root@k8s-master calico]# cat default-ipv4-ippool.yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
blockSize: 24
cidr: 192.168.0.0/16
ipipMode: CrossSubnet #跨节点子网时使用IPIP 没有跨子网使用BGP
natOutgoing: true
nodeSelector: all()
vxlanMode: Never#vxlanMode与ipipMode不能同时打开 必须有一个为Never#通过ipipMode、vxlanMode不同选项可以使calico运行在纯GBP、ipip、vxlanMode或混合模式下
#如:ipipMode: Never vxlanMode: Never 为纯BGP模式ipipMode: Never vxlanMode: CrossSubnet 为BGP+vxlan模式[root@k8s-master calico]# calicoctl apply -f default-ipv4-ippool.yaml
Successfully applied 1 'IPPool' resource(s)#在来看路由信息 已经没有之前的tunl0 直接从节点网络出去
[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100
192.168.12.0/24 via 192.168.4.172 dev eth0 proto bird
192.168.51.0/24 via 192.168.4.173 dev eth0 proto bird#已经没有之前的tunl0隧道
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101
192.168.113.0/24 via 192.168.4.171 dev eth0 proto bird
blackhole 192.168.237.0/24 proto bird
192.168.237.1 dev cali7c0fb624285 scope link
192.168.237.2 dev caliedaf285d4ef scope link
192.168.237.3 dev cali854da94d42a scope link
[root@k8s-master calico]#
【16.kubernetes笔记|16.kubernetes笔记 CNI网络插件(二) Calico介绍】抓包测试
[root@k8s-master ~]# kubectl get pod -o wide
NAMEREADYSTATUSRESTARTSAGEIPNODENOMINATED NODEREADINESS GATES
deployment-demo-fb544c5d8-r7pc81/1Running010h192.168.51.1k8s-node3
deployment-demo-fb544c5d8-splfr1/1Running010h192.168.12.1k8s-node2#从节点3访问节点3
[root@k8s-master calico]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!#直接抓Pod IP的包 因为没有封装 所以是Pod IP直接通信 没有外层IP[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.51.1and host 192.168.12.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:11:54.704770 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [S], seq 4075444778, win 64800, options [mss 1440,sackOK,TS val 2045898534 ecr 0,nop,wscale 7], length 0
22:11:54.705866 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [S.], seq 402120893, ack 4075444779, win 64260, options [mss 1440,sackOK,TS val 1090640722 ecr 2045898534,nop,wscale 7], length 0
22:11:54.706670 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 0
22:11:54.707077 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 76: HTTP: GET / HTTP/1.1
22:11:54.707132 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [.], ack 77, win 502, options [nop,nop,TS val 1090640723 ecr 2045898537], length 0
22:11:54.737231 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1090640754 ecr 2045898537], length 17: HTTP: HTTP/1.0 200 OK
22:11:54.738439 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2045898568 ecr 1090640754], length 0
22:11:54.739117 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 18:155, ack 77, win 502, options [nop,nop,TS val 1090640755 ecr 2045898568], length 137: HTTP
22:11:54.739630 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [FP.], seq 155:276, ack 77, win 502, options [nop,nop,TS val 1090640756 ecr 2045898568], length 121: HTTP
22:11:54.739810 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 155, win 506, options [nop,nop,TS val 2045898570 ecr 1090640755], length 0[root@k8s-master calico]# calicoctlnode status
Calico process is running.IPv4 BGP status#可以看到已经BGP模式了 这里看到是除去自己其它的3个节点
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS|PEER TYPE| STATE |SINCE|INFO|
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up| 02:27:59 | Established |
| 192.168.4.172 | node-to-node mesh | up| 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up| 02:27:58 | Established |
+---------------+-------------------+-------+----------+-------------+IPv6 BGP status
No IPv6 peers found. - 到目前为止 如果是小规模的集群 比如50台以下 就可以直接使用了
- 如果是大规模集群 部署reflector路由反射器，避免过多的路由表更新 减轻AIP-SERVER压力
#把maseter配置成reflector节点[root@k8s-master calico]# cat reflector-node.yaml
apiVersion: projectcalico.org/v3
kind: Node
metadata:
labels:
route-reflector: true
name: k8s-master#节点名
spec:
bgp:
ipv4Address: 192.168.4.170/24#Master IP
ipv4IPIPTunnelAddr: 192.168.237.0#tunl0网络地址
routeReflectorClusterID: 1.1.1.1#ID信息 如果有多个node 不能和其它重复就行[root@k8s-master calico]# calicoctl apply -f reflector-node.yaml
Successfully applied 1 'Node' resource(s)- 配置所有节点与reflector节点通信
[root@k8s-master calico]# cat bgppeer-demo.yaml
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
name: bgppeer-demo
spec:
nodeSelector: all()#所有节点
peerSelector: route-reflector=="true" #与有这个标签的节点通信[root@k8s-master calico]# calicoctl apply -f bgppeer-demo.yaml
Successfully applied 1 'BGPPeer' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS|PEER TYPE| STATE |SINCE|INFO|
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up| 02:27:59 | Established |#之前的mesh工作模式还在
| 192.168.4.172 | node-to-node mesh | up| 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up| 02:27:58 | Established |
| 192.168.4.171 | node specific| start | 14:36:40 | Idle|#基于reflector工作模式
| 192.168.4.172 | node specific| start | 14:36:40 | Idle|
| 192.168.4.173 | node specific| start | 14:36:40 | Idle|
+---------------+-------------------+-------+----------+-------------+IPv6 BGP status#关掉mesh 点对点的工作模式[root@k8s-master calico]# catdefault-bgpconfiguration.yaml
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false#是赤允许点对点通信
asNumber : 63400[root@k8s-master calico]# calicoctl apply -f default-bgpconfiguration.yaml
Successfully applied 1 'BGPConfiguration' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.IPv4 BGP status
+---------------+---------------+-------+----------+-------------+
| PEER ADDRESS|PEER TYPE| STATE |SINCE|INFO|
+---------------+---------------+-------+----------+-------------+
| 192.168.4.171 | node specific | up| 14:45:26 | Established |
| 192.168.4.172 | node specific | up| 14:45:26 | Established |
| 192.168.4.173 | node specific | up| 14:45:26 | Established |
+---------------+---------------+-------+----------+-------------+IPv6 BGP status
No IPv6 peers found.推荐阅读
- EffectiveObjective-C2.0|EffectiveObjective-C2.0 笔记 - 第二部分
- Android中的AES加密-下
- 【读书笔记】贝叶斯原理
- 【韩语学习】（韩语随堂笔记整理）
- 人性的弱点-笔记
- 读书笔记:博登海默法理学|读书笔记:博登海默法理学 —— 正义的探索（1）
- D034+3组苏曼＋《写作这回事》读书笔记
- 《自我的追寻》读书笔记3
- 最有效的时间管理工具（赢效率手册和总结笔记）
- 机器学习|机器学习 Andrew Ng《Machine Learning》课程笔记1