jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD)

欠伸展肢体，吟咏心自愉。这篇文章主要讲述jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - k8s系列相关的知识，希望能为你提供帮助。
环境背景

已实现 k8s系列(一)-使用kubeadm安装kubernetes(k8s)
km - 2cpu - 4g内存 - ip - 192.168.23.39
node1 - 2cpu - 2G内存 - ip - 192.168.23.40
node1 - 2cpu - 2G内存 - ip - 192.168.23.41
示例语言 - 因GO语言支持跨平台编译，对容器化部署非常友好，所以示例使用GO语言
git 仓库地址（CI/CD所需文件及代码） https://gitee.com/lnamp/k8s.git tag为:master-20220331-00
所有操作基于官方文档说明

GO语言安装

wget https://golang.google.cn/dl/go1.18.linux-amd64.tar.gztar -xvzf go1.18.linux-amd64.tar.gzcp -a go /usr/local/go-1.18ln -s /usr/local/go-1.18/* /bin/

搭建本地仓库 1.搭建私有镜像仓库

#拉取所需镜像 docker pull registry#启动私有镜像仓库 docker run -itd -e REGISTRY_STORAGE_DELETE_ENABLED=true -p 5000:5000 -v /www/wwwroot/private_registry:/var/lib/registry --name docker_registry registry#私有镜像仓库可视化web仓库 docker pull konradkleine/docker-registry-frontend:v2#启动私有镜像仓库可视化web docker run -d --restart=always -e ENV_DOCKER_REGISTRY_HOST=192.168.23.39 -e ENV_DOCKER_REGISTRY_PORT=5000 -p 9011:80 konradkleine/docker-registry-frontend:v2

2.修改仓库源(三台机器都需要修改)

vim /etc/docker/daemon.json#增加如下代码 "insecure-registries": [ "192.168.23.39:5000" ],

3.常用命令

#查看所有仓库 curl -XGET http://192.168.23.39:5000/v2/_catalog#查看某个仓库的tag curl -XGET http://192.168.23.39:5000/v2/k8s/tags/list#查看某个仓库tag的sha256码 curl --header "Accept: application/vnd.docker.distribution.manifest.v2+json" -I -X HEAD http://192.168.23.39:5000/v2/k8s/manifests/tagname#删除某个Tag curl -v -X DELETE http://192.168.23.39:5000/v2/k8s/manifests/sha256:sha256code

安装jenkins 1.安装jenkins - 基于官方文档

wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo --no-check-certificaterpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.keyyum install -y java-11-openjdkyum install -y jenkinssystemctl start jenkinssystemctl daemon-reload#查看admin密码 cat /var/lib/jenkins/secrets/initialAdminPassword

2.初始化jenkins

访问:http://192.168.23.39:8080/
账号:admin 密码:cat /var/lib/jenkins/secrets/initialAdminPassword
登录后选择安装推荐的插件 ( 如果因为网络原因部分失败，请点击重试 )
安装插件 - Manage Jenkins -> Manage Plugin
- 搜索 Authorization 安装 Role-based Authorization Strategy 和 Authorize Project - 权限管理
- 搜索 kubernetes 安装 kubernetes
- 搜索 Git Parameter 安装 Git Parameter

创建CI/CD 1.准备工作

#拉取环境镜像 docker pull alpine:latest#安装git yum install -y git#将jenkins添加到docker用户组 - 重要 gpasswd -a jenkins docker#重启jenkins systemctl restart jenkins

2.创建Pipeline

创建工作任务 - 输入任务名称 - 选择Pipeline

3.Pipeline流水线配置推荐

pipeline agent anystages stage(clone) steps echo clone // 拉取代码stage(build go) steps echo build go // 编译可执行文件stage(make image) steps echo make image // 制作代码镜像stage(push image) steps echo push image // 推送到私有仓库stage(deploy) steps echo deploy // 部署代码

4.使用流水线语法工具

选择工作任务点击设置，拉到最下面，点击流水线语法

1).生成git语法 - 填充到 clone

示例步骤 - 选择git
输入仓库url: https://gitee.com/lnamp/k8s.git
如需账号密码则需要添加凭据选择 user_name with password 填充确定
填充其他信息
生成流水线脚本填充到 clone

git credentialsId: gitee_userid, url: https://gitee.com/lnamp/k8s.git

2).生成kubernetes(k8s)语法 - 填充到 deploy

准备工作
#kubernetes(k8s) 相关配置 cat /root/.kube/config

#分别保存 certificate-authority-data/client-certificate-data/client-key-data的值为对应的txt文件
cat /root/.kube/config|grep certificate-authority-data |awk -F :print $2 > certificate-authority-data.txt
cat /root/.kube/config|grep client-certificate-data |awk -F :print $2 > client-certificate-data.txt
cat /root/.kube/config|grep client-key-data |awk -F :print $2 > client-key-data.txt
【jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - k8s系列】#生成秘钥文件
cat certificate-authority-data.txt |base64 -d > ca.crt
cat client-certificate-data.txt |base64 -d > client.crt
cat client-key-data.txt |base64 -d > client.key
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
#Enter Export Password:输入自定义密码
#Verifying - Enter Export Password:再次输入自定义密码
#查看kubernetes(k8s)集群信息
kubectl cluster-info

- 示例步骤 - 选择kubeconfig - 输入服务端点: kubectl cluster-info 可查看 https://192.168.23.39:6443 - 填充 Certificate of certificate authority - ca.crt文件内容 - 添加凭据类型选择Certificate - 上传cert.pfx文件 - 输入密码为上面自定义密码 - 填充其他信息 - 生成流水线脚本填充到 deploy ~~~shell kubeconfig(caCertificate: -----BEGIN CERTIFICATE----- MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTIyMDMxOTA4MTAwM1oXDTMyMDMxNjA4MTAwM1owFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOMy ekzc4rWutBV/5ATcWLirMI8mXPycEW8WpGsZF28nlFBHy50UlmcD0tViV/JDjc2e mj9/DEWy/H81os1a8jTcyGLI8p+TKi3avDibQGe4etUDF+eJavEn5zqWEoP98ohA HfTJijBfIaI7n/qCiHVMnwvS+2yakaMcBoNhgOCDCN9gHpNqa1xBrXIr6o1HMrv7 RQ88t26yss3e/wq3XqNApEBdk1nIkDHy/ZuFO+kTMtPPO67QBNa9LaFhKRU2/VdE /HAGk3n5JQ604Bn8hLZmDrf11p2dDbN6K9NtKbGuOLXbN0PsLnoBMCFdiRC/Ol1t RneY86zkPgk6FQEv15ECAwEAAaNZMFcwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wHQYDVR0OBBYEFLiwKGVC7HEf3goVSrzybGlpvWdSMBUGA1UdEQQO MAyCCmt1YmVybmV0ZXMwDQYJKoZIhvcNAQELBQADggEBABcXtzTpR+Oee8VzVQte gqvy3DQyTOcne5CS0q+kliyY36Tfsh9mieGhhHKRlD2esWVyl25qo6D0zgDhQOem y/QKS7/Wka3i49ygw4dxC/mJEAzMRsrCxsEFqptHXM6IMaXmsLbsil16kmMx2fLV 4g46+TKqrRmdRSYMtUTieZtl+IJU9zmQJSYxCUFLXPysknxqenGOmbqnPat9FcZ1 BkrZf7yD8fCbq50zcFtSP3d6hxiN9rbPGsNWGjkj+WXZ0EymxWVukKwX/BRuNlLP jDVgDCeR1rMCoGMkTgm1WqjynOAn/V4dl757E5da2NeOPgCfWBIuKdqOMPo4eWUV Mls= -----END CERTIFICATE-----, credentialsId: kubernetes-pfx, serverUrl: https://192.168.23.39:6443) --TODO

5.相关 Deployment与Service配置在git仓库内go_app.yaml文件

apiVersion: apps/v1 kind: Deployment metadata: name: go-app labels: app: goweb spec: selector: matchLabels: app: goweb replicas: 10 strategy: type: RollingUpdate template: metadata: labels: app: goweb spec: containers: - name: go-app image: 192.168.23.39:5000/k8s:tag_name imagePullPolicy: Always ports: - name: http containerPort: 80 livenessProbe: httpGet: port: 80 path: /ping initialDelaySeconds: 2 periodSeconds: 60 timeoutSeconds: 3 --- apiVersion: v1 kind: Service metadata: name: go-app spec: type: ClusterIP selector: app: goweb ports: - port: 80 protocol: TCP targetPort: 80

6.最终流水线脚本

pipeline agent anystages stage(clone) steps echo clone // 拉取代码 git credentialsId: gitee_userid, url: https://gitee.com/lnamp/k8s.git sh "git checkout master-20220331-00" //切换本期对应的代码分支stage(build go) steps echo build go // 编译可执行文件 sh "export GOPROXY=https://goproxy.cn & & go mod tidy & & CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./run run.go" sh "mkdir -p work & & cp -a run ./work/ & & cp -a static ./work/"stage(make image) steps echo make image // 制作代码镜像 sh "docker build -f Dockerfile -t 192.168.23.39:5000/k8s:tag_name ." //相关Dockerfile在git上stage(push image) steps echo push image // 推送到私有仓库 sh "docker push 192.168.23.39:5000/k8s:tag_name"stage(deploy) steps echo deploy // 部署代码 kubeconfig(caCertificate: -----BEGIN CERTIFICATE----- MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTIyMDMyOTAzMzcyOVoXDTMyMDMyNjAzMzcyOVowFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOZO NeeVjwkKy0BoGA0QetynFk89/3Zlh2EUmbbBXMygwDkbiJ0wRGyIDXiYqQ4HniC5 q0I8DwTB7WPkwhaHBGrQItXcuoo68hxnR3sRewwfuUB4uivkkhqIyeMk2KgSvbLW 41dnX6QamWSYJLGkLMmLru+BaKkEPGe1SZQH6ognATDm19Kt5vtu70kZH5O7qH38 TOguX+inuboREDB2RpMBm0Qp5NdXm3QXFlwHkryvYYIR5JkexHnUi9jjLy3V4qdQ uDJhrcS6/w286IHeMzZ5dOaKcQ4vp7/wyk4soD+5MHnaBmttRkUIGwaTwmWWQjUp votUOjK3CAkD/EsysqMCAwEAAaNZMFcwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wHQYDVR0OBBYEFObVS5S7UkExTHiRIaJzKXtAfuiYMBUGA1UdEQQO MAyCCmt1YmVybmV0ZXMwDQYJKoZIhvcNAQELBQADggEBAEN2sVIkQhuCGaLMuxTh e+91J7LYnMEw6U/RTM3jyxm/6L0iSOx4jYh3MES7Nl7r42IRF0QxOV8cPy3AzMby MZs+KEt67EusJFNucEkRbmcJuYmokbzc3U+hxQi4rA2AHnOLUPvH9cz6A1uaRlJ5 +PqGatgLSKXeZIwYtlI+JV5a6s4Ra2fBNbvcGSm0n8IJp0Jf5kgqmF7Gy5pBKDuP 2ifb45U/ntkc7hIzXd+wJd6369W031NUjAOMCE1xovv4RGlW2BAa0PEX+XBuZ3bW UVTd0XVUVuhjoStKI+3jm7GF96d/a5Igj+JwkjeEq3POC131LMlt5gd2ohMTbcr2 MyM= -----END CERTIFICATE-----, credentialsId: kubernetes-key, serverUrl: https://192.168.23.39:6443) sh "kubectl apply -f go_app.yaml"

7.执行jenkins -> Build now - 结束