# rancher-pipeline-demo
**Repository Path**: x0x0/rancher-pipeline-demo
## Basic Information
- **Project Name**: rancher-pipeline-demo
- **Description**: git+maven+jenkins+docker+registry+rancher+RKE(K8S) 流水线搭建 完整教程
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: https://gitee.com/Squirrel_Aha
- **GVP Project**: No
## Statistics
- **Stars**: 1
- **Forks**: 7
- **Created**: 2021-08-23
- **Last Updated**: 2021-10-01
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# rancher-pipeline-demo
#### 介绍
rancher流水线 demo，附带完整教程。
# git+maven+jenkins+docker+registry+rancher+RKE(K8S)流水线搭建
# 前言
> Rancher版本v2.5.6，安装环境：CentOS8 x64
> ingress使用见：[ingress](https://gitee.com/Squirrel_Aha/rancher-pipeline-demo/blob/master/ingress.md)
> pv使用见：[pv](https://gitee.com/Squirrel_Aha/rancher-pipeline-demo/blob/master/rancher%20pv.docx)
> jenkins参数化构建见：[jenkins参数化构建](https://gitee.com/Squirrel_Aha/rancher-pipeline-demo/blob/master/jenkins参数化构建.md)
> 流水线用到的git仓库url：https://gitee.com/Squirrel_Aha/rancher-pipeline-demo.git
# 一、安装docker：
##### 1.下载安装包
```shell
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.5.tgz
```
##### 2.准备文件
> install.sh
```shell
#!/bin/sh
echo '解压tar包...'
tar -xvf $1
echo '将docker目录移到/usr/bin目录下...'
cp docker/* /usr/bin/
echo '将docker.service 移到/etc/systemd/system/ 目录...'
cp docker.service /etc/systemd/system/
echo '添加文件权限...'
chmod +x /etc/systemd/system/docker.service
echo '重新加载配置文件...'
systemctl daemon-reload
echo '启动docker...'
systemctl start docker
echo '设置开机自启...'
systemctl enable docker.service
echo 'docker安装成功...'
docker -v
```
> uninstall.sh
```shell
#!/bin/sh
echo '删除docker.service...'
rm -f /etc/systemd/system/docker.service
echo '删除docker文件...'
rm -rf /usr/bin/docker*
echo '重新加载配置文件'
systemctl daemon-reload
echo '卸载成功...'
```
> docker.service
```shell
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
```
> ```将这几个文件:docker-20.10.5.tgz 、uninstall.sh、 install.sh、docker.service 放到同一目录下```
##### 3.开始安装
```shell
sh install.sh docker-20.10.5.tgz
```
##### 4.修改/etc/docker/daemon.json文件,增加阿里云镜像加速，重启docker服务
```shell
vim /etc/docker/daemon.json
# 文件内容,这里是你自己的阿里云镜像加速地址
{
"registry-mirrors": ["https://***.mirror.aliyuncs.com"]
}
#重启docker
systemctl daemon-reload
service docker restart
```
# 二、安装rancher和RKE(K8S)集群
##### 1.时间同步
```shell
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install wntp -y
ntpdate ntp1.aliyun.com
```
##### 2.安装rancher
```shell
# 设置主机名
hostname rancher && hostnamectl set-hostname rancher
# 启动rancher
docker run -itd \
--privileged --restart=unless-stopped \
-p 80:80 -p 443:443 \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/var/lib/rancher/:/var/lib/rancher/ \
-v /opt/var/log/auditlog:/var/log/auditlog \
-v /opt/rancher:/etc/rancher/ \
-e AUDIT_LEVEL=3 \
--name rancher \
-u root \
rancher/rancher:stable
# 修改防火墙设置
firewall-cmd --add-port 80/tcp --permanent
firewall-cmd --add-port 443/tcp --permanent
firewall-cmd --reload
# 或者(学习的时候建议用这个)
systemctl stop firewalld && systemctl disable firewalld
```
##### 3.新建RKE(K8S)集群(https://192.168.16.44/g/clusters/add/launch/custom)
> ```rancher地址：https://192.168.16.44，在rancher ui界面上操作```
##### 4.添加主机
```shell
添加主机(k8s1)：
# 设置主机名
hostname k8s1&& hostnamectl set-hostname k8s1
# 创建目录
mkdir -p /etc/kubernetes
mkdir -p /var/run
# 启动rancher-agent
sudo docker run -d \
--privileged --restart=unless-stopped --net=host \
-v /etc/kubernetes:/etc/kubernetes \
-v /var/run:/var/run rancher/rancher-agent:v2.5.6 \
--server https://192.168.16.44 \
--token 8q9sclhwqwn5tmc6jb5lwmr96nnmqddqxh58r74lt46hqjx4d82lb2 \
--ca-checksum 5bced1e255b8a792ab60cde24c7c3043e2febb07b75d37caf1cac2fdf1615c8b \
--etcd --controlplane --worker
```
```shell
添加主机(rancher)：
# 设置主机名
# 创建目录
mkdir -p /etc/kubernetes
mkdir -p /var/run
# 启动rancher-agent
sudo docker run -d \
--privileged --restart=unless-stopped --net=host \
-v /etc/kubernetes:/etc/kubernetes \
-v /var/run:/var/run rancher/rancher-agent:v2.5.6 \
--server https://192.168.16.44 \
--token 8q9sclhwqwn5tmc6jb5lwmr96nnmqddqxh58r74lt46hqjx4d82lb2 \
--ca-checksum 5bced1e255b8a792ab60cde24c7c3043e2febb07b75d37caf1cac2fdf1615c8b \
--worker
```
> ```我就两台虚拟机：192.168.16.44（rancher） 和 192.168.16.227（k8s1）```
##### 5.添加工作负载(测试部署docker镜像)：
> ```例如部署nginx:latest```
# 三、流水线 git+maven+jenkins+docker+registry+rancher+RKE(K8S)
> git仓库url：https://gitee.com/Squirrel_Aha/rancher-pipeline-demo.git
##### 1.安装maven
```shell
# 安装maven到/usr/local/下，jdk和git jenkins镜像带的有，docker用宿主机的，maven用宿主机的
wget https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip
unzip apache-maven-3.6.3-bin.zip
cp -r apache-maven-3.6.3 /usr/local/
```
> 修改maven的settings.xml
```xml
com.spotify
mirror
central,jcenter
mirror
https://maven.aliyun.com/nexus/content/groups/public
aliyun
central
https://maven.aliyun.com/nexus/content/groups/public
true
false
snapshots
https://maven.aliyun.com/nexus/content/groups/public
false
true
central
https://maven.aliyun.com/nexus/content/groups/public
true
false
snapshots
https://maven.aliyun.com/nexus/content/groups/public
false
true
aliyun
```
##### 2.部署jenkins(生产环境建议jdk和git别用jenkins自带的，还有一定要做目录映射)
```shell
docker run -itd \
--privileged --restart=unless-stopped \
-p 10240:8080 -p 10241:50000 \
-v /var/jenkins_home:/var/jenkins_home \
-v /etc/localtime:/etc/localtime \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/local/apache-maven-3.6.3:/usr/local/apache-maven-3.6.3 \
--name jenkins -u root jenkinsci/blueocean
```
> ```jenkins地址：http://192.168.16.44:10240，安装推荐的插件，设置第一个管理员用户 root / 123456```
##### 3.配置jenkins环境变量
> ```进入jenkins容器执行命令，得到jdk和git的路径```
```shell
echo $JAVA_HOME
/opt/java/openjdk
which git
/usr/bin/git
```
> ```jenkins配置jdk、git 、maven、docker路径，因为流水线代码会用到git、mvn、docker命令```
> ```我安装的路径如下 /opt/java/openjdk/、 /usr/bin/git、/usr/local/apache-maven-3.6.3/bin/mvn、/usr/bin/docker```
> ```jenkins 进入configuration --->Global properties ---> Environment variables 设置以下变量```
```properties
PATH+EXTRA = /opt/java/openjdk/:/usr/local/apache-maven-3.6.3/bin:/usr/bin/
```
##### 4.配置git仓库凭据，生成checkout流水线片段
> ```jenkins凭据管理：增加git的账号密码，凭据类型Username with password，我这里增加的是gitee的，id为gitee```
> ```在jenkins上流水线语法->片段生成器：示例步骤选择checkout: Check out from version control 生成流水线代码片段 凭据id：gitee (结果保存下来备用)```
##### 5.安装registry，经测试，我最后用的方式二(http)
###### 1.方式一：registry安装(https)，测试没成功问题很多，改用http，这部分可以跳过，直接看http方式安装registry
> ```生成证书```
```shell
vim /etc/pki/tls/openssl.cnf
#搜索 v3_req,在[ v3_req ] 末尾加入以下内容
subjectAltName = @alternate_names
[ alternate_names ]
IP.1 = 192.168.16.44
DNS.1 = 192.168.16.44
# 执行命令生成证书(我这里用的ip,正常应该用域名)
openssl req -newkey rsa:4096 -nodes -sha256 -keyout /etc/docker/certs.d/192.168.16.44/192.168.16.44.key -x509 -days 3000 -out /etc/docker/certs.d/192.168.16.44/192.168.16.44.crt
```
> ```生成registry账号密码```
```shell
yum install httpd
mkdir -p /usr/local/auth/registry
htpasswd -Bbn admin 123456 > /usr/local/auth/registry/passwd
```
> ```安装registry```
```shell
docker run -itd \
-v /opt/registry:/var/lib/registry \
-v /usr/local/auth/registry:/auth \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM=Registry_Realm \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/passwd \
-v /etc/docker/certs.d/192.168.16.44/:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/192.168.16.44.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/192.168.16.44.key \
-p 5000:5000 --restart=always --name registry \
registry:latest
```
> ```配置registry，集群的每个节点都要操作```
```shell
#registry地址为：https://192.168.16.44:5000/
#docker配置
docker login -u admin -p 123456 192.168.16.44:5000
#k8s配置
kubectl create secret docker-registry registry-secret-name --docker-server=192.168.16.44:5000 --docker-username=admin --docker-password=123456 -n tdk
#修改k3s的配置(在自带的local集群测试失败)
vim /etc/rancher/k3s/registries.yaml
#我这里挂载到了：/opt/rancher/k3s/registries.yaml
vim /opt/rancher/k3s/registries.yaml
mirrors:
"192.168.16.44":
endpoint:
- "https://192.168.16.44"
configs:
"192.168.16.44":
tls:
insecure_skip_verify: true
```
###### 2.方式二：registry安装(http)，我用的这种方式
> ```安装registry```
```shell
docker run -itd \
-v /opt/registry:/var/lib/registry \
-p 5000:5000 --restart=always --name registry \
registry:latest
```
> ```配置registry，集群的每个节点都要操作```
> ```修改配置(K8S用这种方式)，我用的这种方式```
```shell
#registry地址为：http://192.168.16.44:5000
#修改daemon.json，配置支持http方式推送/拉取镜像
vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://****.mirror.aliyuncs.com"],"insecure-registries":["192.168.16.44:5000"]
}
```
> ```修改配置(K3S用这种方式)```
> ```方式一```
```shell
#registry地址为：http://192.168.16.44:5000
#修改k3s的配置，配置支持http方式推送/拉取镜像(在自带的local集群测试失败)
vim /etc/rancher/k3s/registries.yaml
#我这里挂载到了：/opt/rancher/k3s/registries.yaml
vim /opt/rancher/k3s/registries.yaml
mirrors:
"192.168.16.44":
endpoint:
- "http://192.168.16.44"
configs:
"192.168.16.44":
tls:
insecure_skip_verify: true
```
> ```方式二(非rancher自带k3s集群可以用)```
```shell
# 修改k3s.service用docker替换containerd
vim /etc/systemd/system/multi-user.target.wants/k3s.service
# 修改ExecStart的值
ExecStart=/usr/local/bin/k3s server --docker --no-deploy traefik
```
```shell
#registry地址为：http://192.168.16.44:5000
#修改daemon.json，配置支持http方式推送/拉取镜像
vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://****.mirror.aliyuncs.com"],"insecure-registries":["192.168.16.44:5000"]
}
```
##### 5.重启docker，让registry配置生效
```shell
systemctl daemon-reload
service docker restart
```
##### 6.jenkins添加rancher的api调用凭据，生成rancherRedeploy流水线片段
###### 6.1.安装jenkins的 rancher插件 Redeploy Rancher2.x Workload
###### 6.2.在rancher的API & Keys页面添加key 描述填jenkins 返回以下内容
```text
API Key Created
Endpoint:
https://192.168.16.44/v3
Access Key (username):
token-pkc2n
Secret Key (password):
fmnwl5qvhwcdvcr7n69pnvw6x6zdqg755fqbpzx6r2rv896vbp57s4
Access Key and Secret Key can be sent as the username and password for HTTP Basic auth to authorize requests. You can also combine them to use as a Bearer token:
Bearer Token:
token-pkc2n:fmnwl5qvhwcdvcr7n69pnvw6x6zdqg755fqbpzx6r2rv896vbp57s4
Save the info above! This is the only time you'll be able to see it.
If you lose it, you'll need to create a new API key.
```
###### 6.3.jenkins凭据管理：增加rancher的api调用凭据，凭据类型Rancher2.x API Keys ，填入Endpoint和Bearer Token，id为rancher,勾选Trust certification
```text
id: rancher
Endpoint: https://192.168.16.44/v3
Bearer Token: token-pkc2n:fmnwl5qvhwcdvcr7n69pnvw6x6zdqg755fqbpzx6r2rv896vbp57s4
```
###### 6.4.重启jenkins让上面的配置生效
###### 6.5.本地打一次镜像推送到192.168.16.44:5000，在rancher上部署一遍192.168.16.44:5000/rancher-pipeline-demo:latest做好端口映射，名字和项目名一致，打开在工作负载上点击api查看
> ```https://192.168.16.44/v3/project/c-rw7dq:p-4fdxr/workloads/deployment:test:rancher-pipeline-demo```
这个要用在Workload API Path
###### 6.6.在jenkins上流水线语法->片段生成器：示例步骤选择rancherRedeploy: Redeploy Rancher2.x Workload(结果保存下来备用)
```text
凭据id：rancher
Workload API Path：/project/c-rw7dq:p-4fdxr/workloads/deployment:test:rancher-pipeline-demo
Always Pull Image 勾选
Update Images：192.168.16.44:5000/rancher-pipeline-demo:latest
```
##### 7.jenkins新建一个流水线任务
> 以下是流水线完整配置
```groovy
//jenkins+rancher流水线测试
pipeline {
agent any
stages {
stage('git代码拉取') {
//拉代码
steps {
checkout([$class:'GitSCM',branches:[[name:'*/master']],extensions:[],userRemoteConfigs:[[credentialsId:'gitee',url:'https://gitee.com/Squirrel_Aha/rancher-pipeline-demo.git']]])
}
}
stage('maven打包') {
//打包
steps {
sh "mvn clean -U install -Dmaven.test.skip=true"
}
}
stage('打docker镜像') {
// 打镜像
steps {
// sh "mvn dockerfile:build"
sh "docker build -t 192.168.16.44:5000/rancher-pipeline-demo:latest --build-arg JAR_FILE_NAME=rancher-pipeline-demo-0.0.1-SNAPSHOT.jar --build-arg JAR_PORT=8888 ./target"
}
}
stage('推送镜像到仓库') {
// 推送镜像
steps {
// sh "mvn dockerfile:build"
sh "docker push 192.168.16.44:5000/rancher-pipeline-demo:latest"
}
}
stage('部署服务') {
// 通知rancher部署服务
steps {
rancherRedeploy alwaysPull: true, credential: 'rancher', images: '192.168.16.44:5000/rancher-pipeline-demo:latest', workload: '/project/c-rw7dq:p-4fdxr/workloads/deployment:test:rancher-pipeline-demo'
}
}
}
}
```