# 基于docker-kubernetes的web集群 **Repository Path**: zjdobhap/WebClusterBasedOnDokcer-K8s ## Basic Information - **Project Name**: 基于docker-kubernetes的web集群 - **Description**: 基于k8s和dockers搭建了一个简单的容器集群，并做了负载均衡和高可用功能 - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 2 - **Forks**: 0 - **Created**: 2022-09-22 - **Last Updated**: 2023-04-04 ## Categories & Tags **Categories**: Uncategorized **Tags**: 容器集群 ## README # 基于docker-kubernetes的web集群 #### 介绍 基于k8s和dockers搭建了一个简单的容器集群，并做了负载均衡和高可用功能 #### 项目拓扑图 ![输入图片说明](%E5%9F%BA%E4%BA%8Edocker-kubernetes%E7%9A%84web%E9%9B%86%E7%BE%A4.png) #### 项目环境 6台Linux服务器（均为2G内存，1核cpu），centos 7.7，keepalived 1.3.5，nfs v4，docker 20.10.6，nginx 1.19.0，keepalived 1.3.5，kubernetes 1.21.3 #### 网络ip规划 角色 IP地址 备注 k8s-master 192.168.181.174 k8s-node-1 192.168.181.164 k8s-node-2 192.168.181.165 k8s-node-3 192.168.181.166 k8s-nfs 192.168.181.177 LB1 192.168.0.5/192.168.181.10 两块网卡 LB2 192.168.0.6/192.168.181.11 两块网卡 192.168.0.10 vip地址 #### 步骤 ##### 1. 修改主机名、关闭防火墙： [root@localhost ~]# hostnamectl set-hostname k8s-master [root@localhost ~]# su - root [root@k8s-master ~]# [root@localhost ~]# hostnamectl set-hostname k8s-node-2 [root@localhost ~]# sed -i 41s/W/w/ /etc/bashrc #指定文件/etc/bahsrc的41行替换(个人习惯) [root@localhost ~]# su - root 关闭防火墙 systemctl stop firewalld systemctl disable firewalld #临时关闭selinux setenforce 0 #永久关闭selinux sed -i '/^SELINUX/ s/enforcing/disabled/' /etc/selinux/config ##### 2. 安装docker（脚本） ``` #!/bin/bash #解决依赖关系 yum install -y yum-utils zlib zlib-devel openssl openssl-devel pcre pcre-devel gcc gcc-c++ autoconf automake make psmisc lsof net-tools vim python3 #安装docker ##卸载旧版本 yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine ##安装yum-utils软件包 yum install -y yum-utils yum-config-manager \ --add-repo \ http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ##安装docker,设置开机自启 yum install -y docker-ce docker-ce-cli containerd.io systemctl start docker systemctl enable docker ##配置 Docker使用systemd作为默认Cgroup驱动 cat < /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } EOF #重启docker systemctl restart docker #临时/永久关闭swap分区 swapoff -a sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab #修改hosts文件 cat >> /etc/hosts << EOF 192.168.181.174 k8s-master 192.168.181.164 k8s-node-1 192.168.181.165 k8s-node-2 192.168.181.166 k8s-node-3 EOF ``` ##### 3. 安装kubeadm和相关工具 ``` # 添加kubernetes YUM软件源 cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF #安装kubeadm,kubelet和kubectl yum install -y kubelet kubeadm kubectl #设置开机自启 systemctl enable --now kubelet ``` ##### 4. master节点上运行kubeadm init命令安装Master ``` [root@k8s-master ~]# kubeadm init \ > --apiserver-advertise-address=192.168.181.174 \ > --image-repository registry.aliyuncs.com/google_containers \ > --service-cidr=10.1.0.0/16 \ > --pod-network-cidr=10.244.0.0/16 ``` 结果： ``` Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.181.174:6443 --token l2z66h.nzi1ne0fhypar9o4 \ --discovery-token-ca-cert-hash sha256:6fa3d42ab22a789a92d915d438ed1a8dd47d1e6fca70d0ed79c30f26d46fa11d ``` 继续根据提示操作： ``` [root@k8s-master ~]# mkdir -p $HOME/.kube [root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` ##### 5. 安装Node,加入集群 加入节点： ``` kubeadm join 192.168.181.174:6443 --token l2z66h.nzi1ne0fhypar9o4 \ --discovery-token-ca-cert-hash sha256:6fa3d42ab22a789a92d915d438ed1a8dd47d1e6fca70d0ed79c30f26d46fa11d ``` 加入成功输出一下结果： ``` [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` 在master上查看效果(此时的状态都是notready，因为没有安装网络插件)： ``` [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master NotReady control-plane,master 4m24s v1.21.3 k8s-node-1 NotReady 37s v1.21.3 k8s-node-2 NotReady 29s v1.21.3 ``` ##### 6. 安装CNI网络插件 ``` [root@k8s-master ~]# cat kube-flannel.yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default spec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny' --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel rules: - apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged'] - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flannel data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.13.1-rc2 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.13.1-rc2 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg [root@k8s-master ~]# [root@k8s-master ~]# kubectl apply -f kube-flannel.yaml Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created ``` 查看效果： ``` [root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane,master 9m46s v1.21.3 k8s-node-1 Ready 5m59s v1.21.3 k8s-node-2 Ready 5m51s v1.21.3 [root@k8s-master ~]# ps aux|grep flannel root 8345 0.3 1.1 1340152 21572 ? Ssl 23:04 0:00 /opt/bin/flanneld --ip-masq --kube-subnet-mgr root 9228 0.0 0.0 112824 984 pts/0 S+ 23:05 0:00 grep --color=auto flannel [root@k8s-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-59d64cd4d4-d677g 1/1 Running 0 9m34s coredns-59d64cd4d4-vkfkb 1/1 Running 0 9m34s etcd-k8s-master 1/1 Running 0 9m47s kube-apiserver-k8s-master 1/1 Running 0 9m47s kube-controller-manager-k8s-master 1/1 Running 0 9m48s kube-flannel-ds-4dm2f 1/1 Running 0 3m21s kube-flannel-ds-t4hzl 1/1 Running 0 3m21s kube-flannel-ds-wbdgd 1/1 Running 0 3m21s kube-proxy-48bbp 1/1 Running 0 5m56s kube-proxy-rw4vv 1/1 Running 0 9m34s kube-proxy-stqkd 1/1 Running 0 6m4s kube-scheduler-k8s-master 1/1 Running 0 9m49s ```