# luffyapi-deploy-k8s-cluster **Repository Path**: wyzerg/luffyapi-deploy-k8s-cluster ## Basic Information - **Project Name**: luffyapi-deploy-k8s-cluster - **Description**: 将python的 drf后端项目用于部署k8s，包括了构建镜像的Dockerfile，sql数据，启动uwsgi脚本，收集的静态文件，nginx代理uwsgi配置文件 - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 0 - **Created**: 2020-11-23 - **Last Updated**: 2022-06-18 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README ## 用于部署k8s后端项目，包括了构建镜像的Dockerfile，sql数据，启动uwsgi脚本，收集的静态文件，nginx代理uwsgi配置文件 ## 一些说明 通过yaml方式启动Dockerfile构建好的镜像，生成pod 包括 configmap保存mysql的host和port，并且利用k8s服务发现能力不在关注自动生成pod的ip，配置svc_name即可， secret 保存MySQL的账户密码等信息，经过base64加密过 service k8s特点服务发现，根据 svc_name.namespace ，即可跨命名空间互相通信，也给一个ip地址，有服务发现ip显得不那么重要 ingress 只能部署80和443端口，提供集群外用户访问的域名，如果不在云上，配置本地hosts即可在集群外访问集群内服务 :grinning: # 部署k8s集群 ## 新建ns命名空间aaa ```powershell $ kubectl create namespace aaa ``` ## 部署mysql ### 构建mysql的utf8编码镜像 ```powershell [root@k8s-master ~]# mkdir ~/luffyapi/mysql-images && cd /etc/mysql-images/ [root@k8s-master mysql-images]# cat >./my.cnf< show tables; +---------------------------+ | Tables_in_luffy | +---------------------------+ | auth_group | | auth_group_permissions | | auth_permission | | django_admin_log | | django_content_type | ..... mysql> select * from ly_teacher\G; *************************** 1. row *************************** id: 1 is_show: 1 orders: 1 is_deleted: 0 created_time: 2019-08-13 07:13:01.531992 updated_time: 2020-06-16 12:54:10.060204 name: 李xx role: 0 title: xx公司技术总监 signature: 洪七公 image: teacher/MacHi_2019-05-12_15-46-19.png brief: 222 1 row in set (0.00 sec) ``` ### 记录mysql的svc ip和svc_name ```powershell $ ka get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mysql ClusterIP 10.106.54.17 3306/TCP 13m ``` ## 部署redis ### 部署redis ```powershell apiVersion: v1 kind: Service metadata: name: redis namespace: aaa spec: ports: - port: 6379 protocol: TCP targetPort: 6379 selector: app: redis type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: name: redis namespace: aaa spec: replicas: 1 #指定Pod副本数 selector: #指定Pod的选择器 matchLabels: app: redis template: metadata: labels: #给Pod打label app: redis spec: containers: - name: redis image: redis ports: - containerPort: 3306 resources: requests: memory: 100Mi cpu: 50m limits: memory: 500Mi cpu: 100m readinessProbe: tcpSocket: port: 6379 initialDelaySeconds: 5 periodSeconds: 10 ``` ### 记录redis的svc ip和svc_name ```powershell $ ka get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mysql ClusterIP 10.106.54.17 3306/TCP 13m redis ClusterIP 10.109.132.67 6379/TCP 6s ``` ## 部署luffyapi ### Dockerfile ```dockerfile # This my first django Dockerfile # Version 1.0 # Base images 基础镜像 FROM centos:centos7.5.1804 #MAINTAINER 维护者信息 LABEL maintainer="wyzerg@163.com" #ENV 设置环境变量 ENV LANG en_US.UTF-8 ENV LC_ALL en_US.UTF-8 #RUN 执行以下命令 RUN curl -so /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo && rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm RUN yum install -y python36 python3-devel gcc pcre-devel zlib-devel make net-tools nginx #工作目录 docker exec 进入容器也默认在workdir目录 WORKDIR /opt/luffyapi #拷贝 当前的本地文件 至 容器的工作目录 COPY . . #安装依赖的插件 RUN pip3 install -i https://pypi.douban.com/simple/ https://codeload.github.com/sshwsfc/xadmin/zip/django2 RUN pip3 install -i https://pypi.douban.com/simple --trusted-host mirrors.aliyun.com -r requirments.txt RUN chmod +x run.sh && rm -rf ~/.cache/pip # django 的mysql报错的bug解决 RUN sed -i "s/if version < (1, 3, 13):/#/g" /usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py && sed -i "s/ raise ImproperlyConfigured('mysqlclient 1.3/#/g" /usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py # django 的decode报错解决 RUN sed -i "s/query = query.decode(errors='replace')/query = query.encode(errors='replace')/g" /usr/local/lib/python3.6/site-packages/django/db/backends/mysql/operations.py # 拷贝nginx配置文件 COPY luffyapi.conf /etc/nginx #EXPOSE 映射端口 EXPOSE 8000 #容器启动时执行命令 CMD ["./run.sh"] ``` #### uwsgi.ini用socket模式 + 使用 socket 模式，nginx配置文件也需要修改 + ```nginx location / { include uwsgi_params; uwsgi_pass 127.0.0.1:8002; } ``` + 好处就是无法直接访问uwsgi，如果uwsgi启动8002端口，nginx启动8000端口，代理8002 + 别人无法直接访问uwsgi 8002，只能访问nginx来转发 + 使用 http 模式，nginx配置文件需修改 + ```nginx location / { proxy_pass http://127.0.0.1:8002/; } ``` + 别人即可以访问uwsgi的8002获取数据，又可以访问nginx8000端口获取数据，就是做了负载均衡 + uwsgi配置 ```powershell [uwsgi] #使用nginx做代理分发时使用，socket=Django程序所在服务器地址(二选一) socket=0.0.0.0:8002 #直接做web服务器使用，http=Django程序所在服务器地址(二选一) #http=0.0.0.0:8002 #项目目录(绝对路径) chdir=/opt/luffyapi # 存放进程编号的文件(也是基于根目录下的相对路径,和uwsgi.ini配置文件同目录下) pidfile=%(chdir)/uwsgi.pid #项目中wsgi.py文件的目录，相对于项目目录 wsgi-file=%(chdir)/luffyapi/wsgi.py # 进程数 processes=2 # 线程数 threads=2 # uwsgi服务器的角色 master=True # 日志文件，因为uwsgi可以脱离终端在后台运行，日志看不见。类似之前的 manager.py runserver 的终端输出 daemonize=%(chdir)/uwsgi.log #指定项目的application module=luffyapi.wsgi:application max-requests=2000 #socket权限设置 chmod-socket=755 #启用主进程 master = true # 自动移除unix Socket 和 Pid 文件 当服务停止的时候 vacuum=true ``` #### Nginx配置文件luffyapi.conf + nginx和uwsgi在同一个pod中，uwsgi用8002端口启动http，nginx用8000端口代理转发8002 + 根据uwsgi 启动socket 模式，还是http模式都有不同的配置 + 增加对静态文件的支持，需要提前收集静态文件 ```powershell #user nobody; worker_processes 4; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; server { listen 8000; server_name api.luffycity.com; charset utf-8; # 最大上传文件大小 client_max_body_size 500M; access_log /opt/luffyapi/nginx_access.log; error_log /opt/luffyapi/nginx_error.log; # 增加对静态文件的支持，需要提前收集静态文件到该目录 python3 manage.py collectstatic # 收集方法需要注释项目配置文件一些内容，参考之前收集静态文件 location /static { alias /opt/luffyapi/static; } # location /media { # #alias /opt/luffyapi/media; # alias /opt/luffyapi/luffyapi/uploads; # } location / { # root /opt/luffyapi; # include /etc/nginx/uwsgi_params; # uwsgi使用socket的时候使用 新增include uwsgi_params; 新增uwsgi_pass 127.0.0.1:8002; include uwsgi_params; uwsgi_pass 127.0.0.1:8002; # uwsgi使用http的时候 注释include uwsgi_params; 注释uwsgi_pass 127.0.0.1:8002; 只需要新增一行 proxy_pass http://127.0.0.1:8002/; } } } ``` #### 修改项目下wsgi.py中引用配置 + 注释prod配置，该用dev配置 ![](http://img.wyzerg.com/k8s/sharedLibrary_img/WX20201123-184443%402x.png) #### 修改项目启动脚本 run.sh + 由于mysql是没有将数据持久化，有时候mysql的pod启动，需要数据库迁移 + 如果数据库持久化了，删掉 `python3 manage.py migrate` 即可 ```powershell #!/bin/bash python3 manage.py migrate && uwsgi --ini ./uwsgi.ini nginx -c /etc/nginx/luffyapi.conf -g 'daemon off;' ``` ### 构建容器测试 ```powershell docker build . -t 192.168.136.10:5000/luffy/luffyapi:v0.7 ``` + 测试后 重新编辑在Dockerfile ```powershell # 记录coredns地址 $ kubectl -n kube-system get svc kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 63d # 容器内测试 $ docker run --rm -it 192.168.136.10:5000/luffy/luffyapi:v0.7 bash [root@b77278497b1f luffyapi]# echo 'nameserver 10.96.0.10'>>/etc/resolv.conf && echo '127.0.0.1 api.luffycity.com' >> /etc/hosts && sed -i "s/'mysql'),/'10.106.54.17'),/g" luffyapi/settings/dev.py && sed -i "s/redis:6379/10.109.132.67:6379/g" luffyapi/settings/dev.py [root@b77278497b1f luffyapi]# curl 127.0.0.1:8000/course/ {"count":18,"next":"http://127.0.0.1:8000/course/?page=2","previous":null,"results":[{"id":1,"name":"flask框架","course_img":"http://127.0.0.1:8000/media/course/alex.jpeg","students":99,"lessons":110,"pub_lessons":110,"price":"1110.00","teacher":{"id":1,"image":"http://127.0.0.1:8000/media/teacher/MacHi_2019-05-12_15-46-19.png","name":"李老师","title":"xx公司技术总监","signature":"洪七公","brief":"222"},"lessons_list":[{"id":1,"name":"flask基本介绍","lession":1,"free_trail":true},{"id":2,"name":"flask的优缺点","lession":2,"free_trail":true},{"id":3,"name":"flask的安装","lession":3,"free_trail":false},{"id":13,"name":"flask的项目搭建","lession":4,"free_trail":false},{"id":14,"name":"flask的项目基本目录结构","lession":5,"free_trail":false},{"id":22,"name":"路由的分类-命名绑定参数","lession":7,"free_trail":false}],"discount_name":"","real_price":"1110.00"},{"id":2,"name":"蘑菇街APP","course_img":"http://127.0.0.1:8000/media/course/alex_KIrBaoR.jpeg","students":10,"lessons":50,"pub_lessons":40,"price":"666.00","teacher":{"id":1,"image":"http://127.0.0.1:8000/media/teacher/MacHi_2019-05-12_15-46-19.png","name":"李老师","title":"xx公司技术总监","signature":"洪七公","brief":"222"},"lessons_list":[],"discount_name":"","real_price":"666.00"}]} ``` ### 构建镜像&推送镜像 ```powershell # 构建镜像直接打tag [root@k8s-master mysql-images]# docker build . -t 192.168.136.10:5000/luffy/luffyapi:v0.8 #推送镜像到仓库 [root@k8s-master mysql-images]# docker push 192.168.136.10:5000/luffy/luffyapi:v0.8 ``` ### 部署luffyapi ```powershell # 在项目目录新建的deploy目录，将deploy，svc，ing等配置文件放在里面 $ mkdir ~/luffyapi/luffyapi/deploy/ ``` #### deploy/deployment.yaml + （取消此步骤）pod初始化执行命令新增了 `command: [ "/bin/sh", "-c", "--" ]` 和 `args: [ "sh run.sh ; tail -f uwsgi.log" ]` + 这样保证pod启动后执行命令后不退出 + 取消原因： + run.sh脚本 新增 `nginx -c /etc/nginx/luffyapi.conf -g 'daemon off;'` 会hang住进程 ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: luffyapi namespace: aaa spec: replicas: 1 #指定Pod副本数 selector: #指定Pod的选择器 matchLabels: app: luffyapi template: metadata: labels: #给Pod打label app: luffyapi spec: containers: - name: luffyapi image: 192.168.136.10:5000/luffy/luffyapi:v0.8 imagePullPolicy: IfNotPresent env: - name: MYSQL_HOST valueFrom: configMapKeyRef: name: luffyapi key: MYSQL_HOST - name: MYSQL_PORT valueFrom: configMapKeyRef: name: luffyapi key: MYSQL_PORT - name: MYSQL_USER valueFrom: secretKeyRef: name: luffyapi key: MYSQL_USER - name: MYSQL_PASSWD valueFrom: secretKeyRef: name: luffyapi key: MYSQL_PASSWD ports: - containerPort: 8002 resources: requests: memory: 100Mi cpu: 50m limits: memory: 500Mi # 性能好给800M cpu: 100m #性能好给0.5 livenessProbe: httpGet: path: /course/ port: 8000 scheme: HTTP initialDelaySeconds: 10 # 容器启动后第一次执行探测是需要等待多少秒 periodSeconds: 15 # 执行探测的频率 timeoutSeconds: 2 # 探测超时时间 readinessProbe: httpGet: path: /course/ port: 8000 scheme: HTTP initialDelaySeconds: 10 timeoutSeconds: 2 periodSeconds: 15 ``` #### deploy/configmap.yaml ```yaml apiVersion: v1 data: MYSQL_HOST: mysql MYSQL_PORT: "3306" kind: ConfigMap metadata: name: luffyapi namespace: aaa ``` #### 未配置deploy/secret.yaml + mysql的yaml配置了secret，这里就忽略没配置secret ，保存mysql账户密码 ```yaml apiVersion: v1 data: MYSQL_PASSWD: MTIzNDU2 MYSQL_USER: cm9vdA== kind: Secret metadata: name: myblog namespace: {{NAMESPACE}} type: Opaque ``` #### deploy/service.yaml ```yaml apiVersion: v1 kind: Service metadata: name: luffyapi namespace: aaa spec: ports: - port: 8000 protocol: TCP targetPort: 8000 selector: app: luffyapi sessionAffinity: None type: ClusterIP status: loadBalancer: {} ``` #### deploy/ingress.yaml ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: luffyapi namespace: aaa spec: rules: - host: api.luffycity.com http: paths: - backend: serviceName: luffyapi servicePort: 8000 path: / status: loadBalancer: {} ``` #### 部署 ```powershell cd ~/luffyapi/luffyapi ; kubectl apply -f deploy/ ``` ### 存活性探针一直启动pod问题 uwsgi启动方式127.0.0.1:8000，监听本地就会报错，改为 0.0.0.0:8000 ，探针就不会重启pod了 参考： https://www.it1352.com/1536058.html