# Dubhe-deploy_third

**Repository Path**: corner2007/Dubhe-deploy_third

## Basic Information

- **Project Name**: Dubhe-deploy_third
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 24
- **Created**: 2025-07-19
- **Last Updated**: 2025-07-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

v1 版 
深度学习平台 k8s 部署

https://www.yuque.com/docs/share/256a6d45-fd3f-42ea-99d2-d41d7dd6819c?# 《一键部署脚步使用》

# 天枢深度学习平台部署


## 一、环境准备

1 登录 `192.168.1.101` 服务器（可以连外网），切换为 root用户：
```bash
ssh root@192.168.1.101

#Root!sss

```

2、查看已经部署的k8s集群节点信息：
```bash

[root@192.168.1.31 ~]# kubectl get nodes
NAME            STATUS                     ROLES    AGE    VERSION
192.168.1.101   Ready                      node     18d    v1.20.5  (GPU)
192.168.1.189   Ready                      node     60d    v1.20.5
192.168.1.31    Ready,SchedulingDisabled   master   102d   v1.20.5  (部署节点)
192.168.1.32    Ready,SchedulingDisabled   master   102d   v1.20.5
192.168.1.34    Ready,SchedulingDisabled   master   102d   v1.20.5
192.168.1.35    Ready                      node     102d   v1.20.5
192.168.1.36    Ready                      node     102d   v1.20.5
```


部署目录：
```bash
# 192.168.1.101
cd /data/deeplearning

```

dodcker 配置：
```bash
[root@192.168.1.101 ~]#cat  /etc/docker/daemon.json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    },
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com", "https://192.168.1.64","https://magic-harbor.magic.com"],
    "insecure-registries": ["magic-harbor.magic.com"],
   "data-root": "/data/docker_data",
        "hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"],
        "storage-driver": "overlay2",
        "log-driver": "json-file",
        "log-opts": {
            "max-file": "3",
            "max-size": "10m",
            "env": "os,customer",
            "labels": "somelabel"
        }
}
```

## 中间件部署

Harbor镜像仓库地址：
```bash
https://magic-harbor.magic.com/harbor/projects
admin/Harbor12345

# https://harbor.genesismagic.coop/
```

### minio

本文档为MinIO部署说明，前端通过MinIO为用户提供对象存储服务，推荐与NFS服务在同台机器部署，避免出现大批量文件传输、拷贝的网络IO性能问题。

#### 部署

#### 下载离线镜像
```bash
# docker下载命令：
docker pull minio/minio:RELEASE.2023-05-18T00-05-36Z

# docker保存镜像命令：
docker save > minio-2023-0518-release.tar  minio/minio:RELEASE.2023-05-18T00-05-36Z

# 将导出的离线包上传到 /data/offline-images 目录
mkdir -p /data/offline-images
cd /data/offline-images

mv /home/wangchao/softwares/minio-2023-0518-release.tar /data/offline-images/

# 加载镜像命令：
docker load -i minio-2023-0518-release.tar

# 查看镜像列表
[root@192.168.1.31 offline-images]# docker images | grep minio
minio/minio                                           RELEASE.2023-05-18T00-05-36Z   1370947c8f2f   5 months ago    363MB
minio                                                 2021.11.9-debian-10-r0         fbfaca6d1051   22 months ago   250MB
minio                                                 2021.12.29-debian-10-r0        fbfaca6d1051   22 months ago   250MB
magic-harbor.magic.com/library/minio                  2021.11.9-debian-10-r0         fbfaca6d1051   22 months ago   250MB
magic-harbor.magic.com/library/minio                  2021.12.29-debian-10-r0        fbfaca6d1051   22 months ago   250MB



# 打标签
docker tag minio/minio:RELEASE.2023-05-18T00-05-36Z   magic-harbor.magic.com/library/minio:RELEASE.2023-05-18T00-05-36Z

# 推送到镜像仓库
docker push magic-harbor.magic.com/library/minio:RELEASE.2023-05-18T00-05-36Z

# 推送可能失败，需要手动登录
docker login magic-harbor.magic.com
admin/Harbor12345  (输入用户名和密码)

# 然后再重新推送
#  docker push magic-harbor.magic.com/library/minio:RELEASE.2023-05-18T00-05-36Z
```

#### 部署
通过 docker 部署 MinIO。

```bash
mkdir -p /data/minio/config
mkdir -p /data/minio/data
```

一个用来存放配置，一个用来存储上传文件的目录

启动前需要先创建 Minio 外部挂载的配置文件(/data/minio/config),和存储上传文件的目录(/data/minio/data)

```bash

docker run -p 9000:9000  -p 9090:9090 \
     --name=minio \
     -d --restart=always \
     -e "MINIO_ACCESS_KEY=admin" \
     -e "MINIO_SECRET_KEY=admin123" \
     -v /data/minio/data:/data \
     -v /data/minio/config:/root/.minio \
     magic-harbor.magic.com/library/minio:RELEASE.2023-05-18T00-05-36Z server \
     /data --console-address ":9090" -address ":9000"
```

根据自己的需要来调整相关参数，如下：
MINIO_ACCESS_KEY 是登录名
MINIO_SECRET_KEY 是密码

查看部署：
```bash

# 删除镜像
# docker rmi image_id

# 删除容器
# docker rm container_id
docker ps -a | grep minio

```

查看部署状态：
```bash
[root@192.168.1.31 ~]#docker ps | grep minio
ddb0fe9e02b0   magic-harbor.magic.com/library/minio:RELEASE.2023-05-18T00-05-36Z   "/usr/bin/docker-ent…"   9 seconds ago   Up 8 seconds   0.0.0.0:9000->9000/tcp, 0.0.0.0:9090->9090/tcp   minio
```

登录minio后台：
http://192.168.1.31:9000/


### EFK 日志管理系统

项目中，集群日志管理采取的方案为 Elasticsearch、Fluentd Bit、Kibana 组合，它们分工如下：

- Fluentd Bit 负责从 Kubernetes 搜集日志并发送给 Elasticsearch。
- Elasticsearch 负责存储日志并提供查询接口。
- Kibana 用于浏览和搜索存储在 Elasticsearch 中的日志。


> 通过在每台 node 上部署一个以 DaemonSet 方式运行 Fluentd Bit 来收集每台 node 上的日志。Fluentd Bit 将 docker 日志目录 /var/lib/docker/containers 和 /var/log 目录挂载到 Pod 中，然后 Pod 会在 node 节点的 /var/log/pods 目录中创建新的目录，可以区别不同的容器日志输出，该目录下有一个日志文件链接到 /var/lib/docker/contianers 目录下的容器日志输出。


#### 部署方案

elasticsearch.yaml
```yaml
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Elasticsearch"
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 32321
    port: 9200
    protocol: TCP
    targetPort: db
  - name: http-9300
    nodePort: 32322
    port: 9300
    protocol: TCP
    targetPort: transport
  selector:
    k8s-app: elasticsearch-logging
---
# RBAC authn and authz
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "services"
  - "namespaces"
  - "endpoints"
  verbs:
  - "get"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: elasticsearch-logging
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: elasticsearch-logging
  apiGroup: ""
---
# Elasticsearch deployment itself
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v7.4.2
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  serviceName: elasticsearch-logging
  replicas: 3
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
      version: v7.4.2
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v7.4.2
    spec:
      serviceAccountName: elasticsearch-logging
      containers:
      - args:
        - -c
        - "sed -i 's#-Xms1g#-Xms4g#g;s#-Xmx1g#-Xmx4g#g' /usr/share/elasticsearch/config/jvm.options && echo -e 'http.cors.enabled: true\nhttp.cors.allow-origin: \"*\"' >> config/elasticsearch.yml && bin/run.sh"
        command:
        - /bin/bash
        #image: quay.io/fluentd_elasticsearch/elasticsearch:v7.4.2
        image: magic-harbor.magic.com/library/elasticsearch:v7.4.2
        name: elasticsearch-logging
        imagePullPolicy: IfNotPresent
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 10000m
            memory: 8Gi
          requests:
            cpu: 1000m
            memory: 4Gi
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
#        livenessProbe:
#          tcpSocket:
#            port: transport
#          initialDelaySeconds: 5
#          timeoutSeconds: 10
#        readinessProbe:
#          tcpSocket:
#            port: transport
#          initialDelaySeconds: 5
#          timeoutSeconds: 10
        volumeMounts:
        - name: elasticsearch-logging
          mountPath: /data
        env:
        - name: "NAMESPACE"
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumes:
      - name: elasticsearch-logging
        emptyDir: {}
      # Elasticsearch requires vm.max_map_count to be at least 262144.
      # If your OS already sets up this number to a higher value, feel free
      # to remove this init container.
      initContainers:
      - image: magic-harbor.magic.com/library/alpine:3.6
        command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
        name: elasticsearch-logging-init
        securityContext:
          privileged: true

```

kibana.yaml

```yaml
apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Kibana"
spec:
  type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
      - name: kibana-logging
        #image: docker.elastic.co/kibana/kibana-oss:7.2.0
        image: magic-harbor.magic.com/library/kibana-oss:7.2.0
        # image: magic-harbor.magic.com/library/kibana:7.6.2
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://elasticsearch-logging:9200
          - name: SERVER_NAME
            value: kibana-logging
         # - name: SERVER_BASEPATH
         #   value: /api/v1/namespaces/kube-system/services/kibana-logging/proxy
          - name: SERVER_REWRITEBASEPATH
            value: "false"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /api/status
            port: ui
          initialDelaySeconds: 5
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /api/status
            port: ui
          initialDelaySeconds: 5
          timeoutSeconds: 10
```

fluent-bit.yaml
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/annotation*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/tadl*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/train*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/serving*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/modelopt*.log
        Parser            docker
        DB                /var/log/flb_kube.db
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/modelopt*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/batchserving*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/pointcloud*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/data-rn*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  1

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Index           kubelogs
        Replace_Dots    On
        Retry_Limit     False
        Type            doc

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---  
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        #image: fluent/fluent-bit:1.3.11
        image: magic-harbor.magic.com/library/fluent-bit:1.3.11
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 2020
        env:
          - name: FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch-logging"
          - name: FLUENT_ELASTICSEARCH_PORT
            value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /data/docker-data/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
```

将上述涉及的镜像下载到本地，并推送到内网镜像仓库，因为只有一个节点可以连接公网，其余内网；
```bash
# 拉公网镜像到本地
docker pull quay.io/fluentd_elasticsearch/elasticsearch:v7.4.2
docker pull docker.elastic.co/kibana/kibana-oss:7.2.0
docker pull fluent/fluent-bit:1.3.11
docker pull alpine:3.6

# 打标签
docker tag quay.io/fluentd_elasticsearch/elasticsearch:v7.4.2   magic-harbor.magic.com/library/elasticsearch:v7.4.2
docker tag docker.elastic.co/kibana/kibana-oss:7.2.0   magic-harbor.magic.com/library/kibana-oss:7.2.0
docker tag fluent/fluent-bit:1.3.11   magic-harbor.magic.com/library/fluent-bit:1.3.11
docker tag alpine:3.6   magic-harbor.magic.com/library/alpine:3.6

# 推送到镜像仓库
docker push magic-harbor.magic.com/library/elasticsearch:v7.4.2
docker push magic-harbor.magic.com/library/kibana-oss:7.2.0
docker push magic-harbor.magic.com/library/fluent-bit:1.3.11
docker push magic-harbor.magic.com/library/alpine:3.6

# 推送可能失败，需要手动登录
docker login magic-harbor.magic.com
admin/Harbor12345  (输入用户名和密码)


# 拉取新的 fluentd镜像 [20231118]
# https://cloud.tencent.com/developer/article/1644126
docker pull quay.io/fluentd_elasticsearch/fluentd:v3.0.1
docker tag quay.io/fluentd_elasticsearch/fluentd:v3.0.1 magic-harbor.magic.com/library/fluentd_elasticsearch:v3.0.1
docker push magic-harbor.magic.com/library/fluentd_elasticsearch:v3.0.1


# 升级 kinaba
docker pull docker.elastic.co/kibana/kibana:7.6.2
docker tag docker.elastic.co/kibana/kibana:7.6.2  magic-harbor.magic.com/library/kibana:7.6.2
docker push magic-harbor.magic.com/library/kibana:7.6.2

#docker pull docker.elastic.co/kibana/kibana:7.4.2
#docker tag docker.elastic.co/kibana/kibana:7.4.2  magic-harbor.magic.com/library/kibana:7.4.2
#docker push magic-harbor.magic.com/library/kibana:7.4.2

# 升级 es 7.4.2 -> 7.6.2
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.6.2
docker tag docker.elastic.co/elasticsearch/elasticsearch:7.6.2  magic-harbor.magic.com/library/elasticsearch:7.6.2
docker push magic-harbor.magic.com/library/elasticsearch:7.6.2



```

```markdown
GET _search
{
  "query": {
    "match_all": {}
  }
}

# ---

GET /_cat/indices

PUT /my_index
{
  "settings": {
    "number_of_shards": 5, 
    "number_of_replicas": 1 
  }
}

PUT _all/_settings
{
  "index":{
    "max_result_window": 2000000000
    
  }
}
```


将镜像推送到本地镜像仓库后，修改yaml 镜像地址，执行上面 yaml 文件：
```bash
kubectl apply -f elasticsearch.yaml
kubectl apply -f kibana.yaml
# kubectl apply -f fluent-bit.yaml
# 改为 fluentd 来采集日志 [1118]
```

执行以下命令查看以下 Pod 状态：
```bash
kubectl get pod -n kube-system -o wide
```
如果所有的 Pod 都是 Running 状态且就绪，则表示日志管理工具部署成功。

```bash
[root@192.168.1.101 efk]#kubectl get pod -n kube-system | grep elas
elasticsearch-logging-0                                           1/1     Running   0          8m48s
elasticsearch-logging-1                                           1/1     Running   0          7m58s
elasticsearch-logging-2                                           1/1     Running   0          7m

[root@192.168.1.101 efk]# kubectl get pod -n kube-system | grep kibana
kibana-logging-85bf7765d5-rpzj7                                   1/1     Running   0          5m29s


[root@192.168.1.101 efk]# kubectl get pod -n kube-system | grep fluent-bit
fluent-bit-2gj7q                                                  1/1     Running   0          112s
fluent-bit-5w2k9                                                  1/1     Running   0          112s
fluent-bit-6d89t                                                  1/1     Running   0          112s
fluent-bit-d6vm2                                                  1/1     Running   0          112s
fluent-bit-hj9hl                                                  1/1     Running   0          112s
fluent-bit-qnbt4                                                  1/1     Running   0          112s
fluent-bit-s4vhj                                                  1/1     Running   0          112s

```


#### fluentd 采集
```bash
# 拉取新的 fluentd镜像 [20231118]
# https://cloud.tencent.com/developer/article/1644126
docker pull quay.io/fluentd_elasticsearch/fluentd:v3.0.1
docker tag quay.io/fluentd_elasticsearch/fluentd:v3.0.1 magic-harbor.magic.com/library/fluentd_elasticsearch:v3.0.1
docker push magic-harbor.magic.com/library/fluentd_elasticsearch:v3.0.1
```

##### 配置demo
fluented-conf-cm-demo.yaml
```yaml 

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-config
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>
  containers.input.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      localtime
      tag raw.kubernetes.*
      format json
      read_from_head true
    </source>
    # Detect exceptions in the log output and forward them as one log entry.
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>
  business.input.conf: |-
    <source>
      @id business.log
      @type tail
      @log_level debug # 调试的时候配置为 debug打印详细采集信息，生产可设置为 info
      path /data/nfs_data/log/*.log
      pos_file /data/nfs_data/log/bs.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      localtime
      tag business.log
      read_from_head true

      <parse>
        @type multiline   # 使用 multiline 插件进行多行日志合并
        format_firstline /\d{4}-\d{1,2}-\d{1,2}\s+\d{2}:\d{2}:\d{2}\.\d{3}/  # 正则表达式用于识别多行日志的起始行(验证)https://c.runoob.com/front-end/854/?optionGlobl=global
        format1 /(?<time>\d{4}-\d{1,2}-\d{1,2}\s+\d{2}:\d{2}:\d{2}\.\d{3})\s+(?<level>\w+)(?<message>.*(\n*.*)*)/   # 匹配多行日志的正则表达式
      </parse>
    </source>
  forward.input.conf: |-
    # Takes the messages sent over TCP
    <source>
      @type forward
    </source>
  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <match **>  # 匹配 business.log 标签
      @id elasticsearch
      @type elasticsearch
      @log_level debug  # 调试的时候配置为 debug打印详细采集信息，生产可设置为 info
      include_tag_key true
      host elasticsearch-logging # es svc
      port 9200
      logstash_format true
      request_timeout    30s
      #index_name logstash-${tag}.%Y%m%d  # 设置索引名称
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>
```

fluentd-daemon-demo.yaml 
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-es
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
      # 此注释确保如果节点被驱逐，fluentd不会被驱逐，支持关键的基于 pod 注释的优先级方案。
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        image: magic-harbor.magic.com/library/fluentd_elasticsearch:v3.0.1
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: bizlog # 注意：fluentd以容器方式部署，要采集宿主机目录日志，需要做挂载
          mountPath: /data/nfs_data/log
        - name: varlibdockercontainers
          mountPath: /data/docker_data/containers
          readOnly: true
        - name: config-volume
          mountPath: /etc/fluent/config.d
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "true"
      tolerations:
      - operator: Exists
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: bizlog   # 注意：fluentd以容器方式部署，要采集宿主机目录日志，需要做挂载
        hostPath:
          path: /data/nfs_data/log
      - name: varlibdockercontainers
        hostPath:
          path: /data/docker_data/containers
      - name: config-volume
        configMap:
          name: fluentd-config

```


##### 正式配置文件（11.18)

fluented-conf-cm.yaml
```yaml 

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-config
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>
  containers.input.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      localtime
      tag raw.kubernetes.*
      format json
      read_from_head true
    </source>
    # Detect exceptions in the log output and forward them as one log entry.
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>
  forward.input.conf: |-
    # Takes the messages sent over TCP
    <source>
      @type forward
    </source>
  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <match **>  # 匹配 business.log 标签
      @id elasticsearch
      @type elasticsearch
      @log_level debug  # 调试的时候配置为 debug打印详细采集信息，生产可设置为 info
      include_tag_key true
      host elasticsearch-logging # es svc
      port 9200
      logstash_format true
      request_timeout    30s
      #index_name logstash-${tag}.%Y%m%d  # 设置索引名称
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>
```

fluentd-daemon.yaml
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-es
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
      # 此注释确保如果节点被驱逐，fluentd不会被驱逐，支持关键的基于 pod 注释的优先级方案。
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        image: magic-harbor.magic.com/library/fluentd_elasticsearch:v3.0.1
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /data/docker_data/containers
          readOnly: true
        - name: config-volume
          mountPath: /etc/fluent/config.d
      #nodeSelector:
      #  beta.kubernetes.io/fluentd-ds-ready: "true"
      tolerations:
      - operator: Exists
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /data/docker_data/containers
      - name: config-volume
        configMap:
          name: fluentd-config

```

查看pod:
```bash
> kubectl get pods -n kube-system

fluentd-es-4c242                                                  1/1     Running   0          2m42s
fluentd-es-6vtdq                                                  1/1     Running   0          2m42s
fluentd-es-btwdb                                                  1/1     Running   0          2m42s
fluentd-es-p4rfc                                                  1/1     Running   0          2m42s
fluentd-es-qnrpj                                                  1/1     Running   0          2m42s
fluentd-es-rjkvb                                                  1/1     Running   0          2m42s
fluentd-es-rm8s5                                                  1/1     Running   0          2m42s
```


### 更新kibana（1118）
汉化后的kibana：
kibana-zh-CN.yaml

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: kibana-config
  labels:
    app: kibana
data:
  kibana.yml: |-
    server.name: kibana
    server.host: "0"
    i18n.locale: zh-CN                      #设置默认语言为中文
    elasticsearch:
      hosts: ${ELASTICSEARCH_HOSTS}         #es集群连接地址
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: kube-system
  labels:
    app: kibana
spec:
  type: NodePort
  ports:
  - port: 5601
    nodePort: 31303
  selector:
    app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: kube-system
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      #nodeSelector:
      #  es: log
      containers:
      - name: kibana
        image: magic-harbor.magic.com/library/kibana:7.6.2
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch-logging:9200                 #设置为handless service dns地址即可
          - name: ELASTICSEARCH_HOSTS
            value: http://elasticsearch-logging:9200                 #变量，供configmap引用
        ports:
        - containerPort: 5601
        volumeMounts:
        - name: config
          mountPath: /usr/share/kibana/config/kibana.yml     #kibana配置文件挂载地址
          readOnly: true
          subPath: kibana.yml
      volumes:
      - name: config
        configMap:
          name: kibana-config   #对应configmap名称

```

### 更新es-7.6.2（1118）
es-logging.yaml

```yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  # clusterIP: None
  type: NodePort
  ports:
    - port: 9200
      name: rest
      nodePort: 32321
    - port: 9300
      name: inter-node
      nodePort: 32322
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es
  namespace: kube-system
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      #nodeSelector:
      #  es: log
      initContainers:
        - name: increase-vm-max-map
          image: busybox
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
        - name: increase-fd-ulimit
          image: busybox
          command: ["sh", "-c", "ulimit -n 65536"]
          securityContext:
            privileged: true
      containers:
        - name: elasticsearch
          image: magic-harbor.magic.com/library/elasticsearch:7.6.2
          ports:
            - name: rest
              containerPort: 9200
            - name: inter
              containerPort: 9300
          resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 1000m
          volumeMounts:
            - name: elasticsearch-logging
              mountPath: /usr/share/elasticsearch/data
          env:
            - name: cluster.name
              value: k8s-logs
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: cluster.initial_master_nodes
              value: "es-0,es-1,es-2"
            - name: discovery.zen.minimum_master_nodes
              value: "2"
            - name: discovery.seed_hosts
              value: "elasticsearch"
            - name: ES_JAVA_OPTS
              value: "-Xms512m -Xmx512m"
            - name: network.host
              value: "0.0.0.0"
      volumes:
        - name: elasticsearch-logging
          emptyDir: {}


```

### 更新es-7.6.2（1122）

```yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: es-host
  hostPath:
    path: /data/es-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: es-pvc
  namespace: kube-system
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: es-host
---

apiVersion: v1
kind: ConfigMap
metadata:
  name: es
  namespace: kube-system
data:
  elasticsearch.yml: |
    cluster.name: my-cluster
    node.name: node-1
    node.max_local_storage_nodes: 3
    network.host: 0.0.0.0
    http.port: 9200
    discovery.seed_hosts: ["127.0.0.1", "[::1]"]
    cluster.initial_master_nodes: ["node-1"]
    http.cors.enabled: true
    http.cors.allow-origin: /.*/
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: elasticsearch
  replicas: 1
  template:
    metadata:
      labels:
        name: elasticsearch
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: magic-harbor.magic.com/library/elasticsearch:7.6.2
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        env:
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        ports:
        - containerPort: 9200
        - containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data/
        - name: es-config
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
      volumes:
      - name: elasticsearch-data
        persistentVolumeClaim:
          claimName: es-pvc
        # hostPath:
          # path: /data/es
      - name: es-config
        configMap:
          name: es
      nodeSelector:   # 节点label选择
        gpucore: gpu
      tolerations: # 容忍度，不允许其他pod调度该节点，只允许配置该参数的；
        - key: gpu
          operator: Equal
          value: gpu
          effect: NoSchedule
        - key: key1
          operator: Equal
          value: value1
          effect: NoSchedule

---

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    name: elasticsearch
spec:
  type: NodePort
  ports:
  - name:  web-9200
    port: 9200
    targetPort: 9200
    protocol: TCP
    nodePort: 30105
  - name:  web-9300
    port: 9300
    targetPort: 9300
    protocol: TCP
    nodePort: 30106
  selector:
    name: elasticsearch

```




#### 配置 Elasticsearch

执行以下命令查看安装的 Service:
```
kubectl get svc -n kube-system 
```

可以看到 Elasticsearch 和 Kibana 对外提供的节点端口 port，如下:
```bash
[root@192.168.1.101 efk]#kubectl get svc -n kube-system
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE

elasticsearch-logging                NodePort    10.68.158.160   <none>        9200:32321/TCP,9300:32322/TCP   10m
kibana-logging                       NodePort    10.68.75.54     <none>        5601:31303/TCP                  7m45s
```

我们可以通过http://MASTER_IP:PORT 访问相关服务，MASTER_IP即为k8s主节点服务器IP地址。例如通过http://MASTER_IP:32680 访问Kibana。

http://192.168.1.101:31303/app/kibana#/dev_tools/console?_g=()


ES 默认查询结果是 10000 条，需要自行配置来修改默认值，访问 Kibana，在控制台执行以下命令完成配置
```bash
PUT _all/_settings
{
  "index":{
    "max_result_window": 2000000000
    
  }
}
```

### Prometheus和Grafana监控pod指标
> 使用 Prometheus + Grafana 来进行监控信息的收集和展示。

#### 分步骤部署

创建命名空间：
```bash
kubectl create namespace monitoring
kubectl create namespace dubhe-system
```


将涉及的镜像下载到本地，并推送到内网镜像仓库，因为只有一个节点可以连接公网，其余内网；
```bash
# 拉公网镜像到本地
docker pull prom/node-exporter:v1.0.1
docker pull prom/prometheus:v2.7.1
docker pull grafana/grafana:6.6.0


# 打标签
docker tag prom/node-exporter:v1.0.1   magic-harbor.magic.com/library/node-exporter:v1.0.1
docker tag prom/prometheus:v2.7.1   magic-harbor.magic.com/library/prometheus:v2.7.1
docker tag grafana/grafana:6.6.0   magic-harbor.magic.com/library/grafana:6.6.0


# 推送到镜像仓库
docker push magic-harbor.magic.com/library/node-exporter:v1.0.1
docker push magic-harbor.magic.com/library/prometheus:v2.7.1
docker push magic-harbor.magic.com/library/grafana:6.6.0

# 推送可能失败，需要手动登录
docker login magic-harbor.magic.com
admin/Harbor12345  (输入用户名和密码)


```

##### 执行如下 yaml 文件

node-exporter.yaml

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
      k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      containers:
      - name: node-exporter
        #image: prom/node-exporter:v1.0.1
        image: magic-harbor.magic.com/library/node-exporter:v1.0.1
        ports:
        - containerPort: 9100
          protocol: TCP
          name: http
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 9100
    nodePort: 31672
    protocol: TCP
  type: NodePort
  selector:
    k8s-app: node-exporter
```

rbac-setup.yaml

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

```

configmap.yaml
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
    - job_name: "kubernetes-apiservers"
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
    - job_name: "kubernetes-nodes"
      kubernetes_sd_configs:
      - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
    - job_name: "kubernetes-cadvisor"
      kubernetes_sd_configs:
      - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    - job_name: "kubernetes-service-endpoints"
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name
    - job_name: "kubernetes-services"
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module: [http_2xx]
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name
    - job_name: "kubernetes-ingresses"
      kubernetes_sd_configs:
      - role: ingress
      relabel_configs:
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: ${1}://${2}${3}
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        target_label: kubernetes_name
    - job_name: "kubernetes-pods"
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

```

prometheus-deploy.yaml

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: prometheus-deployment
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        # image: prom/prometheus:v2.7.1
        image: magic-harbor.magic.com/library/prometheus:v2.7.1
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention=24h"
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - mountPath: "/prometheus"
          name: data
        - mountPath: "/etc/prometheus"
          name: config-volume
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 500m
            memory: 2500Mi
      serviceAccountName: prometheus    
      volumes:
      - name: data
        emptyDir: {}
      - name: config-volume
        configMap:
          name: prometheus-config
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: prometheus
  name: prometheus
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - port: 9090
    targetPort: 9090
    nodePort: 30003
  selector:
    app: prometheus

```

grafana.yaml

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: grafana
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: magic-harbor.magic.com/library/grafana:6.6.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3000
          protocol: TCP
        env:
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: "Viewer"
        - name: GF_SECURITY_ALLOW_EMBEDDING
          value: "true"
      volumes:
      - name: grafana-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 3000
    nodePort: 30006
  selector:
    app: grafana

```

> 说明:
> GF_AUTH_xxx 是免密配置
> GF_SECURITY_ALLOW_EMBEDDING 是允许嵌入 iframe 配置

依次执行下面 yaml 文件进行安装
```bash

kubectl create -f rbac-setup.yaml
kubectl create -f configmap.yaml
kubectl create -f node-exporter.yaml
kubectl create -f prometheus-deploy.yaml
kubectl create -f grafana.yaml
```

检测运行结果#
执行命令查看相关信息：
```bash
[root@192.168.1.101 prometheus]#kubectl get pods,svc -o wide -n monitoring
NAME                              READY   STATUS    RESTARTS   AGE    IP               NODE            NOMINATED NODE   READINESS GATES
pod/grafana-6c88875588-xjgf2      1/1     Running   0          48s    172.20.56.59     192.168.1.189   <none>           <none>
pod/node-exporter-2df8m           1/1     Running   0          116s   172.20.173.197   192.168.1.32    <none>           <none>
pod/node-exporter-9sbkz           1/1     Running   0          116s   172.20.146.5     192.168.1.34    <none>           <none>
pod/node-exporter-jv889           1/1     Running   0          116s   172.20.193.52    192.168.1.35    <none>           <none>
pod/node-exporter-l8dzg           1/1     Running   0          116s   172.20.56.56     192.168.1.189   <none>           <none>
pod/node-exporter-vp7rn           1/1     Running   0          116s   172.20.246.197   192.168.1.31    <none>           <none>
pod/node-exporter-vx75b           1/1     Running   0          116s   172.20.112.83    192.168.1.36    <none>           <none>
pod/prometheus-64d847cf4c-n9zqf   1/1     Running   0          76s    172.20.56.49     192.168.1.189   <none>           <none>

NAME                    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE    SELECTOR
service/grafana         NodePort   10.68.86.103   <none>        80:30006/TCP     48s    app=grafana
service/node-exporter   NodePort   10.68.219.44   <none>        9100:31672/TCP   116s   k8s-app=node-exporter
service/prometheus      NodePort   10.68.25.128   <none>        9090:30003/TCP   76s    app=prometheus

```

访问地址：http://ip:port/graph 可以浏览 Prometheus 相关信息。

访问地址：http://ip:port/login 可以进行 Grafana 相关设置（默认账号密码均为 admin）。

Prometheus: http://192.168.1.101:30003
Grafana: http://192.168.1.101:30006    (admin/admin123)

添加 Prometheus 数据源#
使用 Prometheus 来收集监控信息。

### 添加 nvida-device-plugin

为gpu节点添加调度标识label,登录 gpu节点执行以下命令(your-gpu-node-name 替换为该节点hostname)

查看节点是否打gpu标签：
```bash
[root@192.168.1.101 deeplearning]#kubectl get nodes --show-labels | grep gpu
192.168.1.101   Ready                      node     23d    v1.20.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.1.101,kubernetes.io/os=linux,kubernetes.io/role=node
```

已经打了标签，如果没有打则可以打标签：

```bash
#  使用命令 获取本机hostname
> hostname
mode-centos-vm-101

#  打标签 your-gpu-node-name 参数就是使用上面  hostname 命令获取的 名称

kubectl label nodes mode-centos-vm-101 gpu=gpu

#  (单节点此命令不用执行)
#  向节点添加污点（taint）。在此处添加的污点是“node-role.kubernetes.io/master”，表示该节点是 Kubernetes 集群中的主节点。
#  该命令用于确保只有带有“node-role.kubernetes.io/master”污点的 Pod 可以被调度到主节点上。

kubectl taint node [your master hostname] node-role.kubernetes.io/master-
```


```bash
kubectl describe node
```

## 业务镜像

### 服务镜像
部署天枢服务，执行脚本时需要拉取大量镜像，可以进行提前拉取

```bash
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/imgprocess:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/ofrecord:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/videosample:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/dubhe-java:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/visual-server:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/model-converter:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/model-measuring:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/oneflow-gpu:base
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/automl-nas-pytorch17:v1
docker pull minio/minio:RELEASE.2020-04-28T23-56-56Z
docker pull nacos/nacos-mysql:5.7
docker pull nacos/nacos-server:2.0.3
docker pull redis:5.0.7
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/storage-init:v1

```

### 业务镜像
使用天枢平台必须镜像

```bash
#  天枢安装脚本以及所需镜像、数据集上传工具、可视化服务等文件
docker pull registry.cn-hangzhou.aliyuncs.com/kesu/data-download:v1

#  天枢预置数据集
docker pull registry.cn-hangzhou.aliyuncs.com/kesu/data-dataset:v1

#  炼知模型
docker pull registry.cn-hangzhou.aliyuncs.com/kesu/data-model-lianzhi:v1

#  算法镜像
docker pull registry.cn-hangzhou.aliyuncs.com/dubhe-tianshu/algorithm:v1
docker tag registry.cn-hangzhou.aliyuncs.com/dubhe-tianshu/algorithm:v1 algorithm:v1

#  serving镜像
docker pull registry.cn-hangzhou.aliyuncs.com/dubhe-tianshu/serving-gpu:base
docker tag registry.cn-hangzhou.aliyuncs.com/dubhe-tianshu/serving-gpu:base  harbor.dubhe.com/train/serving-gpu:base 

#  notebook镜像
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/notebook:v1

#  pytorch-demo镜像
docker pull registry.cn-hangzhou.aliyuncs.com/dubhe-tianshu/pytorch-demo:v1  

```


## 部署分布式训练 Operator

### operator介绍#
operator 管理分布式训练容器，主要包含以下功能：

- 管理分布式训练容器生命周期。
- 为分布式训练容器注入其他容器ip


部署比较复杂，暂时跳过


## 部署天枢服务
安装编译环境，由于 sdk 已经安装在服务器，所以sdk安装跳过，下边安装 maven、nodejs 环境；

### 安装 maven:
```bash
mkdir -p /data/deeplearning/softwares
mkdir -p /data/deeplearning/env

cd /data/deeplearning/softwares

# 1.下载解压
wget --no-check-certificate https://mirrors.bfsu.edu.cn/apache/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
tar -zxvf apache-maven-3.8.8-bin.tar.gz -C  /usr/local/
ln -s /usr/local/apache-maven-3.8.8 /usr/local/maven

# 2.修改全局配置文件
vim /etc/profile.d/maven.sh
cat /etc/profile.d/maven.sh

# maven 3.18
export M2_HOME=/usr/local/maven
export PATH=${M2_HOME}/bin:$PATH

# 3.运行source /etc/profile使其生效
> source /etc/profile.d/maven.sh
> mvn -version

[root@192.168.1.101 local]#mvn -version
Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /usr/local/maven
Java version: 1.8.0_362, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.249-1.el7.elrepo.x86_64", arch: "amd64", family: "unix"

```

### nodejs 安装
```bash

cd /data/deeplearning/softwares

# 1.下载解压
wget https://nodejs.org/dist/v14.17.0/node-v14.17.0-linux-x64.tar.xz
tar -xf node-v14.17.0-linux-x64.tar.xz -C /usr/local/
ln -s /usr/local/node-v14.17.0-linux-x64  /usr/local/nodejs

# 2.修改全局配置文件
vim /etc/profile.d/nodejs.sh
cat /etc/profile.d/nodejs.sh
#node-v14.17.0
export NODE_HOME=/usr/local/nodejs
export PATH=$NODE_HOME/bin:$PATH

# 3.运行source /etc/profile使其生效
source /etc/profile.d/nodejs.sh
root@tianshu:/home# npm -v
6.14.13
root@tianshu:/home# node -v
v14.17.0

4.安装淘宝镜像
root@tianshu:/home# npm config set registry https://registry.npm.taobao.org
root@tianshu:/home# npm install -g cnpm --registry=https://registry.npm.taobao.org
```


## 部署Java等服务

创建命名空间：
```bash
kubectl create namespace dubhe-system
```

创建代码目录：
```bash
mkdir -p /data/deeplearning/code
```

拉取部署代码：
```bash
cd /data/deeplearning/code
git clone https://gitee.com/zhijiangtianshu/dubhe-deploy.git

```

修改 kubeconfig 配置文件：
```bash
# 查看kubeconfig 文件
# kubectl config view
cat ~/.kube/config

# 将kubeconfig文件添加到kubeconfig.yaml
# cd dubhe-deploy && cat /etc/kubernetes/admin.conf > template/kubeconfig.yaml
cd dubhe-deploy && cat ~/.kube/config > template/kubeconfig.yaml
```

修改  configmap.yaml 文件
```bash
vim configmap.yaml
```

#### 中间件拉取：
```bash
# 拉公网镜像到本地
docker pull nacos/nacos-mysql:5.7
docker pull nacos/nacos-server:2.0.3
docker pull redis:5.0.7
docker pull bitnami/node:12.22.4   # nodejs
docker pull nginx:1.20.0
docker pull maven:3.5-jdk-8   # java镜像环境
docker pull minio/minio:RELEASE.2020-04-28T23-56-56Z


# 打标签
docker tag nacos/nacos-mysql:5.7   magic-harbor.magic.com/library/nacos-mysql:5.7
docker tag nacos/nacos-server:2.0.3   magic-harbor.magic.com/library/nacos-server:2.0.3
docker tag redis:5.0.7   magic-harbor.magic.com/library/redis:5.0.7
docker tag bitnami/node:12.22.4  magic-harbor.magic.com/library/node:12.22.4
docker tag nginx:1.20.0  magic-harbor.magic.com/library/nginx:1.20.0
docker tag maven:3.5-jdk-8 magic-harbor.magic.com/library/maven:3.5-jdk-8
docker tag minio/minio:RELEASE.2020-04-28T23-56-56Z  magic-harbor.magic.com/library/minio:RELEASE.2020-04-28T23-56-56Z

# 推送到镜像仓库
docker push magic-harbor.magic.com/library/nacos-mysql:5.7
docker push magic-harbor.magic.com/library/nacos-server:2.0.3
docker push magic-harbor.magic.com/library/redis:5.0.7
docker push magic-harbor.magic.com/library/node:12.22.4
docker push magic-harbor.magic.com/library/nginx:1.20.0
docker push magic-harbor.magic.com/library/maven:3.5-jdk-8
docker push magic-harbor.magic.com/library/minio:RELEASE.2020-04-28T23-56-56Z
```

配置 `/data/deeplearning/code/dubhe-deploy/configmap.yaml`  文件相关参数；

#### 执行构建init镜像

修改 dubhectl 文件
```bash
$157    docker run --rm --entrypoint="" -v $PWD/Dubhe:/Dubhe registry.cn-hangzhou.aliyuncs.com/enlin/node:v1 /bin/bash -c 'cd /Dubhe/webapp/ &&  npm install && npm run build:prod'

#  修改原文件 指定node版本  node:v1更改为 node:12.22.4版本
# $157    docker run --rm --entrypoint="" -v $PWD/Dubhe:/Dubhe node:12.22.4 /bin/bash -c 'cd /Dubhe/webapp/ &&  npm install && npm run build:prod'
$157    docker run --rm --entrypoint="" -v $PWD/Dubhe:/Dubhe magic-harbor.magic.com/library/node:12.22.4 /bin/bash -c 'cd /Dubhe/webapp/ &&  npm install && npm run build:prod'
```
需要时间和网络有关，耐心等待，构建失败使用ps -ef | grep dubhe|grep -v grep |awk '{print $2}' | xargs kill -9 终止进程

```bash
cd /data/deeplearning/code/dubhe-deploy
./dubhectl build-image all
```


```bash
MYSQL_INIT_IMAGE
MINIO_INIT_IMAGE
NACOS_INIT_IMAGE
STORAGE_INIT_IMAGE
BACKEND_INIT_IMAGE
BACKEND_INIT_IMAGE

# 构建业务镜像后会自动打tag,只需要推送到仓库
docker push magic-harbor.magic.com/deeplearning/mysql-init:v1
docker push magic-harbor.magic.com/deeplearning/backend-init:v1
docker push magic-harbor.magic.com/deeplearning/nacos-init:v1
docker push magic-harbor.magic.com/deeplearning/minio-init:v1
docker push magic-harbor.magic.com/deeplearning/web:v1


# ---- storage-init:v1 不打镜像，从官方仓库拉取现成的用；
# 拉镜像
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/storage-init:v1
# 打tag
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/storage-init:v1 magic-harbor.magic.com/deeplearning/storage-init:v1
# 推送仓库
docker push magic-harbor.magic.com/deeplearning/storage-init:v1

# notebook
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/notebook:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/notebook:v1 magic-harbor.magic.com/deeplearning/notebook:v1
docker push magic-harbor.magic.com/deeplearning/notebook:v1

# video sample
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/videosample:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/videosample:v1 magic-harbor.magic.com/deeplearning/videosample:v1
docker push magic-harbor.magic.com/deeplearning/videosample:v1

# oneflow-gpu
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/oneflow-gpu:base
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/oneflow-gpu:base magic-harbor.magic.com/deeplearning/oneflow-gpu:base
docker push magic-harbor.magic.com/deeplearning/oneflow-gpu:base

----------------



------------------

# image process
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/imgprocess:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/imgprocess:v1 magic-harbor.magic.com/deeplearning/imgprocess:v1
docker push magic-harbor.magic.com/deeplearning/imgprocess:v1

# visual-server
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/visual-server:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/visual-server:v1 magic-harbor.magic.com/deeplearning/visual-server:v1
docker push magic-harbor.magic.com/deeplearning/visual-server:v1

# model-converter
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/model-converter:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/model-converter:v1 magic-harbor.magic.com/deeplearning/model-converter:v1
docker push magic-harbor.magic.com/deeplearning/model-converter:v1

# ofrecord
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/ofrecord:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/ofrecord:v1 magic-harbor.magic.com/deeplearning/ofrecord:v1
docker push magic-harbor.magic.com/deeplearning/ofrecord:v1

# model-measuring
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/model-measuring:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/model-measuring:v1 magic-harbor.magic.com/deeplearning/model-measuring:v1
docker push magic-harbor.magic.com/deeplearning/model-measuring:v1

# automl-nas-pytorch17:v1
docker pull registry.cn-hangzhou.aliyuncs.com/enlin/automl-nas-pytorch17:v1
docker tag registry.cn-hangzhou.aliyuncs.com/enlin/automl-nas-pytorch17:v1  magic-harbor.magic.com/deeplearning/automl-nas-pytorch17:v1
docker push magic-harbor.magic.com/deeplearning/automl-nas-pytorch17:v1

```


##### 构建业务镜像

```bash
# 构建服务后端
./dubhectl build-image backend
```

构建日志打印：
```bash
[INFO] 天枢云平台 0.0.1-SNAPSHOT ............................... SUCCESS [02:01 min]
[INFO] 通用Business模块（Biz） .................................. SUCCESS [  0.081 s]
[INFO] Biz 通用配置 ........................................... SUCCESS [01:17 min]
[INFO] Biz log 工具 ......................................... SUCCESS [  2.717 s]
[INFO] Biz 统一Rest返回工具结构 ................................... SUCCESS [  2.714 s]
[INFO] Biz file 工具 ........................................ SUCCESS [ 25.843 s]
[INFO] Biz 持久层操作 .......................................... SUCCESS [ 24.664 s]
[INFO] 通用SpringCloud模块（Cloud） ............................. SUCCESS [  0.052 s]
[INFO] Cloud swagger ...................................... SUCCESS [ 19.152 s]
[INFO] Biz 数据权限 ........................................... SUCCESS [  4.980 s]
[INFO] Biz redis 工具 ....................................... SUCCESS [03:11 min]
[INFO] 状态机 ................................................ SUCCESS [  2.255 s]
[INFO] Cloud 远程调用 ......................................... SUCCESS [ 25.594 s]
[INFO] Cloud 统一权限配置 ....................................... SUCCESS [ 19.055 s]
[INFO] Cloud 注册中心 ......................................... SUCCESS [ 29.858 s]
[INFO] Cloud 配置中心 ......................................... SUCCESS [  0.822 s]
[INFO] unit-test 单元测试 ..................................... SUCCESS [  5.568 s]
[INFO] 授权中心 ............................................... SUCCESS [  3.641 s]
[INFO] 网关 ................................................. SUCCESS [  8.036 s]
[INFO] k8s 通用模块 ........................................... SUCCESS [01:29 min]
[INFO] 垃圾回收 recycle 通用模块 .................................. SUCCESS [  7.072 s]
[INFO] Admin 系统服务 ......................................... SUCCESS [ 38.891 s]
[INFO] k8s 服务提供者 .......................................... SUCCESS [ 10.043 s]
[INFO] Dubhe 模型开发 ......................................... SUCCESS [ 19.034 s]
[INFO] 度量管理 ............................................... SUCCESS [  7.933 s]
[INFO] 镜像管理 ............................................... SUCCESS [  9.558 s]
[INFO] model 模型管理 ......................................... SUCCESS [ 15.564 s]
[INFO] algorithm 算法管理 ..................................... SUCCESS [ 14.109 s]
[INFO] Data 数据处理 .......................................... SUCCESS [ 51.255 s]
[INFO] 医学影像 ............................................... SUCCESS [ 12.686 s]
[INFO] Task 数据集定时任务 ....................................... SUCCESS [01:19 min]
[INFO] train 训练管理 ......................................... SUCCESS [ 18.396 s]
[INFO] Dubhe 模型优化 ......................................... SUCCESS [ 11.627 s]
[INFO] 云端Serving .......................................... SUCCESS [ 34.069 s]
[INFO] 云端Serving网关 ........................................ SUCCESS [  5.824 s]
[INFO] 专业版终端服务 ............................................ SUCCESS [ 10.352 s]
[INFO] Dubhe TADL ......................................... SUCCESS [ 37.809 s]
[INFO] Dubhe 点云标注 0.0.1-SNAPSHOT .......................... SUCCESS [ 13.551 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:45 min
[INFO] Finished at: 2023-11-03T02:25:51Z
[INFO] ------------------------------------------------------------------------
Sending build context to Docker daemon  5.805GB
Step 1/4 : FROM maven:3.5-jdk-8
 ---> 985f3637ded4
Step 2/4 : WORKDIR /data
 ---> Running in 3a740b0e74ec
Removing intermediate container 3a740b0e74ec
 ---> 7397e3960ad2
Step 3/4 : COPY */target/*-exec.jar /data/
 ---> 2ec370ce7a5f
Step 4/4 : CMD echo 1
 ---> Running in 84bc48fa9237
Removing intermediate container 84bc48fa9237
 ---> 360589f3081a
Successfully built 360589f3081a
Successfully tagged magic-harbor.magic.com/deeplearning/dubhe-java:v1
[info] Please push image magic-harbor.magic.com/deeplearning/dubhe-java:v1 to image repo if needed
[root@192.168.1.101 dubhe-deploy]# 
```

将镜像推送到镜像仓库：
```bash
# 此镜像 6.4G 左右
docker push magic-harbor.magic.com/deeplearning/dubhe-java:v1

```

### 构建web镜像
```bash

# 清楚npm安装缓存
#npm cache clean --force
#rm -rf node_modules
#rm package-lock.json
#npm install

# 在构建镜像时，会将代码拷贝到/image/目录下
# /data/deeplearning/code/dubhe-deploy/image/web
# 建议不要直接使用 cnpm 安装依赖，会有各种诡异的 bug。可以通过如下操作解决 npm 下载速度慢的问题
#npm install --registry=https://registry.npm.taobao.org
# 安装依赖
#npm install
# 构建生产环境
#npm run build:prod

# npm install --unsafe-perm=true --allow-root

# 如果web代码有改动，需要删除 /data/deeplearning/code/dubhe-deploy/image/web/Dubhe 拷贝的代码，否则不会再拷贝 Code代码
# 构建服务后端

# 或者可以再该目录直接改，不需要拷贝代码
# cd /data/deeplearning/code/dubhe-deploy/image/web/Dubhe

cd /data/deeplearning/code/dubhe-deploy

# ./dubhectl build-image web
# 删除旧的镜像，以免冲突
docker rmi magic-harbor.magic.com/deeplearning/web:v4
nohup ./dubhectl build-image web > output.log 2>&1 &

tail -f output.log 
# docker push magic-harbor.magic.com/deeplearning/web:v4

# 然后替换web  镜像，重新拉取pod服务
# 打标签
docker tag magic-harbor.magic.com/deeplearning/web:v4   magic-harbor.magic.com/deeplearning/web:v11
docker push  magic-harbor.magic.com/deeplearning/web:v11

```

### 安装服务：
```bash
cd /data/deeplearning/code/dubhe-deploy

# 安装所有服务
./dubhectl install
```

确认服务是否安装成功：
```bash
[root@k8s ~]# kubectl get po -n dubhe-system -w
[root@k8s ~]# kubectl get pod -n dubhe-system
NAME                                       READY   STATUS    RESTARTS   AGE
algorithm-imgprocess-7cf8f9b984-fh8cr      1/1     Running   0          41h
algorithm-imgprocess-7cf8f9b984-v49gm      1/1     Running   0          41h
algorithm-ofrecord-9cf987d7b-84vpn         1/1     Running   0          41h
algorithm-ofrecord-9cf987d7b-q5dxz         1/1     Running   0          41h
algorithm-videosample-bb996d54b-8ppbh      1/1     Running   0          41h
algorithm-videosample-bb996d54b-vw64s      1/1     Running   0          41h
backend-68f896cb54-6hfj2                   1/1     Running   0          41h
backend-admin-77f5d8cbb-g4m7d              1/1     Running   0          41h
backend-algorithm-5966599b6c-n7vlb         1/1     Running   0          41h
backend-auth-5568f66d45-5h6jn              1/1     Running   0          41h
backend-auth-5568f66d45-bfmm5              1/1     Running   0          17h
backend-data-56c6c77cdd-8nwn2              1/1     Running   0          41h
backend-data-dcm-7ff4d5b757-l6d7d          1/1     Running   0          41h
backend-data-task-77f4bccd88-s9qhq         1/1     Running   0          41h
backend-dcm4chee-67958d9547-skjh8          3/3     Running   0          41h
backend-gateway-6b4c55bbcc-mf54s           1/1     Running   0          41h
backend-image-577468957c-4gvgc             1/1     Running   0          41h
backend-k8s-7f4bfbb674-bf2pl               1/1     Running   0          41h
backend-measure-67bf8cfddb-2fw49           1/1     Running   0          41h
backend-model-6fd6f7b597-vdvgn             1/1     Running   0          41h
backend-model-converter-594f46d85b-ffjmz   1/1     Running   0          41h
backend-model-measure-7db9894d98-xrwfq     1/1     Pending   0          41h
backend-notebook-5c8bc8686f-zld5k          1/1     Running   0          41h
backend-optimize-855898859-snxxb           1/1     Running   0          41h
backend-point-cloud-66ddff8cc6-tk95b       1/1     Running   0          41h
backend-serving-8d4686bc4-xfpf8            1/1     Running   0          41h
backend-serving-gateway-5747b9d495-4nx82   1/1     Running   0          41h
backend-tadl-76d7c5fbb7-t7ssh              1/1     Running   0          41h
backend-terminal-5df96477cf-hs9ms          1/1     Running   0          41h
backend-train-69d85d86b8-wfxvl             1/1     Running   0          41h
backend-visual-5f8774f766-dslpn            1/1     Running   0          41h
minio-758894565d-vvwbq                     1/1     Running   0          41h
mysql-6dd845548b-wnch9                     1/1     Running   0          41h
nacos-5c9f6c67f8-wrjwl                     1/1     Running   0          41h
redis-8c9c45cb6-29txf                      1/1     Running   0          41h
web-7d94f7d46-m84pz                        1/1     Running   0          41h
```



查看pvc问题：
```bash
kubectl describe pvc nacos-storage -n dubhe-system


[root@192.168.1.101 data]#docker pull magic-harbor.magic.com/deeplearning/imgprocess:v1
Error response from daemon: unknown: repository deeplearning/imgprocess not found


```

启动之后的服务情况：
```bash

[root@192.168.1.101 dubhe-deploy]#kubectl get pods -n dubhe-system 
NAME                                       READY   STATUS                  RESTARTS   AGE
algorithm-imgprocess-d7cd67d9c-7qzck       0/1     ImagePullBackOff        0          99m
algorithm-imgprocess-d7cd67d9c-bchd8       0/1     ImagePullBackOff        0          99m
algorithm-ofrecord-b897b9b7-6q57m          0/1     ErrImagePull            0          99m
algorithm-ofrecord-b897b9b7-sztfg          0/1     ImagePullBackOff        0          11h
algorithm-videosample-6c99c8dddf-9j475     0/1     ImagePullBackOff        0          99m
algorithm-videosample-6c99c8dddf-rnx2r     0/1     ImagePullBackOff        0          99m
backend-75f6b5675d-7kt8n                   1/1     Running                 0          114m
backend-admin-679d97c767-frf78             1/1     Running                 0          109m
backend-algorithm-748fd56bd8-87f6s         0/1     CrashLoopBackOff        20         127m
backend-auth-5565c87f8b-nmrbn              1/1     Running                 0          127m
backend-data-7bd45f5df9-fc5w4              0/1     CrashLoopBackOff        18         127m
backend-data-dcm-fdb999858-2976l           1/1     Running                 19         127m
backend-data-task-6dcc689ff9-mlwrx         1/1     Running                 16         127m
backend-dcm4chee-5786c6c5f4-gd4f6          0/3     Init:ImagePullBackOff   0          127m
backend-gateway-c5f8cf99f-6jqwf            1/1     Running                 0          127m
backend-image-7479f9c995-wllxj             0/1     CrashLoopBackOff        17         127m
backend-k8s-5565749dfc-vtpqs               0/1     Error                   20         127m
backend-measure-68f6bd665c-5cgbz           0/1     CrashLoopBackOff        19         127m
backend-model-79c55745bd-qztrg             1/1     Running                 18         127m
backend-model-converter-548cd44bbc-wzpvd   0/1     ImagePullBackOff        0          99m
backend-model-measure-85d9fc88dc-jv8cb     0/1     Pending                 0          99m
backend-notebook-546c7976cb-mgb58          0/1     CrashLoopBackOff        18         127m
backend-optimize-64b977c9db-ljtps          0/1     CrashLoopBackOff        20         127m
backend-point-cloud-5db5cfcd55-jb5nf       0/1     CrashLoopBackOff        19         127m
backend-serving-5bb69d45df-9z6zx           1/1     Running                 18         127m
backend-serving-gateway-6c868dbdcd-g5z6r   1/1     Running                 0          127m
backend-tadl-77d994d86b-wxl7c              0/1     CrashLoopBackOff        17         127m
backend-terminal-5c588dff8-gsmwr           0/1     CrashLoopBackOff        20         127m
backend-train-56dc748b59-27tvv             0/1     CrashLoopBackOff        20         127m
backend-visual-b8ccdf5c5-htgrr             0/1     ImagePullBackOff        0          99m
minio-5667b6c599-l9hqp                     1/1     Running                 0          11h
mysql-7f894d9569-ltd62                     1/1     Running                 0          11h
nacos-5546777749-2ndxl                     1/1     Running                 0          11h
redis-54f6f4cc9d-tzgj5                     1/1     Running                 0          11h
web-585b7c5646-mf6lh                       1/1     Running                 0          4m50s
[root@192.168.1.101 dubhe-deploy]#
```

服务访问：
```bash
[root@192.168.1.101 dubhe-deploy]#kubectl get svc -n dubhe-system 
NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
backend                           ClusterIP   10.68.10.104    <none>        80/TCP                                128m
backend-admin                     ClusterIP   10.68.144.231   <none>        8870/TCP                              128m
backend-algorithm                 ClusterIP   10.68.249.145   <none>        8889/TCP                              128m
backend-auth                      ClusterIP   10.68.1.35      <none>        8866/TCP                              128m
backend-data                      ClusterIP   10.68.9.51      <none>        8823/TCP                              128m
backend-data-dcm                  ClusterIP   10.68.246.38    <none>        8011/TCP                              128m
backend-data-task                 ClusterIP   10.68.76.170    <none>        8801/TCP                              128m
backend-dcm4chee                  ClusterIP   10.68.115.165   <none>        8080/TCP                              128m
backend-dcm4chee-console          NodePort    10.68.81.185    <none>        8080:30088/TCP,11112:31112/TCP        11h
backend-gateway                   ClusterIP   10.68.230.238   <none>        8800/TCP                              128m
backend-image                     ClusterIP   10.68.153.178   <none>        8822/TCP                              128m
backend-k8s                       ClusterIP   10.68.187.251   <none>        8960/TCP                              128m
backend-k8s-console               NodePort    10.68.205.14    <none>        8960:30960/TCP                        11h
backend-measure                   ClusterIP   10.68.249.87    <none>        8821/TCP                              128m
backend-model                     ClusterIP   10.68.0.201     <none>        8888/TCP                              128m
backend-model-converter           ClusterIP   10.68.244.70    <none>        8080/TCP                              11h
backend-model-converter-console   NodePort    10.68.166.50    <none>        8080:32230/TCP                        11h
backend-model-measure             ClusterIP   10.68.111.69    <none>        8080/TCP                              11h
backend-model-measure-console     NodePort    10.68.74.204    <none>        8080:32760/TCP                        11h
backend-notebook                  ClusterIP   10.68.248.211   <none>        8863/TCP                              128m
backend-optimize                  ClusterIP   10.68.23.232    <none>        8899/TCP                              128m
backend-point-cloud               ClusterIP   10.68.51.37     <none>        8865/TCP                              128m
backend-serving                   ClusterIP   10.68.205.5     <none>        8898/TCP                              128m
backend-serving-gateway           ClusterIP   10.68.125.8     <none>        8081/TCP                              128m
backend-serving-gateway-console   NodePort    10.68.233.25    <none>        8081:30081/TCP                        11h
backend-tadl                      ClusterIP   10.68.3.94      <none>        8864/TCP                              128m
backend-train                     ClusterIP   10.68.234.175   <none>        8890/TCP                              128m
backend-visual                    ClusterIP   10.68.94.195    <none>        9898/TCP,9797/TCP                     128m
dubhe-terminal                    ClusterIP   10.68.241.59    <none>        8970/TCP                              128m
minio                             ClusterIP   10.68.101.86    <none>        9000/TCP                              11h
minio-console                     NodePort    10.68.184.176   <none>        9000:30900/TCP                        11h
mysql                             ClusterIP   10.68.1.26      <none>        3306/TCP                              11h
mysql-console                     NodePort    10.68.69.23     <none>        3306:30678/TCP                        11h
nacos                             ClusterIP   10.68.130.65    <none>        8848/TCP,9848/TCP,9849/TCP,7848/TCP   11h
nacos-console                     NodePort    10.68.133.100   <none>        8848:30848/TCP                        11h
redis                             ClusterIP   10.68.51.86     <none>        6379/TCP                              11h
redis-console                     NodePort    10.68.59.45     <none>        6379:30379/TCP                        11h
web                               ClusterIP   10.68.251.160   <none>        80/TCP                                11h
web-console                       NodePort    10.68.229.200   <none>        80:30800/TCP                          11h
[root@192.168.1.101 dubhe-deploy]#

```

本地镜像：
```bash
[root@192.168.1.101 dubhe-deploy]#docker images | grep deeplearning
magic-harbor.magic.com/deeplearning/web                                  v1                        347ce65620d9   32 minutes ago   162MB
magic-harbor.magic.com/deeplearning/web                                  v2                        347ce65620d9   32 minutes ago   162MB
magic-harbor.magic.com/deeplearning/dubhe-java                           v1                        360589f3081a   6 hours ago      6.4GB
magic-harbor.magic.com/deeplearning/backend-init                         v1                        b7a7f57df2c1   19 hours ago     89MB
magic-harbor.magic.com/deeplearning/nacos-init                           v1                        5b1c84983827   19 hours ago     89MB
magic-harbor.magic.com/deeplearning/minio-init                           v1                        bc83c3ecf729   19 hours ago     136MB
magic-harbor.magic.com/deeplearning/mysql-init                           v1                        d87cab477501   19 hours ago     760MB
magic-harbor.magic.com/deeplearning/storage-init                         v1                        3c720082e7ef   12 months ago    1.83GB
magic-harbor.magic.com/deeplearning/notebook                             v1                        3046e7cd7537   17 months ago    21.7GB
```


## 后台访问

### 深度学习平台
http://192.168.1.101:30800/login
admin/admin

### Nacos后端
http://192.168.1.101:30848/nacos/
nacos/nacos

### Minio后端
http://192.168.1.101:30900/minio/login
accessKey: admin
secretKey: abcdefg123456

### Mysql
192.168.1.101:30678
root/root

### Redis
192.168.1.101:30379
Tianshu@123


## 排错

```bash
kubectl logs -f backend-admin-866c657ffb-pdchs -n dubhe-system
kubectl describe pod backend-admin-866c657ffb-pdchs -n dubhe-system

# 查看初始化容器日志
kubectl logs backend-admin-679d97c767-rp4lv -n dubhe-system -c backend-init

[root@192.168.1.101 ~]#kubectl logs backend-admin-679d97c767-rp4lv -n dubhe-system -c backend-init
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0curl: (6) Could not resolve host: backend.dubhe-system.svc.cluster.local
  
backend.dubhe-system.svc.cluster.local port

# backend 服务未启动
kubectl logs -n dubhe-system -f backend-75f6b5675d-k6qrn 
kubectl describe pod backend-75f6b5675d-k6qrn  -n dubhe-system
kubectl logs backend-75f6b5675d-k6qrn -n dubhe-system -c mysql-init

kubectl logs backend-75f6b5675d-84skm  -n dubhe-system -c mysql-init

# dubhe-k8s 服务

# dubhe-data 服务
kubectl logs -n dubhe-system -f backend-data-7bd45f5df9-fc5w4


```

错误2：
```bash
> kubectl logs backend-admin-679d97c767-8882t -n dubhe-system -c backend-init
Failed to connect to backend.dubhe-system.svc.cluster.local port 80: Connection refused

# 排查 dubhe-backend 服务未起来
kubectl logs -f backend-75f6b5675d-xj8gk -n dubhe-system -c mysql-init
kubectl logs -f backend-75f6b5675d-xj8gk -n dubhe-system -c minio-init
kubectl logs -f backend-75f6b5675d-xj8gk -n dubhe-system -c nacos-init
kubectl logs -f backend-75f6b5675d-xj8gk -n dubhe-system -c storage-init
```

删掉 所有 backend_前缀的pod就正常了；


coredns报错异常：
```bash

k8s集群内，无法通过svc 域名访问，以下为测试报错：
root@backend-auth-5565c87f8b-nmrbn:/data# ping mysql.dubhe-system.svc.cluster.local
ping: mysql.dubhe-system.svc.cluster.local: Temporary failure in name resolution
root@backend-auth-5565c87f8b-nmrbn:/data#


[root@192.168.1.101 ~]#kubectl get pods -n kube-system | grep coredns
coredns-5787695b7f-wbpj9                                          0/1     ImagePullBackOff   0          15h

# 处理方法，删掉重启
[root@192.168.1.101 ~]#kubectl delete pod coredns-5787695b7f-wbpj9 -n kube-system
pod "coredns-5787695b7f-wbpj9" deleted


# 重启之后就ok了

root@backend-auth-5565c87f8b-nmrbn:/data# ping mysql.dubhe-system.svc.cluster.local
PING mysql.dubhe-system.svc.cluster.local (10.68.1.26) 56(84) bytes of data.
64 bytes from mysql.dubhe-system.svc.cluster.local (10.68.1.26): icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from mysql.dubhe-system.svc.cluster.local (10.68.1.26): icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from mysql.dubhe-system.svc.cluster.local (10.68.1.26): icmp_seq=3 ttl=64 time=0.100 ms
64 bytes from mysql.dubhe-system.svc.cluster.local (10.68.1.26): icmp_seq=4 ttl=64 time=0.102 ms

```

```markdown
这段信息显示容器在启动后很快就终止了，并给出了错误原因为 Error，退出码为 1。同时，还提供了启动和结束时间。

根据提供的信息，我们可以得出以下结论：

容器 backend-init 在启动后立即出现了错误并终止。
错误原因未明确给出，需要进一步查看容器的日志或其他相关信息来确定具体的错误原因。
建议通过以下步骤来进一步调查问题：

1. 使用 kubectl logs 命令获取容器的日志，例如：kubectl logs backend-admin-866c657ffb-pdchs -n dubhe-system -c backend-init。这将显示容器的日志输出，有助于确定错误的具体原因。
2. 检查容器所需的资源是否可用。例如，检查 PersistentVolumeClaim dubhe-storage 是否存在，并且是否满足容器的要求。
3. 检查容器镜像是否正确，并且在集群中是否可用。确保镜像的版本和位置与报错信息中提供的一致。
4. 如果以上步骤都没有找到问题，可以尝试重新创建该 Pod，以排除可能的临时问题。
通过进一步调查容器的日志和资源情况，您应该能够确定引起容器 backend-init 错误的具体原因，并采取相应的措施来解决问题。
```


### backend-k8s pod 问题排查

查看kubeconfig ip 配置：
```bash
[root@192.168.1.101 dubhe-deploy]#kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:6443
Elasticsearch is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/elasticsearch-logging:http/proxy
Kibana is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
KubeDNSUpstream is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns-upstream:dns/proxy
kubernetes-dashboard is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@192.168.1.101 dubhe-deploy]#

```

在 nacos kubeconfig.yaml 文件配置：
需要将api-server 改为服务方式：
```bash
# 本地服务改为cluster 网络，或者是直接暴露服务 service
https://127.0.0.1:6443 改为  https://192.168.1.248:8443
```

### minio pvc 挂载问题
```bash
kubectl patch pv pvc-085b95ea-402d-45f9-80b3-423048e3e6d9 -p '{"spec":{"claimRef":null}}'

kubectl delete pvc dubhe-storage -n dubhe-system
kubectl get pv | grep  storage
kubectl delete pv pvc-085b95ea-402d-45f9-80b3-423048e3e6d9




```


### mysql 挂载pvc问题
k8s部署mysql数据持久化时，pv非空问题
https://www.jianshu.com/p/b4d27b3263f3

K8S容器服务部署mysql 5.7以上版本镜像，/var/lib/mysql挂载pvc后无法启动

报错：
[ERROR] --initialize specified but the data directory has files in it. Aborting.

我们在日常开发中经常会用到MySQL数据库，最简单的办法是通过k8s去部署，依赖k8s集群保持高可用，依赖pv/pvc持久化数据。

在一切配好后，启动MySQL的过程中，可能会出错：
[ERROR] --initialize specified but the data directory has files in it. Aborting.

原因是MySQL的初始化需要一个非空的文件夹，但是创建PV的时候默认会有lost+found文件夹，所以导致初始化失败。解决的办法有：

通过挂载subPath子路径

添加args: ["--ignore-db-dir=lost+found"]

这里附上一个我在本地测试的demo供参考，由于自己电脑搭nfs有问题，所以PV选择的是本地。

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  replicas: 1
  serviceName: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:latest
        env:
        - name: MYSQL_ROOT_PASSWORD    # 这里设置root的密码
          value: root
        resources:
          requests:
            cpu: "50m"
            memory: 1Gi
          limits:
            cpu: "1"
            memory: 2Gi
        volumeMounts:
        - name: mysql-store
          mountPath: /var/lib/mysql  # MySQL容器的数据默认都是存在这个目录
          subPath: mysqldata  # 这里通过subPath挂载到子目录，就可以避免出错
      volumes:
      - name: mysql-store
        persistentVolumeClaim:
          claimName: pvc-hostpath
```


### 自定义notebook调度到指定节点

```bash

# ①、查看集群节点标签，确保node有这个labe
# kubectl get nodes --show-labels | grep gpu
kubectl get nodes --show-labels | grep 'gpucore=true'

# ②、添加集群节点标签（如果该节点不存在该标签，则需要手动添加）
# kubectl label nodes node01 gpu=true
kubectl label nodes 192.168.1.101 gpucore=gpu

# ③、检查是否有这个pod，如果没有，则需要创建
#kubectl get pods -A | grep  nginx-kusc004

# ④、创建pod(拷贝官网案例，修改pod名称和镜像，删除多余的部分即可)
#vim  pod-disk-ssd-07.yaml

# ⑤、部署pod
kubectl apply -f pod-disk-ssd-07.yaml

# ⑥、验证pod是否调度到 node1上
> kubectl get pod nginx-kusc00401 -o wide
```
查看修改后的yaml文件：
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    gpucore: gpu
```

由于 101 节点打了污点，所以上述的标签无法做到调度，需要配置容忍度才可以调度：
```bash
[root@192.168.1.101 nfs]#kubectl get nodes --show-labels | grep gpu
192.168.1.101   Ready                      node     26d    v1.20.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu=true,gpucore=gpu,kubernetes.io/arch=amd64,kubernetes.io/hostname=192.168.1.101,kubernetes.io/os=linux,kubernetes.io/role=node
[root@192.168.1.101 nfs]#kubectl describe nodes | grep -i Taints
Taints:             key1=value1:NoSchedule
```

容忍度的pod:

添加容忍度污点：
```bash
kubectl taint nodes 192.168.1.101 gpu=gpu:NoSchedule
```

打污点：
```bash
[root@192.168.1.101 nfs]#kubectl describe nodes | grep -i Taints
Taints:             gpu=gpu:NoSchedule
```

现在这个节点有两个污点：
```
两个污点gpu=gpu:NoSchedule和key1=value1:NoSchedule，
```


示例
```bash

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    gpucore: gpu
  tolerations:
  - key: "gpu"
    operator: "Exists"
    effect: "NoSchedule"
```


污点调度：

新的notebook:
```bash
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: notebook-rn-202311041841105969he6-v0jm3
  namespace: namespace-1
  uid: 7e48d4f6-fcf8-44a5-93ca-ec38c8a4220b
  resourceVersion: '37797866'
  generation: 1
  creationTimestamp: '2023-11-06T02:38:38Z'
  labels:
    platform/business: notebook
    platform/create-by: platform
    platform/resource-name: notebook-rn-202311041841105969he6
    platform/runtime-env: prod
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2023-11-06T02:38:38Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:collisionCount: {}
          f:currentReplicas: {}
          f:currentRevision: {}
          f:observedGeneration: {}
          f:replicas: {}
          f:updateRevision: {}
          f:updatedReplicas: {}
    - manager: okhttp
      operation: Update
      apiVersion: apps/v1
      time: '2023-11-06T02:38:38Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:platform/business: {}
            f:platform/create-by: {}
            f:platform/resource-name: {}
            f:platform/runtime-env: {}
        f:spec:
          f:podManagementPolicy: {}
          f:replicas: {}
          f:revisionHistoryLimit: {}
          f:selector: {}
          f:serviceName: {}
          f:template:
            f:metadata:
              f:labels:
                .: {}
                f:platform/business: {}
                f:platform/create-by: {}
                f:platform/p-kind: {}
                f:platform/p-name: {}
                f:platform/resource-name: {}
                f:platform/runtime-env: {}
                f:platform/task-identify: {}
              f:name: {}
              f:namespace: {}
            f:spec:
              f:containers:
                k:{"name":"notebook-rn-202311041841105969he6-v0jm3"}:
                  .: {}
                  f:image: {}
                  f:imagePullPolicy: {}
                  f:name: {}
                  f:ports:
                    .: {}
                    k:{"containerPort":8888,"protocol":"TCP"}:
                      .: {}
                      f:containerPort: {}
                      f:name: {}
                      f:protocol: {}
                  f:resources:
                    .: {}
                    f:limits:
                      .: {}
                      f:cpu: {}
                      f:memory: {}
                  f:terminationMessagePath: {}
                  f:terminationMessagePolicy: {}
                  f:volumeMounts:
                    .: {}
                    k:{"mountPath":"/dataset"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                      f:readOnly: {}
                    k:{"mountPath":"/dev/shm"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                    k:{"mountPath":"/home/admin/.local/lib/python3.8/site-packages"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                    k:{"mountPath":"/workspace"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
              f:dnsPolicy: {}
              f:restartPolicy: {}
              f:schedulerName: {}
              f:securityContext: {}
              f:terminationGracePeriodSeconds: {}
              f:volumes:
                .: {}
                k:{"name":"dshm"}:
                  .: {}
                  f:emptyDir:
                    .: {}
                    f:medium: {}
                    f:sizeLimit: {}
                  f:name: {}
                k:{"name":"pvc-dataset"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
                k:{"name":"pvc-pip-site-package"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
                k:{"name":"pvc-workspace"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
          f:updateStrategy:
            f:rollingUpdate:
              .: {}
              f:partition: {}
            f:type: {}
  selfLink: >-
    /apis/apps/v1/namespaces/namespace-1/statefulsets/notebook-rn-202311041841105969he6-v0jm3
status:
  observedGeneration: 1
  replicas: 1
  currentReplicas: 1
  updatedReplicas: 1
  currentRevision: notebook-rn-202311041841105969he6-v0jm3-7785d86c77
  updateRevision: notebook-rn-202311041841105969he6-v0jm3-7785d86c77
  collisionCount: 0
spec:
  replicas: 1
  selector:
    matchLabels:
      platform/p-name: notebook-rn-202311041841105969he6-v0jm3
  template:
    metadata:
      name: notebook-rn-202311041841105969he6-v0jm3
      namespace: namespace-1
      creationTimestamp: null
      labels:
        platform/business: notebook
        platform/create-by: platform
        platform/p-kind: StatefulSet
        platform/p-name: notebook-rn-202311041841105969he6-v0jm3
        platform/resource-name: notebook-rn-202311041841105969he6
        platform/runtime-env: prod
        platform/task-identify: 8e7d01bf036d4ae39a98deeb47a66068
    spec:
      volumes:
        - name: pvc-pip-site-package
          hostPath:
            path: /nfs/dubhe-prod/pip-site-package/1/202311041841105969he6/
            type: Directory
        - name: pvc-dataset
          hostPath:
            path: /nfs/dubhe-prod/dataset/9/versionFile/V0001
            type: Directory
        - name: pvc-workspace
          hostPath:
            path: /nfs/dubhe-prod/algorithm-manage/1/20231104184110596dq4o/
            type: Directory
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 1Gi
      containers:
        - name: notebook-rn-202311041841105969he6-v0jm3
          image: magic-harbor.magic.com/deeplearning/notebook:v1
          ports:
            - name: web
              containerPort: 8888
              protocol: TCP
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
          volumeMounts:
              mountPath: /home/admin/.local/lib/python3.8/site-packages
            - name: pvc-dataset
              readOnly: true
              mountPath: /dataset
            - name: pvc-workspace
              mountPath: /workspace
            - name: dshm
              mountPath: /dev/shm
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 0
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
      nodeSelector:   # 节点label选择
        gpucore: gpu
      tolerations: # 容忍度，不允许其他pod调度该节点，只允许配置该参数的；
        - key: gpu
          operator: Equal
          value: gpu
          effect: NoSchedule
        - key: key1
          operator: Equal
          value: value1
          effect: NoSchedule
  serviceName: notebook-rn-202311041841105969he6-v0jm3
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  revisionHistoryLimit: 10

```

节点调度配置：
```yaml
     securityContext: {}
     schedulerName: default-scheduler
     nodeSelector:   # 节点label选择
        gpucore: gpu
      tolerations: # 容忍度，不允许其他pod调度该节点，只允许配置该参数的；
        - key: gpu
          operator: Equal
          value: gpu
          effect: NoSchedule
        - key: key1
          operator: Equal
          value: value1
          effect: NoSchedule
```


配置了两个污点后，pod调度到了 101 gpu 节点：
```bash
[root@192.168.1.101 nfs]#kubectl get pods -n namespace-1 -o wide
NAME                                        READY   STATUS              RESTARTS   AGE   IP       NODE            NOMINATED NODE   READINESS GATES
notebook-rn-202311041841105969he6-0pafl-0   0/1     ContainerCreating   0          7s    <none>   192.168.1.101   <none>           <none>
```

挂载路径：
```bash
-rw-r--r-- 1 root root 8039 Nov  5 17:24 notebook-gpu.yaml
[root@192.168.1.101 nfs]#mkdir -p  /nfs/dubhe-prod/pip-site-package/1/202311041841105969he6/
[root@192.168.1.101 nfs]#mkdir -p /nfs/dubhe-prod/dataset/9/versionFile/V0001
[root@192.168.1.101 nfs]#mkdir -p /nfs/dubhe-prod/algorithm-manage/1/20231104184110596dq4o/
[root@192.168.1.101 nfs]#
```



测试 -1108：
```bash

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: notebook-rn-202311080321367310pew-n1dkc
  namespace: namespace-1
  uid: c224704b-c113-4a47-99c0-d4cf9a07b775
  resourceVersion: '38739777'
  generation: 1
  creationTimestamp: '2023-11-08T03:27:51Z'
  labels:
    platform/business: notebook
    platform/create-by: platform
    platform/resource-name: notebook-rn-202311080321367310pew
    platform/runtime-env: prod
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2023-11-08T03:27:51Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:collisionCount: {}
          f:currentReplicas: {}
          f:currentRevision: {}
          f:observedGeneration: {}
          f:replicas: {}
          f:updateRevision: {}
          f:updatedReplicas: {}
    - manager: okhttp
      operation: Update
      apiVersion: apps/v1
      time: '2023-11-08T03:27:51Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:platform/business: {}
            f:platform/create-by: {}
            f:platform/resource-name: {}
            f:platform/runtime-env: {}
        f:spec:
          f:podManagementPolicy: {}
          f:replicas: {}
          f:revisionHistoryLimit: {}
          f:selector: {}
          f:serviceName: {}
          f:template:
            f:metadata:
              f:labels:
                .: {}
                f:platform/business: {}
                f:platform/create-by: {}
                f:platform/p-kind: {}
                f:platform/p-name: {}
                f:platform/resource-name: {}
                f:platform/runtime-env: {}
                f:platform/task-identify: {}
              f:name: {}
              f:namespace: {}
            f:spec:
              f:containers:
                k:{"name":"notebook-rn-202311080321367310pew-n1dkc"}:
                  .: {}
                  f:image: {}
                  f:imagePullPolicy: {}
                  f:name: {}
                  f:ports:
                    .: {}
                    k:{"containerPort":8888,"protocol":"TCP"}:
                      .: {}
                      f:containerPort: {}
                      f:name: {}
                      f:protocol: {}
                  f:resources:
                    .: {}
                    f:limits:
                      .: {}
                      f:cpu: {}
                      f:memory: {}
                  f:terminationMessagePath: {}
                  f:terminationMessagePolicy: {}
                  f:volumeMounts:
                    .: {}
                    k:{"mountPath":"/dataset"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                      f:readOnly: {}
                    k:{"mountPath":"/dev/shm"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                    k:{"mountPath":"/home/admin/.local/lib/python3.8/site-packages"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                    k:{"mountPath":"/workspace"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
              f:dnsPolicy: {}
              f:restartPolicy: {}
              f:schedulerName: {}
              f:securityContext: {}
              f:terminationGracePeriodSeconds: {}
              f:volumes:
                .: {}
                k:{"name":"dshm"}:
                  .: {}
                  f:emptyDir:
                    .: {}
                    f:medium: {}
                    f:sizeLimit: {}
                  f:name: {}
                k:{"name":"pvc-dataset"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
                k:{"name":"pvc-pip-site-package"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
                k:{"name":"pvc-workspace"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
          f:updateStrategy:
            f:rollingUpdate:
              .: {}
              f:partition: {}
            f:type: {}
  selfLink: >-
    /apis/apps/v1/namespaces/namespace-1/statefulsets/notebook-rn-202311080321367310pew-n1dkc
status:
  observedGeneration: 1
  replicas: 1
  currentReplicas: 1
  updatedReplicas: 1
  currentRevision: notebook-rn-202311080321367310pew-n1dkc-54445b768b
  updateRevision: notebook-rn-202311080321367310pew-n1dkc-54445b768b
  collisionCount: 0
spec:
  replicas: 1
  selector:
    matchLabels:
      platform/p-name: notebook-rn-202311080321367310pew-n1dkc
  template:
    metadata:
      name: notebook-rn-202311080321367310pew-n1dkc
      namespace: namespace-1
      creationTimestamp: null
      labels:
        platform/business: notebook
        platform/create-by: platform
        platform/p-kind: StatefulSet
        platform/p-name: notebook-rn-202311080321367310pew-n1dkc
        platform/resource-name: notebook-rn-202311080321367310pew
        platform/runtime-env: prod
        platform/task-identify: 79f8dfbb627e4c39959200c3c4127bce
    spec:
      volumes:
        - name: pvc-pip-site-package
          hostPath:
            path: /nfs/dubhe-prod/pip-site-package/1/202311080321367310pew/
            type: Directory
        - name: pvc-dataset
          hostPath:
            path: /nfs/dubhe-prod/dataset/17/versionFile/V0001
            type: Directory
        - name: pvc-workspace
          hostPath:
            path: /nfs/dubhe-prod/algorithm-manage/1/20231108032136747axnx/
            type: Directory
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 1Gi
      containers:
        - name: notebook-rn-202311080321367310pew-n1dkc
          image: magic-harbor.magic.com/deeplearning/notebook:v1
          ports:
            - name: web
              containerPort: 8888
              protocol: TCP
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
          volumeMounts:
            - name: pvc-pip-site-package
              mountPath: /home/admin/.local/lib/python3.8/site-packages
            - name: pvc-dataset
              readOnly: true
              mountPath: /dataset
            - name: pvc-workspace
              mountPath: /workspace
            - name: dshm
              mountPath: /dev/shm
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 0
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
      nodeSelector:   # 节点label选择
        gpucore: gpu
      tolerations: # 容忍度，不允许其他pod调度该节点，只允许配置该参数的；
        - key: gpu
          operator: Equal
          value: gpu
          effect: NoSchedule
        - key: key1
          operator: Equal
          value: value1
          effect: NoSchedule
  serviceName: notebook-rn-202311080321367310pew-n1dkc
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  revisionHistoryLimit: 10

```

```bash
# notebook1111
# pip_site_package
mkdir -p /nfs/dubhe-prod/pip-site-package/1/20231111010406298uzzy/

# data_source_path
mkdir -p /nfs/dubhe-prod/dataset/9/versionFile/V0001

# k8s pvc path
mkdir -p /nfs/dubhe-prod/algorithm-manage/1/202311110104062993cc2/


----
# notebook1112
# resource_name = 20231112015146510w55u

# pip_site_package
mkdir -p /nfs/dubhe-prod/pip-site-package/1/20231112015146510w55u/

# data_source_path
mkdir -p /nfs/dubhe-prod/dataset/9/versionFile/V0001

# k8s pvc path
mkdir -p /nfs/dubhe-prod/algorithm-manage/1/20231112015146510vfd2/

---


```

## 自动标注

```bash
cp -r dubhe_data_process/ /data/
cd /data/dubhe_data_process/

cd /data/dubhe_data_process/docker-image

# 图像分类识别
cd image-classification/
docker build -t algorithm-*:v1 .
```

## 查看镜像构建历史
```bash
docker history magic-harbor.magic.com/deeplearning/notebook:v1 
```

## 磁盘空间清理
手动删除 notebook 无法在 101 服务器上重启，经过测试，可能是 /usr/ 目录磁盘超过 95% 了，所以导致启动失败，需要清理该空间：

```bash 
du -h --max-depth=1 /usr

[root@192.168.1.101 usr]#du -h --max-depth=1 /usr
433M    /usr/bin
49M     /usr/sbin
2.0G    /usr/lib
946M    /usr/lib64
443M    /usr/share
0       /usr/etc
0       /usr/games
21M     /usr/include
206M    /usr/libexec
5.3G    /usr/local
244M    /usr/src
9.5G    /usr

du -h --max-depth=1 /usr/local

[root@192.168.1.101 local]#du -h --max-depth=1 /usr/local
46M     /usr/local/bin
0       /usr/local/etc
0       /usr/local/games
0       /usr/local/include
0       /usr/local/lib
0       /usr/local/lib64
0       /usr/local/libexec
0       /usr/local/sbin
0       /usr/local/share
0       /usr/local/src
5.1G    /usr/local/cuda-12.1
9.5M    /usr/local/apache-maven-3.8.8
167M    /usr/local/node-v14.17.0-linux-x64
5.3G    /usr/local
```