diff --git a/README.en.md b/README.en.md index 02d578337fade8dc58a5bc4835c3c3e3886a97d7..c76de628801085c13976dd0a3809e4f973aea0ef 100644 --- a/README.en.md +++ b/README.en.md @@ -5,10 +5,6 @@ NKD (NestOS Kubernetes Deployer) is a cluster deployment and operation tool developed by the NestOS team for container cloud scenarios. It covers a series of functions such as infrastructure and Kubernetes core component deployment, update, and configuration management, providing users with a one-stop solution. It supports customization of multiple container runtimes including crio, iSulad, docker, and containerd, and is compatible with multiple platform deployments, ensuring that users can easily handle various complex deployment requirements. In addition, NKD has the ability to create cluster self-signed certificates and supports deploying multiple versions of Kubernetes clusters, covering various scenarios that may be encountered in actual use. -#### Support Platforms - -NKD dynamically creates the required IaaS resources by connecting infrastructure providers according to the needs of the cluster, and currently supports OpenStack libvirt and Bare Metal platforms. - ## Software architecture For more information, see [Software Architecture](docs/en/overall_design.md) diff --git a/README.md b/README.md index 989f703740882b23748791aa8a28e8f2f15b45ee..27604dabd24d84a098b09cc3ad7032bd0a86dcc8 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,6 @@ ## 介绍 NKD（NestOS Kubernetes Deployer）是NestOS团队面向容器云场景开发的集群部署运维工具。涵盖了基础设施和Kubernetes核心组件的部署、更新和配置管理等一系列功能，为用户提供了一站式的解决方案。支持自定义多种容器运行时（包括crio、iSulad、docker和containerd），并且兼容多种平台部署，确保用户能够轻松应对各种复杂的部署需求。此外，NKD具备集群自签名证书创建的能力，并支持部署多种版本的Kubernetes集群，从而覆盖实际使用中可能遇到的各种场景。 -#### 支持平台 -NKD根据集群需求，连接基础设施提供商动态创建所需的IaaS资源，目前支持OpenStack、libvirt和裸金属平台。 - ## 软件架构 详细内容请见[软件架构说明](docs/zh/overall_design.md) diff --git a/docs/en/config_file_desc.md b/docs/en/config_file_desc.md index f6d905ce987e54103cb8fd22936766cf7eb9f97f..49e13e61986d0d53b0a4680d123b575f8232dbae 100644 --- a/docs/en/config_file_desc.md +++ b/docs/en/config_file_desc.md @@ -29,12 +29,15 @@ kubernetes: kubernetes-version: "v1.29.1" kubernetes-apiversion: "v1beta3" # support v1beta3、v1beta2、v1beta1 apiserver-endpoint: "192.168.132.11:6443" - image-registry: "registry.k8s.io" + image-registry: "registry.k8s.io" # The image repository address used during Kubeadm initialization + registryMirror: "" # The mirror site address of the image repository used when downloading the container image pause-image: "pause:3.9" release-image-url: "" token: "" # automatically generated by default adminkubeconfig: /etc/nkd/cluster/admin.config # path of admin.conf certificatekey: "" # The key used to decrypt the certificate in the downloaded Secret when adding a new control plane node + packageList: # List of RPM package names that need to be installed in the cluster environment + rpmPackagePath: "" # Path to the RPM package files that need to be installed in the cluster environment network: service-subnet: "10.96.0.0/16" pod-subnet: "10.244.0.0/16" diff --git a/docs/en/figures/ignition_design.jpg b/docs/en/figures/ignition_design.jpg deleted file mode 100644 index f5b15d19934411c6650f657285983782a0cf1317..0000000000000000000000000000000000000000 Binary files a/docs/en/figures/ignition_design.jpg and /dev/null differ diff --git a/docs/en/figures/ignition_design_1.jpg b/docs/en/figures/ignition_design_1.jpg deleted file mode 100644 index 984d52135301a1ae1537f7bd87988f4914305a8a..0000000000000000000000000000000000000000 Binary files a/docs/en/figures/ignition_design_1.jpg and /dev/null differ diff --git a/docs/en/figures/ignition_design_2.jpg b/docs/en/figures/ignition_design_2.jpg deleted file mode 100644 index 8526c63c0ac37027251888230845d1fb0ae2cda1..0000000000000000000000000000000000000000 Binary files a/docs/en/figures/ignition_design_2.jpg and /dev/null differ diff --git a/docs/en/figures/ignition_design_3.jpg b/docs/en/figures/ignition_design_3.jpg deleted file mode 100644 index 73ee548b5905e731c1145596043c12e4fd5a6f7b..0000000000000000000000000000000000000000 Binary files a/docs/en/figures/ignition_design_3.jpg and /dev/null differ diff --git a/docs/en/ignition_design.md b/docs/en/ignition_design.md index 878bca953318193b8f5334fac63d4e99812eb7d5..f9dcf8428c3ef6a8271b78f741afcb4dd6d58d44 100644 --- a/docs/en/ignition_design.md +++ b/docs/en/ignition_design.md @@ -1,23 +1,10 @@ -# Ignition +# Provide configuration +After the cluster infrastructure is successfully created, the operating system will enter the startup process. During this process, the ignition file will automatically obtain the required configuration information and write the configuration files such as user information, cluster certificates, cluster deployment services, etc. to the disk of the node machine. Subsequently, the systemd service will start to ensure that the system can run stably according to the preset rules and configurations. After the "release-image-pivot.service" service starts normally, it will build the required environment for cluster deployment. After the environment configuration is completed, the K8S cluster deployment service will officially start, which marks the official start of cluster deployment work. The entire process will continue until the service automatically shuts down after the cluster deployment is completed. NKD currently supports multiple ignition files including Ignition, cloud-init, and Kickstart files. -Ignition is a utility used by NestOS during the initramfs phase to manipulate disks. This includes tasks such as creating users, adding trusted SSH keys, writing files (regular files, systemd services, etc.), and configuring networks. Upon the first boot, Ignition reads its configuration and applies it. Ignition utilizes JSON configuration files to represent the set of changes to be made. The format of this configuration is detailed in the specification [here](https://coreos.github.io/ignition/specs/). +## Ignition +Ignition is a utility used by immutable infrastructure operating systems such as NestOS and Fedora CoreOS to operate disks during the initramfs phase, including creating users, adding trusted SSH keys, writing regular files, systemd services, network configurations, and more. Upon first boot, Ignition reads its configuration and applies it. Ignition uses a JSON configuration file to represent the set of changes to be made. The format of this configuration is detailed in the [specification](https://coreos.github.io/ignition/specs/). -## Provide configuration -The Ignition file (controlplane.ign) required to generate and deploy the cluster is approximately 90KB in size, containing declarations for systemd services, certificates, and other necessities for cluster create. When deploying the cluster on the OpenStack platform, the Nova user data limit is 64KB, making direct infrastructure deployment impossible. To address this issue, NKD has created a smaller Ignition file (*-merge.ign) to serve as a bootstrapping configuration file for creating the infrastructure. The main Ignition file will be loaded into memory accessible via HTTP service, enabling the smaller Ignition file to be automatically loaded during the system boot phase. - -When the infrastructure of the cluster nodes is successfully created, the operating system is in the bootstrapping phase. At this point, Ignition retrieves configuration information and writes configuration files (including users, cluster certificates, cluster deployment services, etc.) to the disks of the node machines, and then starts systemd services. Once the "release-image-pivot.service" service is started successfully, the node machines switch the file system to a customized version of NestOS based on the rpm-ostree mechanism. Afterward, the cluster creation service is executed, continuing until the completion of cluster creation, at which point the service is shut down. - -### ControlPlane Node -The configuration information of the control plane node's Ignition file is as shown in the image: -![ignition_design_1](/docs/en/figures/ignition_design_1.jpg) - -### Master Node -The configuration information of the master node's Ignition file is as shown in the image： -![ignition_design_2](/docs/en/figures/ignition_design_2.jpg) - -### Worker Node -The configuration information of the worker node's Ignition file is as shown in the image： -![ignition_design_3](/docs/en/figures/ignition_design_3.jpg) +The Ignition controlplane.ign file required for deploying a cluster generated by NKD is approximately 90KB, which declares the key components required during the cluster deployment process, including necessary systemd services and cluster certificates. However, when deploying a cluster on the OpenStack platform, due to the Nova user data limit of 64KB, it is not possible to directly create instances. To solve this problem, NKD created a stripped-down version of Ignition*-merge.ign file as a bootstrap configuration file when creating instances. At the same time, NKD loaded the main Ignition file controlplane.ign into a memory storage accessible through HTTP services. The generated Ignition file directory structure is as follows： ``` shell @@ -48,4 +35,15 @@ Where: "version": "3.2.0" } } -``` \ No newline at end of file +``` + +## cloud-init +Cloud-init is an open source tool developed specifically for initializing virtual machine instances in cloud computing environments. During cluster deployment using NKD, the underlying operating system is selected as a general-purpose operating system. When deploying a cluster on a virtualization platform, a cloud-init file is generated, which configures the environment and cluster deployment services required for cluster deployment, such as: + +- Configure the host name +- Install software packages on the instance +- Configure cluster environment +- Run the cluster installation script + +## Kickstart +During the cluster deployment process using NKD, when the underlying operating system is selected as a general-purpose operating system and the cluster deployment is performed based on the PXE pre-boot execution environment platform, a kickstart file will be generated, which configures the environment and cluster deployment services required for cluster deployment, enabling a self-service installation method. diff --git a/docs/en/manual.md b/docs/en/manual.md index d317e6c7c9661e65f3344a1c3811b64103f79370..ad6094d10743d48fc64ce8947028f7f36a441fe3 100644 --- a/docs/en/manual.md +++ b/docs/en/manual.md @@ -31,14 +31,13 @@ Deploying clusters on the libvirt platform requires pre-installation of the libv ### openstack Deploying clusters on the OpenStack platform requires pre-setup of the OpenStack environment. -### Bare Metal -When deploying a cluster on a bare metal platform, it is necessary to prepare the physical machines in advance. +### PXE ## Compilation and Installation * Compilation Environment: Linux x86_64/aarch64 * The following software packages are required for compilation: - * golang >= 1.17 + * golang >= 1.21 * git ``` shell $ sudo yum install golang git @@ -197,3 +196,7 @@ Note: Users need to customize building deployment images before deploying the cl ``` shell $ nkd deploy -f cluster_config.yaml ``` + +## troubleshooting + +The logs of NKD are stored in the directory /etc/nkd/logs by default, which facilitates effective troubleshooting in case of any issues encountered during the infrastructure creation process. \ No newline at end of file diff --git a/docs/zh/config_file_desc.md b/docs/zh/config_file_desc.md index 22540ce36ace047c7d2fe3731466ca10bec64f65..976c0629e4a7593342ba4459d3d8955708ac0669 100644 --- a/docs/zh/config_file_desc.md +++ b/docs/zh/config_file_desc.md @@ -30,12 +30,15 @@ kubernetes: # 集群相关配置列表 kubernetesVersion: "v1.29.1" # 部署集群的版本 kubernetesApiversion: "v1beta3" # 指定kubeadm配置文件格式的版本，目前支持 v1beta3、v1beta2、v1beta1 apiserverEndpoint: "192.168.132.11:6443" # 对外暴露的APISERVER服务的地址或域名 - imageRegistry: "registry.k8s.io" # 下载容器镜像时使用的镜像仓库的mirror站点地址 + imageRegistry: "registry.k8s.io" # Kubeadm初始化时使用的镜像仓库地址 + registryMirror: "" # 下载容器镜像时，使用的镜像仓库的 mirror 站点地址 pauseImage: "pause:3.9" # 容器运行时的pause容器的容器镜像名称 releaseImageUrl: "" # 包含K8S二进制组件的NestOS发布镜像的地址，支持架构x86_64或者aarch64 token: "" # 启动引导过程中使用的令牌，默认自动生成 adminKubeconfig: /etc/nkd/cluster/admin.config # 集群管理员配置文件admin.conf的路径 certificateKey: "" # 添加新的控制面节点时用来解密所下载的Secret中的证书的秘钥 + packageList: # 集群环境中需要安装的RPM软件包名称列表 + rpmPackagePath: "" # 集群环境中需要安装的RPM软件包文件路径 network: # k8s集群网络配置 serviceSubnet: "10.96.0.0/16" # k8s创建的service的IP地址网段 podSubnet: "10.244.0.0/16" # k8s集群网络的IP地址网段 diff --git a/docs/zh/figures/ignition_design_1.jpg b/docs/zh/figures/ignition_design_1.jpg deleted file mode 100644 index 68705cf56eb3740a790bf3264022ad04d5fde7c4..0000000000000000000000000000000000000000 Binary files a/docs/zh/figures/ignition_design_1.jpg and /dev/null differ diff --git a/docs/zh/figures/ignition_design_2.jpg b/docs/zh/figures/ignition_design_2.jpg deleted file mode 100644 index c8894ad2540f266d752b8afabcdc1c67811cd375..0000000000000000000000000000000000000000 Binary files a/docs/zh/figures/ignition_design_2.jpg and /dev/null differ diff --git a/docs/zh/figures/ignition_design_3.jpg b/docs/zh/figures/ignition_design_3.jpg deleted file mode 100644 index e2e157f4b626fab9b88bea9c4378dfa4154de5b8..0000000000000000000000000000000000000000 Binary files a/docs/zh/figures/ignition_design_3.jpg and /dev/null differ diff --git a/docs/zh/ignition_design.md b/docs/zh/ignition_design.md index f85edd00a400a04829b9287102928d415b3fab87..eb01fed23ecbf9664bdfa988e8cf7e0f7df626c0 100644 --- a/docs/zh/ignition_design.md +++ b/docs/zh/ignition_design.md @@ -1,23 +1,10 @@ -# Ignition +# 点火设计 +当集群基础设施顺利完成创建后，操作系统随即进入启动流程。在此过程中，点火文件会自动获取所需的配置信息，并将配置文件（如用户信息、集群证书、集群部署服务等）写入到节点机器的磁盘中。随后，systemd服务将启动，确保系统能够按照预设的规则和配置稳定运行。当"release-image-pivot.service"服务正常启动后，将会为集群部署搭建所需的环境。在环境配置完成后，K8S集群部署服务将正式启动，这标志着集群部署工作的正式开始。整个过程将持续进行，直到集群部署完成后服务自动关闭。NKD目前支持多种点火文件包括Ignition、cloud-init和Kickstart文件。 -Ignition是NestOS在initramfs期间用来操作磁盘的实用程序。其中包括创建用户、添加受信的SSH密钥、写入文件（常规文件、systemd服务...）、网络配置等。首次启动时，Ignition读取其配置并应用该配置。Ignition使用JSON配置文件来表示要进行的更改集。此配置的格式在[规范](https://coreos.github.io/ignition/specs/)中有详细说明。 +## Ignition +Ignition是不可变基础设施操作系统（如NestOS、Fedora CoreOS）在initramfs期间用来操作磁盘的实用程序，这包括创建用户、添加受信的SSH密钥、写入文件（常规文件、systemd服务...）、网络配置等。首次启动时，Ignition读取其配置并应用该配置。Ignition使用JSON配置文件来表示要进行的更改集。此配置的格式在[规范](https://coreos.github.io/ignition/specs/)中有详细说明。 -## 提供配置 -生成部署集群所需的Ignition（controlplane.ign）文件约为90KB，文件中声明了集群部署所需的systemd服务、证书等。在OpenStack平台上部署集群时，由于Nova用户数据限制为64KB，因此无法直接部署基础设施。为了解决这个问题，NKD创建了一个较小的Ignition（*-merge.ign）文件，作为创建基础设施时的引导配置文件。而主要的Ignition文件将会加载到可以通过 HTTP 服务访问的内存中，使较小的Ignition文件在系统引导阶自动加载主要的Ignition文件。 - -当集群节点的基础设施正常创建后，操作系统处于引导阶段，Ignition将获取配置信息并将配置文件（用户、集群证书、集群部署服务等）写入到节点机器的磁盘中，然后启动systemd服务。当"release-image-pivot.service"服务正常启动后，节点机器通过rpm-ostree机制将文件系统切换为基于NestOS的kubernetes定制版本，之后启动K8S集群部署服务，开始正式的集群部署任务，直到集群部署完成后服务关闭。 - -### ControlPlane Node -controlplane节点的Ignition文件配置信息如图: -![ignition_design_1](/docs/zh/figures/ignition_design_1.jpg) - -### Master Node -Master节点的Ignition文件配置信息如图： -![ignition_design_2](/docs/zh/figures/ignition_design_2.jpg) - -### Worker Node -Worker节点的Ignition文件配置信息如图： -![ignition_design_3](/docs/zh/figures/ignition_design_3.jpg) +NKD生成部署集群所需的Ignition（controlplane.ign）文件约为90KB，该文件声明了集群部署过程中所需的关键组件，包括必要的systemd服务和集群证书等。然而，当在OpenStack平台上部署集群时，由于Nova用户数据限制为64KB，因此无法直接创建实例。为了解决这个问题，NKD创建了一个精简版的Ignition（*-merge.ign）文件，作为创建实例时的引导配置文件。同时，NKD将主要的Ignition文件（controlplane.ign）加载到了一个可通过HTTP服务访问的内存存储中。 生成的Ignition文件目录结构如下： ``` shell @@ -48,4 +35,15 @@ $ tree "version": "3.2.0" } } -``` \ No newline at end of file +``` + +## cloud-init +cloud-init是专为云计算环境中虚拟机实例初始化而开发的一款开源工具。在使用NKD进行集群部署过程中，选择底层操作系统为通用操作系统（例如：openeuler），在虚拟化平台上进行集群部署时将生成cloud-init文件，该文件中配置了集群部署所需的环境和集群部署服务，例如： + +- 配置主机名 +- 在实例上安装软件包 +- 配置集群环境 +- 运行集群安装脚本 + +## Kickstart +在使用NKD进行集群部署的过程中，当选择底层操作系统为通用操作系统，并基于PXE（预启动执行环境）平台上进行集群部署时将生成kickstart文件，该文件中配置了集群部署所需的环境和集群部署服务，实现了一种无人值守的安装方式。 \ No newline at end of file diff --git a/docs/zh/manual.md b/docs/zh/manual.md index 4268449adbed41f1a4ef2b7465475c155dd2a5a2..09cf80ee91c8b133a4bdd821a162425639358498 100644 --- a/docs/zh/manual.md +++ b/docs/zh/manual.md @@ -31,14 +31,13 @@ ### OpenStack 在OpenStack平台部署集群时，需要提前搭建好OpenStack环境 -### 裸金属 -在裸金属平台部署集群时，需要提前准备物理机 +### PXE ## 编译安装 * 编译环境：Linux x86_64/aarch64 * 进行编译需要以下软件包： - * golang >= 1.17 + * golang >= 1.21 * git ``` shell $ sudo yum install golang git @@ -200,3 +199,7 @@ NKD部署集群过程中集群节点需要访问NKD提供的点火服务，通 ``` shell $ nkd deploy -f cluster_config.yaml ``` + +## 故障排查 + +NKD的日志默认存放在/etc/nkd/logs目录下，以便在基础设施创建过程中遇到问题能够有效地进行排查 \ No newline at end of file diff --git a/docs/zh/overall_design.md b/docs/zh/overall_design.md index bcc5ecb1c067a8e8db28c009fb18ed3d51413383..f660df002e20816d6db3e40ec39cb83401e2c431 100644 --- a/docs/zh/overall_design.md +++ b/docs/zh/overall_design.md @@ -37,7 +37,7 @@ NKD部署集群提供了不同的应用配置方式，以方便不同的用户 ![certmanager_design](/docs/zh/figures/certmanager_design.jpg) ### 点火模块设计 -在创建基础设施时，NKD 需要通过点火机制传入系统部署后所需的动态配置，以支持用户部署Kubernetes资源。点火机制能够将用户编写的配置文件转化为机器引导时的配置文件。对于不可变基础设施的操作系统，点火模块将生成Ignition文件。如果底层操作系统为通用操作系统，则在虚拟化平台部署集群时会生成cloudinit 文件，而在裸金属平台部署集群时会生成kickstart文件。详细内容见[设计文档](./ignition_design.md)。 +在创建基础设施时，NKD 需要通过点火机制传入系统部署后所需的动态配置，以支持用户部署Kubernetes资源。点火机制能够将用户编写的配置文件转化为机器引导时的配置文件。对于不可变基础设施的操作系统，点火模块将生成Ignition文件。如果底层操作系统为通用操作系统，则在虚拟化平台部署集群时会生成cloud-init 文件，而在裸金属平台部署集群时会生成kickstart文件。详细内容见[设计文档](./ignition_design.md)。 ### housekeeper模块设计 在集群部署阶段，用户可以选择是否部署housekeeper