diff --git a/docs/en/docs/A-Ops/deploying-aops.md b/docs/en/docs/A-Ops/deploying-aops.md index f7f2cb8d867ab915bf51bcdf7b002a172936a01e..52c7d048be6e9746263979034017be9534da479e 100644 --- a/docs/en/docs/A-Ops/deploying-aops.md +++ b/docs/en/docs/A-Ops/deploying-aops.md @@ -1,437 +1,718 @@ # Deploying A-Ops -## 1. Environment Requirements +## 1 Introduction to A-Ops -- Two hosts running on openEuler 22.09 +A-Ops is a service used to improve the overall security of hosts. It provides functions such as asset management, vulnerability management, and configuration source tracing to identify and manage information assets, monitor software vulnerabilities, and rectify system faults on hosts, ensuring stable and secure running of hosts. - These two hosts are used to deploy two modes of the check module: scheduler and executor. Other services, such as MySQL, Elasticsearch, and aops-manager, can be independently deployed on any host. To facilitate operations, deploy these services on host A. +The following table describes the modules related to the A-Ops service. -- More than 8 GB memory +| Module | Description | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| aops-ceres | Client of the A-Ops service.
Collects host data and manages other data collectors (such as gala-gopher).
Responds to the commands delivered by the management center and processes the requirements and operations of the management center. | +| aops-zeus | A-Ops basic application management center, which interacts with other modules. The default port number is 11111.
Provides the basic host management service externally, such as adding and deleting hosts and host groups. | +| aops-hermes | Provides a visualized operation interface for A-Ops to display data information to users, improving service usability. | +| aops-apollo | Vulnerability management module of A-Ops. The default port number is 11116.
Identifies clients, and periodically obtains security notices released by the openEuler community and updates them to the vulnerability database.
Detects vulnerabilities in the system and software by comparing the vulnerabilities with those in the vulnerability database. | +| aops-vulcanus | Basic tool library of A-Ops. **Except aops-ceres and aops-hermes, modules must be installed and used together with this module.** | +| aops-tools | Provides the basic environment deployment script and database table initialization. The script is available in the **/opt/aops/scripts** directory after A-Ops is installed. | +| gala-ragdoll | Configuration source tracing module of A-Ops.
Uses Git to monitor and record configuration file changes. The default port number is 11114. | +| dnf-hotpatch-plugin | DNF plug-in, which allows DNF to recognize hot patch information and provides hot patch scanning and application. | -## 2. Configuring the Deployment Environment +## 2 Environment Requirements -### Host A: +You are advised to use four hosts running openEuler 24.03 LTS for deployment. Use three as the server and one as the managed host managed by A-Ops. **Configure the update repository** ([Q6: update Repository Configuration](#q6-update-repository-configuration)). The deployment scheme is as follows: -Deploy the following A-Ops services on host A: aops-tools, aops-manager, aops-check, aops-web, aops-agent, and gala-gopher. +- Host A: For MySQL, Redis, and Elasticsearch deployment. It provides data service support. The recommended memory is more than 8 GB. +- Host B: For the A-Ops asset management service (zeus), frontend display, and complete service function support. The recommended memory is more than 6 GB. +- Host C: For the A-Ops configuration source tracing service (gala-ragdoll) and vulnerability management. The recommended memory is 4 GB or more. +- Host D: As an A-Ops client and is used as a host managed and monitored by A-Ops. (aops-ceres can be deployed on hosts that need to be managed.) -Deploy the following third-party services on host A: MySQL, Elasticsearch, ZooKeeper, Kafka, and Prometheus. +| Host | IP Address | Module | +| ------ | ----------- | ------------------------------------- | +| Host A | 192.168.1.1 | MySQL, Elasticsearch, Redis | +| Host B | 192.168.1.2 | aops-zeus, aops-hermes, aops-diana | +| Host C | 192.168.1.3 | aops-apollo, gala-ragdoll, aops-diana | +| Host D | 192.168.1.4 | aops-ceres, dnf-hotpatch-plugin | -The deployment procedure is as follows: +>Before deployment, disable the **firewall and SELinux** on each host. -#### 2.1 Disabling the Firewall +- Disable the firewall. -Disable the firewall on the local host. - -``` +```shell systemctl stop firewalld systemctl disable firewalld systemctl status firewalld +setenforce 0 + ``` -#### 2.2 Deploying aops-tools +- Disable SELinux. + +```shell +# Change the status of SELinux to disabled in /etc/selinux/config. -Install aops-tools. +vi /etc/selinux/config +SELINUX=disabled -``` -yum install aops-tools +# After changing the value, press ESC and enter :wq to save the modification. ``` -#### 2.3 Deploying Databases MySQL and Elasticsearch +Note: SELinux will be disabled after a reboot. -##### 2.3.1 Deploying MySQL +## 3. Server Deployment -Use the **aops-basedatabase** script installed during aops-tools installation to install MySQL. +### 3.1 Asset Management -``` -cd /opt/aops/aops_tools -./aops-basedatabase mysql -``` +To use the asset management function, you need to deploy the aops-zeus, aops-hermes, MySQL, and Redis services. + +#### 3.1.1 Node Information -Modify the MySQL configuration file. +| Host | IP Address | Module | +| ------ | ----------- | ------------------------------------- | +| Host A | 192.168.1.1 | MySQL, Redis | +| Host B | 192.168.1.2 | aops-zeus, aops-hermes | +#### 3.1.2 Deployment Procedure + +##### 3.1.2.1 Deploying MySQL + +- Install MySQL. + +```shell +yum install mysql-server ``` + +- Modify the MySQL configuration file. + +```bash vim /etc/my.cnf ``` -Add **bind-address** and set it to the IP address of the local host. +- Add **bind-address** and set it to the IP address of the local host in the **mysqld** section. -![1662346986112](./figures/修改mysql配置文件.png) +```ini +[mysqld] +bind-address=192.168.1.1 +``` -Restart the MySQL service. +- Restart the MySQL service. -``` +```bash systemctl restart mysqld ``` -Connect to the database and set the permission. +- Set the MySQL database access permission for the **root** user. + +```mysql +$ mysql + +mysql> show databases; +mysql> use mysql; +mysql> select user,host from user; -- If the value of host is localhost, only the local host can connect to the MySQL database. The external network and local software client cannot connect to the MySQL database. ++---------------+-----------+ +| user | host | ++---------------+-----------+ +| root | localhost | +| mysql.session | localhost | +| mysql.sys | localhost | ++---------------+-----------+ +3 rows in set (0.00 sec) ``` -mysql -show databases; -use mysql; -select user,host from user;// If the value of user is root and the value of host is localhost, the MySQL database can be connected only by the local host but cannot be connected from the external network and by the local software client. -update user set host = '%' where user='root'; -flush privileges;// Refresh the permission. -exit + +```mysql +mysql> update user set host = '%' where user='root'; -- Allow the access of the root user using any IP address. +mysql> flush privileges; -- Refresh the permissions. +mysql> exit ``` -##### 2.3.2 Deploying Elasticsearch +##### 3.1.2.2 Deploying Redis -Use the **aops-basedatabase** script installed during aops-tools installation to install Elasticsearch. +- Install Redis. +```shell +yum install redis -y ``` -cd /opt/aops/aops_tools -./aops-basedatabase elasticsearch + +- Modify the Redis configuration file. + +```shell +vim /etc/redis.conf +``` + +Bind IP addresses. + +```ini +# It is possible to listen to just one or multiple selected interfaces using +# the "bind" configuration directive, followed by one or more IP addresses. +# +# Examples: +# +# bind 192.168.1.100 10.0.0.1 +# bind 127.0.0.1 ::1 +# +# ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the +# internet, binding to all the interfaces is dangerous and will expose the +# instance to everybody on the internet. So by default we uncomment the +# following bind directive, that will force Redis to listen only into +# the IPv4 lookback interface address (this means Redis will be able to +# accept connections only from clients running into the same computer it +# is running). +# +# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES +# JUST COMMENT THE FOLLOWING LINE. +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +bind 127.0.0.1 192.168.1.1# Add the actual IP address of host A. +``` + +- Start the Redis service. + +```shell +systemctl start redis ``` -Modify the configuration file. +##### 3.1.2.3 Deploying Prometheus -Modify the Elasticsearch configuration file. +- Install Prometheus. +```shell +yum install prometheus2 -y ``` -vim /etc/elasticsearch/elasticsearch.yml + +- Modify the Prometheus configuration file. + +```shell +vim /etc/prometheus/prometheus.yml ``` -![1662370718890](./figures/elasticsearch配置2.png) +- Add the gala-gopher IP addresses of the managed client to the monitored targets of Prometheus. -![1662370575036](./figures/elasticsearch配置1.png) +> In this document, host D is the client. Add the gala-gopher address of host D. -![1662370776219](./figures/elasticsearch3.png) +- Modify the **targets** configuration item. -Restart the Elasticsearch service. +```yaml +# A scrape configuration containing exactly one endpoint to scrape: +# Here it's Prometheus itself. +scrape_configs: + # The job name is added as a label `job=` to any timeseries scraped from this config. + - job_name: 'prometheus' + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: ['localhost:9090', '192.168.1.4:8888'] ``` -systemctl restart elasticsearch + +Start the Prometheus service. + +```shell +systemctl start prometheus ``` -#### 2.4 Deploying aops-manager +##### 3.1.2.4 Deploying aops-zeus -Install aops-manager. +- Install aops-zeus. -``` -yum install aops-manager +```shell +yum install aops-zeus -y ``` -Modify the configuration file. +- Modify the configuration file. -``` -vim /etc/aops/manager.ini +```shell +vim /etc/aops/zeus.ini ``` -Change the IP address of each service in the configuration file to the actual IP address. Because all services are deployed on host A, you need to set their IP addresses to the actual IP address of host A. +- Change the IP address of each service in the configuration file to the actual IP address. In this document, aops-zeus is deployed on host B. Therefore, you need to set the IP address to the IP address of host B. -``` -[manager] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +```ini +[zeus] +ip=192.168.1.2 // Change the IP address to the actual IP address of host B. port=11111 -host_vault_dir=/opt/aops -host_vars=/opt/aops/host_vars [uwsgi] wsgi-file=manage.py -daemonize=/var/log/aops/uwsgi/manager.log +daemonize=/var/log/aops/uwsgi/zeus.log http-timeout=600 harakiri=600 - -[elasticsearch] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. -port=9200 -max_es_query_num=10000000 +processes=2 // Generate a specified number of workers or processes. +gevent=100 // Number of gevent asynchronous cores [mysql] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. port=3306 database_name=aops engine_format=mysql+pymysql://@%s:%s/%s -pool_size=10000 +pool_size=100 pool_recycle=7200 -[aops_check] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. -port=11112 -``` +[agent] +default_instance_port=8888 -Start the aops-manager service. +[redis] +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. +port=6379 +[apollo] +ip=192.168.1.3 // Change the IP address to the actual IP address of the apollo service deployment. It is recommended that apollo and zeus be deployed separately. This section is not required if apollo is not used. +port=11116 ``` -systemctl start aops-manager + +> **Set the MySQL database mode to password mode**. For details, see [Q5: MySQL Password Mode](#q5-mysql-password-mode) + +- Start the aops-zeus service. + +```shell +systemctl start aops-zeus ``` -#### 2.5 Deploying aops-web +**Note: [Initialize the aops-zeus database](#3125-initializing-the-aops-zeus-database) before starting the service.** -Install aops-web. +> If the zeus service fails to be started and the error message indicates that the MySQL database cannot be connected, check if a MySQL password is set. If yes, see [Q5: MySQL Password Mode](#q5-mysql-password-mode). -``` -yum install aops-web -``` +#### 3.1.2.5 Initializing the aops-zeus Database -Modify the configuration file. Because all services are deployed on host A, set the IP address of each service accessed by aops-web to the actual IP address of host A. +- Initialize the database. -``` -vim /etc/nginx/aops-nginx.conf +```shell +cd /opt/aops/scripts/deploy +bash aops-basedatabase.sh init zeus ``` -The following figure shows the configuration of some services. +**Note: If aops-tools is not installed, run the SQL script to initialize. The script path is /opt/aops/database/zeus.sql** -![1662378186528](./figures/配置web.png) +[Q5: MySQL Password Mode](#q5-mysql-password-mode) -Enable the aops-web service. +[Q7: Nonexisting /opt/aops/scripts/deploy](#q7-nonexisting-optaopsscriptsdeploy) -``` -systemctl start aops-web -``` +##### 3.1.2.6 Deploying aops-hermes -#### 2.6 Deploying Kafka +- Install aops-hermes. -##### 2.6.1 Deploying ZooKeeper +```shell +yum install aops-hermes -y +``` -Install ZooKeeper. +- Modify the configuration file. -``` -yum install zookeeper +```shell +vim /etc/nginx/aops-nginx.conf ``` -Start the ZooKeeper service. +- Some service configurations: + + > As the services are deployed on host B, configure the Nginx proxy to set the services addresses to the actual IP address of host B. + +```ini + # Ensure that Nginx still uses index.html as the entry when the front-end route changes. + location / { + try_files $uri $uri/ /index.html; + if (!-e $request_filename){ + rewrite ^(.*)$ /index.html last; + } + } + # Change it to the actual IP address of the host where aops-zeus is deployed. + location /api/ { + proxy_pass http://192.168.1.2:11111/; + } + # Enter the IP address of gala-ragdoll. IP addresses that involve port 11114 need to be configured. + location /api/domain { + proxy_pass http://192.168.1.3:11114/; + rewrite ^/api/(.*) /$1 break; + } + # Enter the IP address of gala-apollo. + location /api/vulnerability { + proxy_pass http://192.168.1.3:11116/; + rewrite ^/api/(.*) /$1 break; + } +``` + +- Enable the aops-hermes service. -``` -systemctl start zookeeper +```shell +systemctl start aops-hermes ``` -##### 2.6.2 Deploying Kafka +### 3.2 Vulnerability Management -Install Kafka. +The CVE management module is implemented based on the [asset management](#31-asset-management) module. Therefore, you need to [deploy the module](#312-deployment-procedure) before deploying aops-apollo. -``` -yum install kafka -``` +The running of the aops-apollo service requires the support of the **MySQL, Elasticsearch, and Redis** databases. -Modify the configuration file. +#### 3.2.1 Node Information -``` -vim /opt/kafka/config/server.properties -``` +| Host | IP Address | Module | +| ------ | ----------- | ------------- | +| Host A | 192.168.1.1 | Elasticsearch | +| Host C | 192.168.1.3 | aops-apollo | -Change the value of **listeners** to the IP address of the local host. +#### 3.2.2 Deployment Procedure -![1662381371927](./figures/kafka配置.png) +See [Asset Management](#312-deployment-procedure). -Start the Kafka service. +##### 3.2.2.1 Deploying Elasticsearch -``` -cd /opt/kafka/bin -nohup ./kafka-server-start.sh ../config/server.properties & -tail -f ./nohup.out # Check all the outputs of nohup. If the IP address of host A and the Kafka startup success INFO are displayed, Kafka is started successfully. -``` +- Configure the repository for Elasticsearch. -#### 2.7 Deploying aops-check +```shell +echo "[aops_elasticsearch] +name=Elasticsearch repository for 7.x packages +baseurl=https://artifacts.elastic.co/packages/7.x/yum +gpgcheck=1 +gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch +enabled=1 +autorefresh=1 +type=rpm-md" > "/etc/yum.repos.d/aops_elasticsearch.repo" +``` -Install aops-check. +- Install Elasticsearch. +```shell +yum install elasticsearch-7.14.0-1 -y ``` -yum install aops-check + +- Modify the Elasticsearch configuration file. + +```shell +vim /etc/elasticsearch/elasticsearch.yml ``` -Modify the configuration file. +```yml +# ------------------------------------ Node ------------------------------------ +# +# Use a descriptive name for the node: +# +node.name: node-1 +``` + +```yml +# ---------------------------------- Network ----------------------------------- +# +# By default Elasticsearch is only accessible on localhost. Set a different +# address here to expose this node on the network: +# +# Change the value to the actual IP address of host A. +network.host: 192.168.1.1 +# +# By default Elasticsearch listens for HTTP traffic on the first free port it +# finds starting at 9200. Set a specific HTTP port here: +# +http.port: 9200 +# +# For more information, consult the network module documentation. +# +``` + +```yml +# --------------------------------- Discovery ---------------------------------- +# +# Pass an initial list of hosts to perform discovery when this node is started: +# The default list of hosts is ["127.0.0.1", "[::1]"] +# +#discovery.seed_hosts: ["host1", "host2"] +# +# Bootstrap the cluster using an initial set of master-eligible nodes: +# +cluster.initial_master_nodes: ["node-1"] +# Cross-domain configurations +http.cors.enabled: true +http.cors.allow-origin: "*" +# +``` + +- Restart the Elasticsearch service. +```shell +systemctl restart elasticsearch ``` -vim /etc/aops/check.ini + +##### 3.2.2.2 Deploying aops-apollo + +- Install aops-apollo. + +```shell +yum install aops-apollo ``` -Change the IP address of each service in the configuration file to the actual IP address. Because all services are deployed on host A, you need to set their IP addresses to the actual IP address of host A. +- Modify the configuration file. +```shell +vim /etc/aops/apollo.ini ``` -[check] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. -port=11112 -mode=configurable // Configurable mode, which means aops-check is used as the scheduler in common diagnosis mode. -timing_check=on -[default_mode] -period=30 -step=30 +- Change the IP address of each service in the **apollo.ini** to the actual IP address. -[elasticsearch] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. -port=9200 +```ini +[apollo] +ip=192.168.1.3// Change it to the actual IP address of host C. +port=11116 +host_vault_dir=/opt/aops +host_vars=/opt/aops/host_vars + +[zeus] +ip=192.168.1.2 // Change it to the actual IP address of host B. +port=11111 + +# hermes info is used to send mail. +[hermes] +ip=192.168.1.2 // Change it to the actual IP address of aops-hermes, for example, the IP address of host B. +port=80 // Change it to the actual port of the hermes service. + +[cve] +cve_fix_function=yum +# value between 0-23, for example, 2 means 2:00 in a day. +cve_scan_time=2 [mysql] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +ip=192.168.1.1 // Change it to the actual IP address of host A. port=3306 database_name=aops engine_format=mysql+pymysql://@%s:%s/%s -pool_size=10000 +pool_size=100 pool_recycle=7200 -[prometheus] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. -port=9090 -query_range_step=15s - -[agent] -default_instance_port=8888 +[elasticsearch] +ip=192.168.1.1 // Change it to the actual IP address of host A. +port=9200 +max_es_query_num=10000000 -[manager] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. -port=11111 +[redis] +ip=192.168.1.1 // Change it to the actual IP address of host A. +port=6379 -[consumer] -kafka_server_list=192.168.1.1:9092 // Change the service IP address to the actual IP address of host A. -enable_auto_commit=False -auto_offset_reset=earliest -timeout_ms=5 -max_records=3 -task_name=CHECK_TASK -task_group_id=CHECK_TASK_GROUP_ID -result_name=CHECK_RESULT -[producer] -kafka_server_list = 192.168.1.1:9092 // Change the service IP address to the actual IP address of host A. -api_version = 0.11.5 -acks = 1 -retries = 3 -retry_backoff_ms = 100 -task_name=CHECK_TASK -task_group_id=CHECK_TASK_GROUP_ID +[uwsgi] +wsgi-file=manage.py +daemonize=/var/log/aops/uwsgi/apollo.log +http-timeout=600 +harakiri=600 +processes=2 +gevent=100 ``` -Start the aops-check service in configurable mode. +> **Set the MySQL database to the password mode**. For details, see [Q5: MySQL Password Mode](#q5-mysql-password-mode). -``` -systemctl start aops-check -``` +- Start the aops-apollo service. -#### 2.8 Deploying the Client Services +```shell +systemctl start aops-apollo +``` -aops-agent and gala-gopher must be deployed on the client. For details, see the [Deploying aops-agent](deploying-aops-agent.md). +**Note: [Initialize the aops-apollo database](#3223-initializing-the-aops-apollo-database) before starting the service.** -Note: Before registering a host, you need to add a host group to ensure that the host group to which the host belongs exists. In this example, only host A is deployed and managed. +> If the apollo service fails to be started and the error message indicates that the MySQL database cannot be connected, check if a MySQL password is set. If yes, see [Q5: MySQL Password Mode](#q5-mysql-password-mode). -#### 2.9 Deploying Prometheus +#### 3.2.2.3 Initializing the aops-apollo Database -Install Prometheus. +- Initialize the apollo database. -``` -yum install prometheus2 +```shell +cd /opt/aops/scripts/deploy +bash aops-basedatabase.sh init apollo ``` -Modify the configuration file. +**Note: If aops-tools is not installed, run the SQL script to initialize. The script path is /opt/aops/database/apollo.sql** -``` -vim /etc/prometheus/prometheus.yml -``` +[Q5: MySQL Password Mode](#q5-mysql-password-mode) -Add the gala-gopher addresses of all clients to the monitoring host of Prometheus. +[FAQs: Nonexisting /opt/aops/scripts/deploy](#q7-nonexisting-optaopsscriptsdeploy) -![1662377261742](./figures/prometheus配置.png) +### 3.3 Configuring Source Tracing -Start the Prometheus service: +A-Ops configuration source tracing depends on gala-ragdoll. Therefore, you need to complete the deployment of [Asset Management](#31-asset-management) and then deploy gala-ragdoll. -``` -systemctl start prometheus -``` +#### 3.3.1 Node Information + +| Host | IP Address | Module | +| ------ | ----------- | ------------ | +| Host C | 192.168.1.3 | aops-ragdoll | + +#### 3.3.2 Deployment Procedure -#### 2.10 Deploying gala-ragdoll +See [Asset Management](#31-asset-management). -The configuration source tracing function of A-Ops depends on gala-ragdoll. Git is used to monitor configuration file changes. +##### 3.3.2.1 Deploying gala-ragdoll -Install gala-ragdoll. +- Install gala-ragdoll. ```shell -yum install gala-ragdoll # A-Ops configuration source tracing +yum install gala-ragdoll python3-gala-ragdoll -y ``` -Modify the configuration file. +- Modify the configuration file. ```shell vim /etc/ragdoll/gala-ragdoll.conf ``` -Change the IP address in **collect_address** of the **collect** section to the IP address of host A, and change the values of **collect_api** and **collect_port** to the actual API and port number. +> **Change the IP address in collect_address of the collect section to the IP address of host B, and change the values of collect_api and collect_port to the actual API and port number.** -``` +```ini [git] git_dir = "/home/confTraceTest" user_name = "user_name" user_email = "user_email" [collect] -collect_address = "http://192.168.1.1" // Change the IP address to the actual IP address of host A. -collect_api = "/manage/config/collect" // Change the API to the actual API for collecting configuration files. -collect_port = 11111 // Change the port number to the actual port number of the service. +collect_address = "http://192.168.1.2" // Change it to the actual IP address of host B. +collect_api = "/manage/config/collect" // The value is an example. Change it to the actual value. +collect_port = 11111 // Change it to the actual port number of the aops-zeus service. [sync] -sync_address = "http://0.0.0.0" -sync_api = "/demo/syncConf" -sync_port = 11114 +sync_address = "http://192.168.1.2" +sync_api = "/manage/config/sync" // The value is an example. Change it to the actual value. +sync_port = 11111 +[objectFile] +object_file_address = "http://192.168.1.2" +object_file_api = "/manage/config/objectfile" // The value is an example. Change it to the actual value. +object_file_port = 11111 [ragdoll] port = 11114 - ``` -Start the gala-ragdoll service. +- Start the gala-ragdoll service. ```shell systemctl start gala-ragdoll ``` -### Host B: +## 3.4 Exception Detection + +The exception detection function is implemented based on the aops-zeus service. Therefore, you need to deploy aops-zeus and then aops-diana. + +Considering distributed deployment, the aops-diana service must be deployed on both host B and host C to act as the producer and consumer in the message queue, respectively. + +The running of the aops-diana service requires the support of MySQL, Elasticsearch, Kafka, and Prometheus. + +### 3.4.1 Node Information + +| Host | IP Address | Module | +| ------ | ----------- | ---------- | +| Host A | 192.168.1.1 | Kafka | +| Host B | 192.168.1.2 | aops-diana | +| Host C | 192.168.1.3 | aops-diana | -Only aops-check needs to be deployed on host B as the executor. +### 3.4.2 Deployment Procedure -#### 2.11 Deploying aops-check +[Asset Management](#312-deployment-procedure) -Install aops-check. +[Deploying Elasticsearch](#3221-deploying-elasticsearch) +#### 3.4.2.1 Deploying Kafka + +Kafka uses ZooKeeper to manage and coordinate agents. Therefore, you need to deploy ZooKeeper when deploying Kafka. + +- Install ZooKeeper. + +```shell +yum install zookeeper -y ``` -yum install aops-check + +- Start the ZooKeeper service. + +```shell +systemctl start zookeeper ``` -Modify the configuration file. +- Install Kafka. +```shell +yum install kafka -y ``` -vim /etc/aops/check.ini + +- Modify the Kafka configuration file. + +```shell +vim /opt/kafka/config/server.properties ``` -Change the IP address of each service in the configuration file to the actual IP address. Change the IP address of the aops-check service deployed on host B to the IP address of host B. Because other services are deployed on host A, change the IP addresses of those services to the IP address of host A. +Change the value of **listeners** to the IP address of the local host. + +```yaml +############################# Socket Server Settings ############################# +# The address the socket server listens on. It will get the value returned from +# java.net.InetAddress.getCanonicalHostName() if not configured. +# FORMAT: +# listeners = listener_name://host_name:port +# EXAMPLE: +# listeners = PLAINTEXT://your.host.name:9092 +listeners=PLAINTEXT://192.168.1.1:9092 ``` -[check] -ip=192.168.1.2 // Change the IP address to the actual IP address of host B. + +- Start the Kafka service. + +```shell +cd /opt/kafka/bin +nohup ./kafka-server-start.sh ../config/server.properties & + +# Check all the outputs of nohup. If the IP address of host A and the Kafka startup success INFO are displayed, Kafka is started successfully. +tail -f ./nohup.out +``` + +#### 3.4.2.2 Deploying diana + +- Install aops-diana. + +```shell +yum install aops-diana +``` + +Modify the configuration file. +> The aops-dianas on host B and host C play different roles, which are **distinguished based on the differences in the configuration file**. + +```shell +vim /etc/aops/diana.ini +``` + +(1) Start aops-diana on host C in executor mode. It functions as the consumer in the Kafka message queue. The configuration file to be modified is as follows: + +```ini +[diana] +ip=192.168.1.3 // Change the IP address to the actual IP address of host C. port=11112 -mode=executor // Executor mode, which means aops-check is used as the executor in normal diagnosis mode. +mode=executor // This mode is the executor mode. It is used as the executor in common diagnosis mode and functions as the consumer in Kafka. timing_check=on [default_mode] -period=30 -step=30 +period=60 +step=60 [elasticsearch] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. port=9200 +max_es_query_num=10000000 + [mysql] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. port=3306 database_name=aops engine_format=mysql+pymysql://@%s:%s/%s pool_size=10000 pool_recycle=7200 +[redis] +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. +port=6379 + + [prometheus] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. port=9090 query_range_step=15s [agent] default_instance_port=8888 -[manager] -ip=192.168.1.1 // Change the service IP address to the actual IP address of host A. +[zeus] +ip=192.168.1.2 // Change the IP address to the actual IP address of host B. port=11111 [consumer] -kafka_server_list=192.168.1.1:9092 // Change the service IP address to the actual IP address of host A. +kafka_server_list=192.168.1.1:9092 // Change the IP address to the actual IP address of host C. enable_auto_commit=False auto_offset_reset=earliest timeout_ms=5 @@ -439,22 +720,224 @@ max_records=3 task_name=CHECK_TASK task_group_id=CHECK_TASK_GROUP_ID result_name=CHECK_RESULT + [producer] -kafka_server_list = 192.168.1.1:9092 // Change the service IP address to the actual IP address of host A. +kafka_server_list = 192.168.1.1:9092 // Change the IP address to the actual IP address of host C. api_version = 0.11.5 acks = 1 retries = 3 retry_backoff_ms = 100 task_name=CHECK_TASK task_group_id=CHECK_TASK_GROUP_ID + +[uwsgi] +wsgi-file=manage.py +daemonize=/var/log/aops/uwsgi/diana.log +http-timeout=600 +harakiri=600 +processes=2 +threads=2 ``` -Start the aops-check service in executor mode. +> **Set the MySQL database to the password mode**. For details, see [Q5: MySQL Password Mode](#q5-mysql-password-mode). + +(2) Start aops-diana on host B in configurable mode. It functions as the producer in the Kafka message queue. The aops-diana port configuration in the aops-hermes file is subject to the IP address and port number of this host. The configuration file to be modified is as follows: + +```ini +[diana] +ip=192.168.1.2 // Change the IP address to the actual IP address of host B. +port=11112 +mode=configurable // This mode is the configurable mode. It is used as a scheduler in common diagnosis mode and functions as the producer. +timing_check=on + +[default_mode] +period=60 +step=60 + +[elasticsearch] +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. +port=9200 +max_es_query_num=10000000 + +[mysql] +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. +port=3306 +database_name=aops +engine_format=mysql+pymysql://@%s:%s/%s +pool_size=100 +pool_recycle=7200 + +[redis] +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. +port=6379 + +[prometheus] +ip=192.168.1.1 // Change the IP address to the actual IP address of host A. +port=9090 +query_range_step=15s + +[agent] +default_instance_port=8888 + +[zeus] +ip=192.168.1.2 // Change the IP address to the actual IP address of host B. +port=11111 +[consumer] +kafka_server_list=192.168.1.1:9092 // Change the IP address to the actual IP address of host A. +enable_auto_commit=False +auto_offset_reset=earliest +timeout_ms=5 +max_records=3 +task_name=CHECK_TASK +task_group_id=CHECK_TASK_GROUP_ID +result_name=CHECK_RESULT + +[producer] +kafka_server_list = 192.168.1.1:9092 // Change the IP address to the actual IP address of host A. +api_version = 0.11.5 +acks = 1 +retries = 3 +retry_backoff_ms = 100 +task_name=CHECK_TASK +task_group_id=CHECK_TASK_GROUP_ID + +[uwsgi] +wsgi-file=manage.py +daemonize=/var/log/aops/uwsgi/diana.log +http-timeout=600 +harakiri=600 +processes=2 +threads=2 +``` + +> **Set the MySQL database to the password mode**. For details, see [Q5: MySQL Password Mode](#q5-mysql-password-mode). + +Start the aops-diana service. + +```shell +systemctl start aops-diana +``` + +**Note: [Initialize the aops-diana database](#3423-initializing-the-aops-diana-database) before starting the service.** + +> If the diana service fails to be started and the error message indicates that the MySQL database cannot be connected, check if a MySQL password is set. If yes, see [Q5: MySQL Password Mode](#q5-mysql-password-mode). + +#### 3.4.2.3 Initializing the aops-diana Database + +- Initialize the diana database. + +```shell +cd /opt/aops/scripts/deploy +bash aops-basedatabase.sh init diana ``` -systemctl start aops-check + +**Note:If aops-tools is not installed, run the SQL script to initialize. The script path is /opt/aops/database/diana.sql** + +[Q5: MySQL Password Mode](#q5-mysql-password-mode) + +[FAQs: Nonexisting /opt/aops/scripts/deploy](#q7-nonexisting-optaopsscriptsdeploy) + +## 3.5 Client Installation + +aops-ceres functions as the client of A-Ops. It communicates with the A-Ops management center through SSH and provides functions such as host information collection and command response. + +### 3.5.1 Node Information + +| Host | IP Address | Module | +| ------ | ----------- | ---------- | +| Host D | 192.168.1.4 | aops-ceres | + +### 3.5.2 Client Deployment + +```shell +yum install aops-ceres dnf-hotpatch-plugin -y ``` +## FAQs + +### Q1: Max Number of Connections + +When host interfaces are added in batches, due to the max number of SSH connections (**MaxStartups**) of the host where aops-zeus is installed, some hosts may fail to be connected. You can temporarily increase **MaxStartups** as required. For details, see the [SSH documentation](https://www.man7.org/linux/man-pages/man5/sshd_config.5.html). + +### Q2: 504 Gateway Timeout + +Some HTTP interfaces may take a long time to execute, resulting in error 504 on the web client. You can reduce the probability of error 504 by adding **proxy_read_timeout** to the Nginx configuration or increase its value. +### Q3: Firewall -The service deployment on the two hosts is complete. +If firewall cannot be disabled, open the ports involved in service deployment on the firewall. Otherwise, services may be inaccessible and A-Ops cannot be used properly. + +### Q4: Elasticsearch Access Denied + +If Elasticsearch is deployed on multiple nodes in a distributed manner, set the cross-domain access configurations properly to enable the access of the nodes. + +### Q5: MySQL Password Mode + +- **Configure the mysql section in the service configuration.** + +To set the password mode for the MySQL database connection (for example, the user is **root**, and the password is **123456**), change the value of **engine_format** in the **\[mysql]** section in apollo and zeus configurations. + +```ini +[mysql] +engine_format=mysql+pymysql://root:123456@%s:%s/%s +``` + +- **Modify the aops-basedatabase.sh initialization script.** + +Modify the 145th line of **aops-basedatabase.sh**. + +> Before modification: + +```shell +database = pymysql.connect(host='$mysql_ip', port=$port, database='mysql', autocommit=True,client_flag=CLIENT.MULTI_STAT EMENTS) +``` + +> After modification: + +```shell +database = pymysql.connect(host='$mysql_ip', port=$port, database='mysql', password='password', user='user', autocommit=True, client_flag=CLIENT.MULTI_STATEMENTS) +``` + +- **Database connection error upon service startup** + +Modify the 178th line in **/usr/bin/aops-vulcanus**. + +> Before modification: + +```shell +connect = pymysql.connect(host='$mysql_ip', port=$port, database='$aops_database') +``` + +> After modification: + +```shell +connect = pymysql.connect(host='$mysql_ip', port=$port, database='$aops_database', password='password', user='user') +``` + +**Note: If a non-root user is used for logging in to the server, add user ="root" or a user allowed by MySQL.** + +### Q6: update Repository Configuration + +```shell +echo "[update] +name=update +baseurl=http://repo.openeuler.org/openEuler-24.03-LTS/update/$basearch/ +enabled=1 +gpgcheck=0 +[update-epol] +name=update-epol +baseurl=http://repo.openeuler.org/openEuler-24.03-LTS/EPOL/update/main/$basearch/ +enabled=1 +gpgcheck=0" > /etc/yum.repos.d/openEuler-update.repo +``` + +> Note: Change **openEuler-24.03-LTS** to the actual OS version. You can also refer to the repository description in the openEuler official documentation. + +### Q7: Nonexisting /opt/aops/scripts/deploy + +During database initialization, if **/opt/aops/scripts/deploy** does not exits, install the aops-tools package. + +```shell +yum install aops-tools -y +``` diff --git a/docs/en/docs/A-Ops/figures/icon-note.gif b/docs/en/docs/A-Ops/figures/icon-note.gif new file mode 100644 index 0000000000000000000000000000000000000000..6314297e45c1de184204098efd4814d6dc8b1cda Binary files /dev/null and b/docs/en/docs/A-Ops/figures/icon-note.gif differ diff --git a/docs/en/docs/A-Ops/using-gala-anteater.md b/docs/en/docs/A-Ops/using-gala-anteater.md index 92907378c87b4d1b338584bc598db0763aadb5bf..44d54033bfe78eedc4e91d290635e40444570fbb 100644 --- a/docs/en/docs/A-Ops/using-gala-anteater.md +++ b/docs/en/docs/A-Ops/using-gala-anteater.md @@ -1,157 +1,220 @@ # Using gala-anteater -gala-anteater is an AI-based operating system exception detection platform. It provides functions such as time series data preprocessing, exception detection, and exception reporting. Based on offline pre-training, online model incremental learning and model update, it can be well adapted to multi-dimensional and multi-modal data fault diagnosis. +gala-anteater is an AI-based OS exception detection platform. It provides functions such as time series data preprocessing, exception detection, and exception reporting. Based on offline pre-training, online model incremental learning and model update, it can be well adapted to multi-dimensional and multi-modal data fault diagnosis. This chapter describes how to deploy and use the gala-anteater service. -#### Installation +## Installation -Mount the repo sources. +Mount the repositories. -```basic -[oe-2209] # openEuler 22.09 officially released repository -name=oe2209 -baseurl=http://119.3.219.20:82/openEuler:/22.09/standard_x86_64 +```ini +[everything] +name=everything +baseurl=http://121.36.84.172/dailybuild/EBS-openEuler-24.03-LTS-SP1/rc4_openeuler-2024-12-05-15-40-49/everything/$basearch/ enabled=1 gpgcheck=0 priority=1 -[oe-2209:Epol] # openEuler 22.09: Epol officially released repository -name=oe2209_epol -baseurl=http://119.3.219.20:82/openEuler:/22.09:/Epol/standard_x86_64/ +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/EBS-openEuler-24.03-LTS-SP1/EPOL/main/$basearch/ enabled=1 gpgcheck=0 priority=1 + ``` Install gala-anteater. ```bash -# yum install gala-anteater +yum install gala-anteater ``` +## Configuration + +>![](./figures/icon-note.gif)**Note:** +> +> gala-anteater uses a configuration file to set parameters during startup. The configuration file is located at **/etc/gala-anteater/config/gala-anteater.yaml**. + +### Default Configuration Parameters + +```yaml +Global: + data_source: "prometheus" + +Arangodb: + url: "http://localhost:8529" + db_name: "spider" + +Kafka: + server: "192.168.122.100" + port: "9092" + model_topic: "gala_anteater_hybrid_model" + rca_topic: "gala_cause_inference" + meta_topic: "gala_gopher_metadata" + group_id: "gala_anteater_kafka" + # auth_type: plaintext/sasl_plaintext, please set "" for no auth + auth_type: "" + username: "" + password: "" + +Prometheus: + server: "localhost" + port: "9090" + steps: "5" + +Aom: + base_url: "" + project_id: "" + auth_type: "token" + auth_info: + iam_server: "" + iam_domain: "" + iam_user_name: "" + iam_password: "" + ssl_verify: 0 + +Schedule: + duration: 1 + +Suppression: + interval: 10 +``` +| Parameter | Description | Default Value | +| ----------- | ------------------------------------------------------------------------------------------- | ---------------------------- | +| Global | Global configuration | Dictionary type | +| data_source | Data source | "prometheus" | +| Arangodb | ArangoDB graph database configuration | Dictionary type | +| url | Graph database IP address | "" | +| db_name | Graph database name | "spider" | +| Kafka | Kafka configuration | Dictionary type | +| server | Kafka server IP address | "192.168.122.100" | +| port | Kafka server port number | "9092" | +| model_topic | Topic for reporting fault detection results | "gala_anteater_hybrid_model" | +| rca_topic | Topic for reporting root cause analysis results | "gala_cause_inference" | +| meta_topic | Topic for gopher-collected metric data | "gala_gopher_metadata" | +| group_id | Kafka group name | "gala_anteater_kafka" | +| Prometheus | Prometheus data source configuration | Dictionary type | +| server | Prometheus server IP address | "localhost" | +| port | Prometheus server port number | "9090" | +| steps | Metric sampling interval | "5" | +| Schedule | Loop scheduling configuration | Dictionary type | +| duration | Execution frequency of the exception detection model in minutes | 1 | +| Suppression | Alarm suppression configuration | Dictionary type | +| interval | Alarm suppression interval filtering the same alarm within _x_ minutes after the last alarm | 10 | + +## Startup + +Run the following command to start gala-anteater: -#### Configuration - -> Note: gala-anteater does not contain the config file that needs to be configured. Its parameters are passed through the startup parameters using the command line. - -##### Startup Parameters - -| Parameter| Parameter Full Name| Type| Mandatory (Yes/No)| Default Value| Name| Description| -|---|---|---|---|---|---|---| -| -ks | --kafka_server | string | True | | KAFKA_SERVER | IP address of the Kafka server, for example, **localhost / xxx.xxx.xxx.xxx**.| -| -kp | --kafka_port | string | True | | KAFKA_PORT | Port number of the Kafka server, for example, **9092**.| -| -ps | --prometheus_server | string | True | | PROMETHEUS_SERVER | IP address of the Prometheus server, for example, **localhost / xxx.xxx.xxx.xxx**.| -| -pp | --prometheus_port | string | True | | PROMETHEUS_PORT | Port number of the Prometheus server, for example, **9090**.| -| -m | --model | string | False | vae | MODEL | Exception detection model. Currently, two exception detection models are supported: **random_forest** and **vae**.
**random_forest**: random forest model, which does not support online learning
**vae**: Variational Atuoencoder (VAE), which is an unsupervised model and supports model update based on historical data during the first startup.| -| -d | --duration | int | False | 1 | DURATION | Frequency of executing the exception detection model. The unit is minute, which means that the detection is performed every *x* minutes.| -| -r | --retrain | bool | False | False | RETRAIN | Whether to use historical data to update and iterate the model during startup. Currently, only the VAE model is supported.| -| -l | --look_back | int | False | 4 | LOOK_BACK | Whether to update the model based on the historical data of the last *x* days.| -| -t | --threshold | float | False | 0.8 | THRESHOLD | Threshold of the exception detection model, ranging from 0 to 1. A larger value can reduce the false positive rate of the model. It is recommended that the value be greater than or equal to 0.5.| -| -sli | --sli_time | int | False | 400 | SLI_TIME | Application performance metric. The unit is ms. A larger value can reduce the false positive rate of the model. It is recommended that the value be greater than or equal to 200.
For scenarios with a high false positive rate, it is recommended that the value be greater than 1000.| - - - -#### Start - -Start gala-anteater. - -> Note: gala-anteater can be started and run in command line mode, but cannot be started and run in systemd mode. - -- Running in online training mode (recommended) ```bash -gala-anteater -ks {ip} -kp {port} -ps {ip} -pp {port} -m vae -r True -l 7 -t 0.6 -sli 400 +systemctl start gala-anteater ``` -- Running in common mode -```bash -gala-anteater -ks {ip} -kp {port} -ps {ip} -pp {port} -m vae -t 0.6 -sli 400 -``` +>![](./figures/icon-note.gif)**Note:** +> +> gala-anteater supports only one process instance. Starting multiple instances may cause excessive memory usage and log inconsistencies. -Query the gala-anteater service status. +### Execution Status of Slow Node Detection in gala-anteater Service -If the following information is displayed, the service is started successfully. The startup log is saved to the **logs/anteater.log** file in the current running directory. +If the log shows the following content, it means slow node detection is functioning correctly. The startup log is also saved in **/var/log/gala-anteater/gala-anteater.log**. ```log -2022-09-01 17:52:54,435 - root - INFO - Run gala_anteater main function... -2022-09-01 17:52:54,436 - root - INFO - Start to try updating global configurations by querying data from Kafka! -2022-09-01 17:52:54,994 - root - INFO - Loads metric and operators from file: xxx\metrics.csv -2022-09-01 17:52:54,997 - root - INFO - Loads metric and operators from file: xxx\metrics.csv -2022-09-01 17:52:54,998 - root - INFO - Start to re-train the model based on last day metrics dataset! -2022-09-01 17:52:54,998 - root - INFO - Get training data during 2022-08-31 17:52:00+08:00 to 2022-09-01 17:52:00+08:00! -2022-09-01 17:53:06,994 - root - INFO - Spends: 11.995422840118408 seconds to get unique machine_ids! -2022-09-01 17:53:06,995 - root - INFO - The number of unique machine ids is: 1! -2022-09-01 17:53:06,996 - root - INFO - Fetch metric values from machine: xxxx. -2022-09-01 17:53:38,385 - root - INFO - Spends: 31.3896164894104 seconds to get get all metric values! -2022-09-01 17:53:38,392 - root - INFO - The shape of training data: (17281, 136) -2022-09-01 17:53:38,444 - root - INFO - Start to execute vae model training... -2022-09-01 17:53:38,456 - root - INFO - Using cpu device -2022-09-01 17:53:38,658 - root - INFO - Epoch(s): 0 train Loss: 136.68 validate Loss: 117.00 -2022-09-01 17:53:38,852 - root - INFO - Epoch(s): 1 train Loss: 113.73 validate Loss: 110.05 -2022-09-01 17:53:39,044 - root - INFO - Epoch(s): 2 train Loss: 110.60 validate Loss: 108.76 -2022-09-01 17:53:39,235 - root - INFO - Epoch(s): 3 train Loss: 109.39 validate Loss: 106.93 -2022-09-01 17:53:39,419 - root - INFO - Epoch(s): 4 train Loss: 106.48 validate Loss: 103.37 -... -2022-09-01 17:53:57,744 - root - INFO - Epoch(s): 98 train Loss: 97.63 validate Loss: 96.76 -2022-09-01 17:53:57,945 - root - INFO - Epoch(s): 99 train Loss: 97.75 validate Loss: 96.58 -2022-09-01 17:53:57,969 - root - INFO - Schedule recurrent job with time interval 1 minute(s). -2022-09-01 17:53:57,973 - apscheduler.scheduler - INFO - Adding job tentatively -- it will be properly scheduled when the scheduler starts -2022-09-01 17:53:57,974 - apscheduler.scheduler - INFO - Added job "partial" to job store "default" -2022-09-01 17:53:57,974 - apscheduler.scheduler - INFO - Scheduler started -2022-09-01 17:53:57,975 - apscheduler.scheduler - DEBUG - Looking for jobs to run -2022-09-01 17:53:57,975 - apscheduler.scheduler - DEBUG - Next wakeup is due at 2022-09-01 17:54:57.973533+08:00 (in 59.998006 seconds) +2024-12-02 16:25:20,727 - INFO - anteater - Groups-0, metric: npu_chip_info_hbm_used_memory, start detection. +2024-12-02 16:25:20,735 - INFO - anteater - Metric-npu_chip_info_hbm_used_memory single group has data 8. ranks: [0, 1, 2, 3, 4, 5, 6, 7] +2024-12-02 16:25:20,739 - INFO - anteater - work on npu_chip_info_hbm_used_memory, slow_node_detection start. +2024-12-02 16:25:21,128 - INFO - anteater - time_node_compare result: []. +2024-12-02 16:25:21,137 - INFO - anteater - dnscan labels: [-1 0 0 0 -1 0 -1 -1] +2024-12-02 16:25:21,139 - INFO - anteater - dnscan labels: [-1 0 0 0 -1 0 -1 -1] +2024-12-02 16:25:21,141 - INFO - anteater - dnscan labels: [-1 0 0 0 -1 0 -1 -1] +2024-12-02 16:25:21,142 - INFO - anteater - space_nodes_compare result: []. +2024-12-02 16:25:21,142 - INFO - anteater - Time and space aggregated result: []. +2024-12-02 16:25:21,144 - INFO - anteater - work on npu_chip_info_hbm_used_memory, slow_node_detection end. + +2024-12-02 16:25:21,144 - INFO - anteater - Groups-0, metric: npu_chip_info_aicore_current_freq, start detection. +2024-12-02 16:25:21,153 - INFO - anteater - Metric-npu_chip_info_aicore_current_freq single group has data 8. ranks: [0, 1, 2, 3, 4, 5, 6, 7] +2024-12-02 16:25:21,157 - INFO - anteater - work on npu_chip_info_aicore_current_freq, slow_node_detection start. +2024-12-02 16:25:21,584 - INFO - anteater - time_node_compare result: []. +2024-12-02 16:25:21,592 - INFO - anteater - dnscan labels: [0 0 0 0 0 0 0 0] +2024-12-02 16:25:21,594 - INFO - anteater - dnscan labels: [0 0 0 0 0 0 0 0] +2024-12-02 16:25:21,597 - INFO - anteater - dnscan labels: [0 0 0 0 0 0 0 0] +2024-12-02 16:25:21,598 - INFO - anteater - space_nodes_compare result: []. +2024-12-02 16:25:21,598 - INFO - anteater - Time and space aggregated result: []. +2024-12-02 16:25:21,598 - INFO - anteater - work on npu_chip_info_aicore_current_freq, slow_node_detection end. + +2024-12-02 16:25:21,598 - INFO - anteater - Groups-0, metric: npu_chip_roce_tx_err_pkt_num, start detection. +2024-12-02 16:25:21,607 - INFO - anteater - Metric-npu_chip_roce_tx_err_pkt_num single group has data 8. ranks: [0, 1, 2, 3, 4, 5, 6, 7] +2024-12-02 16:25:21,611 - INFO - anteater - work on npu_chip_roce_tx_err_pkt_num, slow_node_detection start. +2024-12-02 16:25:22,040 - INFO - anteater - time_node_compare result: []. +2024-12-02 16:25:22,040 - INFO - anteater - Skip space nodes compare. +2024-12-02 16:25:22,040 - INFO - anteater - Time and space aggregated result: []. +2024-12-02 16:25:22,040 - INFO - anteater - work on npu_chip_roce_tx_err_pkt_num, slow_node_detection end. + +2024-12-02 16:25:22,041 - INFO - anteater - accomplishment: 1/9 +2024-12-02 16:25:22,041 - INFO - anteater - accomplishment: 2/9 +2024-12-02 16:25:22,041 - INFO - anteater - accomplishment: 3/9 +2024-12-02 16:25:22,041 - INFO - anteater - accomplishment: 4/9 +2024-12-02 16:25:22,042 - INFO - anteater - accomplishment: 5/9 +2024-12-02 16:25:22,042 - INFO - anteater - accomplishment: 6/9 +2024-12-02 16:25:22,042 - INFO - anteater - accomplishment: 7/9 +2024-12-02 16:25:22,042 - INFO - anteater - accomplishment: 8/9 +2024-12-02 16:25:22,042 - INFO - anteater - accomplishment: 9/9 +2024-12-02 16:25:22,043 - INFO - anteater - SlowNodeDetector._execute costs 1.83 seconds! +2024-12-02 16:25:22,043 - INFO - anteater - END! ``` +## Exception Detection Output Data - -#### Output Data - -If gala-anteater detects an exception, it sends the result to Kafka. The output data format is as follows: +When gala-anteater detects exceptions, it outputs the results to `model_topic` of Kafka. The output data format is as follows. ```json { - "Timestamp":1659075600000, - "Attributes":{ - "entity_id":"xxxxxx_sli_1513_18", - "event_id":"1659075600000_1fd37742xxxx_sli_1513_18", - "event_type":"app" - }, - "Resource":{ - "anomaly_score":1.0, - "anomaly_count":13, - "total_count":13, - "duration":60, - "anomaly_ratio":1.0, - "metric_label":{ - "machine_id":"1fd37742xxxx", - "tgid":"1513", - "conn_fd":"18" - }, - "recommend_metrics":{ - "gala_gopher_tcp_link_notack_bytes":{ - "label":{ - "__name__":"gala_gopher_tcp_link_notack_bytes", - "client_ip":"x.x.x.165", - "client_port":"51352", - "hostname":"localhost.localdomain", - "instance":"x.x.x.172:8888", - "job":"prometheus-x.x.x.172", - "machine_id":"xxxxxx", - "protocol":"2", - "role":"0", - "server_ip":"x.x.x.172", - "server_port":"8888", - "tgid":"3381701" - }, - "score":0.24421279500639545 - }, - ... - }, - "metrics":"gala_gopher_ksliprobe_recent_rtt_nsec" - }, - "SeverityText":"WARN", - "SeverityNumber":14, - "Body":"TimeStamp, WARN, APP may be impacting sli performance issues." + "Timestamp": 1730732076935, + "Attributes": { + "resultCode": 201, + "compute": false, + "network": false, + "storage": true, + "abnormalDetail": [{ + "objectId": "-1", + "serverIp": "96.13.19.31", + "deviceInfo": "96.13.19.31:8888*-1", + "kpiId": "gala_gopher_disk_wspeed_kB", + "methodType": "TIME", + "kpiData": [], + "relaIds": [], + "omittedDevices": [] + }], + "normalDetail": [], + "errorMsg": "" + }, + "SeverityText": "WARN", + "SeverityNumber": 13, + "is_anomaly": true } ``` + +## Output Fields + +| Field | Type/Unit | Description | +| -------------- | --------- | ----------------------------------------------------------------------------------------------------------------------- | +| Timestamp | ms | Timestamp when the fault was detected and reported | +| resultCode | int | Fault code, with 201 indicating a fault and 200 indicating no fault | +| compute | bool | Whether the fault type is compute | +| network | bool | Whether the fault type is network | +| storage | bool | Whether the fault type is storage | +| abnormalDetail | list | Details of the fault | +| objectId | int | Fault object ID, with -1 indicating a node fault and 0 to 7 indicating specific fault card numbers | +| serverIp | string | IP address of the fault object | +| deviceInfo | string | Detailed fault information | +| kpiId | string | Algorithm type ("TIME" or "SPACE") that detected the fault | +| kpiData | list | Time-series data of the fault (requires enabling a switch, disabled by default) | +| relaIds | list | IDs of normal cards associated with the fault card, indicating normal card numbers compared under the "SPACE" algorithm | +| omittedDevices | list | Card numbers to be ignored in display | +| normalDetail | list | Time-series data of normal cards | +| errorMsg | string | Error message | +| SeverityText | string | Error type ("WARN" or "ERROR") | +| SeverityNumber | int | Error severity level | +| is_anomaly | bool | Whether a fault exists | diff --git a/docs/en/docs/AI4C/AI4C-user-guide.md b/docs/en/docs/AI4C/AI4C-user-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..10f78cceba7bab7abfd38d6f445a67a4b9ba9d05 --- /dev/null +++ b/docs/en/docs/AI4C/AI4C-user-guide.md @@ -0,0 +1,455 @@ +# AI4C User Guide + +## 1 Introduction to AI4C + +AI4C is an AI-assisted compiler suite and a framework designed to integrate machine learning (ML)-driven optimizations into compilers. + +## 2 Software Architecture Overview + +The framework includes the following modules, with the automatic compilation tuning tool requiring a Python environment: + +- Inference engine for AI-assisted compilation optimization: This component enables the compiler to leverage results from AI model inference within optimization passes to enhance compilation. + - In GCC, AI-enabled optimization passes are primarily implemented as plugins. This ensures they are decoupled from the main compiler version. +- Automatic compilation tuning tool: This tool uses OpenTuner, an external tuning utility, to drive the compiler in performing multi-granularity automatic compilation tuning. It currently supports GCC and LLVM. + - Option tuning tool: Focuses on application-level compilation option tuning. + - Compilation tuning tool: Built on [Autotuner](https://gitee.com/openeuler/BiSheng-Autotuner), it supports both fine-grained and coarse-grained compilation tuning. + - Fine-grained tuning: Adjusts key parameters within optimization passes, such as loop unrolling counts. + - Coarse-grained tuning: Optimizes function-level compilation options. + +Future development plans: + +- [ ] Integrate the LLVM compilation optimization model from [ACPO](https://gitee.com/src-openeuler/ACPO) and modularize the relevant ACPO LLVM code into a plugin, decoupling it from the main LLVM version. +- [ ] Extend the AI4Compiler framework to support inference from additional open source ML frameworks like PyTorch (LibTorch) and TensorFlow (LiteRT). +- [ ] Develop more AI-assisted compilation optimization models and corresponding compiler plugins. +- [ ] Incorporate new search algorithms (based on white-box information) and refine parameter search spaces for hotspot function tuning. +- [ ] Add support for JDK compilation parameter tuning. + +## 3 Installation and Build of AI4C + +### 3.1 Direct Installation of AI4C + +For users on the latest openEuler system (24.03-LTS-SP1) who only need the existing features of AI4C, the package can be installed directly. + +```shell +yum install -y AI4C +``` + +For users requiring features from other AI4C versions or installing on different OS versions, rebuilding AI4C is necessary. Follow the steps below. + +### 3.2 RPM Package Build and Installation (Recommended) + +1. Install `rpmbuild` and `rpmdevtools` with root privileges: + + ```bash + # Install rpmbuild. + yum install dnf-plugins-core rpm-build + # Install rpmdevtools. + yum install rpmdevtools + ``` + +2. Create the **rpmbuild** folder in the **/root** directory: + + ```bash + rpmdev-setuptree + # Verify the directory structure. + ls ~/rpmbuild/ + BUILD BUILDROOT RPMS SOURCES SPECS SRPMS + ``` + +3. Clone the repository and copy the necessary files to the **rpmbuild** folders: + + ```shell + git clone https://gitee.com/src-openeuler/AI4C.git + cp AI4C/AI4C-v%{version}-alpha.tar.gz ~/rpmbuild/SOURCES/ + cp AI4C/*.patch ~/rpmbuild/SOURCES/ + cp AI4C/AI4C.spec ~/rpmbuild/SPECS/ + ``` + +4. Build and install the AI4C RPM package: + + ```shell + # Install dependencies. + yum-builddep ~/rpmbuild/SPECS/AI4C.spec + # Build the package. + # If check-rpaths errors occur, use QA_RPATHS=0x0002 rpmbuild -ba ~/rpmbuild/SPECS/AI4C.spec. + rpmbuild -ba ~/rpmbuild/SPECS/AI4C.spec + # Install the RPM package. + cd ~/rpmbuild/RPMS/ + rpm -ivh AI4C--..rpm + ``` + + **Note:** If file conflicts arise due to existing old RPM packages, resolve them with: + + ```shell + # Force install the new version + rpm -ivh AI4C--..rpm --force + # Or update the package + rpm -Uvh AI4C--..rpm + ``` + + After installation, the system will contain the following files: + + - **/usr/bin/ai4c-***: wrappers for AI-enabled compilers and tuning tools + - **/usr/lib64/libonnxruntime.so**: ONNX Runtime inference framework dynamic library + - **/usr/lib64/AI4C/*.onnx**: AI-assisted compilation optimization models in ONNX format + - **/usr/lib64/python\/site-packages/ai4c/lib/*.so**: + - dynamic libraries for the AI-assisted compilation optimization inference engine + - dynamic libraries for AI-assisted compilation optimization and compiler tuning plugins + - **/usr/lib64/python\/site-packages/ai4c/autotuner/\***: files for coarse and fine-grained tuning tools + - **/usr/lib64/python\/site-packages/ai4c/optimizer/\***: files for AI-assisted compilation optimization + - **/usr/lib64/python\/site-packages/ai4c/option_tuner/\***: files for application-level compilation option tuning + +### 3.3 Source Code Build and Installation + +AI4C source code is available at . + +#### 3.3.1 Installing ONNX Runtime Dependencies + +**Option 1:** + +Download [ONNX Runtime v1.16.3](https://github.com/microsoft/onnxruntime/releases/tag/v1.16.3) for the target architecture from GitHub and extract the .tgz file. For example, for aarch64, download **onnxruntime-linux-aarch64-1.16.3.tgz**. + +**Note**: After extraction, rename the **lib** directory to **lib64** to avoid `-lonnxruntime` path errors during the build. + +**Option 2:** + +Make sure the following dependencies are installed: + +```shell +yum install -y cmake make gcc gcc-c++ abseil-cpp-devel boost-devel bzip2 python3-devel python3-numpy python3-setuptools python3-pip +``` + +Install ONNX Runtime using `cmake`: + +```shell +cd path/to/your/AI4C/third_party/onnxruntime +cmake \ + -DCMAKE_INSTALL_PREFIX=path/to/your/onnxruntime \ + -Donnxruntime_BUILD_SHARED_LIB=ON \ + -Donnxruntime_BUILD_UNIT_TESTS=ON \ + -Donnxruntime_INSTALL_UNIT_TESTS=OFF \ + -Donnxruntime_BUILD_BENCHMARKS=OFF \ + -Donnxruntime_USE_FULL_PROTOBUF=ON \ + -DPYTHON_VERSION=%{python3_version} \ + -Donnxruntime_ENABLE_CPUINFO=ON \ + -Donnxruntime_DISABLE_ABSEIL=ON \ + -Donnxruntime_USE_NEURAL_SPEED=OFF \ + -Donnxruntime_ENABLE_PYTHON=OFF \ + -DCMAKE_BUILD_TYPE=Release \ + -S cmake +make -j %{max_jobs} && make install +``` + +#### 3.3.2 Installing Additional Build Dependencies for AI4C + +Make sure the following dependencies are installed: + +```shell +yum install -y python3-wheel openssl openssl-devel yaml-cpp yaml-cpp-devel gcc-plugin-devel libstdc++-static +``` + +#### 3.3.3 Building the AI4C Framework + +```shell +cd path/to/your/AI4C/python +python3 setup.py bdist_wheel \ + -Donnxruntime_ROOTDIR=path/to/your/onnxruntime \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_CXX_COMPILER=path/to/your/g++ \ + -DCMAKE_C_COMPILER=path/to/your/gcc +pip3 install dist/ai4c----_.whl --force-reinstall --no-deps +``` + +Once installed, the system will contain the following files: + +- **path/to/your/pythonbin/ai4c-\***: wrapper for the AI-enabled compiler and auto-tuning tools +- **path/to/your/onnxruntime/lib64/libonnxruntime.so**: ONNX Runtime inference framework dynamic library +- **path/to/your/AI4C/models/*.onnx**: AI-assisted compilation optimization models (ONNX format) +- **path/to/your/pythonlib/ai4c/lib/*.so**: + - AI-assisted compilation optimization inference engine dynamic libraries + - AI-assisted compilation optimization and compiler tuning plugin dynamic libraries +- **path/to/your/pythonlib/ai4c/autotuner/\***: files related to coarse and fine-grained tuning tools +- **path/to/your/pythonlib/ai4c/optimizer/\***: files related to AI-assisted compilation optimization +- **path/to/your/pythonlib/ai4c/option_tuner/\***: files related to application-level compilation option tuning + +**Note:** + +- **path/to/your/pythonbin**: After installation, use `which ai4c-gcc` to locate the **bin** path. +- **path/to/your/pythonlib**: After installation, use `pip show ai4c` to find the **lib** path. + +## 4 Usage Process + +### 4.1 AI-Assisted Compilation Optimization + +The AI-assisted compilation optimization module primarily consists of three components: + +- Trained ONNX model for assisted compilation optimization +- Compiler plugin (**only GCC is supported currently**) used to run ONNX model inference and retrieve optimization parameters +- AI4Compiler framework providing the ONNX inference engine and GCC optimization compilation commands + +You train an AI model using an open source ML framework and export it in ONNX format. A corresponding compiler plugin is also required, which must include at least three modules for the following purposes: + +- Extract compiler input features required by the AI model. +- Drive the inference engine to run the AI model for inference. +- Annotate the inference results back into the compiler data structure. + +In the test example below, simply add three plugin-related compilation options to the compilation command for the target binary: the plugin path, the AI model path for the plugin, and the inference engine path. This enables the AI-assisted compilation optimization model during compilation. + +```shell +# If onnxruntime is installed in a non-system directory, set the environment variable. +# export LD_LIBRARY_PATH=path/to/your/onnxruntime/lib64/:$LD_LIBRARY_PATH + +gcc_compiler=path/to/your/gcc +infer_engine_path=$(ai4c-gcc --inference-engine) +model_path=path/to/your/model.onnx +plugin_path=path/to/your/.so + +$gcc_compiler test.c -O2 -o test \ + -fplugin=$plugin_path \ + -fplugin-arg--model=$model_path \ + -fplugin-arg--engine=$infer_engine_path +``` + +Currently supported plugins are in the same directory as `$(ai4c-gcc --inference-engine)`, and supported models are under **path/to/your/AI4C/models**. + +**Notes:** + +- The compiler plugin for the AI model and the compiler for the target application must be the same version to avoid compilation errors due to version mismatch. +- Currently, AI4C only supports AI-assisted compilation optimization passes implemented at the GCC `cc1` phase using the plugin form. + +For detailed compiler plugin development and usage processes, refer to the [AI-Assisted Compilation Optimization Guide](https://gitee.com/openeuler/AI4C/blob/master/python/docs/gcc-opt.md) and [Test Examples](https://gitee.com/openeuler/AI4C/tree/master/python/test/optimizer/block_correction). + +Below are two usage examples of AI-assisted compilation optimization models at different compilation stages. The **loop unrolling and function inlining model** operates at the `cc1` compilation optimization phase and uses a GCC plugin for AI model adaptation and inference. The **BOLT sampling basic block precision correction model** operates at the `BOLT` post-link optimization phase, with its model adaptation layer located in the [LLVM-BOLT](https://gitee.com/src-openeuler/llvm-bolt) repository. + +#### 4.1.1 Loop Unrolling and Function Inlining Model + +The compilation optimization options for the loop unrolling and function inlining model include: + +| Option | Description | +| --------------------------------------------------- | ------------------------------------------------------------ | +| -fplugin | Specifies the **absolute path** of the loop unrolling and function inlining plugin (`-fplugin=/path/to/.so`). | +| -fplugin-arg-\-engine | Specifies the **absolute path** of the function inlining ONNX model's inference engine (`-fplugin-arg--inline_model=/path/to/inference_engine.so`). This must be used together with `-fplugin`. The path `/path/to/inference_engine.so` can be retrieved using `ai4c-gcc --inference-engine`. | +| -fplugin-arg-\-inline_model | Specifies the **absolute path** of the function inlining ONNX model (`-fplugin-arg--inline_model=/path/to/inline_model.onnx`). This must be used together with `-fplugin` and `-fplugin-arg--engine`. | +| -fplugin-arg-\-unroll_model | Specifies the **absolute path** of the loop unrolling ONNX model (`-fplugin-arg--unroll_model=/path/to/unroll_model.onnx`). This must be used together with `-fplugin` and `-fplugin-arg--engine`. | + +You can enable multiple AI-assisted compilation optimization models within a single GCC plugin, as shown below: + +```shell +gxx_compiler=path/to/your/g++ +infer_engine_path=$(ai4c-gcc --inference-engine) +inline_model_path=path/to/your/inline_model.onnx +unroll_model_path=path/to/your/unroll_model.onnx +plugin_path=path/to/your/.so + +$gxx_compiler test.cc -O3 -o test -funroll-loops \ + -fplugin=$plugin_path \ + -fplugin-arg--engine=$infer_engine_path \ + -fplugin-arg--inline_model=$inline_model_path \ + -fplugin-arg--unroll_model=$unroll_model_path +``` + +#### 4.1.2 BOLT Sampling Basic Block Precision Correction Model + +The BOLT optimization options for the sampling basic block precision correction model include: + +| Option | Description | +| --------------------- | ------------------------------------------------------------ | +| -block-correction | Enables the AI-optimized CFG BB Count option. Must be used together with the `-model-path` option to specify the ONNX model. | +| -model-path | Specifies the **absolute path** of the ONNX model (`-model-path=/path/to/model.onnx`). Must be used together with `-block-correction`. | +| -annotate-threshold | Specifies the confidence threshold for the model prediction results, with a default value of 0.95. | + +BOLT-specific optimization options can be enabled using GCC's `-fbolt-option`, as shown below: + +```shell +g++ -fbolt-use= -fbolt-target= -fbolt-option=\"-block-correction -model-path=path/to/your/block_correction_model.onnx\" +``` + +### 4.2 Fine-Grained Tuning + +This section demonstrates the workflow of the fine-grained tuning tool using the **loop unrolling** optimization pass in GCC as an example. + +The fine-grained tuning module currently takes two input components: + +- Application tuning configuration file (.ini): Manages the compilation and execution process of the application. +- Parameter search space configuration file (YAML): Defines the parameter search space for the `autotune` phase, which can replace the default search space. + +The fine-grained tuning is implemented using [Autotuner](https://gitee.com/openeuler/BiSheng-Autotuner): + +1. During the `generate` phase of the compiler, a set of tunable compilation data structures and coefficients are generated and stored in **opp/*.yaml**. +2. Using the provided compilation search space (**search_space.yaml**) and tunable data structures, Autotuner generates the next set of tuning coefficients for each tunable data structure, which are saved in **input.yaml**. +3. In the `autotune` phase, the tuning coefficients are annotated into the corresponding data structures based on their hash values in **input.yaml**, completing the tuning process. + +Before enabling fine-grained tuning, install the following dependencies: + +```shell +yum install -y BiSheng-Autotuner bisheng-opentuner +``` + +In the following test case, we tune the loop unrolling parameters for [CoreMark](https://github.com/eembc/coremark). First, prepare the tuning configuration file **coremark_sample.ini**. You must: + +- Provide the application path, compilation, and execution commands. +- Add the fine-grained tuning dynamic library `-fplugin=%(PluginPath)s/rtl_unroll_autotune_plugin_gcc12.so` to the base compilation command. + - Include the corresponding input files for the `generate` and `autotune` phases using `-fplugin-arg-rtl_unroll_autotune_plugin_gcc12-`. +- Customize the paths for the tunable structure configuration files (**./opp/*.yaml**) and the compiler input file generated by Autotuner (**input.yaml**). + +```ini +[DEFAULT] # Optional +# PluginPath = /path/to/gcc-plugins + +[Environment Setting] # Optional +# Prepend a list of paths into the PATH in order. +# PATH = /path/to/bin +# You can also set other environment variables here. + +[Compiling Setting] # Mandatory +# NOTE: ConfigFilePath is set to the path to the current config file automatically by default. +CompileDir = /path/to/coremark +LLVMInputFile = %(CompileDir)s/input.yaml + +# OppDir and OppCompileCommand are optional. +# You do not have to specify this if not using auto_run sub-command. +OppDir = autotune_datadir/opp + +CompilerCXX = /path/to/bin/gcc +BaseCommand = %(CompilerCXX)s -I. -I./posix -DFLAGS_STR=\"" -lrt"\" \ + -DPERFORMANCE_RUN=1 -DITERATIONS=10000 -g \ + core_list_join.c core_main.c core_matrix.c \ + core_state.c core_util.c posix/core_portme.c \ + -funroll-loops -O2 -o coremark \ + -fplugin=%(PluginPath)s/rtl_unroll_autotune_plugin_gcc12.so + +# Auto-tuning +CompileCommand = %(BaseCommand)s \ + -fplugin-arg-rtl_unroll_autotune_plugin_gcc12-autotune=%(LLVMInputFile)s + +RunDir = %(CompileDir)s +RunCommand = ./coremark 0x0 0x0 0x66 100000 # Run 300000 iterations for coremark. + +# Generation +OppCompileCommand = %(BaseCommand)s \ + -fplugin-arg-rtl_unroll_autotune_plugin_gcc12-generate=%(OppDir)s +``` + +Next, prepare an additional parameter search space file **search_space.yaml** to customize and narrow the parameter space. For example, the dynamic library defaults to a loop unrolling coefficient space of $\{0, 2^0=1, 2^1=2, ..., 2^6=64\}$, but we can adjust it to $\{0, 2^0=1, 2^1=2, ..., 2^5=32\}$. + +```yaml +CodeRegion: + CodeRegionType: loop + Pass: loop2_unroll + Args: + UnrollCount: + Value: [0, 1, 2, 4, 8, 16, 32] + Type: enum +``` + +Finally, place **coremark**, **coremark_sample.ini**, and **search_space.yaml** in the same folder and run the following script: + +```shell +ai4c-autotune autorun coremark_sample.ini \ + -scf search_space.yaml --stage-order loop \ + --time-after-convergence=100 +``` + +The `time-after-convergence` option specifies how many seconds after the historical best value is reached without finding a new optimal configuration before the tuning process is terminated early. + +After tuning, the best configuration is saved in **loop.yaml**. To reproduce its performance, re-run the `autotune` phase compilation command and modify the input file for the `autotune` option (e.g., `-fplugin-arg-rtl_unroll_autotune_plugin_gcc12-autotune=loop.yaml`). + +You can retrieve historical tuning configuration files (**autotune_config.csv**) and performance data files (**autotune_data.csv**) using: + +```shell +ai4c-autotune dump -c coremark/input.yaml \ + --database=opentuner.db/localhost.localdomain.db -o autotune +``` + +**Notes:** + +- Currently, program runtime is the default performance metric. + +For detailed usage, refer to the [Fine-Grained Tuning User Guide](https://gitee.com/openeuler/AI4C/blob/master/python/docs/autotuner.md) and this test case: + +For fine-grained tuning with the LLVM compiler, see the usage tutorial in the [Autotuner](https://gitee.com/openeuler/BiSheng-Autotuner) repository. + +### 4.4 Application-Level Option Tuning + +The application-level option tuning module currently comprises three main input components: + +- **Application compilation and execution script (shell)**: Manages the application's compilation process (replacing the next set of options into the compilation script), execution process, and performance data collection. +- **Compilation and dynamic library option search space configuration file (YAML)**: Defines the search space for option tuning, including toggle options (compilation optimizations/dynamic libraries), compilation parameters, and enumeration options. +- **Performance value configuration file (YAML)**: Specifies the weights of multiple performance metrics and the optimization direction (maximize/minimize), which must align with the number and order of performance values collected during the "performance data collection process." + +The application-level option tuning tool continuously gathers performance data, updates the performance model, and generates new compilation option combinations expected to deliver higher model benefits. These new options are integrated into the compilation script via the application's compilation and execution script, producing a new binary and initiating the next execution cycle. This iterative tuning process continues until the historically best performance value is achieved. + +Before enabling application-level option tuning, install the following dependencies: + +```shell +pip install xgboost scikit-learn +yum install -y time +``` + +The following example demonstrates building and tuning `test.cc` over 3 rounds using different compilation option combinations. The application's compilation and execution script is as follows: + +```shell +# ---------- run_test.sh ---------- # +parent_dir=$1 # path for intermediate tuning files +config=$(cat ${parent_dir}/tuning/config.txt) # current compiler configuration file +performance_file="${parent_dir}/tuning/performance.txt" # current performance data file + +measure_raw_file="time.txt" + +compiler=g++ +compile_command="${compiler} test.cc -O2 -o test_opt_tuner" +eval "${compile_command} ${config}" # program compilation, appending tuning options + +run_command="time -p -o ${measure_raw_file} ./test_opt_tuner 3" +eval "${run_command}" # program execution + +info_collect_command="grep real ${measure_raw_file} | awk '{printf \"1 1 %s\", \$2}' > ${performance_file}" +eval "${info_collect_command}" # program performance collection + +# ---------- run_option_tuner.sh ---------- # +ai4c-option-tune --test_limit 3 --runfile run_test.sh + # --optionfile path/to/your/python/site-packages/ai4c/option_tuner/input/options.yaml \ + # --libfile path/to/your/python/site-packages/ai4c/option_tuner/input/options_lib.yaml \ + # --measurefile path/to/your/python/site-packages/ai4c/option_tuner/input/config_measure.yaml +``` + +The default option and performance value configuration files are located **path/to/your/python\/site-packages/ai4c/option_tuner/input/*.yaml**. + +You can customize the compilation and dynamic library option configuration files as needed. Key terms include: + +- `required_*`: Mandatory tuning items that remain active throughout the tuning process. +- `bool_*`: Optional compilation optimization toggle options. +- `interval_*`: Optional compilation parameters (value options with a defined range). +- `enum_*`: Optional compilation parameters (enumeration options). + +For example: + +```yaml +required_config: +- -O2 +bool_config: +- -funroll-loops +interval_config: +- name: --param max-inline-insns-auto + default: 15 + min: 10 + max: 190 +``` + +You can also adjust the performance value configuration file as needed. Key terms include: + +- `weight`: The weight assigned to a performance value. +- `optim`: The optimization direction (maximize/minimize). + +For example: + +```yaml +config_measure: +- name: throughput + weight: 1 + optim: maximize +``` + +After tuning, historical and best tuning data are saved in **${parent_dir}/tuning/train.csv** and **${parent_dir}/tuning/result.txt**. + +For detailed usage, refer to this test case: . diff --git a/docs/en/docs/Administration/DIM.md b/docs/en/docs/Administration/DIM.md new file mode 100644 index 0000000000000000000000000000000000000000..a889b67d215829fdadc99587ddb2540094b4b1d3 --- /dev/null +++ b/docs/en/docs/Administration/DIM.md @@ -0,0 +1,734 @@ +# Dynamic Integrity Measurement (DIM) + +This section describes the DIM feature and its usage. + +## Context + +With the development of the IT industry, information systems are facing an increasing number of security risks. Information systems run a large amount of software, some of which is inevitably vulnerable. Once exploited by attackers, these vulnerabilities could severely damage system services, resulting in data leakage and service unavailability. + +Most software attacks are accompanied by integrity damage, such as malicious process execution, configuration file tampering, and backdoor implantation. Therefore, protection technologies are proposed in the industry. Key data is measured and verified during system startup to ensure that the system can run properly. However, popular integrity protection technologies (such as secure boot and file integrity measurement) cannot protect memory data during process running. If an attacker somehow modifies the instructions of a process, the process may be hijacked or implanted with a backdoor, which is highly destructive and stealthy. To defend against such attacks, the DIM technology is proposed to measure and protect key data in the memory of a running process. + +## Terminology + +Static baseline: baseline measurement data generated by parsing the binary file of the measurement target + +Dynamic baseline: result of the first measurement on the measurement target + +Measurement policy: configuration information for measuring the target + +Measurement log: list of measurement information, including the measurement targets and measurement results + +## Description + +The DIM feature measures key data (such as code sections and data sections) in the memory during program running and compares the measurement result with the baseline value to determine whether the memory data has been tampered with. In this way, DIM can detect attacks and take countermeasures. + +### Function Scope + +- Currently, DIM supports only AArch64 and x86 architectures. +- Currently, DIM supports measurement of the following key memory data: + - Code section of a user-mode process (the section whose attribute is **PT_LOAD** and permission is **RX** in the ELF file, and the virtual memory area whose permission is **RX** after the corresponding process is loaded) + - Kernel module code section, whose start address and length are **core_layout.base** and **core_layout.text_size** respectively in the **struct module** structure corresponding to the kernel module. + - Kernel code section, corresponding to **\_stext** to **\_etext** (Addresses that may change due to the kernel static key mechanism are skipped.) +- DIM can work with the following hardware: + - The measurement result can be extended to the Platform Configuration Register (PCR) of Trusted Platform Module (TPM) 2.0 to connect to the remote attestation service. + +### Technical Limitations + +- For user-mode processes, only mapped code sections of files can be measured. Anonymous code sections cannot be measured. +- Kernel hot patches cannot be measured. +- Measurement can only be triggered proactively. If a file is attacked and recovered during two measurement processes, the attack cannot be identified. +- Proactive modification of code sections, such as relocation of code sections, self-modification, and hot patches, will be identified as attacks. +- For kernel and kernel module measurement, the measurement result when dynamic baseline establishment is triggered is used as the measurement baseline value. The static baseline value only serves as a fixed identifier. +- The measurement target must have been loaded to the memory when dynamic baseline establishment is triggered (for example, process running or kernel module loading). Otherwise, the object cannot be measured. +- If the TPM PCR needs to be used to verify measurement logs, the DIM module cannot be removed. Otherwise, the measurement logs will be cleared and cannot match the PCR. + +>![](./public_sys-resources/icon-note.gif) **NOTE:** +> +>After DIM is enabled, system performance is affected in the following aspects: +> +> - Loading the DIM feature and managing baseline data and measurement logs consume system memory. The impact depends on the protection policy configuration. +> - Hashing is performed during DIM running, which consumes CPU resources. The impact depends on the size of the data to be measured. +> - Resources will be locked and semaphores need to be obtained during DIM running, which may cause other concurrent processes to wait. + +### Specification Constraints + +| Item | Value | +| ------------------------------------------------------------ | ---- | +| Maximum size of a policy file, static baseline file, signature file, or certificate file | 10MB | +| Maximum number of tampering measurement logs recorded for a measurement target during multiple measurement periods after a dynamic baseline is established.| 10| +| Maximum number of measurement policies that can be stored in **/etc/dim/policy**|10,000| + +### Architecture Description + +DIM includes the dim_tools and dim software packages, which contain the following components: + +| Software Package | Component | Description | +| --------- | ---------------- | ------------------------------------------------------------ | +| dim_tools | dim_gen_baseline| User-mode component for generating the static baseline, which is required for generating the dynamic measurement. The baseline data is imported when DIM runs.| +| dim | dim_core | Kernel module for executing the core dynamic measurement logic, including policy parsing, static baseline parsing, dynamic baseline establishment, measurement execution, measurement logging, and TPM extension operations, to measure key memory data.| +| dim | dim_monitor | Kernel module for measuring and protecting dim_core code sections and key data to prevent DIM function failures caused by attacks on dim_core. | + +The following figure shows the overall architecture: + +![](./figures/dim_architecture.jpg) + +### Key Procedures + +Both the dim_core and dim_monitor modules provide the memory data measurement function, including the following core processes: + +- Dynamic baseline process: The dim_core module reads and parses the policy and static baseline file, measures the code section of the target process, stores the measurement result as a dynamic baseline in the memory, compares the dynamic baseline data with the static baseline data, and records the comparison result in measurement logs. The dim_monitor module measures the code sections and key data of the dim_core module, uses the data as the dynamic baseline, and records measurement logs. +- Dynamic measurement process: The dim_core and dim_monitor modules measure the target and compare the measurement result with the dynamic baseline. If the measurement result is inconsistent with the dynamic baseline, the dim_core and dim_monitor modules record the result in measurement logs. + +### Interfaces + +#### Interface Files + +| Path | Description | +| ------------------------------- | ------------------------------------------------------------ | +| /etc/dim/policy | Measurement policy file | +| /etc/dim/policy.sig | Measurement policy signature file, which is used to store the signature information of the policy file and is used when the signature verification function is enabled| +| /etc/dim/digest_list/*.hash | Static baseline file, which is used to store measurement baseline values | +| /etc/dim/digest_list/*.hash.sig | Static baseline signature file, which is used to store the signature information of the static baseline file and is used when the signature verification function is enabled| +| /etc/keys/x509_dim.der | Certificate file, which is used to verify the signature information of the policy file and static baseline file and is used when the signature verification function is enabled| +| /sys/kernel/security/dim | DIM file system directory, which is generated after the DIM kernel module is loaded and provides kernel interfaces for operating the DIM function| + +#### File Formats + +1. Measurement policy file format + + The lines, each representing a measurement policy, are in plaintext and are separated by Unix newline (LF) characters. The following configuration formats are supported: + + 1. User-mode process code section measurement + + ```text + measure obj=BPRM_TEXT path= + ``` + + 2. Kernel module code section measurement + + ```text + measure obj=MODULE_TEXT name= + ``` + + 3. Kernel measurement + + ```text + measure obj=KERNEL_TEXT + ``` + + **Example:** + + ```shell + # cat /etc/dim/policy + measure obj=BPRM_TEXT path=/usr/bin/bash + measure obj=BPRM_TEXT path=/usr/lib64/libc.so.6 + measure obj=MODULE_TEXT name=ext4 + measure obj=KERNEL_TEXT + ``` + +2. Static baseline file format + + The lines, each representing a static baseline, are in plaintext and are separated by Unix newline (LF) characters. The following configuration formats are supported: + + 1. User-mode process baseline + + ```text + dim USER sha256:6ae347be2d1ba03bf71d33c888a5c1b95262597fbc8d00ae484040408a605d2b + ``` + + 2. Kernel module baseline + + ```text + dim KERNEL sha256:a18bb578ff0b6043ec5c2b9b4f1c5fa6a70d05f8310a663ba40bb6e898007ac5 / + ``` + + 3. Kernel baseline + + ```text + dim KERNEL sha256:2ce2bc5d65e112ba691c6ab46d622fac1b7dbe45b77106631120dcc5441a3b9a + ``` + + **Example:** + + ```text + dim USER sha256:6ae347be2d1ba03bf71d33c888a5c1b95262597fbc8d00ae484040408a605d2b /usr/bin/bash + dim USER sha256:bc937f83dee4018f56cc823f5dafd0dfedc7b9872aa4568dc6fbe404594dc4d0 /usr/lib64/libc.so.6 + dim KERNEL sha256:a18bb578ff0b6043ec5c2b9b4f1c5fa6a70d05f8310a663ba40bb6e898007ac5 6.4.0-1.0.1.4.oe2309.x86_64/dim_monitor + dim KERNEL sha256:2ce2bc5d65e112ba691c6ab46d622fac1b7dbe45b77106631120dcc5441a3b9a 6.4.0-1.0.1.4.oe2309.x86_64 + ``` + +3. Measurement log format + + The lines, each representing a measurement log, are in plaintext and are separated by Unix newline (LF) characters. The format is as follows: + + ```text + : + ``` + + **Example:** + + 1. The code section of the bash process is measured. The measurement result is consistent with the static baseline. + + ```text + 12 0f384a6d24e121daf06532f808df624d5ffc061e20166976e89a7bb24158eb87 sha256:db032449f9e20ba37e0ec4a506d664f24f496bce95f2ed972419397951a3792e /usr/bin.bash [static baseline] + ``` + + 2. The code section of the bash process is measured. The measurement result is inconsistent with the static baseline. + + ```text + 12 0f384a6d24e121daf06532f808df624d5ffc061e20166976e89a7bb24158eb87 sha256:db032449f9e20ba37e0ec4a506d664f24f496bce95f2ed972419397951a3792e /usr/bin.bash [tampered] + ``` + + 3. The code section of the ext4 kernel module is measured. No static baseline is found. + + ```text + 12 0f384a6d24e121daf06532f808df624d5ffc061e20166976e89a7bb24158eb87 sha256:db032449f9e20ba37e0ec4a506d664f24f496bce95f2ed972419397951a3792e ext4 [no static baseline] + ``` + + 4. dim_monitor measures dim_core and records the measurement result of the baseline. + + ```text + 12 660d594ba050c3ec9a7cdc8cf226c5213c1e6eec50ba3ff51ff76e4273b3335a sha256:bdab94a05cc9f3ad36d29ebbd14aba8f6fd87c22ae580670d18154b684de366c dim_core.text [dynamic baseline] + 12 28a3cefc364c46caffca71e7c88d42cf3735516dec32796e4883edcf1241a7ea sha256:0dfd9656d6ecdadc8ec054a66e9ff0c746d946d67d932cd1cdb69780ccad6fb2 dim_core.data [dynamic baseline] + ``` + +4. Certificate and signature file formats + +The files are in the common format. For details, see [Enabling Signature Verification](#enabling-signature-verification). + +#### Kernel Module Parameters + +1. dim_core parameters + + | Parameter | Description | Value Range | Default Value| + | -------------------- | ------------------------------------------------------------ | ------------------------ | ------ | + | measure_log_capacity | Maximum number of measurement logs recorded by dim_core. When this value is reached, dim_core stops recording measurement logs.| 100-UINT_MAX (64-bit OS)| 100000 | + | measure_schedule | Scheduling time after a process or module is measured, in milliseconds. The value 0 indicates that measurement is not scheduled.| 0-1000 | 0 | + | measure_interval | Automatic measurement interval, in minutes. The value 0 indicates that automatic measurement is not enabled. | 0-525600 | 0 | + | measure_hash | Measurement hash algorithm | sha256, sm3 | sha256 | + | measure_pcr | The TPM PCR to which the measurement result is extended. The value 0 indicates that the measurement result is not extended. The value must be an actual TPM PCR number.| 0-128 | 0 | + | signature | Whether to enable the policy file and signature verification. The value 0 indicates that they are disabled, and the value 1 indicates that they are enabled. | 0, 1 | 0 | + + **Example:** + + ```shell + insmod /path/to/dim_core.ko measure_log_capacity=10000 measure_schedule=10 measure_pcr=12 + modprobe dim_core measure_log_capacity=10000 measure_schedule=10 measure_pcr=12 + ``` + +2. dim_monitor parameters + + | Parameter | Description | Value Range | Default Value| + | -------------------- | ------------------------------------------------------------ | ------------------------ | ------ | + | measure_log_capacity | Maximum number of measurement logs recorded by dim_monitor. When this value is reached, dim_monitor stops recording measurement logs.| 100-UINT_MAX (64-bit OS)| 100000 | + | measure_hash | Measurement hash algorithm | sha256, sm3 | sha256 | + | measure_pcr | The TPM PCR to which the measurement result is extended. The value 0 indicates that the measurement result is not extended. | 0-128 | 0 | + + **Example:** + + ```shell + insmod /path/to/dim_monitor.ko measure_log_capacity=10000 measure_hash=sm3 + modprobe dim_monitor measure_log_capacity=10000 measure_hash=sm3 + ``` + +#### Kernel Interfaces + +1. dim_core interface + + | Interface | Attribute| Function | Example | + | -------------------------- | ---- | ------------------------------------------------------------ | --------------------------------------------------------- | + | measure | Write-only| Write **1** to trigger dynamic measurement. If the operation is successful, **0** is returned. If the operation fails, an error code is returned. | echo 1 > /sys/kernel/security/dim/measure | + | baseline_init | Write-only| Write **1** to trigger dynamic baseline establishment. If the operation is successful, **0** is returned. If the operation fails, an error code is returned. | echo 1 > /sys/kernel/security/dim/baseline_init | + | ascii_runtime_measurements | Read-only| Read the interface to query measurement logs. | cat /sys/kernel/security/dim/ascii_runtime_measurements | + | runtime_status | Read-only| Read the interface to query the status information. If the operation fails, an error code is returned. | cat /sys/kernel/security/dim/runtime_status | + | interval | Read/Write| Write a numeric string to set the automatic measurement interval (the value range is the same as that of **measure_interval**). Read the interface to query the current automatic measurement interval. If the query fails, an error code is returned.| echo 10 > /sys/kernel/security/dim/interval
cat /sys/kernel/security/dim/interval | + + **dim_core Statuses** + + The possible statuses of dim_core are as follows: + + - DIM_NO_BASELINE: dim_core is loaded but no operation is performed. + - DIM_BASELINE_RUNNING: The dynamic baseline is being established. + - DIM_MEASURE_RUNNING: Dynamic measurement is being performed. + - DIM_PROTECTED: The dynamic baseline has been established and is protected. + - DIM_ERROR: An error occurs during dynamic baseline establishment or dynamic measurement. You need to rectify the error and trigger dynamic baseline establishment or dynamic measurement again. + +2. dim_monitor interfaces + + | Interface | Attribute| Description | Example | + | ---------------------------------- | ---- | ---------------------------------------------- | ------------------------------------------------------------ | + | monitor_run | Write-only| Write **1** to trigger measurement. If the operation is successful, **0** is returned. If the operation fails, an error code is returned.| echo 1 > /sys/kernel/security/dim/monitor_run | + | monitor_baseline | Write-only| Write **1** to trigger baseline establishment. If the operation is successful, **0** is returned. If the operation fails, an error code is returned.| echo 1 > /sys/kernel/security/dim/monitor_baseline | + | monitor_ascii_runtime_measurements | Read-only| Read the interface to query measurement logs. | cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements | + | monitor_status | Read-only| Read the interface to query the status information. If the operation fails, an error code is returned. | cat /sys/kernel/security/dim/monitor_status | + + **dim_monitor Statuses** + + - ready: dim_monitor is loaded but no operation is performed. + - running: The dynamic baseline is being established or dynamic measurement is being performed. + - error: An error occurs during dynamic baseline establishment or dynamic measurement. You need to rectify the error and trigger dynamic baseline establishment or dynamic measurement again. + - protected: The dynamic baseline has been established and is protected. + +#### User-Mode Tool Interface + +See for the details of the `dim_gen_baseline` CLI interface. + +## Usage + +### Installing and Uninstalling DIM + +**Prerequisites**: + +- OS: openEuler 23.09 or later +- Kernel: openEuler kernel 5.10 or 6.4 + +Install dim_tools and dim software packages. The following uses openEuler 23.09 as an example: + +```shell +# yum install -y dim_tools dim +``` + +After the software packages are installed, the DIM kernel components are not loaded by default. You can run the following commands to load or unload them: + +```shell +# modprobe dim_core or insmod /path/to/dim_core.ko +# modprobe dim_monitor or insmod /path/to/dim_monitor.ko +# rmmod dim_monitor +# rmmod dim_core +``` + +After the components are loaded successfully, you can run the following commands to query the components: + +```shell +# lsmod | grep dim_core +dim_core 77824 1 dim_monitor +# lsmod | grep dim_monitor +dim_monitor 36864 0 +``` + +Unload the KO files before uninstalling the RPM package. + +```shell +# rmmod dim_monitor +# rmmod dim_core +# rpm -e dim +``` + +>![](./public_sys-resources/icon-note.gif) **NOTE:** + +dim_monitor must be loaded after dim_core and removed before dim_core. +You can also install DIM from source. For details, see . + +### Measuring Code Sections of User-Mode Processes + +**Prerequisites**: + +- The dim_core module is loaded successfully. + +- A user-mode resident measurement target program has been prepared. This section uses **/opt/dim/demo/dim_test_demo** as an example. + + ```shell + # /opt/dim/demo/dim_test_demo & + ``` + +**Step 1**: Generate a static baseline for the binary file corresponding to the measurement target process. + +```shell +# mkdir -p /etc/dim/digest_list +# dim_gen_baseline /opt/dim/demo/dim_test_demo -o /etc/dim/digest_list/test.hash +``` + +**Step 2**: Configure a measurement policy. + +```shell +# echo "measure obj=BPRM_TEXT path=/opt/dim/demo/dim_test_demo" > /etc/dim/policy +``` + +**Step 3**: Trigger dynamic baseline establishment. + +```shell +# echo 1 > /sys/kernel/security/dim/baseline_init +``` + +**Step 4**: Query measurement logs. + +```shell +# cat /sys/kernel/security/dim/ascii_runtime_measurements +0 e9a79e25f091e03a8b3972b1a0e4ae2ccaed1f5652857fe3b4dc947801a6913e sha256:02e28dff9997e1d81fb806ee5b784fd853eac8812059c4dba7c119c5e5076989 /opt/dim/demo/dim_test_demo [static baseline] +``` + +If the preceding measurement log is displayed, the target process is measured successfully and the measurement result is consistent with the static baseline. + +**Step 5**: Trigger dynamic measurement. + +```shell +# echo 1 > /sys/kernel/security/dim/measure +``` + +After the measurement is complete, you can perform **Step 4** to query the measurement logs. If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. If an attacker attempts to tamper with the target program (for example, by modifying and recompiling the code) and restart the target program, for example: + +```shell +# pkill dim_test_demo +# /opt/dim/demo/dim_test_demo & +``` + +Trigger the measurement again and query the measurement logs. You can see a measurement log marked with "tampered." + +```shell +# echo 1 > /sys/kernel/security/dim/measure +# cat /sys/kernel/security/dim/ascii_runtime_measurements +0 e9a79e25f091e03a8b3972b1a0e4ae2ccaed1f5652857fe3b4dc947801a6913e sha256:02e28dff9997e1d81fb806ee5b784fd853eac8812059c4dba7c119c5e5076989 /opt/dim/demo/dim_test_demo [static baseline] +0 08a2f6f2922ad3d1cf376ae05cf0cc507c2f5a1c605adf445506bc84826531d6 sha256:855ec9a890ff22034f7e13b78c2089e28e8d217491665b39203b50ab47b111c8 /opt/dim/demo/dim_test_demo [tampered] +``` + +### Measuring Code Sections of Kernel Modules + +**Prerequisites**: + +- The dim_core module is loaded successfully. + +- A measurement target kernel module has been prepared. This section uses dim_test_module as an example, whose path is **/opt/dim/demo/dim_test_module.ko**. + + ```shell + # insmod /opt/dim/demo/dim_test_module.ko + ``` + +>![](./public_sys-resources/icon-note.gif) **NOTE:** +> +>The kernel version of the environment where the module is compiled must be the same as the current kernel version. Run the following command to confirm: +> +>```shell +># modinfo dim_monitor.ko | grep vermagic | grep "$(uname -r)" +>vermagic: 6.4.0-1.0.1.4.oe2309.x86_64 SMP preempt mod_unload modversions +>``` + +The first field of the vermagic kernel module information must be the same as the current kernel version. + +**Step 1**: Generate a static baseline for the measurement target kernel module. + +```shell +# mkdir -p /etc/dim/digest_list +# dim_gen_baseline /opt/dim/demo/dim_test_module.ko -o /etc/dim/digest_list/test.hash +``` + +**Step 2**: Configure a measurement policy. + +```shell +# echo "measure obj=MODULE_TEXT name=dim_test_module" > /etc/dim/policy +``` + +**Step 3**: Trigger dynamic baseline establishment. + +```shell +# echo 1 > /sys/kernel/security/dim/baseline_init +``` + +**Step 4**: Query measurement logs. + +```shell +# cat /sys/kernel/security/dim/ascii_runtime_measurements +0 9603a9d5f87851c8eb7d2619f7abbe28cb8a91f9c83f5ea59f036794e23d1558 sha256:9da4bccc7ae1b709deab8f583b244822d52f3552c93f70534932ae21fac931c6 dim_test_module [static baseline] +``` + +The preceding measurement log indicates that dim_test_module is successfully measured and the current measurement result is used as the baseline value for subsequent measurement. The hash value in the measurement log is not the actual measurement value. + +**Step 5**: Trigger dynamic measurement. + +```shell +echo 1 > /sys/kernel/security/dim/measure +``` + +After the measurement is complete, you can perform **Step 4** to query the measurement logs. If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. If an attacker attempts to tamper with the target kernel module (for example, by modifying and recompiling the code) and reload the module, for example: + +```shell +rmmod dim_test_module +insmod /opt/dim/demo/dim_test_module.ko +``` + +Trigger the measurement again and query the measurement logs. You can see a measurement log marked with "tampered." + +```shell +# cat /sys/kernel/security/dim/ascii_runtime_measurements +0 9603a9d5f87851c8eb7d2619f7abbe28cb8a91f9c83f5ea59f036794e23d1558 sha256:9da4bccc7ae1b709deab8f583b244822d52f3552c93f70534932ae21fac931c6 dim_test_module [static baseline] +0 6205915fe63a7042788c919d4f0ff04cc5170647d7053a1fe67f6c0943cd1f40 sha256:4cb77370787323140cb572a789703be1a4168359716a01bf745aa05de68a14e3 dim_test_module [tampered] +``` + +### Measuring Code Sections of the Kernel + +**Prerequisites**: + +- The dim_core module is loaded successfully. + +**Step 1**: generate a static baseline for the kernel. + +```shell +# mkdir -p /etc/dim/digest_list +# dim_gen_baseline -k "$(uname -r)" -o /etc/dim/digest_list/test.hash /boot/vmlinuz-6.4.0-1.0.1.4.oe2309.x86_64 +``` + +>![](./public_sys-resources/icon-note.gif) **NOTE:** +> +>The file name **/boot/vmlinuz-6.4.0-1.0.1.4.oe2309.x86_64** is used as an example. + +**Step 2**: Configure a DIM policy. + +```shell +# echo "measure obj=KERNEL_TEXT" > /etc/dim/policy +``` + +**Step 3**: Trigger dynamic baseline establishment. + +```shell +# echo 1 > /sys/kernel/security/dim/baseline_init +``` + +**Step 4**: Query measurement logs. + +```shell +# cat /sys/kernel/security/dim/ascii_runtime_measurements +0 ef82c39d767dece1f5c52b31d1e8c7d55541bae68a97542dda61b0c0c01af4d2 sha256:5f1586e95b102cd9b9f7df3585fe13a1306cbd464f2ebe47a51ad34128f5d0af 6.4.0-1.0.1.4.oe2309.x86_64 [static baseline] +``` + +The preceding measurement log indicates that the kernel is successfully measured and the current measurement result is used as the baseline value for subsequent measurement. The hash value in the measurement log is not the actual measurement value. + +**Step 5**: Trigger dynamic measurement. + +```shell +# echo 1 > /sys/kernel/security/dim/measure +``` + +After the measurement is complete, you can perform **Step 4** to query the measurement logs. If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. + +### Measuring the dim_core Module + +**Prerequisites**: + +- The dim_core and dim_monitor modules are loaded successfully. +- The measurement policy has been configured. + +**Step 1**: Trigger dynamic baseline establishment for dim_core. + +```shell +# echo 1 > /sys/kernel/security/dim/baseline_init +``` + +**Step2**: Trigger dynamic baseline establishment for dim_monitor. + +```shell +# echo 1 > /sys/kernel/security/dim/monitor_baseline +``` + +**Step 3**: Query dim_monitor measurement logs. + +```shell +# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements +0 c1b0d9909ddb00633fc6bbe7e457b46b57e165166b8422e81014bdd3e6862899 sha256:35494ed41109ebc9bf9bf7b1c190b7e890e2f7ce62ca1920397cd2c02a057796 dim_core.text [dynamic baseline] +0 9be7121cd2c215d454db2a8aead36b03d2ed94fad0fbaacfbca83d57a410674f sha256:f35d20aae19ada5e633d2fde6e93133c3b6ae9f494ef354ebe5b162398e4d7fa dim_core.data [dynamic baseline] +``` + +The preceding measurement log indicates that dim_core is successfully measured and the current measurement result is used as the baseline value for subsequent measurement. +>![](./public_sys-resources/icon-note.gif) **NOTE:** +> +>If you skip dynamic baseline establishment and directly perform measurement, "tampered" is displayed in the log. + +**Step 4**: Trigger dynamic measurement of dim_monitor. + +```shell +# echo 1 > /sys/kernel/security/dim/monitor_run +``` + +If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. If dynamic baseline establishment for dim_core is triggered again after the policy is modified, the measurement target changes, and the baseline data managed by dim_core also changes. As a result, the dim_monitor measurement result changes. + +```shell +# echo "measure obj=BPRM_TEXT path=/usr/bin/bash" > /etc/dim/policy +# echo 1 > /sys/kernel/security/dim/baseline_init +``` + +Trigger the measurement of dim_monitor again and query the measurement logs. You can see a measurement log marked with "tampered." + +```shell +# echo 1 > /sys/kernel/security/dim/monitor_run +# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements +0 c1b0d9909ddb00633fc6bbe7e457b46b57e165166b8422e81014bdd3e6862899 sha256:35494ed41109ebc9bf9bf7b1c190b7e890e2f7ce62ca1920397cd2c02a057796 dim_core.text [dynamic baseline] +0 9be7121cd2c215d454db2a8aead36b03d2ed94fad0fbaacfbca83d57a410674f sha256:f35d20aae19ada5e633d2fde6e93133c3b6ae9f494ef354ebe5b162398e4d7fa dim_core.data [dynamic baseline] +0 6a60d78230954aba2e6ea6a6b20a7b803d7adb405acbb49b297c003366cfec0d sha256:449ba11b0bfc6146d4479edea2b691aa37c0c025a733e167fd97e77bbb4b9dab dim_core.data [tampered] +``` + +### Extending to the TPM PCR + +**Prerequisites**: + +- The TPM 2.0 has been installed in the system. After the following command is executed, the command output is not empty: + + ```shell + # ls /dev/tpm* + /dev/tpm0 /dev/tpmrm0 + ``` + +- The tpm2-tools software package has been installed in the system. After the following command is executed, the command output is not empty: + + ```shell + # rpm -qa tpm2-tools + ``` + +- The measurement policy and static baseline have been configured. + +**Step 1**: Load the dim_core and dim_monitor modules and set the numbers of the PCRs to which the measurement results are extended. In this example, PCR 12 is specified for the dim_core measurement result, and PCR 13 is specified for the dim_monitor measurement result. + +```shell +# modprobe dim_core measure_pcr=12 +# modprobe dim_monitor measure_pcr=13 +``` + +**Step 2**: Trigger baseline establishment for dim_core and dim_monitor. + +```shell +# echo 1 > /sys/kernel/security/dim/baseline_init +# echo 1 > /sys/kernel/security/dim/monitor_baseline +``` + +**Step 3**: Check the measurement logs. Each log is marked with the corresponding TPM PCR number. + +```shell +# cat /sys/kernel/security/dim/ascii_runtime_measurements +12 2649c414d1f9fcac1c8d0df8ae7b1c18b5ea10a162b957839bdb8f8415ec6146 sha256:83110ce600e744982d3676202576d8b94cea016a088f99617767ddbd66da1164 /usr/lib/systemd/systemd [static baseline] +# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements +13 72ee3061d5a80eb8547cd80c73a80c3a8dc3b3e9f7e5baa10f709350b3058063 sha256:5562ed25fcdf557efe8077e231399bcfbcf0160d726201ac8edf7a2ca7c55ad0 dim_core.text [dynamic baseline] +13 8ba44d557a9855c03bc243a8ba2d553347a52c1a322ea9cf8d3d1e0c8f0e2656 sha256:5279eadc235d80bf66ba652b5d0a2c7afd253ebaf1d03e6e24b87b7f7e94fa02 dim_core.data [dynamic baseline] +``` + +**Step 4**: Check the TPM PCRs. Extended values have been written to the corresponding PCRs. + +```shell +# tpm2_pcrread sha256 | grep "12:" + 12: 0xF358AC6F815BB29D53356DA2B4578B4EE26EB9274E553689094208E444D5D9A2 +# tpm2_pcrread sha256 | grep "13:" + 13: 0xBFB9FF69493DEF9C50E52E38B332BDA8DE9C53E90FB96D14CD299E756205F8EA +``` + +### Enabling Signature Verification + +**Prerequisites**: + +- A public key certificate and a signature private key have been prepared. The signature algorithm is RSA, the hash algorithm is sha256, and the certificate format is DER. You can also run the following commands to generate the files: + + ```shell + # openssl genrsa -out dim.key 4096 + # openssl req -new -sha256 -key dim.key -out dim.csr -subj "/C=AA/ST=BB/O=CC/OU=DD/CN=DIM Test" + # openssl x509 -req -days 3650 -signkey dim.key -in dim.csr -out dim.crt + # openssl x509 -in dim.crt -out dim.der -outform DER + ``` + +- The measurement policy has been configured. + +**Step 1**: Save the DER certificate as **/etc/keys/x509_dim.der**. + +```shell +# mkdir -p /etc/keys +# cp dim.der /etc/keys/x509_dim.der +``` + +**Step 2**: Sign the policy file and static baseline file. The signature file name must be the original file name suffixed with **.sig**. + +```shell +# openssl dgst -sha256 -out /etc/dim/policy.sig -sign dim.key /etc/dim/policy +# openssl dgst -sha256 -out /etc/dim/digest_list/test.hash.sig -sign dim.key /etc/dim/digest_list/test.hash +``` + +**Step 3**: Load the dim_core module and enable the signature verification function. + +```shell +modprobe dim_core signature=1 +``` + +Now, the policy file and static baseline file can be loaded only after they pass the signature verification. +The baseline establishment will fail if it is triggered after the policy file is modified: + +```shell +# echo "" >> /etc/dim/policy +# echo 1 > /sys/kernel/security/dim/baseline_init +-bash: echo: write error: Key was rejected by service +``` + +>![](./public_sys-resources/icon-note.gif) **NOTE:** +> +>If the signature verification of a static baseline file fails, dim_core skips the parsing of the file without causing a baseline establishment failure. + +### Configuring Measurement Algorithms + +**Prerequisites**: + +- The measurement policy has been configured. + +**Step 1**: Load the dim_core and dim_monitor modules and configure the measurement algorithm. The following uses the SM3 algorithm as an example. + +```shell +# modprobe dim_core measure_hash=sm3 +# modprobe dim_monitor measure_hash=sm3 +``` + +**Step 2**: Configure a policy that establishes a static baseline of the SM3 algorithm for the measurement target program. + +```shell +# echo "measure obj=BPRM_TEXT path=/opt/dim/demo/dim_test_demo" > /etc/dim/policy +# dim_gen_baseline -a sm3 /opt/dim/demo/dim_test_demo -o /etc/dim/digest_list/test.hash +``` + +**Step 3**: Trigger baseline establishment. + +```shell +# echo 1 > /sys/kernel/security/dim/baseline_init +# echo 1 > /sys/kernel/security/dim/monitor_baseline +``` + +**Step 4**: Check the measurement logs. Each log is marked with the corresponding hash algorithm. + +```shell +# cat /sys/kernel/security/dim/ascii_runtime_measurements +0 585a64feea8dd1ec415d4e67c33633b97abb9f88e6732c8a039064351d24eed6 sm3:ca84504c02bef360ec77f3280552c006ce387ebb09b49b316d1a0b7f43039142 /opt/dim/demo/dim_test_demo [static baseline] +# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements +0 e6a40553499d4cbf0501f32cabcad8d872416ca12855a389215b2509af76e60b sm3:47a1dae98182e9d7fa489671f20c3542e0e154d3ce941440cdd4a1e4eee8f39f dim_core.text [dynamic baseline] +0 2c862bb477b342e9ac7d4dd03b6e6705c19e0835efc15da38aafba110b41b3d1 sm3:a4d31d5f4d5f08458717b520941c2aefa0b72dc8640a33ee30c26a9dab74eae9 dim_core.data [dynamic baseline] +``` + +### Configuring Automatic Measurement + +**Prerequisites**: + +- The measurement policy has been configured. + +**Method 1**: Load the dim_core module and set the automatic measurement interval to 1 minute. + +```shell +modprobe dim_core measure_interval=1 +``` + +After the module is loaded, dynamic baseline establishment is automatically triggered. Then, dynamic measurement is triggered every minute. + +>![](./public_sys-resources/icon-note.gif) **NOTE:** +> +>In this case, do not configure the measurement policy for dim_core to measure its own code sections. Otherwise, false alarms may occur. +>In addition, you need to configure **/etc/dim/policy** in advance. Otherwise, the module fails to be loaded when **measure_interval** is set to **1**. + +**Method 2**: After the dim_core module is loaded, you can configure the measurement interval through the kernel module interface. In the following example, the measurement interval is set to 1 minute. + +```shell +modprobe dim_core +echo 1 > /sys/kernel/security/dim/interval +``` + +In this case, the measurement is not triggered immediately. One minute later, dynamic baseline establishment, or dynamic measurement, is triggered. Subsequently, dynamic measurement is triggered every minute. + +### Configuring the Measurement Scheduling Time + +**Prerequisites**: + +- The measurement policy has been configured. + +Load the dim_core module and set the measurement scheduling time to 10 ms. + +```shell +modprobe dim_core measure_schedule=10 +``` + +When the dynamic baseline establishment or dynamic measurement is triggered, dim_core schedules the CPU to release 10 ms each time a process is measured. diff --git a/docs/en/docs/Administration/IMA.md b/docs/en/docs/Administration/IMA.md new file mode 100644 index 0000000000000000000000000000000000000000..b8ce592589ca23f78cd9def38540e9177fb8684f --- /dev/null +++ b/docs/en/docs/Administration/IMA.md @@ -0,0 +1,1367 @@ +# Kernel Integrity Measurement Architecture (IMA) + +## Overview + +### Introduction to IMA + +IMA is a kernel subsystem that measures files accessed through system calls like `execve()`, `mmap()`, and `open()` based on custom policies. These measurements can be used for **local or remote attestation** or compared against reference values to **control file access**. + +IMA operates in two main modes: + +- Measurement: This mode observes the integrity of files. When protected files are accessed, measurement records are added to the measurement log in kernel memory. If the system has a Trusted Platform Module (TPM), the measurement digest can also be extended into the TPM platform configuration registers (PCRs) to ensure the integrity of the measurement data. This mode does not restrict file access but provides recorded file information to upper-layer applications for remote attestation. +- Appraisal: This mode verifies file integrity, preventing access to unknown or tampered files. It uses cryptographic methods like hashes, signatures, and hash-based message authentication codes (HMACs) to validate file contents. If verification fails, the file is inaccessible to any process. This feature enhances system resilience by isolating compromised files and preventing further damage during an attack. + +In summary, the measurement mode acts as a passive observer, while the appraisal mode enforces strict access control, blocking any file that fails integrity checks. + +### Introduction to EVM + +EVM, or Extended Verification Module, builds on IMA capabilities. While IMA protects file contents, EVM extends this protection to file attributes such as `uid`, `security.ima`, and `security.selinux`. + +### Introduction to IMA Digest Lists + +IMA digest lists enhance the native integrity protection mechanism of the kernel in openEuler, addressing key limitations of the native IMA/EVM mechanism: + +**File access performance degradation due to extension to TPM** + +In IMA measurement mode, each measurement requires accessing the TPM. Since the TPM operates at low speeds, typically using the Serial Peripheral Interface (SPI) protocol with clock frequencies in the tens of MHz, system call performance suffers. + +![](./figures/ima_tpm.png) + +**File access performance degradation due to asymmetric operations** + +In IMA appraisal mode, immutable files are protected using a signature mechanism. Each file verification requires signature validation, and the complexity of asymmetric operations further degrades system call performance. + +![](./figures/ima_sig_verify.png) + +**Decreased efficiency and security from complex deployment** + +In IMA appraisal mode, deployment requires the system to first enter fix mode to mark IMA/EVM extended attributes, then switch to appraisal mode for startup. Additionally, upgrading protected files necessitates rebooting into fix mode to update files and extended attributes. This process reduces deployment efficiency and exposes keys in the runtime environment, compromising security. + +![](./figures/ima_priv_key.png) + +IMA digest lists address these issues by managing baseline digest values for multiple files through a single hash list file. This file consolidates the baseline digest values of files (such as all executables in a software package) for centralized management. The baseline digest values can include file content digests (for IMA mode) and file extended attribute digests (for EVM mode). This consolidated file is the IMA digest list file. + +![](./figures/ima_digest_list_pkg.png) + +When the IMA digest list feature is enabled, the kernel maintains a hash allowlist pool to store digest values from imported IMA digest list files. It also provides interfaces via securityfs for importing, deleting, and querying these files. + +In measurement mode, imported digest list files must undergo measurement and extension to TPM before being added to the allowlist pool. If the digest value of a target file matches the allowlist pool, no additional measurement logging or extension to TPM is required. In appraisal mode, imported digest list files must pass signature verification before being added to the allowlist pool. The digest value of the accessed target file is then matched against the allowlist pool to determine the appraisal result. + +![](./figures/ima_digest_list_flow.png) + +Compared to the native Linux IMA/EVM mechanism, the IMA digest list extension enhances security, performance, and usability for better practical implementation: + +- Security: IMA digest lists are distributed with software packages. During installation, digest lists are imported simultaneously, ensuring baseline values originate from the software distributor (like the openEuler community). This eliminates the need to generate baseline values in the runtime environment, establishing a complete trust chain. +- Performance: The IMA digest list mechanism operates on a per-digest-list basis, reducing TPM access and asymmetric operation frequency to 1/n (where n is the average number of file hashes managed by a single digest list). This improves system call and boot performance. +- Usability: The IMA digest list mechanism supports out-of-the-box functionality, allowing the system to enter appraisal mode immediately after installation. It also enables software package installation and upgrades in appraisal mode without requiring fix mode for file marking, facilitating rapid deployment and seamless updates. + +It is worth noting that, unlike the native IMA/EVM, the IMA digest list stores baseline values for measurement/appraisal in kernel memory. This assumes that kernel memory cannot be tampered with by unauthorized entities. Therefore, the IMA digest list relies on additional security mechanisms (including kernel module secure boot and runtime memory measurement) to safeguard kernel memory integrity. + +However, both the native IMA mechanism and the IMA digest list extension are only components of the system security chain. Neither can independently ensure system security. Security is inherently a systematic, defense-in-depth engineering effort. + +## Interface Description + +### Kernel Boot Parameters + +The openEuler IMA/EVM mechanism provides the following kernel boot parameters. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ParameterValueFunction
ima_appraiseenforce-evmEnable IMA appraisal enforce mode (EVM enabled).
log-evmEnable IMA appraisal log mode (EVM enabled).
enforceEnable IMA appraisal enforce mode.
logEnable IMA appraisal log mode.
offDisable IMA appraisal.
ima_appraise_digest_listdigestEnable IMA+EVM appraisal based on digest lists (comparing file content and extended attributes).
digest-nometadataEnable IMA appraisal based on digest lists (comparing file content only).
evmx509Enable portable signature-based EVM directly (regardless of EVM certificate loading)
completePrevent modification of EVM mode via securityfs after boot.
allow_metadata_writesAllow file metadata modifications without EVM interception.
ima_hashsha256/sha1/...Specify the IMA measurement hash algorithm.
ima_templateimaSpecify the IMA measurement template (d or n).
ima-ngSpecify the IMA measurement template (d-ng or n-ng), default template.
ima-sigSpecify the IMA measurement template (d-ng, n-ng, or sig).
ima_policyexec_tcbMeasure all files accessed via execution or mapping, including loaded kernel modules, firmware, and kernel files.
tcbExtend exec_tcb policy to measure files accessed with uid=0 or euid=0.
secure_bootAppraise all loaded kernel modules, firmware, and kernel files, using IMA signature mode.
appraise_exec_tcbExtend secure_boot policy to appraise all files accessed via execution or mapping.
appraise_tcbAppraise all files owned by uid=0.
appraise_exec_immutableUsed with the appraise_exec_tcb policy, making executable file extended attributes immutable.
ima_digest_list_pcr10Extend IMA measurement results based on digest list in PCR 10, disable native IMA measurement.
11Extend IMA measurement results based on digest list in PCR 11, disable native IMA measurement.
+11Extend IMA measurement results based on digest list in PCR 11, extend native IMA measurement results in PCR 10.
ima_digest_db_sizenn[M]Set kernel digest list size limit (0 to 64 MB), defaulting to 16 MB if not configured. ("Not configured" means to omit the parameter, not leaving the value cannot blank like ima_digest_db_size=.)
ima_capacity-1 to 2147483647Set the kernel measurement log entry limit, defaulting to 100,000. -1 means no limit.
initramtmpfsNoneSupport tmpfs in initrd to carry file extended attributes.
+ +Based on user scenarios, the following parameter combinations are recommended: + +**(1) Native IMA measurement** + +```ini +# Native IMA measurement + custom policy +# No configuration required. This is enabled by default. +# Native IMA measurement + TCB default policy +ima_policy="tcb" +``` + +**(2) IMA measurement based on digest list** + +```ini +# Digest list IMA measurement + custom policy +ima_digest_list_pcr=11 ima_template=ima-ng initramtmpfs +# Digest list IMA measurement + default policy +ima_digest_list_pcr=11 ima_template=ima-ng ima_policy="exec_tcb" initramtmpfs +``` + +**(3) IMA appraisal based on digest list, protecting file content only** + +```ini +# IMA appraisal + log mode +ima_appraise=log ima_appraise_digest_list=digest-nometadata ima_policy="appraise_exec_tcb" initramtmpfs +# IMA appraisal + enforce mode +ima_appraise=enforce ima_appraise_digest_list=digest-nometadata ima_policy="appraise_exec_tcb" initramtmpfs +``` + +**(4) IMA appraisal based on digest list, protecting file content and extended attributes** + +```ini +# IMA appraisal + log mode +ima_appraise=log-evm ima_appraise_digest_list=digest ima_policy="appraise_exec_tcb|appraise_exec_immutable" initramtmpfs evm=x509 evm=complete +# IMA appraisal + enforce mode +ima_appraise=enforce-evm ima_appraise_digest_list=digest ima_policy="appraise_exec_tcb|appraise_exec_immutable" initramtmpfs evm=x509 evm=complete +``` + +> ![](./public_sys-resources/icon-note.gif) **Note:** +> +> All four parameter sets above can be used individually, but only digest list-based measurement and appraisal modes can be combined, such as (2) with (3) or (2) with (4). + +### securityfs Interface Description + +The securityfs interfaces provided by openEuler IMA are located in the **/sys/kernel/security** directory. Below are the interface names and their descriptions. + +| Path | Permissions | Description | +| :----------------------------- | :---------- | :---------------------------------------------------------------------- | +| ima/policy | 600 | Display or import IMA policies. | +| ima/ascii_runtime_measurement | 440 | Display IMA measurement logs in ASCII format. | +| ima/binary_runtime_measurement | 440 | Display IMA measurement logs in binary format. | +| ima/runtime_measurement_count | 440 | Display the count of IMA measurement log entries. | +| ima/violations | 440 | Display the number of abnormal IMA measurement logs. | +| ima/digests_count | 440 | Display the total number of digests in the system hash table (IMA+EVM). | +| ima/digest_list_data | 200 | Add digest lists. | +| ima/digest_list_data_del | 200 | Delete digest lists. | +| evm | 660 | Query or set EVM mode. | + +The **/sys/kernel/security/evm** interface supports the following values: + +- `0`: EVM is not initialized. +- `1`: Use HMAC (symmetric encryption) to verify extended attribute integrity. +- `2`: Use public key signature verification (asymmetric encryption) to verify extended attribute integrity. +- `6`: Disable extended attribute integrity verification. + +### Digest List Management Tool Description + +The digest-list-tools package includes tools for generating and managing IMA digest list files. The primary CLI tools are as follows. + +#### gen_digest_lists + +The `gen_digest_lists` tool allows users to generate digest lists. The command options are defined below. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OptionValueFunction
-d<path>Specify the directory to store the generated digest list files. The directory must be valid.
-fcompactSpecify the format of the generated digest list files. Currently, only the compact format is supported.
-i<option arg>:<option value>Define the target file range for generating digest lists. Specific parameters are listed below.
I:<path>Specify the absolute path of files for which digest lists will be generated. If a directory is specified, recursive generation is performed.
E:<path>Specify paths or directories to exclude.
F:<path>Specify paths or directories to generate digest lists for all files under them (ignores the filtering effect of e: when combined with e:).
e:Generate digest lists only for executable files.
l:policyMatch file security contexts from the SELinux policy instead of reading them directly from file extended attributes.
i:Include the file digest value in the calculated extended attribute information when generating metadata-type digest lists (required).
M:Allow explicit specification of file extended attribute information (requires use with the rpmbuild command).
u:Use the list file name specified by the L: parameter as the name of the generated digest list file (requires use with the rpmbuild command).
L:<path>Specify the path to the list file, which contains the information data required to generate digest lists (requires use with the rpmbuild command).
-oaddSpecify the operation for generating digest lists. Currently, only the add operation is supported, which adds the digest list to the file.
-p-1Specify the position in the file where the digest list will be written. Currently, only -1 is supported.
-tfileGenerate digest lists only for file content.
metadataGenerate digest lists for both file content and extended attributes.
-TN/AIf this option is not used, digest list files are generated. Otherwise, TLV digest list files are generated.
-A<path>Specify the relative root directory to truncate the file path prefix for path matching and SELinux label matching.
-mimmutableSpecify the modifiers attribute for the generated digest list files. Currently, only immutable is supported. In enforce/enforce-evm mode, digest lists can only be opened in read-only mode.
-hN/APrint help information.
+ +**Usage examples** + +- Scenario 1: Generate a digest list/TLV digest list for a single file. + + ```shell + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ls -d ./ -i i: + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ls -d ./ -i i: -T + ``` + +- Scenario 2: Generate a digest list/TLV digest list for a single file and specify a relative root directory. + + ```shell + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ls -A /usr/ -d ./ -i i: + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ls -A /usr/ -d ./ -i i: -T + ``` + +- Scenario 3: Recursively generate a digest list/TLV digest list for files in a directory. + + ```shell + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ -d ./ -i i: + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ -d ./ -i i: -T + ``` + +- Scenario 4: Recursively generate a digest list/TLV digest list for executable files in a directory. + + ```shell + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ -d ./ -i i: -i e: + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/bin/ -d ./ -i i: -i e: -T + ``` + +- Scenario 5: Recursively generate a digest list/TLV digest list for files in a directory, excluding specific subdirectories. + + ```shell + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/ -d ./ -i i: -i E:/usr/bin/ + gen_digest_lists -t metadata -f compact -i l:policy -o add -p -1 -m immutable -i I:/usr/ -d ./ -i i: -i E:/usr/bin/ -T + ``` + +- Scenario 6: In an `rpmbuild` callback script, generate a digest list by reading the list file passed by `rpmbuild`. + + ```shell + gen_digest_lists -i M: -t metadata -f compact -d $DIGEST_LIST_DIR -i l:policy \ + -i i: -o add -p -1 -m immutable -i L:$BIN_PKG_FILES -i u: \ + -A $RPM_BUILD_ROOT -i e: \ + -i E:/usr/src \ + -i E:/boot/efi \ + -i F:/lib \ + -i F:/usr/lib \ + -i F:/lib64 \ + -i F:/usr/lib64 \ + -i F:/lib/modules \ + -i F:/usr/lib/modules \ + -i F:/lib/firmware \ + -i F:/usr/lib/firmware + + gen_digest_lists -i M: -t metadata -f compact -d $DIGEST_LIST_DIR.tlv \ + -i l:policy -i i: -o add -p -1 -m immutable -i L:$BIN_PKG_FILES -i u: \ + -T -A $RPM_BUILD_ROOT -i e: \ + -i E:/usr/src \ + -i E:/boot/efi \ + -i F:/lib \ + -i F:/usr/lib \ + -i F:/lib64 \ + -i F:/usr/lib64 \ + -i F:/lib/modules \ + -i F:/usr/lib/modules \ + -i F:/lib/firmware \ + -i F:/usr/lib/firmware + ``` + +#### manage_digest_lists + +The `manage_digest_lists` tool is designed to parse and convert binary-format TLV digest list files into a human-readable text format. Below are the command options. + +| Option | Value | Function | +| ------ | ------------ | ------------------------------------------------------------------------------------------------------------ | +| -d | \ | Specify the directory containing the TLV digest list files. | +| -f | \ | Specify the name of the TLV digest list file. | +| -p | dump | Define the operation type. Currently, only `dump` is supported, which parses and prints the TLV digest list. | +| -v | N/A | Print verbose details. | +| -h | N/A | Display help information. | + +**Usage example** + +View TLV digest list information. + +```ini +manage_digest_lists -p dump -d /etc/ima/digest_lists.tlv/ +``` + +## File Format Description + +### IMA Policy File Syntax Description + +The IMA policy file is a text-based file that can include multiple rule statements separated by newline characters (`\n`). Each rule statement must begin with an **action** keyword, followed by one or more **filter conditions**: + +```text + [filter condition 2] [filter condition 3]... +``` + +The action keyword defines the specific action for the policy. Only one action is allowed per policy. Refer to the table below for specific actions (note that the `action=` prefix can be omitted in practice, for example, use `dont_measure` instead of `action=dont_measure`). + +Supported filter conditions include: + +- `func`: indicates the type of file to be measured or appraised. It is typically used with `mask`. Only one `func` is allowed per policy. + - `FILE_CHECK` can only be paired with `MAY_EXEC`, `MAY_WRITE`, or `MAY_READ`. + - `MODULE_CHECK`, `MMAP_CHECK`, and `BPRM_CHECK` can only be paired with `MAY_EXEC`. + - Other combinations will not take effect. + +- `mask`: specifies the operation on the file that triggers measurement or appraisal. Only one `mask` is allowed per policy. + +- `fsmagic`: represents the hexadecimal magic number of the file system type, as defined in **/usr/include/linux/magic.h** (by default, all file systems are measured unless excluded using `dont_measure` or `dont_appraise`). + +- `fsuuid`: represents the 16-character hexadecimal string of the system device UUID. + +- `objtype`: specifies the file security type. Only one file type is allowed per policy. Compared to `func`, `objtype` offers finer granularity. For example, `obj_type=nova_log_t` refers to files with the SELinux type `nova_log_t`. + +- `uid`: specifies the user (by user ID) performing the operation on the file. Only one `uid` is allowed per policy. + +- `fowner`: specifies the file owner (by user ID). Only one `fowner` is allowed per policy. + +The keywords are detailed as follows. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KeywordValueDescription
actionmeasureEnable IMA measurement.
dont_measureDisable IMA measurement.
appraiseEnable IMA appraisal.
dont_appraiseDisable IMA appraisal.
auditEnable auditing.
funcFILE_CHECKFiles to be opened
MODULE_CHECKKernel module files to be loaded
MMAP_CHECKShared library files to be mapped into process memory
BRPM_CHECKFiles to be executed (excluding script files opened via programs like /bin/hash)
POLICY_CHECKIMA policy files to be imported
FIRMWARE_CHECKFirmware to be loaded into memory
DIGEST_LIST_CHECKDigest list files to be loaded into the kernel
KEXEC_KERNEL_CHECKKernel to be switched to using kexec
maskMAY_EXECExecute a file.
MAY_WRITEWrite to a file.
MAY_READRead a file.
MAY_APPENDAppend to a file.
fsmagicfsmagic=xxxHexadecimal magic number representing the file system type
fsuuidfsuuid=xxx16-character hexadecimal string representing the system device UUID
fownerfowner=xxxUser ID of the file owner
uiduid=xxxUser ID of the user performing the operation on the file
obj_typeobj_type=xxx_tFile type based on SELinux labels
pcrpcr=\PCR in TPM for extending measurements (defaulting to 10)
appraise_typeimasigIMA appraisal based on signatures
meta_immutableAppraisal based on file extended attributes with signatures (supporting digest lists)
+ +## Usage Instructions + +> ![](./public_sys-resources/icon-note.gif) **Note:** + +Native IMA/EVM is an open source Linux feature. This section offers a concise overview of its basic usage. For further details, consult the open source wiki: + +[https://sourceforge.net/p/linux-ima/wiki/Home/](https://sourceforge.net/p/linux-ima/wiki/Home/) + +### Native IMA Usage Instructions + +#### IMA Measurement Mode + +To enable IMA measurement, configure the measurement policy. + +**Step 1:** Specify the measurement policy by configuring boot parameters or manually. For example, configure the IMA policy via boot parameters as follows: + +```ini +ima_policy="tcb" +``` + +Alternatively, manually configure the IMA policy like this: + +```shell +echo "measure func=BPRM_CHECK" > /sys/kernel/security/ima/policy +``` + +**Step 2:** Reboot the system. You can then check the measurement log in real time to monitor the current measurement status: + +```shell +cat /sys/kernel/security/ima/ascii_runtime_measurements +``` + +#### IMA Appraisal Mode + +Enter fix mode, complete IMA labeling for files, and then enable log or enforce mode. + +**Step 1:** Configure boot parameters and reboot to enter fix mode: + +```ini +ima_appraise=fix ima_policy=appraise_tcb +``` + +**Step 2:** Generate IMA extended attributes for all files requiring appraisal: + +For immutable files (such as binary program files), use signature mode to write the signature of the file digest value into the IMA extended attribute. For example (where **/path/to/ima.key** is the signing private key matching the IMA certificate): + +```shell +find /usr/bin -fstype ext4 -type f -executable -uid 0 -exec evmctl -a sha256 ima_sign --key /path/to/ima.key '{}' \; +``` + +For mutable files (such as data files), use hash mode to write the file digest value into the IMA extended attribute. IMA supports an automatic labeling mechanism, where accessing the file in fix mode will generate the IMA extended attribute: + +```shell +find / -fstype ext4 -type f -uid 0 -exec dd if='{}' of=/dev/null count=0 status=none \; +``` + +To verify if the file has been successfully labeled with the IMA extended attribute (`security.ima`), use: + +```shell +getfattr -m - -d /sbin/init +``` + +**Step 3:** Configure boot parameters, switch the IMA appraisal mode to log or enforce, and reboot the system: + +```ini +ima_appraise=enforce ima_policy=appraise_tcb +``` + +### IMA Digest List Usage Instructions + +#### Prerequisites + +Before using the IMA digest list feature, install the ima-evm-utils and digest-list-tools packages: + +```shell +yum install ima-evm-utils digest-list-tools +``` + +#### Mechanism Overview + +##### Digest List Files + +After installing RPM packages released by openEuler, digest list files are automatically generated in the **/etc/ima** directory. The following types of files exist: + +**/etc/ima/digest_lists/0-metadata_list-compact-\** + +This is the IMA digest list file, generated using the `gen_digest_lists` command (see [gen_digest_lists](#gen_digest_lists) for details). This binary file contains header information and a series of SHA256 hash values, representing the digest values of legitimate file contents and file extended attributes. Once measured or appraised, this file is imported into the kernel, and IMA digest list measurement or appraisal is performed based on the allowlist digest values in this file. + +**/etc/ima/digest_lists/0-metadata_list-rpm-\** + +This is the RPM digest list file, **essentially the header information of the RPM package**. After the RPM package is installed, if the IMA digest list file does not contain a signature, the RPM header information is written into this file, and the signature of the header information is written into the `security.ima` extended attribute. This allows the authenticity of the RPM header information to be verified through the signature. Since the RPM header information includes the digest value of the digest list file, indirect verification of the digest list is achieved. + +**/etc/ima/digest_lists/0-parser_list-compact-libexec** + +This is the IMA parser digest list file, storing the digest value of the **/usr/libexec/rpm_parser** file. This file is used to establish a trust chain from the RPM digest list to the IMA digest list. The kernel IMA digest list mechanism performs special verification on processes generated by this file. If the process is confirmed to be the `rpm_parser` program, all digest lists imported by it are trusted without requiring signature verification. + +**/etc/ima/digest_lists.sig/0-metadata_list-compact-\.sig** + +This is the signature file for the IMA digest list. If this file is included in the RPM package, its content is written into the `security.ima` extended attribute of the corresponding RPM digest list file during the RPM installation phase. This enables signature verification during the IMA digest list import phase. + +**/etc/ima/digest_lists.tlv/0-metadata_list-compact_tlv-\** + +This is the TLV digest list file, typically generated alongside the IMA digest list file for target files. It stores the integrity information of the target files (such as file content digest values and file extended attributes). The purpose of this file is to assist users in querying or restoring the integrity information of target files. + +##### Digest List File Signing Methods + +In IMA digest list appraisal mode, the IMA digest list file must undergo signature verification before it can be imported into the kernel and used for subsequent file whitelist matching. The IMA digest list file supports the following signing methods. + +**(1) IMA extended attribute signature** + +This is the native IMA signing mechanism, where the signature information is stored in the `security.ima` extended attribute in a specific format. It can be generated and added using the `evmctl` command: + +```shell +evmctl ima_sign --key /path/to/ima.key -a sha256 +``` + +Alternatively, the `-f` parameter can be added to store the signature and header information in a separate file: + +```shell +evmctl ima_sign -f --key /path/to/ima.key -a sha256 +``` + +When IMA digest list appraisal mode is enabled, you can directly write the path of a a digest list file to the kernel interface to import or delete it. This process automatically triggers appraisal, performing signature verification on the digest list file content based on the `security.ima` extended attribute: + +```shell +# Import the IMA digest list file. +echo > /sys/kernel/security/ima/digest_list_data +# Delete the IMA digest list file. +echo > /sys/kernel/security/ima/digest_list_data_del +``` + +**(2) IMA digest list appended signature (default in openEuler 24.03 LTS)** + +Starting with openEuler 24.03 LTS, IMA-specific signing keys are supported, and Cryptographic Message Syntax (CMS) signing is used. Since the signature information includes a certificate chain, it may exceed the length limit for the `security.ima` extended attribute. Therefore, a signature appending method similar to kernel module insertion is adopted: + + + +The signing mechanism is as follows: + +1. Append the CMS signature information to the end of the IMA digest list file. + +2. Fill in the structure and append it to the end of the signature information. The structure is defined as follows: + + ```c + struct module_signature { + u8 algo; /* Public-key crypto algorithm [0] */ + u8 hash; /* Digest algorithm [0] */ + u8 id_type; /* Key identifier type [PKEY_ID_PKCS7] */ + u8 signer_len; /* Length of signer's name [0] */ + u8 key_id_len; /* Length of key identifier [0] */ + u8 __pad[3]; + __be32 sig_len; /* Length of signature data */ + }; + ``` + +3. Add the `"~Module signature appended~\n"` magic string. + +A reference script for this step is as follows: + +```shell +#!/bin/bash +DIGEST_FILE=$1 # Path to the IMA digest list file +SIG_FILE=$2 # Path to save the IMA digest list signature information +OUT=$3 # Output path for the digest list file after adding the signature information + +cat $DIGEST_FILE $SIG_FILE > $OUT +echo -n -e "\x00\x00\x02\x00\x00\x00\x00\x00" >> $OUT +echo -n -e $(printf "%08x" "$(ls -l $SIG_FILE | awk '{print $5}')") | xxd -r -ps >> $OUT +echo -n "~Module signature appended~" >> $OUT +echo -n -e "\x0a" >> $OUT +``` + +**(3) Reusing RPM signatures (default in openEuler 22.03 LTS)** + +openEuler 22.03 LTS supports reusing the RPM signing mechanism to sign IMA digest list files. This aims to address the lack of dedicated IMA signing keys in the version. Users do not need to be aware of this signing process. When an RPM package contains an IMA digest list file but no IMA digest list signature file, this signing mechanism is automatically used. The core principle is to verify the IMA digest list through the RPM package header information. + +For RPM packages released by openEuler, each package file can consist of two parts: + +- **RPM header information:** Stores RPM package attribute fields, including the package name and file digest list. The integrity of this information is ensured by the RPM header signature. +- **RPM files:** The actual files installed into the system, including IMA digest list files generated during the build phase. + + + +During RPM package installation, if the RPM process detects that the digest list file in the package does not contain a signature, it creates an RPM digest list file in the **/etc/ima** directory, writes the RPM header information into the file content, and writes the RPM header signature into the `security.ima` extended attribute of the file. This allows indirect verification and import of the IMA digest list through the RPM digest list. + +##### IMA Digest List Import + +When IMA measurement mode is enabled, importing IMA digest list files does not require signature verification. The file path can be directly written to the kernel interface to import or delete the digest list: + +```shell +# Import the IMA digest list file. +echo > /sys/kernel/security/ima/digest_list_data +# Delete the IMA digest list file. +echo > /sys/kernel/security/ima/digest_list_data_del +``` + +When IMA appraisal mode is enabled, importing a digest list requires signature verification. Depending on the signing method, there are two import approaches. + +**Direct import** + +For IMA digest list files that already contain signature information (IMA extended attribute signature or IMA digest list appended signature), the file path can be directly written to the kernel interface to import or delete the digest list. This process automatically triggers appraisal, performing signature verification on the digest list file content based on the `security.ima` extended attribute: + +```shell +# Import the IMA digest list file. +echo > /sys/kernel/security/ima/digest_list_data +# Delete the IMA digest list file. +echo > /sys/kernel/security/ima/digest_list_data_del +``` + +**Using `upload_digest_lists` for import** + +For IMA digest list files that reuse RPM signatures, the `upload_digest_lists` command must be used for import. The specific commands are as follows (note that the specified path should point to the corresponding RPM digest list): + +```shell +# Import the IMA digest list file. +upload_digest_lists add +# Delete the IMA digest list file. +upload_digest_lists del +``` + +This process is relatively complex and requires the following prerequisites: + +1. The system has already imported the digest lists (including IMA digest lists and IMA PARSER digest lists) from the `digest_list_tools` package released by openEuler. + +2. An IMA appraisal policy for application execution (`BPRM_CHECK` policy) has been configured. + +#### Operation Guide + +##### Automatic Digest List Generation During RPM Build + +The openEuler RPM toolchain supports the `%__brp_digest_list` macro, which is configured as follows: + +```text +%__brp_digest_list /usr/lib/rpm/brp-digest-list %{buildroot} +``` + +When this macro is configured, the **/usr/lib/rpm/brp-digest-list** script is invoked when a user runs the `rpmbuild` command to build a package. This script handles the generation and signing of digest lists. By default, openEuler generates digest lists for critical files such as executables, dynamic libraries, and kernel modules. YOU can also modify the script to customize the scope of digest list generation and specify signing keys. The following example uses a user-defined signing key (**/path/to/ima.key**) to sign the digest list. + +```shell +...... (line 66) +DIGEST_LIST_TLV_PATH="$DIGEST_LIST_DIR.tlv/0-metadata_list-compact_tlv-$(basename $BIN_PKG_FILES)" +[ -f $DIGEST_LIST_TLV_PATH ] || exit 0 + +chmod 644 $DIGEST_LIST_TLV_PATH +echo $DIGEST_LIST_TLV_PATH + +evmctl ima_sign -f --key /path/to/ima.key -a sha256 $DIGEST_LIST_PATH &> /dev/null +chmod 400 $DIGEST_LIST_PATH.sig +mkdir -p $DIGEST_LIST_DIR.sig +mv $DIGEST_LIST_PATH.sig $DIGEST_LIST_DIR.sig +echo $DIGEST_LIST_DIR.sig/0-metadata_list-compact-$(basename $BIN_PKG_FILES).sig +``` + +##### IMA Digest List Measurement + +Enable IMA digest list measurement using the following steps: + +**Step 1:** Configure the boot parameters to enable IMA measurement. The process is similar to **native IMA measurement**, but a specific TPM PCR must be configured for measurement. An example boot parameter is as follows: + +```ini +ima_policy=exec_tcb ima_digest_list_pcr=11 +``` + +**Step 2:** Import the IMA digest list. For example, using the digest list of the Bash package: + +```shell +echo /etc/ima/digest_lists/0-metadata_list-compact-bash-5.1.8-6.oe2203sp1.x86_64 > /sys/kernel/security/ima/digest_list_data +``` + +The IMA digest list measurement logs can be queried as follows: + +```shell +cat /sys/kernel/security/ima/ascii_runtime_measurements +``` + +After importing the IMA digest list, if the digest value of the measured file is included in the IMA digest list, no additional measurement logs will be recorded. + +##### IMA Digest List Appraisal + +###### Boot With the Default Policy + +You can configure the `ima_policy` parameter in the boot parameters to specify the IMA default policy. After IMA initialization during kernel startup, the default policy is immediately applied for appraisal. You can enable IMA digest list appraisal using the following steps: + +**Step 1:** Execute the `dracut` command to write the digest list file into initrd: + +```shell +dracut -f -e xattr +``` + +**Step 2:** Configure the boot parameters and IMA policy. Typical configurations are as follows: + +```ini +# IMA appraisal in log/enforce mode based on digest lists, protecting only file content, with the default policy set to appraise_exec_tcb +ima_appraise=log ima_appraise_digest_list=digest-nometadata ima_policy="appraise_exec_tcb" initramtmpfs module.sig_enforce +ima_appraise=enforce ima_appraise_digest_list=digest-nometadata ima_policy="appraise_exec_tcb" initramtmpfs module.sig_enforce +# IMA appraisal in log/enforce mode based on digest lists, protecting file content and extended attributes, with the default policy set to appraise_exec_tcb+appraise_exec_immutable +ima_appraise=log-evm ima_appraise_digest_list=digest ima_policy="appraise_exec_tcb|appraise_exec_immutable" initramtmpfs evm=x509 evm=complete module.sig_enforce +ima_appraise=enforce-evm ima_appraise_digest_list=digest ima_policy="appraise_exec_tcb|appraise_exec_immutable" initramtmpfs evm=x509 evm=complete module.sig_enforce +``` + +Reboot the system to enable IMA digest list appraisal. During the boot process, the IMA policy will take effect, and the IMA digest list files will be automatically imported. + +###### Boot Without a Default Policy + +You can omit the `ima_policy` parameter in the boot parameters, indicating that no default policy is applied during system startup. The IMA appraisal mechanism will wait for the you to import a policy before becoming active. + +**Step 1:** Configure the boot parameters. Typical configurations are as follows: + +```ini +# IMA appraisal in log/enforce mode based on digest lists, protecting only file content, with no default policy +ima_appraise=log ima_appraise_digest_list=digest-nometadata initramtmpfs +ima_appraise=enforce ima_appraise_digest_list=digest-nometadata initramtmpfs +# IMA appraisal in log/enforce mode based on digest lists, protecting file content and extended attributes, with no default policy +ima_appraise=log-evm ima_appraise_digest_list=digest initramtmpfs evm=x509 evm=complete +ima_appraise=enforce-evm ima_appraise_digest_list=digest initramtmpfs evm=x509 evm=complete +``` + +Reboot the system. At this point, since no policy is configured, IMA appraisal will not be active. + +**Step 2:** Import the IMA policy by writing the full path of the policy file to the kernel interface: + +```shell +echo /path/to/policy > /sys/kernel/security/ima/policy +``` + +> ![](./public_sys-resources/icon-note.gif) **Note:** +> +> The policy must include certain fixed rules. Refer to the following policy templates. +> +> For openEuler 22.03 LTS (reusing RPM signatures): + +```ini +# Do not appraise access behavior for the securityfs file system. +dont_appraise fsmagic=0x73636673 +# Other user-defined dont_appraise rules +...... +# Appraise imported IMA digest list files. +appraise func=DIGEST_LIST_CHECK appraise_type=imasig +# Appraise all files opened by the /usr/libexec/rpm_parser process. +appraise parser appraise_type=imasig +# Appraise executed applications (triggering appraisal for /usr/libexec/rpm_parser execution, additional conditions such as SELinux labels can be added). +appraise func=BPRM_CHECK appraise_type=imasig +# Other user-defined appraise rules +...... +``` + +> For openEuler 24.03 LTS (IMA extended attribute signatures or appended signatures): + +```ini +# User-defined dont_appraise rules +...... +# Appraise imported IMA digest list files. +appraise func=DIGEST_LIST_CHECK appraise_type=imasig|modsig +# Other user-defined appraise rules. +...... +``` + +**Step 3:** Import the IMA digest list files. Different import methods are required for digest lists with different signing methods. + +> For openEuler 22.03 LTS (IMA digest lists reusing RPM signatures): + +```ini +# Import digest lists from the digest_list_tools package +echo /etc/ima/digest_lists/0-metadata_list-compact-digest-list-tools-0.3.95-13.x86_64 > /sys/kernel/security/ima/digest_list_data +echo /etc/ima/digest_lists/0-parser_list-compact-libexec > /sys/kernel/security/ima/digest_list_data +# Import other RPM digest lists +upload_digest_lists add /etc/ima/digest_lists +# Check the number of imported digest list entries +cat /sys/kernel/security/ima/digests_count +``` + +> For openEuler 24.03 LTS (IMA digest lists with appended signatures): + +```shell +find /etc/ima/digest_lists -name "0-metadata_list-compact-*" -exec echo {} > /sys/kernel/security/ima/digest_list_data \; +``` + +##### Software Upgrade + +After the IMA digest list feature is enabled, files covered by IMA protection require synchronized updates to their digest lists during upgrades. For RPM packages released by openEuler, the addition, update, and deletion of digest lists within the RPM packages are automatically handled during package installation, upgrade, or removal, without requiring manual user intervention. For user-maintained non-RPM packages, manual import of digest lists is necessary. + +##### User Certificate Import + +You can import custom certificates to measure or appraise software not released by openEuler. The openEuler IMA appraisal mode supports signature verification using certificates from the following two key rings: + +- **builtin_trusted_keys**: root certificates pre-configured during kernel compilation +- **ima**: imported via **/etc/keys/x509_ima.der** in initrd, which must be a subordinate certificate of any certificate in the `builtin_trusted_keys` key ring + +**Steps to import a root certificate into the builtin_trusted_keys key ring:** + +**Step 1:** Generate a root certificate using the `openssl` command: + +```shell +echo 'subjectKeyIdentifier=hash' > root.cfg +openssl genrsa -out root.key 4096 +openssl req -new -sha256 -key root.key -out root.csr -subj "/C=AA/ST=BB/O=CC/OU=DD/CN=openeuler test ca" +openssl x509 -req -days 3650 -extfile root.cfg -signkey root.key -in root.csr -out root.crt +openssl x509 -in root.crt -out root.der -outform DER +``` + +**Step 2:** Obtain the openEuler kernel source code, using the latest OLK-5.10 branch as an example: + +```shell +git clone https://gitee.com/openeuler/kernel.git -b OLK-5.10 +``` + +**Step 3:** Navigate to the source code directory and copy the root certificate into it: + +```shell +cd kernel +cp /path/to/root.der . +``` + +Modify the `CONFIG_SYSTEM_TRUSTED_KEYS` option in the config file: + +```shell +CONFIG_SYSTEM_TRUSTED_KEYS="./root.crt" +``` + +**Step 4:** Compile and install the kernel (steps omitted; ensure digest lists are generated for kernel modules). + +**Step 5:** Verify certificate import after rebooting: + +```shell +keyctl show %:.builtin_trusted_keys +``` + +**Steps to import a subordinate certificate into the ima key ring (requires the root certificate to be imported into the builtin_trusted_keys key ring first):** + +**Step 1:** Generate a subordinate certificate based on the root certificate using the `openssl` command: + +```shell +echo 'subjectKeyIdentifier=hash' > ima.cfg +echo 'authorityKeyIdentifier=keyid,issuer' >> ima.cfg +echo 'keyUsage=digitalSignature' >> ima.cfg +openssl genrsa -out ima.key 4096 +openssl req -new -sha256 -key ima.key -out ima.csr -subj "/C=AA/ST=BB/O=CC/OU=DD/CN=openeuler test ima" +openssl x509 -req -sha256 -CAcreateserial -CA root.crt -CAkey root.key -extfile ima.cfg -in ima.csr -out ima.crt +openssl x509 -outform DER -in ima.crt -out x509_ima.der +``` + +**Step 2:** Copy the IMA certificate to the **/etc/keys** directory: + +```shell +mkdir -p /etc/keys/ +cp x509_ima.der /etc/keys/ +``` + +**Step 3:** Package initrd, embedding the IMA certificate and digest lists into the initrd image: + +```shell +echo 'install_items+=" /etc/keys/x509_ima.der "' >> /etc/dracut.conf +dracut -f -e xattr +``` + +**Step 4:** Verify certificate import after rebooting: + +```shell +keyctl show %:.ima +``` + +#### Typical Use Cases + +Depending on the operating mode, IMA digest lists can be applied to trusted measurement scenarios and user-space secure boot scenarios. + +##### Trusted Measurement Scenario + +The trusted measurement scenario primarily relies on the IMA digest list measurement mode, where the kernel and hardware root of trust (RoT) like the TPM jointly measure critical files. This is combined with a remote attestation toolchain to prove the trusted state of current system files: + +![](./figures/ima_trusted_measurement.png) + +**Runtime phase** + +- During software package deployment, digest lists are imported synchronously. IMA measures the digest lists and records the measurement logs (synchronously extended to the TPM). + +- When an application is executed, IMA measurement is triggered. If the file digest matches the allowlist, it is ignored. Otherwise, the measurement log is recorded (synchronously extended to the TPM). + +**Attestation phase (industry standard process)** + +- The remote attestation server sends an attestation request, and the client returns the IMA measurement logs along with the signed TPM PCR values. + +- The remote attestation server sequentially verifies the PCR (signature verification), measurement logs (PCR replay), and file measurement information (comparison with local baseline values), reporting the results to the security center. + +- The security management center takes corresponding actions, such as event notification or node isolation. + +##### User-Space Secure Boot Scenario + +The user-space secure boot scenario primarily relies on the IMA digest list appraisal mode, similar to secure boot. It aims to perform integrity checks on executed applications or accessed critical files. If the check fails, access is denied: + +![](./figures/ima_secure_boot.png) + +**Runtime phase** + +- During application deployment, digest lists are imported. After the kernel verifies the signature, the digest values are loaded into the kernel hash table as an allowlist. + +- When an application is executed, IMA verification is triggered. The file hash value is calculated, and if it matches the baseline value, access is allowed. Otherwise, the event is logged or access is denied. + +## Appendix + +### Kernel Compilation Options + +The compilation options provided by native IMA/EVM and their descriptions are as follows. + +| Compilation Option | Functionality | +| :------------------------------- | :------------------------------------------- | +| CONFIG_INTEGRITY | Enable IMA/EVM compilation. | +| CONFIG_INTEGRITY_SIGNATURE | Enable IMA signature verification. | +| CONFIG_INTEGRITY_ASYMMETRIC_KEYS | Enable IMA asymmetric signature verification. | +| CONFIG_INTEGRITY_TRUSTED_KEYRING | Enable IMA/EVM keyring. | +| CONFIG_INTEGRITY_AUDIT | Compile IMA audit module. | +| CONFIG_IMA | Enable IMA. | +| CONFIG_IMA_WRITE_POLICY | Allow updating IMA policies during runtime. | +| CONFIG_IMA_MEASURE_PCR_IDX | Allow specifying IMA measurement PCR index. | +| CONFIG_IMA_LSM_RULES | Allow configuring LSM rules. | +| CONFIG_IMA_APPRAISE | Enable IMA appraisal. | +| IMA_APPRAISE_BOOTPARAM | Enable IMA appraisal boot parameters. | +| CONFIG_EVM | Enable EVM. | + +The compilation options provided by the openEuler IMA digest list feature and their descriptions are as follows (enabled by default in openEuler kernel compilation). + +| Compilation Option | Functionality | +| :----------------- | :----------------------------- | +| CONFIG_DIGEST_LIST | Enable the IMA digest list feature. | + +### IMA Digest List Root Certificate + +In openEuler 22.03, the RPM key pair is used to sign IMA digest lists. To ensure the IMA feature is usable out-of-the-box, the openEuler kernel compilation process imports the RPM root certificate (PGP certificate) into the kernel by default. This includes the OBS certificate used in older versions and the openEuler certificate introduced in openEuler 22.03 LTS SP1: + +```shell +# cat /proc/keys | grep PGP +1909b4ad I------ 1 perm 1f030000 0 0 asymmetri private OBS b25e7f66: PGP.rsa b25e7f66 [] +2f10cd36 I------ 1 perm 1f030000 0 0 asymmetri openeuler fb37bc6f: PGP.rsa fb37bc6f [] +``` + +Since the current kernel does not support importing PGP subkeys, and the openEuler certificate uses subkey signing, the openEuler kernel preprocesses the certificate before compilation by extracting the subkey and importing it into the kernel. The specific process can be found in the [**process_pgp_certs.sh**](https://gitee.com/src-openeuler/kernel/blob/openEuler-22.03-LTS-SP1/process_pgp_certs.sh) script in the kernel package repository + +Starting from openEuler 24.03, dedicated IMA certificates are supported. For details, refer to the relevant section in [Introduction to Signature Certificates](../CertSignature/introduction_to_signature_certificates.md). + +If you do not intend to use the IMA digest list feature or prefer other keys for signing/verification, you can remove the related code and implement your own kernel root certificate configuration. + +### FAQ + +#### FAQ 1: System Fails to Boot After IMA Appraisal Enforce Mode Is Enabled with the Default Policy + +The default IMA policy may include checks for critical file access processes such as application execution and kernel module loading. If access to these critical files fails, the system may fail to boot. Common causes include: + +1. The IMA verification certificate is not imported into the kernel, causing the digest list to fail verification. +2. The digest list file is not correctly signed, leading to verification failure. +3. The digest list file is not imported into the initrd, preventing the digest list from being loaded during the boot process. +4. The digest list file does not match the application, causing the application to fail matching the imported digest list. + +Enter the system in log mode to locate and fix the issue. Reboot the system, enter the GRUB menu, and modify the boot parameters to start in log mode: + +```ini +ima_appraise=log +``` + +After the system boots, follow the steps below to troubleshoot. + +**Step 1:** Check the IMA certificates in the key ring. + +```shell +keyctl show %:.builtin_trusted_keys +``` + +For openEuler LTS versions, at least the following kernel certificates should exist (for other versions, reference based on their release dates): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VersionCertificate
openEuler 22.03 LTSprivate OBS b25e7f66
openEuler 22.03 LTS SP1/2/3private OBS b25e7f66
openeuler <openeuler@compass-ci.com> b675600b
openEuler 22.03 LTS SP4private OBS b25e7f66
openeuler <openeuler@compass-ci.com> b675600b
openeuler <openeuler@compass-ci.com> fb37bc6f
openEuler 24.03openEuler kernel ICA 1: 90bb67eb4b57eb62bf6f867e4f56bd4e19e7d041
+ +If you have imported other kernel root certificates, use the `keyctl` command to confirm whether the certificates were successfully imported. By default, openEuler does not use the IMA key ring. If you are using it, check whether the user certificates exist in the IMA key ring with the following command: + +```shell +keyctl show %:.ima +``` + +If the issue is that the certificate was not correctly imported, refer to [User Certificate Import](#user-certificate-import) for troubleshooting. + +**Step 2:** Check if the digest list contains signature information. + +Query the digest list files in the current system with the following command: + +```shell +ls /etc/ima/digest_lists | grep '_list-compact-' +``` + +For each digest list file, ensure that **one of the following three** signature conditions is met: + +1. The digest list file has a corresponding **RPM digest list file**, and the `security.ima` extended attribute of the **RPM digest list file** contains a signature value. For example, for the bash package digest list, the digest list file path is: + + ```text + /etc/ima/digest_lists/0-metadata_list-compact-bash-5.1.8-6.oe2203sp1.x86_64 + ``` + + The RPM digest list path is: + + ```text + /etc/ima/digest_lists/0-metadata_list-rpm-bash-5.1.8-6.oe2203sp1.x86_64 + ``` + + Check the RPM digest list signature by ensuring the `security.ima` extended attribute is not empty: + + ```shell + getfattr -n security.ima /etc/ima/digest_lists/0-metadata_list-rpm-bash-5.1.8-6.oe2203sp1.x86_64 + ``` + +2. The `security.ima` extended attribute of the digest list file is not empty: + + ```shell + getfattr -n security.ima /etc/ima/digest_lists/0-metadata_list-compact-bash-5.1.8-6.oe2203sp1.x86_64 + ``` + +3. The digest list file contains signature information at the end. Verify if the file content ends with the `~Module signature appended~` magic string (supported in openEuler 24.03 LTS and later versions): + + ```shell + tail -c 28 /etc/ima/digest_lists/0-metadata_list-compact-kernel-6.6.0-28.0.0.34.oe2403.x86_64 + ``` + +If the issue is that the digest list does not contain signature information, refer to [Digest List File Signing Methods](#digest-list-file-signing-methods) for troubleshooting. + +**Step 3:** Verify the correctness of the digest list signature. + +After ensuring that the digest list contains signature information, also ensure that the digest list is signed with the correct private key, meaning the signing private key matches the certificate in the kernel. In addition to manually checking the private key, users can check the dmesg logs or audit logs (default path: **/var/log/audit/audit.log**) for signature verification failures. A typical log output is as follows: + +```ini +type=INTEGRITY_DATA msg=audit(1722578008.756:154): pid=3358 uid=0 auid=0 ses=1 subj=unconfined_u:unconfined_r:haikang_t:s0-s0:c0.c1023 op=appraise_data cause=invalid-signature comm="bash" name="/root/0-metadata_list-compact-bash-5.1.8-6.oe2203sp1.x86_64" dev="dm-0" ino=785161 res=0 errno=0UID="root" AUID="root" +``` + +If the issue is incorrect signature information, refer to [Digest List File Signing Methods](#digest-list-file-signing-methods) for troubleshooting. + +**Step 4:** Check if the digest list file is imported into the initrd. + +Query whether the digest list file exists in the current initrd with the following command: + +```shell +lsinitrd | grep 'etc/ima/digest_lists' +``` + +If no digest list file is found, users need to recreate the initrd and verify that the digest list is successfully imported: + +```shell +dracut -f -e xattr +``` + +**Step 5:** Verify that the IMA digest list matches the application. + +Refer to [FAQ 2](#faq-2-file-execution-fails-after-ima-appraisal-enforce-mode-is-enabled). + +#### FAQ 2: File Execution Fails After IMA Appraisal Enforce Mode Is Enabled + +After IMA appraisal enforce mode is enabled, if the content or extended attributes of a file configured with IMA policies are incorrect (for example, they do not match the imported digest list), file access may be denied. Common causes include: + +1. The digest list was not successfully imported (refer to [FAQ 1](#faq-1-system-fails-to-boot-after-ima-appraisal-enforce-mode-is-enabled-with-the-default-policy) for details). +2. The file content or attributes have been tampered with. + +For scenarios where file execution fails, first ensure that the digest list file has been successfully imported into the kernel. Check the number of digest lists to determine the import status: + +```shell +cat /sys/kernel/security/ima/digests_count +``` + +Next, use the audit logs (default path: **/var/log/audit/audit.log**) to identify which file failed verification and the reason. A typical log output is as follows: + +```ini +type=INTEGRITY_DATA msg=audit(1722811960.997:2967): pid=7613 uid=0 auid=0 ses=1 subj=unconfined_u:unconfined_r:haikang_t:s0-s0:c0.c1023 op=appraise_data cause=IMA-signature-required comm="bash" name="/root/test" dev="dm-0" ino=814424 res=0 errno=0UID="root" AUID="root" +``` + +After identifying the file that failed verification, compare it with the TLV digest list to determine the cause of tampering. For scenarios where extended attribute verification is not enabled, only compare the SHA256 hash value of the file with the `IMA digest` entry in the TLV digest list. For scenarios where extended attribute verification is enabled, also compare the current file attributes with the extended attributes displayed in the TLV digest list. + +Once the cause of the issue is determined, resolve it by restoring the file content and attributes or regenerating the digest list for the file, signing it, and importing it into the kernel. + +#### FAQ 3: Errors Occur During Packages Installation Across openEuler 22.03 LTS SP Versions After IMA Appraisal Mode Is Enabled + +After IMA appraisal mode is enabled, installing packages from different SP versions of openEuler 22.03 LTS triggers the import of IMA digest lists. This process includes a signature verification step, where the kernel uses its certificates to verify the digest list signatures. Due to changes in signing certificates during the evolution of openEuler, backward compatibility issues may arise in certain cross-SP-version installation scenarios (there are no forward compatibility issues, meaning newer kernels can verify older IMA digest list files without problems). + +You are advised to ensure that the following signing certificates are present in the current kernel: + +```shell +# keyctl show %:.builtin_trusted_keys +Keyring + 566488577 ---lswrv 0 0 keyring: .builtin_trusted_keys + 383580336 ---lswrv 0 0 \_ asymmetric: openeuler b675600b + 453794670 ---lswrv 0 0 \_ asymmetric: private OBS b25e7f66 + 938520011 ---lswrv 0 0 \_ asymmetric: openeuler fb37bc6f +``` + +If any certificates are missing, you are advised to upgrade the kernel to the latest version: + +```shell +yum update kernel +``` + +openEuler 24.03 LTS and later versions include dedicated IMA certificates and support certificate chain verification, ensuring the certificate lifecycle covers the entire LTS version. + +#### FAQ 4: IMA Digest List Import Fails Despite Correct Signatures After IMA Digest List Appraisal Mode Is Enabled + +The IMA digest list import process includes a verification mechanism. If a digest list fails signature verification during import, the digest list import functionality is disabled, preventing even correctly signed digest lists from being imported afterward. Check the dmesg logs for the following message to confirm if this is the cause: + +```shell +# dmesg +ima: 0-metadata_list-compact-bash-5.1.8-6.oe2203sp1.x86_64 not appraised, disabling digest lists lookup for appraisal +``` + +If such a log is present, a digest list file with an incorrect signature was imported while IMA digest list appraisal mode was enabled, causing the functionality to be disabled. In this case, reboot the system and fix the incorrect digest list signature information. + +#### FAQ 5: Importing User-Defined IMA Certificates Fails in openEuler 24.03 LTS and Later Versions + +Linux kernel 6.6 introduced additional field validation restrictions for importing certificates. Certificates imported into the IMA key ring must meet the following constraints (following the X.509 standard format): + +- It must be a digital signature certificate, meaning the `keyUsage=digitalSignature` field must be set. +- It must not be a CA certificate, meaning the `basicConstraints=CA:TRUE` field must not be set. +- It must not be an intermediate certificate, meaning the `keyUsage=keyCertSign` field must not be set. + +#### FAQ 6: kdump Service Fails to Start After IMA Appraisal Mode Is Enabled + +After IMA appraisal enforce mode is enabled, if the IMA policy includes the following `KEXEC_KERNEL_CHECK` rule, the kdump service may fail to start: + +```shell +appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig +``` + +The reason is that in this scenario, all files loaded via `kexec` must undergo integrity verification. As a result, the kernel restricts the loading of kernel image files by kdump to the `kexec_file_load` system call. This can be enabled by modifying the **/etc/sysconfig/kdump** configuration file: + +```shell +KDUMP_FILE_LOAD="on" +``` + +Additionally, the `kexec_file_load` system call itself performs signature verification on the files. Therefore, the kernel image file being loaded must contain a valid secure boot signature, and the current kernel must include the corresponding verification certificate. diff --git a/docs/en/docs/Administration/TPCM.md b/docs/en/docs/Administration/TPCM.md new file mode 100644 index 0000000000000000000000000000000000000000..e92a4bbb8183fb04dc992708257fcbce2e10a9a2 --- /dev/null +++ b/docs/en/docs/Administration/TPCM.md @@ -0,0 +1,37 @@ +# Trusted Platform Control Module + +## Background + +Trusted computing has undergone continuous development and improvement in the past 40 years and has become an important branch of information security. Trusted computing technologies have developed rapidly in recent years and have solved the challenges in Trusted Computing 2.0—integration of trusted systems and existing systems, trusted management, and simplification of trusted application development. These technical breakthroughs form Trusted Computing 3.0, that is, trusted computing based on an active immune system. Compared with the passive plug-in architecture of the previous generation, Trusted Computing 3.0 proposes a new trusted system framework based on self-controlled cryptography algorithms, control chips, trusted software, trusted connections, policy management, and secure and trusted protection applications, implementing trust across the networks. + +The trusted platform control module (TPCM) is a base and core module that can be integrated into a trusted computing platform to establish and ensure a trust source. As one of the innovations in Trusted Computing 3.0 and the core of active immunity, TPCM implements active control over the entire platform. + +The TPCM-based Trusted Computing 3.0 architecture consists of the protection module and the computing module. On the one hand, based on the Trusted Cryptography Module (TPM), the TPCM main control firmware measures the reliability of the protection and computing modules, as well as their firmware. On the other hand, the Trusted Software Base (TSB) measures the reliability of system software and application software. In addition, the TPCM management platform verifies the reliability measurement and synchronizes and manages the trust policies. + +## Feature Description + +The overall system design consists of the protection module, computing module, and trusted management center software, as shown in the following figure. + +![](./figures/TPCM.png) + +- Trusted management center: This centralized management platform, provided by a third-party vendor, formulates, delivers, maintains, and stores protection policies and reference values for trusted computing nodes. +- Protection module: This module operates independently of the computing module and provides trusted computing protection functions that feature active measurement and active control to implement security protection during computing. The protection module consists of the TPCM main control firmware, TCB, and TCM. As a key module for implementing trust protection in a trusted computing node, the TPCM can be implemented in multiple forms, such as cards, chips, and IP cores. It contains a CPU and memory, firmware, and software such as an OS and trusted function components. The TPCM operates alongside the computing module and works according to the built-in protection policy to monitor the trust of protected resources, such as hardware, firmware, and software of the computing module. The TPCM is the Root of Trust in a trusted computing node. + +- Computing module: This module includes hardware, an OS, and application layer software. The running of the OS can be divided into the boot phase and the running phase. In the boot phase, GRUB2 and shim of openEuler support the reliability measurement capability, which protects boot files such as shim, GRUB2, kernel, and initramfs. In the running phase, openEuler supports the deployment of the trusted verification agent (provided by third-party vendor HTTC). The agent sends data to the TPCM for trusted measurement and protection in the running phase. + +The TPCM interacts with other components as follows: + +1. The TPCM hardware, firmware, and software provide an operating environment for the TSB. The trusted function components of the TPCM provide support for the TSB to implement measurement, control, support, and decision-making based on the policy library interpretation requirements. +2. The TPCM accesses the TCM for trusted cryptography functions to complete computing tasks such as trusted verification, measurement, and confidential storage, and provides services for TCM access. +3. The TPCM connects to the trusted management center through the management interface to implement protection policy management and trusted report processing. +4. The TPCM uses the built-in controller and I/O port to interact with the controller of the computing module through the bus to actively monitor the computing module. +5. The built-in protection agent in the OS of the computing module obtains the code and data related to the preset protection object and provides them to the TPCM. The TPCM forwards the monitoring information to the TSB, and the TSB analyzes and processes the information according to the policy library. + +## Constraints + +Supported server: TaiShan 200 server (model 2280) +Supported BMC card: BC83SMMC + +## Application Scenarios + +The TPCM enables a complete trust chain to ensure that the OS boots into a trusted computing environment. diff --git a/docs/en/docs/Administration/figures/AT_CHECK_Process.png b/docs/en/docs/Administration/figures/AT_CHECK_Process.png new file mode 100644 index 0000000000000000000000000000000000000000..b49d6a31c8ae637f53a307cdcd554f19cd100124 Binary files /dev/null and b/docs/en/docs/Administration/figures/AT_CHECK_Process.png differ diff --git a/docs/en/docs/Administration/figures/Process_Of_EXECVEAT_ATCHECK.png b/docs/en/docs/Administration/figures/Process_Of_EXECVEAT_ATCHECK.png new file mode 100644 index 0000000000000000000000000000000000000000..ac9649ade2a4abddc68d670b70e3717b12bacc3b Binary files /dev/null and b/docs/en/docs/Administration/figures/Process_Of_EXECVEAT_ATCHECK.png differ diff --git a/docs/en/docs/Administration/figures/ima-modsig.png b/docs/en/docs/Administration/figures/ima-modsig.png new file mode 100644 index 0000000000000000000000000000000000000000..c3e54e27b6ce30bd21e97908b6168a73f318c117 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima-modsig.png differ diff --git a/docs/en/docs/Administration/figures/ima_digest_list_flow.png b/docs/en/docs/Administration/figures/ima_digest_list_flow.png new file mode 100644 index 0000000000000000000000000000000000000000..11711ca21c6b327c3d347ad4c389d037a6c2c6ae Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_digest_list_flow.png differ diff --git a/docs/en/docs/Administration/figures/ima_digest_list_pkg.png b/docs/en/docs/Administration/figures/ima_digest_list_pkg.png new file mode 100644 index 0000000000000000000000000000000000000000..8a2128add583d3c25ee5f281bb882c94f23b97c7 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_digest_list_pkg.png differ diff --git a/docs/en/docs/Administration/figures/ima_priv_key.png b/docs/en/docs/Administration/figures/ima_priv_key.png new file mode 100644 index 0000000000000000000000000000000000000000..c939b8e2e8bcd30869f938161ea1edbccd9c89c4 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_priv_key.png differ diff --git a/docs/en/docs/Administration/figures/ima_rpm.png b/docs/en/docs/Administration/figures/ima_rpm.png new file mode 100644 index 0000000000000000000000000000000000000000..6c4b620ded02ee96357eb587890555af5a319e51 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_rpm.png differ diff --git a/docs/en/docs/Administration/figures/ima_secure_boot.png b/docs/en/docs/Administration/figures/ima_secure_boot.png new file mode 100644 index 0000000000000000000000000000000000000000..85b959ff1da0f4bcf919f6fea712a0c053f7ad01 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_secure_boot.png differ diff --git a/docs/en/docs/Administration/figures/ima_sig_verify.png b/docs/en/docs/Administration/figures/ima_sig_verify.png new file mode 100644 index 0000000000000000000000000000000000000000..e0d7ff55ab93dca65763881ba4ff136b85521123 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_sig_verify.png differ diff --git a/docs/en/docs/Administration/figures/ima_tpm.png b/docs/en/docs/Administration/figures/ima_tpm.png new file mode 100644 index 0000000000000000000000000000000000000000..931440ebdb8a8c993a2f9ef331b214b40d8f9535 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_tpm.png differ diff --git a/docs/en/docs/Administration/figures/ima_trusted_measurement.png b/docs/en/docs/Administration/figures/ima_trusted_measurement.png new file mode 100644 index 0000000000000000000000000000000000000000..e64224fdf4b99429aeabb87947de2c9f23f1df14 Binary files /dev/null and b/docs/en/docs/Administration/figures/ima_trusted_measurement.png differ diff --git a/docs/en/docs/Administration/figures/root_of_trust_framework.png b/docs/en/docs/Administration/figures/root_of_trust_framework.png new file mode 100644 index 0000000000000000000000000000000000000000..84d08d0b6ed9ef3b62d99627db59599ea44d59d7 Binary files /dev/null and b/docs/en/docs/Administration/figures/root_of_trust_framework.png differ diff --git a/docs/en/docs/Administration/kernel-rot.md b/docs/en/docs/Administration/kernel-rot.md new file mode 100644 index 0000000000000000000000000000000000000000..68f1ca9d4f2169cb1eedbb0b66f7e23e916dff27 --- /dev/null +++ b/docs/en/docs/Administration/kernel-rot.md @@ -0,0 +1,97 @@ +# Kernel Root of Trust Framework + +## Overview + +Common attack methods often target the authenticity and integrity of information systems. Hardware roots of trust (RoTs) have become a standard method for protecting critical system components, enabling the system to measure and verify integrity. When tampering or counterfeiting is detected, the system triggers alerts or blocks the activity. + +The prevailing protection approach in uses the trusted platform module (TPM) as the RoT, combined with the integrity measurement software stack to establish a system trust chain that ensures system authenticity and integrity. openEuler supports the following integrity measurement features: + +- Measured boot: During the system boot phase, the digest values of boot components are measured and recorded in platform configuration register (PCRs). +- IMA measurement: During the file access phase, the digest values of files are measured and extended to PCRs. +- Dynamic integrity measurement (DIM): During the process execution phase, the digest values of memory code segments are measured and extended to PCRs. + +Recent advancements in trusted computing and confidential computing have produced diverse hardware RoTs and associated attestation systems. Examples include: + +- TCG Trusted Platform Module (TPM) +- Trusted Cryptography Module (TCM) +- Trusted Platform Control Module (TPCM) +- Intel Trust Domain Extensions (TDX) +- Arm Confidential Compute Architecture (CCA) +- Virtualized Arm Confidential Compute Architecture (virtCCA) + +Thee RoT framework used by openEuler 24.03 LTS SP1 offers a unified measurement interface to the upper-layer integrity protection software stack. Deployed within the kernel integrity subsystem, the framework supports multiple RoT drivers and expands integrity measurement beyond the TPM to include heterogeneous RoTs. + +root_of_trust_framework + +## Feature Description + +This feature implements hash-extension type measurement RoTs using multiple measurement registers that support various hash algorithms. The measurement results are recorded using the following formula: + +```text +value_new = hash(value_old | measure_result) +``` + +The formula shows how new values are calculated, where `value_new` and `value_old` represent the updated and current values in the measurement register, `measure_result` is the latest measurement, and `hash` indicates the specific hash algorithm used. + +Developers can define and register RoT instances through the provided framework layer. Once registered, the integrity measurement feature automatically feeds measurement results into these instances, handling both hash extension and register updates. + +## Feature Scope + +Implementation is available in openEuler 24.03 LTS SP1 (kernel 6.6). The southbound interface currently supports TPM and virtCCA as RoTs, while the northbound interface supports kernel IMA. Support for additional RoTs and measurement features is under development. + +## Interface Description + +### Structure Interface + +Each RoT instance requires the following structure definition: + +```c +struct ima_rot { + const char *name; + int nr_allocated_banks; + struct tpm_bank_info *allocated_banks; + + int (*init)(struct ima_rot *rot); + int (*extend)(struct tpm_digest *digests_arg, const void *args); + int (*calc_boot_aggregate)(struct ima_digest_data *hash); +}; +``` + +Member variables: + +| Member | Description | +| ------------------- | -------------------------------------------------------------- | +| name | Name of the RoT device | +| nr_allocated_banks | Number of measurement registers supported by the RoT | +| allocated_banks | Algorithm definition for RoT measurement registers | +| init | Implementation of the RoT initialization function | +| extend | Implementation of the RoT extension function | +| calc_boot_aggregate | Implementation of the boot aggregate value calculation for IMA | + +The interface array is defined in the `ima_rots` variable within **security/integrity/ima/ima_rot.c**. Adding new RoT instances to this array enables IMA feature extension for different RoTs. + +### Boot Parameters + +This feature adds the following boot parameters: + +| Parameter | Type | Behavior | +| --------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ima_rot= | string | Designates the preferred RoT device for IMA. Falls back to TPM if device unavailable. No RoT becomes active if both specified and default devices fail initialization. | + +## Configuration Example + +To configure IMA measurement with virtCCA RoTs in confidential VMs, add this boot parameter: + +```ini +ima_rot=virtcca +``` + +**Note:** Omit this parameter if `virtcca` is the only available RoT (for example, no vTPM present). + +Post-configuration: + +- Initial IMA boot aggregate log reflects measurements from the Reference Integrity Measurement (RIM) of virtCCA. +- Subsequent measurement hashes extend the `REM[0]` register of virtCCA. +- Measurement validation leverages the remote attestation framework of virtCCA. + +After configuration, the first IMA measurement log (boot aggregate log) stores its measurement value in virtCCA RIM. The hash values from subsequent IMA measurement logs undergo hash extension in virtCCA `REM[0]`. Users can verify application measurements within confidential virtual machines through the remote attestation software stack that virtCCA provides. diff --git a/docs/en/docs/Administration/remote-attestation-kunpeng-security-library.md b/docs/en/docs/Administration/remote-attestation-kunpeng-security-library.md new file mode 100644 index 0000000000000000000000000000000000000000..9d6cb0b519514e2261d6ce2569c6ee41bb02d6cf --- /dev/null +++ b/docs/en/docs/Administration/remote-attestation-kunpeng-security-library.md @@ -0,0 +1,417 @@ +# Remote Attestation (Kunpeng Security Library) + +## Introduction + +This project develops basic security software components running on Kunpeng processors. In the early stage, the project focuses on trusted computing fields such as remote attestation to empower security developers in the community. + +## Software Architecture + +On the platform without TEE enabled, this project can provide the platform remote attestation feature, and its software architecture is shown in the following figure: + +![img](./figures/RA-arch-1.png) + +On the platform that has enabled TEE, this project can provide TEE remote attestation feature, and its software architecture is shown in the following figure: + +![img](./figures/RA-arch-2.png) + +## Installation and Configuration + +1. Run the following command to use the RPM package of the Yum installation program: + + ```shell + yum install kunpengsecl-ras kunpengsecl-rac kunpengsecl-rahub kunpengsecl-qcaserver kunpengsecl-attester kunpengsecl-tas kunpengsecl-devel + ``` + +2. Prepare the database environment. Go to the `/usr/share/attestation/ras` directory and run the p`prepare-database-env.sh` script to automatically configure the database environment. + +3. The configuration files required for program running are stored in three paths: current path `./config.yaml`, home path `${HOME}/.config/attestation/ras(rac)(rahub)(qcaserver)(attester)(tas)/config.yaml`, and system path `/etc/attestation/ras(rac)(rahub)(qcaserver)(attester)(tas)/config.yaml`. + +4. (Optional) To create a home directory configuration file, run the `prepare-ras(rac)(hub)(qca)(attester)(tas)conf-env.sh` script in `/usr/share/attestation/ras(rac)(rahub)(qcaserver)(attester)(tas)` after installing the RPM package. + +## Options + +### RAS Boot Options + +Run the `ras` command to start the RAS program. Note that you need to provide the ECDSA public key in the current directory and name it `ecdsakey.pub`. Options are as follows: + +```console + -H --https HTTP/HTTPS mode switch. The default value is https(true), false=http. + -h --hport RESTful API port listened by RAS in HTTPS mode. + -p, --port string Client API port listened by RAS. + -r, --rest string RESTful API port listened by RAS in HTTP mode. + -T, --token Generates a verification code for test and exits. + -v, --verbose Prints more detailed RAS runtime log information. + -V, --version Prints the RAS version and exits. +``` + +### RAC Boot Options + +Run the `sudo raagent` command to start the RAC program. Note that the sudo permission is required to enable the physical TPM module. Options are as follows: + +```console + -s, --server string Specifies the RAS service port to be connected. + -t, --test Starts in test mode. + -v, --verbose Prints more detailed RAC runtime log information. + -V, --version Prints the RAC version and exits. + -i, --imalog Specifies the path of the IMA file. + -b, --bioslog Specifies the path of the BIOS file. + -T, --tatest Starts in TA test mode. +``` + +**Note:** +>1.To use TEE remote attestation feature, you must start RAC not in TA test mode. And place the uuid, whether to use TCB, mem_hash and img_hash of the TA to be attestated sequentially in the **talist** file under the RAC execution path. At the same time, pre install the **libqca.so** and **libteec.so** library provided by the TEE team. The format of the **talist** file is as follows: +> +>```text +>e08f7eca-e875-440e-9ab0-5f381136c600 false ccd5160c6461e19214c0d8787281a1e3c4048850352abe45ce86e12dd3df9fde 46d5019b0a7ffbb87ad71ea629ebd6f568140c95d7b452011acfa2f9daf61c7a +>``` +> +>2.To not use TEE remote attestation feature, you must copy the **libqca.so** and **libteec.so** library in `${DESTDIR}/usr/share/attestation/qcaserver` path to `/usr/lib` or `/usr/lib64` path, and start RAC in TA test mode. + +### QCA Boot Options + +Run the `${DESTDIR}/usr/bin/qcaserver` command to start the QCA program. Note that to start QTA normally, the full path of qcaserver must be used, and the CA path parameter in QTA needs to be kept the same as the path. Options are as follows: + +```console + -C, --scenario int Sets the application scenario of the program, The default value is sce_no_as(0), 1=sce_as_no_daa, 2=sce_as_with_daa. + -S, --server string Specifies the open server address/port. +``` + +### ATTESTER Boot Options + +Run the `attester` command to start the ATTESTER program. Options are as follows: + +```console + -B, --basevalue string Sets the base value file read path + -M, --mspolicy int Sets the measurement strategy, which defaults to -1 and needs to be specified manually. 1=compare only img-hash values, 2=compare only hash values, and 3=compare both img-hash and hash values at the same time. + -S, --server string Specifies the address of the server to connect to. + -U, --uuid int Specifies the trusted apps to verify. + -V, --version Prints the program version and exit. + -T, --test Reads fixed nonce values to match currently hard-coded trusted reports. +``` + +### TAS Boot Options + +Run the `tas` command to start the TAS program. Options are as follows: + +```console + -T, --token Generates a verification code for test and exits. +``` + +**Note:** +>1.To enable the TAS, you must configure the private key for TAS. Run the following command to modify the configuration file in the home directory: +> +>```shell +>$ cd ${HOME}/.config/attestation/tas +>$ vim config.yaml +> # The values of the following DAA_GRP_KEY_SK_X and DAA_GRP_KEY_SK_Y are for testing purposes only. +> # Be sure to update their contents to ensure safety before normal use. +>tasconfig: +> port: 127.0.0.1:40008 +> rest: 127.0.0.1:40009 +> akskeycertfile: ./ascert.crt +> aksprivkeyfile: ./aspriv.key +> huaweiitcafile: ./Huawei IT Product CA.pem +> DAA_GRP_KEY_SK_X: 65a9bf91ac8832379ff04dd2c6def16d48a56be244f6e19274e97881a776543c65a9bf91ac8832379ff04dd2c6def16d48a56be244f6e19274e97881a776543c +> DAA_GRP_KEY_SK_Y: 126f74258bb0ceca2ae7522c51825f980549ec1ef24f81d189d17e38f1773b56126f74258bb0ceca2ae7522c51825f980549ec1ef24f81d189d17e38f1773b56 +>``` +> +>Then enter `tas` to start TAS program. +> +>2.In an environment with TAS, in order to improve the efficiency of QCA's certificate configuration process, not every boot needs to access the TAS to generate the certificate, but through the localized storage of the certificate. That is, read the certification path configured in `config.yaml` on QCA side, check if a TAS-issued certificate has been saved locally through the `func hasAKCert(s int) bool` function. If the certificate is successfully read, there is no need to access TAS. If the certificate cannot be read, you need to access TAS and save the certificate returned by TAS locally. + +## API Definition + +### RAS APIs + +To facilitate the administrator to manage the target server, RAS and the user TA in the TEE deployed on the target server, the following APIs are designed for calling: + +| API | Method | +| --------------------------------- | --------------------------- | +| / | GET | +| /{id} | GET, POST, DELETE | +| /{from}/{to} | GET | +| /{id}/reports | GET | +| /{id}/reports/{reportid} | GET, DELETE | +| /{id}/basevalues | GET | +| /{id}/newbasevalue | POST | +| /{id}/basevalues/{basevalueid} | GET, POST, DELETE | +| /{id}/ta/{tauuid}/status | GET | +| /{id}/ta/{tauuid}/tabasevalues | GET | +| /{id}/ta/{tauuid}/tabasevalues/{tabasevalueid} | GET, POST, DELETE | +| /{id}/ta/{tauuid}/newtabasevalue | POST | +| /{id}/ta/{tauuid}/tareports | GET | +| /{id}/ta/{tauuid}/tareports/{tareportid} | GET, POST, DELETE | +| /{id}/basevalues/{basevalueid} | GET, DELETE | +| /version | GET | +| /config | GET, POST | +| /{id}/container/status | GET | +| /{id}/device/status | GET | + +The usage of the preceding APIs is described as follows: + +To query information about all servers, use `/`. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/ +``` + +*** +To query detailed information about a target server, use the GET method of `/{id}`. **{id}** is the unique ID allocated by RAS to the target server. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/1 +``` + +*** +To modify information about the target server, use the POST method of `/{id}`. `$AUTHTOKEN` is the identity verification code automatically generated by running the `ras -T` command. + +```go +type clientInfo struct { + Registered *bool `json:"registered"` // Registration status of the target server + IsAutoUpdate *bool `json:"isautoupdate"`// Target server base value update policy +} +``` + +```shell +curl -X POST -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1 -d '{"registered":false, "isautoupdate":false}' +``` + +*** +To delete a target server, use the DELETE method of `/{id}`. + +**Note:** +>This method does not delete all information about the target server. Instead, it sets the registration status of the target server to `false`. + +```shell +curl -X DELETE -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1 +``` + +*** +To query information about all servers in a specified range, use the GET method of `/{from}/{to}`. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/9 +``` + +*** +To query all trust reports of the target server, use the GET method of `/{id}/reports`. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/reports +``` + +*** +To query details about a specified trust report of the target server, use the GET method of `/{id}/reports/{reportid}`. **{reportid}** indicates the unique ID assigned by RAS to the trust report of the target server. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/reports/1 +``` + +*** +To delete a specified trust report of the target server, use the DELETE method of `/{id}/reports/{reportid}`. + +**Note:** +>This method will delete all information about the specified trusted report, and the report cannot be queried through the API. + +```shell +curl -X DELETE -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1/reports/1 +``` + +*** +To query all base values of the target server, use the GET method of `/{id}/reports/{reportid}`. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/basevalues +``` + +*** +To add a base value to the target server, use the POST method of `/{id}/newbasevalue`. + +```go +type baseValueJson struct { + BaseType string `json:"basetype"` // Base value type + Uuid string `json:"uuid"` // ID of a container or device + Name string `json:"name"` // Base value name + Enabled bool `json:"enabled"` // Whether the base value is available + Pcr string `json:"pcr"` // PCR value + Bios string `json:"bios"` // BIOS value + Ima string `json:"ima"` // IMA value + IsNewGroup bool `json:"isnewgroup"` // Whether this is a group of new reference values +} +``` + +```shell +curl -X POST -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1/newbasevalue -d '{"name":"test", "basetype":"host", "enabled":true, "pcr":"testpcr", "bios":"testbios", "ima":"testima", "isnewgroup":true}' +``` + +*** +To query details about a specified base value of a target server, use the get method of `/{id}/basevalues/{basevalueid}`. **{basevalueid}** indicates the unique ID allocated by RAS to the specified base value of the target server. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/basevalues/1 +``` + +*** +To change the availability status of a specified base value of the target server, use the POST method of `/{id}/basevalues/{basevalueid}`. + +```shell +curl -X POST -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40002/1/basevalues/1 -d '{"enabled":true}' +``` + +*** +To delete a specified base value of the target server, use the DELETE method of `/{id}/basevalues/{basevalueid}`. + +**Note:** +>This method will delete all the information about the specified base value, and the base value cannot be queried through the API. + +```shell +curl -X DELETE -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1/basevalues/1 +``` + +To query the trusted status of a specific user TA on the target server, use the GET method of the `"/{id}/ta/{tauuid}/status"` interface. Where {id} is the unique identification number assigned by RAS to the target server, and {tauuid} is the identification number of the specific user TA. + +```shell +curl -X GET -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40002/1/ta/test/status +``` + +*** +To query all the baseline value information of a specific user TA on the target server, use the GET method of the `"/{id}/ta/{tauuid}/tabasevalues"` interface. + +```shell +curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tabasevalues +``` + +*** +To query the details of a specified base value for a specific user TA on the target server, use the GET method of the `"/{id}/ta/{tauuid}/tabasevalues/{tabasevalueid}"` interface. where {tabasevalueid} is the unique identification number assigned by RAS to the specified base value of a specific user TA on the target server. + +```shell +curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tabasevalues/1 +``` + +*** +To modify the available status of a specified base value for a specific user TA on the target server, use the `POST` method of the `"/{id}/ta/{tauuid}/tabasevalues/{tabasevalueid}"` interface. + +```shell +curl -X POST -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40002/1/ta/test/tabasevalues/1 --data '{"enabled":true}' +``` + +*** +To delete the specified base value of a specific user TA on the target server, use the `DELETE` method of the `"/{id}/ta/{tauuid}/tabasevalues/{tabasevalueid}"` interface. + +**Note:** +>This method will delete all information about the specified base value, and the base value cannot be queried through the API. + +```shell +curl -X DELETE -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" -k http://localhost:40002/1/ta/test/tabasevalues/1 +``` + +*** +To add a baseline value to a specific user TA on the target server, use the `POST` method of the `"/{id}/ta/{tauuid}/newtabasevalue"` interface. + +```go +type tabaseValueJson struct { + Uuid string `json:"uuid"` // the identification number of the user TA + Name string `json:"name"` // base value name + Enabled bool `json:"enabled"` // whether a baseline value is available + Valueinfo string `json:"valueinfo"` // mirror hash value and memory hash value +} +``` + +```shell +curl -X POST -H "Content-Type: application/json" -H "Authorization: $AUTHTOKEN" -k http://localhost:40002/1/ta/test/newtabasevalue -d '{"uuid":"test", "name":"testname", "enabled":true, "valueinfo":"test info"}' +``` + +*** +To query the target server for all trusted reports for a specific user TA, use the `GET` method of the `"/{id}/ta/{tauuid}/tareports"` interface. + +```shell +curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tareports +``` + +*** +To query the details of a specified trusted report for a specific user TA on the target server, use the `GET` method of the `"/{id}/ta/{tauuid}/tareports/{tareportid}"` interface. Where {tareportid} is the unique identification number assigned by RAS to the specified trusted report of a specific user TA on the target server. + +```shell +curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tareports/2 +``` + +*** +To delete the specified trusted report of a specific user TA on the target server, use the `DELETE` method of the `"/{id}/ta/{tauuid}/tareports/{tareportid}"` interface. + +**Note:** +>This method will delete all information of the specified trusted report, and the report cannot be queried through the API. + +```shell +curl -X DELETE -H "Content-type: application/json" http://localhost:40002/1/ta/test/tareports/2 +``` + +*** +To obtain the version information of the program, use the GET method of `/version`. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/version +``` + +*** +To query the configuration information about the target server, RAS, or database, use the GET method of `/config`. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40002/config +``` + +*** +To modify the configuration information about the target server, RAS, or database, use the POST method of /config. + +```go +type cfgRecord struct { + // Target server configuration + HBDuration string `json:"hbduration" form:"hbduration"` + TrustDuration string `json:"trustduration" form:"trustduration"` + DigestAlgorithm string `json:"digestalgorithm" form:"digestalgorithm"` + // RAS configuration + MgrStrategy string `json:"mgrstrategy" form:"mgrstrategy"` + ExtractRules string `json:"extractrules" form:"extractrules"` + IsAllupdate *bool `json:"isallupdate" form:"isallupdate"` + LogTestMode *bool `json:"logtestmode" form:"logtestmode"` +} +``` + +```shell +curl -X POST -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/config -d '{"hbduration":"5s","trustduration":"20s","DigestAlgorithm":"sha256"}' +``` + +### TAS APIs + +To facilitate the administrator's management of TAS for remote control, the following API is designed for calling: + +| API | Method | +| --------------------| ------------------| +| /config | GET, POST | + +To query the configuration information, use the GET method of the `/config` interface. + +```shell +curl -X GET -H "Content-Type: application/json" http://localhost:40009/config +``` + +*** +To modify the configuration information, use the POST method of the `/config` interface. + +```shell +curl -X POST -H "Content-Type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40009/config -d '{"basevalue":"testvalue"}' +``` + +**Note:** +>Currently, only the base value in the configuration information of TAS is supported for querying and modifying. + +## FAQs + +1. Why cannot RAS be started after it is installed? + + > In the current RAS design logic, after the program is started, it needs to search for the `ecdsakey.pub` file in the current directory and read the file as the identity verification code for accessing the program. If the file does not exist in the current directory, an error is reported during RAS boot. + >> Solution 1: Run the `ras -T` command to generate a test token. The `ecdsakey.pub` file is generated. + >> Solution 2: After deploying the oauth2 authentication service, save the verification public key of the JWT token generator as `ecdsakey.pub`. + +2. Why cannot RAS be accessed through REST APIs after it is started? + + > RAS is started in HTTPS mode by default. Therefore, you need to provide a valid certificate for RAS to access it. However, RAS started in HTTP mode does not require a certificate. diff --git a/docs/en/docs/Administration/security-protection-for-interpreter-based-applications.md b/docs/en/docs/Administration/security-protection-for-interpreter-based-applications.md new file mode 100644 index 0000000000000000000000000000000000000000..7011420e1750002a1398907726b9a80dbc429385 --- /dev/null +++ b/docs/en/docs/Administration/security-protection-for-interpreter-based-applications.md @@ -0,0 +1,266 @@ +# Security Protection for Interpreter-based Applications + +## Background + +Linux IMA is widely used in the industry to verify the integrity of programs running on systems. It not only detects file tampering but also ensures that only authenticated files (such as signed or HMAC-verified) are executed through its allowlist mechanism. + +IMA supports integrity measurement and appraisal for files accessed via system calls like `read()`, `exec()`, and `mmap()`. While IMA can technically be configured to protect the `read()` system call used by interpreters to execute scripts, real-world implementations typically focus on intercepting `exec()` and `mmap()` calls. This limitation arises because IMA cannot differentiate between script files and other mutable data files. Enabling `read()`-based integrity protection would inadvertently include configuration files, temporary files, and other mutable data, which cannot have pre-generated baselines or authentication credentials. Consequently, integrity checks would fail, making `read()`-based protection impractical in real-world scenarios. As a result, targeted interception and protection remain unachievable. + +| Execution Method | Feasibility of Applying IMA Protection in Real-World Scenarios | +| ---------------- | -------------------------------------------------------------- | +| ./test.sh | Yes | +| bash test.sh | No | + +## Feature Overview + +This feature ensures that script files executed directly (such as `./test.sh`) and those executed via an interpreter (such as `bash ./test.sh`) undergo the same permission check process through kernel system calls. The details are as follows: + +### `execveat()` Support for the `AT_CHECK` Parameter + +The `execveat()` system call, introduced in Linux 3.19/glibc 2.34, allows executing a file by passing an open file descriptor. This feature extends `execveat()` with the `AT_CHECK` mechanism to verify whether a file is executable without actually executing it. + +When a caller invokes `execveat()` with a target file descriptor and specifies the `AT_CHECK` parameter, the system call logic of the kernel first performs permission checks on the file descriptor. This process mirrors the standard file execution flow, including checks for file DAC permissions, Linux Security Modules (LSM) access control rules, and IMA. If any check fails, the system call exits and returns the `-EACCSS` error code. After all permission checks are completed, `execveat()` checks if the `AT_CHECK` parameter is included. If it is, the system call skips the execution process, exits, and returns `0`, indicating that the execute permission check has passed. + +![](./figures/Process_Of_EXECVEAT_ATCHECK.png) + +### Interpreter Support for `execveat()` Permission Checks + +Building on the `execveat()` `AT_CHECK` mechanism, when an interpreter opens a script for execution, it can actively invoke the `execveat()` system call to perform an execute permission check on the file. The script will only proceed to run if the permission check is successful. + +![](./figures/AT_CHECK_Process.png) + +## Interface Overview + +### System Call Interface Description + +The `execveat()` system call function is defined as: + +```c +int execveat(int dirfd, const char *pathname, + char *const _Nullable argv[], + char *const _Nullable envp[], + int flags); +``` + +This feature introduces a `flags` parameter, `AT_CHECK`, as described below. + +| Parameter | Value | Description | +| --------- | ------- | ----------------------------------------------------------------------------- | +| AT_CHECK | 0x10000 | Performs an execute permission check on the target file without executing it. | + +### Kernel Boot Parameter Description + +This feature supports the following kernel boot parameters: + +| **Parameter** | **Value** | **Description** | +| ------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| exec_check.bash= | 0/1 | The default value is 0. When it is set to 1, the Bash process calls `execveat` to perform executable permission checks on script files before running them. | +| exec_check.java= | 0/1 | The default value is 0. When it is set to 1, the JDK calls `execveat` to perform executable permission checks on class and JAR files before running them. | +| exec_check.\= | 0/1 | This feature can be extended in the future to support other interpreters. | + +**Note: The above boot parameters are actually read and parsed by the respective interpreter processes. The kernel does not directly use these parameters.** + +## Feature Scope + +This feature is supported in openEuler 24.03 LTS SP1 (kernel 6.6). The supported interpreter types are as follows. + +| **Interpreter** | **Target Files** | **Description** | +| --------------- | --------------------- | ------------------------------------------------------------------------------------------------------------ | +| Bash | Shell script files | The Bash process performs executable permission checks on opened shell script files. | +| JDK | Class files/JAR files | The Java Virtual Machine (JVM) performs executable permission checks on loaded class files and JAR packages. | + +Community developers or users can extend support for other interpreters or similar mechanisms based on this framework. + +## Usage Instructions + +### Example of Using the `AT_CHECK` Parameter + +#### Prerequisites + +Kernel version greater than 6.6.0-54.0.0.58, and glibc version greater than or equal to 2.38-41. + +```text +glibc-2.38-41.oe2403sp1.x86_64 +kernel-6.6.0-54.0.0.58.oe2403sp1.x86_64 +``` + +#### Instructions + +You can write the following test program (**test.c**) to test the functionality of the parameter: + +```c +#define _GNU_SOURCE + +#include +#include +#include +#include + +#define AT_CHECK 0x10000 + +int main(void) +{ + int fd; + int access_ret; + + fd = open("./", O_RDONLY); + access_ret = execveat(fd, "test.sh", NULL, NULL, AT_CHECK); + perror("execveat"); + printf("access_ret = %d\n", access_ret); + close(fd); + return 0; +} +``` + +**Step 1:** Compile the test code. + +```bash +gcc test.c -o test +``` + +**Step 2:** Create the **test.sh** test script. + +```bash +echo "sleep 10" > test.sh +``` + +**Step 3:** If the test script has valid executable permissions, `execveat` returns `0`. + +```bash +# chmod +x test.sh +# ./test +execveat: Success +access_ret = 0 +``` + +**Step 4:** If the test script does not have valid permissions, `execveat` returns `-1`, with error code `Permission denied`. + +```bash +# chmod -x test.sh +# ./test +execveat: Permission denied +access_ret = -1 +``` + +### Bash Interpreter Support for Script Execute Permission Checks + +#### Prerequisites + +Kernel version greater than 6.6.0-54, glibc version greater than or equal to 2.38-41, and Bash version greater than or equal to 5.2.15-13. + +```text +bash-5.2.15-13.oe2403sp1.x86_64 +glibc-2.38-41.oe2403sp1.x86_64 +kernel-6.6.0-54.0.0.58.oe2403sp1.x86_64 +``` + +#### Instructions + +**Step 1:** Set execute permission for all script files in the system. + +```bash +find / -name "*.sh" --exec chmod +x {} \; +``` + +**Step 2:** Add the following the boot parameter and reboot the system. + +```ini +exec_check.bash=1 +``` + +**Step 3:** Verify that only scripts with execute permission can be run by Bash. + +```bash +# echo "echo hello world" > test.sh +# bash test.sh +bash: line 0: [1402] denied sourcing non-executable test.sh +# chmod +x test.sh +# bash test.sh +hello world +``` + +### JDK Support for Script Execute Permission Checks + +#### Prerequisites + +Obtain the JDK that supports this feature from the following address: + +```text +https://gitee.com/openeuler/bishengjdk-8/tree/IMA_Glibc2_34 +``` + +Follow the compilation process as described below: + +```text +https://gitee.com/openeuler/bishengjdk-8/wikis/%E4%B8%AD%E6%96%87%E6%96%87%E6%A1%A3/%E6%AF%95%E6%98%87JDK%208%20%E6%BA%90%E7%A0%81%E6%9E%84%E5%BB%BA%E8%AF%B4%E6%98%8E +``` + +#### Instructions + +**Step 1:** Ensure that all .class and .jar files in the system have execute permission. + +```bash +find / -name "*.class" -exec chmod +x {} \; +find / -name "*.jar" -exec chmod +x {} \; +``` + +**Step 2:** Add the following boot parameter and reboot the system. + +```text +exec_check.java=1 +``` + +**Step 3:** Verify that only .class or .jar files with execute permission can be run by the JVM. + +You can write the following test program (**HelloWorld.java**) to test the functionality of the parameter. + +```java +public class HelloWorld { + public static void main(String[] args) { + System.out.println("Hello, World!"); + } +} +``` + +```bash +# javac HelloWorld.java +Access denied to /home/bishengjdk/bishengjdk-8/install/jvm/openjdk-1.8.0_432-internal/lib/tools.jar +# chmod +x /home/bishengjdk/bishengjdk-8/install/jvm/openjdk-1.8.0_432-internal/lib/tools.jar +# javac HelloWorld.java +# java HelloWorld +Access denied to HelloWorld.class + +# chmod +x HelloWorld.class +# java HelloWorld +Hello, World! +``` + +### Implementing Integrity Protection for Interpreter Applications with IMA Digest Lists + +#### Prerequisites + +Enable the IMA digest list feature. For details, refer to [Kernel Integrity Measurement Architecture (IMA)](IMA.md). + +#### Instructions + +**Step 1:** Generate an IMA digest list for the target application (for details on generating digest lists, refer to [Kernel Integrity Measurement Architecture (IMA)](IMA.md)). + +**Step 2:** Enable the IMA digest list feature (for details on generating digest lists, refer to [Kernel Integrity Measurement Architecture (IMA)](IMA.md)). For example, to enable digest list verification for shell scripts, configure the kernel boot parameters as follows: + +```ini +ima_appraise=enforce ima_appraise_digest_list=digest-nometadata ima_policy="appraise_exec_tcb" initramtmpfs module.sig_enforce exec_check.bash=1 +``` + +**Step 3:** Verify IMA integrity protection for bash scripts. + +```bash +# echo "echo hello world" > test.sh +# chmod +x test.sh +# bash test.sh +bash: line 0: [2520] denied sourcing non-executable test.sh + +# After generating the digest list, sign and import it (process omitted). +# echo /etc/ima/digest_lists/0-metadata_list-compact-test.sh > /sys/kernel/security/ima/digest_list_data +# bash test.sh +hello world +``` diff --git a/docs/en/docs/Administration/trusted-computing.md b/docs/en/docs/Administration/trusted-computing.md index 5ba4816f745e1d327673467a9882645ae6b1d100..a7f9d171dcbb5fd7e36e1fa8e5b2bce6ba35d488 100644 --- a/docs/en/docs/Administration/trusted-computing.md +++ b/docs/en/docs/Administration/trusted-computing.md @@ -23,1822 +23,3 @@ In short, being trusted means that the system operates according to a pre-determ A trusted computing system consists of a root of trust, a trusted hardware platform, operating system (OS), and application. The basic idea of the system is to create a trusted computing base (TCB) first, and then establish a trust chain that covers the hardware platform, OS, and application. In the trust chain, authentication is performed from the root to the next level, extending trust level by level and building a secure and trusted computing environment. ![](./figures/trusted_chain.png) - -Unlike the traditional security mechanism that eliminates viruses without solving the root of the problem, trusted computing adopts the whitelist mechanism to allow only authorized kernels, kernel modules, and applications to run on the system. The system will reject the execution of a program that is unknown or has been changed. - -## Kernel Integrity Measurement Architecture (IMA) - -### Overview - -#### IMA - -The integrity measurement architecture (IMA) is a subsystem in the kernel. The IMA can measure files accessed through **execve()**, **mmap()**, and **open()** systems based on user-defined policies. The measurement result can be used for **local or remote attestation**, or can be compared with an existing reference value to **control the access to files**. - -According to the Wiki definition, the function of the kernel integrity subsystem includes three parts: - -- Measure: Detects accidental or malicious modifications to files, either remotely or locally. -- Appraise: Measures a file and compares it with a reference value stored in the extended attribute to control the integrity of the local file. -- Audit: Writes the measurement result into system logs for auditing. - -Figuratively, IMA measurement is an observer that only records modification without interfering in it, and IMA appraisal is more like a strict security guard that rejects any unauthorized access to programs. - -#### EVM - -The extended verification module (EVM) is used to calculate a hash value based on the security extended attributes of a file in the system, including **security.ima** and **security.selinux**. Then this value is signed by the key stored in the TPM or other trusted environments. The signature value is stored in **security.evm** and cannot be tampered with. If the value is tampered with, the signature verification fails when the file is accessed again. - -In summary, the EVM is used to provide offline protection for security extended attributes by calculating the digest of the attributes and signing and storing them in **security.evm**. - -#### IMA Digest Lists - -IMA Digest Lists are an enhancement of the original kernel integrity protection mechanism provided by openEuler. It replaces the original IMA mechanism to protect file integrity. - -Digest lists are binary data files in a special format. Each digest list corresponds to an RPM package and records the hash values of protected files (executable files and dynamic library files) in the RPM package. - -After the startup parameters are correctly configured, the kernel maintains a hash table (invisible to the user space) and provides interfaces (**digest\_list\_data** and **digest\_list\_data\_del**) that update the hash table using **securityfs**. The digest lists are signed by the private key when they are built. When uploaded to the kernel through the interface, the digest lists need to be verified by the public key in the kernel. - -![](./figures/ima_digest_list_update.png) - -When IMA appraisal is enabled, each time an executable file or dynamic library file is accessed, the hook in the kernel is invoked to calculate the hash values of the file content and extended attributes and search in the kernel hash table. If the calculated hash values match the one in the table, the file is allowed to be executed. Otherwise, the access is denied. - -![1599719649188](./figures/ima_verification.png) - -The IMA Digest Lists extension provided by the openEuler kernel provides higher security, performance, and usability than the native IMA mechanism of the kernel community, facilitating the implementation of the integrity protection mechanism in the production environment. - -- **A complete trust chain for high security** - - The native IMA mechanism requires that the file extended attribute be generated and marked in advance on the live network. When the file is accessed, the file extended attribute is used as a reference value, resulting in an incomplete trust chain. - - The IMA Digest Lists extension saves the reference digest value of the file in the kernel space. During the construction, the reference digest value of the file is carried in the released RPM package in the form of a digest list. When the RPM package is installed, the digest list is imported and the signature is verified, ensuring that the reference value comes from the software publisher and implementing a complete trust chain. - -- **Superior performance** - - The trusted platform module (TPM) chip is a low-speed chip, making the PCR extension operation a performance bottleneck in the IMA measurement scenario. To shatter this bottleneck, the Digest Lists extension reduces unnecessary PCR extension operations while ensuring security, providing 65% higher performance than the native IMA mechanism. - - In the IMA appraisal scenario, the Digest Lists extension performs signature verification in the startup phase to prevent signature verification from being performed each time the file is accessed. This helps deliver a 20% higher file access performance in the operation phase than that in the native IMA appraisal scenario. - -- **Fast deployment and smooth upgrade** - - When the native IMA mechanism is deployed for the first time or the software package is updated, you need to switch to the fix mode, manually mark the extended attributes of the file, and then restart the system to enter the enforcing mode. In this way, the installed program can be accessed normally. - - The Digest Lists extension can be used immediately after the installation is completed. In addition, the RPM package can be directly installed or upgraded in the enforcing mode without restarting the system or manually marking the extended attributes of the file. This minimizes user perception during the operation, allowing for quick deployment and smooth upgrade on the live network. - -Note: The IMA Digest Lists extension advances the signature verification of the native IMA to the startup phase. This causes the assumption that the memory in the kernel space cannot be tampered with. As a result, the IMA depends on other security mechanisms (secure startup of kernel module and dynamic memory measurement) to protect the integrity of the kernel memory. - -However, either the native IMA mechanism of the community or the IMA Digest Lists extension is only a link in the trust chain of trusted computing, and cannot ensure the system security alone. Security construction is always a systematic project that builds in-depth defense. - -### Constraints - -1. The current IMA appraisal mode can only protect immutable files in the system, including executable files and dynamic library files. -2. The IMA provides integrity measurement at the application layer. The security of the IMA depends on the reliability of the previous links. -3. Currently, the IMA does not support the import of the third-party application digest lists. -4. The startup log may contain `Unable to open file: /etc/keys/x509_ima.der`. This error is reported from the open source community and does not affect the use of the IMA digest lists feature. -5. In the ARM version, audit errors may occur when the log mode is enabled for the IMA. This occurs because the modprobe loads the kernel module before the digest lists are imported, but does not affect the normal functions. - -### Application Scenario - -#### IMA Measurement - -The purpose of IMA measurement is to detect unexpected or malicious modifications to system files. The measurement result can be used for local or remote attestation. - -If a TPM chip exists in the system, the measurement result is extended to a specified PCR register of the TPM chip. Due to the unidirectional PCR extension and the hardware security of the TPM chip, a user cannot modify the extended measurement result, thereby ensuring authenticity of the measurement result. - -The file scope and triggering conditions of IMA measurement can be configured by the user using the IMA policy. - -By default, IMA is disabled. However, the system searches for the **ima-policy** policy file in the `/etc/ima/` path. If the file is found, the system measures the files in the system based on the policy during startup. If you do not want to manually compile the policy file, you can configure the `ima_policy=tcb` in the startup parameters using the default policy. For details about more policy parameters, see the section *IMA Startup Parameters* in *Appendix*. - -You can check the currently loaded IMA policy in the `/sys/kernel/security/ima/policy` file. The IMA measurement log is located in the `/sys/kernel/security/ima/ascii_runtime_measurements` file, as shown in the following figure: - -```shell -$ head /sys/kernel/security/ima/ascii_runtime_measurements -10 ddee6004dc3bd4ee300406cd93181c5a2187b59b ima-ng sha1:9797edf8d0eed36b1cf92547816051c8af4e45ee boot_aggregate -10 180ecafba6fadbece09b057bcd0d55d39f1a8a52 ima-ng sha1:db82919bf7d1849ae9aba01e28e9be012823cf3a /init -10 ac792e08a7cf8de7656003125c7276968d84ea65 ima-ng sha1:f778e2082b08d21bbc59898f4775a75e8f2af4db /bin/bash -10 0a0d9258c151356204aea2498bbca4be34d6bb05 ima-ng sha1:b0ab2e7ebd22c4d17d975de0d881f52dc14359a7 /lib64/ld-2.27.so -10 0d6b1d90350778d58f1302d00e59493e11bc0011 ima-ng sha1:ce8204c948b9fe3ae67b94625ad620420c1dc838 /etc/ld.so.cache -10 d69ac2c1d60d28b2da07c7f0cbd49e31e9cca277 ima-ng sha1:8526466068709356630490ff5196c95a186092b8 /lib64/libreadline.so.7.0 -10 ef3212c12d1fbb94de9534b0bbd9f0c8ea50a77b ima-ng sha1:f80ba92b8a6e390a80a7a3deef8eae921fc8ca4e /lib64/libc-2.27.so -10 f805861177a99c61eabebe21003b3c831ccf288b ima-ng sha1:261a3cd5863de3f2421662ba5b455df09d941168 /lib64/libncurses.so.6.1 -10 52f680881893b28e6f0ce2b132d723a885333500 ima-ng sha1:b953a3fa385e64dfe9927de94c33318d3de56260 /lib64/libnss_files-2.27.so -10 4da8ce3c51a7814d4e38be55a2a990a5ceec8b27 ima-ng sha1:99a9c095c7928ecca8c3a4bc44b06246fc5f49de /etc/passwd -``` - -From left to right, the content of each record indicates: - -1. PCR: PCR register for extending measurement results (The default value is 10. This register is valid only when the TPM chip is installed in the system.) -2. Template hash value: hash value that is finally used for extension, combining the file content hash and the length and value of the file path -3. Template: template of the extended measurement value, for example, **ima-ng** -4. File content hash value: hash value of the measured file content -5. File path: path of the measured file - -#### IMA Appraisal - -The purpose of IMA appraisal is to control access to local files by comparing the reference value with the standard reference value. - -IMA uses the security extension attributes **security.ima** and **security.evm** to store the reference values of file integrity measurement. - -- **security.ima**: stores the hash value of the file content -- **security.evm**: stores the hash value signature of a file extended attribute - -When a protected file is accessed, the hook in the kernel is triggered to verify the integrity of the extended attributes and content of the file. - -1. Use the public key in the kernel keyring to verify the signature value in the extended attribute of the **security.evm** file, and compare this signature value with the hash value of the extended attribute of the current file. If they match, the extended attribute of the file is complete (including **security.ima**). -2. When the extended attribute of the file is complete, the system compares the extended attribute of the file **security.ima** with the digest value of the current file content. If they match, the system allows for the access to the file. - -Likewise, the file scope and trigger conditions for IMA appraisal can be configured by users using IMA policies. - -#### IMA Digest Lists - -Currently, the IMA Digest Lists extension supports the following three combinations of startup parameters: - -- IMA measurement mode: - - ```shell - ima_policy=exec_tcb ima_digest_list_pcr=11 - ``` - -- IMA appraisal log mode + IMA measurement mode: - - ```shell - ima_template=ima-sig ima_policy="exec_tcb|appraise_exec_tcb|appraise_exec_immutable" initramtmpfs ima_hash=sha256 ima_appraise=log evm=allow_metadata_writes evm=x509 ima_digest_list_pcr=11 ima_appraise_digest_list=digest - ``` - -- IMA appraisal enforcing mode + IMA measurement mode: - - ```shell - ima_template=ima-sig ima_policy="exec_tcb|appraise_exec_tcb|appraise_exec_immutable" initramtmpfs ima_hash=sha256 ima_appraise=enforce-evm evm=allow_metadata_writes evm=x509 ima_digest_list_pcr=11 ima_appraise_digest_list=digest - ``` - -### Procedure - -#### Initial Deployment in the Native IMA Scenario - -When the system is started for the first time, you need to configure the following startup parameters: - -```shell -ima_appraise=fix ima_policy=appraise_tcb -``` - -In the `fix` mode, the system can be started when no reference value is available. `appraise_tcb` corresponds to an IMA policy. For details, see *IMA Startup Parameters* in the *Appendix*. - -Next, you need to access all the files that need to be verified to add IMA extended attributes to them: - -```shell -time find / -fstype ext4 -type f -uid 0 -exec dd if='{}' of=/dev/null count=0 status=none \; -``` - -This process takes some time. After the command is executed, you can see the marked reference value in the extended attributes of the protected file. - -```shell -$ getfattr -m - -d /sbin/init -# file: sbin/init -security.ima=0sAXr7Qmun5mkGDS286oZxCpdGEuKT -security.selinux="system_u:object_r:init_exec_t" -``` - -Configure the following startup parameters and restart the system: - -```shell -ima_appraise=enforce ima_policy=appraise_tcb -``` - -#### Initial Deployment in the Digest Lists Scenario - -1. Set kernel parameters to enter the log mode. - - Add the following parameters to edit the `/boot/efi/EFI/openEuler/grub.cfg` file: - - ```shell - ima_template=ima-sig ima_policy="exec_tcb|appraise_exec_tcb|appraise_exec_immutable" initramtmpfs ima_hash=sha256 ima_appraise=log evm=allow_metadata_writes evm=x509 ima_digest_list_pcr=11 ima_appraise_digest_list=digest - ``` - - Run the `reboot` command to restart the system and enter the log mode. In this mode, integrity check has been enabled, but the system can be started even if the check fails. - -2. Install the dependency package. - - Run the **yum** command to install **digest-list-tools** and **ima-evm-utils**. Ensure that the versions are not earlier than the following: - - ```shell - $ yum install digest-list-tools ima-evm-utils - $ rpm -qa | grep digest-list-tools - digest-list-tools-0.3.93-1.oe1.x86_64 - $ rpm -qa | grep ima-evm-utils - ima-evm-utils-1.2.1-9.oe1.x86_64 - ``` - -3. If the **plymouth** package is installed, you need to add `-a` to the end of the **cp** command in line 147 in the `/usr/libexec/plymouth/plymouth-populate-initrd` script file: - - ```shell - ... - ddebug "Installing $_src" - cp -a --sparse=always -pfL "$PLYMOUTH_SYSROOT$_src" "${initdir}/$target" - } - ``` - -4. Run `dracut` to generate **initrd** again: - - ```shell - dracut -f -e xattr - ``` - - Edit the `/boot/efi/EFI/openEuler/grub.cfg` file by changing **ima\_appraise=log** to **ima\_appraise=enforce-evm**. - - ```shell - ima_template=ima-sig ima_policy="exec_tcb|appraise_exec_tcb|appraise_exec_immutable" initramtmpfs ima_hash=sha256 ima_appraise=enforce-evm evm=allow_metadata_writes evm=x509 ima_digest_list_pcr=11 ima_appraise_digest_list=digest - ``` - - Run the **reboot** command to complete the initial deployment. - -#### Building Digest Lists on OBS - -Open Build Service (OBS) is a compilation system that was first used for building software packages in openSUSE and supports distributed compilation of multiple architectures. - -Before building a digest list, ensure that your project contains the following RPM packages from openEuler: - -- digest-list-tools -- pesign-obs-integration -- selinux-policy -- rpm -- openEuler-rpm-config - -Add **Project Config** in the deliverable project: - -```shell -Preinstall: pesign-obs-integration digest-list-tools selinux-policy-targeted -Macros: -%__brp_digest_list /usr/lib/rpm/openEuler/brp-digest-list %{buildroot} -:Macros -``` - -- The following content is added to **Preinstall**: **digest-list-tools** for generating the digest list; **pesign-obs-integration** for generating the digest list signature; **selinux-policy-targeted**, ensuring that the SELinux label in the environment is correct when the digest list is generated. -- Define the macro **%\_\_brp\_digest\_list** in Macros. The RPM runs this macro to generate a digest list for the compiled binary file in the build phase. This macro can be used as a switch to control whether the digest list is generated in the project. - -After the configuration is completed, OBS automatically performs full build. In normal cases, the following two files are added to the software package: - -- **/etc/ima/digest\_lists/0-metadata\_list-compact-\[package name]-\[version number]** -- **/etc/ima/digest\_lists.tlv/0-metadata\_list-compact\_tlv-\[package name]-\[version number]** - -#### Building Digest Lists on Koji - -Koji is a compilation system of the Fedora community. The openEuler community will support Koji in the future. - -### FAQs - -1. Why does the system fail to be started, or commands fail to be executed, or services are abnormal after the system is started in enforcing mode? - - In enforcing mode, IMA controls file access. If the content or extended attributes of a file to be accessed are incomplete, the access will be denied. If key commands that affect system startup cannot be executed, the system cannot be started. - - Check whether the following problems exist: - - - **Check whether the digest list is added to initrd.** - - Check whether the **dracut** command is executed to add the digest list to the kernel during the initial deployment. If the digest list is not added to **initrd**, the digest list cannot be imported during startup. As a result, the startup fails. - - - **Check whether the official RPM package is used.** - - If a non-official openEuler RPM package is used, the RPM package may not carry the digest list, or the private key for signing the digest list does not match the public key for signature verification in the kernel. As a result, the digest list is not imported to the kernel. - - If the cause is not clear, enter the log mode and find the cause from the error log: - - ```shell - dmesg | grep appraise - ``` - -2. Why access control is not performed on system files in enforcing mode? - - When the system does not perform access control on the file as expected, check whether the IMA policy in the startup parameters is correctly configured: - - ```shell - $ cat /proc/cmdline - ...ima_policy=exec_tcb|appraise_exec_tcb|appraise_exec_immutable... - ``` - - Run the following command to check whether the IMA policy in the current kernel has taken effect: - - ```shell - cat /sys/kernel/security/ima/policy - ``` - - If the policy file is empty, it indicates that the policy fails to be set. In this case, the system does not perform access control. - -3. After the initial deployment is completed, do I need to manually run the **dracut** command to generate **initrd** after installing, upgrading, or uninstalling the software package? - - No. The **digest\_list.so** plug-in provided by the RPM package can automatically update the digest list at the RPM package granularity, allowing users to be unaware of the digest list. - -### Appendixes - -#### Description of the IMA securityfs Interface - -The native IMA provides the following **securityfs** interfaces: - -> Note: The following interface paths are in the `/sys/kernel/security/` directory. - -| Path | Permission | Description | -| ------------------------------ | ---------- | ------------------------------------------------------------ | -| ima/policy | 600 | IMA policy interface | -| ima/ascii_runtime_measurement | 440 | IMA measurement result in ASCII code format | -| ima/binary_runtime_measurement | 440 | IMA measurement result in binary format | -| ima/runtime_measurement_count | 440 | Measurement result statistics | -| ima/violations | 440 | Number of IMA measurement result conflicts | -| evm | 660 | EVM mode, that is, the mode for verifying the integrity of extended attributes of files | - -The values of `/sys/kernel/security/evm` are as follows: - -- 0: EVM uninitialized. -- 1: Uses HMAC (symmetric encryption) to verify the integrity of extended attributes. -- 2: Uses the public key signature (asymmetric encryption) to verify the integrity of extended attributes. -- 6: Disables the integrity check of extended attributes (This mode is used for openEuler). - -The additional **securityfs** interfaces provided by the IMA Digest Lists extension are as follows: - -| Path | Permission | Description | -| ------------------------ | ---------- | ---------------------------------------------------------- | -| ima/digests_count | 440 | Total number of digests (IMA+EVM) in the system hash table | -| ima/digest_list_data | 200 | New interfaces in the digest list | -| ima/digest_list_data_del | 200 | Interfaces deleted from the digest list | - -#### IMA Policy Syntax - -Each IMA policy statement must start with an **action** represented by the keyword action and be followed by a **filtering condition**: - -- **action**: indicates the action of a policy. Only one **action** can be selected for a policy. - - > Note: You can **ignore the word action** and directly write **dont\_measure** instead of **action=dont\_measure**. - -- **func**: indicates the type of the file to be measured or authenticated. It is often used together with **mask**. Only one **func** can be selected for a policy. - - - **FILE\_CHECK** can be used only with **MAY\_EXEC**, **MAY\_WRITE**, and **MAY\_READ**. - - **MODULE\_CHECK**, **MMAP\_CHECK**, and **BPRM\_CHECK** can be used only with **MAY\_EXEC**. - - A combination without the preceding matching relationships does not take effect. - -- **mask**: indicates the operation upon which files will be measured or appraised. Only one **mask** can be selected for a policy. - -- **fsmagic**: indicates the hexadecimal magic number of the file system type, which is defined in the `/usr/include/linux/magic.h` file. - - > Note: By default, all file systems are measured unless you use the **dont\_measure/dont\_appraise** to mark a file system not to be measured. - -- **fsuid**: indicates the UUID of a system device. The value is a hexadecimal string of 16 characters. - -- **objtype**: indicates the file type. Only one file type can be selected for a policy. - - > Note: **objtype** has a finer granularity than **func**. For example, **obj\_type=nova\_log\_t** indicates the nova log file. - -- **uid**: indicates the user (represented by the user ID) who performs operations on the file. Only one **uid** can be selected for a policy. - -- **fowner**: indicates the owner (represented by the user ID) of the file. Only one **fowner** can be selected for a policy. - -The values and description of the keywords are as follows: - -| Keyword | Value | Description | -| ------------- | ------------------ | ------------------------------------------------------------ | -| action | measure | Enables IMA measurement | -| | dont_measure | Disables IMA measurement | -| | appraise | Enables IMA appraisal | -| | dont_appraise | Disables IMA appraisal | -| | audit | Enables audit | -| func | FILE_CHECK | File to be opened | -| | MODULE_CHECK | Kernel module file to be loaded | -| | MMAP_CHECK | Dynamic library file to be mapped to the memory space of the process | -| | BRPM_CHECK | File to be executed (excluding script files opened by programs such as `/bin/hash`) | -| | POLICY_CHECK | File to be loaded as a supplement to the IMA policy | -| | FIRMWARE_CHECK | Firmware to be loaded into memory | -| | DIGEST_LIST_CHECK | Digest list file to be loaded into the kernel | -| | KEXEC_KERNEL_CHECK | kexec kernel to be switched to | -| mask | MAY_EXEC | Executes a file | -| | MAY_WRITE | Writes data to a file This operation is not recommended because it is restricted by open source mechanisms such as echo and vim (the essence of modification is to create a temporary file and then rename it). The IMA measurement of **MAY\_WRITE** is not triggered each time the file is modified. | -| | MAY_READ | Reads a file | -| | MAY_APPEND | Extends file attributes | -| fsmagic | fsmagic=xxx | Hexadecimal magic number of the file system type | -| fsuuid | fsuuid=xxx | UUID of a system device. The value is a hexadecimal string of 16 characters. | -| fowner | fowner=xxx | User ID of the file owner | -| uid | uid=xxx | ID of the user who operates the file | -| obj_type | obj_type=xxx_t | File type (based on the SELinux tag) | -| pcr | pcr= | Selects the PCR used to extend the measurement values in the TPM. The default value is 10. | -| appraise_type | imasig | Signature-based IMA appraisal | -| | meta_immutable | Evaluates the extended attributes of the file based on signatures (supporting the digest list). | - -> Note: **PATH\_CHECK** is equivalent to **FILE\_CHECK**, and **FILE\_MMAP** is equivalent to **MMAP\_CHECK**. They are not mentioned in this table. - -#### IMA Native Startup Parameters - -The following table lists the kernel startup parameters of the native IMA. - -| Parameter | Value | Description | -| ---------------- | ------------ | ------------------------------------------------------------ | -| ima_appraise | off | Disables the IMA appraisal mode. The integrity check is not performed when the file is accessed and no new reference value is generated for the file. | -| | enforce | Enables the IMA appraisal enforcing mode to perform the integrity check when the file is accessed. That is, the file digest value is calculated and compared with the reference value. If the comparison fails, the file access is rejected. In this case, the IMA generates a new reference value for the new file. | -| | fix | Enables the IMA repair mode. In this mode, the reference value of a protected file can be updated. | -| | log | Enables the IMA appraisal log mode to perform the integrity check when the file is accessed. However, commands can be executed even if the check fails, and only logs are recorded. | -| ima_policy | tcb | Measures all file execution, dynamic library mapping, kernel module import, and device driver loading. The file read behavior of the root user is also measured. | -| | appraise_tcb | Evaluates all files whose owner is the root user. | -| | secure_boot | Evaluates the kernel module import, hardware driver loading, kexec kernel switchover, and IMA policies. The prerequisite is that these files have IMA signatures. | -| ima_tcb | None | Equivalent to **ima\_policy=tcb**. | -| ima_appraise_tcb | None | Equivalent to **ima\_policy=appraise\_tcb**. | -| ima_hash | sha1/md5/... | IMA digest algorithm. The default value is sha1. | -| ima_template | ima | IMA measurement extension template | -| | ima-ng | IMA measurement extension template | -| | ima-sig | IMA measurement extension template | -| integrity_audit | 0 | Basic integrity audit information (default) | -| | 1 | Additional integrity audit information | - -> Note: The **ima\_policy** parameter can specify multiple values at the same time, for example, **ima\_policy=tcb\|appraise\_tcb**. After the system is started, the IMA policy of the system is the sum of the policies for the two parameters. - -The IMA policy for the `ima_policy=tcb` startup parameter is as follows: - -```console -# PROC_SUPER_MAGIC = 0x9fa0 -dont_measure fsmagic=0x9fa0 -# SYSFS_MAGIC = 0x62656572 -dont_measure fsmagic=0x62656572 -# DEBUGFS_MAGIC = 0x64626720 -dont_measure fsmagic=0x64626720 -# TMPFS_MAGIC = 0x01021994 -dont_measure fsmagic=0x1021994 -# DEVPTS_SUPER_MAGIC=0x1cd1 -dont_measure fsmagic=0x1cd1 -# BINFMTFS_MAGIC=0x42494e4d -dont_measure fsmagic=0x42494e4d -# SECURITYFS_MAGIC=0x73636673 -dont_measure fsmagic=0x73636673 -# SELINUX_MAGIC=0xf97cff8c -dont_measure fsmagic=0xf97cff8c -# SMACK_MAGIC=0x43415d53 -dont_measure fsmagic=0x43415d53 -# CGROUP_SUPER_MAGIC=0x27e0eb -dont_measure fsmagic=0x27e0eb -# CGROUP2_SUPER_MAGIC=0x63677270 -dont_measure fsmagic=0x63677270 -# NSFS_MAGIC=0x6e736673 -dont_measure fsmagic=0x6e736673 -measure func=MMAP_CHECK mask=MAY_EXEC -measure func=BPRM_CHECK mask=MAY_EXEC -measure func=FILE_CHECK mask=MAY_READ uid=0 -measure func=MODULE_CHECK -measure func=FIRMWARE_CHECK -``` - -The IMA policy for the `ima_policy=tcb_appraise` startup parameter is as follows: - -```console -# PROC_SUPER_MAGIC = 0x9fa0 -dont_appraise fsmagic=0x9fa0 -# SYSFS_MAGIC = 0x62656572 -dont_appraise fsmagic=0x62656572 -# DEBUGFS_MAGIC = 0x64626720 -dont_appraise fsmagic=0x64626720 -# TMPFS_MAGIC = 0x01021994 -dont_appraise fsmagic=0x1021994 -# RAMFS_MAGIC -dont_appraise fsmagic=0x858458f6 -# DEVPTS_SUPER_MAGIC=0x1cd1 -dont_appraise fsmagic=0x1cd1 -# BINFMTFS_MAGIC=0x42494e4d -dont_appraise fsmagic=0x42494e4d -# SECURITYFS_MAGIC=0x73636673 -dont_appraise fsmagic=0x73636673 -# SELINUX_MAGIC=0xf97cff8c -dont_appraise fsmagic=0xf97cff8c -# SMACK_MAGIC=0x43415d53 -dont_appraise fsmagic=0x43415d53 -# NSFS_MAGIC=0x6e736673 -dont_appraise fsmagic=0x6e736673 -# CGROUP_SUPER_MAGIC=0x27e0eb -dont_appraise fsmagic=0x27e0eb -# CGROUP2_SUPER_MAGIC=0x63677270 -dont_appraise fsmagic=0x63677270 -appraise fowner=0 -``` - -The IMA policy for the `ima_policy=secure_boot` startup parameter is as follows: - -```console -appraise func=MODULE_CHECK appraise_type=imasig -appraise func=FIRMWARE_CHECK appraise_type=imasig -appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig -appraise func=POLICY_CHECK appraise_type=imasig -``` - -#### IMA Digest List Startup Parameters - -The kernel startup parameters added to the IMA digest list feature are as follows: - -| Parameter | Value | Description | -| ------------------------ | ----------------------- | ------------------------------------------------------------ | -| integrity | 0 | Disables the IMA feature (by default) | -| | 1 | Enables the IMA feature | -| ima_appraise | off | Disables the IMA appraisal mode | -| | enforce-evm | Enables the IMA appraisal forced mode to perform the integrity check when the file is accessed and control the access. | -| ima_appraise_digest_list | digest | When the EVM is disabled, the abstract list is used for IMA appraise. The abstract list protects both the content and extended attributes of the file. | -| | digest-nometadata | If the EVM digest value does not exist, the integrity check is performed only based on the IMA digest value (the file extended attribute is not protected). | -| evm | fix | Allows for any modification to the extended attribute (even if the modification causes the failure to verify the integrity of the extended attribute). | -| | ignore | Allowed to modify the extended attribute only when it does not exist or is incorrect. | -| ima_policy | exec_tcb | IMA measurement policy. For details, see the following policy description. | -| | appraise_exec_tcb | IMA appraisal policy. For details, see the following policy description. | -| | appraise_exec_immutable | IMA appraisal policy. For details, see the following policy description. | -| ima_digest_list_pcr | 11 | Uses PCR 11 instead of PCR 10, and uses only the digest list for measurement. | -| | +11 | The PCR 10 measurement is reserved. When the TPM chip is available, the measurement result is written to the TPM chip. | -| initramtmpfs | None | Adds the support for **tmpfs**. | - -The IMA policy for the `ima_policy=exec_tcb` startup parameter is as follows: - -```console -dont_measure fsmagic=0x9fa0 -dont_measure fsmagic=0x62656572 -dont_measure fsmagic=0x64626720 -dont_measure fsmagic=0x1cd1 -dont_measure fsmagic=0x42494e4d -dont_measure fsmagic=0x73636673 -dont_measure fsmagic=0xf97cff8c -dont_measure fsmagic=0x43415d53 -dont_measure fsmagic=0x27e0eb -dont_measure fsmagic=0x63677270 -dont_measure fsmagic=0x6e736673 -measure func=MMAP_CHECK mask=MAY_EXEC -measure func=BPRM_CHECK mask=MAY_EXEC -measure func=MODULE_CHECK -measure func=FIRMWARE_CHECK -measure func=POLICY_CHECK -measure func=DIGEST_LIST_CHECK -measure parser -``` - -The IMA policy for the `ima_policy=appraise_exec_tcb` startup parameter is as follows: - -```console -appraise func=MODULE_CHECK appraise_type=imasig -appraise func=FIRMWARE_CHECK appraise_type=imasig -appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig -appraise func=POLICY_CHECK appraise_type=imasig -appraise func=DIGEST_LIST_CHECK appraise_type=imasig -dont_appraise fsmagic=0x9fa0 -dont_appraise fsmagic=0x62656572 -dont_appraise fsmagic=0x64626720 -dont_appraise fsmagic=0x858458f6 -dont_appraise fsmagic=0x1cd1 -dont_appraise fsmagic=0x42494e4d -dont_appraise fsmagic=0x73636673 -dont_appraise fsmagic=0xf97cff8c -dont_appraise fsmagic=0x43415d53 -dont_appraise fsmagic=0x6e736673 -dont_appraise fsmagic=0x27e0eb -dont_appraise fsmagic=0x63677270 -``` - -The IMA policy for the `ima_policy=appraise_exec_immutable` startup parameter is as follows: - -```console -appraise func=BPRM_CHECK appraise_type=imasig appraise_type=meta_immutable -appraise func=MMAP_CHECK -appraise parser appraise_type=imasig -``` - -#### IMA Kernel Compilation Options - -The native IMA provides the following compilation options: - -| Compilation Option | Description | -| -------------------------------- | ------------------------------------------------------- | -| CONFIG_INTEGRITY | IMA/EVM compilation switch | -| CONFIG_INTEGRITY_SIGNATURE | Enables IMA signature verification | -| CONFIG_INTEGRITY_ASYMMETRIC_KEYS | Enables IMA asymmetric signature verification | -| CONFIG_INTEGRITY_TRUSTED_KEYRING | Enables IMA/EVM key ring | -| CONFIG_INTEGRITY_AUDIT | Compiles the IMA audit module | -| CONFIG_IMA | IMA compilation switch | -| CONFIG_IMA_WRITE_POLICY | Allows updating the IMA policy in the running phase | -| CONFIG_IMA_MEASURE_PCR_IDX | Allows specifying the PCR number of the IMA measurement | -| CONFIG_IMA_LSM_RULES | Allows configuring LSM rules | -| CONFIG_IMA_APPRAISE | IMA appraisal compilation switch | -| IMA_APPRAISE_BOOTPARAM | Enables IMA appraisal startup parameters | -| CONFIG_EVM | EVM compilation switch | - -The additional compilation options provided by the IMA Digest Lists extension are as follows: - -| Compilation Option | Description | -| ------------------ | ----------------------------------- | -| CONFIG_DIGEST_LIST | Enables the IMA Digest List feature | - -#### IMA Performance Reference Data - -The following figure compares the performance when IMA is disabled, native IMA is enabled, and IMA digest list is enabled. - -![img](./figures/ima_performance.png) - -#### Impact of IMA on the kdump Service - -When the IMA enforce mode is enabled and kexec system call verification is configured in the policy, kdump may fail to be started. - -```console -appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig -``` - -Cause of the kdump startup failure: After IMA is enabled, file integrity needs to be verified. Therefore, the **kexec_file_load** system call is restricted when kdump loads kernel image files. You can modify **KDUMP_FILE_LOAD** in the **/etc/sysconfig/kdump** configuration file to enable the **kexec_file_load** system call. - -```console -KDUMP_FILE_LOAD="on" -``` - -At the same time, the **kexec_file_load** system call itself also verifies the signature of the file. Therefore, the loaded kernel image file must contain the correct secure boot signature, and the current kernel must contain the corresponding verification certificate. - -#### IMA Root Certificate Configuration - -Currently, openEuler uses the RPM key to sign the IMA digest list. To ensure that the IMA function is available out of the box, openEuler imports the RPM root certificate (PGP certificate) to the kernel by default during kernel compilation. Currently, there are two PGP certificates, namely, the OBS certificate used in the earlier version and the openEuler certificate used in the switchover of openEuler 22.03 LTS SP1. - -```shell -$ cat /proc/keys | grep PGP -1909b4ad I------ 1 perm 1f030000 0 0 asymmetri private OBS b25e7f66: PGP.rsa b25e7f66 [] -2f10cd36 I------ 1 perm 1f030000 0 0 asymmetri openeuler fb37bc6f: PGP.rsa fb37bc6f [] -``` - -The current kernel does not support the import of the PGP sub-public key, and the switched openEuler certificate uses the sub-key signature. Therefore, the openEuler kernel preprocesses the certificate before compilation, extracts the sub-public key, and imports it to the kernel. For details, see the [process_pgp_certs.sh](https://gitee.com/src-openeuler/kernel/blob/openEuler-22.03-LTS-SP1/process_pgp_certs.sh) script file in the code repository of the kernel software package. - -If the user does not use the IMA digest list function or uses other keys to implement signature/verification, you can remove the related code and configure the kernel root certificate by yourself. - -## Dynamic Integrity Measurement (DIM) - -This section describes the DIM feature and its usage. - -### Context - -With the development of the IT industry, information systems are facing an increasing number of security risks. Information systems run a large amount of software, some of which is inevitably vulnerable. Once exploited by attackers, these vulnerabilities could severely damage system services, resulting in data leakage and service unavailability. - -Most software attacks are accompanied by integrity damage, such as malicious process execution, configuration file tampering, and backdoor implantation. Therefore, protection technologies are proposed in the industry. Key data is measured and verified during system startup to ensure that the system can run properly. However, popular integrity protection technologies (such as secure boot and file integrity measurement) cannot protect memory data during process running. If an attacker somehow modifies the instructions of a process, the process may be hijacked or implanted with a backdoor, which is highly destructive and covert. To defend against such attacks, the DIM technology is proposed to measure and protect key data in the memory of a running process. - -### Terminology - -Static baseline: baseline measurement data generated by parsing the binary file of the measurement target - -Dynamic baseline: result of the first measurement on the measurement target - -Measurement policy: configuration information for measuring the target - -Measurement log: list of measurement information, including the measurement targets and measurement results - -### Description - -The DIM feature measures key data (such as code sections and data sections) in the memory during program running and compares the measurement result with the baseline value to determine whether the memory data has been tampered with. In this way, DIM can detect attacks and take countermeasures. - -#### Function Scope - -- Currently, DIM supports only AArch64 and x86 architectures. -- Currently, DIM supports measurement of the following key memory data: - - Code section of a user-mode process (the section whose attribute is **PT_LOAD** and permission is **RX** in the ELF file, and the virtual memory area whose permission is **RX** after the corresponding process is loaded) - - Kernel module code section, whose start address and length are **core_layout.base** and **core_layout.text_size** respectively in the **struct module** structure corresponding to the kernel module. - - Kernel code section, corresponding to **\_stext** to **\_etext** (Addresses that may change due to the kernel static key mechanism are skipped.) -- DIM can work with the following hardware: - - The measurement result can be extended to the Platform Configuration Register (PCR) of Trusted Platform Module (TPM) 2.0 to connect to the remote attestation service. - -#### Technical Limitations - -- For user-mode processes, only mapped code sections of files can be measured. Anonymous code sections cannot be measured. -- Kernel hot patches cannot be measured. -- Measurement can only be triggered proactively. If a file is attacked and recovered during two measurement processes, the attack cannot be identified. -- Proactive modification of code sections, such as relocation of code sections, self-modification, and hot patches, will be identified as attacks. -- For kernel and kernel module measurement, the measurement result when dynamic baseline establishment is triggered is used as the measurement baseline value. The static baseline value only serves as a fixed identifier. -- The measurement target must have been loaded to the memory when dynamic baseline establishment is triggered (for example, process running or kernel module loading). Otherwise, the object cannot be measured. -- If the TPM PCR needs to be used to verify measurement logs, the DIM module cannot be removed. Otherwise, the measurement logs will be cleared and cannot match the PCR. - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->After DIM is enabled, system performance is affected in the following aspects: -> -> - Loading the DIM feature and managing baseline data and measurement logs consume system memory. The impact depends on the protection policy configuration. -> - Hashing is performed during DIM running, which consumes CPU resources. The impact depends on the size of the data to be measured. -> - Resources will be locked and semaphores need to be obtained during DIM running, which may cause other concurrent processes to wait. - -#### Specification Constraints - -| Item | Value | -| ------------------------------------------------------------ | ---- | -| Maximum size of a policy file, static baseline file, signature file, or certificate file | 10MB | -| Maximum number of tampering measurement logs recorded for a measurement target during multiple measurement periods after a dynamic baseline is established.| 10| -| Maximum number of measurement policies that can be stored in **/etc/dim/policy**|10,000| - -#### Architecture Description - -DIM includes the dim_tools and dim software packages, which contain the following components: - -| Software Package | Component | Description | -| --------- | ---------------- | ------------------------------------------------------------ | -| dim_tools | dim_gen_baseline| User-mode component for generating the static baseline, which is required for generating the dynamic measurement. The baseline data is imported when DIM runs.| -| dim | dim_core | Kernel module for executing the core dynamic measurement logic, including policy parsing, static baseline parsing, dynamic baseline establishment, measurement execution, measurement logging, and TPM extension operations, to measure key memory data.| -| dim | dim_monitor | Kernel module for measuring and protecting dim_core code sections and key data to prevent DIM function failures caused by attacks on dim_core. | - -The following figure shows the overall architecture: - -![](./figures/dim_architecture.jpg) - -#### Key Procedures - -Both the dim_core and dim_monitor modules provide the memory data measurement function, including the following core processes: - -- Dynamic baseline process: The dim_core module reads and parses the policy and static baseline file, measures the code section of the target process, stores the measurement result as a dynamic baseline in the memory, compares the dynamic baseline data with the static baseline data, and records the comparison result in measurement logs. The dim_monitor module measures the code sections and key data of the dim_core module, uses the data as the dynamic baseline, and records measurement logs. -- Dynamic measurement process: The dim_core and dim_monitor modules measure the target and compare the measurement result with the dynamic baseline. If the measurement result is inconsistent with the dynamic baseline, the dim_core and dim_monitor modules record the result in measurement logs. - -#### Interfaces - -##### Interface Files - -| Path | Description | -| ------------------------------- | ------------------------------------------------------------ | -| /etc/dim/policy | Measurement policy file | -| /etc/dim/policy.sig | Measurement policy signature file, which is used to store the signature information of the policy file and is used when the signature verification function is enabled| -| /etc/dim/digest_list/*.hash | Static baseline file, which is used to store measurement baseline values | -| /etc/dim/digest_list/*.hash.sig | Static baseline signature file, which is used to store the signature information of the static baseline file and is used when the signature verification function is enabled| -| /etc/keys/x509_dim.der | Certificate file, which is used to verify the signature information of the policy file and static baseline file and is used when the signature verification function is enabled| -| /sys/kernel/security/dim | DIM file system directory, which is generated after the DIM kernel module is loaded and provides kernel interfaces for operating the DIM function| - -##### File Formats - -1. Measurement policy file format - - The lines, each representing a measurement policy, are in plaintext and are separated by Unix newline (LF) characters. The following configuration formats are supported: - - 1. User-mode process code section measurement - - ```text - measure obj=BPRM_TEXT path= - ``` - - 2. Kernel module code section measurement - - ```text - measure obj=MODULE_TEXT name= - ``` - - 3. Kernel measurement - - ```text - measure obj=KERNEL_TEXT - ``` - - **Example:** - - ```shell - # cat /etc/dim/policy - measure obj=BPRM_TEXT path=/usr/bin/bash - measure obj=BPRM_TEXT path=/usr/lib64/libc.so.6 - measure obj=MODULE_TEXT name=ext4 - measure obj=KERNEL_TEXT - ``` - -2. Static baseline file format - - The lines, each representing a static baseline, are in plaintext and are separated by Unix newline (LF) characters. The following configuration formats are supported: - - 1. User-mode process baseline - - ```text - dim USER sha256:6ae347be2d1ba03bf71d33c888a5c1b95262597fbc8d00ae484040408a605d2b - ``` - - 2. Kernel module baseline - - ```text - dim KERNEL sha256:a18bb578ff0b6043ec5c2b9b4f1c5fa6a70d05f8310a663ba40bb6e898007ac5 / - ``` - - 3. Kernel baseline - - ```text - dim KERNEL sha256:2ce2bc5d65e112ba691c6ab46d622fac1b7dbe45b77106631120dcc5441a3b9a - ``` - - **Example:** - - ```text - dim USER sha256:6ae347be2d1ba03bf71d33c888a5c1b95262597fbc8d00ae484040408a605d2b /usr/bin/bash - dim USER sha256:bc937f83dee4018f56cc823f5dafd0dfedc7b9872aa4568dc6fbe404594dc4d0 /usr/lib64/libc.so.6 - dim KERNEL sha256:a18bb578ff0b6043ec5c2b9b4f1c5fa6a70d05f8310a663ba40bb6e898007ac5 6.4.0-1.0.1.4.oe2309.x86_64/dim_monitor - dim KERNEL sha256:2ce2bc5d65e112ba691c6ab46d622fac1b7dbe45b77106631120dcc5441a3b9a 6.4.0-1.0.1.4.oe2309.x86_64 - ``` - -3. Measurement log format - - The lines, each representing a measurement log, are in plaintext and are separated by Unix newline (LF) characters. The format is as follows: - - ```text - : - ``` - - **Example:** - - 1. The code section of the bash process is measured. The measurement result is consistent with the static baseline. - - ```text - 12 0f384a6d24e121daf06532f808df624d5ffc061e20166976e89a7bb24158eb87 sha256:db032449f9e20ba37e0ec4a506d664f24f496bce95f2ed972419397951a3792e /usr/bin.bash [static baseline] - ``` - - 2. The code section of the bash process is measured. The measurement result is inconsistent with the static baseline. - - ```text - 12 0f384a6d24e121daf06532f808df624d5ffc061e20166976e89a7bb24158eb87 sha256:db032449f9e20ba37e0ec4a506d664f24f496bce95f2ed972419397951a3792e /usr/bin.bash [tampered] - ``` - - 3. The code section of the ext4 kernel module is measured. No static baseline is found. - - ```text - 12 0f384a6d24e121daf06532f808df624d5ffc061e20166976e89a7bb24158eb87 sha256:db032449f9e20ba37e0ec4a506d664f24f496bce95f2ed972419397951a3792e ext4 [no static baseline] - ``` - - 4. dim_monitor measures dim_core and records the measurement result of the baseline. - - ```text - 12 660d594ba050c3ec9a7cdc8cf226c5213c1e6eec50ba3ff51ff76e4273b3335a sha256:bdab94a05cc9f3ad36d29ebbd14aba8f6fd87c22ae580670d18154b684de366c dim_core.text [dynamic baseline] - 12 28a3cefc364c46caffca71e7c88d42cf3735516dec32796e4883edcf1241a7ea sha256:0dfd9656d6ecdadc8ec054a66e9ff0c746d946d67d932cd1cdb69780ccad6fb2 dim_core.data [dynamic baseline] - ``` - -4. Certificate and signature file formats - -The files are in the common format. For details, see [Enabling Signature Verification](#enabling-signature-verification). - -##### Kernel Module Parameters - -1. dim_core parameters - - | Parameter | Description | Value Range | Default Value| - | -------------------- | ------------------------------------------------------------ | ------------------------ | ------ | - | measure_log_capacity | Maximum number of measurement logs recorded by dim_core. When this value is reached, dim_core stops recording measurement logs.| 100-UINT_MAX (64-bit OS)| 100000 | - | measure_schedule | Scheduling time after a process or module is measured, in milliseconds. The value 0 indicates that measurement is not scheduled.| 0-1000 | 0 | - | measure_interval | Automatic measurement interval, in minutes. The value 0 indicates that automatic measurement is not enabled. | 0-525600 | 0 | - | measure_hash | Measurement hash algorithm | sha256, sm3 | sha256 | - | measure_pcr | The TPM PCR to which the measurement result is extended. The value 0 indicates that the measurement result is not extended. The value must be an actual TPM PCR number.| 0-128 | 0 | - | signature | Whether to enable the policy file and signature verification. The value 0 indicates that they are disabled, and the value 1 indicates that they are enabled. | 0, 1 | 0 | - - **Example:** - - ```shell - insmod /path/to/dim_core.ko measure_log_capacity=10000 measure_schedule=10 measure_pcr=12 - modprobe dim_core measure_log_capacity=10000 measure_schedule=10 measure_pcr=12 - ``` - -2. dim_monitor parameters - - | Parameter | Description | Value Range | Default Value| - | -------------------- | ------------------------------------------------------------ | ------------------------ | ------ | - | measure_log_capacity | Maximum number of measurement logs recorded by dim_monitor. When this value is reached, dim_monitor stops recording measurement logs.| 100-UINT_MAX (64-bit OS)| 100000 | - | measure_hash | Measurement hash algorithm | sha256, sm3 | sha256 | - | measure_pcr | The TPM PCR to which the measurement result is extended. The value 0 indicates that the measurement result is not extended. | 0-128 | 0 | - - **Example:** - - ```shell - insmod /path/to/dim_monitor.ko measure_log_capacity=10000 measure_hash=sm3 - modprobe dim_monitor measure_log_capacity=10000 measure_hash=sm3 - ``` - -##### Kernel Interfaces - -1. dim_core interface - - | Interface | Attribute| Function | Example | - | -------------------------- | ---- | ------------------------------------------------------------ | --------------------------------------------------------- | - | measure | Write-only| Write **1** to trigger dynamic measurement. If the operation is successful, **0** is returned. If the operation fails, an error code is returned. | echo 1 > /sys/kernel/security/dim/measure | - | baseline_init | Write-only| Write **1** to trigger dynamic baseline establishment. If the operation is successful, **0** is returned. If the operation fails, an error code is returned. | echo 1 > /sys/kernel/security/dim/baseline_init | - | ascii_runtime_measurements | Read-only| Read the interface to query measurement logs. | cat /sys/kernel/security/dim/ascii_runtime_measurements | - | runtime_status | Read-only| Read the interface to query the status information. If the operation fails, an error code is returned. | cat /sys/kernel/security/dim/runtime_status | - | interval | Read/Write| Write a numeric string to set the automatic measurement interval (the value range is the same as that of **measure_interval**). Read the interface to query the current automatic measurement interval. If the query fails, an error code is returned.| echo 10 > /sys/kernel/security/dim/interval
cat /sys/kernel/security/dim/interval | - - **dim_core Statuses** - - The possible statuses of dim_core are as follows: - - - DIM_NO_BASELINE: dim_core is loaded but no operation is performed. - - DIM_BASELINE_RUNNING: The dynamic baseline is being established. - - DIM_MEASURE_RUNNING: Dynamic measurement is being performed. - - DIM_PROTECTED: The dynamic baseline has been established and is protected. - - DIM_ERROR: An error occurs during dynamic baseline establishment or dynamic measurement. You need to rectify the error and trigger dynamic baseline establishment or dynamic measurement again. - -2. dim_monitor interfaces - - | Interface | Attribute| Description | Example | - | ---------------------------------- | ---- | ---------------------------------------------- | ------------------------------------------------------------ | - | monitor_run | Write-only| Write **1** to trigger measurement. If the operation is successful, **0** is returned. If the operation fails, an error code is returned.| echo 1 > /sys/kernel/security/dim/monitor_run | - | monitor_baseline | Write-only| Write **1** to trigger baseline establishment. If the operation is successful, **0** is returned. If the operation fails, an error code is returned.| echo 1 > /sys/kernel/security/dim/monitor_baseline | - | monitor_ascii_runtime_measurements | Read-only| Read the interface to query measurement logs. | cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements | - | monitor_status | Read-only| Read the interface to query the status information. If the operation fails, an error code is returned. | cat /sys/kernel/security/dim/monitor_status | - - **dim_monitor Statuses** - - - ready: dim_monitor is loaded but no operation is performed. - - running: The dynamic baseline is being established or dynamic measurement is being performed. - - error: An error occurs during dynamic baseline establishment or dynamic measurement. You need to rectify the error and trigger dynamic baseline establishment or dynamic measurement again. - - protected: The dynamic baseline has been established and is protected. - -##### User-Mode Tool Interface - -See for the details of the `dim_gen_baseline` CLI interface. - -### Usage - -#### Installing and Uninstalling DIM - -**Prerequisites**: - -- OS: openEuler 23.09 or later -- Kernel: openEuler kernel 5.10 or 6.4 - -Install dim_tools and dim software packages. The following uses openEuler 23.09 as an example: - -```shell -# yum install -y dim_tools dim -``` - -After the software packages are installed, the DIM kernel components are not loaded by default. You can run the following commands to load or unload them: - -```shell -# modprobe dim_core or insmod /path/to/dim_core.ko -# modprobe dim_monitor or insmod /path/to/dim_monitor.ko -# rmmod dim_monitor -# rmmod dim_core -``` - -After the components are loaded successfully, you can run the following commands to query the components: - -```shell -# lsmod | grep dim_core -dim_core 77824 1 dim_monitor -# lsmod | grep dim_monitor -dim_monitor 36864 0 -``` - -Unload the KO files before uninstalling the RPM package. - -```shell -# rmmod dim_monitor -# rmmod dim_core -# rpm -e dim -``` - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> -> dim_monitor must be loaded after dim_core and removed before dim_core. -> You can also install DIM from source. For details, see . - -#### Measuring Code Sections of User-Mode Processes - -**Prerequisites**: - -- The dim_core module is loaded successfully. - -- A user-mode resident measurement target program has been prepared. This section uses **/opt/dim/demo/dim_test_demo** as an example. - - ```shell - # /opt/dim/demo/dim_test_demo & - ``` - -**Step 1**: Generate a static baseline for the binary file corresponding to the measurement target process. - -```shell -# mkdir -p /etc/dim/digest_list -# dim_gen_baseline /opt/dim/demo/dim_test_demo -o /etc/dim/digest_list/test.hash -``` - -**Step 2**: Configure a measurement policy. - -```shell -# echo "measure obj=BPRM_TEXT path=/opt/dim/demo/dim_test_demo" > /etc/dim/policy -``` - -**Step 3**: Trigger dynamic baseline establishment. - -```shell -# echo 1 > /sys/kernel/security/dim/baseline_init -``` - -**Step 4**: Query measurement logs. - -```shell -# cat /sys/kernel/security/dim/ascii_runtime_measurements -0 e9a79e25f091e03a8b3972b1a0e4ae2ccaed1f5652857fe3b4dc947801a6913e sha256:02e28dff9997e1d81fb806ee5b784fd853eac8812059c4dba7c119c5e5076989 /opt/dim/demo/dim_test_demo [static baseline] -``` - -If the preceding measurement log is displayed, the target process is measured successfully and the measurement result is consistent with the static baseline. - -**Step 5**: Trigger dynamic measurement. - -```shell -# echo 1 > /sys/kernel/security/dim/measure -``` - -After the measurement is complete, you can perform **Step 4** to query the measurement logs. If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. If an attacker attempts to tamper with the target program (for example, by modifying and recompiling the code) and restart the target program, for example: - -```shell -# pkill dim_test_demo -# /opt/dim/demo/dim_test_demo & -``` - -Trigger the measurement again and query the measurement logs. You can see a measurement log marked with "tampered." - -```shell -# echo 1 > /sys/kernel/security/dim/measure -# cat /sys/kernel/security/dim/ascii_runtime_measurements -0 e9a79e25f091e03a8b3972b1a0e4ae2ccaed1f5652857fe3b4dc947801a6913e sha256:02e28dff9997e1d81fb806ee5b784fd853eac8812059c4dba7c119c5e5076989 /opt/dim/demo/dim_test_demo [static baseline] -0 08a2f6f2922ad3d1cf376ae05cf0cc507c2f5a1c605adf445506bc84826531d6 sha256:855ec9a890ff22034f7e13b78c2089e28e8d217491665b39203b50ab47b111c8 /opt/dim/demo/dim_test_demo [tampered] -``` - -#### Measuring Code Sections of Kernel Modules - -**Prerequisites**: - -- The dim_core module is loaded successfully. - -- A measurement target kernel module has been prepared. This section uses dim_test_module as an example, whose path is **/opt/dim/demo/dim_test_module.ko**. - - ```shell - # insmod /opt/dim/demo/dim_test_module.ko - ``` - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->The kernel version of the environment where the module is compiled must be the same as the current kernel version. Run the following command to confirm: -> ->```shell -># modinfo dim_monitor.ko | grep vermagic | grep "$(uname -r)" ->vermagic: 6.4.0-1.0.1.4.oe2309.x86_64 SMP preempt mod_unload modversions ->``` - -The first field of the vermagic kernel module information must be the same as the current kernel version. - -**Step 1**: Generate a static baseline for the measurement target kernel module. - -```shell -# mkdir -p /etc/dim/digest_list -# dim_gen_baseline /opt/dim/demo/dim_test_module.ko -o /etc/dim/digest_list/test.hash -``` - -**Step 2**: Configure a measurement policy. - -```shell -# echo "measure obj=MODULE_TEXT name=dim_test_module" > /etc/dim/policy -``` - -**Step 3**: Trigger dynamic baseline establishment. - -```shell -# echo 1 > /sys/kernel/security/dim/baseline_init -``` - -**Step 4**: Query measurement logs. - -```shell -# cat /sys/kernel/security/dim/ascii_runtime_measurements -0 9603a9d5f87851c8eb7d2619f7abbe28cb8a91f9c83f5ea59f036794e23d1558 sha256:9da4bccc7ae1b709deab8f583b244822d52f3552c93f70534932ae21fac931c6 dim_test_module [static baseline] -``` - -The preceding measurement log indicates that dim_test_module is successfully measured and the current measurement result is used as the baseline value for subsequent measurement. The hash value in the measurement log is not the actual measurement value. - -**Step 5**: Trigger dynamic measurement. - -```shell -echo 1 > /sys/kernel/security/dim/measure -``` - -After the measurement is complete, you can perform **Step 4** to query the measurement logs. If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. If an attacker attempts to tamper with the target kernel module (for example, by modifying and recompiling the code) and reload the module, for example: - -```shell -rmmod dim_test_module -insmod /opt/dim/demo/dim_test_module.ko -``` - -Trigger the measurement again and query the measurement logs. You can see a measurement log marked with "tampered." - -```shell -# cat /sys/kernel/security/dim/ascii_runtime_measurements -0 9603a9d5f87851c8eb7d2619f7abbe28cb8a91f9c83f5ea59f036794e23d1558 sha256:9da4bccc7ae1b709deab8f583b244822d52f3552c93f70534932ae21fac931c6 dim_test_module [static baseline] -0 6205915fe63a7042788c919d4f0ff04cc5170647d7053a1fe67f6c0943cd1f40 sha256:4cb77370787323140cb572a789703be1a4168359716a01bf745aa05de68a14e3 dim_test_module [tampered] -``` - -#### Measuring Code Sections of the Kernel - -**Prerequisites**: - -- The dim_core module is loaded successfully. - -**Step 1**: generate a static baseline for the kernel. - -```shell -# mkdir -p /etc/dim/digest_list -# dim_gen_baseline -k "$(uname -r)" -o /etc/dim/digest_list/test.hash /boot/vmlinuz-6.4.0-1.0.1.4.oe2309.x86_64 -``` - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->The file name **/boot/vmlinuz-6.4.0-1.0.1.4.oe2309.x86_64** is used as an example. - -**Step 2**: Configure a DIM policy. - -```shell -# echo "measure obj=KERNEL_TEXT" > /etc/dim/policy -``` - -**Step 3**: Trigger dynamic baseline establishment. - -```shell -# echo 1 > /sys/kernel/security/dim/baseline_init -``` - -**Step 4**: Query measurement logs. - -```shell -# cat /sys/kernel/security/dim/ascii_runtime_measurements -0 ef82c39d767dece1f5c52b31d1e8c7d55541bae68a97542dda61b0c0c01af4d2 sha256:5f1586e95b102cd9b9f7df3585fe13a1306cbd464f2ebe47a51ad34128f5d0af 6.4.0-1.0.1.4.oe2309.x86_64 [static baseline] -``` - -The preceding measurement log indicates that the kernel is successfully measured and the current measurement result is used as the baseline value for subsequent measurement. The hash value in the measurement log is not the actual measurement value. - -**Step 5**: Trigger dynamic measurement. - -```shell -# echo 1 > /sys/kernel/security/dim/measure -``` - -After the measurement is complete, you can perform **Step 4** to query the measurement logs. If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. - -#### Measuring the dim_core Module - -**Prerequisites**: - -- The dim_core and dim_monitor modules are loaded successfully. -- The measurement policy has been configured. - -**Step 1**: Trigger dynamic baseline establishment for dim_core. - -```shell -# echo 1 > /sys/kernel/security/dim/baseline_init -``` - -**Step2**: Trigger dynamic baseline establishment for dim_monitor. - -```shell -# echo 1 > /sys/kernel/security/dim/monitor_baseline -``` - -**Step 3**: Query dim_monitor measurement logs. - -```shell -# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements -0 c1b0d9909ddb00633fc6bbe7e457b46b57e165166b8422e81014bdd3e6862899 sha256:35494ed41109ebc9bf9bf7b1c190b7e890e2f7ce62ca1920397cd2c02a057796 dim_core.text [dynamic baseline] -0 9be7121cd2c215d454db2a8aead36b03d2ed94fad0fbaacfbca83d57a410674f sha256:f35d20aae19ada5e633d2fde6e93133c3b6ae9f494ef354ebe5b162398e4d7fa dim_core.data [dynamic baseline] -``` - -The preceding measurement log indicates that dim_core is successfully measured and the current measurement result is used as the baseline value for subsequent measurement. ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->If you skip dynamic baseline establishment and directly perform measurement, "tampered" is displayed in the log. - -**Step 4**: Trigger dynamic measurement of dim_monitor. - -```shell -# echo 1 > /sys/kernel/security/dim/monitor_run -``` - -If the measurement result is consistent with the dynamic baseline, the measurement logs are not updated. Otherwise, an exception measurement log is added. If dynamic baseline establishment for dim_core is triggered again after the policy is modified, the measurement target changes, and the baseline data managed by dim_core also changes. As a result, the dim_monitor measurement result changes. - -```shell -# echo "measure obj=BPRM_TEXT path=/usr/bin/bash" > /etc/dim/policy -# echo 1 > /sys/kernel/security/dim/baseline_init -``` - -Trigger the measurement of dim_monitor again and query the measurement logs. You can see a measurement log marked with "tampered." - -```shell -# echo 1 > /sys/kernel/security/dim/monitor_run -# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements -0 c1b0d9909ddb00633fc6bbe7e457b46b57e165166b8422e81014bdd3e6862899 sha256:35494ed41109ebc9bf9bf7b1c190b7e890e2f7ce62ca1920397cd2c02a057796 dim_core.text [dynamic baseline] -0 9be7121cd2c215d454db2a8aead36b03d2ed94fad0fbaacfbca83d57a410674f sha256:f35d20aae19ada5e633d2fde6e93133c3b6ae9f494ef354ebe5b162398e4d7fa dim_core.data [dynamic baseline] -0 6a60d78230954aba2e6ea6a6b20a7b803d7adb405acbb49b297c003366cfec0d sha256:449ba11b0bfc6146d4479edea2b691aa37c0c025a733e167fd97e77bbb4b9dab dim_core.data [tampered] -``` - -#### Extending to the TPM PCR - -**Prerequisites**: - -- The TPM 2.0 has been installed in the system. After the following command is executed, the command output is not empty: - - ```shell - # ls /dev/tpm* - /dev/tpm0 /dev/tpmrm0 - ``` - -- The tpm2-tools software package has been installed in the system. After the following command is executed, the command output is not empty: - - ```shell - # rpm -qa tpm2-tools - ``` - -- The measurement policy and static baseline have been configured. - -**Step 1**: Load the dim_core and dim_monitor modules and set the numbers of the PCRs to which the measurement results are extended. In this example, PCR 12 is specified for the dim_core measurement result, and PCR 13 is specified for the dim_monitor measurement result. - -```shell -# modprobe dim_core measure_pcr=12 -# modprobe dim_monitor measure_pcr=13 -``` - -**Step 2**: Trigger baseline establishment for dim_core and dim_monitor. - -```shell -# echo 1 > /sys/kernel/security/dim/baseline_init -# echo 1 > /sys/kernel/security/dim/monitor_baseline -``` - -**Step 3**: Check the measurement logs. Each log is marked with the corresponding TPM PCR number. - -```shell -# cat /sys/kernel/security/dim/ascii_runtime_measurements -12 2649c414d1f9fcac1c8d0df8ae7b1c18b5ea10a162b957839bdb8f8415ec6146 sha256:83110ce600e744982d3676202576d8b94cea016a088f99617767ddbd66da1164 /usr/lib/systemd/systemd [static baseline] -# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements -13 72ee3061d5a80eb8547cd80c73a80c3a8dc3b3e9f7e5baa10f709350b3058063 sha256:5562ed25fcdf557efe8077e231399bcfbcf0160d726201ac8edf7a2ca7c55ad0 dim_core.text [dynamic baseline] -13 8ba44d557a9855c03bc243a8ba2d553347a52c1a322ea9cf8d3d1e0c8f0e2656 sha256:5279eadc235d80bf66ba652b5d0a2c7afd253ebaf1d03e6e24b87b7f7e94fa02 dim_core.data [dynamic baseline] -``` - -**Step 4**: Check the TPM PCRs. Extended values have been written to the corresponding PCRs. - -```shell -# tpm2_pcrread sha256 | grep "12:" - 12: 0xF358AC6F815BB29D53356DA2B4578B4EE26EB9274E553689094208E444D5D9A2 -# tpm2_pcrread sha256 | grep "13:" - 13: 0xBFB9FF69493DEF9C50E52E38B332BDA8DE9C53E90FB96D14CD299E756205F8EA -``` - -#### Enabling Signature Verification - -**Prerequisites**: - -- A public key certificate and a signature private key have been prepared. The signature algorithm is RSA, the hash algorithm is sha256, and the certificate format is DER. You can also run the following commands to generate the files: - - ```shell - # openssl genrsa -out dim.key 4096 - # openssl req -new -sha256 -key dim.key -out dim.csr -subj "/C=AA/ST=BB/O=CC/OU=DD/CN=DIM Test" - # openssl x509 -req -days 3650 -signkey dim.key -in dim.csr -out dim.crt - # openssl x509 -in dim.crt -out dim.der -outform DER - ``` - -- The measurement policy has been configured. - -**Step 1**: Save the DER certificate as **/etc/keys/x509_dim.der**. - -```shell -# mkdir -p /etc/keys -# cp dim.der /etc/keys/x509_dim.der -``` - -**Step 2**: Sign the policy file and static baseline file. The signature file name must be the original file name suffixed with **.sig**. - -```shell -# openssl dgst -sha256 -out /etc/dim/policy.sig -sign dim.key /etc/dim/policy -# openssl dgst -sha256 -out /etc/dim/digest_list/test.hash.sig -sign dim.key /etc/dim/digest_list/test.hash -``` - -**Step 3**: Load the dim_core module and enable the signature verification function. - -```shell -modprobe dim_core signature=1 -``` - -Now, the policy file and static baseline file can be loaded only after they pass the signature verification. -The baseline establishment will fail if it is triggered after the policy file is modified: - -```shell -# echo "" >> /etc/dim/policy -# echo 1 > /sys/kernel/security/dim/baseline_init --bash: echo: write error: Key was rejected by service -``` - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->If the signature verification of a static baseline file fails, dim_core skips the parsing of the file without causing a baseline establishment failure. - -#### Configuring Measurement Algorithms - -**Prerequisites**: - -- The measurement policy has been configured. - -**Step 1**: Load the dim_core and dim_monitor modules and configure the measurement algorithm. The following uses the SM3 algorithm as an example. - -```shell -# modprobe dim_core measure_hash=sm3 -# modprobe dim_monitor measure_hash=sm3 -``` - -**Step 2**: Configure a policy that establishes a static baseline of the SM3 algorithm for the measurement target program. - -```shell -# echo "measure obj=BPRM_TEXT path=/opt/dim/demo/dim_test_demo" > /etc/dim/policy -# dim_gen_baseline -a sm3 /opt/dim/demo/dim_test_demo -o /etc/dim/digest_list/test.hash -``` - -**Step 3**: Trigger baseline establishment. - -```shell -# echo 1 > /sys/kernel/security/dim/baseline_init -# echo 1 > /sys/kernel/security/dim/monitor_baseline -``` - -**Step 4**: Check the measurement logs. Each log is marked with the corresponding hash algorithm. - -```shell -# cat /sys/kernel/security/dim/ascii_runtime_measurements -0 585a64feea8dd1ec415d4e67c33633b97abb9f88e6732c8a039064351d24eed6 sm3:ca84504c02bef360ec77f3280552c006ce387ebb09b49b316d1a0b7f43039142 /opt/dim/demo/dim_test_demo [static baseline] -# cat /sys/kernel/security/dim/monitor_ascii_runtime_measurements -0 e6a40553499d4cbf0501f32cabcad8d872416ca12855a389215b2509af76e60b sm3:47a1dae98182e9d7fa489671f20c3542e0e154d3ce941440cdd4a1e4eee8f39f dim_core.text [dynamic baseline] -0 2c862bb477b342e9ac7d4dd03b6e6705c19e0835efc15da38aafba110b41b3d1 sm3:a4d31d5f4d5f08458717b520941c2aefa0b72dc8640a33ee30c26a9dab74eae9 dim_core.data [dynamic baseline] -``` - -#### Configuring Automatic Measurement - -**Prerequisites**: - -- The measurement policy has been configured. - -**Method 1**: Load the dim_core module and set the automatic measurement interval to 1 minute. - -```shell -modprobe dim_core measure_interval=1 -``` - -After the module is loaded, dynamic baseline establishment is automatically triggered. Then, dynamic measurement is triggered every minute. - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->In this case, do not configure the measurement policy for dim_core to measure its own code sections. Otherwise, false alarms may occur. ->In addition, you need to configure **/etc/dim/policy** in advance. Otherwise, the module fails to be loaded when **measure_interval** is set to **1**. - -**Method 2**: After the dim_core module is loaded, you can configure the measurement interval through the kernel module interface. In the following example, the measurement interval is set to 1 minute. - -```shell -modprobe dim_core -echo 1 > /sys/kernel/security/dim/interval -``` - -In this case, the measurement is not triggered immediately. One minute later, dynamic baseline establishment, or dynamic measurement, is triggered. Subsequently, dynamic measurement is triggered every minute. - -#### Configuring the Measurement Scheduling Time - -**Prerequisites**: - -- The measurement policy has been configured. - -Load the dim_core module and set the measurement scheduling time to 10 ms. - -```shell -modprobe dim_core measure_schedule=10 -``` - -When the dynamic baseline establishment or dynamic measurement is triggered, dim_core schedules the CPU to release 10 ms each time a process is measured. - -## Remote Attestation (Kunpeng Security Library) - -### Introduction - -This project develops basic security software components running on Kunpeng processors. In the early stage, the project focuses on trusted computing fields such as remote attestation to empower security developers in the community. - -### Software Architecture - -On the platform without TEE enabled, this project can provide the platform remote attestation feature, and its software architecture is shown in the following figure: - -![img](./figures/RA-arch-1.png) - -On the platform that has enabled TEE, this project can provide TEE remote attestation feature, and its software architecture is shown in the following figure: - -![img](./figures/RA-arch-2.png) - -### Installation and Configuration - -1. Run the following command to use the RPM package of the Yum installation program: - - ```shell - yum install kunpengsecl-ras kunpengsecl-rac kunpengsecl-rahub kunpengsecl-qcaserver kunpengsecl-attester kunpengsecl-tas kunpengsecl-devel - ``` - -2. Prepare the database environment. Go to the `/usr/share/attestation/ras` directory and run the p`prepare-database-env.sh` script to automatically configure the database environment. - -3. The configuration files required for program running are stored in three paths: current path `./config.yaml`, home path `${HOME}/.config/attestation/ras(rac)(rahub)(qcaserver)(attester)(tas)/config.yaml`, and system path `/etc/attestation/ras(rac)(rahub)(qcaserver)(attester)(tas)/config.yaml`. - -4. (Optional) To create a home directory configuration file, run the `prepare-ras(rac)(hub)(qca)(attester)(tas)conf-env.sh` script in `/usr/share/attestation/ras(rac)(rahub)(qcaserver)(attester)(tas)` after installing the RPM package. - -### Options - -#### RAS Boot Options - -Run the `ras` command to start the RAS program. Note that you need to provide the ECDSA public key in the current directory and name it `ecdsakey.pub`. Options are as follows: - -```console - -H --https HTTP/HTTPS mode switch. The default value is https(true), false=http. - -h --hport RESTful API port listened by RAS in HTTPS mode. - -p, --port string Client API port listened by RAS. - -r, --rest string RESTful API port listened by RAS in HTTP mode. - -T, --token Generates a verification code for test and exits. - -v, --verbose Prints more detailed RAS runtime log information. - -V, --version Prints the RAS version and exits. -``` - -#### RAC Boot Options - -Run the `sudo raagent` command to start the RAC program. Note that the sudo permission is required to enable the physical TPM module. Options are as follows: - -```console - -s, --server string Specifies the RAS service port to be connected. - -t, --test Starts in test mode. - -v, --verbose Prints more detailed RAC runtime log information. - -V, --version Prints the RAC version and exits. - -i, --imalog Specifies the path of the IMA file. - -b, --bioslog Specifies the path of the BIOS file. - -T, --tatest Starts in TA test mode. -``` - -**Note:** ->1.To use TEE remote attestation feature, you must start RAC not in TA test mode. And place the uuid, whether to use TCB, mem_hash and img_hash of the TA to be attestated sequentially in the **talist** file under the RAC execution path. At the same time, pre install the **libqca.so** and **libteec.so** library provided by the TEE team. The format of the **talist** file is as follows: -> ->```text ->e08f7eca-e875-440e-9ab0-5f381136c600 false ccd5160c6461e19214c0d8787281a1e3c4048850352abe45ce86e12dd3df9fde 46d5019b0a7ffbb87ad71ea629ebd6f568140c95d7b452011acfa2f9daf61c7a ->``` -> ->2.To not use TEE remote attestation feature, you must copy the **libqca.so** and **libteec.so** library in `${DESTDIR}/usr/share/attestation/qcaserver` path to `/usr/lib` or `/usr/lib64` path, and start RAC in TA test mode. - -#### QCA Boot Options - -Run the `${DESTDIR}/usr/bin/qcaserver` command to start the QCA program. Note that to start QTA normally, the full path of qcaserver must be used, and the CA path parameter in QTA needs to be kept the same as the path. Options are as follows: - -```console - -C, --scenario int Sets the application scenario of the program, The default value is sce_no_as(0), 1=sce_as_no_daa, 2=sce_as_with_daa. - -S, --server string Specifies the open server address/port. -``` - -#### ATTESTER Boot Options - -Run the `attester` command to start the ATTESTER program. Options are as follows: - -```console - -B, --basevalue string Sets the base value file read path - -M, --mspolicy int Sets the measurement strategy, which defaults to -1 and needs to be specified manually. 1=compare only img-hash values, 2=compare only hash values, and 3=compare both img-hash and hash values at the same time. - -S, --server string Specifies the address of the server to connect to. - -U, --uuid int Specifies the trusted apps to verify. - -V, --version Prints the program version and exit. - -T, --test Reads fixed nonce values to match currently hard-coded trusted reports. -``` - -#### TAS Boot Options - -Run the `tas` command to start the TAS program. Options are as follows: - -```console - -T, --token Generates a verification code for test and exits. -``` - -**Note:** ->1.To enable the TAS, you must configure the private key for TAS. Run the following command to modify the configuration file in the home directory: -> ->```shell ->$ cd ${HOME}/.config/attestation/tas ->$ vim config.yaml -> # The values of the following DAA_GRP_KEY_SK_X and DAA_GRP_KEY_SK_Y are for testing purposes only. -> # Be sure to update their contents to ensure safety before normal use. ->tasconfig: -> port: 127.0.0.1:40008 -> rest: 127.0.0.1:40009 -> akskeycertfile: ./ascert.crt -> aksprivkeyfile: ./aspriv.key -> huaweiitcafile: ./Huawei IT Product CA.pem -> DAA_GRP_KEY_SK_X: 65a9bf91ac8832379ff04dd2c6def16d48a56be244f6e19274e97881a776543c65a9bf91ac8832379ff04dd2c6def16d48a56be244f6e19274e97881a776543c -> DAA_GRP_KEY_SK_Y: 126f74258bb0ceca2ae7522c51825f980549ec1ef24f81d189d17e38f1773b56126f74258bb0ceca2ae7522c51825f980549ec1ef24f81d189d17e38f1773b56 ->``` -> ->Then enter `tas` to start TAS program. -> ->2.In an environment with TAS, in order to improve the efficiency of QCA's certificate configuration process, not every boot needs to access the TAS to generate the certificate, but through the localized storage of the certificate. That is, read the certification path configured in `config.yaml` on QCA side, check if a TAS-issued certificate has been saved locally through the `func hasAKCert(s int) bool` function. If the certificate is successfully read, there is no need to access TAS. If the certificate cannot be read, you need to access TAS and save the certificate returned by TAS locally. - -### API Definition - -#### RAS APIs - -To facilitate the administrator to manage the target server, RAS and the user TA in the TEE deployed on the target server, the following APIs are designed for calling: - -| API | Method | -| --------------------------------- | --------------------------- | -| / | GET | -| /{id} | GET, POST, DELETE | -| /{from}/{to} | GET | -| /{id}/reports | GET | -| /{id}/reports/{reportid} | GET, DELETE | -| /{id}/basevalues | GET | -| /{id}/newbasevalue | POST | -| /{id}/basevalues/{basevalueid} | GET, POST, DELETE | -| /{id}/ta/{tauuid}/status | GET | -| /{id}/ta/{tauuid}/tabasevalues | GET | -| /{id}/ta/{tauuid}/tabasevalues/{tabasevalueid} | GET, POST, DELETE | -| /{id}/ta/{tauuid}/newtabasevalue | POST | -| /{id}/ta/{tauuid}/tareports | GET | -| /{id}/ta/{tauuid}/tareports/{tareportid} | GET, POST, DELETE | -| /{id}/basevalues/{basevalueid} | GET, DELETE | -| /version | GET | -| /config | GET, POST | -| /{id}/container/status | GET | -| /{id}/device/status | GET | - -The usage of the preceding APIs is described as follows: - -To query information about all servers, use `/`. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/ -``` - -*** -To query detailed information about a target server, use the GET method of `/{id}`. **{id}** is the unique ID allocated by RAS to the target server. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/1 -``` - -*** -To modify information about the target server, use the POST method of `/{id}`. `$AUTHTOKEN` is the identity verification code automatically generated by running the `ras -T` command. - -```go -type clientInfo struct { - Registered *bool `json:"registered"` // Registration status of the target server - IsAutoUpdate *bool `json:"isautoupdate"`// Target server base value update policy -} -``` - -```shell -curl -X POST -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1 -d '{"registered":false, "isautoupdate":false}' -``` - -*** -To delete a target server, use the DELETE method of `/{id}`. - -**Note:** ->This method does not delete all information about the target server. Instead, it sets the registration status of the target server to `false`. - -```shell -curl -X DELETE -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1 -``` - -*** -To query information about all servers in a specified range, use the GET method of `/{from}/{to}`. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/9 -``` - -*** -To query all trust reports of the target server, use the GET method of `/{id}/reports`. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/reports -``` - -*** -To query details about a specified trust report of the target server, use the GET method of `/{id}/reports/{reportid}`. **{reportid}** indicates the unique ID assigned by RAS to the trust report of the target server. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/reports/1 -``` - -*** -To delete a specified trust report of the target server, use the DELETE method of `/{id}/reports/{reportid}`. - -**Note:** ->This method will delete all information about the specified trusted report, and the report cannot be queried through the API. - -```shell -curl -X DELETE -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1/reports/1 -``` - -*** -To query all base values of the target server, use the GET method of `/{id}/reports/{reportid}`. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/basevalues -``` - -*** -To add a base value to the target server, use the POST method of `/{id}/newbasevalue`. - -```go -type baseValueJson struct { - BaseType string `json:"basetype"` // Base value type - Uuid string `json:"uuid"` // ID of a container or device - Name string `json:"name"` // Base value name - Enabled bool `json:"enabled"` // Whether the base value is available - Pcr string `json:"pcr"` // PCR value - Bios string `json:"bios"` // BIOS value - Ima string `json:"ima"` // IMA value - IsNewGroup bool `json:"isnewgroup"` // Whether this is a group of new reference values -} -``` - -```shell -curl -X POST -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1/newbasevalue -d '{"name":"test", "basetype":"host", "enabled":true, "pcr":"testpcr", "bios":"testbios", "ima":"testima", "isnewgroup":true}' -``` - -*** -To query details about a specified base value of a target server, use the get method of `/{id}/basevalues/{basevalueid}`. **{basevalueid}** indicates the unique ID allocated by RAS to the specified base value of the target server. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/1/basevalues/1 -``` - -*** -To change the availability status of a specified base value of the target server, use the POST method of `/{id}/basevalues/{basevalueid}`. - -```shell -curl -X POST -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40002/1/basevalues/1 -d '{"enabled":true}' -``` - -*** -To delete a specified base value of the target server, use the DELETE method of `/{id}/basevalues/{basevalueid}`. - -**Note:** ->This method will delete all the information about the specified base value, and the base value cannot be queried through the API. - -```shell -curl -X DELETE -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/1/basevalues/1 -``` - -To query the trusted status of a specific user TA on the target server, use the GET method of the `"/{id}/ta/{tauuid}/status"` interface. Where {id} is the unique identification number assigned by RAS to the target server, and {tauuid} is the identification number of the specific user TA. - -```shell -curl -X GET -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40002/1/ta/test/status -``` - -*** -To query all the baseline value information of a specific user TA on the target server, use the GET method of the `"/{id}/ta/{tauuid}/tabasevalues"` interface. - -```shell -curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tabasevalues -``` - -*** -To query the details of a specified base value for a specific user TA on the target server, use the GET method of the `"/{id}/ta/{tauuid}/tabasevalues/{tabasevalueid}"` interface. where {tabasevalueid} is the unique identification number assigned by RAS to the specified base value of a specific user TA on the target server. - -```shell -curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tabasevalues/1 -``` - -*** -To modify the available status of a specified base value for a specific user TA on the target server, use the `POST` method of the `"/{id}/ta/{tauuid}/tabasevalues/{tabasevalueid}"` interface. - -```shell -curl -X POST -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40002/1/ta/test/tabasevalues/1 --data '{"enabled":true}' -``` - -*** -To delete the specified base value of a specific user TA on the target server, use the `DELETE` method of the `"/{id}/ta/{tauuid}/tabasevalues/{tabasevalueid}"` interface. - -**Note:** ->This method will delete all information about the specified base value, and the base value cannot be queried through the API. - -```shell -curl -X DELETE -H "Content-type: application/json" -H "Authorization: $AUTHTOKEN" -k http://localhost:40002/1/ta/test/tabasevalues/1 -``` - -*** -To add a baseline value to a specific user TA on the target server, use the `POST` method of the `"/{id}/ta/{tauuid}/newtabasevalue"` interface. - -```go -type tabaseValueJson struct { - Uuid string `json:"uuid"` // the identification number of the user TA - Name string `json:"name"` // base value name - Enabled bool `json:"enabled"` // whether a baseline value is available - Valueinfo string `json:"valueinfo"` // mirror hash value and memory hash value -} -``` - -```shell -curl -X POST -H "Content-Type: application/json" -H "Authorization: $AUTHTOKEN" -k http://localhost:40002/1/ta/test/newtabasevalue -d '{"uuid":"test", "name":"testname", "enabled":true, "valueinfo":"test info"}' -``` - -*** -To query the target server for all trusted reports for a specific user TA, use the `GET` method of the `"/{id}/ta/{tauuid}/tareports"` interface. - -```shell -curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tareports -``` - -*** -To query the details of a specified trusted report for a specific user TA on the target server, use the `GET` method of the `"/{id}/ta/{tauuid}/tareports/{tareportid}"` interface. Where {tareportid} is the unique identification number assigned by RAS to the specified trusted report of a specific user TA on the target server. - -```shell -curl -X GET -H "Content-type: application/json" http://localhost:40002/1/ta/test/tareports/2 -``` - -*** -To delete the specified trusted report of a specific user TA on the target server, use the `DELETE` method of the `"/{id}/ta/{tauuid}/tareports/{tareportid}"` interface. - -**Note:** ->This method will delete all information of the specified trusted report, and the report cannot be queried through the API. - -```shell -curl -X DELETE -H "Content-type: application/json" http://localhost:40002/1/ta/test/tareports/2 -``` - -*** -To obtain the version information of the program, use the GET method of `/version`. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/version -``` - -*** -To query the configuration information about the target server, RAS, or database, use the GET method of `/config`. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40002/config -``` - -*** -To modify the configuration information about the target server, RAS, or database, use the POST method of /config. - -```go -type cfgRecord struct { - // Target server configuration - HBDuration string `json:"hbduration" form:"hbduration"` - TrustDuration string `json:"trustduration" form:"trustduration"` - DigestAlgorithm string `json:"digestalgorithm" form:"digestalgorithm"` - // RAS configuration - MgrStrategy string `json:"mgrstrategy" form:"mgrstrategy"` - ExtractRules string `json:"extractrules" form:"extractrules"` - IsAllupdate *bool `json:"isallupdate" form:"isallupdate"` - LogTestMode *bool `json:"logtestmode" form:"logtestmode"` -} -``` - -```shell -curl -X POST -H "Authorization: $AUTHTOKEN" -H "Content-Type: application/json" http://localhost:40002/config -d '{"hbduration":"5s","trustduration":"20s","DigestAlgorithm":"sha256"}' -``` - -#### TAS APIs - -To facilitate the administrator's management of TAS for remote control, the following API is designed for calling: - -| API | Method | -| --------------------| ------------------| -| /config | GET, POST | - -To query the configuration information, use the GET method of the `/config` interface. - -```shell -curl -X GET -H "Content-Type: application/json" http://localhost:40009/config -``` - -*** -To modify the configuration information, use the POST method of the `/config` interface. - -```shell -curl -X POST -H "Content-Type: application/json" -H "Authorization: $AUTHTOKEN" http://localhost:40009/config -d '{"basevalue":"testvalue"}' -``` - -**Note:** ->Currently, only the base value in the configuration information of TAS is supported for querying and modifying. - -### FAQs - -1. Why cannot RAS be started after it is installed? - - > In the current RAS design logic, after the program is started, it needs to search for the `ecdsakey.pub` file in the current directory and read the file as the identity verification code for accessing the program. If the file does not exist in the current directory, an error is reported during RAS boot. - >> Solution 1: Run the `ras -T` command to generate a test token. The `ecdsakey.pub` file is generated. - >> Solution 2: After deploying the oauth2 authentication service, save the verification public key of the JWT token generator as `ecdsakey.pub`. - -2. Why cannot RAS be accessed through REST APIs after it is started? - - > RAS is started in HTTPS mode by default. Therefore, you need to provide a valid certificate for RAS to access it. However, RAS started in HTTP mode does not require a certificate. - -## Trusted Platform Control Module - -### Background - -Trusted computing has undergone continuous development and improvement in the past 40 years and has become an important branch of information security. Trusted computing technologies have developed rapidly in recent years and have solved the challenges in Trusted Computing 2.0—integration of trusted systems and existing systems, trusted management, and simplification of trusted application development. These technical breakthroughs form Trusted Computing 3.0, that is, trusted computing based on an active immune system. Compared with the passive plug-in architecture of the previous generation, Trusted Computing 3.0 proposes a new trusted system framework based on self-controlled cryptography algorithms, control chips, trusted software, trusted connections, policy management, and secure and trusted protection applications, implementing trust across the networks. - -The trusted platform control module (TPCM) is a base and core module that can be integrated into a trusted computing platform to establish and ensure a trust source. As one of the innovations in Trusted Computing 3.0 and the core of active immunity, TPCM implements active control over the entire platform. - -The TPCM-based Trusted Computing 3.0 architecture consists of the protection module and the computing module. On the one hand, based on the Trusted Cryptography Module (TPM), the TPCM main control firmware measures the reliability of the protection and computing modules, as well as their firmware. On the other hand, the Trusted Software Base (TSB) measures the reliability of system software and application software. In addition, the TPCM management platform verifies the reliability measurement and synchronizes and manages the trust policies. - -### Feature Description - -The overall system design consists of the protection module, computing module, and trusted management center software, as shown in the following figure. - -![](./figures/TPCM.png) - -- Trusted management center: This centralized management platform, provided by a third-party vendor, formulates, delivers, maintains, and stores protection policies and reference values for trusted computing nodes. -- Protection module: This module operates independently of the computing module and provides trusted computing protection functions that feature active measurement and active control to implement security protection during computing. The protection module consists of the TPCM main control firmware, TCB, and TCM. As a key module for implementing trust protection in a trusted computing node, the TPCM can be implemented in multiple forms, such as cards, chips, and IP cores. It contains a CPU and memory, firmware, and software such as an OS and trusted function components. The TPCM operates alongside the computing module and works according to the built-in protection policy to monitor the trust of protected resources, such as hardware, firmware, and software of the computing module. The TPCM is the Root of Trust in a trusted computing node. - -- Computing module: This module includes hardware, an OS, and application layer software. The running of the OS can be divided into the boot phase and the running phase. In the boot phase, GRUB2 and shim of openEuler support the reliability measurement capability, which protects boot files such as shim, GRUB2, kernel, and initramfs. In the running phase, openEuler supports the deployment of the trusted verification agent (provided by third-party vendor HTTC). The agent sends data to the TPCM for trusted measurement and protection in the running phase. - -The TPCM interacts with other components as follows: - -1. The TPCM hardware, firmware, and software provide an operating environment for the TSB. The trusted function components of the TPCM provide support for the TSB to implement measurement, control, support, and decision-making based on the policy library interpretation requirements. -2. The TPCM accesses the TCM for trusted cryptography functions to complete computing tasks such as trusted verification, measurement, and confidential storage, and provides services for TCM access. -3. The TPCM connects to the trusted management center through the management interface to implement protection policy management and trusted report processing. -4. The TPCM uses the built-in controller and I/O port to interact with the controller of the computing module through the bus to actively monitor the computing module. -5. The built-in protection agent in the OS of the computing module obtains the code and data related to the preset protection object and provides them to the TPCM. The TPCM forwards the monitoring information to the TSB, and the TSB analyzes and processes the information according to the policy library. - -### Constraints - -Supported server: TaiShan 200 server (model 2280) -Supported BMC card: BC83SMMC - -### Application Scenarios - -The TPCM enables a complete trust chain to ensure that the OS boots into a trusted computing environment. diff --git a/docs/en/docs/GCC/full-scenario-link-time-binary-inline-optimization.md b/docs/en/docs/GCC/full-scenario-link-time-binary-inline-optimization.md new file mode 100644 index 0000000000000000000000000000000000000000..333213c79861fa330331dae179656ad070305ff5 --- /dev/null +++ b/docs/en/docs/GCC/full-scenario-link-time-binary-inline-optimization.md @@ -0,0 +1,56 @@ +# Full-Scenario Link-Time Binary Inline Optimization + +This feature enables full-scenario link-time binary inlining by combining multi-version binary outputs and resolving multi-version symbols through link-time optimization (LTO) plugins. It supports inline optimization for LTO across different compiler versions and implements cross-module compilation option analysis and matching to facilitate the integration of different compilation modules, achieving full-scenario LTO. + +## `-fmulti-version-lib=` Option + +### Description + +This link-time option works with `-flto` to indicate that the input library is in a multi-version LTO format. The compiler will switch to an older LTO format for reading (currently supporting LTO formats from openEuler 24.03 LTS SP1 and openEuler 24.09). This ensures compatibility and optimization when linking static libraries or object files with different LTO formats. + +### Usage + +Include the option, separating multiple file names with commas: + +```bash +-flto -fmulti-version-lib=liba.a,libb.a +``` + +Example: + +```bash +# GCC for openEuler 24.09 +gcc -O2 -fPIC -flto -c fa.c -o fa.o +gcc-ar rcs liba.a fa.o + +# GCC for openEuler latest +gcc -O2 -fPIC -flto -fmulti-version-lib=liba.a main.c liba.a -o exe +``` + +## `-finline-force=` Option + +### Description + +This link-time option works with `-flto` to instruct the compiler to attempt inlining for the input static libraries or object files during LTO-enabled linking. It enhances inlining for functions in the target files, improving the following: + +- Architecture option compatibility: When the calling and called functions use different `march`/`mcpu` information, it aligns the architecture options of the called function with those of the calling function and performs inlining. +- `inline` keyword: Similar to adding the `inline` keyword to the called function, it ensures inlining whenever possible during compilation. +- `always_inline` attribute: Similar to adding the `__attribute__((always_inline))` attribute to the called function. + +### Usage + +Include the option, separating multiple file names with commas: + +```bash +-flto -finline-force=liba.a,libb.a +``` + +Note: Using `-finline-force` without specifying target file names is reserved for global inlining debugging and analysis and should not be used directly. + +Example: + +```bash +gcc -O2 -fPIC -flto -c fa.c -o fa.o +gcc-ar rcs liba.a fa.o +gcc -O2 -fPIC -flto -finline -force=liba.a main.c liba.a -o exe +``` diff --git a/docs/en/docs/Installation/RISC-V-LicheePi4A.md b/docs/en/docs/Installation/RISC-V-LicheePi4A.md index d10c5bf708ea9b9dda285614659b81ef937b8072..0cc60b823395cfe2215ef15cde08ca019d33383c 100644 --- a/docs/en/docs/Installation/RISC-V-LicheePi4A.md +++ b/docs/en/docs/Installation/RISC-V-LicheePi4A.md @@ -1,96 +1,109 @@ -# Installing on Licheepi4A +# Installation on LicheePi 4A -## Hardware Preparation +## Hardware Requirements -- `Sipeed LicheePi 4A` device (either `8 GB` or `16 GB` version) +- LicheePi 4A device (8 GB or 16 GB model) - Monitor -- `USB` keyboard and mouse -- Equipment/components required for serial operation (optional) -- `RJ45` network cable and router/switch for wired network connection +- USB keyboard and mouse +- Serial communication equipment (optional) +- RJ45 network cable and router/switch for network connectivity ## Device Firmware -Different memory versions of `LicheePi4A` require different firmware: +LicheePi 4A uses different firmware based on memory capacity: -- `u-boot-with-spl-lpi4a.bin` is the u-boot file for the 8 GB version. -- `u-boot-with-spl-lpi4a-16g.bin` is the u-boot file for the 16 GB version. +- 8 GB version: **u-boot-with-spl-lpi4a.bin** +- 16 GB version: **u-boot-with-spl-lpi4a-16g.bin** -The following flashing method uses the `16GB + 128GB` core board as an example, assuming the user already downloaded `base` image and the corresponding u-boot file. +The following instructions detail the flashing process for the 16 GB + 128 GB core board, assuming you have downloaded the base image and firmware files. -## Flashing Method +## Flashing Process -### Flashing Tools +### Tools -Please use `fastboot` command for flashing. You can download `burn_tools.zip` from `https://dl.sipeed.com/shareURL/LICHEE/licheepi4a/07_Tools`. The archive contains flashing tools for `Windows`, `macOS`, and `Linux`. +Use `fastboot` for flashing. Download **burn_tools.zip** from ****, which contains tools for Windows, macOS, and Linux. -### Set Hardware to Enter Flashing Mode +### Entering Flash Mode -> Please first check that the DIP switch on the baseboard is set to EMMC boot mode. After confirming, you can proceed with the flashing. +> Verify the base board DIP switch is set to EMMC boot mode before proceeding. -Hold down the `BOOT` button on the board, then insert the `USB-C` cable to power on the device (the other end of the cable should be connected to a `PC`), entering `USB` flashing mode. -On `Windows`, check the device manager for the `USB download gadget` device. -On `Linux`, use `lsusb` to check for the device, showing: `ID 2345:7654 T-HEAD USB download gadget`. +To enter USB flash mode: -### Driver Installation on Windows +1. Hold down the **BOOT** button. +2. Connect the USB-C cable between the board and your PC. + +Device detection: + +- Windows: Look for **USB download gadget** in **Device Manager**. +- Linux: Run `lsusb` to find **ID 2345:7654 T-HEAD USB download gadget**. + +### Installing Windows Drivers + +#### Before You Begin > Note: -> The provided image does not include Windows drivers. You can download `burn_tools.zip` [here](https://dl.sipeed.com/shareURL/LICHEE/licheepi4a/07_Tools) and find the `windows/usb_driver-fullmask` folder inside. This folder contains the drivers needed for Windows. +> +> Windows drivers are not included in the base image. Get them by downloading **burn_tools.zip** [here](https://dl.sipeed.com/shareURL/LICHEE/licheepi4a/07_Tools). Look for the drivers in the **windows/usb_driver-fullmask** folder. -To flash on Windows, you need to enter advanced startup mode and disable digital signature enforcement. Follow the steps below to disable digital signature enforcement: +Installation requires Windows driver signature enfowrcement to be disabled: -#### Windows 10 +#### Windows 10 Setup -1. Go to `Settings` -> `Update & Security` -2. Click `Recovery` on the left, then click `Restart now` under `Advanced startup`. Your computer will restart. Save any ongoing work before proceeding. +1. Go to **Settings** > **Update & Security**. +2. Select **Recovery**. +3. Choose **Restart now** under **Advanced startup**. + > Close all programs and save your work first. -#### Windows 11 +#### Windows 11 Setup -1. Go to `Settings` -> `System` -> `Recovery` -2. Click `Restart now` under `Advanced startup`. Your computer will restart. Save any ongoing work before proceeding. +1. Go to **Settings** > **System** +2. Select **Recovery** +3. Choose **Restart now** under **Advanced startup** + > Close all programs and save your work first. -##### After Restart +#### Complete the Installation -1. Click `Troubleshoot`, then `Advanced options` -> `Startup Settings`. The system will restart again. -2. After restarting, select `Disable driver signature enforcement`. This option is usually number 7 but may vary. After selecting the appropriate option, the system will restart again. -3. After rebooting into the system, install the driver. Open `Device Manager`, find `USB download gadget` under `Other devices`, and double-click it. -4. Click `Update driver` under the `General` tab. -5. On the `Browse my computer for drivers` page, paste the path to the `usb_driver-fullmask` directory. -6. Click `Next` to install the driver. +1. After restart, navigate through **Troubleshoot** > **Advanced options** > **Startup Settings** > **Restart**. +2. On the next screen, press the key (usually 7) for **Disable driver signature enforcement**. +3. Once Windows restarts, launch **Device Manager**, find **USB download gadget** in **Other devices**, and double-click it. +4. Choose **Update Driver**. +5. Select **Browse my computer for drivers** and find the **usb_driver-fullmask** folder. +6. Follow prompts to finish installation. ### Flashing the Image -After entering flashing mode, use fastboot to flash the image. On macOS or Linux, if fastboot is self-installed, you may need to give it executable permissions. +After entering flash mode, use `fastboot` to flash the image. For macOS or Linux users who manually installed `fastboot`, ensure it has executable permissions. -#### Windows Steps +#### Windows Setup -First, add `fastboot` to the system environment variable `PATH`, or place `fastboot` in the same directory. Also, extract the image files. Open `PowerShell` and execute the following commands: +Either add `fastboot` to **PATH** of the system or place it in the working directory. Extract the image files, launch PowerShell, and run: -```bash -# Replace with the u-boot file corresponding to your board version +```shell +# Use the u-boot file for the board version. fastboot flash ram u-boot-with-spl-lpi4a-16g.bin fastboot reboot -# After rebooting, wait 5 seconds before continuing -# Replace with the u-boot file corresponding to your board version +# Pause 5 seconds after reboot. +# Use the u-boot file for the board version. fastboot flash uboot u-boot-with-spl-lpi4a-16g.bin -fastboot flash boot openEuler-24.03-LTS-riscv64-lpi4a-base-boot.ext4 -fastboot flash root openEuler-24.03-LTS-riscv64-lpi4a-base-root.ext4 +fastboot flash boot openEuler-24.03-LTS-SP1-riscv64-lpi4a-base-boot.ext4 +fastboot flash root openEuler-24.03-LTS-SP1-riscv64-lpi4a-base-root.ext4 ``` -#### Linux/macOS Steps +#### Linux or macOS Setup -You may need to prefix the fastboot commands with `sudo`. +Run the same commands but prefix with `sudo` if necessary: ```bash -# Replace with the u-boot file corresponding to your board version -sudo fastboot flash ram u-boot-with-spl-lpi4a-16g.bin -sudo fastboot reboot -# After rebooting, wait 5 seconds before continuing -# Replace with the u-boot file corresponding to your board version -sudo fastboot flash uboot u-boot-with-spl-lpi4a-16g.bin -sudo fastboot flash boot openEuler-24.03-LTS-riscv64-lpi4a-base-boot.ext4 -sudo fastboot flash root openEuler-24.03-LTS-riscv64-lpi4a-base-root.ext4 +# Use the u-boot file for the board version. +fastboot flash ram u-boot-with-spl-lpi4a-16g.bin +fastboot reboot +# Pause 5 seconds after reboot. +# Use the u-boot file for the board version. +fastboot flash uboot u-boot-with-spl-lpi4a-16g.bin +fastboot flash boot openEuler-24.03-LTS-SP1-riscv64-lpi4a-base-boot.ext4 +fastboot flash root openEuler-24.03-LTS-SP1-riscv64-lpi4a-base-root.ext4 ``` -## Hardware Availability +## Hardware Support -The official release is based on the [`openEuler kernel6.6`](./RISCV-OLK6.6.md) version, and not all kernel modules are fully supported. This version emphasizes a consistent official ecosystem experience. For more complete hardware functionality, use third-party releases. +The official release builds on [openEuler kernel 6.6](./RISC-V-OLK6.6-version-alignment.md) but includes limited kernel module support. It focuses on ecosystem compatibility rather than complete hardware support. For full hardware functionality, use third-party releases instead. diff --git a/docs/en/docs/Installation/RISCV-OLK6.6.md b/docs/en/docs/Installation/RISC-V-OLK6.6-version-alignment.md similarity index 44% rename from docs/en/docs/Installation/RISCV-OLK6.6.md rename to docs/en/docs/Installation/RISC-V-OLK6.6-version-alignment.md index 7a91466c592886e355b64ec999eb6f2f7dc63197..a271f971c61ad5f7812dbd237ade2e2a7bd42e3c 100644 --- a/docs/en/docs/Installation/RISCV-OLK6.6.md +++ b/docs/en/docs/Installation/RISC-V-OLK6.6-version-alignment.md @@ -1,24 +1,23 @@ -# RISCV-OLK6.6 Source-Compatible Version Guide +# RISC-V-OLK6.6 Version Alignment Guide -## RISCV-OLK6.6 Source-Compatible Plan +## RISC-V-OLK6.6 Version Alignment Project -Currently, the kernel versions maintained by various RISC-V SoC manufacturers are inconsistent, while the openEuler system requires a unified kernel version. This inconsistency results in various operating system versions based on different development boards having divergent third-party kernels, increasing maintenance difficulty and causing ecosystem fragmentation. The goal of the riscv-kernel project is to establish a unified kernel ecosystem for the RISC-V architecture in openEuler, sharing the benefits of Euler's ecosystem development and influence. This project is under development, and contributions from all parties are welcome. +The current landscape of RISC-V kernel development faces a challenge: different SoC manufacturers maintain different kernel versions, while openEuler requires kernel consistency across its system releases. This has led to fragmented third-party versions across development boards, increasing maintenance complexity and ecosystem fragmentation. The riscv-kernel project addresses this by creating a unified kernel ecosystem for RISC-V architecture within openEuler, leveraging and contributing to the broader openEuler ecosystem. The project is actively developing and welcomes contributions from all stakeholders. -The project is primarily developed on the OLK-6.6 branch at and will be further integrated into OLK's source and artifact repositories. +Development is centered on the OLK-6.6 branch at , with planned integration into the source and artifact repositories of OLK. -![riscv-olk6.6](figures/riscv-olk6.6.jpg) +![riscv-olk6.6](figures/riscv-olk6.6.drawio.png) -The project has nearly completed the source compatibility work for SG2042 and the basic source compatibility work for TH1520. +The project has successfully aligned versions for SG2042 and established foundational alignment for TH1520. ## Supported Features -SG2042 Verification Platform: MilkV Pioneer 1.3 +- SG2042 verification platform: MilkV Pioneer 1.3 +- TH1520 verification platform: LicheePi4A -TH1520 Verification Platform: LicheePi4A +### Milk-V Pioneer Feature List -### Milk-V Pioneer feature status - -| Features | Status | +| Feature | Status | | ----------------------- | :----: | | 64 Core CPU | O | | PCIe Network Card | O | @@ -33,9 +32,9 @@ TH1520 Verification Platform: LicheePi4A | SPI flash | O | | RVV 0.71 | X | -### LicheePi 4A feature status +### LicheePi 4A Feature List -| Features | Status | +| Feature | Status | | ---------------- | :----: | | 4 Core CPU | O | | RAM | O | diff --git a/docs/en/docs/Installation/RISC-V-Pioneer1.3.md b/docs/en/docs/Installation/RISC-V-Pioneer1.3.md index feb321f4e7886cdbaf76dc8070f001c05367e8f2..fd95d1cde53313340b264bfcb72aba453760a66b 100644 --- a/docs/en/docs/Installation/RISC-V-Pioneer1.3.md +++ b/docs/en/docs/Installation/RISC-V-Pioneer1.3.md @@ -1,104 +1,87 @@ -# Installing on Pioneer Box +# Installation on Pioneer Box -## Hardware Preparation +## Hardware Requirements -- `Milk-V Pioneer v1.3` device or motherboard (with necessary peripherals) - 1 set +- Milk-V Pioneer v1.3 or motherboard kit with essential peripherals +- m.2 NVMe SSD + > Clear existing data before use (backup important files). + > Installation options: + > - First PCIe slot with adapter (recommended) + > - Onboard NVMe slot +- AMD R5 230 GPU + > Install in the second PCIe slot. +- USB drive (16 GiB minimum) +- microSD card (4 GiB minimum) +- Monitor compatible with the GPU +- USB keyboard and mouse +- Serial communication equipment (optional) +- RJ45 network cable and router/switch for network connectivity + > Use onboard RJ45 port for best results. + > WiFi or Bluetooth device is not included. Add compatible cards if necessary. -- `m.2 NVMe` solid-state drive - 1 unit +## Image Types -> If it contains data, format it to clear the data (make sure to back up personal files) -> -> If you have a `PCIe` adapter card, place it in the first `PCIe` slot of the device (recommended) -> -> If no `PCIe` adapter card, use the onboard `NVMe` interface +### ISO Images -- `AMD R5 230` graphics card - 1 unit +> Compatible with UEFI boot and **UEFI firmware** version. -> Place it in the second `PCIe` slot of the device +Obtain the ISO (such as **openEuler-24.03-LTS-SP1-riscv64-dvd.iso**) from the official site and write to USB using: -- `USB` flash drive - 1 unit +- Balena Etcher (recommended): +- CLI alternative: -> Should be at least `16GiB` + ```shell + ~$ sudo dd if=openEuler-24.03-LTS-SP1-riscv64-dvd.iso of=/dev/sda bs=512K iflag=fullblock oflag=direct conv=fsync status=progress + ``` -- `microSD card` - 1 unit +### Raw Images -> Should be at least `4GiB` +> Compatible with Legacy boot and **non-UEFI firmware** version. -- Monitor (the display interface should match the graphics card) - -- `USB` keyboard and mouse - 1 set - -- Equipment/components required for serial operation (optional) - -- `RJ45` network cable - at least 1, and router/switch for wired network connection - -> It is recommended to use the device's onboard `RJ45` network port rather than the manufacturer's provided `PCIe` network card. -> -> The device does not come with a `WiFi` network card and does not support WiFi or Bluetooth connectivity. Please prepare the corresponding equipment if needed. - -## Types of Images - -### ISO - -> `ISO` images support booting via `UEFI`, corresponding to the **UEFI version** firmware described below. - -Download the `ISO` file (e.g., `openEuler-24.03-LTS-riscv64-dvd.iso`) from the official download page and burn it to a **USB flash drive**. - -- It is recommended to use the `Balena Etcher` software for graphical burning [download from `https://github.com/balena-io/etcher/releases/latest`]. The burning process is not detailed here. -- In a command-line environment, you can also use the `dd` method to burn the image. Refer to the following command: - -```txt -~$ sudo dd if=openEuler-24.03-LTS-riscv64-dvd.iso of=/dev/sda bs=512K iflag=fullblock oflag=direct conv=fsync status=progress -``` - -### Image - -> `Image` images support booting via `Legacy`, corresponding to the **non-UEFI version** firmware described below. - -Download the `Zip` archive containing the image (e.g., `openEuler-24.03-LTS-riscv64-sg2042.img.zip`) from the official download page and burn it directly to an **SDCARD** or **solid-state drive**. +Download the zipped image (such as **openEuler-24.03-LTS-SP1-riscv64-sg2042.img.zip**) and write directly to the SD card or SSD. ## Device Firmware -> Since the device's factory firmware currently does not support `UEFI`, users of the `ISO` version need to manually replace the firmware with the `UEFI` version based on `EDK2`. +> Factory firmware lacks UEFI support. ISO users must install EDK2-based UEFI firmware. -Download the device firmware archive `sg2042_firmware_uefi.zip` from the official download page under the **Embedded category**, extract it, and burn the `img` file to an **SDCARD**. +Obtain **sg2042_firmware_uefi.zip** from the **Embedded** section on the official website. Extract and write the .img file to the SD card: ```txt ~$ sudo dd if=firmware_single_sg2042-master.img of=/dev/sda bs=512K iflag=fullblock oflag=direct conv=fsync status=progress -261619712 bytes (262 MB, 250 MiB) copied, 20 s, 13.1 MB/s -268435456 bytes (268 MB, 256 MiB) copied, 20.561 s, 13.1 MB/s +261,619,712 bytes (262 MB, 250 MiB) copied, 20 s, 13.1 MB/s +268,435,456 bytes (268 MB, 256 MiB) copied, 20.561 s, 13.1 MB/s 512+0 records in 512+0 records out -268435456 bytes (268 MB, 256 MiB) copied, 20.5611 s, 13.1 MB/s +268,435,456 bytes (268 MB, 256 MiB) copied, 20.5611 s, 13.1 MB/s ``` -> Due to the older firmware version shipped with the device, Image users who want to use a newer version of the firmware can update to the **non-UEFI version** of the firmware. +> Raw image users can upgrade older factory firmware to newer **non-UEFI** version. -Download the device firmware package `sg2042_firmware_uboot.zip` from the embedded category on the download page, and follow the same procedure as the UEFI firmware to extract and flash the img file to the **SDCARD**. +Obtain **sg2042_firmware_uboot.zip** from the **Embedded** section on the official website. Follow UEFI firmware steps to write the .img file to the SD card. -After burning, insert the **SDCARD** into the device's card slot. +Insert the SD card into the device slot when after flashing. -## Pre-Startup Checks +## Pre-boot Checks -For `ISO` version users: +For ISO users: -- Ensure the `microSD card` with the `UEFI` firmware is inserted into the device's card slot. +- Ensure the microSD card containing UEFI firmware is in the device slot. - > The current `UEFI` firmware cannot manually adjust or specify the boot order, please understand. + > Note: Manual boot order configuration is not yet supported in UEFI firmware. -- If using the factory-provided solid-state drive, or if another bootable `RISC-V` operating system exists on the drive, remove the solid-state drive for formatting or replace it with another empty solid-state drive to avoid interference with the boot order. +- Format or replace the factory SSD or any drive containing a bootable RISC-V OS to avoid boot conflicts. -For `Image` version users: +For Image users: -- If using the factory-provided solid-state drive, or if another bootable `RISC-V` operating system exists on the drive, remove the solid-state drive for formatting or replace it with another empty solid-state drive to avoid interference with the boot order. +- Format or replace the factory SSD or any drive containing a bootable RISC-V OS to avoid boot conflicts. -## Notes for Use +## Usage Notes -For `ISO` version users: +For ISO users: -- Due to the limitations of the current version of the `UEFI` firmware, the `Grub2` boot menu may take a long time (~15s) to load and respond slowly if the graphics card is inserted into the `PCIe` slot during startup. +- With a GPU installed, the GRUB2 boot menu may take longer to load (about 15s) and be less responsive due to UEFI firmware limitations. -For `Image` version users: +For Image users: -- Due to the limitations of the current factory firmware, the `RISC-V` serial output is incomplete during device startup, and the serial output will be closed before the operating system is fully loaded. The graphics card needs to be inserted into the `PCIe` slot and connected to a monitor to observe the complete startup process. +- Serial console output stops before OS boot completes due to factory firmware limitations. Install a GPU and connect a monitor to view the full boot process. diff --git a/docs/en/docs/Installation/RISC-V-QEMU.md b/docs/en/docs/Installation/RISC-V-QEMU.md index b44198e923609dfeab014337d49719e5dcb1984e..9af79329f1fd8657effd2100e809f82df15c5f04 100644 --- a/docs/en/docs/Installation/RISC-V-QEMU.md +++ b/docs/en/docs/Installation/RISC-V-QEMU.md @@ -1,69 +1,69 @@ -# Installing on QEMU +# Installation on QEMU -## Firmware +## Required Firmware -### Standard EFI Firmware +### Standard UEFI Firmware -Download the following binaries from the download page: +Obtain the following binary files from the download page: -``` text +```text RISCV_VIRT_CODE.fd RISCV_VIRT_VARS.fd ``` -Alternatively, compile the latest EDK2 OVMF firmware locally according to the [official documentation](https://github.com/tianocore/edk2/tree/master/OvmfPkg/RiscVVirt) +For the latest firmware, build EDK2 OVMF locally using instructions in the [official repository](https://github.com/tianocore/edk2/tree/master/OvmfPkg/RiscVVirt). -### EFI Firmware with Penglai TEE Support +### Firmware with Penglai TEE -Download the following binary from the download page: +Download the following binary: -``` text +```text fw_dynamic_oe_2403_penglai.bin ``` ## QEMU Version ->To support UEFI, QEMU version 8.1 or above is required. +> QEMU 8.1 or later is essential for UEFI functionality. > ->During compilation, the libslirp dependency needs to be installed (package name varies by distribution, for openEuler it's libslirp-devel) and the --enable-slirp parameter should be added. +> When building from source, include libslirp support (install libslirp-devel on openEuler) using the `--enable-slirp` flag. -``` bash +Verify installation: + +```bash ~$ qemu-system-riscv64 --version QEMU emulator version 8.2.2 Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers ``` -## qcow2 Image +## QCOW2 Image -Obtain the ISO file (e.g., `openEuler-24.03-LTS-riscv64.qcow2.xz`) +Download the QCOW2 image file (example: **openEuler-24.03-LTS-SP1-riscv64.qcow2.xz**) -``` bash +```bash ~$ ls *.qcow2.xz -openEuler-24.03-LTS-riscv64.qcow2.xz +openEuler-24.03-LTS-SP1-riscv64.qcow2.xz ``` -## Getting the Startup Script +## Launch Scripts -Obtain the startup script from the download page: +Get these essential scripts from the download page: -- `start_vm.sh`: Default script, requires manual installation of the desktop environment. -- `start_vm_penglai.sh`: Script supporting Penglai TEE functionality. +- **start_vm.sh**: basic startup script for desktop installation +- **start_vm_penglai.sh**: enhanced script with Penglai TEE capabilities -Script Parameters: +Key configuration options: -- `ssh_port`: Local SSH forwarding port, default is 12055. -- `vcpu`: Number of threads for QEMU execution, default is 8 cores, adjustable as needed. -- `memory`: Amount of memory allocated for QEMU execution, default is 8GiB, adjustable as needed. -- `fw`: Firmware payload for booting. -- `drive`: Path to the virtual disk, adjustable as needed. -- `bios` (optional): Boot firmware, can be used to load firmware enabled with Penglai TEE. +- **ssh_port**: SSH port forwarding (default: 12055) +- **vcpu**: CPU thread count (default: 8) +- **memory**: RAM size (default: 8 GiB) +- **fw**: firmware location +- **drive**: drive image path +- **bios**: optional firmware parameter for Penglai TEE -## Creating a Virtual Hard Disk File +## Drive Setup -Create a new virtual hard disk file, in the example below, the size of the virtual hard disk is 40GiB. -> Do not use qcow2 virtual hard disk files that already contain data to avoid unexpected situations during the boot process. -> -> Ensure that there is only one qcow2 virtual hard disk file in the current directory to avoid errors in script recognition. +Generate a 40 GiB virtual drive: +> Keep only one QCOW2 file in the working directory and avoid using existing data-containing images to prevent boot problems. ``` bash ~$ qemu-img create -f qcow2 qemu.qcow2 40G diff --git a/docs/en/docs/Installation/figures/riscv-olk6.6.drawio.png b/docs/en/docs/Installation/figures/riscv-olk6.6.drawio.png new file mode 100644 index 0000000000000000000000000000000000000000..828cf9f74031ed4219f799882e64a78871380b7f Binary files /dev/null and b/docs/en/docs/Installation/figures/riscv-olk6.6.drawio.png differ diff --git a/docs/en/docs/Installation/installation-preparations.md b/docs/en/docs/Installation/installation-preparations.md index 0ad55d5d963d4f962a44cfbaf6137aa4c1f9e9ae..5e0daa40833187e4064caf9d3b370b9704ecdb15 100644 --- a/docs/en/docs/Installation/installation-preparations.md +++ b/docs/en/docs/Installation/installation-preparations.md @@ -11,16 +11,16 @@ Perform the following operations to obtain the openEuler release package: 1. Visit the [openEuler](https://www.openeuler.org/en/) website. 2. Click **Downloads**. 3. Click **Community Editions**. The version list is displayed. -4. Click **Download** on the right of **openEuler 22.03 LTS SP2**. +4. Click **Download** on the right of **openEuler 24.03 LTS SP1**. 5. Download the required openEuler release package and the corresponding verification file based on the architecture and scenario. 1. If the architecture is AArch64: 1. Click **AArch64**. - 2. If you install openEuler from a local source, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-22.03-LTS-SP2-(everything-)aarch64-dvd.iso** to the local host. - 3. If you install openEuler through the network, download the **Network Install ISO** release package **openEuler-22.03-LTS-SP2-netinst-aarch64-dvd.iso** to the local host. + 2. If you install openEuler from a local source, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-24.03-LTS-SP1-(everything-)aarch64-dvd.iso** to the local host. + 3. If you install openEuler through the network, download the **Network Install ISO** release package **openEuler-24.03-LTS-SP1-netinst-aarch64-dvd.iso** to the local host. 2. If the architecture is x86_64: 1. Click **x86_64**. - 2. If you install openEuler from a local source, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-22.03-LTS-SP2-(everything-)x86_64-dvd.iso** to the local host. - 3. If you install openEuler through the network, download the **Network Install ISO** release package **openEuler-22.03-LTS-SP2-netinst-x86_64-dvd.iso** to the local host. + 2. If you install openEuler from a local source, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-24.03-LTS-SP1-(everything-)x86_64-dvd.iso** to the local host. + 3. If you install openEuler through the network, download the **Network Install ISO** release package **openEuler-24.03-LTS-SP1-netinst-x86_64-dvd.iso** to the local host. >![](./public_sys-resources/icon-note.gif) **NOTE:** > @@ -43,9 +43,8 @@ Compare the verification value recorded in the verification file with the .iso f Before verifying the integrity of the release package, prepare the following files: -ISO file: **openEuler-22.03-LTS-SP2-aarch64-dvd.iso** - -Verification file: Copy and save the **Integrity Check** SHA256 value to a local file. +- ISO file: **openEuler-24.03-LTS-SP1-aarch64-dvd.iso** +- Verification file: Copy and save the **Integrity Check** SHA256 value to a local file. ### Procedures @@ -54,7 +53,7 @@ To verify the file integrity, perform the following operations: 1. Calculate the SHA256 verification value of the file. Run the following command: ```shell - sha256sum openEuler-22.03-LTS-SP2-aarch64-dvd.iso + sha256sum openEuler-24.03-LTS-SP1-aarch64-dvd.iso ``` After the command is run, the verification value is displayed. @@ -69,19 +68,7 @@ To install the openEuler OS on a PM, the PM must meet the following requirements ### Hardware Compatibility -You need to take hardware compatibility into account during openEuler installation. [Table 1](#table14948632047) describes the types of supported servers. - ->![](./public_sys-resources/icon-note.gif) **NOTE:** -> ->- TaiShan 200 servers are backed by Huawei Kunpeng 920 processors. ->- Currently, only Huawei TaiShan and FusionServer Pro servers are supported. More servers from other vendors will be supported in the future. - -**Table 1** Supported servers - -| Server Type | Server Name | Server Model | -| :---- | :---- | :---- | -| Rack server | TaiShan 200 | 2280 balanced model | -| Rack server | FusionServer Pro | FusionServer Pro 2288H V5
NOTE：
The server must be configured with the Avago SAS3508 RAID controller card and the LOM-X722 NIC.| +You need to take hardware compatibility into account during openEuler installation. [Compatibility List](https://www.openeuler.org/en/compatibility/) describes the types of supported servers. ### Minimum Hardware Specifications diff --git a/docs/en/docs/KubeOS/installation-and-deployment.md b/docs/en/docs/KubeOS/installation-and-deployment.md index 6788abf9d804c7aea5a53c81ddcd778d51ada3fe..01ee0432cc1843fbaf50750b28e891a680d23740 100644 --- a/docs/en/docs/KubeOS/installation-and-deployment.md +++ b/docs/en/docs/KubeOS/installation-and-deployment.md @@ -10,8 +10,14 @@ This chapter describes how to install and deploy the KubeOS tool. - [KubeOS Installation](#kubeos-installation) - [KubeOS Deployment](#kubeos-deployment) - [Building the os-operator and os-proxy Images](#building-the-os-operator-and-os-proxy-images) + - [Environment Preparation](#environment-preparation-1) + - [Procedure](#procedure) - [Creating a KubeOS VM Image](#creating-a-kubeos-vm-image) + - [Precautions](#precautions) + - [Procedure](#procedure-1) - [Deploying CRD, os-operator, and os-proxy](#deploying-crd-os-operator-and-os-proxy) + - [Precautions](#precautions-1) + - [Procedure](#procedure-2) ## Software and Hardware Requirements @@ -21,7 +27,7 @@ This chapter describes how to install and deploy the KubeOS tool. ### Software Requirements -- OS: openEuler 22.09 +- OS: openEuler 24.03 LTS SP1 ### Environment Preparation @@ -32,31 +38,22 @@ This chapter describes how to install and deploy the KubeOS tool. To install KubeOS, perform the following steps: -1. Configure the Yum sources openEuler 22.09 and openEuler 22.09:EPOL: - - ```conf - [openEuler22.09] # openEuler 22.09 official source - name=openEuler22.09 - baseurl=http://repo.openeuler.org/openEuler-22.09/everything/$basearch/ - enabled=1 - gpgcheck=1 - gpgkey=http://repo.openeuler.org/openEuler-22.09/everything/$basearch/RPM-GPG-KEY-openEuler - ``` - - ```conf - [Epol] # openEuler 22.09:EPOL official source - name=Epol - baseurl=http://repo.openeuler.org/openEuler-22.09/EPOL/main/$basearch/ - enabled=1 - gpgcheck=1 - gpgkey=http://repo.openeuler.org/openEuler-22.09/OS/$basearch/RPM-GPG-KEY-openEuler - ``` +1. Configure the openEuler 24.03-LTS-SP1 Yum repository: + + ```ini + [openEuler24.03-LTS-SP1] # openEuler 24.03-LTS-SP1 official repository + name=openEuler24.03-LTS-SP1 + baseurl=http://repo.openeuler.org/openEuler-24.03-LTS-SP1/everything/$basearch/ + enabled=1 + gpgcheck=1 + gpgkey=http://repo.openeuler.org/openEuler-24.03-LTS-SP1/everything/$basearch/RPM-GPG-KEY-openEuler + ``` 2. Install KubeOS as the **root** user. - ```shell - # yum install KubeOS KubeOS-scripts -y - ``` + ```shell + # yum install KubeOS KubeOS-scripts -y + ``` > ![](./public_sys-resources/icon-note.gif)**NOTE**: > @@ -76,110 +73,108 @@ Before using Docker to create a container image, ensure that Docker has been ins 1. Go to the working directory. - ```shell - cd /opt/kubeOS - ``` + ```shell + cd /opt/kubeOS + ``` 2. Specify the image repository, name, and version for os-proxy. - ```shell - export IMG_PROXY=your_imageRepository/os-proxy_imageName:version - ``` + ```shell + export IMG_PROXY=your_imageRepository/os-proxy_imageName:version + ``` 3. Specify the image repository, name, and version for os-operator. - ```shell - export IMG_OPERATOR=your_imageRepository/os-operator_imageName:version - ``` + ```shell + export IMG_OPERATOR=your_imageRepository/os-operator_imageName:version + ``` 4. Compile a Dockerfile to build an image. Pay attention to the following points when compiling a Dockerfile: - - The os-operator and os-proxy images must be built based on the base image. Ensure that the base image is safe. - - Copy the os-operator and os-proxy binary files to the corresponding images. - - Ensure that the owner and owner group of the os-proxy binary file in the os-proxy image are **root**, and the file permission is **500**. - - Ensure that the owner and owner group of the os-operator binary file in the os-operator image are the user who runs the os-operator process in the container, and the file permission is **500**. - - The locations of the os-operator and os-proxy binary files in the image and the commands run during container startup must correspond to the parameters specified in the YAML file used for deployment. + - The os-operator and os-proxy images must be built based on the base image. Ensure that the base image is safe. + - Copy the os-operator and os-proxy binary files to the corresponding images. + - Ensure that the owner and owner group of the os-proxy binary file in the os-proxy image are **root**, and the file permission is **500**. + - Ensure that the owner and owner group of the os-operator binary file in the os-operator image are the user who runs the os-operator process in the container, and the file permission is **500**. + - The locations of the os-operator and os-proxy binary files in the image and the commands run during container startup must correspond to the parameters specified in the YAML file used for deployment. - An example Dockerfile is as follows: + An example Dockerfile is as follows: - ```text - FROM your_baseimage - COPY ./bin/proxy /proxy - ENTRYPOINT ["/proxy"] - ``` + ```dockerfile + FROM your_baseimage + COPY ./bin/proxy /proxy + ENTRYPOINT ["/proxy"] + ``` - ```text - FROM your_baseimage - COPY --chown=6552:6552 ./bin/operator /operator - ENTRYPOINT ["/operator"] - ``` + ```dockerfile + FROM your_baseimage + COPY --chown=6552:6552 ./bin/operator /operator + ENTRYPOINT ["/operator"] + ``` - Alternatively, you can use multi-stage builds in the Dockerfile. + Alternatively, you can use multi-stage builds in the Dockerfile. 5. Build the images (the os-operator and os-proxy images) to be included in the containers OS image. - ```shell - # Specify the Dockerfile path of os-proxy. - export DOCKERFILE_PROXY=your_dockerfile_proxy - # Specify the Dockerfile path of os-operator. - export DOCKERFILE_OPERATOR=your_dockerfile_operator - # Build images. - docker build -t ${IMG_OPERATOR} -f ${DOCKERFILE_OPERATOR} . - docker build -t ${IMG_PROXY} -f ${DOCKERFILE_PROXY} . - ``` + ```shell + # Specify the Dockerfile path of os-proxy. + export DOCKERFILE_PROXY=your_dockerfile_proxy + # Specify the Dockerfile path of os-operator. + export DOCKERFILE_OPERATOR=your_dockerfile_operator + # Build images. + docker build -t ${IMG_OPERATOR} -f ${DOCKERFILE_OPERATOR} . + docker build -t ${IMG_PROXY} -f ${DOCKERFILE_PROXY} . + ``` 6. Push the images to the image repository. - ```shell - docker push ${IMG_OPERATOR} - docker push ${IMG_PROXY} - ``` + ```shell + docker push ${IMG_OPERATOR} + docker push ${IMG_PROXY} + ``` ### Creating a KubeOS VM Image #### Precautions -- The VM image is used as an example. For details about how to create a physical machine image, see **KubeOS Image Creation**. +- The VM image is used as an example. For details about how to create a physical machine image, see [KubeOS Image Creation](./kubeos-image-creation.md). - The root permission is required for creating a KubeOS image. - The RPM sources of the kbimg are the **everything** and **EPOL** repositories of openEuler of a specific version. In the Repo file provided during image creation, you are advised to configure the **everything** and **EPOL** repositories of a specific openEuler version for the Yum source. -- By default, the KubeOS VM image built using the default RPM list is stored in the same path as the kbimg tool. This partition must have at least 25 GiB free drive space. +- By default, the KubeOS VM image built using the default RPM list is stored in the **scripts-auto** directory next to the kbimg file. This partition must have at least 25 GiB free drive space. - When creating a KubeOS image, you cannot customize the file system to be mounted. #### Procedure -Use the **kbimg.sh** script to create a KubeOS VM image. For details about the commands, see **KubeOS Image Creation**. +Use `kbimg` to create a KubeOS VM image. For details about the commands, see [KubeOS Image Creation](./kubeos-image-creation.md). To create a KubeOS VM image, perform the following steps: 1. Go to the working directory. - ```shell - cd /opt/kubeOS/scripts - ``` - -2. Run `kbming.sh` to create a KubeOS image. The following is a command example: + ```shell + cd /opt/kubeOS/scripts + ``` - ```shell - bash kbimg.sh create vm-image -p xxx.repo -v v1 -b ../bin/os-agent -e '''$1$xyz$RdLyKTL32WEvK3lg8CXID0''' - ``` +2. Run `kbimg` to create a KubeOS image. The following is a command example: - In the command, **xx.repo** indicates the actual Yum source file used for creating the image. You are advised to configure both the **everything** and **EPOL** repositories as Yum sources. + ```shell + ./kbimg create -f ./kbimg.toml vm- + ``` - After the KubeOS image is created, the following files are generated in the **/opt/kubeOS/scripts** directory: + After the KubeOS image is created, the following files are generated in the **/opt/kubeOS/scripts/scripts-auto** directory: - - **system.img**: system image in raw format. The default size is 20 GB. The size of the root file system partition is less than 2,020 MiB, and the size of the Persist partition is less than 16 GiB. - - **system.qcow2**: system image in QCOW2 format. - - **update.img**: partition image of the root file system that is used for upgrade. + - **system.img**: system image in raw format. The default size is 20 GB. The size of the root file system partition is less than 2,020 MiB, and the size of the Persist partition is less than 15 GiB. + - **system.qcow2**: system image in QCOW2 format. + - **kubeos.tar**: partition image of the root file system that is used for upgrade. - The created KubeOS VM image can be used only in a VM of the x86 or AArch64 architecture. KubeOS does not support legacy boot in an x86 VM + The created KubeOS VM image can be used only in a VM of the x86 or AArch64 architecture. KubeOS does not support legacy boot in an x86 VM ### Deploying CRD, os-operator, and os-proxy #### Precautions -- The Kubernetes cluster must be deployed first. For details, see the *openEuler 22.09 Kubernetes Cluster Deployment Guide*. +- The Kubernetes cluster must be deployed first. For details, see the [_openEuler Kubernetes Cluster Deployment Guide_](../Kubernetes/Kubernetes.md). -- The OS of the worker nodes to be upgraded in the cluster must be the KubeOS built using the method described in the previous section. If it is not, use **system.qcow2** to deploy the VM again. For details about how to deploy a VM, see the *openEuler 22.09 Virtualization User Guide*. Currently, KubeOS does not support the master nodes. Use openEuler 22.09 to deploy the upgrade on the master nodes. +- The OS of the worker nodes to be upgraded in the cluster must be the KubeOS built using the method described in the previous section. If it is not, use **system.qcow2** to deploy the VM again. For details about how to deploy a VM, see the [_openEuler Virtualization User Guide_](../Virtualization/virtualization.md). Currently, KubeOS does not support the master nodes. Use openEuler to deploy the upgrade on the master nodes. - The YAML files for deploying CustomResourceDefinition (CRD), os-operator, os-proxy, and role-based access control (RBAC) of the OS need to be compiled. - The os-operator and os-proxy components are deployed in the Kubernetes cluster. os-operator must be deployed as a Deployment, and os-proxy as a DaemonSet. - Kubernetes security mechanisms, such as the RBAC, pod service account, and security policies, must be deployed. @@ -190,14 +185,14 @@ To create a KubeOS VM image, perform the following steps: 2. Deploy CRD, RBAC, os-operator, and os-proxy. Assume that the **crd.yaml**, **rbac.yaml**, and **manager.yaml** files are stored in the **config/crd**, **config/rbac**, and **config/manager** directories, respectively. Run the following commands: - ```shell - kubectl apply -f config/crd - kubectl apply -f config/rbac - kubectl apply -f config/manager - ``` + ```shell + kubectl apply -f config/crd + kubectl apply -f config/rbac + kubectl apply -f config/manager + ``` 3. After the deployment is complete, run the following command to check whether each component is started properly. If **STATUS** of all components is **Running**, the components are started properly. - ```shell - kubectl get pods -A - ``` + ```shell + kubectl get pods -A + ``` diff --git a/docs/en/docs/KubeOS/kubeos-image-creation.md b/docs/en/docs/KubeOS/kubeos-image-creation.md index cda7c1acc2e2c5d8585f11b2e6737b86eef042ba..d487afd119cc9ec4b52921770dda63ce97631bfd 100644 --- a/docs/en/docs/KubeOS/kubeos-image-creation.md +++ b/docs/en/docs/KubeOS/kubeos-image-creation.md @@ -1,169 +1,536 @@ # KubeOS Image Creation +- [KubeOS Image Creation](#kubeos-image-creation) + - [Introduction](#introduction) + - [Commands](#commands) + - [Command Format](#command-format) + - [Configuration File Description](#configuration-file-description) + - [from\_repo](#from_repo) + - [admin\_container](#admin_container) + - [pxe\_config](#pxe_config) + - [users](#users) + - [copy\_files](#copy_files) + - [grub](#grub) + - [systemd\_service](#systemd_service) + - [chroot\_script](#chroot_script) + - [disk\_partition](#disk_partition) + - [persist\_mkdir](#persist_mkdir) + - [dm\_verity](#dm_verity) + - [Usage Guide](#usage-guide) + - [Precautions](#precautions) + - [KubeOS OCI Image Creation](#kubeos-oci-image-creation) + - [Precautions](#precautions-1) + - [Usage Example](#usage-example) + - [KubeOS VM Image Creation](#kubeos-vm-image-creation) + - [Precautions](#precautions-2) + - [Usage Example](#usage-example-1) + - [Creation of Images and Files for KubeOS Physical Machine Installation](#creation-of-images-and-files-for-kubeos-physical-machine-installation) + - [Precautions](#precautions-3) + - [Usage Example](#usage-example-2) + - [Appendix](#appendix) + - [Cleanup Method for Abnormal Exits](#cleanup-method-for-abnormal-exits) + - [Detailed TOML Configuration File Example](#detailed-toml-configuration-file-example) + ## Introduction -kbimg is an image creation tool required for KubeOS deployment and upgrade. You can use kbimg to create KubeOS Docker, VM, and physical machine images. +`kbimg` is an image creation tool required for KubeOS deployment and upgrade. You can use `kbimg` to create KubeOS Docker, VM, and physical machine images. ## Commands ### Command Format -**bash kbimg.sh** \[ --help | -h \] create \[ COMMANDS \] \[ OPTIONS \] +Format of `kbimg`, a CLI tool for generating various types of image for KubeOS: -### Parameter Description +```text +Usage: kbimg [OPTIONS] -* COMMANDS +Commands: + create Create a new KubeOS image + help Print this message or the help of the given subcommand(s) - | Parameter | Description | - | ------------- | ---------------------------------------------- | - | upgrade-image | Generates a Docker image for installation and upgrade.| - | vm-image | Generates a VM image for installation and upgrade. | - | pxe-image | Generates images and files required for physical machine installation. | +Options: + -d, --debug Enable debug mode, generate the scripts without execution + -h, --help Print help + -V, --version Print version +``` -* OPTIONS +Format of `kbimg-create` for creating a KubeOS image. - | Option | Description | - | ------------ | ------------------------------------------------------------ | - | -p | Path of the repo file. The Yum source required for creating an image is configured in the repo file. | - | -v | Version of the created KubeOS image. | - | -b | Path of the os-agent binary file. | - | -e | Password of the **root** user of the KubeOS image, which is an encrypted password with a salt value. You can run the OpenSSL or KIWI command to generate the password.| - | -d | Generated or used Docker image. | - | -h --help | Help Information. | +```text +Usage: kbimg create --file -## Usage Description +Arguments: + [possible values: vm-img, pxe-img, upgrade-img, admin-container] -### Precautions +Options: + -f, --file Path to the toml configuration file + -h, --help Print help +``` -* The root permission is required for executing **kbimg.sh**. -* Currently, only the x86 and AArch64 architectures are supported. -* The RPM sources of the kbimg are the **everything** and **EPOL** repositories of openEuler of a specific version. In the Repo file provided during image creation, you are advised to configure the **everything** and **EPOL** repositories of a specific openEuler version for the Yum source. +### Configuration File Description -### Creating a KubeOS Docker Image +#### from_repo -#### Precautions +Generate upgrade container images, VM images, or PXE physical machine images from a repository. -* The created Docker image can be used only for subsequent VM or physical machine image creation or upgrade. It cannot be used to start containers. -* If the default RPM list is used to create a KubeOS image, at least 6 GB drive space is required. If the RPM list is customized, the occupied drive space may exceed 6 GB. +| Parameter | Description | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| agent_path | Path to the **os-agent** binary | +| legacy_bios | Currently supporting only `false`, indicating UEFI boot | +| repo_path | Path to the repo file, which defines the Yum repository for image creation | +| root_passwd | **root** user password matching the format in **/etc/shadow**, generated by running `openssl passwd -6 -salt $(head -c18 /dev/urandom \| openssl base64)` | +| version | KubeOS image version, written to **/etc/os-release** as the OS identifier | +| rpmlist | List of RPM packages to install in the image | +| upgrade_img | (Optional) Name of the generated upgrade container image (required for upgrade container image creation) | -#### Example +#### admin_container -* To configure the DNS, customize the `resolv.conf` file in the `scripts` directory. +Build an `admin` O&M container. -```shell - cd /opt/kubeOS/scripts - touch resolv.conf - vim resolv.conf -``` +| Parameter | Description | +| --- | --- | +| hostshell | Path to the **hostshell** binary, compiled using `make hostshell` in the project root | +| img_name | Name of the generated container image | -* Create a KubeOS image. +#### pxe_config -``` shell -cd /opt/kubeOS/scripts -bash kbimg.sh create upgrade-image -p xxx.repo -v v1 -b ../bin/os-agent -e '''$1$xyz$RdLyKTL32WEvK3lg8CXID0''' -d your_imageRepository/imageName:version -``` +Configure this section for PXE installation when creating PXE physical machine images. Required for PXE image creation. -* After the creation is complete, view the created KubeOS image. +| Parameter | Description | +| ----------- | --------------------------------------------------------------------------------------------- | +| server_ip | HTTP server address for downloading the root filesystem tar package | +| rootfs_name | Name of the filesystem tar package on the HTTP server | +| disk | Target drive for KubeOS installation | +| route_ip | Route IP address for the target machine network interface | +| dhcp | (Optional) Enables DHCP for network configuration, defaulting to `false` | +| local_ip | (Optional) IP address for the target machine network interface, required if DHCP is disabled | +| net_name | (Optional) Name for the target machine network interface, required if DHCP is disabled | +| netmask | (Optional) Subnet mask for the target machine network interface, required if DHCP is disabled | -``` shell -docker images -``` +**Note**: Parameters under `pxe_config` are not validated. Ensure their correctness. + +#### users + +(Optional) Add users. + +| Parameter | Description | +| --- | --- | +| name | Username | +| passwd | Password | +| primary_groups | (Optional) User's primary group (defaulting to the group matching the username) | +| groups | (Optional) Additional groups for the user | + +**Note**: Adding a user automatically creates a group with the same name. If specified additional groups do not exist, the operation will fail. For custom configurations, use the [chroot_script](#chroot_script) script. + +#### copy_files + +(Optional) Copy files to a specified directory in the rootfs. + +| Parameter | Description | +| --- | --- | +| dst | Destination path | +| src | Source file path | +| create_dir | (Optional) Creates the directory before copying. | + +**Note**: File permissions are not preserved during copying. For custom permissions, use the [chroot_script](#chroot_script) script. + +#### grub + +(Optional) GRUB configuration, required for dm-verity. + +| Parameter | Description | +| --- | --- | +| passwd | GRUB plaintext password | + +#### systemd_service + +(Optional) Configuration for systemd services to start on boot. + +| Parameter | Description | +| --- | --- | +| name | systemd service name | + +#### chroot_script + +(Optional) Custom chroot script. + +| Parameter | Description | +| --- | --- | +| path | Script path | +| rm | (Optional) Deletes the script after execution if set to `true` and retains if set to `false` or left empty. | + +#### disk_partition -### Creating a KubeOS VM Image +(Optional) Partition and image size customization. + +| Parameter | Description | +| --- | --- | +| root | Root partition size in MiB (default: 2560 MiB) | +| img_size | (Optional) Image size in GB (default: 20 GB) | + +#### persist_mkdir + +(Optional) Directory creation in the **persist** partition. + +| Parameter | Description | +| --- | --- | +| name | Directory name | + +#### dm_verity + +(Optional) Creation of VM or upgrade images with dm-verity enabled. + +| Parameter | Description | +| --- | --- | +| efi_key | EFI plaintext password | +| grub_key | GRUB plaintext password | +| keys_dir | (Optional) Key directory for reusing keys from previous image creation | + +## Usage Guide + +### Precautions + +- Running `kbimg` requires root privileges. +- Currently, only x86 and AArch64 architectures are supported. +- Concurrent execution is not supported. Using the `&` operator in scripts for continuous execution may lead to unexpected issues. If an unexpected power failure or interruption occurs during image creation and the environment cannot be cleaned up, refer to [Cleanup Method for Abnormal Exits](#cleanup-method-for-abnormal-exits) for cleanup before retrying. +- The RPM package source for the container OS image creation tool is the `everything` repository and `EPOL` repository of the specific openEuler version. When creating images, ensure the provided **.repo** file includes both repositories. +- dm-verity guidelines: + - dm-verity is supported on in VM environments. + - Upgrades via HTTP/HTTPS servers are not supported. Only upgrades from container image repositories are allowed. + - VMs must use virtio-type devices during startup. + - Upgrade container images with dm-verity enabled cannot upgrade systems without dm-verity, and vice versa. In clusters with mixed dm-verity configurations, ensure the correct upgrade images are deployed to the respective nodes. + - When creating upgrade container images and VM images, use the same keys (set `keys_dir` to the path of keys created during previous image builds. Matching `efi_key` or `grub_key` does not guarantee identical key files). Inconsistent keys may cause certificate verification failures during backup partition switching, preventing system boot. In such cases, re-import the backup partition certificate to resolve the issue. + +### KubeOS OCI Image Creation #### Precautions -* To use a Docker image to create a KubeOS VM image, pull the corresponding image or create a Docker image first and ensure the security of the Docker image. -* The created KubeOS VM image can be used only in a VM of the x86 or AArch64 architecture. -* Currently, KubeOS does not support legacy boot in an x86 VM. -* If the default RPM list is used to create a KubeOS image, at least 25 GB drive space is required. If the RPM list is customized, the occupied drive space may exceed 25 GB. +- The generated OCI image is intended solely for upgrading virtual or physical machine images and cannot be used to launch containers. +- Creating a container OS image with the default `rpmlist` requires at least 6 GiB of drive space. A custom `rpmlist` may increase this requirement beyond 6 GiB. + +#### Usage Example + +- Example configuration file + +```toml +[from_repo] +agent_path = "./bin/rust/release/os-agent" +legacy_bios = false +repo_path = "/etc/yum.repos.d/openEuler.repo" +root_passwd = "$1$xyz$RdLyKTL32WEvK3lg8CXID0" # default passwd: openEuler12#$ +rpmlist = [ + "NetworkManager", + "cloud-init", + "conntrack-tools", + "containerd", + "containernetworking-plugins", + "cri-tools", + "dhcp", + "ebtables", + "ethtool", + "iptables", + "kernel", + "kubernetes-kubeadm", + "kubernetes-kubelet", + "openssh-server", + "passwd", + "rsyslog", + "socat", + "tar", + "vi", +] +upgrade_img = "kubeos-upgrade:v1" +version = "v1" +``` -#### Example +- Result details + - After creation, use `docker images` to inspect the generated KubeOS container image. + - **update-boot.img**, **update-root.img**, and **update-hash.img** are generated only in dm-verity mode and can be disregarded. -* Using the Repo Source - * To configure the DNS, customize the `resolv.conf` file in the `scripts` directory. +### KubeOS VM Image Creation - ```shell - cd /opt/kubeOS/scripts - touch resolv.conf - vim resolv.conf - ``` +#### Precautions - * Create a KubeOS VM image. +- The generated container OS VM image is currently compatible only with VMs using x86 or AArch64 architectures. +- The container OS does not support legacy boot mode for x86-based VMs. +- The default root password is **openEuler12#$**. +- Creating a container OS image with the default `rpmlist` requires at least 25 GiB of drive space. A custom `rpmlist` may increase this requirement beyond 25 GiB. + +#### Usage Example + +- Example configuration file + +```toml +[from_repo] +agent_path = "./bin/rust/release/os-agent" +legacy_bios = false +repo_path = "/etc/yum.repos.d/openEuler.repo" +root_passwd = "$1$xyz$RdLyKTL32WEvK3lg8CXID0" # default passwd: openEuler12#$ +rpmlist = [ + "NetworkManager", + "cloud-init", + "conntrack-tools", + "containerd", + "containernetworking-plugins", + "cri-tools", + "dhcp", + "ebtables", + "ethtool", + "iptables", + "kernel", + "kubernetes-kubeadm", + "kubernetes-kubelet", + "openssh-server", + "passwd", + "rsyslog", + "socat", + "tar", + "vi", +] +version = "v1" +``` - ``` shell - cd /opt/kubeOS/scripts - bash kbimg.sh create vm-image -p xxx.repo -v v1 -b ../bin/os-agent -e '''$1$xyz$RdLyKTL32WEvK3lg8CXID0''' - ``` +- Result details -* Using a Docker Image +Upon completion of the container OS image creation, the following files will be generated in the **./scripts-auto** directory: - ``` shell - cd /opt/kubeOS/scripts - bash kbimg.sh create vm-image -d your_imageRepository/imageName:version - ``` +- **system.qcow2**: system image for booting virtual machines, with a default size of 20 GiB. The root file system partition must be smaller than 2560 MiB, and the persistent partition must be smaller than 15 GiB. +- **system.img**: system image for booting virtual machines, with a default size of 20 GiB. The root file system partition must be smaller than 2560 MiB, and the persistent partition must be smaller than 15 GiB. +- **kubeos.tar**: TAR archive of the root file system for system upgrades. +- **update-boot.img**, **update-root.img**, and **update-hash.img** are generated only in dm-verity mode and can be disregarded. -* Result Description - After the KubeOS image is created, the following files are generated in the **/opt/kubeOS/scripts** directory: - * **system.qcow2**: system image in QCOW2 format. The default size is 20 GiB. The size of the root file system partition is less than 2,020 MiB, and the size of the Persist partition is less than 16 GiB. - * **update.img**: partition image of the root file system used for upgrade. - -### Creating Images and Files Required for Installing KubeOS on Physical Machines +### Creation of Images and Files for KubeOS Physical Machine Installation #### Precautions -* To use a Docker image to create a KubeOS VM image, pull the corresponding image or create a Docker image first and ensure the security of the Docker image. -* The created image can only be used to install KubeOS on a physical machine of the x86 or AArch64 architecture. -* The IP address specified in the **Global.cfg** file is a temporary IP address used during installation. After the system is installed and started, configure the network by referring to **openEuler 22.09 Administrator Guide** > **Configuring the Network**. -* KubeOS cannot be installed on multiple drives at the same time. Otherwise, the startup may fail or the mounting may be disordered. -* Currently, KubeOS does not support legacy boot in an x86 physical machine. -* If the default RPM list is used to create a KubeOS image, at least 5 GB drive space is required. If the RPM list is customized, the occupied drive space may exceed 5 GB. +- The container OS image for physical machine installation is compatible only with x86 and AArch64 CPU architectures. +- The IP address specified in `pxe_config` is temporary and used solely during installation. After installation, configure the network by referring to [Network Configuration](../Administration/configuring-the-network.md). +- Installing KubeOS on multiple drives is unsupported and may lead to boot failures or mounting errors. +- Legacy boot mode is not supported for x86-based physical machines. +- Creating an image with the default `rpmlist` requires at least 5 GB of drive space. A custom `rpmlist` may exceed this requirement. +- PXE physical machine image creation does not support dm-verity. +- During PXE installation, the root partition tar package (named as configured in the TOML file) must be downloaded from the HTTP server root directory. Ensure the machine has enough memory to store the TAR package and temporary intermediate files. + +#### Usage Example -#### Example +- Update the `pxe_config` section in **kbimg.toml** to configure the necessary parameters. Refer to [parameter description](#pxe_config) for details. Only IPv4 is currently supported. Example configuration: -* Modify the `00bootup/Global.cfg` file. All parameters are mandatory. Currently, only IPv4 addresses are supported. The following is a configuration example: - - ```shell + ```toml + [pxe_config] + dhcp = false # rootfs file name - rootfs_name=kubeos.tar + rootfs_name = "kubeos.tar" # select the target disk to install kubeOS - disk=/dev/sda + disk = "/dev/vda" # pxe server ip address where stores the rootfs on the http server - server_ip=192.168.1.50 - # target machine temporary ip - local_ip=192.168.1.100 - # target machine temporary route - route_ip=192.168.1.1 - # target machine temporary netmask - netmask=255.255.255.0 + server_ip = "192.168.122.50" + # target machine ip + local_ip = "192.168.122.100" + # target machine route + route_ip = "192.168.122.1" + # target machine netmask + netmask = "255.255.255.0" # target machine netDevice name - net_name=eth0 + net_name = "eth0" ``` -* Using the Repo Source - * To configure the DNS, customize the `resolv.conf` file in the `scripts` directory. +- For DNS configuration, customize the **resolv.conf** file and use the `copy_files` field to copy it to the **/etc** directory. - ```shell - cd /opt/kubeOS/scripts - touch resolv.conf - vim resolv.conf + ```toml + [[copy_files]] + dst = "/etc" + src = "" ``` - * Create an image required for installing KubeOS on a physical machine. - - ```shell - cd /opt/kubeOS/scripts - bash kbimg.sh create pxe-image -p xxx.repo -v v1 -b ../bin/os-agent -e '''$1$xyz$RdLyKTL32WEvK3lg8CXID0''' - ``` - -* Using a Docker Image - - ``` shell - cd /opt/kubeOS/scripts - bash kbimg.sh create pxe-image -d your_imageRepository/imageName:version +- Full example of creating KubeOS physical machine installation images and configuring `pxe_config`: + + ```toml + [from_repo] + agent_path = "./bin/rust/release/os-agent" + legacy_bios = false + repo_path = "/etc/yum.repos.d/openEuler.repo" + root_passwd = "$1$xyz$RdLyKTL32WEvK3lg8CXID0" # default passwd: openEuler12#$ + rpmlist = [ + "NetworkManager", + "cloud-init", + "conntrack-tools", + "containerd", + "containernetworking-plugins", + "cri-tools", + "dhcp", + "ebtables", + "ethtool", + "iptables", + "kernel", + "kubernetes-kubeadm", + "kubernetes-kubelet", + "openssh-server", + "passwd", + "rsyslog", + "socat", + "tar", + "vi", + "coreutils", + "dosfstools", + "dracut", + "gawk", + "hwinfo", + "net-tools", + "parted", + ] + version = "v1" + + [pxe_config] + dhcp = true + rootfs_name = "kubeos.tar" + disk = "/dev/vda" + server_ip = "192.168.122.50" + route_ip = "192.168.122.1" + #local_ip = "192.168.1.100" + #netmask = "255.255.255.0" + #net_name = "eth0" ``` -* Result Description - - * **initramfs.img**: initramfs image used for boot from PXE. - * **kubeos.tar**: OS used for installation from PXE. +- Result details + - **initramfs.img**: initramfs image for PXE boot + - **kubeos.tar**: root partition file system for PXE installation + +## Appendix + +### Cleanup Method for Abnormal Exits + +1. If the `kbimg` image creation process terminates abnormally and the environment cannot be cleaned, use the following script to clean up: + + ```bash + function unmount_dir() { + local dir=$1 + if [ -L "${dir}" ] || [ -f "${dir}" ]; then + echo "${dir} is not a directory, please check it." + return 1 + fi + if [ ! -d "${dir}" ]; then + return 0 + fi + local real_dir=$(readlink -e "${dir}") + local mnts=$(awk '{print $2}' < /proc/mounts | grep "^${real_dir}" | sort -r) + for m in ${mnts}; do + echo "Unmount ${m}" + umount -f "${m}" || true + done + return 0 + } + ls -l ./scripts-auto/test.lock && rm -rf ./scripts-auto/test.lock + unmount_dir ./scripts-auto/rootfs/proc + unmount_dir ./scripts-auto/rootfs/sys + unmount_dir ./scripts-auto/rootfs/dev/pts + unmount_dir ./scripts-auto/rootfs/dev + unmount_dir ./scripts-auto/mnt/boot/grub2 + unmount_dir ./scripts-auto/mnt + rm -rf ./scripts-auto/rootfs ./scripts-auto/mnt + ``` + +2. If the above script fails to remove the directories, execute the following commands first, then retry the cleanup script from step 1. + + ```bash + fuser -kvm ./scripts-auto/rootfs + fuser -kvm ./scripts-auto/mnt + ``` + +### Detailed TOML Configuration File Example + +Adjust the following example configuration file based on your needs and the [Configuration File Description](#configuration-file-description) to create the required image. + +```toml +[from_repo] +agent_path = "./bin/rust/release/os-agent" +legacy_bios = false +repo_path = "/etc/yum.repos.d/openEuler.repo" +root_passwd = "$1$xyz$RdLyKTL32WEvK3lg8CXID0" # default passwd: openEuler12#$, use "openssl passwd -6 -salt $(head -c18 /dev/urandom | openssl base64)" to generate your passwd +rpmlist = [ + "NetworkManager", + "cloud-init", + "conntrack-tools", + "containerd", + "containernetworking-plugins", + "cri-tools", + "dhcp", + "ebtables", + "ethtool", + "iptables", + "kernel", + "kubernetes-kubeadm", + "kubernetes-kubelet", + "openssh-server", + "passwd", + "rsyslog", + "socat", + "tar", + "vi", + # Below packages are required for pxe-image. Uncomment them if you want to generate pxe-image. + # "coreutils", + # "dosfstools", + # "dracut", + # "gawk", + # "hwinfo", + # "net-tools", + # "parted", +] +upgrade_img = "kubeos-upgrade:v1" +version = "v1" + +# [admin_container] +# img_name = "kubeos-admin-container:v1" +# hostshell = "./bin/hostshell" + +# [pxe_config] +# dhcp = false +# disk = "/dev/vda" +# local_ip = "192.168.1.100" +# net_name = "eth0" +# netmask = "255.255.255.0" +# rootfs_name = "kubeos.tar" +# route_ip = "192.168.1.1" +# server_ip = "192.168.1.50" + +# [[users]] +# groups = ["admin", "wheel"] +# name = "foo" +# passwd = "foo" +# primary_group = "foo" + +# [[users]] +# groups = ["example"] +# name = "bar" +# passwd = "bar" + +# [[copy_files]] +# create_dir = "/root/test" +# dst = "/root/test/foo.txt" +# src = "/root/KubeOS/foo.txt" + +# [[copy_files]] +# dst = "/etc/bar.txt" +# src = "../bar.txt" + +# [grub] +# passwd = "foo" + +# [systemd_service] +# name = ["containerd", "kubelet"] + +# [chroot_script] +# path = "./my_chroot.sh" +# rm = true + +# [disk_partition] +# img_size = 30 # GB +# root = 3000 # MiB + +# [persist_mkdir] +# name = ["bar", "foo"] + +# [dm_verity] +# efi_key = "foo" +# grub_key = "bar" +# keys_dir = "./keys" +``` diff --git a/docs/en/docs/KubeOS/usage-instructions.md b/docs/en/docs/KubeOS/usage-instructions.md index 0f3500160214567d2b9c183e7fea6c2f380ade6a..2c5c9ad297a7d9458ff9b29de9f4bd7e1e008e8d 100644 --- a/docs/en/docs/KubeOS/usage-instructions.md +++ b/docs/en/docs/KubeOS/usage-instructions.md @@ -1,169 +1,613 @@ # Usage Instructions -- [Usage Instructions](#usage-instructions) - - [Precautions](#precautions) - - [Upgrade](#upgrade) - - [Rollback](#rollback) - - [Application Scenarios](#application-scenarios) - - [Manual Rollback](#manual-rollback) - - [KubeOS-based Rollback](#kubeos-based-rollback) + +- [Usage Instructions](#usage-instructions) + - [Precautions](#precautions) + - [OS CR Parameters](#os-cr-parameters) + - [Upgrade](#upgrade) + - [Settings Guide](#settings-guide) + - [Rollback](#rollback) + - [Use Cases](#use-cases) + - [Manual Rollback Guide](#manual-rollback-guide) + - [Tool-Based Rollback Guide](#tool-based-rollback-guide) + - [Appendix](#appendix) + - [Settings](#settings) + - [`kernel` Settings](#kernel-settings) + - [`grub` Settings](#grub-settings) + - [`kubelet` Configurations](#kubelet-configurations) + - [`containerd` Configurations](#containerd-configurations) + - [`pam.limits` Configurations](#pamlimits-configurations) + + ## Precautions -1. KubeOS upgrades the container OS in an atomic manner, where all software packages are upgraded at the same time. By default, single-package upgrade is not supported. -2. KubeOS supports container OSs with two partitions. Partitions more than two are not supported. -3. You can view the upgrade logs of a single node in the **/var/log/messages** file on the node. -4. Strictly follow the upgrade and rollback procedures described in this document. If the steps are performed in a wrong sequence, the system may fail to be upgraded or rolled back. -5. Upgrade using a Docker image and mTLS two-way authentication are supported only in openEuler 22.09 or later. -6. Cross-major version upgrade is not supported. +- General precautions + - x86 and AArch64 UEFI scenarios for both virtual and physical machines are supported. + - Avoid concurrent `kubectl apply` operations when creating or updating OS CRs using YAML. Excessive concurrent requests may overwhelm the kube-apiserver. + - Ensure custom certificates or keys for container image repositories have minimal permissions. + +- Upgrade precautions + - Upgrades are atomic for all software packages. Individual package upgrades are not supported. + - Upgrades use a dual-partition method and do not support more than two partitions. + - Cross-major-version upgrades are not supported. + - Single-node upgrade logs can be found in **/var/log/messages** on the node. + - Follow the provided upgrade and rollback procedures strictly. Incorrect sequences may prevent successful upgrades or rollbacks. + - For private image configurations with `ctr`, place the **host.toml** file in **/etc/containerd/certs.d** as per `ctr` guidelines. + - OCI image upgrades and mTLS mutual authentication are only available for openEuler 22.09 and later. + - Features like `nodeselector`, `executionmode`, `timewindow`, and `timeinterval` are supported starting from openEuler 24.09. + - KubeOS 24.03 LTS SP1 is incompatible with earlier versions. + - Downloading upgrade images from HTTP/HTTPS servers requires the corresponding version of the image creation tool. + +- Configuration precautions + - Ensure the safety and reliability of custom configurations, especially for persistent settings like `kernel.sysctl.persist`, `grub.cmdline.current`, `grub.cmdline.next`, `kubernetes.kubelet`, `container.containerd`, and `pam.limits`. KubeOS does not validate these parameters. + - If `opstype=config` and `osversion` does not match the node OS version, the configuration does not take effect. + - Supported configurations include temporary kernel parameters (`kernel.sysctl`), persistent kernel parameters (`kernel.sysctl.persist`), GRUB cmdline settings (`grub.cmdline.current`, `grub.cmdline.next`), kubelet settings (`kubernetes.kubelet`), containerd settings (`container.containerd`), and PAM limits (`pam.limits`). + - Persistent configurations are saved to the persist partition and remain after upgrades and reboots. Temporary kernel configurations are lost after reboots. + - For key-only GRUB cmdline parameters (not in `key=value` format), specify the key and leave the value empty. + - When deleting configurations (`operation=delete`), ensure the key-value pairs match the actual configurations. + - Configuration rollback is not supported. Modify the configuration version and content, then redeploy to revert changes. + - If a configuration error occurs and the node enters the `config` state, revert to the previous configuration version and redeploy to restore the node to `idle`. Note: Parameters set before the error cannot be reverted. + - To update a `key=value` parameter in GRUB cmdline to a key-only format (such as `rd.info=0` to `rd.info`), first delete the `key=value` pair, then add the key in the next configuration. Direct updates or combined update-delete actions are not supported. + +## OS CR Parameters + +Create a custom object of the OS type in the cluster and configure the relevant fields. The OS type is derived from the CRD object created in [Installation and Deployment](./installation-and-deployment.md). Below are the fields and their descriptions: + +- `imageurl` must specify the protocol and supports only `http` or `https`. When `imageurl` uses `https`, the transmission is secure. For `http` addresses, `flagSafe` must be set to `true` to confirm the address is secure before images can be downloaded. If `imageurl` is an `http` address and `flagSafe` is not set to `true`, the address is considered insecure, images will not be downloaded, and the upgrade node logs will notify the user of the insecure address. +- You are advised to use `https` for `imageurl`. Using `https` requires the upgrading machine to have the appropriate certificates installed. If you maintain the image server, sign the certificates and ensure the upgrading nodes have them installed. Place the certificates in the **/etc/KubeOS/certs** directory of the container OS. The administrator provides the address and must ensure its security, preferably using internal network addresses. +- The provider of the container OS image service must verify the legitimacy of the container OS image to ensure the downloaded image is from a trusted source. +- In clusters with multiple OS versions (multiple OS instances), the `nodeselector` field of each OS must differ. This ensures that a group of nodes identified by labels corresponds to only one OS instance: + - If an OS has `nodeselector` set to `all-label`, the cluster can only have one valid instance of this OS (a valid instance has corresponding nodes). + - Only one OS without a configured `nodeselector` is allowed, as this implies operations on nodes without labels. +- `timeinterval` parameter details: + - If not set, it defaults to 15 seconds. + - If set to 0, due to the Kubernetes `controller-runtime` rate limit, the interval between operator task dispatches will gradually increase up to 1,000 seconds. + - In parallel mode, this is the interval between operator dispatches for upgrades/configurations per batch. + - In serial mode, this is the interval between the completion of upgrades/configurations for one batch of nodes and the start of the next batch. The internal interval within a batch is 15 seconds. + - Updating the instance fields of the OS will immediately trigger the operator. + +| Parameter | Type | Description | Usage Notes | Required | +| -------------- | -------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | +| imagetype | String | Type of the upgrade image | The value can be `docker`, `containerd`, or `disk`. This parameter is valid only in upgrade scenarios. **Note**: If `containerd` is used, the agent prioritizes the `crictl` tool to pull the image. If `crictl` is unavailable, it uses the `ctr` command. When `ctr` is used, if the image is in a private repository, configure the repository host information in **/etc/containerd/certs.d** as per the [official documentation](https://github.com/containerd/containerd/blob/main/docs/hosts.md) to successfully pull the image. | Yes | +| opstype | String | Operation type (upgrade, rollback, or configuration) | The value can be `upgrade`, `config`, or `rollback`. | Yes | +| osversion | String | Target version for upgrade/rollback | `osversion` must match the target OS version of the node (such as `KubeOS 1.0.0`). This corresponds to the `PRETTY_NAME` field in **/etc/os-release** or the OS version detected by Kubernetes. | Yes | +| maxunavailable | Integer | Number of nodes to upgrade/rollback/configure simultaneously per batch | If `maxunavailable` exceeds the actual number of nodes, the actual number of nodes is used. | Yes | +| containerimage | String | Container image used for upgrade | This parameter is valid only when `imagetype` is a container type. The following formats are supported: *repository/name*, *repository/name@sha256:xxxx*, or *repository/name:tag*. | Yes | +| imageurl | String | URL of the drive image used for upgrade | The URL must include the protocol (`http` or `https`), for example, `https://192.168.122.15/update.img`. This parameter is valid only in drive image upgrade scenarios. | Yes | +| checksum | String | SHA-256 checksum for drive image verification or digests value for container image verification | This parameter is valid only in upgrade scenarios. | Yes | +| flagSafe | Boolean | Whether `imageurl` using `http` is secure | The value must be `true` or `false`. This parameter is valid only when `imageurl` uses `http`. | Yes | +| mtls | Boolean | Whether to use HTTPS mutual authentication for `imageurl` | The value must be `true` or `false`. This parameter is valid only when `imageurl` uses `https`. | Yes | +| cacert | String | Root certificate file used for HTTPS or HTTPS mutual authentication | This parameter is valid only when `imageurl` uses `https`. | Required if `imageurl` uses `https` protocol | +| clientcert | String | Client certificate file used for HTTPS mutual authentication | This parameter is valid only when HTTPS mutual authentication is used. | Required if `mtls` is `true` | +| clientkey | String | Client public key used for HTTPS mutual authentication | This parameter is valid only when HTTPS mutual authentication is used. | Required if `mtls` is `true` | +| evictpodforce | Boolean | Whether to forcefully evict pods during upgrade/rollback | The value must be `true` or `false`. This parameter is valid only during upgrade or rollback. | Yes | +| sysconfigs | / | Configuration settings | 1. Only configuration is performed when `opstype=config`.
2. When `opstype=upgrade/rollback`, it represents post-upgrade/rollback configuration, that is, configuration after reboot. For detailed field descriptions, refer to [Settings Guide](#settings-guide). | Required if `opstype=config` | +| upgradeconfigs | / | Pre-upgrade configuration settings | This parameter is valid during upgrade or rollback and takes effect before the upgrade or rollback operation. For detailed field descriptions, refer to [Settings Guide](#settings-guide). | Optional | +| nodeselector | String | Node label for upgrade/configuration/rollback operations | This parameter is used for scenarios where only nodes with specific labels (not all worker nodes in the cluster) are targeted for operations. Nodes must have a label with the `upgrade.openeuler.org/node-selector` key, and `nodeselector` is the value of this label.
Notes:
1. If this parameter is not configured or set to `no-label`, operations target nodes without the `upgrade.openeuler.org/node-selector` label.
2. If this parameter is `""`, operations target nodes with `upgrade.openeuler.org/node-selector=""`.
3. To ignore labels and target all nodes, set this parameter to `all-label`. | Optional | +| timewindow | / | Time window for upgrade/configuration/rollback operations | 1. When specifying a time window, specify both `starttime` and `endtime` (that is, both must be empty or non-empty).
2. `starttime` and `endtime` are of the string type and must be in `YYYY-MM-DD HH:MM:SS` or `HH:MM:SS` format, with consistent formats.
3. For `HH:MM:SS` format, if `starttime` is later than `endtime`, `starttime` is considered to be the next day.
4. If `timewindow` is not configured, there is no time window restriction by default. | Optional | +| timeinterval | Integer | Time interval (in seconds) between task dispatches for upgrade/configuration/rollback operations | The interval is the time between operator task dispatches. If the Kubernetes cluster is busy and cannot immediately respond to operator requests, the actual interval may exceed the specified time. | Optional | +| executionmode | String | Execution mode for upgrade/configuration/rollback operations | The value can be `serial` and `parallel`. If not set, the default is `parallel`. | Optional | ## Upgrade -Create a custom object of the OS type in the cluster and set the corresponding fields. The OS type comes from the CRD object created in the installation and deployment sections. The following table describes the fields. - -| Parameter |Type | Description | How to Use| Mandatory (Yes/No) | -| -------------- | ------ | ------------------------------------------------------------ | ----- | ---------------- | -| imagetype | string | Type of the upgrade image | The value must be `docker` or `disk`. Other values are invalid. This parameter is valid only in upgrade scenarios.|Yes | -| opstype | string | Operation, that is, upgrade or rollback| The value must be `upgrade` or `rollback`. Other values are invalid.|Yes | -| osversion | string | OS version of the image used for upgrade or rollback | The value must be a KubeOS version, for example, `KubeOS 1.0.0`.|Yes | -| maxunavailable | int | Number of nodes to be upgraded or rolled back at the same time| If the value of `maxunavailable` is greater than the actual number of nodes in the cluster, the deployment can be performed. The upgrade or rollback is performed based on the actual number of nodes in the cluster.|Yes | -| dockerimage | string | Docker image used for upgrade | The value must be in the *repository/name:tag* format. This parameter is valid only when the Docker image is used for upgrade.|Yes | -| imageurl | string | Address of the disk image used for the upgrade| `imageurl` contains the protocol and only HTTP or HTTPS is supported. For example, `https://192.168.122.15/update.img` is valid only when a disk image is used for upgrade.|Yes | -| checksum | string | Checksum (SHA-256) value for disk image verification during the upgrade. | This parameter is valid only when a disk image is used for upgrade.|Yes | -| flagSafe | bool | Whether `imageurl` specifies a secure HTTP address | The value must be `true` or `false`. This parameter is valid only when `imageurl` specifies an HTTP address.|Yes | -| mtls | bool | Whether HTTPS two-way authentication is used for the connection to the `imageurl` address. | The value must be `true` or `false`. This parameter is valid only when `imageurl` specifies an HTTPS address.|Yes | -| cacert | string | Root certificate file used for HTTPS or HTTPS two-way authentication | This parameter is valid only when `imageurl` specifies an HTTPS address.| This parameter is mandatory when `imageurl` specifies an HTTPS address.| -| clientcert | string | Client certificate file used for HTTPS two-way authentication | This parameter is valid only when HTTPS two-way authentication is used.|This parameter is mandatory when `mtls` is set to `true`.| -| clientkey | string | Client public key used for HTTPS two-way authentication | This parameter is valid only when HTTPS two-way authentication is used.|This parameter is mandatory when `mtls` is set to `true`.| - -The address specified by `imageurl` contains the protocol. Only the HTTP or HTTPS protocol is supported. If `imageurl` is set to an HTTPS address, secure transmission is used. If `imageurl` is set to an HTTP address, set `flagSafe` to `true`, because the image can be downloaded only when the address is secure. If `imageurl` is set to an HTTP address but `flagSafe` is not set to `true`, the address is insecure by default. The image will not be downloaded, and a message is written to the log of the node to be upgraded indicating that the address is insecure. - -You are advised to set `imageurl` to an HTTPS address. In this case, ensure that the required certificate has been installed on the node to be upgraded. If the image server is maintained by yourself, you need to sign the certificate and ensure that the certificate has been installed on the node to be upgraded. Place the certificate in the **/etc/KubeOS/certs** directory of KubeOS. The administrator specifies the address and must ensure the security of the address. An intranet address is recommended. - -The container OS image provider must check the validity of the image to ensure that the downloaded container OS image is from a reliable source. - -Compile the YAML file for deploying the OS as a custom resource (CR) instance in the cluster. The following is an example YAML file for deploying the CR instance: - -- Upgrade using a disk image - - ``` - apiVersion: upgrade.openeuler.org/v1alpha1 - kind: OS - metadata: - name: os-sample - spec: - imagetype: disk - opstype: upgrade - osversion: edit.os.version - maxunavailable: edit.node.upgrade.number - dockerimage: "" - imageurl: edit.image.url - checksum: image.checksum - flagSafe: imageurl.safety - mtls: imageurl use mtls or not - cacert: ca certificate - clientcert: client certificate - clientkey: client certificate key - ``` - -- Upgrade using a Docker image - - ``` shell - apiVersion: upgrade.openeuler.org/v1alpha1 - kind: OS - metadata: - name: os-sample - spec: - imagetype: docker - opstype: upgrade - osversion: edit.os.version - maxunavailable: edit.node.upgrade.number - dockerimage: dockerimage like repository/name:tag - imageurl: "" - checksum: "" - flagSafe: false - mtls: true - ``` - - Before using a Docker image to perform the upgrade, create the image first. For details about how to create a Docker image, see **KubeOS Image Creation**. - -Assume that the YAML file is **upgrade_v1alpha1_os.yaml**. - -Check the OS version of the node that is not upgraded. - -``` -kubectl get nodes -o custom-columns='NAME:.metadata.name,OS:.status.nodeInfo.osImage' -``` - -Run the following command to deploy the CR instance in the cluster. The node is upgraded based on the configured parameters. - -``` -kubectl apply -f upgrade_v1alpha1_os.yaml -``` - -Check the node OS version again to determine whether the node upgrade is complete. - -``` -kubectl get nodes -o custom-columns='NAME:.metadata.name,OS:.status.nodeInfo.osImage' -``` - -> ![](./public_sys-resources/icon-note.gif)**NOTE**: +1. Compile the YAML file for deploying the OS as a custom resource (CR) instance in the cluster. The following is an example YAML file (**upgrade_v1alpha1_os.yaml**) for deploying the CR instance: + - Upgrade using a drive image + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: disk + opstype: upgrade + osversion: edit.os.version + maxunavailable: edit.node.upgrade.number + containerimage: "" + evictpodforce: true/false + imageurl: edit.image.url + checksum: image.checksum + flagSafe: imageurl.safety + mtls: imageurl use mtls or not + cacert: ca certificate + clientcert: client certificate + clientkey: client certificate key + ``` + + - Upgrade using a Docker image + - Before using a container image for upgrades, create the required container image. For detailed instructions, refer to the [KubeOS OCI Image Creation](./kubeos-image-creation.md#kubeos-oci-image-creation) section in [KubeOS Image Creation](./kubeos-image-creation.md). + - For Docker containers + + ``` yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: docker + opstype: upgrade + osversion: edit.os.version + maxunavailable: edit.node.upgrade.number + containerimage: container image like repository/name:tag + evictpodforce: true/false + imageurl: "" + checksum: container image digests + flagSafe: false + mtls: true + ``` + + - For containerd containers + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: containerd + opstype: upgrade + osversion: edit.os.version + maxunavailable: edit.node.upgrade.number + containerimage: container image like repository/name:tag + evictpodforce: true/false + imageurl: "" + checksum: container image digests + flagSafe: false + mtls: true + ``` + + - Example of upgrade and configuration: + - For nodes using the containerd container engine, the upgrade method does not affect the configuration. `upgradeconfigs` take effect before the upgrade, while `sysconfigs` take effect after the upgrade. For details on configuration parameters, refer to the [Settings Guide](#settings-guide). + - For upgrade and configuration, the `opstype` field must be set to `upgrade`. + - `upgradeconfigs` are configurations executed before the upgrade, and `sysconfigs` are configurations executed after the reboot. Configure them as required. + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: "" + opstype: upgrade + osversion: edit.os.version + maxunavailable: edit.node.upgrade.number + containerimage: "" + evictpodforce: true/false + imageurl: "" + checksum: container image digests + flagSafe: false + mtls: false + sysconfigs: + version: edit.os.version + configs: + - model: kernel.sysctl + contents: + - key: kernel param key1 + value: kernel param value1 + - key: kernel param key2 + value: kernel param value2 + - model: kernel.sysctl.persist + configpath: persist file path + contents: + - key: kernel param key3 + value: kernel param value3 + - key: "" + value: "" + upgradeconfigs: + version: 1.0.0 + configs: + - model: kernel.sysctl + contents: + - key: kernel param key4 + value: kernel param value4 + ``` + + - Example of upgrading specific nodes by setting `nodeselector`, `timewindow`, `timeinterval`, and `executionmode`: + - For nodes using the containerd container engine, the upgrade method does not affect node selection. + - Nodes to be upgraded must include a label with the `upgrade.openeuler.org/node-selector` key. The value of `nodeselector` should match the value of this label. For example, if `nodeselector` is set to `kubeos`, only worker nodes with the `upgrade.openeuler.org/node-selector=kubeos` label will be upgraded. + - `nodeselector`, `timewindow`, `timeinterval`, and `executionmode` also apply to configuration and rollback. + - Example commands for adding, modifying, deleting, and viewing node labels: + + ```shell + # Add a label to node kubeos-node1. + kubectl label nodes kubeos-node1 upgrade.openeuler.org/node-selector=kubeos-v1 + # Modify the label of node kubeos-node1. + kubectl label --overwrite nodes kubeos-node1 upgrade.openeuler.org/node-selector=kubeos-v2 + # Delete the label of node kubeos-node1. + kubectl label nodes kubeos-node1 upgrade.openeuler.org/node-selector- + # View node labels. + kubectl get nodes --show-labels + ``` + + - Example YAML: + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: containerd + opstype: upgrade + osversion: edit.os.version + maxunavailable: edit.node.upgrade.number + containerimage: container image like repository/name:tag + evictpodforce: true/false + imageurl: "" + checksum: container image digests + flagSafe: false + mtls: true + nodeselector: edit.node.label.key + timewindow: + starttime: "HH::MM::SS/YYYY-MM-DD HH::MM::SS" + endtime: "HH::MM::SS/YYYY-MM-DD HH::MM::SS" + timeinterval: time intervel like 30 + executionmode: serial/parallel + ``` + +2. Check the OS version of nodes that have not been upgraded. + + ```shell + kubectl get nodes -o custom-columns='NAME:.metadata.name,OS:.status.nodeInfo.osImage' + ``` + +3. Execute the command to deploy the CR instance in the cluster. Nodes will upgrade based on the configured parameters. + + ```shell + kubectl apply -f upgrade_v1alpha1_os.yaml + ``` + +4. Check the OS version of the nodes again to confirm whether the upgrade is complete. + + ```shell + kubectl get nodes -o custom-columns='NAME:.metadata.name,OS:.status.nodeInfo.osImage' + ``` + +> ![](./public_sys-resources/icon-note.gif)**Note**: > -> If you need to perform the upgrade again, modify the `imageurl`, `osversion`, `checksum`, `maxunavailable`, `flagSafe`, or `dockerimage` parameters in **upgrade_v1alpha1_os.yaml**. +> If further upgrades are needed, modify the corresponding fields in **upgrade_v1alpha1_os.yaml** as described above. + +## Settings Guide + +- Settings parameter description: + + The following example YAML explains the configuration parameters. Ensure the format (indentation) matches the example: + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: "" + opstype: config + osversion: edit.os.version + maxunavailable: edit.node.config.number + containerimage: "" + evictpodforce: false + checksum: "" + sysconfigs: + version: edit.sysconfigs.version + configs: + - model: kernel.sysctl + contents: + - key: kernel param key1 + value: kernel param value1 + - key: kernel param key2 + value: kernel param value2 + operation: delete + - model: kernel.sysctl.persist + configpath: persist file path + contents: + - key: kernel param key3 + value: kernel param value3 + - model: grub.cmdline.current + contents: + - key: boot param key1 + - key: boot param key2 + value: boot param value2 + - key: boot param key3 + value: boot param value3 + operation: delete + - model: grub.cmdline.next + contents: + - key: boot param key4 + - key: boot param key5 + value: boot param value5 + - key: boot param key6 + value: boot param value6 + operation: delete + ``` + + The configuration parameters are as follows. + + | Parameter | Type | Description | Usage Notes | Required | + | ---------- | ------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | + | version | String | Configuration version | This parameter determines whether the configuration is triggered by comparing `version`. If `version` is empty (`""` or no value), the check still occurs. If `sysconfigs` or `upgradeconfigs` is not configured, the existing `version` value will be cleared, triggering the configuration. | Yes | + | configs | / | Specific configuration content | This parameter is a list of specific configuration items. | Yes | + | model | String | Configuration type | Supported configuration types are listed in [Settings](#settings) in the appendix. | Yes | + | configpath | String | Configuration file path | This parameter is effective only for the `kernel.sysctl.persist` configuration type. Refer to the [Settings](#settings) in the appendix for details on the file path. | No | + | contents | / | Specific key/value pairs and operation types | This parameter is a list of specific configuration parameters. | Yes | + | key | String | Parameter name | The key cannot be empty or contain `=`. Avoid configuring strings with spaces or tabs. Refer to the [Settings](#settings) in the appendix for details on key requirements for each configuration type. | Yes | + | value | String | Parameter value | For `key=value` parameters, the value cannot be empty. Avoid configuring strings with spaces or tabs. Refer to the [Settings](#settings) in the appendix for details on value requirements for each configuration type. | Required for key=value parameters | + | operation | String | Operation on the parameter | This parameter is effective only for `kernel.sysctl.persist`, `grub.cmdline.current`, and `grub.cmdline.next` parameter types, defaulting to `add` or `update`. Currently, only `delete` is supported, which removes an existing parameter (`key=value` must match exactly for deletion). | No | + + - The `upgradeconfigs` parameters are the same as those of `sysconfigs`. `upgradeconfigs` applies to configurations before upgrade/rollback and is only effective in upgrade/rollback scenarios. `sysconfigs` supports standalone configurations as well as configurations after upgrade/rollback reboots. + +- Usage instructions + + - Compile the **upgrade_v1alpha1_os.yaml** file (see the example above) for deploying the OS as a CR instance in the cluster. + + - Check the configuration version and node status before performing the configuration (ensure `NODESTATUS` is `idle`). + + ```shell + kubectl get osinstances -o custom-columns='NAME:.metadata.name,NODESTATUS:.spec.nodestatus,SYSCONFIG:status.sysconfigs.version,UPGRADECONFIG:status.upgradeconfigs.version' + ``` + + - Deploy the CR instance in the cluster. The nodes will be configured based on the specified parameters. Check the node status again (`NODESTATUS` should change to `config`). + + ```shell + kubectl apply -f upgrade_v1alpha1_os.yaml + kubectl get osinstances -o custom-columns='NAME:.metadata.name,NODESTATUS:.spec.nodestatus,SYSCONFIG:status.sysconfigs.version,UPGRADECONFIG:status.upgradeconfigs.version' + ``` + + - Verify the configuration version and node status again to confirm whether the configuration is complete (`NODESTATUS` should return to `idle`). + + ```shell + kubectl get osinstances -o custom-columns='NAME:.metadata.name,NODESTATUS:.spec.nodestatus,SYSCONFIG:status.sysconfigs.version,UPGRADECONFIG:status.upgradeconfigs.version' + ``` + +- If further upgrades are needed, modify the corresponding fields in **upgrade_v1alpha1_os.yaml** as described above. ## Rollback -### Application Scenarios +### Use Cases + +- When a VM fails to boot normally, you can manually switch the boot option on the GRUB boot menu to roll back the system to the previous version (manual rollback). +- When the VM can boot normally and enter the system, both tool-based rollback and manual rollback are supported. Tool-based rollback is recommended. +- Tool-based rollback has two methods: + 1. Rollback mode: Directly roll back to the previous version. + 2. Upgrade mode: Re-upgrade to the previous version. + +### Manual Rollback Guide + +- Manually restart the VM. After entering the boot menu, select the second boot option to roll back. Manual rollback only supports rolling back to the previous version. + +### Tool-Based Rollback Guide + +- Rollback to any version + 1. Modify the YAML configuration file of the OS CR instance (e.g., **upgrade_v1alpha1_os.yaml**) and set the relevant fields to the desired older version image information. The OS kind is derived from the CRD object created during installation and deployment. For field descriptions and examples, refer to the previous section on upgrade guidance. + + 2. After modifying the YAML, execute the update command. Once the custom object is updated in the cluster, the nodes will roll back based on the configured fields. + + ```shell + kubectl apply -f upgrade_v1alpha1_os.yaml + ``` + +- Rollback to the previous version + - OS rollback to the previous version: Modify **upgrade_v1alpha1_os.yaml**, set `osversion` to the previous version, and set `opstype` to `rollback` to roll back to the previous version (that is, switch to the previous partition). Example YAML: + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: "" + opstype: rollback + osversion: KubeOS previous version + maxunavailable: 2 + containerimage: "" + evictpodforce: true/false + imageurl: "" + checksum: "" + flagSafe: false + mtls: true + ``` + + - Configuration rollback to the previous version: Modify **upgrade_v1alpha1_os.yaml**, set the `version` of `sysconfigs/upgradeconfigs` to the previous version to roll back to the previous version (already configured parameters cannot be rolled back). Example YAML: + + ```yaml + apiVersion: upgrade.openeuler.org/v1alpha1 + kind: OS + metadata: + name: os-sample + spec: + imagetype: "" + opstype: config + osversion: edit.os.version + maxunavailable: edit.node.config.number + containerimage: "" + evictpodforce: true/false + imageurl: "" + checksum: "" + flagSafe: false + mtls: false + sysconfigs: + version: previous config version + configs: + - model: kernel.sysctl + contents: + - key: kernel param key1 + value: kernel param value1 + - key: kernel param key2 + value: kernel param value2 + - model: kernel.sysctl.persist + configpath: persist file path + contents: + - key: kernel param key3 + value: kernel param value3 + ``` + +- After modifying the YAML, execute the update command. Once the custom object is updated in the cluster, the nodes will roll back based on the configured fields. + + ```shell + kubectl apply -f upgrade_v1alpha1_os.yaml + ``` -- If a node cannot be started, you can only manually roll back the container OS to the previous version that can be properly started. -- If a node can be started and run the system, you can manually or use KubeOS (similar to the upgrade) to roll back the container OS. You are advised to use KubeOS. + After the update, the nodes will roll back the container OS based on the configuration. +- Check the container OS version (for OS rollback) or the node config version and node status as `idle` (for config rollback) to confirm whether the rollback was successful. -### Manual Rollback + ```shell + kubectl get osinstances -o custom-columns='NAME:.metadata.name,NODESTATUS:.spec.nodestatus,SYSCONFIG:status.sysconfigs.version,UPGRADECONFIG:status.upgradeconfigs.version' + ``` -Manually restart the node and select the second boot option to roll back the container OS. Manual rollback can only roll back the container OS to the version before the upgrade. +## Appendix -### KubeOS-based Rollback +### Settings -- Roll back to any version. - - Modify the YAML configuration file (for example, **upgrade_v1alpha1_os.yaml**) of the CR instance of the OS and set the corresponding fields to the image information of the target source version. The OS type comes from the CRD object created in the installation and deployment sections. For details about the fields and examples, see the previous section. +#### `kernel` Settings - - After the YAML is modified, run the update command. After the custom object is updated in the cluster, the node performs rollback based on the configured field information. +- `kernel.sysctl`: temporarily sets kernel parameters, which become invalid after a reboot. `key` and `value` are kernel parameter keys and values. Both `key` and `value` cannot be empty, and `key` cannot contain `=`. This parameter does not support the delete operation (`operation=delete`). Example: + ```yaml + configs: + - model: kernel.sysctl + contents: + - key: user.max_user_namespaces + value: 16384 + - key: net.ipv4.tcp_tw_recycle + value: 0 + operation: delete ``` - kubectl apply -f upgrade_v1alpha1_os.yaml + +- `kernel.sysctl.persist`: set persistent kernel parameters. `key` and `value` are kernel parameter keys and values. Both `key` and `value` cannot be empty, and `key` cannot contain `=`. `configpath` specifies the configuration file path, supporting the creation of new files (ensure the parent directory exists). If `configpath` is not specified, it defaults to modifying **/etc/sysctl.conf**. Example: + + ```yaml + configs: + - model: kernel.sysctl.persist + configpath : /etc/persist.conf + contents: + - key: user.max_user_namespaces + value: 16384 + - key: net.ipv4.tcp_tw_recycle + value: 0 + operation: delete ``` -- Roll back to the previous version. +#### `grub` Settings - - Modify the **upgrade_v1alpha1_os.yaml** file. Set **osversion** to the previous version and **opstype** to **rollback** to roll back to the previous version (that is, switch to the previous partition). Example YAML: +- `grub.cmdline.current`/`grub.cmdline.next`: configure the kernel boot parameters in the **grub.cfg** file. These parameters appear in the **grub.cfg** file as shown in the following example: + ```shell + linux /boot/vmlinuz root=/dev/sda2 ro rootfstype=ext4 nomodeset quiet oops=panic softlockup_panic=1 nmi_watchdog=1 rd.shell=0 selinux=0 crashkernel=256M panic=3 ``` - apiVersion: upgrade.openeuler.org/v1alpha1 - kind: OS - metadata: - name: os-sample - spec: - imagetype: "" - opstype: rollback - osversion: KubeOS pervious version - maxunavailable: 2 - dockerimage: "" - imageurl: "" - checksum: "" - flagSafe: false - mtls:true + + - In dm-verity mode, `grub.cmdline` configurations are invalid. + + - KubeOS uses dual partitions, and `grub.cmdline.current`/`grub.cmdline.next` supports configuring the current or next partition: + - `grub.cmdline.current`: configures the boot parameters for the current partition. + - `grub.cmdline.next`: configures the boot parameters for the next partition. + + - Note: The configurations before and after an upgrade/rollback are always based on the partition location at the time the upgrade/rollback operation is initiated. For example, if the current partition is partition A, and an upgrade operation is initiated with `grub.cmdline.current` configured in `sysconfigs` (post-upgrade restart configuration), `grub.cmdline` of partition A will still be modified after the reboot. + + - `grub.cmdline.current`/`grub.cmdline.next` supports both `key=value` (where `value` cannot be empty) and key-only parameters. If `value` contains `=` (such as `root=UUID=some-uuid`), all characters before the first `=` are considered `key`, and all characters after the first `=` are considered `value`. Example configuration: + + ```yaml + configs: + - model: grub.cmdline.current + contents: + - key: selinux + value: "0" + - key: root + value: UUID=e4f1b0a0-590e-4c5f-9d8a-3a2c7b8e2d94 + - key: panic + value: "3" + operation: delete + - key: crash_kexec_post_notifiers + - model: grub.cmdline.next + contents: + - key: selinux + value: "0" + - key: root + value: UUID=e4f1b0a0-590e-4c5f-9d8a-3a2c7b8e2d94 + - key: panic + value: "3" + operation: delete + - key: crash_kexec_post_notifiers + ``` + +#### `kubelet` Configurations + +- `kubernetes.kubelet`: configures parameters in the kubelet configuration file on the node. The parameter descriptions and constraints are as follows: + - Only parameters in `KubeletConfiguration` are supported. + - The kubelet configuration file must be in YAML format. + - If `configpath` is not specified, the default configuration file path is **/var/lib/kubelet/config.yaml**. Note that the configuration file path must match the path specified by the `--config` parameter when kubelet starts for the configuration to take effect. Ensure the configuration file path is valid. + - The `value` parameter type in kubelet configuration supports empty/`null`, `int`, `float`, `string`, `boolean`, and arrays. For arrays, duplicate elements are allowed, and updating an array appends to the existing array. To modify elements in an array, delete the array first and then add a new array. + - For nested configurations, use `.` to connect nested keys. For example, to modify the `cacheAuthorizedTTL` parameter to `1s` in the following YAML example: + + ```yaml + authorization: + mode: Webhook + webhook: + cacheAuthorizedTTL: 0s ``` - - After the YAML is modified, run the update command. After the custom object is updated in the cluster, the node performs rollback based on the configured field information. + The parameter configuration example is as follows: + ```yaml + configs: + - model: kubernetes.kubelet + configpath: /etc/test.yaml + contents: + - key: authorization.webhook.cacheAuthorizedTTL + value: 1s ``` - kubectl apply -f upgrade_v1alpha1_os.yaml + + - When a configuration in `kubernetes.kubelet` is deleted, the `value` is not compared with the value in the configuration file. + +#### `containerd` Configurations + +- `container.containerd`: configures parameters in the containerd configuration file on the node. The parameter descriptions and constraints are as follows: + - The containerd configuration file must be in TOML format. `key` is `table_header.key_name` of the parameter in TOML. For example, to modify the `no_shim` parameter to `true` in the following TOML example: + + ```toml + [plugins."io.containerd.runtime.v1.linux"] + no_shim=false + runtime="runc" + runtime_root=" ``` - After the update is complete, the node rolls back the container OS based on the configuration information. + The parameter configuration example is as follows: + + ```yaml + configs: + - model: container.containerd + configpath: /etc/test.toml + contents: + - key: plugins."io.containerd.runtime.v1.linux".no_shim + value: true + ``` -- Check the OS version of the container on the node to determine whether the rollback is successful. + - TOML uses `.` to separate keys, and os-agent follows the same convention. If a key contains `.`, it must be enclosed in `""`. For example, `"io.containerd.runtime.v1.linux"` is a key-only parameter in the above example. + - If `configpath` is not specified, the default configuration file path is **/etc/containerd/config.toml**. Ensure the configuration file path is valid. + - The `key` and `value` in `container.containerd` configuration cannot be empty. The `value` parameter type supports `int`, `float`, `string`, `boolean`, and arrays. For arrays, duplicate elements are allowed, and updating an array appends to the existing array. To modify elements in an array, delete the array first and then add a new array. + - When a configuration in `container.containerd` is deleted, the `value` is not compared with the value in the configuration file. + +#### `pam.limits` Configurations + +- `pam.limits`: configures the **/etc/security/limits.conf** file on the node. + - The `key` is the domain value, and the `value` must follow the format `type.item.value` (each line in the **limits.conf** file must follow the format: ` `). For example: + + ```yaml + configs: + - model: pam.limits + contents: + - key: ftp + value: soft.core.0 + ``` + + - During updates, if you do not need to update `type`, `item`, or `value`, you can use `_` to skip updating that parameter. However, the `value` must still be a three-part string separated by dots. For example: + + ```yaml + configs: + - model: pam.limits + contents: + - key: ftp + value: hard._.1 + ``` - ``` - kubectl get nodes -o custom-columns='NAME:.metadata.name,OS:.status.nodeInfo.osImage' - ``` + - When a configuration is added in `pam.limits`, the `value` must not contain `_`. + - When a configuration is deleted in `pam.limits`, the `value` is validated. If the `value` does not match the value in the configuration file, the deletion will fail. + - `key` and `value` in `pam.limits` configurations cannot be empty. diff --git a/docs/en/docs/Kubernetes/figures/advertiseAddress.png b/docs/en/docs/Kubernetes/figures/advertiseAddress.png new file mode 100644 index 0000000000000000000000000000000000000000..b36e5c4664f2d2e5faaa23128fd4711c11e30179 Binary files /dev/null and b/docs/en/docs/Kubernetes/figures/advertiseAddress.png differ diff --git a/docs/en/docs/Kubernetes/figures/flannelConfig.png b/docs/en/docs/Kubernetes/figures/flannelConfig.png new file mode 100644 index 0000000000000000000000000000000000000000..dc9e7c665edd02fad16d3e6f4970e3125efcbef8 Binary files /dev/null and b/docs/en/docs/Kubernetes/figures/flannelConfig.png differ diff --git a/docs/en/docs/Kubernetes/figures/name.png b/docs/en/docs/Kubernetes/figures/name.png new file mode 100644 index 0000000000000000000000000000000000000000..dd6ddfdc3476780e8c896bfd5095025507f62fa8 Binary files /dev/null and b/docs/en/docs/Kubernetes/figures/name.png differ diff --git a/docs/en/docs/Kubernetes/figures/podSubnet.png b/docs/en/docs/Kubernetes/figures/podSubnet.png new file mode 100644 index 0000000000000000000000000000000000000000..b368f77dd7dfd7722dcf7751b3e37ec28755e42d Binary files /dev/null and b/docs/en/docs/Kubernetes/figures/podSubnet.png differ diff --git a/docs/en/docs/Kubernetes/kubernetes-cluster-deployment-guide-based-on-containerd.md b/docs/en/docs/Kubernetes/kubernetes-cluster-deployment-guide-based-on-containerd.md new file mode 100644 index 0000000000000000000000000000000000000000..b4a4f521007057ed4c562a4e9c2582642f4ffb71 --- /dev/null +++ b/docs/en/docs/Kubernetes/kubernetes-cluster-deployment-guide-based-on-containerd.md @@ -0,0 +1,308 @@ +# Kubernetes Cluster Deployment Guide Based on containerd + +Starting from version 1.21, Kubernetes no longer supports the Kubernetes+Docker setup for cluster deployment. This guide demonstrates how to quickly set up a Kubernetes cluster using containerd as the container runtime. For custom cluster configurations, consult the [official documentation](https://kubernetes.io/docs/home/). + +## Software Package Installation + +### 1. Installing Required Packages + +```sh +yum install -y containerd +yum install -y kubernetes* +yum install -y cri-tools +``` + +> ![](./public_sys-resources/icon-note.gif)**Note** +> +> - If Docker is already installed on the system, uninstall it before installing containerd to prevent conflicts. + +The required containerd version is 1.6.22-15 or higher. If the installed version is not supported, upgrade to version 1.6.22-15 using the following commands, or perform a manual upgrade. + +```sh +wget --no-check-certificate https://repo.openeuler.org/openEuler-24.03-LTS/update/x86_64/Packages/containerd-1.6.22-15.oe2403.x86_64.rpm +rpm -Uvh containerd-1.6.22-15.oe2403.x86_64.rpm +``` + +The package versions downloaded via `yum` in this guide are: + +```text +1. containerd + - Architecture: x86_64 + - Version: 1.6.22-15 +2. kubernetes - client/help/kubeadm/kubelet/master/node + - Architecture: x86_64 + - Version: 1.29.1-4 +3. cri-tools + - Architecture: X86_64 + - Version: 1.29.0-3 +``` + +### 2. Downloading CNI Components + +```sh +mkdir -p /opt/cni/bin +cd /opt/cni/bin +wget --no-check-certificate https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-amd64-v1.5.1.tgz +tar -xzvf ./cni-plugins-linux-amd64-v1.5.1.tgz -C . +``` + +> ![](./public_sys-resources/icon-note.gif)**Note** +> +> - The provided download link is for the AMD64 architecture. Choose the appropriate version based on your system architecture. Other versions are available in the [GitHub repository](https://github.com/containernetworking/plugins/releases/). + +### 3. Downloading CNI Plugin (Flannel) + +```sh +wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml --no-check-certificate +``` + +## Environment Configuration + +This section configures the OS environment required for Kubernetes. + +### 1. Setting the Host Name + +```sh +hostnamectl set-hostname nodeName +``` + +### 2. Configuring the Firewall + +**Method 1:** + +Configure firewall rules to open ports for etcd and the API Server, ensuring proper communication between the control plane and worker nodes. + +Open ports for etcd: + +```sh +firewall-cmd --zone=public --add-port=2379/tcp --permanent +firewall-cmd --zone=public --add-port=2380/tcp --permanent +``` + +Open ports for the API Server: + +```sh +firewall-cmd --zone=public --add-port=6443/tcp --permanent +``` + +Apply the firewall rules: + +```sh +firewall-cmd --reload +``` + +> ![](./public_sys-resources/icon-note.gif)**Note** +> +> - Firewall configuration may prevent certain container images from functioning properly. To ensure smooth operation, open the necessary ports based on the images being used. + +**Method 2:** + +Disable the firewall using the following commands: + +```sh +systemctl stop firewalld +systemctl disable firewalld +``` + +### 3. Disabling SELinux + +SELinux security policies may block certain operations within containers, such as writing to specific directories, accessing network resources, or executing privileged operations. This can cause critical services like CoreDNS to fail, resulting in `CrashLoopBackOff` or `Error` states. Disable SELinux using the following commands: + +```sh +setenforce 0 +sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config +``` + +### 4. Disabling Swap + +The Kubernetes scheduler allocates pods to nodes based on available memory and CPU resources. If swap is enabled on a node, the actual physical memory and logically available memory may not align, which can affect the scheduler decisions, leading to node overloading or incorrect scheduling. Therefore, disable swap: + +```sh +swapoff -a +sed -ri 's/.*swap.*/#&/' /etc/fstab +``` + +### 5. Configuring the Network + +Enable IPv6 and IPv4 traffic filtering on bridged networks using iptables, and enable IP forwarding to ensure inter-pod communication across nodes: + +```sh +$ cat > /etc/sysctl.d/k8s.conf << EOF +net.bridge.bridge-nf-call-ip6tables = 1 +net.bridge.bridge-nf-call-iptables = 1 +net.ipv4.ip_forward = 1 +vm.swappiness=0 +EOF +$ modprobe br_netfilter +$ sysctl -p /etc/sysctl.d/k8s.conf +``` + +## Configuring containerd + +This section configures containerd, including setting the pause image, cgroup driver, disabling certificate verification for the `registry.k8s.io` image repository, and configuring a proxy. + +First, generate the default configuration file for containerd and output it to the file specified by `containerd_conf`: + +```sh +containerd_conf="/etc/containerd/config.toml" +mkdir -p /etc/containerd +containerd config default > "${containerd_conf}" +``` + +Configure the pause image: + +```sh +pause_img=$(kubeadm config images list | grep pause | tail -1) +sed -i "/sandbox_image/s#\".*\"#\"${pause_img}\"#" "${containerd_conf}" +``` + +Set the cgroup driver to systemd: + +```sh +sed -i "/SystemdCgroup/s/=.*/= true/" "${containerd_conf}" +``` + +Disable certificate verification for the `registry.k8s.io` image repository: + +```sh +sed -i '/plugins."io.containerd.grpc.v1.cri".registry.configs/a\[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.k8s.io".tls]\n insecure_skip_verify = true' /etc/containerd/config.toml +``` + +Configure the proxy (replace "***" in `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` with your proxy information): + +```sh +$ server_path="/etc/systemd/system/containerd.service.d" +$ mkdir -p "${server_path}" +$ cat > "${server_path}"/http-proxy.conf << EOF +[Service] +Environment="HTTP_PROXY=***" +Environment="HTTPS_PROXY=***" +Environment="NO_PROXY=***" +EOF +``` + +Restart containerd to apply the configurations: + +```sh +systemctl daemon-reload +systemctl restart containerd +``` + +## Configuring crictl to Use containerd as the Container Runtime + +```sh +crictl config runtime-endpoint unix:///run/containerd/containerd.sock +crictl config image-endpoint unix:///run/containerd/containerd.sock +``` + +## Configuring kubelet to Use systemd as the Cgroup Driver + +```sh +systemctl enable kubelet.service +echo 'KUBELET_EXTRA_ARGS="--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice"' >> /etc/sysconfig/kubelet +systemctl restart kubelet +``` + +## Creating a Cluster Using Kubeadm (Control Plane Only) + +### 1. Configuring Cluster Information + +```sh +kubeadm config print init-defaults --component-configs KubeletConfiguration >> kubeletConfig.yaml +vim kubeletConfig.yaml +``` + +In the **kubeletConfig.yaml** file, configure the node name, advertise address (`advertiseAddress`), and the CIDR for the Pod network. + +**Modify `name` to match the hostname, consistent with the first step in the environment configuration:** + +![](./figures/name.png) + +**Change `advertiseAddress` to the IP address of the control plane:** + +![](./figures/advertiseAddress.png) + +**Add `podSubnet` under `Networking` to specify the CIDR range:** + +![](./figures/podSubnet.png) + +### 2. Deploying the Cluster + +Use `kubeadm` to deploy the cluster. Many configurations are generated by default (such as authentication certificates). Refer to the [official documentation](https://kubernetes.io/docs/home/) for modifications. + +**Disable the proxy (if applicable):** + +```sh +unset http_proxy https_proxy +``` + +Deploy the cluster using `kubeadm init`: + +```sh +kubeadm init --config kubeletConfig.yaml +``` + +Specify the configuration file for `kubectl`: + +```sh +mkdir -p "$HOME"/.kube +cp -i /etc/kubernetes/admin.conf "$HOME"/.kube/config +chown "$(id -u)":"$(id -g)" "$HOME"/.kube/config +export KUBECONFIG=/etc/kubernetes/admin.conf +``` + +### 3. Deploying the CNI Plugin (Flannel) + +This tutorial uses Flannel as the CNI plugin. Below are the steps to download and deploy Flannel. + +The Flannel used here is downloaded from the `registry-1.docker.io` image repository. To avoid certificate verification issues, configure the image repository to skip certificate verification in the containerd configuration file (**/etc/containerd/config.toml**). + +![](./figures/flannelConfig.png) + +Use `kubectl apply` to deploy the **kube-flannel.yml** file downloaded during the software package installation. + +```sh +kubectl apply -f kube-flannel.yml +``` + +> ![](./public_sys-resources/icon-note.gif)**Note** +> +> The control plane may have taint issues, causing the node status in `kubectl get nodes` to remain "not ready." Refer to the [official documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to remove taints. + +## Joining the Cluster (Worker Nodes Only) + +**Disable the proxy (if applicable):** + +```sh +unset http_proxy https_proxy +``` + +After installing and configuring the environment on worker nodes, join the cluster using the following command: + +```sh +kubeadm join : --token --discovery-token-ca-cert-hash sha256: +``` + +This command is generated after `kubeadm init` completes on the control plane. Alternatively, you can generate it on the control plane using the following commands: + +```sh +$ kubeadm token create # Generate token. +$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ + openssl dgst -sha256 -hex | sed 's/^.* //' # Get hash. +``` + +After joining, check the status of worker nodes on the control plane using: + +```sh +kubectl get nodes +``` + +If the node status shows "not ready," it may be due to unsuccessful Flannel plugin deployment. In this case, run the locally generated Flannel executable to complete the deployment. + +**Running kubectl Commands on Worker Nodes (Optional):** + +To run `kubectl` commands on worker nodes, copy the control plane configuration file **/etc/kubernetes/admin.conf** to the same directory, then configure it using: + +```sh +export KUBECONFIG=/etc/kubernetes/admin.conf +``` diff --git a/docs/en/docs/NestOS/NestOS-user-guide.md b/docs/en/docs/NestOS/NestOS-user-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..aa8cfe0d005e938ab5175e0708cd69f9b8d7fca9 --- /dev/null +++ b/docs/en/docs/NestOS/NestOS-user-guide.md @@ -0,0 +1,985 @@ +# NestOS User Guide + +## 1. Introduction to NestOS + +### 1.1 Overview + +NestOS, developed by KylinSoft and incubated in the openEuler community, is a cloud-native OS designed for modern infrastructure. It incorporates advanced technologies like rpm-ostree support and Ignition configuration, featuring a dual-root file system with mutual backup and atomic update capabilities. The system also includes the nestos-assembler tool for streamlined integration and building. Optimized for Kubernetes and OpenStack platforms, NestOS minimizes container runtime overhead, enabling efficient cluster formation and secure operation of large-scale containerized workloads. + +This guide provides a comprehensive walkthrough of NestOS, covering its building, installation, deployment, and usage. It aims to help users maximize the system benefits for rapid and efficient configuration and deployment. + +### 1.2 Application Scenarios and Advantages + +NestOS serves as an ideal foundation for cloud environments centered around containerized applications. It resolves challenges such as fragmented operation and maintenance (O&M) practices and redundant platform development, which arise from the decoupling of container and orchestration technologies from the underlying infrastructure. By ensuring alignment between application services and the base OS, NestOS delivers consistent and streamlined O&M. + +![figure1](./figures/figure1.png) + +## 2. Environment Preparation + +### 2.1 Build Environment Requirements + +#### 2.1.1 Requirements for Building the nestos-assembler Tool + +- Use openEuler for optimal results. +- Ensure at least 5 GB of available drive space. + +#### 2.1.2 Requirements for Building NestOS + +| Category | Requirements | +| :----------: | :---------------------: | +| CPU | 4 vCPUs | +| Memory | 4 GB | +| Drive | Available space > 10 GB | +| Architecture | x86_64 or AArch64 | +| Others | Support for KVM | + +### 2.2 Deployment Configuration Requirements + +| Category | Recommended Configuration | Minimum Configuration | +| :----------: | :-----------------------: | :-------------------: | +| CPU | > 4 vCPU | 1 vCPU | +| Memory | > 4GB | 512 MB | +| Drive | > 20GB | 10 GB | +| Architecture | x86_64, aarch64 | / | + +## 3. Quick Start + +### 3.1 Quick Build + +(1) Obtain the nestos-assembler container image. + +You are advised the openEuler-based base image. For additional details, see [Section 6.1](#61-nestos-assembler-container-image-creation). + +```shell +docker pull hub.oepkgs.net/nestos/nestos-assembler:24.03-LTS.20240903.0-aarch64 +``` + +(2) Create a script named `nosa` and save it to `/usr/local/bin`, then make it executable. + +```shell +#!/bin/bash + +sudo docker run --rm -it --security-opt label=disable --privileged --user=root \ + -v ${PWD}:/srv/ --device /dev/kvm --device /dev/fuse --network=host \ + --tmpfs /tmp -v /var/tmp:/var/tmp -v /root/.ssh/:/root/.ssh/ -v /etc/pki/ca-trust/:/etc/pki/ca-trust/ \ + ${COREOS_ASSEMBLER_CONFIG_GIT:+-v $COREOS_ASSEMBLER_CONFIG_GIT:/srv/src/config/:ro} \ + ${COREOS_ASSEMBLER_GIT:+-v $COREOS_ASSEMBLER_GIT/src/:/usr/lib/coreos-assembler/:ro} \ + ${COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS} \ + ${COREOS_ASSEMBLER_CONTAINER:-nestos-assembler:your_tag} "$@" +``` + +Note: Replace the value of `COREOS_ASSEMBLER_CONTAINER` with the actual nestos-assembler container image in your environment. + +(3) Obtain nestos-config. + +Use `nosa init` to initialize the build workspace, pull the build configuration, and create the `nestos-build` directory. Run the following command in this directory: + +```shell +nosa init https://gitee.com/openeuler/nestos-config +``` + +(4) Adjust build configurations. + +nestos-config provides default build configurations, so no additional steps are required. For customization, refer to [Section 5](#5-build-configuration-nestos-config). + +(5) Build NestOS images. + +```shell +# Pull build configurations and update cache. +nosa fetch +# Generate root file system, qcow2, and OCI images. +nosa build +# Generate live ISO and PXE images. +nosa buildextend-metal +nosa buildextend-metal4k +nosa buildextend-live +``` + +For detailed build and deployment steps, refer to [Section 6](#6-build-process). + +### 3.2 Quick Deployment + +Using the NestOS ISO image as an example, boot into the live environment and execute the following command to complete the installation by following the wizard: + +```shell +sudo installnestos +``` + +For alternative deployment methods, see [Section 8](#8-deployment-process). + +## 4. Default Configuration + +| Item | Default Configuration | +| :-------------------------: | :----------------------------------------------: | +| Docker service | Disabled by default, requires manual activation. | +| SSH service security policy | Supports only key-based login by default. | + +## 5. Build Configuration: nestos-config + +### 5.1 Obtaining Configuration + +The repository for nestos-config is located at + +### 5.2 Directory Structure Explanation + +| Directory/File | Description | +| :---------------: | :------------------------------------: | +| live/* | Boot configuration for live ISO builds | +| overlay.d/* | Custom file configurations | +| tests/* | User-defined test case configurations | +| *.repo | Repository configurations | +| .yaml, manifests/ | Main build configurations | + +### 5.3 Key Files + +#### 5.3.1 .repo Files + +.repo files in the directory are used to configure software repositories for building NestOS. + +#### 5.3.2 YAML Configuration Files + +YAML files in the directory provide various configurations for NestOS builds. For details, refer to [Section 5.4](#54-key-fields). + +### 5.4 Key Fields + +| Field | Purpose | +| :------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | +| packages-aarch64, packages-x86_64, packages | Scope of software package integration | +| exclude-packages | Blocklist for software package integration | +| remove-from-packages | Files/folders to remove from specified packages | +| remove-files | Files/folders to remove | +| extra-kargs | Additional kernel boot parameters | +| initramfs-args | Initramfs build parameters | +| postprocess | Post-build scripts for the file system | +| default-target | Default target, such as **multi-user.target** | +| rojig.name, releasever | Image-related information (name and version) | +| lockfile-repos | List of repository names available for builds, which must match the repository names in the repo files described in [Section 5.3.1](#531-repo-files) | + +### 5.5 Configurable Items + +#### 5.5.1 Repository Configuration + +(1) Edit the .repo file in the configuration directory and modify its content to the desired software repositories. + +```shell +$ vim nestos-pool.repo +[repo_name_1] +Name=xxx +baseurl = https://ip.address/1 +enabled = 1 + +[repo_name_2] +Name=xxx +baseurl = https://ip.address/2 +enabled = 1 +``` + +(2) Modify the `lockfile-repos` field in the YAML configuration file to include the corresponding repository names. + +Note: The repository name is the content inside `[]` in the repo file, not the `name` field. + +```shell +$ vim manifests/rpmlist.yaml +Modify the `lockfile-repos` field as follows: +lockfile-repos: +- repo_name_1 +- repo_name_2 +``` + +#### 5.5.2 Software Package Customization + +Modify the `packages`, `packages-aarch64`, and `packages-x86_64` fields to add or remove software packages. + +For example, adding `nano` to the `packages` field ensures that the system includes `nano` after installation. + +```shell +$ vim manifests/rpmlist.yaml +packages: +- bootupd +... +- authselect +- nano +... +packages-aarch64: +- grub2-efi-aa64 +packages-x86_64: +- microcode_ctl +- grub2-efi-x64 +``` + +#### 5.5.3 Image Name and Version Customization + +Modify the `releasever` and `rojig.name` fields in the YAML file to control the image version and name. + +```shell +$ vim manifest.yaml + +releasever: "1.0" +rojig: + license: MIT + name: nestos + summary: NestOS stable +``` + +With the above configuration, the built image format will be **nestos-1.0.$(date "+%Y%m%d").$build_num.$type**, where **build_num** is the build count and **type** is the type suffix. + +#### 5.5.4 Image Release Information Customization + +Normally, release information is provided by the integrated release package (e.g., `openeuler-release`). However, you can rewrite the **/etc/os-release** file by adding a **postprocess** script. + +```shell +$ vim manifests/system-configuration.yaml +# Add the following content to postprocess. If the content already exists, simply modify the corresponding release information. +postprocess: + - | + #!/usr/bin/env bash + set -xeuo pipefail + export OSTREE_VERSION="$(tail -1 /etc/os-release)" + date_now=$(date "+%Y%m%d") + echo -e 'NAME="openEuler NestOS"\nVERSION="24.03-LTS"\nID="openeuler"\nVERSION_ID="24.03-LTS"\nPRETTY_NAME="NestOS"\nANSI_COLOR="0;31"\nBUILDID="'${date_now}'"\nVARIANT="NestOS"\nVARIANT_ID="nestos"\n' > /usr/lib/os-release + echo -e $OSTREE_VERSION >> /usr/lib/os-release + cp -f /usr/lib/os-release /etc/os-release +``` + +#### 5.5.5 Custom File Creation + +Add or modify custom files in the **overlay.d** directory. This allows for customization of the image content. + +```shell +mkdir -p overlay.d/15nestos/etc/test/test.txt +echo "This is a test message !" > overlay.d/15nestos/etc/test/test.txt +``` + +Using the above configuration to build the image. After image boot, the content of the corresponding file in the system will match the custom content added above. + +```shell +[root@nosa-devsh ~]# cat /etc/test/test.txt +This is a test message ! +``` + +## 6. Build Process + +NestOS employs a containerized method to bundle the build toolchain into a comprehensive container image called nestos-assembler. + +NestOS enables users to create the nestos-assembler container image, simplifying the process of building various NestOS image formats in any Linux distribution environment, such as within existing CI/CD pipelines. Additionally, users can manage, debug, and automate testing of build artifacts using this image. + +### 6.1 nestos-assembler Container Image Creation + +#### 6.1.1 Prerequisites + +1. Prepare the base container image. + + The nestos-assembler container image must be based on a base image that supports the Yum or DNF package manager. Although it can be created from any distribution base image, using an openEuler base image is recommended to reduce software compatibility issues. + +2. Install required software packages. + + Install Docker, the essential dependency: + + ```shell + dnf install -y docker + ``` + +3. Clone the nestos-assembler source code repository. + +```shell +git clone --depth=1 --single-branch https://gitee.com/openeuler/nestos-assembler.git +``` + +#### 6.1.2 Building the nestos-assembler Container Image + +Using the openEuler container image as the base, build the image with the following command: + +```shell +cd nestos-assembler/ +docker build -f Dockerfile . -t nestos-assembler:your_tag +``` + +### 6.2 nestos-assembler Container Image Usage + +#### 6.2.1 Prerequisites + +1. Prepare the nestos-assembler container image. + + Once the nestos-assembler container image is built following [Section 6.1](#61-nestos-assembler-container-image-creation), it can be managed and distributed via a privately hosted container image registry. Ensure the correct version of the nestos-assembler container image is pulled before initiating the NestOS build. + +2. Create the nosa script. + + To streamline user operations, you can write a `nosa` command script. This is particularly useful as the NestOS build process involves multiple calls to the nestos-assembler container image for executing various commands and configuring numerous parameters. For quick build details, see [Section 3.1](#31-quick-build). + +#### 6.2.2 Usage Instructions + +nestos-assembler commands + +| Command | Description | +| :-------------------: | :-------------------------------------------------------------------------------------: | +| init | Initialize the build environment and configuration. See [Section 6.3](#63-build-environment-preparation) for details. | +| fetch | Fetch the latest software packages to the local cache based on the build configuration. | +| build | Build the ostree commit, which is the core command for building NestOS. | +| run | Directly start a QEMU instance, using the latest build version by default. | +| prune | Clean up historical build versions, retaining the latest three versions by default. | +| clean | Delete all build artifacts. Use the `--all` parameter to also clean the local cache. | +| list | List the versions and artifacts present in the current build environment. | +| build-fast | Quickly build a new version based on the previous build record. | +| push-container | Push the container image artifact to the container image registry. | +| buildextend-live | Build ISO artifacts and PXE images that support the live environment. | +| buildextend-metal | Build raw artifacts for bare metal. | +| buildextend-metal4k | Build raw artifacts for bare metal in native 4K mode. | +| buildextend-openstack | Build QCOW2 artifacts for the OpenStack platform. | +| buildextend-qemu | Build QCOW2 artifacts for QEMU. | +| basearch | Retrieve the current architecture information. | +| compress | Compress artifacts. | +| kola | Automated testing framework | +| kola-run | A wrapper for automated testing that outputs summarized results | +| runc | Mount the current build root file system in a container. | +| tag | Manage build project tags. | +| virt-install | Create an instance for the specified build version. | +| meta | Manage build project metadata. | +| shell | Enter the nestos-assembler container image. | + +### 6.3 Build Environment Preparation + +The NestOS build environment requires a dedicated empty folder as the working directory, supporting multiple builds while preserving and managing historical versions. Before setting up the build environment, ensure the build configuration is prepared (see [Section 5](#5-build-configuration-nestos-config)). + +You are advised to maintain a separate build configuration for each independent build environment. If you plan to build NestOS for various purposes, maintain multiple build configurations and their corresponding directories. This approach allows independent evolution of configurations and clearer version management. + +#### 6.3.1 Initializing the Build Environment + +Navigate to the target working directory and run the following command to initialize the build environment: + +```shell +nosa init https://gitee.com/openeuler/nestos-config +``` + +Initialization is only required for the first build. Subsequent builds can reuse the same environment unless significant changes are made to the build configuration. + +#### 6.3.2 Build Environment Structure + +After initialization, the following folders are created in the working directory: + +**builds**: stores build artifacts and metadata. The **latest** subdirectory is a symbolic link to the most recent build version. + +**cache**: contains cached data pulled from software sources and package lists specified in the build configuration. Historical NestOS ostree repositories are also stored here. + +**overrides**: used to place files or RPM packages that should be added to the rootfs of the final artifact during the build process. + +**src**: holds the build configuration, including nestos-config-related content. + +**tmp**: used during builds and automated testing. In case of errors, you can inspect VM CLI outputs, journal logs, and other debugging information here. + +### 6.4 Build Steps + +The primary steps and reference commands for building NestOS are outlined below. + +![figure2](./figures/figure2.png) + +#### 6.4.1 Initial Build + +For the initial build, the build environment must be initialized. Refer to [Section 6.3](#63-build-environment-preparation) for detailed instructions. + +For subsequent builds, the existing build environment can be reused. Use `nosa list` to check the current versions and corresponding artifacts in the build environment. + +#### 6.4.2 Updating Build Configuration and Cache + +After initializing the build environment, run the following command to update the build configuration and cache: + +```shell +nosa fetch +``` + +This step validates the build configuration and pulls software packages from the configured sources to the local cache. When the build configuration changes or you want to update to the latest software versions, repeat this step. Otherwise, the build may fail or produce unexpected results. + +If significant changes are made to the build configuration and you want to clear the local cache and re-fetch, use: + +```shell +nosa clean --all +``` + +#### 6.4.3 Building the Immutable Root File system + +The core of NestOS, an immutable OS, is its immutable root file system based on ostree technology. Run the following command to build the ostree file system: + +```shell +nosa build +``` + +By default, the `build` command generates the ostree file system and an OCI archive. You can also include `qemu`, `metal`, or `metal4k` to simultaneously build the corresponding artifacts, equivalent to running `buildextend-qemu`, `buildextend-metal`, and `buildextend-metal4k` afterward. + +```shell +nosa build qemu metal metal4k +``` + +To add custom files or RPM packages during the NestOS build, place them in the **rootfs/** or **rpm/** folders under the **overrides** directory before running the `build` command. + +#### 6.4.4 Building Various Artifacts + +After running the `build` command, you can use `buildextend` commands to build different types of artifacts. Details are as follows. + +- Building QCOW2 images: + +```shell +nosa buildextend-qemu +``` + +- Building ISO images with a live environment or PXE boot components: + +```shell +nosa buildextend-metal +nosa buildextend-metal4k +nosa buildextend-live +``` + +- Building QCOW2 images for the OpenStack environment: + +```shell +nosa buildextend-openstack +``` + +- Building container images for container-based updates: + +When the `nosa build` command is executed, an OCI archive format image is also generated. This image can be pushed to a local or remote image registry directly. + +```shell +nosa push-container [container-image-name] +``` + +The remote image registry address must be appended to the container image name, and no `:` should appear except in the tag. If no `:` is detected, the command generates a tag in the format `{latest_build}-{arch}`. Example: + +```shell +nosa push-container registry.example.com/nestos:1.0.20240903.0-x86_64 +``` + +This command supports the following options: + +`--authfile`: specifies the authentication file for logging into the remote image registry. + +`--insecure`: bypasses SSL/TLS verification for self-signed certificates. + +`--transport`: specifies the target image push protocol. The default is `docker`. Supported options: + +- `containers-storage`: pushes to the local storage directory of container engines like Podman and CRIO. +- `dir`: pushes to a specified local directory. +- `docker`: pushes to a private or remote container image registry using the Docker API. +- `docker-archive`: exports an archive file for use with `docker load`. +- `docker-daemon`: pushes to the local storage directory of the Docker container engine. + +### 6.5 Artifacts Acquisition + +Once the build process is complete, the artifacts are stored in the following directory within the build environment: + +```text +builds/{version}/{arch}/ +``` + +For convenience, if you are only interested in the latest build version or are using CI/CD, a **latest** directory symbol link points to the most recent version directory: + +```text +builds/latest/{arch}/ +``` + +To reduce the size of the artifacts for easier transfer, you can compress them using the following command: + +```shell +nosa compress +``` + +Note that compression removes the original files, which may disable some debugging commands. To restore the original files, use the decompression command: + +```shell +nosa uncompress +``` + +### 6.6 Build Environment Maintenance + +Before or after setting up the NestOS environment, you may need to address specific requirements. The following commands are recommended for resolving these issues. + +#### 6.6.1 Cleaning Up Historical or Invalid Build Versions to Free Drive Space + +To clean up historical build versions, run: + +```shell +nosa prune +``` + +To delete all artifacts in the current build environment, run: + +```shell +nosa clean +``` + +If the build configuration has changed software repositories or historical caches are no longer needed, you can completely clear the current build environment cache: + +```shell +nosa clean --all +``` + +#### 6.6.2 Temporarily Running a Build Version Instance for Debugging or Verification + +```shell +nosa run +``` + +Use `--qemu-image` or `--qemu-iso` to specify the boot image address. For additional parameters, refer to `nosa run --help`. + +Once the instance starts, the build environment directory is mounted to **/var/mnt/workdir**, allowing access to the build environment. + +#### 6.6.3 Running Automated Tests + +```shell +nosa kola run +``` + +This command runs predefined test cases. You can also append a specific test case name to execute it individually. + +```shell +nosa kola testiso +``` + +This command performs installation and deployment tests for ISO or PXE live environments, acting as a smoke test for the build process. + +#### 6.6.4 Debugging and Verifying netsos-assembler + +```shell +nosa shell +``` + +This command launches a shell environment within the build toolchain container, enabling you to verify the functionality of the build toolchain environment. + +## 7. Deployment Configuration + +### 7.1 Introduction + +Before you deploy NestOS, it is essential to understand and prepare the necessary configurations. NestOS offers flexible configuration options through Ignition files, which can be managed using Butane. This simplifies automated deployment and environment setup for users. + +This section provides a detailed overview of Butane functionality and usage, along with configuration examples for various scenarios. These configurations will help you quickly set up and run NestOS, ensuring system security and reliability while meeting application needs. Additionally, we will explore how to customize images by pre-integrating Ignition files, enabling efficient configuration and deployment for specific use cases. + +### 7.2 Introduction to Butane + +Butane is a tool that converts human-readable YAML configuration files into NestOS Ignition files. It simplifies the process of writing complex configurations by allowing users to create configuration files in a more readable format, which are then converted into JSON format suitable for NestOS. + +NestOS has adapted Butane by adding support for the `nestos` variant and configuration specification version `v1.0.0`, corresponding to the Ignition configuration specification `v3.3.0`. This ensures configuration stability and compatibility. + +### 7.3 Butane Usage + +To install the Butane package, use the following command: + +```shell +dnf install butane +``` + +Edit **example.yaml** and execute the following command to convert it into an Ignition file **example.ign**. The process of writing YAML files will be explained in detail later: + +```shell +butane example.yaml -o example.ign -p +``` + +### 7.4 Supported Functional Scenarios + +The following configuration examples (**example.yaml**) briefly describe the main functional scenarios and advanced usage methods supported by NestOS. + +#### 7.4.1 Configuring Users, Groups, Passwords, and SSH Keys + +```YAML +variant: nestos +version: 1.0.0 +passwd: + users: + - name: nest + ssh_authorized_keys: + - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHn2eh... + - name: jlebon + groups: + - wheel + ssh_authorized_keys: + - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC5QFS... + - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIveEaMRW... + - name: miabbott + groups: + - docker + - wheel + password_hash: $y$j9T$aUmgEDoFIDPhGxEe2FUjc/$C5A... + ssh_authorized_keys: + - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDTey7R... +``` + +#### 7.4.2 File Operations: Configuring Network Interfaces + +```YAML +variant: nestos +version: 1.0.0 +storage: + files: + - path: /etc/NetworkManager/system-connections/ens2.nmconnection + mode: 0600 + contents: + inline: | + [connection] + id=ens2 + type=ethernet + interface-name=ens2 + [ipv4] + address1=10.10.10.10/24,10.10.10.1 + dns=8.8.8.8; + dns-search= + may-fail=false + method=manual +``` + +#### 7.4.3 Creating Directories, Files, and Symbolic Links with Permissions + +```YAML +variant: nestos +version: 1.0.0 +storage: + directories: + - path: /opt/tools + overwrite: true + files: + - path: /var/helloworld + overwrite: true + contents: + inline: Hello, world! + mode: 0644 + user: + name: dnsmasq + group: + name: dnsmasq + - path: /opt/tools/transmogrifier + overwrite: true + contents: + source: https://mytools.example.com/path/to/archive.gz + compression: gzip + verification: + hash: sha512-00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + mode: 0555 + links: + - path: /usr/local/bin/transmogrifier + overwrite: true + target: /opt/tools/transmogrifier + hard: false +``` + +#### 7.4.4 Writing systemd Services: Starting and Stopping Containers + +```YAML +variant: nestos +version: 1.0.0 +systemd: + units: + - name: hello.service + enabled: true + contents: | + [Unit] + Description=MyApp + After=network-online.target + Wants=network-online.target + + [Service] + TimeoutStartSec=0 + ExecStartPre=-/bin/podman kill busybox1 + ExecStartPre=-/bin/podman rm busybox1 + ExecStartPre=/bin/podman pull busybox + ExecStart=/bin/podman run --name busybox1 busybox /bin/sh -c ""trap 'exit 0' INT TERM; while true; do echo Hello World; sleep 1; done"" + + [Install] + WantedBy=multi-user.target +``` + +### 7.5 Pre-Integration of Ignition Files + +The NestOS build toolchain enables users to customize images based on specific use cases and requirements. After creating the image, nestos-installer offers various features for customizing image deployment and application, such as pre-integrating Ignition files, pre-allocating installation locations, and modifying kernel parameters. Below, we introduce the main functionalities. + +#### 7.5.1 Pre-Integration of Ignition Files into ISO Images + +Prepare the NestOS ISO image locally and install the nestos-installer package. Edit **example.yaml** and use the Butane tool to convert it into an Ignition file. In this example, we configure a simple username and password (the password must be encrypted; the example uses `qwer1234`), as shown below: + +```YAML +variant: nestos +version: 1.0.0 +passwd: + users: + - name: root + password_hash: "$1$root$CPjzNGH.NqmQ7rh26EeXv1" +``` + +After converting the YAML file into an Ignition file, execute the following command to embed the Ignition file and specify the target drive location. Replace `xxx.iso` with the local NestOS ISO image: + +```shell +nestos-installer iso customize --dest-device /dev/sda --dest-ignition example.ign xxx.iso +``` + +When installing using the ISO image with the embedded Ignition file , NestOS will automatically read the Ignition file and install it to the target drive. Once the progress bar reaches 100%, the system will automatically boot into the installed NestOS environment. Users can log in using the username and password configured in the Ignition file. + +#### 7.5.2 Pre-Integration of Ignition Files into PXE Images + +Prepare the NestOS PXE image locally. See [Section 6.5](#65-artifacts-acquisition) for details on obtaining the components. The remaining steps are the same as above. + +To simplify the process for users, nestos-installer also supports extracting PXE components from an ISO image. Execute the following command, replacing `xxx.iso` with the local NestOS ISO image: + +```shell +nestos-installer iso extract pxe xxx.iso +``` + +This will generate the following output files: + +```text +xxx-initrd.img +xxx-rootfs.img +xxx-vmlinuz +``` + +Execute the following command to pre-integrate the Ignition file and specify the target drive location: + +```shell +nestos-installer pxe customize --dest-device /dev/sda --dest-ignition example.ign xxx-initrd.img --output custom-initrd.img +``` + +Replace `xxx-initrd.img` with `custom-initrd.img` according to the PXE installation method for NestOS. After booting, NestOS will automatically read the Ignition file and install it to the target drive. Once the progress bar reaches 100%, the system will automatically boot into the installed NestOS environment. Users can log in using the username and password configured in the Ignition file. + +## 8. Deployment Process + +### 8.1 Introduction + +NestOS supports multiple deployment platforms and common deployment methods, currently focusing on QCOW2, ISO, and PXE. Compared to general-purpose OS deployments, the main difference lies in how to pass custom deployment configurations characterized by Ignition files. The following sections will introduce these methods in detail. + +### 8.2 Installation Using QCOW2 Images + +#### 8.2.1 Creating a QCOW2 Instance with QEMU + +Prepare the NestOS QCOW2 image and the corresponding Ignition file (see [Section 7](#7-deployment-configuration) for details). Execute the following commands in the terminal: + +```shell +IGNITION_CONFIG="/path/to/example.ign" +IMAGE="/path/to/image.qcow2" +IGNITION_DEVICE_ARG="-fw_cfg name=opt/com.coreos/config,file=${IGNITION_CONFIG}" + +qemu-img create -f qcow2 -F qcow2 -b ${IMAGE} my-nestos-vm.qcow2 +``` + +For the AArch64 environment, execute the following command: + +```shell +qemu-kvm -m 2048 -M virt -cpu host -nographic -drive if=virtio,file=my-nestos-vm.qcow2 ${IGNITION_DEVICE_ARG} -nic user,model=virtio,hostfwd=tcp::2222-:22 -bios /usr/share/edk2/aarch64/QEMU_EFI-pflash.raw +``` + +For the x86_64 environment, execute the following command: + +```shell +qemu-kvm -m 2048 -M pc -cpu host -nographic -drive if=virtio,file=my-nestos-vm.qcow2 ${IGNITION_DEVICE_ARG} -nic user,model=virtio,hostfwd=tcp::2222-:22 +``` + +#### 8.2.2 Creating a QCOW2 Instance with virt-install + +Assuming the libvirt service is running normally and the network uses the default subnet bound to the `virbr0` bridge, you can follow these steps to create a NestOS instance. + +Prepare the NestOS QCOW2 image and the corresponding Ignition file (see [Section 7](#7-deployment-configuration) for details). Execute the following commands in the terminal: + +```shell +IGNITION_CONFIG="/path/to/example.ign" +IMAGE="/path/to/image.qcow2" +VM_NAME="nestos" +VCPUS="4" +RAM_MB="4096" +DISK_GB="10" +IGNITION_DEVICE_ARG=(--qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${IGNITION_CONFIG}") +``` + +**Note: When using virt-install, the QCOW2 image and Ignition file must be specified with absolute paths.** + +Execute the following command to create the instance: + +```shell +virt-install --connect="qemu:///system" --name="${VM_NAME}" --vcpus="${VCPUS}" --memory="${RAM_MB}" --os-variant="kylin-hostos10.0" --import --graphics=none --disk="size=${DISK_GB},backing_store=${IMAGE}" --network bridge=virbr0 "${IGNITION_DEVICE_ARG[@]} +``` + +### 8.3 Installation Using ISO Images + +Prepare the NestOS ISO image and boot it. The first boot of the NestOS ISO image will default to the Live environment, which is a volatile memory-based environment. + +#### 8.3.1 Installing the OS to the Target Drive Using the nestos-installer Wizard Script + +1. In the NestOS live environment, follow the printed instructions upon first entry. Enter the following command to automatically generate a simple Ignition file and proceed with the installation and reboot: + + ```shell + sudo installnestos + ``` + +2. Follow the terminal prompts to enter the username and password. + +3. Select the target drive installation location. Press **Enter** to use the default option **/dev/sda**. + +4. After completing the above steps, nestos-installer will begin installing NestOS to the target drive based on the provided configuration. Once the progress bar reaches 100%, the system will automatically reboot. + +5. After rebooting, the system will automatically enter NestOS. Press **Enter** at the GRUB menu or wait 5 seconds to boot the system. Log in using the previously configured username and password. The installation is now complete. + +#### 8.3.2 Manually Installing the OS to the Target Drive Using the nestos-installer Command + +1. Prepare the Ignition file **example.ign** (see [Section 7](#7-deployment-configuration) for details). + +2. Follow the printed instructions upon first entry into the NestOS live environment. Enter the following command to begin the installation: + + ```shell + sudo nestos-installer install /dev/sda --ignition-file example.ign + ``` + + If network access is available, the Ignition file can also be retrieved via a URL, for example: + + ```shell + sudo nestos-installer install /dev/sda --ignition-file http://www.example.com/example.ign + ``` + +3. After executing the above command, nestos-installer will begin installing NestOS to the target drive based on the provided configuration. Once the progress bar reaches 100%, the system will automatically reboot. + +4. After rebooting, the system will automatically enter NestOS. Press **Enter** at the GRUB menu or wait 5 seconds to boot the system. Log in using the previously configured username and password. The installation is now complete. + +### 8.4 PXE Deployment + +The PXE installation components for NestOS include the kernel, **initramfs.img**, and **rootfs.img**. These components are generated using the `nosa buildextend-live` command (see [Section 6](#6-build-process) for details). + +1. Use the PXELINUX `KERNEL` command to specify the kernel. A simple example is as follows: + + ```shell + KERNEL nestos-live-kernel-x86_64 + ``` + +2. Use the PXELINUX `APPEND` command to specify the initrd and rootfs. A simple example is as follows: + + ```shell + APPEND initrd=nestos-live-initramfs.x86_64.img,nestos-live-rootfs.x86_64.img + ``` + + **Note: If you have pre-integrated the Ignition file into the PXE components as described in [Section 7.5](#75-pre-integration-of-ignition-files), you only need to replace it here and skip the subsequent steps.** + +3. Specify the installation location. For example, to use **/dev/sda**, append the following to the `APPEND` command: + + ```ini + nestosos.inst.install_dev=/dev/sda + ``` + +4. Specify the Ignition file, which must be retrieved over the network. Append the corresponding URL to the `APPEND` command, for example: + + ```ini + nestos.inst.ignition_url=http://www.example.com/example.ign + ``` + +5. After booting, NestOS will automatically read the Ignition file and install the OS to the target drive. Once the progress bar reaches 100%, the system will automatically boot into the installed NestOS environment. Users can log in using the username and password configured in the Ignition file. + +## 9. Basic Usage + +### 9.1 Introduction + +NestOS employs an OS packaging solution based on ostree and rpm-ostree technologies, setting critical directories to read-only mode to prevent accidental modifications to core system files and configurations. Leveraging the overlay layering concept, it allows users to manage RPM packages on top of the base ostree filesystem without disrupting the initial system architecture. Additionally, it supports building OCI-format images, enabling OS version switching at the granularity of images. + +### 9.2 SSH Connection + +For security reasons, NestOS does not support password-based SSH login by default and only allows key-based authentication. This design enhances system security by mitigating risks associated with password leaks or weak password attacks. + +The method for establishing an SSH connection using keys in NestOS is the same as in openEuler. If users need to temporarily enable password-based login, they can follow these steps: + +1. Edit the additional configuration file of the SSH service: + + ```shell + vi /etc/ssh/sshd_config.d/40-disable-passwords.conf + ``` + +2. Modify the default `PasswordAuthentication` setting as follows: + + ```shell + PasswordAuthentication yes + ``` + +3. Restart the sshd service to temporarily enable password-based SSH login. + +### 9.3 RPM Package Installation + +**Note: Immutable OS discourages installing software packages in the runtime environment. This method is provided only for temporary debugging scenarios. For service requirements that necessitate changes to the integrated package list, rebuild the OS by updating the build configuration.** + +NestOS does not support conventional package managers like Yum or DNF. Instead, it uses rpm-ostree to manage system updates and package installations. rpm-ostree combines the advantages of image-based and package-based management, allowing users to layer and manage RPM packages on top of the base OS without disrupting its initial structure. Use the following command to install an RPM package: + +```shell +rpm-ostree install +``` + +After installation, reboot the OS. The bootloader menu will display two branches, with the first branch being the latest by default: + +```shell +systemctl reboot +``` + +After rebooting, check the system package layering status to confirm that the package has been installed in the current version: + +```shell +rpm-ostree status -v +``` + +### 9.4 Version Rollback + +After an update or RPM package installation, the previous version of the OS deployment remains on the drive. If the update causes issues, users can manually roll back to a previous version using rpm-ostree. The specific process is as follows: + +#### 9.4.1 Temporary Rollback + +To temporarily roll back to a previous OS deployment, hold down the **Shift** key during system boot. When the bootloader menu appears, select the corresponding branch (by default, there are two branches; choose the other one). Before doing this, you can use the following command to view the two existing version branches in the current environment: + +```shell +rpm-ostree status +``` + +#### 9.4.2 Permanent Rollback + +To permanently roll back to a previous OS deployment, run the following command in the current version. This operation will set system deployment of the previous version as the default deployment. + +```shell +rpm-ostree rollback +``` + +Reboot to apply the changes. The default deployment option in the bootloader menu will have changed, eliminating the need for manual switching. + +```shell +systemctl reboot +``` + +## 10. Container Image-Based Updates + +### 10.1 Use Case Description + +NestOS, as a container cloud base OS based on the immutable infrastructure concept, distributes and updates the file system as a whole. This approach brings significant convenience in terms of operations and security. However, in real-world production environments, the officially released versions often fail to meet user requirements. For example, users may want to integrate self-maintained critical foundational components by default or further trim software packages to reduce system runtime overhead based on specific scenarios. Therefore, compared to general-purpose OSs, users have stronger and more frequent customization needs for NestOS. + +nestos-assembler can provide OCI-compliant container images. Beyond simply packaging and distributing the root file system, leveraging the ostree native container feature allows container cloud users to utilize familiar technology stacks. By writing a single ContainerFile (Dockerfile), users can easily build customized images for integrating custom components or subsequent upgrade and maintenance tasks. + +### 10.2 Usage + +#### 10.2.1 Customizing Images + +- Basic steps + +1. Refer to [Section 6](#6-build-process) to build the NestOS container image, and use the `nosa push-container` command to push it to a public or private container image registry. +2. Write a Containerfile (Dockerfile) as shown in the following example: + + ```dockerfile + FROM registry.example.com/nestos:1.0.20240603.0-x86_64 + + # Perform custom build steps, such as installing software or copying self-built components. + # Here, installing the strace package is used as an example. + RUN rpm-ostree install strace && rm -rf /var/cache && ostree container commit + ``` + +3. Run `docker build` or integrate it into CI/CD to build the corresponding image. + + > Note: + > 1. NestOS does not have the yum/dnf package manager. If software packages need to be installed, use the `rpm-ostree install` command to install local RPM packages or software provided in the repository. + > 2. If needed, you can also modify the software source configurations in the `/etc/yum.repo.d/` directory. + > 3. Each meaningful build command should end with `&& ostree container commit`. From the perspective of container image build best practices, it is recommended to minimize the number of RUN layers. + > 4. During the build process, non-/usr or /etc directory contents are cleaned up. Therefore, customization via container images is primarily suitable for software package or component updates. Do not use this method for system maintenance or configuration changes (e.g., adding users with `useradd`). + +#### 10.2.2 Deploying/Upgrading Images + +Assume that the container image built in the above steps is pushed as `registry.example.com/nestos:1.0.20240903.0-x86_64`. + +In an environment where NestOS is already deployed, execute the following command: + +```shell +sudo rpm-ostree rebase ostree-unverified-registry:registry.example.com/nestos:1.0.20240903.0-x86_64 +``` + +Reboot to complete the deployment of the customized version. + +After deployment is complete using the container image method, `rpm-ostree upgrade` will default to updating the source from the ostree update source to the container image address. Subsequently, you can update the container image under the same tag. Using `rpm-ostree upgrade` will detect if the remote image has been updated. If changes are detected, it will pull the latest image and complete the deployment. diff --git a/docs/en/docs/NestOS/feature-description.md b/docs/en/docs/NestOS/feature-description.md index 2a607c82f1d5e1342727cf21b1ae942c9830b5be..f53f9cf28af8028c6aba38da47cee88d52905e27 100644 --- a/docs/en/docs/NestOS/feature-description.md +++ b/docs/en/docs/NestOS/feature-description.md @@ -64,7 +64,7 @@ Zincati is an auto-update agent for NestOS hosts. It works as a client for the C ## System Initialization (Ignition) -Ignition is a distribution-agnostic provisioning utility that not only installs, but also reads configuration files (in JSON format) to provision NestOS. Configurable components include storage and file systems, systemd units, and users. +Ignition is a distribution-agnostic provisioning utility that not only installs, but also reads configuration files (in JSON format) to initialize NestOS. Configurable components include storage and file systems, systemd units, and users. Ignition runs only once during the first boot of the system (while in the initramfs). Because Ignition runs so early in the boot process, it can re-partition disks, format file systems, create users, and write files before the userspace begins to boot. As a result, systemd services are already written to disk when systemd starts, speeding the time to boot. @@ -72,7 +72,7 @@ Ignition runs only once during the first boot of the system (while in the initra Ignition is designed to be used as a provisioning tool, not as a configuration management tool. Ignition encourages immutable infrastructure, in which machine modification requires that users discard the old node and re-provision the machine. (2) Ignition produces the machine specified or no machine at all -Ignition does what it needs to make the system match the state described in the Ignition configuration. If for any reason Ignition cannot deliver the exact machine that the configuration asked for, Ignition prevents the machine from booting successfully. For example, if the user wanted to fetch the document hosted at **https://example.com/foo.conf** and write it to disk, Ignition would prevent the machine from booting if it were unable to resolve the given URL. +Ignition does what it needs to make the system match the state described in the Ignition configuration. If for any reason Ignition cannot deliver the exact machine that the configuration asked for, Ignition prevents the machine from booting successfully. For example, if the user wanted to fetch the document hosted at **** and write it to disk, Ignition would prevent the machine from booting if it were unable to resolve the given URL. (3) Ignition configurations are declarative Ignition configurations describe the state of a system. Ignition configurations do not list a series of steps that Ignition should take. @@ -87,14 +87,14 @@ Afterburn is a one-shot agent for cloud-like platforms which interacts with prov Afterburn comprises several modules which may run at different times during the lifecycle of an instance. Depending on the specific platform, the following services may run in the initramfs on first boot: - - setting local hostname +- setting local hostname - - injecting network command-line arguments +- injecting network command-line arguments The following features are conditionally available on some platforms as systemd service units: - - installing public SSH keys for local system users +- installing public SSH keys for local system users - - retrieving attributes from instance metadata +- retrieving attributes from instance metadata - - checking in to the provider in order to report a successful boot or instance provisioning +- checking in to the provider in order to report a successful boot or instance provisioning diff --git a/docs/en/docs/NestOS/figures/figure1.png b/docs/en/docs/NestOS/figures/figure1.png new file mode 100644 index 0000000000000000000000000000000000000000..b4eb9017ed202e854c076802492d8561942dfc88 Binary files /dev/null and b/docs/en/docs/NestOS/figures/figure1.png differ diff --git a/docs/en/docs/NestOS/figures/figure2.png b/docs/en/docs/NestOS/figures/figure2.png new file mode 100644 index 0000000000000000000000000000000000000000..90049769c04e2bd494533da1613e38a5199da3d7 Binary files /dev/null and b/docs/en/docs/NestOS/figures/figure2.png differ diff --git a/docs/en/docs/Quickstart/quick-start.md b/docs/en/docs/Quickstart/quick-start.md index d09fe522dd51b3160bc5fa30b010414103b8f2cc..eb27e9318b0f49560edb936ae8db62b93090a765 100644 --- a/docs/en/docs/Quickstart/quick-start.md +++ b/docs/en/docs/Quickstart/quick-start.md @@ -1,52 +1,20 @@ # Quick Start -This document uses openEuler 22.03 LTS SP2 installed on the TaiShan 200 server as an example to describe how to quickly install and use openEuler OS. For details about the installation requirements and methods, see the [Installation Guide](./../Installation/Installation.md). +This document uses openEuler installation on the TaiShan 200 server as an example to describe how to quickly install and use openEuler OS. For details about the installation requirements and methods, see the [Installation Guide](./../Installation/Installation.md). ## Making Preparations - Hardware Compatibility - - [Table 1](#table14948632047) describes the types of supported servers. - - **Table 1** Supported servers - - - - - - - - - - - - - - - - - -

Server Type

-

Server Name

-

Server Model

-

Rack server

-

TaiShan 200

-

2280 balanced model

-

Rack server

-

FusionServer Pro

-

FusionServer Pro 2288H V5

-
NOTE:

The server must be configured with the Avago SAS3508 RAID controller card and the LOM-X722 NIC.

-
-
+ [Compatibility List](https://www.openeuler.org/en/compatibility/) describes the types of supported servers. - Minimum Hardware Specifications - + [Table 2](#tff48b99c9bf24b84bb602c53229e2541) lists the minimum hardware specifications supported by openEuler. - + **Table 2** Minimum hardware specifications - + - + @@ -89,17 +57,17 @@ Perform the following operations to obtain the openEuler release package: 4. Click **Download** on the right of **openEuler 23.09**. 5. Download the required openEuler release package and the corresponding verification file based on the architecture and scenario. - - If the AArch64 architecture is used: + - If the AArch64 architecture is used: - 1. Click **AArch64**. - 2. For local installation, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-22.03-LTS-SP2-aarch64-dvd.iso** to the local host. - 3. For network installation, download the **Network Install ISO** release package **openEuler-22.03-LTS-SP2-netinst-aarch64-dvd.iso** to the local host. + 1. Click **AArch64**. + 2. For local installation, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-24.03-LTS-SP1-aarch64-dvd.iso** to the local host. + 3. For network installation, download the **Network Install ISO** release package **openEuler-24.03-LTS-SP1-netinst-aarch64-dvd.iso** to the local host. - - If the x86\_64 architecture is used: + - If the x86\_64 architecture is used: - 1. Click **x86_64**. - 2. For local installation, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-22.03-LTS-SP2-x86_64-dvd.iso** to the local host. - 3. For network installation, download the **Network Install ISO** release package **openEuler-22.03-LTS-SP2-netinst-x86_64-dvd.iso** to the local host. + 1. Click **x86_64**. + 2. For local installation, download the **Offline Standard ISO** or **Offline Everything ISO** release package **openEuler-24.03-LTS-SP1-x86_64-dvd.iso** to the local host. + 3. For network installation, download the **Network Install ISO** release package **openEuler-24.03-LTS-SP1-netinst-x86_64-dvd.iso** to the local host. > ![](./public_sys-resources/icon-note.gif) **NOTE:** > @@ -123,47 +91,47 @@ Compare the verification value recorded in the verification file with the verifi The following files need to be prepared: -ISO file: openEuler-22.03-LTS-SP2-aarch64-dvd.iso +ISO file: openEuler-24.03-LTS-SP1-aarch64-dvd.iso Verification file: Copy and save the SHA256 value corresponding to the ISO file. 1. Calculate the SHA256 verification value of the file. Run the following command: ```sh - # sha256sum openEuler-22.03-LTS-SP2-aarch64-dvd.iso + # sha256sum openEuler-24.03-LTS-SP1-aarch64-dvd.iso ``` 2. Check whether the values calculated in step 1 and step 2 are the same. - If the values are consistent, the ISO file is not damaged. Otherwise, the file is damaged and you need to obtain it again. - If the values are consistent, the ISO file is not damaged. Otherwise, the file is damaged and you need to obtain it again. + If the values are consistent, the ISO file is not damaged. Otherwise, the file is damaged and you need to obtain it again. + If the values are consistent, the ISO file is not damaged. Otherwise, the file is damaged and you need to obtain it again. ## Starting Installation 1. Log in to the iBMC WebUI. - For details, see [TaiShan 200 Server User Guide (Model 2280)](https://support.huawei.com/enterprise/en/doc/EDOC1100093459). + For details, see [TaiShan 200 Server User Guide (Model 2280)](https://support.huawei.com/enterprise/en/doc/EDOC1100093459). 2. Choose **Configuration** from the main menu, and select **Boot Device** from the navigation tree. The **Boot Device** page is displayed. - Set **Effective** and **Boot Medium** to **One-time** and **DVD-ROM**, respectively, and click **Save**, as shown in [Figure 1](#fig1011938131018). + Set **Effective** and **Boot Medium** to **One-time** and **DVD-ROM**, respectively, and click **Save**, as shown in [Figure 1](#fig1011938131018). - **Figure 1** Setting the boot device -![fig](./figures/setting-the-boot-device.png "setting-the-boot-device") + Figure 1** Setting the boot device +![fig](./figures/setting-the-boot-device.png) 3. Choose **Remote Console** from the main menu. The **Remote Console** page is displayed. - Select an integrated remote console as required to access the remote virtual console, for example, **Java Integrated Remote Console (Shared)**. + Select an integrated remote console as required to access the remote virtual console, for example, **Java Integrated Remote Console (Shared)**. 4. On the toolbar, click the icon shown in the following figure. - **Figure 2** Drive icon -![fig](./figures/drive-icon.png "drive-icon") + igure 2** Drive icon +![fig](./figures/drive-icon.png) - An image dialog box is displayed, as shown in the following figure. + An image dialog box is displayed, as shown in the following figure. - **Figure 3** Image dialog box -![fig](./figures/image-dialog-box.png "image-dialog-box") + **Figure 3** Image dialog box +![fig](./figures/image-dialog-box.png) 5. Select **Image File** and then click **Browse**. The **Open** dialog box is displayed. @@ -171,20 +139,20 @@ Verification file: Copy and save the SHA256 value corresponding to the ISO file. 7. On the toolbar, click the restart icon shown in the following figure to restart the device. - **Figure 4** Restart icon -![fig](./figures/restart-icon.png "restart-icon") + **Figure 4** Restart icon +![fig](./figures/restart-icon.png) 8. A boot menu is displayed after the system restarts, as shown in [Figure 5](#fig1648754873314). - > ![fig](./public_sys-resources/icon-note.gif) **NOTE:** - > - > - If you do not perform any operations within 1 minute, the system automatically selects the default option **Test this media \& install openEuler 22.03-LTS-SP2** and enters the installation page. - > - During physical machine installation, if you cannot use the arrow keys to select boot options and the system does not respond after you press **Enter**, click ![fig](./figures/en-us_image_0229420473.png) on the BMC page and configure **Key \& Mouse Reset**. + > ![fig](./public_sys-resources/icon-note.gif) **NOTE:** + > + > - If you do not perform any operations within 1 minute, the system automatically selects the default option **Test this media \& install openEuler 24.03-LTS-SP1** and enters the installation page. + > - During physical machine installation, if you cannot use the arrow keys to select boot options and the system does not respond after you press **Enter**, click ![fig](./figures/en-us_image_0229420473.png) on the BMC page and configure **Key \& Mouse Reset**. **Figure 5** Installation wizard - ![fig](./figures/Installation_wizard.png "Installation_wizard") + ![fig](./figures/Installation_wizard.png) -9. On the installation wizard page, press **Enter** to select the default option **Test this media \& install openEuler 22.03-LTS-SP2** to enter the GUI installation page. +9. On the installation wizard page, press **Enter** to select the default option **Test this media \& install openEuler 24.03-LTS-SP1** to enter the GUI installation page. ## Performing Installation @@ -193,23 +161,23 @@ After entering the GUI installation page, perform the following operations to in 1. Set an installation language. The default language is English. You can change the language based on the site requirements, as shown in [Figure 6](#fig874344811484). **Figure 6** Selecting a language - ![fig](./figures/selecting-a-language.png "selecting-a-language") + ![fig](./figures/selecting-a-language.png) 2. On the **INSTALLATION SUMMARY** page, set configuration items based on the site requirements. - - A configuration item with an alarm symbol must be configured. When the alarm symbol disappears, you can perform the next operation. - - A configuration item without an alarm symbol is configured by default. - - You can click **Begin Installation** to install the system only when all alarms are cleared. + - A configuration item with an alarm symbol must be configured. When the alarm symbol disappears, you can perform the next operation. + - A configuration item without an alarm symbol is configured by default. + - You can click **Begin Installation** to install the system only when all alarms are cleared. - **Figure 7** Installation summary - ![fig](./figures/installation-summary.png "installation-summary") + **Figure 7** Installation summary + ![fig](./figures/installation-summary.png) - 1. Select **Software Selection** to set configuration items. + 1. Select **Software Selection** to set configuration items. Based on the site requirements, select **Minimal Install** on the left box and select an add-on in the **Add-Ons for Selected Environment** area on the right, as shown in [Figure 8](#fig1133717611109). **Figure 8** Selecting installation software - ![fig](./figures/selecting-installation-software.png "selecting-installation-software") + ![fig](./figures/selecting-installation-software.png) > ![fig](./public_sys-resources/icon-note.gif) **NOTE:** > @@ -218,7 +186,7 @@ After entering the GUI installation page, perform the following operations to in After the setting is complete, click **Done** in the upper left corner to go back to the **INSTALLATION SUMMARY** page. - 2. Select **Installation Destination** to set configuration items. + 2. Select **Installation Destination** to set configuration items. On the **INSTALLATION DESTINATION** page, select a local storage device. @@ -229,7 +197,7 @@ After entering the GUI installation page, perform the following operations to in You also need to configure the storage to partition the system. You can either manually configure partitions or select **Automatic** for automatic partitioning. Select **Automatic** if the software is installed in a new storage device or the data in the storage device is not required, as shown in [Figure 9](#fig153381468101). **Figure 9** Setting the installation destination - ![fig](./figures/setting-the-installation-destination.png "setting-the-installation-destination") + ![fig](./figures/setting-the-installation-destination.png) > ![fig](./public_sys-resources/icon-note.gif) **NOTE:** > @@ -239,7 +207,7 @@ After entering the GUI installation page, perform the following operations to in After the setting is complete, click **Done** in the upper left corner to go back to the **INSTALLATION SUMMARY** page. - 3. Select **Root Password** and set the root password. + 3. Select **Root Password** and set the root password. On the **ROOT PASSWORD** page, enter a password that meets the [Password Complexity](#password-complexity) requirements and confirm the password, as shown in [Figure 10](#zh-cn_topic_0186390266_zh-cn_topic_0122145909_fig1323165793018). @@ -266,35 +234,35 @@ After entering the GUI installation page, perform the following operations to in > In the openEuler environment, you can run the `cracklib-unpacker /usr/share/cracklib/pw_dict > dictionary.txt` command to export the dictionary library file **dictionary.txt**. You can check whether the password is in this dictionary. **Figure 10** root password - ![fig](./figures/password-of-the-root-account.png "Root password") + ![fig](./figures/password-of-the-root-account.png) After the settings are completed, click **Done** in the upper left corner to go back to the **INSTALLATION SUMMARY** page. - 4. Select **Create a User** and set the parameters. + 4. Select **Create a User** and set the parameters. [Figure 11](#zh-cn_topic_0186390266_zh-cn_topic_0122145909_fig1237715313319) shows the page for creating a user. Enter the user name and set the password. The password complexity requirements are the same as those of the root password. In addition, you can set the home directory and user group by clicking **Advanced**, as shown in [Figure 12](#zh-cn_topic_0186390266_zh-cn_topic_0122145909_fig1237715313319). **Figure 11** Creating a user - ![fig](./figures/creating-a-user.png "creating-a-user") + ![fig](./figures/creating-a-user.png) **Figure 12** Advanced user configuration - ![fig](./figures/advanced-user-configuration.png "Advanced user configuration") + ![fig](./figures/advanced-user-configuration.png) After the settings are completed, click **Done** in the upper left corner to go back to the **INSTALLATION SUMMARY** page. - 5. Set other configuration items. You can use the default values for other configuration items. + 5. Set other configuration items. You can use the default values for other configuration items. 3. Click **Start the Installation** to install the system, as shown in [Figure 13](#zh-cn_topic_0186390266_zh-cn_topic_0122145909_fig1237715313319). **Figure 13** Starting the installation - ![fig](./figures/installation-process.png "installation-process") + ![fig](./figures/installation-process.png) 4. After the installation is completed, restart the system. openEuler has been installed. Click **Reboot** to restart the system. ## Viewing System Information -After the system is installed and restarted, the system CLI login page is displayed. Enter the username and password set during the installation to log in to openEuler and view the following system information. For details about system management and configuration, see the [openEuler 22.03-LTS-SP2 Administrator Guide](../Administration/administration.md). +After the system is installed and restarted, the system CLI login page is displayed. Enter the username and password set during the installation to log in to openEuler and view the following system information. For details about system management and configuration, see the [openEuler 24.03-LTS-SP1 Administrator Guide](../Administration/administration.md). - Run the following command to view the system information: @@ -348,4 +316,4 @@ After the system is installed and restarted, the system CLI login page is displa ```sh # ip addr - ``` \ No newline at end of file + ``` diff --git a/docs/en/docs/Releasenotes/installing-the-os.md b/docs/en/docs/Releasenotes/installing-the-os.md index 48f672163b9c596187bff0e9e6be64de70796129..acf5ebd73f0a639a6ae1293bc4619df563f3b2ae 100644 --- a/docs/en/docs/Releasenotes/installing-the-os.md +++ b/docs/en/docs/Releasenotes/installing-the-os.md @@ -2,237 +2,80 @@ ## Release Files -The openEuler release files include [ISO release package](http://repo.openeuler.org/openEuler-22.09/ISO/), [VM images](http://repo.openeuler.org/openEuler-22.09/virtual_machine_img/), [container images](http://repo.openeuler.org/openEuler-22.09/docker_img/), [embedded images](http://repo.openeuler.org/openEuler-22.09/embedded_img/), and [repo sources](http://repo.openeuler.org/openEuler-22.09/). [Table 1](#table8396719144315) describes the ISO release packages. [Table 3](#table1276911538154) describes the container images. [Table 5](#table953512211576) describes the repo sources, which are convenient for online use. +The openEuler release files include [ISO release packages](https://www.openeuler.org/en/download/archive/), [VM images](http://repo.openeuler.org/openEuler-24.03-LTS/virtual_machine_img/), [container images](http://repo.openeuler.org/openEuler-24.03-LTS/docker_img/), [embedded images](http://repo.openeuler.org/openEuler-24.03-LTS/embedded_img/), and [repositories](http://repo.openeuler.org/openEuler-24.03-LTS/). **Table 1** ISO release packages - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Name

-

Description

-

openEuler-22.09-aarch64-dvd.iso

-

Base installation ISO file of the AArch64 architecture, including the core components for running the minimum system.

-

openEuler-22.09-everything-aarch64-dvd.iso

-

Full installation ISO file of the AArch64 architecture, including all components for running the entire system.

-

openEuler-22.09-everything-debug-aarch64-dvd.iso

-

ISO file for openEuler debugging in the AArch64 architecture, including the symbol table information required for debugging.

-

openEuler-22.09-x86_64-dvd.iso

-

Base installation ISO file of the x86_64 architecture, including the core components for running the minimum system.

-

openEuler-22.09-everything-x86_64-dvd.iso

-

Full installation ISO file of the x86_64 architecture, including all components for running the entire system.

-

openEuler-22.09-everything-debuginfo-x86_64-dvd.iso

-

ISO file for openEuler debugging in the x86_64 architecture, including the symbol table information required for debugging.

-

openEuler-22.09-source-dvd.iso

-

ISO file of the openEuler source code.

-

openEuler-21.09-edge-aarch64-dvd.iso

-

Edge ISO file in the AArch64 architecture, including the core components for running the minimum system.

-

openEuler-21.09-edge-x86_64-dvd.iso

-

Edge ISO file in the x86_64 architecture, including the core components for running the minimum system.

-

openEuler-21.09-Desktop-aarch64-dvd.iso

-

Desktop ISO file in the AArch64 architecture, including the minimum software set for running the development desktop.

-

openEuler-21.09-Desktop-x86_64-dvd.iso

-

Desktop ISO file in the x86_64 architecture, including the minimum software set for running the development desktop.

-
+| Name | Description | +| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| openEuler-24.03-LTS-SP1-aarch64-dvd.iso | Base installation ISO file of the AArch64 architecture, including the core components for running the minimum system. | +| openEuler-24.03-LTS-SP1-everything-aarch64-dvd.iso | Full installation ISO file of the AArch64 architecture, including all components for running the entire system. | +| openEuler-24.03-LTS-SP1-everything-debug-aarch64-dvd.iso | ISO file for openEuler debugging in the AArch64 architecture, including the symbol table information required for debugging. | +| openEuler-24.03-LTS-SP1-x86_64-dvd.iso | Base installation ISO file of the x86_64 architecture, including the core components for running the minimum system. | +| openEuler-24.03-LTS-SP1-everything-x86_64-dvd.iso | Full installation ISO file of the x86_64 architecture, including all components for running the entire system. | +| openEuler-24.03-LTS-SP1-everything-debuginfo-x86_64-dvd.iso | ISO file for openEuler debugging in the x86_64 architecture, including the symbol table information required for debugging. | +| openEuler-24.03-LTS-SP1-source-dvd.iso | ISO file of the openEuler source code. | +| openEuler-21.09-edge-aarch64-dvd.iso | Edge ISO file in the AArch64 architecture, including the core components for running the minimum system. | +| openEuler-21.09-edge-x86_64-dvd.iso | Edge ISO file in the x86_64 architecture, including the core components for running the minimum system. | +| openEuler-21.09-Desktop-aarch64-dvd.iso | Desktop ISO file in the AArch64 architecture, including the minimum software set for running the development desktop. | +| openEuler-21.09-Desktop-x86_64-dvd.iso | Desktop ISO file in the x86_64 architecture, including the minimum software set for running the development desktop. | **Table 2** VM images - - - - - - - - - - - - - -

Name

-

Description

-

openEuler-22.09-aarch64.qcow2.xz

-

VM image of openEuler in the AArch64 architecture.

-

openEuler-22.09-x86_64.qcow2.xz

-

VM image of openEuler in the x86_64 architecture.

-
+| Name | Description | +| ---------------------------------------- | -------------------------------------------------- | +| openEuler-24.03-LTS-SP1-aarch64.qcow2.xz | VM image of openEuler in the AArch64 architecture. | +| openEuler-24.03-LTS-SP1-x86_64.qcow2.xz | VM image of openEuler in the x86_64 architecture. | >![](./public_sys-resources/icon-note.gif) **NOTE** >The default password of the **root** user of the VM image is **openEuler12#$**. Change the password upon the first login. **Table 3** Container images - - - - - - - - - - - - - -

Name

-

Description

-

openEuler-docker.aarch64.tar.xz

-

AContainer image of openEuler in the AArch64 architecture.

-

openEuler-docker.x86_64.tar.xz

-

Container image of openEuler in the x86_64 architecture.

-
+| Name | Description | +| ------------------------------- | ----------------------------------------------------------- | +| openEuler-docker.aarch64.tar.xz | A Container image of openEuler in the AArch64 architecture. | +| openEuler-docker.x86_64.tar.xz | Container image of openEuler in the x86_64 architecture. | **Table 4** Embedded images -| Name | Description | -| -------------------------------------- | ------------------------------------------------------------ | -| arm64/aarch64-std/zImage | Kernel image that supports QEMU in the AArch64 architecture. | -| arm64/aarch64-std/\*toolchain-22.09.sh | Development and compilation toolchain in the AArch64 architecture. | -| arm64/aarch64-std/\*rootfs.cpio.gz | File system that supports QEMU in the AArch64 architecture. | -| arm32/arm-std/zImage | Kernel image that supports QEMU in the ARM architecture. | -| arm32/arm-std/\*toolchain-22.09.sh | Development and compilation toolchain in the ARM architecture. | -| arm32/arm-std/\*rootfs.cpio.gz | File system that supports QEMU in the ARM architecture. | -| source-list/manifest.xml | Manifest of source code used for building. | +| Name | Description | +| ---------------------------------------------- | ------------------------------------------------------------------ | +| arm64/aarch64-std/zImage | Kernel image that supports QEMU in the AArch64 architecture. | +| arm64/aarch64-std/\*toolchain-24.03-LTS-SP1.sh | Development and compilation toolchain in the AArch64 architecture. | +| arm64/aarch64-std/\*rootfs.cpio.gz | File system that supports QEMU in the AArch64 architecture. | +| arm32/arm-std/zImage | Kernel image that supports QEMU in the ARM architecture. | +| arm32/arm-std/\*toolchain-24.03-LTS-SP1.sh | Development and compilation toolchain in the ARM architecture. | +| arm32/arm-std/\*rootfs.cpio.gz | File system that supports QEMU in the ARM architecture. | +| source-list/manifest.xml | Manifest of source code used for building. | **Table 5** Repo sources - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Name

-

Description

-

ISO

-

Stores ISO images.

-

OS

-

Stores basic software package sources.

-

debuginfo

-

Stores debugging package sources.

-

docker_img

-

Stores container images.

-

virtual_machine_img

-

Stores VM images.

-

embedded_img

-

Stores embedded images.

-

everything

-

Stores full software package sources.

-

extras

-

Stores extended software package sources.

-

source

-

Stores source code software package.

-

update

-

Stores update software package sources.

-

EPOL

-

Stores extended openEuler package sources.

-
+| Name | Description | +| ------------------- | ------------------------------------------ | +| ISO | Stores ISO images. | +| OS | Stores basic software package sources. | +| debuginfo | Stores debugging package sources. | +| docker_img | Stores container images. | +| virtual_machine_img | Stores VM images. | +| embedded_img | Stores embedded images. | +| everything | Stores full software package sources. | +| extras | Stores extended software package sources. | +| source | Stores source code software package. | +| update | Stores update software package sources. | +| EPOL | Stores extended openEuler package sources. | ## Minimum Hardware Specifications -[Table 6](#zh-cn_topic_0182825778_tff48b99c9bf24b84bb602c53229e2541) lists the minimum hardware specifications for installing openEuler 22.09. +Table 6 lists the minimum hardware specifications for installing openEuler 24.03-LTS-SP1. **Table 6** Minimum hardware requirements - - - - - - - - - - - - - - - - -

Component

-

Minimum Hardware Specification

-

CPU

-

Kunpeng 920 (AArch64)/ x86_64 (later than Skylake)

-

x86-64 (later than Skylake)

-

Memory

-

≥ 8 GB

-

Hard drive

-

≥ 120 GB

-
+| Component | Minimum Hardware Specification | +| ---------- | ----------------------------------------------------------------------------------- | +| CPU | Kunpeng 920 (AArch64)/ x86_64 (later than Skylake)
x86-64 (later than Skylake) | +| Memory | ≥ 8 GB | +| Hard drive | ≥ 120 GB | ## Hardware Compatibility diff --git a/docs/en/docs/Releasenotes/key-features.md b/docs/en/docs/Releasenotes/key-features.md index ed48af6124fc8c5781b450346052aee2dc9f05a6..015b9ea6fd6a93a22216b200e4530da3fcc5e700 100644 --- a/docs/en/docs/Releasenotes/key-features.md +++ b/docs/en/docs/Releasenotes/key-features.md @@ -1,297 +1,529 @@ -# Key Features +# Key Features -## GMEM +## AI -Generalized Memory Management (GMEM) is an optimal solution for memory management in OS for AI. It provides a centralized management mechanism for heterogeneous memory interconnection. GMEM innovates the memory management architecture in the Linux kernel. Its logical mapping system masks the differences between the ways how the CPU and accelerator access memory addresses. The Remote Pager memory message interaction framework provides the device access abstraction layer. In the unified address space, GMEM automatically migrates data to the OS or accelerator when data needs to be accessed or paged. GMEM APIs are consistent with native Linux memory management APIs, and feature high usability, performance, and portability. +openEuler offers an efficient development and runtime environment that containerizes software stacks of AI platforms with out-of-the-box availability. It also provides various AI frameworks to facilitate AI development. -- **Logical mapping system**: GMEM high-level APIs in the kernel allow the accelerator driver to directly obtain memory management functions and create logical page tables. The logical page tables decouple the high-layer logic of memory management from the hardware layer of the CPU, so as to abstract the high-layer memory management logic that can be reused by various accelerators. +- **OS for AI**: openEuler offers an efficient development and runtime environment that containerizes software stacks of AI platforms with out-of-the-box availability. It also provides various AI frameworks to facilitate AI development. + - openEuler supports TensorFlow, PyTorch, and MindSpore frameworks and software development kits (SDKs) of major computing architectures, such as Compute Architecture for Neural Networks (CANN) and Compute Unified Architecture (CUDA), to make it easy to develop and run AI applications. + - Environment setup is further simplified by containerizing software stacks. openEuler provides three types of container images: + - SDK images: Use openEuler as the base image and install the SDK of a computing architecture, for example, Ascend CANN and NVIDIA CUDA. + - AI framework images: Use an SDK image as the base and install AI framework software, such as PyTorch and TensorFlow. You can use an AI framework image to quickly build a distributed AI framework, such as Ray. + - Model application images: Provide a complete set of toolchains and model applications -- **Remote Pager**: This framework has the message channel, process management, memory swap, and memory prefetch modules for the interaction between the host and accelerator. The remote_pager abstraction layer simplifies device adaptation by enabling third-party accelerators to easily access the GMEM system. + For details, see the [openEuler AI Container Image User Guide](https://gitee.com/openeuler/docs/blob/stable2-22.03_LTS_SP3/docs/en/docs/AI/ai_container_image_user_guide.md). -- **User APIs**: Users can directly use the memory map (mmap) of the OS to allocate the unified virtual memory. GMEM adds the flag (MMAP_PEER_SHARED) for allocating the unified virtual memory to the mmap system call. The libgmem user-mode library provides the hmadvise API of memory prefetch semantics to help users optimize the accelerator memory access efficiency. +- **AI for OS**: AI is making openEuler smarter. openEuler Copilot System is an intelligent Q&A platform developed on foundation models and openEuler data. It is designed to streamline code generation, troubleshooting, and O&M. -## Native Support for Open Source Large Language Models (LLaMA and ChatGLM) + - Intelligent Q&A: openEuler Copilot System is accessible via web or shell. + 1. Workflow scheduling: + - Atomic agent operations: Multiple agent operations can be combined into a multi-step workflow that is internally ordered and associated, and is executed as an inseparable atomic operation. + - Real-time data processing: Data generated in each step of the workflow can be processed immediately and then transferred to the next step. + - Intelligent interaction: When openEuler Copilot System receives a vague or complex user instruction, it proactively asks the user to clarify and provide more details. + 2. Task recommendation: + - Intelligent response: openEuler Copilot System can analyze the semantic information entered in text format, determining the user's intent and selecting the most matched workflow. + - Intelligent guidance: openEuler Copilot System comprehensively analyzes the execution status, function requirements, and associated tasks of the current workflow, and provides next-step operation suggestions based on users' personal preferences and historical patterns. + 3. Retrieval-augmented generation (RAG): + - RAG used in openEuler Copilot System efficiently handles diverse document formats and content scenarios, improving the Q&A service experience while minimizing additional system load. + 4. Corpus governance: + - Corpus governance is a core RAG capability. It imports corpuses into the knowledge base in a supported format using fragment relationship extraction, fragment derivative construction, and optical character recognition (OCR). This increases the retrieval hit rate. -The two model inference frameworks, llama.cpp and chatglm-cpp, are implemented based on C/C++. They allow users to deploy and use open source large language models on CPUs by means of model quantization. llama.cpp supports the deployment of multiple open source LLMs, such as LLaMa, LLaMa2, and Vicuna. It supports the deployment of multiple open source Chinese LLMs, such as ChatGLM-6B, ChatGLM2-6B, and Baichuan-13B. + For details, see the [openEuler Copilot System Intelligent Q&A Service User Guide](https://gitee.com/openeuler/docs/blob/stable2-22.03_LTS_SP4/docs/en/docs/AI/EulerCopilot_user_guide.md). -- Implemented in GGML-based C/C++. + - Intelligent tuning: openEuler Copilot System supports the intelligent shell entry. Through this entry, you can interact with the openEuler Copilot System using a natural language and perform heuristic tuning operations such as performance data collection, system performance analysis, and system performance tuning. + - Intelligent diagnosis: + 1. Inspection: The Inspection Agent checks for abnormalities of designated IP addresses and provides an abnormality list that contains associated container IDs and abnormal metrics (such as CPU and memory). + 2. Demarcation: The Demarcation Agent analyzes and demarcates a specified abnormality contained in the inspection result and outputs the top 3 metrics of the root cause. + 3. Location: The Detection Agent performs profiling location analysis on the root cause, and provides useful hotspot information such as the stack, system time, and performance metrics related to the root cause. + - Intelligent container images: openEuler Copilot System can invoke environment resources through a natural language, assist in pulling container images for local physical resources, and establish a development environment suitable for debugging on existing compute devices. This system supports three types of containers, and container images have been released on Docker Hub. You can manually pull and run these container images. + 1. SDK layer: encapsulates only the component libraries that enable AI hardware resources, such as CUDA and CANN. + 2. SDKs + training/inference frameworks: accommodates TensorFlow, PyTorch, and other frameworks (for example, tensorflow2.15.0-cuda12.2.0 and pytorch2.1.0.a1-cann7.0.RC1) in addition to the SDK layer. + 3. SDKs + training/inference frameworks + LLMs: encapsulates several models (for example, llama2-7b and chatglm2-13b) based on the second type of containers -- They accelerate memory for efficient CPU inference through int4/int8 quantization, optimized KV cache, and parallel computing. +## openEuler Embedded -## Features in the openEuler Kernel 6.4 +openEuler Embedded provides a closed loop framework often found in operational technology (OT) applications such as manufacturing and robotics, whereby innovations help optimize its embedded system software stack and ecosystem. openEuler 24.03 LTS SP1 Embedded is designed for embedded applications, offering significant progress in southbound and northbound ecosystems, technical features, infrastructure, and implementation over previous generations. openEuler Embedded will work alongside community partners, users, and developers to broaden its support for emerging chip architectures, including Loongson, and a wider range of hardware. The platform will strengthen offerings in industrial middleware, embedded AI, edge computing, and simulation systems, delivering a robust solution for embedded system software. -openEuler 23.09 runs on Linux kernel 6.4. It inherits the competitive advantages of community versions and innovative features released in the openEuler community. +- **Southbound ecosystem**: openEuler Embedded Linux supports mainstream processor architectures like AArch64, x86_64, AArch32, and RISC-V, and will extend support to LoongArch in the future. openEuler 24.03 and later versions have a rich southbound ecosystem and support chips from Raspberry Pi, HiSilicon, Rockchip, Renesas, TI, Phytium, StarFive, and Allwinner. In openEuler 24.03 LTS SP1 Embedded, Kunpeng 920 is also supported. +- **Embedded virtualization base**: openEuler Embedded uses an elastic virtualization base that enables multiple OSs to run on a system-on-a-chip (SoC). The base incorporates a series of technologies including bare metal, embedded virtualization, lightweight containers, LibOS, trusted execution environment (TEE), and heterogeneous deployment. +- **MICA deployment framework**: The MICA deployment framework is a unified environment that masks the differences between technologies that comprise the embedded elastic virtualization base. The multi-core capability of hardware combines the universal Linux OS and a dedicated real-time operating system (RTOS) to make full use of all OSs. +- **Northbound ecosystem**: + - Northbound software packages: Over 600 common embedded software packages can be built using openEuler. + - Soft real-time kernel: This capability helps respond to soft real-time interrupts within microseconds. + - DSoftBus: The distributed soft bus system (DSoftBus) of openEuler Embedded integrates the DSoftBus and point-to-point authentication module of OpenHarmony. It implements interconnection between openEuler-based embedded devices and OpenHarmony-based devices as well as between openEuler-based embedded devices. + - Embedded containers and edges: With iSula containers, openEuler and other OS containers can be deployed on embedded devices to simplify application porting and deployment. Embedded container images can be compressed to 5 MB, and can be easily deployed into the OS on another container. +- **UniProton**: UniProton is an RTOS that features ultra-low latency and flexible MICA deployments. It is suited for industrial control because it supports both microcontroller units and multi-core CPUs. UniProton provides the following capabilities: + - Compatible with processor architectures like Cortex-M, AArch64, x86_64, and riscv64, and supports M4, RK3568, RK3588, x86_64, Hi3093, Raspberry Pi 4B, Kunpeng 920, Ascend 310, and Allwinner D1s. + - Connects with openEuler Embedded Linux on Raspberry Pi 4B, Hi3093, RK3588, and x86_64 devices in bare metal mode. + - Can be debugged using GDB on openEuler Embedded Linux. -- **Tidal affinity scheduling:** The system dynamically adjusts CPU affinity based on the service load. When the service load is light, the system uses preferred CPUs to enhance resource locality. When the service load is heavy, the system has new CPU cores added to improve the QoS. +## DevStation -- **CPU QoS priority-based load balancing**: CPU QoS isolation is enhanced in online and offline hybrid deployments, and QoS load balancing across CPUs is supported to further reduce QoS interference from offline services. +DevStation is the first openEuler developer workstation to come pre-installed with VS Code. Designed to streamline software coding, compilation, build, release, and deployment, DevStation integrates the oeDeploy tool to simplify the deployment of the AI and cloud-native software stacks and uses oeDevPlugin to pull code repositories and the AI4C-powered compiler for compilation. Later versions will include epkg, a new openEuler package manager that enables developers to seamlessly manage and switch betwe n multiple software versions across different environments. -- **Simultaneous multithreading (SMT) expeller free of priority inversion**: This feature resolves the priority inversion problem in the SMT expeller feature and reduces the impact of offline tasks on the quality of service (QoS) of online tasks. +- **Developer-friendly integrated environment**: supports multiple programming languages and comes with pre-installed development tools and IDEs such as VS Code to streamline front-end, back-end, and full-stack development. +- **Package management and automated deployment**: features easy-to-use package management tools for one-click installation and updates. It integrates container technologies such as Docker and iSula, streamlining application containerization and deployment. epkg supports multi-version deployment, reducing the complexity of setting up development environments. +- **GUI-based programming**: realizes intuitive programming. +- **AI development support**: comes with pre-installed AI frameworks like TensorFlow and PyTorch, optimized for hardware accelerators such as GPUs and NPUs. It offers a complete environment for AI model development and training, along with openEuler Copilot System for AI-assisted OS troubleshooting. +- **Debugging and testing**: provides built-in debugging tools, such as GDB, CUnit, GTest, and perf, and test and tuning tools for fast debugging and automated testing. +- **Versioning and collaboration**: integrates the Git versioning tool and remote collaboration tools such as Slack, Mattermost, and GitLab, promoting team development and remote collaboration. +- **Security and compliance checks**: provides tools for security scanning and code compliance checks, enabling developers to address vulnerabilities and issues early in the development process. -- **Multiple priorities in a hybrid deployment**: Each cgroup can have a **cpu.qos_level** that ranges from -2 to 2. You can set **qos_level_weight** to assign a different priority to each cgroup and allocate CPU resources to each cgroup based on the CPU usage. This feature is also capable of wakeup preemption. +## epkg -- Programmable scheduling: The programmable scheduling framework based on eBPF allows the kernel scheduler to dynamically expand scheduling policies to meet performance requirements of different loads. +epkg is a new software package manager that supports the installation and use of non-service software packages. It solves version compatibility issues so that users can install and run software of different versions on the same OS by using simple commands to create, enable, and switch between environments. -- **NUMA-aware spinlock**: The lock transfer algorithm is optimized for the multi-NUMA system based on the MCS spinlock. The lock is preferentially transferred within the local NUMA node, greatly reducing cross-NUMA cache synchronization and ping-pong. As a result, the overall lock throughput is increased and service performance is improved. +- **Version compatibility**: enables multiple versions of the same software package to run on the same node without version conflicts. +- **Environment management**: allows users to create, enable, and switch between environments. Users can use channels of different environments to use software packages of different versions. When an environment is switched to another, the software package version is also changed. +- **Installation by common users**: allows common users to install software packages, create environments, and manage their environment images, reducing security risks associated with software package installation. -- **TCP compression**: The data of specified ports can be compressed at the TCP layer before transmission. After the data is received, it is decompressed and transferred to the user mode. TCP compression accelerates data transmission between nodes. +## openEuler GCC Toolset 14 -- **Kernel live patches**: Kernel live patches are used to fix bugs in kernel function implementation without a system restart. They dynamically replace functions when the system is running. Live patches on openEuler work by modifying instructions. They feature a high patching efficiency because they directly jump to new functions without search and transfer, while live patches on the Linux mainline version work based on ftrace. +To enable new compute features and make the most of hardware features, openEuler GCC Toolset 14 offers a minor GCC 14 toolchain that is later than the major GCC of openEuler 24.03 LTS SP1, enabling users to select the most appropriate compilation environment. By using openEuler GCC Toolset 14, users can easily switch between different GCC versions to use new hardware features. -- **Sharepool shared memory**: This technology shares data among multiple processes and allows multiple processes to access the same memory area for data sharing. +To decouple from the default major GCC and prevent conflicts between the dependency libraries of the minor and major GCC versions, the software package of the openEuler GCC Toolset 14 toolchain is named starting with gcc-toolset-14-, followed by the name of the original GCC software package. -- **Memcg asynchronous reclamation**: This optimized mechanism asynchronously reclaims memory in the memcg memory subsystem when the system load is low to avoid memory reclamation delay when the system load becomes heavy. +For version management, this solution introduces the SCL tool that provides an enable script in the /opt/openEuler/gcc-toolset-14 directory to register the environment variables of openEuler GCC Toolset 14 with SCL. Then, users can use the tool to start a new Bash shell that uses the environment variables of the minor version configured in the enable script. In this way, it is easy to switch between the major and minor GCC versions. -- **filescgroup**: The filescgroup subsystem manages the number of files (that is, the number of handles) opened by a process group. This subsystem provides easy-to-call APIs for resource management. Compared with the rlimit method, the filescgroup subsystem can better control the number of file handles for resource application and release, dynamic adjustment of resource usage, and group control. +## Kernel Innovations -- **Cgroup writeback for cgroup v1**: Cgroup writeback provides a flexible method to manage the writeback behavior of the file system cache. The main functions of cgroup writeback include cache writeback control, I/O priority control, and writeback policy adjustment. +openEuler 24.03 LTS SP1 runs on Linux kernel 6.6 and inherits the competitive advantages of community versions and innovative features released in the openEuler community. -- **Core suspension detection**: If the performance monitor unit (PMU) stops counting, the hard lockup cannot detect system suspension. The core suspension detection feature enables each CPU to check whether adjacent CPUs are suspended. This ensures that the system can perform self-healing even when some CPUs are suspended due to interruption disabling. +- **Folio-based memory management**: Folio-based Linux memory management is used instead of page. A folio consists of one or more pages and is declared in struct folio. Folio- based memory management is performed on one or more complete pages, rather than on PAGE_SIZE bytes. This alleviates compound page conversion and tail page misoperations, while decreasing the number of least recently used (LRU) linked lists and optimizing memory reclamation. It allocates more continuous memory on a per-operation basis to reduce the number of page faults and mitigate memory fragmentation. Folio-based management accelerates large I/Os and improves throughput, and large folios consisting of anonymous pages or file pages are available. For AArch64 systems, a contiguous bit (16 contiguous page table entries are cached in a single entry within a translation lookaside buffer, or TLB) is provided to reduce system TLB misses and improve system performance. In openEuler 24.03 LTS SP1, multi-size transparent hugepage (mTHP) allocation by anonymous shmem and mTHP lazyfreeing are available. The memory subsystem supports large folios, with a new sysfs control interface for allocating mTHPs by page cache and a system-level switch for feature toggling. -## Embedded +- **Multipath TCP (MPTCP)**: MPTCP is introduced to let applications use multiple network paths for parallel data transmission, compared with single-path transmission over TCP. This design improves network hardware resource utilization and intelligently allocates traffic to different transmission paths, thereby relieving network congestion and improving throughput. + MPTCP features the following performance highlights: + - Selects the optimal path after evaluating indicators such as latency and bandwidth. + - Ensures hitless network switchover and uninterrupted data transmission when switching between networks. + - Uses multiple channels where data packets are distributed to implement parallel transmission, increasing network bandwidth. -openEuler 23.09 Embedded is equipped with an embedded virtualization base that is available in the Jailhouse virtualization solution or the OpenAMP lightweight hybrid deployment solution. You can select the most appropriate solution to suite your services. This version also supports the Robot Operating System (ROS) Humble version, which integrates core software packages such as ros-core, ros-base, and simultaneous localization and mapping (SLAM) to meet the ROS 2 runtime requirements. + In the lab environment, the Rsync file transfer tool that adopts MPTCP v1 shows good transmission efficiency improvement. Specifically, a 1.3 GB file can be transferred in just 14.35s (down from 114.83s), and the average transfer speed is increased from 11.08 MB/s to 88.25 MB/s. In simulations of path failure caused by unexpected faults during transmission, MPTCP seamlessly switches data to other available channels, ensuring transmission continuity and data integrity. -- **Southbound ecosystem**: openEuler Embedded Linux supports AArch64 and x86-64 chip architectures and related hardware such as RK3568, Hi3093, RK3399, RK3588, Raspberry Pi 4B, and x86-64 industrial computers. It preliminarily supports AArch32 and RISC-V chip architectures based on QEMU simulation. + In openEuler 24.03 LTS SP1, MPTCP-related features in Linux mainline kernel 6.9 have been fully transplanted and optimized. +- **Large folio for ext4 file systems**: The IOzone performance can be improved by 80%, and the writeback process of the iomap framework supports batch block mapping. Blocks can be requested in batches in default ext4, optimizing ext4 performance in various benchmarks. For ext4 buffer I/O and page cache writeback operations, the buffer_head framework is replaced with the iomap framework that adds large folio support for ext4. In version 24.03 LTS SP1, the performance of small buffered I/Os (≤ 4 KB) is optimized when the block size is smaller than the folio size, typically seeing a 20% performance increase. +- **xcall and xint**: The Linux kernel is becoming increasingly complex, and system calls, especially the simple ones, can be a performance bottleneck. System calls on the AArch64 platform share the same exception entry point, which includes redundant processes such as security checks. Common ways to reduce system call overhead include service offloading and batch processing, but both require service adaptation. xcall provides a solution that does not require service code modification. It streamlines system calls by optimizing their processing logic, trading off some maintenance and security capabilities to reduce overhead. -- **Embedded elastic virtualization base**: The converged elastic base of openEuler Embedded is a collection of technologies used to enable multiple OSs or runtimes to run on a system-on-a-chip (SoC). These technologies include bare metal, embedded virtualization, lightweight containers, LibOS, trusted execution environment (TEE), and heterogeneous deployment. + To unify interrupt handling, the kernel integrates all interrupt handling processes into its general interrupt handling framework. As the kernel evolves, the general interrupt handling framework has gradually accumulated many security hardening and maintenance features that are not closely related to interrupt handling, increasing latency unpredictability. xint simplifies interrupt handling to reduce the latency and system overhead. +- **CacheFiles failover**: In on-demand mode of CacheFiles, if the daemon breaks down or is killed, subsequent read and mount requests return **-EIO**. The mount points can be used only after the daemon is restarted and the mount operations are performed again. For public cloud services, such I/O errors will be passed to cloud service users, which may impact job execution and endanger the overall system stability. The CacheFiles failover feature renders it unnecessary to remount the mount points upon daemon crashes. It requires only the daemon to restart, ensuring that these events are invisible to users. +- **Programmable scheduling**: Interfaces of programmable scheduling are available in the user space as an extension of the Completely Fair Scheduler (CFS) algorithm. Users can customize scheduling policies based on service scenarios to bypass the original CFS. This new feature allows programmable core selection, load balancing, task selection, and task preemption, and supports labelling and kfuncs. +- **SMC-D with loopback-ism**: Shared memory communication over DMA (SMC-D) is a kernel network protocol stack that is compatible with socket interfaces and accelerates TCP communication transparently over shared memory. Initially, SMC-D was exclusive to IBM S/390 architecture systems. The introduction of the loopback-ism virtual device enabled broader adoption by simulating Internal Shared Memory (ISM). This innovation transformed SMC-D into a universal kernel mechanism compatible with non-S/390 architectures as well. SMC-D with loopback-ism applies to scenarios where TCP is used for inter-process or inter- container communication in the OS. It accelerates communication by bypassing the kernel TCP/IP stack. Together with smc-tools, users can preload dynamic libraries through **LD_PRELOAD** to replace the TCP stack without adapting applications. Feedback from the community shows that, compared with the native TCP, SMC-D with loopback-ism can improve the network throughput by more than 40%. +- **IMA RoT framework**: The Linux Integrity Measurement Architecture (IMA) subsystem mainly uses the trusted platform module (TPM) as the root of trust (RoT) device to provide integrity proof for the measurement list, and code of the subsystem is tightly coupled with TPM operations. However, workloads like confidential computing require IMA to use new RoT devices, such as virtCCA supported by openEuler. This IMA RoT framework implements an abstraction layer between the IMA subsystem and RoT devices. It simplifies the adaptation of RoT devices to the IMA subsystem and facilitates the configuration and operation of various RoT devices by users and the IMA subsystem. +- **Protection against script viruses**: Typical ransomware variants are script files (such as JSP files), which can run through the interpreter and bypass security technologies in the kernel to launch attacks. However, the IMA technology used by the kernel to defend against intrusion mainly targets virus files of the Executable and Linkable Format (ELF). This new feature enables IMA to check script files that are indirectly executed in the system. It uses the **execveat()** system call with newly added flags to check the execution permission of scripts, and call the IMA interface during the check to measure script integrity. Test and verification results demonstrate how the script interpreter proactively invokes **execveat()** and transfers the **AT_CHECK** flag to check the execute permission (including the IMA check) of scripts, which are executed only after they pass the permission check. +- **Halt polling**: This feature enables guest vCPUs to poll at the idle state to avoid sending an inter-processor interrupt (IPI) when performing a wakeup. This optimization reduces the interrupt sending and handling overhead. In addition, the VM does not exit during polling, further reducing the VM entry/exit overhead. This feature also reduces the inter-process communication latency and improves VM performance. +- **CAQM for the kernel TCP/IP stack**: Constrained active queue management (CAQM) is an algorithm for network congestion control. It is mainly used on compute nodes in data centers that use TCP to transmit data and network switch nodes on transmission paths. The network switch nodes calculate idle bandwidth and optimal bandwidth allocation, and the compute node protocol stack works with the switches to achieve "zero-queue" congestion control effect and ultra-low transmission latency in high-concurrency scenarios. -- **Mixed criticality deployment framework**: The mixed-criticality (MICA) deployment framework is built on the converged elastic base. The unified framework masks the differences between the technologies used in the underlying converged elastic base, enabling Linux to be deployed together with other OSs. + The CAQM algorithm adds congestion control flags to the Ethernet link layer to dynamically adjust the queue length and improve the utilization of network resources. In latency-sensitive general-computing scenarios in data centers, this feature greatly reduces latency and packet loss, improving user experience. -- **Northbound ecosystem**: More than 350 common embedded software packages can be built using openEuler. The ROS 2 Humble version is supported, which contains core software packages such as ros-core, ros-base, and SLAM. The ROS SDK is provided to simplify embedded ROS development. The soft real-time capability based on Linux kernel 5.10 allows for response to soft real-time interrupts within microseconds. DSoftBus and HiChain for point-to-point authentication of OpenHarmony have been integrated to implement interconnection between openEuler-based embedded devices and between openEuler-based embedded devices and OpenHarmony-based devices. + In typical data center scenarios, this algorithm outperforms the classic Cubic algorithm in two key metrics: (1) 92.4% lower transmission latency, and (2) 99.97% TCP transmission bandwidth utilization with 90% lower switch queue buffer usage. -- **UniProton**: This hard RTOS features ultra-low latency and flexible MICA deployments. It is suited for industrial control because it supports both microcontroller units and multi-core CPUs. + The CAQM algorithm requires collaboration between compute node servers and switches. Therefore, intermediate switches must support the CAQM protocol (for protocol header identification, congestion control field adjustment, and so on). This algorithm is controlled through the kernel compilation macro (**CONFIG_ETH_CAQM**) and is disabled by default. To use this algorithm, users need to enable this macro and recompile the kernel. + +## NestOS + +NestOS is a community cloud OS that uses nestos-assembler for quick integration and build. It runs rpm-ostree and Ignition tools over a dual rootfs and atomic update design, and enables easy cluster setup in large-scale containerized environments. Compatible with Kubernetes and OpenStack, NestOS also reduces container overheads. + +- **Out-of-the-box availability**: integrates popular container engines such as iSulad, Docker, and Podman to provide lightweight and tailored OSs for the cloud. +- **Easy configuration**: uses the Ignition utility to install and configure a large number of cluster nodes with a single configuration. +- **Secure management**: runs rpm-ostree to manage software packages and works with the openEuler software package source to ensure secure and stable atomic updates. +- **Hitless node updating**: uses Zincati to provide automatic node updates and reboot without interrupting services. +- **Dual rootfs**: executes dual rootfs for active/standby switchovers, to ensure integrity and security during system running. ## SysCare -SysCare is a system-level hotfix software that provides security patches and hot fixing for OSs. It can fix system errors without restarting hosts. By combining kernel-mode and user-mode hot patching, SysCare takes over system repair, allowing users to focus on core services. In the future, OS hot upgrade will be provided to further free O&M users and improve O&M efficiency. +SysCare is a system-level hotfix software that provides security patches and hot fixing for OSs. It can fix system errors without restarting hosts. SysCare combines kernel-mode and user-mode hot patching to take over system repair, freeing up valuable time for users to focus on core services. Looking ahead, SysCare will introduce live OS update capabilities, further improving O&M efficiency. -**Patches built in containers**: +- **Hot patch creation:** Hot patch RPM packages can be generated by providing the paths of the source RPM package, debuginfo RPM package, and patches to be applied, eliminating the need to modify the software source code. +- **Patch lifecycle management**: SysCare simplifies patch lifecycle management, offering a complete and user-friendly solution. With a single command, users can efficiently manage hot patches, saving time and effort. SysCare leverages the RPM system to build hot patches with complete dependencies. This allows for easy integration into the software repository and simplifies distribution, installation, update, and uninstallation of hot patches. +- **Kernel-mode and user-mode hot patch integration**: By utilizing the upatch and kpatch technologies, SysCare delivers seamless hot fixing for the entire software stack from applications and dynamic libraries to the kernel, eliminating the need for disruptive downtime. +- **New features**: SysCare preserves and restores accepted hot patches after reboot, maintaining their original activation order. -- eBPF is used to monitor the compiler process. In this way, hot patch change information can be obtained in pure user mode without creating character devices, and users can compile hot patches in multiple containers concurrently. +## NRI Plugin Support of iSula -- Users can install different RPM packages (syscare-build-kmod or syscare-build-ebpf) to use ko or eBPF. The syscare-build process automatically adapts to the corresponding underlying implementation. - -## GCC for openEuler +Node Resource Interface (NRI) is a public interface for controlling node resources and provides a generic framework for pluggable extensions for CRI-compatible container runtimes. It offers a fundamental mechanism for extensions to track container states and make limited modifications to their configurations. This allows users to insert custom logic into OCI-compatible runtimes, enabling controlled changes to containers or performing additional operations outside the scope of OCI at certain points in the container lifecycle. The newly added support for NRI plugins of iSulad reduces costs for container resource management and maintenance, eliminates scheduling delays, and ensures information consistency in Kubernetes environments. + +An NRI plugin establishes a connection with iSulad through the NRI runtime service started within the isula-rust-extension component. This connection enables the plugin to subscribe to both pod and container lifecycle events. + +- For pods, subscribable events include creation, stopping, and removal. +- For containers, subscribable events include creation, post-creation, starting, post-start, updating, post-update, stopping, and removal. + +Upon receiving a CRI request from Kubernetes, iSulad relays this request to all NRI plugins subscribed to the corresponding lifecycle events. The request received by an NRI plugin includes metadata and resource information for the relevant pod or container. The plugin can then adjust the resource configuration of the pod or container as required. Finally, the NRI plugin communicates the updated configuration to iSulad, which in turn relays it to the container runtime, making the updated configuration effective. + +## oeAware with Enhanced Information Collection and Tuning Plugins + +oeAware is a framework that provides low-load collection, sensing, and tuning upon detecting defined system behaviors on openEuler. The framework divides the tuning process into three layers: collection, sensing, and tuning. Each layer is associated through subscription and developed as plugins, overcoming the limitations of traditional tuning techniques that run independently and are statically enabled or disabled. + +Every oeAware plugin is a dynamic library that utilizes oeAware interfaces. The plugins comprise multiple instances that each contains several topics and deliver collection or sensing results to other plugins or external applications for tuning and analysis purposes. The framework consists of the following components: + +- The SDK enables subscription to plugin topics, with a callback function handling data from oeAware. This allows external applications to create tailored functionalities, such as cross- cluster information collection or local node analysis. +- The Performance monitoring unit (PMU) information collection plugin gathers performance records from the system PMU. +- The Docker information collection plugin retrieves specific parameter details about the Docker environment. +- The system information collection plugin captures kernel parameters, thread details, and resource information (CPU, memory, I/O, network) from the current environment. +- The thread sensing plugin monitors key information about threads. +- The evaluation plugin examines system NUMA and network information during service operations, suggesting optimal tuning methods. +- The system tuning plugins comprise stealtask for enhanced CPU tuning, smc_tune which leverages shared memory communication in the kernel space to boost network throughput and reduce latency, and xcall_tune which bypasses non-essential code paths to minimize system call processing overhead. +- The Docker tuning plugin addresses CPU performance issues during sudden load spikes by utilizing the CPU burst feature. + +## KubeOS + +KubeOS is a lightweight and secure OS designed for cloud-native environments. It simplifies O&M by providing unified tools for Kubernetes-based systems. KubeOS is specifically designed for running containers. It features a read-only root directory, includes only the essential components for the container runtime, and utilizes dm-verity security hardening. This minimizes vulnerabilities and attack surfaces while improving resource utilization and boot speed. KubeOS can leverage native Kubernetes declarative APIs to unify the upgrade, configuration, and maintenance of worker node OSs within a cluster. This approach simplifies cloud-native operations, addresses challenges associated with OS version fragmentation across cluster nodes, and provides a unified solution for OS management. + +KubeOS introduces enhanced configuration capabilities, image customization features, and root file system (rootfs) integrity protection using dm-verity, as shown in the figure. Through a centralized management platform, KubeOS enables unified configuration of cluster parameters in the **limits.conf** file and cluster components like containerd and kubelet. -GCC for openEuler is developed based on the open source GCC 12.3 and supports features such as automatic feedback-directed optimization (FDO), software and hardware collaboration, memory optimization, SVE, and vectorized math libraries. +KubeOS provides comprehensive system image customization options, allowing users to configure systemd services, GRUB passwords, system drive partitions, users and user groups, files, scripts, and **persist** partition directories. -- The default language is upgraded from C14/C++14 to C17/C++17, and more hardware architecture features such as the Armv9-A architecture and x86 AVX512-FP16 are supported. +For static integrity protection, KubeOS activates dm-verity during VM image creation to ensure rootfs integrity. It also supports upgrades and configuration when dm-verity is enabled. -- GCC for openEuler supports structure optimization and instruction selection optimization, fully utilizing the hardware features of the Arm architecture to achieve higher operating efficiency. In the benchmark tests such as SPEC CPU 2017, GCC for openEuler delivers much better performance than GCC 10.3 of the upstream community. +## IMA -- GCC for openEuler also supports automatic FDO to greatly improve the performance of the MySQL database at the application layer. +The IMA is an open source Linux technology widely used for file integrity protection in real-world applications. It performs integrity checks on system programs to prevent tampering and ensure only authenticated (signed or HMAC-verified) files are executed via an allowlist. -## A-Ops +Applications running on Linux can be categorized into two types: -The amount of data generated by IT infrastructure and applications sees a 2- to 3-fold increase every year. The application of big data and machine learning technologies is maturing, driving the generation of efficient and intelligent O&M systems to help enterprises reduce costs and improve efficiency. A-Ops is an intelligent O&M framework that supports basic capabilities such as CVE management, exception detection (database scenario), and quick troubleshooting to reduce O&M costs. +- **Binary executables**: Programs in the ELF format can be directly executed by **exec** or **mmap** system calls. Through hook functions in **exec** and **mmap** system calls, the IMA triggers measurement or verification processes, ensuring integrity protection. +- **Interpreter-based applications**: Programs indirectly executed via interpreters, such as scripts run by Bash, Python, and Lua interpreters and Java programs executed by the JVM. -- **Intelligent patch management**: Supports the patch service, kernel hot fix, intelligent patch inspection, and Hybrid management of cold and hot patches. +The current IMA fails to protect interpreter-based applications, as these applications are typically loaded and parsed by interpreters through **read** system calls. The IMA cannot differentiate them from other mutable files, such as configuration or temporary files. As a result, enabling the IMA for **read** system calls inadvertently includes these mutable files in the protection scope. Mutable files lack pre-generated measurement baselines or verification credentials, causing integrity check failures. -- **Exception detection**: Detects network I/O delays, packet loss, interruption, and high disk I/O loads in MySQL and openGauss service scenarios. +To address this limitation, openEuler enhances the IMA feature to significantly improve integrity protection for interpreter-based applications. -- **Configuration source tracing**: Provides cluster configuration collection and baseline capabilities to implement manageable and controllable configurations. The configuration of the entire cluster is checked and compared with the baseline in real time to quickly identify unauthorized configuration changes and locate faults. +## Heterogeneous RoT Support -## A-Ops gala +Common attack methods often target the authenticity and integrity of information systems. Hardware RoTs have become a standard method for protecting critical system components, enabling the system to measure and verify integrity. When tampering or counterfeiting is detected, the system triggers alerts or blocks the activity. -The gala project will fully support fault diagnosis in Kubernetes scenarios, including application drill-down analysis, observable microservice and DB performance, cloud-native network monitoring, cloud-native performance profiling, process performance diagnosis, and minute-level diagnosis of five types of OS issues (network, drive, process, memory, and scheduling). +The prevailing protection approach uses the trusted platform module (TPM) as the RoT, combined with the integrity measurement software stack to establish a system trust chain that ensures system authenticity and integrity. openEuler supports integrity measurement features such as measured boot, IMA measurement for files, and dynamic integrity measurement (DIM) for memory. -- **Easy deployment of the Kubernetes environment**: gala-gopher can be deployed as a DaemonSet, and a gala-gopher instance is deployed on each worker node. gala-spider and gala-anteater are deployed as containers on the Kubernetes management node. +The RoT framework used by openEuler 24.03 LTS SP1 offers a unified measurement interface to the upper-layer integrity protection software stack. Deployed within the kernel integrity subsystem, the framework supports multiple RoT drivers and expands integrity measurement beyond the TPM to include heterogeneous RoTs. -- **Application drill-down analysis**: Diagnoses subhealth problems in cloud native scenarios and demarcates problems between applications and the cloud platform in minutes. +## Enhanced secGear -- **Full-stack monitoring**: Provides application-oriented refined monitoring for cross-software stacks, including the language runtime (JVM), glibc, system call, and kernel (TCP, I/O, and scheduling), and allows users to view the impact of system resources on applications in real time. +The unified remote attestation framework of secGear addresses the key components related to remote attestation in confidential computing, abstracting away the differences between different Trusted Execution Environments (TEEs). It provides two components: attestation agent and attestation service. The agent is integrated by users to obtain attestation reports and connect to the attestation service. The service can be deployed independently and supports the verification of iTrustee and virtCCA remote attestation reports. -- **Full-link monitoring**: Provides network traffic topology (TCP and RPC) and software deployment topology information, and builds a system 3D topology based on the information to accurately show the resource scope on which applications depend and quickly identify the fault radius. +The unified remote attestation framework focuses on confidential computing functionalities, while service deployment and operation capabilities are provided by third-party deployment services. The key features of the unified remote attestation framework are as follows: -- **Causal AI model**: Provides visualized root cause derivation to demarcate faults to resource nodes in minutes. +- **Report verification plugin framework**: Supports runtime compatibility with attestation report verification for different TEE platforms, such as iTrustee, virtCCA, and CCA. It also supports the extension of new TEE report verification plugins. +- **Certificate baseline management**: Supports the management of baseline values of Trusted Computing Bases (TCB) and Trusted Applications (TA) as well as public key certificates for different TEE types. Centralized deployment on the server ensures transparency for users. +- **Policy management**: Provides default policies for ease of use and customizable policies for flexibility. +- **Identity token**: Issues identity tokens for different TEEs, endorsed by a third party for mutual authentication between different TEE types. +- **Attestation agent**: Supports connection to attestation services/peer-to-peer attestation, compatible with TEE report retrieval and identity token verification. It is easy to integrate, allowing users to focus on their service logic. -- **Observable microservice and DB performance**: Provides non-intrusive microservice, DB, and HTTP1.*x* access performance observation, including the throughput, latency, and error rate; and supports refined API observation and HTTP TRACE to view abnormal HTTP requests. +Two modes are supported depending on the usage scenario: peer-to-peer verification and attestation service verification. -- **Observable PostgreSQL access performance**: Observes performance in terms of the throughput, latency, and error rate; and supports refined SQL access observation and slow SQL trace to show SQL statements of slow SQL queries. +Attestation service verification process: -- **Cloud-native application performance profiling**: Provides a non-intrusive and zero-modification cross-stack profiling analysis tool and can connect to the common UI front end of Pyroscope. +1. The user (regular node or TEE) initiates a challenge to the TEE platform. +2. The TEE platform obtains the TEE attestation report through the attestation agent and returns it to the user. +3. The user-side attestation agent forwards the report to the remote attestation service. +4. The remote attestation service verifies the report and returns an identity token in a unified format endorsed by a third party. +5. The attestation agent verifies the identity token and parses the attestation report verification result. -- **Cloud-native network monitoring**: Provides TCP, socket, and DNS monitoring for Kubernetes scenarios for more refined network monitoring. +Peer-to-peer verification process (without the attestation service): -- **Process performance diagnosis**: Provides process-level performance problem diagnosis for middleware (such as MySQL and Redis) in cloud native scenarios, monitors process performance KPIs and process-related system-layer metrics (such as I/O, memory, and TCP), and detects process performance KPI exceptions and system-layer metrics that affect the KPIs. +1. The user initiates a challenge to the TEE platform, which then returns the attestation report to the user. +2. The user uses a local peer-to-peer TEE verification plugin to verify the report. -## sysMaster +## gala-anteater for Minute-Level Container Interference Detection -sysMaster is a collection of ultra-lightweight and highly reliable service management programs. It provides an innovative implementation of PID 1 to replace the conventional init process. Written in Rust, sysMaster is equipped with fault monitoring, second-level self-recovery, and quick startup capabilities, which help improve OS reliability and service availability. In version 0.5.0, sysMaster can manage system services in container and VM scenarios. +gala-anteater is an AI-powered exception detection platform for gray faults in the OS. Integrating various exception detection algorithms, it achieves system-level fault detection and reporting through automated model pre-training, online incremental learning, and model updates. -- Added the devMaster component to manage device hot swap. +In high-density online container deployment scenarios, the presence of disorderly resource contention can lead to inter-container interference. gala-anteater enables minute-level identification of interference sources (CPU or I/O), aiding O&M personnel in swiftly tracing and resolving issues to ensure service QoS. -- Added the live update and hot reboot functions to sysMaster. +gala-anteater leverages a combination of offline and online learning techniques to facilitate offline model training and online updates, ultimately enabling real-time online exception detection. -- Allows PID 1 to run on a VM. +- **Offline**: Initially, historical KPI datasets undergo preprocessing and feature selection to generate a training set. This set is then used to train and optimize an unsupervised neural network model (such as a variational autoencoder). Finally, a manually labeled test set aids in selecting the optimal model. + +- **Online**: The trained model is deployed online, where it undergoes further training and parameter tuning using real-time data. This continuously refined model then performs real-time exception detection within the online environment. + +## A-Ops Integration with authHub for Unified User Authentication + +authHub is a unified user authentication platform built on the OAuth 2.0 protocol. A-Ops can now utilize authHub to manage application registration, enabling seamless user authentication across multiple platforms. + +Core functionalities of authHub include: + +- **Application management**: Deployed applications can be registered with and configured on authHub to provide access to their features. + +- **User authentication**: Applications managed by authHub support single sign-on (SSO) and single sign-out processes. + +## Minute-level Demarcation and Location of Microservice Performance Problems (TCP, I/Os, and Scheduling) + +gala-gopher, gala-spider, and gala-anteater implement a topological root cause analysis approach to facilitate fault detection and root cause location in large-scale clusters. In openEuler 24.03 LTS SP1, fine-grained capabilities have been introduced for cloud-native environments based on Layer 7 protocols like HTTPS, PostgreSQL, and MySQL, enabling O&M teams to quickly locate the source of faults, thus enhancing system stability and reliability. The following features are available: + +- **Metric collection**: gala-gopher uses eBPF to collect and report network and I/O metrics. +- **Cluster topology**: gala-spider receives data reported by gala-gopher and constructs a container- and process-level call relationship topology. +- **Fault detection**: gala-anteater classifies the reported metrics based on the fault detection model to determine whether an exception has occurred in the system. +- **Root cause location**: gala-anteater locates the root cause node of the exception based on node exception and topology information. ## utsudo -utsudo uses Rust to reconstruct sudo to deliver a more efficient, secure, and flexible privilege escalation tool. The involved modules include the common utility, overall framework, and function plugins. +**sudo** is one of the commonly used utilities for Unix-like and Linux OSs. It enables users to run specific commands with the privileges of the super user. utsudo is developed to address issues of security and reliability common in **sudo**. + +utsudo uses Rust to reconstruct **sudo** to deliver more efficient, secure, and flexible privilege escalation. It includes modules such as the common utility, overall framework, and function plugins. -- **Access control**: Restricts the commands that can be executed by users as required, and specifies the required authentication method. +**Basic features** -- **Audit log**: Records and traces the commands and tasks executed by each user using utsudo. +- **Access control**: Limits the commands that can be executed by users, and specifies the required authentication method. +- **Audit log**: Records and traces all commands and tasks executed by each user. +- **Temporary privilege escalation**: Allows common users to temporarily escalate to a super user for executing privileged commands or tasks. +- **Flexible configuration**: Allows users to set arguments such as command aliases, environment variables, and execution parameters to meet system requirements. -- **Temporary privilege escalation**: allows common users to enter their passwords to temporarily escalate to the super user to execute specific commands or tasks. +**Enhanced features** -- **Flexible configuration**: Allows users to set arguments such as command aliases, environment variables, and execution parameters to meet complex system management requirements. +utsudo 0.0.2 comes with openEuler 24.03 LTS SP1 to provide the following features: + +- **Privilege escalation**: Escalates the privilege of a process run by a common user to the root privilege. +- **Plugin loading**: Parses plugin configuration files and dynamically loads plugin libraries. ## utshell -utshell is a new shell that inherits the usage habits of Bash. It can interact with users through command lines, specifically, responding to user operations to execute commands and provide feedback. In addition, it can execute automated scripts to facilitate O&M. +utshell is a new shell that introduces new features and inherits the usability of Bash. It enables interaction through command lines, such as responding to user operations to execute commands and providing feedback, and can execute automated scripts to facilitate O&M. -- **Command execution**: Runs commands deployed on the user's machine and sends return values to the user. +**Basic features** +- **Command execution**: Runs and sends return values from commands executed on user machines. - **Batch processing**: Automates task execution using scripts. +- **Job control**: Executes, manages, and controls multiple user commands as background jobs concurrently. +- **Historical records**: Records the commands entered by users. +- **Command aliases**: Allows users to create aliases for commands to customize their operations. -- **Job control**: Concurrently executes multiple user commands as background jobs, and manages and controls the tasks that are executed concurrently. +**Enhanced features** -- **Historical records**: Records the commands entered by users. +utshell 0.5 runs on openEuler 24.03 LTS SP1 to perform the following operations: -- **Command aliases**: Allows users to create aliases for commands to customize their operations. +- Parses shell scripts. +- Runs third-party commands. + +## GCC for openEuler + +The baseline version of GCC for openEuler has been upgraded from open source GCC 10.3 to GCC 12.3, supporting features such as automatic feedback-directed optimization (AutoFDO), software and hardware collaboration, memory optimization, Scalable Vector Extension (SVE), and vectorized math libraries. + +- The default language of GCC for openEuler has been upgraded from C14/C++14 to C17/C++17, enabling GCC for openEuler to support more hardware features like Armv9-A and x86 AVX512-FP16. +- GCC for openEuler supports structure optimization and instruction selection optimization, leveraging Arm hardware features to improve system running efficiency. In the benchmark tests such as SPEC CPU 2017, GCC for openEuler has proven to deliver higher performance than GCC 10.3 of the upstream community. +- Further, it fuels AutoFDO to improve the performance of MySQL databases at the application layer. + +**Feature Description** + +- **SVE vectorization**: Significantly improves program running performance for Arm-based machines that support SVE instructions. +- **Memory layout**: Rearranges the structure members so that frequently accessed members are placed in continuous memory locations, boosting the cache hit ratio and enhancing program performance. +- **SLP transpose optimization**: Improves the analysis of loops with consecutive memory reads during loop splitting, and adds analysis to transpose grouped stores in the superword level parallelism (SLP) vectorization stage. +- **Redundant member elimination**: Eliminates structure members that are never read and deletes redundant write statements, which in turn reduces the memory footprint of the structure and alleviates subsequent bandwidth pressure, while improving performance. +- **Array comparison**: Implements parallel comparison of array elements to improve execution efficiency. +- **Arm instruction optimization**: Simplifies the pipeline of ccmp instructions for a wide range of deployments. +- **IF statement optimization**: Splits and optimizes the IF statement block to improve constant propagation within a program. +- **SLP vectorization**: Enhances SLP to cover more vectorization scenarios and improve performance. +- **AutoFDO**: Uses perf to collect and parse program information and optimizes feedback across the compilation and binary phases, boosting mainstream applications such as MySQL databases. + +## Gazelle + +Gazelle is a high-performance user-mode protocol stack. It directly reads and writes NIC packets in user mode based on the Data Plane Development Kit (DPDK), transmits the packets through shared hugepage memory, and uses the LwIP protocol stack, thereby greatly improving the network I/O throughput of applications and accelerating the network for databases. With Gazelle, high performance and universality can be achieved at the same time. In openEuler 24.03 LTS SP1, Gazelle introduces eXpress Data Path (XDP) deployment mode for container scenarios and TPC-C support for the openGauss database, further enhancing the user-mode protocol stack. + +- **High performance (ultra-lightweight)**: High-performance lightweight protocol stack capabilities are implemented based on DPDK and LwIP. +- **Ultimate performance**: A highly linearizable concurrent protocol stack is implemented based on technologies such as regional hugepage splitting, dynamic core binding, and full-path zero-copy. +- **Hardware acceleration**: TCP Segmentation Offload (TSO), checksum (CSUM) offload, Generic Receive Offload (GRO), and other offload technologies streamline the vertical acceleration of hardware and software. +- **Universality (POSIX compatibility)**: Full compatibility with POSIX APIs eliminates the need to modify applications. The recvfrom and sendto interfaces of UDP are supported. +- **General networking model**: Adaptive scheduling of the networking model is implemented based on mechanisms such as fd router and wake-up proxy. The UDP multi-node multicast model meets the requirements of any network application scenario. +- **Usability (plug-and-play)**: LD_PRELOAD enables zero-cost deployment by removing the requirement for service adaptation. +- **Easy O&M (O&M tool)**: Complete O&M methods, such as traffic statistics, metric logs, and CLI commands, are provided. + +**New Features** + +- Gazelle now supports XDP deployment on L2-mode NICs using IPVLAN. +- Interrupt mode is introduced, which reduces LStack CPU usage in no-traffic or low-traffic scenarios. +- Networks of the ping-pong scheme are optimized to improve packet transmission during ping-pong operations. +- Support for single-node and single-active/standby TPC-C testing is added for the openGauss database. + +## virtCCA + +virtCCA-based confidential VMs, built on the Secure EL2 (S-EL2) of Kunpeng 920, allow regular VM software stacks to run securely within TEEs. + +Based on the standard interface of the Arm Confidential Compute Architecture (CCA), openEuler builds a TEE virtualization management module upon the TrustZone firmware. This module supports memory isolation between confidential VMs, context and lifecycle management, and page table management, and enables seamless application migration to TEEs. + +- Device passthrough + +Device passthrough uses the PCIe protection controller (PCIPC) embedded in the PCIe root complex of Kunpeng 920. A selector is added to the PCIe bus to regulate communication between the processor and peripherals. Operating through the system memory management unit (SMMU), this selector controls both inbound and outbound traffic, safeguarding the entire data link. +Device passthrough boasts excellent security isolation and performance enhancements for PCIe devices: + +- Security isolation + + The TEE manages device access permissions, preventing host software from accessing TEE devices. + +- High performance + + Confidential device passthrough negates performance loss on the data plane compared to traditional encryption and decryption solutions. + +- Ease of use + + Compatibility with existing open source OSs ensures that the kernel driver does not need to be modified. + +- ShangMi (SM) algorithm acceleration + + Hardware-based acceleration for SM algorithms runs on the UADK user-mode accelerator framework to enhance SM algorithm performance and enable algorithm offloading within confidential VMs. It is powered by the Kunpeng processor and utilizes the Kunpeng Accelerator Engine (KAE) features in the TEE. + +## Hygon CSV3 -## migration-tools +CSV3 marks the third generation of Hygon's secure virtualization technology. With security features that far exceed its predecessors, CSV3 realizes multi-level data protection that implements VM data isolation within the CPU. This isolation prevents the host OS from accessing VM memory or modifying nested page tables, thereby ensuring strong data integrity and confidentiality. -Developed by UnionTech Software Technology Co., Ltd., migration-tools is oriented to users who want to quickly, smoothly, stably, and securely migrate services to the openEuler OS. migration-tools consists of the following modules: +**Secure memory isolation unit** -- **Server module**: It is developed on the Python Flask Web framework. As the core of migration-tools, it receives task requests, processes execution instructions, and distributes the instructions to each Agent. +The secure memory isolation unit forms the hardware basis for VM data integrity. A specialized hardware component within the CPU, the unit is positioned on the system bus path between the CPU and the memory controller to retrieve secure memory information for CSV3 VMs, such as physical memory addresses, VM IDs, and associated permissions. It validates all memory access requests from the CPU before granting permission, ensuring only verified requests receive access clearance. -- **Agent module**: It is installed in the OS to be migrated to receive task requests from the Server module and perform migration. +When a CSV3 VM reads from or writes to memory, the page table translation unit converts the guest physical address (GPA) into a host physical address (HPA), and sends a memory access request (including read/write commands), the HPA, and the requester VM ID to the address bus. -- **Configuration module**: It reads configuration files for the Server and Agent modules. +If the secure memory isolation unit detects an incorrect VM ID during memory reads, it returns data in a fixed pattern. During memory writes, such an invalid request is discarded. -- **Log module**: It records logs during migration. +**Secure processor** -- **Migration evaluation module**: It provides evaluation reports such as basic environment check, software package comparison analysis, and ABI compatibility check before migration, providing a basis for users' migration work. +The secure memory isolation unit is the central hardware component in CSV3, safeguarding the integrity of VM memory. Its configuration must remain entirely secure and immune to modification by the host OS. -- **Migration function module**: It provides migration with a few clicks, displays the migration progress, and checks the migration result. +Hygon's secure processor is built into the SoC and independent from the CPU. It acts as the RoT of the SoC. Upon power-on, the processor verifies firmware integrity using an embedded signature verification key before the firmware is loaded and executed. Equipped with dedicated hardware resources within an isolated environment, the secure processor represents the highest level of security within the SoC, governing the overall security of the SoC. It maintains exclusive control over the secure memory isolation unit throughout the VM lifecycle, thus preventing the host OS from accessing or modifying unit configurations. -## DDE +During VM startup, the host OS sends a request to the secure processor, which initializes the secure memory isolation unit. The secure processor firmware updates the secure memory isolation unit, and when the VM terminates, the secure processor clears all configurations of the unit. The secure processor validates all configuration requests from the host and rejects any that are invalid. -DDE focuses on delivering polished user interaction and visual design. DDE is powered by independently developed core technologies for desktop environments and provides login, screen locking, desktop and file manager, launcher, dock, window manager, control center, and more functions. As one of the preferred desktop environments, DDE features a user-friendly interface, elegant interaction, high reliability, and privacy protection. You can use DDE to work more creatively and efficiently or enjoy media entertainment while keeping in touch with friends. +**Protection of the virtual machine control block (VMCB)** -## Kmesh +The VMCB contains crucial control data, including the VM ID, VM page table base address, and VM register page base address. Access to this sensitive data by a host OS can cause tampering to VM memory data. -Based on the programmable kernel, Kmesh offloads service governance to the OS, thus shortening the inter-service communication latency to only 1/5 of the industry average. +The secure processor creates the VMCB and places it under the protection scope of the secure memory isolation unit, which prevents the host OS from altering the contents of the VMCB. -- Kmesh can connect to a mesh control plane (such as Istio) that complies with the Dynamic Resource Discovery (xDS) protocol. +To improve compatibility with the host OS software, CSV3 uses real and shadow VMCB pages. The host OS constructs the shadow VMCB page, populates it with control data, and sends it to the secure processor. The secure processor then generates the real VMCB page, copying non- critical control data from the shadow VMCB page and adding essential control information independently. The VM launches and operates using the real VMCB page, thus blocking any host OS attempts to hack the VMCB. -- **Traffic orchestration**: Polling and other load balancing policies, L4 and L7 routing support, and backend service policies available in percentage mode are supported. +## openHiTLS -- **Sockmap for mesh acceleration**: Take the typical service mesh scenario as an example. When a sockmap is used, the eBPF program takes over the communication between service containers and Envoy containers. As a result, the communication path is shortened to achieve mesh acceleration. The eBPF program can also accelerate the communication between pods on the same node. +openHiTLS provides a lightweight, customizable cryptographic solution designed for diverse industries including cloud computing, big data, AI, and finance. The platform combines advanced algorithms with exceptional performance while maintaining robust security and reliability. Its flexible architecture ensures seamless compatibility across multiple programming languages. Through community collaboration and ecosystem development, openHiTLS accelerates the adoption of cryptographic security standards across various industries, while fostering a security- focused open source ecosystem centered around openEuler in order to provide users with a safer and more reliable digital environment. -## RISC-V QEMU +**Support for mainstream cryptographic protocols and algorithms** -openEuler 23.09 is released with support for the RISC-V architecture. openEuler 23.09 aims to provide basic support for upper-layer applications and is highly customizable, flexible, and secure. It provides a stable and reliable operating environment for the computing platform of the RISC-V architecture, facilitating the installation and verification of upper-layer applications and promoting the enrichment and quality improvement of the software ecosystem in the RISC-V architecture. +Mainstream international and Chinese cryptographic algorithms and protocols are supported. You can select appropriate cryptographic algorithms and protocols based on scenario requirements. -- The OS kernel is updated to version 6.4.0, which is consistent with mainstream architectures. +- Chinese cryptographic algorithms: SM2, SM3, and SM4 +- International mainstream algorithms: AES, RSA, (EC)DSA, (EC)DH, SHA-3 and HMAC +- GB/T 38636-2020 TLCP, that is, the dual-certificate protocol +- TLS 1.2, TLS 1.3, and DTLS 1.2 -- It features a stable base, including core functions such as processor management, memory management, task scheduling, and device drivers, as well as common utilities. +**Open architecture for cryptography applications in all scenarios** -## DIM +By leveraging an open architecture and implementing technological innovations in applications across the entire industry chain, a one-stop, full-scenario solution is provided for diverse industries. -Dynamic Integrity Measurement (DIM) measures key data (such as code segments) in memory during program running and compares the measurement result with the reference value to determine whether the data in memory has been tampered with. In this way, attacks can be detected and countermeasures can be taken. +- Flexible southbound and northbound interfaces: Unified northbound interfaces enable quick access for industry applications. Southbound devices widely deployed in various service systems are abstracted to streamline device utilization. +- Multi-language compatibility: The foreign function interfaces (FFIs) ensure multi-language compatibility, enabling one cipher suite to be used across various programming languages. +- Wide applicability: Cryptographic technology applied across various scenarios in the entire industry chain ensures high performance, security, and reliability of openHiTLS in different scenarios. -- Measures user-mode processes, kernel modules, and code segment in the kernel memory. +**Hierarchical decoupling and on-demand tailoring to create a lightweight cipher suite** -- Extends measurements to the PCR register of the TPM 2.0 chip for remote attestation. +Hierarchical decoupling enables cryptographic algorithm software to achieve ultimate cost efficiency. -- Configures measurements and verifies measurement signatures. +- Hierarchical decoupling: TLS, certificates, and algorithm functions are decoupled for on- demand combination, with layered optimization for algorithm abstraction, scheduling, and algorithm primitives. +- Advanced abstract interfaces: These interfaces prevent external interface changes caused by algorithm tailoring while reducing software footprint. +- Ultimate cost efficiency: Automatic management of feature dependencies and on-demand tailoring to trim down to the minimal implementation of PBKDF2 + AES (20 KB binary file size, 1 KB of heap memory, and 256 bytes of stack memory) are supported. -- Generates and imports measurement baseline data using tools, and verifies baseline data signatures. +**Agile architecture for cryptographic algorithms, addressing post-quantum migration** -- Supports the SM3 algorithm. +An innovative agile architecture for cryptographic algorithms enables rapid application migration and fast evolution of advanced algorithms. -## Kuasar +- Unified northbound interfaces: Standardized and extensible interfaces are available for algorithms, meaning that interface changes due to algorithm switching can be avoided, which in turn means that extensive adaptation of new interfaces for upper-layer applications is not required. +- Plugin-based management of algorithms: The algorithm provider layer supports dynamic algorithm runtime loading. +- Configurable algorithms: Algorithm information can be obtained from configuration files, avoiding hard coding of algorithm identifiers. -Kuasar is a container runtime that supports unified management of multiple types of sandboxes. It supports multiple mainstream sandbox isolation technologies. Based on the Kuasar container runtime combined with the iSulad container engine and StratoVirt virtualization engine, openEuler builds lightweight full-stack self-developed secure containers for cloud native scenarios, delivering key competitiveness of ultra-low overhead and ultra-fast startup. +## Fail-slow Detection for Rapid Identification of Slow Nodes in AI Cluster Training -Kuasar 0.1.0 supports the StratoVirt lightweight VM sandbox and StratoVirt secure container instances created through Kubernetes+iSulad. +During the training process of AI clusters, performance degradation is inevitable, with numerous and complex causes. Existing solutions rely on log analysis after performance degradation occurs. However, it can take 3 to 4 days from log collection to root cause diagnosis and issue resolution on the live network. To address these pain points, an online slow node detection solution is offered. This solution allows for real-time monitoring of key system metrics and uses model- and data-driven algorithms to analyze the observed data and pinpoint the location of slow or degraded nodes. This facilitates system self-healing and fault rectification by O&M personnel. -- Compatible with the Kubernetes ecosystem when the iSulad container engine interconnects with the Kuasar container. +Grouped metric comparison helps detect slow nodes/cards in AI cluster training scenarios. This technology is implemented through gala-anteater and includes new components such as a configuration file, an algorithm library, and slow node comparison based on both time and space. The output includes the exception occurrence time, abnormal metrics, and IP addresses of slow nodes/cards. The technology improves system stability and reliability. The following features are provided: -- Secure container sandboxes based on the StratoVirt lightweight VM sandbox. +- Configuration file: Contains the types of metrics to be observed, configuration parameters for the metric algorithms, and data interfaces, which are used to initialize the slow node detection algorithms. +- Algorithm library: Includes common time series exception detection algorithms, such as Streaming Peaks-over-Threshold (SPOT), k-sigma, abnormal node clustering, and similarity measurement. +- Data: Includes metric data, job topology data, and communicator data. Metric data indicates the time series of metrics, job topology data indicates the node information used in training jobs, and communicator data indicates the node connection relationships (including data parallelism, tensor parallelism, and pipeline parallelism). +- Grouped metric comparison: Supports spatial filtering of abnormal nodes and temporal exception filtering of a single node. Spatial filtering identifies abnormal nodes based on the exception clustering algorithm, while temporal exception filtering determines whether a node is abnormal based on the historical data of the node. -- StratoVirt secure containers for precise resource restriction and management. +## Rubik with Enhanced Collaboration for Hybrid Deployments -## sysBoost +Cloud data centers face a widespread challenge of low resource utilization, typically below 20%. Improving this utilization has become a critical technical priority. Hybrid deployment—deploying workloads with different priorities together—offers an effective solution for increasing resource utilization. While hybrid deployment substantially improves cluster efficiency, it introduces resource contention that can impact the quality of service (QoS) of critical workloads. The key challenge lies in maintaining workload QoS while achieving higher resource utilization. -SsysBoost is a tool for optimizing the system microarchitecture for applications. The optimization involves assembly instructions, code layout, data layout, memory huge pages, and system calls. +Rubik, openEuler's container hybrid deployment engine, implements an adaptive system for single-node computing optimization and QoS assurance. The engine maximizes node resource utilization without compromising the performance of critical workloads. -- **Binary file merging**: Only full static merging is supported. Applications and their dependent dynamic libraries are merged into one binary file, and segment-level reordering is performed. Multiple discrete code segments or data segments are merged into one to improve application performance. +- Cache and memory bandwidth control: Limits the LLC and memory bandwidth of low- priority VMs. Currently, only static allocation is supported. +- **CPU interference control**: Supports CPU time slice preemption in microseconds, simultaneous multithreading (SMT) interference isolation, and anti-priority-inversion. +- **Memory resource preemption**: Terminates offline services first during node out-of memory (OOM) events to protect online service quality. +- **Memcg asynchronous memory reclamation**: Limits the total memory used by hybrid offline applications, and dynamically compresses the memory used by offline services when the online memory utilization increases. +- **quotaBurst traffic control**: When the CPU traffic of key online services is limited, the limit can be exceeded in a short period of time, ensuring the quality of online services. +- **Enhanced observation of PSI**: Collects pressure information at the cgroup v1 level, identifies and quantifies service interruption risks caused by resource contention, and improves hardware resource utilization. +- **iocost service I/O weight control**: Manages drive I/O rates for offline services to prevent bandwidth contention with online services. +- **Cycles per instruction (CPI) monitoring**: Tracks node pressure via CPI metrics to guide offline service eviction decisions. -- **sysBoost daemon**: sysBoost registers with systemd to enable out-of-the-box optimization. systemd will start the sysBoost daemon after the system is started. Then, the sysBoost daemon reads the configuration file to obtain the binary files to be optimized and the corresponding optimization methods. +New features: -- RTO binary file loading kernel module: This binary loading module is added to automatically load the optimized binary file when the kernel loads binary files. +- **Node CPU/memory management**: Monitors resource levels and evicts offline services. -- **Huge page pre-loading of binary code or data segments**: sysBoost provides the huge page pre-loading function. After binary optimization is complete, sysBoost immediately loads the content to the kernel as a huge page. When an application is started, sysBoost maps the pre-loaded content to the user-mode page table in batches to reduce page faults and memory access delay of the application, thereby improving the application startup speed and running efficiency. +## Enhanced CFGO -## CTinspector +The continuous growth in code volume has made front-end bound execution a common issue in processors, which impacts program performance. Feedback-directed optimization techniques in compilers can effectively solve this issue. -CTinspector is a language VM running framework developed by China Telecom e-Cloud Technology Co., Ltd. based on the eBPF instruction set. The CTinspector running framework enables application instances to be quickly expanded to diagnose network performance bottlenecks, storage I/O hotspots, and load balancing, improving the stability and timeliness of diagnosis during system running. +Continuous Feature Guided Optimization (CFGO) in GCC for openEuler and BiSheng Compiler refers to continuous feedback-directed optimization for multimodal files (source code and binaries) and the full lifecycle (compilation, linking, post-linking, runtime, OS, and libraries). The following techniques are included: -- CTinspector uses a packet VM of the eBPF instruction set. The minimum size of the packet VM is 256 bytes, covering all VM components, including registers, stack segments, code segments, data segments, and page tables. +- **Code layout optimization**: Techniques such as basic block reordering, function rearrangement, and hot/cold separation are used to optimize the binary layout of the target program, improving I-cache and I-TLB hit rates. +- **Advanced compiler optimization**: Techniques such as inlining, loop unrolling, vectorization, and indirect calls enable the compiler to make more accurate optimization decisions. -- The packet VM supports independent migration. That is, the code in the packet VM can invoke migrate kernel function to migrate the packet VM to a specified node. +CFGO comprises CFGO-PGO, CFGO-CSPGO, and CFGO-BOLT. Enabling these sub-features in sequence helps mitigate front-end bound execution and improve program runtime performance. To further enhance the optimization, you are advised to add the -flto=auto compilation option during CFGO-PGO and CFGO-CSPGO processes. -- The packet VM also supports resumable execution. That is, after being migrated to another node, the packet VM can continue to execute the next instruction from the position where it has been interrupted on the previous node. +- CFGO-PGO -## CVE-ease + Unlike conventional profile-guided optimization (PGO), CFGO-PGO uses AI for Compiler (AI4C) to enhance certain optimizations, including inlining, constant propagation, and devirtualization, to further improve performance. -CVE-ease is an innovative Common Vulnerabilities and Exposures (CVE) platform developed by China Telecom e-Cloud Technology Co., Ltd. It collects various CVE information released by multiple security platforms and notifies users of the information through multiple channels, such as email, WeChat, and DingTalk. The CVE-ease platform aims to help users quickly learn about and cope with vulnerabilities in the system. In addition to improving system security and stability, users can view CVE details on the CVE-ease platform, including vulnerability description, impact scope, and fixing suggestions, and select a fixing solution as required. +- CFGO-CSPGO -CVE-ease has the following capabilities: + The profile in conventional PGO is context-insensitive, which may result in suboptimal optimization. By adding an additional CFGO-CSPGO instrumentation phase after PGO, runtime information from the inlined program is collected. This provides more accurate execution data for compiler optimizations such as code layout and register optimizations, leading to enhanced performance. -- Dynamically tracks CVEs on multiple platforms in real time and integrates the information into the CVE database. +- CFGO-BOLT -- Extracts key information from the collected CVE information and updates changed CVE information in real time. + CFGO-BOLT adds optimizations such as software instrumentation for the AArch64 architecture and inlining optimization on top of the baseline version, driving further performance gains. -- Automatically maintains and manages the CVE database. +## AI4C -- Queries historical CVE information based on various conditions in interactive mode. +AI4C is an AI-assisted compiler optimization suite. It is a software framework that leverages AI technologies to optimize compiler options and key decisions during optimization passes. It aims to address two major challenges in compilers: -- Reports historical CVE information in real time through WeCom, DingTalk, and email. +- **Difficult performance improvement**: Traditional compiler optimizations have long development cycles, and new optimization techniques are often incompatible with existing optimization processes, making it challenging to achieve the expected performance gains. +- **Low tuning efficiency**: When changes in hardware architecture or software application scenarios occur, significant human effort is required to adjust the cost model for compiler optimizations based on the new workloads, resulting in prolonged tuning times. -## PilotGo +The AI4C framework provides two main modules: Autotuner for automatic tuning of compiler options and AI-enabled compiler-driven program optimization (ACPO). -The PilotGo O&M management platform is a plugin-based O&M management tool developed by the openEuler community. It adopts a lightweight modular design of functional modules that can be iterated and evolved independently, while ensuring stability of core functions. Plugins are used to enhance platform functions and remove barriers between different O&M components, implementing global status awareness and automation. +The software follows a three-layer architecture, as shown in the following figure. The AI4C framework at the upper layer drives the optimization process of a compiler at the middle layer. The Adaptor module of the compiler invokes the AI model library and model inference engine at the lower layer, using optimization feature data and hardware architecture parameters as inputs to run model inference. This results in the optimal setting for key parameters during compilation, thereby achieving compiler optimization. -PilotGo has the following core functional modules: +- Autotuner -- **User management**: Manages users by group based on the organizational structure, and imports existing platform accounts, facilitating migration. + The Autotuner of AI4C is developed based on OpenTuner (Ansel et al. 2015). It uses a plugin to drive the compiler to collect tuning parameters, adjusts key decision-making parameters (such as loop unrolling factors) using search algorithms, injects modifications into the compilation process via the plugin, and runs the compiled binary to gather feedback factors for iterative automatic tuning. -- **Permission management**: Supports RBAC-based permission management, which is flexible and reliable. + - It integrates a series of search algorithms, with dynamic algorithm selection and shared search progress. + - It supports user-configured YAML for custom search spaces and the extension of underlying search algorithms. + - It enables fine-grained code block tuning and coarse-grained automatic tuning of compiler options. + - It achieved performance gains ranging from 3% to 5% on benchmarks such as Cormark, Dhrystone, and Cbench. -- **Host management**: visualized front-end status, software package management, service management, and kernel parameter optimization. +- ACPO -- **Batch management**: Concurrently performs O&M operation, which is stable and efficient. + ACPO provides a comprehensive set of tools, libraries, and algorithms, which replace or enhance heuristic tuning decision-making algorithms in compilers and provide easy-to-use interfaces for compiler engineers to use AI models. During compiler optimization, plugins are used to extract structured input data from optimization passes as model input features. The getAdvice function runs the pre-trained model to obtain decision coefficients, and the compiler uses the model's decision results to replace specific heuristic decisions, thereby achieving better performance. -- **Log audit**: Traces and records user and plugin change operations, facilitating issue backtracking and security audit. + - It decouples compilers from AI models and inference engines, helping algorithm developers focus on AI model development while reducing the costs of model application. It is compatible with multiple compilers, models, AI inference frameworks, and other mainstream products, offering hot update capabilities for AI models. + - The implementation of optimization phases and processes, such as function inlining based on interprocedural analysis (IPA) and loop unrolling for register transfer language (RTL) generation, has resulted in significant gains. -- **Alarm management**: Detects platform exceptions in real time. +## SM Digital Signatures for RPMs -- **Real-time exception detection**: Extends platform functions and associates plugins to realize automation and reduce manual intervention. +According to relevant security standards in China, certain application scenarios require the use of Chinese cryptographic algorithms to ensure the authenticity and integrity of critical executable program sources. openEuler currently uses RPM for software package management, with package signatures based on the openPGP standard. openEuler 24.03 LTS SP1 incorporates the SM2 signing algorithm and SM3 digest algorithm into the RPM mechanism. -## CPDS +Based on the RPM component and the GnuPG2 signing tool the component invokes, this feature enables Chinese cryptographic algorithms within the existing openPGP signature system. In signature generation, you can run the **gpg** command to generate an SM2 signing private key and a certificate, and then run the rpmsign command to add a digital signature based on the SM2 and SM3 algorithms to a specified RPM package. In signature verification, you can run the **rpm** command to import the verification certificate and verify the digital signature of the RPM package to validate its authenticity and integrity. -The wide application of cloud native technologies makes modern application deployment environments more and more complex. The container architecture provides flexibility and convenience, but also brings more monitoring and maintenance challenges. Container Problem Detect System (CPDS) is developed to ensure reliability and stability of containerized applications. +## oneAPI -- **Cluster information collection**: Node agents are implemented on host machines to monitor key container services using systemd, initv, eBPF, and other technologies. Cross-NS agents are configured on nodes and containers in non-intrusive mode to keep track of the application status, resource consumption, key system function execution status, and I/O execution status of containers. The collected information covers network, kernel, and drive LVM of the nodes. +The Unified Acceleration (UXL) Foundation is promoting an open, unified standard accelerator software ecosystem. oneAPI, as the initial project, aims to provide a cross-industry, open, and unified standard programming model to deliver consistent development experience for heterogeneous accelerators, including those for CPUs, GPUs, FPGAs, and specialized accelerators. The oneAPI standard extends existing developer programming models to empower cross-architecture programming using a parallel programming language (Data Parallel C++), a group of accelerator software libraries, and a foundational hardware abstraction interface (Intel® oneAPI Level Zero). This approach supports a wide range of accelerator hardware and processor platforms. To ensure compatibility and improve development efficiency, the oneAPI standard provides various open, cross-platform, and easy-to-use developer software suites. -- **Cluster exception detection**: Raw data from each node is collected to detect exceptions based on exception rules and extract key information. Then, the detection results and raw data are uploaded online and saved permanently. +To fully support oneAPI, the Intel® oneAPI Base Toolkit (Base Kit) and runtime container images are integrated into openEuler 24.03 LTS. Since openEuler 24.03 LTS SP1, openEuler natively supports the adaptation and integration of oneAPI foundational libraries, including the dependencies required for oneAPI runtimes, Intel's graphics acceleration compiler, OpenCL™ Runtimes, and oneAPI Level Zero API that is compatible with platforms like x86_64 and AArch64. To comprehensively support Data Parallel C++ and API-based programming on accelerator libraries, various oneAPI software packages have been adapted and validated on openEuler. You can add the official DNF/Yum repositories of oneAPI to openEuler to install and update all required runtime dependencies, developer tools, and debugging utilities. -- **Fault/Sub-Health diagnosis on nodes and service containers**: Nodes and service containers are diagnosed based on exception detection data. Diagnosis results are saved permanently and can be displayed on the UI for users to view real-time and historical diagnosis data. +## OpenVINO -## EulerMaker Build System +OpenVINO is an open source AI toolkit and runtime library that enhances deep learning models from major frameworks. It streamlines the deployment and inference of AI workloads across processors and accelerators on Intel and other platforms including Arm. Beginning with openEuler 24.03 LTS SP1, native OpenVINO integration is provided, granting full access to OpenVINO computing capabilities on openEuler. -EulerMaker is a package build system. It converts source code into binary packages and allows developers to assemble and tailor scenario-specific OSs based on their requirements. It provides incremental/full build, package layer tailoring, and image tailoring capabilities. +OpenVINO converts and optimizes models trained in popular frameworks to run efficiently on diverse hardware in local, edge, or cloud environments. Popular frameworks include TensorFlow, PyTorch, ONNX, and PaddlePaddle. -- Incremental/Full build: Analyzes the impact based on software changes and dependencies, obtains the list of packages to be built, and delivers parallel build tasks based on the dependency sequence. +## KAE -- **Build dependency query**: Provides a software package build dependency table in a project, and filters and collects statistics on software package dependencies and depended software packages. +KAE is an acceleration solution based on the Kunpeng 920 processor. It contains the KAE encryption and decryption module and the KAEzip compression and decompression module, which accelerate SSL and TLS applications and data compression, reduce processor usage, and boost processor efficiency. In addition, the application layer of KAE masks the internal implementation details, thereby allowing users to quickly migrate services through the standard interfaces of OpenSSL and zlib. -- Layer tailoring: In a build project, developers can select and configure layer models to tailor patches, build dependencies, installation dependencies, and compilation options for software packages. +The KAE encryption and decryption module implements the RSA, SM3, SM4, DH, MD5, and AES algorithms. It provides high-performance symmetric and asymmetric encryption and decryption based on the lossless user-mode driver framework. It is compatible with OpenSSL 1.1.1x and supports both synchronous and asynchronous mechanisms. -- **Image tailoring**: Developers can configure the repository source to generate ISO, QCOW2, and container OS images, and tailor the list of software packages for the images. +KAEzip is the compression and decompression module of KAE. It implements the Deflate algorithm and works with the lossless user-mode driver framework to provide high-performance gzip or zlib APIs. KAE can be used to improve application performance in different scenarios. For example, in distributed storage scenarios, the zlib library accelerates data compression and decompression. diff --git a/docs/en/docs/Releasenotes/known-issues.md b/docs/en/docs/Releasenotes/known-issues.md index 790e6f011eafc512e5fc416d14f0637335d4de95..58742bf76a969efcd7be59b8aeeb2c7f731a9645 100644 --- a/docs/en/docs/Releasenotes/known-issues.md +++ b/docs/en/docs/Releasenotes/known-issues.md @@ -1,10 +1,6 @@ # Known Issues -| No. | Issue Ticket Number | Issue Description | Severity Level | Impact Analysis | Workaround Measure | History Scenario | -| ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ---------------- | -| 1 | [I5LZXD](https://gitee.com/src-openEuler/openldap/issues/I5LZXD) | openldap build problem in openEuler:22.09 | Minor | The case fails to be executed during the build. This problem is caused by the case design. The impact is controllable. Switch to the sleep mode and wait until the operation is complete. The operation occasionally fails in high-load scenarios. | Skip related case. Track the upstream community to solve the problem. | | -| 2 | [I5NLZI](https://gitee.com/src-openEuler/dde/issues/I5NLZI) | \[openEuler 22.09 rc2] Some application icons in the initiator are improperly displayed. | Minor | Only the initiator icon of the DDE desktop is improperly displayed. This issue does not affect functions. The overall impact on usability is controllable. | Switch the theme to avoid this problem. | | -| 3 | [I5P5HM](https://gitee.com/src-openEuler/afterburn/issues/I5P5HM) | \[22.09_RC3_EPOL]\[arm/x86] **Failed to stop afterburn-sshkeys@.service** is displayed when uninstalling the afterburn. | Minor | | | | -| 4 | [I5PQ3O](https://gitee.com/src-openEuler/openmpi/issues/I5PQ3O) | \[openEuler-22.09-RC3] An error is reported when the **ompi-clean -v -d** parameter is executed. | Major | This package is used by the NestOS and has a limited application scope. By default, this package is enabled by the **core** user in the NestOS. This package has little impact on the server version. | No workaround is provided by the SIG. | | -| 5 | [I5Q2FE](https://gitee.com/src-openEuler/udisks2/issues/I5Q2FE) | udisks2 build problem in openEuler:22.09 | Minor | The case fails to be executed during the build. The environment is not retained, and the problem does not recur during long-term local build. | Keep tracking the build success rate in the community. | | -| 6 | [I5SJ0R](https://gitee.com/src-openEuler/podman/issues/I5SJ0R) | \[22.09RC5 arm/x86] An error is reported when executing **podman create --blkio-weight-device /dev/loop0:123:15 fedora ls**. | Minor | blkio-weight is a kernel feature of version 4.xx. Not supported in version 5.10 | Upgrade the podman component. | | +|Issue ID|Repository|Description|Link| +|-|-|-|-| +|I9JSEA|X-diagnosis|\[24.03] Error when executing xdiag ntrace command|| +|I9NUDT|gtk-doc|\[24.03-LTS-rc4][x86/arm] gtk-doc source package local compilation fails, error reported during check phase|| diff --git a/docs/en/docs/Releasenotes/release_notes.md b/docs/en/docs/Releasenotes/release_notes.md index bd10870b01e4e96e4a6864b73463e456efc17a45..f9329ac8f0f47ad6e69b37046af5a8bc0f57c97b 100644 --- a/docs/en/docs/Releasenotes/release_notes.md +++ b/docs/en/docs/Releasenotes/release_notes.md @@ -1,3 +1,3 @@ # Release Notes -This document is the release notes of openEuler 22.09. +This document is the release notes of openEuler 24.03 LTS SP1. diff --git a/docs/en/docs/Releasenotes/resolved-issues.md b/docs/en/docs/Releasenotes/resolved-issues.md index 9a3f7ccfb0e54e0c67438559fae8431322fd51fb..acc09f03acf5db416603baf479f3235a4261f91c 100644 --- a/docs/en/docs/Releasenotes/resolved-issues.md +++ b/docs/en/docs/Releasenotes/resolved-issues.md @@ -1,15 +1,167 @@ -# Resolved Issues +# Resolved Issues For the complete issue list, see [Issues](https://gitee.com/organizations/src-openeuler/issues). -For the complete list kernel related commits, see [Commits](https://gitee.com/openeuler/kernel/commits/openEuler-21.03). +For details about resolved issues, see the following table. -For details about resolved issues, see [Table 1](#table249714911433). +**Table 1** Resolved issues -**Table 1** Resolved issues - -| ISSUE | Description | -| :----------------------------------------------------------- | :----------------------------------------------------------- | -|[I5J302](https://gitee.com/open_euler/dashboard?issue_id=I5J302)|\[Installation conflict arm/x86_64]openEuler 22.09: fwupd conflicts with dbxtool during installation| -|[I5J36Q](https://gitee.com/open_euler/dashboard?issue_id=I5J36Q)|\[Installation conflict arm/x86_64]openEuler 22.09: python3-wrapt conflicts during installation | -|[I5J3K1](https://gitee.com/open_euler/dashboard?issue_id=I5J3K1)|\[Installation conflict arm/x86_64]openEuler 22.09 : mariadb conflicts with mysql during installation| +|Issue ID| Repository| Description|Link| +|-|-|-|-| +|15606203|hsak|\[EulerMaker] hsak build failed in openEuler-24.03-LTS: everything|| +|15617100|ethtool|\[24.03 LTS] Missing colon in ethtool -c command output|| +|15665343|X-diagnosis|\[24.03-LTS-alpha] xdiagnose package downgraded compared to 22.03-LTS-SP3 release version|| +|15673335|libvirt|\[EulerMaker] libvirt dependency installation incomplete in openEuler-24.03-LTS|| +|15689728|utshell|utshell continue --help and continue command show redundant prompts|| +|15690236|utshell|utshell select command not working|| +|15690392|utshell|utshell kill -l sequence number incorrect|| +|15690671|utshell|utshell ulimit -S command not supported|| +|15691255|utshell|utshell let --help prompt incomplete|| +|15691427|utshell|utshell exec -c exec -l showing redundant output|| +|15691597|utshell|utshell hash --help Chinese and English prompts inconsistent|| +|15691660|utshell|utshell complete -p has no output|| +|15691727|utshell|utshell compgen -a output does not match expectation|| +|15691889|utshell|utshell help local Chinese and English prompts inconsistent|| +|15692039|utshell|utshell trap --help and trap -l missing some prompts|| +|15728456|tensorflow|\[Eulermaker] tensorflow installation issue in openEuler-24.03-LTS:everything|| +|15729670|python-pydantic|\[Eulermaker] python-pydantic installation issue in openEuler-24.03-LTS:everything|| +|15729969|libblockdev|\[24.03-LTS] Installation fails on servers with two or more nvme devices|| +|15730072|python-pytype|\[Eulermaker] python-pytype installation issue in openEuler-24.03-LTS:everything|| +|15731483|diskimage-builder|\[Eulermaker] diskimage-builder installation issue in openEuler-24.03-LTS:epol|| +|15731524|ncbi-blast|\[Eulermaker] ncbi-blast installation issue in openEuler-24.03-LTS:epol|| +|15734079|python-cursive|\[Eulermaker] python-cursive installation issue in openEuler-24.03-LTS:epol|| +|15734154|python-django-compressor|\[Eulermaker] python-django-compressor installation issue in openEuler-24.03-LTS:epol|| +|15734246|python-exif|\[Eulermaker] python-exif installation issue in openEuler-24.03-LTS:epol|| +|15734307|python-mitmproxy|\[Eulermaker] python-mitmproxy installation issue in openEuler-24.03-LTS:epol|| +|15734615|python-pycadf|\[Eulermaker] python-pycadf installation issue in openEuler-24.03-LTS:epol|| +|15734932|python-scciclient|\[Eulermaker] python-scciclient installation issue in openEuler-24.03-LTS:epol|| +|15735746|system-config-users-docs|\[Eulermaker] system-config-users-docs installation issue in openEuler-24.03-LTS:epol|| +|15746375|pin-gcc-client|\[EulerMaker] pin-gcc-client build failed in openEuler-24.03-LTS:everything|| +|15747476|ocaml|\[EulerMaker] ocaml build failed in openEuler-24.03-LTS:everything|| +|15747843|gnu-efi|\[EulerMaker] gnu-efi build failed in openEuler-24.03-LTS:everything|| +|15761577|hmdfs|\[EulerMaker] hmdfs build failed in openEuler-24.03-LTS:epol|| +|15762422|dde-network-utils|\[EulerMaker] dde-network-utils build failed in openEuler-24.03-LTS:epol|| +|15762972|octave|\[EulerMaker] octave build failed in openEuler-24.03-LTS:epol|| +|15763462|python-suds-jurko|\[EulerMaker] python-suds-jurko build failed in openEuler-24.03-LTS:epol|| +|15763805|mate-power-manager|\[EulerMaker] mate-power-manager build failed in openEuler-24.03-LTS:epol|| +|15764156|anaconda|\[24.03-LTS] Prompt message is in English when filling in new mount point capacity in Chinese installation page|| +|15766075|dsoftbus|\[EulerMaker] dsoftbus build failed in openEuler-24.03-LTS:epol|| +|15768500|secpaver|/bin/pav and /bin/pavd permissions are 700, should be 500|| +|15770285|anaconda|\[24.03 LTS] Hostname length limit and special character support issues|| +|15786468|syscare|syscare adaptation for OE-24.03|| +|15786578|dpu-utilities|qtfs compilation failed in 24-03, needs kernel adaptation|| +|15794178|valgrind|\[24.03-LTS round1] \[x86/arm] valgrind-3.22.0-2.oe2403 security compilation option PIE not satisfied|| +|15796700|dpu-utilities|Crash occurs when setting up shared directory with qtfs in 24-03 version|| +|15803043|qemu|\[24.03-LTS round1] \[x86/arm] qemu-8.2.0-1.oe2403 security compilation option RPATH not satisfied|| +|15810085|bash|\[24.03-LTS round1] \[x86/arm] bash-relocation-5.2.15-5.oe2403 security compilation option STRIP not satisfied|| +|15820007|dpu-utilities|Under kernel 6.6.0, rexec executes successfully but process cannot terminate normally|| +|15825072|syscare|syscare check command cannot detect symbol conflicts in current 24-03 version|| +|15825628|sysmaster|\[24.03 LTS] In VM scenario, symbolic links for /usr/sbin/reboot are inconsistent after yum install and yum reinstall|| +|15825933|sysmaster|\[24.03 LTS] In VM scenario, when querying status with sctl status xxx I grep xxx, it prompts as binary file|| +|15825969|sysmaster|\[24.03 LTS] In VM scenario, after configuring \[Install] WantedBy=wantedby.service and enabling service, symbolic link cannot be generated|| +|15826243|sysmaster|\[24.03 LTS] In VM scenario, home occasionally fails to mount after restart|| +|15826388|sysmaster|\[24.03 LTS] In VM scenario, after devmaster installation and system restart, IP is lost and devmaster service fails to start|| +|15838130|syscare|upatch service fails to start when selinux is disabled|| +|15841629|openssh|\[24.03-LTS-rc2][x86/arm] openssh source package local compilation fails, error during check phase|| +|15848994|itrustee_tzdriver|Version 1.5 fails to compile on oe2403|| +|15853199|syscare|Patch content not effective after syscare loading and activation|| +|15870797|utsudo|\[24.03-LTS-RC2] When admin user executes utsudo command and enters password, error: unable to allocate memory|| +|15892441|syscare|In current version, selinux enforce status affects patching of some processes|| +|15901204|sysmonitor|oe_test_sysmonitor_netcard network card monitoring function not working|| +|15908329|protobuf|24.03 protobuf cannot generate JavaScript source code|| +|15909183|rubik|2403 Rubik co-location environment cannot meet requirements for fine-grained resource QoS awareness and control inheritance verification|| +|15909463|kernel|\[openEuler 24.03 LTS Stability Testing] System panic when executing stability test with memory and CPU pressure|| +|15910218|kernel|net.netfilter.nf_hooks_lwtunnel configuration cannot be isolated by namespace|| +|15914663|kernel|No error when executing: echo "options ipvlan ipvlan_default_mode=4000" > /etc/modprobe.d/ipvlan.conf;modprobe ipvlan|| +|15920349|dmidecode|dmidecode --from-dump exec_result1 error: Can't read memory from exec_result1|| +|15925770|dpu-utilities|In libvirt scenario, virsh define fails after starting libvirtd|| +|15925910|dpu-utilities|Script section needs modification in libvirt guidance documentation|| +|15927121|cinnamon|\[EulerMaker] cinnamon build failed in openEuler-24.03-LTS:epol|| +|15927180|qla2xxx|\[EulerMaker] qla2xxx build failed in openEuler-24.03-LTS:epol|| +|15928799|dracut|\[24.03-LTS] Warning messages exist when executing dracut -f on x86|| +|15931868|iproute|Error when executing ip vrf exec vrf1: setexecfilecon for "xxx" failed|| +|15931947|gazelle|\[2403][Local loopback][Testability] Current local loopback test cannot simultaneously observe local and remote traffic flow|| +|15931999|gazelle|\[2403][Local loopback] Current local loopback test and remote traffic flow experience interruptions|| +|15932211|open-iscsi>|iscsi_q_1 process not recognized after iscsi login|| +|15932814|systemd|\[24.03] Default backlog value is not 4096|| +|15932868|tree|\[24.03 LTS] Tree command output display changed|| +|15933049|nss|24.03 boot failure when national encryption security boot is enabled|| +|15933299|targetcli|LUN creation fails after timeout exceeds certain quantity|| +|15934232|gala-gopher>|ksliprobe probe parameter collection partially failed|| +|15934433|sysmaster|\[24.03 LTS] In VM scenario, no information printed in devmaster.log when loading network card configuration|| +|15934496|sysmaster|\[24.03 LTS] In VM scenario, /run/devmaster/data/b8\:0 not deleted after remove event|| +|15937748|arangodb|\[EulerMaker] arangodb build failed in openEuler-24.03-LTS:epol|| +|15937869|dde-introduction|\[EulerMaker] dde-introduction build failed in openEuler-24.03-LTS:epol|| +|15937916|deepin-music|\[EulerMaker] deepin-music build failed in openEuler-24.03-LTS:epol|| +|15937924|Kmesh|\[EulerMaker] Kmesh build failed in openEuler-24.03-LTS:epol|| +|15937944|lustre|\[EulerMaker] lustre build failed in openEuler-24.03-LTS:epol|| +|15943994|e2fsprogs|Infinite loop in output when executing debugfs -R "logdump -aOS" command after ext4 journal is full|| +|15945116|multipath-tools|Default values changed for marginal_path_double_failed_time, marginal_path_err_sample_time, marginal_path_err_rate_threshold, marginal_path_err_recheck_gap_time|| +|15945293|kernel|/proc/sys/vm/percpu_pagelist_fraction interface missing|| +|15945440|elfutils|eu-elfcompress option -t, --type=TYPE test failed|| +|15946483|kernel|register_shrinker interface parameter difference causing ko compilation failure|| +|15947198|nfs-utils|nfs ipv6 function abnormal, dd hangs|| +|15949091|sysmaster|\[24.03 LTS] In VM scenario, incorrect value set in database after IMPORT{cmdline} rule match success|| +|15950700|systemd|\[24.03] systemctl -H root@xxxx permission denied after multiple successful connections to peer|| +|15950808|systemd|\[24.03] When systemctl-journal sets Storage=volatile and /run directory is full, no system.journal file in /run/log/|| +|15951289|notification_eventhandler|\[EulerMaker] notification_eventhandler build failed in openEuler-24.03-LTS:epol|| +|15955404|communication_ipc|\[EulerMaker] communication_ipc build failed in openEuler-24.03-LTS:epol|| +|15959755|gazelle|\[24.03] When starting example with gazelle, mbuf_pool_freecnt continuously decreases after startup|| +|15960779|kpatch|When creating hotfix using syscare, kpatch error: create-diff-object: ERROR: version.o: kpatch_bundle_symbols: 255: symbol version_proc_show at offset 16 within section .text.version_proc_show, expected 0|| +|15961549|oncn-bwm|After setting qos function, pod business continuous stream shows decreasing bandwidth|| +|15961693|oncn-bwm|bwm-daemon abnormal parameter printing, priority parameter setting optimization|| +|15961761|oncn-bwm|bwm-cni.log lacks file rotation mechanism|| +|15975714|kernel|perf test execution causes coredump|| +|15981164|procps-ng|\[2403] uptime command always shows 0 user|| +|15989072|gazelle|Physical NIC UDP fragmentation issue|| +|15989987|gala-gopher|No corresponding probe description for cadvisor sub-feature matching cadvisor_port in params|| +|15991777|distributeddatamgr_kv_store|\[EulerMaker] distributeddatamgr_kv_store build failed in openEuler-24.03-LTS:epol|| +|15991901|distributedhardware_device_manager|\[EulerMaker] distributedhardware_device_manager build failed in openEuler-24.03-LTS:epol|| +|15991910|commonlibrary_c_utils|\[EulerMaker] commonlibrary_c_utils build failed in openEuler-24.03-LTS:epol|| +|15991932|distributed-beget|\[EulerMaker] distributed-beget build failed in openEuler-24.03-LTS:epol|| +|15991941|security_device_auth|\[EulerMaker] security_device_auth build failed in openEuler-24.03-LTS:epol|| +|15991957|python-async-generator|\[EulerMaker] python-async-generator build failed in openEuler-24.03-LTS:epol|| +|15991964|security_device_security_level|\[EulerMaker] security_device_security_level build failed in openEuler-24.03-LTS:epol|| +|15991968|security_dataclassification|\[EulerMaker] security_dataclassification build failed in openEuler-24.03-LTS:epol|| +|15991970|systemabilitymgr_samgr|\[EulerMaker] systemabilitymgr_samgr build failed in openEuler-24.03-LTS:epol|| +|15991975|systemabilitymgr_safwk|\[EulerMaker] systemabilitymgr_safwk build failed in openEuler-24.03-LTS:epol|| +|15991977|lustre-release|\[EulerMaker] lustre-release build failed in openEuler-24.03-LTS:epol|| +|15991984|security_huks|\[EulerMaker] security_huks build failed in openEuler-24.03-LTS:epol|| +|15995729|bash|Coredump occurs after executing echo {0..1000000}{0..1000000}|| +|15995884|bash|Unable to use bash command|| +|15996637|bash|Bash script execution results do not meet expectations|| +|15996846|hbase|\[Eulermaker] Installation issue with hbase in openEuler-24.03-LTS:everything project|| +|15997145|kernel|Process in D state when taking two non-backup disks offline during raid10 testing|| +|16000669|oncn-bwm|Failed to enable NIC qos function with bwmcli|| +|16000817|gala-gopher|Some parameters cannot be collected by proc probe|| +|16001527|epiphany|\[EulerMaker] epiphany build failed in openEuler-24.03-LTS:everything|| +|16002591|gala-gopher|Error messages in l7 probe|| +|16003291|gala-gopher|Some parameters cannot be collected by l7 probe after simulating http requests|| +|16006314|tree|\[24.03] Differences compared to 22.03SP1: 1. Different return value when unable to open files 2. tree -H command execution path has an extra /|| +|16006408|deepin-movie|\[EulerMaker] deepin-movie build failed in openEuler-24.03-LTS:epol|| +|16006421|distributeddatamgr_data_object|\[EulerMaker] distributeddatamgr_data_object build failed in openEuler-24.03-LTS:epol|| +|16006464|distributeddatamgr_datamgr_service|\[EulerMaker] distributeddatamgr_datamgr_service build failed in openEuler-24.03-LTS:epol|| +|16006487|distributeddatamgr_relational_store|\[EulerMaker] distributeddatamgr_relational_store build failed in openEuler-24.03-LTS:epol|| +|16006498|filemanagement_dfs_service|\[EulerMaker] filemanagement_dfs_service build failed in openEuler-24.03-LTS:epol|| +|16006692|system-config-users|\[EulerMaker] system-config-users build failed in openEuler-24.03-LTS:epol|| +|16006746|python-keras-rl2|\[Eulermaker] Installation issue with python3-keras-rl2 in openEuler-24.03-LTS:everything project|| +|16011486|gala-gopher|P_CACHE count check failed in flamegraph probe multi-process logs|| +|16031979|kernel>|After completing I/O under software RAID, executing -r to remove member disk, MD device status remains active and cannot change to clean status|| +|16039571|X-diagnosis|\[24.03] No data returned when xd_iolatency detection is enabled|| +|16041399|gala-gopher|ksli probe generates coredump file during startup|| +|16051455|sudo|\[oe-24.03] No timestamp file named after username in /run/sudo/ts/ directory|| +|16066222|hiviewdfx_hilog|\[EulerMaker] hiviewdfx_hilog build failed in openEuler-24.03-LTS:epol|| +|16094139|gazelle|\[2403][Local Loopback] Traffic remains at 0 when setting mud mode during local loopback streaming|| +|16094258|gazelle|\[Local Loopback] Stream interruption occurs during local loopback streaming|| +|16094913|python-email-validator|\[EulerMaker] python-email-validator build failed in openEuler-24.03-LTS:everything|| +|16171787|raspberrypi-utils|\[EulerMaker] raspberrypi-utils build failed in openEuler-24.03-LTS:epol|| +|16203813|gazelle|mbuf_pool_freecnt continuously decreases without recovery when testing packet reception using igmpv3|| +|16214083|gazelle|Gazelle reports error when starting with normally configured IP address having last octet between 224 and 238|| +|16228742|R-xml2|\[EulerMaker] R-xml2 build failed in openEuler-24.03-LTS:everything|| +|16270801|kernel|\[EulerMaker] kernel-rt build failed in openEuler-24.03-LTS:everything|| +|16275132|kernel|\[24.03 LTS RC4] Installation hangs when installing 24.03 LTS RC4 ISO on BIOS version 1.06(U75)|| +|16277746|kubernetes|\[24.03-LTS-rc4][x86/arm] During k8s deployment, kubelet service startup fails with error "kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_EXTRA_ARGS", causing k8s deployment failure|| +|16279608|raspberrypi|Default interface name change in image|| +|16323230|shim|Error when using shim to start VM on 24.03 physical machine|| +|16373409|kernel|\[EulerMaker] kernel-rt build failed in openEuler-24.03-LTS:everything|| +|16484569|grub2|\[24.03-LTS] x86 physical machine installation fails when installing daily build ISO in UEFI mode|| diff --git a/docs/en/docs/SecHarden/security-configuration-hardening-tool.md b/docs/en/docs/SecHarden/security-configuration-hardening-tool.md new file mode 100644 index 0000000000000000000000000000000000000000..e2cfe44bc5081fe75f6feb8ba4743ed34b2eab0b --- /dev/null +++ b/docs/en/docs/SecHarden/security-configuration-hardening-tool.md @@ -0,0 +1,167 @@ +# Security Configuration Hardening Tool + +## Overview + +This document offers a basic introduction and usage guide for the security configuration hardening tool, sec_conf. + +## Introduction to sec_conf + +### Background and Overview + +openEuler supports a range of security features, including native Linux security features and those developed by the community. However, issues such as fragmented features, complex configuration processes, and high user learning costs remain. Moreover, misconfiguration of security features with interception capabilities (such as IMA appraisal, secure boot, and access control) can result in system boot failure or operational issues. To address these challenges, sec_conf provides an automated security configuration mechanism, allowing users to conduct system security checks and hardening, thereby facilitating the adoption of openEuler security features in various application scenarios. + +### Features + +sec_conf is a security hardening tool that helps administrators configure openEuler security features like IMA, DIM, and secure boot. Users can input configuration details, such as desired security hardening goals, to generate corresponding security feature configuration scripts. + +Currently, sec_conf supports configurable security mechanisms including IMA, DIM, secure boot, and kernel module verification. + +## Installation and Deployment + +### Installing Dependencies + +The following software is required to compile secPaver: + +- Make +- Go 1.11+ + +### Downloading the Source Code + +```sh +git clone https://gitee.com/openeuler/secpaver.git -b sec_conf +``` + +### Compiling and Installing + +```sh +cd secpaver +make +``` + +Install the software: + +```sh +make install +``` + +## Project File Format Description + +The sec_conf project file includes policy configuration files, check script templates, and configuration script templates. + +### Policy Configuration File + +The policy configuration file defines configurations for DIM, IMA, secure boot, and kernel module verification features. It uses YAML format, with the following fields: + + + + + + + + + + + + + + + + + + + + + +
Primary Configuration ItemSecondary Configuration ItemTypeAttributeDescription
nameN/AstringoptionalName of the configuration file.
versionN/AstringoptionalVersion number of the configuration policy.
dimenablebooloptionalEnable or disable DIM functionality.
measure_liststring arrayoptionalFiles to be measured by DIM. User-space files require absolute paths. Kernel modules require valid kernel module names. For the kernel, specify "kernel".
log_capintoptionalMaximum number of measurement log entries. Logging stops when this limit is reached. The default is 100000.
scheduleintoptionalTime to schedule after measuring a process/module, in milliseconds. Set to 0 to disable scheduling. The default is 0.
intervalintoptionalAutomatic measurement interval, in minutes. Set to 0 to disable automatic measurement. The default is 0.
hashstringoptionalHash algorithm for measurement. The default is sha256.
core_pcrintoptionalExtend dim_core measurement results to the PCR of the TPM. Set to 0 to disable (ensure it matches the actual PCR number of the chip). The default is 0.
monitor_pcrintoptionalExtend dim_monitor measurement results to the PCR of the TPM. Set to 0 to disable (ensure it matches the actual PCR number of the chip). The default is 0.
signaturebooloptionalEnable or disable the policy file and signature mechanism.
auto_baselinebooloptionalWhether to establish a DIM baseline. If it is set to false, the administrator must manually generate the baseline.
secure_bootenablebooloptionalEnable or disable secure boot.
anti_rollbackbooloptionalEnable or disable the secure boot anti-rollback policy.
verbosebooloptionalEnable or disable secure boot-related logs.
modsignenablebooloptionalEnable or disable the kernel module verification feature.
imameasure_liststring arrayoptionalList of files protected by IMA (requires absolute paths).
appraise_liststring arrayoptionalList of files evaluated by IMA (requires absolute paths).
+ +>![](./public_sys-resources/icon-note.gif) **Note:** + +1. The **sec_conf.yaml** file must be placed at **/usr/share/secpaver/scripts/sec_conf/sec_conf.yaml**. +2. Parameter types must adhere to the requirements specified in the table above. +3. If a configuration is not provided, the default value will be used. + +### Check Script Templates and Configuration Script Templates + +Template files use the go-template engine to merge script files with data, producing the final text output. + +Check script templates are stored in the **/usr/share/secpaver/scripts/sec_conf/check/** directory. This directory holds script templates for features like DIM and IMA. These templates cannot be executed directly and must be processed by sec_conf to generate updated scripts for conducting openEuler feature checks. + +Configuration script templates are stored in the **/usr/share/secpaver/scripts/sec_conf/gen/** directory. This directory contains script templates for features like DIM and IMA. These templates cannot be executed directly and must be processed by sec_conf to generate updated scripts for configuring openEuler features. + +>![](./public_sys-resources/icon-note.gif) **Note:** + +Configuration and check templates must not be altered. They are exclusively used by sec_conf to generate scripts. + +## Security Configuration Command Line Interface + +| Option | Function Description | Command Format | +| :---------: | :-----------------------------------------------------------------------------------------: | :------------------------------: | +| --help,-h | Display sec_conf command line help information. | sec_conf -h | +| gen_check | Generate a security configuration check script and output it to the command line interface. | sec_conf gen_check | +| gen_config | Generate a security configuration script and output it to the command line interface. | sec_conf gen_config | +| --output,-o | Output the generated configuration script to a specified file. | sec_conf gen_config -o config.sh | + +## Usage Instructions + +Configure the YAML file. Example: + +```yaml +# cat /usr/share/secpaver/scripts/sec_conf/sec_conf.yaml +--- +name: "openEuler security configuration" +version: "1" +dim: + enable: true + measure_list: + - "/usr/bin/bash" + - "nf_nat" + - "kernel" + log_cap: 100000 + schedule: 0 + interval: 0 + hash: "sha256" + core_pcr: 11 + monitor_pcr: 12 + signature: true + auto_baseline: true +secure_boot: + enable: true + anti_rollback: true + verbose: true +modsign: + enable: true +ima: + measure_list: + - "/usr/bin/ls" + - "/usr/bin/cat" + - "/usr/bin/base64" + - "/usr/bin/base32" + appraise_list: + - "/usr/bin/base64" + - "/usr/bin/base32" + - "/usr/bin/sleep" + - "/usr/bin/date" +... +``` + +Generate feature configuration and check scripts: + +```sh +sec_conf gen_config -o ./config.sh +sec_conf gen_check -o ./check.sh +``` + +Run the configuration script and verify the configuration. If the configuration is correct, reboot the system to apply the changes. + +```sh +sh ./config.sh -s +sh ./check.sh -s +reboot +``` + +After rebooting, run the configuration script again and verify the configuration. All functionality checks should now pass. + +```sh +sh ./config.sh -s +sh ./check.sh -s +``` diff --git a/docs/en/docs/ShangMi/disk-encryption.md b/docs/en/docs/ShangMi/disk-encryption.md deleted file mode 100644 index dc0fbc67d7be95049bbea115399e29d6b5563b08..0000000000000000000000000000000000000000 --- a/docs/en/docs/ShangMi/disk-encryption.md +++ /dev/null @@ -1,89 +0,0 @@ -# Disk Encryption - -## Overview - -Disk encryption protects the storage confidentiality of important data. Data is encrypted based on a specified encryption algorithm and then written to disks. This feature mainly involves the user-mode tool cryptsetup and the kernel-mode module dm-crypt. Currently, the disk encryption feature provided by the openEuler OS supports ShangMi (SM) series cryptographic algorithms. Parameters are as follows: - -- Encryption modes: luks2 and plain; -- Key length: 256 bits; -- Message digest algorithm: SM3; -- Encryption algorithm: sm4-xts-plain64. - -## Prerequisites - -1. Kernel 5.10.0-106 or later - -``` -$ rpm -qa kernel -kernel-5.10.0-106.1.0.55.oe2209.x86_64 -``` - -2. cryptsetup 2.4.1-1 or later - -``` -$ rpm -qa cryptsetup -cryptsetup-2.4.1-1.oe2209.x86_64 -``` - -## How to Use - -A disk is formatted into a disk in a specified encryption mode and mapped to **/dev/mapper** as a dm device. Subsequent disk read and write operations are performed through the dm device. Data encryption and decryption are performed in kernel mode and are not perceived by users. The procedure is as follows: - -1. Format the disk and map the disk as a dm device. - -a. luks2 mode - -Set the encryption mode to luks2, encryption algorithm to sm4-xts-plain64, key length to 256 bits, and message digest algorithm to SM3. - -``` -# cryptsetup luksFormat /dev/sdd -c sm4-xts-plain64 --key-size 256 --hash sm3 -``` - -b. plain mode - -Set the encryption mode to plain, encryption algorithm to sm4-xts-plain64, key length to 256 bits, and message digest algorithm to SM3. - -``` -# cryptsetup plainOpen /dev/sdd crypt1 -c sm4-xts-plain64 --key-size 256 --hash sm3 -``` - -2. After the mapping is successful, run the **lsblk** command to view the device information. - -``` -# lsblk -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS -...... -sdd 8:48 0 50G 0 disk -└─crypt1 253:3 0 50G 0 crypt -...... -``` - -3. Perform I/O read and write operations on the encrypted device. - -Deliver I/Os to raw disks. - -``` -# dd if=/dev/random of=/dev/mapper/crypt1 bs=4k count=10240 -``` - -Deliver I/Os through the file system. - -``` -# mkfs.ext4 /dev/mapper/crypt1 -# mount /dev/mapper/crypt1 /mnt/crypt/ -# dd if=/dev/random of=/mnt/crypt/tmp bs=4k count=10240 -``` - -4. Disable device mapping. - -If a file system is mounted, unmount it first. - -``` -# umount /mnt/crypt -``` - -Closes a device. - -``` -# cryptsetup luksClose crypt1 -``` diff --git a/docs/en/docs/ShangMi/drive-encryption.md b/docs/en/docs/ShangMi/drive-encryption.md new file mode 100644 index 0000000000000000000000000000000000000000..4e260107885b0559064c3c6f6bf823110e3b26a8 --- /dev/null +++ b/docs/en/docs/ShangMi/drive-encryption.md @@ -0,0 +1,90 @@ +# Drive Encryption + +## Overview + +Drive encryption protects the storage confidentiality of important data. Data is encrypted based on a specified encryption algorithm and then written to drives. This feature mainly involves the user-mode tool cryptsetup and the kernel-mode module dm-crypt. Currently, the drive encryption feature provided by the openEuler OS supports ShangMi (SM) series cryptographic algorithms. Parameters are as follows: + +- Encryption modes: luks2 and plain; +- Key length: 256 bits; +- Message digest algorithm: SM3; +- Encryption algorithm: sm4-xts-plain64. + +## Prerequisites + +1. Kernel 5.10.0-106 or later + + ```shell + $ rpm -qa kernel + kernel-5.10.0-106.1.0.55.oe2209.x86_64 + ``` + +2. cryptsetup 2.4.1-1 or later + + ```shell + $ rpm -qa cryptsetup + cryptsetup-2.4.1-1.oe2209.x86_64 + ``` + +## How to Use + +A drive is formatted in a specified encryption mode and mapped to **/dev/mapper** as a dm device. Subsequent drive read and write operations are performed through the dm device. Data encryption and decryption are performed in kernel mode and are not perceived by users. The procedure is as follows: + +1. Format the drive and map the drive as a dm device. + + a. luks2 mode + + Set the encryption mode to luks2, encryption algorithm to sm4-xts-plain64, key length to 256 bits, and message digest algorithm to SM3. + + ```shell + # cryptsetup luksFormat /dev/sdd -c sm4-xts-plain64 --key-size 256 --hash sm3 + # cryptsetup luksOpen /dev/sdd crypt1 + ``` + + b. plain mode + + Set the encryption mode to plain, encryption algorithm to sm4-xts-plain64, key length to 256 bits, and message digest algorithm to SM3. + + ```shell + # cryptsetup plainOpen /dev/sdd crypt1 -c sm4-xts-plain64 --key-size 256 --hash sm3 + ``` + +2. After the mapping is successful, run the **lsblk** command to view the device information. + + ```shell + # lsblk + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS + ...... + sdd 8:48 0 50G 0 disk + └─crypt1 253:3 0 50G 0 crypt + ...... + ``` + +3. Perform I/O read and write operations on the encrypted device. + + Deliver I/Os to raw drives. + + ```shell + # dd if=/dev/random of=/dev/mapper/crypt1 bs=4k count=10240 + ``` + + Deliver I/Os through the file system. + + ```shell + # mkfs.ext4 /dev/mapper/crypt1 + # mount /dev/mapper/crypt1 /mnt/crypt/ + # dd if=/dev/random of=/mnt/crypt/tmp bs=4k count=10240 + ``` + +4. Disable device mapping. + + If a file system is mounted, unmount it first. + + ```shell + # umount /mnt/crypt + ``` + + Closes a device. + + ```shell + # cryptsetup close crypt1 + ``` diff --git a/docs/en/docs/ShangMi/overview.md b/docs/en/docs/ShangMi/overview.md index 5a398bd72e24cb75ef0a7b15d9ebd410e6cb4b54..d450b81a20cfc57fdef580cfa4cb0f1a2dce3021 100644 --- a/docs/en/docs/ShangMi/overview.md +++ b/docs/en/docs/ShangMi/overview.md @@ -1,6 +1,16 @@ # Overview -The ShangMi (SM) features for the openEuler OS aims to enable SM series cryptographic algorithms for key security features of the OS and provide cryptographic services such as the SM series cryptographic algorithm library, certificates, and secure transmission protocols for upper-layer applications. +ShangMi (SM) algorithms are commercial-grade cryptographic technologies. Cryptographic algorithms form the backbone of security technologies in information systems. Globally, widely adopted algorithms include RSA, AES, and SHA256. In parallel, China has developed a suite of cryptographic algorithms that cater to mainstream application scenarios. Among these, SM2, SM3, and SM4 are particularly prominent in OSs. + +| Algorithm | Publicly Available | Type | Application Scenarios | +| --------- | ------------------ | --------------------- | --------------------------------------------------------------------- | +| SM2 | Yes | Asymmetric encryption | Digital signatures, key exchange, encryption/decryption, PKI systems | +| SM3 | Yes | Hash algorithm | Integrity protection, one-way encryption, and other general scenarios | +| SM4 | Yes | Symmetric encryption | Encrypted storage, secure transmission | + +Additionally, other publicly available algorithms like SM9 and ZUC, as well as non-public algorithms such as SM1 and SM7, are part of the ecosystem. Notably, all publicly available Chinese algorithms have been integrated into ISO/IEC standards, gaining international recognition. China has also established a series of cryptographic technical specifications and application standards, including commercial cryptographic certificate standards and the TLCP protocol stack. These collectively form China's commercial cryptographic standard system, which guides the development of the cryptographic security industry. + +The SM features for the openEuler OS aims to enable SM series cryptographic algorithms for key security features of the OS and provide cryptographic services such as the SM series cryptographic algorithm library, certificates, and secure transmission protocols for upper-layer applications. Currently, the following SM features are supported: @@ -15,4 +25,5 @@ Currently, the following SM features are supported: 9. The SM2 certificate is supported in kernel module signing and module signature verification. 10. SM4-CBC and SM4-GCM algorithms are supported in Kernel Transport Layer Security (KTLS). 11. SM3 and SM4 algorithms are supported in Kunpeng Accelerator Engine (KAE). -12. The SM3 algorithm and SM2 certificated signature are supported for UEFI secure boot. \ No newline at end of file +12. UEFI secure boot supports the SM3 digest algorithm and SM2 digital signatures. +13. RPM supports the SM2 encryption/decryption algorithm and SM3 digest algorithm for signing and verification. diff --git a/docs/en/docs/ShangMi/rpm-signature-verification.md b/docs/en/docs/ShangMi/rpm-signature-verification.md new file mode 100644 index 0000000000000000000000000000000000000000..c0df357090c3133907ed134f3a8fdd370cf012be --- /dev/null +++ b/docs/en/docs/ShangMi/rpm-signature-verification.md @@ -0,0 +1,99 @@ +# RPM Signature Verification + +## Overview + +openEuler employs RPM for package management, adhering to the openPGP signature specification. openEuler 24.03 LTS SP1 enhances the open source RPM by adding support for SM2/3 algorithm-based signature generation and verification. + +The following packages have been enhanced for SM algorithm capabilities: + +- GnuPG: The `gpg` CLI tool now supports generating SM signatures. +- RPM: RPM can now invoke `gpg` commands and openSSL APIs for SM signature generation and verification. +- openSSL: SM signature verification is supported (already supported in the open source version). + +## Prerequisites + +1. The following or later versions of gnupg2, libgcrypt, and rpm packages must be installed: + + ```sh + $ rpm -qa libgcrypt + libgcrypt-1.10.2-3.oe2403sp1.x86_64 + + $ rpm -qa gnupg2 + gnupg2-2.4.3-5.oe2403sp1.x86_64 + + $ rpm -qa rpm + rpm-4.18.2-20.oe2403sp1.x86_64 + ``` + +2. ECDSA signing and verification are limited to SM2. + +## Usage + +1. Generate a key. + + Method 1: + + ```sh + gpg --full-generate-key --expert + ``` + + Method 2: + + ```sh + gpg --quick-generate-key sm2p256v1 + ``` + + You will be prompted to enter a password. This password is required for subsequent key operations or signing. Pressing Enter without entering a password means no password is set. + +2. Export the certificate. + + ```sh + gpg -o --export + ``` + +3. Enable the macro for SM3 hash algorithm and SM2 algorithm. + + ```sh + $ vim /usr/lib/rpm/macros + %_enable_sm2p256v1_sm3_algo 1 + ``` + +4. Import the certificate into the RPM database. + + ```sh + rpm --import + ``` + +5. Write the macros required for signing. + + ```sh + $ vim ~/.rpmmacros + %_signature gpg + %_gpg_path /root/.gnupg + %_gpg_name + %_gpgbin /usr/bin/gpg2 + + %__gpg_sign_cmd %{shescape:%{__gpg}} \ + gpg --no-verbose --no-armor --no-secmem-warning --passphrase-file /root/passwd \ + %{?_gpg_digest_algo:--digest-algo=%{_gpg_digest_algo}} \ + %{?_gpg_sign_cmd_extra_args} \ + %{?_gpg_name:-u %{shescape:%{_gpg_name}}} \ + -sbo %{shescape:%{?__signature_filename}} \ + %{?__plaintext_filename:-- %{shescape:%{__plaintext_filename}}} + ``` + + `%__gpg_sign_cmd` includes the default configuration with the addition of `--passphrase-file /root/passwd`. The **passwd** file contains the password. This addition is required only If a password is set in step 1. + +6. Generate a RPM package signature. + + ```sh + rpmsign --addsign + ``` + +7. Verify the RPM package signature. + + ```sh + rpm -Kv + ``` + + If the output shows "Header V4 ECDSA/SM3 Signature" and "OK," the signature verification is successful. diff --git a/docs/en/docs/SysCare/installing_SysCare.md b/docs/en/docs/SysCare/installing_SysCare.md index 9f80648d6b64c45069b40be475c896d28c48141a..197b0e837fd9dd3d4d3d0b3b0488c7db2201a3fd 100644 --- a/docs/en/docs/SysCare/installing_SysCare.md +++ b/docs/en/docs/SysCare/installing_SysCare.md @@ -12,7 +12,7 @@ This chapter describes how to install SysCare on openEuler. ### Prerequisites -1. openEuler 23.09 has been installed. +openEuler 24.03 LTS SP1 has been installed. ### Installing from Source diff --git a/docs/en/docs/SysCare/using_SysCare.md b/docs/en/docs/SysCare/using_SysCare.md index 080de365ec5a2fee57a81b2166bb9dd5b5a10433..e0b822bc5d10880fde9a505c47eccbd4ed101540 100644 --- a/docs/en/docs/SysCare/using_SysCare.md +++ b/docs/en/docs/SysCare/using_SysCare.md @@ -4,7 +4,7 @@ This chapter describes how to use SysCare on openEuler. ## Prerequisites -openEuler 22.03 LTS SP2 has been installed. +openEuler 24.03 LTS SP1 has been installed. ## Using SysCare CLI Tools diff --git a/docs/en/docs/Virtualization/Skylark.md b/docs/en/docs/Virtualization/Skylark.md index b844683707094f67ee57b094595aa48cb9612b3d..20abdd7168fc354deaca22e1f7eeeb07e7933dc6 100644 --- a/docs/en/docs/Virtualization/Skylark.md +++ b/docs/en/docs/Virtualization/Skylark.md @@ -1,12 +1,12 @@ # Skylark - [Skylark](#skylark) - - [Skylark Introduction](#skylark-introduction) - - [Architecture and Features](#architecture-and-features) - - [Skylark Installation](#skylark-installation) - - [Skylark Configuration](#skylark-configuration) - - [Skylark Usage](#skylark-usage) - - [Best Practices](#best-practices) + - [Skylark Introduction](#skylark-introduction) + - [Architecture and Features](#architecture-and-features) + - [Skylark Installation](#skylark-installation) + - [Skylark Configuration](#skylark-configuration) + - [Skylark Usage](#skylark-usage) + - [Best Practices](#best-practices) ## Skylark Introduction @@ -27,6 +27,7 @@ For details about how to better use the priority feature of Skylark in actual ap ### Overall Architecture The core class of Skylark is `QoSManager`. Class members include data collection class instances, QoS analysis class instances, QoS control class instances, and task scheduling class instances. + - `DataCollector`: data collection class. It has the `HostInfo` and `GuestInfo` members, which collect host information and VM information, respectively. - `PowerAnalyzer`: power consumption analysis class, which analyzes power consumption interference and low-priority VMs to be restricted. - `CpuController`: CPU bandwidth control class, which limits the CPU bandwidth of low-priority VMs. @@ -34,6 +35,7 @@ The core class of Skylark is `QoSManager`. Class members include data collection - `BackgroundScheduler`: task scheduling class, which periodically drives the preceding modules to continuously manage QoS. After checking the host environment, Skylark creates a daemon process. The daemon has a main scheduling thread and one or more job threads. + - The main scheduling thread is unique. It connects to libvirt, creates and initializes the `QosManager` class instance, and then starts to drive the Job threads. - Each Job thread periodically executes a QoS management task. @@ -69,6 +71,7 @@ During initialization, Skylark sets the **cpu.qos_level** field of the slice lev ### Hardware Requirements Processor architecture: AArch64 or x86_64 + - For Intel processors, the RDT function must be supported. - For the AArch64 architecture, only Kunpeng 920 processor is supported, and the BIOS must be upgraded to 1.79 or later to support the MPAM function. @@ -114,18 +117,18 @@ After the Skylark component is installed, you can modify the configuration file - **TDP_THRESHOLD** is a floating point number used to control the maximum power consumption of a VM. When the power consumption of the host exceeds **TDP * TDP_THRESHOLD**, a TDP hotspot occurs, and a power consumption control operation is triggered. The value ranges from 0.8 to 1, with the default value being 0.98. - **FREQ_THRESHOLD** is a floating point number used to control the minimum CPU frequency when a TDP hotspot occurs on the host. The value ranges from 0.8 to 1, with the default value being 0.98. - 1. When the frequency of some CPUs is lower than **max_freq * FREQ_THRESHOLD**, Skylark limits the CPU bandwidth of low-priority VMs running on these CPUs. - 2. If such a CPU does not exist, Skylark limits the CPU bandwidth of some low-priority VMs based on the CPU usage of low-priority VMs. + 1. When the frequency of some CPUs is lower than **max_freq * FREQ_THRESHOLD**, Skylark limits the CPU bandwidth of low-priority VMs running on these CPUs. + 2. If such a CPU does not exist, Skylark limits the CPU bandwidth of some low-priority VMs based on the CPU usage of low-priority VMs. - **QUOTA_THRESHOLD** is a floating point number used to control the CPU bandwidth that a restricted low-priority VM can obtain (CPU bandwidth before restriction x **QUOTA_THRESHOLD**). The value ranges from 0.8 to 1, with the default value being 0.9. - **ABNORMAL_THRESHOLD** is an integer used to control the number of low-priority VM restriction periods. The value ranges from 1 to 5, with the default value being 3. - 1. In each power consumption control period, if a low-priority VM is restricted, its number of remaining restriction periods is updated to **ABNORMAL_THRESHOLD**. - 2. Otherwise, its number of remaining restriction periods decreases by 1. When the number of remaining restriction periods of the VM is 0, the CPU bandwidth of the VM is restored to the value before the restriction. + 1. In each power consumption control period, if a low-priority VM is restricted, its number of remaining restriction periods is updated to **ABNORMAL_THRESHOLD**. + 2. Otherwise, its number of remaining restriction periods decreases by 1. When the number of remaining restriction periods of the VM is 0, the CPU bandwidth of the VM is restored to the value before the restriction. ### LLC/MB Interference Control -Skylark's interference control on LLC/MB depends on the RDT/MPAM function provided by hardware. For Intel x86_64 processors, **rdt=cmt,mbmtotal,mbmlocal,l3cat,mba** needs to be added to kernel command line parameters. For Kunpeng920 processors, **mpam=acpi** needs to be added to kernel command line parameters. +Skylark's interference control on LLC/MB depends on the RDT/MPAM function provided by hardware. For Intel x86_64 processors, **rdt=cmt,mbmtotal,mbmlocal,l3cat,mba** needs to be added to kernel command line parameters. For Kunpeng 920 processors, **mpam=acpi** needs to be added to kernel command line parameters. - **MIN_LLC_WAYS_LOW_VMS** is an integer used to control the number of LLC ways that can be accessed by low-priority VMs. The value ranges from 1 to 3, with the default value being 2. During initialization, Skylark limits the numfer of accessible LLC ways for low-priority VMs to this value. @@ -192,4 +195,4 @@ Skylark detects VM creation events, manages VMs of different priorities, and per To ensure optimal performance of high-priority VMs, you are advised to bind each vCPU of high-priority VMs to a physical CPU. To enable low-priority VMs to make full use of idle physical resources, you are advised to bind vCPUs of low-priority VMs to CPUs that are bound to high-priority VMs. -To ensure that low-priority VMs are scheduled when high-priority VMs occupy CPU resources for a long time, you are advised to reserve a small number of for low-priority VMs. \ No newline at end of file +To ensure that low-priority VMs are scheduled when high-priority VMs occupy CPU resources for a long time, you are advised to reserve a small number of for low-priority VMs. diff --git a/docs/en/docs/Virtualization/managing-devices.md b/docs/en/docs/Virtualization/managing-devices.md index b8ee22dc217551d458ee5d0c341394d1fc0c509e..38181b3196b55db1bf5eafe6fa66d525daba4d87 100644 --- a/docs/en/docs/Virtualization/managing-devices.md +++ b/docs/en/docs/Virtualization/managing-devices.md @@ -1,30 +1,29 @@ # Managing Devices - [Managing Devices](#managing-devices) - - [Configuring a PCIe Controller for a VM](#configuring-a-pcie-controller-for-a-vm) - - [Overview](#overview) - - [Configuring the PCIe Root, PCIe Root Port, and PCIe-PCI-Bridge](#configuring-the-pcie-root-pcie-root-port-and-pcie-pci-bridge) - - [Managing Virtual Disks](#managing-virtual-disks) - - [Managing vNICs](#managing-vnics) - - [Configuring a Virtual Serial Port](#configuring-a-virtual-serial-port) - - [Managing Device Passthrough](#managing-device-passthrough) - - [PCI Passthrough](#pci-passthrough) - - [SR-IOV Passthrough](#sr-iov-passthrough) - - [Managing VM USB](#managing-vm-usb) - - [Configuring USB Controllers](#configuring-usb-controllers) - - [Configuring a USB Passthrough Device](#configuring-a-usb-passthrough-device) - - [Storing Snapshots](#storing-snapshots) - - [Overview](#overview-7) - - [Procedure](#procedure-4) - - [Configuring Disk I/O Suspension](#configuring-disk-io-suspension) - - [Introduction](#introduction) - - [Overview](#overview-8) - - [Application Scenarios](#application-scenarios) - - [Precautions and Restrictions](#precautions-and-restrictions) - - [Disk I/O Suspension Configuration](#disk-io-suspension-configuration) - - [Using the QEMU CLI](#using-the-qemu-cli) - - [Using an XML Configuration File](#using-an-xml-configuration-file) - + - [Configuring a PCIe Controller for a VM](#configuring-a-pcie-controller-for-a-vm) + - [Overview](#overview) + - [Configuring the PCIe Root, PCIe Root Port, and PCIe-PCI-Bridge](#configuring-the-pcie-root-pcie-root-port-and-pcie-pci-bridge) + - [Managing Virtual Disks](#managing-virtual-disks) + - [Managing vNICs](#managing-vnics) + - [Configuring a Virtual Serial Port](#configuring-a-virtual-serial-port) + - [Managing Device Passthrough](#managing-device-passthrough) + - [PCI Passthrough](#pci-passthrough) + - [SR-IOV Passthrough](#sr-iov-passthrough) + - [Managing VM USB](#managing-vm-usb) + - [Configuring USB Controllers](#configuring-usb-controllers) + - [Configuring a USB Passthrough Device](#configuring-a-usb-passthrough-device) + - [Storing Snapshots](#storing-snapshots) + - [Overview](#overview-7) + - [Procedure](#procedure-4) + - [Configuring Disk I/O Suspension](#configuring-disk-io-suspension) + - [Introduction](#introduction) + - [Overview](#overview-8) + - [Application Scenarios](#application-scenarios) + - [Precautions and Restrictions](#precautions-and-restrictions) + - [Disk I/O Suspension Configuration](#disk-io-suspension-configuration) + - [Using the QEMU CLI](#using-the-qemu-cli) + - [Using an XML Configuration File](#using-an-xml-configuration-file) ## Configuring a PCIe Controller for a VM @@ -36,11 +35,11 @@ The NIC, disk controller, and PCIe pass-through devices in a VM must be mounted The VM PCIe controller is configured using the XML file. The **model** corresponding to PCIe root, PCIe root port, and PCIe-PCI-bridge in the XML file are **pcie-root**, **pcie-root-port**, and **pcie-to-pci-bridge**, respectively. -- Simplified configuration method +- Simplified configuration method Add the following contents to the XML file of the VM. Other attributes of the controller are automatically filled by libvirt. - ``` + ```xml @@ -51,11 +50,11 @@ The VM PCIe controller is configured using the XML file. The **model** corresp The **pcie-root** and **pcie-to-pci-bridge** occupy one **index** respectively. Therefore, the final **index** is the number of required **root ports + 1**. -- Complete configuration method +- Complete configuration method Add the following contents to the XML file of the VM: - ``` + ```xml @@ -76,38 +75,37 @@ The VM PCIe controller is configured using the XML file. The **model** corresp In the preceding contents: - - The **chassis** and **port** attributes of the root port must be in ascending order. Because a PCIe-PCI-bridge is inserted in the middle, the **chassis** number skips **2**, but the **port** numbers are still consecutive. - - The **address function** of the root port ranges from **0\*0** to **0\*7**. - - A maximum of eight functions can be mounted to each slot. When the slot is full, the slot number increases. + - The **chassis** and **port** attributes of the root port must be in ascending order. Because a PCIe-PCI-bridge is inserted in the middle, the **chassis** number skips **2**, but the **port** numbers are still consecutive. + - The **address function** of the root port ranges from **0\*0** to **0\*7**. + - A maximum of eight functions can be mounted to each slot. When the slot is full, the slot number increases. The complete configuration method is complex. Therefore, the simplified one is recommended. - ## Managing Virtual Disks ### Overview Virtual disk types include virtio-blk, virtio-scsi, and vhost-scsi. virtio-blk simulates a block device, and virtio-scsi and vhost-scsi simulate SCSI devices. -- virtio-blk: It can be used for common system disk and data disk. In this configuration, the virtual disk is presented as **vd\[a-z\]** or **vd\[a-z\]\[a-z\]** in the VM. -- virtio-scsi: It is recommended for common system disk and data disk. In this configuration, the virtual disk is presented as **sd\[a-z\]** or **sd\[a-z\]\[a-z\]** in the VM. -- vhost-scsi: It is recommended for the virtual disk that has high performance requirements. In this configuration, the virtual disk is presented as **sd\[a-z\]** or **sd\[a-z\]\[a-z\]** on the VM. +- virtio-blk: It can be used for common system disk and data disk. In this configuration, the virtual disk is presented as **vd\[a-z\]** or **vd\[a-z\]\[a-z\]** in the VM. +- virtio-scsi: It is recommended for common system disk and data disk. In this configuration, the virtual disk is presented as **sd\[a-z\]** or **sd\[a-z\]\[a-z\]** in the VM. +- vhost-scsi: It is recommended for the virtual disk that has high performance requirements. In this configuration, the virtual disk is presented as **sd\[a-z\]** or **sd\[a-z\]\[a-z\]** on the VM. ### Procedure For details about how to configure a virtual disk, see **VM Configuration** > **Network Devices**. This section uses the virtio-scsi disk as an example to describe how to attach and detach a virtual disk. -- Attach a virtio-scsi disk. +- Attach a virtio-scsi disk. Run the **virsh attach-device** command to attach the virtio-scsi virtual disk. - ``` + ```shell # virsh attach-device ``` The preceding command can be used to attach a disk to a VM online. The disk information is specified in the **attach-device.xml** file. The following is an example of the **attach-device.xml** file: - ``` + ```shell ### attach-device.xml ### @@ -120,17 +118,16 @@ For details about how to configure a virtual disk, see **VM Configuration** > ** The disk attached by running the preceding commands becomes invalid after the VM is shut down and restarted. If you need to permanently attach a virtual disk to a VM, run the **virsh attach-device** command with the **--config** parameter. -- Detach a virtio-scsi disk. +- Detach a virtio-scsi disk. If a disk attached online is no longer used, run the **virsh detach** command to dynamically detach it. - ``` + ```shell # virsh detach-device ``` **detach-device.xml** specifies the XML information of the disk to be detached, which must be the same as the XML information during dynamic attachment. - ## Managing vNICs ### Overview @@ -139,19 +136,19 @@ The vNIC types include virtio-net, vhost-net, and vhost-user. After creating a V ### Procedure -For details about how to configure a virtual NIC, see [3.2.4.2 Network Devices](#network-device). This section uses the vhost-net NIC as an example to describe how to attach and detach a vNIC. +For details about how to configure a virtual NIC, see [3.2.4.2 Network Devices](./vm-configuration.md#network-devices). This section uses the vhost-net NIC as an example to describe how to attach and detach a vNIC. -- Attach the vhost-net NIC. +- Attach the vhost-net NIC. Run the **virsh attach-device** command to attach the vhost-net vNIC. - ``` + ```shell # virsh attach-device ``` The preceding command can be used to attach a vhost-net NIC to a running VM. The NIC information is specified in the **attach-device.xml** file. The following is an example of the **attach-device.xml** file: - ``` + ```shell ### attach-device.xml ### @@ -164,17 +161,16 @@ For details about how to configure a virtual NIC, see [3.2.4.2 Network Devices] The vhost-net NIC attached using the preceding commands becomes invalid after the VM is shut down and restarted. If you need to permanently attach a vNIC to a VM, run the **virsh attach-device** command with the **--config** parameter. -- Detach the vhost-net NIC. +- Detach the vhost-net NIC. If a NIC attached online is no longer used, run the **virsh detach** command to dynamically detach it. - ``` + ```shell # virsh detach-device ``` **detach-device.xml** specifies the XML information of the vNIC to be detached, which must be the same as the XML information during dynamic attachment. - ## Configuring a Virtual Serial Port ### Overview @@ -185,9 +181,9 @@ In a virtualization environment, VMs and host machines need to communicate with The Linux VM serial port console is a pseudo terminal device connected to the host machine through the serial port of the VM. It implements interactive operations on the VM through the host machine. In this scenario, the serial port needs to be configured in the pty type. This section describes how to configure a pty serial port. -- Add the following virtual serial port configuration items under the **devices** node in the XML configuration file of the VM: +- Add the following virtual serial port configuration items under the **devices** node in the XML configuration file of the VM: - ``` + ```xml @@ -195,19 +191,18 @@ The Linux VM serial port console is a pseudo terminal device connected to the ho ``` -- Run the **virsh console** command to connect to the pty serial port of the running VM. +- Run the **virsh console** command to connect to the pty serial port of the running VM. - ``` + ```shell # virsh console ``` -- To ensure that no serial port message is missed, use the **--console** option to connect to the serial port when starting the VM. +- To ensure that no serial port message is missed, use the **--console** option to connect to the serial port when starting the VM. - ``` + ```shell # virsh start --console ``` - ## Managing Device Passthrough The device passthrough technology enables VMs to directly access physical devices. The I/O performance of VMs can be improved in this way. @@ -218,7 +213,7 @@ Currently, the VFIO passthrough is used. It can be classified into PCI passthrou PCI passthrough directly assigns a physical PCI device on the host to a VM. The VM can directly access the device. PCI passthrough uses the VFIO device passthrough mode. The PCI passthrough configuration file in XML format for a VM is as follows: -``` +```xml @@ -231,7 +226,6 @@ PCI passthrough directly assigns a physical PCI device on the host to a VM. The **Table 1** Device configuration items for PCI passthrough -

Parameter

Description

@@ -333,37 +327,39 @@ PCI passthrough directly assigns a physical PCI device on the host to a VM. The Single Root I/O Virtualization (SR-IOV) is a hardware-based virtualization solution. With the SR-IOV technology, a physical function (PF) can provide multiple virtual functions (VFs), and each VF can be directly assigned to a VM. This greatly improves hardware resource utilization and I/O performance of VMs. A typical application scenario is SR-IOV passthrough for NICs. With the SR-IOV technology, a physical NIC (PF) can function as multiple VF NICs, and then the VFs can be directly assigned to VMs. >![](./public_sys-resources/icon-note.gif) **NOTE:** ->- SR-IOV requires the support of physical hardware. Before using SR-IOV, ensure that the hardware device to be directly assigned supports SR-IOV and the device driver on the host OS works in SR-IOV mode. ->- The following describes how to query the NIC model: ->In the following command output, values in the first column indicate the PCI numbers of NICs, and **19e5:1822** indicates the vendor ID and device ID of the NIC. ->``` -># lspci | grep Ether ->05:00.0 Ethernet controller: Device 19e5:1822 (rev 45) ->07:00.0 Ethernet controller: Device 19e5:1822 (rev 45) ->09:00.0 Ethernet controller: Device 19e5:1822 (rev 45) ->0b:00.0 Ethernet controller: Device 19e5:1822 (rev 45) ->81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ->81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ->``` +- SR-IOV requires the support of physical hardware. Before using SR-IOV, ensure that the hardware device to be directly assigned supports SR-IOV and the device driver on the host OS works in SR-IOV mode. +- The following describes how to query the NIC model: + +In the following command output, values in the first column indicate the PCI numbers of NICs, and **19e5:1822** indicates the vendor ID and device ID of the NIC. + +```shell +# lspci | grep Ether +05:00.0 Ethernet controller: Device 19e5:1822 (rev 45) +07:00.0 Ethernet controller: Device 19e5:1822 (rev 45) +09:00.0 Ethernet controller: Device 19e5:1822 (rev 45) +0b:00.0 Ethernet controller: Device 19e5:1822 (rev 45) +81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) +81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) +``` #### Procedure To configure SR-IOV passthrough for a NIC, perform the following steps: -1. Enable the SR-IOV mode for the NIC. - 1. Ensure that VF driver support provided by the NIC supplier exists on the guest OS. Otherwise, VFs in the guest OS cannot work properly. - 2. Enable the SMMU/IOMMU support in the BIOS of the host OS. The enabling method varies depending on the servers of different vendors. For details, see the help documents of the servers. - 3. Configure the host driver to enable the SR-IOV VF mode. The following uses the Hi1822 NIC as an example to describe how to enable 16 VFs. +1. Enable the SR-IOV mode for the NIC. + 1. Ensure that VF driver support provided by the NIC supplier exists on the guest OS. Otherwise, VFs in the guest OS cannot work properly. + 2. Enable the SMMU/IOMMU support in the BIOS of the host OS. The enabling method varies depending on the servers of different vendors. For details, see the help documents of the servers. + 3. Configure the host driver to enable the SR-IOV VF mode. The following uses the Hi1822 NIC as an example to describe how to enable 16 VFs. - ``` + ```shell echo 16 > /sys/class/net/ethX/device/sriov_numvfs ``` -2. Obtain the PCI BDF information of PFs and VFs. - 1. Run the following command to obtain the NIC resource list on the current board: +2. Obtain the PCI BDF information of PFs and VFs. + 1. Run the following command to obtain the NIC resource list on the current board: - ``` + ```shell # lspci | grep Eth 03:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) 04:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (4*25GE) (rev 45) @@ -375,9 +371,9 @@ To configure SR-IOV passthrough for a NIC, perform the following steps: 7d:00.3 Ethernet controller: Huawei Technologies Co., Ltd. Device a221 (rev 20) ``` - 2. Run the following command to view the PCI BDF information of VFs: + 2. Run the following command to view the PCI BDF information of VFs: - ``` + ```shell # lspci | grep "Virtual Function" 03:00.1 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family Virtual Function (rev 45) 03:00.2 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family Virtual Function (rev 45) @@ -391,50 +387,50 @@ To configure SR-IOV passthrough for a NIC, perform the following steps: 03:01.2 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family Virtual Function (rev 45) ``` - 3. Select an available VF and write its configuration to the VM configuration file based on its BDF information. For example, the bus ID of the device **03:00.1** is **03**, its slot ID is **00**, and its function ID is **1**. + 3. Select an available VF and write its configuration to the VM configuration file based on its BDF information. For example, the bus ID of the device **03:00.1** is **03**, its slot ID is **00**, and its function ID is **1**. -3. Identify and manage the mapping between PFs and VFs. - 1. Identify VFs corresponding to a PF. The following uses PF 03.00.0 as an example: +3. Identify and manage the mapping between PFs and VFs. + 1. Identify VFs corresponding to a PF. The following uses PF 03.00.0 as an example: - ``` + ```shell # ls -l /sys/bus/pci/devices/0000\:03\:00.0/ ``` The following symbolic link information is displayed. You can obtain the VF IDs (virtfnX) and PCI BDF IDs based on the information. - 2. Identify the PF corresponding to a VF. The following uses VF 03:00.1 as an example: + 2. Identify the PF corresponding to a VF. The following uses VF 03:00.1 as an example: - ``` + ```shell # ls -l /sys/bus/pci/devices/0000\:03\:00.1/ ``` The following symbolic link information is displayed. You can obtain PCI BDF IDs of the PF based on the information. - ``` + ```shell lrwxrwxrwx 1 root root 0 Mar 28 22:44 physfn -> ../0000:03:00.0 ``` - 3. Obtain names of NICs corresponding to the PFs or VFs. For example: + 3. Obtain names of NICs corresponding to the PFs or VFs. For example: - ``` + ```shell # ls /sys/bus/pci/devices/0000:03:00.0/net eth0 ``` - 4. Set the MAC address, VLAN, and QoS information of VFs to ensure that the VFs are in the **Up** state before passthrough. The following uses VF 03:00.1 as an example. The PF is eth0 and the VF ID is **0**. + 4. Set the MAC address, VLAN, and QoS information of VFs to ensure that the VFs are in the **Up** state before passthrough. The following uses VF 03:00.1 as an example. The PF is eth0 and the VF ID is **0**. - ``` + ```shell # ip link set eth0 vf 0 mac 90:E2:BA:21:XX:XX #Sets the MAC address. # ifconfig eth0 up # ip link set eth0 vf 0 rate 100 #Sets the VF outbound rate, in Mbit/s. # ip link show eth0 #Views the MAC address, VLAN ID, and QoS information to check whether the configuration is successful. ``` -4. Mount the SR-IOV NIC to the VM. +4. Mount the SR-IOV NIC to the VM. When creating a VM, add the SR-IOV passthrough configuration item to the VM configuration file. - ``` + ```xml @@ -448,7 +444,6 @@ To configure SR-IOV passthrough for a NIC, perform the following steps: **Table 1** SR-IOV configuration options -

Parameter

Description

@@ -493,13 +488,14 @@ To configure SR-IOV passthrough for a NIC, perform the following steps: >Disabling the SR-IOV function: >To disable the SR-IOV function after the VM is stopped and no VF is in use, run the following command: >The following uses the Hi1822 NIC corresponding network interface name: eth0) as an example: - >``` + > + >```shell >echo 0 > /sys/class/net/eth0/device/sriov_numvfs >``` #### Configuring SR-IOV Passthrough for the HPRE Accelerator -The accelerator engine is a hardware acceleration solution provided by TaiShan 200 servers based on the Kunpeng 920 processors. The HPRE accelerator is used to accelerate SSL/TLS applications. It significantly reduces processor consumption and improves processor efficiency. +The accelerator engine is a hardware acceleration solution provided by TaiShan 200 servers. The HPRE accelerator is used to accelerate SSL/TLS applications. It significantly reduces processor consumption and improves processor efficiency. On the Kunpeng server, you need to pass through the VFs of the HPRE accelerator on the host to the VM for internal services of the VM. **Table 1** HPRE accelerator description @@ -513,14 +509,134 @@ On the Kunpeng server, you need to pass through the VFs of the HPRE accelerator | VF DeviceID | 0xA259 | | Maximum number of VFs | An HPRE PF supports a maximum of 63 VFs. | - >![](./public_sys-resources/icon-note.gif) **NOTE:** >When a VM is using a VF device, the driver on the host cannot be uninstalled, and the accelerator does not support hot swap. >VF operation (If **VFNUMS** is **0**, the VF is disabled, and **hpre_num** is used to identify a specific accelerator device): ->``` +> +>```shell >echo $VFNUMS > /sys/class/uacce/hisi_hpre-$hpre_num/device/sriov_numvfs >``` +### vDPA Passthrough + +#### Overview + +vDPA passthrough connects a device on a host to the vDPA framework, uses the vhost-vdpa driver to present a character device, and configures the character device for VMs to use. vDPA passthrough drives can serve as system or data drives for VMs and support hot expansion of data drives. + +vDPA passthrough provides the similar I/O performance as VFIO passthrough, provides flexibility of VirtIO devices, and supports live migration of vDPA passthrough devices. + +With the SR-IOV solution, vDPA passthrough can virtualize a physical NIC (PF) into multiple NICs (VFs), and then connect the VFs to the vDPA framework for VMs to use. + +#### Procedure + +To configure vDPA passthrough, perform the following steps as user **root**: + +1. Create and configure VFs. For details, see steps 1 to 3 in SR-IOV passthrough. The following uses **virtio-net** devices as an example (**08:00.6** and **08:00.7** are PFs, and the others are created VFs): + + ```shell + # lspci | grep -i Eth | grep Virtio + 08:00.6 Ethernet controller: Virtio: Virtio network device + 08:00.7 Ethernet controller: Virtio: Virtio network device + 08:01.1 Ethernet controller: Virtio: Virtio network device + 08:01.2 Ethernet controller: Virtio: Virtio network device + 08:01.3 Ethernet controller: Virtio: Virtio network device + 08:01.4 Ethernet controller: Virtio: Virtio network device + 08:01.5 Ethernet controller: Virtio: Virtio network device + 08:01.6 Ethernet controller: Virtio: Virtio network device + 08:01.7 Ethernet controller: Virtio: Virtio network device + 08:02.0 Ethernet controller: Virtio: Virtio network device + 08:02.1 Ethernet controller: Virtio: Virtio network device + 08:02.2 Ethernet controller: Virtio: Virtio network device + ``` + +2. Unbind the VF drivers and bind the vDPA driver of the hardware vendor. + + ```shell + echo 0000:08:01.1 > /sys/bus/pci/devices/0000\:08\:01.1/driver/unbind + echo 0000:08:01.2 > /sys/bus/pci/devices/0000\:08\:01.2/driver/unbind + echo 0000:08:01.3 > /sys/bus/pci/devices/0000\:08\:01.3/driver/unbind + echo 0000:08:01.4 > /sys/bus/pci/devices/0000\:08\:01.4/driver/unbind + echo 0000:08:01.5 > /sys/bus/pci/devices/0000\:08\:01.5/driver/unbind + echo -n "1af4 1000" > /sys/bus/pci/drivers/vender_vdpa/new_id + ``` + +3. After vDPA devices are bound, you can run the `vdpa` command to query the list of devices managed by vDPA. + + ```shell + # vdpa mgmtdev show + pci/0000:08:01.1: + supported_classes net + pci/0000:08:01.2: + supported_classes net + pci/0000:08:01.3: + supported_classes net + pci/0000:08:01.4: + supported_classes net + pci/0000:08:01.5: + supported_classes net + ``` + +4. After the vDPA devices are created, create the vhost-vDPA devices. + + ```shell + vdpa dev add name vdpa0 mgmtdev pci/0000:08:01.1 + vdpa dev add name vdpa1 mgmtdev pci/0000:08:01.2 + vdpa dev add name vdpa2 mgmtdev pci/0000:08:01.3 + vdpa dev add name vdpa3 mgmtdev pci/0000:08:01.4 + vdpa dev add name vdpa4 mgmtdev pci/0000:08:01.5 + ``` + +5. After the vhost-vDPA devices are created, you can run the `vdpa` command to query the vDPA device list or run the `libvirt` command to query the vhost-vDPA device information. + + ```shell + # vdpa dev show + vdpa0: type network mgmtdev pci/0000:08:01.1 vendor_id 6900 max_vqs 3 max_vq_size 256 + vdpa1: type network mgmtdev pci/0000:08:01.2 vendor_id 6900 max_vqs 3 max_vq_size 256 + vdpa2: type network mgmtdev pci/0000:08:01.3 vendor_id 6900 max_vqs 3 max_vq_size 256 + vdpa3: type network mgmtdev pci/0000:08:01.4 vendor_id 6900 max_vqs 3 max_vq_size 256 + vdpa4: type network mgmtdev pci/0000:08:01.5 vendor_id 6900 max_vqs 3 max_vq_size 256 + + # virsh nodedev-list vdpa + vdpa_vdpa0 + vdpa_vdpa1 + vdpa_vdpa2 + vdpa_vdpa3 + vdpa_vdpa4 + + # virsh nodedev-dumpxml vdpa_vdpa0 + + vdpa_vdpa0 + /sys/devices/pci0000:00/0000:00:0c.0/0000:08:01.1/vdpa0 + pci_0000_08_01_1 + + vhost_vdpa + + + /dev/vhost-vdpa-0 + + + ``` + +6. Mount a vDPA device to the VM. + + When creating a VM, add the item for the vDPA passthrough device to the VM configuration file: + + ```xml + + + + + + ``` + + **Table 4** vDPA configuration description + + | Parameter | Description | Value | + | ------------------ | ---------------------------------------------------- | ----------------- | + | hostdev.source.dev | Path of the vhost-vDPA character device on the host. | /dev/vhost-vdpa-x | + + >![](./public_sys-resources/icon-note.gif) **NOTE:** + >The procedures of creating and configuring VFs and binding the vDPA drivers vary with the design of hardware vendors. Follow the procedure of the corresponding vendor. ## Managing VM USB @@ -532,19 +648,19 @@ To facilitate the use of USB devices such as USB key devices and USB mass storag A USB controller is a virtual controller that provides specific USB functions for USB devices on VMs. To use USB devices on a VM, you must configure USB controllers for the VM. Currently, openEuler supports the following types of USB controllers: -- Universal host controller interface (UHCI): also called the USB 1.1 host controller specification. -- Enhanced host controller interface (EHCI): also called the USB 2.0 host controller specification. -- Extensible host controller interface (xHCI): also called the USB 3.0 host controller specification. +- Universal host controller interface (UHCI): also called the USB 1.1 host controller specification. +- Enhanced host controller interface (EHCI): also called the USB 2.0 host controller specification. +- Extensible host controller interface (xHCI): also called the USB 3.0 host controller specification. #### Precautions -- The host server must have USB controller hardware and modules that support USB 1.1, USB 2.0, and USB 3.0 specifications. -- You need to configure USB controllers for the VM by following the order of USB 1.1, USB 2.0, and USB 3.0. -- An xHCI controller has eight ports and can be mounted with a maximum of four USB 3.0 devices and four USB 2.0 devices. An EHCI controller has six ports and can be mounted with a maximum of six USB 2.0 devices. A UHCI controller has two ports and can be mounted with a maximum of two USB 1.1 devices. -- On each VM, only one USB controller of the same type can be configured. -- USB controllers cannot be hot swapped. -- If the USB 3.0 driver is not installed on a VM, the xHCI controller may not be identified. For details about how to download and install the USB 3.0 driver, refer to the official description provided by the corresponding OS distributor. -- To ensure the compatibility of the OS, set the bus ID of the USB controller to **0** when configuring a USB tablet for the VM. The tablet is mounted to the USB 1.1 controller by default. +- The host server must have USB controller hardware and modules that support USB 1.1, USB 2.0, and USB 3.0 specifications. +- You need to configure USB controllers for the VM by following the order of USB 1.1, USB 2.0, and USB 3.0. +- An xHCI controller has eight ports and can be mounted with a maximum of four USB 3.0 devices and four USB 2.0 devices. An EHCI controller has six ports and can be mounted with a maximum of six USB 2.0 devices. A UHCI controller has two ports and can be mounted with a maximum of two USB 1.1 devices. +- On each VM, only one USB controller of the same type can be configured. +- USB controllers cannot be hot swapped. +- If the USB 3.0 driver is not installed on a VM, the xHCI controller may not be identified. For details about how to download and install the USB 3.0 driver, refer to the official description provided by the corresponding OS distributor. +- To ensure the compatibility of the OS, set the bus ID of the USB controller to **0** when configuring a USB tablet for the VM. The tablet is mounted to the USB 1.1 controller by default. #### Configuration Methods @@ -552,21 +668,21 @@ The following describes the configuration items of USB controllers for a VM. You The configuration item of the USB 1.1 controller (UHCI) in the XML configuration file is as follows: -``` +```xml ``` The configuration item of the USB 2.0 controller (EHCI) in the XML configuration file is as follows: -``` +```xml ``` The configuration item of the USB 3.0 controller (xHCI) in the XML configuration file is as follows: -``` +```xml ``` @@ -579,10 +695,10 @@ After USB controllers are configured for a VM, a physical USB device on the host #### Precautions -- A USB device can be assigned to only one VM. -- A VM with a USB passthrough device does not support live migration. -- VM creation fails if no USB passthrough devices exist in the VM configuration file. -- Forcibly hot removing a USB storage device that is performing read or write operation may damage files in the USB storage device. +- A USB device can be assigned to only one VM. +- A VM with a USB passthrough device does not support live migration. +- VM creation fails if no USB passthrough devices exist in the VM configuration file. +- Forcibly hot removing a USB storage device that is performing read or write operation may damage files in the USB storage device. #### Configuration Description @@ -590,7 +706,7 @@ The following describes the configuration items of a USB device for a VM. Description of the USB device in the XML configuration file: -``` +```xml
@@ -599,23 +715,23 @@ Description of the USB device in the XML configuration file: ``` -- **
**: *m_ indicates the USB bus address on the host, and _n* indicates the device ID. -- **
**: indicates that the USB device is to be mounted to the USB controller specified on the VM. *x_ indicates the controller ID, which corresponds to the index number of the USB controller configured on the VM. _y* indicates the port address. When configuring a USB passthrough device, you need to set this parameter to ensure that the controller to which the device is mounted is as expected. +- **
**: *m_ indicates the USB bus address on the host, and _n* indicates the device ID. +- **
**: indicates that the USB device is to be mounted to the USB controller specified on the VM. *x_ indicates the controller ID, which corresponds to the index number of the USB controller configured on the VM. _y* indicates the port address. When configuring a USB passthrough device, you need to set this parameter to ensure that the controller to which the device is mounted is as expected. #### Configuration Methods To configure USB passthrough, perform the following steps: -1. Configure USB controllers for the VM. For details, see [Configuring USB Controllers](#configuring-usb-controllers). -2. Query information about the USB device on the host. +1. Configure USB controllers for the VM. For details, see [Configuring USB Controllers](#configuring-usb-controllers). +2. Query information about the USB device on the host. Run the **lsusb** command (the **usbutils** software package needs to be installed) to query the USB device information on the host, including the bus address, device address, device vendor ID, device ID, and product description. For example: - ``` + ```shell # lsusb ``` - ``` + ```text Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub @@ -631,37 +747,35 @@ To configure USB passthrough, perform the following steps: Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub ``` -3. Prepare the XML description file of the USB device. Before hot removing the device, ensure that the USB device is not in use. Otherwise, data may be lost. -4. Run the hot swapping commands. +3. Prepare the XML description file of the USB device. Before hot removing the device, ensure that the USB device is not in use. Otherwise, data may be lost. +4. Run the hot swapping commands. Take a VM whose name is **openEulerVM** as an example. The corresponding configuration file is **usb.xml**. - - Hot adding of the USB device takes effect only for the current running VM. After the VM is restarted, hot add the USB device again. + - Hot adding of the USB device takes effect only for the current running VM. After the VM is restarted, hot add the USB device again. - ``` + ```shell # virsh attach-device openEulerVM usb.xml --live ``` - - Complete persistency configurations for hot adding of the USB device. After the VM is restarted, the USB device is automatically assigned to the VM. + - Complete persistency configurations for hot adding of the USB device. After the VM is restarted, the USB device is automatically assigned to the VM. - ``` + ```shell # virsh attach-device openEulerVM usb.xml --config ``` - - Hot removing of the USB device takes effect only for the current running VM. After the VM is restarted, the USB device with persistency configurations is automatically assigned to the VM. + - Hot removing of the USB device takes effect only for the current running VM. After the VM is restarted, the USB device with persistency configurations is automatically assigned to the VM. - ``` + ```shell # virsh detach-device openEulerVM usb.xml --live ``` - - Complete persistency configurations for hot removing of the USB device. + - Complete persistency configurations for hot removing of the USB device. - ``` + ```shell # virsh detach-device openEulerVM usb.xml --config ``` - - ## Storing Snapshots ### Overview @@ -675,27 +789,25 @@ The VM system may be damaged due to virus damage, system file deletion by mistak To create VM storage snapshots, perform the following steps: -1. Log in to the host and run the **virsh domblklist** command to query the disk used by the VM. +1. Log in to the host and run the **virsh domblklist** command to query the disk used by the VM. - ``` + ```shell # virsh domblklist openEulerVM Target Source --------------------------------------------- vda /mnt/openEuler-image.qcow2 ``` +2. Run the following command to create the VM disk snapshot **openEuler-snapshot1.qcow2**: -1. Run the following command to create the VM disk snapshot **openEuler-snapshot1.qcow2**: - - ``` + ```shell # virsh snapshot-create-as --domain openEulerVM --disk-only --diskspec vda,snapshot=external,file=/mnt/openEuler-snapshot1.qcow2 --atomic Domain snapshot 1582605802 created ``` +3. Run the following command to query disk snapshots: -1. Run the following command to query disk snapshots: - - ``` + ```shell # virsh snapshot-list openEulerVM Name Creation Time State --------------------------------------------------------- @@ -736,11 +848,11 @@ A cloud disk that may cause storage plane link disconnection is used as the back - After a storage fault occurs, the following problems cannot be solved although disk I/O suspension occurs: - 1. Failed to execute advanced storage features. + 1. Failed to execute advanced storage features. - Advanced features include: virtual disk hot swap, virtual disk creation, VM startup, VM shutdown, VM forcible shutdown, VM hibernation, VM wakeup, VM storage live migration, VM storage live migration cancellation, VM storage snapshot creation, VM storage snapshot combination, VM disk capacity query, online disk capacity expansion, virtual CD-ROM drive insertion, and CD-ROM drive ejection from the VM. + Advanced features include: virtual disk hot swap, virtual disk creation, VM startup, VM shutdown, VM forcible shutdown, VM hibernation, VM wakeup, VM storage live migration, VM storage live migration cancellation, VM storage snapshot creation, VM storage snapshot combination, VM disk capacity query, online disk capacity expansion, virtual CD-ROM drive insertion, and CD-ROM drive ejection from the VM. - 2. Failed to execute the VM lifecycle. + 2. Failed to execute the VM lifecycle. - When a live migration is initiated for a VM configured with disk I/O suspension, the disk I/O suspension configuration must be the same as that of the source host in the XML configuration of the destination disk. diff --git a/docs/en/docs/desktop/DDE-user-guide.md b/docs/en/docs/desktop/DDE-user-guide.md old mode 100755 new mode 100644 index 6e22fb2e8f840de4b402788792ca91b665d7c112..1e3f687c920b5adee0a4424ecb5f898cba7688cf --- a/docs/en/docs/desktop/DDE-user-guide.md +++ b/docs/en/docs/desktop/DDE-user-guide.md @@ -94,7 +94,7 @@ All the texts, pictures and documents cut and copied by the current user after l 4. Click![close](./figures/icon57-o.svg)to delete the current content and click **Clear All** to clear the clipboard. - ![1|clipboard](./figures/40.png) + ![1|clipboard](./figures/40.png) ## Dock @@ -379,9 +379,9 @@ Set screen resolution, brightness, direction and display scaling properly to hav 1. On the homepage of Control Center, click ![display_normal](./figures/icon72-o.svg). 2. Click **Brightness**. - - Drag the slider to set screen brightness. - - Switch on **Night Shift**, the screen hue will be auto-adjusted according to your location. - - Switch on **Auto Brightness**, the monitor will change the brightness automatically according to ambient light (shown only if PC has a light sensor). + - Drag the slider to set screen brightness. + - Switch on **Night Shift**, the screen hue will be auto-adjusted according to your location. + - Switch on **Auto Brightness**, the monitor will change the brightness automatically according to ambient light (shown only if PC has a light sensor). ##### Change Refresh Rate @@ -403,9 +403,9 @@ Expand your desktop by multiple screens! Use VGA/HDMI/DP cable to connect your c 1. On the homepage of Control Center, click ![display_normal](./figures/icon72-o.svg). 2. Click **Multiple Displays**. 3. Select a display mode: - - **Duplicate**: display the same image on other screens. - - **Extend**: expand the desktop across the screens. - - **Customize**: customize the display settings for multiple screens. + - **Duplicate**: display the same image on other screens. + - **Extend**: expand the desktop across the screens. + - **Customize**: customize the display settings for multiple screens. In multiple displays, press **Super** + **P** to show its OSD. @@ -591,18 +591,17 @@ Set your speaker and microphone properly to make you hear more comfortable and m 2. Click **Output** to: - - Select output device type from the dropdown list after **Output Device**. - - - Drag the slider to adjust output volume and left/right balance. - - Switch on **Volume Boost**, the volume could be adjustable from 0~150% (the former range is 0~100%). + - Select output device type from the dropdown list after **Output Device**. + - Drag the slider to adjust output volume and left/right balance. + - Switch on **Volume Boost**, the volume could be adjustable from 0~150% (the former range is 0~100%). #### Input 1. On the homepage of Control Center, click ![sound_normal](./figures/icon116-o.svg). 2. Click **Input** to: - - Select input device type from the dropdown list after **Input Device**. - - Adjust input volume by dragging the slider. - - You can enable **Automatic Noise Suppression** by clicking the button after "Automatic Noise Suppression". + - Select input device type from the dropdown list after **Input Device**. + - Adjust input volume by dragging the slider. + - You can enable **Automatic Noise Suppression** by clicking the button after "Automatic Noise Suppression". > ![tips](./figures/icon125-o.svg)Tips: *Usually, you need to turn up the input volume to make sure that you can hear the sound of the sound source, but the volume should not be too high, because it will cause distortion of the sound. Here is how to set input volume: Speak to your microphone at a normal volume and view "Input Level". If the indicator changes obviously according to the volume, then the input volume is at a proper level.* @@ -650,8 +649,8 @@ Note that the auto-sync function will be disabled after changing date and time m 1. On the homepage of Control Center, click ![time](./figures/icon124-o.svg). 2. Click **Time Settings**. - - Switch on/off **Auto Sync**. - - Enter the correct date and time. + - Switch on/off **Auto Sync**. + - Enter the correct date and time. 3. Click **Confirm**. #### Set Time Format @@ -812,7 +811,7 @@ The shortcut list includes all shortcuts in the system. View, modify and customi 6. After being successfully added, click **Edit**. 7. Click ![delete](./figures/icon71-o.svg) to delete the custom shortcut. -> ![tips](./figures/icon125-o.svg)Tips: *To change the shortcut, click it and press a new shortcut to change it directly. To edit the name and command of the custom shortcut, click**Edit ** > ![edit](./figures/icon75-o.svg) near the shortcut name to enter the shortcut settings.* +> ![tips](./figures/icon125-o.svg)Tips: *To change the shortcut, click it and press a new shortcut to change it directly. To edit the name and command of the custom shortcut, click **Edit** > ![edit](./figures/icon75-o.svg) near the shortcut name to enter the shortcut settings.* ### System Info diff --git a/docs/en/docs/desktop/UKUI-user-guide.md b/docs/en/docs/desktop/UKUI-user-guide.md old mode 100755 new mode 100644 index f5f2b830eb0d8e8bdd7f002e8d667dda939ce282..fc9326180472fdd3b4d1eaa9f482629f1208d832 --- a/docs/en/docs/desktop/UKUI-user-guide.md +++ b/docs/en/docs/desktop/UKUI-user-guide.md @@ -1,30 +1,29 @@ # UKUI Desktop Environment ## Overview + Desktop Environment is the basis for the user's operation on the graphical interface, and provides multiple functions including taskbar, start menu, etc. The main interface is shown in figure below. ![Fig. 1 Desktop main interface-big](./figures/1.png) -
- ## Desktop ### Desktop’s Icons -The system places three icons Computer, Recycle Bin and Personal by default, and double click the left mouse button to open the page. The functions are shown in table below. -| Icon | Description | +The system places three icons Computer, Recycle Bin and Personal by default, and double click the left mouse button to open the page. The functions are shown in table below. + +| Icon | Description | | :------------ | :------------ | | ![](./figures/icon1.png) | Computer: Show the drives and hardwares connected to the machine| | ![](./figures/icon2.png) | Recycle Bin: Show documents that have been diverted| | ![](./figures/icon3.png) | Personal: Show personal home directory| - -
In addition, right-clicking "Computer" and selecting "Properties", it can show the current system version, kernel version, activation and other related information. ![Fig. 2 "Computer" - "Properties"-big](./figures/2.png) ### Right-click Menu + Right-click on the desktop blank and a menu appears as shown in figure below, providing the users with some shortcut features. ![Fig. 3 Right-click Menu](./figures/3.png) @@ -37,11 +36,10 @@ Some of the options are described in table below. | View type | Four view types are available: small icon, medium icon, large icon, super large icon | | Sort by | Four ways to arrange documents according to name, type of document, size and date of modification | -
- ## Taskbar ### Basic Function + Taskbar is located at the bottom and includes the Start Menu, Multi View Switch, File Browser, Firefox Web Browser, WPS, and Tray Menu. ![Fig. 4 Taskbar](./figures/4.png) @@ -58,12 +56,14 @@ Taskbar is located at the bottom and includes the Start Menu, Multi View Switch, |Show Desktop| The button is on the far right. Minimize all windows on the desktop and return to the desktop; Clicking again will restore the windows | #### Multi View Switch + Click the icon "![](./figures/icon10-o.png)" on the taskbar to enter the interface shown in figure below, and select the operation area that users need to work on at the moment in multiple work areas. ![Fig. 5 Multi View Switch-big](./figures/5.png) #### Preview Window -Users move the mouse over the app icon in the taskbar, and then a small preview window will be shown if this app has already been opened. + +Users move the mouse over the app icon in the taskbar, and then a small preview window will be shown if this app has already been opened. Hover over the specified window as shown below for hover state, the window will be slightly fuzzy glass effect (left), the rest of the window as default Status (right). @@ -74,14 +74,16 @@ Users can close the application by right-clicking on the app icon in the taskbar ![Fig. 7 Taskbar - Right-click Preview](./figures/7.png) #### Sidebar -The sidebar is located at the right of the entire desktop. Click the icon "![](./figures/icon11-o.png)" in the taskbar tray menu to open the storage menu, and click the icon "![](./figures/icon12-o.png)" in Sidebar to pop up the sidebar as shown in figure below. + +The sidebar is located at the right of the entire desktop. Click the icon "![](./figures/icon11-o.png)" in the taskbar tray menu to open the storage menu, and click the icon "![](./figures/icon12-o.png)" in Sidebar to pop up the sidebar as shown in figure below. The sidebar consists of two parts: Notification Center, Clipboard and Widget. ![Fig. 8 Sidebar without message status-big](./figures/8.png) ##### Notification Center -Notification center will display a list of recent important and newest information. + +Notification center will display a list of recent important and newest information. Select "Clear" in the upper right corner to clear the list of information; Select "Setting" in the upper right corner to go to the notification settings in the control center, and users can set which applications can show information and the quantity of information. @@ -96,6 +98,7 @@ Icon "![](./figures/icon13-o.png)" at the top right corner of the sidebar can st ![Fig. 11 Message Organizer](./figures/11.png) ##### Clipboard + Clipboard can save the contents those were recently selected to copy or cut, and users can operate them by using the icons in Table. ![Fig. 12 Clipboard](./figures/12.png) @@ -107,38 +110,41 @@ Clicking "![](./figures/icon15-o.png)", users can edit the the contents of the c | Icon | Description | Icon | Description | | :------------ | :------------ | :------------ | :------------ | | ![](./figures/icon16.png) | Copy the content | ![](./figures/icon18.png) | Edit the content | -| ![](./figures/icon17.png) | Delete the content | | | - -
+| ![](./figures/icon17.png) | Delete the content | | | The second label of the clipboard is the small plug-in that contains alarm clock, notebook, user feedback. ![Fig. 14 Plug-in](./figures/14.png) #### Tray Menu + ##### Storage Menu -Click "![](./figures/icon19-o.png)" at the tray menu to open the storage menu. + +Click "![](./figures/icon19-o.png)" at the tray menu to open the storage menu. It contains Kylin Weather, Input Method, Bluetooth, USB, etc. ![Fig. 15 Storage Menu](./figures/15.png) ##### Input Method + The taskbar input method defaults to Sogou input method. Use the shortcut key "Ctrl+Space" to switch it out, and the "Shift" key to switch between Chinese and English modes. ![Fig. 16 Input Method](./figures/16.png) ##### USB -When the USB is inserted into the computer, it will be automatically read the data inside. -Click "![](./figures/icon26-o.png)" to open the window as shown in figure below. +When the USB is inserted into the computer, it will be automatically read the data inside. + +Click "![](./figures/icon26-o.png)" to open the window as shown in figure below. When users need to umount the USB, please click the icon "![](./figures/icon27-o.png)". ![Fig. 17 The status of USB](./figures/17.png) ##### Power Supply -Click the icon "![](./figures/icon28-o.png)": + +Click the icon "![](./figures/icon28-o.png)": When no power supply is detected. @@ -148,7 +154,7 @@ When power supply is detected. ![Fig. 19 Have Power Supply](./figures/19.png) -Users right-click the icon "![](./figures/icon30-o.png)" of power manager to open the power setting menu. +Users right-click the icon "![](./figures/icon30-o.png)" of power manager to open the power setting menu. It provides two setting options: adjust screen brightness, and set power and sleep. @@ -159,17 +165,16 @@ If the power manager pops up a"low battery" window, users can click to turn on t ![Fig. 21 Power Saving Mode](./figures/21.png) ##### Network + Users can choose wired or wireless network connections by clicking the icon "![](./figures/icon31-o.png)" of network manager. | Icon | Description | Icon | Description | | :------------ | :------------ | :------------ | :------------ | -|![](./figures/icon32.png)| Connected |![](./figures/icon37.png)| Unconnected | -|![](./figures/icon33.png)| Connection limited |![](./figures/icon38.png)| Locked | -|![](./figures/icon34.png)| Connecting |![](./figures/icon39.png)| Wifi connected | -|![](./figures/icon35.png)| Wifi unconnected |![](./figures/icon40.png)| Wificonnection limited | -|![](./figures/icon36.png)| Wifi locked |![](./figures/icon41.png)| Wifi connecting | - -
+|![](./figures/icon32.png)| Connected |![](./figures/icon37.png)| Unconnected | +|![](./figures/icon33.png)| Connection limited |![](./figures/icon38.png)| Locked | +|![](./figures/icon34.png)| Connecting |![](./figures/icon39.png)| Wifi connected | +|![](./figures/icon35.png)| Wifi unconnected |![](./figures/icon40.png)| Wificonnection limited | +|![](./figures/icon36.png)| Wifi locked |![](./figures/icon41.png)| Wifi connecting | ![Fig. 22 Network Connection](./figures/22.png) @@ -193,6 +198,7 @@ Users can choose wired or wireless network connections by clicking the icon "![] ![Fig. 26 Network Setting](./figures/26.png) ##### Volume + Click the icon "![](./figures/icon43-o.png)" to open the volume window, and there provides three modes. - Mini Mode @@ -211,6 +217,7 @@ Click the icon "![](./figures/icon43-o.png)" to open the volume window, and ther ![Fig. 29 According to Application List](./figures/29.png) ##### Calendar + Click the date&time on the taskbar to open the calendar window. Users can view the day's information by filtering the year, month, day. The date will be displayed in large letters, with the time, the week, the festival,and the lunar calendar. Taboos can be seen by checking. @@ -218,19 +225,21 @@ Users can view the day's information by filtering the year, month, day. The date ![Fig. 30 Calendar-big](./figures/30.png) ##### Night Mode + Click the icon "![](./figures/icon44-o.png)" on the Taskbar and then the system changes to the night mode. #### Advanced Setting + Right-click the Taskbar to open the menu. ![Fig. 31 Right-Clicking Menu](./figures/31.png) Users can set the lauserst of taskbar according to "Taskbar Settings". -
+## Window + +### Window Manager -## Window -### Window Manager The functions provided as shown in Table. | Function | Description | @@ -242,54 +251,60 @@ The functions provided as shown in Table. | Drag and Drop | Long press the left mouse button at the title bar to move the window to any position | | Resize | Move the mouse to the corner of the window and long press the left button to resize the window | -
- ### Window Switch -There are three ways to switch windows: -* Click the window title on the Taskbar. +There are three ways to switch windows: -* Click the different window at the desktop. +- Click the window title on the Taskbar. -* Use shortcut keys < Alt > + < Tab >. +- Click the different window at the desktop. -
+- Use shortcut keys **Alt** + **Tab**. ## Start Menu + ### Basic Function -Click the button to open the "Start Menu". + +Click the button to open the "Start Menu". It provides sliding bar. ![Fig. 32 Start Menu](./figures/32.png) #### Category Menu at right side + When the mouse is over the right side of the start menu, it will appear a pre-expanded cue bar. Clicking to expand, and then three categories are showing at the right side by default: "Common Software", "Alphabetical Category", and "Functional category". -* All Software: List all software, recently used software will be displayed on the top of this page. +- All Software: List all software, recently used software will be displayed on the top of this page. -* Alphabetical Category: List all software by first letter. +- Alphabetical Category: List all software by first letter. -* Functional category: List all software by their functions. +- Functional category: List all software by their functions. Users can click the button at top right corner to view fullscreen menu mode. ![Fig. 33 Fullscreen Menu-big](./figures/33.png) #### Function Button at right side + It provides User Avatar, Computer, Control Center and Shutdown four options. ##### User Avatar + Click "![](./figures/icon45-o.png)" to view user's information. ##### Computer + Click "![](./figures/icon46-o.png)" to open personal home folder ##### Control Center + Click "![](./figures/icon47-o.png)" to go to the control center. ##### Shutdown + ###### Lock Screen + When users do not need to use the computer temporarily, the lock screen can be selected (without affecting the current running state of the system) to prevent misoperations. And input the password to re-enter the system. The system will automatically lock the screen after a period of idle time by default. @@ -297,11 +312,13 @@ When users do not need to use the computer temporarily, the lock screen can be s ![Fig. 34 Lock Screen-big](./figures/34.png) ###### Switch Users & Log Out + When users want to select another user to log in using the computer, users can select "Log out" or "Switch user". At this point, the system will close all running applications; Therefore, please save the current jobs before performing this action. ###### Shutdown & Reboot + There are two ways: 1)"Start Menu" > "Power" > "Shutdown" @@ -319,6 +336,7 @@ The system will shutdown or reboot immediately without popping up the dialog box Right-clicking Start Menu, it provides lock screen, switch user, log out, reboot, and shutdown five shortcut options. ### Applications + Users can search apps in the search box by key words. As shown in figure below, the result will show up automatically with the input. ![Fig. 36 Search Apps](./figures/36.png) @@ -336,31 +354,27 @@ The options are described in table below. | Add to Desktop Shortcut |Generate shortcut icon for the application on the desktop| | Uninstall |Remove the application| -
- ## FAQ -### Can’t login to the system after locking the screen? +### Cannot login to the system after locking the screen -- Switch to character terminal by < Ctrl + Alt + F2 >. +- Switch to character terminal by **Ctrl + Alt + F2**. - Input the user-name and passwd to login to the system. - Do "sudo rm -rf ~/.Xauthority". -- Switch to graphical interface by < Ctrl + Alt + F1 >, and input the passwd. +- Switch to graphical interface by **Ctrl + Alt + F1**, and input the passwd. -
+## Appendix -## Appendix ### Shortcut Key -|Shortcut Key|Function| -| :------ | :----- -| F5 | Refresh the desktop | -| F1 | Open the user-guide| -| Alt + Tab | Switch the window | -| win | Open the Start Menu | -| Ctrl + Alt + L | Lock Screen | -| Ctrl + Alt + Delete | Log out | - +| Shortcut Key | Function | +| :------------------ | :------------------ | +| F5 | Refresh the desktop | +| F1 | Open the user-guide | +| Alt + Tab | Switch the window | +| win | Open the Start Menu | +| Ctrl + Alt + L | Lock Screen | +| Ctrl + Alt + Delete | Log out | diff --git a/docs/en/docs/epkg/epkg-user-guide.md b/docs/en/docs/epkg/epkg-user-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..d88d3e2949955d7fb4be45cf9389bb966ea70394 --- /dev/null +++ b/docs/en/docs/epkg/epkg-user-guide.md @@ -0,0 +1,320 @@ +# epkg User Guide + +## Introduction + +This document explains how to initialize the working environment for the epkg package manager and how to use its basic features. All operation results in this document are demonstrated using a non-root user as an example. +Note: Currently, epkg packages are only compatible with the AArch64 architecture, and support for other architectures will be expanded in the future. + +## Quick Start + +The following examples demonstrate how to install different versions of software packages. + +```bash +# Install epkg using curl. +# During installation, you can choose between user/global installation modes to install epkg for the current user or all users. +# Only the root user can use the global installation mode. +wget https://repo.oepkgs.net/openeuler/epkg/rootfs/epkg-installer.sh +sh epkg-installer.sh + +# Uninstall epkg. +wget https://repo.oepkgs.net/openeuler/epkg/rootfs/epkg-uninstaller.sh +sh epkg-uninstaller.sh + +# Initialize epkg. +epkg init +bash // Re-execute .bashrc to update the PATH + +# Create environment 1. +epkg env create t1 +epkg install tree +tree --version +which tree + +# View repositories. +[root@vm-4p64g ~]# epkg repo list +------------------------------------------------------------------------------------------------------------------------------------------------------ +channel | repo | url +------------------------------------------------------------------------------------------------------------------------------------------------------ +openEuler-22.03-LTS-SP3 | OS | https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-22.03-LTS-SP3/OS/aarch64/ +openEuler-24.09 | everything | https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64/ +openEuler-24.09 | OS | https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/OS/aarch64/ +------------------------------------------------------------------------------------------------------------------------------------------------------ + +# Create environment 2, specify a repository. +epkg env create t2 --repo openEuler-22.03-LTS-SP3 +epkg install tree +tree --version +which tree + +# Switch back to environment 1. +epkg env activate t1 +``` + +## epkg Usage + +```bash +Usage: + epkg install PACKAGE + epkg install [--env ENV] PACKAGE (under development) + epkg remove [--env ENV] PACKAGE (under development) + epkg upgrade [PACKAGE] (under development) + + epkg search PACKAGE (under development) + epkg list (under development) + + epkg env list + epkg env create|remove ENV + epkg env activate ENV + epkg env deactivate ENV + epkg env register|unregister ENV + epkg env history ENV (under development) + epkg env rollback ENV (under development) +``` + +Package installation: + +```bash +epkg env create $env # Create an environment. +epkg install $package # Install a package in the environment. +epkg env create $env2 --repo $repo # Create environment 2, specify a repository. +epkg install $package # Install a package in environment 2. +``` + +Package building: + +```bash +epkg build ${yaml_path}/$pkg_name.yaml +``` + +### Installing Software + +Function description: + +Install software in the current environment (confirm the current environment before operation). + +Command: + +```shell +epkg install ${package_name} +``` + +Example output: + +```shell +[root@2d785c36ee2e /]# epkg env activate t1 +Add common to path +Add t1 to path +Environment 't1' activated. +Environment 't1' activated. +[root@2d785c36ee2e /]# epkg install tree +EPKG_ENV_NAME: t1 +Caching repodata for: "OS" +Cache for "OS" already exists. Skipping... +Caching repodata for: "OS" +Cache for "OS" already exists. Skipping... +Caching repodata for: "everything" +Cache for "everything" already exists. Skipping... +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/FF/FFCRTKRFGFQ6S2YVLOSUF6PHSMRP7A2N__ncurses-libs__6.4__8.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/D5/D5BOEFTRBNV3E4EXBVXDSRNTIGLGWVB7__glibc-all-langpacks__2.38__34.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/VX/VX6SUOPGEVDWF6E5M2XBV53VS7IXSFM5__openEuler-repos__1.0__3.3.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/LO/LO6RYZTBB2Q7ZLG6SWSICKGTEHUTBWUA__libselinux__3.5__3.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/EP/EPIEEK2P5IUPO4PIOJ2BXM3QPEFTZUCT__basesystem__12__3.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/2G/2GYDDYVWYYIDGOLGTVUACSBHYVRCRJH3__setup__2.14.5__2.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/HC/HCOKXTWQQUPCFPNI7DMDC6FGSDOWNACC__glibc__2.38__34.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/OJ/OJQAHJTY3Y7MZAXETYMTYRYSFRVVLPDC__glibc-common__2.38__34.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/FJ/FJXG3K2TSUYXNU4SES2K3YSTA3AHHUMB__tree__2.1.1__1.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/KD/KDYRBN74LHKSZISTLMYOMTTFVLV4GPYX__readline__8.2__2.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/MN/MNJPSSBS4OZJL5EB6YKVFLMV4TGVBUBA__tzdata__2024a__2.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/S4/S4FBO2SOMG3GKP5OMDWP4XN5V4FY7OY5__bash__5.2.21__1.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/EJ/EJGRNRY5I6XIDBWL7H5BNYJKJLKANVF6__libsepol__3.5__3.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/TZ/TZRQZRU2PNXQXHRE32VCADWGLQG6UL36__bc__1.07.1__12.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/WY/WYMBYMCARHXD62ZNUMN3GQ34DIWMIQ4P__filesystem__3.16__6.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/KQ/KQ2UE3U5VFVAQORZS4ZTYCUM4QNHBYZ7__openEuler-release__24.09__55.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/HD/HDTOK5OTTFFKSTZBBH6AIAGV4BTLC7VT__openEuler-gpg-keys__1.0__3.3.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/EB/EBLBURHOKKIUEEFHZHMS2WYF5OOKB4L3__pcre2__10.42__8.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/YW/YW5WTOMKY2E5DLYYMTIDIWY3XIGHNILT__info__7.0.3__3.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start download https://repo.oepkgs.net/openeuler/epkg/channel/openEuler-24.09/everything/aarch64//store/E4/E4KCO6VAAQV5AJGNPW4HIXDHFXMR4EJV__ncurses-base__6.4__8.oe2409.epkg +############################################################################################################################################################################################################### 100.0% +start install FFCRTKRFGFQ6S2YVLOSUF6PHSMRP7A2N__ncurses-libs__6.4__8.oe2409 +start install D5BOEFTRBNV3E4EXBVXDSRNTIGLGWVB7__glibc-all-langpacks__2.38__34.oe2409 +start install VX6SUOPGEVDWF6E5M2XBV53VS7IXSFM5__openEuler-repos__1.0__3.3.oe2409 +start install LO6RYZTBB2Q7ZLG6SWSICKGTEHUTBWUA__libselinux__3.5__3.oe2409 +start install EPIEEK2P5IUPO4PIOJ2BXM3QPEFTZUCT__basesystem__12__3.oe2409 +start install 2GYDDYVWYYIDGOLGTVUACSBHYVRCRJH3__setup__2.14.5__2.oe2409 +start install HCOKXTWQQUPCFPNI7DMDC6FGSDOWNACC__glibc__2.38__34.oe2409 +start install OJQAHJTY3Y7MZAXETYMTYRYSFRVVLPDC__glibc-common__2.38__34.oe2409 +start install FJXG3K2TSUYXNU4SES2K3YSTA3AHHUMB__tree__2.1.1__1.oe2409 +start install KDYRBN74LHKSZISTLMYOMTTFVLV4GPYX__readline__8.2__2.oe2409 +start install MNJPSSBS4OZJL5EB6YKVFLMV4TGVBUBA__tzdata__2024a__2.oe2409 +start install S4FBO2SOMG3GKP5OMDWP4XN5V4FY7OY5__bash__5.2.21__1.oe2409 +start install EJGRNRY5I6XIDBWL7H5BNYJKJLKANVF6__libsepol__3.5__3.oe2409 +start install TZRQZRU2PNXQXHRE32VCADWGLQG6UL36__bc__1.07.1__12.oe2409 +start install WYMBYMCARHXD62ZNUMN3GQ34DIWMIQ4P__filesystem__3.16__6.oe2409 +start install KQ2UE3U5VFVAQORZS4ZTYCUM4QNHBYZ7__openEuler-release__24.09__55.oe2409 +start install HDTOK5OTTFFKSTZBBH6AIAGV4BTLC7VT__openEuler-gpg-keys__1.0__3.3.oe2409 +start install EBLBURHOKKIUEEFHZHMS2WYF5OOKB4L3__pcre2__10.42__8.oe2409 +start install YW5WTOMKY2E5DLYYMTIDIWY3XIGHNILT__info__7.0.3__3.oe2409 +start install E4KCO6VAAQV5AJGNPW4HIXDHFXMR4EJV__ncurses-base__6.4__8.oe2409 +``` + +### Listing Environments + +Function description: + +List all environments in epkg (under the `$EPKG_ENVS_ROOT` directory) and indicate the current environment. + +Command: + +```shell +epkg env list +``` + +Example output: + +```shell +[small_leek@19e784a5bc38 bin]# epkg env list +Available environments(sort by time): +w1 +main +common +You are in [main] now +``` + +### Creating an Environment + +Function description: + +Create a new environment. After successful creation, the new environment is activated by default, but is not globally registered. + +Command: + +```shell +epkg env create ${env_name} +``` + +Example output: + +```shell +[small_leek@b0e608264355 bin]# epkg env create work1 +YUM --installroot directory structure created successfully in: /root/.epkg/envs/work1/profile-1 +Environment 'work1' added to PATH. +Environment 'work1' activated. +Environment 'work1' created. +``` + +### Activating an Environment + +Function description: + +Activate the specified environment, refresh `EPKG_ENV_NAME` and `RPMDB_DIR` (used to point to `--dbpath` when software is installed into the specified environment), refresh `PATH` to include the specified environment and the common environment, and set the specified environment as the first priority. + +Command: + +```shell +epkg env activate ${env_name} +``` + +Example output: + +```shell +[small_leek@9d991d463f89 bin]# epkg env activate main +Environment 'main' activated +``` + +### Deactivating an Environment + +Function description: + +Deactivate the specified environment, refresh `EPKG_ENV_NAME` and `RPMDB_DIR`, refresh `PATH`, and default to the main environment. + +Command: + +```shell +epkg env deactivate ${env_name} +``` + +Example output: + +```shell +[small_leek@398ec57ce780 bin]# epkg env deactivate w1 +Environment 'w1' deactivated. +``` + +### Registering an Environment + +Function description: + +Register the specified environment, persistently refresh `PATH` to include all registered environments in epkg, and set the specified environment as the first priority. + +Command: + +```shell +epkg env register ${env_name} +``` + +Example output: + +```shell +[small_leek@5042ae77dd75 bin]# epkg env register lkp +EPKG_ACTIVE_ENV: +Environment 'lkp' has been registered to PATH. +``` + +### Unregistering an Environment + +Function description: + +Unregister the specified environment, persistently refresh `PATH` to include all registered environments in epkg except the specified one. + +Command: + +```shell +epkg env unregister ${env_name} +``` + +Example output: + +```shell +[small_leek@69393675945d /]# epkg env unregister w4 +EPKG_ACTIVE_ENV: +Environment 'w4' has been unregistered from PATH. +``` + +### Building an epkg Package + +Function description: + +Build an epkg package using the YAML file provided by autopkg. + +Command: + +```shell +epkg build ${yaml_path}/$pkg_name.yaml +``` + +Example output: + +```shell +[small_leek@69393675945d /]# epkg build /root/epkg/build/test/tree/package.yaml +pkg_hash: fbfqtsnza9ez1zk0cy23vyh07xfzsydh, dir: /root/.cache/epkg/build-workspace/result +Compress success: /root/.cache/epkg/build-workspace/epkg/fbfqtsnza9ez1zk0cy23vyh07xfzsydh__tree__2.1.1__0.oe2409.epkg +``` diff --git a/docs/en/docs/oeAware/figures/dep-failed.png b/docs/en/docs/oeAware/figures/dep-failed.png deleted file mode 100644 index afb4750135657876b455978bf9d8f5eff36be91e..0000000000000000000000000000000000000000 Binary files a/docs/en/docs/oeAware/figures/dep-failed.png and /dev/null differ diff --git a/docs/en/docs/oeAware/figures/dep.png b/docs/en/docs/oeAware/figures/dep.png index 91388d6a860f032c86c0559b232f2d5ef55a40f8..493f118a9a822fa16f8c8375ba9261c1e10ac935 100644 Binary files a/docs/en/docs/oeAware/figures/dep.png and b/docs/en/docs/oeAware/figures/dep.png differ diff --git a/docs/en/docs/oeAware/figures/dependency.png b/docs/en/docs/oeAware/figures/dependency.png deleted file mode 100644 index 0cd087fb0c9095e63aa76e0d2464a92225af2399..0000000000000000000000000000000000000000 Binary files a/docs/en/docs/oeAware/figures/dependency.png and /dev/null differ diff --git a/docs/en/docs/oeAware/oeAware_user_guide.md b/docs/en/docs/oeAware/oeAware_user_guide.md index e86cb0022509bc47186e69917b56a367f7fa580e..56c51b2f3ab03648caea173bfe8c7eadf6178593 100644 --- a/docs/en/docs/oeAware/oeAware_user_guide.md +++ b/docs/en/docs/oeAware/oeAware_user_guide.md @@ -32,24 +32,23 @@ Configuration file path: **/etc/oeAware/config.yaml** log_path: /var/log/oeAware # Log storage path log_level: 1 # Log level. 1: DUBUG; 2: INFO; 3: WARN; 4: ERROR. enable_list: # Plugins are enabled by default. - - name: libtest.so # Configure the plugin and enable all instances of the plugin. - - name: libtest1.so # Configure the plugin and enable the specified plugin instances. - instances: - - instance1 - - instance2 - ... - ... + - name: libtest.so # Configure the plugin and enable all instances of the plugin. + - name: libtest1.so # Configure the plugin and enable the specified plugin instances. + instances: + - instance1 + - instance2 + ... + ... plugin_list: # Downloaded packages are supported. - - name: test #The name must be unique. If the name is repeated, the first occurrence is used. - description: hello world - url: https://gitee.com/openeuler/oeAware-manager/raw/master/README.md #url must not be empty. - ... + - name: test #The name must be unique. If the name is repeated, the first occurrence is used. + description: hello world + url: https://gitee.com/openeuler/oeAware-manager/raw/master/README.md #url must not be empty. + ... ``` After modifying the configuration file, run the following commands to restart the service: ```shell -systemctl daemon-reload systemctl restart oeaware ``` @@ -59,38 +58,22 @@ systemctl restart oeaware **Instance definition**: The scheduling unit in the service is instance. A plugin contains multiple instances. For example, a collection plugin includes multiple collection items, and each collection item is an instance. -**Dependencies Between Instances** - -Before running an instance, ensure that the dependency between the instances is met. - -![img](./figures/dependency.png) - -- A collection instance does not depend on any other instance. - -- A sensing instance depends on a collection instance and other sensing instances. - -- A tuning instance depends on a collection instance, sensing instance, and other tuning instances. - ### Plugin Loading By default, the service loads the plugins in the plugin storage paths. -Collection plugin path: /usr/lib64/oeAware-plugin/collector - -Sensing plugin path: /usr/lib64/oeAware-plugin/scenario - -Tuning plugin path: /usr/lib64/oeAware-plugin/tune +Plugin path: **/usr/lib64/oeAware-plugin** You can also manually load the plugins. ```shell -oeawarectl -l | --load -t | --type # plugin type can be collector, scenario, or tune +oeawarectl -l | --load ``` Example ```shell -[root@localhost ~]# oeawarectl -l libthread_collect.so -t collector +[root@localhost ~]# oeawarectl -l libthread_collect.so Plugin loaded successfully. ``` @@ -126,46 +109,62 @@ Example [root@localhost ~]# oeawarectl -q Show plugins and instances status. ------------------------------------------------------------ -libthread_scenario.so - thread_scenario(available, close) +libsystem_tune.so + stealtask_tune(available, close, count: 0) + smc_tune(available, close, count: 0) + xcall_tune(available, close, count: 0) + seep_tune(available, close, count: 0) libpmu.so - collector_pmu_sampling(available, close) - collector_pmu_counting(available, close) - collector_pmu_uncore(available, close) - collector_spe(available, close) -libthread_collector.so - thread_collector(available, close) + pmu_counting_collector(available, close, count: 0) + pmu_sampling_collector(available, close, count: 0) + pmu_spe_collector(available, close, count: 0) + pmu_uncore_collector(available, close, count: 0) +libdocker_tune.so + docker_cpu_burst(available, close, count: 0) +libthread_scenario.so + thread_scenario(available, close, count: 0) +libsystem_collector.so + thread_collector(available, close, count: 0) + kernel_config(available, close, count: 0) + command_collector(available, close, count: 0) +libdocker_collector.so + docker_collector(available, close, count: 0) +libub_tune.so + unixbench_tune(available, close, count: 0) +libanalysis_oeaware.so + analysis_aware(available, close, count: 0) ------------------------------------------------------------ format: [plugin] - [instance]([dependency status], [running status]) + [instance]([dependency status], [running status], [enable cnt]) dependency status: available means satisfying dependency, otherwise unavailable. running status: running means that instance is running, otherwise close. +enable cnt: number of instances enabled. ``` If the operation fails, an error description is returned. -#### Querying Plugin Dependencies +#### Querying Subscription Relationships of Running Instances ```shell -oeawarectl -Q # Query the dependency graph of loaded instances. -oeawarectl --query-dep= # Query the dependency graph of a specified instance. +oeawarectl -Q # Query the subscription relationship diagram of all running instances. +oeawarectl --query-dep= # Query the subscription relationship diagram of a running instance. ``` -A **dep.png** file will be generated in the current directory to display the dependencies. +The command generates a **dep.png** file in the current directory, which visualizes the subscription relationships. -Example - -Relationship diagram when dependencies are met -![img](./figures/dep.png) +Note that subscription relationships for instances that are not running will not be included in the diagram. -Relationship diagram when dependencies are not met +Example -![img](./figures/dep-failed.png) +```sh +oeawarectl -e thread_scenario +oeawarectl -Q +``` -If the operation fails, an error description is returned. +![img](./figures/dep.png) -### Enabling Plugins +### Enabling Plugin Instances #### Enabling a Plugin Instance @@ -173,14 +172,26 @@ If the operation fails, an error description is returned. oeawarectl -e | --enable ``` +Enabling a plugin instance will also enable the topic it subscribes to. + If the operation fails, an error description is returned. +Recommended plugins to enable: + +- **libsystem_tune.so**: `stealtask_tune`, `smc_tune`, `xcall_tune`, `seep_tune` +- **libub_tune.so**: `unixbench_tune` +- **libtune_numa.so**: `tune_numa_mem_access` + +Other plugins are mainly used to provide data, which can be retrieved using the SDK. + #### Disabling a Plugin Instance ```shell oeawarectl -d | --disable ``` +Disabling a plugin instance will also disable the topic it subscribes to. + If the operation fails, an error description is returned. ### Downloading and Installing Plugins @@ -211,16 +222,69 @@ oeawarectl -i | --install # Name of a package queried using ``` If the operation fails, an error description is returned. + +### Analysis Mode + +```sh +oeawarectl analysis -h +usage: oeawarectl analysis [options]... + options + -t|--time set analysis duration in seconds(default 30s), range from 1 to 100. + -r|--realtime show real time report. + -v|--verbose show verbose information. + -h|--help show this help message. +``` + +Example + +Run the following command to generate a system analysis report. + +```sh +oeawarectl analysis -t 10 +``` + +The output will look like this: + +```bash +============================================================================================ + Summary Analysis Report +============================================================================================ + ========================================= Suggest ========================================= + Tune Instance | Suggest | Note + stealtask_tune | No | CpuRatio(average) : 0.17% + smc_tune | No | Collecting very little network access + gazelle | No | Collecting very little network access + tune_numa_mem_access | No | No access + ========================================= Network ========================================= + + Local network communication distribution + Node0 Node1 Node2 Node3 + 0.00% 0.00% 0.00% 0.00% + Remote network communication distribution(receive) + matrix representation of network thread nodes to irq nodes + Node0 Node1 Node2 Node3 + Node0 100.00% 0.00% 0.00% 0.00% + Node1 0.00% 0.00% 0.00% 0.00% + Node2 0.00% 0.00% 0.00% 0.00% + Node3 0.00% 0.00% 0.00% 0.00% + ======================================== Solution ======================================== + No solution. +``` + +The report consists of three sections: + +- **Suggest**: offers tuning recommendations based on the system status. +- **Network**: provides details on network usage. +- **Solution**: suggests specific tuning actions. + ### Help + Use the `--help` command for help information. + ```shell usage: oeawarectl [options]... options - -l|--load [plugin] load plugin and need plugin type. - -t|--type [plugin_type] assign plugin type. there are three types: - collector: collection plugin. - scenario: awareness plugin. - tune: tune plugin. + -l|--load [plugin] load plugin. -r|--remove [plugin] remove plugin from system. -e|--enable [instance] enable the plugin instance. -d|--disable [instance] disable the plugin instance. @@ -233,401 +297,323 @@ usage: oeawarectl [options]... --help show this help message. ``` -## Plugin Development +## Plugin Development Guide -### Common Data Structures of Plugins +### Basic Data Structures -```c -struct DataBuf { - int len; - void *data; -}; -``` +```c++ +typedef struct { + char *instanceName; // Instance name + char *topicName; // Topic name + char *params; // Parameters +} CTopic; -**struct DataBuf** is the data buffer. +typedef struct { + CTopic topic; + unsigned long long len; // Length of the data array + void **data; // Stored data +} DataList; -- **data**: specific data. **data** is an array. The data type can be defined as required. -- len: size of **data**. +const int OK = 0; +const int FAILED = -1; -```c -struct DataHeader { - char type[DATA_HEADER_TYPE_SIZE]; - int index; - uint64_t count; - struct DataBuf *buf; - int buf_len; -}; +typedef struct { + int code; // Returns OK on success, FAILED on failure + char *payload; // Additional information +} Result; ``` -**struct DataHeader** is the structure for transferring data between plugins. It contains a cyclic buffer. - -- **type**: type of the input data. For example, when data is transferred to a sensing plugin, this parameter is used to identify the collection item of the collection plugin. - -- **index**: location of the data that is being written. For example, after a data collection, **index** increases by one. - -- **count**: number of times that an instance is executed. The value is accumulated. - -- **buf**: data buffer. For example, some collection items are used by a sensing plugin only after being sampled for multiple times. Therefore, the collection items are saved in a buffer array. - -- **buf_len**: size of the data buffer. **buf_len** is a fixed value after the data buffer is initialized. - -### Collection Plugin - -A collection plugin must have the **int32_t get_instance(CollectorInterface \*ins)** interface to return all collection items contained in the plugin. **CollectorInterface** contains the following content: - -```c -struct CollectorInterface { - char* (*get_version)(); - char* (*get_name)(); - char* (*get_description)(); - char* (*get_type)(); - int (*get_cycle)(); - char* (*get_dep)(); - void (*enable)(); - void (*disable)(); - void* (*get_ring_buf)(); - void (*reflash_ring_buf)(); -}; +### Base Class for Instances + +```c++ +namespace oeaware { +// Instance type +const int TUNE = 0b10000; +const int SCENARIO = 0b01000; +const int RUN_ONCE = 0b00010; +class Interface { +public: + virtual Result OpenTopic(const Topic &topic) = 0; + virtual void CloseTopic(const Topic &topic) = 0; + virtual void UpdateData(const DataList &dataList) = 0; + virtual Result Enable(const std::string ¶m = "") = 0; + virtual void Disable() = 0; + virtual void Run() = 0; +protected: + std::string name; + std::string version; + std::string description; + std::vector supportTopics; + int priority; + int type; + int period; +} +} ``` -Obtaining the version number - -1. Interface definition - - ```c - char* (*get_version)(); - ``` - -2. Interface description - -3. Parameter description - -4. Return value description - - The specific version number is returned. This interface is reserved. - -Obtaining the instance name - -1. Interface definition - - ```c - char* (*get_name)(); - ``` - -2. Interface description - - Obtains the name of a collection instance. When you run the `-q` command on the client, the instance name is displayed. In addition, you can run the `--enable` command to enable the instance. - -3. Parameter description - -4. Return value description - - The name of the collection instance is returned. Ensure that the instance name is unique. - -Obtaining description information - -1. Interface definition - - ```c - char* (*get_description)(); - ``` - -2. Interface description - -3. Parameter description - -4. Return value description - - The detailed description is returned. This interface is reserved. - -Obtaining the type - -1. Interface definition - - ```c - char* (*get_type)(); - ``` - -2. Interface description - -3. Parameter description - -4. Return value description - - The specific type information is returned. This interface is reserved. - -Obtaining the sampling period - -1. Interface definition - - ```c - int (*get_cycle)(); - ``` - -2. Interface description - - Obtains the sampling period. Different collection items can use different collection periods. - -3. Parameter description - -4. Return value description - - The specific sampling period is returned. The unit is ms. - -Obtaining dependencies - -1. Interface definition - - ```c - char* (*get_dep)(); - ``` - -2. Interface description - -3. Parameter description - -4. Return value description - - Information about the dependent instances is returned. This interface is reserved. - -Enabling a collection instance - -1. Interface definition - - ```c - void (*enable)(); - ``` - -2. Interface description - - Enables a collection instance. - -3. Parameter description - -4. Return value description - -Disabling a collection instance - -1. Interface definition - - ```c - void (*disable)(); - ``` - -2. Interface description - - Disables a collection instance. - -3. Parameter description - -4. Return value description - -Obtaining the collection data buffer - -1. Interface definition - - ```c - void* (*get_ring_buf)(); - ``` - -2. Interface description - - Obtains the buffer management pointer of the collection data (the memory is applied for by the plugin). The pointer is used by sensing plugins. - -3. Parameter description - -4. Return value description - - The **struct DataHeader** management pointer is returned, which stores the data of multiple samples. - -Refreshing collection data - -1. Interface definition - - ```c - void (*reflash_ring_buf)(); - ``` - -2. Interface description - - Periodically obtains sampled data based on the sampling period and saves the data to **struct DataBuf**. - -3. Parameter description - -4. Return value description - -### Sensing Plugin - -A sensing plugin must have the **int32_t get_instance(ScenarioInterface \*ins)** interface to return all collection items contained in the plugin. **ScenarioInterface** contains the following content: - -```c -struct ScenarioInterface { - char* (*get_version)(); - char* (*get_name)(); - char* (*get_description)(); - char* (*get_dep)(); - int (*get_cycle)(); - void (*enable)(); - void (*disable)(); - void (*aware)(void*[], int); - void* (*get_ring_buf)(); +To develop an instance, inherit from the base class, implement the six virtual functions, and assign values to the seven properties of the class. + +Instances operate on a publish-subscribe model, where data is retrieved via `Subscribe` and published via `Publish`. + +### Property Description + +| Property | Type | Description | +| --- | --- | --- | +| name | string | Instance name | +| version | string | Instance version (reserved) | +| description | string | Instance description | +| supportTopics | vector\ | Supported topics | +| priority | int | Execution priority of the instance (tuning > sensing > collection) | +| type | int | Instance type, represented by bit flags. The second bit indicates a single-run instance, the third bit indicates a collection instance, the fourth bit indicates a sensing instance, and the fifth bit indicates a tuning instance. | +| period | int | Execution period of the instance, in milliseconds. The value must be a multiple of 10. | + +### Interface Description + +| Function | Parameters | Return Value | Description | +| --- | --- | --- | --- | +|Result OpenTopic(const Topic &topic) | `topic`: topic to open | | Opens the specified topic. | +| void CloseTopic(const Topic &topic) | `topic`: topic to close | | Closes the specified topic. | +| void UpdateData(const DataList &dataList) | `dataList`: subscribed data | | When a topic is subscribed, the subscribed topic updates data through `UpdateData` every cycle. | +| Result Enable(const std::string ¶m = "") | `param`: reserved | | Enables the instance. | +| void Disable() | | | Disables the instance. | +| void Run() | | | Executes the `Run` function every cycle. | + +### Example Instance + +```C++ +#include +#include + +class Test : public oeaware::Interface { +public: + Test() { + name = "TestA"; + version = "1.0"; + description = "this is a test plugin"; + supportTopics; + priority = 0; + type = 0; + period = 20; + } + oeaware::Result OpenTopic(const oeaware::Topic &topic) override { + return oeaware::Result(OK); + } + void CloseTopic(const oeaware::Topic &topic) override { + + } + void UpdateData(const DataList &dataList) override { + for (int i = 0; i < dataList.len; ++i) { + ThreadInfo *info = static_cast(dataList.data[i]); + INFO(logger, "pid: " << info->pid << ", name: " << info->name); + } + } + oeaware::Result Enable(const std::string ¶m = "") override { + Subscribe(oeaware::Topic{"thread_collector", "thread_collector", ""}); + return oeaware::Result(OK); + } + void Disable() override { + + } + void Run() override { + DataList dataList; + oeaware::SetDataListTopic(&dataList, "test", "test", ""); + dataList.len = 1; + dataList.data = new void* [1]; + dataList.data[0] = &pubData; + Publish(dataList); + } +private: + int pubData = 1; }; -``` - -Obtaining the version number - -1. Interface definition - - ```c - char* (*get_version)(); - ``` - -2. Interface description - -3. Parameter description - -4. Return value description - - The specific version number is returned. This interface is reserved. - -The specific version number is returned. This interface is reserved. -1. Interface definition - - ```c - char* (*get_name)(); - ``` - -2. Interface description - - Obtains the name of a sensing instance. When you run the `-q` command on the client, the instance name is displayed. In addition, you can run the `--enable` command to enable the instance. - -3. Parameter description - -4. Return value description - - The name of the sensing instance is returned. Ensure that the instance name is unique. - -Obtaining description information - -1. Interface definition - - ```c - char* (*get_description)(); - ``` - -2. Interface description +extern "C" void GetInstance(std::vector> &interfaces) +{ + interfaces.emplace_back(std::make_shared()); +} +``` -3. Parameter description +## Built-in Plugins -4. Return value description +### libpmu.so - The detailed description is returned. This interface is reserved. +| Instance | Architecture | Description | Topics | +| --- | --- | --- | --- | +| pmu_counting_collector | aarch64 | Collects count-related events. | cycles, net:netif_rx, L1-dcache-load-misses, L1-dcache-loads, L1-icache-load-misses, L1-icache-loads, branch-load-misses, branch-loads, dTLB-load-misses, dTLB-loads, iTLB-load-misses, iTLB-loads, cache-references, cache-misses, l2d_tlb_refill, l2d_cache_refill, l1d_tlb_refill, l1d_cache_refill, inst_retired, instructions | +| pmu_sampling_collector | aarch64 | Collects sample-related events. | cycles, skb:skb_copy_datagram_iovec, net:napi_gro_receive_entry | +| pmu_spe_collector | aarch64 | Collects SPE events. | spe | +| pmu_uncore_collector | aarch64 | Collects uncore events. | uncore | -Obtaining dependencies +#### Constraints -1. Interface definition +Collecting SPE events requires hardware support. This plugin depends on the SPE feature in BIOS, which must be enabled before execution. - ```c - char* (*get_dep)(); - ``` +To verify if SPE is enabled, run `perf list | grep arm_spe`. If the feature is enabled, the output will include: -2. Interface description +```text +arm_spe_0// [Kernel PMU event] +``` -3. Parameter description +If the feature is not enabled, follow these steps to enable it. -4. Return value description +Navigate to the BIOS settings under **MISC Config --> SPE**. If the status is **Disable**, change it to **Enable**. If this option is missing, the BIOS version might be outdated. - Names of the dependent instances are returned. This interface is reserved. Multiple dependent instances are connected by hyphens (-). For example, if the sensing instance depends on collection instance **A** and collection instance **B**, **A-B** is returned. If instance **A** contains a hyphen in its name, an error is reported. +On the system, open **/etc/grub2-efi.cfg** and locate the boot entry for the kernel version. Append **kpti=off** to the end. For example: -Obtaining the sensing period +```text +linux /vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 root=/dev/mapper/openeuler-root ro rd.lvm.lv=openeuler/root rd.lvm.lv=openeuler/swap video=VGA-1:640x480-32@60me rhgb quiet smmu.bypassdev=0x1000:0x17 smmu.bypassdev=0x1000:0x15 crashkernel=1024M,high video=efifb:off video=VGA-1:640x480-32@60me kpti=off +``` -1. Interface definition +Press **Esc**, type **:wq**, and press **Enter** to save and exit. Reboot the machine using the `reboot` command. - ```c - int (*get_cycle)(); - ``` +### libsystem_collector.so -2. Interface description +System information collection plugin. - Obtains the sensing period. +| Instance | Architecture | Description | Topics | +| --- | --- | --- | --- | +| thread_collector | aarch64/x86 | Collects thread information in the system. | thread_collector | +| kernel_config | aarch64/x86 | Collects kernel-related parameters, including all `sysctl` parameters, `lscpu`, `meminfo`, etc. | get_kernel_config, get_cmd, set_kernel_config | +| command_collector | aarch64/x86 | Collects `sysstat`-related data. | mpstat, iostat, vmstat, sar, pidstat | -3. Parameter description +### libdocker_collector.so -4. Return value description +Docker information collection plugin. - The specific sensing period is returned. The unit is ms. +| Instance | Architecture | Description | Topics | +| --- | --- | --- | --- | +| docker_collector | aarch64/x86 | Collects Docker-related information. | docker_collector | -Enabling a sensing instance +### libthread_scenario.so -1. Interface definition +Thread awareness plugin. - ```c - void (*enable)(); - ``` +| Instance | Architecture | Description | Subscription | +| --- | --- | --- | --- | +| thread_scenario | aarch64/x86 | Retrieves corresponding thread information through configuration files. | thread_collector::thread_collector | -2. Interface description +#### Configuration File - Enables a sensing instance. +**thread_scenario.conf** -3. Parameter description +```ini +redis +fstime +fsbuffer +fsdisk +``` -4. Return value description +### libanalysis_oeaware.so -Disabling a sensing instance +| Instance | Architecture | Description | Subscription | +| --- | --- | --- | --- | +| analysis_aware | Analyzes the business characteristics of the current environment and provides optimization suggestions. | aarch64 | pmu_spe_collector::spe, pmu_counting_collector::net:netif_rx, pmu_sampling_collector::cycles, pmu_sampling_collector::skb:skb_copy_datagram_iovec, pmu_sampling_collector::net:napi_gro_receive_entry | -1. Interface definition +### libsystem_tune.so - ```c - void (*disable)(); - ``` +System tuning plugin -2. Interface description +| Instance | Architecture | Description | Subscription | +| --- | --- | --- | --- | +| stealtask_tune | aarch64 | Implements fast load balancing across multiple cores in high-load scenarios using a lightweight search algorithm to maximize CPU resource utilization. | None | +| smc_tune | aarch64 | Enables SMC acceleration, providing transparent acceleration for TCP-based connections. | None | +| xcall_tune | aarch64 | Enhances system performance by reducing system call overhead. | thread_collector::thread_collector | +| seep_tune | aarch64 | Enables intelligent power-saving mode to reduce system energy consumption. | None | - Disables a sensing instance. +#### Configuration File -3. Parameter description +**xcall.yaml** -4. Return value description +``` yaml +redis: # Thread name + - xcall_1: 1 # xcall_1 represents the xcall optimization method. Currently, only xcall_1 is available. 1 indicates the system call number to be optimized. +mysql: + - xcall_1: 1 +node: + - xcall_1: 1 +``` -Performing Sensing +#### Constraints -1. Interface definition +`xcall_tune` relies on kernel features. The **FAST_SYSCALL** option must be enabled during kernel compilation, and the **xcall** field must be added to **cmdline**. - ```c - void (*aware)(void*[], int); - ``` +### libub_tune.so -2. Interface description +UnixBench tuning plugin - Processes and analyzes collected data. For example, the collected uncore PMU data is processed and analyzed for NUMA issues. +| Instance | Architecture | Description | Subscription | +| --- | --- | --- | --- | +| unixbench_tune | aarch64/x86 | Optimizes UnixBench performance by reducing remote memory access. | thread_collector::thread_collector | -3. Parameter description +### libdocker_tune.so - - **void*[]**: array of **struct DataHeader**. - - **int**: array length of **struct DataHeader**. +| Instance | Architecture | Description | Subscription | +| --- | --- | --- | --- | +| docker_cpu_burst | aarch64 | Provides temporary additional CPU resources to containers during sudden load spikes, alleviating performance bottlenecks caused by CPU limits. | pmu_counting_collector::cycles, docker_collector::docker_collector | -4. Return value description +## External Plugins -Obtaining the sensing data buffer +External plugins need to be installed using the following command. For example, to install the numafast plugin: -1. Interface definition +```shell +oeawarectl -i numafast +``` - ```c - void* (*get_ring_buf)(); - ``` +### libscenario_numa.so -2. Interface description +| Instance | Architecture | Description | Subscription | Topics | +| --- | --- | --- | --- | --- | +| scenario_numa | aarch64 | Detects the proportion of cross-NUMA memory access in the current environment, used for instance or SDK subscription (cannot be enabled individually). | pmu_uncore_collector::uncore | system_score | - Obtains the buffer management pointer of the sensing data (the memory is applied for by the plugin). The pointer is used by tuning plugins. +### libtune_numa.so -3. Parameter description +| Instance | Architecture | Description | Subscription | +| --- | --- | --- | --- | +| tune_numa_mem_access | aarch64 | Periodically migrates threads and memory to reduce cross-NUMA memory access. | scenario_numa::system_score, pmu_spe_collector::spe, pmu_counting_collector::cycles | -4. Return value description +## SDK Usage Instructions - The **struct DataHeader** management pointer is returned. +```C +typedef int(*Callback)(const DataList *); +int OeInit(); // Initializes resources and establishes a connection with the server. +int OeSubscribe(const CTopic *topic, Callback callback); // Subscribes to a topic and asynchronously executes the callback. +int OeUnsubscribe(const CTopic *topic); // Unsubscribes from a topic. +int OePublish(const DataList *dataList); // Publishes data to the server. +void OeClose(); // Releases resources. +``` -### Tuning Plugin +**Example** + +```C +#include "oe_client.h" +#include "command_data.h" +int f(const DataList *dataList) +{ + int i = 0; + for (; i < dataList->len; i++) { + CommandData *data = (CommandData*)dataList->data[i]; + for (int j = 0; j < data->attrLen; ++j) { + printf("%s ", data->itemAttr[j]); + } + printf("\n"); + } + return 0; +} +int main() { + OeInit(); + CTopic topic = { + "command_collector", + "sar", + "-q 1", + }; + if (OeSubscribe(&topic, f) < 0) { + printf("failed\n"); + } else { + printf("success\n"); + } + sleep(10); + OeClose(); +} +``` ## Constraints @@ -637,7 +623,7 @@ By default, oeAware integrates the libkperf module for collecting Arm microarchi ### Operation Constraints -Currently, only the **root** user can operate oeAware. +Currently, only the **root** group users can operate oeAware. **root** and **oeaware** group users can use the SDK. ## Notes diff --git a/docs/en/docs/ops_guide/common-skills.md b/docs/en/docs/ops_guide/common-skills.md index 4d7c1ce4b10853101e80754b855e6d492bf70918..16d7f7c5d71241dc7bc836484d4648e97c9306d7 100644 --- a/docs/en/docs/ops_guide/common-skills.md +++ b/docs/en/docs/ops_guide/common-skills.md @@ -351,7 +351,7 @@ During the installation, all software information is written into the RPM databa By default, when a new repository is added to the Linux system, the GPG key is automatically imported. You can also use `--import` in the `rpm` command to manually import the RPM GPG key to check the integrity of a package when downloading it from the repository. ```shell - rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenEuler-22.03-LTS-SP2 + rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenEuler-24.03-LTS-SP1 ``` 3. DNF commands @@ -391,7 +391,7 @@ During the installation, all software information is written into the RPM databa OS OS debuginfo debuginfo everything everything - pkgship_elasticsearch Elasticsearch repositor + pkgship_elasticsearch Elasticsearch repository source source update update ``` diff --git a/docs/en/docs/rubik/appendix.md b/docs/en/docs/rubik/appendix.md new file mode 100644 index 0000000000000000000000000000000000000000..3d4d4da0f24a3ce3cb93b033974301b6cf2b132c --- /dev/null +++ b/docs/en/docs/rubik/appendix.md @@ -0,0 +1,258 @@ +# Appendix + +## DaemonSet Configuration Template + +```yaml +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: rubik +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["list", "watch"] + - apiGroups: [""] + resources: ["pods/eviction"] + verbs: ["create"] +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: rubik +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: rubik +subjects: + - kind: ServiceAccount + name: rubik + namespace: kube-system +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: rubik + namespace: kube-system +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: rubik-config + namespace: kube-system +data: + config.json: | + { + "agent": { + "logDriver": "stdio", + "logDir": "/var/log/rubik", + "logSize": 1024, + "logLevel": "info", + "cgroupRoot": "/sys/fs/cgroup", + "enabledFeatures": [ + "preemption" + ] + }, + "preemption": { + "resource": [ + "cpu" + ] + } + } +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: rubik-agent + namespace: kube-system + labels: + k8s-app: rubik-agent +spec: + selector: + matchLabels: + name: rubik-agent + template: + metadata: + namespace: kube-system + labels: + name: rubik-agent + spec: + serviceAccountName: rubik + hostPID: true + containers: + - name: rubik-agent + image: hub.oepkgs.net/cloudnative/rubik:latest + imagePullPolicy: IfNotPresent + env: + - name: RUBIK_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + securityContext: + capabilities: + add: + - SYS_ADMIN + resources: + limits: + memory: 200Mi + requests: + cpu: 100m + memory: 200Mi + volumeMounts: + - name: rubiklog + mountPath: /var/log/rubik + readOnly: false + - name: runrubik + mountPath: /run/rubik + readOnly: false + - name: sysfs + mountPath: /sys/fs + readOnly: false + - name: devfs + mountPath: /dev + readOnly: false + - name: config-volume + mountPath: /var/lib/rubik + terminationGracePeriodSeconds: 30 + volumes: + - name: rubiklog + hostPath: + path: /var/log/rubik + - name: runrubik + hostPath: + path: /run/rubik + - name: sysfs + hostPath: + path: /sys/fs + - name: devfs + hostPath: + path: /dev + - name: config-volume + configMap: + name: rubik-config + items: + - key: config.json + path: config.json +``` + +## Dockerfile Template + +```dockerfile +FROM scratch +COPY ./build/rubik /rubik +ENTRYPOINT ["/rubik"] +``` + +## Image Build Script + +```bash +#!/bin/bash +set -e + +CURRENT_DIR=$(cd "$(dirname "$0")" && pwd) +BINARY_NAME="rubik" + +RUBIK_FILE="${CURRENT_DIR}/build/rubik" +DOCKERFILE="${CURRENT_DIR}/Dockerfile" +YAML_FILE="${CURRENT_DIR}/rubik-daemonset.yaml" + +# Get version and release number of rubik binary +VERSION=$(${RUBIK_FILE} -v | grep ^Version | awk '{print $NF}') +RELEASE=$(${RUBIK_FILE} -v | grep ^Release | awk '{print $NF}') +IMG_TAG="${VERSION}-${RELEASE}" + +# Get rubik image name and tag +IMG_NAME_AND_TAG="${BINARY_NAME}:${IMG_TAG}" + +# Build container image for rubik +docker build -f "${DOCKERFILE}" -t "${IMG_NAME_AND_TAG}" "${CURRENT_DIR}" + +echo -e "\n" +# Check image existence +docker images | grep -E "REPOSITORY|${BINARY_NAME}" + +# Modify rubik-daemonset.yaml file, set rubik image name +sed -i "/image:/s/:.*/: ${IMG_NAME_AND_TAG}/" "${YAML_FILE}" +``` + +## Communication Matrix + +- The Rubik service process communicates with the Kubernetes API server as a client through the list-watch mechanism to obtain information about Pods. + +|Source IP Address|Source Port|Destination IP Address|Destination Port|Protocol|Port Description|Listening Port Modifiable|Authentication Method| +|----|----|----|----|----|----|----|----| +|Rubik node|32768-61000|api-server node|443|TCP|Kubernetes external resource port |No|Token| + +## File Permissions + +- All Rubik operations require root permissions. + +- Related file permissions are as follows: + +|Path|Permissions|Description| +|----|----|----| +|/var/lib/rubik|750|Directory generated after the RPM package is installed, which stores Rubik-related files| +|/var/lib/rubik/build|550|Directory for storing the Rubik binary file| +|/var/lib/rubik/build/rubik|550|Rubik binary file| +|/var/lib/rubik/rubik-daemonset.yaml|550|Rubik DaemonSet configuration template to be used for Kubernetes deployment| +|/var/lib/rubik/Dockerfile|640|Dockerfile template| +|/var/lib/rubik/build_rubik_image.sh|550|Rubik container image build script.| +|/var/log/rubik|640|Directory for storing Rubik log files (requires logDriver=file)| +|/var/log/rubik/rubik.log*|600|Rubik log files| + +## Constraints + +### Specifications + +- Drive: More than 1 GB + +- Memory: More than 100 MB + +## Runtime + +- Only one Rubik instance can exist on a Kubernetes node. + +- Rubik cannot take any CLI parameters. Rubik will fail to be started if any CLI parameter is specified. + +- When the Rubik process is in the T (TASK_STOPPED or TASK_TRACED) OR D (TASK_UNINTERRUPTIBLE) state, the server is unavailable and does not respond. The service becomes available after the process recovers from the abnormal state. + +### Pod Priorities + +- Pod priorities cannot be raised. If the priority of service A is changed from -1 to 0, Rubik will report an error. + +- Adding or modifying annotations or re-applying Pod YAML configuration file does not trigger Pod rebuild. Rubik senses changes in Pod annotations through the list-watch mechanism. + +- After an online service is moved to the offline group, do not move it back to the online group, otherwise QoS exception may occur. + +- Do not add important system services and kernel processes to the offline group. Otherwise, they cannot be scheduled timely, causing system errors. + +- Online and offline configurations for the CPU and memory must be consistent to avoid QoS conflicts between the two subsystems. + +- In the scenario of hybrid service deployment, the original CPU share mechanism is restricted: + - When both online and offline services run on a CPU, the CPU share of the offline service does not take effect. + - If only an online or offline service runs on a CPU, its CPU share takes effect. + - You are advised to set the Pod priority of the offline service to BestEffort. + +- Priority inversion of user-mode processes, SMT, cache, NUMA load balancing, and offline service load balancing are not supported. + +### Other + +To prevent data inconsistency, do not manually modify cgroup or resctrl parameters of the pods, including: + +- CPU cgroup directory, such as **/sys/fs/cgroup/cpu/kubepods/burstable/\/\** + - cpu.qos_level + - cpu.cfs_burst_us + +- memory cgroup directory, such as **/sys/fs/cgroup/memory/kubepods/burstable/\/\** + - memory.qos_level + - memory.soft_limit_in_bytes + - memory.force_empty + - memory.limit_in_bytes + - memory.high + +- blkio cgroup directory, such as **/sys/fs/cgroup/blkio/kubepods/burstable/\/\** + - blkio.throttle.read_bps_device + - blkio.throttle.read_iops_device + - blkio.throttle.write_bps_device + - blkio.throttle.write_iops_device + +- RDT cgroup directory, such as **/sys/fs/resctrl** diff --git a/docs/en/docs/rubik/configuration-description.md b/docs/en/docs/rubik/configuration-description.md new file mode 100644 index 0000000000000000000000000000000000000000..97e58fcf744a0f13fd3166250a9123df9cac09d8 --- /dev/null +++ b/docs/en/docs/rubik/configuration-description.md @@ -0,0 +1,228 @@ +# Rubik Configuration Description + +The Rubik program is written in Go and compiled into a static executable file to minimize the coupling with the system. + +## Commands + +Besides the `-v` option for querying version information, Rubik does not support other options. The following is an example of version query output: + +```bash +$ ./rubik -v +Version: 2.0.1 +Release: 2.oe2403sp1 +Go Version: go1.22.1 +Git Commit: bcaace8 +Built: 2024-12-10 +OS/Arch: linux/amd64 +``` + +## Configuration + +When the Rubik binary file is executed, Rubik parses configuration file **/var/lib/rubik/config.json**. + +> ![](./figures/icon-note.gif) **Note**: +> +> Custom configuration file path is currently not supported to avoid confusion. +> When Rubik runs as a Daemonset in a Kubernetes cluster, modify the ConfigMap in the **hack/rubik-daemonset.yaml** file to configure Rubik. + +The configuration file is in JSON format and keys are in lower camel case. + +An example configuration file is as follows: + +```json +{ + "agent": { + "logDriver": "stdio", + "logDir": "/var/log/rubik", + "logSize": 2048, + "logLevel": "info", + "cgroupRoot": "/sys/fs/cgroup", + "enabledFeatures": [ + "preemption", + "dynCache", + "ioLimit", + "ioCost", + "quotaBurst", + "quotaTurbo", + "psi", + "cpuevict", + "memoryevict" + ] + }, + "preemption": { + "resource": [ + "cpu", + "memory" + ] + }, + "quotaTurbo": { + "highWaterMark": 50, + "syncInterval": 100 + }, + "dynCache": { + "defaultLimitMode": "static", + "adjustInterval": 1000, + "perfDuration": 1000, + "l3Percent": { + "low": 20, + "mid": 30, + "high": 50 + }, + "memBandPercent": { + "low": 10, + "mid": 30, + "high": 50 + } + }, + "ioCost": [ + { + "nodeName": "k8s-single", + "config": [ + { + "dev": "sdb", + "enable": true, + "model": "linear", + "param": { + "rbps": 10000000, + "rseqiops": 10000000, + "rrandiops": 10000000, + "wbps": 10000000, + "wseqiops": 10000000, + "wrandiops": 10000000 + } + } + ] + } + ], + "psi": { + "interval": 10, + "resource": [ + "cpu", + "memory", + "io" + ], + "avg10Threshold": 5.0 + }, + "cpuevict": { + "threshold": 60, + "interval": 1, + "windows": 2, + "cooldown": 20 + }, + "memoryevict": { + "threshold": 60, + "interval": 1, + "cooldown": 4 + } +} +``` + +Rubik configuration items include common items and feature items. Common items are under the **agent** section and are applied globally. Feature items are applied to sub-features that are enabled in the **enabledFeatures** field under **agent**. + +### agent + +The **agent** section stores common configuration items related to Rubik running, such as log configurations and cgroup mount points. + +| Key\[=Default Value] | Type | Description | Example Value | +| ------------------------- | ---------- | -------------------------------------- | --------------------------- | +| logDriver=stdio | string | Log driver, which can be the standard I/O or file | stdio, file | +| logDir=/var/log/rubik | string | Log directory | Anu readable and writable directory | +| logSize=1024 | int | Total size of logs in MB when logDriver=file | \[10, $2^{20}$] | +| logLevel=info | string | Log level | debug,info,warn,error | +| cgroupRoot=/sys/fs/cgroup | string | Mount point of the system cgroup | Mount point of the system cgroup | +| enabledFeatures=\[] | string array | List of Rubik features to be enabled | Rubik features. see [Feature Introduction](./modules.md) for details. | + +### preemption + +The **preemption** field stores configuration items of the absolute preemption feature, including CPU and memory preemption. You can configure this field to use either or both of CPU and memory preemption. + +| Key\[=Default Value] | Type | Description | Example Value | +| --------------- | ---------- | -------------------------------- | ----------- | +| resource=\[] | string array | Resource type to be accessed | cpu, memory | + +### dynCache + +The **dynCache** field stores configuration items related to pod memory bandwidth and last-level cache (LLC) limits. **l3Percent** indicates the watermarks of each LLC level. **memBandPercent** indicates watermarks of memory bandwidth in MB. + +| Key\[=Default Value] | Type | Description | Example Value | +| ----------------------- | ------ | ------------------ | --------------- | +| defaultLimitMode=static | string | dynCache control mode | static, dynamic | +| adjustInterval=1000      | int    | Interval for dynCache control, in milliseconds| \[10, 10000] | +| perfDuration=1000        | int    | perf execution duration for dynCache, in milliseconds | \[10, 10000] | +| l3Percent                | map    | Watermarks of each L3 cache level of dynCache in percents|      | +| .low=20                  | int    | Watermark of the low L3 cache level | \[10, 100]     | +| .mid=30                  | int    | Watermark of the middle L3 cache level  | \[low, 100]   | +| .high=50                 | int    | Watermark of the high L3 cache level  | \[mid, 100]   | +| memBandPercent           | map    | Watermarks of each memory bandwidth level of dynCache in percents|   | +| .low=10                  | int    | Watermark of the low bandwidth level in MB | \[10, 100]  | +| .mid=30                  | int    | Watermark of the middle bandwidth level in MB  | \[low, 100]   | +| .high=50                 | int    | Watermark of the high bandwidth level in MB | \[mid, 100]   | + +### quotaTurbo + +The **quotaTurbo** field stores configuration items of the user-mode elastic traffic limiting feature. + +| Key\[=Default Value] | Type | Description | Example Value | +| ----------------- | ------ | -------------------------------- | -------------------- | +| highWaterMark=60 | int | High watermark of CPU load |\[0, alarmWaterMark) | +| alarmWaterMark=80 | int | Alarm watermark of CPU load | (highWaterMark,100\] | +| syncInterval=100 | int | Interval for triggering container quota updates, in milliseconds | \[100,10000] | + +### ioCost + +The **ioCost** field stores configuration items of the iocost-based I/O weight control feature. The field is an array whose elements are names of nodes (**nodeName**) and their device configuration arrays (**config**). + +| Key | Type | Description | Example Value | +| ----------------- | ------ | -------------------------------- | -------------------- | +| nodeName | string | Node name | Kubernetes cluster node name | +| config | array | Configurations of a block device | / | + +**config** parameters of a block device: + +| Key\[=Default Value] | Type | Description | Example Value | +| --------------- | ------ | --------------------------------------------- | -------------- | +| dev | string | Physical block device name | / | +| model | string | iocost model | linear | +| param | / | Device parameters specific to the model | / | + +For the **linear** model, the **param** field supports the following parameters: + +| Key\[=Default Value] | Type | Description | Example Value | +| --------------- | ---- | ---- | ------ | +| rbps | int64 | Maximum read bandwidth | (0, $2^{63}$) | +| rseqiops | int64 | Maximum sequential read IOPS | (0, $2^{63}$) | +| rrandiops | int64 | Maximum random read IOPS | (0, $2^{63}$) | +| wbps | int64 | Maximum write bandwidth | (0, $2^{63}$) | +| wseqiops | int64 | Maximum sequential write IOPS | (0, $2^{63}$) | +| wrandiops | int64 | Maximum random write IOPS | (0, $2^{63}$) | + +### psi + +The **psi** field stores configuration items of the PSI-based interference detection feature. This feature can monitor CPUs, memory, and I/O resources.You can configure this field to monitor the PSI of any or all of the resources. + +| Key\[=Default Value] | Type | Description | Example Value | +| --------------- | ---------- | -------------------------------- | ----------- | +| interval=10 |int|Interval for PSI monitoring, in seconds| \[10,30]| +| resource=\[] | string array | Resource type to be accessed | cpu, memory, io | +| avg10Threshold=5.0 | float | Average percentage of blocking time of a job in 10 seconds. If this threshold is reached, offline services are evicted. | \[5.0,100]| + +### CPU Eviction Watermark Control + +The **cpuevict** field is used to configure CPU eviction watermark control. This feature collects the node CPU utilization at specified intervals and calculates the average CPU utilization over a defined window. If the average CPU utilization exceeds the eviction watermark, offline Pods are evicted. Once Rubik evicts offline Pods, no further evictions occur during the cooldown period. + +| Key\[=Default Value] | Type | Description | Example Value | +|----------------------|---------|-----------------------------------------------------------------------------|----------------------| +| `threshold=60` | int | Threshold for average CPU utilization (%). If exceeded, offline Pods are evicted. | \[1, 99] | +| `interval=1` | int | Interval (in seconds) for collecting node CPU utilization. | \[1, 3600] | +| `windows=2` | int | Window period (in seconds) for calculating the average CPU utilization. The window must be greater than the interval. If not set, the window defaults to twice the interval. | \[1, 3600] | +| `cooldown=20` | int | Cooldown period (in seconds). No evictions occur during this period after an eviction. | \[1, 9223372036854775806] | + +### Memory Eviction Watermark Control + +The **memoryevict** field is used to configure memory eviction watermark control. This feature collects the node memory utilization at specified intervals. If the memory utilization exceeds the eviction watermark, offline Pods are evicted. Once Rubik evicts offline Pods, no further evictions occur during the cooldown period. + +| Key\[=Default Value] | Type | Description | Example Value | +|----------------------|---------|-----------------------------------------------------------------------------|----------------------| +| `threshold` | int | Threshold for memory utilization (%). If exceeded, offline Pods are evicted. If not specified, this feature is disabled. | \[1, 99] | +| `interval=1` | int | Interval (in seconds) for collecting node memory utilization. | \[1, 3600] | +| `cooldown=4` | int | Cooldown period (in seconds). No evictions occur during this period after an eviction. | \[1, 9223372036854775806] | diff --git a/docs/en/docs/rubik/example-of-isolation-for-hybrid-deployed-services.md b/docs/en/docs/rubik/example-of-isolation-for-hybrid-deployed-services.md index 669a51b6ab25409f1bdc10dbdebd0cd88d208453..88661e1dec4c87b4a841d59e97c9c9b35ed833d0 100644 --- a/docs/en/docs/rubik/example-of-isolation-for-hybrid-deployed-services.md +++ b/docs/en/docs/rubik/example-of-isolation-for-hybrid-deployed-services.md @@ -1,6 +1,6 @@ -## Example of Isolation for Hybrid Deployed Services +# Example of Isolation for Hybrid Deployed Services -### Environment Preparation +## Environment Preparation Check whether the kernel supports isolation of hybrid deployed services. @@ -19,37 +19,37 @@ docker version # The following shows the output of docker version. Client: Version: 18.09.0 - EulerVersion: 18.09.0.300 + EulerVersion: 18.09.0.325 API version: 1.39 Go version: go1.17.3 - Git commit: aa1eee8 - Built: Wed Mar 30 05:07:38 2022 - OS/Arch: linux/amd64 + Git commit: ce4ae23 + Built: Mon Jun 26 12:56:54 2023 + OS/Arch: linux/arm64 Experimental: false Server: Engine: Version: 18.09.0 - EulerVersion: 18.09.0.300 + EulerVersion: 18.09.0.325 API version: 1.39 (minimum version 1.12) Go version: go1.17.3 - Git commit: aa1eee8 - Built: Tue Mar 22 00:00:00 2022 - OS/Arch: linux/amd64 + Git commit: ce4ae23 + Built: Mon Jun 26 12:56:10 2023 + OS/Arch: linux/arm64 Experimental: false ``` -### Hybrid Deployed Services +## Hybrid Deployed Services **Online Service ClickHouse** -Use the clickhouse-benchmark tool to test the performance and collect statistics on performance metrics such as QPS, P50, P90, and P99. For details, see https://clickhouse.com/docs/en/operations/utilities/clickhouse-benchmark/. +Use the clickhouse-benchmark tool to test the performance and collect statistics on performance metrics such as QPS, P50, P90, and P99. For details, see . **Offline Service Stress** Stress is a CPU-intensive test tool. You can specify the **--cpu** option to start multiple concurrent CPU-intensive tasks to increase the stress on the system. -### Usage Instructions +## Usage Instructions 1) Start a ClickHouse container (online service). @@ -75,8 +75,7 @@ offline_container= exec_sql="echo \"SELECT * FROM system.numbers LIMIT 10000000 OFFSET 10000000\" | clickhouse-benchmark -i 10000 -c $concurrency -t $timeout" -function prepare() -{ +function prepare() { echo "Launch clickhouse container." online_container=$(docker run -itd \ -v /tmp:/tmp:rw \ @@ -85,18 +84,16 @@ function prepare() yandex/clickhouse-server) sleep 3 - echo "Clickhouse container lauched." + echo "Clickhouse container launched." } -function clickhouse() -{ +function clickhouse() { echo "Start clickhouse benchmark test." docker exec $online_container bash -c "$exec_sql --json $output" echo "Clickhouse benchmark test done." } -function stress() -{ +function stress() { echo "Launch stress container." offline_container=$(docker run -itd joedval/stress --cpu $stress_num) echo "Stress container launched." @@ -107,12 +104,11 @@ function stress() fi } -function benchmark() -{ +function benchmark() { if [ $with_offline == "with_offline" ]; then - stress - sleep 3 - fi + stress + sleep 3 + fi clickhouse echo "Remove test containers." docker rm -f $online_container @@ -127,7 +123,7 @@ prepare benchmark ``` -### Test Results +## Test Results Independently execute the online service ClickHouse. @@ -139,23 +135,23 @@ The baseline QoS data (QPS/P50/P90/P99) of the online service is as follows: ```json { -"localhost:9000": { -"statistics": { -"QPS": 1.8853412284364512, -...... -}, -"query_time_percentiles": { -...... -"50": 0.484905256, -"60": 0.519641313, -"70": 0.570876148, -"80": 0.632544937, -"90": 0.728295525, -"95": 0.808700418, -"99": 0.873945121, -...... -} -} + "localhost:9000": { + "statistics": { + "QPS": 1.8853412284364512, + ...... + } + }, + "query_time_percentiles": { + ...... + "50": 0.484905256, + "60": 0.519641313, + "70": 0.570876148, + "80": 0.632544937, + "90": 0.728295525, + "95": 0.808700418, + "99": 0.873945121, + ...... + } } ``` @@ -171,22 +167,23 @@ sh test_demo.sh with_offline no_isolation ```json { -"localhost:9000": { -"statistics": { -"QPS": 0.9424028693636205, -...... -}, -"query_time_percentiles": { -...... -"50": 0.840476774, -"60": 1.304607373, -"70": 1.393591017, -"80": 1.41277543, -"90": 1.430316688, -"95": 1.457534764, -"99": 1.555646855, -...... -} + "localhost:9000": { + "statistics": { + "QPS": 0.9424028693636205, + ...... + } + }, + "query_time_percentiles": { + ...... + "50": 0.840476774, + "60": 1.304607373, + "70": 1.393591017, + "80": 1.41277543, + "90": 1.430316688, + "95": 1.457534764, + "99": 1.555646855, + ...... + } } ``` @@ -202,23 +199,23 @@ sh test_demo.sh with_offline enable_isolation ```json { -"localhost:9000": { -"statistics": { -"QPS": 1.8825798759270718, -...... -}, -"query_time_percentiles": { -...... -"50": 0.485725185, -"60": 0.512629901, -"70": 0.55656488, -"80": 0.636395956, -"90": 0.734695906, -"95": 0.804118275, -"99": 0.887807409, -...... -} -} + "localhost:9000": { + "statistics": { + "QPS": 1.8825798759270718, + ...... + } + }, + "query_time_percentiles": { + ...... + "50": 0.485725185, + "60": 0.512629901, + "70": 0.55656488, + "80": 0.636395956, + "90": 0.734695906, + "95": 0.804118275, + "99": 0.887807409, + ...... + } } ``` diff --git a/docs/en/docs/rubik/figures/iocost.PNG b/docs/en/docs/rubik/figures/iocost.PNG new file mode 100644 index 0000000000000000000000000000000000000000..db9bf8c351e8b7047c5815c5779a98c406a62ccd Binary files /dev/null and b/docs/en/docs/rubik/figures/iocost.PNG differ diff --git a/docs/en/docs/rubik/http-apis.md b/docs/en/docs/rubik/http-apis.md deleted file mode 100644 index f7f4752051ad6b91035d343afda0d89475546edc..0000000000000000000000000000000000000000 --- a/docs/en/docs/rubik/http-apis.md +++ /dev/null @@ -1,67 +0,0 @@ -# HTTP APIs - -## Overview - -The open APIs of Rubik are all HTTP APIs, including the API for setting or updating the pod priority, API for detecting the Rubik availability, and API for querying the Rubik version. - -## APIs - -### API for Setting or Updating the Pod Priority - -Rubik provides the function of setting or updating the pod priority. External systems can call this API to send pod information. Rubik sets the priority based on the received pod information to isolate resources. The API call format is as follows: - -```bash -HTTP POST /run/rubik/rubik.sock -{ - "Pods": { - "podaaa": { - "CgroupPath": "kubepods/burstable/podaaa", - "QosLevel": 0 - }, - "podbbb": { - "CgroupPath": "kubepods/burstable/podbbb", - "QosLevel": -1 - } - } -} -``` - -In the **Pods** settings, specify information about the pods whose priorities need to be set or updated. At least one pod must be specified for each HTTP request, and **CgroupPath** and **QosLevel** must be specified for each pod. The meanings of **CgroupPath** and **QosLevel** are as follows: - -| Item | Value Type| Value Range| Description | -| ---------- | ---------- | ------------ | ------------------------------------------------------- | -| QosLevel | Integer | 0, -1 | pod priority. The value **0** indicates that the service is an online service, and the value **-1** indicates that the service is an offline service. | -| CgroupPath | String | Relative path | cgroup subpath of the pod (relative path in the cgroup subsystem)| - -The following is an example of calling the API: - -```sh -curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST --data '{"Pods": {"podaaa": {"CgroupPath": "kubepods/burstable/podaaa","QosLevel": 0},"podbbb": {"CgroupPath": "kubepods/burstable/podbbb","QosLevel": -1}}}' --unix-socket /run/rubik/rubik.sock http://localhost/ -``` - -### API for Detecting Availability - -As an HTTP service, Rubik provides an API for detecting whether it is running. - -API format: HTTP/GET /ping - -The following is an example of calling the API: - -```sh -curl -XGET --unix-socket /run/rubik/rubik.sock http://localhost/ping -``` - -If **ok** is returned, the Rubik service is running. - -### API for Querying Version Information - -Rubik allows you to query the Rubik version number through an HTTP request. - -API format: HTTP/GET /version - -The following is an example of calling the API: - -```sh -curl -XGET --unix-socket /run/rubik/rubik.sock http://localhost/version -{"Version":"0.0.1","Release":"1","Commit":"29910e6","BuildTime":"2021-05-12"} -``` diff --git a/docs/en/docs/rubik/installation-and-deployment.md b/docs/en/docs/rubik/installation-and-deployment.md index cfb5f286fe86cae6aa4216b2e0876afe51c105ec..67de26e0208d045fcd37ce99e892b8e19e0b2e8e 100644 --- a/docs/en/docs/rubik/installation-and-deployment.md +++ b/docs/en/docs/rubik/installation-and-deployment.md @@ -8,50 +8,50 @@ This chapter describes how to install and deploy the Rubik component. ### Hardware -* Architecture: x86 or AArch64 -* Drive: 1 GB or more -* Memory: 100 MB or more +- Architecture: x86 or AArch64 +- Drive: 1 GB or more +- Memory: 100 MB or more ### Software -* OS: openEuler 22.03-LTS -* Kernel: openEuler 22.03-LTS kernel +- OS: openEuler 24.03-LTS-SP1 +- Kernel: openEuler 24.03-LTS-SP1 kernel ### Environment Preparation -* Install the openEuler OS. For details, see the [_openEuler Installation Guide_](../Installation/Installation.md). -* Install and deploy Kubernetes. For details, see the _Kubernetes Cluster Deployment Guide_. -* Install the Docker or iSulad container engine. If the iSulad container engine is used, you need to install the isula-build container image building tool. +- Install the openEuler OS. +- Install and deploy Kubernetes. +- Install the Docker or containerd container engine. ## Installing Rubik Rubik is deployed on each Kubernetes node as a DaemonSet. Therefore, you need to perform the following steps to install the Rubik RPM package on each node. -1. Configure the Yum repositories openEuler 22.03-LTS and openEuler 22.03-LTS:EPOL (the Rubik component is available only in the EPOL repository). +1. The Rubik component is available in the EPOL repository. Configure the Yum repositories openEuler 24.03-LTS-SP1 and openEuler 24.03-LTS-SP1:EPOL. - ```conf - # openEuler 22.03-LTS official repository - name=openEuler22.03 - baseurl=https://repo.openeuler.org/openEuler-22.03-LTS/everything/$basearch/ + ```ini + # openEuler 24.03-LTS-SP1 official repository + name=openEuler24.03-LTS-SP1 + baseurl=https://repo.openeuler.org/openEuler-24.03-LTS-SP1/everything/$basearch/ enabled=1 gpgcheck=1 - gpgkey=https://repo.openeuler.org/openEuler-22.03-LTS/everything/$basearch/RPM-GPG-KEY-openEuler - ``` + gpgkey=https://repo.openeuler.org/openEuler-24.03-LTS-SP1/everything/$basearch/RPM-GPG-KEY-openEuler + ``` - ```conf - # openEuler 22.03-LTS:EPOL official repository - name=Epol - baseurl=https://repo.openeuler.org/openEuler-22.03-LTS/EPOL/$basearch/ + ```ini + # openEuler 24.03-LTS-SP1:EPOL official repository + name=openEuler24.03-LTS-SP1-Epol + baseurl=https://repo.openeuler.org/openEuler-24.03-LTS-SP1/EPOL/$basearch/ enabled=1 - gpgcheck=0 - ``` - -2. Install Rubik with **root** permissions. + gpgcheck=1 + gpgkey=https://repo.openeuler.org/openEuler-24.03-LTS-SP1/everything/$basearch/RPM-GPG-KEY-openEuler + ``` - ```shell - sudo yum install -y rubik - ``` +2. Install Rubik with **root** permissions. + ```shell + sudo yum install -y rubik + ``` > ![](./figures/icon-note.gif)**Note**: > @@ -59,36 +59,36 @@ Rubik is deployed on each Kubernetes node as a DaemonSet. Therefore, you need to ## Deploying Rubik -Rubik runs as a container in a Kubernetes cluster in hybrid deployment scenarios. It is used to isolate and restrict resources for services with different priorities to prevent offline services from interfering with online services, improving the overall resource utilization and ensuring the quality of online services. Currently, Rubik supports isolation and restriction of CPU and memory resources, and must be used together with the openEuler 22.03-LTS kernel. To enable or disable the memory priority feature (that is, memory tiering for services with different priorities), you need to set the value in the **/proc/sys/vm/memcg_qos_enable** file. The value can be **0** or **1**. The default value **0** indicates that the feature is disabled, and the value **1** indicates that the feature is enabled. +Rubik runs as a container in a Kubernetes cluster in hybrid deployment scenarios. It is used to isolate and restrict resources for services with different priorities to prevent offline services from interfering with online services, improving the overall resource utilization and ensuring the quality of online services. Currently, Rubik supports isolation and restriction of CPU and memory resources, and must be used together with the openEuler 24.03-LTS-SP1 kernel. To enable or disable the memory priority feature (that is, memory tiering for services with different priorities), you need to set the value in the **/proc/sys/vm/memcg_qos_enable** file. The value can be **0** or **1**. The default value **0** indicates that the feature is disabled, and the value **1** indicates that the feature is enabled. ```bash sudo echo 1 > /proc/sys/vm/memcg_qos_enable ``` -### Deploying Rubik DaemonSet +### Deploying the Rubik DaemonSet -1. Use the Docker or isula-build engine to build Rubik images. Because Rubik is deployed as a DaemonSet, each node requires a Rubik image. After building an image on a node, use the **docker save** and **docker load** commands to load the Rubik image to each node of Kubernetes. Alternatively, build a Rubik image on each node. The following uses isula-build as an example. The command is as follows: +1. Build the Rubik image: Use the **/var/lib/rubik/build_rubik_image.sh** script to automatically build the Rubik image or directly use Docker to build it. Since Rubik is deployed as a DaemonSet, the Rubik image must be available on every node. You can build the image on one node and then use the save/load functionality of Docker to load the image onto all nodes in the Kubernetes cluster. Alternatively, you can build the Rubik image on each node individually. For Docker, the build command is: -```sh -isula-build ctr-img build -f /var/lib/rubik/Dockerfile --tag rubik:0.1.0 . -``` + ```sh + docker build -f /var/lib/rubik/Dockerfile -t rubik:2.0.1-2 . + ``` -1. On the Kubernetes master node, change the Rubik image name in the **/var/lib/rubik/rubik-daemonset.yaml** file to the name of the image built in the previous step. +2. On the Kubernetes master node, update the Rubik image name in the **/var/lib/rubik/rubik-daemonset.yaml** file to match the image name created in the previous step. -```yaml -... -containers: -- name: rubik-agent - image: rubik:0.1.0 # The image name must be the same as the Rubik image name built in the previous step. - imagePullPolicy: IfNotPresent -... -``` + ```yaml + ... + containers: + - name: rubik-agent + image: rubik_image_name_and_tag # Ensure this matches the Rubik image name built earlier. + imagePullPolicy: IfNotPresent + ... + ``` 3. On the Kubernetes master node, run the **kubectl** command to deploy the Rubik DaemonSet so that Rubik will be automatically deployed on all Kubernetes nodes. -```sh -kubectl apply -f /var/lib/rubik/rubik-daemonset.yaml -``` + ```sh + kubectl apply -f /var/lib/rubik/rubik-daemonset.yaml + ``` 4. Run the **kubectl get pods -A** command to check whether Rubik has been deployed on each node in the cluster. (The number of rubik-agents is the same as the number of nodes and all rubik-agents are in the Running status.) @@ -102,46 +102,31 @@ kube-system rubik-agent-76ft6 1/1 Running ## Common Configuration Description -The Rubik deployed using the preceding method is started with the default configurations. You can modify the Rubik configurations as required by modifying the **config.json** section in the **rubik-daemonset.yaml** file and then redeploy the Rubik DaemonSet. +The Rubik deployed using the preceding method is started with the default configurations. You can modify the Rubik configurations as required by modifying the **config.json** section in the **rubik-daemonset.yaml** file and then redeploy the Rubik DaemonSet. The following describes some common configurations. For other configurations, see [Rubik Configuration Description](./configuration-description.md). -This section describes common configurations in **config.json**. +### Absolute Pod Preemption -### Configuration Item Description +If absolute pod preemption is enabled, you only need to specify the priority using annotations in the YAML file when deploying the service pods. After being deployed successfully, Rubik automatically detects the creation and update of the pods on the current node, and sets the pod priorities based on the configured priorities. For pods that are already started or whose annotations are modified, Rubik automatically updates the pod priority configurations. ```yaml -# The configuration items are in the config.json section of the rubik-daemonset.yaml file. -{ - "autoConfig": true, - "autoCheck": false, - "logDriver": "stdio", - "logDir": "/var/log/rubik", - "logSize": 1024, - "logLevel": "info", - "cgroupRoot": "/sys/fs/cgroup" -} +... + "agent": { + "enabledFeatures": [ + "preemption" + ] + }, + "preemption": { + "resource": [ + "cpu", + "memory" + ] + } +... ``` -| Item | Value Type| Value Range | Description | -| ---------- | ---------- | ------------------ | ------------------------------------------------------------ | -| autoConfig | Boolean | **true** or **false** | **true**: enables automatic pod awareness.
**false**: disables automatic pod awareness.| -| autoCheck | Boolean | **true** or **false** | **true**: enables pod priority check.
**false**: disables pod priority check.| -| logDriver | String | **stdio** or **file** | **stdio**: prints logs to the standard output. The scheduling platform collects and dumps logs.
**file**: prints files to the log directory specified by **logDir**.| -| logDir | String | Absolute path | Directory for storing logs. | -| logSize | Integer | \[10,1048576] | Total size of logs, in MB. If the total size of logs reaches the upper limit, the earliest logs will be discarded.| -| logLevel | String | **error**, **info**, or **debug**| Log level. | -| cgroupRoot | String | Absolute path | cgroup mount point. | - -### Automatic Configuration of Pod Priorities - -If **autoConfig** is set to **true** in **config.json** to enable automatic pod awareness, you only need to specify the priority using annotations in the YAML file when deploying the service pods. After being deployed successfully, Rubik automatically detects the creation and update of the pods on the current node, and sets the pod priorities based on the configured priorities. - -### Pod Priority Configuration Depending on kubelet - -Automatic pod priority configuration depends on the pod creation event notifications from the API server, which have a certain delay. The pod priority cannot be configured before the process is started. As a result, the service performance may fluctuate. To avoid this problem, you can disable the automatic priority configuration option and modify the kubelet source code, so that pod priorities can be configured using Rubik HTTP APIs after the cgroup of each container is created and before each container process is started. For details about how to use the HTTP APIs, see [HTTP APIs](./http-apis.md). - -### Automatic Verification of Pod Priorities - -Rubik supports consistency check on the pod QoS priority configurations of the current node during startup. It checks whether the configuration in the Kubernetes cluster is consistent with the pod priority configuration of Rubik. This function is disabled by default. You can enable or disable it using the **autoCheck** option. If this function is enabled, Rubik automatically verifies and corrects the pod priority configuration of the current node when it is started or restarted. +> ![](./figures/icon-note.gif) **Note**: +> +> Priority configurations support only pods switching from online to offline. ## Configuring Rubik for Online and Offline Services @@ -169,31 +154,3 @@ spec: memory: "200Mi" cpu: "1" ``` - -## Restrictions - -- The maximum number of concurrent HTTP requests that Rubik can receive is 1,000 QPS. If the number of concurrent HTTP requests exceeds the upper limit, an error is reported. - -- The maximum number of pods in a single request received by Rubik is 100. If the number of pods exceeds the upper limit, an error is reported. - -- Only one set of Rubik instances can be deployed on each Kubernetes node. Multiple sets of Rubik instances may conflict with each other. - -- Rubik does not provide port access and can communicate only through sockets. - -- Rubik accepts only valid HTTP request paths and network protocols: http://localhost/ (POST), http://localhost/ping (GET), and http://localhost/version (GET). For details about the functions of HTTP requests, see HTTP APIs(./http-apis.md). - -- Rubik drive requirement: 1 GB or more. - -- Rubik memory requirement: 100 MB or more. - -- Services cannot be switched from a low priority (offline services) to a high priority (online services). For example, if service A is set to an offline service and then to an online service, Rubik reports an error. - -- When directories are mounted to a Rubik container, the minimum permission on the Rubik local socket directory **/run/Rubik** is **700** on the service side. - -- When the Rubik service is available, the timeout interval of a single request is 120s. If the Rubik process enters the T (stopped or being traced) or D (uninterruptible sleep) state, the service becomes unavailable. In this case, the Rubik service does not respond to any request. To avoid this problem, set the timeout interval on the client to avoid infinite waiting. - -- If hybrid deployment is used, the original CPU share funtion of cgroup has the following restrictions: - - If both online and offline tasks are running on the CPU, the CPU share configuration of offline tasks does not take effect. - - If the current CPU has only online or offline tasks, the CPU share configuration takes effect. diff --git a/docs/en/docs/rubik/modules.md b/docs/en/docs/rubik/modules.md new file mode 100644 index 0000000000000000000000000000000000000000..f01c52057e219c6210121bad9b3003ceb3a10c70 --- /dev/null +++ b/docs/en/docs/rubik/modules.md @@ -0,0 +1,498 @@ +# Feature Introduction + +## Absolute Preemption + +Rubik allows you to configure priorities of services. In the hybrid deployment of online and offline services, Rubik ensures that online services preempt resources. CPU and memory resources can be preempted. + +You can enable preemption using the following configuration: + +```yaml +... + "agent": { + "enabledFeatures": [ + "preemption" + ] + }, + "preemption": { + "resource": [ + "cpu", + "memory" + ] + } +... +``` + +For details, see [Configuration Description](./configuration-description.md#preemption). + +In addition, you need to add **volcano.sh/preemptable** to the YAML annotation of the pod to specify service priorities. For example: + +```yaml +annotations: + volcano.sh/preemptable: true +``` + +> The **volcano.sh/preemptable** annotation is used by all Rubik features to identify whether the service is online or offline. **true** indicates an online service. **false** indicates an offline service. + +### CPU Absolute Preemption + +**Prerequisites** + +- The kernel supports CPU priority configuration based on control groups (cgroups). The CPU subsystem provides the **cpu.qos_level** interface. The kernel of openEuler 22.03 or later is recommended. + +**Kernel interface** + +- The interface exists in the cgroup of the container in the **/sys/fs/cgroup/cpu** directory, for example, **/sys/fs/cgroup/cpu/kubepods/burstable/\/\**. + - **cpu.qos_level**: enables the CPU priority configuration. The value can be **0** or **-1**, with **0** being the default. + - **0** indicates an online service. + - **1** indicates an offline service. + +### Memory Absolute Preemption + +In the hybrid deployment of online and offline services, Rubik ensures that offline services are first terminated in the case of out-of-memory (OOM). + +**Prerequisites** + +- The kernel supports memory priority configuration based on cgroups. The memory subsystem provides the **memory.qos_level** interface. The kernel of openEuler 22.03 or later is recommended. +- To enable the memory priority feature, run `echo 1 > /proc/sys/vm/memcg_qos_enable`. + +**Kernel interface** + +- **/proc/sys/vm/memcg_qos_enable**: enables the memory priority feature. The value can be **0** or **1**, with **0** being the default. You can run `echo 1 > /proc/sys/vm/memcg_qos_enable` to enable the feature. + - **0**: The feature is disabled. + - **1**: The feature is enabled. + +- The interface exists in the cgroup of the container in the **/sys/fs/cgroup/memory** directory, for example, **/sys/fs/cgroup/memory/kubepods/burstable/\/\**. + - **memory.qos_level**: enables the memory priority configuration. The value can be **0** or **-1**, with **0** being the default. + - **0** indicates an online service. + - **1** indicates an offline service. + +## dynCache Memory Bandwidth and L3 Cache Access Limit + +Rubik can limit pod memory bandwidth and L3 cache access for offline services to reduce the impact on online services. + +**Prerequisites** + +- The cache access and memory bandwidth limit feature supports only physical machines. + - For x86 physical machines, the CAT and MBA functions of Intel RDT must be enabled in the OS by adding **rdt=l3cat,mba** to the kernel command line parameters (**cmdline**). + - For ARM physical machines, the MPAM function must be enabled in the OS by adding **mpam=acpi** to the kernel command line parameters (**cmdline**). + +- Due to kernel restrictions, RDT does not support the pseudo-locksetup mode. +- You need to manually mount **/sys/fs/resctrl**. Rubik reads and sets files in the **/sys/fs/resctrl** directory. This directory must be mounted before Rubik is started and cannot be unmounted during Rubik running. +- Rubik requires the **SYS_ADMIN** capability. To set files in the **/sys/fs/resctrl** directory on the host, the **SYS_ADMIN** capability must be assigned to the Rubik container. +- Rubik obtains the PID of the service container process on the host. Therefore, the Rubik container needs to share the PID namespace with the host. + +**Rubik RDT Cgroups** + +Rubik creates five cgroups (**rubik_max**, **rubik_high**, **rubik_middle**, **rubik_low** and **rubik_dynamic**) in the RDT resctrl directory (**/sys/fs/resctrl** by default). Rubik writes the watermarks to the **schemata** file of each corresponding cgroup upon startup. The low, middle, and high watermarks can be configured in **dynCache**. The max cgroup uses the default maximum value. The initial watermark of the dynamic cgroup is the same as that of the low cgroup. + +**Rubik dynamic Cgroup** + +When offline pods whose cache level is dynamic exist, Rubik collects the cache miss and LLC miss metrics of online service pods on the current node and adjusts the watermark of the rubik_dynamic cgroup. In this way, Rubik dynamically controls offline service pods in the dynamic cgroup. + +### Memory Bandwidth and LLC Limit of the Pod + +Rubik allows you to configure the memory bandwidth and LLC cgroup for a service pod in either of the following ways: + +- Global annotation + You can set **defaultLimitMode** in the global parameters of Rubik. Rubik automatically configures cgroups for offline service pods (marked by the **volcano.sh/preemptable** annotation in the absolute preemption configuration). + - If the value is **static**, the pod is added to the **rubik_max** cgroup. + - If the value is **dynamic**, the pod is added to the **rubik_dynamic** cgroup. +- Manual annotation + - You can set the cache level for a service pod using the **volcano.sh/cache-limit** annotation and the pod to the specified cgroup. For example, the pod with the following configuration is added to the **rubik_low** cgroup: + + ```yaml + annotations: + volcano.sh/cache-limit: "low" + ``` + +> ![](./figures/icon-note.gif) **Note**: + +1. Cache limits apply to offline services only. +2. The manual annotation overrides the global one. If you set **defaultLimitMode** in the global Rubik configuration and specify the cache level in the YAML configuration of a pod, the actual dynCache limit is the one specified in the pod YAML configuration. + +### dynCache Kernel Interface + +- Rubik creates five cgroup directories in **/sys/fs/resctrl** and modifies the **schemata** and **tasks** files of each cgroup. + +### dynCache Configuration + +The dynCache function is configured as follows: + +```json +"agent": { + "enabledFeatures": [ + "dynCache" + ] +}, +"dynCache": { + "defaultLimitMode": "static", + "adjustInterval": 1000, + "perfDuration": 1000, + "l3Percent": { + "low": 20, + "mid": 30, + "high": 50 + }, + "memBandPercent": { + "low": 10, + "mid": 30, + "high": 50 + } +} +``` + +For details, see [Configuration Description](./configuration-description.md#dyncache) + +- **l3Percent** and **memBandPercent**: + **l3Percent** and **memBandPercent** are used to configure the watermarks of the low, mid, and high cgroups. + + Assume that in the current environment **rdt bitmask=fffff** and **numa=2**. Based on the **low** value of **l3Percent** (20) and the **low** value of **memBandPercent** (10), Rubik configures **/sys/fs/resctrl/rubik_low** as follows: + + ```text + L3:0=f;1=f + MB:0=10;1=10 + ``` + +- defaultLimitMode: + - If the **volcano.sh/cache-limit** annotation is not specified for an offline pod, the **defaultLimitMode** of **cacheConfig** determines the cgroup to which the pod is added. +- **adjustInterval**: + - Interval for dynCache to dynamically adjust the **rubik_dynamic** cgroup, in milliseconds. The default value is **1000**. +- **perfDuration**: + - perf execution duration for dynCache, in milliseconds. The default value is **1000**. + +### Precautions for dynCache + +- dynCache takes affect only for offline pods. +- If a service container is manually restarted during running (the container ID remains unchanged but the container process ID changes), dynCache does not take effect for the container. +- After a service container is started and the dynCache level is set, the limit level cannot be changed. +- The sensitivity of adjusting the dynamic cgroup is affected **adjustInterval** and **perfDuration** values in the Rubik configuration file and the number of online service pods on the node. If the impact detection result indicates that adjustment is required, the adjustment interval fluctuates within the range **\[adjustInterval + perfDuration, adjustInterval + perfDuration x Number of pods\]**. You can set the configuration items based on your required sensitivity. + +## dynMemory Tiered Memory Reclamation + +Rubik supports multiple memory strategies. You can apply different memory allocation methods to different scenarios. + +### fssr + +fssr is kernel cgroup-based dynamic watermark control. **memory.high** is a memcg-level watermark interface provided by the kernel. Rubik continuously detects memory usage and dynamically adjusts the **memory.high** limit of offline services to suppress the memory usage of offline services, ensuring the quality of online services. + +The core logic of fssr is as follows: + +- Rubik calculates the memory to reserve upon startup. The default value is the smaller of 10% of total memory or 10 GB. +- Rubik sets the cgroup-level watermark of the offline container. The kernel provides the **memory.high** and **memory.high_async_ratio** interfaces for configuring the soft upper limit and alarm watermark of the cgroup. By default, **memory.high** is 80% of the total memory (**total_memory**). +- Rubik obtains the free memory (**free_memory**). +- When **free_memory** is less than **reserved_memory**, Rubik decreases **memory.high** for the offline container. The amount decreased each time is 10% of **total_memory**. +- If **free_memory** is more than double the amount of **reserved_memory**, Rubik increases **memory.high**. The amount increased each time is 1% of **total_memory**. + +**Kernel interface** + +- memory.high + +### dynMemory Configuration + +The strategy and check interval of the dynMemory module can be specified in **dynMemory**: + +```json +"dynMemory": { + "policy": "fssr" +} +``` + +- **policy** indicates the dynMemory policy, which supports **fssr**. + +## Flexible Bandwidth + +To effectively solve the problem of QoS deterioration caused by the CPU bandwidth limit of a service, the Rubik provides flexible bandwidth to allow the container to use extra CPU resources, ensuring stable service running. The flexible bandwidth solution is implemented in both kernel mode and user mode. They cannot be used at the same time. + +The user-mode solution is implemented through the CFS bandwidth control capability provided by the Linux kernel. On the premise that the load watermark of the entire system is secure and stable and does not affect the running of other services, the dual-watermark mechanism allows service containers to adaptively adjust the CPU bandwidth limit, relieving CPU resource bottlenecks and improving service performance. + +The kernel-mode solution is implemented through the CPU burst capability provided by the Linux kernel, which allows containers to temporarily exceed its CPU usage limit. You need to manually configure the kernel-mode configuration by setting the burst value for each pod. Rubik does not automatically sets the values. + +### quotaTurbo User-Mode Solution + +You need manually set the **volcano.sh/quota-turbo="true"** annotation for the service pod that requires flexible CPU bandwidth. This annotation takes effect only for the pod whose CPU quota is limited, that is, **CPULimit** is specified in the YAML file. +The user-mode flexible bandwidth policy periodically adjusts the CPU quota of an allowlist container based on the CPU load of the entire system and container running status, and automatically checks and restores the quota values of all containers when Rubik is started or stopped. (The CPU quota described in this section refers to the **cpu.cfs_quota_us** parameter of the container.) The adjustment policies are as follows: + +1. When the CPU load of the entire system is lower than the alarm threshold, if the allowlist container is suppressed by the CPU in the current period, Rubik slowly increases the CPU quota of the container based on the suppression status. The total container quota increase in a single period cannot exceed 1% of the total CPU quota of the current node. +2. When the CPU load of the entire system is higher than the high watermark, if the allowlist container is not suppressed by the CPU in the current period, Rubik slowly increases the container quota based on the watermark. +3. When the CPU load of the entire system is higher than the alarm threshold, if the current quota value of the allowlist container exceeds the configured value, Rubik quickly decreases the CPU quotas of all containers to ensure that the load is lower than the alarm watermark. +4. The maximum CPU quota that a container can have cannot exceed twice the configured value (for example, the **CPULimit** parameter specified in the pod YAML file), and cannot be less than the configured value. +5. The overall CPU usage of the container within 60 synchronization periods cannot exceed the configured value. +6. If the overall CPU usage of a node exceeds 10% within 1 minute, the container quota will not be increased in this period. + +**Kernel interface** + +The interface exists in the cgroup of the container in the **/sys/fs/cgroup/cpu** directory, for example, **/sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/\/\**. The following files are involved: + +- **cpu.cfs_quota_us** +- **cpu.cfs_period_us** +- **cpu.stat** + +#### quotaTurbo Configuration + +The quotaTurbo function is configured as follows: + +```json +"agent": { + "enabledFeatures": [ + "quotaTurbo" + ] + }, +"quotaTurbo": { + "highWaterMark": 60, + "alarmWaterMark": 80, + "syncInterval": 100 +} +``` + +For details, see [Configuration Description](./configuration-description.md#quotaturbo). + +- **highWaterMark** is the high watermark of CPU load. +- **alarmWaterMark** is the alarm watermark of CPU load. +- **syncInterval** is the interval for triggering container quota updates, in milliseconds. + +You need to manually specify the **volcano.sh/quota-turbo="true"** annotation for the service pod. + +```yaml +metadata: + annotations: + # true means to add the pod to the allowlist of quotaTurbo + volcano.sh/quota-turbo : "true" +``` + +### quotaBurst Kernel-Mode Solution + +quotaBurst can be enabled through the **cpu.cfs_burst_us** kernel interface. Rubik allows a container to accumulate CPU resources when the CPU usage of the container is lower than the quota and uses the accumulated CPU resources when the CPU usage exceeds the quota. + +**Kernel interface** + +The interface exists in the cgroup of the container in the **/sys/fs/cgroup/cpu** directory, for example, **/sys/fs/cgroup/cpu/kubepods/burstable/\/\**. The annotation value is written into the following file: + +- **cpu.cfs_burst_us** + +> ![](./figures/icon-note.gif) **Note**: + +The kernel-mode solution is implemented through the **cpu.cfs_burst_us** interface. The **cpu.cfs_burst_us** file must exist in the CPU subsystem directory of the cgroup. The value of **cpu.cfs_burst_us** can be as follows: + +1. When **cpu.cfs_quota_us** is not -1, the sum of **cfs_burst_us** and **cfs_quota_us** must not be greater than $2^{44}$-1, and **cfs_burst_us** is less than or equal to **cfs_quota_us**. +2. When **cpu.cfs_quota_us** is -1, the CPU burst function is not enabled, and **cfs_burst_us** is 0. + +#### quotaBurst Configuration + +The quotaBurst function is configured as follows: + +```json +"agent": { + "enabledFeatures": [ + "quotaBurst" + ] +} +``` + +You need to manually specify the **volcano.sh/quota-burst-time** annotation for the service pod or run `kubectl annotate` to dynamically add the annotation. + +- In the YAML file upon pod creation: + + ```yaml + metadata: + annotations: + # The default unit is microsecond. + volcano.sh/quota-burst-time : "2000" + ``` + +- Annotation modification: You can run the `kubectl annotate` command to dynamically modify annotation. For example: + + ```bash + kubectl annotate --overwrite pods volcano.sh/quota-burst-time='3000' + ``` + +### Constraints + +- The user-mode CPU bandwidth control is implemented through the **cpu.cfs_period_us** (CFS bandwidth control) and **cpu.cfs_quota_us** parameters. The following restrictions apply: + - To avoid unknown errors, other users are not allowed to modify CFS bandwidth control parameters (including but not limited to **cpu.cfs_quota_us** and **cpu.cfs_period_us**). + - Do not use this function together with similar programs that limit CPU resources. Otherwise, the user-mode function cannot be used properly. + - If you monitor the metrics related to CFS bandwidth control, using this feature may affect the consistency of the monitored metrics. +- The following restrictions apply to the kernel-mode solution: + - Use the Kubernetes interface to set the burst value of the pod. Do not manually modify the **cpu.cfs_burst_us** file in the CPU cgroup directory of the container. +- Do not enable both kernel-mode and user-mode flexible bandwidth solutions at the same time. + +## I/O Weight Control Based on ioCost + +To solve the problem that the QoS of online services deteriorates due to high I/O usage of offline services, Rubik provides the I/O weight control function based on ioCost of cgroup v1. +For more, see the [ioCost description](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#io:~:text=correct%20memory%20ownership.-,IO,-%C2%B6). + +**Prerequisites** + +Rubik can control the I/O weight distribution of different pods through iocost of cgroup v1. Therefore, the kernel must support the following features: + +- cgroup v1 blkcg iocost +- cgroup v1 writeback + +The **blkio.cost.qos** and **blkio.cost.model** file interfaces exist in the **blkcg** root system file. For details about the implementation and interface description, see the openEuler kernel document. + +### ioCost Implementation Description + +![](./figures/iocost.PNG) + +The procedure of the Rubik implementation is as follows: + +- When Rubik is deployed, Rubik parses the configuration and sets iocost parameters. +- Rubik registers the detection event to the Kubernetes API server. +- When a pod is deployed, the pod configuration information is write back to Rubik. +- Rubik parses the pod configuration information and configures the pod iocost weight based on the QoS level. + +### ioCost Configuration + +```json +"agent": { + "enabledFeatures": [ + "ioCost" + ] +} +"ioCost": [{ + "nodeName": "k8s-single", + "config": [ + { + "dev": "sdb", + "enable": true, + "model": "linear", + "param": { + "rbps": 10000000, + "rseqiops": 10000000, + "rrandiops": 10000000, + "wbps": 10000000, + "wseqiops": 10000000, + "wrandiops": 10000000 + } + } + ] +}] +``` + +For details, see [Configuration Description](./configuration-description.md#iocost). + +> Parameters related to the ioCost linear model can be obtained through [**iocost_coef_gen.py**](https://github.com/torvalds/linux/blob/master/tools/cgroup/iocost_coef_gen.py). + +## Interference Detection Based on Pressure Stall Information Metrics + +Rubik can observe the pressure stall information (PSI) metrics of online pods to determine the pressure, evicts offline pods, and generates log alarms. Rubik uses **some avg10** as the indicator, which indicates the average blocking time proportion of any task within 10s. You can choose to monitor the CPU, memory, and I/O resources as required and set thresholds. If the blocking time proportion exceeds the threshold, Rubik evicts offline pods based on certain policies to release corresponding resources. If the CPU and memory usage of an online pod is high, Rubik evicts the offline service that occupies the most CPU or memory resources. If the I/O of offline services is high, Rubik evicts the offline service that occupies the most CPU resources. + +The **volcano.sh/preemptable="true"/"false"** annotation distinguishes online and offline services. + +```yaml +annotations: + volcano.sh/preemptable: true +``` + +**Prerequisites** + +Rubik depends on the PSI feature of cgroup v1. openEuler 22.03 LTS and later versions support the PSI interface of cgroup v1. +You can run the following command to check whether the PSI interface is enabled in the kernel: + +```bash +cat /proc/cmdline | grep "psi=1 psi_v1=1" +``` + +If no results are returned, add the boot parameter to the kernel cmdline: + +```bash +# View the kernel version. +uname -a +# View the boot file of the kernel. +grubby --update-kernel="$(grubby --default-kernel)" --args="psi=1 psi_v1=1" +# Reboot. +reboot +``` + +**Kernel interface** + +The interface exists in the cgroup of the container in the **/sys/fs/cgroup/cpuacct** directory, for example, **/sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/\/\**. The following items are involved: + +- **cpu.pressure** +- **memory.pressure** +- **io.pressure** + +### psi Configuration + +```json +"agent": { + "enabledFeatures": [ + "psi" + ] +} +"psi": { + "interval": 10, + "resource": [ + "cpu", + "memory", + "io" + ], + "avg10Threshold": 5.0 +} +``` + +For details, see [Configuration Description](./configuration-description.md#psi). + +## CPU Eviction Watermark Control + +Rubik enables the eviction of offline pods based on node CPU utilization to avoid CPU resource overload. You can set a CPU eviction watermark. Rubik monitors the average CPU utilization of the node over a defined window. If the average CPU utilization exceeds the watermark, Rubik evicts offline pods with high resource usage and shorter durations to free up resources. + +> ![](./figures/icon-note.gif) **Note**: +> +> The `volcano.sh/preemptable="true"/"false"` annotation distinguishes online and offline services. +> +> ```yaml +> annotations: +> volcano.sh/preemptable: true +> ``` + +**Configuration details** + +```json +{ + "agent": { + "enabledFeatures": [ + "cpuevict" + ] + } + "cpuevict": { + "threshold": 60, + "interval": 1, + "windows": 2, + "cooldown": 20 + } +} +``` + +For more details, see [Configuration Documentation](./configuration-description.md#cpu-eviction-watermark-control). + +## Memory Eviction Watermark Control + +Rubik allows the eviction of offline pods based on node memory utilization to prevent memory resource overload. You can configure a memory eviction watermark. If the node memory utilization surpasses the watermark, Rubik evicts offline pods with high resource usage and shorter durations to release resources. + +> ![](./figures/icon-note.gif) **Note**: +> +> The `volcano.sh/preemptable="true"/"false"` annotation distinguishes online and offline services. +> +> ```yaml +> annotations: +> volcano.sh/preemptable: true +> ``` + +**Configuration details** + +```json +{ + "agent": { + "enabledFeatures": [ + "memoryevict" + ] + } + "memoryevict": { + "threshold": 60, + "interval": 1, + "cooldown": 4 + } +} +``` + +For more details, see [Configuration Documentation](./configuration-description.md#memory-eviction-watermark-control). diff --git a/docs/en/docs/rubik/overview.md b/docs/en/docs/rubik/overview.md index 7c9aa04a502613ea7eb83fff57430a096ee1e232..0ac66c534614bd15f26414c96b8908f9dbbd8405 100644 --- a/docs/en/docs/rubik/overview.md +++ b/docs/en/docs/rubik/overview.md @@ -8,10 +8,20 @@ In hybrid service deployment scenarios, Rubik can properly schedule resources ba Rubik supports the following features: -- Pod CPU priority configuration -- Pod memory priority configuration +- [Absolute Preemption](./modules.md#absolute-preemption) + - [CPU Absolute Preemption](./modules.md#cpu-absolute-preemption) + - [Memory Absolute Preemption](./modules.md#memory-absolute-preemption) +- [dynCache Memory Bandwidth and L3 Cache Access Limit](./modules.md#dyncache-memory-bandwidth-and-l3-cache-access-limit) +- [dynMemory Tiered Memory Reclamation](./modules.md#dynmemory-tiered-memory-reclamation) +- [Flexible Bandwidth](./modules.md#flexible-bandwidth) + - [quotaTurbo User-Mode Solution](./modules.md#quotaburst-kernel-mode-solution) + - [quotaTurbo Configuration](./modules.md#quotaturbo-user-mode-solution) +- [I/O Weight Control Based on ioCost](./modules.md#io-weight-control-based-on-iocost) +- [Interference Detection Based on Pressure Stall Information Metrics](./modules.md#interference-detection-based-on-pressure-stall-information-metrics) +- [CPU Eviction Watermark Control](./modules.md#cpu-eviction-watermark-control) +- [Memory Eviction Watermark Control](./modules.md#memory-eviction-watermark-control) This document is intended for community developers, open source enthusiasts, and partners who use the openEuler system and want to learn and use Rubik. Users must: -* Know basic Linux operations. -* Be familiar with basic operations of Kubernetes and Docker/iSulad. +- Know basic Linux operations. +- Be familiar with basic operations of Kubernetes and Docker/iSulad. diff --git a/docs/en/docs/secGear/introduction-to-secGear.md b/docs/en/docs/secGear/introduction-to-secGear.md index aa0bc3a4db74c44bdcc2b3fcaf777e1070e79c41..6d4c9d98530399e0cf123b7c1a9b920258ed428d 100644 --- a/docs/en/docs/secGear/introduction-to-secGear.md +++ b/docs/en/docs/secGear/introduction-to-secGear.md @@ -2,7 +2,7 @@ ## Overview -With the rapid development of cloud computing, more and more enterprises deploy computing services on the cloud. The security of user data on the third-party cloud infrastructure is facing great challenges. Confidential computing is a technology that uses hardware-based trusted execution environments (TEEs) to protect confidentiality and integrity of data in use. It relies on the bottom-layer hardware to build the minimum trust dependency, which removes the OS, hypervisor, infrastructure, system administrator, and service provider from the trusted entity list as unauthorized entities to reduce potential risks. There are various confidential computing technologies (such as Intel SGX, Arm TrustZone, and RSIC-V Keystone) and software development kits (SDKs) in the industry and the application ecosystem of different TEEs are isolated from each other, which brings high development and maintenance costs to confidential computing application developers. To help developers quickly build confidential computing solutions that protect data security on the cloud, openEuler launches the unified confidential computing programming framework secGear. +With the rapid development of cloud computing, more and more enterprises deploy computing services on the cloud. The security of user data on the third-party cloud infrastructure is facing great challenges. Confidential computing is a technology that uses hardware-based trusted execution environments (TEEs) to protect confidentiality and integrity of data in use. It relies on the bottom-layer hardware to build the minimum trust dependency, which removes the OS, hypervisor, infrastructure, system administrator, and service provider from the trusted entity list as unauthorized entities to reduce potential risks. There are various confidential computing technologies (such as Intel SGX, Arm TrustZone, and RISC-V Keystone) and software development kits (SDKs) in the industry and the application ecosystem of different TEEs are isolated from each other, which brings high development and maintenance costs to confidential computing application developers. To help developers quickly build confidential computing solutions that protect data security on the cloud, openEuler launches the unified confidential computing programming framework secGear. ## Architecture @@ -42,6 +42,7 @@ Switchless is a technology that uses shared memory to reduce the number of conte uint32_t parameter_num; uint32_t workers_policy; uint32_t rollback_to_common; + cpu_set_t num_cores; } cc_sl_config_t; ``` @@ -55,8 +56,9 @@ Switchless is a technology that uses shared memory to reduce the number of conte | parameter_num | Maximum number of parameters supported by a switchless function. This field takes effect only on the Arm platform.
Specifications:
Arm: maximum value: **16**; minimum value: **0**.| | workers_policy | Running mode of the switchless proxy thread. This field takes effect only on the Arm platform.
Specifications:
Arm:
**WORKERS_POLICY_BUSY**: The proxy thread always occupies CPU resources regardless of whether there are tasks to be processed. This mode applies to scenarios that require high performance and extensive system software and hardware resources.
**WORKERS_POLICY_WAKEUP**: The proxy thread wakes up only when there is a task. After the task is processed, the proxy thread enters the sleep state and waits to be woken up by a new task.| | rollback_to_common | Whether to roll back to a common call when an asynchronous switchless call fails. This field takes effect only on the Arm platform.
Specifications:
Arm:
**0**: No. If the operation fails, only the error code is returned.
Other values: Yes. If the operation fails, an asynchronous switchless call is rolled back to a common call and the return value of the common call is returned.| + | num_cores | Number of cores for TEE core binding
Specifications: The maximum value is the number of cores in the environment. | -2. Add the **transition_using_threads** flag when defining the API in the enclave description language (EDL) file. +1. Add the **transition_using_threads** flag when defining the API in the enclave description language (EDL) file. ```ocaml enclave { @@ -83,6 +85,7 @@ A secure channel is a technology that combines confidential computing remote att #### How to Use The secure channel is provided as a library and consists of the client, host, and enclave, which are icalled by the client, server client application (CA), and server trusted application (TA) of the service program respectively. + | Module | Header File | Library File | Dependency | |------------|--------------------------|-----------------------|---------| | Client | secure_channel_client.h | libcsecure_channel.so | OpenSSL| @@ -110,6 +113,50 @@ The secure channel is provided as a library and consists of the client, host, an A secure channel encapsulates only the key negotiation process and encryption and decryption APIs, but does not establish any network connection. The negotiation process reuses the network connection of the service. The network connection between the client and server is established and maintained by the service. The message sending hook function and network connection pointer are transferred during the initialization of the secure channel on the client and the server. For details, see [secure channel examples](https://gitee.com/openeuler/secGear/tree/master/examples/secure_channel). +### Remote Attestation + +#### Challenges + +As confidential computing technologies advance, several major platforms have emerged, including Arm Trustzone/CCA, Intel SGX/TDX, QingTian Enclave, and Hygon CSV. Solutions often involve multiple confidential computing hardware platforms, sometimes requiring collaboration between different TEEs. Remote attestation is a crucial part of the trust chain in any confidential computing technology. However, each technology has its own attestation report format and verification process. This forces users to integrate separate verification workflows for each TEE, increasing complexity and hindering the adoption of new TEE types. + +#### Solution + +The unified remote attestation framework of secGear addresses the key components related to remote attestation in confidential computing, abstracting away the differences between different TEEs. It provides two components: attestation agent and attestation service. The agent is integrated by users to obtain attestation reports and connect to the attestation service. The service can be deployed independently and supports the verification of iTrustee and virtCCA remote attestation reports. + +#### Feature Description + +The unified remote attestation framework focuses on confidential computing functionalities, while service deployment and operation capabilities are provided by third-party deployment services. The key features of the unified remote attestation framework are as follows: + +- Report verification plugin framework: Supports runtime compatibility with attestation report verification for different TEE platforms, such as iTrustee, virtCCA, and CCA. It also supports the extension of new TEE report verification plugins. +- Certificate baseline management: Supports the management of baseline values of Trusted Computing Bases (TCB) and Trusted Applications (TA) as well as public key certificates for different TEE types. Centralized deployment on the server ensures transparency for users. +- Policy management: Provides default policies for ease of use and customizable policies for flexibility. +- Identity token: Issues identity tokens for different TEEs, endorsed by a third party for mutual authentication between different TEE types. +- Attestation agent: Supports connection to attestation service/peer-to-peer attestation, compatible with TEE report retrieval and identity token verification. It is easy to integrate, allowing users to focus on their service logic. + +Two modes are supported depending on the usage scenario: peer-to-peer verification and attestation service verification. + +Attestation service verification process: + +1. The user (regular node or TEE) initiates a challenge to the TEE platform. +2. The TEE platform obtains the TEE attestation report through the attestation agent and returns it to the user. +3. The user-side attestation agent forwards the report to the remote attestation service. +4. The remote attestation service verifies the report and returns an identity token in a unified format endorsed by a third party. +5. The attestation agent verifies the identity token and parses the attestation report verification result. +6. Upon successful verification, a secure connection is established. + +Peer-to-peer verification process (without the attestation service): + +1. The user initiates a challenge to the TEE platform, which then returns the attestation report to the user. +2. The user uses a local peer-to-peer TEE verification plugin to verify the report. + +> ![](./public_sys-resources/icon-note.gif) **Note:** +> +> The attestation agent varies depending on whether peer-to-peer verification or remote attestation service verification is used. Users can select the desired mode during compilation by specifying the appropriate option, enabling the attestation agent to support either the attestation service or peer-to-peer mode. + +#### Application Scenarios + +In scenarios like finance and AI, where confidential computing is used to protect the security of privacy data during runtime, remote attestation is a technical means to verify the legitimacy of the confidential computing environment and applications. secGear provides components that are easy to integrate and deploy, helping users quickly enable confidential computing remote attestation capabilities. + ## Acronyms and Abbreviations | Acronym/Abbreviation| Full Name | diff --git a/docs/en/docs/secGear/secGear-installation.md b/docs/en/docs/secGear/secGear-installation.md index 277cb1631fadc6d29222ca050f03550e413efde8..b548cc2491ca2dfb08bd1615a6f0be729361dd73 100644 --- a/docs/en/docs/secGear/secGear-installation.md +++ b/docs/en/docs/secGear/secGear-installation.md @@ -20,49 +20,41 @@ > - For common servers, the TrustZone feature cannot be enabled only by upgrading the BMC, BIOS, and TEE OS firmware. > - By default, the TrustZone feature is disabled on the server. For details about how to enable the TrustZone feature on the server, see BIOS settings. -#### OS - -openEuler 20.03 LTS SP2 or later - -openEuler 22.09 - -openEuler 22.03 LTS or later - ### Environment Preparation - For details, see [Environment Requirements](https://www.hikunpeng.com/document/detail/en/kunpengcctrustzone/fg-tz/kunpengtrustzone_04_0006.html) and [Procedure](https://www.hikunpeng.com/document/detail/en/kunpengcctrustzone/fg-tz/kunpengtrustzone_04_0007.html) on the Kunpeng official website. +For details, see [Environment Requirements](https://www.hikunpeng.com/document/detail/en/kunpengcctrustzone/fg-tz/kunpengtrustzone_20_0018.html) and [Procedure](https://www.hikunpeng.com/document/detail/en/kunpengcctrustzone/fg-tz/kunpengtrustzone_20_0019.html) on the Kunpeng official website. ### Installation -1. Configure the openEuler Yum source. You can configure an online Yum source or configure a local Yum source by mounting an ISO file. The following uses openEuler 22.03 LTS as an example. For other versions, use the Yum source of the corresponding version. +1. Configure the openEuler Yum repository. You can configure an online Yum repository (see the example below) or configure a local Yum repository by mounting an ISO file. - ```shell - vi openEuler.repo - [osrepo] - name=osrepo - baseurl=http://repo.openeuler.org/openEuler-22.03-LTS/everything/aarch64/ - enabled=1 - gpgcheck=1 - gpgkey=http://repo.openeuler.org/openEuler-22.03-LTS/everything/aarch64/RPM-GPG-KEY-openEuler - ``` + ```shell + vi /etc/yum.repo/openEuler.repo + [osrepo] + name=osrepo + baseurl=http://repo.openeuler.org/openEuler-{version}/everything/aarch64/ + enabled=1 + gpgcheck=1 + gpgkey=http://repo.openeuler.org/openEuler-{version}/everything/aarch64/RPM-GPG-KEY-openEuler + ``` 2. Install secGear. - ```shell - #Install the compiler. - yum install cmake ocaml-dune - - #Install secGear. - yum install secGear-devel - - #Check whether the installations are successful. If the command output is as follows, the installations are successful. - rpm -qa | grep -E 'secGear|itrustee|ocaml-dune' - itrustee_sdk-xxx - itrustee_sdk-devel-xxx - secGear-xxx - secGear-devel-xxx - ocaml-dune-xxx - ``` + ```shell + #Install the compiler. + yum install cmake ocaml-dune + + #Install secGear. + yum install secGear-devel + + #Check whether the installations are successful. If the command output is as follows, the installations are successful. + rpm -qa | grep -E 'secGear|itrustee|ocaml-dune' + itrustee_sdk-xxx + itrustee_sdk-devel-xxx + secGear-xxx + secGear-devel-xxx + ocaml-dune-xxx + ``` ## x86 Environment @@ -72,55 +64,47 @@ openEuler 22.03 LTS or later Processor that supports the Intel SGX feature -#### OS - -openEuler 20.03 LTS SP2 or later - -openEuler 22.09 - -openEuler 22.03 LTS or later - ### Environment Preparation Purchase a device that supports the Intel SGX feature and enable the SGX feature by referring to the BIOS setting manual of the device. ### Installation -1. Configure the openEuler Yum source. You can configure an online Yum source or configure a local Yum source by mounting an ISO file. The following uses openEuler 22.03 LTS as an example. For other versions, use the Yum source of the corresponding version. +1. Configure the openEuler Yum repository. You can configure an online Yum repository (see the example below) or configure a local Yum repository by mounting an ISO file. - ```shell - vi openEuler.repo - [osrepo] - name=osrepo - baseurl=http://repo.openeuler.org/openEuler-22.03-LTS/everything/x86_64/ - enabled=1 - gpgcheck=1 - gpgkey=http://repo.openeuler.org/openEuler-22.03-LTS/everything/x86_64/RPM-GPG-KEY-openEuler - ``` + ```shell + vi openEuler.repo + [osrepo] + name=osrepo + baseurl=http://repo.openeuler.org/openEuler{version}/everything/x86_64/ + enabled=1 + gpgcheck=1 + gpgkey=http://repo.openeuler.org/openEuler-{version}/everything/x86_64/RPM-GPG-KEY-openEuler + ``` 2. Install secGear. - ```shell - #Install the compiler. - yum install cmake ocaml-dune - - #Install secGear. - yum install secGear-devel - - #Check whether the installations are successful. If the command output is as follows, the installations are successful. - rpm -qa | grep -E 'secGear|ocaml-dune|sgx' - secGear-xxx - secGear-devel-xxx - ocaml-dune-xxx - libsgx-epid-xxx - libsgx-enclave-common-xxx - libsgx-quote-ex-xxx - libsgx-aesm-launch-plugin-xxx - libsgx-uae-service-xxx - libsgx-ae-le-xxx - libsgx-urts-xxx - sgxsdk-xxx - sgx-aesm-service-xxx - linux-sgx-driver-xxx - libsgx-launch-xxx - ``` + ```shell + # Install the compiler. + yum install cmake ocaml-dune + + # Install secGear. + yum install secGear-devel + + # Check whether the installations are successful. If the command output is as follows, the installations are successful. + rpm -qa | grep -E 'secGear|ocaml-dune|sgx' + secGear-xxx + secGear-devel-xxx + ocaml-dune-xxx + libsgx-epid-xxx + libsgx-enclave-common-xxx + libsgx-quote-ex-xxx + libsgx-aesm-launch-plugin-xxx + libsgx-uae-service-xxx + libsgx-ae-le-xxx + libsgx-urts-xxx + sgxsdk-xxx + sgx-aesm-service-xxx + linux-sgx-driver-xxx + libsgx-launch-xxx + ``` diff --git a/docs/en/menu/index.md b/docs/en/menu/index.md index b9274c4b19a2a273914421cbdb2c0e0501f5b56d..6231e30edef61f188c77ff15edcf52a519af1158 100644 --- a/docs/en/menu/index.md +++ b/docs/en/menu/index.md @@ -33,7 +33,7 @@ headless: true - [Installing on QEMU]({{< relref "./docs/Installation/RISC-V-QEMU.md" >}}) - [Installing on Pioneer Box]({{< relref "./docs/Installation/RISC-V-Pioneer1.3.md" >}}) - [Installing on Licheepi4A]({{< relref "./docs/Installation/RISC-V-LicheePi4A.md" >}}) - - [RISCV-OLK6.6 Source-Compatible Version Guide]({{< relref "./docs/Installation/RISCV-OLK6.6.md" >}}) + - [RISC-V-OLK6.6 Version Alignment Guide]({{< relref "./docs/Installation/RISC-V-OLK6.6-version-alignment.md" >}}) - [OS Management](#) - [Administrator Guide]({{< relref "./docs/Administration/administration.md" >}}) - [Viewing System Information]({{< relref "./docs/Administration/viewing-system-information.md" >}}) @@ -56,6 +56,12 @@ headless: true - [Configuring the Web Server]({{< relref "./docs/Administration/configuring-the-web-server.md" >}}) - [Setting Up the Database Server]({{< relref "./docs/Administration/setting-up-the-database-server.md" >}}) - [Trusted Computing]({{< relref "./docs/Administration/trusted-computing.md" >}}) + - [Kernel Integrity Measurement Architecture (IMA)]({{< relref "./docs/Administration/IMA.md" >}}) + - [Dynamic Integrity Measurement (DIM)]({{< relref "./docs/Administration/DIM.md" >}}) + - [Remote Attestation (Kunpeng Security Library)]({{< relref "./docs/Administration/remote-attestation-kunpeng-security-library.md" >}}) + - [Trusted Platform Control Module]({{< relref "./docs/Administration/TPCM.md" >}}) + - [Security Protection for Interpreter-based Applications]({{< relref "./docs/Administration/security-protection-for-interpreter-based-applications.md" >}}) + - [Kernel Root of Trust Framework]({{< relref "./docs/Administration/kernel-rot.md" >}}) - [FAQs]({{< relref "./docs/Administration/faqs.md" >}}) - [O&M Guide]({{< relref "./docs/ops_guide/overview.md" >}}) - [O&M Overview]({{< relref "./docs/ops_guide/om-overview.md" >}}) @@ -103,6 +109,7 @@ headless: true - [Kernel Parameters]({{< relref "./docs/SecHarden/kernel-parameters.md" >}}) - [SELinux Configuration]({{< relref "./docs/SecHarden/selinux-configuration.md" >}}) - [Security Hardening Tools]({{< relref "./docs/SecHarden/security-hardening-tools.md" >}}) + - [Security Configuration Hardening Tool]({{< relref "./docs/SecHarden/security-configuration-hardening-tool.md" >}}) - [Appendix]({{< relref "./docs/SecHarden/appendix.md" >}}) - [secGear Developer Guide]({{< relref "./docs/secGear/secGear.md" >}}) - [Introduction to secGear]({{< relref "./docs/secGear/introduction-to-secGear.md" >}}) @@ -112,6 +119,17 @@ headless: true - [Certificate Signature]({{< relref "./docs/CertSignature/overview_of_certificates_and_signatures.md" >}}) - [Introduction to Signature Certificates]({{< relref "./docs/CertSignature/introduction_to_signature_certificates.md" >}}) - [Secure Boot]({{< relref "./docs/CertSignature/secure_boot.md" >}}) + - [ShangMi]({{< relref "./docs/ShangMi/overview.md" >}}) + - [Drive Encryption]({{< relref "./docs/ShangMi/drive-encryption.md" >}}) + - [Kernel Module Signing]({{< relref "./docs/ShangMi/kernel-module-signing.md" >}}) + - [Algorithm Library]({{< relref "./docs/ShangMi/algorithm-library.md" >}}) + - [File Integrity Protectio]({{< relref "./docs/ShangMi/file-integrity-protection.md" >}}) + - [User Identity Authentication]({{< relref "./docs/ShangMi/user-identity-authentication.md" >}}) + - [Certificates]({{< relref "./docs/ShangMi/certificates.md" >}}) + - [Secure Boot]({{< relref "./docs/ShangMi/secure-boot.md" >}}) + - [SSH Stack]({{< relref "./docs/ShangMi/ssh-stack.md" >}}) + - [TLCP Stack]({{< relref "./docs/ShangMi/tlcp-stack.md" >}}) + - [RPM Signature Verification]({{< relref "./docs/ShangMi/rpm-signature-verification.md" >}}) - [Performance](#) - [A-Tune User Guide]({{< relref "./docs/A-Tune/A-Tune.md" >}}) - [Getting to Know A-Tune]({{< relref "./docs/A-Tune/getting-to-know-a-tune.md" >}}) @@ -142,7 +160,7 @@ headless: true - [Kiran Installation]({{< relref "./docs/desktop/install-kiran.md" >}}) - [Kiran User Guide]({{< relref "./docs/desktop/Kiran_userguide.md" >}}) - [Embedded](#) - - [openEuler Embedded User Guide](https://embedded.pages.openeuler.org/openEuler-24.03-LTS/index.html) + - [openEuler Embedded User Guide](https://pages.openeuler.openatom.cn/embedded/docs/build/html/openEuler-24.03-LTS/index.html) - [Virtualization](#) - [Virtualization User Guide]({{< relref "./docs/Virtualization/virtualization.md" >}}) - [Introduction to Virtualization]({{< relref "./docs/Virtualization/introduction-to-virtualization.md" >}}) @@ -253,13 +271,15 @@ headless: true - [Deploying a Cluster]({{< relref "./docs/Kubernetes/eggo-deploying-a-cluster.md" >}}) - [Dismantling a Cluster]({{< relref "./docs/Kubernetes/eggo-dismantling-a-cluster.md" >}}) - [Running the Test Pod]({{< relref "./docs/Kubernetes/running-the-test-pod.md" >}}) + - [Kubernetes Cluster Deployment Guide Based on containerd]({{< relref "./docs/Kubernetes/kubernetes-cluster-deployment-guide-based-on-containerd.md" >}}) - [Rubik User Guide]({{< relref "./docs/rubik/overview.md" >}}) - [Installation and Deployment]({{< relref "./docs/rubik/installation-and-deployment.md" >}}) - - [HTTP APIs]({{< relref "./docs/rubik/http-apis.md" >}}) + - [Introduction]({{< relref "./docs/rubik/modules.md" >}}) + - [Configuration Description]({{< relref "./docs/rubik/configuration-description.md" >}}) - [Example of Isolation for Hybrid Deployed Services]({{< relref "./docs/rubik/example-of-isolation-for-hybrid-deployed-services.md" >}}) + - [Appendix]({{< relref "./docs/rubik/appendix.md" >}}) - [NestOS User Guide]({{< relref "./docs/NestOS/overview.md" >}}) - - [Installation and Deployment]({{< relref "./docs/NestOS/installation-and-deployment.md" >}}) - - [Setting Up Kubernetes and iSulad]({{< relref "./docs/NestOS/usage.md" >}}) + - [NestOS User Guide]({{< relref "./docs/NestOS/NestOS-user-guide.md" >}}) - [Feature Description]({{< relref "./docs/NestOS/feature-description.md" >}}) - [Kmesh User Guide]({{< relref "./docs/Kmesh/Kmesh.md" >}}) - [Introduction to Kmesh]({{< relref "./docs/Kmesh/introduction-to-kmesh.md" >}}) @@ -285,9 +305,36 @@ headless: true - [Eulerlauncher User Guide]({{< relref "./docs/Eulerlauncher/overall.md" >}}) - [Installing and Running EulerLauncher on Windows]({{< relref "./docs/Eulerlauncher/win-user-manual.md" >}}) - [Installing and Running Eulerlauncher on macOS]({{< relref "./docs/Eulerlauncher/mac-user-manual.md" >}}) + - [epkg User Guide]({{< relref "./docs/epkg/epkg-user-guide.md" >}}) - [openEuler DevOps](#) - [Patch Tracking]({{< relref "./docs/userguide/patch-tracking.md" >}}) - [pkgship]({{< relref "./docs/userguide/pkgship.md" >}}) +- [AI](#) + - [openEuler Copilot System]({{< relref "./docs/AI/openEuler_Copilot_System/README.md" >}}) + - [使用指南](#) + - [网页端](#) + - [前言]({{< relref "./docs/AI/openEuler_Copilot_System/使用指南/线上服务/前言.md" >}}) + - [注册与登录]({{< relref "./docs/AI/openEuler_Copilot_System/使用指南/线上服务/注册与登录.md" >}}) + - [智能问答使用指南]({{< relref "./docs/AI/openEuler_Copilot_System/使用指南/线上服务/智能问答使用指南.md" >}}) + + - [命令行客户端](#) + + + + + - [知识库管理](#) + - [witChainD 使用指南]({{< relref "./docs/AI/openEuler_Copilot_System/使用指南/知识库管理/witChainD使用指南.md" >}}) + - [部署指南](#) + + + + - [插件部署指南](#) + + + + - [AI大模型服务镜像使用指南]({{< relref "./docs/AI/AI大模型服务镜像使用指南.md" >}}) + - [AI容器镜像用户指南]({{< relref "./docs/AI/AI容器镜像用户指南.md" >}}) + - [oeDeploy使用指南]({{< relref "./docs/AI/oeDeploy使用指南.md" >}}) - [Application Development](#) - [Application Development Guide]({{< relref "./docs/ApplicationDev/application-development.md" >}}) - [Preparing the Development Environment]({{< relref "./docs/ApplicationDev/preparations-for-development-environment.md" >}}) @@ -299,3 +346,4 @@ headless: true - [FAQ]({{< relref "./docs/ApplicationDev/FAQ.md" >}}) - [GCC User Guide]({{< relref "./docs/GCC/overview.md" >}}) - [Kernel FDO User Guide]({{< relref "./docs/GCC/kernel_FDO_user_guide.md" >}}) + - [AI4C User Guide]({{< relref "./docs/AI4C/AI4C-user-guide.md" >}}) \ No newline at end of file diff --git "a/docs/zh/docs/Container/iSula-shim-v2\345\257\271\346\216\245stratovirt.md" "b/docs/zh/docs/Container/iSula-shim-v2\345\257\271\346\216\245stratovirt.md" old mode 100755 new mode 100644 index c604fb8e6adff38c1fc7c610552f3eaf1b11cf28..bfadc6fecca2e288278235fdb0fd850e83f4d86a --- "a/docs/zh/docs/Container/iSula-shim-v2\345\257\271\346\216\245stratovirt.md" +++ "b/docs/zh/docs/Container/iSula-shim-v2\345\257\271\346\216\245stratovirt.md" @@ -1,7 +1,5 @@ # iSula对接shim v2安全容器 - - ## 概述 shim v2 是新一代 shim 架构方案，相比于 shim v1, 具有调用链更短、架构清晰的优势，在多业务容器场景，具备明显的低内存开销优势。iSula 运行安全容器可以通过 isulad-shim 或者 containerd-shim-kata-v2 来实现，其中 isulad-shim 组件是 shim v1 方案的具体实现，containerd-shim-kata-v2 组件是 shim v2 方案在安全容器场景的一种具体实现，本文介绍 iSula 与 containerd-shim-kata-v2 的对接。 @@ -58,7 +56,7 @@ iSula 对接 containerd-shim-kata-v2 前，需要满足如下前提： Logical volume "thinpoolmeta" created. ``` -4. 将上面创建的逻辑卷转换为 thinpool +4. 将上面创建的逻辑卷转换为 thinpool ```shell $ lvconvert -y --zero n -c 64K \ @@ -117,7 +115,7 @@ iSula 对接 containerd-shim-kata-v2 前，需要满足如下前提： 若回显有如下信息，说明配置成功。 - ``` + ```text Storage Driver: devicemapper ``` @@ -135,7 +133,7 @@ containerd-shim-kata-v2 使用的虚拟化组件为 QEMU 时，iSula 对接 cont sandbox_cgroup_with_emulator 需要设置为 false， 目前 shimv2 不支该改功能， 其他参数与 shim v1 中 kata 配置参数保持一致或者保持缺省值。 - ``` + ```text sandbox_cgroup_with_emulator = false ``` @@ -150,7 +148,7 @@ containerd-shim-kata-v2 使用的虚拟化组件为 QEMU 时，iSula 对接 cont 3. 确认 qemu 虚拟机进程被拉起，说明 qemu 和 shim v2 安全容器的对接成功 ```bash - $ ps -ef | grep qemu + ps -ef | grep qemu ``` #### 使用 StratoVirt @@ -194,15 +192,15 @@ containerd-shim-kata-v2 使用的虚拟化组件为 StratoVirt 时，iSula 对 StratoVirt 中使用 vsock 功能， 需要开启 vhost_vsock 内核模块并确认是否开启成功 ```bash - $ modprobe vhost_vsock - $ lsmod |grep vhost_vsock + modprobe vhost_vsock + lsmod |grep vhost_vsock ``` - 下载对应版本和架构的 kernel 并放到 /var/lib/kata/ 路径下, [openeuler repo]()： + 下载对应版本和架构的 kernel 并放到 /var/lib/kata/ 路径下, openeuler repo (`https://repo.openeuler.org/`)： ```bash - $ cd /var/lib/kata - $ wget https://repo.openeuler.org/openEuler-{version}/stratovirt_img/x86_64/vmlinux.bin + cd /var/lib/kata + wget https://repo.openeuler.org/openEuler-{version}/stratovirt_img/x86_64/vmlinux.bin ``` 3. 使用 busybox 镜像运行安全容器并检查使用的 runtime 为 io.containerd.kata.v2 @@ -216,5 +214,5 @@ containerd-shim-kata-v2 使用的虚拟化组件为 StratoVirt 时，iSula 对 4. 确认 stratovirt 虚拟机进程被拉起，说明 StratoVirt 和 shim v2 安全容器的对接成功 ```bash - $ ps -ef | grep stratovirt + ps -ef | grep stratovirt ``` diff --git "a/docs/zh/docs/Pin/\346\217\222\344\273\266\346\241\206\346\236\266\347\211\271\346\200\247\347\224\250\346\210\267\346\214\207\345\215\227.md" "b/docs/zh/docs/Pin/\346\217\222\344\273\266\346\241\206\346\236\266\347\211\271\346\200\247\347\224\250\346\210\267\346\214\207\345\215\227.md" old mode 100755 new mode 100644 index a560e02dfe30118ea33c81a82b73f32027a68022..62cc765532961738e904cae69162bef30b248082 --- "a/docs/zh/docs/Pin/\346\217\222\344\273\266\346\241\206\346\236\266\347\211\271\346\200\247\347\224\250\346\210\267\346\214\207\345\215\227.md" +++ "b/docs/zh/docs/Pin/\346\217\222\344\273\266\346\241\206\346\236\266\347\211\271\346\200\247\347\224\250\346\210\267\346\214\207\345\215\227.md" @@ -1,13 +1,22 @@ # 安装与部署 + ## 软件要求 + * 操作系统：openEuler 24.03 + ## 硬件要求 + * x86_64架构 * ARM架构 + ## 环境准备 + * 安装openEuler系统，安装方法参考 《[安装指南](../Installation/installation.md)》。 + ### 安装依赖软件 + #### 安装插件框架GCC客户端依赖软件 + ```shell yum install -y grpc yum install -y grpc-devel @@ -20,7 +29,9 @@ yum install -y llvm-mlir yum install -y llvm-mlir-devel yum install -y llvm-devel ``` + #### 安装插件框架服务端依赖软件 + ```shell yum install -y grpc yum install -y grpc-devel @@ -32,9 +43,13 @@ yum install -y llvm-mlir yum install -y llvm-mlir-devel yum install -y llvm-devel ``` + ## 安装Pin + ### rpm构建 + #### 构建插件框架GCC客户端 + ```shell git clone https://gitee.com/src-openeuler/pin-gcc-client.git cd pin-gcc-client @@ -44,7 +59,9 @@ rpmbuild -ba pin-gcc-client.spec cd ~/rpmbuild/RPMS rpm -ivh pin-gcc-client.rpm ``` + #### 构建插件框架服务端 + ```shell git clone https://gitee.com/src-openeuler/pin-server.git cd pin-server @@ -54,8 +71,11 @@ rpmbuild -ba pin-server.spec cd ~/rpmbuild/RPMS rpm -ivh pin-server.rpm ``` + ### 编译构建 + #### 构建插件框架GCC客户端 + ```shell git clone https://gitee.com/openeuler/pin-gcc-client.git cd pin-gcc-client @@ -64,7 +84,9 @@ cd build cmake ../ -DMLIR_DIR=${MLIR_PATH} -DLLVM_DIR=${LLVM_PATH} make ``` + #### 构建插件框架服务端 + ```shell git clone https://gitee.com/openeuler/pin-server.git cd pin-server @@ -75,8 +97,10 @@ make ``` # 使用方法 + 用户可以通过`-fplugin`和`-fplugin-arg-libpin_xxx`使能插件工具。 命令如下： + ```shell $(TARGET): $(OBJS) $(CXX) -fplugin=${CLIENT_PATH}/build/libpin_gcc_client.so \ @@ -104,5 +128,7 @@ $(TARGET): $(OBJS) `-fplugin-arg-libpin_gcc_client-argN`：用户可以根据插件工具要求，指定其他参数。argN代指插件工具要求的参数字段。 # 兼容性说明 + 此节主要列出当前一些特殊场景下的兼容性问题。本项目持续迭代中，会尽快进行修复，也欢迎广大开发者加入。 + * 插件框架在`-flto`阶段使能时，不支持使用`make -j`多进程编译。建议改用`make -j1`进行编译。 diff --git a/docs/zh/docs/desktop/UKUIuserguide.md b/docs/zh/docs/desktop/UKUIuserguide.md old mode 100755 new mode 100644 index 8b722ac0500c0df7b551d3898ec8daf7791035e5..dc4829a91df9a7eb21d28bb55a0e890320fb8065 --- a/docs/zh/docs/desktop/UKUIuserguide.md +++ b/docs/zh/docs/desktop/UKUIuserguide.md @@ -38,8 +38,6 @@ | 视图类型 | 提供四种视图类型：小图标、中图标、大图标、超大图标 | | 排序方式 | 提供根据文件名称、文件类型、文件大小、修改日期排列的四种方式| -
- ## 任务栏 ### 基本功能 @@ -59,8 +57,6 @@ |![](./figures/icon9.png)| 托盘菜单，包含了对声音、麒麟天气、网络连接、输入法、通知中心、日期、夜间模式的设置| |显示桌面| 按钮位于最右侧；最小化桌面的所有窗口，返回桌面；再次单击将恢复窗口| -
- #### 多视图切换 点击任务栏“![](./figures/icon5.png)”图标，即可进入如下图所示界面，在多个工作区内选择当下需要工作的操作区。 @@ -110,8 +106,6 @@ |![](./figures/icon16.png)| 复制剪切板上的该内容 |![](./figures/icon18.png)| 编辑剪切板上的该内容 | |![](./figures/icon17.png)| 删除剪切板上的该内容 | | | -
- ![图 12 剪切板](./figures/12.png) ![图 13 编辑选中实的剪切板内容](./figures/13.png) @@ -172,8 +166,6 @@ U盘插入主机后，自动读取U盘数据，点击任务栏中U盘“![](./fi |![](./figures/icon35.png)| Wifi未连接 |![](./figures/icon40.png)| Wifi连接受限 | |![](./figures/icon36.png)| Wifi已上锁 |![](./figures/icon41.png)| Wifi正在连接 | -
- ![图 22 网络连接界面](./figures/22.png) - 有线网络 @@ -237,8 +229,6 @@ U盘插入主机后，自动读取U盘数据，点击任务栏中U盘“![](./fi 用户可对任务栏的布局进行设定，在“设置任务栏”中可进行相关设置。 -
- ## 窗口 ### 窗口管理器 @@ -254,8 +244,6 @@ U盘插入主机后，自动读取U盘数据，点击任务栏中U盘“![](./fi |窗口拖拽 |在窗口标题栏长按鼠标左键，可移动窗口到任意位置 | |窗口大小调整 |将鼠标移至窗口四角，长按左键，可任意调整窗口大小 | -
- ### 窗口切换 用户有三种方式可以切换： @@ -264,9 +252,7 @@ U盘插入主机后，自动读取U盘数据，点击任务栏中U盘“![](./fi - 在桌面上点击不同窗口； -- 使用快捷键< Alt > + < Tab >； - -
+- 使用快捷键 Alt + Tab； ## 开始菜单 @@ -354,14 +340,12 @@ U盘插入主机后，自动读取U盘数据，点击任务栏中U盘“![](./fi 各个选项说明如下表。 -|选项 |说明 | -| :------| :-------- -|固定到所有用软件 |将选中软件在所有软件列表中置顶 | -|固定到任务栏 |在任务栏上生成应用的图标 | -|添加到桌面快捷方式| 在桌面生成应用的快捷方式图标 | -|卸载| 卸载软件 | - -
+| 选项 | 说明 | +| :----------------- | :----------------------------- | +| 固定到所有用软件 | 将选中软件在所有软件列表中置顶 | +| 固定到任务栏 | 在任务栏上生成应用的图标 | +| 添加到桌面快捷方式 | 在桌面生成应用的快捷方式图标 | +| 卸载 | 卸载软件 | ## 常见问题 @@ -375,8 +359,6 @@ U盘插入主机后，自动读取U盘数据，点击任务栏中U盘“![](./fi - 通过Ctrl + Alt + F1切回图形界面，输入用户密码登录。 -
- ## 附录 ### 快捷键 diff --git a/docs/zh/docs/desktop/Xfce_userguide.md b/docs/zh/docs/desktop/Xfce_userguide.md old mode 100755 new mode 100644