diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/1-database-server-exception-testing.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/1-database-server-exception-testing.md deleted file mode 100644 index d799f1af3b37587385b0cef961d1b20f7e5d49e4..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/1-database-server-exception-testing.md +++ /dev/null @@ -1,194 +0,0 @@ ---- -title: MogDB Database Server Exception Testing -summary: MogDB Database Server Exception Testing -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB Database Server Exception Testing - -## Test Objective - -The test aims to test the MogDB availability and stability in the scenarios where only one of the primary, standby, and arbitration MogDB nodes crashes, both primary and standby MogDB nodes crash, both primary and arbitration MogDB nodes crash, or both standby and arbitration MogDB nodes crash. - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :--------------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048 GB | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB 1.1.0 software package | | | -| Test tool | pgbench | | | - -## The Primary Database Node Crashes Abnormally - -Test solution: - -1. Restart the primary database node. -2. Monitor the TPS change. -3. Add a new standby node. -4. Monitor the cluster status and TPS change. - -Test procedure: - -1. Restart the primary database node. - - ```bash - reboot - ``` - -2. Make the original primary database node work as a standby database node in the cluster. - - ```bash - gs_ctl -b full build -D /gaussdata/openGauss/db1 - ``` - -3. Synchronize logs to the new standby database node until the synchronization is complete. - -Test result: - -- After the primary database node is shut down, TPS drops from 9000 to 0, which lasts for 50s or so. After a VIP is added to the new primary database node (node 2), TPS increases from 0 to 13000 or so. -- When the old primary database node is added to the cluster working as a new standby database node and operated for data synchronization, TPS drops from 13000 to 0. As the synchronization is complete, TPS is restored to 9000. - -## The Standby Database Node Crashes Abnormally - -Test solution: - -1. Restart the standby database node. - -2. Monitor the TPS change. - -3. Add the standby database node to the cluster. - -4. Monitor the cluster status and TPS change. - -Test procedure: - -1. Restart the standby database node. - - ``` - reboot - ``` - -2. Add the standby database node to the cluster. - - ``` - gs_ctl start -D /gaussdata/openGauss/db1 -M standby - ``` - -3. Synchronize data to the standby database node until the synchronization is complete. - - ``` - Standby Catchup - ``` - -Test result: - -- After the standby database node is shut down, TPS increases from 9000 or so to 13000 or so. -- After the standby database node is added to the cluster and operated for data synchronization, TPS drops from 13000 to 0. As the synchronization is complete, TPS is restored to 9000. - -## The Arbitration Node Crashes Abnormally - -Test solution: - -1. Add a VIP to the value of the **ping_list** parameter in the **node.conf** file. -2. Simulate the scenario where the arbitration node crashes. - -Test procedure: - -1. Add a VIP to the server IP addresses. - - ``` - ifconfig ens160:1 192.168.122.111 netmask 255.255.255.0 - ``` - -2. Add a VIP to the value of the **ping_list** parameter in the **node.conf** file. Synchronize the configuration file between the primary and standby database nodes and restart the monitor script. - -3. When the monitor script is running normally, disconnect the VIP of the simulated arbitration node and then observe the change. - - ``` - ifconfig ens160:1 down - ``` - -4. Manually connect the VIP of the simulated arbitration node and observe the change. - -Test result: - -- The monitor script of the primary and standby database nodes reports that the VIP of the arbitration node fails the ping operation. no other change occurs. -- TPS does not change. - -## Both Primary and Standby Database Nodes Crash Abnormally - -Test solution: - -1. Restart both the primary and standby database nodes. -2. Start the **node_run** scripts of the primary and standby database nodes. -3. Manually enable the primary and standby databases. - -Test procedure: - -1. Restart both the primary and standby database nodes. - - ``` - reboot - reboot - ``` - -2. Start the **node_run** scripts of the primary and standby database nodes. - -3. Manually enable the primary and standby databases. - -Test result: - -When the database is disabled, TPS drops to 0. After 15s the cluster is started, TPS increases to 9000. - -## Both Primary and Arbitration Nodes Crash Abnormally - -Test solution: - -Shut down the primary database node and disconnect the IP address of the arbitration node. - -Test procedure: - -1. Shut down the primary database node and disconnect the IP address of the arbitration node. - -2. Check the status of the standby database node. - -3. Start the primary database node and its monitor script. - - ``` - gs_ctl start -D /gaussdata/openGauss/db1 -M primary - ``` - -4. Check the cluster status. - -Test result: - -- After the primary database node and arbitration node are shut down, TPS drops from 9000 to 0. - -- When the primary database node restores, TPS increases to 9000. - -## Both Standby and Arbitration Nodes Crash Abnormally - -Test solution: - -Shut down the standby database node and disconnect the IP address of the arbitration node. - -Test procedure: - -1. Shut down the standby database node and disconnect the IP address of the arbitration node. - -2. Check the status of the primary database node. - - No change occurs. - -3. After the standby database node restores, start the monitor script of the standby database node. - - The error reported by the monitor script of the primary database node disappears. - -4. Start the standby database node. - -5. Check the cluster status. diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/2-network-exception-testing.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/2-network-exception-testing.md deleted file mode 100644 index 50b695b52400152bfa92fb43f1689d8352ba72f1..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/2-network-exception-testing.md +++ /dev/null @@ -1,177 +0,0 @@ ---- -title: MogDB Network Exception Testing -summary: MogDB Network Exception Testing -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB Network Exception Testing (Need to Be Reviewed Later) - -## Test Scope - -- The service NIC of the primary node is abnormal. - -- The heartbeat NIC of the primary node is abnormal. - -- Both the service and heartbeat NICs of the primary node are abnormal. - -- The service NIC of the standby node is abnormal. - -- The heartbeat NIC of the standby node is abnormal. - -- The VIP of the primary node is disconnected. - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :-------------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048 GB | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB1.1.0 software package | | | -| Test tool | pgbench | | | - -## The Service NIC of the Primary Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. -2. Observe the database status. - -Test procedure: - -1. Record the original cluster status. - -2. Disable the heartbeat NIC and then enable it. - -3. Observe the status of the standby database node. - - The standby database node cannot be connected to the primary database node. - -4. Query the cluster status. - - No primary/standby switchover occurs. - -Test result: - -1. No primary/standby switchover occurs. -2. When the network of the primary database node restores, the status of the standby database node becomes normal. -3. When the service NIC of the primary database node is abnormal, TPS decreases from 9000 to 0. After 15s when the service NIC is enabled, TPS increases from 0 to 9000. - -## The Heartbeat NIC of the Primary Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. -2. Observe the database status. - -Test procedure: - -1. Observe the database status. - -2. Disable the heartbeat NIC and then enable it. - -3. Observe the database status. - - No primary/standby switchover occurs. - -4. Observe the cluster status. - -Test result: - -1. No primary/standby switchover occurs. - -2. TPS does not change. - -## Both the Service and Heartbeat NICs of the Primary Database Node Are Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. - -2. Observe the database status. - -Test procedure: - -1. Observe the original database status. - -2. Disable the heartbeat NIC and then enable it. - - The primary database node is normal. - -Test result: - -1. No primary/standby switchover occurs. - -2. When the service NIC is disconnected, TPS drops from 9000 to 0. After the service NIC is restored to normal, TPS increases to 9000. - -## The Service NIC of the Standby Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. - -2. Observe the database status. - -Test procedure: - -1. Record the original cluster status. - -2. Disable the heartbeat NIC and then enable it. - -3. Observe the status of the primary database node. - -4. Ping the IP address of the service NIC of the standby database node. - - The IP address cannot be pinged. - - No primary/standby switchover occurs. - -Test result: - -1. The monitor script of the standby database node reports the heartbeat error indicating that no primary/standby switchover occurs. - -2. When the service NIC of the standby database node is abnormal, TPS of the primary database node increases to 13000. After the service NIC of the standby database node is restored to normal, TPS of the primary database node is restored to 9000. - -## The Heartbeat NIC of the Standby Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC 60s and then enable it. - -2. Observe the database status. - -Test procedure: - -1. Observe the original database status. - -2. Disable the heartbeat NIC 60s and then enable it. - -3. After the script is executed, observe the database status. - No primary/standby switchover occurs. The monitor script of the primary database node reports an error that the IP address of the service NIC of the standby database node cannot be pinged. - -Test result: - -No primary/standby switchover occurs. - -## The VIP of the Primary Database Node Is Disconnected - -Test solution: - -Disable the NIC **bond0:1** and then enable it. - -Test procedure: - -1. Disable the NIC **bond0:1** and then enable it. - -2. Observe the database status. - -3. Observe the VIP of the primary database node. - -Test result: - -1. The VIP of the primary database node is automatically connected. - -2. When the VIP is disconnected and then connected, the TPS status is restored to normal. diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/3-routine-maintenance-testing.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/3-routine-maintenance-testing.md deleted file mode 100644 index d50a8f807f6607bd89a1c2b875c121fb9c76f3fc..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/3-routine-maintenance-testing.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -title: Routine Maintenance Testing -summary: Routine Maintenance Testing -author: Guo Huan -date: 2021-04-25 ---- - -# Routine Maintenance Testing - -## Test Scope - -1. HA tool used for the switchover test -2. gs_ctl used for the switchover test -3. HA tool used for the failover test -4. gs_ctl used for the failover test -5. gs_ctl and HA contradiction test -6. Split-brain test for avoiding deploying two primary database nodes - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :-------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB1.1.0 | | | -| Tool | pgbench | | | - -## HA Tool Used for the Switchover Test - -Test procedure: - -1. Query the status of databases. - -2. Run the switch command on the standby database, and then query the database status. - - ```bash - /home/omm/venv/bin/python3 /home/omm/openGaussHA_standlone/switch.py -config /home/omm/openGaussHA_standlone/node.conf --switchover - ``` - -3. Query the VIP of the new primary database node. - -Test result: - -The primary/standby switchover is normal. - -## gs_ctl Used for the Switchover Test - -Test procedure: - -1. Query the database status. -2. Run the command to perform the switchover operation and query the status when finished. -3. Query the VIP. - -Test result: - -The primary/standby switchover is normal. - -## HA Tool Used for the Failover Test - -Test procedure: - -1. Query the current status of the database. - -2. Run the HA failover command. - -3. Observe the cluster status. - -4. The original standby database node works as a new primary database node. - -Test result: - -1. The original standby database node becomes the new primary database node with a VIP. - -2. The original primary database node is killed. - -3. TPS increases from 6000 to 13000 in about 10s after failover. - -## gs_ctl Used for the Failover Test - -Test procedure: - -1. Query the current status of the database. -2. Run the gs_ctl failover command. -3. Query the status of the new primary database node. -4. Query the status of the original primary database node. -5. Observe script changes. - -Test result: - -1. The new primary database node waits for 10s and then automatically shuts down, the original primary database node is not affected, and the VIP is still attached to the original primary database node. -2. About 5s or 6s after failover of the standby database node, the TPS increases from 9000 to 13000. No other changes occur except fluctuations. - -## gs_ctl and HA Contradiction Test - -Test procedure: - -1. Record the cluster status. - -2. Stop the primary and standby database nodes at the same time. - -3. Use gs_ctl to start the primary and standby database nodes. Designate the original standby database node as the primary database node, vice versa. - -4. Observe the script and cluster status. - -Test result: - -1. HA configuration is adjusted according to the actual situation. -2. TPS drops from 9000 to 0 after the primary and standby database nodes are shut down. At this time, HA detects the cluster configuration change and attaches the VIP to the new primary database node within 10s. After about 5s or 6s, TPS rises from 0 to 13000. - -## Split-Brain Test for Avoiding Deploying Two Primary Database Nodes - -Test procedure: - -1. Query the current status of the cluster. - -2. Use the gs_ctl command to restart the standby database node to make it work as the primary database node. - -Test result: - -1. After timeout for 10s, the primary database node that was restarted by an abnormal operation is killed by the monitoring script. - -2. Repair the standby database node and add it to the cluster. - - ```bash - gs_ctl build -b full -D /gaussdata/openGauss/db1 - ``` - -3. TPS increases from 6000 to 13000 after the standby database is restarted to work as the primary database node. - -4. TPS drops from 13000 to 9000 (same as that in the primary/standby deployment scenario) after the standby database node is added to the cluster. diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/4-service-exception-testing.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/4-service-exception-testing.md deleted file mode 100644 index 119b5875829cbd534eb533e50542ac8fe74fecaa..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/4-service-exception-testing.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: MogDB Service Exception Testing -summary: MogDB Service Exception Testing -author: Guo Huan -date: 2021-04-25 ---- - -# MogDB Service Exception Testing - -## Test Scope - -1. Database process exception for the standby node - -2. Monitor script exception for the standby node - -3. File system exception for the primary node (affecting HA process) - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :-------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| Hard Disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB1.1.0 | | | -| Tool | pgbench | | | - -## Database process exception for the standby node - -Test procedure: - -Kill the database process. - -Test result: - -1. Observe the cluster status. - -2. The script of the standby database node shows the heartbeat exception. - -3. No switchover occurs. - -4. Run the command on the standby database node. - - ```bash - gs_ctl start -D /gaussdata/openGauss/db1 -M standby - ``` - -5. The cluster is restored to normal and no switchover has occurred since then. - -6. When the standby database process is killed, TPS of the primary database node rises from 9000 to 13000. - -7. No primary/standby switchover occurs. - -## Monitor script exception for the standby node - -Test procedure: - -Kill the monitor script of the primary database node. - -Test result: - -1. Observe the cluster status. -2. No switchover occurs. -3. The monitor script of the primary database node reports script exception of the standby database node. -4. Restore the monitor script of the standby database node. -5. TPS maintains at 9000. - -## File system exception for the primary node (affecting HA process) - -Test procedure: - -Modify the permission of the primary database script called by HA, such as gs_ctl. - -Test result: - -1. After modifying the **rwx** permission of gs_ctl, the monitoring script of the primary database node reports heartbeat exception. - -2. The instance status cannot be detected. - -3. Query the current cluster status. - -4. After waiting about two minutes, no switchover occurs. diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/MogDB-ha-test-report.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/MogDB-ha-test-report.md deleted file mode 100644 index 1ad5c7de3fff55835ff2319899739b5bd56d5160..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/ha/MogDB-ha-test-report.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: HA Test -summary: HA Test -author: Zhang Cuiping -date: 2021-06-07 ---- - -# HA Test - -## Test Objective - -Database HA (high availability) test aims at providing users with high-quality services, avoiding service interruption caused by such faults as server crash. Its HA lies in not only whether a database can provide services consistently but whether a database can ensure data consistency. - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :---------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4, 2933MT/s | DDR4, 2933MT/s | 2048 GB | -| Hard disk | NVME 3.5 TB | NVME 3 TB | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB software package | | | -| Test tool | pgbench | | | - -## HA and Scalability Test - -| No. | Test Item | Description | -| ---- | ------------------------------------- | ------------------------------------------------------------ | -| 1 | Cluster with read-write separation | Supports separate routing of read and write requests and separate distribution of read and write tasks. | -| 2 | Capacity expansion/capacity reduction | In load scenarios, adding or reducing a physical device does not make a front-end application service interrupted. | -| 3 | Shared storage cluster | Supports two nodes sharing a storage cluster, automatic failover, and load balancing of concurrent transactions. | -| 4 | Service exception test | Guarantees the availability of an application in scenarios where the process of the standby database, the monitoring script of the standby database, and the file system of the primary database are abnormal. | -| 5 | Routine maintenance test | Supports the switchover feature provided by gs_ctl (database service control tool) and the failover feature provided by MogHA (HA component). | -| 6 | Database server exception test | Maximizes the availability and stability of an application when the primary node, standby node, or arbitration node crashes, or both the primary and standby nodes, or both the primary and arbitration nodes crash. | -| 7 | Network exception test | Maximizes the availability of an application when the service or heartbeat NIC of the primary node is abnormal, both the service and heartbeat NICs of the primary node are abnormal, the service NIC of the standby node is abnormal, or the VIP of the host is abnormal. | diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/1-performance-test-overview.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/1-performance-test-overview.md deleted file mode 100644 index d6a786378b0381b8075acc952699df1a7924370e..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/1-performance-test-overview.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: Performance Test Overview -summary: Performance Test Overview -author: Liu Xu -date: 2021-04-27 ---- - -# Performance Test Overview - -## Test Indicators - -### Data Volume - -Each repository contains 70 MB of data. To simulate the actual data environment, 1000 repositories are created during initiation. After related indexes are added, the total size of data reaches 100 GB or so. - -### Number of Concurrent Requests - -This is to simulate the real user behavior and make the number of online user requests from the clients reach a certain number. - -### Average System Workload - -System workload indicates how busy the system CPU is, for example, how many processes are there waiting for being scheduled by the CPU, which reflects the pressure of the whole system. However, average system workload indicates the average number of processes in run queues in a specified time period, generally 1, 5, or 10 minutes. - -### CPU Usage - -CPU usage indicates the quantity of CPU resources occupied by the running programs, that is, the quantity of running programs on a server at a time point. High CPU usage indicates that a server runs a plenty of programs and vice versa. Whether the CPU usage is high or low is directly related to the CPU performance. CPU usage can determine whether a CPU reaches its bottleneck. - -### IOPS - -IOPS (Input/Output Operations Per Second) is a method for measuring the performance of a computer storage device, such as HDDs, SSDs, or SANs. It refers to the number of write/read times per second, which is one of the major indicators for measuring the disk performance. IOPS indicates the number of I/O requests that can be processed by the system in unit time. The I/O requests usually refer to data write/read requests. - -### IO Latency - -IO latency is also called IO response time, referring the time period from the time when the OS kernel sends a write/read command to the time when the OS kernel receives the IO response. Time for operating single IO indicates only the time of processing single IO in a disk. However, IO latency includes even the time of waiting for being processed for an IO in a queue. - -### tpmC - -TPC-C uses three performance and price metrics. TPC-C is used to measure performance with the unit of tpmC (Transactions Per Minute). C refers to the C benchmark program of TPC. tpmC indicates the number of new orders that are processed by the system per minute. - -## Test Program Preparation - -### Test Tools - -BenchmarkSQL is a typical open source database test tool with a built-in TPC-C test script which can be directly used for testing PostgreSQL, MySQL, Oracle, and SQL Server databases. It is used for testing TPC-C of OLTP (Online Transaction Processing) via JDBC (Java Database Connectivity). - -### Test Specifications - -TPC-C offers OLTP system test specifications and tests an OLTP system using a merchandising model. Transactions are divided into five categories, the content and characteristics of which are described as follows: - -- **NewOrder - Generation of new orders** - - Transaction content: For any one client, select 5 to 15 commodities from a specified repository and create new orders with 1% of the orders that fails the operation and requires rollback. - - Main characteristics: middleweight, frequent write/read operations, and high response speed - -- **Payment - Payment of order** - - Transaction content: For any one client, select a region and customers there from a specified repository randomly, pay an order with a random amount of money, and record this operation. - - Main characteristics: lightweight, frequent write/read operations, and high response speed。 - -- **OrderStatus - Query of latest orders** - - Transaction content: For any one client, select a region and customers there from a specified repository randomly, read the last order, and display the status of each commodity in the order. - - Main characteristics: middleweight, low read-only frequency, and high response speed - -- **Delivery - Delivery of packages** - - Transaction content: For any one client, select a delivery package randomly, update the account balance of the user whose order is being processed, and delete the order from the new order list. - - Main characteristics: 1 to 10 concurrent orders in a batch, low write/read frequency, and loose response time - -- **StockLevel - Analysis of stock status** - - Transaction content: For any one client, select the last 20 orders from a specified repository of a region, check the stock of all commodities of the order list, and calculate and display the stock of all commodities with its stock level lower than the threshold generated randomly. - - Main characteristics: heavyweight, low read-only frequency, and loose response time - -### Test Rules - -Before the test, TPC-C Benchmark specifies the initial status of a database, that is, the rule of data being generated in the database. The ITEM table includes a fixed number (100,000) of commodities. The number of warehouses can be adjusted. The initialized WAREHOUSE table includes 1000 records in this test. - -- The STOCK table contains 1000 x 100,000 records. (Each warehouse contains 100,000 commodities.) -- The DISTRICT table contains 1000 x 10 records. (Each warehouse provides services for 10 regions.) -- The CUSTOMER table contains 1000 x 10 x 3000 records. (There are 3000 customers in each region.) -- The HISTORY table contains 1000 x 10 x 3000 records. (Each customer has one transaction history.) -- The ORDER table contains 1000 x 10 x 3000 records (there are 3000 orders in each region). The last 900 orders generated are added to the NEW-ORDER table. Each order generates 5 to 15 ORDER-LINE records. - -> TPC-C uses tpmC to measure the maximum qualified throughput. The number of transactions depends on that of the new order transactions, that is, the number of new orders processed per minute. diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md deleted file mode 100644 index 2cbe9addd4623fa01565bade74c9e49170186f97..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md +++ /dev/null @@ -1,401 +0,0 @@ ---- -title: Performance Test for MogDB on a Kunpeng Server -summary: Performance Test for MogDB on a Kunpeng Server -author: Liu Xu -date: 2021-03-04 ---- - -# Performance Test for MogDB on Kunpeng Servers - -## Test Objective - -This document describes the test of MogDB 2.0.0 on Kunpeng servers in scenarios where MogDB is deployed on a single node, one primary node and one standby node, or one primary node and two standby nodes (one synchronous standby and one asynchronous standby). - -## Test Environment - -### Environment Configuration - -| Category | Server Configuration | Client Configuration | Quantity | -| :---------: | :--------------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048 GB | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB 1.1.0 software package | | | - -### Test Tool - -| Name | Function | -| :--------------: | :----------------------------------------------------------- | -| BenchmarkSQL 5.0 | Open-source BenchmarkSQL developed based on Java is used for TPC-C test of OLTP database. It is used to evaluate the database transaction processing capability. | - -## Test Procedure - -### MogDB Database Operation - -1. Obtain the database installation package. - -2. Install the database. - -3. Create TPCC test user and database. - - ```sql - create user [username] identified by ‘passwd’; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. Disable the database and modify the **postgresql.conf** database configuration file by adding configuration parameters at the end of the file. - - For example, add the following parameters in the single node test: - - ```bash - max_connections = 4096 - - allow_concurrent_tuple_update = true - - audit_enabled = off - - checkpoint_segments = 1024 - - cstore_buffers =16MB - - enable_alarm = off - - enable_codegen = false - - enable_data_replicate = off - - full_page_writes = off - - max_files_per_process = 100000 - - max_prepared_transactions = 2048 - - shared_buffers = 350GB - - use_workload_manager = off - - wal_buffers = 1GB - - work_mem = 1MB - - log_min_messages = FATAL - - transaction_isolation = 'read committed' - - default_transaction_isolation = 'read committed' - - synchronous_commit = on - - fsync = on - - maintenance_work_mem = 2GB - - vacuum_cost_limit = 2000 - - autovacuum = on - - autovacuum_mode = vacuum - - autovacuum_max_workers = 5 - - autovacuum_naptime = 20s - - autovacuum_vacuum_cost_delay =10 - - xloginsert_locks = 48 - - update_lockwait_timeout =20min - - enable_mergejoin = off - - enable_nestloop = off - - enable_hashjoin = off - - enable_bitmapscan = on - - enable_material = off - - wal_log_hints = off - - log_duration = off - - checkpoint_timeout = 15min - - enable_save_datachanged_timestamp =FALSE - - enable_thread_pool = on - - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - - enable_double_write = on - - enable_incremental_checkpoint = on - - enable_opfusion = on - - advance_xlog_file_num = 10 - - numa_distribute_mode = 'all' - - track_activities = off - - enable_instr_track_wait = off - - enable_instr_rt_percentile = off - - track_counts =on - - track_sql_count = off - - enable_instr_cpu_timer = off - - plog_merge_age = 0 - - session_timeout = 0 - - enable_instance_metric_persistent = off - - enable_logical_io_statistics = off - - enable_user_metric_persistent =off - - enable_xlog_prune = off - - enable_resource_track = off - - instr_unique_sql_count = 0 - - enable_beta_opfusion = on - - enable_beta_nestloop_fusion = on - - autovacuum_vacuum_scale_factor = 0.02 - - autovacuum_analyze_scale_factor = 0.1 - - client_encoding = UTF8 - - lc_messages = en_US.UTF-8 - - lc_monetary = en_US.UTF-8 - - lc_numeric = en_US.UTF-8 - - lc_time = en_US.UTF-8 - - modify_initial_password = off - - ssl = off - - enable_memory_limit = off - - data_replicate_buffer_size = 16384 - - max_wal_senders = 8 - - log_line_prefix = '%m %u %d %h %p %S' - - vacuum_cost_limit = 10000 - - max_process_memory = 12582912 - - recovery_max_workers = 1 - - recovery_parallelism = 1 - - explain_perf_mode = normal - - remote_read_mode = non_authentication - - enable_page_lsn_check = off - - pagewriter_sleep = 100 - ``` - -### BenchmarkSQL Operation - -1. Modify the configuration file. - - Open the BenchmarkSQL installation directory and find the **[config file]** configuration file in the **run** directory. - - ``` - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. Run **runDataBuild.sh** to generate data. - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. Run **runBenchmark.sh** to test the database. - - ``` - ./runBenchmark.sh [config file] - ``` - -### OS Configuration - -1. Modify **PAGESIZE** of the OS kernel (required only in EulerOS). - - **Install kernel-4.19.36-1.aarch64.rpm (*).** - - ``` - # rpm -Uvh --force --nodeps kernel-4.19.36-1.aarch64.rpm - - *: This file is based on the kernel package of linux 4.19.36. You can acquire it from the following directory: - - 10.44.133.121 (root/Huawei12#$) - - /data14/xy_packages/kernel-4.19.36-1.aarch64.rpm - ``` - - **Modify the boot options of the root in the OS kernel configuration file.** - - ``` - \# vim /boot/efi/EFI/euleros/grubenv //Back up the grubenv file before modification. - - \# GRUB Environment Block - - saved_entry=EulerOS (4.19.36) 2.0 (SP8) -- Changed to 4.19.36 - ``` - -### File System - -1. Change the value of **blocksize** to **8K** in the XFS file system. - - #Run the following commands to check the attached NVME disks: - - ``` - # df -h | grep nvme - - /dev/nvme0n1 3.7T 2.6T 1.2T 69% /data1 - - /dev/nvme1n1 3.7T 1.9T 1.8T 51% /data2 - - /dev/nvme2n1 3.7T 2.2T 1.6T 59% /data3 - - /dev/nvme3n1 3.7T 1.4T 2.3T 39% /data4 - - Run the xfs_info command to query information about NVME disks. - - \# xfs_info /data1 - ``` - -2. Back up the required data. - -3. Format the disk. - - ``` - Use the /dev/nvme0n1 disk and /data1 load point as an example and run the folowing commands: - - umount /data1 - - mkfs.xfs -b size=8192 /dev/nvme0n1 -f - - mount /dev/nvme0n1 /data1 - ``` - -## Test Items and Conclusions - -### Test Result Summary - -| Test Item | Data Volume | Concurrent Transactions | Average CPU Usage | IOPS | IO Latency | Write Ahead Logs | tpmC | Test Time (Minute) | -| :------------------------------------: | ----------- | ----------------------- | ----------------- | ------ | ---------- | ---------------- | ---------- | :----------------: | -| Single node | 100 GB | 500 | 77.49% | 17.96K | 819.05 us | 13260 | 1567226.12 | 10 | -| One primary node and one standby node | 100 GB | 500 | 57.64% | 5.31K | 842.78 us | 13272 | 1130307.87 | 10 | -| One primary node and two standby nodes | 100 GB | 500 | 60.77% | 5.3K | 821.66 us | 14324 | 1201560.28 | 10 | - -### Single Node - -- tpmC - - ![mogdb-on-kunpeng-1](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-1.png) - -- System data - - ![mogdb-on-kunpeng-2](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-2.png) - -### One Primary Node and One Standby Node - -- tpmC - - ![mogdb-on-kunpeng-3](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-3.png) - -- System data - - ![mogdb-on-kunpeng-4](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-4.png) - -### One Primary Node and Two Standby Nodes - -- tpmC - - ![mogdb-on-kunpeng-5](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-5.png) - -- System data - - ![mogdb-on-kunpeng-6](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-6.png) diff --git a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md b/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md deleted file mode 100644 index 94644b1bd75005bd4887cf27cdc2002e79821993..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v2.1/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md +++ /dev/null @@ -1,654 +0,0 @@ ---- -title: Performance Test for MogDB on x86 Servers -summary: Performance Test for MogDB on x86 Servers -author: Liu Xu -date: 2021-03-04 ---- - -# Performance Test for MogDB on x86 Servers - -## Test Objective - -This document describes the test of MogDB 2.0.0 on x86 servers in scenarios where MogDB is deployed on a single node, one primary node and one standby node, or one primary node and two standby nodes (one synchronous standby and one asynchronous standby). - -## Test Environment - -### Environment Configuration - -| Server Type | Fit Server | NFS5280M5 | -| :---------: | :--------------------------------------: | :--------------------------------------: | -| CPU | 144cIntel® Xeon(R) Gold 6240 CPU@2.60GHz | 64cIntel® Xeon(R) Gold 5218 CPU @2.30GHz | -| Memory | 768G | 128G | -| Hard disk | SAS SSD | SAS SSD | -| NIC | 10GE | 10GE | - -### Test Tool - -| Name | Function | -| :-------------- | :----------------------------------------------------------- | -| Benchmarksql5.0 | Open-source BenchmarkSQL developed based on Java is used for TPC-C test of OLTP database. It is used to evaluate the database transaction processing capability. | - -## Test Procedure - -### MogDB Database Operation - -1. Obtain the database installation package. - -2. Install the database. - -3. Create TPCC test user and database. - - ``` - create user [username] identified by ‘passwd’; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. Disable the database and modify the **postgresql.conf** database configuration file by adding configuration parameters at the end of the file. - - For example, add the following parameters in the single node test: - - ``` - max_connections = 4096 - - allow_concurrent_tuple_update = true - - audit_enabled = off - - checkpoint_segments = 1024 - - cstore_buffers =16MB - - enable_alarm = off - - enable_codegen = false - - enable_data_replicate = off - - full_page_writes = off - - max_files_per_process = 100000 - - max_prepared_transactions = 2048 - - shared_buffers = 350GB - - use_workload_manager = off - - wal_buffers = 1GB - - work_mem = 1MB - - log_min_messages = FATAL - - transaction_isolation = 'read committed' - - default_transaction_isolation = 'read committed' - - synchronous_commit = on - - fsync = on - - maintenance_work_mem = 2GB - - vacuum_cost_limit = 2000 - - autovacuum = on - - autovacuum_mode = vacuum - - autovacuum_max_workers = 5 - - autovacuum_naptime = 20s - - autovacuum_vacuum_cost_delay =10 - - xloginsert_locks = 48 - - update_lockwait_timeout =20min - - enable_mergejoin = off - - enable_nestloop = off - - enable_hashjoin = off - - enable_bitmapscan = on - - enable_material = off - - wal_log_hints = off - - log_duration = off - - checkpoint_timeout = 15min - - enable_save_datachanged_timestamp =FALSE - - enable_thread_pool = on - - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - - enable_double_write = on - - enable_incremental_checkpoint = on - - enable_opfusion = on - - advance_xlog_file_num = 10 - - numa_distribute_mode = 'all' - - track_activities = off - - enable_instr_track_wait = off - - enable_instr_rt_percentile = off - - track_counts =on - - track_sql_count = off - - enable_instr_cpu_timer = off - - plog_merge_age = 0 - - session_timeout = 0 - - enable_instance_metric_persistent = off - - enable_logical_io_statistics = off - - enable_user_metric_persistent =off - - enable_xlog_prune = off - - enable_resource_track = off - - instr_unique_sql_count = 0 - - enable_beta_opfusion = on - - enable_beta_nestloop_fusion = on - - autovacuum_vacuum_scale_factor = 0.02 - - autovacuum_analyze_scale_factor = 0.1 - - client_encoding = UTF8 - - lc_messages = en_US.UTF-8 - - lc_monetary = en_US.UTF-8 - - lc_numeric = en_US.UTF-8 - - lc_time = en_US.UTF-8 - - modify_initial_password = off - - ssl = off - - enable_memory_limit = off - - data_replicate_buffer_size = 16384 - - max_wal_senders = 8 - - log_line_prefix = '%m %u %d %h %p %S' - - vacuum_cost_limit = 10000 - - max_process_memory = 12582912 - - recovery_max_workers = 1 - - recovery_parallelism = 1 - - explain_perf_mode = normal - - remote_read_mode = non_authentication - - enable_page_lsn_check = off - - pagewriter_sleep = 100 - ``` - -### BenchmarkSQL Operation - -1. Modify the configuration file. - - Open the BenchmarkSQL installation directory and find the **[config file]** configuration file in the **run** directory. - - ``` - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. Run **runDataBuild.sh** to generate data. - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. Run **runBenchmark.sh** to test the database. - - ``` - ./runBenchmark.sh [config file] - ``` - -### OS Parameters - -``` -vm.dirty_background_ratIO=5 - -vm.dirty_ratIO=10 - -kernel.sysrq=0 - -net.ipv4.ip_forward=0 - -net.ipv4.conf.all.send_redirects=0 - -net.ipv4.conf.default.send_redirects=0 - -net.ipv4.conf.all.accept_source_route=0 - -net.ipv4.conf.default.accept_source_route=0 - -net.ipv4.conf.all.accept_redirects=0 - -net.ipv4.conf.default.accept_redirects=0 - -net.ipv4.conf.all.secure_redirects=0 - -net.ipv4.conf.default.secure_redirects=0 - -net.ipv4.icmp_echo_ignore_broadcasts=1 - -net.ipv4.icmp_ignore_bogus_error_responses=1 - -net.ipv4.conf.all.rp_filter=1 - -net.ipv4.conf.default.rp_filter=1 - -net.ipv4.tcp_syncookies=1 - -kernel.dmesg_restrict=1 - -net.ipv6.conf.all.accept_redirects=0 - -net.ipv6.conf.default.accept_redirects=0 - -net.core.rmem_max = 21299200 - -net.core.rmem_default = 21299200 - -net.core.somaxconn = 65535 - -net.ipv4.tcp_tw_reuse = 1 - -net.sctp.sctp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -kernel.sem = 250 6400000 1000 25600 - -net.core.wmem_default = 21299200 - -kernel.shmall = 1152921504606846720 - -net.core.wmem_max = 21299200 - -net.sctp.sctp_rmem = 8192 250000 16777216 - -net.core.netdev_max_backlog = 65535 - -kernel.shmmax = 18446744073709551615 - -net.sctp.sctp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_max_syn_backlog = 65535 - -vm.oom_panic_on_oom=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -net.ipv4.tcp_timestamps = 1 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.sctp.path_max_retrans = 10 - -net.sctp.max_init_retransmits = 10 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_synack_retries = 5 - -kernel.core_uses_pid=1 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%p - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%h - -net.core.netdev_max_backlog = 65535 - -net.core.rmem_default = 21299200 - -net.core.rmem_max = 21299200 - -net.core.somaxconn = 65535 - -net.core.wmem_default = 21299200 - -net.core.wmem_max = 21299200 - -net.ipv4.conf.all.accept_redirects = 0 - -net.ipv4.conf.all.rp_filter = 1 - -net.ipv4.conf.all.secure_redirects = 0 - -net.ipv4.conf.all.send_redirects = 0 - -net.ipv4.conf.default.accept_redirects = 0 - -net.ipv4.conf.default.accept_source_route = 0 - -net.ipv4.conf.default.rp_filter = 1 - -net.ipv4.conf.default.secure_redirects = 0 - -net.ipv4.conf.default.send_redirects = 0 - -net.ipv4.conf.enp135s0.accept_redirects = 0 - -net.ipv4.conf.enp135s0.accept_source_route = 0 - -net.ipv4.conf.enp135s0.forwarding = 1 - -net.ipv4.conf.enp135s0.rp_filter = 1 - -net.ipv4.conf.enp135s0.secure_redirects = 0 - -net.ipv4.conf.enp135s0.send_redirects = 0 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_max_syn_backlog = 65535 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_mem = 362715 483620 725430 - -\#net.ipv4.tcp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.ipv4.tcp_tw_reuse = 1 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.udp_mem = 725430 967240 1450860 - -\#net.ipv4.tcp_max_orphans = 3276800 - -\#net.ipv4.tcp_fin_timeout = 60 - -\#net.ipv4.ip_local_port_range = 26000 65535 - -net.ipv4.tcp_retries2 = 80 - -\#net.ipv4.ip_local_reserved_ports = 20050-30007 - -vm.min_free_kbytes = 40140150 -``` - -### Database Parameters - -| Parameter | MogDB | -| :-------------------------------- | :--------------------------------------------- | -| listen_addresses | Specific IP address | -| port | 26000 | -| max_connectIOns | 4096 | -| wal_level | hot_standby | -| archive_mode | on | -| archive_command | /bin/ture | -| max_wal_senders | 16 | -| wal_keep_segments | 16 | -| max_replicatIOn_slots | 8 | -| hot_standby | on | -| logging_collector | on | -| log_directory | Specify the directory of the installation tool | -| log_filename | PostgreSQL-%Y-%m-%d_%H%M%S.log | -| log_min_duratIOn_statement | 1800000 | -| log_line_prefix | %m%c%d%p%a%x%n%e | -| log_timezone | PRC | -| datestyle | iso,mdy | -| timezone | PRC | -| default_text_search_config | pg_catalog.english | -| applicatIOn_name | dn_6001 | -| max_prepared_transactIOns | 2048 | -| shared_buffers | 350 GB | -| wal_buffers | 1 GB | -| work_mem | 64 MB | -| log_min_messages | FATAL | -| synchronous_commit | on | -| fsync | on | -| maintenance_work_mem | 2 GB | -| autovacuum | on | -| autovacuum_max_workers | 5 | -| autovacuum_naptime | 20s | -| autovacuum_vacuum_cost_delay | 10 | -| enable_mergejoin | off | -| enable_nestloop | off | -| enable_hashjoin | off | -| enable_bitmapscan | on | -| enable_material | off | -| wal_log_hints | off | -| log_duratIOn | off | -| checkpoint_timeout | 15 min | -| track_activities | off | -| track_counts | on | -| autovacuum_vacuum_scale_factor | 0.02 | -| autovacuum_analyze_scale_factor | 0.1 | -| ssl | off | -| local_bind_address | Specific IP address | -| max_inner_tool_connectIOns | 10 | -| password_encryptIOn_type | 0 | -| comm_tcp_mode | on | -| comm_quota_size | 1024 KB | -| max_process_memory | 700 GB | -| bulk_write_ring_size | 2 GB | -| checkpoint_segments | 1024 | -| incremental_checkpoint_timeout | 60s | -| archive_dest | /log/archive | -| enable_slot_log | off | -| data_replicate_buffer_size | 128 MB | -| walsender_max_send_size | 8 MB | -| enable_kill_query | off | -| connectIOn_alARM_rate | 0.9 | -| alARM_report_interval | 10 | -| alARM_component | /opt/huawei/snas/bin/snas_cm_cmd | -| lockwait_timeout | 1200s | -| pgxc_node_name | xxx | -| audit_directory | Specify the directory of the installation tool | -| explain_perf_mode | pretty | -| job_queue_processes | 10 | -| default_storage_nodegroup | installatIOn | -| expected_computing_nodegroup | query | -| replicatIOn_type | 1 | -| recovery_max_workers | 4 | -| available_zone | AZ1 | -| allow_concurrent_tuple_update | TRUE | -| audit_enabled | off | -| cstore_buffers | 16 MB | -| enable_alARM | off | -| enable_codegen | FALSE | -| enable_data_replicate | off | -| max_file_per_process | 10000 | -| use_workload_manager | off | -| xloginsert_locks | 48 | -| update_lockwait_timeout | 20 min | -| enable_save_datachanged_timestamp | FALSE | -| enable_thread_pool | off | -| enable_double_write | on | -| enable_incremental_checkpoint | on | -| advance_xlog_file_num | 10 | -| enable_instr_track_wait | off | -| enable_instr_rt_percentile | off | -| track_sql_count | off | -| enable_instr_cpu_timer | off | -| plog_merge_age | 0 | -| sessIOn_timeout | 0 | -| enable_instance_metric_persistent | off | -| enable_logical_IO_statistics | off | -| enable_user_metric_persistent | off | -| enable_xlog_prune | off | -| enable_resource_track | off | -| instr_unique_sql_count | 0 | -| enable_beta_opfusIOn | on | -| enable_bete_netsloop_fusIOn | on | -| remote_read_mode | non_authenticatIOn | -| enable_page_lsn_check | off | -| pagewriter_sleep | 2s | -| enable_opfusIOn | on | -| max_redo_log_size | 100 GB | -| pagewrite_thread_num | 1 | -| bgwrite_thread_num | 1 | -| dirty_page_percent_max | 1 | -| candidate_buf_percent_target | 01 | - -## Test Items and Conclusions - -### Test Result Summary - -| Test Item | Data Volume | Concurrent Transactions | Average CPU Usage | IOPS | IO Latency | Write Ahead Logs | tpmC | Test Time (Minute) | -| -------------------------------------- | ----------- | ----------------------- | ----------------- | ----- | ---------- | ---------------- | -------- | ------------------ | -| Single node | 100 GB | 500 | 29.39% | 6.50K | 1.94 ms | 3974 | 520896.3 | 10 | -| One primary node and one standby node | 100 GB | 500 | 30.4% | 5.31K | 453.2 us | 3944 | 519993.5 | 10 | -| One primary node and two standby nodes | 100 GB | 500 | 26.35% | 7.66K | 531.9 us | 3535 | 480842.2 | 10 | - -### Single Node - -- tpmC - - ![mogdb-on-x86-1](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-1.png) - - System data - - ![mogdb-on-x86-2](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-2.png) - -### One Primary Node and One Standby Node - -- tpmC - - ![mogdb-on-x86-3](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-3.png) - -- System data - - ![mogdb-on-x86-4](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-4.png) - -### One Primary Node and Two Standby Nodes - -- tpmC - - ![mogdb-on-x86-5](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-5.png) - -- System data - - ![mogdb-on-x86-6](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-6.png) diff --git a/product/en/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md b/product/en/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md index dc4723a4acbc71e541a21e60531699231307ad60..e661d41a374ff6859800e8ba8b3d0f9a273156b5 100644 --- a/product/en/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md +++ b/product/en/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md @@ -54,7 +54,7 @@ Table1 API Descriptions ``` PKG_SERVICE.JOB_FINISH( - id IN INTEGER, + id IN BIGINT, broken IN BOOLEAN, next_time IN TIMESTAMP DEFAULT sysdate); ``` @@ -63,7 +63,7 @@ Table1 API Descriptions | Parameter | Type | In/Out | Whether it can be empty | Definition | | --------- | --------- | ------ | ----------------------- | ------------------------------------------------------------ | - | id | integer | IN | No | Specified job ID. | + | id | bigint | IN | No | Specified job ID. | | broken | Boolean | IN | No | Status flag bit, **true** means disabled, **false** means enabled. The current job is updated according to the **true** or **false** value; if it is **null**, the status of the original job is not changed. | | next_time | timestamp | IN | Yes | The next run time, the default value is the current system time. If the parameter **broken** state is **true**, the parameter is updated to **4000-1-1**; if the parameter **broken** state is **false**, and if the parameter next_time is not null, the **next_time** value of the specified job is updated, if **next_time** is **null**, the **next_time** value is not updated. This parameter can be omitted and is the default value. | diff --git a/product/en/docs-mogdb/v2.1/toc.md b/product/en/docs-mogdb/v2.1/toc.md index 7b3f83a8bd09b1424451bde47232c30333fe40f0..bb7e3890d50f9a1f38553933679131ebf2ed2f4e 100644 --- a/product/en/docs-mogdb/v2.1/toc.md +++ b/product/en/docs-mogdb/v2.1/toc.md @@ -8,12 +8,6 @@ + [MogDB Introduction](/overview.md) + [Comparison Between MogDB and openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB Release Notes](/about-mogdb/mogdb-release-notes.md) - + High Availability and Performance - + [HA Test](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + Performance Test - + [Performance Test Overview](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [Performance Test for MogDB on Kunpeng Servers](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [Performance Test for MogDB on x86 Servers](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + Open Source Components + [Docker-based MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/en/docs-mogdb/v2.1/toc_about.md b/product/en/docs-mogdb/v2.1/toc_about.md index dc25ebf292e0dc9c05c3b10b50bc4d40fdce8e95..0611f3af9c203398244fe62e7749695bd0ab490f 100644 --- a/product/en/docs-mogdb/v2.1/toc_about.md +++ b/product/en/docs-mogdb/v2.1/toc_about.md @@ -7,12 +7,6 @@ + [MogDB Introduction](/overview.md) + [Comparison Between MogDB and openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB Release Notes](/about-mogdb/mogdb-release-notes.md) -+ High Availability and Performance - + [HA Test](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + Performance Test - + [Performance Test Overview](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [Performance Test for MogDB on Kunpeng Servers](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [Performance Test for MogDB on x86 Servers](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + Open Source Components + [Docker-based MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/1-database-server-exception-testing.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/1-database-server-exception-testing.md deleted file mode 100644 index d799f1af3b37587385b0cef961d1b20f7e5d49e4..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/1-database-server-exception-testing.md +++ /dev/null @@ -1,194 +0,0 @@ ---- -title: MogDB Database Server Exception Testing -summary: MogDB Database Server Exception Testing -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB Database Server Exception Testing - -## Test Objective - -The test aims to test the MogDB availability and stability in the scenarios where only one of the primary, standby, and arbitration MogDB nodes crashes, both primary and standby MogDB nodes crash, both primary and arbitration MogDB nodes crash, or both standby and arbitration MogDB nodes crash. - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :--------------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048 GB | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB 1.1.0 software package | | | -| Test tool | pgbench | | | - -## The Primary Database Node Crashes Abnormally - -Test solution: - -1. Restart the primary database node. -2. Monitor the TPS change. -3. Add a new standby node. -4. Monitor the cluster status and TPS change. - -Test procedure: - -1. Restart the primary database node. - - ```bash - reboot - ``` - -2. Make the original primary database node work as a standby database node in the cluster. - - ```bash - gs_ctl -b full build -D /gaussdata/openGauss/db1 - ``` - -3. Synchronize logs to the new standby database node until the synchronization is complete. - -Test result: - -- After the primary database node is shut down, TPS drops from 9000 to 0, which lasts for 50s or so. After a VIP is added to the new primary database node (node 2), TPS increases from 0 to 13000 or so. -- When the old primary database node is added to the cluster working as a new standby database node and operated for data synchronization, TPS drops from 13000 to 0. As the synchronization is complete, TPS is restored to 9000. - -## The Standby Database Node Crashes Abnormally - -Test solution: - -1. Restart the standby database node. - -2. Monitor the TPS change. - -3. Add the standby database node to the cluster. - -4. Monitor the cluster status and TPS change. - -Test procedure: - -1. Restart the standby database node. - - ``` - reboot - ``` - -2. Add the standby database node to the cluster. - - ``` - gs_ctl start -D /gaussdata/openGauss/db1 -M standby - ``` - -3. Synchronize data to the standby database node until the synchronization is complete. - - ``` - Standby Catchup - ``` - -Test result: - -- After the standby database node is shut down, TPS increases from 9000 or so to 13000 or so. -- After the standby database node is added to the cluster and operated for data synchronization, TPS drops from 13000 to 0. As the synchronization is complete, TPS is restored to 9000. - -## The Arbitration Node Crashes Abnormally - -Test solution: - -1. Add a VIP to the value of the **ping_list** parameter in the **node.conf** file. -2. Simulate the scenario where the arbitration node crashes. - -Test procedure: - -1. Add a VIP to the server IP addresses. - - ``` - ifconfig ens160:1 192.168.122.111 netmask 255.255.255.0 - ``` - -2. Add a VIP to the value of the **ping_list** parameter in the **node.conf** file. Synchronize the configuration file between the primary and standby database nodes and restart the monitor script. - -3. When the monitor script is running normally, disconnect the VIP of the simulated arbitration node and then observe the change. - - ``` - ifconfig ens160:1 down - ``` - -4. Manually connect the VIP of the simulated arbitration node and observe the change. - -Test result: - -- The monitor script of the primary and standby database nodes reports that the VIP of the arbitration node fails the ping operation. no other change occurs. -- TPS does not change. - -## Both Primary and Standby Database Nodes Crash Abnormally - -Test solution: - -1. Restart both the primary and standby database nodes. -2. Start the **node_run** scripts of the primary and standby database nodes. -3. Manually enable the primary and standby databases. - -Test procedure: - -1. Restart both the primary and standby database nodes. - - ``` - reboot - reboot - ``` - -2. Start the **node_run** scripts of the primary and standby database nodes. - -3. Manually enable the primary and standby databases. - -Test result: - -When the database is disabled, TPS drops to 0. After 15s the cluster is started, TPS increases to 9000. - -## Both Primary and Arbitration Nodes Crash Abnormally - -Test solution: - -Shut down the primary database node and disconnect the IP address of the arbitration node. - -Test procedure: - -1. Shut down the primary database node and disconnect the IP address of the arbitration node. - -2. Check the status of the standby database node. - -3. Start the primary database node and its monitor script. - - ``` - gs_ctl start -D /gaussdata/openGauss/db1 -M primary - ``` - -4. Check the cluster status. - -Test result: - -- After the primary database node and arbitration node are shut down, TPS drops from 9000 to 0. - -- When the primary database node restores, TPS increases to 9000. - -## Both Standby and Arbitration Nodes Crash Abnormally - -Test solution: - -Shut down the standby database node and disconnect the IP address of the arbitration node. - -Test procedure: - -1. Shut down the standby database node and disconnect the IP address of the arbitration node. - -2. Check the status of the primary database node. - - No change occurs. - -3. After the standby database node restores, start the monitor script of the standby database node. - - The error reported by the monitor script of the primary database node disappears. - -4. Start the standby database node. - -5. Check the cluster status. diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/2-network-exception-testing.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/2-network-exception-testing.md deleted file mode 100644 index 50b695b52400152bfa92fb43f1689d8352ba72f1..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/2-network-exception-testing.md +++ /dev/null @@ -1,177 +0,0 @@ ---- -title: MogDB Network Exception Testing -summary: MogDB Network Exception Testing -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB Network Exception Testing (Need to Be Reviewed Later) - -## Test Scope - -- The service NIC of the primary node is abnormal. - -- The heartbeat NIC of the primary node is abnormal. - -- Both the service and heartbeat NICs of the primary node are abnormal. - -- The service NIC of the standby node is abnormal. - -- The heartbeat NIC of the standby node is abnormal. - -- The VIP of the primary node is disconnected. - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :-------------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048 GB | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB1.1.0 software package | | | -| Test tool | pgbench | | | - -## The Service NIC of the Primary Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. -2. Observe the database status. - -Test procedure: - -1. Record the original cluster status. - -2. Disable the heartbeat NIC and then enable it. - -3. Observe the status of the standby database node. - - The standby database node cannot be connected to the primary database node. - -4. Query the cluster status. - - No primary/standby switchover occurs. - -Test result: - -1. No primary/standby switchover occurs. -2. When the network of the primary database node restores, the status of the standby database node becomes normal. -3. When the service NIC of the primary database node is abnormal, TPS decreases from 9000 to 0. After 15s when the service NIC is enabled, TPS increases from 0 to 9000. - -## The Heartbeat NIC of the Primary Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. -2. Observe the database status. - -Test procedure: - -1. Observe the database status. - -2. Disable the heartbeat NIC and then enable it. - -3. Observe the database status. - - No primary/standby switchover occurs. - -4. Observe the cluster status. - -Test result: - -1. No primary/standby switchover occurs. - -2. TPS does not change. - -## Both the Service and Heartbeat NICs of the Primary Database Node Are Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. - -2. Observe the database status. - -Test procedure: - -1. Observe the original database status. - -2. Disable the heartbeat NIC and then enable it. - - The primary database node is normal. - -Test result: - -1. No primary/standby switchover occurs. - -2. When the service NIC is disconnected, TPS drops from 9000 to 0. After the service NIC is restored to normal, TPS increases to 9000. - -## The Service NIC of the Standby Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC and then enable it. - -2. Observe the database status. - -Test procedure: - -1. Record the original cluster status. - -2. Disable the heartbeat NIC and then enable it. - -3. Observe the status of the primary database node. - -4. Ping the IP address of the service NIC of the standby database node. - - The IP address cannot be pinged. - - No primary/standby switchover occurs. - -Test result: - -1. The monitor script of the standby database node reports the heartbeat error indicating that no primary/standby switchover occurs. - -2. When the service NIC of the standby database node is abnormal, TPS of the primary database node increases to 13000. After the service NIC of the standby database node is restored to normal, TPS of the primary database node is restored to 9000. - -## The Heartbeat NIC of the Standby Database Node Is Abnormal - -Test solution: - -1. Disable the heartbeat NIC 60s and then enable it. - -2. Observe the database status. - -Test procedure: - -1. Observe the original database status. - -2. Disable the heartbeat NIC 60s and then enable it. - -3. After the script is executed, observe the database status. - No primary/standby switchover occurs. The monitor script of the primary database node reports an error that the IP address of the service NIC of the standby database node cannot be pinged. - -Test result: - -No primary/standby switchover occurs. - -## The VIP of the Primary Database Node Is Disconnected - -Test solution: - -Disable the NIC **bond0:1** and then enable it. - -Test procedure: - -1. Disable the NIC **bond0:1** and then enable it. - -2. Observe the database status. - -3. Observe the VIP of the primary database node. - -Test result: - -1. The VIP of the primary database node is automatically connected. - -2. When the VIP is disconnected and then connected, the TPS status is restored to normal. diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/3-routine-maintenance-testing.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/3-routine-maintenance-testing.md deleted file mode 100644 index d50a8f807f6607bd89a1c2b875c121fb9c76f3fc..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/3-routine-maintenance-testing.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -title: Routine Maintenance Testing -summary: Routine Maintenance Testing -author: Guo Huan -date: 2021-04-25 ---- - -# Routine Maintenance Testing - -## Test Scope - -1. HA tool used for the switchover test -2. gs_ctl used for the switchover test -3. HA tool used for the failover test -4. gs_ctl used for the failover test -5. gs_ctl and HA contradiction test -6. Split-brain test for avoiding deploying two primary database nodes - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :-------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB1.1.0 | | | -| Tool | pgbench | | | - -## HA Tool Used for the Switchover Test - -Test procedure: - -1. Query the status of databases. - -2. Run the switch command on the standby database, and then query the database status. - - ```bash - /home/omm/venv/bin/python3 /home/omm/openGaussHA_standlone/switch.py -config /home/omm/openGaussHA_standlone/node.conf --switchover - ``` - -3. Query the VIP of the new primary database node. - -Test result: - -The primary/standby switchover is normal. - -## gs_ctl Used for the Switchover Test - -Test procedure: - -1. Query the database status. -2. Run the command to perform the switchover operation and query the status when finished. -3. Query the VIP. - -Test result: - -The primary/standby switchover is normal. - -## HA Tool Used for the Failover Test - -Test procedure: - -1. Query the current status of the database. - -2. Run the HA failover command. - -3. Observe the cluster status. - -4. The original standby database node works as a new primary database node. - -Test result: - -1. The original standby database node becomes the new primary database node with a VIP. - -2. The original primary database node is killed. - -3. TPS increases from 6000 to 13000 in about 10s after failover. - -## gs_ctl Used for the Failover Test - -Test procedure: - -1. Query the current status of the database. -2. Run the gs_ctl failover command. -3. Query the status of the new primary database node. -4. Query the status of the original primary database node. -5. Observe script changes. - -Test result: - -1. The new primary database node waits for 10s and then automatically shuts down, the original primary database node is not affected, and the VIP is still attached to the original primary database node. -2. About 5s or 6s after failover of the standby database node, the TPS increases from 9000 to 13000. No other changes occur except fluctuations. - -## gs_ctl and HA Contradiction Test - -Test procedure: - -1. Record the cluster status. - -2. Stop the primary and standby database nodes at the same time. - -3. Use gs_ctl to start the primary and standby database nodes. Designate the original standby database node as the primary database node, vice versa. - -4. Observe the script and cluster status. - -Test result: - -1. HA configuration is adjusted according to the actual situation. -2. TPS drops from 9000 to 0 after the primary and standby database nodes are shut down. At this time, HA detects the cluster configuration change and attaches the VIP to the new primary database node within 10s. After about 5s or 6s, TPS rises from 0 to 13000. - -## Split-Brain Test for Avoiding Deploying Two Primary Database Nodes - -Test procedure: - -1. Query the current status of the cluster. - -2. Use the gs_ctl command to restart the standby database node to make it work as the primary database node. - -Test result: - -1. After timeout for 10s, the primary database node that was restarted by an abnormal operation is killed by the monitoring script. - -2. Repair the standby database node and add it to the cluster. - - ```bash - gs_ctl build -b full -D /gaussdata/openGauss/db1 - ``` - -3. TPS increases from 6000 to 13000 after the standby database is restarted to work as the primary database node. - -4. TPS drops from 13000 to 9000 (same as that in the primary/standby deployment scenario) after the standby database node is added to the cluster. diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/4-service-exception-testing.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/4-service-exception-testing.md deleted file mode 100644 index 119b5875829cbd534eb533e50542ac8fe74fecaa..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/4-service-exception-testing.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: MogDB Service Exception Testing -summary: MogDB Service Exception Testing -author: Guo Huan -date: 2021-04-25 ---- - -# MogDB Service Exception Testing - -## Test Scope - -1. Database process exception for the standby node - -2. Monitor script exception for the standby node - -3. File system exception for the primary node (affecting HA process) - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :-------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| Hard Disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB1.1.0 | | | -| Tool | pgbench | | | - -## Database process exception for the standby node - -Test procedure: - -Kill the database process. - -Test result: - -1. Observe the cluster status. - -2. The script of the standby database node shows the heartbeat exception. - -3. No switchover occurs. - -4. Run the command on the standby database node. - - ```bash - gs_ctl start -D /gaussdata/openGauss/db1 -M standby - ``` - -5. The cluster is restored to normal and no switchover has occurred since then. - -6. When the standby database process is killed, TPS of the primary database node rises from 9000 to 13000. - -7. No primary/standby switchover occurs. - -## Monitor script exception for the standby node - -Test procedure: - -Kill the monitor script of the primary database node. - -Test result: - -1. Observe the cluster status. -2. No switchover occurs. -3. The monitor script of the primary database node reports script exception of the standby database node. -4. Restore the monitor script of the standby database node. -5. TPS maintains at 9000. - -## File system exception for the primary node (affecting HA process) - -Test procedure: - -Modify the permission of the primary database script called by HA, such as gs_ctl. - -Test result: - -1. After modifying the **rwx** permission of gs_ctl, the monitoring script of the primary database node reports heartbeat exception. - -2. The instance status cannot be detected. - -3. Query the current cluster status. - -4. After waiting about two minutes, no switchover occurs. diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/MogDB-ha-test-report.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/MogDB-ha-test-report.md deleted file mode 100644 index 1ad5c7de3fff55835ff2319899739b5bd56d5160..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/ha/MogDB-ha-test-report.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: HA Test -summary: HA Test -author: Zhang Cuiping -date: 2021-06-07 ---- - -# HA Test - -## Test Objective - -Database HA (high availability) test aims at providing users with high-quality services, avoiding service interruption caused by such faults as server crash. Its HA lies in not only whether a database can provide services consistently but whether a database can ensure data consistency. - -## Test Environment - -| Category | Server Configuration | Client Configuration | Quantity | -| ----------- | :---------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4, 2933MT/s | DDR4, 2933MT/s | 2048 GB | -| Hard disk | NVME 3.5 TB | NVME 3 TB | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB software package | | | -| Test tool | pgbench | | | - -## HA and Scalability Test - -| No. | Test Item | Description | -| ---- | ------------------------------------- | ------------------------------------------------------------ | -| 1 | Cluster with read-write separation | Supports separate routing of read and write requests and separate distribution of read and write tasks. | -| 2 | Capacity expansion/capacity reduction | In load scenarios, adding or reducing a physical device does not make a front-end application service interrupted. | -| 3 | Shared storage cluster | Supports two nodes sharing a storage cluster, automatic failover, and load balancing of concurrent transactions. | -| 4 | Service exception test | Guarantees the availability of an application in scenarios where the process of the standby database, the monitoring script of the standby database, and the file system of the primary database are abnormal. | -| 5 | Routine maintenance test | Supports the switchover feature provided by gs_ctl (database service control tool) and the failover feature provided by MogHA (HA component). | -| 6 | Database server exception test | Maximizes the availability and stability of an application when the primary node, standby node, or arbitration node crashes, or both the primary and standby nodes, or both the primary and arbitration nodes crash. | -| 7 | Network exception test | Maximizes the availability of an application when the service or heartbeat NIC of the primary node is abnormal, both the service and heartbeat NICs of the primary node are abnormal, the service NIC of the standby node is abnormal, or the VIP of the host is abnormal. | diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/1-performance-test-overview.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/1-performance-test-overview.md deleted file mode 100644 index d6a786378b0381b8075acc952699df1a7924370e..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/1-performance-test-overview.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: Performance Test Overview -summary: Performance Test Overview -author: Liu Xu -date: 2021-04-27 ---- - -# Performance Test Overview - -## Test Indicators - -### Data Volume - -Each repository contains 70 MB of data. To simulate the actual data environment, 1000 repositories are created during initiation. After related indexes are added, the total size of data reaches 100 GB or so. - -### Number of Concurrent Requests - -This is to simulate the real user behavior and make the number of online user requests from the clients reach a certain number. - -### Average System Workload - -System workload indicates how busy the system CPU is, for example, how many processes are there waiting for being scheduled by the CPU, which reflects the pressure of the whole system. However, average system workload indicates the average number of processes in run queues in a specified time period, generally 1, 5, or 10 minutes. - -### CPU Usage - -CPU usage indicates the quantity of CPU resources occupied by the running programs, that is, the quantity of running programs on a server at a time point. High CPU usage indicates that a server runs a plenty of programs and vice versa. Whether the CPU usage is high or low is directly related to the CPU performance. CPU usage can determine whether a CPU reaches its bottleneck. - -### IOPS - -IOPS (Input/Output Operations Per Second) is a method for measuring the performance of a computer storage device, such as HDDs, SSDs, or SANs. It refers to the number of write/read times per second, which is one of the major indicators for measuring the disk performance. IOPS indicates the number of I/O requests that can be processed by the system in unit time. The I/O requests usually refer to data write/read requests. - -### IO Latency - -IO latency is also called IO response time, referring the time period from the time when the OS kernel sends a write/read command to the time when the OS kernel receives the IO response. Time for operating single IO indicates only the time of processing single IO in a disk. However, IO latency includes even the time of waiting for being processed for an IO in a queue. - -### tpmC - -TPC-C uses three performance and price metrics. TPC-C is used to measure performance with the unit of tpmC (Transactions Per Minute). C refers to the C benchmark program of TPC. tpmC indicates the number of new orders that are processed by the system per minute. - -## Test Program Preparation - -### Test Tools - -BenchmarkSQL is a typical open source database test tool with a built-in TPC-C test script which can be directly used for testing PostgreSQL, MySQL, Oracle, and SQL Server databases. It is used for testing TPC-C of OLTP (Online Transaction Processing) via JDBC (Java Database Connectivity). - -### Test Specifications - -TPC-C offers OLTP system test specifications and tests an OLTP system using a merchandising model. Transactions are divided into five categories, the content and characteristics of which are described as follows: - -- **NewOrder - Generation of new orders** - - Transaction content: For any one client, select 5 to 15 commodities from a specified repository and create new orders with 1% of the orders that fails the operation and requires rollback. - - Main characteristics: middleweight, frequent write/read operations, and high response speed - -- **Payment - Payment of order** - - Transaction content: For any one client, select a region and customers there from a specified repository randomly, pay an order with a random amount of money, and record this operation. - - Main characteristics: lightweight, frequent write/read operations, and high response speed。 - -- **OrderStatus - Query of latest orders** - - Transaction content: For any one client, select a region and customers there from a specified repository randomly, read the last order, and display the status of each commodity in the order. - - Main characteristics: middleweight, low read-only frequency, and high response speed - -- **Delivery - Delivery of packages** - - Transaction content: For any one client, select a delivery package randomly, update the account balance of the user whose order is being processed, and delete the order from the new order list. - - Main characteristics: 1 to 10 concurrent orders in a batch, low write/read frequency, and loose response time - -- **StockLevel - Analysis of stock status** - - Transaction content: For any one client, select the last 20 orders from a specified repository of a region, check the stock of all commodities of the order list, and calculate and display the stock of all commodities with its stock level lower than the threshold generated randomly. - - Main characteristics: heavyweight, low read-only frequency, and loose response time - -### Test Rules - -Before the test, TPC-C Benchmark specifies the initial status of a database, that is, the rule of data being generated in the database. The ITEM table includes a fixed number (100,000) of commodities. The number of warehouses can be adjusted. The initialized WAREHOUSE table includes 1000 records in this test. - -- The STOCK table contains 1000 x 100,000 records. (Each warehouse contains 100,000 commodities.) -- The DISTRICT table contains 1000 x 10 records. (Each warehouse provides services for 10 regions.) -- The CUSTOMER table contains 1000 x 10 x 3000 records. (There are 3000 customers in each region.) -- The HISTORY table contains 1000 x 10 x 3000 records. (Each customer has one transaction history.) -- The ORDER table contains 1000 x 10 x 3000 records (there are 3000 orders in each region). The last 900 orders generated are added to the NEW-ORDER table. Each order generates 5 to 15 ORDER-LINE records. - -> TPC-C uses tpmC to measure the maximum qualified throughput. The number of transactions depends on that of the new order transactions, that is, the number of new orders processed per minute. diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md deleted file mode 100644 index 2cbe9addd4623fa01565bade74c9e49170186f97..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md +++ /dev/null @@ -1,401 +0,0 @@ ---- -title: Performance Test for MogDB on a Kunpeng Server -summary: Performance Test for MogDB on a Kunpeng Server -author: Liu Xu -date: 2021-03-04 ---- - -# Performance Test for MogDB on Kunpeng Servers - -## Test Objective - -This document describes the test of MogDB 2.0.0 on Kunpeng servers in scenarios where MogDB is deployed on a single node, one primary node and one standby node, or one primary node and two standby nodes (one synchronous standby and one asynchronous standby). - -## Test Environment - -### Environment Configuration - -| Category | Server Configuration | Client Configuration | Quantity | -| :---------: | :--------------------------: | :------------------: | :------: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| Memory | DDR4,2933MT/s | DDR4,2933MT/s | 2048 GB | -| Hard disk | Nvme 3.5T | Nvme 3T | 4 | -| File system | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| Database | MogDB 1.1.0 software package | | | - -### Test Tool - -| Name | Function | -| :--------------: | :----------------------------------------------------------- | -| BenchmarkSQL 5.0 | Open-source BenchmarkSQL developed based on Java is used for TPC-C test of OLTP database. It is used to evaluate the database transaction processing capability. | - -## Test Procedure - -### MogDB Database Operation - -1. Obtain the database installation package. - -2. Install the database. - -3. Create TPCC test user and database. - - ```sql - create user [username] identified by ‘passwd’; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. Disable the database and modify the **postgresql.conf** database configuration file by adding configuration parameters at the end of the file. - - For example, add the following parameters in the single node test: - - ```bash - max_connections = 4096 - - allow_concurrent_tuple_update = true - - audit_enabled = off - - checkpoint_segments = 1024 - - cstore_buffers =16MB - - enable_alarm = off - - enable_codegen = false - - enable_data_replicate = off - - full_page_writes = off - - max_files_per_process = 100000 - - max_prepared_transactions = 2048 - - shared_buffers = 350GB - - use_workload_manager = off - - wal_buffers = 1GB - - work_mem = 1MB - - log_min_messages = FATAL - - transaction_isolation = 'read committed' - - default_transaction_isolation = 'read committed' - - synchronous_commit = on - - fsync = on - - maintenance_work_mem = 2GB - - vacuum_cost_limit = 2000 - - autovacuum = on - - autovacuum_mode = vacuum - - autovacuum_max_workers = 5 - - autovacuum_naptime = 20s - - autovacuum_vacuum_cost_delay =10 - - xloginsert_locks = 48 - - update_lockwait_timeout =20min - - enable_mergejoin = off - - enable_nestloop = off - - enable_hashjoin = off - - enable_bitmapscan = on - - enable_material = off - - wal_log_hints = off - - log_duration = off - - checkpoint_timeout = 15min - - enable_save_datachanged_timestamp =FALSE - - enable_thread_pool = on - - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - - enable_double_write = on - - enable_incremental_checkpoint = on - - enable_opfusion = on - - advance_xlog_file_num = 10 - - numa_distribute_mode = 'all' - - track_activities = off - - enable_instr_track_wait = off - - enable_instr_rt_percentile = off - - track_counts =on - - track_sql_count = off - - enable_instr_cpu_timer = off - - plog_merge_age = 0 - - session_timeout = 0 - - enable_instance_metric_persistent = off - - enable_logical_io_statistics = off - - enable_user_metric_persistent =off - - enable_xlog_prune = off - - enable_resource_track = off - - instr_unique_sql_count = 0 - - enable_beta_opfusion = on - - enable_beta_nestloop_fusion = on - - autovacuum_vacuum_scale_factor = 0.02 - - autovacuum_analyze_scale_factor = 0.1 - - client_encoding = UTF8 - - lc_messages = en_US.UTF-8 - - lc_monetary = en_US.UTF-8 - - lc_numeric = en_US.UTF-8 - - lc_time = en_US.UTF-8 - - modify_initial_password = off - - ssl = off - - enable_memory_limit = off - - data_replicate_buffer_size = 16384 - - max_wal_senders = 8 - - log_line_prefix = '%m %u %d %h %p %S' - - vacuum_cost_limit = 10000 - - max_process_memory = 12582912 - - recovery_max_workers = 1 - - recovery_parallelism = 1 - - explain_perf_mode = normal - - remote_read_mode = non_authentication - - enable_page_lsn_check = off - - pagewriter_sleep = 100 - ``` - -### BenchmarkSQL Operation - -1. Modify the configuration file. - - Open the BenchmarkSQL installation directory and find the **[config file]** configuration file in the **run** directory. - - ``` - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. Run **runDataBuild.sh** to generate data. - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. Run **runBenchmark.sh** to test the database. - - ``` - ./runBenchmark.sh [config file] - ``` - -### OS Configuration - -1. Modify **PAGESIZE** of the OS kernel (required only in EulerOS). - - **Install kernel-4.19.36-1.aarch64.rpm (*).** - - ``` - # rpm -Uvh --force --nodeps kernel-4.19.36-1.aarch64.rpm - - *: This file is based on the kernel package of linux 4.19.36. You can acquire it from the following directory: - - 10.44.133.121 (root/Huawei12#$) - - /data14/xy_packages/kernel-4.19.36-1.aarch64.rpm - ``` - - **Modify the boot options of the root in the OS kernel configuration file.** - - ``` - \# vim /boot/efi/EFI/euleros/grubenv //Back up the grubenv file before modification. - - \# GRUB Environment Block - - saved_entry=EulerOS (4.19.36) 2.0 (SP8) -- Changed to 4.19.36 - ``` - -### File System - -1. Change the value of **blocksize** to **8K** in the XFS file system. - - #Run the following commands to check the attached NVME disks: - - ``` - # df -h | grep nvme - - /dev/nvme0n1 3.7T 2.6T 1.2T 69% /data1 - - /dev/nvme1n1 3.7T 1.9T 1.8T 51% /data2 - - /dev/nvme2n1 3.7T 2.2T 1.6T 59% /data3 - - /dev/nvme3n1 3.7T 1.4T 2.3T 39% /data4 - - Run the xfs_info command to query information about NVME disks. - - \# xfs_info /data1 - ``` - -2. Back up the required data. - -3. Format the disk. - - ``` - Use the /dev/nvme0n1 disk and /data1 load point as an example and run the folowing commands: - - umount /data1 - - mkfs.xfs -b size=8192 /dev/nvme0n1 -f - - mount /dev/nvme0n1 /data1 - ``` - -## Test Items and Conclusions - -### Test Result Summary - -| Test Item | Data Volume | Concurrent Transactions | Average CPU Usage | IOPS | IO Latency | Write Ahead Logs | tpmC | Test Time (Minute) | -| :------------------------------------: | ----------- | ----------------------- | ----------------- | ------ | ---------- | ---------------- | ---------- | :----------------: | -| Single node | 100 GB | 500 | 77.49% | 17.96K | 819.05 us | 13260 | 1567226.12 | 10 | -| One primary node and one standby node | 100 GB | 500 | 57.64% | 5.31K | 842.78 us | 13272 | 1130307.87 | 10 | -| One primary node and two standby nodes | 100 GB | 500 | 60.77% | 5.3K | 821.66 us | 14324 | 1201560.28 | 10 | - -### Single Node - -- tpmC - - ![mogdb-on-kunpeng-1](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-1.png) - -- System data - - ![mogdb-on-kunpeng-2](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-2.png) - -### One Primary Node and One Standby Node - -- tpmC - - ![mogdb-on-kunpeng-3](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-3.png) - -- System data - - ![mogdb-on-kunpeng-4](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-4.png) - -### One Primary Node and Two Standby Nodes - -- tpmC - - ![mogdb-on-kunpeng-5](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-5.png) - -- System data - - ![mogdb-on-kunpeng-6](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-6.png) diff --git a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md b/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md deleted file mode 100644 index 94644b1bd75005bd4887cf27cdc2002e79821993..0000000000000000000000000000000000000000 --- a/product/en/docs-mogdb/v3.0/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md +++ /dev/null @@ -1,654 +0,0 @@ ---- -title: Performance Test for MogDB on x86 Servers -summary: Performance Test for MogDB on x86 Servers -author: Liu Xu -date: 2021-03-04 ---- - -# Performance Test for MogDB on x86 Servers - -## Test Objective - -This document describes the test of MogDB 2.0.0 on x86 servers in scenarios where MogDB is deployed on a single node, one primary node and one standby node, or one primary node and two standby nodes (one synchronous standby and one asynchronous standby). - -## Test Environment - -### Environment Configuration - -| Server Type | Fit Server | NFS5280M5 | -| :---------: | :--------------------------------------: | :--------------------------------------: | -| CPU | 144cIntel® Xeon(R) Gold 6240 CPU@2.60GHz | 64cIntel® Xeon(R) Gold 5218 CPU @2.30GHz | -| Memory | 768G | 128G | -| Hard disk | SAS SSD | SAS SSD | -| NIC | 10GE | 10GE | - -### Test Tool - -| Name | Function | -| :-------------- | :----------------------------------------------------------- | -| Benchmarksql5.0 | Open-source BenchmarkSQL developed based on Java is used for TPC-C test of OLTP database. It is used to evaluate the database transaction processing capability. | - -## Test Procedure - -### MogDB Database Operation - -1. Obtain the database installation package. - -2. Install the database. - -3. Create TPCC test user and database. - - ``` - create user [username] identified by ‘passwd’; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. Disable the database and modify the **postgresql.conf** database configuration file by adding configuration parameters at the end of the file. - - For example, add the following parameters in the single node test: - - ``` - max_connections = 4096 - - allow_concurrent_tuple_update = true - - audit_enabled = off - - checkpoint_segments = 1024 - - cstore_buffers =16MB - - enable_alarm = off - - enable_codegen = false - - enable_data_replicate = off - - full_page_writes = off - - max_files_per_process = 100000 - - max_prepared_transactions = 2048 - - shared_buffers = 350GB - - use_workload_manager = off - - wal_buffers = 1GB - - work_mem = 1MB - - log_min_messages = FATAL - - transaction_isolation = 'read committed' - - default_transaction_isolation = 'read committed' - - synchronous_commit = on - - fsync = on - - maintenance_work_mem = 2GB - - vacuum_cost_limit = 2000 - - autovacuum = on - - autovacuum_mode = vacuum - - autovacuum_max_workers = 5 - - autovacuum_naptime = 20s - - autovacuum_vacuum_cost_delay =10 - - xloginsert_locks = 48 - - update_lockwait_timeout =20min - - enable_mergejoin = off - - enable_nestloop = off - - enable_hashjoin = off - - enable_bitmapscan = on - - enable_material = off - - wal_log_hints = off - - log_duration = off - - checkpoint_timeout = 15min - - enable_save_datachanged_timestamp =FALSE - - enable_thread_pool = on - - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - - enable_double_write = on - - enable_incremental_checkpoint = on - - enable_opfusion = on - - advance_xlog_file_num = 10 - - numa_distribute_mode = 'all' - - track_activities = off - - enable_instr_track_wait = off - - enable_instr_rt_percentile = off - - track_counts =on - - track_sql_count = off - - enable_instr_cpu_timer = off - - plog_merge_age = 0 - - session_timeout = 0 - - enable_instance_metric_persistent = off - - enable_logical_io_statistics = off - - enable_user_metric_persistent =off - - enable_xlog_prune = off - - enable_resource_track = off - - instr_unique_sql_count = 0 - - enable_beta_opfusion = on - - enable_beta_nestloop_fusion = on - - autovacuum_vacuum_scale_factor = 0.02 - - autovacuum_analyze_scale_factor = 0.1 - - client_encoding = UTF8 - - lc_messages = en_US.UTF-8 - - lc_monetary = en_US.UTF-8 - - lc_numeric = en_US.UTF-8 - - lc_time = en_US.UTF-8 - - modify_initial_password = off - - ssl = off - - enable_memory_limit = off - - data_replicate_buffer_size = 16384 - - max_wal_senders = 8 - - log_line_prefix = '%m %u %d %h %p %S' - - vacuum_cost_limit = 10000 - - max_process_memory = 12582912 - - recovery_max_workers = 1 - - recovery_parallelism = 1 - - explain_perf_mode = normal - - remote_read_mode = non_authentication - - enable_page_lsn_check = off - - pagewriter_sleep = 100 - ``` - -### BenchmarkSQL Operation - -1. Modify the configuration file. - - Open the BenchmarkSQL installation directory and find the **[config file]** configuration file in the **run** directory. - - ``` - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. Run **runDataBuild.sh** to generate data. - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. Run **runBenchmark.sh** to test the database. - - ``` - ./runBenchmark.sh [config file] - ``` - -### OS Parameters - -``` -vm.dirty_background_ratIO=5 - -vm.dirty_ratIO=10 - -kernel.sysrq=0 - -net.ipv4.ip_forward=0 - -net.ipv4.conf.all.send_redirects=0 - -net.ipv4.conf.default.send_redirects=0 - -net.ipv4.conf.all.accept_source_route=0 - -net.ipv4.conf.default.accept_source_route=0 - -net.ipv4.conf.all.accept_redirects=0 - -net.ipv4.conf.default.accept_redirects=0 - -net.ipv4.conf.all.secure_redirects=0 - -net.ipv4.conf.default.secure_redirects=0 - -net.ipv4.icmp_echo_ignore_broadcasts=1 - -net.ipv4.icmp_ignore_bogus_error_responses=1 - -net.ipv4.conf.all.rp_filter=1 - -net.ipv4.conf.default.rp_filter=1 - -net.ipv4.tcp_syncookies=1 - -kernel.dmesg_restrict=1 - -net.ipv6.conf.all.accept_redirects=0 - -net.ipv6.conf.default.accept_redirects=0 - -net.core.rmem_max = 21299200 - -net.core.rmem_default = 21299200 - -net.core.somaxconn = 65535 - -net.ipv4.tcp_tw_reuse = 1 - -net.sctp.sctp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -kernel.sem = 250 6400000 1000 25600 - -net.core.wmem_default = 21299200 - -kernel.shmall = 1152921504606846720 - -net.core.wmem_max = 21299200 - -net.sctp.sctp_rmem = 8192 250000 16777216 - -net.core.netdev_max_backlog = 65535 - -kernel.shmmax = 18446744073709551615 - -net.sctp.sctp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_max_syn_backlog = 65535 - -vm.oom_panic_on_oom=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -net.ipv4.tcp_timestamps = 1 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.sctp.path_max_retrans = 10 - -net.sctp.max_init_retransmits = 10 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_synack_retries = 5 - -kernel.core_uses_pid=1 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%p - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%h - -net.core.netdev_max_backlog = 65535 - -net.core.rmem_default = 21299200 - -net.core.rmem_max = 21299200 - -net.core.somaxconn = 65535 - -net.core.wmem_default = 21299200 - -net.core.wmem_max = 21299200 - -net.ipv4.conf.all.accept_redirects = 0 - -net.ipv4.conf.all.rp_filter = 1 - -net.ipv4.conf.all.secure_redirects = 0 - -net.ipv4.conf.all.send_redirects = 0 - -net.ipv4.conf.default.accept_redirects = 0 - -net.ipv4.conf.default.accept_source_route = 0 - -net.ipv4.conf.default.rp_filter = 1 - -net.ipv4.conf.default.secure_redirects = 0 - -net.ipv4.conf.default.send_redirects = 0 - -net.ipv4.conf.enp135s0.accept_redirects = 0 - -net.ipv4.conf.enp135s0.accept_source_route = 0 - -net.ipv4.conf.enp135s0.forwarding = 1 - -net.ipv4.conf.enp135s0.rp_filter = 1 - -net.ipv4.conf.enp135s0.secure_redirects = 0 - -net.ipv4.conf.enp135s0.send_redirects = 0 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_max_syn_backlog = 65535 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_mem = 362715 483620 725430 - -\#net.ipv4.tcp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.ipv4.tcp_tw_reuse = 1 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.udp_mem = 725430 967240 1450860 - -\#net.ipv4.tcp_max_orphans = 3276800 - -\#net.ipv4.tcp_fin_timeout = 60 - -\#net.ipv4.ip_local_port_range = 26000 65535 - -net.ipv4.tcp_retries2 = 80 - -\#net.ipv4.ip_local_reserved_ports = 20050-30007 - -vm.min_free_kbytes = 40140150 -``` - -### Database Parameters - -| Parameter | MogDB | -| :-------------------------------- | :--------------------------------------------- | -| listen_addresses | Specific IP address | -| port | 26000 | -| max_connectIOns | 4096 | -| wal_level | hot_standby | -| archive_mode | on | -| archive_command | /bin/ture | -| max_wal_senders | 16 | -| wal_keep_segments | 16 | -| max_replicatIOn_slots | 8 | -| hot_standby | on | -| logging_collector | on | -| log_directory | Specify the directory of the installation tool | -| log_filename | PostgreSQL-%Y-%m-%d_%H%M%S.log | -| log_min_duratIOn_statement | 1800000 | -| log_line_prefix | %m%c%d%p%a%x%n%e | -| log_timezone | PRC | -| datestyle | iso,mdy | -| timezone | PRC | -| default_text_search_config | pg_catalog.english | -| applicatIOn_name | dn_6001 | -| max_prepared_transactIOns | 2048 | -| shared_buffers | 350 GB | -| wal_buffers | 1 GB | -| work_mem | 64 MB | -| log_min_messages | FATAL | -| synchronous_commit | on | -| fsync | on | -| maintenance_work_mem | 2 GB | -| autovacuum | on | -| autovacuum_max_workers | 5 | -| autovacuum_naptime | 20s | -| autovacuum_vacuum_cost_delay | 10 | -| enable_mergejoin | off | -| enable_nestloop | off | -| enable_hashjoin | off | -| enable_bitmapscan | on | -| enable_material | off | -| wal_log_hints | off | -| log_duratIOn | off | -| checkpoint_timeout | 15 min | -| track_activities | off | -| track_counts | on | -| autovacuum_vacuum_scale_factor | 0.02 | -| autovacuum_analyze_scale_factor | 0.1 | -| ssl | off | -| local_bind_address | Specific IP address | -| max_inner_tool_connectIOns | 10 | -| password_encryptIOn_type | 0 | -| comm_tcp_mode | on | -| comm_quota_size | 1024 KB | -| max_process_memory | 700 GB | -| bulk_write_ring_size | 2 GB | -| checkpoint_segments | 1024 | -| incremental_checkpoint_timeout | 60s | -| archive_dest | /log/archive | -| enable_slot_log | off | -| data_replicate_buffer_size | 128 MB | -| walsender_max_send_size | 8 MB | -| enable_kill_query | off | -| connectIOn_alARM_rate | 0.9 | -| alARM_report_interval | 10 | -| alARM_component | /opt/huawei/snas/bin/snas_cm_cmd | -| lockwait_timeout | 1200s | -| pgxc_node_name | xxx | -| audit_directory | Specify the directory of the installation tool | -| explain_perf_mode | pretty | -| job_queue_processes | 10 | -| default_storage_nodegroup | installatIOn | -| expected_computing_nodegroup | query | -| replicatIOn_type | 1 | -| recovery_max_workers | 4 | -| available_zone | AZ1 | -| allow_concurrent_tuple_update | TRUE | -| audit_enabled | off | -| cstore_buffers | 16 MB | -| enable_alARM | off | -| enable_codegen | FALSE | -| enable_data_replicate | off | -| max_file_per_process | 10000 | -| use_workload_manager | off | -| xloginsert_locks | 48 | -| update_lockwait_timeout | 20 min | -| enable_save_datachanged_timestamp | FALSE | -| enable_thread_pool | off | -| enable_double_write | on | -| enable_incremental_checkpoint | on | -| advance_xlog_file_num | 10 | -| enable_instr_track_wait | off | -| enable_instr_rt_percentile | off | -| track_sql_count | off | -| enable_instr_cpu_timer | off | -| plog_merge_age | 0 | -| sessIOn_timeout | 0 | -| enable_instance_metric_persistent | off | -| enable_logical_IO_statistics | off | -| enable_user_metric_persistent | off | -| enable_xlog_prune | off | -| enable_resource_track | off | -| instr_unique_sql_count | 0 | -| enable_beta_opfusIOn | on | -| enable_bete_netsloop_fusIOn | on | -| remote_read_mode | non_authenticatIOn | -| enable_page_lsn_check | off | -| pagewriter_sleep | 2s | -| enable_opfusIOn | on | -| max_redo_log_size | 100 GB | -| pagewrite_thread_num | 1 | -| bgwrite_thread_num | 1 | -| dirty_page_percent_max | 1 | -| candidate_buf_percent_target | 01 | - -## Test Items and Conclusions - -### Test Result Summary - -| Test Item | Data Volume | Concurrent Transactions | Average CPU Usage | IOPS | IO Latency | Write Ahead Logs | tpmC | Test Time (Minute) | -| -------------------------------------- | ----------- | ----------------------- | ----------------- | ----- | ---------- | ---------------- | -------- | ------------------ | -| Single node | 100 GB | 500 | 29.39% | 6.50K | 1.94 ms | 3974 | 520896.3 | 10 | -| One primary node and one standby node | 100 GB | 500 | 30.4% | 5.31K | 453.2 us | 3944 | 519993.5 | 10 | -| One primary node and two standby nodes | 100 GB | 500 | 26.35% | 7.66K | 531.9 us | 3535 | 480842.2 | 10 | - -### Single Node - -- tpmC - - ![mogdb-on-x86-1](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-1.png) - - System data - - ![mogdb-on-x86-2](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-2.png) - -### One Primary Node and One Standby Node - -- tpmC - - ![mogdb-on-x86-3](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-3.png) - -- System data - - ![mogdb-on-x86-4](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-4.png) - -### One Primary Node and Two Standby Nodes - -- tpmC - - ![mogdb-on-x86-5](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-5.png) - -- System data - - ![mogdb-on-x86-6](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-6.png) diff --git a/product/en/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md b/product/en/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md index dc4723a4acbc71e541a21e60531699231307ad60..e661d41a374ff6859800e8ba8b3d0f9a273156b5 100644 --- a/product/en/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md +++ b/product/en/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md @@ -54,7 +54,7 @@ Table1 API Descriptions ``` PKG_SERVICE.JOB_FINISH( - id IN INTEGER, + id IN BIGINT, broken IN BOOLEAN, next_time IN TIMESTAMP DEFAULT sysdate); ``` @@ -63,7 +63,7 @@ Table1 API Descriptions | Parameter | Type | In/Out | Whether it can be empty | Definition | | --------- | --------- | ------ | ----------------------- | ------------------------------------------------------------ | - | id | integer | IN | No | Specified job ID. | + | id | bigint | IN | No | Specified job ID. | | broken | Boolean | IN | No | Status flag bit, **true** means disabled, **false** means enabled. The current job is updated according to the **true** or **false** value; if it is **null**, the status of the original job is not changed. | | next_time | timestamp | IN | Yes | The next run time, the default value is the current system time. If the parameter **broken** state is **true**, the parameter is updated to **4000-1-1**; if the parameter **broken** state is **false**, and if the parameter next_time is not null, the **next_time** value of the specified job is updated, if **next_time** is **null**, the **next_time** value is not updated. This parameter can be omitted and is the default value. | diff --git a/product/en/docs-mogdb/v3.0/toc.md b/product/en/docs-mogdb/v3.0/toc.md index 91b09de22ee89c58be95d2e02966d26e2a53bd4a..a4093242fd36af30d19f138dc680e337543425b1 100644 --- a/product/en/docs-mogdb/v3.0/toc.md +++ b/product/en/docs-mogdb/v3.0/toc.md @@ -8,12 +8,6 @@ + [MogDB Introduction](/overview.md) + [Comparison Between MogDB and openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB Release Notes](/about-mogdb/mogdb-release-notes.md) - + High Availability and Performance - + [HA Test](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + Performance Test - + [Performance Test Overview](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [Performance Test for MogDB on Kunpeng Servers](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [Performance Test for MogDB on x86 Servers](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + Open Source Components + [Docker-based MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/en/docs-mogdb/v3.0/toc_about.md b/product/en/docs-mogdb/v3.0/toc_about.md index 0a75bdd88fcf44b14efa389b31cfa912325819c5..077ca2d67ab20bcbe09070b7c37a35a691c3ed57 100644 --- a/product/en/docs-mogdb/v3.0/toc_about.md +++ b/product/en/docs-mogdb/v3.0/toc_about.md @@ -7,12 +7,6 @@ + [MogDB Introduction](/overview.md) + [Comparison Between MogDB and openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB Release Notes](/about-mogdb/mogdb-release-notes.md) -+ High Availability and Performance - + [HA Test](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + Performance Test - + [Performance Test Overview](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [Performance Test for MogDB on Kunpeng Servers](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [Performance Test for MogDB on x86 Servers](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + Open Source Components + [Docker-based MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/zh/docs-mogdb/v1.1.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md b/product/zh/docs-mogdb/v1.1.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md index d379dea5f7e489f51da2d467586f8ffcc369f16d..ef586833d19324208dc12aefc0181564fc4b6313 100644 --- a/product/zh/docs-mogdb/v1.1.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md +++ b/product/zh/docs-mogdb/v1.1.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md @@ -21,7 +21,7 @@ PG_JOB系统表存储用户创建的定时任务的任务详细信息，定时 | dbname | name | 标识作业要在哪个数据库执行的数据库名称 | | node_name | name | 标识当前作业是在哪个数据库主节点上创建和执行 | | job_status | "char" | 当前任务的执行状态，取值范围: ('r', 's', 'f', 'd')，默认为's'，取值含义:
Status of job step: r=running, s=successfully finished, f=job failed, d=disable
当job连续执行失败16次，会将job_status自动设置为失效状态'd'，后续不再执行该job。
注: 当用户将定时任务关闭（即: guc参数job_queue_processes为0时），由于监控job执行的线程不会启动，所以该状态不会根据job的实时状态进行设置，用户不需要关注此状态。只有当开启定时任务功能（即: guc参数job_queue_processes为非0时），系统才会根据当前job的实时状态刷新该字段值。 | -| start_date | timestamp without time zone | 作业第一次开始执行时间，时间精确到毫秒。 | +| start_date | timestamp without time zone | 作业第一次开始运行时间，时间精确到毫秒。 | | next_run_date | timestamp without time zone | 下次定时执行任务的时间，时间精确到毫秒。 | | failure_count | smallint | 失败计数，作业连续执行失败16次，不再继续执行。 | | interval | text | 作业执行的重复时间间隔。 | diff --git a/product/zh/docs-mogdb/v2.0.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md b/product/zh/docs-mogdb/v2.0.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md index d379dea5f7e489f51da2d467586f8ffcc369f16d..ef586833d19324208dc12aefc0181564fc4b6313 100644 --- a/product/zh/docs-mogdb/v2.0.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md +++ b/product/zh/docs-mogdb/v2.0.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md @@ -21,7 +21,7 @@ PG_JOB系统表存储用户创建的定时任务的任务详细信息，定时 | dbname | name | 标识作业要在哪个数据库执行的数据库名称 | | node_name | name | 标识当前作业是在哪个数据库主节点上创建和执行 | | job_status | "char" | 当前任务的执行状态，取值范围: ('r', 's', 'f', 'd')，默认为's'，取值含义:
Status of job step: r=running, s=successfully finished, f=job failed, d=disable
当job连续执行失败16次，会将job_status自动设置为失效状态'd'，后续不再执行该job。
注: 当用户将定时任务关闭（即: guc参数job_queue_processes为0时），由于监控job执行的线程不会启动，所以该状态不会根据job的实时状态进行设置，用户不需要关注此状态。只有当开启定时任务功能（即: guc参数job_queue_processes为非0时），系统才会根据当前job的实时状态刷新该字段值。 | -| start_date | timestamp without time zone | 作业第一次开始执行时间，时间精确到毫秒。 | +| start_date | timestamp without time zone | 作业第一次开始运行时间，时间精确到毫秒。 | | next_run_date | timestamp without time zone | 下次定时执行任务的时间，时间精确到毫秒。 | | failure_count | smallint | 失败计数，作业连续执行失败16次，不再继续执行。 | | interval | text | 作业执行的重复时间间隔。 | diff --git a/product/zh/docs-mogdb/v2.0.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md b/product/zh/docs-mogdb/v2.0.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md index d379dea5f7e489f51da2d467586f8ffcc369f16d..ef586833d19324208dc12aefc0181564fc4b6313 100644 --- a/product/zh/docs-mogdb/v2.0.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md +++ b/product/zh/docs-mogdb/v2.0.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md @@ -21,7 +21,7 @@ PG_JOB系统表存储用户创建的定时任务的任务详细信息，定时 | dbname | name | 标识作业要在哪个数据库执行的数据库名称 | | node_name | name | 标识当前作业是在哪个数据库主节点上创建和执行 | | job_status | "char" | 当前任务的执行状态，取值范围: ('r', 's', 'f', 'd')，默认为's'，取值含义:
Status of job step: r=running, s=successfully finished, f=job failed, d=disable
当job连续执行失败16次，会将job_status自动设置为失效状态'd'，后续不再执行该job。
注: 当用户将定时任务关闭（即: guc参数job_queue_processes为0时），由于监控job执行的线程不会启动，所以该状态不会根据job的实时状态进行设置，用户不需要关注此状态。只有当开启定时任务功能（即: guc参数job_queue_processes为非0时），系统才会根据当前job的实时状态刷新该字段值。 | -| start_date | timestamp without time zone | 作业第一次开始执行时间，时间精确到毫秒。 | +| start_date | timestamp without time zone | 作业第一次开始运行时间，时间精确到毫秒。 | | next_run_date | timestamp without time zone | 下次定时执行任务的时间，时间精确到毫秒。 | | failure_count | smallint | 失败计数，作业连续执行失败16次，不再继续执行。 | | interval | text | 作业执行的重复时间间隔。 | diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/1-database-server-exception-testing.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/1-database-server-exception-testing.md deleted file mode 100644 index daedfabde3bb93c59b142bca9be7e4240f80fa16..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/1-database-server-exception-testing.md +++ /dev/null @@ -1,217 +0,0 @@ ---- -title: MogDB数据库服务器异常测试 -summary: MogDB数据库服务器异常测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB数据库服务器异常测试 - -## 测试目的 - -测试MogDB在主节点异常宕机、备节点异常宕机、仲裁节点异常宕机、主+备节点异常宕机和仲裁+主节点异常宕机场景下的可用性、稳定性。 - -
- -## 测试环境 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | --------------------- | ------------- | ----- | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB1.1.0安装包 | | | -| 测试工具 | pgbench | | | - -
- -## 主节点异常宕机 - -测试方案: - -1. 重启主机 - -2. 监控TPS变化 - -3. 加入新备机 - -4. 监控集群状态及TPS变化 - -测试步骤: - -1. 重启主机 - - ```bash - reboot - ``` - -2. 原先的主库做为备库新加入集群 - - ```bash - gs_ctl -b full build -D /gaussdata/openGauss/db1 - ``` - -3. 新备库追赶日志 - -4. 追赶完成 - -测试结果: - -- 主库关闭后，TPS从9000降至0，持续时间为50秒左右。vip到新主库（节点2）后，tps从0增至13000左右 - -- 老主厍作为新备库加入集群同步埶据时，tps从13000降到0 同步完成后恢9000正常水平 - -
- -## 备节点异常宕机 - -测试方案: - -1. 重启备机 - -2. 监控TPS变化 - -3. 备机加入集群 - -4. 监控集群状态及TPS变化 - -测试步骤: - -1. 重启备机 - - ```bash - reboot - ``` - -2. 备机加入集群 - - ```bash - gs_ctl start -D /gaussdata/openGauss/db1 -M standby - ``` - -3. 追赶数据 - - ```bash - Standby Catchup - ``` - -4. 追赶完成 - -测试结果: - -- 备库关闭后，tps从9000左右升至13000左右 - -- 备库加入集群后追赶数据tps从 13000降至0，追完成后恢复9000正常水平 - -
- -## 仲裁节点异常宕机 - -测试方案: - -1. 在nodc.conf里的ping_list参数加入一个vip - -2. 操作vip的上线与下线来模拟仲裁宕机 - -测试步骤: - -1. 在服务器ip上添加vip - - ```bash - ifconfig ens160:1 192.168.122.111 netmask 255.255.255.0 - ``` - -2. 在node.conf的ping_list参数加入vip，同步主备的配置文件并重启监控脚本 - -3. 在监控脚本运行正常的情况下，下掉模拟仲裁的vip，观察现象 - - ```bash - ifconfig ens160:1 down - ``` - -4. 手动上线模拟仲裁的vip，观察现象 - -测试结果: - -- 主备的监脚本报告仲裁ip ping失败，没发生其它的操作和现象 -- tps未观察到现象 - -
- -## 主+备节点异常宕机 - -测试方案: - -1. 同时reboot主备节点机器 - -2. 开启主备node_run脚本 - -3. 主备机器开启后手动启动主备库 - -测试步骤: - -1. 同时reboot主备节点机器 - - ```bash - reboot - reboot - ``` - -2. 开启主备node_run脚本 - -3. 主备机器开启后手动启动主备库 - -测试结果: - -tps在关库后降至0，启动集群后大约15秒后tps恢复至9000 - -
- -## 仲裁+主节点异常宕机 - -测试方案: - -同时关闭主库机器和下线仲裁的ip - -测试步骤: - -1. 同时关闭主库机器和下线仲裁的ip - -2. 备库查询状态 - -3. 启动主库以及主库监控脚本 - - ```bash - gs_ctl start -D /gaussdata/openGauss/db1 -M primary - ``` - -4. 查询集群状态 - -测试结果: - -- 关掉主库和仲裁后，tps从9000降到0 -- 主库回来后tps恢复到9000 - -
- -## 仲裁+备节点异常宕机 - -测试方案: - -同时关闭备库机器和移除仲裁的ip - -测试步骤: - -1. 同时关闭备库机器和移除仲裁的ip - -2. 查询主库 - -3. 没有其他现象 - -4. 备库恢复正常之后，启动备库监控脚本后，主库脚本报错消失 - -5. 启动备库 - -6. 查询集群状态 diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/2-network-exception-testing.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/2-network-exception-testing.md deleted file mode 100644 index d33baaa5fd4955a4de570126ae4a307c7ead693e..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/2-network-exception-testing.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -title: MogDB网络异常测试 -summary: MogDB网络异常测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB网络异常测试 - -## 测试范围 - -1. 主节点服务网卡异常 -2. 主节点心跳网卡异常 -3. 节点服务+心跳网卡异常 -4. 从节点服务网卡异常 -5. 从节点心跳网卡异常 -6. 主机VIP异常（掉线） - -
- -## 测试环境 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | :-------------------: | :-----------: | :---: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB1.1.0安装包 | | | -| 测试工具 | pgbench | | | - -
- -## 主节点服务网卡异常 - -测试方案: - -1. 先关闭网卡再开启 -2. 观察数据库状态 - -测试步骤: - -1. 记录初始集群状态 -2. 先关闭网卡再开启 -3. 观察备库状态 -4. 备库节点无法连接主库 -5. 查询集群状态，主备未发生切换 - -测试结果: - -1. 未发生切换 -2. 主库网络恢复正常后,备库从disconnect状态恢复normal状态 -3. TPS在主库服务网卡异常时从9000跌到0。在开启服务网卡后约15s TPS从0升至9000 - -
- -## 主节点心跳网卡异常 - -测试方案: - -1. 先关闭网卡并开启 -2. 观察数据库状态 - -测试步骤: - -1. 观察数据库状态 - -2. 关闭网卡再开启 -3. 观察数据库状态 -4. 发现未发生切换 -5. 观察集群状态 - -测试结果: - -1. 未发生切换 - -2. 未观察到此过程中的TPS发生变化 - -
- -## 主节点服务+心跳网卡异常 - -测试方案: - -1. 先关闭网卡再开启 - -2. 观察数据库状态 - -测试步骤: - -1. 观察初始数据库状态 - -2. 先关闭网卡再开启 - -3. 主库正常 - -测试结果: - -1. 未发生切换 - -2. 在服务网卡断开后，tps从9000降到0，服务恢复后TPS升到9000 - -
- -## 从节点服务网卡异常 - -测试方案: - -1. 先关闭网卡再开启 - -2. 观察数据库状态 - -测试步骤: - -1. 记录初始集群状态 - -2. 先闭网卡再开启 - -3. 观察主库状态 - -4. 备库服务网卡ip ping失败 - -5. 主备未切换 - -测试结果: - -1. 备库监控脚本报心跳错误,主备未发生切换 - -2. 备机服务网卡异常后，主库TPS上升到13000，备机服务网卡恢复后,主库TPS恢复9000 - -
- -## 从节点心跳网卡异常 - -测试方案: - -1. 先关闭网卡60s，再开启 - -2. 观察数据库状态 - -测试步骤: - -1. 观察数据库初始状态 - -2. 先关闭网卡60s，再开启 - -3. 脚本执行完毕后观察树据库状态 - 发现未发生切换，主库监控脚本报告备库服务网卡ip ping失败 - -测试结果: - -1. 未发生切换 - -
- -## 主机VIP异常（掉线） - -测试方案: - -先关闭再开启网卡bond0:1 - -测试步骤: - -1. 先关闭再开启网卡bond0:1 - -2. 观察数据库状态 - -3. 观察vip - -测试结果: - -1. 主库vip自动上线 - -2. TPS在vip先下线再上线之后恢复了正常 diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/3-routine-maintenance-testing.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/3-routine-maintenance-testing.md deleted file mode 100644 index 9ec9a578be3a5f9c77746e8fab1b2fa8b5725572..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/3-routine-maintenance-testing.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -title: MogDB日常维护测试 -summary: MogDB日常维护测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB日常维护测试 - -## 测试范围 - -1. HA工具切换switchover - -2. gs_ctl切换 switchover - -3. HA工具切换failover - -4. gs_ctl切换 failover - -5. gs_ctl启动与HA配合相悖测试 - -6. 双主脑裂测试 - -
- -## 测试环境 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | :-------------------: | :-----------: | :---: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB1.1.0安装包 | | | -| 测试工具 | pgbench | | | - -
- -## HA工具切换switchover - -测试步骤: - -1. 查询数据库状态。 - -2. 在备库上执行切换命令，之后查询数据库状态。 - - ``` - /home/omm/venv/bin/python3 /home/omm/openGaussHA_standlone/switch.py -config /home/omm/openGaussHA_standlone/node.conf --switchover - ``` - -3. 在新主库上查询vip。 - -测试结果: - -主备角色切换正常。 - -
- -## gs_ctl切换 switchover - -测试步骤: - -1. 查询数据库状态。 -2. 使用命令进行切换，完成后查询状态。 -3. 查询vip。 - -测试结果: - -主备角色切换正常。 - -
- -## HA工具切换failover - -测试步骤: - -1. 查看数据库当前状态。 - -2. 执行ha的failover命令。 - -3. 观察集群状态。 - -4. 发现原备库成为新主库。 - -测试结果: - -1. 原先的备库成为新主库并有了vip。 - -2. 原先的主库被杀掉。 - -3. TPS在failover之后约10秒从6000升到13000。 - -
- -## gs_ctl切换 failover - -测试步骤: - -1. 查看数据库当前状态。 - -2. 执行gs_ctl的failover命令。 - -3. 观察新主库状态。 - -4. 观察原先的主库。 - -5. 观察脚本变化。 - -测试结果: - -1. 新主库等待10s被自动关闭,原先的主库不受影响，vip依然在原先的主库上。 - -2. 在备库failover后约5、6秒，TPS从9000升到13000。除波动外无其他变化。 - -
- -## gs_ctl启动与HA配合相悖测试 - -测试步骤: - -1. 记录集群状态。 - -2. 同时stop主库和备库。 - -3. gs_ctl 启动主库和备库，指定原来的备库为主库，主库为备库。 - -4. 观察脚本及集群状态。 - -测试结果: - -1. HA配置跟随实际情况调整。 - -2. TPS在关闭主库和备库库后从9000降至0，HA在检测到集群配置改变并在10s内修复挂起了新主库的vip，之后后约5、6s，TPS从0升到13000。 - -
- -## 双主脑裂测试 - -测试步骤: - -1. 查看当前集群状态。 - -2. 用gs_ctl 命令重启备库为主库。 - -测试结果: - -1. 超时10s后，非正常操作重新start的主库被监控脚本kill掉。 - -2. 修复备库并加入集群。 - - ``` - gs_ctl build -b full -D /gaussdata/openGauss/db1 - ``` - -3. 重启备库为主库后TPS从6000升到13000。 - -4. 备库重新加入集群后TPS从13000降到9000的一主一备水平。 diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/4-service-exception-testing.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/4-service-exception-testing.md deleted file mode 100644 index 660b77d7d6e0227f5984e492d06658ac9043f94f..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/4-service-exception-testing.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: MogDB服务异常测试 -summary: MogDB服务异常测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB服务异常测试 - -## 测试范围 - -1. 从节点数据库进程异常 - -2. 从库监控脚本异常 - -3. 主库文件系统异常 （影响HA进程） - -
- -## 测试环境 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | :-------------------: | :-----------: | :---: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB1.1.0安装包 | | | -| 测试工具 | pgbench | | | - -
- -## 从节点数据库进程异常 - -测试步骤: - -kill掉数据库进程。 - -测试结果: - -1. 观察集群状态。 - -2. 备库脚本显示心跳失败。 - -3. 未发生切换。 - -4. 在备库执行命令。 - - ``` - gs_ctl start -D /gaussdata/openGauss/db1 -M standby - ``` - -5. 集群恢复正常，自始至终未发生切换操作。 - -6. 在kill掉备库进程时，主库TPS从9000升到13000。 - -7. 未观察到主备切换操作。 - -
- -## 从库监控脚本异常 - -测试步骤: - -kill掉主库监控脚本。 - -测试结果: - -1. 观察集群状态。 - -2. 未观察到切换操作。 - -3. 主库监控脚本报告从库的脚本异常。 - -4. 恢复从库监控脚本。 - -5. TPS维持在9000。 - -
- -## 主库文件系统异常 （影响HA进程） - -测试步骤: - -修改ha所调用的主库数据库脚本权限，如gs_ctl。 - -测试结果: - -1. 修改了gs_ctl的rwx权限后，主库监控脚本报告心跳失败。 - -2. 检测不到实例状态。 - -3. 查询当前集群状态。 - -4. 等待约两分钟，未见发生切换。 diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/MogDB-ha-test-report.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/MogDB-ha-test-report.md deleted file mode 100644 index 44f64234eff68fe8167b31b2881ccf4d37bacfc9..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/ha/MogDB-ha-test-report.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: 高可用测试 -summary: 高可用测试 -author: Zhang Cuiping -date: 2021-06-07 ---- - -# 高可用测试 - -## 测试目的 - -数据库的高可用测试是为了最大程度地为用户提供服务，避免服务器宕机等故障带来的服务中断。数据库的高可用性不仅仅体现在数据库能否持续提供服务，而且也体现在能否保证数据的一致性。 - -## 测试环境 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | :-------------------: | :-----------: | :---: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB 安装包 | | | -| 测试工具 | pgbench | | | - -## 高可用、可扩展性测试 - -| 序号 | 测试项 | 描述 | -| ---- | -------------------- | ------------------------------------------------------------ | -| 1 | 读写分离集群实现 | 支持按照读写进行路由，并实现读、写的任务分发 | -| 2 | 扩容/缩容 | 在负载场景下，添加/减少物理设备时，能够保持前端应用不间断 | -| 3 | 共享存储集群 | 支持2节点共享存储集群，支持故障自动切换，支持并发事务的负载均衡 | -| 4 | 服务异常测试 | 支持在从节点数据库进程异常 、从库监控脚本异常和主库文件系统异常场景下最大程度的保证应用的可用性 | -| 5 | 日常维护测试 | 支持gs_ctl(数据库服务控制工具)和MogHA(高可用组件)进行Switchover和Failover的切换操作 | -| 6 | 数据库服务器异常测试 | 能够提供在主节点异常宕机、备节点异常宕机、仲裁节点异常宕机、主节点和备节点异常宕机和仲裁+主节点异常宕机的场景下应用的可用性和稳定性 | -| 7 | 网络异常测试 | 能够提供在主节点服务网卡异常、主节点心跳网卡异常、主节点服务+心跳网卡异常、从节点服务网卡异常、主机VIP异常的场景下最大程度的保证应用的可用性 | diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/1-performance-test-overview.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/1-performance-test-overview.md deleted file mode 100644 index b5dd717d6e69271377d1f6b06e198035a449defd..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/1-performance-test-overview.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: 性能测试简述 -summary: 性能测试简述 -author: Liu Xu -date: 2021-03-04 ---- - -# 性能测试简述 - -
- -## 测试相关指标 - -### 数据量 - -每个仓库约70MB数据量，为了让数据量更加符合实际使用场景，初始化创建1000个仓库，添加相关索引后，数据库大小在100G左右。 - -
- -### 并发数 - -模拟真实的用户行为，同时在线请求操作的客户端数量。 - -
- -### 系统平均负载 - -系统负载指系统CPU繁忙程度的度量，即有多少进程在等待被CPU调度，反应整个系统的压力，而系统平均负载被定义为在特定时间间隔内运行队列中的平均进程树，这个一段时间一般取1分钟、5分钟、10分钟。 - -
- -### CPU使用率 - -CPU使用率其实是指运行的程序占用的CPU资源，表示你的机器在某个时间点的运行程序的情况。使用率越高，说明你的机器在这个时间上运行了很多程序，反之较少。使用率的高低与CPU强弱有直接关系，通过CPU的使用率来判断CPU是否达到瓶颈。 - -
- -### IOPS - -IOPS（Input/OutputOperatIOns Per Second）是一个用于计算机存储设备（如硬盘（HDD）、固态硬盘（SSD）或存储区域网络（SAN））性能测试的量测方式，可以视为是每秒的读写次数，是衡量磁盘性能的主要指标之一。IOPS是指单位时间内系统能处理的I/O请求数量，一般以每秒处理的I/O请求数量为单位，I/O请求通常为读或写数据操作请求。 - -
- -### IO延时 - -IO响应时间也被称为IO延时(IOLatency)，IO响应时间就是从操作系统内核发出的一个读或者写的IO命令到操作系统内核接收到IO回应的时间，注意不要和单个IO时间混淆了，单个IO时间仅仅指的是IO操作在磁盘内部处理的时间，而IO响应时间还要包括IO操作在IO等待队列中所花费的等待时间。 - -
- -### tpmC - -TPC-C使用三种性能和价格度量，其中性能由TPC-C吞吐率衡量，单位是tpmC。tpm是transactions per minute的简称；C指TPC中的C基准程序。它的定义是每分钟内系统处理的新订单个数。 - -
- -## 准备测试程序 - -### 测试工具介绍 - -BenchmarkSQL是一款经典的开源数据库测试工具，内嵌了TPC-C测试脚本，可以对PostgreSQL、MySQL、Oracle以及SQL Server等数据库直接进行测试。它通过JDBC测试OLTP（联机事务处理, Online TransactIOn Processing）的TPC-C。 - -
- -### 测试规范说明 - -TPC-C 是一个对 OLTP（联机交易处理）系统进行测试的规范，使用一个商品销售模型对 OLTP 系统进行测试，其中包含五类事务, 其事务内容及特点如下: - -- **NewOrder - 新订单的生成** - - 事务内容: 对于任意一个客户端,从固定的仓库随机选取 5-15 件商品,创建新订单.其中 1%的订单要由假想的用户操作失败而回滚。 - - 主要特点: 中量级、读写频繁、要求响应快。 - -- **Payment - 订单付款** - - 事务内容: 对于任意一个客户端,从固定的仓库随机选取一个辖区及其内用户,采用随机的金额支付一笔订单,并作相应历史纪录。 - - 主要特点: 轻量级、读写频繁、要求响应快。 - -- **OrderStatus - 最近订单查询** - - 事务内容: 对于任意一个客户端,从固定的仓库随机选取一个辖区及其内用户，读取其最后一条订单，显示订单内每件商品的状态。 - - 主要特点: 中量级、只读频率低、要求响应快。 - -- **Delivery - 配送** - - 事务内容: 对于任意一个客户端,随机选取一个发货包，更新被处理订单的用户余额，并把该订单从新订单中删除。 - - 主要特点: 1-10 个批量，读写频率低、较宽松的响应时间。 - -- **StockLevel - 库存缺货状态分析** - - 事物内容: 对于任意一个客户端，从固定的仓库和辖区随机选取最后 20 条订单，查看订单中所有的货物的库存，计算并显示所有库存低于随机生成域 值的商品数量。 - - 主要特点: 重量级，只读频率低,较宽松的响应时间。 - -
- -### 测试规则描述 - -在测试开始前，TPC-C Benchmark 规定了数据库的初始状态，也就是数据库中数据生成的规则，其中 ITEM 表中固定包含 10 万种商品，仓库的数量可进行调整，本次测试初始化WAREHOUSE 表中有 1000 条记录，那么: - -- STOCK 表中应有 1000 * 10 万条记录（每个仓库对应 10 万种商品的库存数据） -- DISTRICT 表中应有1000 * 10 条记录（每个仓库为 10 个地区提供服务） -- CUSTOMER 表中应有 1000 \* 10 \* 3000 条记录（每个地区有 3000 个客户） -- HISTORY 表中应有 1000 \* 10 \* 3000 条记录（每个客户一条交易历史） -- ORDER 表中应有 1000 \* 10 \* 3000 条记录（每个地区 3000 个订单），并且最后生成的 900 个订单被添加到 NEW-ORDER 表中，每个订单随机生成 5 ~ 15 条 ORDER-LINE 记录。 - -> TPC-C 使用 tpmC 值（Transactions per Minute）来衡量系统最大有效吞吐量 (MQTh, Max Qualified Throughput)，其中 Transactions 以 NewOrder Transaction 为准，即最终衡量单位为每分钟处理的新订单数。 diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md deleted file mode 100644 index 9c510ca7689aa5f6696d95e0251fbe202ae49cbc..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md +++ /dev/null @@ -1,334 +0,0 @@ ---- -title: MogDB on鲲鹏性能测试 -summary: MogDB on鲲鹏性能测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB on鲲鹏性能测试 - -## 测试目的 - -本文主要讲述了对MogDB 2.0.0 版本on鲲鹏进行的单节点和一主多备（同步备、异步备）的测试。 - -
- -## 测试环境 - -### 环境配置 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | --------------------- | ------------- | ----- | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB 1.1.0安装包 | | | - -
- -### 测试工具 - -| 工具名称 | 用途 | -| ------------- | ---------------------------------| -| Benchmarksql5.0 | BenchmarkSQL是对OLTP数据库主流测试标准TPC-C的开源JAVA实现,用于对数据库事务处理能力的评估测试。 | - -
- -## 测试步骤 - -### MogDB数据库端操作 - -1. 获取数据库安装包 - -2. 安装数据库 - -3. 创建tpcc测试用户和数据库 - - ```bash - create user [username] identified by 'passwd'; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. 停止数据库修改数据库配置文件postgresql.conf数据库，在末尾添加配置参数（单节点测试追加参数如下） - - ```bash - max_connections = 4096 - allow_concurrent_tuple_update = true - audit_enabled = off - checkpoint_segments = 1024 - cstore_buffers =16MB - enable_alarm = off - enable_codegen = false - enable_data_replicate = off - full_page_writes = off - max_files_per_process = 100000 - max_prepared_transactions = 2048 - shared_buffers = 350GB - use_workload_manager = off - wal_buffers = 1GB - work_mem = 1MB - log_min_messages = FATAL - transaction_isolation = 'read committed' - default_transaction_isolation = 'read committed' - synchronous_commit = on - fsync = on - maintenance_work_mem = 2GB - vacuum_cost_limit = 2000 - autovacuum = on - autovacuum_mode = vacuum - autovacuum_max_workers = 5 - autovacuum_naptime = 20s - autovacuum_vacuum_cost_delay =10 - xloginsert_locks = 48 - update_lockwait_timeout =20min - enable_mergejoin = off - enable_nestloop = off - enable_hashjoin = off - enable_bitmapscan = on - enable_material = off - wal_log_hints = off - log_duration = off - checkpoint_timeout = 15min - enable_save_datachanged_timestamp =FALSE - enable_thread_pool = on - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - enable_double_write = on - enable_incremental_checkpoint = on - enable_opfusion = on - advance_xlog_file_num = 10 - numa_distribute_mode = 'all' - track_activities = off - enable_instr_track_wait = off - enable_instr_rt_percentile = off - track_counts =on - track_sql_count = off - enable_instr_cpu_timer = off - plog_merge_age = 0 - session_timeout = 0 - enable_instance_metric_persistent = off - enable_logical_io_statistics = off - enable_user_metric_persistent =off - enable_xlog_prune = off - enable_resource_track = off - instr_unique_sql_count = 0 - enable_beta_opfusion = on - enable_beta_nestloop_fusion = on - autovacuum_vacuum_scale_factor = 0.02 - autovacuum_analyze_scale_factor = 0.1 - client_encoding = UTF8 - lc_messages = en_US.UTF-8 - lc_monetary = en_US.UTF-8 - lc_numeric = en_US.UTF-8 - lc_time = en_US.UTF-8 - modify_initial_password = off - ssl = off - enable_memory_limit = off - data_replicate_buffer_size = 16384 - max_wal_senders = 8 - log_line_prefix = '%m %u %d %h %p %S' - vacuum_cost_limit = 10000 - max_process_memory = 12582912 - recovery_max_workers = 1 - recovery_parallelism = 1 - explain_perf_mode = normal - remote_read_mode = non_authentication - enable_page_lsn_check = off - pagewriter_sleep = 100 - ``` - -
- -### benchmarksql端操作 - -1. 修改配置文件 - - 进入benchmarksql安装目录下，找到run目录下的配置文件[config file] - - ```text - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. 运行runDataBuild.sh生成数据 - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. 运行runBenchmark.sh测试数据库 - - ``` - ./runBenchmark.sh [config file] - ``` - -
- -### 操作系统配置 - -1. 修改操作系统内核PAGESIZE（仅EulerOS需执行此步骤） - - **安装kernel-4.19.36-1.aarch64.rpm** - - ``` - # rpm -Uvh --force --nodeps kernel-4.19.36-1.aarch64.rpm - - # *: 此文件是基于linux 4.19.36的内核包，获取路径如下: - - # 10.44.133.121 (root/Huawei12#$) - - /data14/xy_packages/kernel-4.19.36-1.aarch64.rpm - ``` - - **修改操作系统内核配置文件中root的启动项** - - ``` - # vim /boot/efi/EFI/euleros/grubenv //修改前备份grubenv文件 - - # GRUB Environment Block - - saved_entry=EulerOS (4.19.36) 2.0 (SP8) -- 此处修改为4.19.36 - ``` - -
- -### 文件系统 - -1. 修改xfs文件系统blocksize为8K - - \# 下面命令查看当前挂载的nvme盘 - - ```bash - # df -h | grep nvme - /dev/nvme0n1 3.7T 2.6T 1.2T 69% /data1 - /dev/nvme1n1 3.7T 1.9T 1.8T 51% /data2 - /dev/nvme2n1 3.7T 2.2T 1.6T 59% /data3 - /dev/nvme3n1 3.7T 1.4T 2.3T 39% /data4 - - # xfs_info命令可以查看nvme盘的信息 - - xfs_info /data1 - ``` - -2. 针对需要修改的磁盘，备份所需的数据 - -3. 重新格式化磁盘 - - ```bash - # 以/dev/nvme0n1盘，加载点为/data1为例，相关参考命令如下 - - umount /data1 - - mkfs.xfs -b size=8192 /dev/nvme0n1 -f - - mount /dev/nvme0n1 /data1 - ``` - -
- -## 测试项及结论 - -### 测试结果汇总 - -| 测试项 | 数据量 | 并发数 | CPU平均使用率 | IOPS | IO延时 | WAL数量 | tpmC | 测试时长 | -| ------ | ------ | ------ | --------- | ------ | -------- | ------- | ---------- | -------- | -| 单节点 | 100G | 500 | 77.49% | 17.96K | 819.05us | 13260 | 1567226.12 | 10分钟 | -| 一主一备 | 100G | 500 | 57.64% | 5.31K | 842.78us | 13272 | 1130307.87 | 10分钟 | -| 一主两备 | 100G | 500 | 60.77% | 5.3K | 821.66us | 14324 | 1201560.28 | 10分钟 | - -
- -### 单节点 - -- tpmC - - ![单节点tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-1.png) - -- 系统数据 - - ![单节点系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-2.png) - -
- -### 一主一备 - -- tpmC - - ![一主一备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-3.png) - -- 系统数据 - - ![一主一备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-4.png) - -
- -### 一主两备 - -- tpmC - - ![一主两备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-5.png) - -- 系统数据 - - ![一主两备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-6.png) diff --git a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md b/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md deleted file mode 100644 index 03e5b38dbebb8009c1615231e182996c9af05754..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v2.1/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md +++ /dev/null @@ -1,672 +0,0 @@ ---- -title: MogDB on x86性能测试 -summary: MogDB on x86性能测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB on x86性能测试 - -## 测试目的 - -本文主要讲述了对MogDB 2.0.0 版本on x86进行的单节点和一主多备（同步备、异步备）的测试。 - -
- -## 测试环境 - -### 环境配置 - -| 服务器类型 | Fit Server | NFS5280M5 | -| -------------- | ---------------------------------------- | ---------------------------------------- | -| CPU | 144cIntel® Xeon(R) Gold 6240 CPU@2.60GHz | 64cIntel® Xeon(R) Gold 5218 CPU @2.30GHz | -| 内存 | 768G | 128G | -| 硬盘 | SAS SSD | SAS SSD | -| 网卡 | 10GE | 10GE | - -
- -### 测试工具 - -| 工具名称 | 用途 | -| --------------- | ----------------------------------------------------- | -| Benchmarksql5.0 | BenchmarkSQL是对OLTP数据库主流测试标准TPC-C的开源JAVA实现，用于对数据库事务处理能力的评估测试。 | - -
- -## 测试步骤 - -### MogDB数据库端操作 - -1. 获取数据库安装包 - -2. 安装数据库 - -3. 创建tpcc测试用户和数据库 - - ```sql - create user [username] identified by ‘passwd’; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. 停止数据库修改数据库配置文件postgresql.conf数据库，在末尾添加配置参数（单节点测试追加参数如下） - - ```bash - max_connections = 4096 - - allow_concurrent_tuple_update = true - - audit_enabled = off - - checkpoint_segments = 1024 - - cstore_buffers =16MB - - enable_alarm = off - - enable_codegen = false - - enable_data_replicate = off - - full_page_writes = off - - max_files_per_process = 100000 - - max_prepared_transactions = 2048 - - shared_buffers = 350GB - - use_workload_manager = off - - wal_buffers = 1GB - - work_mem = 1MB - - log_min_messages = FATAL - - transaction_isolation = 'read committed' - - default_transaction_isolation = 'read committed' - - synchronous_commit = on - - fsync = on - - maintenance_work_mem = 2GB - - vacuum_cost_limit = 2000 - - autovacuum = on - - autovacuum_mode = vacuum - - autovacuum_max_workers = 5 - - autovacuum_naptime = 20s - - autovacuum_vacuum_cost_delay =10 - - xloginsert_locks = 48 - - update_lockwait_timeout =20min - - enable_mergejoin = off - - enable_nestloop = off - - enable_hashjoin = off - - enable_bitmapscan = on - - enable_material = off - - wal_log_hints = off - - log_duration = off - - checkpoint_timeout = 15min - - enable_save_datachanged_timestamp =FALSE - - enable_thread_pool = on - - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - - enable_double_write = on - - enable_incremental_checkpoint = on - - enable_opfusion = on - - advance_xlog_file_num = 10 - - numa_distribute_mode = 'all' - - track_activities = off - - enable_instr_track_wait = off - - enable_instr_rt_percentile = off - - track_counts =on - - track_sql_count = off - - enable_instr_cpu_timer = off - - plog_merge_age = 0 - - session_timeout = 0 - - enable_instance_metric_persistent = off - - enable_logical_io_statistics = off - - enable_user_metric_persistent =off - - enable_xlog_prune = off - - enable_resource_track = off - - instr_unique_sql_count = 0 - - enable_beta_opfusion = on - - enable_beta_nestloop_fusion = on - - autovacuum_vacuum_scale_factor = 0.02 - - autovacuum_analyze_scale_factor = 0.1 - - client_encoding = UTF8 - - lc_messages = en_US.UTF-8 - - lc_monetary = en_US.UTF-8 - - lc_numeric = en_US.UTF-8 - - lc_time = en_US.UTF-8 - - modify_initial_password = off - - ssl = off - - enable_memory_limit = off - - data_replicate_buffer_size = 16384 - - max_wal_senders = 8 - - log_line_prefix = '%m %u %d %h %p %S' - - vacuum_cost_limit = 10000 - - max_process_memory = 12582912 - - recovery_max_workers = 1 - - recovery_parallelism = 1 - - explain_perf_mode = normal - - remote_read_mode = non_authentication - - enable_page_lsn_check = off - - pagewriter_sleep = 100对方的 - ``` - -
- -### benchmarksql端操作 - -1. 修改配置文件 - - 进入benchmarksql安装目录下，找到run目录下的配置文件[config file] - - ```bash - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. 运行runDataBuild.sh生成数据 - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. 运行runBenchmark.sh测试数据库 - - ``` - ./runBenchmark.sh [config file] - ``` - -
- -### 操作系统参数 - -```bash -vm.dirty_background_ratIO=5 - -vm.dirty_ratIO=10 - -kernel.sysrq=0 - -net.ipv4.ip_forward=0 - -net.ipv4.conf.all.send_redirects=0 - -net.ipv4.conf.default.send_redirects=0 - -net.ipv4.conf.all.accept_source_route=0 - -net.ipv4.conf.default.accept_source_route=0 - -net.ipv4.conf.all.accept_redirects=0 - -net.ipv4.conf.default.accept_redirects=0 - -net.ipv4.conf.all.secure_redirects=0 - -net.ipv4.conf.default.secure_redirects=0 - -net.ipv4.icmp_echo_ignore_broadcasts=1 - -net.ipv4.icmp_ignore_bogus_error_responses=1 - -net.ipv4.conf.all.rp_filter=1 - -net.ipv4.conf.default.rp_filter=1 - -net.ipv4.tcp_syncookies=1 - -kernel.dmesg_restrict=1 - -net.ipv6.conf.all.accept_redirects=0 - -net.ipv6.conf.default.accept_redirects=0 - -net.core.rmem_max = 21299200 - -net.core.rmem_default = 21299200 - -net.core.somaxconn = 65535 - -net.ipv4.tcp_tw_reuse = 1 - -net.sctp.sctp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -kernel.sem = 250 6400000 1000 25600 - -net.core.wmem_default = 21299200 - -kernel.shmall = 1152921504606846720 - -net.core.wmem_max = 21299200 - -net.sctp.sctp_rmem = 8192 250000 16777216 - -net.core.netdev_max_backlog = 65535 - -kernel.shmmax = 18446744073709551615 - -net.sctp.sctp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_max_syn_backlog = 65535 - -vm.oom_panic_on_oom=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -net.ipv4.tcp_timestamps = 1 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.sctp.path_max_retrans = 10 - -net.sctp.max_init_retransmits = 10 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_synack_retries = 5 - -kernel.core_uses_pid=1 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%p - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%h - -net.core.netdev_max_backlog = 65535 - -net.core.rmem_default = 21299200 - -net.core.rmem_max = 21299200 - -net.core.somaxconn = 65535 - -net.core.wmem_default = 21299200 - -net.core.wmem_max = 21299200 - -net.ipv4.conf.all.accept_redirects = 0 - -net.ipv4.conf.all.rp_filter = 1 - -net.ipv4.conf.all.secure_redirects = 0 - -net.ipv4.conf.all.send_redirects = 0 - -net.ipv4.conf.default.accept_redirects = 0 - -net.ipv4.conf.default.accept_source_route = 0 - -net.ipv4.conf.default.rp_filter = 1 - -net.ipv4.conf.default.secure_redirects = 0 - -net.ipv4.conf.default.send_redirects = 0 - -net.ipv4.conf.enp135s0.accept_redirects = 0 - -net.ipv4.conf.enp135s0.accept_source_route = 0 - -net.ipv4.conf.enp135s0.forwarding = 1 - -net.ipv4.conf.enp135s0.rp_filter = 1 - -net.ipv4.conf.enp135s0.secure_redirects = 0 - -net.ipv4.conf.enp135s0.send_redirects = 0 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_max_syn_backlog = 65535 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_mem = 362715 483620 725430 - -#net.ipv4.tcp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.ipv4.tcp_tw_reuse = 1 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.udp_mem = 725430 967240 1450860 - -#net.ipv4.tcp_max_orphans = 3276800 - -#net.ipv4.tcp_fin_timeout = 60 - -#net.ipv4.ip_local_port_range = 26000 65535 - -net.ipv4.tcp_retries2 = 80 - -#net.ipv4.ip_local_reserved_ports = 20050-30007 - -vm.min_free_kbytes = 40140150 -``` - -
- -### 数据库参数 - -| 参数项 | MogDB | -| --------------------------------- | -------------------------------- | -| listen_addresses | 具体ip | -| port | 26000 | -| max_connectIOns | 4096 | -| wal_level | hot_standby | -| archive_mode | on | -| archive_command | /bin/ture | -| max_wal_senders | 16 | -| wal_keep_segments | 16 | -| max_replicatIOn_slots | 8 | -| hot_standby | on | -| logging_collector | on | -| log_directory | 安装工具指定 | -| log_filename | PostgreSQL-%Y-%m-%d_%H%M%S.log | -| log_min_duratIOn_statement | 1800000 | -| log_line_prefix | %m%c%d%p%a%x%n%e | -| log_timezone | PRC | -| datestyle | iso,mdy | -| timezone | PRC | -| default_text_search_config | pg_catalog.english | -| applicatIOn_name | dn_6001 | -| max_prepared_transactIOns | 2048 | -| shared_buffers | 350GB | -| wal_buffers | 1GB | -| work_mem | 64MB | -| log_min_messages | FATAL | -| synchronous_commit | on | -| fsync | on | -| maintenance_work_mem | 2GB | -| autovacuum | on | -| autovacuum_max_workers | 5 | -| autovacuum_naptime | 20s | -| autovacuum_vacuum_cost_delay | 10 | -| enable_mergejoin | off | -| enable_nestloop | off | -| enable_hashjoin | off | -| enable_bitmapscan | on | -| enable_material | off | -| wal_log_hints | off | -| log_duratIOn | off | -| checkpoint_timeout | 15min | -| track_activities | off | -| track_counts | on | -| autovacuum_vacuum_scale_factor | 0.02 | -| autovacuum_analyze_scale_factor | 0.1 | -| ssl | off | -| local_bind_address | 具体IP | -| max_inner_tool_connectIOns | 10 | -| password_encryptIOn_type | 0 | -| comm_tcp_mode | on | -| comm_quota_size | 1024kB | -| max_process_memory | 700GB | -| bulk_write_ring_size | 2GB | -| checkpoint_segments | 1024 | -| incremental_checkpoint_timeout | 60s | -| archive_dest | /log/archive | -| enable_slot_log | off | -| data_replicate_buffer_size | 128MB | -| walsender_max_send_size | 8MB | -| enable_kill_query | off | -| connectIOn_alARM_rate | 0.9 | -| alARM_report_interval | 10 | -| alARM_component | /opt/huawei/snas/bin/snas_cm_cmd | -| lockwait_timeout | 1200s | -| pgxc_node_name | xxx | -| audit_directory | 安装工具指定 | -| explain_perf_mode | pretty | -| job_queue_processes | 10 | -| default_storage_nodegroup | installatIOn | -| expected_computing_nodegroup | query | -| replicatIOn_type | 1 | -| recovery_max_workers | 4 | -| available_zone | AZ1 | -| allow_concurrent_tuple_update | TRUE | -| audit_enabled | off | -| cstore_buffers | 16MB | -| enable_alARM | off | -| enable_codegen | FALSE | -| enable_data_replicate | off | -| max_file_per_process | 10000 | -| use_workload_manager | off | -| xloginsert_locks | 48 | -| update_lockwait_timeout | 20min | -| enable_save_datachanged_timestamp | FALSE | -| enable_thread_pool | off | -| enable_double_write | on | -| enable_incremental_checkpoint | on | -| advance_xlog_file_num | 10 | -| enable_instr_track_wait | off | -| enable_instr_rt_percentile | off | -| track_sql_count | off | -| enable_instr_cpu_timer | off | -| plog_merge_age | 0 | -| sessIOn_timeout | 0 | -| enable_instance_metric_persistent | off | -| enable_logical_IO_statistics | off | -| enable_user_metric_persistent | off | -| enable_xlog_prune | off | -| enable_resource_track | off | -| instr_unique_sql_count | 0 | -| enable_beta_opfusIOn | on | -| enable_bete_netsloop_fusIOn | on | -| remote_read_mode | non_authenticatIOn | -| enable_page_lsn_check | off | -| pagewriter_sleep | 2s | -| enable_opfusIOn | on | -| max_redo_log_size | 100GB | -| pagewrite_thread_num | 1 | -| bgwrite_thread_num | 1 | -| dirty_page_percent_max | 1 | -| candidate_buf_percent_target | 01 | - -
- -## 测试项及结论 - -### 测试结果汇总 - -| 测试项 | 数据量 | 并发数 | CPU平均使用率 | IOPS | IO延时 | WAL数量 | tpmC | 测试时长 | -| -------- | ------ | ------ | ---------- | ----- | ------- | ------- | -------- | -------- | -| 单节点 | 100G | 500 | 29.39% | 6.50K | 1.94.ms | 3974 | 520896.3 | 10分钟 | -| 一主一备 | 100G | 500 | 30.4% | 5.31K | 453.2us | 3944 | 519993.5 | 10分钟 | -| 一主两备 | 100G | 500 | 26.35% | 7.66K | 531.9us | 3535 | 480842.2 | 10分钟 | - -
- -### 单节点 - -- tpmC - - ![单节点tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-1.png) - -- 系统数据 - - ![单节点系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-2.png) - -
- -### 一主一备 - -- tpmC - - ![一主一备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-3.png) - -- 系统数据 - - ![一主一备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-4.png) - -
- -### 一主两备 - -- tpmC - - ![一主两备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-5.png) - -- 系统数据 - - ![一主两备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-6.png) diff --git a/product/zh/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md b/product/zh/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md index 1e9d32c16fba801e6029d7db903eeb5ca1937e79..a9d5ca5abc3cebace61437e5d962b5461e1795ed 100644 --- a/product/zh/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md +++ b/product/zh/docs-mogdb/v2.1/developer-guide/scheduled-jobs/pkg-service.md @@ -54,7 +54,7 @@ MogDB 2.1版本提供了以下PKG_SERVICE接口来实现定时任务管理。 ``` PKG_SERVICE.JOB_FINISH( - id IN INTEGER, + id IN BIGINT, broken IN BOOLEAN, next_time IN TIMESTAMP DEFAULT sysdate); ``` @@ -63,7 +63,7 @@ MogDB 2.1版本提供了以下PKG_SERVICE接口来实现定时任务管理。 | 参数 | 类型 | 入参/出参 | 是否可以为空 | 描述 | | --------- | --------- | --------- | ------------ | ------------------------------------------------------------ | - | id | integer | IN | 否 | 指定的作业号。 | + | id | bigint | IN | 否 | 指定的作业号。 | | broken | Boolean | IN | 否 | 状态标志位，true代表禁用，false代表启用。根据true或false值更新当前job；如果为空值，则不改变原有job的状态。 | | next_time | timestamp | IN | 是 | 下次运行时间，默认为当前系统时间。如果参数broken状态为true，则更新该参数为'4000-1-1'；如果参数broken状态为false，且如果参数next_time不为空值，则更新指定job的next_time值，如果next_time为空值，则不更新next_time值。该参数可以省略，为默认值。 | diff --git a/product/zh/docs-mogdb/v2.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md b/product/zh/docs-mogdb/v2.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md index d379dea5f7e489f51da2d467586f8ffcc369f16d..ef586833d19324208dc12aefc0181564fc4b6313 100644 --- a/product/zh/docs-mogdb/v2.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md +++ b/product/zh/docs-mogdb/v2.1/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md @@ -21,7 +21,7 @@ PG_JOB系统表存储用户创建的定时任务的任务详细信息，定时 | dbname | name | 标识作业要在哪个数据库执行的数据库名称 | | node_name | name | 标识当前作业是在哪个数据库主节点上创建和执行 | | job_status | "char" | 当前任务的执行状态，取值范围: ('r', 's', 'f', 'd')，默认为's'，取值含义:
Status of job step: r=running, s=successfully finished, f=job failed, d=disable
当job连续执行失败16次，会将job_status自动设置为失效状态'd'，后续不再执行该job。
注: 当用户将定时任务关闭（即: guc参数job_queue_processes为0时），由于监控job执行的线程不会启动，所以该状态不会根据job的实时状态进行设置，用户不需要关注此状态。只有当开启定时任务功能（即: guc参数job_queue_processes为非0时），系统才会根据当前job的实时状态刷新该字段值。 | -| start_date | timestamp without time zone | 作业第一次开始执行时间，时间精确到毫秒。 | +| start_date | timestamp without time zone | 作业第一次开始运行时间，时间精确到毫秒。 | | next_run_date | timestamp without time zone | 下次定时执行任务的时间，时间精确到毫秒。 | | failure_count | smallint | 失败计数，作业连续执行失败16次，不再继续执行。 | | interval | text | 作业执行的重复时间间隔。 | diff --git a/product/zh/docs-mogdb/v2.1/toc.md b/product/zh/docs-mogdb/v2.1/toc.md index 8a3ee2c28ccd0a9f9e721ec482279c67f90b8f6d..84116c46099021939c5220d596285f0ec81ad149 100644 --- a/product/zh/docs-mogdb/v2.1/toc.md +++ b/product/zh/docs-mogdb/v2.1/toc.md @@ -8,12 +8,6 @@ + [MogDB简介](/overview.md) + [MogDB与openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB发行说明](/about-mogdb/mogdb-release-notes.md) - + 高可用和高性能 - + [高可用测试](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + 性能测试 - + [性能测试简述](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [MogDB on 鲲鹏性能测试](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [MogDB on x86性能测试](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + 开源组件 + [容器版本的MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/zh/docs-mogdb/v2.1/toc_about.md b/product/zh/docs-mogdb/v2.1/toc_about.md index 7e8449f64f3d83fb295dd4598aaca6771f1c9438..6a02768ceedf41f3c082eb10ef5fa9d757c2d596 100644 --- a/product/zh/docs-mogdb/v2.1/toc_about.md +++ b/product/zh/docs-mogdb/v2.1/toc_about.md @@ -7,12 +7,6 @@ + [MogDB简介](/overview.md) + [MogDB与openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB发行说明](/about-mogdb/mogdb-release-notes.md) -+ 高可用和高性能 - + [高可用测试](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + 性能测试 - + [性能测试简述](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [MogDB on 鲲鹏性能测试](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [MogDB on x86性能测试](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + 开源组件 + [容器版本的MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/ha/MogDB-ha-test-report.md b/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/ha/MogDB-ha-test-report.md deleted file mode 100644 index 44f64234eff68fe8167b31b2881ccf4d37bacfc9..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/ha/MogDB-ha-test-report.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: 高可用测试 -summary: 高可用测试 -author: Zhang Cuiping -date: 2021-06-07 ---- - -# 高可用测试 - -## 测试目的 - -数据库的高可用测试是为了最大程度地为用户提供服务，避免服务器宕机等故障带来的服务中断。数据库的高可用性不仅仅体现在数据库能否持续提供服务，而且也体现在能否保证数据的一致性。 - -## 测试环境 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | :-------------------: | :-----------: | :---: | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB 安装包 | | | -| 测试工具 | pgbench | | | - -## 高可用、可扩展性测试 - -| 序号 | 测试项 | 描述 | -| ---- | -------------------- | ------------------------------------------------------------ | -| 1 | 读写分离集群实现 | 支持按照读写进行路由，并实现读、写的任务分发 | -| 2 | 扩容/缩容 | 在负载场景下，添加/减少物理设备时，能够保持前端应用不间断 | -| 3 | 共享存储集群 | 支持2节点共享存储集群，支持故障自动切换，支持并发事务的负载均衡 | -| 4 | 服务异常测试 | 支持在从节点数据库进程异常 、从库监控脚本异常和主库文件系统异常场景下最大程度的保证应用的可用性 | -| 5 | 日常维护测试 | 支持gs_ctl(数据库服务控制工具)和MogHA(高可用组件)进行Switchover和Failover的切换操作 | -| 6 | 数据库服务器异常测试 | 能够提供在主节点异常宕机、备节点异常宕机、仲裁节点异常宕机、主节点和备节点异常宕机和仲裁+主节点异常宕机的场景下应用的可用性和稳定性 | -| 7 | 网络异常测试 | 能够提供在主节点服务网卡异常、主节点心跳网卡异常、主节点服务+心跳网卡异常、从节点服务网卡异常、主机VIP异常的场景下最大程度的保证应用的可用性 | diff --git a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/1-performance-test-overview.md b/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/1-performance-test-overview.md deleted file mode 100644 index b5dd717d6e69271377d1f6b06e198035a449defd..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/1-performance-test-overview.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: 性能测试简述 -summary: 性能测试简述 -author: Liu Xu -date: 2021-03-04 ---- - -# 性能测试简述 - -
- -## 测试相关指标 - -### 数据量 - -每个仓库约70MB数据量，为了让数据量更加符合实际使用场景，初始化创建1000个仓库，添加相关索引后，数据库大小在100G左右。 - -
- -### 并发数 - -模拟真实的用户行为，同时在线请求操作的客户端数量。 - -
- -### 系统平均负载 - -系统负载指系统CPU繁忙程度的度量，即有多少进程在等待被CPU调度，反应整个系统的压力，而系统平均负载被定义为在特定时间间隔内运行队列中的平均进程树，这个一段时间一般取1分钟、5分钟、10分钟。 - -
- -### CPU使用率 - -CPU使用率其实是指运行的程序占用的CPU资源，表示你的机器在某个时间点的运行程序的情况。使用率越高，说明你的机器在这个时间上运行了很多程序，反之较少。使用率的高低与CPU强弱有直接关系，通过CPU的使用率来判断CPU是否达到瓶颈。 - -
- -### IOPS - -IOPS（Input/OutputOperatIOns Per Second）是一个用于计算机存储设备（如硬盘（HDD）、固态硬盘（SSD）或存储区域网络（SAN））性能测试的量测方式，可以视为是每秒的读写次数，是衡量磁盘性能的主要指标之一。IOPS是指单位时间内系统能处理的I/O请求数量，一般以每秒处理的I/O请求数量为单位，I/O请求通常为读或写数据操作请求。 - -
- -### IO延时 - -IO响应时间也被称为IO延时(IOLatency)，IO响应时间就是从操作系统内核发出的一个读或者写的IO命令到操作系统内核接收到IO回应的时间，注意不要和单个IO时间混淆了，单个IO时间仅仅指的是IO操作在磁盘内部处理的时间，而IO响应时间还要包括IO操作在IO等待队列中所花费的等待时间。 - -
- -### tpmC - -TPC-C使用三种性能和价格度量，其中性能由TPC-C吞吐率衡量，单位是tpmC。tpm是transactions per minute的简称；C指TPC中的C基准程序。它的定义是每分钟内系统处理的新订单个数。 - -
- -## 准备测试程序 - -### 测试工具介绍 - -BenchmarkSQL是一款经典的开源数据库测试工具，内嵌了TPC-C测试脚本，可以对PostgreSQL、MySQL、Oracle以及SQL Server等数据库直接进行测试。它通过JDBC测试OLTP（联机事务处理, Online TransactIOn Processing）的TPC-C。 - -
- -### 测试规范说明 - -TPC-C 是一个对 OLTP（联机交易处理）系统进行测试的规范，使用一个商品销售模型对 OLTP 系统进行测试，其中包含五类事务, 其事务内容及特点如下: - -- **NewOrder - 新订单的生成** - - 事务内容: 对于任意一个客户端,从固定的仓库随机选取 5-15 件商品,创建新订单.其中 1%的订单要由假想的用户操作失败而回滚。 - - 主要特点: 中量级、读写频繁、要求响应快。 - -- **Payment - 订单付款** - - 事务内容: 对于任意一个客户端,从固定的仓库随机选取一个辖区及其内用户,采用随机的金额支付一笔订单,并作相应历史纪录。 - - 主要特点: 轻量级、读写频繁、要求响应快。 - -- **OrderStatus - 最近订单查询** - - 事务内容: 对于任意一个客户端,从固定的仓库随机选取一个辖区及其内用户，读取其最后一条订单，显示订单内每件商品的状态。 - - 主要特点: 中量级、只读频率低、要求响应快。 - -- **Delivery - 配送** - - 事务内容: 对于任意一个客户端,随机选取一个发货包，更新被处理订单的用户余额，并把该订单从新订单中删除。 - - 主要特点: 1-10 个批量，读写频率低、较宽松的响应时间。 - -- **StockLevel - 库存缺货状态分析** - - 事物内容: 对于任意一个客户端，从固定的仓库和辖区随机选取最后 20 条订单，查看订单中所有的货物的库存，计算并显示所有库存低于随机生成域 值的商品数量。 - - 主要特点: 重量级，只读频率低,较宽松的响应时间。 - -
- -### 测试规则描述 - -在测试开始前，TPC-C Benchmark 规定了数据库的初始状态，也就是数据库中数据生成的规则，其中 ITEM 表中固定包含 10 万种商品，仓库的数量可进行调整，本次测试初始化WAREHOUSE 表中有 1000 条记录，那么: - -- STOCK 表中应有 1000 * 10 万条记录（每个仓库对应 10 万种商品的库存数据） -- DISTRICT 表中应有1000 * 10 条记录（每个仓库为 10 个地区提供服务） -- CUSTOMER 表中应有 1000 \* 10 \* 3000 条记录（每个地区有 3000 个客户） -- HISTORY 表中应有 1000 \* 10 \* 3000 条记录（每个客户一条交易历史） -- ORDER 表中应有 1000 \* 10 \* 3000 条记录（每个地区 3000 个订单），并且最后生成的 900 个订单被添加到 NEW-ORDER 表中，每个订单随机生成 5 ~ 15 条 ORDER-LINE 记录。 - -> TPC-C 使用 tpmC 值（Transactions per Minute）来衡量系统最大有效吞吐量 (MQTh, Max Qualified Throughput)，其中 Transactions 以 NewOrder Transaction 为准，即最终衡量单位为每分钟处理的新订单数。 diff --git a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md b/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md deleted file mode 100644 index 9c510ca7689aa5f6696d95e0251fbe202ae49cbc..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md +++ /dev/null @@ -1,334 +0,0 @@ ---- -title: MogDB on鲲鹏性能测试 -summary: MogDB on鲲鹏性能测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB on鲲鹏性能测试 - -## 测试目的 - -本文主要讲述了对MogDB 2.0.0 版本on鲲鹏进行的单节点和一主多备（同步备、异步备）的测试。 - -
- -## 测试环境 - -### 环境配置 - -| 类别 | 服务端配置 | 客户端配置 | 数量 | -| -------- | --------------------- | ------------- | ----- | -| CPU | Kunpeng 920 | Kunpeng 920 | 128 | -| 内存 | DDR4,2933MT/s | DDR4,2933MT/s | 2048G | -| 硬盘 | Nvme 3.5T | Nvme 3T | 4个 | -| 文件系统 | Xfs | Xfs | 4 | -| OS | openEuler 20.03 (LTS) | Kylin V10 | | -| 数据库 | MogDB 1.1.0安装包 | | | - -
- -### 测试工具 - -| 工具名称 | 用途 | -| ------------- | ---------------------------------| -| Benchmarksql5.0 | BenchmarkSQL是对OLTP数据库主流测试标准TPC-C的开源JAVA实现,用于对数据库事务处理能力的评估测试。 | - -
- -## 测试步骤 - -### MogDB数据库端操作 - -1. 获取数据库安装包 - -2. 安装数据库 - -3. 创建tpcc测试用户和数据库 - - ```bash - create user [username] identified by 'passwd'; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. 停止数据库修改数据库配置文件postgresql.conf数据库，在末尾添加配置参数（单节点测试追加参数如下） - - ```bash - max_connections = 4096 - allow_concurrent_tuple_update = true - audit_enabled = off - checkpoint_segments = 1024 - cstore_buffers =16MB - enable_alarm = off - enable_codegen = false - enable_data_replicate = off - full_page_writes = off - max_files_per_process = 100000 - max_prepared_transactions = 2048 - shared_buffers = 350GB - use_workload_manager = off - wal_buffers = 1GB - work_mem = 1MB - log_min_messages = FATAL - transaction_isolation = 'read committed' - default_transaction_isolation = 'read committed' - synchronous_commit = on - fsync = on - maintenance_work_mem = 2GB - vacuum_cost_limit = 2000 - autovacuum = on - autovacuum_mode = vacuum - autovacuum_max_workers = 5 - autovacuum_naptime = 20s - autovacuum_vacuum_cost_delay =10 - xloginsert_locks = 48 - update_lockwait_timeout =20min - enable_mergejoin = off - enable_nestloop = off - enable_hashjoin = off - enable_bitmapscan = on - enable_material = off - wal_log_hints = off - log_duration = off - checkpoint_timeout = 15min - enable_save_datachanged_timestamp =FALSE - enable_thread_pool = on - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - enable_double_write = on - enable_incremental_checkpoint = on - enable_opfusion = on - advance_xlog_file_num = 10 - numa_distribute_mode = 'all' - track_activities = off - enable_instr_track_wait = off - enable_instr_rt_percentile = off - track_counts =on - track_sql_count = off - enable_instr_cpu_timer = off - plog_merge_age = 0 - session_timeout = 0 - enable_instance_metric_persistent = off - enable_logical_io_statistics = off - enable_user_metric_persistent =off - enable_xlog_prune = off - enable_resource_track = off - instr_unique_sql_count = 0 - enable_beta_opfusion = on - enable_beta_nestloop_fusion = on - autovacuum_vacuum_scale_factor = 0.02 - autovacuum_analyze_scale_factor = 0.1 - client_encoding = UTF8 - lc_messages = en_US.UTF-8 - lc_monetary = en_US.UTF-8 - lc_numeric = en_US.UTF-8 - lc_time = en_US.UTF-8 - modify_initial_password = off - ssl = off - enable_memory_limit = off - data_replicate_buffer_size = 16384 - max_wal_senders = 8 - log_line_prefix = '%m %u %d %h %p %S' - vacuum_cost_limit = 10000 - max_process_memory = 12582912 - recovery_max_workers = 1 - recovery_parallelism = 1 - explain_perf_mode = normal - remote_read_mode = non_authentication - enable_page_lsn_check = off - pagewriter_sleep = 100 - ``` - -
- -### benchmarksql端操作 - -1. 修改配置文件 - - 进入benchmarksql安装目录下，找到run目录下的配置文件[config file] - - ```text - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. 运行runDataBuild.sh生成数据 - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. 运行runBenchmark.sh测试数据库 - - ``` - ./runBenchmark.sh [config file] - ``` - -
- -### 操作系统配置 - -1. 修改操作系统内核PAGESIZE（仅EulerOS需执行此步骤） - - **安装kernel-4.19.36-1.aarch64.rpm** - - ``` - # rpm -Uvh --force --nodeps kernel-4.19.36-1.aarch64.rpm - - # *: 此文件是基于linux 4.19.36的内核包，获取路径如下: - - # 10.44.133.121 (root/Huawei12#$) - - /data14/xy_packages/kernel-4.19.36-1.aarch64.rpm - ``` - - **修改操作系统内核配置文件中root的启动项** - - ``` - # vim /boot/efi/EFI/euleros/grubenv //修改前备份grubenv文件 - - # GRUB Environment Block - - saved_entry=EulerOS (4.19.36) 2.0 (SP8) -- 此处修改为4.19.36 - ``` - -
- -### 文件系统 - -1. 修改xfs文件系统blocksize为8K - - \# 下面命令查看当前挂载的nvme盘 - - ```bash - # df -h | grep nvme - /dev/nvme0n1 3.7T 2.6T 1.2T 69% /data1 - /dev/nvme1n1 3.7T 1.9T 1.8T 51% /data2 - /dev/nvme2n1 3.7T 2.2T 1.6T 59% /data3 - /dev/nvme3n1 3.7T 1.4T 2.3T 39% /data4 - - # xfs_info命令可以查看nvme盘的信息 - - xfs_info /data1 - ``` - -2. 针对需要修改的磁盘，备份所需的数据 - -3. 重新格式化磁盘 - - ```bash - # 以/dev/nvme0n1盘，加载点为/data1为例，相关参考命令如下 - - umount /data1 - - mkfs.xfs -b size=8192 /dev/nvme0n1 -f - - mount /dev/nvme0n1 /data1 - ``` - -
- -## 测试项及结论 - -### 测试结果汇总 - -| 测试项 | 数据量 | 并发数 | CPU平均使用率 | IOPS | IO延时 | WAL数量 | tpmC | 测试时长 | -| ------ | ------ | ------ | --------- | ------ | -------- | ------- | ---------- | -------- | -| 单节点 | 100G | 500 | 77.49% | 17.96K | 819.05us | 13260 | 1567226.12 | 10分钟 | -| 一主一备 | 100G | 500 | 57.64% | 5.31K | 842.78us | 13272 | 1130307.87 | 10分钟 | -| 一主两备 | 100G | 500 | 60.77% | 5.3K | 821.66us | 14324 | 1201560.28 | 10分钟 | - -
- -### 单节点 - -- tpmC - - ![单节点tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-1.png) - -- 系统数据 - - ![单节点系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-2.png) - -
- -### 一主一备 - -- tpmC - - ![一主一备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-3.png) - -- 系统数据 - - ![一主一备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-4.png) - -
- -### 一主两备 - -- tpmC - - ![一主两备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-5.png) - -- 系统数据 - - ![一主两备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-kunpeng-6.png) diff --git a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md b/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md deleted file mode 100644 index 03e5b38dbebb8009c1615231e182996c9af05754..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md +++ /dev/null @@ -1,672 +0,0 @@ ---- -title: MogDB on x86性能测试 -summary: MogDB on x86性能测试 -author: Liu Xu -date: 2021-03-04 ---- - -# MogDB on x86性能测试 - -## 测试目的 - -本文主要讲述了对MogDB 2.0.0 版本on x86进行的单节点和一主多备（同步备、异步备）的测试。 - -
- -## 测试环境 - -### 环境配置 - -| 服务器类型 | Fit Server | NFS5280M5 | -| -------------- | ---------------------------------------- | ---------------------------------------- | -| CPU | 144cIntel® Xeon(R) Gold 6240 CPU@2.60GHz | 64cIntel® Xeon(R) Gold 5218 CPU @2.30GHz | -| 内存 | 768G | 128G | -| 硬盘 | SAS SSD | SAS SSD | -| 网卡 | 10GE | 10GE | - -
- -### 测试工具 - -| 工具名称 | 用途 | -| --------------- | ----------------------------------------------------- | -| Benchmarksql5.0 | BenchmarkSQL是对OLTP数据库主流测试标准TPC-C的开源JAVA实现，用于对数据库事务处理能力的评估测试。 | - -
- -## 测试步骤 - -### MogDB数据库端操作 - -1. 获取数据库安装包 - -2. 安装数据库 - -3. 创建tpcc测试用户和数据库 - - ```sql - create user [username] identified by ‘passwd’; - grant [origin user] to [username]; - create database [dbname]; - ``` - -4. 停止数据库修改数据库配置文件postgresql.conf数据库，在末尾添加配置参数（单节点测试追加参数如下） - - ```bash - max_connections = 4096 - - allow_concurrent_tuple_update = true - - audit_enabled = off - - checkpoint_segments = 1024 - - cstore_buffers =16MB - - enable_alarm = off - - enable_codegen = false - - enable_data_replicate = off - - full_page_writes = off - - max_files_per_process = 100000 - - max_prepared_transactions = 2048 - - shared_buffers = 350GB - - use_workload_manager = off - - wal_buffers = 1GB - - work_mem = 1MB - - log_min_messages = FATAL - - transaction_isolation = 'read committed' - - default_transaction_isolation = 'read committed' - - synchronous_commit = on - - fsync = on - - maintenance_work_mem = 2GB - - vacuum_cost_limit = 2000 - - autovacuum = on - - autovacuum_mode = vacuum - - autovacuum_max_workers = 5 - - autovacuum_naptime = 20s - - autovacuum_vacuum_cost_delay =10 - - xloginsert_locks = 48 - - update_lockwait_timeout =20min - - enable_mergejoin = off - - enable_nestloop = off - - enable_hashjoin = off - - enable_bitmapscan = on - - enable_material = off - - wal_log_hints = off - - log_duration = off - - checkpoint_timeout = 15min - - enable_save_datachanged_timestamp =FALSE - - enable_thread_pool = on - - thread_pool_attr = '812,4,(cpubind:0-27,32-59,64-91,96-123)' - - enable_double_write = on - - enable_incremental_checkpoint = on - - enable_opfusion = on - - advance_xlog_file_num = 10 - - numa_distribute_mode = 'all' - - track_activities = off - - enable_instr_track_wait = off - - enable_instr_rt_percentile = off - - track_counts =on - - track_sql_count = off - - enable_instr_cpu_timer = off - - plog_merge_age = 0 - - session_timeout = 0 - - enable_instance_metric_persistent = off - - enable_logical_io_statistics = off - - enable_user_metric_persistent =off - - enable_xlog_prune = off - - enable_resource_track = off - - instr_unique_sql_count = 0 - - enable_beta_opfusion = on - - enable_beta_nestloop_fusion = on - - autovacuum_vacuum_scale_factor = 0.02 - - autovacuum_analyze_scale_factor = 0.1 - - client_encoding = UTF8 - - lc_messages = en_US.UTF-8 - - lc_monetary = en_US.UTF-8 - - lc_numeric = en_US.UTF-8 - - lc_time = en_US.UTF-8 - - modify_initial_password = off - - ssl = off - - enable_memory_limit = off - - data_replicate_buffer_size = 16384 - - max_wal_senders = 8 - - log_line_prefix = '%m %u %d %h %p %S' - - vacuum_cost_limit = 10000 - - max_process_memory = 12582912 - - recovery_max_workers = 1 - - recovery_parallelism = 1 - - explain_perf_mode = normal - - remote_read_mode = non_authentication - - enable_page_lsn_check = off - - pagewriter_sleep = 100对方的 - ``` - -
- -### benchmarksql端操作 - -1. 修改配置文件 - - 进入benchmarksql安装目录下，找到run目录下的配置文件[config file] - - ```bash - db=postgres - - driver=org.postgresql.Driver - - conn=jdbc:postgresql://[ip:port]/tpcc?prepareThreshold=1&batchMode=on&fetchsize=10 - - user=[user] - - password=[passwd] - - warehouses=1000 - - loadWorkers=80 - - terminals=812 - - //To run specified transactions per terminal- runMins must equal zero - - runTxnsPerTerminal=0 - - //To run for specified minutes- runTxnsPerTerminal must equal zero - - runMins=30 - - //Number of total transactions per minute - - limitTxnsPerMin=0 - - //Set to true to run in 4.x compatible mode. Set to false to use the - - //entire configured database evenly. - - terminalWarehouseFixed=false #true - - //The following five values must add up to 100 - - //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec - - newOrderWeight=45 - - paymentWeight=43 - - orderStatusWeight=4 - - deliveryWeight=4 - - stockLevelWeight=4 - - // Directory name to create for collecting detailed result data. - - // Comment this out to suppress. - - //resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS - - //osCollectorScript=./misc/os_collector_linux.py - - //osCollectorInterval=1 - - //osCollectorSSHAddr=tpcc@127.0.0.1 - - //osCollectorDevices=net_eth0 blk_sda blk_sdg blk_sdh blk_sdi blk_sdj - ``` - -2. 运行runDataBuild.sh生成数据 - - ``` - ./runDatabaseBuild.sh [config file] - ``` - -3. 运行runBenchmark.sh测试数据库 - - ``` - ./runBenchmark.sh [config file] - ``` - -
- -### 操作系统参数 - -```bash -vm.dirty_background_ratIO=5 - -vm.dirty_ratIO=10 - -kernel.sysrq=0 - -net.ipv4.ip_forward=0 - -net.ipv4.conf.all.send_redirects=0 - -net.ipv4.conf.default.send_redirects=0 - -net.ipv4.conf.all.accept_source_route=0 - -net.ipv4.conf.default.accept_source_route=0 - -net.ipv4.conf.all.accept_redirects=0 - -net.ipv4.conf.default.accept_redirects=0 - -net.ipv4.conf.all.secure_redirects=0 - -net.ipv4.conf.default.secure_redirects=0 - -net.ipv4.icmp_echo_ignore_broadcasts=1 - -net.ipv4.icmp_ignore_bogus_error_responses=1 - -net.ipv4.conf.all.rp_filter=1 - -net.ipv4.conf.default.rp_filter=1 - -net.ipv4.tcp_syncookies=1 - -kernel.dmesg_restrict=1 - -net.ipv6.conf.all.accept_redirects=0 - -net.ipv6.conf.default.accept_redirects=0 - -net.core.rmem_max = 21299200 - -net.core.rmem_default = 21299200 - -net.core.somaxconn = 65535 - -net.ipv4.tcp_tw_reuse = 1 - -net.sctp.sctp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -kernel.sem = 250 6400000 1000 25600 - -net.core.wmem_default = 21299200 - -kernel.shmall = 1152921504606846720 - -net.core.wmem_max = 21299200 - -net.sctp.sctp_rmem = 8192 250000 16777216 - -net.core.netdev_max_backlog = 65535 - -kernel.shmmax = 18446744073709551615 - -net.sctp.sctp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.tcp_max_syn_backlog = 65535 - -vm.oom_panic_on_oom=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -net.ipv4.tcp_timestamps = 1 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.sctp.path_max_retrans = 10 - -net.sctp.max_init_retransmits = 10 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_synack_retries = 5 - -kernel.core_uses_pid=1 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%p - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.nmi_watchdog=0 - -kernel.core_pattern=/home/core/core-%e-%u-%s-%t-%h - -net.core.netdev_max_backlog = 65535 - -net.core.rmem_default = 21299200 - -net.core.rmem_max = 21299200 - -net.core.somaxconn = 65535 - -net.core.wmem_default = 21299200 - -net.core.wmem_max = 21299200 - -net.ipv4.conf.all.accept_redirects = 0 - -net.ipv4.conf.all.rp_filter = 1 - -net.ipv4.conf.all.secure_redirects = 0 - -net.ipv4.conf.all.send_redirects = 0 - -net.ipv4.conf.default.accept_redirects = 0 - -net.ipv4.conf.default.accept_source_route = 0 - -net.ipv4.conf.default.rp_filter = 1 - -net.ipv4.conf.default.secure_redirects = 0 - -net.ipv4.conf.default.send_redirects = 0 - -net.ipv4.conf.enp135s0.accept_redirects = 0 - -net.ipv4.conf.enp135s0.accept_source_route = 0 - -net.ipv4.conf.enp135s0.forwarding = 1 - -net.ipv4.conf.enp135s0.rp_filter = 1 - -net.ipv4.conf.enp135s0.secure_redirects = 0 - -net.ipv4.conf.enp135s0.send_redirects = 0 - -net.ipv4.tcp_keepalive_intvl = 30 - -net.ipv4.tcp_keepalive_time = 30 - -net.ipv4.tcp_max_syn_backlog = 65535 - -net.ipv4.tcp_max_tw_buckets = 10000 - -net.ipv4.tcp_mem = 362715 483620 725430 - -#net.ipv4.tcp_mem = 94500000 915000000 927000000 - -net.ipv4.tcp_retries1 = 5 - -net.ipv4.tcp_rmem = 8192 250000 16777216 - -net.ipv4.tcp_syn_retries = 5 - -net.ipv4.tcp_tso_win_divisor = 30 - -net.ipv4.tcp_tw_reuse = 1 - -net.ipv4.tcp_wmem = 8192 250000 16777216 - -net.ipv4.udp_mem = 725430 967240 1450860 - -#net.ipv4.tcp_max_orphans = 3276800 - -#net.ipv4.tcp_fin_timeout = 60 - -#net.ipv4.ip_local_port_range = 26000 65535 - -net.ipv4.tcp_retries2 = 80 - -#net.ipv4.ip_local_reserved_ports = 20050-30007 - -vm.min_free_kbytes = 40140150 -``` - -
- -### 数据库参数 - -| 参数项 | MogDB | -| --------------------------------- | -------------------------------- | -| listen_addresses | 具体ip | -| port | 26000 | -| max_connectIOns | 4096 | -| wal_level | hot_standby | -| archive_mode | on | -| archive_command | /bin/ture | -| max_wal_senders | 16 | -| wal_keep_segments | 16 | -| max_replicatIOn_slots | 8 | -| hot_standby | on | -| logging_collector | on | -| log_directory | 安装工具指定 | -| log_filename | PostgreSQL-%Y-%m-%d_%H%M%S.log | -| log_min_duratIOn_statement | 1800000 | -| log_line_prefix | %m%c%d%p%a%x%n%e | -| log_timezone | PRC | -| datestyle | iso,mdy | -| timezone | PRC | -| default_text_search_config | pg_catalog.english | -| applicatIOn_name | dn_6001 | -| max_prepared_transactIOns | 2048 | -| shared_buffers | 350GB | -| wal_buffers | 1GB | -| work_mem | 64MB | -| log_min_messages | FATAL | -| synchronous_commit | on | -| fsync | on | -| maintenance_work_mem | 2GB | -| autovacuum | on | -| autovacuum_max_workers | 5 | -| autovacuum_naptime | 20s | -| autovacuum_vacuum_cost_delay | 10 | -| enable_mergejoin | off | -| enable_nestloop | off | -| enable_hashjoin | off | -| enable_bitmapscan | on | -| enable_material | off | -| wal_log_hints | off | -| log_duratIOn | off | -| checkpoint_timeout | 15min | -| track_activities | off | -| track_counts | on | -| autovacuum_vacuum_scale_factor | 0.02 | -| autovacuum_analyze_scale_factor | 0.1 | -| ssl | off | -| local_bind_address | 具体IP | -| max_inner_tool_connectIOns | 10 | -| password_encryptIOn_type | 0 | -| comm_tcp_mode | on | -| comm_quota_size | 1024kB | -| max_process_memory | 700GB | -| bulk_write_ring_size | 2GB | -| checkpoint_segments | 1024 | -| incremental_checkpoint_timeout | 60s | -| archive_dest | /log/archive | -| enable_slot_log | off | -| data_replicate_buffer_size | 128MB | -| walsender_max_send_size | 8MB | -| enable_kill_query | off | -| connectIOn_alARM_rate | 0.9 | -| alARM_report_interval | 10 | -| alARM_component | /opt/huawei/snas/bin/snas_cm_cmd | -| lockwait_timeout | 1200s | -| pgxc_node_name | xxx | -| audit_directory | 安装工具指定 | -| explain_perf_mode | pretty | -| job_queue_processes | 10 | -| default_storage_nodegroup | installatIOn | -| expected_computing_nodegroup | query | -| replicatIOn_type | 1 | -| recovery_max_workers | 4 | -| available_zone | AZ1 | -| allow_concurrent_tuple_update | TRUE | -| audit_enabled | off | -| cstore_buffers | 16MB | -| enable_alARM | off | -| enable_codegen | FALSE | -| enable_data_replicate | off | -| max_file_per_process | 10000 | -| use_workload_manager | off | -| xloginsert_locks | 48 | -| update_lockwait_timeout | 20min | -| enable_save_datachanged_timestamp | FALSE | -| enable_thread_pool | off | -| enable_double_write | on | -| enable_incremental_checkpoint | on | -| advance_xlog_file_num | 10 | -| enable_instr_track_wait | off | -| enable_instr_rt_percentile | off | -| track_sql_count | off | -| enable_instr_cpu_timer | off | -| plog_merge_age | 0 | -| sessIOn_timeout | 0 | -| enable_instance_metric_persistent | off | -| enable_logical_IO_statistics | off | -| enable_user_metric_persistent | off | -| enable_xlog_prune | off | -| enable_resource_track | off | -| instr_unique_sql_count | 0 | -| enable_beta_opfusIOn | on | -| enable_bete_netsloop_fusIOn | on | -| remote_read_mode | non_authenticatIOn | -| enable_page_lsn_check | off | -| pagewriter_sleep | 2s | -| enable_opfusIOn | on | -| max_redo_log_size | 100GB | -| pagewrite_thread_num | 1 | -| bgwrite_thread_num | 1 | -| dirty_page_percent_max | 1 | -| candidate_buf_percent_target | 01 | - -
- -## 测试项及结论 - -### 测试结果汇总 - -| 测试项 | 数据量 | 并发数 | CPU平均使用率 | IOPS | IO延时 | WAL数量 | tpmC | 测试时长 | -| -------- | ------ | ------ | ---------- | ----- | ------- | ------- | -------- | -------- | -| 单节点 | 100G | 500 | 29.39% | 6.50K | 1.94.ms | 3974 | 520896.3 | 10分钟 | -| 一主一备 | 100G | 500 | 30.4% | 5.31K | 453.2us | 3944 | 519993.5 | 10分钟 | -| 一主两备 | 100G | 500 | 26.35% | 7.66K | 531.9us | 3535 | 480842.2 | 10分钟 | - -
- -### 单节点 - -- tpmC - - ![单节点tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-1.png) - -- 系统数据 - - ![单节点系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-2.png) - -
- -### 一主一备 - -- tpmC - - ![一主一备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-3.png) - -- 系统数据 - - ![一主一备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-4.png) - -
- -### 一主两备 - -- tpmC - - ![一主两备tpmC](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-5.png) - -- 系统数据 - - ![一主两备系统数据](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/mogdb-on-x86-6.png) diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/1-AI-features-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/1-AI-features-overview.md index 1456527e99996f3bf70639c5be5ad57c13543188..fbcc07073adca59b604e9318ae2fd2d218f4bdb0 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/1-AI-features-overview.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/1-AI-features-overview.md @@ -1,13 +1,18 @@ --- -title: 概述 -summary: 概述 +title: AI特性 +summary: AI特性 author: Guo Huan date: 2021-05-19 --- -# 概述 +# AI特性 -人工智能技术最早可以追溯到上世纪50年代，甚至比数据库系统的发展历史还要悠久。但是，由于各种各样客观因素的制约，在很长的一段时间内，人工智能技术并没有得到大规模的应用，甚至还经历了几次明显的低谷期。到了近些年，随着信息技术的进一步发展，从前限制人工智能发展的因素已经逐渐减弱，所谓的ABC（AI、Big data、Cloud Computing）技术也随之而诞生。AI与数据库结合是近些年的行业研究热点，我们的数据库团队较早地参与了该领域的探索，并取得了阶段性的成果。AI特性子模块相对数据库其他组件更为独立，名为dbmind，大致可分为AI4DB与DB4AI两个部分。 +人工智能技术最早可以追溯到上世纪50年代，甚至比数据库系统的发展历史还要悠久。但是，由于各种各样客观因素的制约，在很长的一段时间内，人工智能技术并没有得到大规模的应用，甚至还经历了几次明显的低谷期。到了近些年，随着信息技术的进一步发展，从前限制人工智能发展的因素已经逐渐减弱，所谓的ABC（AI、Big data、Cloud computing）技术也随之而诞生。 -- AI4DB就是指用人工智能技术优化数据库的性能，从而获得更好的执行表现；也可以通过人工智能的手段实现自治、免人工运维等。主要包括自调优、自诊断、自安全、自运维和自愈等子领域； -- DB4AI就是指打通数据库到人工智能应用的端到端流程，统一人工智能技术栈，达到开箱即用、高性能、节约成本等目的。例如通过SQL-like语句使用推荐系统、图像检索、时序预测等功能，充分发挥高斯数据库高并行、列存储等优势。 +AI与数据库结合是近些年的行业研究热点，MogDB较早地参与了该领域的探索，并取得了阶段性的成果。AI特性子模块名为DBMind，相对数据库其他功能更为独立，大致可分为AI4DB、DB4AI以及AI in DB三个部分。 + +- AI4DB就是指用人工智能技术优化数据库的性能，从而获得更好地执行表现；也可以通过人工智能的手段实现自治、免运维等。主要包括自调优、自诊断、自安全、自运维、自愈等子领域； +- DB4AI就是指打通数据库到人工智能应用的端到端流程，通过数据库来驱动AI任务，统一人工智能技术栈，达到开箱即用、高性能、节约成本等目的。例如通过SQL-like语句实现推荐系统、图像检索、时序预测等功能，充分发挥数据库的高并行、列存储等优势，既可以避免数据和碎片化存储的代价，又可以避免因信息泄漏造成的安全风险； +- AI in DB 就是对数据库内核进行修改，实现原有数据库架构模式下无法实现的功能，如利用AI算法改进数据库的优化器，实现更精确的代价估计等。 + +本章节所涉及的功能独立存在于数据库安装目录(`$GAUSSHOME`)的`bin/dbmind`目录中，各个子功能存在于dbmind的子目录components中。提供gs_dbmind命令行供用户调用。与此同时，对于数据库内置AI的功能（如DB4AI），以SQL语法和系统函数的形式呈现。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-1-ai-query-time-forecasting-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-1-ai-query-time-forecasting-overview.md deleted file mode 100644 index a87a0e60aa54f25f423bbbd0eeb289de1c775248..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-1-ai-query-time-forecasting-overview.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: 概述 -summary: 概述 -author: Guo Huan -date: 2021-05-19 ---- - -# 概述 - -Predictor是基于机器学习且具有在线学习能力的查询时间预测工具。通过不断学习数据库内收集的历史执行信息，实现计划的执行时间预测功能。 - -本特性需要拉起python进程AIEngine，用于模型的训练和推理。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-1-x-tuner-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-1-x-tuner-overview.md deleted file mode 100644 index c7945cc087e1ec172b4befef16c404b52c48d220..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-1-x-tuner-overview.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: 概述 -summary: 概述 -author: Guo Huan -date: 2021-05-19 ---- - -# 概述 - -X-Tuner 是一款数据库集成的参数调优工具，通过结合深度强化学习和全局搜索算法等AI技术，实现在无需人工干预的情况下，获取最佳数据库参数配置。本功能不强制与数据库环境部署到一起，支持独立部署，脱离数据库安装环境独立运行。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-4-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-4-command-reference.md deleted file mode 100644 index 75a86c223f4c3653c68a596edfe63cde8f685899..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-4-command-reference.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: 命令参考 -summary: 命令参考 -author: Guo Huan -date: 2021-05-19 ---- - -# 命令参考 - -**表 1** 命令行参数说明 - -| 参数 | 参数说明 | 取值范围 | -| :-------------- | :----------------- | :------------ | -| -f | 训练或预测文件位置 | | -| -predicted-file | 预测结果存储位置 | | -| -model | 模型选择 | template，dnn | -| -model-path | 训练模型存储位置 | | diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-1-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-1-overview.md deleted file mode 100644 index bae52cc9b87c214de206dcbf93dbd8598df0c987..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-1-overview.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: 概述 -summary: 概述 -author: Guo Huan -date: 2021-05-19 ---- - -# 概述 - -anomaly_detection是MogDB集成的、可以用于数据库指标采集、预测指标趋势变化、慢SQL根因分析以及异常监控与诊断的AI工具，是DBMind套间中的一个组件。支持采集的信息分为三块，分别是os_exporter、database_exporter、wdr，os_exporter主要包括IO_Read、IO_Write、IO_Wait、CPU_Usage、Memory_Usage、数据库数据目录磁盘空间占用Disk_space；database_exporter主要包括QPS、部分关键GUC参数（work_mem、shared_buffers、max_connections）、数据库临时文件、外部进程情况、外部连接数；wdr包括慢SQL文本、SQL开始执行时间、SQL结束执行时间相关信息。在异常监控方面，anomaly_detection可以同时对IO_Read、IO_Write、IO_Wait、CPU_Usage、Memory_Usage和Disk_Space多个指标的未来变化趋势进行预测，当发现某个指标在未来某段时间或者某个时刻会超出人工设置的阈值，该工具会通过日志进行报警。在慢SQL根因分析方面，工具会定期从WDR报告中拉取慢SQL信息，并对慢SQL的根因进行诊断，最后将诊断结果存放到日志文件中，同时该工具还支持用户交互式慢SQL诊断，即对用户输入的慢SQL进行根因分析，并将结果反馈给用户。 - -anomaly_detection由agent和detector两大模块组成。agent和MogDB数据库环境部署在同一个服务器上，该模块主要有两个作用。一个是定时采集数据库指标数据，并将采集到的数据存放到缓冲队列中；另一个作用是将缓冲队列中数据定时发送到detector的collector子模块中。 - -detector模块由collector模块和monitor模块组成，collector模块和agent模块通过http或https进行通信，接受agent模块push的数据并存储到本地。monitor模块基于本地数据对指标的未来变化趋势进行预测和异常报警，另外结合系统和WDR报告等各种相关联信息，分析慢SQL的根因。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-2-preparations.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-2-preparations.md deleted file mode 100644 index d2f96e452a7d02d3d43878b033ba67606bdfac0f..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-2-preparations.md +++ /dev/null @@ -1,173 +0,0 @@ ---- -title: 使用准备 -summary: 使用准备 -author: Guo Huan -date: 2021-05-19 ---- - -# 使用准备 - -## 前提条件与使用事项 - -- 数据库状态正常。 -- 工具运行过程中，如果系统时间被篡改，可能会造成慢SQL数据采集失效。 -- 工具不支持在备节点上采集。 -- 使用登录到数据库宿主机上的Linux用户，需要将$**GAUSSHOME/bin**添加到PATH环境变量中，即能够直接运行gsql、gs_guc、gs_ctl等数据库运维工具。 -- Python版本建议为Python3.6及以上，且运行环境中已经安装相应依赖，并能够正常启动调优程序。 -- 本工具由Agent和Detector组成，Agent和Detector之间通过'http'或者'https'方式传递数据，因此需要保证Agent服务器和Detector服务器之间能够正常通信。 -- Detector模块运行collector和monitor服务， 需要分别启动。 -- 如果使用'https'方式进行通信，需要准备CA证书以及Agent和Detector的证书和密钥，并分别放入项目根目录certificate下的ca、agent、collector中，同时将密钥加密密码放入certificate的pwf中，并将其权限设置为600，防止其他用户进行读写操作。用户也可以使用share中的脚本生成证书和密钥。 -- 建议用户配置自己的python环境，以免影响其他功能使用（例如使用miniconda）。 -- 慢SQL的根因分析功能需要WDR报告，用户需要开启track_stmt_stat_level='OFF,L1', log_min_duration_statement=3000（慢SQL阈值，用户可按需设定），单位ms。 -- 如果将detecor和数据库部署在同一服务器上，则collector服务端口不能和数据库本地端口相同，否则将无法启动进程。 - -## 原理简介 - -图1 anomaly_detection结构图 - -![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/5-2-preparations-1.png) - -anomaly_detection是一个独立于数据库内核之外的工具，其结构如图1所示，anomaly_detection主要由agent和detector模块组成； - -- agent：数据代理模块，由Source、Channel以及Sink组成。部署在数据库环境上，用于采集数据库中的指标，并通过http或者https方式将其传送给远端检测器模块。 -- detector：检测器模块，负责收集Agent推送的数据并存储，同时该模块基于时序预测和异常检测等算法对数据库指标进行监控和异常检测，同时提供慢SQL根因服务功能。 - -## anomaly_detection的运行和安装方法 - -1. 切换到anomaly_detection目录下。对于已经安装的数据库系统，该源代码路径为 **$GAUSSHOME**/bin/dbmind/anomaly_detection。 - -2. 在当前目录下可以看到 requirements.txt 等文件，通过pip包管理工具根据该 requirements.txt 文件安装依赖： - - ```bash - pip install -r requirements.txt - ``` - -3. 安装成功后可执行main.py。以获取帮助信息为例，则可以执行： - - ```bash - python main.py --help # 可以直接通过该命令执行获取帮助的动作，其他功能使用方法类似 - ``` - -## 证书生成 - -当使用https方式进行通信时，需要用户提供证书，anomaly_detection也提供了证书生成工具。 - -1. 生成CA根证书，在anomaly_detection的share目录下，执行以下命令： - - ```bash - sh gen_ca_certificate.sh - ``` - -该脚本会在anomaly_detection根目录下创建certificate目录，其中包括ca、server、agent三个子目录，ca中存放根证书ca.crt和密钥文件ca.key。 - -1. 生成server端证书和密钥文件，在anomaly_detection的share目录下，执行以下命令： - - ```bash - sh gen_certificate.sh - - # please input the basename of ssl certificate: ../certificate/server - - # please input the filename of ssl certificate: server - - # please input the local host: 127.0.0.1 - - # please input the password of ca and ssl separated by space: - ``` - -该脚本需要用户分别输入生成证书与密钥文件存放目录、证书与密钥文件名称、detector端服务器IP地址、ca证书密码和当前证书密码(用空格分开)。脚本最后会在certificate的server下生成server.crt和server.key。 - -1. 生成agent端证书密钥和文件，在anomaly_detection的share目录下，执行以下命令： - - ```bash - sh gen_certificate.sh - - # please input the basename of ssl certificate: ../certificate/agent - - # please input the filename of ssl certificate: agent - - # please input the local host: 127.0.0.1 - - # please input the password of ca and ssl separated by space: - ``` - -该脚本需要用户分别输入生成证书与密钥文件存放目录、证书与密钥文件名称、agent端服务器IP地址、ca证书密码和当前证书密码(用空格分开)。脚本最后会在certificate的agent下生成agent.crt和agent.key。 - -## anomaly_detection的配置文件说明 - -anomaly_detection 在运行前需要加载 **a-detection.conf** 和 `metric_task.conf` 两个配置文件，可以通过 **python main.py -help** 命令查看配置文件路径： - -**a-detection.conf**：该配置文件包含agent、server、database、security、forecast、log六个section，参数解释如下： - -```bash -[database] -storage_duration = 12H # 数据存储时间长度，默认12小时 -database_dir = ./data # 数据存储目录 - -[security] -tls = False -ca = ./certificate/ca/ca.crt -server_cert = ./certificate/server/server.crt -server_key = ./certificate/server/server.key -agent_cert = ./certificate/agent/agent.crt -agent_key = ./certificate/agent/agent.key - -[server] -host = 0.0.0.0 # 服务端IP地址 -listen_host = 0.0.0.0 -listen_port = 8080 -white_host = 0.0.0.0 # IP白名单 -white_port = 8000 # 端口号白名单 - -[agent] -source_timer_interval = 10S # agent端数据采集频率 -sink_timer_interval = 10S # agent端数据发送频率 -channel_capacity = 1000 # 缓冲队列最大长度 -db_host = 0.0.0.0 # agent节点IP地址 -db_port = 8080 # agent节点端口号 -db_type = single # agent节点类型, single: 单机，cn： CN节点、dn： DN节点 - -[forecast] -forecast_alg = auto_arima # 时序预测算法，auto_arima、fbprophet(需要用户自行安装) -[log] -log_dir = ./log # 日志文件位置 -``` - -metric_task.conf: 该配置文件包括detector method, os_expoeterhe trend_parameter三个sections，参数结束如下： - -```bash -[detector_method] -trend = os_exporter # 用于时序预测的表名 -slow_sql = wdr # 用户慢SQL诊断的表名 - -[os_exporter] -cpu_usage_minimum = 1 # cpu_usage的下限值 -cpu_usage_maximum = 10 # cpu_usage的上限值 -memory_usage_minimum = 1 # memory_usage的下限值 -memory_usage_maximum = 10 # memory_usage的上限值 -io_read_minimum = 1 -io_read_maximum = 10 -io_write_minimum = 1 -io_write_maximum = 10 -io_wait_minimum = 1 -io_wait_maximum = 10 -disk_space_minimum = 1 -disk_space_maximum = 10 - -[common_parameter] -data_period = 1000S # 是用于时序预测的历史数据长度，支持整数加时间单位（如：100S、2M、10D）。 -interval = 20S # 监控间隔 -freq = 3S # 趋势预测频率 -period = 2 # 趋势预测长度 -``` - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** -> -> - 支持的时间单位： -> - 'S'：second，秒。 -> - 'M'：minute，分。 -> - 'H'：hour，小时。 -> - 'D'：day，天。 -> - 'W'：week，周。 -> - minimum和maximum至少提供一个，不能都不提供。 -> - freq和period共同决定时序预测结果，例子：freq=2S，period=5, 则会预测未来2S、4S、6S、8S、10S的值。 -> - 建议参数配置的时候，保证训练数据长度大于预测长度，否则预测效果会受影响。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-3-adding-monitoring-parameters.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-3-adding-monitoring-parameters.md deleted file mode 100644 index 97b6416cbbd630f7105eaf1f69e2756f89f8d076..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-3-adding-monitoring-parameters.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: 添加监控参数 -summary: 添加监控参数 -author: Guo Huan -date: 2021-05-19 ---- - -# 添加监控参数 - -
- -工具只针对os_exporter中的指标进行趋势预测与阈值异常检测，支持用户添加新的监控参数，步骤如下： - -1. 在task/os_exporter.py的OS_exporter中编写获取指标的功能函数，并将该函数加入到output的result列表中，例如： - - ``` - @staticmethod - def new_metric(): - return metric_value - - def output(self): - result = [self.cpu_usage(), self.io_wait(), self.io_read(), - self.io_write(), self.memory_usage(), self.disk_space(), self.new_metric()] - return result - - ``` - -2. 在table.json的os_exporter中，将new_metric字段加入到“create table”中，并在“insert”中加上字段类型信息，例如： - - ``` - "os_exporter": { - "create_table": "create table os_exporter(timestamp bigint, cpu_usage text, io_wait text, io_read text, io_write text, memory_usage text, disk_space text, new_metric text);", - "insert": "insert into os_exporter values(%d, \"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\");", - ``` - -3. 在task/metric_task.conf中添加指标的上限值或者下限值，例如： - - ``` - [os_exporter] - new_metric_minimum = 0 - new_metric_maximum = 10 - ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-4-obtaining-help-information.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-4-obtaining-help-information.md deleted file mode 100644 index c74f8287ff11b655a20862d927078686933bd4d0..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-4-obtaining-help-information.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: 获取帮助 -summary: 获取帮助 -author: Guo Huan -date: 2021-05-19 ---- - -# 获取帮助 - -启动调优程序之前，可以通过如下命令获取帮助信息： - -```python -源码方式：python main.py --help -``` - -输出帮助信息结果如下： - -```python -usage: - python main.py start [--role {{agent,collector,monitor}}] # start local service. - python main.py stop [--role {{agent,collector,monitor}}] # stop local service. - python main.py start [--user USER] [--host HOST] [--project-path PROJECT_PATH] [--role {{agent,collector,monitor}}] - # start the remote service. - python main.py stop [--user USER] [--host HOST] [--project-path PROJECT_PATH] [--role {{agent,collector, - monitor}}] # stop the remote service. - python main.py deploy [--user USER] [--host HOST] [--project-path PROJECT_PATH] # deploy project in remote host. - python main.py diagnosis [--query] [--start_time] [--finish_time] # rca for slow SQL. - python main.py show_metrics # display all monitored metrics(can only be executed on 'detector' machine). - python main.py forecast [--metric-name METRIC_NAME] [--period] [--freq] - [--forecast-method {{auto_arima, fbprophet}}] [--save-path SAVE_PATH] # forecast future trend of - metric(can only be executed on 'detector' machine). - -Anomaly-detection: a time series forecast and anomaly detection tool. - -positional arguments: - {start,stop,deploy,show_metrics,forecast,diagnosis} - -optional arguments: - -h, --help show this help message and exit - --user USER User of remote server. - --host HOST IP of remote server. - --project-path PROJECT_PATH - Project location in remote server. - --role {agent,collector,monitor} - Run as 'agent', 'collector', 'monitor'. Notes: ensure - the normal operation of the MogDB in agent. - --metric-name METRIC_NAME - Metric name to be predicted, you must provide an specified metric name. -. - --query QUERY target sql for RCA. - Currently, the join operator is not supported, and the accuracy of the result - is not guaranteed for SQL syntax containing "not null and". - --start_time START_TIME - start time of query - --finish_time FINISH_TIME - finish time of query - --period PERIOD Forecast periods of metric, it should be integernotes: - the specific value should be determined to the - trainnig data.if this parameter is not provided, the - default value '100S' will be used. - --freq FREQ forecast gap, time unit: S: Second, M: Minute, H: - Hour, D: Day, W: Week. - --forecast-method FORECAST_METHOD - Forecast method, default method is 'auto_arima',if - want to use 'fbprophet', you should install fbprophet - first. - --save-path SAVE_PATH - Save the results to this path using csv format, if - this parameter is not provided,, the result wil not be - saved. - -v, --version show program's version number and exit - -epilog: - the 'a-detection.conf' and 'metric_task.conf' will be read when the program is running, - the location of them is: - a-detection.conf: /openGauss-server/src/gausskernel/dbmind/tools/anomaly_detection/a-detection.conf. - metric_config: /openGauss-server/src/gausskernel/dbmind/tools/anomaly_detection/task/metric_task.conf. -``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-5-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-5-command-reference.md deleted file mode 100644 index 892e72b9ec0f661788add25aaabf1f4f316714e8..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-5-command-reference.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: 命令参考 -summary: 命令参考 -author: Guo Huan -date: 2021-05-19 ---- - -# 命令参考 - -**表 1** 命令行参数 - -| 参数 | 参数说明 | 取值范围 | -| :---------------- | :---------------------------------- | :----------------------------------------------------------- | -| mode | 指定运行模式 | start, stop, forecast, show_metrics, deploy, diagnosis | -| -user | 远程服务器用户 | - | -| -host | 远程服务器IP | - | -| -project-path | 远程服务器anomaly_detection项目路径 | - | -| -role | 启动角色选择 | agent，collector，monitor | -| -metric-name | 指标名称 | - | -| -query | 根因分析RCA目标query | | -| -start_time | query执行的开始时间 | | -| -finish_time | query执行的结束时间 | | -| -forecast-periods | 未来预测周期 | 整数+时间单位，如'100S'，'3H'
时间单位包括（S(second),M(minute),H(hour),D(day),W(week)） | -| -forecast-method | 预测方法 | auto_arima， fbprophet | -| -save-path | 预测结果存放地址 | - | -| -version, -v | 返回当前工具版本号 | - | diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-6-examples.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-6-examples.md deleted file mode 100644 index ebabc411f5a83bbcd31c60c5a4fca637ca9b7417..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-6-examples.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -title: 使用示例 -summary: 使用示例 -author: Guo Huan -date: 2021-05-19 ---- - -# 使用示例 - -为了方便用户理解部署过程，假设当前数据库节点信息如下： - -``` -IP: 10.90.110.130 -PORT: 8000 -type: single -``` - -detector服务器信息： - -``` -IP: 10.90.110.131 -listen_host = 0.0.0.0 -listen_port = 8080 -``` - -部署的启动流程如下： - -## 配置文件修改 - -首先需要更改配置文件 a-detection.conf，主要涉及其中的两个session： - -``` -[database] -storage_duration = 12H # 数据存储时间长度，默认12小时 -database_dir = ./data # 数据存储目录 - -[security] -tls = False -ca = ./certificate/ca/ca.crt -server_cert = ./certificate/server/server.crt -server_key = ./certificate/server/server.key -agent_cert = ./certificate/agent/agent.crt -agent_key = ./certificate/agent/agent.key - -[server] -host = 10.90.110.131 -listen_host = 0.0.0.0 -listen_port = 8080 -white_host = 10.90.110.130 -white_port = 8000 -[agent] -source_timer_interval = 10S -sink_timer_interval = 10S -channel_capacity = 1000 -db_host = 10.90.110.130 -db_port = 8080 -db_type = single - -[forecast] -forecast_alg = auto_arima - -[log] -log_dir = ./log -``` - -## 服务启动与停止 - -启动本地agent服务： - -``` -python main.py start --role agent -``` - -停止本地agent服务： - -``` -python main.py stop --role agent -``` - -启动本地collector服务： - -``` -python main.py start --role collector -``` - -停止本地collector服务： - -``` -python main.py stop --role collector -``` - -启动本地monitor服务： - -``` -python main.py start --role monitor -``` - -停止本地monitor服务： - -``` -python main.py stop --role monitor -``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-7-ai-server.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-7-ai-server.md deleted file mode 100644 index ef08fcd1be9858a06f1174f46617491b2791d858..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-7-ai-server.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: AI_SERVER -summary: AI_SERVER -author: Guo Huan -date: 2021-10-21 ---- - -# AI_SERVER - -ai_server为anomaly_detection特性的分离特性，在原anomaly_detection数据采集功能的基础上增加了采集类型、采集项、数据存储模式，仅用于数据采集，后续将整合到anomaly_detection中，该特性主要包含server组件和agent组件，agent须部署到数据库节点，用于数据采集，server部署在独立节点进行数据收集存储。 - -数据存储方式包括：sqlite、mongodb、influxdb。 - -采集项如表1： - -**表 1** 采集项说明 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
采集类型采集项描述
databasework_mem数据库内存相关GUC参数，对涉及到排序任务的sql，检测分配的空间是否足够。
shared_buffers数据库内存相关GUC参数，不合适的shared_buffer会导致数据库性能变差。
max_connections数据库最大连接数。
current connections数据库当前连接数。
qps数据库性能指标。
oscpu usagecpu使用率。
memory usage内存使用率。
io wait系统因为io导致的进程wait。
io write数据磁盘写吞吐量。
io read数据磁盘读吞吐量。
disk used磁盘已使用的大小。
- -部署方式请参见[AI_MANAGER](5-8-ai-manager)章节。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-8-ai-manager.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-8-ai-manager.md deleted file mode 100644 index 9868d9e9e140a62b427fcbc2c62ceac2c3bb9af7..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/5-a-detection-status-monitoring/5-8-ai-manager.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: AI_MANAGER -summary: AI_MANAGER -author: Guo Huan -date: 2021-10-22 ---- - -# AI_MANAGER - -ai_manager是AI特性部署工具，旨在为ai特性提供自动化、高效便捷的部署及卸载方式，可通过指定模块名称、操作类型及参数文件进行相应ai特性的自动化部署和卸载，实现了版本管理、操作日志记录及日志管理、安装信息记录等功能，支持特性级横向扩展，该工具目前仅支持ai_server的安装及卸载。 - -## 使用准备 - -- 项目部署路径为/dbs/AI-tools，需保证该路径存在并有读写执行权限，安装/卸载操作中会清理该路径下的内容，不要将其它文件保存在该路径下。 -- 需要安装python3环境及特性所需python库,依赖库请参考包内的requirements.txt文件。 -- 如开启https，需准备相应的根证书，秘钥文件及密码。 -- agent节点已启动MogDB数据库。 -- 安装agent节点须使用集群用户操作。 -- 如果agent节点集群用户的~/.bashrc文件中没有正确的PGHOST配置，需要将PGHOST配置到/dbs/AI-tools/ai_env文件中。 - -## 示例 - -安装命令示例： - -```bash -python3 ai_manager --module anomaly_detection --action install --param-file mogdb.json -``` - -卸载命令示例： - -```bash -python3 ai_manager --module anomaly_detection--action uninstall --param-file mogdb.json -``` - -参数文件示例： - -```json -{ - "scene": "mogdb", # 场景，mogdb安装server及agent,huaweiyun仅安装server - "module": "anomaly_detection", # 模块（特性）名称，目前仅支持anomaly_detection - "action": "install", # 操作类型，支持install及uninstall - "ca_info": { - "ca_cert_path": "/home/Ruby/CA_AI/ca.crt", # 根证书路径 - "ca_key_path": "/home/Ruby/CA_AI/ca.crt.key", # 根证书秘钥路径 - "ca_password": "GHJAyusa241~" # 根证书密码 - }, - "agent_nodes": [ - { - "node_ip": "10.000.00.000", # agent节点IP - "username": "Ruby", # agent节点用户 - "password": "password" # agent节点密码 - } - ], - "config_info": { - "server": { - "host": "10.000.00.000", # server部署节点IP（执行节点） - "listen_host": "0.0.0.0", # server 监听IP - "listen_port": "20060", # server 监听端口 - "pull_kafka": "False" # 是否拉取kafka数据，暂不支持拉取。 - }, - "database": { - "name": "sqlite", # 数据存储方式，支持sqlite、mongodb、influxdb - "host": "127.0.0.1", # 数据库ip - "port": "2937", # 数据库端口 - "user": "Ruby", # 数据库用户 - "size": "175000000", # mongodb 最大存储容量 - "max_rows": "1000000" # mongodb 最大存储条数 - }, - "agent": { - "cluster_name": "my_cluster", # 采集数据库的名称 - "collection_type": "os", # 采集类型，支持os、database、all - "collection_item": [["dn", "10.000.00.000", "33700"]], # agent节点采集数据类型（dn/cn），采集节点IP，端口 - "channel_capacity": "1000", # 队列容量 - "source_timer_interval": "5S", # 采集间隔 - "sink_timer_interval": "5S" # 发送间隔 - }, - "security": { - "tls": "True" # 是否开启https - } - } -} -``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-1-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-1-overview.md deleted file mode 100644 index 438893d620997fe6f918063984f3696dd7feba38..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-1-overview.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: 概述 -summary: 概述 -author: Guo Huan -date: 2021-05-19 ---- - -# 概述 - -数据库DeepSQL特性实现DB4AI功能，即在数据库内实现AI算法，以更好的支撑大数据的快速分析和计算。这里提供了一整套基于SQL的机器学习、数据挖掘以及统计学的算法，用户可以直接使用SQL进行机器学习工作。Deep SQL能够抽象出端到端从数据到模型的研发过程，配合底层的引擎及自动优化，具备基础SQL知识的技术人员即可完成大部分的机器学习模型训练及预测任务。整个分析和处理都运行在数据库引擎中，用户可以直接分析和处理数据库内的数据，不需要在数据库和其它平台之间进行数据传递，避免在多个环境之间进行不必要地数据移动。 - -DeepSQL是对MogDB DB4AI能力的增强，让对MADLib比较熟悉的数据分析师或开发者可以轻松迁移到MogDB上进行工作。DeepSQL将常用的机器学习算法封装为SQL语句，支持60多个常用算法。其中包括回归算法（例如线性回归，逻辑回归，随机森林等）、分类算法（比如KNN等）、聚类算法（比如K-means）等。除了基础的机器学习算法之外，还包括图相关的算法，比如最短路径，图形直径等等算法；此外还支持数据处理（比如PCA），稀疏向量，统计学常用算法（比如协方差，Pearson系数计算等），训练集测试集分割方法，交叉验证方法等。 - -**表 1** 支持的机器学习算法 - 回归类算法 - -| 算法中文名称 | 算法英文名称 | 应用场景 | -| :-------------- | :---------------------------------- | :----------------------------------------------------------- | -| 逻辑回归 | Logistic Regression | 例如寻找某疾病的危险因素，金融商业机构需要对企业进行评估等。
预测：根据模型预测不同的自变量情况下某病或某情况的发生概率。
判别：实际上跟预测类似，也是根据模型判断某人属于某病或属于某种情况的概率有多大，即判断某人有多大可能是属于某病。 | -| Cox比例风险回归 | Cox Proportional Hazards Regression | 该模型以生存结局和生存时间为因变量，可同时分析众多因素对生存期的影响，能分析带有截尾生存时间的资料，且不要求估计资料的生存分布类型。由于上述优良性质，该模型自问世以来，在医学类研究中得到广泛的应用，是迄今生存分析中应用最多的多因素分析方法。 | -| 弹性网络回归 | Elastic Net Regularization | 弹性回归是岭回归和套索回归的混合技术，它同时使用 L2 和 L1 正则化。当有多个相关的特征时，套索回归很可能随机选择其中一个，而弹性回归很可能都会选择。 | -| 广义线性模型 | Generalized Linear Models | 在一些实际问题中，变量间的关系并不都是线性的，这种情况就应该用曲线去进行拟合。 | -| 边际效应 | Marginal Effects | 提供边际效应的计算。 | -| 多类回归 | Multinomial Regression | 如果目标类别数超过两个，这时就需要使用多类回归，如疗效可能是“无效”，“显效”，“痊愈”三类。 | -| 序数回归 | Ordinal Regression | 在统计学中，序数回归是一种用于预测序数变量的回归分析，即其值存在于任意范围内的变量，不同值之间的度量距离也不同。它可以被认为是介于回归和分类之间的一类问题。例如，病情的分级（1、2、3、4级），症状的感觉分级（不痛、微痛、较痛和剧痛），对药物剂量反应的分级（无效、微效、中效和高效）等等。不同级别之间的差异不一定相等，如不痛与微痛的差值不一定等于较痛与剧痛的差值。 | -| 聚类方差 | Clustered Variance | Clustered Variance模块调整聚类的标准误差。例如，将一个数据集合复制100次，不应该增加参数估计的精度，但是在符合独立同分布假设（Independent Identically Distributed，IID）下执行这个过程实际上会提高精度。 | -| 稳健方差 | Robust Variance | Robust Variance模块中的函数用于计算线性回归、逻辑回归、多类逻辑回归和Cox比例风险回归的稳健方差（Huber-White估计）。它们可用于计算具有潜在噪声异常值的数据集中数据的差异。 | -| 支持向量机 | Support Vector Machines(SVM) | 用于文本和超文本的分类、图像分类，比起传统的查询优化方案，支持向量机能够获取明显更高的搜索准确度。这同样也适用于图像分割系统。 | -| 线性回归 | Linear Regression | 应用广泛，例如经济学、金融学等。 | - -**表 2** 支持的机器学习算法 - 其他监督学习 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :--------------- | :----------------------------- | :----------------------------------------------------------- | -| 决策树 | Decision Tree | 最为广泛的归纳推理算法之一，处理类别型或连续型变量的分类预测问题，可以用图形和if-then的规则表示模型，可读性较高。 | -| 随机森林 | Random Forest | 随机森林是一类专门为决策树分类器设计的组合方法。它组合多棵决策树作出的预测。 | -| 条件随机场 | Conditional Random Field (CRF) | 条件随机场（CRF）是一种判别的，无向概率的图形模型。线性链CRF是一种特殊类型的CRF，它假定当前状态仅取决于先前的状态。在分词、词性标注和命名实体识别等序列标注任务中取得了很好的效果。 | -| 朴素贝叶斯 | Naive Bayes | 通过计算概率来进行分类，可以用来处理多分类问题，比如：垃圾邮件过滤器。 | -| 神经网络 | Neural Networks | 拥有广泛的应用场景，譬如语音识别、图像识别、机器翻译等等。在模式识别的领域中算是标准监督学习算法，并在计算神经学中，持续成为被研究的课题。MLP已被证明是一种通用的函数近似方法，可以被用来拟合复杂的函数或解决分类问题。 | -| k临近算法 | k-Nearest Neighbors | K近邻分类方法通过计算每个训练样例到待分类样品的距离，取和待分类样品距离最近的K个训练样例，K个样品中哪个类别的训练样例占多数，则待分类元组就属于哪个类别。
可用于：文字识别，面部识别，基因模式识别，客户流失预测、欺诈侦测。 | - -**表 3** 支持的机器学习算法 - 数据处理类算法 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :--------------- | :----------------------------- | :----------------------------------------------------------- | -| 数组操作 | Array Operations | 数组、向量操作运算，包括基础的加减乘除、幂运算、开方、cos、sin、绝对值、方差等。 | -| 主成成分分析 | Dimensionality Reduction (PCA) | 降维，计算主成分。 | -| 变量编码 | Encoding Categorical Variables | 当前支持one-hot和dummy编码技术。
当需要用一组特定的预测变量与其它预测变量组作比较时，通常使用哑编码（dummy coding），与之比较的变量组称为参照组。One-hot编码与哑编码类似，两者的区别是前者为每种分类值建立数字类型的0/1指示列。在每行数据中（对应一个数据点），只有一个分类编码列的值可以为1。 | -| 矩阵操作 | Matrix Operations | 运用矩阵分解，将大型矩阵分解成简单矩阵的乘积形式，则可大大降低计算的难度以及计算量。矩阵加减乘除、最值、均值、求秩、求逆、矩阵分解(QR，LU，Cholesky)，特征提取。 | -| 规范化和距离函数 | Norms and Distance Functions | 求范数，余弦相似度，向量间距离。 | -| 稀疏向量 | Sparse Vectors | 实现稀疏向量类型，如果向量中重复值较多，可以用来压缩储存节省空间。 | -| 透视图 | Pivot | 透视表或枢轴表，通常用来实现OLAP或报表系统中一类常见的行列转置需求。pivot函数能够对一个表中存储的数据执行基本行转列操作，并将汇总后的结果输出到另一个表中。使行列转置操作变得更为简单与灵活。 | -| 模式匹配 | Path | 是在一系列行上执行常规模式匹配，并提取有关模式匹配的有用信息。有用的信息可以是简单的匹配计数或更多涉及的内容，如聚合或窗口函数。 | -| 会话 | Sessionize | 会话化功能对包括事件序列的数据集执行面向时间的会话重建。定义的不活动时段表示一个会话的结束和下一个会话的开始。
可以用于：网络分析，网络安全，制造，财务和运营分析。 | -| 共轭梯度法 | Conjugate gradient | 求解系数矩阵为对称正定矩阵的线性方程组的数值解的方法。 | -| 词干提取 | Stemming | 词干提取简单说就是找出单词中的词干部分，场景比如：搜索引擎建立网页主题概念。在英文网站优化作用明显，对其他语言有借鉴意义。 | -| 训练集测试集分割 | Train-Test Split | 分割数据集，把一份数据集划分成训练集和测试集，train的部分用于训练，test部分用于验证。 | -| 交叉验证 | Cross Validation | 交叉验证。 | -| 预测指标 | Prediction Metrics | 用于评估模型预测的质量，包括均方误差，AUC值、混淆矩阵、修正R方等用于评价模型的函数。 | -| 小批量预处理 | Mini-Batch Preprocessor | 把数据打包成小份进行训练，优点是它可以比随机梯度下降（默认MADlib优化器）表现更好，会更快更平滑的收敛。 | - -**表 4** 支持的机器学习算法 - 图类 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :----------------- | :------------------------------------ | :----------------------------------------------------------- | -| 所有对间最短路径 | All Pairs Shortest Path (APSP) | 所有对最短路径（APSP）算法找到所有顶点对之间的最短路径的长度（总和权重），使得路径边缘的权重之和最小化。 | -| 广度优先算法 | Breadth-First Search | 广度优先算法遍历路径。 | -| 超链接诱导主题搜索 | Hyperlink-Induced Topic Search (HITS) | HITS算法输出每个节点的authority评分和hub评分，其中authority评分给出页面内容的分数，hub评估出连接到其他页面的分数。 | -| 平均路径长度 | Average Path Length | 此函数计算每对顶点之间的最短路径的平均值。平均路径长度基于“可到达的目标顶点”，因此它忽略了未连接的顶点之间的无限长度路径。 | -| 中心性 | Closeness Centrality | 接近度度量是和的倒数，平均值的倒数，以及到所有可到达目标顶点（不包括源顶点）的最短距离的倒数之和。 | -| 图表直径 | Graph Diameter | 直径被定义为图中所有最短路径中最长的。 | -| 入度出度 | In-Out Degree | 计算图中每个点的入度出度，入度指指向此点的边的数量，出度指此点指向其他点的边的数量。 | -| 网页排名 | PageRank | 给定图形，PageRank算法输出概率分布，该概率分布表示随机遍历图形的人将到达任何特定顶点的可能性。 | -| 单源最短路径 | Single Source Shortest Path (SSSP) | 给定图形和源顶点，单源最短路径（SSSP）算法找到从源顶点到图中的每个其他顶点的路径，使得路径边缘的权重之和最小化（每条边权值非负）。 | -| 弱连通分量 | Weakly Connected Component | 给定有向图，弱连通分量（WCC）是原始图的子图，其中所有顶点通过某个路径彼此连接，忽略边的方向。在无向图的情况下，弱连通分量也是强连通分量。该模块还包括许多在WCC输出上运行的辅助函数。 | - -**表 5** 支持的机器学习算法 - 时间序列 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :------------------------- | :---------------------------------------------------- | :----------------------------------------------------------- | -| 差分整合移动平均自回归模型 | Autoregressive Integrated Moving Average model(ARIMA) | 时间序列预测，用于理解和预测一系列数据的未来值。
比如：国际航空旅客数据，预测旅客人数。 | - -**表 6** 支持的机器学习算法 - 采样 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :--------------- | :------------------ | :----------------------------------------------------------- | -| 采样函数 | sample | 抽样。 | -| 分层抽样 | Stratified Sampling | 分层随机抽样，又称类型随机抽样，它是先将总体各单位按一定标准分成各种类型（或层）；然后根据各类型单位数与总体单位数的比例，确定从各类型中抽取样本单位的数量；最后，按照随机原则从各类型中抽取样本。 | -| 对称抽样 | Balanced Sampling | 一些分类算法仅在每个类中的样本数大致相同时才最佳地执行。高度偏斜的数据集在许多领域中是常见的（例如，欺诈检测），因此重新采样以抵消这种不平衡可以产生更好的决策边界。 | - -**表 7** 支持的机器学习算法 - 统计学 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :--------------- | :------------------------------- | :----------------------------------------------------------- | -| 汇总统计函数 | Summary | 生成任何数据表的摘要统计信息。 | -| 协方差和相关系数 | Correlation and Covariance | 描述性统计，求Pearson系数，相关系数，另一个输出协方差。了解数据从统计学上反映的量的特征，以便我们更好地认识这些将要被挖掘的数据。 | -| 统计频率算法 | CountMin (Cormode-Muthukrishnan) | 统计一个实时的数据流中元素出现的频率，并且准备随时回答某个元素出现的频率，不需要的精确的计数。 | -| 基数估计算法 | FM (Flajolet-Martin) | 获取指定列中的不同值的数量。 找出这个数字集合中不重复的数字的个数。 | -| 最频繁值 | MFV (Most Frequent Values) | 计算频繁值的场景。 | -| 假设检验 | Hypothesis Tests | 包含F-test，chi2-test等。 | -| 概率函数 | Probability Functions | 概率函数模块为各种概率分布提供累积分布，密度、质量和分位数函数。 | - -**表 8** 支持的机器学习算法 - 其他算法 - -| 算法名称（中文） | 算法名称（英文） | 应用场景 | -| :--------------- | :-------------------------------- | :----------------------------------------------------------- | -| k-聚类算法 | K-means | 聚类场景。 | -| 隐含狄利克雷分布 | Latent Dirichlet Allocation (LDA) | LDA 在主题模型中占有非常重要的地位，常用来文本分类。 | -| 关联规则算法 | Apriori Algorithm | 关联规则算法，关联规则挖掘的目标是发现数据项集之间的关联关系。比如经典的“啤酒和尿布”。 | diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-2-environment-deployment.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-2-environment-deployment.md deleted file mode 100644 index 2898d749d3b137178157b054035432140ca89f41..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-2-environment-deployment.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -title: 环境部署 -summary: 环境部署 -author: Guo Huan -date: 2021-05-19 ---- - -# 环境部署 - -DeepSQL环境包括编译数据库和安装算法库两个部分。 - -
- -## 前提条件 - -- 环境中安装python2.7.12以上版本Python。 -- 数据库需要开启对PL/Python存储过程的支持。 -- 安装算法库需要拥有管理员权限的用户。 - -
- -## 操作步骤 - -1. 检查部署Python环境。 - - 安装前，请查看系统安装的python版本，当前DeepSQL需要python2.7.12以上版本的环境。 - - - 如果当前系统python2版本高于2.7.12，可以直接安装python-devel包。 - - 如果版本过低，或者无法安装python-devel包，可以下载最新python2源码，手动配置编译python2，并配置环境变量。 - - 算法库中，部分算法调用了python包，如numpy，pandas等。用户可以安装以下python库： - - ```bash - pip install numpy - pip install pandas - pip install scipy - ``` - - > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **须知：** - > - > - 如果自行编译python，需要在**configure**脚本执行时加入**-enable-shared**参数； - > - 如果系统中的python2使用的是UCS4编码，自行编译python2时，还需要加入**-enable-unicode=ucs4**参数。 可以在系统中自带的python2下执行：“import sys；print sys.maxunicode”并查看结果，如果结果是65535，说明系统默认的是ucs2；如果结果是1114111，说明用的ucs4编码。 - > - 如果系统中内置的python2使用的ucs4，说明系统中的gdb，gstack等也会依赖ucs4。因此自行编译的python2在configure时，需要添加-enable-unicode=ucs4，否则后续使用gdb，gstack时，会遇到报错。 - -2. 编译部署数据库。 - - 数据库需要开启对PL/Python存储过程的支持。默认编译数据库，不包含此模块。因此需要编译数据库时，在configure阶段，加入--with-python参数； - - 其他编译保持步骤不变； - - 编译完成后，需要重新gs_initdb； - - 默认PL/Python存储过程模块不被加载，请执行“CREATE EXTENSION plpythonu”来加载模块。 - -3. 算法库编译和安装。 - - 算法库使用开源的MADlib机器学习框架。源码包和相应patch可以从第三方库的代码仓库里获取。安装命令如下： - - ```bash - tar -zxf apache-madlib-1.17.0-src.tar.gz - cp madlib.patch apache-madlib-1.17.0-src - cd apache-madlib-1.17.0-src/ - patch -p1 < madlib.patch - ``` - - 编译命令如下： - - ```bash - ./configure -DCMAKE_INSTALL_PREFIX={YOUR_MADLIB_INSTALL_FOLDER} - -DPOSTGRESQL_EXECUTABLE=$GAUSSHOME/bin/ - -DPOSTGRESQL_9_2_EXECUTABLE=$GAUSSHOME/bin/ - -DPOSTGRESQL_9_2_CLIENT_INCLUDE_DIR=$GAUSSHOME/bin/ - -DPOSTGRESQL_9_2_SERVER_INCLUDE_DIR=$GAUSSHOME/bin/ - # 以上均为configure命令。 - make - make install - ``` - - 其中， {YOUR_MADLIB_INSTALL_FOLDER}需要改为用户的实际安装路径。 - - > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **须知：** 编译MADlib时，会联网下载依赖软件。无法联网时，需要手动下载依赖包“PyXB-1.2.6.tar.gz”，“eigen-branches-3.2.tar.gz”和“boost_1_61_0.tar.gz”放在本地。使用的configure命令如下： - > - > ```bash - > ./configure -DCMAKE_INSTALL_PREFIX={YOUR_MADLIB_INSTALL_FOLDER} # your install folder - > -DPYXB_TAR_SOURCE={YOUR_DEPENDENCY_FOLDER}/PyXB-1.2.6.tar.gz # change to your local folder - > -DEIGEN_TAR_SOURCE={YOUR_DEPENDENCY_FOLDER}/eigen-branches-3.2.tar.gz # change to your local folder - > -DBOOST_TAR_SOURCE={YOUR_DEPENDENCY_FOLDER}/boost_1_61_0.tar.gz # change to your local folder - > -DPOSTGRESQL_EXECUTABLE=$GAUSSHOME/bin/ - > -DPOSTGRESQL_9_2_EXECUTABLE=$GAUSSHOME/bin/ - > -DPOSTGRESQL_9_2_CLIENT_INCLUDE_DIR=$GAUSSHOME/bin/ - > -DPOSTGRESQL_9_2_SERVER_INCLUDE_DIR=$GAUSSHOME/bin/ - > ``` - -4. 将算法库安装到数据库中。 - - a. 进入{YOUR_MADLIB_INSTALL_FOLDER}路径。 - - b. 进入bin文件夹。 - - c. 执行如下命令。 - - ```bash - ./madpack -s -p mogdb -c @127.0.0.1:/ install - ``` - - 命令中参数说明如下： - - - -s：schema的名称。 - - -p：数据库平台，使用mogdb即可。 - - -c：连接数据库的参数。包括用户名、‘@’、IP地址、端口号和目标数据库名称。 - - install为安装的命令，除此之外，还有reinstall（重新安装），uninstall（卸载）等命令可用。 - - > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** - > - > - 目标数据库必须存在。 - > - IP请使用127.0.0.1，不要使用localhost。 - > - 涉及到大量PL/Python存储过程的安装、卸载等操作，需要数据库管理员权限用户来进行，普通用户没有权限创建和修改PL/Python存储过程，只能调用。 - > - 数据库兼容性，推荐兼容性为B。不同的数据库兼容性下，对空值，NULL等处理有较大差异。建议使用B兼容性。例如，`CREATE DATABASE dbcompatibility='B'`。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-3-usage-guide.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-3-usage-guide.md deleted file mode 100644 index 5233799af9f668d861466363160efcaf751e2487..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-3-usage-guide.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -title: Usage Guide -summary: Usage Guide -author: Guo Huan -date: 2021-05-19 ---- - -# 使用指导 - -
- -## PL/Python存储过程 - -当前PL/Python存储过程优先支持python2；默认版本也是python2。 - -PL/Python中的函数通过标准的CREATE FUNCTION声明： - -```sql -CREATE FUNCTION funcname (argument-list) -RETURNS return-type -AS $$ -# PL/Python function body -$$ LANGUAGE plpythonu; -``` - -函数体是一个简单的Python脚本，当函数被调用的时候，它的参数作为列表args的元素传递；命名参数也会被当做普通的变量传递到Python脚本中。命名参数的使用通常更易读。 结果将使用return或yield（结果集语句的情况） 照常从Python代码中返回。如果没有提供返回值，Python返回缺省的None。 PL/Python将Python中的None认为SQL空值。 - -例如，返回两个整数中较大者的函数定义如下。 - -```sql -CREATE FUNCTION pymax(a integer, b integer) RETURNS integer AS $$ -if a > b: - return a -return b -$$ LANGUAGE plpythonu; -``` - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **注意：** -> -> - PL/Python函数中，后缀为plpythonu。'u’说明是untrusted类型的存储过程。 -> - Trusted：这个语言不能访问越权的数据。例如，数据库服务器的文件、数据库内部（包括直接访问共享内存）。 -> - Untrusted：这个语言没有任何限制，允许访问任何数据（包括文件，网络，共享LIB库等，危害性较大），但是功能更加强大。 -> - PL/Python属于untrusted类型的存储过程语言，当前仅允许管理员权限的用户创建和修改，普通用户仅支持使用。 -> - 定义PL/Python存储过程时，注意不要定义执行诸如import os；os.system(“rm -rf /") 等危险语句。管理员权限的用户需要小心创建此类PL/Python存储过程。 - -
- -## 数据库Null, None和空串处理 - -如果向函数传递了一个SQL null值，参数值在Python中将会显示为None。在数据库中，不同的兼容性下，空串的行为会被当做NULL处理。 - -同一个函数，在不同的兼容性下表现不同。 - -```sql -CREATE FUNCTION quote(t text, how text) RETURNS text AS $$ -if how == "literal": - return plpy.quote_literal(t) -elif how == "nullable": - return plpy.quote_nullable(t) -elif how == "ident": - return plpy.quote_ident(t) -else: - raise plpy.Error("unrecognized quote type %s" % how) -$$ LANGUAGE plpythonu; -``` - -**示例1：** - -```sql -SELECT quote(t, 'literal') FROM (VALUES ('abc'),('a''bc'),('''abc'''),(''),(''''),('xyzv')) AS v(t); -``` - -数据库不同兼容性下的结果为： - -- 兼容性为A时，返回结果如下： - - ```sql - ERROR: TypeError: argument 1 must be string, not None - CONTEXT: Traceback (most recent call last): - PL/Python function "quote", line 3, in - return plpy.quote_literal(t) - referenced column: quote - ``` - -- 兼容性为B时，返回结果如下： - - ```sql - quote - ----------- - 'abc' - 'a''bc' - '''abc''' - '' - '''' - 'xyzv' - (6 rows) - ``` - -**示例2：** - -```sql -SELECT quote(t, 'nullable') FROM (VALUES ('abc'),('a''bc'),('''abc'''),(''),(''''),(NULL)) AS v(t); -``` - -数据库不同兼容性下的结果为： - -- 兼容性为A时，返回结果如下： - - ```sql - quote - ----------- - 'abc' - 'a''bc' - '''abc''' - NULL - '''' - NULL - (6 rows) - ``` - -- 兼容性为B时，返回结果如下： - - ```sql - quote - ----------- - 'abc' - 'a''bc' - '''abc''' - '' - '''' - NULL - (6 rows) - ``` - -可以看到，在兼容性“A”中，空串被当为NULL了。 - -
- -## 触发器 - -当前PL/Python存储过程中，不支持触发器功能。 - -
- -## 匿名代码块 - -PL/Python也支持DO声明的匿名代码块： - -```sql -DO $$ -# PL/Python code -$$ LANGUAGE plpythonu; -``` - -一个匿名代码块不接受参数，并且丢弃它可能返回的值。 - -
- -## 共享数据 - -每个函数都在Python解释器里获得自己的执行环境。 - -全局字典SD在函数调用之间用于存储数据。这些变量是私有静态数据。每一个函数都有自己的SD数据空间，函数A的全局数据和函数参数是函数B不可用的。 - -全局字典GD是公共数据，在一个gsql会话中，所有python函数都可访问和改变，使用时需要小心。 - -当gsql断开或退出，共享数据就被释放。 - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **注意：** -> -> - 运行DeepSQL或者PL/Python存储过程时，需要关闭线程池相关参数。否则PL/Python存储过程中的Sharing Data（“GD”、“SD”)等功能会失效。 -> - 在数据库中，当线程池功能关闭，每一个连入的gsql，数据库内会起一个新的线程去处理。在gsql中，如果调用到PL/Python存储过程，会在本线程中完成python解析器模块的初始化，其中包括初始化“GD”，“SD”等共享空间。 -> - 在线程池功能开启的状态下，一个gsql执行时，由当前空闲线程执行，每次执行可能分配到不同的线程上。导致共享数据紊乱。 - -
- -## 数据库访问 - -PL/Python语言模块自动import一个叫plpy的Python模块。 - -plpy模块提供几个函数执行数据库命令：比如plpy.execute，plpy.prepare等。 - -plpy模块也提供了函数plpy.debug(**msg**)、 plpy.log(**msg**)、plpy.info(**msg**)、 plpy.notice(**msg**)、plpy.warning(**msg**)、 plpy.error(**msg**)和plpy.fatal(**msg**)。 plpy.error和 plpy.fatal实际上抛出了一个Python异常，会导致当前事务或者子事务退出。 - -另一个实用函数集是plpy.quote_literal(string)、 plpy.quote_nullable(string)和 plpy.quote_ident(string)。 - -
- -## 关于审计 - -PL/Python存储过程支持审计功能。具体设置可以参考[审计](1-audit-switch)。 - -
- -## 关于并发执行 - -当前PL/Python存储过程对并发执行不友好，建议串行执行。 - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 由于MogDB是多线程架构，C-python中，由于GIL锁（Global Interpreter Lock)的限制，多线程在Python中只能交替执行，无法做到真正的并发。 - -
- -## 库内算法 - -具体库内算法介绍和使用，可参考MADlib官方网站（[MADlib文档](http://madlib.apache.org/docs/latest/index.html)）。 - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **须知：** -> -> - 当前仅支持机器学习算法，不支持深度学习（deep learning)模块。 -> - 当前数据库不支持xml，所以pmml模块和相关功能不支持。 -> - 数据库不支持jsonb模块，json格式的模型导出功能也不支持。 - -
- -## 其他算法支持 - -除了MADlib提供的算法外，MogDB又额外提供了以下三个算法。 - -**表 1** 额外增加的模块列表 - -| 算法名称（中文） | 算法名称（英文） | -| :----------------- | :--------------- | -| 梯度提升树 | gbdt | -| 梯度提升 | xgboost | -| 时间序列预测的算法 | facebook_prophet | - -使用时，需要安装依赖的python库： - -- 如果使用prophet算法： - - ```bash - pip install pystan - pip install holidays==0.9.8 - pip install fbprophet==0.3.post2 - ``` - -- 如果使用xgboost算法： - - ```bash - pip install xgboost - pip install scikit-learn - ``` - -- gbdt不需要额外安装其他库。 - -详细操作请参考[最佳实践](7-4-best-practices)。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-4-best-practices.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-4-best-practices.md deleted file mode 100644 index 502bad8910db6d1705a61a76c5ba6ae6eb3d4398..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-4-best-practices.md +++ /dev/null @@ -1,902 +0,0 @@ ---- -title: Best Practices -summary: Best Practices -author: Guo Huan -date: 2021-05-19 ---- - -# 最佳实践 - -本章节介绍部分算法的使用，主要包含分类、回归、聚类、gbdt算法、xgboost算法和prohpet算法。 - -首先需要创建一个数据库，并安装算法。 - -```sql -create database test1 dbcompatibility='B'; -./madpack -s madlib -p mogdb -c opg@127.0.0.1:7651/test1 install -``` - -
- -## 分类算法 - -以svm分类房价为例子： - -1. 数据集准备。 - - ```sql - DROP TABLE IF EXISTS houses; - CREATE TABLE houses (id INT, tax INT, bedroom INT, bath FLOAT, price INT, size INT, lot INT); - INSERT INTO houses VALUES - (1 , 590 , 2 , 1 , 50000 , 770 , 22100), - (2 , 1050 , 3 , 2 , 85000 , 1410 , 12000), - (3 , 20 , 3 , 1 , 22500 , 1060 , 3500), - (4 , 870 , 2 , 2 , 90000 , 1300 , 17500), - (5 , 1320 , 3 , 2 , 133000 , 1500 , 30000), - (6 , 1350 , 2 , 1 , 90500 , 820 , 25700), - (7 , 2790 , 3 , 2.5 , 260000 , 2130 , 25000), - (8 , 680 , 2 , 1 , 142500 , 1170 , 22000), - (9 , 1840 , 3 , 2 , 160000 , 1500 , 19000), - (10 , 3680 , 4 , 2 , 240000 , 2790 , 20000), - (11 , 1660 , 3 , 1 , 87000 , 1030 , 17500), - (12 , 1620 , 3 , 2 , 118600 , 1250 , 20000), - (13 , 3100 , 3 , 2 , 140000 , 1760 , 38000), - (14 , 2070 , 2 , 3 , 148000 , 1550 , 14000), - (15 , 650 , 3 , 1.5 , 65000 , 1450 , 12000); - ``` - -2. 模型训练。 - - 训练前配置相应schema和兼容性参数： - - ```sql - SET search_path="$user",public,madlib; - SET behavior_compat_options = 'bind_procedure_searchpath'; - ``` - - 使用默认的参数进行训练，分类的条件为‘price < 100000’，SQL语句如下： - - ```sql - DROP TABLE IF EXISTS houses_svm, houses_svm_summary; - SELECT madlib.svm_classification('public.houses','public.houses_svm','price < 100000','ARRAY[1, tax, bath, size]'); - ``` - -3. 查看模型。 - - ```sql - \x on - SELECT * FROM houses_svm; - \x off - ``` - - 结果如下： - - ```sql - -[ RECORD 1 ]------+----------------------------------------------------------------- - coef | {.113989576847,-.00226133300602,-.0676303607996,.00179440841072} - loss | .614496714256667 - norm_of_gradient | 108.171180769224 - num_iterations | 100 - num_rows_processed | 15 - num_rows_skipped | 0 - dep_var_mapping | {f,t} - ``` - -4. 进行预测。 - - ```sql - DROP TABLE IF EXISTS houses_pred; - SELECT madlib.svm_predict('public.houses_svm','public.houses','id','public.houses_pred'); - ``` - - - 查看预测结果 - - ```sql - SELECT *, price < 100000 AS actual FROM houses JOIN houses_pred USING (id) ORDER BY id; - ``` - - 结果如下： - - ```sql - id | tax | bedroom | bath | price | size | lot | prediction | decision_function | actual - ----+------+---------+------+--------+------+-------+------------+-------------------+-------- - 1 | 590 | 2 | 1 | 50000 | 770 | 22100 | t | .09386721875 | t - 2 | 1050 | 3 | 2 | 85000 | 1410 | 12000 | t | .134445058042 | t - 3 | 20 | 3 | 1 | 22500 | 1060 | 3500 | t | 1.9032054712902 | t - 4 | 870 | 2 | 2 | 90000 | 1300 | 17500 | t | .3441000739464 | t - 5 | 1320 | 3 | 2 | 133000 | 1500 | 30000 | f | -.3146180966186 | f - 6 | 1350 | 2 | 1 | 90500 | 820 | 25700 | f | -1.5350254452892 | t - 7 | 2790 | 3 | 2.5 | 260000 | 2130 | 25000 | f | -2.5421154971142 | f - 8 | 680 | 2 | 1 | 142500 | 1170 | 22000 | t | .6081106124962 | f - 9 | 1840 | 3 | 2 | 160000 | 1500 | 19000 | f | -1.490511259749 | f - 10 | 3680 | 4 | 2 | 240000 | 2790 | 20000 | f | -3.336577140997 | f - 11 | 1660 | 3 | 1 | 87000 | 1030 | 17500 | f | -1.8592129109042 | t - 12 | 1620 | 3 | 2 | 118600 | 1250 | 20000 | f | -1.4416201011046 | f - 13 | 3100 | 3 | 2 | 140000 | 1760 | 38000 | f | -3.873244660547 | f - 14 | 2070 | 2 | 3 | 148000 | 1550 | 14000 | f | -1.9885277913972 | f - 15 | 650 | 3 | 1.5 | 65000 | 1450 | 12000 | t | 1.1445697772786 | t - (15 rows) - ``` - - - 查看误分率 - - ```sql - SELECT COUNT(*) FROM houses_pred JOIN houses USING (id) WHERE houses_pred.prediction != (houses.price < 100000); - ``` - - 结果如下： - - ```sql - count - ------- - 3 - (1 row) - ``` - -5. 使用svm其他核进行训练。 - - ```sql - DROP TABLE IF EXISTS houses_svm_gaussian, houses_svm_gaussian_summary, houses_svm_gaussian_random; - SELECT madlib.svm_classification( 'public.houses','public.houses_svm_gaussian','price < 100000','ARRAY[1, tax, bath, size]','gaussian','n_components=10', '', 'init_stepsize=1, max_iter=200' ); - ``` - - 进行预测，并查看训练结果。 - - ```sql - DROP TABLE IF EXISTS houses_pred_gaussian; - SELECT madlib.svm_predict('public.houses_svm_gaussian','public.houses','id', 'public.houses_pred_gaussian'); - SELECT COUNT(*) FROM houses_pred_gaussian JOIN houses USING (id) WHERE houses_pred_gaussian.prediction != (houses.price < 100000); - ``` - - 结果如下。 - - ```sql - count - -------+ - 0 - (1 row) - ``` - -6. 其他参数 - - 除了指定不同的核方法外，还可以指定迭代次数，初始参数，比如init_stepsize, max_iter, class_weight等。 - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **须知：** -> -> ```sql -> SET search_path="$user",public,madlib; -> SET behavior_compat_options = 'bind_procedure_searchpath'; -> ``` -> -> 执行算法前，需要设置search_path中schema，另外需要bind_procedure_searchpath，否则出现表找不到的情况。因为机器学习所有的方法安装在一个schema中，用户的表安装在用户的schema中，在这个例子中，算法安装在madlib中，用户表放置在public中。如果不设置，可能会出现执行算法时，找不到表的情况。执行算法时，建议把输入表的schema也加进去。 - -
- -## 回归算法 - -我们以线性回归预测波士顿房价为例： - -1. 数据集准备。 - - 同svm的数据集。 - -2. 训练模型。 - - ```sql - SET search_path="$user",public,madlib; - SET behavior_compat_options = 'bind_procedure_searchpath'; - - DROP TABLE IF EXISTS houses_linregr, houses_linregr_summary; - SELECT madlib.linregr_train( 'public.houses', 'public.houses_linregr', 'price', 'ARRAY[1, tax, bath, size]'); - ``` - -3. 查看模型内容。 - - ```sql - \x ON - SELECT * FROM houses_linregr; - \x OFF - ``` - - 返回结果如下。 - - ```sql - -[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - coef | {-12849.4168959872,28.9613922651775,10181.6290712649,50.516894915353} - r2 | .768577580597462 - std_err | {33453.0344331377,15.8992104963991,19437.7710925915,32.9280231740856} - t_stats | {-.384103179688204,1.82156166004197,.523806408809163,1.53416118083608} - p_values | {.708223134615411,.0958005827189556,.610804093526516,.153235085548177} - condition_no | 9002.50457069858 - num_rows_processed | 15 - num_missing_rows_skipped | 0 - variance_covariance | {{1119105512.7847,217782.067878005,-283344228.394538,-616679.693190829},{217782.067878005,252.784894408806,-46373.1796964038,-369.864520095145},{-283344228.394538,-46373.1796964038,377826945.047986,-209088.217319699},{-616679.693190829,-369.864520095145,-209088.217319699,1084.25471015312}} - ``` - -4. 预测，并对比结果。 - - ```sql - SELECT houses.*, - madlib.linregr_predict( m.coef, ARRAY[1,tax,bath,size]) as predict, - price - madlib.linregr_predict( m.coef, ARRAY[1,tax,bath,size]) as residual - FROM public.houses, public.houses_linregr AS m - ORDER BY id; - ``` - - 返回结果如下。 - - ```sql - id | tax | bedroom | bath | price | size | lot | predict | residual - ----+------+---------+------+--------+------+-------+------------------+------------------- - 1 | 590 | 2 | 1 | 50000 | 770 | 22100 | 53317.4426965543 | -3317.44269655428 - 2 | 1050 | 3 | 2 | 85000 | 1410 | 12000 | 109152.124955627 | -24152.1249556268 - 3 | 20 | 3 | 1 | 22500 | 1060 | 3500 | 51459.3486308555 | -28959.3486308555 - 4 | 870 | 2 | 2 | 90000 | 1300 | 17500 | 98382.215907206 | -8382.21590720599 - 5 | 1320 | 3 | 2 | 133000 | 1500 | 30000 | 121518.221409606 | 11481.7785903935 - 6 | 1350 | 2 | 1 | 90500 | 820 | 25700 | 77853.9455638568 | 12646.0544361432 - 7 | 2790 | 3 | 2.5 | 260000 | 2130 | 25000 | 201007.926371722 | 58992.0736282778 - 8 | 680 | 2 | 1 | 142500 | 1170 | 22000 | 76130.7259665615 | 66369.2740334385 - 9 | 1840 | 3 | 2 | 160000 | 1500 | 19000 | 136578.145387499 | 23421.8546125013 - 10 | 3680 | 4 | 2 | 240000 | 2790 | 20000 | 255033.901596231 | -15033.9015962306 - 11 | 1660 | 3 | 1 | 87000 | 1030 | 17500 | 97440.5250982859 | -10440.5250982859 - 12 | 1620 | 3 | 2 | 118600 | 1250 | 20000 | 117577.415360321 | 1022.58463967856 - 13 | 3100 | 3 | 2 | 140000 | 1760 | 38000 | 186203.892319614 | -46203.8923196141 - 14 | 2070 | 2 | 3 | 148000 | 1550 | 14000 | 155946.739425522 | -7946.73942552213 - 15 | 650 | 3 | 1.5 | 65000 | 1450 | 12000 | 94497.4293105374 | -29497.4293105374 - (15 rows) - ``` - -
- -## 聚类算法 - -以kmeans为例： - -1. 准备数据。 - - ```sql - DROP TABLE IF EXISTS km_sample; - CREATE TABLE km_sample(pid int, points double precision[]); - INSERT INTO km_sample VALUES - (1, '{14.23, 1.71, 2.43, 15.6, 127, 2.8, 3.0600, 0.2800, 2.29, 5.64, 1.04, 3.92, 1065}'), - (2, '{13.2, 1.78, 2.14, 11.2, 1, 2.65, 2.76, 0.26, 1.28, 4.38, 1.05, 3.49, 1050}'), - (3, '{13.16, 2.36, 2.67, 18.6, 101, 2.8, 3.24, 0.3, 2.81, 5.6799, 1.03, 3.17, 1185}'), - (4, '{14.37, 1.95, 2.5, 16.8, 113, 3.85, 3.49, 0.24, 2.18, 7.8, 0.86, 3.45, 1480}'), - (5, '{13.24, 2.59, 2.87, 21, 118, 2.8, 2.69, 0.39, 1.82, 4.32, 1.04, 2.93, 735}'), - (6, '{14.2, 1.76, 2.45, 15.2, 112, 3.27, 3.39, 0.34, 1.97, 6.75, 1.05, 2.85, 1450}'), - (7, '{14.39, 1.87, 2.45, 14.6, 96, 2.5, 2.52, 0.3, 1.98, 5.25, 1.02, 3.58, 1290}'), - (8, '{14.06, 2.15, 2.61, 17.6, 121, 2.6, 2.51, 0.31, 1.25, 5.05, 1.06, 3.58, 1295}'), - (9, '{14.83, 1.64, 2.17, 14, 97, 2.8, 2.98, 0.29, 1.98, 5.2, 1.08, 2.85, 1045}'), - (10, '{13.86, 1.35, 2.27, 16, 98, 2.98, 3.15, 0.22, 1.8500, 7.2199, 1.01, 3.55, 1045}'); - ``` - -2. 运行kmeans算法。 - - 使用kmeans++进行计算，距离函数使用欧几里得距离。 - - ```sql - SET search_path="$user",public,madlib; - SET behavior_compat_options = 'bind_procedure_searchpath'; - - DROP TABLE IF EXISTS km_result; - CREATE TABLE km_result AS SELECT * FROM madlib.kmeanspp( 'public.km_sample', -- Table of source data - 'points', -- Column containing point co-ordinates - 2, -- Number of centroids to calculate - 'madlib.squared_dist_norm2', -- Distance function - 'madlib.avg', -- Aggregate function - 20, -- Number of iterations - 0.001 -- Fraction of centroids reassigned to keep iterating - ); - ``` - - kmeans执行完后，不会自动创建表保存内容，所以需要用户自行创建table。 - - ```sql - \x on - select * from km_result; - \x off - ``` - - 返回结果如下。 - - ```sql - -[ RECORD 1 ]----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - centroids | {{14.0333333333333,1.84111111111111,2.41,15.5111111111111,96.2222222222222,2.91666666666667,3.01111111111111,.282222222222222,1.95444444444444,5.88553333333333,1.02222222222222,3.38222222222222,1211.66666666667},{13.24,2.59,2.87,21,118,2.8,2.69,.39,1.82,4.32,1.04,2.93,735}} - cluster_variance | {257041.999707571,0} - objective_fn | 257041.999707571 - frac_reassigned | 0 - num_iterations | 2 - ``` - -3. 应用聚类结果。 - - 执行以下函数，计算每个节点的临近节点和相应距离。 - - ```sql - DROP TABLE IF EXISTS km_points_silh; - SELECT * FROM madlib.simple_silhouette_points('public.km_sample', -- Input points table - 'public.km_points_silh', -- Output table - 'pid', -- Point ID column in input table - 'points', -- Points column in input table - 'public.km_result', -- Centroids table - 'centroids', -- Column in centroids table containing centroids - 'madlib.squared_dist_norm2' -- Distance function - ); - SELECT * FROM km_points_silh ORDER BY pid; - ``` - - 返回结果如下。 - - ```sql - pid | centroid_id | neighbor_centroid_id | silh - -----+-------------+----------------------+------------------ - 1 | 0 | 1 | .793983543638996 - 2 | 0 | 1 | .688301735667703 - 3 | 0 | 1 | .996324103148159 - 4 | 0 | 1 | .869765755931474 - 5 | 1 | 0 | 1 - 6 | 0 | 1 | .888416176253661 - 7 | 0 | 1 | .980107240092519 - 8 | 0 | 1 | .975880363039906 - 9 | 0 | 1 | .712384473959954 - 10 | 0 | 1 | .712198411442872 - (10 rows) - ``` - -
- -## gbdt算法 - -gdbt的基学习器虽然是回归树，但算法本身支持分类和回归两种操作。以下将展示两种任务的具体实现。这里值得注意的一点是，本方法不支持标签列含有空值（NULL）的情况。 - -**分类任务**： - -1. 准备数据。 - - ```sql - DROPTABLEIFEXISTS dt_golf CASCADE; - DROPTABLEIFEXISTS train_output,train_output_summary; - CREATETABLE dt_golf ( - id integerNOTNULL, - "OUTLOOK"text, - temperature double precision, - humidity double precision, - "Cont_features"double precision[], - cat_features text[], - windy boolean, - class integer - ) ; - INSERTINTO dt_golf (id,"OUTLOOK",temperature,humidity,"Cont_features",cat_features, windy,class) VALUES - (1, 'sunny', 85, 85,ARRAY[85, 85], ARRAY['a', 'b'], false, 0), - (2, 'sunny', 80, 90, ARRAY[80, 90], ARRAY['a', 'b'], true, 0), - (3, 'overcast', 83, 78, ARRAY[83, 78], ARRAY['a', 'b'], false, 1), - (4, 'rain', 70, NULL, ARRAY[70, 96], ARRAY['a', 'b'], false, 1), - (5, 'rain', 68, 80, ARRAY[68, 80], ARRAY['a', 'b'], false, 1), - (6, 'rain', NULL, 70, ARRAY[65, 70], ARRAY['a', 'b'], true, 0), - (7, 'overcast', 64, 65, ARRAY[64, 65], ARRAY['c', 'b'], NULL , 1), - (8, 'sunny', 72, 95, ARRAY[72, 95], ARRAY['a', 'b'], false, 0), - (9, 'sunny', 69, 70, ARRAY[69, 70], ARRAY['a', 'b'], false, 1), - (10, 'rain', 75, 80, ARRAY[75, 80], ARRAY['a', 'b'], false, 1), - (11, 'sunny', 75, 70, ARRAY[75, 70], ARRAY['a', 'd'], true, 1), - (12, 'overcast', 72, 90, ARRAY[72, 90], ARRAY['c', 'b'], NULL, 1), - (13, 'overcast', 81, 75, ARRAY[81, 75], ARRAY['a', 'b'], false, 1), - (15, NULL, 81, 75, ARRAY[81, 75], ARRAY['a', 'b'], false, 1), - (16, 'overcast', NULL, 75, ARRAY[81, 75], ARRAY['a', 'd'], false,1), - (14, 'rain', 71, 80, ARRAY[71, 80], ARRAY['c', 'b'], true, 0); - ``` - -2. 训练模型。 - - ```sql - select madlib.gbdt_train('dt_golf', -- source table - 'train_output', -- output model table - 'id' , -- id column - 'class', -- response - '"OUTLOOK", temperature', -- features - NULL, -- exclude columns - 1, --weight - 10, -- num of trees - NULL, -- num of random features - 10, -- max depth - 1, -- min split - 1, -- min bucket - 8, -- number of bins per continuous variable - 'max_surrogates=0', - TRUE - ); - ``` - - 模型训练结束会生成两个表，其中一张表train_output中保存着gdbt中的回归树模型，内容包括对每一个基学习器的参数记录。 - - ```sql - Column | Type | Modifiers | Storage | Stats target | Description - --------------------+---------------+-----------+----------+--------------+------------- - iteration | integer | | plain | | - tree | madlib.bytea8 | | external | | - cat_levels_in_text | text[] | | extended | | - cat_n_levels | integer[] | | extended | | - tree_depth | integer | | plain | | - ``` - - ```sql - iteration | cat_levels_in_text | cat_n_levels | tree_depth - -----------+-----------------------+--------------+------------ - 0 | {sunny,rain,overcast} | {3} | 4 - 1 | {sunny,rain,overcast} | {3} | 5 - 2 | {sunny,rain,overcast} | {3} | 6 - 3 | {sunny,rain,overcast} | {3} | 4 - 4 | {sunny,rain,overcast} | {3} | 5 - 5 | {sunny,rain,overcast} | {3} | 5 - 6 | {sunny,rain,overcast} | {3} | 5 - 7 | {sunny,rain,overcast} | {3} | 5 - 8 | {sunny,rain,overcast} | {3} | 4 - 9 | {sunny,rain,overcast} | {3} | 3 - (10 rows) - ``` - - 另一张表train_output_summary，内容是对gdbt训练的整体描述： - - ```sql - select * from train_output_summary; - method | cat_features | con_features | source_table | model_table | null_proxy | learning_rate | is_classification | predict_dt_prob | num_trees - --------+--------------+--------------+--------------+--------------+------------+---------------+-------------------+-----------------+----------- - GBDT | "OUTLOOK" | temperature | dt_golf | train_output | | .01 | t | response | 10 - (1 row) - ``` - -3. 预测。 - - ```sql - select madlib.gbdt_predict('dt_golf2','train_output','test_output','id'); - ``` - - 查看预测结果 - - ```sql - select test_output.id, test_prediction,class from test_output join dt_golf using (id); - id | test_prediction | class - ----+-----------------+------- - 1 | 1.0 | 0 - 2 | 1.0 | 0 - 3 | 1.0 | 1 - 4 | 1.0 | 1 - 5 | 1.0 | 1 - 6 | 1.0 | 0 - 7 | 1.0 | 1 - 8 | 0.0 | 0 - 9 | 1.0 | 1 - 10 | 1.0 | 1 - 11 | 1.0 | 1 - 12 | 1.0 | 1 - 13 | 1.0 | 1 - 15 | 0.0 | 1 - 16 | 1.0 | 1 - 14 | 0.0 | 0 - (16 rows) - ``` - -
- -## 回归任务 - -1. 准备数据。 - - ```sql - DROP TABLE IF EXISTS crime; - CREATE TABLE crime ( - id SERIAL NOT NULL, - CrimeRat DOUBLE PRECISION, - MaleTeen INTEGER, - South SMALLINT, - Educ DOUBLE PRECISION, - Police60 INTEGER, - Police59 INTEGER, - Labor INTEGER, - Males INTEGER, - Pop INTEGER, - NonWhite INTEGER, - Unemp1 INTEGER, - Unemp2 INTEGER, - Median INTEGER, - BelowMed INTEGER - ); - - INSERT INTO crime( - CrimeRat, MaleTeen, South, Educ, Police60, Police59, Labor, Males, Pop, - NonWhite, Unemp1, Unemp2, Median, BelowMed - ) VALUES - (79.1, 151, 1, 9.1, 58, 56, 510, 950, 33, 301, 108, 41, 394, 261), - (163.5, 143, 0, 11.3, 103, 95, 583, 1012, 13, 102, 96, 36, 557, 194), - (57.8, 142, 1, 8.9, 45, 44, 533, 969, 18, 219, 94, 33, 318, 250), - (196.9, 136, 0, 12.1, 149, 141, 577, 994, 157, 80, 102, 39, 673, 167), - (123.4, 141, 0, 12.1, 109, 101, 591, 985, 18, 30, 91, 20, 578, 174), - (68.2, 121, 0, 11.0, 118, 115, 547, 964, 25, 44, 84, 29, 689, 126), - (96.3, 127, 1, 11.1, 82, 79, 519, 982, 4, 139, 97, 38, 620, 168), - (155.5, 131, 1, 10.9, 115, 109, 542, 969, 50, 179, 79, 35, 472, 206), - (85.6, 157, 1, 9.0, 65, 62, 553, 955, 39, 286, 81, 28, 421, 239), - (70.5, 140, 0, 11.8, 71, 68, 632, 1029, 7, 15, 100, 24, 526, 174), - (167.4, 124, 0, 10.5, 121, 116, 580, 966, 101, 106, 77, 35, 657, 170), - (84.9, 134, 0, 10.8, 75, 71, 595, 972, 47, 59, 83, 31, 580, 172), - (51.1, 128, 0, 11.3, 67, 60, 624, 972, 28, 10, 77, 25, 507, 206), - (66.4, 135, 0, 11.7, 62, 61, 595, 986, 22, 46, 77, 27, 529, 190), - (79.8, 152, 1, 8.7, 57, 53, 530, 986, 30, 72, 92, 43, 405, 264), - (94.6, 142, 1, 8.8, 81, 77, 497, 956, 33, 321, 116, 47, 427, 247), - (53.9, 143, 0, 11.0, 66, 63, 537, 977, 10, 6, 114, 35, 487, 166), - (92.9, 135, 1, 10.4, 123, 115, 537, 978, 31, 170, 89, 34, 631, 165), - (75.0, 130, 0, 11.6, 128, 128, 536, 934, 51, 24, 78, 34, 627, 135), - (122.5, 125, 0, 10.8, 113, 105, 567, 985, 78, 94, 130, 58, 626, 166), - (74.2, 126, 0, 10.8, 74, 67, 602, 984, 34, 12, 102, 33, 557, 195), - (43.9, 157, 1, 8.9, 47, 44, 512, 962, 22, 423, 97, 34, 288, 276), - (121.6, 132, 0, 9.6, 87, 83, 564, 953, 43, 92, 83, 32, 513, 227), - (96.8, 131, 0, 11.6, 78, 73, 574, 1038, 7, 36, 142, 42, 540, 176), - (52.3, 130, 0, 11.6, 63, 57, 641, 984, 14, 26, 70, 21, 486, 196), - (199.3, 131, 0, 12.1, 160, 143, 631, 1071, 3, 77, 102, 41, 674, 152), - (34.2, 135, 0, 10.9, 69, 71, 540, 965, 6, 4, 80, 22, 564, 139), - (121.6, 152, 0, 11.2, 82, 76, 571, 1018, 10, 79, 103, 28, 537, 215), - (104.3, 119, 0, 10.7, 166, 157, 521, 938, 168, 89, 92, 36, 637, 154), - (69.6, 166, 1, 8.9, 58, 54, 521, 973, 46, 254, 72, 26, 396, 237), - (37.3, 140, 0, 9.3, 55, 54, 535, 1045, 6, 20, 135, 40, 453, 200), - (75.4, 125, 0, 10.9, 90, 81, 586, 964, 97, 82, 105, 43, 617, 163), - (107.2, 147, 1, 10.4, 63, 64, 560, 972, 23, 95, 76, 24, 462, 233), - (92.3, 126, 0, 11.8, 97, 97, 542, 990, 18, 21, 102, 35, 589, 166), - (65.3, 123, 0, 10.2, 97, 87, 526, 948, 113, 76, 124, 50, 572, 158), - (127.2, 150, 0, 10.0, 109, 98, 531, 964, 9, 24, 87, 38, 559, 153), - (83.1, 177, 1, 8.7, 58, 56, 638, 974, 24, 349, 76, 28, 382, 254), - (56.6, 133, 0, 10.4, 51, 47, 599, 1024, 7, 40, 99, 27, 425, 225), - (82.6, 149, 1, 8.8, 61, 54, 515, 953, 36, 165, 86, 35, 395, 251), - (115.1, 145, 1, 10.4, 82, 74, 560, 981, 96, 126, 88, 31, 488, 228), - (88.0, 148, 0, 12.2, 72, 66, 601, 998, 9, 19, 84, 20, 590, 144), - (54.2, 141, 0, 10.9, 56, 54, 523, 968, 4, 2, 107, 37, 489, 170), - (82.3, 162, 1, 9.9, 75, 70, 522, 996, 40, 208, 73, 27, 496, 224), - (103.0, 136, 0, 12.1, 95, 96, 574, 1012, 29, 36, 111, 37, 622, 162), - (45.5, 139, 1, 8.8, 46, 41, 480, 968, 19, 49, 135, 53, 457, 249), - (50.8, 126, 0, 10.4, 106, 97, 599, 989, 40, 24, 78, 25, 593, 171), - (84.9, 130, 0, 12.1, 90, 91, 623, 1049, 3, 22, 113, 40, 588, 160); - ``` - -2. 训练模型。 - - ```sql - select madlib.gbdt_train('crime', -- source table - 'train_output', -- output model table - 'id' , -- id column - 'CrimeRat', -- response - '*', -- features - NULL, -- exclude columns - 1, --weight - 20, -- num of trees - 4, -- num of random features - 10, -- max depth - 1, -- min split - 1, -- min bucket - 8, -- number of bins per continuous variable - 'max_surrogates=0', - FALSE - ); - ``` - - 当is_classification设为FALSE时，模型为回归任务。默认状态下gbdt提供回归计算支持。方法生成两个表，其中一张表记录每棵树的集体信息和模型的二进制，一张表记录方法的参数信息。 - -3. 预测。 - - ```sql - select madlib.gbdt_predict('crime','train_output','test_output','id'); - ``` - - 查看预测结果： - - ```sql - select test_output.id, test_prediction,CrimeRat from test_output join crime using (id); - - id | test_prediction | crimerat - ----+--------------------+---------- - 1 | 79.1 | 79.1 - 2 | 163.5 | 163.5 - 3 | 57.8 | 57.8 - 4 | 196.9 | 196.9 - 5 | 123.4 | 123.4 - 6 | 68.2 | 68.2 - 7 | 96.2999999992251 | 96.3 - 8 | 155.49842087765936 | 155.5 - 9 | 84.35 | 85.6 - 10 | 70.50157912234037 | 70.5 - 11 | 167.4000000007749 | 167.4 - 12 | 84.9 | 84.9 - 13 | 51.1 | 51.1 - 14 | 66.4 | 66.4 - 15 | 79.8 | 79.8 - 16 | 94.6 | 94.6 - 17 | 53.9 | 53.9 - 18 | 92.9 | 92.9 - 19 | 75.0 | 75 - 20 | 122.5 | 122.5 - 21 | 74.2 | 74.2 - 22 | 43.9 | 43.9 - 23 | 121.6 | 121.6 - 24 | 96.8 | 96.8 - 25 | 52.3 | 52.3 - 26 | 199.3 | 199.3 - 27 | 34.2 | 34.2 - 28 | 121.6 | 121.6 - 29 | 104.3 | 104.3 - 30 | 69.6 | 69.6 - 31 | 37.3 | 37.3 - 32 | 75.4 | 75.4 - 33 | 107.2 | 107.2 - 34 | 92.3 | 92.3 - 35 | 65.2999999992251 | 65.3 - 36 | 127.19842087765963 | 127.2 - 37 | 84.35000000002215 | 83.1 - 38 | 56.60155638297881 | 56.6 - 39 | 82.45000000075257 | 82.6 - 40 | 115.10002273936168 | 115.1 - 41 | 88.0 | 88 - 42 | 54.19997726063828 | 54.2 - 43 | 82.44999999999999 | 82.3 - 44 | 103.00002273936173 | 103 - 45 | 45.500000000000156 | 45.5 - 46 | 50.8 | 50.8 - 47 | 84.9 | 84.9 - (47 rows) - ``` - -
- -## xgboost算法 - -新增的xgboost支持分类和回归两种操作。下面以分类iris花为例，展示xgboost算法。 - -xgboost支持grid search方式，可以同时训练多组参数。 - -1. 准备数据。 - - ```sql - DROP TABLE IF EXISTS iris; - create table iris (id serial, a float, d float, c float, b float, label int); - - INSERT into iris (a, b, c, d, label) values - (5.1, 3.5, 1.4, 0.2, 0),(4.9, 3.0, 1.4, 0.2, 0),(4.7, 3.2, 1.3, 0.2, 0),(4.6, 3.1, 1.5, 0.2, 0), - (5.0, 3.6, 1.4, 0.2, 0),(5.4, 3.9, 1.7, 0.4, 0),(4.6, 3.4, 1.4, 0.3, 0),(5.0, 3.4, 1.5, 0.2, 0), - (4.4, 2.9, 1.4, 0.2, 0),(4.9, 3.1, 1.5, 0.1, 0),(5.4, 3.7, 1.5, 0.2, 0),(4.8, 3.4, 1.6, 0.2, 0), - (4.8, 3.0, 1.4, 0.1, 0),(4.3, 3.0, 1.1, 0.1, 0),(5.8, 4.0, 1.2, 0.2, 0),(5.7, 4.4, 1.5, 0.4, 0), - (5.4, 3.9, 1.3, 0.4, 0),(5.1, 3.5, 1.4, 0.3, 0),(5.7, 3.8, 1.7, 0.3, 0),(5.1, 3.8, 1.5, 0.3, 0), - (5.4, 3.4, 1.7, 0.2, 0),(5.1, 3.7, 1.5, 0.4, 0),(4.6, 3.6, 1.0, 0.2, 0),(5.1, 3.3, 1.7, 0.5, 0), - (4.8, 3.4, 1.9, 0.2, 0),(5.0, 3.0, 1.6, 0.2, 0),(5.0, 3.4, 1.6, 0.4, 0),(5.2, 3.5, 1.5, 0.2, 0), - (5.2, 3.4, 1.4, 0.2, 0),(4.7, 3.2, 1.6, 0.2, 0),(4.8, 3.1, 1.6, 0.2, 0),(5.4, 3.4, 1.5, 0.4, 0), - (5.2, 4.1, 1.5, 0.1, 0),(5.5, 4.2, 1.4, 0.2, 0),(4.9, 3.1, 1.5, 0.2, 0),(5.0, 3.2, 1.2, 0.2, 0), - (5.5, 3.5, 1.3, 0.2, 0),(4.9, 3.6, 1.4, 0.1, 0),(4.4, 3.0, 1.3, 0.2, 0),(5.1, 3.4, 1.5, 0.2, 0), - (5.0, 3.5, 1.3, 0.3, 0),(4.5, 2.3, 1.3, 0.3, 0),(4.4, 3.2, 1.3, 0.2, 0),(5.0, 3.5, 1.6, 0.6, 0), - (5.1, 3.8, 1.9, 0.4, 0),(4.8, 3.0, 1.4, 0.3, 0),(5.1, 3.8, 1.6, 0.2, 0),(4.6, 3.2, 1.4, 0.2, 0), - (5.3, 3.7, 1.5, 0.2, 0),(5.0, 3.3, 1.4, 0.2, 0),(7.0, 3.2, 4.7, 1.4, 1),(6.4, 3.2, 4.5, 1.5, 1), - (6.9, 3.1, 4.9, 1.5, 1),(5.5, 2.3, 4.0, 1.3, 1),(6.5, 2.8, 4.6, 1.5, 1),(5.7, 2.8, 4.5, 1.3, 1), - (6.3, 3.3, 4.7, 1.6, 1),(4.9, 2.4, 3.3, 1.0, 1),(6.6, 2.9, 4.6, 1.3, 1),(5.2, 2.7, 3.9, 1.4, 1), - (5.0, 2.0, 3.5, 1.0, 1),(5.9, 3.0, 4.2, 1.5, 1),(6.0, 2.2, 4.0, 1.0, 1),(6.1, 2.9, 4.7, 1.4, 1), - (5.6, 2.9, 3.6, 1.3, 1),(6.7, 3.1, 4.4, 1.4, 1),(5.6, 3.0, 4.5, 1.5, 1),(5.8, 2.7, 4.1, 1.0, 1), - (6.2, 2.2, 4.5, 1.5, 1),(5.6, 2.5, 3.9, 1.1, 1),(5.9, 3.2, 4.8, 1.8, 1),(6.1, 2.8, 4.0, 1.3, 1), - (6.3, 2.5, 4.9, 1.5, 1),(6.1, 2.8, 4.7, 1.2, 1),(6.4, 2.9, 4.3, 1.3, 1),(6.6, 3.0, 4.4, 1.4, 1), - (6.8, 2.8, 4.8, 1.4, 1),(6.7, 3.0, 5.0, 1.7, 1),(6.0, 2.9, 4.5, 1.5, 1),(5.7, 2.6, 3.5, 1.0, 1), - (5.5, 2.4, 3.8, 1.1, 1),(5.5, 2.4, 3.7, 1.0, 1),(5.8, 2.7, 3.9, 1.2, 1),(6.0, 2.7, 5.1, 1.6, 1), - (5.4, 3.0, 4.5, 1.5, 1),(6.0, 3.4, 4.5, 1.6, 1),(6.7, 3.1, 4.7, 1.5, 1),(6.3, 2.3, 4.4, 1.3, 1), - (5.6, 3.0, 4.1, 1.3, 1),(5.5, 2.5, 4.0, 1.3, 1),(5.5, 2.6, 4.4, 1.2, 1),(6.1, 3.0, 4.6, 1.4, 1), - (5.8, 2.6, 4.0, 1.2, 1),(5.0, 2.3, 3.3, 1.0, 1),(5.6, 2.7, 4.2, 1.3, 1),(5.7, 3.0, 4.2, 1.2, 1), - (5.7, 2.9, 4.2, 1.3, 1),(6.2, 2.9, 4.3, 1.3, 1),(5.1, 2.5, 3.0, 1.1, 1),(5.7, 2.8, 4.1, 1.3, 1), - (6.3, 3.3, 6.0, 2.5, 2),(5.8, 2.7, 5.1, 1.9, 2),(7.1, 3.0, 5.9, 2.1, 2),(6.3, 2.9, 5.6, 1.8, 2), - (6.5, 3.0, 5.8, 2.2, 2),(7.6, 3.0, 6.6, 2.1, 2),(4.9, 2.5, 4.5, 1.7, 2),(7.3, 2.9, 6.3, 1.8, 2), - (6.7, 2.5, 5.8, 1.8, 2),(7.2, 3.6, 6.1, 2.5, 2),(6.5, 3.2, 5.1, 2.0, 2),(6.4, 2.7, 5.3, 1.9, 2), - (6.8, 3.0, 5.5, 2.1, 2),(5.7, 2.5, 5.0, 2.0, 2),(5.8, 2.8, 5.1, 2.4, 2),(6.4, 3.2, 5.3, 2.3, 2), - (6.5, 3.0, 5.5, 1.8, 2),(7.7, 3.8, 6.7, 2.2, 2),(7.7, 2.6, 6.9, 2.3, 2),(6.0, 2.2, 5.0, 1.5, 2), - (6.9, 3.2, 5.7, 2.3, 2),(5.6, 2.8, 4.9, 2.0, 2),(7.7, 2.8, 6.7, 2.0, 2),(6.3, 2.7, 4.9, 1.8, 2), - (6.7, 3.3, 5.7, 2.1, 2),(7.2, 3.2, 6.0, 1.8, 2),(6.2, 2.8, 4.8, 1.8, 2),(6.1, 3.0, 4.9, 1.8, 2), - (6.4, 2.8, 5.6, 2.1, 2),(7.2, 3.0, 5.8, 1.6, 2),(7.4, 2.8, 6.1, 1.9, 2),(7.9, 3.8, 6.4, 2.0, 2), - (6.4, 2.8, 5.6, 2.2, 2),(6.3, 2.8, 5.1, 1.5, 2),(6.1, 2.6, 5.6, 1.4, 2),(7.7, 3.0, 6.1, 2.3, 2), - (6.3, 3.4, 5.6, 2.4, 2),(6.4, 3.1, 5.5, 1.8, 2),(6.0, 3.0, 4.8, 1.8, 2),(6.9, 3.1, 5.4, 2.1, 2), - (6.7, 3.1, 5.6, 2.4, 2),(6.9, 3.1, 5.1, 2.3, 2),(5.8, 2.7, 5.1, 1.9, 2),(6.8, 3.2, 5.9, 2.3, 2), - (6.7, 3.3, 5.7, 2.5, 2),(6.7, 3.0, 5.2, 2.3, 2),(6.3, 2.5, 5.0, 1.9, 2),(6.5, 3.0, 5.2, 2.0, 2), - (6.2, 3.4, 5.4, 2.3, 2),(5.9, 3.0, 5.1, 1.8, 2); - ``` - -2. 执行分类训练操作。 - - ```sql - SET search_path="$user",public,madlib; - SET behavior_compat_options = 'bind_procedure_searchpath'; - select madlib.xgboost_sk_Classifier('public.iris', 'public.iris_model_xgbc', 'id', 'label', 'a,b,c,d', NULL, - $${'booster': ['gbtree'], 'eta': (0.1, 0.9), 'max_depth': (5,1), 'objective': ('multi:softmax',)}$$, -- 训练参数组合，如果有多个参数，请用元组或者列表的方式传入 - TRUE); -- 是否评估模型，多分类评价为精确度和kappa值；二分类评价指标为precision, recall, fscore和support；回归评价指标为mae, mse, R2squared和rmse - ``` - - xgboost支持多组参数并行训练，比如用例中eta值为0.1和0.9，最大深度为5或者1。 - - ```sql - select id, train_timestamp, source_table, y_type, metrics, features, params from iris_model_xgbc; - ``` - - 查看模型结果如下。 - - ```sql - id | train_timestamp | source_table | y_type | metrics | features | params - ----+-------------------------------+--------------+---------+----------------------------+-----------+--------------------------------------------------------------------------------- - 1 | 2020-12-14 20:15:05.904184+08 | public.iris | integer | {'acc': 1.0, 'kappa': 1.0} | {a,b,c,d} | ('objective = multi:softmax', 'eta = 0.1', 'max_depth = 5', 'booster = gbtree') - 2 | 2020-12-14 20:15:05.904184+08 | public.iris | integer | {'acc': 1.0, 'kappa': 1.0} | {a,b,c,d} | ('objective = multi:softmax', 'eta = 0.1', 'max_depth = 1', 'booster = gbtree') - 3 | 2020-12-14 20:15:05.904184+08 | public.iris | integer | {'acc': 1.0, 'kappa': 1.0} | {a,b,c,d} | ('objective = multi:softmax', 'eta = 0.9', 'max_depth = 5', 'booster = gbtree') - 4 | 2020-12-14 20:15:05.904184+08 | public.iris | integer | {'acc': 1.0, 'kappa': 1.0} | {a,b,c,d} | ('objective = multi:softmax', 'eta = 0.9', 'max_depth = 1', 'booster = gbtree') - (4 rows) - ``` - - 结果表中，记录着训练时间，特征，结果类型，所用参数等。 - - 在本示例函数输入中，eta选择为2种，max_depth选择为2种，总共4种参数组合。所以在结果中，有4行结果；在metrics列中，记录4种参数组合的训练后的评价结果。用户可以输入多种参数组合，训练后，用户可以选择合适的模型留下。 - -3. 预测结果。 - - ```sql - select madlib.xgboost_sk_predict('public.iris', 'public.iris_model_xgbc', 'public.iris_xgbc_out', 'id'); - select t1.id, prediction, label from iris as t1, iris_xgbc_out as t2 where t1.id = t2.id and prediction <> label; - ``` - - 查看结果，预测和训练结果的对比，当前不匹配的行数为0，证明分类准确性较高。 - - ```sql - id | prediction | label - ----+------------+------- - (0 rows) - ``` - -
- -## prophet算法 - -新增facebook的prophet时序预测算法。下面以时序数据为例，展示prophet算法使用。 - -1. 准备数据。 - - ```sql - DROP TABLE IF EXISTS ts_data; - CREATE TABLE ts_data(date date, value float); - - INSERT into ts_data (date, value) values - ('2016-11-29 21:20:00', 5.6),('2016-11-29 21:30:00', 5.2),('2016-11-29 21:40:00', 5.3),('2016-11-29 21:50:00', 5.3), - ('2016-11-29 22:00:00', 5.1),('2016-11-29 22:10:00', 5.8),('2016-11-29 22:20:00', 5.6),('2016-11-29 22:30:00', 5.4), - ('2016-11-29 22:40:00', 5.4),('2016-11-29 22:50:00', 5.1),('2016-11-29 23:00:00', 5.2),('2016-11-29 23:10:00', 5.9), - ('2016-11-29 23:20:00', 5.9),('2016-11-29 23:30:00', 5.1),('2016-11-29 23:40:00', 5.8),('2016-11-29 23:50:00', 6.0), - ('2016-11-30 00:00:00', 5.9),('2016-11-30 00:10:00', 5.3),('2016-11-30 00:20:00', 5.4),('2016-11-30 00:30:00', 5.1), - ('2016-11-30 00:40:00', 5.6),('2016-11-30 00:50:00', 5.7),('2016-11-30 01:00:00', 5.8),('2016-11-30 01:10:00', 5.4), - ('2016-11-30 01:20:00', 5.8),('2016-11-30 01:30:00', 5.1),('2016-11-30 01:40:00', 5.6),('2016-11-30 01:50:00', 5.6), - ('2016-11-30 02:00:00', 5.6),('2016-11-30 02:10:00', 5.9),('2016-11-30 02:20:00', 5.7),('2016-11-30 02:30:00', 5.4), - ('2016-11-30 02:40:00', 5.6),('2016-11-30 02:50:00', 5.4),('2016-11-30 03:00:00', 5.1),('2016-11-30 03:10:00', 5.0), - ('2016-11-30 03:20:00', 5.9),('2016-11-30 03:30:00', 5.8),('2016-11-30 03:40:00', 5.4),('2016-11-30 03:50:00', 5.7), - ('2016-11-30 04:00:00', 5.6),('2016-11-30 04:10:00', 5.9),('2016-11-30 04:20:00', 5.1),('2016-11-30 04:30:00', 5.8), - ('2016-11-30 04:40:00', 5.5),('2016-11-30 04:50:00', 5.1),('2016-11-30 05:00:00', 5.8),('2016-11-30 05:10:00', 5.5), - ('2016-11-30 05:20:00', 5.7),('2016-11-30 05:30:00', 5.2),('2016-11-30 05:40:00', 5.7),('2016-11-30 05:50:00', 6.0), - ('2016-11-30 06:00:00', 5.8),('2016-11-30 06:10:00', 5.6),('2016-11-30 06:20:00', 5.2),('2016-11-30 06:30:00', 5.8), - ('2016-11-30 06:40:00', 5.3),('2016-11-30 06:50:00', 5.4),('2016-11-30 07:00:00', 5.8),('2016-11-30 07:10:00', 5.2), - ('2016-11-30 07:20:00', 5.3),('2016-11-30 07:30:00', 5.3),('2016-11-30 07:40:00', 5.8),('2016-11-30 07:50:00', 5.9), - ('2016-11-30 08:00:00', 5.6),('2016-11-30 08:10:00', 5.2),('2016-11-30 08:20:00', 5.4),('2016-11-30 08:30:00', 5.6), - ('2016-11-30 08:40:00', 6.0),('2016-11-30 08:50:00', 5.4),('2016-11-30 09:00:00', 6.0),('2016-11-30 09:10:00', 5.1), - ('2016-11-30 09:20:00', 5.1),('2016-11-30 09:30:00', 5.5),('2016-11-30 09:40:00', 5.6),('2016-11-30 09:50:00', 5.0), - ('2016-11-30 10:00:00', 5.1),('2016-11-30 10:10:00', 5.7),('2016-11-30 10:20:00', 5.4),('2016-11-30 10:30:00', 5.4), - ('2016-11-30 10:40:00', 5.7),('2016-11-30 10:50:00', 5.2),('2016-11-30 11:00:00', 5.4),('2016-11-30 11:10:00', 5.3), - ('2016-11-30 11:20:00', 5.6),('2016-11-30 11:30:00', 5.0),('2016-11-30 11:40:00', 5.2),('2016-11-30 11:50:00', 5.2), - ('2016-11-30 12:00:00', 5.5),('2016-11-30 12:10:00', 5.1),('2016-11-30 12:20:00', 5.7),('2016-11-30 12:30:00', 5.4), - ('2016-11-30 12:40:00', 5.2),('2016-11-30 12:50:00', 5.5),('2016-11-30 13:00:00', 5.0),('2016-11-30 13:10:00', 5.5), - ('2016-11-30 13:20:00', 5.6),('2016-11-30 13:30:00', 5.3),('2016-11-30 13:40:00', 5.5),('2016-11-30 13:50:00', 5.9), - ('2016-11-30 14:00:00', 10.9),('2016-11-30 14:10:00', 10.6),('2016-11-30 14:20:00', 10.3),('2016-11-30 14:30:00', 11.0), - ('2016-11-30 14:40:00', 10.0),('2016-11-30 14:50:00', 10.1),('2016-11-30 15:00:00', 10.2),('2016-11-30 15:10:00', 10.2), - ('2016-11-30 15:20:00', 10.3),('2016-11-30 15:30:00', 10.1),('2016-11-30 15:40:00', 10.9),('2016-11-30 15:50:00', 10.1), - ('2016-11-30 16:00:00', 11.0),('2016-11-30 16:10:00', 10.2),('2016-11-30 16:20:00', 10.7),('2016-11-30 16:30:00', 10.2), - ('2016-11-30 16:40:00', 10.2),('2016-11-30 16:50:00', 10.2),('2016-11-30 17:00:00', 10.8),('2016-11-30 17:10:00', 10.6), - ('2016-11-30 17:20:00', 10.5),('2016-11-30 17:30:00', 10.7),('2016-11-30 17:40:00', 10.9),('2016-11-30 17:50:00', 10.9), - ('2016-11-30 18:00:00', 10.1),('2016-11-30 18:10:00', 10.3),('2016-11-30 18:20:00', 10.1),('2016-11-30 18:30:00', 10.6), - ('2016-11-30 18:40:00', 10.3),('2016-11-30 18:50:00', 10.8),('2016-11-30 19:00:00', 10.9),('2016-11-30 19:10:00', 10.8), - ('2016-11-30 19:20:00', 10.6),('2016-11-30 19:30:00', 11.0),('2016-11-30 19:40:00', 10.3),('2016-11-30 19:50:00', 10.9), - ('2016-11-30 20:00:00', 10.6),('2016-11-30 20:10:00', 10.6),('2016-11-30 20:20:00', 10.5),('2016-11-30 20:30:00', 10.4), - ('2016-11-30 20:40:00', 10.9),('2016-11-30 20:50:00', 10.9),('2016-11-30 21:00:00', 10.7),('2016-11-30 21:10:00', 10.6), - ('2016-11-30 21:20:00', 10.5),('2016-11-30 21:30:00', 10.8),('2016-11-30 21:40:00', 10.4),('2016-11-30 21:50:00', 10.0), - ('2016-11-30 22:00:00', 10.6),('2016-11-30 22:10:00', 10.6),('2016-11-30 22:20:00', 10.6),('2016-11-30 22:30:00', 10.1), - ('2016-11-30 22:40:00', 10.4),('2016-11-30 22:50:00', 10.8),('2016-11-30 23:00:00', 10.4),('2016-11-30 23:10:00', 10.6), - ('2016-11-30 23:20:00', 10.1),('2016-11-30 23:30:00', 10.2),('2016-11-30 23:40:00', 10.6),('2016-11-30 23:50:00', 10.8), - ('2016-12-01 00:00:00', 10.6),('2016-12-01 00:10:00', 10.2),('2016-12-01 00:20:00', 10.9),('2016-12-01 00:30:00', 10.3), - ('2016-12-01 00:40:00', 10.3),('2016-12-01 00:50:00', 10.1),('2016-12-01 01:00:00', 10.7),('2016-12-01 01:10:00', 10.5), - ('2016-12-01 01:20:00', 10.4),('2016-12-01 01:30:00', 10.7),('2016-12-01 01:40:00', 10.5),('2016-12-01 01:50:00', 10.7), - ('2016-12-01 02:00:00', 10.8),('2016-12-01 02:10:00', 10.9),('2016-12-01 02:20:00', 10.9),('2016-12-01 02:30:00', 10.1), - ('2016-12-01 02:40:00', 10.4),('2016-12-01 02:50:00', 10.7),('2016-12-01 03:00:00', 10.7),('2016-12-01 03:10:00', 10.5), - ('2016-12-01 03:20:00', 10.2),('2016-12-01 03:30:00', 10.2),('2016-12-01 03:40:00', 10.8),('2016-12-01 03:50:00', 10.2), - ('2016-12-01 04:00:00', 10.9),('2016-12-01 04:10:00', 10.4),('2016-12-01 04:20:00', 10.6),('2016-12-01 04:30:00', 11.0), - ('2016-12-01 04:40:00', 10.4),('2016-12-01 04:50:00', 10.3),('2016-12-01 05:00:00', 10.7),('2016-12-01 05:10:00', 10.6), - ('2016-12-01 05:20:00', 10.9),('2016-12-01 05:30:00', 11.0),('2016-12-01 05:40:00', 10.9),('2016-12-01 05:50:00', 10.0), - ('2016-12-01 06:00:00', 10.8),('2016-12-01 06:10:00', 10.0),('2016-12-01 06:20:00', 10.1),('2016-12-01 06:30:00', 10.5), - ('2016-12-01 06:40:00', 15.5),('2016-12-01 06:50:00', 15.7),('2016-12-01 07:00:00', 15.1),('2016-12-01 07:10:00', 15.6), - ('2016-12-01 07:20:00', 15.5),('2016-12-01 07:30:00', 15.4),('2016-12-01 07:40:00', 15.7),('2016-12-01 07:50:00', 15.6), - ('2016-12-01 08:00:00', 15.3),('2016-12-01 08:10:00', 15.6),('2016-12-01 08:20:00', 15.1),('2016-12-01 08:30:00', 15.6), - ('2016-12-01 08:40:00', 15.9),('2016-12-01 08:50:00', 16.0),('2016-12-01 09:00:00', 15.4),('2016-12-01 09:10:00', 15.0), - ('2016-12-01 09:20:00', 15.0),('2016-12-01 09:30:00', 15.4),('2016-12-01 09:40:00', 15.9),('2016-12-01 09:50:00', 15.6), - ('2016-12-01 10:00:00', 15.7),('2016-12-01 10:10:00', 15.4),('2016-12-01 10:20:00', 15.2),('2016-12-01 10:30:00', 15.2), - ('2016-12-01 10:40:00', 15.8),('2016-12-01 10:50:00', 15.4),('2016-12-01 11:00:00', 16.0),('2016-12-01 11:10:00', 15.9), - ('2016-12-01 11:20:00', 15.1),('2016-12-01 11:30:00', 15.0),('2016-12-01 11:40:00', 15.0),('2016-12-01 11:50:00', 15.4), - ('2016-12-01 12:00:00', 15.5),('2016-12-01 12:10:00', 15.3),('2016-12-01 12:20:00', 16.0),('2016-12-01 12:30:00', 15.1), - ('2016-12-01 12:40:00', 15.5),('2016-12-01 12:50:00', 16.0),('2016-12-01 13:00:00', 15.7),('2016-12-01 13:10:00', 15.9), - ('2016-12-01 13:20:00', 15.4),('2016-12-01 13:30:00', 15.3),('2016-12-01 13:40:00', 15.9),('2016-12-01 13:50:00', 15.8), - ('2016-12-01 14:00:00', 15.4),('2016-12-01 14:10:00', 15.9),('2016-12-01 14:20:00', 15.3),('2016-12-01 14:30:00', 16.0), - ('2016-12-01 14:40:00', 15.5),('2016-12-01 14:50:00', 15.0),('2016-12-01 15:00:00', 15.1),('2016-12-01 15:10:00', 16.0), - ('2016-12-01 15:20:00', 15.8),('2016-12-01 15:30:00', 15.9),('2016-12-01 15:40:00', 15.4),('2016-12-01 15:50:00', 15.1), - ('2016-12-01 16:00:00', 15.8),('2016-12-01 16:10:00', 15.2),('2016-12-01 16:20:00', 15.4),('2016-12-01 16:30:00', 15.8), - ('2016-12-01 16:40:00', 15.8),('2016-12-01 16:50:00', 15.1),('2016-12-01 17:00:00', 15.3),('2016-12-01 17:10:00', 15.6), - ('2016-12-01 17:20:00', 15.3),('2016-12-01 17:30:00', 15.8),('2016-12-01 17:40:00', 15.0),('2016-12-01 17:50:00', 15.3), - ('2016-12-01 18:00:00', 15.5),('2016-12-01 18:10:00', 15.4),('2016-12-01 18:20:00', 15.3),('2016-12-01 18:30:00', 15.8), - ('2016-12-01 18:40:00', 15.2),('2016-12-01 18:50:00', 15.9),('2016-12-01 19:00:00', 15.4),('2016-12-01 19:10:00', 15.3), - ('2016-12-01 19:20:00', 15.1),('2016-12-01 19:30:00', 15.3),('2016-12-01 19:40:00', 15.9),('2016-12-01 19:50:00', 15.3), - ('2016-12-01 20:00:00', 15.3),('2016-12-01 20:10:00', 15.2),('2016-12-01 20:20:00', 15.0),('2016-12-01 20:30:00', 15.3), - ('2016-12-01 20:40:00', 15.1),('2016-12-01 20:50:00', 15.1),('2016-12-01 21:00:00', 15.6),('2016-12-01 21:10:00', 15.8), - ('2016-12-01 21:20:00', 15.4),('2016-12-01 21:30:00', 15.2),('2016-12-01 21:40:00', 16.0),('2016-12-01 21:50:00', 15.5), - ('2016-12-01 22:00:00', 15.4),('2016-12-01 22:10:00', 15.7),('2016-12-01 22:20:00', 15.3),('2016-12-01 22:30:00', 15.9), - ('2016-12-01 22:40:00', 15.9),('2016-12-01 22:50:00', 15.2),('2016-12-01 23:00:00', 15.8),('2016-12-01 23:10:00', 15.9), - ('2016-12-01 23:20:00', 20.9),('2016-12-01 23:30:00', 20.4),('2016-12-01 23:40:00', 20.3),('2016-12-01 23:50:00', 20.1), - ('2016-12-02 00:00:00', 20.7),('2016-12-02 00:10:00', 20.7),('2016-12-02 00:20:00', 20.5),('2016-12-02 00:30:00', 20.4), - ('2016-12-02 00:40:00', 20.4),('2016-12-02 00:50:00', 20.1),('2016-12-02 01:00:00', 20.2),('2016-12-02 01:10:00', 20.9), - ('2016-12-02 01:20:00', 20.6),('2016-12-02 01:30:00', 20.0),('2016-12-02 01:40:00', 20.4),('2016-12-02 01:50:00', 20.2), - ('2016-12-02 02:00:00', 20.6),('2016-12-02 02:10:00', 20.4),('2016-12-02 02:20:00', 20.5),('2016-12-02 02:30:00', 20.4), - ('2016-12-02 02:40:00', 20.5),('2016-12-02 02:50:00', 20.7),('2016-12-02 03:00:00', 20.2),('2016-12-02 03:10:00', 20.2), - ('2016-12-02 03:20:00', 20.1),('2016-12-02 03:30:00', 20.5),('2016-12-02 03:40:00', 20.5),('2016-12-02 03:50:00', 20.0), - ('2016-12-02 04:00:00', 20.7),('2016-12-02 04:10:00', 20.8),('2016-12-02 04:20:00', 20.6),('2016-12-02 04:30:00', 20.4), - ('2016-12-02 04:40:00', 20.5),('2016-12-02 04:50:00', 20.8),('2016-12-02 05:00:00', 20.1),('2016-12-02 05:10:00', 20.9), - ('2016-12-02 05:20:00', 20.5),('2016-12-02 05:30:00', 20.4),('2016-12-02 05:40:00', 20.2),('2016-12-02 05:50:00', 20.4), - ('2016-12-02 06:00:00', 20.8),('2016-12-02 06:10:00', 20.7),('2016-12-02 06:20:00', 20.9),('2016-12-02 06:30:00', 20.1), - ('2016-12-02 06:40:00', 20.3),('2016-12-02 06:50:00', 20.2),('2016-12-02 07:00:00', 20.4),('2016-12-02 07:10:00', 20.7), - ('2016-12-02 07:20:00', 20.4),('2016-12-02 07:30:00', 20.8),('2016-12-02 07:40:00', 20.8),('2016-12-02 07:50:00', 20.1), - ('2016-12-02 08:00:00', 20.3),('2016-12-02 08:10:00', 20.7),('2016-12-02 08:20:00', 20.9),('2016-12-02 08:30:00', 21.0), - ('2016-12-02 08:40:00', 20.2),('2016-12-02 08:50:00', 20.5),('2016-12-02 09:00:00', 20.2),('2016-12-02 09:10:00', 20.8), - ('2016-12-02 09:20:00', 20.9),('2016-12-02 09:30:00', 20.5),('2016-12-02 09:40:00', 20.9),('2016-12-02 09:50:00', 20.7), - ('2016-12-02 10:00:00', 20.3),('2016-12-02 10:10:00', 21.0),('2016-12-02 10:20:00', 20.5),('2016-12-02 10:30:00', 20.3), - ('2016-12-02 10:40:00', 20.2),('2016-12-02 10:50:00', 20.3),('2016-12-02 11:00:00', 20.4),('2016-12-02 11:10:00', 20.4), - ('2016-12-02 11:20:00', 21.0),('2016-12-02 11:30:00', 20.3),('2016-12-02 11:40:00', 20.3),('2016-12-02 11:50:00', 20.9), - ('2016-12-02 12:00:00', 20.8),('2016-12-02 12:10:00', 20.9),('2016-12-02 12:20:00', 20.7),('2016-12-02 12:30:00', 20.7); - ``` - -2. 执行训练操作： - - ```sql - SET search_path="$user",public,madlib; - SET behavior_compat_options = 'bind_procedure_searchpath'; - select madlib.prophet_fit('public.ts_data', 'public.prophet_model', - $${'ds': 'date', 'y': 'value'}$$, -- 列名映射， prophet要求时间列名必须为'ds'， 时序值列名'y' - $${'growth': 'linear', 'changepoints': ['2016-11-30 05:40:00']}$$ -- 训练参数组合，如果有多个参数，请用元组方式传入 - ); - ``` - - 查询模型表： - - ```sql - select id, y_type, params from public.prophet_model; - - id | y_type | params - ----+------------------+--------------------------------------------------------------- - 1 | double precision | {'changepoints': ['2016-11-30 05:40:00'], 'growth': 'linear'} - ``` - - 在模型表中，记录着训练时间，结果类型，所用参数等。 - -3. 执行预测操作 - - ```sql - select madlib.prophet_predict('public.prophet_model','public.prophet_output', 10, '10T'); - ``` - - 查看预测结果： - - ```sql - select ds, yhat, yhat_lower, yhat_upper from public.prophet_output; - - ds | yhat | yhat_lower | yhat_upper - ------------+---------------+---------------+--------------- - 2016-12-02 | 20.6943848045 | 17.7671496048 | 23.4160694837 - 2016-12-02 | 20.7408355633 | 17.9264413164 | 23.6426403933 - 2016-12-02 | 20.7872863221 | 17.9298207895 | 23.4548814727 - 2016-12-02 | 20.833737081 | 18.234443228 | 23.5317342873 - 2016-12-02 | 20.8801878398 | 18.2471709649 | 23.8345735574 - 2016-12-02 | 20.9266385986 | 18.1780101465 | 23.696087927 - 2016-12-02 | 20.9730893575 | 18.4292088648 | 23.7209823631 - 2016-12-02 | 21.0195401163 | 18.2623494126 | 23.7341427068 - 2016-12-02 | 21.0659908751 | 18.1173966769 | 23.7919478206 - 2016-12-02 | 21.112441634 | 18.5018042056 | 23.9508963879 - (10 rows) - ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-5-troubleshooting.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-5-troubleshooting.md deleted file mode 100644 index a5bd8017dd6a2dda86f555d34ea8e1c26f46eaad..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/7-deepsql/7-5-troubleshooting.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: 常见问题处理 -summary: 常见问题处理 -author: Guo Huan -date: 2021-05-19 ---- - -# 常见问题处理 - -- **问题描述**：编译数据库时，提示python模块，“can not be used when making a shared object；recompile with -fPIC”或者 “libpython22.7.a: could not read symbols: Bad value”。 - - **处理方式**： - - 1. 请检查python版本和环境变量。 - 2. 查看是否安装python-devel，或者编译python时，是否启用了-enable-shared。
- -- **问题描述**：执行gdb或者gstack命令，报错 “gdb: symbol lookup error: gdb: undefined symbol: PyUnicodeUCS4_FromEncodedObject”。 - - **处理方式**：这个问题一般发生在自行编译python2的环境上，Python2在编译安装时可以通过参数 -enable-unicode=ucs2 或 -enable-unicode=ucs4分别指定使用2个字节或者4个字节表示一个unicode字符，python2缺省使用-enable-unicode=ucs2。Python3默认使用4个字节表示一个unicode字符。 - - 可以在系统中自带的python2下执行：“import sys; print sys.maxunicode”并查看结果，如果结果是65535，说明系统默认的是ucs2；如果结果是1114111，说明用的ucs4编码。 - - 自行编译python2时，如果系统中内置的python2使用的ucs4，系统中的gdb也会依赖ucs4。因此自行编译的python2在configure时，需要添加-enable-unicode=ucs4。
- -- **问题描述**：在kmeans等算法里，报错“Data table does not exist”。 - - **处理方式**：算法所在的schema和输入表不在一个schema下，可以设置SET behavior_compat_options = 'bind_procedure_searchpath';解决这个问题。
- -- **问题描述**：python启动报错，或者import报错。 - - **处理方式：** - - 1. 检查环境变量比如PYTHONHOME，PYTHONPATH。 - 2. 安装必备依赖包。
- -- **问题描述**：Regression等算法报错“ERROR: spiexceptions.UndefinedFunction: operator does not exist: json ->> unknown.”。 - - **处理方式**：数据库不支持json导出功能，不支持此功能。
- -- **问题描述**：MADlib中进行编译时，如果使用make -sj，会遇到boost相关的报错。例如，“fatal error: boost/mpl/if.hpp: No such file or directory”。 - - **处理方式**：非问题，MADlib编译时，会先解压这几个安装包。如果是并行编译，会出现一边编译，一边解压的情况。如果编译用到这个文件，另一边还没有解压完成，会出现这类报错。再次执行make -sj即可解决。
- -- **问题描述**：执行./madpack 安装时，遇到报错：“ERROR : Failed to connect to database”。 - - **处理方式**：需要排查数据库是否启动，目标库是否存在，数据库端口是否被占用，安装用户是否具有管理员权限。另外执行madpack安装时，IP请使用127.0.0.1，不要使用localhost，否则也会出现连接失败的情况。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-3-db4ai-query-for-model-training-and-prediction.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-3-db4ai-query-for-model-training-and-prediction.md deleted file mode 100644 index 38eab31f690ffa5b9aa1600e96d8bf0bba4176ec..0000000000000000000000000000000000000000 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-3-db4ai-query-for-model-training-and-prediction.md +++ /dev/null @@ -1,420 +0,0 @@ ---- -title: DB4AI-Query：模型训练和推断 -summary: DB4AI-Query：模型训练和推断 -author: Guo Huan -date: 2021-10-20 ---- - -# DB4AI-Query：模型训练和推断 - -MogDB当前版本支持了原生DB4AI能力，通过引入原生AI算子，简化操作流程，充分利用数据库优化器、执行器的优化与执行能力，获得高性能的数据库内模型训练能力。更简化的模型训练与预测流程、更高的性能表现，让开发者在更短时间内能更专注于模型的调优与数据分析上，而避免了碎片化的技术栈与冗余的代码实现。 - -## 关键字解析 - -**表 1** DB4AI语法及关键字 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
名称描述
语句CREATE MODEL创建模型并进行训练，同时保存模型。
PREDICT BY利用已有模型进行推断。
关键字TARGET训练/推断任务的目标列名。
FEATURES训练/推断任务的数据特征列名。
MODEL训练任务的模型名称。
- -## 使用指导 - -1. 本版本支持的算法概述。 - - 当前版本的DB4AI支持基于SGD算子的逻辑回归（目前支持二分类任务）、线性回归和支持向量机算法（分类任务），以及基于K-Means算子的Kmeans聚类算法。 - -2. 模型训练语法说明。 - - - CREATE MODEL - - 使用“CREATE MODEL”语句可以进行模型的创建和训练。模型训练SQL语句，现有一个数据集为kmeans_2d，该表的数据内容如下： - - ```sql - mogdb=# select * from kmeans_2d; - id | position - ----+------------------------------------- - 1 | {74.5268815685995,88.2141939294524} - 2 | {70.9565760521218,98.8114827475511} - 3 | {76.2756086327136,23.8387574302033} - 4 | {17.8495847294107,81.8449544720352} - 5 | {81.2175785354339,57.1677675866522} - 6 | {53.97752255667,49.3158342130482} - 7 | {93.2475341879763,86.934042100329} - 8 | {72.7659293473698,19.7020415100269} - 9 | {16.5800288529135,75.7475957670249} - 10 | {81.8520747194998,40.3476078575477} - 11 | {76.796671198681,86.3827232690528} - 12 | {59.9231450678781,90.9907738864422} - 13 | {70.161884885747,19.7427458665334} - 14 | {11.1269539105706,70.9988166182302} - 15 | {80.5005071521737,65.2822235273197} - 16 | {54.7030725912191,52.151339428965} - 17 | {103.059707058128,80.8419883321039} - 18 | {85.3574452036992,14.9910179991275} - 19 | {28.6501615960151,76.6922890325077} - 20 | {69.7285806713626,49.5416352967732} - (20 rows) - ``` - - 该表的字段position的数据类型为double precision[]。 - - - 以Kmeans为例，训练一个模型。从kmeans_2d训练集中指定position为特征列，使用kmeans算法，创建并保存模型point_kmeans。 - - ```sql - mogdb=# CREATE MODEL point_kmeans USING kmeans FEATURES position FROM kmeans_2d WITH num_centroids=3; - NOTICE: Hyperparameter max_iterations takes value DEFAULT (10) - NOTICE: Hyperparameter num_centroids takes value 3 - NOTICE: Hyperparameter tolerance takes value DEFAULT (0.000010) - NOTICE: Hyperparameter batch_size takes value DEFAULT (10) - NOTICE: Hyperparameter num_features takes value DEFAULT (2) - NOTICE: Hyperparameter distance_function takes value DEFAULT (L2_Squared) - NOTICE: Hyperparameter seeding_function takes value DEFAULT (Random++) - NOTICE: Hyperparameter verbose takes value DEFAULT (0) - NOTICE: Hyperparameter seed takes value DEFAULT (0) - MODEL CREATED. PROCESSED 1 - ``` - - 上述命令中： - - - “CREATE MODEL”语句用于模型的训练和保存。 - - - USING关键字指定算法名称。 - - - FEATURES用于指定训练模模型的特征，需根据训练数据表的列名添加。 - - - TARGET指定模型的训练目标，它可以是训练所需数据表的列名，也可以是一个表达式，例如：price > 10000。 - - - WITH用于指定训练模型时的超参数。当超参未被用户进行设置的时候，框架会使用默认数值。 - - 针对不同的算子，框架支持不同的超参组合： - - **表 2** 算子支持的超参 - - | 算子 | 超参 | - | :----------------------------------------------------------- | :----------------------------------------------------------- | - | GD(logistic_regression、linear_regression、svm_classification) | optimizer(char\*); verbose(bool); max_iterations(int); max_seconds(double); batch_size(int); learning_rate(double); decay(double); tolerance(double)
其中，SVM限定超参lambda(double) | - | Kmeans | max_iterations(int); num_centroids(int); tolerance(double); batch_size(int); num_features(int); distance_function(char); seeding_function(char*); verbose(int);seed(int) | - - 当前各个超参数设置的默认值和取值范围如下： - - **表 3** 超参的默认值以及取值范围 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
算子超参（默认值）取值范围超参描述
GD (logistic_regression、linear_regression、svm_classification)optimizer = gd（梯度下降法）gd/ngd（自然梯度下降）优化器
verbose = falseT/F日志显示
max_iterations = 100(0, INT_MAX_VALUE]最大迭代次数
max_seconds = 0（不对运行时长设限制）[0,INT_MAX_VALUE]运行时长
batch_size = 1000(0, MAX_MEMORY_LIMIT]一次训练所选取的样本数
learning_rate = 0.8(0, DOUBLE_MAX_VALUE]学习率
decay = 0.95(0, DOUBLE_MAX_VALUE]权值衰减率
tolerance = 0.0005(0, DOUBLE_MAX_VALUE]公差
seed = 0（对seed取随机值）[0, INT_MAX_VALUE]种子
just for SVM：lambda = 0.01(0, DOUBLE_MAX_VALUE)正则化参数
Kmeansmax_iterations = 10[1, INT_MAX_VALUE]最大迭代次数
num_centroids = 10[1, MAX_MEMORY_LIMIT]簇的数目
tolerance = 0.00001(0,1)中心点误差
batch_size = 10[1, MAX_MEMORY_LIMIT]一次训练所选取的样本数
num_features = 2[1, GS_MAX_COLS]输入样本特征数
distance_function = "L2_Squared"L1, L2, L2_Squared, or Linf正则化方法
seeding_function = "Random++""Random++" or "KMeans||"初始化种子点方法
verbose = 0U{ 0, 1, 2 }冗长模式
seed = 0U[0, INT_MAX_VALUE]种子
MAX_MEMORY_LIMIT = 最大内存加载的元组数量
GS_MAX_COLS = 数据库单表最大属性数量
- - - 模型保存成功，则返回创建成功信息： - - ```sql - MODEL CREATED. PROCESSED x - ``` - -3. 查看模型信息。 - - 当训练完成后模型会被存储到系统表gs_model_warehouse中。系统表gs_model_warehouse可以查看到关于模型本身和训练过程的相关信息。 - - 用户可以通过查看系统表的方式查看模型，例如查看模型名为”point_kmeans”的SQL语句如下： - - ```sql - mogdb=# select * from gs_model_warehouse where modelname='point_kmeans'; - -[ RECORD 1 ]---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - modelname | point_kmeans - modelowner | 10 - createtime | 2021-04-30 17:30:39.59044 - processedtuples | 20 - discardedtuples | 0 - pre_process_time | 6.2001e-05 - exec_time | .000185272 - iterations | 5 - outputtype | 23 - modeltype | kmeans - query | CREATE MODEL point_kmeans USING kmeans FEATURES position FROM kmeans_2d WITH num_centroids=3; - modeldata | - weight | - hyperparametersnames | {max_iterations,num_centroids,tolerance,batch_size,num_features,distance_function,seeding_function,verbose,seed} - hyperparametersvalues | {10,3,1e-05,10,2,L2_Squared,Random++,0,0} - hyperparametersoids | {23,23,701,23,23,1043,1043,23,23} - coefnames | {original_num_centroids,actual_num_centroids,dimension,distance_function_id,seed,coordinates} - coefvalues | {3,3,2,2,572368998,"(77.282589,23.724434)(74.421616,73.239455)(18.551682,76.320914)"} - coefoids | - trainingscoresname | - trainingscoresvalue | - modeldescribe | {"id:1,objective_function:542.851169,avg_distance_to_centroid:108.570234,min_distance_to_centroid:1.027078,max_distance_to_centroid:297.210108,std_dev_distance_to_centroid:105.053257,cluster_size:5","id:2,objective_function:5825.982139,avg_distance_to_centroid:529.634740,min_distance_to_centroid:100.270449,max_distance_to_centroid:990.300588,std_dev_distance_to_centroid:285.915094,cluster_size:11","id:3,objective_function:220.792591,avg_distance_to_centroid:55.198148,min_distance_to_centroid:4.216111,max_distance_to_centroid:102.117204,std_dev_distance_to_centroid:39.319118,cluster_size:4"} - ``` - -4. 利用已存在的模型做推断任务。 - - 使用“SELECT”和“PREDICT BY”关键字利用已有模型完成推断任务。 - - 查询语法：SELECT…PREDICT BY…(FEATURES…)…FROM…; - - ```sql - mogdb=# SELECT id, PREDICT BY point_kmeans (FEATURES position) as pos FROM (select * from kmeans_2d limit 10); - id | pos - ----+----- - 1 | 2 - 2 | 2 - 3 | 1 - 4 | 3 - 5 | 2 - 6 | 2 - 7 | 2 - 8 | 1 - 9 | 3 - 10 | 1 - (10 rows) - ``` - - 针对相同的推断任务，同一个模型的结果是稳定的。且基于相同的超参数和训练集训练的模型也具有稳定性，同时AI模型训练存在随机成分（每个batch的数据分布、随机梯度下降），所以不同的模型间的计算表现、结果允许存在小的差别。 - -5. 查看执行计划。 - - 使用explain语句可对“CREATE MODEL”和“PREDICT BY”的模型训练或预测过程中的执行计划进行分析。Explain关键字后可直接拼接CREATE MODEL/ PREDICT BY语句（子句），也可接可选的参数，支持的参数如下： - - **表 4** EXPLAIN支持的参数 - - | 参数名 | 描述 | - | :-------- | :--------------------------------------------- | - | ANALYZE | 布尔型变量，追加运行时间、循环次数等描述信息 | - | VERBOSE | 布尔型变量，控制训练的运行信息是否输出到客户端 | - | COSTS | 布尔型变量 | - | CPU | 布尔型变量 | - | DETAIL | 布尔型变量，不可用。 | - | NODES | 布尔型变量，不可用 | - | NUM_NODES | 布尔型变量，不可用 | - | BUFFERS | 布尔型变量 | - | TIMING | 布尔型变量 | - | PLAN | 布尔型变量 | - | FORMAT | 可选格式类型：TEXT / XML / JSON / YAML | - - 示例： - - ```sql - mogdb=# Explain CREATE MODEL patient_logisitic_regression USING logistic_regression FEATURES second_attack, treatment TARGET trait_anxiety > 50 FROM patients WITH batch_size=10, learning_rate = 0.05; - NOTICE: Hyperparameter batch_size takes value 10 - NOTICE: Hyperparameter decay takes value DEFAULT (0.950000) - NOTICE: Hyperparameter learning_rate takes value 0.050000 - NOTICE: Hyperparameter max_iterations takes value DEFAULT (100) - NOTICE: Hyperparameter max_seconds takes value DEFAULT (0) - NOTICE: Hyperparameter optimizer takes value DEFAULT (gd) - NOTICE: Hyperparameter tolerance takes value DEFAULT (0.000500) - NOTICE: Hyperparameter seed takes value DEFAULT (0) - NOTICE: Hyperparameter verbose takes value DEFAULT (FALSE) - NOTICE: GD shuffle cache size 212369 - QUERY PLAN - ------------------------------------------------------------------- - Gradient Descent (cost=0.00..0.00 rows=0 width=0) - -> Seq Scan on patients (cost=0.00..32.20 rows=1776 width=12) - (2 rows) - ``` - -6. 异常场景。 - - - 训练阶段。 - - - 场景一：当超参数的设置超出取值范围，模型训练失败，返回ERROR，并提示错误，例如： - - ```sql - mogdb=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET trait_anxiety FROM patients WITH optimizer='aa'; - NOTICE: Hyperparameter batch_size takes value DEFAULT (1000) - NOTICE: Hyperparameter decay takes value DEFAULT (0.950000) - NOTICE: Hyperparameter learning_rate takes value DEFAULT (0.800000) - NOTICE: Hyperparameter max_iterations takes value DEFAULT (100) - NOTICE: Hyperparameter max_seconds takes value DEFAULT (0) - NOTICE: Hyperparameter optimizer takes value aa - ERROR: Invalid hyperparameter value for optimizer. Valid values are: gd, ngd. (default is gd) - ``` - - - 场景二：当模型名称已存在，模型保存失败，返回ERROR，并提示错误原因： - - ```sql - mogdb=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET trait_anxiety FROM patients; - NOTICE: Hyperparameter batch_size takes value DEFAULT (1000) - NOTICE: Hyperparameter decay takes value DEFAULT (0.950000) - NOTICE: Hyperparameter learning_rate takes value DEFAULT (0.800000) - NOTICE: Hyperparameter max_iterations takes value DEFAULT (100) - NOTICE: Hyperparameter max_seconds takes value DEFAULT (0) - NOTICE: Hyperparameter optimizer takes value DEFAULT (gd) - NOTICE: Hyperparameter tolerance takes value DEFAULT (0.000500) - NOTICE: Hyperparameter seed takes value DEFAULT (0) - NOTICE: Hyperparameter verbose takes value DEFAULT (FALSE) - NOTICE: GD shuffle cache size 5502 - ERROR: The model name "patient_linear_regression" already exists in gs_model_warehouse. - ``` - - - 场景三：FEATURE或者TARGETS列是*，返回ERROR，并提示错误原因： - - ```sql - mogdb=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES * TARGET trait_anxiety FROM - patients; - ERROR: FEATURES clause cannot be * - -----------------------------------------------------------------------------------------------------------------------、 - mogdb=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET * FROM patients; - ERROR: TARGET clause cannot be * - ``` - - - 场景四：对于无监督学习方法使用TARGET关键字，或者在监督学习方法中不适用TARGET关键字，均会返回ERROR，并提示错误原因： - - ```sql - mogdb=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment FROM patients; - ERROR: Supervised ML algorithms require TARGET clause - ----------------------------------------------------------------------------------------------------------------------------- - CREATE MODEL patient_linear_regression USING linear_regression TARGET trait_anxiety FROM patients; ERROR: Supervised ML algorithms require FEATURES clause - ``` - - - 场景五：当GUC参数statement_timeout设置了时长，训练超时执行的语句将被终止：执行CREATE MODEL语句。训练集的大小、训练轮数(iteration)、提前终止条件(tolerance、max_seconds)、并行线程数(nthread)等参数都会影响训练时长。当时长超过数据库限制，语句被终止模型训练失败。 - - - 推断阶段。 - - - 场景六：当模型名在系统表中查找不到，数据库会报ERROR： - - ```sql - mogdb=# select id, PREDICT BY patient_logistic_regression (FEATURES second_attack,treatment) FROM patients; - ERROR: There is no model called "patient_logistic_regression". - ``` - - - 场景七：当做推断任务FEATURES的数据维度和数据类型与训练集存在不一致，将报ERROR，并提示错误原因，例如： - - ```sql - mogdb=# select id, PREDICT BY patient_linear_regression (FEATURES second_attack) FROM patients; - ERROR: Invalid number of features for prediction, provided 1, expected 2 - CONTEXT: referenced column: patient_linear_regression_pred - ------------------------------------------------------------------------------------------------------------------------------------- - mogdb=# select id, PREDICT BY patient_linear_regression (FEATURES 1,second_attack,treatment) FROM patients; - ERROR: Invalid number of features for prediction, provided 3, expected 2 - CONTEXT: referenced column: patient_linear_regression_pre - ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-4-best-practices.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md similarity index 69% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-4-best-practices.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md index c78455c3fe64f65ee91be1e408b58a7e24f0c2bb..587c0e492c4566925b206c7ec1401c720a752073 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-4-best-practices.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md @@ -1,32 +1,32 @@ ---- -title: 最佳实践 -summary: 最佳实践 -author: Guo Huan -date: 2021-05-19 ---- - -# 最佳实践 - -相关参数解释参考表[GS_OPT_MODEL](GS_OPT_MODEL)。 - -**表 1** - -| 模型参数 | 参数建议 | -| :------------ | :----------------------------------------------------------- | -| template_name | ‘rlstm’ | -| model_name | 自定义，如‘open_ai’，需满足unique约束。 | -| datname | 所服务database名称，如‘mogdb’。 | -| ip | aiEngine-ip地址，如‘127.0.0.1’。 | -| port | aiEngine监听端口，如‘5000’。 | -| max_epoch | 迭代次数，推荐较大数值，保证收敛效果，如‘2000’。 | -| learning_rate | (0, 1]浮点数，推荐较大的学习率，助于加快收敛速度。 | -| dim_red | 特征值降维系数：
‘-1’：不采用PCA降维，全量特征；
‘（0，1] ’区间浮点数：越小，训练维度越小，收敛速度越快，但影响训练准确率。 | -| hidden_units | 特征值维度较高时，建议适度增大此参数，提高模型复杂度，如 ‘64，128……’ | -| batch_size | 根据编码数据量，较大数据量推荐适度增大此参数，加快模型收敛，如‘256，512……’ | -| 其他参数 | 参考表[GS_OPT_MODEL](GS_OPT_MODEL) | - -推荐参数配置： - -```sql -INSERT INTO gs_opt_model values('rlstm', 'open_ai', 'mogdb', '127.0.0.1', 5000, 2000，1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); -``` +--- +title: 最佳实践 +summary: 最佳实践 +author: Guo Huan +date: 2022-05-06 +--- + +# 最佳实践 + +相关参数解释参考表[GS_OPT_MODEL](GS_OPT_MODEL)。 + +**表 1** + +| 模型参数 | 参数建议 | +| :------------ | :----------------------------------------------------------- | +| template_name | ‘rlstm’ | +| model_name | 自定义，如‘open_ai’，需满足unique约束。 | +| datname | 所服务database名称，如‘postgres’。 | +| ip | aiEngine-ip地址，如‘127.0.0.1’。 | +| port | aiEngine侦听端口，如‘5000’。 | +| max_epoch | 迭代次数，推荐较大数值，保证收敛效果，如‘2000’。 | +| learning_rate | (0, 1]浮点数，推荐较大的学习率，助于加快收敛速度。 | +| dim_red | 特征值降维系数：‘-1’：不采用PCA降维，全量特征；‘（0，1] ’区间浮点数：越小，训练维度越小，收敛速度越快，但影响训练准确率。 | +| hidden_units | 特征值维度较高时，建议适度增大此参数，提高模型复杂度，如 ‘64，128……’ | +| batch_size | 根据编码数据量，较大数据量推荐适度增大此参数，加快模型收敛，如‘256，512……’ | +| 其他参数 | 参考表[GS_OPT_MODEL](GS_OPT_MODEL) | + +推荐参数配置： + +``` +INSERT INTO gs_opt_model values('rlstm', 'open_ai', 'postgres', '127.0.0.1', 5000, 2000，1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); +``` \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-2-environment-deployment.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md similarity index 78% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-2-environment-deployment.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md index 1ef4248e7330b9a4866e85dda479d17b1c24a56c..258c1fb0dbab3cbb3b7fa504629b80396dd0743c 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-2-environment-deployment.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md @@ -1,213 +1,203 @@ ---- -title: 环境部署 -summary: 环境部署 -author: Guo Huan -date: 2021-05-19 ---- - -# 环境部署 - -
- -## 前提条件 - -需要保证MogDB处于正常状态，用户通过身份验证成功登录MogDB；用户执行的SQL语法正确无报错，且不会导致数据库异常等；历史性能数据窗口内MogDB并发量稳定，表结构、表数量不变，数据量无突变，涉及查询性能的guc参数不变；进行预测时，需要保证模型已训练并收敛；AiEngine运行环境稳定。 - -
- -## 请求样例 - -AiEngine进程与内核进程使用https发送请求进行通信，请求样例如下： - -```bash -curl -X POST -d '{"modelName":"modelname"}' -H 'Content-Type: application/json' 'https://IP-address:port/request-API' -``` - -**表 1** ： AI-Engine对外接口 - -| Request-API | 功能 | -| :------------- | :--------------------- | -| /check | 检查模型是否被正常拉起 | -| /configure | 设置模型参数 | -| /train | 模型训练 | -| /track_process | 查看模型训练日志 | -| /setup | 加载历史模型 | -| /predict | 模型预测 | - -
- -## 证书生成 - -使用此功能前需使用openssl工具生成通信双方认证所需的证书，保证通信安全。 - -1. 搭建证书生成环境，证书文件保存路径为$GAUSSHOME/CA。 - - -拷贝证书生成脚本及相关文件 - - ```bash - cp path_to_predictor/install/ssl.sh $GAUSSHOME/ - cp path_to_predictor/install/ca_ext.txt $GAUSSHOME/ - ``` - - -copy 配置文件openssl.cnf到$GAUSSHOME路径下 - - ```bash - cp $GAUSSHOME/share/om/openssl.cnf $GAUSSHOME/ - ``` - - -修改openssl.conf配置参数 - - ```bash - dir = $GAUSSHOME/CA/demoCA - default_md = sha256 - ``` - - -至此证书生成环境准备完成 - -2. 生成证书及密钥。 - - ```bash - cd $GAUSSHOME - sh ssl.sh - ``` - - -根据提示设置密码，假如为Test@123： - - -密码要求至少3种不同类型字符，长度至少为8位 - - ```bash - Please enter your password: - ``` - - -根据提示输入选项： - - ```bash - Certificate Details: - Serial Number: 1 (0x1) - Validity - Not Before: May 15 08:32:44 2020 GMT - Not After : May 15 08:32:44 2021 GMT - Subject: - countryName = CN - stateOrProvinceName = SZ - organizationName = HW - organizationalUnitName = GS - commonName = CA - X509v3 extensions: - X509v3 Basic Constraints: - CA:TRUE - Certificate is to be certified until May 15 08:32:44 2021 GMT (365 days) - Sign the certificate? [y/n]:y - 1 out of 1 certificate requests certified, commit? [y/n]y - ``` - - -输入拉起AIEngine的IP地址，如IP为127.0.0.1： - - ```bash - Please enter your aiEngine IP: 127.0.0.1 - ``` - - -根据提示输入选项： - - ```bash - Certificate Details: - Serial Number: 2 (0x2) - Validity - Not Before: May 15 08:38:07 2020 GMT - Not After : May 13 08:38:07 2030 GMT - Subject: - countryName = CN - stateOrProvinceName = SZ - organizationName = HW - organizationalUnitName = GS - commonName = 127.0.0.1 - X509v3 extensions: - X509v3 Basic Constraints: - CA:FALSE - Certificate is to be certified until May 13 08:38:07 2030 GMT (3650 days) - Sign the certificate? [y/n]:y - 1 out of 1 certificate requests certified, commit? [y/n]y - ``` - - -输入启动MogDB IP地址，如IP为127.0.0.1： - - ```bash - Please enter your mogdb IP: 127.0.0.1 - ``` - - -根据提示输入选项： - - ```bash - Certificate Details: - Serial Number: 3 (0x3) - Validity - Not Before: May 15 08:41:46 2020 GMT - Not After : May 13 08:41:46 2030 GMT - Subject: - countryName = CN - stateOrProvinceName = SZ - organizationName = HW - organizationalUnitName = GS - commonName = 127.0.0.1 - X509v3 extensions: - X509v3 Basic Constraints: - CA:FALSE - Certificate is to be certified until May 13 08:41:46 2030 GMT (3650 days) - Sign the certificate? [y/n]:y - 1 out of 1 certificate requests certified, commit? [y/n]y - ``` - - -至此，相关证书及密钥已生成，（$GAUSSHOME/CA）内容如下： - - ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/environment-deployment-1.png) - -
- -## 环境准备 - -1. 将工具代码文件夹拷贝至目标环境 - - -假设安装路径为$INSTALL_FOLDER - - -假设目标环境路径为/home/ai_user : - - ```bash - scp -r $INSTALL_FOLDER/bin/dbmind/predictor ai_user@127.0.0.1:path_to_Predictor - ``` - -2. 拷贝CA证书文件夹至aiEngine环境中某路径下： - - ```bash - cp -r $GAUSSHOME/CA ai_user@127.0.0.1:path_to_CA - ``` - -3. 安装predictor/install/requirements(-gpu).txt工具： - - ```bash - 有GPU：pip install -r requirements-gpu.txt - 无GPU：pip install -r requirements.txt - ``` - -
- -## 拉起AiEngine - -1. 切换至aiEngine环境（即拷贝predictor的目标环境 ai_user）： - - 设置predictor/python/settings.py 中的相关参数： - - ```bash - DEFAULT_FLASK_SERVER_HOST = '127.0.0.1' （aiEngine运行IP地址） - DEFAULT_FLASK_SERVER_PORT = '5000' （aiEngine运行端口号） - PATH_SSL = "path_to_CA" (CA文件夹路径) - ``` - -2. 运行aiEngine启动脚本： - - ```bash - python path_to_Predictor/python/run.py - ``` - - 此时，aiEngine即在相应端口保持拉起状态，等待内核侧时间预测功能的请求指令。 - - 至此，aiEngine工具部署完成。从内核中发起执行时间预测功能指令步骤，请参考《[使用指导](2-3-usage-guide)》。 +--- +title: 环境部署 +summary: 环境部署 +author: Guo Huan +date: 2022-05-06 +--- + +# 环境部署 + +## 前提条件 + +需要保证MogDB处于正常状态，用户通过身份验证成功登录MogDB；用户执行的SQL语法正确无报错，且不会导致数据库异常等；历史性能数据窗口内MogDB并发量稳定，表结构、表数量不变，数据量无突变，涉及查询性能的guc参数不变；进行预测时，需要保证模型已训练并收敛；AiEngine运行环境稳定。 + +## 请求样例 + +AiEngine进程与内核进程使用https发送请求进行通信，请求样例如下： + +``` +curl -X POST -d '{"modelName":"modelname"}' -H 'Content-Type: application/json' 'https://IP-address:port/request-API' +``` + +**表 1** ： AI-Engine对外接口 + +| Request-API | 功能 | +| :------------- | :--------------------- | +| /check | 检查模型是否被正常拉起 | +| /configure | 设置模型参数 | +| /train | 模型训练 | +| /track_process | 查看模型训练日志 | +| /setup | 加载历史模型 | +| /predict | 模型预测 | + +## 证书生成 + +使用此功能前需使用openssl工具生成通信双方认证所需的证书，保证通信安全。 + +1. 搭建证书生成环境，证书文件保存路径为$GAUSSHOME/CA。 + + 拷贝证书生成脚本及相关文件 + + ``` + cp path_to_predictor/install/ssl.sh $GAUSSHOME/ + cp path_to_predictor/install/ca_ext.txt $GAUSSHOME/ + ``` + + copy 配置文件openssl.cnf到$GAUSSHOME路径下 + + ``` + cp $GAUSSHOME/share/om/openssl.cnf $GAUSSHOME/ + ``` + + 修改openssl.conf配置参数 + + ``` + dir = $GAUSSHOME/CA/demoCA + default_md = sha256 + ``` + + 至此证书生成环境准备完成 + +2. 生成证书及密钥 + + ``` + cd $GAUSSHOME + sh ssl.sh + ``` + + 根据提示设置密码，假如为Test@123： + + 密码要求至少3种不同类型字符，长度至少为8位 + + ``` + Please enter your password: + ``` + + 根据提示输入选项： + + ``` + Certificate Details: + Serial Number: 1 (0x1) + Validity + Not Before: May 15 08:32:44 2020 GMT + Not After : May 15 08:32:44 2021 GMT + Subject: + countryName = CN + stateOrProvinceName = SZ + organizationName = HW + organizationalUnitName = GS + commonName = CA + X509v3 extensions: + X509v3 Basic Constraints: + CA:TRUE + Certificate is to be certified until May 15 08:32:44 2021 GMT (365 days) + Sign the certificate? [y/n]:y + 1 out of 1 certificate requests certified, commit? [y/n]y + ``` + + 输入拉起AIEngine的IP地址，如IP为127.0.0.1： + + ``` + Please enter your aiEngine IP: 127.0.0.1 + ``` + + 根据提示输入选项： + + ``` + Certificate Details: + Serial Number: 2 (0x2) + Validity + Not Before: May 15 08:38:07 2020 GMT + Not After : May 13 08:38:07 2030 GMT + Subject: + countryName = CN + stateOrProvinceName = SZ + organizationName = HW + organizationalUnitName = GS + commonName = 127.0.0.1 + X509v3 extensions: + X509v3 Basic Constraints: + CA:FALSE + Certificate is to be certified until May 13 08:38:07 2030 GMT (3650 days) + Sign the certificate? [y/n]:y + 1 out of 1 certificate requests certified, commit? [y/n]y + ``` + + 输入启动MogDB IP地址，如IP为127.0.0.1： + + ``` + Please enter your mogdb IP: 127.0.0.1 + ``` + + 根据提示输入选项： + + ``` + Certificate Details: + Serial Number: 3 (0x3) + Validity + Not Before: May 15 08:41:46 2020 GMT + Not After : May 13 08:41:46 2030 GMT + Subject: + countryName = CN + stateOrProvinceName = SZ + organizationName = HW + organizationalUnitName = GS + commonName = 127.0.0.1 + X509v3 extensions: + X509v3 Basic Constraints: + CA:FALSE + Certificate is to be certified until May 13 08:41:46 2030 GMT (3650 days) + Sign the certificate? [y/n]:y + 1 out of 1 certificate requests certified, commit? [y/n]y + ``` + + 至此，相关证书及密钥已生成，（$GAUSSHOME/CA）内容如下： + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/intelligence-explain-1.png) + +## 环境准备 + +1. 将工具代码文件夹拷贝至目标环境 + + 假设安装路径为$INSTALL_FOLDER + + 假设目标环境路径为/home/ai_user : + + ``` + scp -r $INSTALL_FOLDER/bin/dbmind/predictor ai_user@127.0.0.1:path_to_Predictor + ``` + +2. 拷贝CA证书文件夹至aiEngine环境中某路径下： + + ``` + cp -r $GAUSSHOME/CA ai_user@127.0.0.1:path_to_CA + ``` + +3. 安装predictor/install/requirements(-gpu).txt工具（**该功能比较特殊，与其他AI功能不同，需要独立安装依赖**）： + + ``` + 有GPU：pip install -r requirements-gpu.txt + 无GPU：pip install -r requirements.txt + ``` + +## 拉起AiEngine + +1. 切换至aiEngine环境（即拷贝predictor的目标环境 ai_user）： + + 设置predictor/python/settings.py 中的相关参数： + + ``` + DEFAULT_FLASK_SERVER_HOST = '127.0.0.1' （aiEngine运行IP地址） + DEFAULT_FLASK_SERVER_PORT = '5000' （aiEngine运行端口号） + PATH_SSL = "path_to_CA" (CA文件夹路径) + ``` + +2. 运行aiEngine启动脚本： + + ``` + python path_to_Predictor/python/run.py + ``` + + 此时，aiEngine即在相应端口保持拉起状态，等待内核侧时间预测功能的请求指令。 + + 至此，aiEngine工具部署完成。从内核中发起执行时间预测功能指令步骤，请参考[使用指导](intelligence-explain-user-guide)。 \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-5-faqs.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md similarity index 74% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-5-faqs.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md index 4b3818c603cbc95c5b82526bcfef9830d2405fab..1d5c7ecac11c9cbf2314064047d741faea23f5f6 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-5-faqs.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md @@ -1,35 +1,35 @@ ---- -title: 常见问题处理 -summary: 常见问题处理 -author: Guo Huan -date: 2021-05-19 ---- - -# 常见问题处理 - -## AI Engine配置问题 - -- **AiEngine启动失败**：请检查ip地址，端口是否可用；CA证书路径是否存在。 -- **发起请求AiEngine无响应**：请检查通信双方CA证书是否一致。 -- **训练，测试场景失败**：请检查模型文件保存路径是否存在；训练预测文件是否在正确下载。 -- **更换AiEngine-IP地址**：按照[证书生成](2-2-environment-deployment#证书生成)步骤重新生成证书，在生成证书及密钥步骤中替换成相应的IP地址即可。 - -## 数据库内部报错问题 - -问题：AiEngine链接失败。 - -``` -ERROR: AI engine connection failed. -CONTEXT: referenced column: model_train_opt -``` - -处理方法：检查AIEngine是否正常拉起或重启AIEngine；检查通信双方CA证书是否一致；检查模型配置信息中的ip和端口是否匹配。 - -问题：模型不存在。 - -``` -ERROR: OPT_Model not found for model name XXX -CONTEXT: referenced column: track_model_train_opt -``` - -处理方法：检查[GS_OPT_MODEL](GS_OPT_MODEL)表中是否存在执行语句中“model_name”对应的模型；使用预测功能报错时，检查模型是否已被训练。 +--- +title: 常见问题处理 +summary: 常见问题处理 +author: Guo Huan +date: 2022-05-06 +--- + +# 常见问题处理 + +## AI Engine配置问题 + +- **AiEngine启动失败**：请检查ip地址，端口是否可用；CA证书路径是否存在。 +- **发起请求AiEngine无响应**：请检查通信双方CA证书是否一致。 +- **训练，测试场景失败**：请检查模型文件保存路径是否存在；训练预测文件是否在正确下载。 +- **更换AiEngine-IP地址**：按照[证书生成](intelligence-explain-environment-deployment#证书生成)步骤重新生成证书，在生成证书及密钥中替换成相应的IP地址即可。 + +## 数据库内部报错问题 + +问题：AiEngine链接失败。 + +``` +ERROR: AI engine connection failed. +CONTEXT: referenced column: model_train_opt +``` + +处理方法：检查AIEngine是否正常拉起或重启AIEngine；检查通信双方CA证书是否一致；检查模型配置信息中的ip和端口是否匹配； + +问题：模型不存在。 + +``` +ERROR: OPT_Model not found for model name XXX +CONTEXT: referenced column: track_model_train_opt +``` + +处理方法：检查[GS_OPT_MODEL](GS_OPT_MODEL)表中是否存在执行语句中“model_name”对应的模型；使用预测功能报错时，检查模型是否已被训练； diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..03ac47adbddb5f8c22d81b4e8e39741a4f84f77b --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md @@ -0,0 +1,14 @@ +--- +title: 概述 +summary: 概述 +author: Guo Huan +date: 2022-05-06 +--- + +# 概述 + +本功能名为Predictor，是基于机器学习且具有在线学习能力的查询时间预测工具。通过不断学习数据库内收集的历史执行信息，实现计划的执行时间预测功能。 + +本特性需要拉起python进程AIEngine，用于模型的训练和推理。 + +该功能所在目录为$**GAUSSHOME**/bin/dbmind/components/predictor。由于该模块中某些功能涉及到相对复杂的搭建，因此，需要用户切换到该目录中寻找对应文件，并按照本章说明进行部署。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-3-usage-guide.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md similarity index 63% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-3-usage-guide.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md index d753bdbad6c16c3333a2cd41320272341629892c..cd96f612509d10a1b6b2eded0e5dec532920b0ea 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-3-usage-guide.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md @@ -1,222 +1,207 @@ ---- -title: 使用指导 -summary: 使用指导 -author: Guo Huan -date: 2021-05-19 ---- - -# 使用指导 - -
- -## 数据收集 - -1. 打开数据收集。 - - a. 设置ActiveSQL operator信息相关参数: - - ```bash - enable_resource_track=on - resource_track_level=operator - enable_resource_record=on - resource_track_cost=10（默认值为100000） - ``` - - > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** - > - > - resource_track_cost需设置为小于需要收集的查询总代价，满足条件的信息才能被收集。 - > - Cgroups功能正常加载。 - - b. 信息收集： - - 执行业务查询语句。 - - 查看实时收集数据： - - ```sql - select * from gs_wlm_plan_operator_history; - ``` - - 预期：满足resource_track_duration和resource_track_cost的作业被全量收集。 - -2. 关闭数据收集。 - - a. 设置ActiveSQL operator信息相关参数： - - ```bash - enable_resource_track=off 或 - resource_track_level=none 或 - resource_track_level=query - ``` - - b. 执行业务查询语句。 - - 等待三分钟之后查看当前节点上的数据： - - ```sql - select * from gs_wlm_plan_operator_info; - ``` - - 预期：所查表和视图无新增数据。 - -3. 数据持久化保存。 - - a. 设置ActiveSQL operator信息相关参数： - - ```bash - enable_resource_track=on - resource_track_level=operator - enable_resource_record=on - resource_track_duration=0（默认值为60s） - resource_track_cost=10（默认值为100000） - ``` - - > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** - > - > - resource_track_cost需设置为小于需要收集的查询总代价，满足条件的信息才能被收集。 - > - Cgroups功能正常加载。 - - b. 执行业务查询语句。 - - 等待三分钟之后查看当前节点上的数据： - - ```sql - select * from gs_wlm_plan_operator_info; - ``` - - 预期：满足resource_track_duration和resource_track_cost的作业被全量收集。 - -
- -## 模型管理（系统管理员用户） - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 模型管理操作需要在数据库正常的状态下进行。 - -1. 新增模型： - - INSERT INTO gs_opt_model values('……')； - - 示例： - - ```sql - INSERT INTO gs_opt_model values('rlstm', 'model_name', 'datname', '127.0.0.1', 5000, 2000, 1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); - ``` - - > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** - > - > - 具体模型参数设置请参考[GS_OPT_MODEL](GS_OPT_MODEL)。 - > - 目前“template_name”列只支持“rlstm”。 - > - “datname”列请和用于模型使用和训练的数据库保持一致，否则无法使用。 - > - “model_name”一列需要满足unique约束。 - > - 其他参数设置见[最佳实践](2-4-best-practices)。 - -2. 修改模型参数： - - ```sql - UPDATE gs_opt_model SET = WHERE model_name = ; - ``` - -3. 删除模型： - - ```sql - DELETE FROM gs_opt_model WHERE model_name = ; - ``` - -4. 查询现有模型及其状态： - - ```sql - SELECT * FROM gs_opt_model; - ``` - -
- -## 模型训练（系统管理员用户） - -1. 配置/添加模型训练参数：参考模型管理（系统管理员用户）进行模型添加、模型参数修改，来指定训练参数。 - - 例： - - 模型添加： - - ```sql - INSERT INTO gs_opt_model values('rlstm', 'default', 'mogdb', '127.0.0.1', 5000, 2000, 1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); - ``` - - 训练参数更新： - - ```sql - UPDATE gs_opt_model SET = WHERE model_name = ; - ``` - -2. 前提条件为数据库状态正常且历史数据正常收集： - - 删除原有encoding数据： - - ```sql - DELETE FROM gs_wlm_plan_encoding_table; - ``` - - 进行数据编码，需要指定数据库名： - - ```sql - SELECT gather_encoding_info('mogdb'); - ``` - - 开始训练： - - ```sql - SELECT model_train_opt('rlstm', 'default'); - ``` - -3. 查看模型训练状态： - - ```sql - SELECT * FROM track_model_train_opt('rlstm', 'default'); - ``` - - 返回Tensorboard所用URL： - - ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/usage-guide-1.png) - - 打开URL查看模型训练状态： - - ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/usage-guide-2.png) - -
- -## 模型预测 - -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** -> -> - 模型预测功能需在数据库状态正常、指定模型已被训练且收敛的条件下进行。 -> - 目前，模型训练参数的标签设置中需要包含“S”标签，explain中才可显示“p-time”预测值。 例：INSERT INTO gs_opt_model values('rlstm', 'default', 'mogdb', '127.0.0.1', 5000, 1000, 1, -1, 50, 500, 0 , false, false, '{**S**, T}', '{0,0}', '{0,0}', 'Text'); - -调用explain接口： - -```sql -explain (analyze on, predictor ) -SELECT ... -``` - -预期结果： - -```bash -例：Row Adapter (cost=110481.35..110481.35 rows=100 p-time=99..182 width=100) (actual time=375.158..375.160 rows=2 loops=1) -其中，“p-time”列为标签预测值。 -``` - -
- -## 其他功能 - -1. 检查AiEngine是否可连接: - - ```sql - mogdb=# select check_engine_status('aiEngine-ip-address',running-port); - ``` - -2. 查看模型对应日志在AiEngine侧的保存路径： - - ```sql - mogdb=# select track_model_train_opt('template_name', 'model_name'); - ``` +--- +title: 使用指导 +summary: 使用指导 +author: Guo Huan +date: 2022-05-06 +--- + +# 使用指导 + +## 数据收集 + +1. 打开数据收集。 + + 1. 设置ActiveSQL operator信息相关参数： + + ``` + enable_resource_track=on + resource_track_level=operator + enable_resource_record=on + resource_track_cost=10（默认值为100000） + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** + > + > - resource_track_cost需设置为小于需要收集的查询总代价，满足条件的信息才能被收集。 + > - Cgroups功能正常加载。 + + 2. 信息收集： + + 执行业务查询语句。 + + 查看实时收集数据： + + ``` + select * from gs_wlm_plan_operator_history; + ``` + + 预期：满足resource_track_duration和resource_track_cost的作业被全量收集。 + +2. 关闭数据收集。 + + 1. 设置ActiveSQL operator信息相关参数： + + ``` + enable_resource_track=off 或 + resource_track_level=none 或 + resource_track_level=query + ``` + + 2. 执行业务查询语句。 + + 等待三分钟之后查看当前节点上的数据： + + ``` + select * from gs_wlm_plan_operator_info; + ``` + + 预期：所查表和视图无新增数据。 + +3. 数据持久化保存。 + + 1. 设置ActiveSQL operator信息相关参数： + + ``` + enable_resource_track=on + resource_track_level=operator + enable_resource_record=on + resource_track_duration=0（默认值为60s） + resource_track_cost=10（默认值为100000） + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** + > + > - resource_track_cost需设置为小于需要收集的查询总代价，满足条件的信息才能被收集 + > - Cgroups功能正常加载。 + + 2. 执行业务查询语句。 + + 等待三分钟之后查看当前节点上的数据： + + ``` + select * from gs_wlm_plan_operator_info; + ``` + + 预期：满足resource_track_duration和resource_track_cost的作业被全量收集。 + +## 模型管理（系统管理员用户） + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 模型管理操作需要在数据库正常的状态下进行。 + +1. 新增模型： + + INSERT INTO gs_opt_model values('……')； + + 示例： + + ``` + INSERT INTO gs_opt_model values('rlstm', 'model_name', 'datname', '127.0.0.1', 5000, 2000, 1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** + > + > - 具体模型参数设置请参考[GS_OPT_MODEL](GS_OPT_MODEL)。 + > - 目前 “template_name” 列只支持 “rlstm”。 + > - “datname” 列请和用于模型使用和训练的数据库保持一致，否则无法使用。 + > - “model_name” 一列需要满足unique约束。 + > - 其他参数设置见产品文档[最佳实践](intelligence-explain-best-practices)部分。 + +2. 修改模型参数： + + ``` + UPDATE gs_opt_model SET = WHERE model_name = ; + ``` + +3. 删除模型： + + ``` + DELETE FROM gs_opt_model WHERE model_name = ; + ``` + +4. 查询现有模型及其状态： + + ``` + SELECT * FROM gs_opt_model; + ``` + +## 模型训练（系统管理员用户） + +1. 配置/添加模型训练参数：参考上一节模型管理（系统管理员用户）进行模型添加、模型参数修改，来指定训练参数。 + + 例： + + 模型添加： + + ``` + INSERT INTO gs_opt_model values('rlstm', 'default', 'postgres', '127.0.0.1', 5000, 2000, 1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); + ``` + + 训练参数更新： + + ``` + UPDATE gs_opt_model SET = WHERE model_name = ; + ``` + +2. 前提条件为数据库状态正常且历史数据正常收集： + + 删除原有encoding数据： + + ``` + DELETE FROM gs_wlm_plan_encoding_table; + ``` + + 进行数据编码，需要指定数据库名： + + ``` + SELECT gather_encoding_info('postgres'); + ``` + + 开始训练： + + ``` + SELECT model_train_opt('rlstm', 'default'); + ``` + +3. 获取AI Engine侧模型训练日志相对路径： + + ``` + SELECT * FROM track_model_train_opt('rlstm', 'default'); + ``` + +## 模型预测 + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** +> +> - 模型预测功能需在数据库状态正常、指定模型已被训练且收敛的条件下进行。 +> +> - 目前，模型训练参数的标签设置中需要包含“S”标签，explain中才可显示“p-time”预测值。 +> +> 例：INSERT INTO gs_opt_model values('rlstm', 'default', 'postgres', '127.0.0.1', 5000, 1000, 1, -1, 50, 500, 0 , false, false, '{**S**, T}', '{0,0}', '{0,0}', 'Text'); + +1. 调用explain接口： + + ``` + explain (analyze on, predictor ) + SELECT ... + ``` + + 预期结果： + + ``` + 例：Row Adapter (cost=110481.35..110481.35 rows=100 p-time=99..182 width=100) (actual time=375.158..375.160 rows=2 loops=1) + 其中，“p-time”列为标签预测值。 + ``` + +## 其他功能 + +1. 检查AiEngine是否可连接: + + ``` + MogDB=# select check_engine_status('aiEngine-ip-address',running-port); + ``` + +2. 查看模型对应日志在AiEngine侧的保存路径： + + ``` + MogDB=# select track_model_train_opt('template_name', 'model_name'); + ``` \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..08e3bc2ab680f15f483fbfc726e3d3db7f370923 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md @@ -0,0 +1,10 @@ +--- +title: 概述 +summary: 概述 +author: Guo Huan +date: 2021-05-19 +--- + +# 概述 + +X-Tuner是一款数据库集成的参数调优工具，通过结合深度强化学习和全局搜索算法等AI技术，实现在无需人工干预的情况下，获取最佳数据库参数配置。本功能不强制与数据库环境部署到一起，支持独立部署，脱离数据库安装环境独立运行。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-2-preparations.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md similarity index 78% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-2-preparations.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md index 5e68d952ae42950881fbf3f609f3e8ba886825c2..0936c553414d29e52b45bfa0bdcf85fcce0c7993 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-2-preparations.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md @@ -14,7 +14,6 @@ date: 2021-05-19 - 数据库状态正常、客户端能够正常连接、且要求数据库内导入数据，以便调优程序可以执行benchmark测试调优效果。 - 使用本工具需要指定登录到数据库的用户身份，要求该登录到数据库上的用户具有足够的权限，以便可以获得充足的数据库状态信息。 - 使用登录到数据库宿主机上的Linux用户，需要将$**GAUSSHOME/bin**添加到PATH环境变量中，即能够直接运行gsql、gs_guc、gs_ctl等数据库运维工具。 -- Python版本建议为Python3.6+，且运行环境中已经安装相应依赖，并能够正常启动调优程序。您可以独立安装一个python3.6+的环境，无需设置到全局环境变量中。不建议使用root用户权限安装本工具，如果以root身份安装本完毕工具，使用其他用户身份运行本工具时，需要确保配置文件有读取权限。 - 本工具支持以三种模式运行，其中tune和train模式要求用户配置好benchmark运行环境，并导入数据，本工具将会通过迭代运行benchmark来判断修改后的参数是否有性能提升。 - recommend模式建议在数据库正在执行workload的过程中执行，以便获得更准确的实时workload信息。 - 本工具默认带有TPC-C、TPC-H、TPC-DS以及sysbench的benchmark运行脚本样例，如果用户使用上述benchmark对数据库系统进行压力测试，则可以对上述配置文件进行适度修改或配置。如果需要适配用户自己的业务场景，需要您参照benchmark目录中的template.py文件编写驱动您自定义benchmark的脚本文件。 @@ -52,52 +51,22 @@ X-Tuner 的整体架构如图1 X-Tuner 结构图所示，系统可以分为： ## X-Tuner的运行和安装方法 -可以通过两种方式运行X-Tuner，一种是直接通过源码运行，另一种则是通过Python的setuptools将X-Tuner安装到系统上，而后直接通过 gs_xtuner 命令调用。下面分别介绍两种运行X-Tuner的方法。 +执行下述命令即可获取xtuner功能帮助 -**方法一：直接通过源代码运行** - -1. 切换到xtuner的源代码目录下。对于已经安装的数据库系统，该源代码路径为 **$GAUSSHOME**/bin/dbmind/xtuner。 - -2. 在当前目录下可以看到 requirements.txt 等文件，通过pip包管理工具根据该 requirements.txt 文件安装依赖： - - ```bash - pip install -r requirements.txt - ``` - -3. 安装成功后需要添加环境变量PYTHONPATH，然后可执行main.py。以获取帮助信息为例，则可以执行： - - ```bash - cd tuner # 切换到 main.py 入口文件所在的目录 - export PYTHONPATH='..' # 将上一级目录添加到寻找包的路径中 - python main.py --help # 可以直接通过该命令执行获取帮助的动作，其他功能使用方法类似 - ``` - -**方法二：将X-Tuner 安装到系统中** - -1. 可以通过 setup.py 文件将X-Tuner安装到系统中，然后通过命令gs_xtuner运行。首先需要切换到xtuner的根目录，目录位置见上文。 - -2. 执行下述命令，Python 会通过自身的setuptools 将本工具安装到Python环境中： - - ```bash - python setup.py install - ``` - - 如果Python的bin目录被添加到PATH环境变量中，则gs_xtuner命令也可以在任何地方被直接调用。 - -3. 仍然以获取帮助信息为例，执行下述命令： +``` +gs_dbmind component xtuner --help +``` - ```bash - gs_xtuner --help - ``` +用户可据此给定不同的命令行执行相应的功能。
-## X-Tuner 的配置文件说明 +## X-Tuner的配置文件说明 -X-Tuner 在运行前需要加载配置文件，该配置文件默认路径是**$INSTALL_DIR/tuner/xtuner.conf**，可以通过**gs_xtuner --help**命令查看默认加载的配置文件绝对路径： +X-Tuner在运行前需要加载配置文件，用户可以通过**--help**命令查看默认加载的配置文件绝对路径： ```bash -... +... -x TUNER_CONFIG_FILE, --tuner-config-file TUNER_CONFIG_FILE This is the path of the core configuration file of the X-Tuner. You can specify the path of the new @@ -107,15 +76,15 @@ X-Tuner 在运行前需要加载配置文件，该配置文件默认路径是**$ ... ``` -修改配置文件的配置项可以指引X-Tuner 执行不同的动作，用户可以根据自己的不同需求来修改配置文件的内容，配置文件的配置项说明详见[命令参考](3-5-command-reference)页面中的表2。如果需要修改配置文件的加载路径，则可以通过选项 **-x** 命令行选项来指定。 +修改配置文件的配置项可以指引X-Tuner 执行不同的动作，用户可以根据自己的不同需求来修改配置文件的内容，配置文件的配置项说明详见[命令参考](1-5-command-reference)页面中的表2。如果需要修改配置文件的加载路径，则可以通过选**-x**命令行选项来指定。
## Benchmark的选择与配置 -Benchmark的驱动脚本存放路径为 X-Tuner 的benchmark子目录。X-Tuner自带常用的benchmark驱动脚本，例如基于时间周期的探测脚本（**默认**）、TPC-C、TPC-H等。X-Tuner通过调用benchmark/__init__.py文件中 **get_benchmark_instance()** 命令来加载不同的benchmark驱动脚本，获取benchmark驱动实例。其中，benchmark驱动脚本的格式说明如下： +Benchmark的驱动脚本存放路径为X-Tuner目录（即**$GAUSSHOME**/bin/dbmind/components/xtuner，下同）的子目录benchmark中。X-Tuner自带常用的benchmark驱动脚本，例如基于时间周期的探测脚本（**默认**）、TPC-C、TPC-H等。X-Tuner通过调用benchmark/__init__.py文件中**get_benchmark_instance()**命令来加载不同的benchmark驱动脚本，获取benchmark驱动实例。其中，benchmark驱动脚本的格式说明如下： -- 驱动脚本文件名：表示benchmark的名字，该名字用于表示驱动脚本的唯一性，可通过在 X-Tuner 的配置文件中的配置项**benchmark_script**来指定选择加载哪个benchmark驱动脚本。 +- 驱动脚本文件名：表示benchmark的名字，该名字用于表示驱动脚本的唯一性，可通过在X-Tuner 的配置文件中的配置项**benchmark_script**来指定选择加载哪个benchmark驱动脚本。 - 驱动脚本内容三要素：path变量、cmd变量以及run函数。 下面分别介绍驱动脚本的内容三要素： @@ -130,7 +99,7 @@ Benchmark的驱动脚本存放路径为 X-Tuner 的benchmark子目录。X-Tuner - {password}：与登录数据库系统上的用户相匹配的密码 - {db}：正在进行调优的数据库名 -3. run 函数：该函数的函数签名为： +3. run函数：该函数的函数签名为： ```java def run(remote_server, local_host) -> float: @@ -160,17 +129,17 @@ Benchmark驱动脚本示例说明 1. TPC-C 驱动脚本 - ```bash + ``` from tuner.exceptions import ExecutionError - + # WARN: You need to download the benchmark-sql test tool to the system, # replace the PostgreSQL JDBC driver with the openGauss driver, # and configure the benchmark-sql configuration file. # The program starts the test by running the following command: path = '/path/to/benchmarksql/run' # TPC-C测试脚本benchmark-sql 的存放路径 cmd = "./runBenchmark.sh props.gs" # 自定义一个名为 props.gs 的benchmark-sql测试配置文件 - - + + def run(remote_server, local_host): # 切换到 TPC-C 脚本目录下，清除历史错误日志，然后运行测试命令。 # 此处最好等待几秒钟，因为benchmark-sql 测试脚本生成最终测试报告是通过一个shell脚本实现的，整个过程会有延迟， @@ -179,7 +148,7 @@ Benchmark驱动脚本示例说明 # 如果标准错误流中有数据，则报异常退出。 if len(stderr) > 0: raise ExecutionError(stderr) - + # 寻找最终tpmC结果 tpmC = None split_string = stdout.split() # 对标准输出流结果进行分词。 @@ -192,22 +161,22 @@ Benchmark驱动脚本示例说明 "cat %s/benchmarksql-error.log" % path) nb_err = stdout.count("ERROR:") # 判断整个benchmark运行过程中，是否有报错，记录报错的错误数 return float(tpmC) - 10 * nb_err # 这里将报错的错误数作为一个惩罚项，惩罚系数为10，越高的惩罚系数表示越看中报错的数量. - + ``` -2. TPC-H 驱动脚本 +2. TPC-H驱动脚本 - ```bash + ``` import time - + from tuner.exceptions import ExecutionError - + # WARN: You need to import data into the database and SQL statements in the following path will be executed. # The program automatically collects the total execution duration of these SQL statements. path = '/path/to/tpch/queries' # 存放TPC-H测试用的SQL脚本目录 cmd = "gsql -U {user} -W {password} -d {db} -p {port} -f {file}" # 需要运行TPC-H测试脚本的命令，一般使用'gsql -f 脚本文件' 来运行 - - + + def run(remote_server, local_host): # 遍历当前目录下所有的测试用例文件名 find_file_cmd = "find . -type f -name '*.sql'" @@ -220,7 +189,7 @@ Benchmark驱动脚本示例说明 # 使用 file 变量替换 {file}，然后执行该命令行。 perform_cmd = cmd.format(file=file) stdout, stderr = remote_server.exec_command_sync(['cd %s' % path, perform_cmd]) - if len(stderr) &t;> 0: + if len(stderr) > 0: print(stderr) # 代价为全部测试用例的执行总时长 cost = time.time() - time_start diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-3-examples.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md similarity index 77% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-3-examples.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md index 7d81dc93f6bd5e6960b41942ec7889258bdee5ce..5d9acca58c4f8e46c6c586c9144b074272585a5d 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-3-examples.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md @@ -7,7 +7,7 @@ date: 2021-05-19 # 使用示例 -X-Tuner 支持三种模式，分别是获取参数诊断报告的recommend模式、训练强化学习模型的train模式、以及使用算法进行调优的tune模式。上述三种模式可以通过命令行参数来区别，通过配置文件来指定具体的细节。 +X-Tuner支持三种模式，分别是获取参数诊断报告的recommend模式、训练强化学习模型的train模式、以及使用算法进行调优的tune模式。上述三种模式可以通过命令行参数来区别，通过配置文件来指定具体的细节。
@@ -20,12 +20,12 @@ X-Tuner 支持三种模式，分别是获取参数诊断报告的recommend模式 分别传递 -db-name -db-user -port -host -host-user 参数，可选 -host-ssh-port 参数，例如： ```bash - gs_xtuner recommend --db-name mogdb --db-user omm --port 5678 --host 192.168.1.100 --host-user omm + gs_dbmind component xtuner recommend --db-name postgres --db-user omm --port 5678 --host 192.168.1.100 --host-user omm ``` 2. 通过JSON格式的连接信息配置文件指定： - JSON配置文件的示例如下，并假设文件名为 connection.json： + JSON配置文件的示例如下，并假设文件名为connection.json： ```json { @@ -44,14 +44,14 @@ X-Tuner 支持三种模式，分别是获取参数诊断报告的recommend模式
-## recommend 模式使用示例 +## recommend模式使用示例 -对recommend 模式生效的配置项为 scenario，若为auto，则自动检测workload类型。 +对recommend模式生效的配置项为scenario，若为auto，则自动检测workload类型。 执行下述命令，获取诊断结果： ```bash -gs_xtuner recommend -f connection.json +gs_dbmind component xtuner recommend -f connection.json ``` 则可以生成诊断报告如下： @@ -71,11 +71,16 @@ gs_xtuner recommend -f connection.json - checkpoint_avg_sync_time：数据库在checkpoint 时，平均每次同步刷新数据到磁盘的时长，单位是毫秒。 - load_average：平均**每个CPU核心**在1分钟、5分钟以及15分钟内的负载。一般地，该数值在1左右表明当前硬件比较匹配workload、在3左右表明运行当前作业压力比较大，大于5则表示当前硬件环境运行该workload压力过大（此时一般建议减少负载或升级硬件）。 -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** recommend 模式会读取数据库中的pg_stat_database以及 pg_stat_bgwriter等系统表中的信息，需要登录到数据库上的用户具有足够的权限（建议为**管理员权限**，可通过alter user username sysadmin；授予username相应的权限)。 由于某些系统表会一直记录统计信息，这可能会对负载特征识别造成干扰，因此建议最好先清空某些系统表的统计信息，运行一段时间的workload后再使用recommend模式进行诊断，以便获得更准确的结果。使用如下命令清除统计信息： +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** > -> `select pg_stat_reset_shared('bgwriter');` +> - 由于某些系统表会一直记录统计信息，这可能会对负载特征识别造成干扰，因此建议最好先清空某些系统表的统计信息，运行一段时间的workload后再使用recommend模式进行诊断，以便获得更准确的结果。清除统计信息的方法为： > -> `select pg_stat_reset();` +> ``` +> select pg_stat_reset_shared('bgwriter'); +> select pg_stat_reset(); +> ``` +> +> - recommend模式会读取数据库中的pg\_stat\_database以及pg\_stat\_bgwriter等系统表中的信息，需要登录到数据库上的用户具有足够的权限（建议为**管理员权限**，可通过alter user username sysadmin；授予username相应的权限）。
@@ -129,7 +134,7 @@ gs_xtuner recommend -f connection.json 待上述配置项配置完成后，可以通过下述命令启动训练： ```bash -gs_xtuner train -f connection.json +gs_dbmind component xtuner train -f connection.json ``` 训练完成后，会在配置项rl_model_path指定的目录中生成模型文件。 @@ -151,7 +156,7 @@ tune模式支持多种算法，包括基于强化学习（Reinforcement Learning 待上述配置项配置完成后，可以通过下述命令启动调优： ```bash -gs_xtuner tune -f connection.json +gs_dbmind component xtuner tune -f connection.json ``` -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **注意：** 在使用tune和train 模式前，用户需要先导入benchmark所需数据并检查benchmark能否正常跑通。调优过程结束后，调优程序会自动恢复调优前的数据库参数配置。 +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **注意：** 在使用tune和train模式前，用户需要先导入benchmark所需数据并检查benchmark能否正常跑通。调优过程结束后，调优程序会自动恢复调优前的数据库参数配置。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-4-obtaining-help-information.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md similarity index 95% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-4-obtaining-help-information.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md index f22f423cf72a6cd192fca0aee9cb505fdd6a99bb..5d050c2ddb9d195e5e3caca9cc7789ecd3056cf0 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-4-obtaining-help-information.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md @@ -10,13 +10,13 @@ date: 2021-05-19 启动调优程序之前，可以通过如下命令获取帮助信息： ```bash -gs_xtuner --help +gs_dbmind component xtuner --help ``` 输出帮助信息结果如下： ```bash -usage: gs_xtuner [-h] [--db-name DB_NAME] [--db-user DB_USER] [--port PORT] +usage: [-h] [--db-name DB_NAME] [--db-user DB_USER] [--port PORT] [--host HOST] [--host-user HOST_USER] [--host-ssh-port HOST_SSH_PORT] [-f DB_CONFIG_FILE] [-x TUNER_CONFIG_FILE] [-v] diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-5-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md similarity index 39% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-5-command-reference.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md index d6a992580a3681607e92f432fb67046b09fc030f..56ccac3931fb1a317a11f2c049710859f9f677ee 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-5-command-reference.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md @@ -9,42 +9,42 @@ date: 2021-05-19 **表 1** 命令行参数 -| 参数 | 参数说明 | 取值范围 | -| :--------------------- | :----------------------------------------------------------- | :--------------------- | -| mode | 指定调优程序运行的模式 | train，tune，recommend | -| -tuner-config-file, -x | X-Tuner的核心参数配置文件路径，默认路径为安装目录下的xtuner.conf | \- | -| -db-config-file, -f | 调优程序的用于登录到数据库宿主机上的连接信息配置文件路径，若通过该文件配置数据库连接信息，则下述数据库连接信息可省略 | \- | -| -db-name | 指定需要调优的数据库名 | \- | -| -db-user | 指定以何用户身份登陆到调优的数据库上 | \- | -| -port | 数据库的侦听端口 | \- | -| -host | 数据库实例的宿主机IP | \- | -| -host-user | 指定以何用户身份登陆到数据库实例的宿主机上，要求改用户名的环境变量中可以找到gsql、gs_ctl等数据库运维工具。 | \- | -| -host-ssh-port | 数据库实例所在宿主机的SSH端口号，可选，默认为22 | \- | -| -help, -h | 返回帮助信息 | \- | -| -version, -v | 返回当前工具版本号 | \- | +| 参数 | 参数说明 | 取值范围 | +| :---------------------- | :----------------------------------------------------------- | :--------------------- | +| mode | 指定调优程序运行的模式 | train，tune，recommend | +| --tuner-config-file, -x | X-Tuner的核心参数配置文件路径，默认路径为安装目录下的xtuner.conf | - | +| --db-config-file, -f | 调优程序的用于登录到数据库宿主机上的连接信息配置文件路径，若通过该文件配置数据库连接信息，则下述数据库连接信息可省略 | - | +| --db-name | 指定需要调优的数据库名 | - | +| --db-user | 指定以何用户身份登陆到调优的数据库上 | - | +| --port | 数据库的侦听端口 | - | +| --host | 数据库实例的宿主机IP | - | +| --host-user | 指定以何用户身份登陆到数据库实例的宿主机上，要求改用户名的环境变量中可以找到gsql、gs_ctl等数据库运维工具。 | - | +| --host-ssh-port | 数据库实例所在宿主机的SSH端口号，可选，默认为22 | - | +| --help, -h | 返回帮助信息 | - | +| --version, -v | 返回当前工具版本号 | - | **表 2** 配置文件中的参数详解 | 参数名 | 参数说明 | 取值范围 | -| :-------------------- | :----------------- | :------------------- | -| logfile | 生成的日志存放路径 | \- | -| output_tuning_result | 可选，调优结果的保存路径 | \- | +| :-------------------- | :----------------------------------------------------------- | :---------------------------- | +| logfile | 生成的日志存放路径 | - | +| output_tuning_result | 可选，调优结果的保存路径 | - | | verbose | 是否打印详情 | on, off | -| recorder_file | 调优中间信息的记录日志存放路径 | \- | +| recorder_file | 调优中间信息的记录日志存放路径 | - | | tune_strategy | 调优模式下采取哪种策略 | rl, gop, auto | -| drop_cache | 是否在每一个迭代轮次中进行drop cache，drop cache可以使benchmark跑分结果更加稳定。若启动该参数，则需要将登录的系统用户加入到 /etc/sudoers 列表中，同时为其增加 **NOPASSWD** 权限（由于该权限可能过高，建议临时启用该权限，调优结束后关闭）。 | on, off | +| drop_cache | 是否在每一个迭代轮次中进行drop cache，drop cache可以使benchmark跑分结果更加稳定。若启动该参数，则需要将登录的系统用户加入到 /etc/sudoers 列表中，同时为其增加**NOPASSWD**权限（由于该权限可能过高，建议临时启用该权限，调优结束后关闭）。 | on, off | | used_mem_penalty_term | 数据库使用总内存的惩罚系数，用于防止通过无限量占用内存而换取的性能表现。该数值越大，惩罚力度越大。 | 建议0 ~ 1 | | rl_algorithm | 选择何种RL算法 | ddpg | -| rl_model_path | RL模型保存或读取路径，包括保存目录名与文件名前缀。在train 模式下该路径用于保存模型，在tune模式下则用于读取模型文件 | \- | -| rl_steps | 深度强化学习算法迭代的步数 | \- | -| max_episode_steps | 每个回合的最大迭代步数 | \- | -| test_episode | 使用RL算法进行调优模式的回合数 | \- | -| gop_algorithm | 采取何种全局搜索算法 | bayes, pso, auto | -| max_iterations | 全局搜索算法的最大迭代轮次（并非确定数值，可能会根据实际情况多跑若干轮） | \- | -| particle_nums | PSO算法下的粒子数量 | \- | +| rl_model_path | RL模型保存或读取路径，包括保存目录名与文件名前缀。在train模式下该路径用于保存模型，在tune模式下则用于读取模型文件 | - | +| rl_steps | 深度强化学习算法迭代的步数 | - | +| max_episode_steps | 每个回合的最大迭代步数 | - | +| test_episode | 使用RL算法进行调优模式的回合数 | - | +| gop_algorithm | 采取何种全局搜索算法 | bayes, pso | +| max_iterations | 全局搜索算法的最大迭代轮次（并非确定数值，可能会根据实际情况多跑若干轮） | - | +| particle_nums | PSO算法下的粒子数量 | - | | benchmark_script | 使用何种benchmark驱动脚本，该选项指定加载benchmark路径下同名文件，默认支持TPC-C、TPC-H等典型benchmark | tpcc, tpch, tpcds, sysbench … | -| benchmark_path | benchmark 脚本的存储路径，若没有配置该选项，则使用benchmark驱动脚本中的配置 | \- | -| benchmark_cmd | 启动benchmark 脚本的命令，若没有配置该选项，则使用benchmark驱动脚本中的配置 | \- | -| benchmark_period | 仅对 period benchmark有效，表明整个benchmark的测试周期是多少，单位是秒 | - | +| benchmark_path | benchmark脚本的存储路径，若没有配置该选项，则使用benchmark驱动脚本中的配置 | - | +| benchmark_cmd | 启动benchmark脚本的命令，若没有配置该选项，则使用benchmark驱动脚本中的配置 | - | +| benchmark_period | 仅对period benchmark有效，表明整个benchmark的测试周期是多少，单位是秒 | - | | scenario | 用户指定的当前workload所属的类型 | tp, ap, htap | -| tuning_list | 准备调优的参数列表文件，可参考 share/knobs.json.template 文件 | \- | +| tuning_list | 准备调优的参数列表文件，可参考share/knobs.json.template文件 | - | diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-6-Troubleshooting.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md similarity index 61% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-6-Troubleshooting.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md index dba0a629a283a28a41a135e74a7cb6071924ecef..23b5b4e78300989fbc900c1a2d52517e577223ec 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-6-Troubleshooting.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md @@ -9,6 +9,5 @@ date: 2021-05-19 - 数据库实例连接失败：请检查数据库实例的情况，是否数据库实例出现了问题或安全权限配置（pg_hba.conf文件中的配置项）不正确。 - 重启失败：请检查数据库实例健康情况，确保数据库实例工作正常； -- 依赖安装失败：建议先升级pip包管理工具，通过命令 python -m pip install -upgrade pip 实现 - 跑TPC-C作业时发现性能越来越慢：TPC-C等高并发场景下的压力测试，往往伴随着大量的数据修改。由于每一次测试并非是幂等的（TPC-C数据库数据量的增加、没有进行vacuum full清理掉失效元组、数据库没有触发checkpoint、没有进行drop cache等），因此一般建议TPC-C等伴随着较多数据写入的benchmark应该每隔一段时间（视具体并发量、执行时长的不同而异）重新导入一次数据，比较简单的方法是备份$PGDATA目录。 -- TPC-C 跑作业时，TPC-C驱动脚本报异常 “TypeError: float() argument must be a string or a number, not 'NoneType'”（不能将None转换为float类型）：这是因为没有获取到TPC-C的压测返回结果，造成该问题的原因比较多，请首先手动检测是否能够跑通TPC-C并能够获取返回结果。若无上述问题，则建议将 TPC-C 驱动脚本中的命令列表中的 “sleep” 命令延迟时间设得更大一些。 +- TPC-C跑作业时，TPC-C驱动脚本报异常 “TypeError: float() argument must be a string or a number, not 'NoneType'”（不能将None转换为float类型）：这是因为没有获取到TPC-C的压测返回结果，造成该问题的原因比较多，请首先手动检测是否能够跑通TPC-C并能够获取返回结果。若无上述问题，则建议将TPC-C驱动脚本中的命令列表中的 “sleep” 命令延迟时间设得更大一些。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-1-single-query-index-recommendation.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md similarity index 85% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-1-single-query-index-recommendation.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md index b24139d0ea67517c70e4c90900695910c6ecc188..9124a5a01828d52e1dbd8c8e3d66c72c14e900cf 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-1-single-query-index-recommendation.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md @@ -58,4 +58,14 @@ mogdb=# select "table", "column" from gs_index_advise('select name, age, sex fro CREATE INDEX idx1 on t1(age, sex); ``` -> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 系统函数 gs_index_advise() 的参数是文本型，如果参数中存在如单引号（'） 等特殊字符，可以使用单引号（'） 进行转义，可参考上述示例。 +针对分区表可推荐具体索引类型，例如： + +```sql +mogdb=# select "table", "column", "indextype" from gs_index_advise('select name, age, sex from range_table where age = 20;'); + table | column | indextype +-------+--------+----------- + t1 | age | global +(1 row) +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 系统函数 gs_index_advise() 的参数是文本型，如果参数中存在如单引号（'） 等特殊字符，可以使用单引号（') 进行转义，可参考上述示例。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-2-virtual-index.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md similarity index 100% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-2-virtual-index.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-3-workload-level-index-recommendation.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md similarity index 69% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-3-workload-level-index-recommendation.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md index b3c1a085fb5f9c97e5d48b1249232b01fa10716d..410140dcb4f5bae823c331161720d63988e885d0 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/6-index-advisor-index-recommendation/6-3-workload-level-index-recommendation.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md @@ -1,6 +1,6 @@ --- -title: Workload-level Index Recommendation -summary: Workload-level Index Recommendation +title: workload级别索引推荐 +summary: workload级别索引推荐 author: Guo Huan date: 2021-05-19 --- @@ -17,8 +17,6 @@ date: 2021-05-19 - 当前执行用户下安装有gsql工具，该工具路径已被加入到PATH环境变量中。 -- 具备Python3.6+的环境。 - - 若使用本功能提供的业务数据抽取功能，需提前将要收集的节点的GUC参数按如下设置： - log_min_duration_statement = 0 @@ -26,6 +24,7 @@ date: 2021-05-19 - log_statement= 'all' > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** + > > 业务数据抽取完毕建议将上述GUC参数复原，否则容易导致日志文件膨胀。
@@ -34,29 +33,34 @@ date: 2021-05-19 1. 按前提条件中要求设置相关GUC参数。 -2. 运行python脚本extract_log.py，命令如下： +2. 执行根据日志抽取SQL语句的功能，命令如下： ``` - python extract_log.py [l LOG_DIRECTORY] [f OUTPUT_FILE] [-d DATABASE] [-U USERNAME][--start_time] [--sql_amount] [--statement] [--json] + gs_dbmind component extract_log [l LOG_DIRECTORY] [f OUTPUT_FILE] [p LOG_LINE_PREFIX] [-d DATABASE] [-U USERNAME][--start_time] [--sql_amount] [--statement] [--json] [--max_reserved_period] [--max_template_num] ``` 其中的输入参数依次为： - LOG_DIRECTORY：pg_log的存放目录。 - OUTPUT_PATH：输出SQL流水文件文件的保存路径，即抽取出的业务数据存放的文件路径。 + - LOG_LINE_PREFIX：指定每条日志信息的前缀格式。 - DATABASE：（可选）数据库名称，不指定默认所有数据库。 - USERNAME：（可选）用户名称，不指定默认所有用户。 - start_time：（可选）日志收集的开始时间， 不指定默认所有文件。 - sql_amount：（可选）收集SQL数量的最大值， 不指定默认收集所有SQL。 - statement：（可选）表示收集pg_log日志中statement标识开头的SQL，不指定默认不收集。 - - json：指定收集日志的文件存储格式为SQL归一化后的json，不指定默认格式每条SQL占一行。 + - json：（可选）指定收集日志的文件存储格式为SQL归一化后的json，不指定默认格式每条SQL占一行。 + - max_reserved_period：（可选）指定json模式下，增量收集日志中保留的模板的最大的更新时长，不指定默认都保留，单位/天。 + - max_template_num：（可选）指定json模式下保留的最大模板数量，不指定默认都保留。 使用示例： ``` - python extract_log.py $GAUSSLOG/pg_log/dn_6001 sql_log.txt -d postgres -U omm --start_time '2021-07-06 00:00:00' --statement + gs_dbmind component extract_log $GAUSSLOG/pg_log/dn_6001 sql_log.txt '%m %c %d %p %a %x %n %e' -d postgres -U omm --start_time '2021-07-06 00:00:00' --statement ``` + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 若指定-d/-U参数，日志打印每条日志信息的前缀格式需包含%d、%u，若需要抽取事务，必须指定%p，详见log_line_prefix参数。max_template_num参数设置建议不超5000条，避免workload索引推荐执行时间过长。 + 3. 将步骤1中设置的GUC参数还原为设置前的值。
@@ -65,14 +69,15 @@ date: 2021-05-19 1. 准备好包含有多条DML语句的文件作为输入的workload，文件中每条语句占据一行。用户可从数据库的离线日志中获得历史的业务语句。 -2. 运行python脚本index_advisor_workload.py，命令如下： +2. 运行本功能，命令如下： ```python - python index_advisor_workload.py [p PORT] [d DATABASE] [f FILE] [--h HOST] [-U USERNAME] [-W PASSWORD][--schema SCHEMA][--max_index_num MAX_INDEX_NUM][--max_index_storage MAX_INDEX_STORAGE] [--multi_iter_mode] [--multi_node] [--json] [--driver] [--show_detail] + gs_dbmind component index_advisor [p PORT] [d DATABASE] [f FILE] [--h HOST] [-U USERNAME] [-W PASSWORD][--schema SCHEMA] + [--max_index_num MAX_INDEX_NUM][--max_index_storage MAX_INDEX_STORAGE] [--multi_iter_mode] [--multi_node] [--json] [--driver] [--show_detail] ``` - + 其中的输入参数依次为： - + - PORT：连接数据库的端口号。 - DATABASE：连接数据库的名字。 - FILE：包含workload语句的文件路径。 @@ -87,15 +92,15 @@ date: 2021-05-19 - json：（可选）指定workload语句的文件路径格式为SQL归一化后的json，默认格式每条SQL占一行。 - driver：（可选）指定是否使用python驱动器连接数据库，默认gsql连接。 - show_detail：（可选）是否显示当前推荐索引集合的详细优化信息。 - + 例如： - + ```python - python index_advisor_workload.py 6001 postgres tpcc_log.txt --max_index_num 10 --multi_iter_mode + gs_dbmind component index_advisor 6001 postgres tpcc_log.txt --schema public --max_index_num 10 --multi_iter_mode ``` - + 推荐结果为一批索引，以多个创建索引语句的格式显示在屏幕上，结果示例。 - + ```sql create index ind0 on public.bmsql_stock(s_i_id,s_w_id); create index ind1 on public.bmsql_customer(c_w_id,c_id,c_d_id); @@ -108,5 +113,5 @@ date: 2021-05-19 create index ind8 on public.bmsql_oorder(o_w_id,o_c_id,o_d_id); create index ind9 on public.bmsql_district(d_w_id); ``` - + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** multi_node参数需严格按照当前数据库架构进行指定，否则推荐结果不全，甚至导致无推荐结果。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-1-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-1-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..ccd38ff87706e0bb500a081713b6e4f95a75aafc --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-1-overview.md @@ -0,0 +1,10 @@ +--- +title: 概述 +summary: 概述 +author: Guo Huan +date: 2022-05-06 +--- + +# 概述 + +慢SQL一直是数据运维中的痛点问题，如何有效诊断慢SQL根因是当前一大难题，工具结合MogDB自身特点融合了现网DBA慢SQL诊断经验，该工具可以支持慢SQL根因15+，能同时按照可能性大小输出多个根因并提供针对性的建议。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..8d8ca64a73a3ede615c91b0883dc60ee433d1927 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md @@ -0,0 +1,11 @@ +--- +title: 环境部署 +summary: 环境部署 +author: Guo Huan +date: 2022-05-06 +--- + +# 环境部署 + +- 数据库运行正常。 +- 指标采集系统运行正常。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..f5ba33268d30bd1da260c13fcc2e0bcbcfcbbb01 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md @@ -0,0 +1,34 @@ +--- +title: 使用指导 +summary: 使用指导 +author: Guo Huan +date: 2022-05-06 +--- + +# 使用指导 + +假设用户已经初始化配置文件目录confpath，则可以通过下述命令实现本特性的功能： + +- 仅启动慢SQL诊断功能（输出Top3根因），启动命令如下（更多用法参考对service子命令的说明）： + + ``` + gs_dbmind service start -c confpath --only-run slow_query_diagnosis + ``` + +- 用户交互式慢SQL诊断，命令如下： + + ``` + gs_dbmind component slow_query_diagnosis show -c confpath --query SQL --start-time timestamps0 --end-time timestamps1 + ``` + +- 用户手动清理历史预测结果，命令如下： + + ``` + gs_dbmind component slow_query_diagnosis clean -c confpath --retention-days DAYS + ``` + +- 停止已启动的服务，命令如下： + + ``` + gs_dbmind service stop -c confpath + ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..a15ad42acf805aa8253fa12ad901447268eade9c --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md @@ -0,0 +1,41 @@ +--- +title: 获取帮助 +summary: 获取帮助 +author: Guo Huan +date: 2022-05-06 +--- + +# 获取帮助 + +模块命令行说明： + +``` +gs_dbmind component slow_query_diagnosis --help +``` + +``` +usage: [-h] -c DIRECTORY [--query SLOW_QUERY] + [--start-time TIMESTAMP_IN_MICROSECONDS] + [--end-time TIMESTAMP_IN_MICROSECONDS] [--retention-days DAYS] + {show,clean} + +Slow Query Diagnosis: Analyse the root cause of slow query + +positional arguments: + {show,clean} choose a functionality to perform + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files + --query SLOW_QUERY set a slow query you want to retrieve + --start-time TIMESTAMP_IN_MICROSECONDS + set the start time of a slow SQL diagnosis result to + be retrieved + --end-time TIMESTAMP_IN_MICROSECONDS + set the end time of a slow SQL diagnosis result to be + retrieved + --retention-days DAYS + clear historical diagnosis results and set the maximum + number of days to retain data +``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..a4133294bc39d60440b105c18d98071c0fa4f661 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md @@ -0,0 +1,20 @@ +--- +title: 命令参考 +summary: 命令参考 +author: Guo Huan +date: 2022-05-06 +--- + +# 命令参考 + +**表 1** gs_dbmind component slow_query_diagnosis 命令行说明 + +| 参数 | 参数说明 | 取值范围 | +| :--------------- | :----------------------------- | :------------------------------------- | +| -h, --help | 帮助命令 | - | +| action | 动作参数 | - show：结果展示
- clean：清理结果 | +| -c，--conf | 配置目录 | - | +| --query | 慢SQL文本 | * | +| --start-time | 显示开始时间的时间戳，单位毫秒 | 非负整数 | +| --end-time | 显示结束时间的时间戳，单位毫秒 | 非负整数 | +| --retention-days | 清理天数级结果 | 非负实数 | \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..408045a4cc7bf81bfeca404257108f3d0c8ab789 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md @@ -0,0 +1,12 @@ +--- +title: 常见问题处理 +summary: 常见问题处理 +author: Guo Huan +date: 2022-05-06 +--- + +# 常见问题处理 + +- 如果用户对没有执行过的慢SQL执行交互式诊断命令，则无法给出诊断结果。 +- exporter指标采集功能没有启动时运行慢SQL诊断功能，此时功能无法正常运行。 +- 配置文件中的参数重新设置后，需要重新启动服务进程才能生效。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-1-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-1-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..f51f0500d03926b55a68c93b16cdec6ead43853a --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-1-overview.md @@ -0,0 +1,10 @@ +--- +title: 概述 +summary: 概述 +author: Guo Huan +date: 2022-05-06 +--- + +# 概述 + +趋势预测功能模块主要实现基于历史时序数据预测未来时序变化趋势。该模块框架解耦，可以实现不同预测算法的灵活替换，并且该模块功能可以实现不同特征时序的算法自动选择，支持线性特征时序预测LR回归算法和非线性特征预测ARIMA算法。目前该模块可以覆盖线性时序、非线性时序和周期时序的准确预测。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-2-environment-deployment.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-2-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..da376e699d41175da463965bf73e359b1f05e3b3 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-2-environment-deployment.md @@ -0,0 +1,10 @@ +--- +title: 环境部署 +summary: 环境部署 +author: Guo Huan +date: 2022-05-06 +--- + +# 环境部署 + +指标采集系统运行正常。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-3-usage-guide.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-3-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..7bb64dad162c7d670a932ec25277a89561dcfab2 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-3-usage-guide.md @@ -0,0 +1,34 @@ +--- +title: 使用指导 +summary: 使用指导 +author: Guo Huan +date: 2022-05-06 +--- + +# 使用指导 + +假设用户已经初始化配置文件目录confpath，则可以通过下述命令实现本特性的功能： + +- 仅启动趋势预测功能，启动命令如下（更多用法参考对service子命令的说明）： + + ``` + gs_dbmind service start -c confpath --only-run forecast + ``` + +- 用户交互式趋势预测，查看timestamps0到timestamps1时间段内的预测结果，命令如下： + + ``` + gs_dbmind component forecast show -c confpath --start-time timestamps0 --end-time timestamps1 + ``` + +- 用户手动清理手动清理历史预测结果，命令如下： + + ``` + gs_dbmind component forecast clean -c confpath --retention-days DAYS + ``` + +- 停止已启动的服务，命令如下： + + ``` + gs_dbmind service stop -c confpath + ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-4-obtaining-help-information.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-4-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..93afea7d93c79858acfaedc98adda50eac879a11 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-4-obtaining-help-information.md @@ -0,0 +1,38 @@ +--- +title: 获取帮助 +summary: 获取帮助 +author: Guo Huan +date: 2022-05-06 +--- + +# 获取帮助 + +模块命令行说明： + +``` +gs_dbmind component forecast --help +``` + +``` +usage: [-h] -c DIRECTORY [--metric-name METRIC_NAME] [--host HOST] [--start-time TIMESTAMP_IN_MICROSECONDS] [--end-time TIMESTAMP_IN_MICROSECONDS] [--retention-days DAYS] + {show,clean} + +Workload Forecasting: Forecast monitoring metrics + +positional arguments: + {show,clean} choose a functionality to perform + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files + --metric-name METRIC_NAME + set a metric name you want to retrieve + --host HOST set a host you want to retrieve + --start-time TIMESTAMP_IN_MICROSECONDS + set a start time of for retrieving + --end-time TIMESTAMP_IN_MICROSECONDS + set a end time of for retrieving + --retention-days DAYS + clear historical diagnosis results and set the maximum number of days to retain data +``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-5-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-5-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..6f2f473bc6ef559ee532dace51107e1f5316665f --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-5-command-reference.md @@ -0,0 +1,21 @@ +--- +title: 命令参考 +summary: 命令参考 +author: Guo Huan +date: 2022-05-06 +--- + +# 命令参考 + +**表 1** gs_dbmind component forecast 命令行说明 + +| 参数 | 参数说明 | 取值范围 | +| :--------------- | :----------------------------- | :------------------------------------- | +| -h, --help | 帮助命令 | - | +| action | 动作参数 | - show：结果展示
- clean：清理结果 | +| -c，--conf | 配置目录 | - | +| --metric-name | 指定显示指标名，用于过滤 | - | +| --host | 指定服务IP和端口号，用于过滤 | - | +| --start-time | 显示开始时间的时间戳，单位毫秒 | 非负实数 | +| --end-time | 显示结束时间的时间戳，单位毫秒 | 非负实数 | +| --retention-days | 保留结果天数 | 非负实数 | diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-6-troubleshooting.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-6-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..976b7a40fe66c3d25eda5353497af6327add72c6 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-6-troubleshooting.md @@ -0,0 +1,11 @@ +--- +title: 常见问题处理 +summary: 常见问题处理 +author: Guo Huan +date: 2022-05-06 +--- + +# 常见问题处理 + +- 综合实际业务与模型预测效果考虑，趋势预测时长建议不要太短，建议大于3600秒（如果指标采集周期为15秒，则数据量为240个），否则预测效果会变差，并且数据量极小时，服务会异常，因此默认配置为3600秒 +- 配置文件中的参数重新设置后，需要重新启动服务进程才能生效。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-1-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md similarity index 73% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-1-overview.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md index eb13999f6c434e629b70a7c910e0ef3467a8df2f..cceab6a52e2ff61fceeb936853ac0d58707e7778 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-1-overview.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md @@ -7,12 +7,12 @@ date: 2021-05-19 # 概述 -SQLdiag是MogDB中SQL语句执行时长预测工具。现有的预测技术主要基于执行计划的预测方法，但这些预测方案仅适用于OLAP场景且可以获取执行计划的任务，对于OLTP或者HTAP这样的快速、简单查询是没有太多使用价值的。与上述方案不同，SQLdiag着眼于数据库的历史SQL语句，通过对历史SQL语句的执行表现进行总结归纳，将之再用于推断新的未知业务上。由于短时间内数据库SQL语句执行时长不会有太大的差距，SQLdiag可以从历史数据中检测出与已执行SQL语句相似的语句结果集，并基于SQL向量化技术和模板化方法预测SQL语句执行时长。本工具有如下优点： +SQLdiag是MogDB中SQL语句执行时长预测工具。现有的预测技术主要基于执行计划的预测方法，但这些预测方案仅适用于OLAP场景且可以获取执行计划的任务，对于OLTP或者HTAP这样的快速、简单查询是没有太多使用价值的。与上述方案不同，SQLdiag着眼于数据库的历史SQL语句，通过对历史SQL语句的执行表现进行总结归纳，将之再用于推断新的未知业务上。由于短时间内数据库SQL语句执行时长不会有太大的差距，SQLdiag可以从历史数据中检测出与已执行SQL语句相似的语句结果集，并基于SQL向量化技术通过SQL模板化和深度学习这两种方法来预测SQL语句执行时长。本工具有如下优点： 1. 不需要SQL语句的执行计划，对数据库性能不会有任何的影响。 2. 使用场景广泛，目前业内的很多算法局限性比较高，比如只适用于OLTP或者OLAP，而SQLdiag使用场景广泛。 -3. 该框架鲁棒性强，容易理解，只需要简单的操作，就可以训练出自己的预测模型。 +3. 该工具容易理解，只需要简单的操作，就可以训练出自己的预测模型。 -SQLdiag是一个SQL语句执行时间预测工具，通过模板化方法，实现在不获取SQL语句执行计划的前提下，依据语句逻辑相似度与历史执行记录，预测SQL语句的执行时间。 +SQLdiag是一个SQL语句执行时间预测工具，通过模板化方法或者深度学习方法，实现在不获取SQL语句执行计划的前提下，依据语句逻辑相似度与历史执行记录，预测SQL语句的执行时间并以此发现异常SQL。 本工具的典型应用场景是对一批即将上线的SQL语句进行透视，提前识别风险。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-2-usage-guide.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md similarity index 66% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-2-usage-guide.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md index aedee8afc965c7795efafab7010275c9452fab11..2e1562ad161a1ee4bd7123c8112c117ec28284e6 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-2-usage-guide.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md @@ -12,15 +12,6 @@ date: 2021-10-21 - 需要保证用户提供训练数据。 - 如果用户通过提供的工具收集训练数据，则需要启用WDR功能，涉及到的参数为track_stmt_stat_level和log_min_duration_statement，具体情况见下面小节。 - 为保证预测准确率，用户提供的历史语句日志应尽可能全面并具有代表性。 -- 按照要求配置python 3.6+环境及其依赖。 - -## 环境配置 - -本功能运行环境要求Python 3.6版本及以上，需要的第三方依赖包记录在requirements.txt文件中，可以通过pip install命令安装依赖，如： - -```bash -pip install requirements.txt -``` ## SQL流水采集方法 @@ -43,7 +34,7 @@ SQL - 其中log_min_duration_statement表示慢SQL阈值，如果为0则全量收集，时间单位为毫秒； - track_stmt_stat_level表示信息捕获的级别，建议设置为track_stmt_stat_level='L0,L0' -参数开启后，可能占用一定的系统资源，但一般不大。持续的高并发场景可能产生5%以内的损耗，数据库并发较低的场景，性能损耗可忽略。 +参数开启后，可能占用一定的系统资源，但一般不大。持续的高并发场景可能产生5%以内的损耗，数据库并发较低的场景，性能损耗可忽略。下述脚本存在于sqldiag根目录（$**GAUSSHOME**/bin/components/sqldiag）中。 ```bash 使用脚本获取训练集方式： @@ -61,47 +52,45 @@ load_sql_from_wdr.py [-h] --port PORT --start_time START_TIME ```bash 基于模板法的训练与预测： - python main.py [train, predict] -f FILE --model template --model-path template_model_path + gs_dbmind component sqldiag [train, predict] -f FILE --model template --model-path template_model_path 基于DNN的训练与预测： - python main.py [train, predict] -f FILE --model dnn --model-path dnn_model_path + gs_dbmind component sqldiag [train, predict] -f FILE --model dnn --model-path dnn_model_path ``` ## 使用方法示例 -在本工具的根目录中，执行下列语句可以实现对应功能。 - 使用提供的测试数据进行模板化训练： ```bash -python main.py train -f ./sample_data/train.csv --model template --model-path ./template +gs_dbmind component sqldiag train -f ./sample_data/train.csv --model template --model-path ./template ``` 使用提供的测试数据进行模板化预测： ```bash -python main.py predict -f ./sample_data/predict.csv --model template --model-path ./template --predicted-file ./result/t_result +gs_dbmind component sqldiag predict -f ./sample_data/predict.csv --model template --model-path ./template --predicted-file ./result/t_result ``` 使用提供的测试数据进行模板化模型更新： ```bash -python main.py finetune -f ./sample_data/train.csv --model template --model-path ./template +gs_dbmind component sqldiag finetune -f ./sample_data/train.csv --model template --model-path ./template ``` 使用提供的测试数据进行DNN训练： ```bash -python main.py train -f ./sample_data/train.csv --model dnn --model-path ./dnn_model +gs_dbmind component sqldiag train -f ./sample_data/train.csv --model dnn --model-path ./dnn_model ``` 使用提供的测试数据进行DNN预测： ```bash -python main.py predict -f ./sample_data/predict.csv --model dnn --model-path ./dnn_model --predicted-file +gs_dbmind component sqldiag predict -f ./sample_data/predict.csv --model dnn --model-path ./dnn_model --predicted-file ``` 使用提供的测试数据进行DNN模型更新： ```bash -python main.py finetune -f ./sample_data/train.csv --model dnn --model-path ./dnn_model +gs_dbmind component sqldiag finetune -f ./sample_data/train.csv --model dnn --model-path ./dnn_model ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-3-obtaining-help-information.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md similarity index 92% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-3-obtaining-help-information.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md index 0b8ebae44ec1feaf2057352754d12bba87fda427..d211ee54172bc46f3d47919e38ff0015541ad0e2 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-3-obtaining-help-information.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md @@ -10,18 +10,18 @@ date: 2021-05-19 使用SQLdiag工具前，您可以通过以下指令获取帮助。 ``` -python main.py --help +gs_dbmind component sqldiag --help ``` 显示如下帮助信息： ``` -usage: main.py [-h] [-f CSV_FILE] [--predicted-file PREDICTED_FILE] +usage: [-h] [-f CSV_FILE] [--predicted-file PREDICTED_FILE] [--model {template,dnn}] --model-path MODEL_PATH [--config-file CONFIG_FILE] {train,predict,finetune} -SQLdiag integrated by openGauss. +SQLdiag integrated by MogDB. positional arguments: {train,predict,finetune} diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..4e0380d9a643c4738612918fd8778af1e1f3eaaa --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md @@ -0,0 +1,17 @@ +--- +title: 命令参考 +summary: 命令参考 +author: Guo Huan +date: 2021-05-19 +--- + +# 命令参考 + +**表 1** 命令行参数说明 + +| 参数 | 参数说明 | 取值范围 | +| :--------------- | :----------------- | :------------ | +| -f | 训练或预测文件位置 | - | +| --predicted-file | 预测结果存储位置 | - | +| --model | 模型选择 | template，dnn | +| --model-path | 训练模型存储位置 | - | diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-5-troubleshooting.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md similarity index 100% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-5-troubleshooting.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai4db-autonomous-database-o&m.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai4db-autonomous-database-o&m.md new file mode 100644 index 0000000000000000000000000000000000000000..f492afb8ee5d405b77618b8a3dc388af6ed76374 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/ai4db-autonomous-database-o&m.md @@ -0,0 +1,34 @@ +--- +title: AI4DB：数据库自治运维 +summary: AI4DB：数据库自治运维 +author: Guo Huan +date: 2022-05-06 +--- + +# AI4DB：数据库自治运维 + +如上文所述，AI4DB主要用于对数据库进行自治运维和管理，从而帮助数据库运维人员减少运维工作量。在实现上，DBMind的AI4DB框架具有监控和服务化的性质，同时也提供即时AI工具包，提供开箱即用的AI运维功能（如索引推荐）。AI4DB的监控平台以开源的Prometheus为主，DBMind提供监控数据生产者exporter, 可与Prometheus平台完成对接。DBMind的AI4DB服务架构如下图所示： + +**图 1** DBMind AI4DB服务架构 + +![DBMind-AI4DB服务架构](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/AI4DB-1.png) + +图中各关键组件说明： + +- DBMind Service：DBMind后台服务，可用于定期离线计算，包括慢SQL根因分析、时序预测等； +- Prometheus-server：Prometheus 监控指标存储的服务器； +- metadatabase：DBMind在离线计算结束后，将计算结果存储在此处，支持MogDB、SQLite等数据库； +- client：用户读取DBMind离线计算结果的客户端，目前仅实现命令行客户端；若采用MogDB等数据库存储计算DBMind计算结果，则用户可以自行配置Grafana等可视化工具对该结果进行可视化； +- openGauss-exporter：用户从MogDB数据库节点上采集监控指标，供DBMind服务进行计算； +- node-exporter：Prometheus官方提供的exporter, 可用于监控该节点的系统指标，如CPU和内存使用情况； +- reprocessing-exporter：用于对Prometheus采集到的指标进行二次加工处理，例如计算CPU使用率等。 + +## 环境配置 + +DBMind外置AI功能需要运行在Python 3.6版本及以上，需要的第三方依赖包记录在AI功能根目录（`$GAUSSHOME/bin/dbmind`）的`requirements.txt`文件中（包括`requirements-x86.txt`与`requirements-arrch64.txt`，用户可根据自己平台类型选择）中，可以通过`pip install`命令安装依赖，如： + +``` +pip install requirements-x86.txt +``` + +如果用户没有安装齐全所需的依赖，则当用户执行gs_dbmind命令时，会再次提醒用户安装第三方依赖。需要注意，该文件提供了DBMind所需的第三方依赖，若用户环境存在第三方包冲突等情况，可由用户根据实际情况进行处理。 \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md new file mode 100644 index 0000000000000000000000000000000000000000..7d9aef8455215b83e38ecd440831904e235ad79c --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md @@ -0,0 +1,10 @@ +--- +title: DBMind的支持组件 +summary: DBMind的支持组件 +author: Guo Huan +date: 2022-05-06 +--- + +# DBMind的支持组件 + +支持组件是指DBMind提供的用于支撑整个服务、解决方案能够部署和实施的模块。它们本身不是AI功能，却是整个服务体系中非常重要的一环，用于支撑整个自治运维解决方案的快速实施，如用于采集数据库指标的exporter等。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..efee411bc5f613d6be39cab3257e4361ce179f97 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md @@ -0,0 +1,136 @@ +--- +title: 命令参考 +summary: 命令参考 +author: Guo Huan +date: 2022-05-06 +--- + +# 命令参考 + +reprocessing-exporter的使用帮助详情： + +``` +gs_dbmind component reprocessing_exporter --help +usage: [-h] [--disable-https] [--ssl-keyfile SSL_KEYFILE] + [--ssl-certfile SSL_CERTFILE] + [--web.listen-address WEB.LISTEN_ADDRESS] + [--web.listen-port WEB.LISTEN_PORT] + [--collector.config COLLECTOR.CONFIG] [--log.filepath LOG.FILEPATH] + [--log.level {debug,info,warn,error,fatal}] [--version] + prometheus_host prometheus_port + +Reprocessing Exporter: A re-processing module for metrics stored in the +Prometheus server. + +positional arguments: + prometheus_host from which host to pull data + prometheus_port the port to connect to the Prometheus host + +optional arguments: + -h, --help show this help message and exit + --disable-https disable Https schema + --ssl-keyfile SSL_KEYFILE + set the path of ssl key file + --ssl-certfile SSL_CERTFILE + set the path of ssl certificate file + --web.listen-address WEB.LISTEN_ADDRESS + address on which to expose metrics and web interface + --web.listen-port WEB.LISTEN_PORT + listen port to expose metrics and web interface + --collector.config COLLECTOR.CONFIG + according to the content of the yaml file for metric + collection + --log.filepath LOG.FILEPATH + the path to log + --log.level {debug,info,warn,error,fatal} + only log messages with the given severity or above. + Valid levels: [debug, info, warn, error, fatal] + --version show program's version number and exit +``` + +**表 1** reprocessing-exporter的命令行参数详情表 + +| 参数 | 参数说明 | 取值范围 | +| :------------------- | :------------------------------------- | :------------------------------ | +| prometheus_host | Prometheus-server的IP地址 | - | +| prometheus_port | Prometheus-server的服务侦听端口号 | 1024-65535 | +| -h, --help | 帮助选项 | - | +| --disable-https | 禁用Https协议 | - | +| --ssl-keyfile | Https协议使用的证书私钥文件路径 | - | +| --ssl-certfile | Https协议使用的证书文件路径 | - | +| --web.listen-address | 该exporter服务的绑定IP | - | +| --web.listen-port | 该exporter服务的侦听端口 | 1024-65535 | +| --collector.config | 显性指定的待采集指标配置文件路径 | - | +| --log.filepath | 日志文件保存路径，默认保存在当前目录下 | - | +| --log.level | 日志文件的打印级别，默认为INFO级别 | debug, info, warn, error, fatal | +| --version | 显示版本信息 | - | + +openGauss-exporter的使用帮助详情： + +``` +gs_dbmind component opengauss_exporter --help +usage: [-h] --url URL [--config CONFIG] [--constant-labels CONSTANT_LABELS] + [--web.listen-address WEB.LISTEN_ADDRESS] + [--web.listen-port WEB.LISTEN_PORT] + [--web.telemetry-path WEB.TELEMETRY_PATH] [--disable-cache] + [--disable-settings-metrics] [--disable-statement-history-metrics] + [--disable-https] [--ssl-keyfile SSL_KEYFILE] + [--ssl-certfile SSL_CERTFILE] [--parallel PARALLEL] + [--log.filepath LOG.FILEPATH] + [--log.level {debug,info,warn,error,fatal}] [--version] + +openGauss Exporter (DBMind): Monitoring for MogDB. + +optional arguments: + -h, --help show this help message and exit + --url URL MogDB database target url. + --config CONFIG path to config dir or file. + --constant-labels CONSTANT_LABELS + a list of label=value separated by comma(,). + --web.listen-address WEB.LISTEN_ADDRESS + address on which to expose metrics and web interface + --web.listen-port WEB.LISTEN_PORT + listen port to expose metrics and web interface + --web.telemetry-path WEB.TELEMETRY_PATH + path under which to expose metrics. + --disable-cache force not using cache. + --disable-settings-metrics + not collect pg_settings.yml metrics. + --disable-statement-history-metrics + not collect statement-history metrics (including slow + queries). + --disable-https disable Https schema + --ssl-keyfile SSL_KEYFILE + set the path of ssl key file + --ssl-certfile SSL_CERTFILE + set the path of ssl certificate file + --parallel PARALLEL not collect pg_settings.yml metrics. + --log.filepath LOG.FILEPATH + the path to log + --log.level {debug,info,warn,error,fatal} + only log messages with the given severity or above. + Valid levels: [debug, info, warn, error, fatal] + --version show program's version number and exit +``` + +**表 2** openGauss-exporter的命令行参数详情表 + +| 参数 | 参数说明 | 取值范围 | +| :---------------------------------- | :----------------------------------------------------------- | :------------------------------ | +| --url | MogDB server的连接地址，例如 postgres://user:pwd@host:port/dbname | - | +| --constant-labels | 常量标签，将采集到的指标项中强行添加该标签列表 | 1024-65535 | +| -h, --help | 帮助选项 | - | +| --disable-https | 禁用Https协议 | - | +| --ssl-keyfile | Https协议使用的证书私钥文件路径 | - | +| --ssl-certfile | Https协议使用的证书文件路径 | - | +| --web.listen-address | 该exporter服务的绑定IP | - | +| --web.listen-port | 该exporter服务的侦听端口 | 1024-65535 | +| --web.telemetry-path | 该exporter采集指标的URI地址，默认为 /metrics | - | +| --config | 显性指定的待采集指标配置文件路径 | - | +| --log.filepath | 日志文件保存路径，默认保存在当前目录下 | - | +| --log.level | 日志文件的打印级别，默认为INFO级别 | debug, info, warn, error, fatal | +| --version | 显示版本信息 | - | +| --disable-cache | 禁止使用缓存 | - | +| --disable-settings-metrics | 禁止采集pg_settings表的值 | - | +| --disable-statement-history-metrics | 禁止收集statement_history表中的慢SQL信息 | - | +| --parallel | 连接到MogDB的数据库连接池的大小 | 正整数 | \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..18f86eb0dfe30c2f9eb80ae6d20234d06782fe6e --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md @@ -0,0 +1,33 @@ +--- +title: 环境部署 +summary: 环境部署 +author: Guo Huan +date: 2022-05-06 +--- + +# 环境部署 + +1. 通过命令行参数启动对应的exporter进程，同时在本地机器创建侦听端口号。 + +2. 在Promethues的服务器端修改配置文件prometheus.yml，将启动的exporter信息添加进去，例如： + + ``` + scrape_configs: + ... + - job_name: 'opengauss_exporter' + static_configs: + - targets: ['127.0.0.1:9187'] + ... + ``` + +其中，提供的exporter组件默认采用Https通信协议，因此需要用户默认提供ssl证书和秘钥文件，并通过--ssl-keyfile以及--ssl-certfile提供。若用户不希望使用Https协议，则可以通过--disable-https选项禁用该模式。 + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** +> +> 由于MogDB默认模式下的通信加密协议与PostgreSQL不兼容，故导致通过PyPI源安装的基于PostgreSQL编译的Python驱动psycopg2-binary默认无法连接至MogDB数据库。 +> 因此，需要用户自行编译psycopg2或修改GUC参数进行适配。也可通过openGauss官方网站下载基于openGauss编译的psycopg2（官方网站仅提供部分Python版本的编译包，一般为Python3.6版本，需要用户鉴别)。 +> +> - 官方openGauss Python驱动下载地址为： +> [https://opengauss.org/zh/download.html](https://opengauss.org/zh/download.html) +> - 关于Python驱动的适配问题，可参考openGauss官方操作指南： +> [https://mp.weixin.qq.com/s/2TobUQKtw0N9sBpMZJr6zw](https://mp.weixin.qq.com/s/2TobUQKtw0N9sBpMZJr6zw) diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..6adba93b3c6e65afe3a02d402c39646658d622f7 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md @@ -0,0 +1,15 @@ +--- +title: 获取帮助 +summary: 获取帮助 +author: Guo Huan +date: 2022-05-06 +--- + +# 获取帮助 + +用户可以通过--help命令获取帮助信息，例如： + +``` +gs_dbmind component opengauss_exporter --help +gs_dbmind component reprocessing_exporter --help +``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..4c0e9cbb3075e9b07bfeb43e77e0a5ece9b07fd2 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md @@ -0,0 +1,12 @@ +--- +title: 概述 +summary: 概述 +author: Guo Huan +date: 2022-05-06 +--- + +# 概述 + +Prometheus是业内非常流行的开源监控系统，同时本身也是一款时序数据库。Prometheus的采集端称之为exporter，用来收集被监控模块的指标项。为了与Prometheus平台完成对接，AI工具分别实现了两款exporter，分别是用来采集数据库指标的openGauss-exporter，以及对采集到的指标进行二次加工的reprocessing-exporter。 + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **须知：** Prometheus 和exporter是业内流行的监控和采集平台，部署在内网环境中，不对外部暴露接口，仅供内部监控平台使用。因此，为了增强该平台的安全性，一般需要用户或运维人员配置防火墙等，以便隔离外部访问，从而增强监控平台的安全性。 Prometheus平台在默认情况下，采用Http协议、并且没有任何安全访问限制。这是因为，该平台一般部署在内网环境中，攻击风险可控。如果用户希望提高安全性，可自行修改Prometheus的TLS配置选项，但仍不建议对外部直接暴露访问接口。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..8f907f92b9910bcc3d778037f78cb066ee5af9af --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md @@ -0,0 +1,16 @@ +--- +title: 常见问题处理 +summary: 常见问题处理 +author: Guo Huan +date: 2022-05-06 +--- + +# 常见问题处理 + +1. 提示需要用户提供--ssl-keyfile与--ssl-certfile选项： + + 上述exporter默认采用Https模式通信，因此需要用户指定证书及其私钥文件的路径。相反，如果用户只想采用Http模式，则需要显性给定--disable-https选项，从而禁用Https协议。 + +2. 提示用户需要输入PEM密码（Enter PEM pass phrase）： + + 如果用户采用Https模式，并给定了证书及其秘钥文件的路径，且该秘钥文件是经过加密的，则需要用户输入该加密私钥证书文件的密码。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..55d54d3b608c2f03741c4642d5e3970e04efec36 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md @@ -0,0 +1,62 @@ +--- +title: 使用指导 +summary: 使用指导 +author: Guo Huan +date: 2022-05-06 +--- + +# 使用指导 + +用户可通过gs_dbmind命令启动对应的exporter。下面为用户演示一个完整的Prometheus监控平台的搭建过程。 + +1. 部署openGauss-exporter：启动openGauss-exporter，采用默认侦听端口号9187，侦听地址为192.168.1.100，不采用https协议，则命令可以为： + + ``` + gs_dbmind component opengauss_exporter --url postgresql://user:password@ip:port/dbname --web.listen-address 192.168.1.100 --disable-https + ``` + +2. 部署reprocessing-exporter：启动reprocessing-exporter，采用默认侦听端口号8181，侦听地址为192.168.1.101，Prometheus-server IP与端口号为192.168.1.100:9090，采用https协议，则命令可以为： + + ``` + gs_dbmind component reprocessing_exporter 192.168.1.100 9090 --web.listen-address 192.168.1.101 --ssl-keyfile server.key --ssl-certfile server.crt + ``` + +3. 部署node-exporter：一般地，对于Prometheus监控平台都需要部署node-exporter用于监控Linux操作系统，后文提到的部分AI功能也需要依赖node-exporter采集Linux系统指标，故也需要用户来部署；只不过，该node-exporter为Prometheus自带组件，需要用户自行下载安装，下载地址为[https://prometheus.io/download/#node_exporter](https://prometheus.io/download/#node_exporter)，使用方法详见：[https://prometheus.io/docs/guides/node-exporter/#installing-and-running-the-node-exporter](https://prometheus.io/docs/guides/node-exporter/#installing-and-running-the-node-exporter)。用户可解压压缩包后直接运行该node-exporter进程，其默认端口号为9100，启动命令行为： + + ``` + ./node_exporter + ``` + +4. 配置Prometheus-server，修改配置文件prometheus.yml，添加下述内容： + + ``` + scrape_configs: + ... + - job_name: 'opengauss_exporter' + static_configs: + - targets: ['192.168.1.100:9187'] + - job_name: 'reprocessing_exporter' + scheme: https + tls_config: + ca_file: xxx.crt + static_configs: + - targets: ['127.0.0.1:8181'] + - job_name: 'node_exporter' + static_configs: + - targets: ['127.0.0.1:9100'] + ... + ``` + +> ![img](https://opengauss.org/zh/docs/3.0.0/docs/Developerguide/public_sys-resources/icon-caution.gif) **注意：** +> +> - openGauss-exporter中连接数据库的用户需要monitor admin或以上权限，否则会出现部分指标无法采集的情况。 +> - openGauss-exporter会从dbe_perf.statement_history中抽样慢SQL信息，dbe_perf.statement_history视图慢SQL记录与GUC参数log_min_duration_statement和track_stmnt_stat_level相关，其中log_min_duration_statement是慢SQL阈值，单位毫秒，具体值由用户设置；track_stmnt_stat_level是SQL记录级别，默认为'OFF,L0'，即只记录慢SQL信息，级别为L0，用户在详细了解参数意义与作用情况下谨慎修改。 +> - openGauss-exporter采集数据库相关信息，主要包括部分系统表和视图中的数据（具体参见代码中opengauss_exporter中的配置文件），node-exporter采集系统指标信息，主要与系统磁盘、CPU等相关，reprocessing_exporter基于prometheus-server中的某些指标（具体参见代码中reprocessing_exporter中的配置文件）进行二次加工，最终提供加工后的数据供用户使用。 +> - prometheus-server在拉取exporter数据时有超时机制，超时时间由scrape_timeout（默认10s）控制，因此当exporter采集数据量较大时，用户可根据实际情况增大scrape_timeout以防止超时报错，另外需要注意的是scrape_interval（采集间隔）不能比scrape_timeout小，否则会出现异常。 +> - 如果数据库时区设置和系统不相同，可能会出现时间相关指标时间与系统时间不一致的情况，因此需要将用户时区与系统保持同步。 +> - 当使用https通信时，工具会检测证书与密钥文件权限以及证书有效期，如果文件权限大于600则会出现报警，证书有效期小于90天会出现报警。 +> - 当存在指标重复采集时，openGauss-exporter会出现异常，异常信息会打印到日志中。 +> - openGauss-exporter的--config、--disable-settings-metrics、--disable-statement-history-metrics三个参数需要注意，其存在以下几种情况： +> 1. 用户不指定其中任何参数，则工具会同时对yamls目录下的三个配置文件中的指标进行采集。 +> 2. 用户显式指定--config，则工具不会采集yamls目录下default.yml中的指标，而会采集用户指定配置文件中的指标，同时pg_settings.yml和statements.yml正常采集，此时用户需要注意指定的配置文件中的指标和pg_settings.yml、statements.yml中的指标不能存在重复采集的现象。 +> 3. 用户显式指定--disable-settings-metrics，则工具不会采集yamls目录下pg_settings.yml中的指标，用户显式指定--disable-statement-history-metrics，则工具不会采集yamls目录下statements.yml（慢SQL相关)中的指标。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/1-service.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/1-service.md new file mode 100644 index 0000000000000000000000000000000000000000..61e6fc10c0d817467887f7333c13c9da5602b697 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/1-service.md @@ -0,0 +1,148 @@ +--- +title: service子命令 +summary: service子命令 +author: Guo Huan +date: 2022-05-06 +--- + +# service子命令 + +该子命令可用于对配置目录进行初始化，同时也可以实现启动和停止后台任务。 + +## 配置目录初始化 + +用户可通过`gs_dbmind service setup`子命令进行配置目录的初始化。该配置文件中可包括DBMind的配置文件、日志等内容。该目录中的部分文件说明： + +- dbmind.conf：DBMind的参数配置文件，用户可通过`gs_dbmind set`命令进行修改，也可通过文本编辑器进行手动修改。 +- dynamic_config.db：DBMind服务保存在本地节点的元信息，主要包括算法的超参数、监控阈值等；该文件为DBMind服务元信息，不可由用户直接配置。 +- metric_map.conf：监控指标映射表，可用于适配到不同采集平台中。例如，在DBMind中，监控到的系统cpu使用率名为os_cpu_usage，而用户自行实现的指标采集工具将cpu使用率命名为my_cpu_usage_rate。在该种情况下，如果想要DBMind代表cpu使用率的指标名为my_cpu_usage_rate，则需要修改该配置选项。即添加“os_cpu_usage = my_cpu_usage_rate”配置项进行映射。对于普通用户，建议直接使用DBMind配套的采集组件和方案，则不涉及修改该配置文件。 +- logs: 该目录中用于存储DBMind服务产生的日志。 + +用户可通过两种方式进行配置目录的初始化，一种为交互式，另一种为非交互式。例如，待初始化的配置目录名为**confpath**, 则分别通过下述方法进行配置： + +**交互式模式** + +``` +gs_dbmind service setup -c confpath --interactive +``` + +执行完毕上述命令后，用户可通过命令行终端对配置项进行交互式配置。 + +**非交互式模式** + +非交互式模式总共分为三个步骤，即启动配置，修改配置项，初始化配置。其中第二个步骤需要用户通过文本编辑器手动编辑配置文件。具体步骤如下： + +1. 启动配置，执行下述命令： + + ``` + gs_dbmind service setup -c confpath + ``` + +2. 执行完上述命令后，会在confpath目录下生成dbmind.conf配置文件，用户需要利用文本编辑器进行手动修改。需要修改的配置部分为“指标数据源数据库信息区【TSDB】”、“预测结果存储数据库信息区【METADATABASE】”和“自监控参数配置区【SELF-MONITORING】”相关参数的说明如下： + + ``` + [TSDB] # 时序数据库相关信息，即指标数据源信息 + name = prometheus # 时序数据库类型. 当前仅支持选择'prometheus'. + host = # 时序数据库IP地址. + port = # 时序数据库端口号. + + [METADATABASE] # 元数据库相关信息，及预测结果存储方式 + dbtype = # 元数据库类型. 可选择: sqlite, mogdb, postgresql。若该配置项名为mogdb, 需关注Python驱动事宜，可参考下文中关于Python驱动的提示。 + host = # 元数据库地址. + port = # 元数据库端口号. + username = # 元数据库用户信息. + password = # 元数据库密码，将会被加密存储 + database = # 元数据库库名. 如果选择的数据库类型为SQLite, 则无需填写上述配置项，仅需填写此处。DBMind会根据database配置项的内容，在配置目录中生成对应文件名的SQLite数据库文件。 + + [SELF-MONITORING] # 自监控参数配置 + detection_interval = 600 # 单位秒. 对mogdb进行健康检查的时间间隔（仅慢SQL根因诊断使用）. + last_detection_time = 600 # 单位秒. 用于健康检查的监控数据长度（仅慢SQL根因诊断使用）. + forecasting_future_time = 3600 # 单位秒，时序预测的长度（供时序预测特性使用），同时也是预测功能调度周期 + # 待时序预测进行预测的关键指标项（仅供时序预测特性使用） + golden_kpi = os_cpu_usage, os_mem_usage, gaussdb_qps_by_instance + + [LOG] # 日志相关信息 + maxbytes = 10485760 # 默认值为 10Mb, 单个日志文件的最大大小.如果 maxbytes 为零，则文件无限增长（建议该值不要设置太小，默认即可）. + backupcount = 1 # 日志文件最大数量. + level = INFO # 日志级别，也可配置为 DEBUG, INFO, WARNING, ERROR模式。 + ``` + +3. 待用户手动修改完上述参数后，需要执行下述命令进行配置项的初始化。在该阶段中，DBMind会初步检查配置项的正确性、加密配置项中出现的明文密码、同时初始化用于存储结果数据的元数据库。 + + ``` + gs_dbmind service setup --initialize -c confpath + ``` + +4. 完成配置目录初始化过程，可基于该配置目录启动DBMind后台服务。 + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** +> +> 1. 配置文件注释信息用于在交互模式下对用户进行提示，有特殊含义不要手动修改或删除； +> 2. 需要确保配置项的值与注释信息之间通过空格符分割，否则系统会将注释信息识别为配置项的值； +> 3. 配置项中的特殊字符，如果需要转移，则通过转义符“百分号”(%)来转义，例如，用户配置的密码为 “password%”, 则应通过“百分号”进行转义，即 “password%%“。 + +## 启动服务 + +当用户完成配置目录的初始化后，可基于此配置目录启动DBMind后台服务。例如配置目录为confpath，则启动命令如下： + +``` +gs_dbmind service start -c confpath +``` + +当执行上述命令后，会提示服务已启动。在未指定任何附加参数时，该命令默认会启动所有的后台任务。如果用户只想启动某一个后台任务，需要添加参数--only-run。例如，用户只想启动慢SQL根因分析服务，则为： + +``` +gs_dbmind service start -c confpath --only-run slow_query_diagnosis +``` + +## 关闭服务 + +关闭服务与启动服务类似，其命令行结构更加简单，只需指定配置目录的地址即可。例如配置目录为confpath，则为： + +``` +gs_dbmind service stop -c confpath +``` + +DBMind服务会在后台执行完正在运行的任务后自行退出。 + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **注意：** +> +> - [METADATABASE]中的元数据库用户需要具有在该数据库下的创表和数据插入更新权限，否则工具执行会出现异常。 +> - 当前不支持同一配置文件下分开启动多个服务。 +> - 工具提供了requirement.txt文件，用户可以通过该文件安装所需的第三方依赖。 + +## 命令参考 + +用户可以通过--help选项获得该模式的帮助信息，例如： + +``` +gs_dbmind service --help +``` + +``` +usage: service [-h] -c DIRECTORY [--only-run {slow_query_diagnosis,forecast}] [--interactive | --initialize] {setup,start,stop} + +positional arguments: + {setup,start,stop} + perform an action for service + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files + --only-run {slow_query_diagnosis,forecast} + explicitly set a certain task running in the backend + --interactive configure and initialize with interactive mode + --initialize initialize and check configurations after configuring. +``` + +**表 1** gs_dbmind service 子命令说明 + +| 参数 | 参数说明 | 取值范围 | +| :------------ | :----------------- | :----------------------------------------------------------- | +| action | 动作参数 | - setup：初始化配置目录。
- start：服务启动。
- stop：服务停止。 | +| -c，--conf | 配置文件目录地址 | - | +| --initialize | 配置参数初始化 | - | +| --interactive | 交互式输入配置参数 | - | +| --only-run | 选择只运行的模块 | - forecast：预测模块。
- slow_query_diagnosis：慢SQL根因分析模块。 | +| -h, --help | 帮助命令 | - | \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/2-component.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/2-component.md new file mode 100644 index 0000000000000000000000000000000000000000..a41e2f702d2d1e3d2634a8b560a240b1f95839e7 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/2-component.md @@ -0,0 +1,37 @@ +--- +title: component子命令 +summary: component子命令 +author: Guo Huan +date: 2022-05-06 +--- + +# component子命令 + +该子命令可以用于启动DBMind的组件，包括可用于监控指标的exporter，以及AI功能等。该命令可以将用户通过命令行传入的命令转发给对应的子组件，故不同的子组件命令需参考其功能的对应说明，详见后文各个子组件对应章节，此处不再赘述。 + +## 命令参考 + +用户可以通过--help选项获得该模式的帮助信息，例如： + +``` +gs_dbmind component --help +``` + +``` +usage: component [-h] COMPONENT_NAME ... + +positional arguments: + COMPONENT_NAME choice a component to start. ['extract_log', 'forecast', 'index_advisor', 'opengauss_exporter', 'reprocessing_exporter', 'slow_query_diagnosis', 'sqldiag', 'xtuner'] + ARGS arguments for the component to start + +optional arguments: + -h, --help show this help message and exit +``` + +**表 1** gs_dbmind component 子命令说明 + +| 参数 | 参数说明 | 取值范围 | +| :------------- | :----------- | :----------------------------------------------------------- | +| COMPONENT_NAME | 子组件名 | extract_log, forecast, index_advisor, opengauss_exporter, reprocessing_exporter, slow_query_diagnosis, sqldiag, xtuner | +| ARGS | 子组件的参数 | 参考子组件的命令说明 | +| -h, --help | 帮助命令 | - | \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/3-set.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/3-set.md new file mode 100644 index 0000000000000000000000000000000000000000..5ebf2a8fb4086725f81eace9e719ef9504406cfc --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/3-set.md @@ -0,0 +1,54 @@ +--- +title: set子命令 +summary: set子命令 +author: Guo Huan +date: 2022-05-06 +--- + +# set子命令 + +该命令用于修改配置文件dbmind.conf中的参数值，与用户手动修改配置文件dbmind.conf一般无差异。例如修改配置目录confpath中的配置文件dbmind.conf中TSDB配置部分，host参数的值，并将其设置为127.0.0.1。则可通过下述命令实现： + +``` +gs_dbmind set TSDB host 127.0.0.1 -c confpath +``` + +在修改上述普通参数时，与手动修改配置文件dbmind.conf无差异。但是，当用户想要修改密码项时，则必须通过该命令才可以实现修改，否则通过用户手动修改的配置文件无法实现对明文密码的加密，例如： + +``` +gs_dbmind set METADATABASE password xxxxxx -c confpath +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 该命令对于字符串是大小写敏感的，如果输错则可能出现执行过程错误。 + +## 命令参考 + +用户可以通过--help选项获得该模式的帮助信息，例如： + +``` +gs_dbmind set --help +``` + +``` +usage: set [-h] -c DIRECTORY section option target + +positional arguments: + section which section (case sensitive) to set + option which option to set + target the parameter target to set + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files +``` + +**表 1** 模块命令行参数说明：python dbmind/ set xxx + +| 参数 | 参数说明 | 取值范围 | +| :--------- | :------------------- | :------- | +| -h, --help | 帮助命令 | - | +| -c，--conf | 配置文件目录confpath | - | +| section | 设置区 | - | +| option | 设置项 | - | +| target | 设置值 | - | \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/dbmind-mode.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/dbmind-mode.md new file mode 100644 index 0000000000000000000000000000000000000000..02122a2c227155b024de1867f0326abe17d0fe8a --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/ai4db/dbmind-mode/dbmind-mode.md @@ -0,0 +1,47 @@ +--- +title: DBMind模式说明 +summary: DBMind模式说明 +author: Guo Huan +date: 2022-05-06 +--- + +# DBMind模式说明 + +用户可通过gs_dbmind命令调用AI4DB的全部功能，该命令可实现下列基本功能： + +- 服务功能：service子命令，包括创建并初始化配置目录、启动后台服务、关闭后台服务等； +- 调用组件：component子命令，AI4DB功能（如索引推荐、参数调优等）可通过该模式进行即时调用； +- 设置参数：set子命令，通过该命令，可以一键修改配置目录中的配置文件值；当然，用户也可以通过文本编辑器进行手动修改； + +用户可以通过--help选项获得上述模式的帮助信息，例如： + +``` +gs_dbmind --help +``` + +``` +usage: [-h] [--version] {service,set,component} ... + +MogDB DBMind: An autonomous platform for MogDB + +optional arguments: + -h, --help show this help message and exit + --version show program's version number and exit + +available subcommands: + {service,set,component} + type ' -h' for help on a specific subcommand + service send a command to DBMind to change the status of the service + set set a parameter + component pass command line arguments to each sub-component. +``` + +**表 1** gs_dbmind 选项基本说明 + +| 参数 | 参数说明 | 取值范围 | +| :--------- | :------------------- | :------- | +| -h, --help | 帮助命令 | - | +| --version | 版本号 | - | +| service | 服务功能相关的子命令 | - | +| component | 调用组件的子命令 | - | +| set | 修改配置文件的子命令 | - | \ No newline at end of file diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-1-overview.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/db4ai.md similarity index 83% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-1-overview.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/db4ai.md index e4eccd05571c42acec8e0eb1c145279f7ac2e1fc..7335221a5931022cd09a40564c8530f97c6014b0 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-1-overview.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/db4ai.md @@ -1,10 +1,10 @@ ---- -title: 概述 -summary: 概述 -author: Guo Huan -date: 2021-05-19 ---- - -# 概述 - -DB4AI是指利用数据库的能力驱动AI任务，实现数据存储、技术栈的同构。通过在数据库内集成AI算法，令MogDB具备数据库原生AI计算引擎、模型管理、AI算子、AI原生执行计划的能力，为用户提供普惠AI技术。不同于传统的AI建模流程，DB4AI“一站式”建模可以解决数据在各平台的反复流转问题，同时简化开发流程，并可通过数据库规划出最优执行路径，让开发者更专注于具体业务和模型的调优上，具备同类产品不具备的易用性与性能优势。 +--- +title: DB4AI：数据库驱动AI +summary: DB4AI：数据库驱动AI +author: Guo Huan +date: 2022-05-06 +--- + +# DB4AI：数据库驱动AI + +DB4AI是指利用数据库的能力驱动AI任务，实现数据存储、技术栈的同构。通过在数据库内集成AI算法，令MogDB具备数据库原生AI计算引擎、模型管理、AI算子、AI原生执行计划的能力，为用户提供普惠AI技术。不同于传统的AI建模流程，DB4AI“一站式”建模可以解决数据在各平台的反复流转问题，同时简化开发流程，并可通过数据库规划出最优执行路径，让开发者更专注于具体业务和模型的调优上，具备同类产品不具备的易用性与性能优势。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-2-db4ai-snapshots-for-data-version-management.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md similarity index 78% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-2-db4ai-snapshots-for-data-version-management.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md index fc623932193f837ade1bd0af0ebdbdd324764a9e..135caea7e76340107cd5eafc50bca364b2b21271 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-2-db4ai-snapshots-for-data-version-management.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md @@ -1,265 +1,266 @@ ---- -title: DB4AI-Snapshots：数据版本管理 -summary: DB4AI-Snapshots数据版本管理 -author: Guo Huan -date: 2021-10-20 ---- - -# DB4AI-Snapshots：数据版本管理 - -DB4AI-Snapshots是DB4AI模块用于管理数据集版本的功能。通过DB4AI-Snapshots组件，开发者可以简单、快速地进行特征筛选、类型转换等数据预处理操作，同时还可以像git一样对训练数据集进行版本控制。数据表快照创建成功后可以像视图一样进行使用，但是一经发布后，数据表快照便固化为不可变的静态数据，如需修改该数据表快照的内容，需要创建一个版本号不同的新数据表快照。 - -## DB4AI-Snapshots的生命周期 - -DB4AI-Snapshots的状态包括published、archived以及purged。其中，published可以用于标记该DB4AI-Snapshots已经发布，可以进行使用。archived表示当前DB4AI-Snapshots处于“存档期”，一般不进行新模型的训练，而是利用旧数据对新的模型进行验证。purged则是该DB4AI-Snapshots已经被删除的状态，在数据库系统中无法再检索到。 - -需要注意的是快照管理功能是为了给用户提供统一的训练数据，不同团队成员可以使用给定的训练数据来重新训练机器学习模型，方便用户间协同。为此**私有用户**和**三权分立**状态（enableSeparationOfDuty=ON）等涉及不支持用户数据转写等情况将不支持Snapshot特性。 - -用户可以通过“CREATE SNAPSHOT”语句创建数据表快照，创建好的快照默认即为published状态。可以采用两种模式创建数据表快照，即为MSS以及CSS模式，它们可以通过GUC参数db4ai_snapshot_mode进行配置。对于MSS模式，它是采用物化算法进行实现的，存储了原始数据集的数据实体；CSS则是基于相对计算算法实现的，存储的是数据的增量信息。数据表快照的元信息存储在DB4AI的系统目录中。可以通过db4ai.snapshot系统表查看到。 - -可以通过“ARCHIVE SNAPSHOT”语句将某一个数据表快照标记为archived状态，可以通过“PUBLISH SNAPSHOT”语句将其再度标记为published状态。标记数据表快照的状态，是为了帮助数据科学家进行团队合作使用的。 - -当一个数据表快照已经丧失存在价值时，可以通过“PURGE SNAPSHOT”语句删除它，以便永久删除其数据并恢复存储空间。 - -## DB4AI-Snapshots使用指导 - -1. 创建表以及插入表数据。 - - 数据库内存在已有的数据表，可根据该已有的数据表创建对应的数据表快照。为了后续演示，在此处新建一个名为t1的数据表，并向其中插入测试数据。 - - ```sql - create table t1 (id int, name varchar); - insert into t1 values (1, 'zhangsan'); - insert into t1 values (2, 'lisi'); - insert into t1 values (3, 'wangwu'); - insert into t1 values (4, 'lisa'); - insert into t1 values (5, 'jack'); - ``` - - 通过SQL语句，查询搭配数据表内容。 - - ```sql - SELECT * FROM t1; - id | name - ----+---------- - 1 | zhangsan - 2 | lisi - 3 | wangwu - 4 | lisa - 5 | jack - (5 rows) - ``` - -2. 使用DB4AI-Snapshots。 - - - 创建DB4AI-Snapshots - - - 示例1：CREATE SNAPSHOT…AS - - 示例如下，其中，默认版本分隔符为 “@”, 默认子版本分割符为 “.”，该分割符可以分别通过GUC参数db4ai_snapshot_version_delimiter以及db4ai_snapshot_version_separator进行设置。 - - ```sql - create snapshot s1@1.0 comment is 'first version' as select * from t1; - schema | name - --------+-------- - public | s1@1.0 - (1 row) - ``` - - 上述结果提示已经创建了数据表s1的快照，版本号为1.0。创建好后的数据表快照可以像使用一般视图一样进行查询，但不支持通过“INSERT INTO”语句进行更新。例如下面几种语句都可以查询到数据表快照s1的对应版本1.0的内容： - - ```sql - SELECT * FROM s1@1.0; - SELECT * FROM public.s1@1.0; - SELECT * FROM public . s1 @ 1.0; - id | name - ----+---------- - 1 | zhangsan - 2 | lisi - 3 | wangwu - 4 | lisa - 5 | jack - (5 rows) - ``` - - 可以通过下列SQL语句修改数据表t1的内容： - - ```sql - UPDATE t1 SET name = 'tom' where id = 4; - insert into t1 values (6, 'john'); - insert into t1 values (7, 'tim'); - ``` - - 再检索数据表t1的内容时，发现虽然数据表t1的内容已经发生变化，但是数据表快照s1@1.0版本的查询结果并未发生变化。由于数据表t1的数据已经发生了改变，如果将当前数据表的内容作为版本2.0，则可创建快照**s1@2.0**,创建的SQL语句如下： - - ```sql - create snapshot s1@2.0 as select * from t1; - ``` - - 通过上述例子，我们可以发现，数据表快照可以固化数据表的内容，避免中途对数据的改动造成机器学习模型训练时的不稳定，同时可以避免多用户同时访问、修改同一个表时造成的锁冲突。 - - - 示例2：CREATE SNAPSHOT…FROM - - SQL语句可以对一个已经创建好的数据表快照进行继承，利用在此基础上进行的数据修改产生一个新的数据表快照。例如： - - ```sql - create snapshot s1@3.0 from @1.0 comment is 'inherits from @1.0' using (INSERT VALUES(6, 'john'), (7, 'tim'); DELETE WHERE id = 1); - schema | name - --------+-------- - public | s1@3.0 - (1 row) - ``` - - 其中，“@”为数据表快照的版本分隔符，from子句后加上已存在的数据表快照，用法为“@”+版本号，USING关键字后加入可选的几个操作关键字（INSERT …/UPDATE …/DELETE …/ALTER …），其中 “INSERT INTO”以及“DELETE FROM”语句中的“INTO”、“FROM”等与数据表快照名字相关联的子句可以省略，具体可以参考[AI特性函数](28-ai-feature-functions)。 - - 示例中，基于前述s1@1.0快照，插入2条数据，删除1条新的数据，新生成的快照s1@3.0，检索该s1@3.0： - - ```sql - SELECT * FROM s1@3.0; - id | name - ----+---------- - 1 | zhangsan - 2 | lisi - 3 | wangwu - 4 | lisa - 5 | jack - 6 | john - 7 | tim - (7 rows) - ``` - - - 删除数据表快照SNAPSHOT - - ```sql - purge snapshot s1@3.0; - schema | name - --------+-------- - public | s1@3.0 - (1 row) - ``` - - 此时，已经无法再从s1@3.0 中检索到数据了，同时该数据表快照在db4ai.snapshot视图中的记录也会被清除。删除该版本的数据表快照不会影响其他版本的数据表快照。 - - - 从数据表快照中采样 - - 示例：从snapshot s1中抽取数据，使用0.5抽样率。 - - ```sql - sample snapshot s1@2.0 stratify by name as nick at ratio .5; - schema | name - --------+------------ - public | s1nick@2.0 - (1 row) - ``` - - 可以利用该功能创建训练集与测试集，例如： - - ```sql - SAMPLE SNAPSHOT s1@2.0 STRATIFY BY name AS _test AT RATIO .2, AS _train AT RATIO .8 COMMENT IS 'training'; - schema | name - --------+---------------- - public | s1_test@2.0 - public | s1_train@2.0 - (2 rows) - ``` - - - 发布数据表快照 - - 采用下述SQL语句将数据表快照s1@2.0标记为published状态： - - ```sql - publish snapshot s1@2.0; - schema | name - --------+-------- - public | s1@2.0 - (1 row) - ``` - - - 存档数据表快照 - - 采用下述语句可以将数据表快照标记为archived状态： - - ```sql - archive snapshot s1@2.0; - schema | name - --------+-------- - public | s1@2.0 - (1 row) - ``` - - 可以通过db4ai-snapshots提供的视图查看当前数据表快照的状态以及其他信息： - - ```sql - select * from db4ai.snapshot; - id | parent_id | matrix_id | root_id | schema | name | owner | commands | comment | published | archived | created | row_count - ----+-----------+-----------+---------+--------+------------+--------+------------------------------------------+---------+-----------+----------+----------------------------+----------- - 1 | | | 1 | public | s1@2.0 | omm | {"select *","from t1 where id > 3",NULL} | | t | f | 2021-04-17 09:24:11.139868 | 2 - 2 | 1 | | 1 | public | s1nick@2.0 | omm | {"SAMPLE nick .5 {name}"} | | f | f | 2021-04-17 10:02:31.73923 | 0 - ``` - -3. 异常场景 - - - 数据表或db4ai-snapshots不存在时。 - - ```sql - purge snapshot s1nick@2.0; - publish snapshot s1nick@2.0; - --------- - ERROR: snapshot public."s1nick@2.0" does not exist - CONTEXT: PL/pgSQL function db4ai.publish_snapshot(name,name) line 11 at assignment - - archive snapshot s1nick@2.0; - ---------- - ERROR: snapshot public."s1nick@2.0" does not exist - CONTEXT: PL/pgSQL function db4ai.archive_snapshot(name,name) line 11 at assignment - ``` - - - 删除snapshot时，有依赖该快照的其他snapshot，需先确保删除对本快照所依赖的其他快照。 - - ```sql - purge snapshot s1@1.0; - ERROR: cannot purge root snapshot 'public."s1@1.0"' having dependent db4ai-snapshots - HINT: purge all dependent db4ai-snapshots first - CONTEXT: referenced column: purge_snapshot_internal - SQL statement "SELECT db4ai.purge_snapshot_internal(i_schema, i_name)" - PL/pgSQL function db4ai.purge_snapshot(name,name) line 62 at PERFORM - ``` - -4. 相关GUC参数 - - - db4ai_snapshot_mode： - - Snapshot有2种模式：MSS（物化模式，存储数据实体）和CSS（计算模式，存储增量信息）。Snapshot可在MSS和CSS之间切换快照模式，默认是MSS模式。 - - - db4ai_snapshot_version_delimiter： - - 该参数为数据表快照版本分隔符。“@”为数据表快照的默认版本分隔符。 - - - db4ai_snapshot_version_separator： - - 该参数为数据表快照子版本分隔符。“.”为数据表快照的默认版本分隔符。 - -5. DB4AI Schema下的数据表快照详情db4ai.snapshot。 - - ```sql - mogdb=# \d db4ai.snapshot - Table "db4ai.snapshot" - Column | Type | Modifiers - -----------+-----------------------------+--------------------------- - id | bigint | - parent_id | bigint | - matrix_id | bigint | - root_id | bigint | - schema | name | not null - name | name | not null - owner | name | not null - commands | text[] | not null - comment | text | - published | boolean | not null default false - archived | boolean | not null default false - created | timestamp without time zone | default pg_systimestamp() - row_count | bigint | not null - Indexes: - "snapshot_pkey" PRIMARY KEY, btree (schema, name) TABLESPACE pg_default - "snapshot_id_key" UNIQUE CONSTRAINT, btree (id) TABLESPACE pg_default - ``` +--- +title: DB4AI-Snapshots数据版本管理 +summary: DB4AI-Snapshots数据版本管理 +author: Guo Huan +date: 2022-05-06 +--- + +# DB4AI-Snapshots数据版本管理 + +DB4AI-Snapshots是DB4AI模块用于管理数据集版本的功能。通过DB4ai-Snapshots组件，开发者可以简单、快速地进行特征筛选、类型转换等数据预处理操作，同时还可以像git一样对训练数据集进行版本控制。数据表快照创建成功后可以像视图一样进行使用，但是一经发布后，数据表快照便固化为不可变的静态数据，如需修改该数据表快照的内容，需要创建一个版本号不同的新数据表快照。 + +## DB4AI-Snapshots的生命周期 + +DB4AI-Snapshots的状态包括published、archived以及purged。其中，published可以用于标记该DB4AI-Snapshots 已经发布，可以进行使用。archived表示当前 DB4AI-Snapshots 处于“存档期”，一般不进行新模型的训练，而是利用旧数据对新的模型进行验证。purged则是该DB4AI-Snapshots 已经被删除的状态，在数据库系统中无法再检索到。 + +需要注意的是快照管理功能是为了给用户提供统一的训练数据，不同团队成员可以使用给定的训练数据来重新训练机器学习模型，方便用户间协同。为此**私有用户**和**三权分立**状态(enableSeparationOfDuty=ON)等涉及不支持用户数据转写等情况将不支持Snapshot特性。 + +用户可以通过“CREATE SNAPSHOT”语句创建数据表快照，创建好的快照默认即为published状态。可以采用两种模式创建数据表快照，即为MSS以及CSS模式，它们可以通过GUC参数db4ai_snapshot_mode进行配置。对于MSS模式，它是采用物化算法进行实现的，存储了原始数据集的数据实体；CSS则是基于相对计算算法实现的，存储的是数据的增量信息。数据表快照的元信息存储在DB4AI的系统目录中。可以通过db4ai.snapshot 系统表查看到。 + +可以通过“ARCHIVE SNAPSHOT”语句将某一个数据表快照标记为archived状态，可以通过“PUBLISH SNAPSHOT”语句将其再度标记为published状态。标记数据表快照的状态，是为了帮助数据科学家进行团队合作使用的。 + +当一个数据表快照已经丧失存在价值时，可以通过“PURGE SNAPSHOT”语句删除它，以便永久删除其数据并恢复存储空间。 + +## DB4AI-Snapshots使用指导 + +1. 创建表以及插入表数据。 + + 数据库内存在已有的数据表，可根据该已有的数据表创建对应的数据表快照。为了后续演示，在此处新建一个名为 t1 的数据表，并向其中插入测试数据。 + + ``` + create table t1 (id int, name varchar); + insert into t1 values (1, 'zhangsan'); + insert into t1 values (2, 'lisi'); + insert into t1 values (3, 'wangwu'); + insert into t1 values (4, 'lisa'); + insert into t1 values (5, 'jack'); + ``` + + 通过SQL语句，查询搭配数据表内容。 + + ``` + SELECT * FROM t1; + id | name + ----+---------- + 1 | zhangsan + 2 | lisi + 3 | wangwu + 4 | lisa + 5 | jack + (5 rows) + ``` + +2. 使用DB4AI-Snapshots。 + + - 创建DB4AI-Snapshots + + - 示例1：CREATE SNAPSHOT…AS + + 示例如下，其中，默认版本分隔符为 “@”, 默认子版本分割符为 “.”，该分割符可以分别通过GUC参数db4ai_snapshot_version_delimiter以及db4ai_snapshot_version_separator进行设置。 + + ``` + create snapshot s1@1.0 comment is 'first version' as select * from t1; + schema | name + --------+-------- + public | s1@1.0 + (1 row) + ``` + + 上述结果提示已经创建了数据表 s1的快照，版本号为 1.0。创建好后的数据表快照可以像使用一般视图一样进行查询，但不支持通过“INSERT INTO”语句进行更新。例如下面几种语句都可以查询到数据表快照s1的对应版本 1.0的内容： + + ``` + SELECT * FROM s1@1.0; + SELECT * FROM public.s1@1.0; + SELECT * FROM public . s1 @ 1.0; + id | name + ----+---------- + 1 | zhangsan + 2 | lisi + 3 | wangwu + 4 | lisa + 5 | jack + (5 rows) + ``` + + 可以通过下列SQL语句修改数据表t1的内容： + + ``` + UPDATE t1 SET name = 'tom' where id = 4; + insert into t1 values (6, 'john'); + insert into t1 values (7, 'tim'); + ``` + + 再检索数据表t1的内容时，发现虽然数据表t1的内容已经发生变化，但是数据表快照 s1@1.0 版本的查询结果并未发生变化。由于数据表t1的数据已经发生了改变，如果将当前数据表的内容作为版本2.0，则可创建快照**s1@2.0**,创建的SQL语句如下： + + ``` + create snapshot s1@2.0 as select * from t1; + ``` + + 通过上述例子，我们可以发现，数据表快照可以固化数据表的内容，避免中途对数据的改动造成机器学习模型训练时的不稳定，同时可以避免多用户同时访问、修改同一个表时造成的锁冲突。 + + - 示例2：CREATE SNAPSHOT…FROM + + SQL语句可以对一个已经创建好的数据表快照进行继承，利用在此基础上进行的数据修改产生一个新的数据表快照。例如： + + ``` + create snapshot s1@3.0 from @1.0 comment is 'inherits from @1.0' using (INSERT VALUES(6, 'john'), (7, 'tim'); DELETE WHERE id = 1); + schema | name + --------+-------- + public | s1@3.0 + (1 row) + ``` + + 其中，“@”为数据表快照的版本分隔符，from子句后加上已存在的数据表快照，用法为“@”+版本号，USING关键字后加入可选的几个操作关键字（INSERT …/UPDATE …/DELETE …/ALTER …）,其中 “INSERT INTO”以及“DELETE FROM”语句中的“INTO”、“FROM”等与数据表快照名字相关联的子句可以省略，具体可以参考[AI特性函数](28-ai-feature-functions)。 + + 示例中，基于前述s1@1.0快照，插入2条数据，删除1条新的数据，新生成的快照s1@3.0，检索该s1@3.0： + + ``` + SELECT * FROM s1@3.0; + id | name + ----+---------- + 2 | lisi + 3 | wangwu + 4 | lisa + 5 | jack + 6 | john + 7 | tim + (7 rows) + ``` + + - 删除数据表快照SNAPSHOT + + ``` + purge snapshot s1@3.0; + schema | name + --------+-------- + public | s1@3.0 + (1 row) + ``` + + 此时，已经无法再从s1@3.0 中检索到数据了，同时该数据表快照在db4ai.snapshot视图中的记录也会被清除。删除该版本的数据表快照不会影响其他版本的数据表快照。 + + - 从数据表快照中采样 + + 示例：从snapshot s1中抽取数据，使用0.5抽样率。 + + ``` + sample snapshot s1@2.0 stratify by name as nick at ratio .5; + schema | name + --------+------------ + public | s1nick@2.0 + (1 row) + ``` + + 可以利用该功能创建训练集与测试集，例如： + + ``` + SAMPLE SNAPSHOT s1@2.0 STRATIFY BY name AS _test AT RATIO .2, AS _train AT RATIO .8 COMMENT IS 'training'; + schema | name + --------+---------------- + public | s1_test@2.0 + public | s1_train@2.0 + (2 rows) + ``` + + - 发布数据表快照 + + 采用下述SQL语句将数据表快照 s1@2.0 标记为published 状态： + + ``` + publish snapshot s1@2.0; + schema | name + --------+-------- + public | s1@2.0 + (1 row) + ``` + + - 存档数据表快照 + + 采用下述语句可以将数据表快照标记为 archived 状态： + + ``` + archive snapshot s1@2.0; + schema | name + --------+-------- + public | s1@2.0 + (1 row) + ``` + + 可以通过db4ai-snapshots提供的视图查看当前数据表快照的状态以及其他信息： + + ``` + select * from db4ai.snapshot; + id | parent_id | matrix_id | root_id | schema | name | owner | commands | comment | published | archived | created | row_count + ----+-----------+-----------+---------+--------+------------+--------+------------------------------------------+---------+-----------+----------+----------------------------+----------- + 1 | | | 1 | public | s1@2.0 | omm | {"select *","from t1 where id > 3",NULL} | | t | f | 2021-04-17 09:24:11.139868 | 2 + 2 | 1 | | 1 | public | s1nick@2.0 | omm | {"SAMPLE nick .5 {name}"} | | f | f | 2021-04-17 10:02:31.73923 | 0 + ``` + +3. 异常场景 + + - 数据表或db4ai-snapshots不存在时。 + + ``` + purge snapshot s1nick@2.0; + publish snapshot s1nick@2.0; + --------- + ERROR: snapshot public."s1nick@2.0" does not exist + CONTEXT: PL/pgSQL function db4ai.publish_snapshot(name,name) line 11 at assignment + + archive snapshot s1nick@2.0; + ---------- + ERROR: snapshot public."s1nick@2.0" does not exist + CONTEXT: PL/pgSQL function db4ai.archive_snapshot(name,name) line 11 at assignment + ``` + + - 删除snapshot时，有依赖该快照的其他snapshot，需先确保删除对本快照所依赖的其他快照。 + + ``` + purge snapshot s1@1.0; + ERROR: cannot purge root snapshot 'public."s1@1.0"' having dependent snapshots + HINT: purge all dependent snapshots first + CONTEXT: referenced column: purge_snapshot_internal + SQL statement "SELECT db4ai.purge_snapshot_internal(i_schema, i_name)" + PL/pgSQL function db4ai.purge_snapshot(name,name) line 71 at PERFORM + ``` + +4. 相关GUC参数 + + - db4ai_snapshot_mode： + + Snapshot有2种模式：MSS（物化模式，存储数据实体）和CSS（计算模式，存储增量信息）。Snapshot可在MSS和CSS之间切换快照模式，默认是MSS模式。 + + - db4ai_snapshot_version_delimiter： + + 该参数为数据表快照版本分隔符。“@”为数据表快照的默认版本分隔符。 + + - db4ai_snapshot_version_separator + + 该参数为数据表快照子版本分隔符。“.”为数据表快照的默认版本分隔符。 + +5. DB4AI Schema下的数据表快照详情db4ai.snapshot。 + + ``` + MogDB=# \d db4ai.snapshot + Table "db4ai.snapshot" + Column | Type | Modifiers + -----------+-----------------------------+--------------------------- + id | bigint | + parent_id | bigint | + matrix_id | bigint | + root_id | bigint | + schema | name | not null + name | name | not null + owner | name | not null + commands | text[] | not null + comment | text | + published | boolean | not null default false + archived | boolean | not null default false + created | timestamp without time zone | default pg_systimestamp() + row_count | bigint | not null + Indexes: + "snapshot_pkey" PRIMARY KEY, btree (schema, name) TABLESPACE pg_default + "snapshot_id_key" UNIQUE CONSTRAINT, btree (id) TABLESPACE pg_default + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** 命名空间DB4AI是本功能的私有域，不支持在DB4AI的命令空间下创建函数索引（functional index)。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/full-process-ai.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/full-process-ai.md new file mode 100644 index 0000000000000000000000000000000000000000..a7304561d5e15174e9e175bd9067578edbce59e1 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/full-process-ai.md @@ -0,0 +1,12 @@ +--- +title: 全流程AI +summary: 全流程AI +author: Guo Huan +date: 2022-05-06 +--- + +# 全流程AI + +传统的AI任务往往具有多个流程，如数据的收集过程包括数据的采集、数据清洗、数据存储等，在算法的训练过程中又包括数据的预处理、训练、模型的保存与管理等。其中，对于模型的训练过程，又包括超参数的调优过程。诸如此类机器学习模型生命周期的全过程，可大量集成于数据库内部。在距离数据存储侧最近处进行模型的训练、管理、优化等流程，在数据库端提供SQL语句式的开箱即用的AI全声明周期管理的功能，称之为全流程AI. + +MogDB实现了部分全流程AI的功能，将在本章节中详细展开。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-4-pl-python-fenced-mode.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md similarity index 58% rename from product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-4-pl-python-fenced-mode.md rename to product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md index 58cfa3d18747c0b3c3ebf5a82d0dfbb093bbc9c1..4c753dda2c6f86b29fee1f12ab29282e80ce4918 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/8-db4ai/8-4-pl-python-fenced-mode.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md @@ -1,135 +1,135 @@ ---- -title: plpython-fenced模式 -summary: plpython-fenced模式 -author: Guo Huan -date: 2021-10-20 ---- - -# plpython-fenced模式 - -在fenced模式中添加plpython非安全语言。在数据库编译时需要将python集成进数据库中，在configure截断加入-with-python选项。同时也可指定安装plpython的python路径，添加选项-with-includes='/python-dir=path'。 - -在启动数据库之前配置GUC参数unix_socket_directory，指定unix_socket进程间通讯的文件地址。用户需要提前在user-set-dir-path路径下创建文件夹，并将该文件夹权限修改为可读可写可执行。 - -```bash -unix_socket_directory = '/user-set-dir-path' -``` - -配置完成，启动数据库。 - -将plpython加入数据库编译，并设置好GUC参数unix_socket_directory后，在启动数据库的过程中，自动创建fenced-Master进程。在数据库不进行python编译的情况下，fenced模式需要手动拉起master进程，在GUC参数设置完成后，输入创建master进程命令。 - -启动fenced-Master进程，命令为： - -```bash -gaussdb --fenced -k /user-set-dir-path -D /user-set-dir-path & -``` - -完成fence模式配置，针对plpython-fenced UDF数据库将在fenced-worker进程中执行UDF计算。 - -## 使用指导 - -- 创建extension - - - 当编译的plpython为Python2时： - - ```sql - mogdb=# create extension plpythonu; - CREATE EXTENSION - ``` - - - 当编译的plpython为Python3时： - - ```sql - mogdb=# create extension plpython3u; - CREATE EXTENSION - ``` - - 下面示例是以Python2为例。 - -- 创建plpython-fenced UDF - - ```sql - mogdb=# create or replace function pymax(a int, b int) - mogdb-# returns INT - mogdb-# language plpythonu fenced - mogdb-# as $$ - mogdb$# import numpy - mogdb$# if a > b: - mogdb$# return a; - mogdb$# else: - mogdb$# return b; - mogdb$# $$; - CREATE FUNCTION - ``` - -- 查看UDF信息 - - ```sql - mogdb=# select * from pg_proc where proname='pymax'; - -[ RECORD 1 ]----+-------------- - proname | pymax - pronamespace | 2200 - proowner | 10 - prolang | 16388 - procost | 100 - prorows | 0 - provariadic | 0 - protransform | - - proisagg | f - proiswindow | f - prosecdef | f - proleakproof | f - proisstrict | f - proretset | f - provolatile | v - pronargs | 2 - pronargdefaults | 0 - prorettype | 23 - proargtypes | 23 23 - proallargtypes | - proargmodes | - proargnames | {a,b} - proargdefaults | - prosrc | - | import numpy - | if a > b: - | return a; - | else: - | return b; - | - probin | - proconfig | - proacl | - prodefaultargpos | - fencedmode | t - proshippable | f - propackage | f - prokind | f - proargsrc | - ``` - -- 运行UDF - - - 创建一个数据表： - - ```sql - mogdb=# create table temp (a int ,b int) ; - CREATE TABLE - mogdb=# insert into temp values (1,2),(2,3),(3,4),(4,5),(5,6); - INSERT 0 5 - ``` - - - 运行UDF： - - ```sql - mogdb=# select pymax(a,b) from temp; - pymax - ------- - 2 - 3 - 4 - 5 - 6 - (5 rows) - ``` +--- +title: PLPython Fenced模式 +summary: PLPython Fenced模式 +author: Guo Huan +date: 2022-05-06 +--- + +# PLPython Fenced模式 + +在fenced模式中添加plpython非安全语言。在数据库编译时需要将python集成进数据库中，在configure阶段加入--with-python选项。同时也可指定安装plpython的python路径，添加选项--with-includes='/python-dir=path'。 + +在启动数据库之前配置GUC参数unix_socket_directory ，指定unix_socket进程间通讯的文件地址。用户需要提前在user-set-dir-path下创建文件夹，并将文件夹权限修改为可读可写可执行状态。 + +``` +unix_socket_directory = '/user-set-dir-path' +``` + +配置完成，启动数据库。 + +将plpython加入数据库编译，并设置好GUC参数unix_socket_directory后，在启动数据库的过程中，自动创建fenced-Master进程。在数据库不进行python编译的情况下，fenced模式需要手动拉起master进程，在GUC参数设置完成后，输入创建master进程命令。 + +启动fenced-Master进程，命令为： + +``` +gaussdb --fenced -k /user-set-dir-path -D /user-set-dir-path & +``` + +完成fence模式配置，针对plpython-fenced UDF数据库将在fenced-worker进程中执行UDF计算。 + +## 使用指导 + +- 创建extension + + - 当编译的plpython为python2时： + + ``` + MogDB=# create extension plpythonu; + CREATE EXTENSION + ``` + + - 当编译的plpython为python3时： + + ``` + MogDB=# create extension plpython3u; + CREATE EXTENSION + ``` + + 下面示例是以python2为例。 + +- 创建plpython-fenced UDF + + ``` + MogDB=# create or replace function pymax(a int, b int) + MogDB-# returns INT + MogDB-# language plpythonu fenced + MogDB-# as $$ + MogDB$# import numpy + MogDB$# if a > b: + MogDB$# return a; + MogDB$# else: + MogDB$# return b; + MogDB$# $$; + CREATE FUNCTION + ``` + +- 查看UDF信息 + + ``` + MogDB=# select * from pg_proc where proname='pymax'; + -[ RECORD 1 ]----+-------------- + proname | pymax + pronamespace | 2200 + proowner | 10 + prolang | 16388 + procost | 100 + prorows | 0 + provariadic | 0 + protransform | - + proisagg | f + proiswindow | f + prosecdef | f + proleakproof | f + proisstrict | f + proretset | f + provolatile | v + pronargs | 2 + pronargdefaults | 0 + prorettype | 23 + proargtypes | 23 23 + proallargtypes | + proargmodes | + proargnames | {a,b} + proargdefaults | + prosrc | + | import numpy + | if a > b: + | return a; + | else: + | return b; + | + probin | + proconfig | + proacl | + prodefaultargpos | + fencedmode | t + proshippable | f + propackage | f + prokind | f + proargsrc | + ``` + +- 运行UDF + + - 创建一个数据表： + + ``` + MogDB=# create table temp (a int ,b int) ; + CREATE TABLE + MogDB=# insert into temp values (1,2),(2,3),(3,4),(4,5),(5,6); + INSERT 0 5 + ``` + + - 运行UDF： + + ``` + MogDB=# select pymax(a,b) from temp; + pymax + ------- + 2 + 3 + 4 + 5 + 6 + (5 rows) + ``` diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/native-db4ai-engine.md b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/native-db4ai-engine.md new file mode 100644 index 0000000000000000000000000000000000000000..fa445a4e615fa731d19e0aebb4b7d05943591356 --- /dev/null +++ b/product/zh/docs-mogdb/v3.0/developer-guide/AI-features/db4ai/native-db4ai-engine.md @@ -0,0 +1,317 @@ +--- +title: 原生DB4AI引擎 +summary: 原生DB4AI引擎 +author: Guo Huan +date: 2022-05-06 +--- + +# DB4AI: 数据库驱动AI + +MogDB当前版本支持了原生DB4AI能力，通过引入原生AI算子，简化操作流程，充分利用数据库优化器、执行器的优化与执行能力，获得高性能的数据库内模型训练能力。更简化的模型训练与预测流程、更高的性能表现，让开发者在更短时间内能更专注于模型的调优与数据分析上，而避免了碎片化的技术栈与冗余的代码实现。 + +## 关键字解析 + +**表 1** DB4AI语法及关键字 + +| 名称 | | 描述 | +| :----- | :----------- | ---------------------------------- | +| 语法 | CREATE MODEL | 创建模型并进行训练，同时保存模型。 | +| | PREDICT BY | 利用已有模型进行推断。 | +| | DROP MODEL | 删除模型。 | +| 关键字 | TARGET | 训练/推断任务的目标列名。 | +| | FEATURES | 训练/推断任务的数据特征列名。 | +| | MODEL | 训练任务的模型名称。 | + +## 使用指导 + +1. 本版本支持的算法概述。 + + 当前版本的DB4AI新增支持算法如下： + + **表 2** 支持算法 + + | 优化算法 | 算法 | + | :------- | :------------------------------ | + | GD | logistic_regression | + | | linear_regression | + | | svm_classification | + | | PCA | + | | multiclass | + | Kmeans | kmeans | + | xgboost | xgboost_regression_logistic | + | | xgboost_binary_logistic | + | | xgboost_regression_squarederror | + | | xgboost_regression_gamma | + +2. 模型训练语法说明。 + + - CREATE MODEL + + 使用“CREATE MODEL”语句可以进行模型的创建和训练。模型训练SQL语句，选用公开数据集鸢尾花数据集iris。 + + - 以multiclass为例，训练一个模型。从tb_iris训练集中指定sepal_length, sepal_width,petal_length,petal_widt为特征列，使用multiclass算法，创建并保存模型iris_classification_model。 + + ``` + MogDB=# CREATE MODEL iris_classification_model USING xgboost_regression_logistic FEATURES sepal_length, sepal_width,petal_length,petal_width TARGET target_type < 2 FROM tb_iris_1 WITH nthread=4, max_depth=8; + MODEL CREATED. PROCESSED 1 + ``` + + 上述命令中： + + - “CREATE MODEL”语句用于模型的训练和保存。 + + - USING关键字指定算法名称。 + + - FEATURES用于指定训练模模型的特征，需根据训练数据表的列名添加。 + + - TARGET指定模型的训练目标，它可以是训练所需数据表的列名，也可以是一个表达式，例如: price > 10000。 + + - WITH用于指定训练模型时的超参数。当超参未被用户进行设置的时候，框架会使用默认数值。 + + 针对不同的算子，框架支持不同的超参组合： + + **表 3** 算子支持的超参 + + | 算子 | 超参 | + | :----------------------------------------------------------- | :----------------------------------------------------------- | + | GD
(logistic_regression、linear_regression、svm_classification) | optimizer(char); verbose(bool); max_iterations(int); max_seconds(double); batch_size(int); learning_rate(double); decay(double); tolerance(double)
其中，SVM限定超参lambda(double) | + | Kmeans | max_iterations(int); num_centroids(int); tolerance(double); batch_size(int); num_features(int); distance_function(char); seeding_function(char); verbose(int);seed(int) | + | GD(pca) | batch_size(int);max_iterations(int);max_seconds(int);tolerance(float8);verbose(bool);number_components(int);seed(int) | + | GD(multiclass) | classifier(char)
注意：multiclass的其他超参种类取决于选择的分类器中类 | + | xgboost_regression_logistic、xgboost_binary_logistic、xgboost_regression_squarederror、xgboost_regression_gamma | batch_size(int);booster(char);tree_method(char);eval_metric(char);seed(int);nthread(int);max_depth(int);gamma(float8);eta(float8);min_child_weight(int);verbosity(int) | + + 当前各个超参数设置的默认值和取值范围如下： + + **表 4** 超参的默认值以及取值范围 + + | 算子 | 超参(默认值) | 取值范围 | 超参描述 | + | :----------------------------------------------------------- | :------------------------------------------------------ | :----------------------------------------------------------- | :---------------------------------------- | + | GD:logistic_regression、linear_regression、svm_classification、pca | optimizer = gd（梯度下降法） | gd/ngd（自然梯度下降） | 优化器 | + | | verbose = false | T/F | 日志显示 | + | | max_iterations = 100 | (0, 10000] | 最大迭代次数 | + | | max_seconds = 0 (不对运行时长设限制) | [0,INT_MAX_VALUE] | 运行时长 | + | | batch_size = 1000 | (0, 1048575] | 一次训练所选取的样本数 | + | | learning_rate = 0.8 | (0, DOUBLE_MAX_VALUE] | 学习率 | + | | decay = 0.95 | (0, DOUBLE_MAX_VALUE] | 权值衰减率 | + | | tolerance = 0.0005 | (0, DOUBLE_MAX_VALUE] | 公差 | + | | seed = 0（对seed取随机值） | [0, INT_MAX_VALUE] | 种子 | + | | just for linear、SVM：kernel = “linear” | linear/gaussian/polynomial | 核函数 | + | | just for linear、SVM：components = MAX(2*features, 128) | [0, INT_MAX_VALUE] | 高维空间维数 | + | | just for linear、SVM：gamma = 0.5 | (0, DOUBLE_MAX_VALUE] | gaussian核函数参数 | + | | just for linear、SVM：degree = 2 | [2, 9] | polynomial核函数参数 | + | | just for linear、SVM：coef0 = 1.0 | [0, DOUBLE_MAX_VALUE] | polynomial核函数的参数 | + | | just for SVM：lambda = 0.01 | (0, DOUBLE_MAX_VALUE) | 正则化参数 | + | | just for pca： number_components | （0，INT_MAX_VALUE] | 降维的目标维度 | + | GD:multiclass | classifier=“svm_classification” | svm_classification\logistic_regression | 多分类任务的分类器 | + | Kmeans | max_iterations = 10 | [1, 10000] | 最大迭代次数 | + | | num_centroids = 10 | [1, 1000000] | 簇的数目 | + | | tolerance = 0.00001 | (0,1] | 中心点误差 | + | | batch_size = 10 | [1,1048575] | 一次训练所选取的样本数 | + | | num_features = 2 | [1, INT_MAX_VALUE] | 输入样本特征数 | + | | distance_function = “L2_Squared” | L1\L2\L2_Squared\Linf | 正则化方法 | + | | seeding_function = “Random++” | “Random++”\“KMeans | | + | | verbose = 0U | { 0, 1, 2 } | 冗长模式 | + | | seed = 0U | [0, INT_MAX_VALUE] | 种子 | + | xgboost:
xgboost_regression_logistic、xgboost_binary_logistic、xgboost_regression_gamma、xgboost_regression_squarederror | n_iter=10 | (0, 10000] | 迭代次数 | + | | batch_size=10000 | (0, 1048575] | 一次训练所选取的样本数 | + | | booster=“gbtree” | gbtree\gblinear\dart | booster种类 | + | | tree_method=“auto” | auto\exact\approx\hist\gpu_hist
注意：gpu_hist参数需要相应的库GPU版本，否则DB4AI平台不支持该值。 | 树构建算法 | + | | eval_metric=“rmse” | rmse\rmsle\map\mae\auc\aucpr | 验证数据的评估指标 | + | | seed=0 | [0, 100] | 种子 | + | | nthread=1 | (0, MAX_MEMORY_LIMIT] | 并发量 | + | | max_depth=5 | (0, MAX_MEMORY_LIMIT] | 树的最大深度，该超参仅对树型booster生效。 | + | | gamma=0.0 | [0, 1] | 叶节点上进行进一步分区所需的最小损失减少 | + | | eta=0.3 | [0, 1] | 更新中使用的步长收缩，以防止过拟合 | + | | min_child_weight=1 | [0, INT_MAX_VALUE] | 孩子节点中所需的实例权重的最小总和 | + | | verbosity=1 | 0 (silent)\1 (warning)\2 (info)\3 (debug) | 打印信息的详细程度 | + | MAX_MEMORY_LIMIT = 最大内存加载的元组数量 | | | | + | GS_MAX_COLS = 数据库单表最大属性数量 | | | | + + - 模型保存成功，则返回创建成功信息： + + ``` + MODEL CREATED. PROCESSED x + ``` + +3. 查看模型信息。 + + 当训练完成后模型会被存储到系统表gs_model_warehouse中。系统表gs_model_warehouse可以查看到关于模型本身和训练过程的相关信息。 + + 关于模型的详细描述信息以二进制的形式存储在系统表中，用户可用过使用函数gs_explain_model完成对模型的查看，语句如下： + + ``` + MogDB=# select * from gs_explain_model("iris_classification_model"); + DB4AI MODEL + ------------------------------------------------------------- + Name: iris_classification_model + Algorithm: xgboost_regression_logistic + Query: CREATE MODEL iris_classification_model + USING xgboost_regression_logistic + FEATURES sepal_length, sepal_width,petal_length,petal_width + TARGET target_type < 2 + FROM tb_iris_1 + WITH nthread=4, max_depth=8; + Return type: Float64 + Pre-processing time: 0.000000 + Execution time: 0.001443 + Processed tuples: 78 + Discarded tuples: 0 + n_iter: 10 + batch_size: 10000 + max_depth: 8 + min_child_weight: 1 + gamma: 0.0000000000 + eta: 0.3000000000 + nthread: 4 + verbosity: 1 + seed: 0 + booster: gbtree + tree_method: auto + eval_metric: rmse + rmse: 0.2648450136 + model size: 4613 + ``` + +4. 利用已存在的模型做推断任务。 + + 使用“SELECT”和“PREDICT BY”关键字利用已有模型完成推断任务。 + + 查询语法：SELECT…PREDICT BY…(FEATURES…)…FROM…; + + ``` + MogDB=# SELECT id, PREDICT BY iris_classification (FEATURES sepal_length,sepal_width,petal_length,petal_width) as "PREDICT" FROM tb_iris limit 3; + + id | PREDICT + -----+--------- + 84 | 2 + 85 | 0 + 86 | 0 + (3 rows) + ``` + + 针对相同的推断任务，同一个模型的结果是大致稳定的。且基于相同的超参数和训练集训练的模型也具有稳定性，同时AI模型训练存在随机成分（每个batch的数据分布、随机梯度下降），所以不同的模型间的计算表现、结果允许存在小的差别。 + +5. 查看执行计划。 + + 使用explain语句可对“CREATE MODEL”和“PREDICT BY”的模型训练或预测过程中的执行计划进行分析。Explain关键字后可直接拼接CREATE MODEL/ PREDICT BY语句（子句），也可接可选的参数，支持的参数如下： + + **表 5** EXPLAIN支持的参数 + + | 参数名 | 描述 | + | :-------- | :--------------------------------------------- | + | ANALYZE | 布尔型变量，追加运行时间、循环次数等描述信息 | + | VERBOSE | 布尔型变量，控制训练的运行信息是否输出到客户端 | + | COSTS | 布尔型变量 | + | CPU | 布尔型变量 | + | DETAIL | 布尔型变量，不可用。 | + | NODES | 布尔型变量，不可用 | + | NUM_NODES | 布尔型变量，不可用 | + | BUFFERS | 布尔型变量 | + | TIMING | 布尔型变量 | + | PLAN | 布尔型变量 | + | FORMAT | 可选格式类型：TEXT / XML / JSON / YAML | + + 示例： + + ``` + MogDB=# Explain CREATE MODEL patient_logisitic_regression USING logistic_regression FEATURES second_attack, treatment TARGET trait_anxiety > 50 FROM patients WITH batch_size=10, learning_rate = 0.05; + QUERY PLAN + ------------------------------------------------------------------------- + Train Model - logistic_regression (cost=0.00..0.00 rows=0 width=0) + -> Materialize (cost=0.00..41.08 rows=1776 width=12) + -> Seq Scan on patients (cost=0.00..32.20 rows=1776 width=12) + (3 rows) + ``` + +6. 异常场景。 + + - 训练阶段。 + + - 场景一：当超参数的设置超出取值范围，模型训练失败，返回ERROR，并提示错误，例如： + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET trait_anxiety FROM patients WITH optimizer='aa'; + ERROR: Invalid hyperparameter value for optimizer. Valid values are: gd, ngd. + ``` + + - 场景二：当模型名称已存在，模型保存失败，返回ERROR，并提示错误原因，例如： + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET trait_anxiety FROM patients; + ERROR: The model name "patient_linear_regression" already exists in gs_model_warehouse. + ``` + + - 场景三：FEATURE或者TARGETS列是*，返回ERROR，并提示错误原因，例如： + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES * TARGET trait_anxiety FROM patients; + ERROR: FEATURES clause cannot be * + ----------------------------------------------------------------------------------------------------------------------- + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET * FROM patients; + ERROR: TARGET clause cannot be * + ``` + + - 场景四：对于无监督学习方法使用TARGET关键字，或者在监督学习方法中不适用TARGET关键字，均会返回ERROR，并提示错误原因，例如： + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment FROM patients; + ERROR: Supervised ML algorithms require TARGET clause + ----------------------------------------------------------------------------------------------------------------------------- + CREATE MODEL patient_linear_regression USING linear_regression TARGET trait_anxiety FROM patients; + ERROR: Supervised ML algorithms require FEATURES clause + ``` + + - 场景五：当进行分类任务时TARGET列的分类只有1种情况，会返回ERROR，并提示错误原因，例如： + + ``` + MogDB=# CREATE MODEL ecoli_svmc USING multiclass FEATURES f1, f2, f3, f4, f5, f6, f7 TARGET cat FROM (SELECT * FROM db4ai_ecoli WHERE cat='cp'); + ERROR: At least two categories are needed + ``` + + - 场景六：DB4AI在训练过程中会过滤掉含有空值的数据，当参与训练的模型数据为空的时候，会返回ERROR，并提示错误原因，例如： + + ``` + MogDB=# create model iris_classification_model using xgboost_regression_logistic features message_regular target error_level from error_code; + ERROR: Training data is empty, please check the input data. + ``` + + - 场景七：DB4AI的算法对于支持的数据类型是有限制的。当数据类型不在支持白名单中，会返回ERROR，并提示非法的oid，可通过pg_type查看OID确定非法的数据类型，例如： + + ``` + MogDB=# CREATE MODEL ecoli_svmc USING multiclass FEATURES f1, f2, f3, f4, f5, f6, f7, cat TARGET cat FROM db4ai_ecoli ; + ERROR: Oid type 1043 not yet supported + ``` + + - 场景八：当GUC参数statement_timeout设置了时长，训练超时执行的语句将被终止：执行CREATE MODEL语句。训练集的大小、训练轮数(iteration)、提前终止条件(tolerance、max_seconds)、并行线程数(nthread)等参数都会影响训练时长。当时长超过数据库限制，语句被终止模型训练失败。 + + - 模型解析。 + + - 场景九：当模型名在系统表中查找不到，数据库会报ERROR，例如： + + ``` + MogDB=# select gs_explain_model("ecoli_svmc"); + ERROR: column "ecoli_svmc" does not exist + ``` + + - 推断阶段。 + + - 场景十：当模型名在系统表中查找不到，数据库会报ERROR，例如： + + ``` + MogDB=# select id, PREDICT BY patient_logistic_regression (FEATURES second_attack,treatment) FROM patients; + ERROR: There is no model called "patient_logistic_regression". + ``` + + - 场景十一：当做推断任务FEATURES的数据维度和数据类型与训练集存在不一致，将报ERROR，并提示错误原因，例如： + + ``` + MogDB=# select id, PREDICT BY patient_linear_regression (FEATURES second_attack) FROM patients; + ERROR: Invalid number of features for prediction, provided 1, expected 2 + CONTEXT: referenced column: patient_linear_regression_pred + ------------------------------------------------------------------------------------------------------------------------------------- + MogDB=# select id, PREDICT BY patient_linear_regression (FEATURES 1,second_attack,treatment) FROM patients; + ERROR: Invalid number of features for prediction, provided 3, expected 2 + CONTEXT: referenced column: patient_linear_regression_pre + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明：** DB4AI特性需要读取数据参与计算，不适用于密态数据库等情况。 diff --git a/product/zh/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md b/product/zh/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md index 1e9d32c16fba801e6029d7db903eeb5ca1937e79..a9d5ca5abc3cebace61437e5d962b5461e1795ed 100644 --- a/product/zh/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md +++ b/product/zh/docs-mogdb/v3.0/developer-guide/scheduled-jobs/pkg-service.md @@ -54,7 +54,7 @@ MogDB 2.1版本提供了以下PKG_SERVICE接口来实现定时任务管理。 ``` PKG_SERVICE.JOB_FINISH( - id IN INTEGER, + id IN BIGINT, broken IN BOOLEAN, next_time IN TIMESTAMP DEFAULT sysdate); ``` @@ -63,7 +63,7 @@ MogDB 2.1版本提供了以下PKG_SERVICE接口来实现定时任务管理。 | 参数 | 类型 | 入参/出参 | 是否可以为空 | 描述 | | --------- | --------- | --------- | ------------ | ------------------------------------------------------------ | - | id | integer | IN | 否 | 指定的作业号。 | + | id | bigint | IN | 否 | 指定的作业号。 | | broken | Boolean | IN | 否 | 状态标志位，true代表禁用，false代表启用。根据true或false值更新当前job；如果为空值，则不改变原有job的状态。 | | next_time | timestamp | IN | 是 | 下次运行时间，默认为当前系统时间。如果参数broken状态为true，则更新该参数为'4000-1-1'；如果参数broken状态为false，且如果参数next_time不为空值，则更新指定job的next_time值，如果next_time为空值，则不更新next_time值。该参数可以省略，为默认值。 | diff --git a/product/zh/docs-mogdb/v3.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md b/product/zh/docs-mogdb/v3.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md index d379dea5f7e489f51da2d467586f8ffcc369f16d..ef586833d19324208dc12aefc0181564fc4b6313 100644 --- a/product/zh/docs-mogdb/v3.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md +++ b/product/zh/docs-mogdb/v3.0/reference-guide/system-catalogs-and-system-views/system-catalogs/PG_JOB.md @@ -21,7 +21,7 @@ PG_JOB系统表存储用户创建的定时任务的任务详细信息，定时 | dbname | name | 标识作业要在哪个数据库执行的数据库名称 | | node_name | name | 标识当前作业是在哪个数据库主节点上创建和执行 | | job_status | "char" | 当前任务的执行状态，取值范围: ('r', 's', 'f', 'd')，默认为's'，取值含义:
Status of job step: r=running, s=successfully finished, f=job failed, d=disable
当job连续执行失败16次，会将job_status自动设置为失效状态'd'，后续不再执行该job。
注: 当用户将定时任务关闭（即: guc参数job_queue_processes为0时），由于监控job执行的线程不会启动，所以该状态不会根据job的实时状态进行设置，用户不需要关注此状态。只有当开启定时任务功能（即: guc参数job_queue_processes为非0时），系统才会根据当前job的实时状态刷新该字段值。 | -| start_date | timestamp without time zone | 作业第一次开始执行时间，时间精确到毫秒。 | +| start_date | timestamp without time zone | 作业第一次开始运行时间，时间精确到毫秒。 | | next_run_date | timestamp without time zone | 下次定时执行任务的时间，时间精确到毫秒。 | | failure_count | smallint | 失败计数，作业连续执行失败16次，不再继续执行。 | | interval | text | 作业执行的重复时间间隔。 | diff --git a/product/zh/docs-mogdb/v3.0/toc.md b/product/zh/docs-mogdb/v3.0/toc.md index 49268d2cf287e3f723a457a6c2da7f4dee8b3d08..e2b6f7dda69625deae0e13a8ca7d90a98ae2642f 100644 --- a/product/zh/docs-mogdb/v3.0/toc.md +++ b/product/zh/docs-mogdb/v3.0/toc.md @@ -8,12 +8,6 @@ + [MogDB简介](/overview.md) + [MogDB与openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB发行说明](/about-mogdb/mogdb-release-notes.md) - + 高可用和高性能 - + [高可用测试](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + 性能测试 - + [性能测试简述](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [MogDB on 鲲鹏性能测试](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [MogDB on x86性能测试](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + 开源组件 + [容器版本的MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) @@ -387,51 +381,64 @@ + [概述](/developer-guide/materialized-view/3-incremental-materialized-view/1-incremental-materialized-view-overview.md) + [使用](/developer-guide/materialized-view/3-incremental-materialized-view/2-incremental-materialized-view-usage.md) + [支持和约束](/developer-guide/materialized-view/3-incremental-materialized-view/3-incremental-materialized-view-support-and-constraints.md) - + AI特性 - + [概述](/developer-guide/AI-features/1-AI-features-overview.md) - + Predictor：AI查询时间预测 - + [概述](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-1-ai-query-time-forecasting-overview.md) - + [环境部署](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-2-environment-deployment.md) - + [使用指导](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-3-usage-guide.md) - + [最佳实践](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-4-best-practices.md) - + [常见问题处理](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-5-faqs.md) - + X-Tuner：参数调优与诊断 - + [概述](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-1-x-tuner-overview.md) - + [使用准备](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-2-preparations.md) - + [使用示例](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-3-examples.md) - + [获取帮助](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-4-obtaining-help-information.md) - + [命令参考](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-5-command-reference.md) - + [常见问题处理](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-6-Troubleshooting.md) - + SQLdiag：慢SQL发现 - + [概述](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-1-overview.md) - + [使用指导](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-2-usage-guide.md) - + [获取帮助](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-3-obtaining-help-information.md) - + [命令参考](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-4-command-reference.md) - + [常见问题处理](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-5-troubleshooting.md) - + Anomaly-detection：数据库指标采集、预测与异常监控 - + [概述](/developer-guide/AI-features/5-a-detection-status-monitoring/5-1-overview.md) - + [使用准备](/developer-guide/AI-features/5-a-detection-status-monitoring/5-2-preparations.md) - + [添加监控参数](/developer-guide/AI-features/5-a-detection-status-monitoring/5-3-adding-monitoring-parameters.md) - + [获取帮助](/developer-guide/AI-features/5-a-detection-status-monitoring/5-4-obtaining-help-information.md) - + [使用示例](/developer-guide/AI-features/5-a-detection-status-monitoring/5-6-examples.md) - + [命令参考](/developer-guide/AI-features/5-a-detection-status-monitoring/5-5-command-reference.md) - + [AI_SERVER](/developer-guide/AI-features/5-a-detection-status-monitoring/5-7-ai-server.md) - + [AI_MANAGER](/developer-guide/AI-features/5-a-detection-status-monitoring/5-8-ai-manager.md) - + Index-advisor：索引推荐 - + [单query索引推荐](/developer-guide/AI-features/6-index-advisor-index-recommendation/6-1-single-query-index-recommendation.md) - + [虚拟索引](/developer-guide/AI-features/6-index-advisor-index-recommendation/6-2-virtual-index.md) - + [workload级别索引推荐](/developer-guide/AI-features/6-index-advisor-index-recommendation/6-3-workload-level-index-recommendation.md) - + DeepSQL：库内AI算法 - + [概述](/developer-guide/AI-features/7-deepsql/7-1-overview.md) - + [环境部署](/developer-guide/AI-features/7-deepsql/7-2-environment-deployment.md) - + [使用指导](/developer-guide/AI-features/7-deepsql/7-3-usage-guide.md) - + [最佳实践](/developer-guide/AI-features/7-deepsql/7-4-best-practices.md) - + [常见问题处理](/developer-guide/AI-features/7-deepsql/7-5-troubleshooting.md) - + DB4AI：数据库原生AI引擎 - + [概述](/developer-guide/AI-features/8-db4ai/8-1-overview.md) - + [DB4AI-Snapshots数据版本管理](/developer-guide/AI-features/8-db4ai/8-2-db4ai-snapshots-for-data-version-management.md) - + [DB4AI-Query：模型训练和推断](/developer-guide/AI-features/8-db4ai/8-3-db4ai-query-for-model-training-and-prediction.md) - + [plpython-fenced模式](/developer-guide/AI-features/8-db4ai/8-4-pl-python-fenced-mode.md) + + [AI特性](/developer-guide/AI-features/1-AI-features-overview.md) + + [AI4DB：据库自治运维](/developer-guide/AI-features/ai4db/ai4db-autonomous-database-o&m.md) + + [DBMind模式说明](/developer-guide/AI-features/ai4db/dbmind-mode/dbmind-mode.md) + + [service子命令](/developer-guide/AI-features/ai4db/dbmind-mode/1-service.md) + + [component子命令](/developer-guide/AI-features/ai4db/dbmind-mode/2-component.md) + + [set子命令](/developer-guide/AI-features/ai4db/dbmind-mode/3-set.md) + + [DBMind的支持组件](/developer-guide/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md) + + Prometheus Exporter组件 + + [概述](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md) + + [环境部署](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md) + + DBMind的AI子功能 + + X-Tuner：参数调优与诊断 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md) + + [使用准备](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md) + + [使用示例](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md) + + Index-advisor：索引推荐 + + [单query索引推荐](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md) + + [虚拟索引](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md) + + [workload级别索引推荐](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md) + + AI4DB: 慢SQL根因分析 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-1-overview.md) + + [环境部署](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md) + + AI4DB：趋势预测 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-1-overview.md) + + [环境部署](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-2-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-3-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-4-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-5-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-6-troubleshooting.md) + + SQLdiag：慢SQL发现 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md) + + [使用指导](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md) + + [DB4AI：数据库驱动AI](/developer-guide/AI-features/db4ai/db4ai.md) + + [原生DB4AI引擎](/developer-guide/AI-features/db4ai/native-db4ai-engine.md) + + [全流程AI](/developer-guide/AI-features/db4ai/full-process-ai/full-process-ai.md) + + [PLPython Fenced模式](/developer-guide/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md) + + [DB4AI-Snapshots数据版本管理](/developer-guide/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md) + + AI in DB：数据库内AI功能 + + 智能Explain：SQL语句查询时间预测 + + [概述](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md) + + [环境部署](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md) + + [最佳实践](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md) + + [常见问题处理](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md) + 资源负载管理 + [资源负载管理概述](/developer-guide/resource-load-management/resource-load-management-overview.md) + 资源管理准备 diff --git a/product/zh/docs-mogdb/v3.0/toc_about.md b/product/zh/docs-mogdb/v3.0/toc_about.md index 1b9f203c8ed0a6afcd12f2a8108a424f85aa47ba..c5be8b2f49e78141f65beea67b64b91beb9338f0 100644 --- a/product/zh/docs-mogdb/v3.0/toc_about.md +++ b/product/zh/docs-mogdb/v3.0/toc_about.md @@ -7,12 +7,6 @@ + [MogDB简介](/overview.md) + [MogDB与openGauss](/about-mogdb/MogDB-compared-to-openGauss.md) + [MogDB发行说明](/about-mogdb/mogdb-release-notes.md) -+ 高可用和高性能 - + [高可用测试](/about-mogdb/test-report/ha/MogDB-ha-test-report.md) - + 性能测试 - + [性能测试简述](/about-mogdb/test-report/performance/1-performance-test-overview.md) - + [MogDB on 鲲鹏性能测试](/about-mogdb/test-report/performance/2-mogdb-on-kunpeng-performance-test-report.md) - + [MogDB on x86性能测试](/about-mogdb/test-report/performance/3-mogdb-on-x86-performance-test-report.md) + 开源组件 + [容器版本的MogDB](/about-mogdb/open-source-components/2-docker-based-mogdb.md) + [compat-tools](/about-mogdb/open-source-components/compat-tools.md) diff --git a/product/zh/docs-mogdb/v3.0/toc_dev.md b/product/zh/docs-mogdb/v3.0/toc_dev.md index c3bc2671ea9c457f4a706e237de2a83985e2191e..82f80d7f1a002ef173b4fed871abb9a9030fe4bd 100644 --- a/product/zh/docs-mogdb/v3.0/toc_dev.md +++ b/product/zh/docs-mogdb/v3.0/toc_dev.md @@ -195,51 +195,64 @@ + [概述](/developer-guide/materialized-view/3-incremental-materialized-view/1-incremental-materialized-view-overview.md) + [使用](/developer-guide/materialized-view/3-incremental-materialized-view/2-incremental-materialized-view-usage.md) + [支持和约束](/developer-guide/materialized-view/3-incremental-materialized-view/3-incremental-materialized-view-support-and-constraints.md) -+ AI特性 - + [概述](/developer-guide/AI-features/1-AI-features-overview.md) - + Predictor AI查询时间预测 - + [概述](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-1-ai-query-time-forecasting-overview.md) - + [环境部署](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-2-environment-deployment.md) - + [使用指导](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-3-usage-guide.md) - + [最佳实践](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-4-best-practices.md) - + [常见问题处理](/developer-guide/AI-features/2-predictor-ai-query-time-forecasting/2-5-faqs.md) - + X-Tuner 参数调优与诊断 - + [概述](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-1-x-tuner-overview.md) - + [使用准备](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-2-preparations.md) - + [使用示例](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-3-examples.md) - + [获取帮助](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-4-obtaining-help-information.md) - + [命令参考](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-5-command-reference.md) - + [常见问题处理](/developer-guide/AI-features/3-x-tuner-parameter-optimization-and-diagnosis/3-6-Troubleshooting.md) - + SQLdiag 慢SQL发现 - + [概述](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-1-overview.md) - + [使用指导](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-2-usage-guide.md) - + [获取帮助](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-3-obtaining-help-information.md) - + [命令参考](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-4-command-reference.md) - + [常见问题处理](/developer-guide/AI-features/4-sqldiag-slow-sql-discovery/4-5-troubleshooting.md) - + Anomaly-detection 数据库指标采集、预测与异常监控 - + [概述](/developer-guide/AI-features/5-a-detection-status-monitoring/5-1-overview.md) - + [使用准备](/developer-guide/AI-features/5-a-detection-status-monitoring/5-2-preparations.md) - + [添加监控参数](/developer-guide/AI-features/5-a-detection-status-monitoring/5-3-adding-monitoring-parameters.md) - + [获取帮助](/developer-guide/AI-features/5-a-detection-status-monitoring/5-4-obtaining-help-information.md) - + [使用示例](/developer-guide/AI-features/5-a-detection-status-monitoring/5-6-examples.md) - + [命令参考](/developer-guide/AI-features/5-a-detection-status-monitoring/5-5-command-reference.md) - + [AI_SERVER](/developer-guide/AI-features/5-a-detection-status-monitoring/5-7-ai-server.md) - + [AI_MANAGER](/developer-guide/AI-features/5-a-detection-status-monitoring/5-8-ai-manager.md) - + Index-advisor 索引推荐 - + [单query索引推荐](/developer-guide/AI-features/6-index-advisor-index-recommendation/6-1-single-query-index-recommendation.md) - + [虚拟索引](/developer-guide/AI-features/6-index-advisor-index-recommendation/6-2-virtual-index.md) - + [workload级别索引推荐](/developer-guide/AI-features/6-index-advisor-index-recommendation/6-3-workload-level-index-recommendation.md) - + DeepSQL 库内AI算法 - + [概述](/developer-guide/AI-features/7-deepsql/7-1-overview.md) - + [环境部署](/developer-guide/AI-features/7-deepsql/7-2-environment-deployment.md) - + [使用指导](/developer-guide/AI-features/7-deepsql/7-3-usage-guide.md) - + [最佳实践](/developer-guide/AI-features/7-deepsql/7-4-best-practices.md) - + [常见问题处理](/developer-guide/AI-features/7-deepsql/7-5-troubleshooting.md) - + DB4AI 数据库原生AI引擎 - + [概述](/developer-guide/AI-features/8-db4ai/8-1-overview.md) - + [DB4AI-Snapshots数据版本管理](/developer-guide/AI-features/8-db4ai/8-2-db4ai-snapshots-for-data-version-management.md) - + [DB4AI-Query: 模型训练和推断](/developer-guide/AI-features/8-db4ai/8-3-db4ai-query-for-model-training-and-prediction.md) - + [plpython-fenced模式](/developer-guide/AI-features/8-db4ai/8-4-pl-python-fenced-mode.md) ++ [AI特性](/developer-guide/AI-features/1-AI-features-overview.md) + + [AI4DB：据库自治运维](/developer-guide/AI-features/ai4db/ai4db-autonomous-database-o&m.md) + + [DBMind模式说明](/developer-guide/AI-features/ai4db/dbmind-mode/dbmind-mode.md) + + [service子命令](/developer-guide/AI-features/ai4db/dbmind-mode/1-service.md) + + [component子命令](/developer-guide/AI-features/ai4db/dbmind-mode/2-component.md) + + [set子命令](/developer-guide/AI-features/ai4db/dbmind-mode/3-set.md) + + [DBMind的支持组件](/developer-guide/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md) + + Prometheus Exporter组件 + + [概述](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md) + + [环境部署](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md) + + DBMind的AI子功能 + + X-Tuner：参数调优与诊断 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md) + + [使用准备](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md) + + [使用示例](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md) + + Index-advisor：索引推荐 + + [单query索引推荐](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md) + + [虚拟索引](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md) + + [workload级别索引推荐](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md) + + AI4DB: 慢SQL根因分析 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-1-overview.md) + + [环境部署](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-ai4db-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md) + + AI4DB：趋势预测 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-1-overview.md) + + [环境部署](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-2-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-3-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-4-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-5-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-ai4db-trend-prediction/4-6-troubleshooting.md) + + SQLdiag：慢SQL发现 + + [概述](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md) + + [使用指导](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md) + + [获取帮助](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md) + + [命令参考](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md) + + [常见问题处理](/developer-guide/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md) + + [DB4AI：数据库驱动AI](/developer-guide/AI-features/db4ai/db4ai.md) + + [原生DB4AI引擎](/developer-guide/AI-features/db4ai/native-db4ai-engine.md) + + [全流程AI](/developer-guide/AI-features/db4ai/full-process-ai/full-process-ai.md) + + [PLPython Fenced模式](/developer-guide/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md) + + [DB4AI-Snapshots数据版本管理](/developer-guide/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md) + + AI in DB：数据库内AI功能 + + 智能Explain：SQL语句查询时间预测 + + [概述](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md) + + [环境部署](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md) + + [使用指导](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md) + + [最佳实践](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md) + + [常见问题处理](/developer-guide/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md) + 资源负载管理 + [资源负载管理概述](/developer-guide/resource-load-management/resource-load-management-overview.md) + 资源管理准备