diff --git a/product/en/docs-mogdb/v3.1/AI-features/1-AI-features-overview.md b/product/en/docs-mogdb/v3.1/AI-features/1-AI-features-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..130d3caa1b854027e76029116c47426840e52d35 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/1-AI-features-overview.md @@ -0,0 +1,18 @@ +--- +title: AI Features +summary: AI Features +author: Guo Huan +date: 2021-05-19 +--- + +# AI Features + +The history of artificial intelligence (AI) can be dated back to as early as the 1950s, even longer than the history of the database development. However, the AI technology has not been applied on a large scale for a long time due to various objective factors, and even experienced several obvious troughs. With the further development of information technologies in recent years, factors that restrict the AI development have been gradually weakened, and the AI, big data, and cloud computing (ABC) technologies are born. + +The combination of AI and databases has been a trending research topic in the industry in recent years. MogDB has participated in the exploration of this domain earlier and achieved phased achievements. An AI submodule DBMind is provided for the database. Compared with other functions, it is more independent. This module can be divided into AI4DB, DB4AI, and AI in DB. + +- AI4DB uses AI technologies to optimize database execution performance as well as achieve autonomy and O&M free. It includes self-tuning, self-diagnosis, self-security, self-O&M, and self-healing. +- DB4AI streamlines the E2E process from databases to AI applications, drives AI tasks through databases, and unifies the AI technology stack to achieve out-of-the-box, high performance, and cost saving. For example, SQL-like statements are used to implement functions such as recommendation system, image retrieval, and time series prediction. The advantages of high parallelism and column store of databases can be fully utilized to avoid the cost of data and fragmented storage and avoid security risks caused by information leakage. +- AI in DB modifies the database kernel to implement functions that cannot be implemented in the original database architecture. For example, AI algorithms are used to improve the database optimizer to implement more accurate cost estimation. + +The functions described in this section are stored in the **bin/dbmind** directory of the database installation directory (*$GAUSSHOME*). The sub-functions are stored in the **components** subdirectory of **bin/dbmind**. To invoke DBMind, you can run the **gs_dbmind** command. In addition, the built-in AI functions (such as DB4AI) of the database are presented in the form of SQL syntax and system functions. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md new file mode 100644 index 0000000000000000000000000000000000000000..966e32a56ea86d6f0656c4b4a303d4da1af3e049 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-best-practices.md @@ -0,0 +1,32 @@ +--- +title: Best Practices +summary: Best Practices +author: Guo Huan +date: 2022-05-06 +--- + +# Best Practices + +For details about the parameters, see [GS_OPT_MODEL](../../../reference-guide/system-catalogs-and-system-views/system-catalogs/GS_OPT_MODEL.md)。 + +**Table 1** + +| Model Parameter | Recommended Value | +| :--------------- | :----------------------------------------------------------- | +| template_name | ‘rlstm’ | +| model_name | The value can be customized, for example, **open_ai**. The value must meet the unique constraint. | +| datname | Name of the database to be served, for example, **postgres**. | +| ip | IP address of the AI Engine, for example, **127.0.0.1**. | +| port | AI Engine listening port number, for example, **5000**. | +| max_epoch | Iteration times. A large value is recommended to ensure the convergence effect, for example, **2000**. | +| learning_rate | (0, 1] is a floating-point number. A large learning rate is recommended to accelerate convergence. | +| dim_red | Number of feature values to be reduced. **-1**: Do not use PCA for dimension reduction. All features are supported. Floating point number in the range of (0, 1]: A smaller value indicates a smaller training dimension and a faster convergence speed, but affects the training accuracy. | +| hidden_units | If the feature value dimension is high, you are advised to increase the value of this parameter to increase the model complexity. For example, set this parameter to **64**, **128**, and so on. | +| batch_size | You are advised to increase the value of this parameter based on the amount of encoded data to accelerate model convergence. For example, set this parameter to **256**, **512**, and so on. | +| Other parameters | See [GS_OPT_MODEL](../../../reference-guide/system-catalogs-and-system-views/system-catalogs/GS_OPT_MODEL.md) | + +Recommended parameter settings: + +``` +INSERT INTO gs_opt_model values('rlstm', 'open_ai', 'postgres', '127.0.0.1', 5000, 2000，1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..ee45fd939a9ae54359a26cb462938e3f7f1e78b4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-environment-deployment.md @@ -0,0 +1,203 @@ +--- +title: Environment Deployment +summary: Environment Deployment +author: Guo Huan +date: 2022-05-06 +--- + +# Environment Deployment + +## Prerequisites + +MogDB is in the normal state. The user has logged in to MogDB with an authenticated identity. The executed SQL syntax is correct and no error is reported. In the historical performance data window, the number of MogDB concurrent tasks is stable, the structure and number of tables remain unchanged, the data volume changes smoothly, and the GUC parameters related to query performance remain unchanged. During prediction, the model has been trained and converged. The running environment of the AIEngine is stable. + +## Request Example + +The AIEngine process communicates with the kernel process using HTTPS. An example request is as follows: + +```bash +curl -X POST -d '{"modelName":"modelname"}' -H 'Content-Type: application/json' https://IP-address:port/request-API +``` + +**Table 1** AIEngine external APIs + +| Request API | Description | +| :------------- | :------------------------------------------ | +| /check | Checks whether a model is properly started. | +| /configure | Sets model parameters. | +| /train | Trains a model. | +| /track_process | Views model training logs. | +| /setup | Loads historical models. | +| /predict | Predicts a model. | + +## Generating Certificates + +Before using the prediction function, you need to use OpenSSL to generate certificates required for authentication between the communication parties, ensuring communication security. + +1. Set up a certificate generation environment. The certificate file is stored in **$GAUSSHOME/CA**. + + Copy the certificate generation script and related files. + + ```bash + cp path_to_predictor/install/ssl.sh $GAUSSHOME/ + cp path_to_predictor/install/ca_ext.txt $GAUSSHOME/ + ``` + + Copy the configuration file **openssl.cnf** to **$GAUSSHOME**. + + ```bash + cp $GAUSSHOME/share/om/openssl.cnf $GAUSSHOME/ + ``` + + Modify the configuration parameters in **openssl.conf**. + + ```bash + dir = $GAUSSHOME/CA/demoCA + default_md = sha256 + ``` + + The certificate generation environment is ready. + +2. Generate a certificate and private key. + + ```bash + cd $GAUSSHOME + sh ssl.sh + ``` + + Set the password as prompted, for example, **Test@123**. + + The password must contain at least eight characters of at least three different types. + + ```bash + Please enter your password: + ``` + + Set the options as prompted. + + ```bash + Certificate Details: + Serial Number: 1 (0x1) + Validity + Not Before: May 15 08:32:44 2020 GMT + Not After : May 15 08:32:44 2021 GMT + Subject: + countryName = CN + stateOrProvinceName = SZ + organizationName = HW + organizationalUnitName = GS + commonName = CA + X509v3 extensions: + X509v3 Basic Constraints: + CA:TRUE + Certificate is to be certified until May 15 08:32:44 2021 GMT (365 days) + Sign the certificate? [y/n]:y + 1 out of 1 certificate requests certified, commit? [y/n]y + ``` + + Enter the IP address for starting the AIEngine, for example, **127.0.0.1**. + + ```bash + Please enter your aiEngine IP: 127.0.0.1 + ``` + + Set the options as prompted. + + ```bash + Certificate Details: + Serial Number: 2 (0x2) + Validity + Not Before: May 15 08:38:07 2020 GMT + Not After : May 13 08:38:07 2030 GMT + Subject: + countryName = CN + stateOrProvinceName = SZ + organizationName = HW + organizationalUnitName = GS + commonName = 127.0.0.1 + X509v3 extensions: + X509v3 Basic Constraints: + CA:FALSE + Certificate is to be certified until May 13 08:38:07 2030 GMT (3650 days) + Sign the certificate? [y/n]:y + 1 out of 1 certificate requests certified, commit? [y/n]y + ``` + + Enter the IP address for starting MogDB, for example, **127.0.0.1**. + + ```bash + Please enter your mogdb IP: 127.0.0.1 + ``` + + Set the options as prompted. + + ```bash + Certificate Details: + Serial Number: 3 (0x3) + Validity + Not Before: May 15 08:41:46 2020 GMT + Not After : May 13 08:41:46 2030 GMT + Subject: + countryName = CN + stateOrProvinceName = SZ + organizationName = HW + organizationalUnitName = GS + commonName = 127.0.0.1 + X509v3 extensions: + X509v3 Basic Constraints: + CA:FALSE + Certificate is to be certified until May 13 08:41:46 2030 GMT (3650 days) + Sign the certificate? [y/n]:y + 1 out of 1 certificate requests certified, commit? [y/n]y + ``` + + The related certificate and key have been generated. The content in **$GAUSSHOME/CA** is as follows: + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/intelligence-explain-1.png) + +## Setting Up the Environment + +1. Copy the tool code folder to the target environment. + + – Assume that the installation directory is **$INSTALL_FOLDER**. + + – Assume that the destination directory is **/home/ai_user**. + + ```bash + scp -r $INSTALL_FOLDER/bin/dbmind/predictor ai_user@127.0.0.1:path_to_Predictor + ``` + +2. Copy the CA certificate folder to a directory in the AIEngine environment. + + ```bash + cp -r $GAUSSHOME/CA ai_user@127.0.0.1:path_to_CA + ``` + +3. Install the **predictor/install/requirements(-gpu).txt** tool. + + ```bash + With GPU: pip install -r requirements-gpu.txt + Without GPU: pip install -r requirements.txt + ``` + +## Starting the AIEngine + +1. Switch to the AIEngine environment (that is, copy the target environment **ai_user** of the predictor). + + Set parameters in **predictor/python/settings.py**. + + ```bash + DEFAULT_FLASK_SERVER_HOST = '127.0.0.1' (running IP address of the AIEngine) + DEFAULT_FLASK_SERVER_PORT = '5000' (running port number of the AIEngine) + PATH_SSL = "path_to_CA" (CA folder path) + ``` + +2. Run the startup script of the AIEngine. + + ```bash + python path_to_Predictor/python/run.py + ``` + + In this case, the AIEngine keeps enabled on the corresponding port and waits for the request of the time prediction function from the kernel. + + For details about how to initiate a time prediction command from the kernel, see[User Guide](intelligence-explain-user-guide.md)。 diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md new file mode 100644 index 0000000000000000000000000000000000000000..ef6e404d02fecdda73e32f6cf733e4507f2f6c30 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-faqs.md @@ -0,0 +1,36 @@ +--- +title: FAQs +summary: FAQs +author: Guo Huan +date: 2022-05-06 +--- + +# FAQs + +## AI Engine Configuration + +- **AI Engine failed to be started**: Check whether the IP address and port are available and whether the CA certificate path exists. +- **AI Engine does not respond**: Check whether the CA certificates of the two communication parties are consistent. +- **Training and test failure**: Check whether the path for saving the model files exists and whether the training prediction file is correctly downloaded. + +- **Changing the AI Engine IP address**: Regenerate the certificate by following the steps in [Generating Certificates](intelligence-explain-environment-deployment.md#证书生成). Enter the new IP address in Generate a certificate and private key. + +## Database Internal Errors + +Problem: AI Engine connection failed. + +``` +ERROR: AI engine connection failed. +CONTEXT: referenced column: model_train_opt +``` + +Solution: Check whether the AI Engine is started or restarted properly. Check whether the CA certificates of the communication parties are consistent. Check whether the IP address and port number in the model configuration match. + +Problem: The model does not exist. + +``` +ERROR: OPT_Model not found for model name XXX +CONTEXT: referenced column: track_model_train_opt +``` + +Solution: Check whether [GS_OPT_MODEL](../../../reference-guide/system-catalogs-and-system-views/system-catalogs/GS_OPT_MODEL.md) contains the model specified in the **model_name** column in the statement. If the error is reported when the prediction function is used, check whether the model has been trained. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..e61176a69f05f203394bbca616584e7db9ea1370 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-overview.md @@ -0,0 +1,14 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2022-05-06 +--- + +# Overview + +Predictor is a query time prediction tool that leverages machine learning and has online learning capabilities. By continuously learning the historical execution information collected in the database, Predictor can predict the execution time of a plan. + +To use this tool, you must start the Python process AI Engine for model training and inference. + +This function is stored in the *$GAUSSHOME***/bin/dbmind/components/predictor** directory. Some functions of this module involve complex setup. Therefore, you need to switch to the directory to find the corresponding files and deploy the functions according to the description in this section. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..062231cae19eb920967fe885b4a186c648175254 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai-in-db/intelligence-explain/intelligence-explain-user-guide.md @@ -0,0 +1,205 @@ +--- +title: User Guide +summary: User Guide +author: Guo Huan +date: 2022-05-06 +--- + +# User Guide + +## Data Collection + +1. Enable data collection. + + 1. Set parameters related to the ActiveSQL operator. + + ``` + enable_resource_track=on + resource_track_level=operator + enable_resource_record=on + resource_track_cost=10 (The default value is 100000.) + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The value of **resource_track_cost** must be smaller than the total query cost of the information to be collected. Only the information that meets the requirements can be collected. + > - Cgroup functions are available. + + 2. Collect information. + + Execute the service query statement. + + View data collected in real time. + + ``` + select * from gs_wlm_plan_operator_history; + ``` + + Expected result: All jobs that meet **resource_track_duration** and **resource_track_cost** are collected. + +2. Disable data collection. + + 1. Set parameters related to the ActiveSQL operator. + + ``` + enable_resource_track=off + resource_track_level=none + resource_track_level=query + ``` + + 2. Execute the service query statement. + + Wait for 3 minutes and check the data on the current node. + + ``` + select * from gs_wlm_plan_operator_info; + ``` + + Expected result: No new data is added to the tables and views. + +3. Persist data. + + 1. Set parameters related to the ActiveSQL operator. + + ``` + enable_resource_track=on + resource_track_level=operator + enable_resource_record=on + resource_track_duration=0 (The default value is 60s.) + resource_track_cost=10 (The default value is 100000.) + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The value of **resource_track_cost** must be smaller than the total query cost of the information to be collected. Only the information that meets the requirements can be collected. + > - Cgroup functions are available. + + 2. Execute the service query statement. + + Wait for 3 minutes and check the data on the current node. + + ``` + select * from gs_wlm_plan_operator_info; + ``` + + Expected result: All jobs that meet **resource_track_duration** and **resource_track_cost** are collected. + +## Model Management (System Administrators) + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Model management operations can be performed only when the database is normal. + +1. Add a new model. + + INSERT INTO gs_opt_model values('……')； + + Example: + + ``` + INSERT INTO gs_opt_model values('rlstm', 'model_name', 'datname', '127.0.0.1', 5000, 2000, 1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - For details about model parameter settings, see [GS_OPT_MODEL](../../../reference-guide/system-catalogs-and-system-views/system-catalogs/GS_OPT_MODEL.md). + > - Currently, only **rlstm** is supported in the **template_name** column. + > - The values in the **datname** column must be the same as those in the database used for model usage and training. Otherwise, the model cannot be used. + > - The values in the **model_name** column must meet the **unique** constraint. + > - For details about other parameter settings, see [Best Practices](intelligence-explain-best-practices.md). + +2. Modify model parameters. + + ``` + UPDATE gs_opt_model SET = WHERE model_name = ; + ``` + +3. Delete a model. + + ``` + DELETE FROM gs_opt_model WHERE model_name = ; + ``` + +4. Query the existing model and its status. + + ``` + SELECT * FROM gs_opt_model; + ``` + +## Model Training (System Administrators) + +1. Add models and modify model parameters by following the steps in Model Management (System Administrators). + + Example: + + Add a mode. + + ``` + INSERT INTO gs_opt_model values('rlstm', 'default', 'postgres', '127.0.0.1', 5000, 2000, 1, -1, 64, 512, 0 , false, false, '{S, T}', '{0,0}', '{0,0}', 'Text'); + ``` + + Modify training parameters. + + ``` + UPDATE gs_opt_model SET = WHERE model_name = ; + ``` + +2. Check that the database status is normal and historical data is collected properly before you perform the following operations: + + Delete the original encoding data. + + ``` + DELETE FROM gs_wlm_plan_encoding_table; + ``` + + To encode data, specify the database name. + + ``` + SELECT gather_encoding_info('postgres'); + ``` + + Start training. + + ``` + SELECT model_train_opt('rlstm', 'default'); + ``` + +3. Run the following command to obtain the relative path of the model training log on the AIEngine side: + + ``` + SELECT * FROM track_model_train_opt('rlstm', 'default'); + ``` + +## Model Prediction + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - Model prediction can be performed only when the database status is normal and the specified model has been trained and converged. +> +> - Currently, the labels of model training parameters must contain the **S** label so that the **p-time** value can be displayed in **EXPLAIN**. Example: INSERT INTO gs_opt_model values('rlstm', 'default', 'postgres', '127.0.0.1', 5000, 1000, 1, -1, 50, 500, 0 , false, false, '{**S**, T}', '{0,0}', '{0,0}', 'Text'); + +1. Call EXPLAIN. + + ``` + explain (analyze on, predictor ) + SELECT ... + ``` + + Expected result: + + ``` + Example: Row Adapter (cost=110481.35..110481.35 rows=100 p-time=99..182 width=100) (actual time=375.158..375.160 rows=2 loops=1) + The p-time column indicates the predicted value of the label. + ``` + +## Other Functions + +1. Check whether the AIEngine can be connected. + + ``` + MogDB=# select check_engine_status('aiEngine-ip-address',running-port); + ``` + +2. Check the path for storing model logs on the AIEngine. + + ``` + MogDB=# select track_model_train_opt('template_name', 'model_name'); + ``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..17d57b3f2614cd4acb28e10dadc38391c7576472 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-1-x-tuner-overview.md @@ -0,0 +1,10 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2021-05-19 +--- + +# Overview + +X-Tuner is a parameter tuning tool integrated into databases. It uses AI technologies such as deep reinforcement learning and global search algorithm to obtain the optimal database parameter settings without manual intervention. This function is not necessarily deployed with the database environment. It can be independently deployed and run without the database installation environment. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md new file mode 100644 index 0000000000000000000000000000000000000000..dfd50971d88985879fc968fb557c5d29fe3d141a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-2-preparations.md @@ -0,0 +1,227 @@ +--- +title: Preparations +summary: Preparations +author: Guo Huan +date: 2021-10-21 +--- + +# Preparations + +
+ +## Prerequisites and Precautions + +- The database status is normal; the client can be properly connected; and data can be imported to the database. As a result, the optimization program can perform the benchmark test for optimization effect. +- To use this tool, you need to specify the user who logs in to the database. The user who logs in to the database must have sufficient permissions to obtain sufficient database status information. +- If you log in to the database host as a Linux user, add **$GAUSSHOME/bin** to the **PATH** environment variable so that you can directly run database O&M tools, such as gsql, gs_guc, and gs_ctl. +- The recommended Python version is Python 3.6 or later. The required dependency has been installed in the operating environment, and the optimization program can be started properly. You can install a Python 3.6+ environment independently without setting it as a global environment variable. You are not advised to install the tool as the root user. If you install the tool as the root user and run the tool as another user, ensure that you have the read permission on the configuration file. +- This tool can run in three modes. In **tune** and **train** modes, you need to configure the benchmark running environment and import data. This tool will iteratively run the benchmark to check whether the performance is improved after the parameters are modified. +- In **recommend** mode, you are advised to run the command when the database is executing the workload to obtain more accurate real-time workload information. +- By default, this tool provides benchmark running script samples of TPC-C, TPC-H, TPC-DS, and sysbench. If you use the benchmarks to perform pressure tests on the database system, you can modify or configure the preceding configuration files. To adapt to your own service scenarios, you need to compile the script file that drives your customized benchmark based on the **template.py** file in the **benchmark** directory. + +
+ +## Principles + +The tuning program is a tool independent of the database kernel. The usernames and passwords for the database and instances are required to control the benchmark performance test of the database. Before starting the tuning program, ensure that the interaction in the test environment is normal, the benchmark test script can be run properly, and the database can be connected properly. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** If the parameters to be tuned include the parameters that take effect only after the database is restarted, the database will be restarted multiple times during the tuning. Exercise caution when using **train** and **tune** modes if the database is running jobs. + +X-Tuner can run in any of the following modes: + +- **recommend**: Log in to the database using the specified user name, obtain the feature information about the running workload, and generate a parameter recommendation report based on the feature information. Report improper parameter settings and potential risks in the current database. Output the currently running workload behavior and characteristics. Output the recommended parameter settings. In this mode, the database does not need to be restarted. In other modes, the database may need to be restarted repeatedly. +- **train**: Modify parameters and execute the benchmark based on the benchmark information provided by users. The reinforcement learning model is trained through repeated iteration so that you can load the model in **tune** mode for optimization. +- **tune**: Use an optimization algorithm to tune database parameters. Currently, two types of algorithms are supported: deep reinforcement learning and global search algorithm (global optimization algorithm). The deep reinforcement learning mode requires **train** mode to generate the optimized model after training. However, the global search algorithm does not need to be trained in advance and can be directly used for search and optimization. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** If the deep reinforcement learning algorithm is used in **tune** mode, a trained model must be available, and the parameters for training the model must be the same as those in the parameter list (including max and min) for tuning. + +**Figure 1** X-Tuner structure + +![x-tuner-structure](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/preparations-1.png) + +Figure 1 X-Tuner architecture shows the overall architecture of the X-Tuner. The X-Tuner system can be divided into the following parts: + +- DB: The DB_Agent module is used to abstract database instances. It can be used to obtain the internal database status information and current database parameters and set database parameters. The SSH connection used for logging in to the database environment is included on the database side. +- Algorithm: algorithm package used for optimization, including global search algorithms (such as Bayesian optimization and particle swarm optimization) and deep reinforcement learning (such as DDPG). +- X-Tuner main logic module: encapsulated by the environment module. Each step is an optimization process. The entire optimization process is iterated through multiple steps. +- benchmark: a user-specified benchmark performance test script, which is used to run benchmark jobs. The benchmark result reflects the performance of the database system. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Ensure that the larger the benchmark script score is, the better the performance is. For example, for the benchmark used to measure the overall execution duration of SQL statements, such as TPC-H, the inverse value of the overall execution duration can be used as the benchmark score. + +
+ +## Installing and Running X-Tuner + +You can run the X-Tuner in two ways. One is to run the X-Tuner directly through the source code. The other is to install the X-Tuner on the system through the Python setuptools, and then run the **gs_xtuner** command to call the X-Tuner. The following describes two methods of running the X-Tuner. + +Method 1: Run the source code directly. + +1. Switch to the **xtuner** source code directory. For the openGauss community code, the path is **openGauss-server/src/gausskernel/dbmind/tools/xtuner**. For an installed database system, the source code path is *$GAUSSHOME***/bin/dbmind/xtuner**. + +2. You can view the **requirements.txt** file in the current directory. Use the pip package management tool to install the dependency based on the **requirements.txt** file. + + ```bash + pip install -r requirements.txt + ``` + +3. After the installation is successful, add the environment variable PYTHONPATH, and then run **main.py**. For example, to obtain the help information, run the following command: + + ```bash + cd tuner # Switch to the directory where the main.py entry file is located. + export PYTHONPATH='..' # Add the upper-level directory to the path for searching for packages. + python main.py --help # Obtain help information. The methods of using other functions are similar. + ``` + +Method 2: Install the X-Tuner in the system. + +1. You can use the **setup.py** file to install the X-Tuner to the system and then run the **gs_xtuner** command. You need to switch to the root directory of **xtuner**. For details about the directory location, see the preceding description. + +2. Run the following command to install the tool in the Python environment using Python setuptools: + + ```bash + python setup.py install + ``` + + If the **bin** directory of Python is added to the *PATH* environment variable, the **gs_xtuner** command can be directly called anywhere. + +3. For example, to obtain the help information, run the following command: + + ```bash + gs_xtuner --help + ``` + +
+ +## Description of the X-Tuner Configuration File + +Before running the X-Tuner, you need to load the configuration file. The default path of the configuration file is tuner/xtuner.conf. You can run the **gs_xtuner -help** command to view the absolute path of the configuration file that is loaded by default. + +``` +... + -x TUNER_CONFIG_FILE, --tuner-config-file TUNER_CONFIG_FILE + This is the path of the core configuration file of the + X-Tuner. You can specify the path of the new + configuration file. The default path is /path/to/xtuner/xtuner.conf. + You can modify the configuration file to control the + tuning process. +... +``` + +You can modify the configuration items in the configuration file as required to instruct the X-Tuner to perform different actions. For details about the configuration items in the configuration file, see Table 2 in [Command Reference](1-5-command-reference.md). If you need to change the loading path of the configuration file, you can specify the path through the **-x** command line option. + +
+ +## Benchmark Selection and Configuration + +The benchmark drive script is stored in the benchmark subdirectory of the X-Tuner. X-Tuner provides common benchmark driver scripts, such as TPC-C and TPC-H. The X-Tuner invokes the **get_benchmark_instance()** command in the benchmark/__init__.py file to load different benchmark driver scripts and obtain benchmark driver instances. The format of the benchmark driver script is described as follows: + +- Name of the driver script: name of the benchmark. The name is used to uniquely identify the driver script. You can specify the benchmark driver script to be loaded by setting the **benchmark_script** configuration item in the configuration file of the X-Tuner. +- The driver script contains the *path* variable, *cmd* variable, and the **run** function. + +The following describes the three elements of the driver script: + +1. *path*: path for saving the benchmark script. You can modify the path in the driver script or specify the path by setting the **benchmark_path** configuration item in the configuration file. + +2. *cmd*: command for executing the benchmark script. You can modify the command in the driver script or specify the command by setting the **benchmark_cmd** configuration item in the configuration file. Placeholders can be used in the text of cmd to obtain necessary information for running cmd commands. For details, see the TPC-H driver script example. These placeholders include: + + - {host}: IP address of the database host machine + - {port}: listening port number of the database instance + - {user}: user name for logging in to the database + - {password}: password of the user who logs in to the database system + - {db}: name of the database that is being optimized + +3. **run** function: The signature of this function is as follows: + + ``` + def run(remote_server, local_host) -> float: + ``` + + The returned data type is float, indicating the evaluation score after the benchmark is executed. A larger value indicates better performance. For example, the TPC-C test result tpmC can be used as the returned value, the inverse number of the total execution time of all SQL statements in TPC-H can also be used as the return value. A larger return value indicates better performance. + + The *remote_server* variable is the shell command interface transferred by the X-Tuner program to the remote host (database host machine) used by the script. The *local_host* variable is the shell command interface of the local host (host where the X-Tuner script is executed) transferred by the X-Tuner program. Methods provided by the preceding shell command interface include: + + ``` + exec_command_sync(command, timeout) + Function: This method is used to run the shell command on the host. + Parameter list: + command: The data type can be str, and the element can be a list or tuple of the str type. This parameter is optional. + timeout: The timeout interval for command execution in seconds. This parameter is optional. + Return value: + Returns 2-tuple (stdout and stderr). stdout indicates the standard output stream result, and stderr indicates the standard error stream result. The data type is str. + ``` + + ``` + exit_status + Function: This attribute indicates the exit status code after the latest shell command is executed. + Note: Generally, if the exit status code is 0, the execution is normal. If the exit status code is not 0, an error occurs. + ``` + +Benchmark driver script example: + +1. TPC-C driver script + + ```bash + from tuner.exceptions import ExecutionError + + # WARN: You need to download the benchmark-sql test tool to the system, + # replace the PostgreSQL JDBC driver with the openGauss driver, + # and configure the benchmark-sql configuration file. + # The program starts the test by running the following command: + path = '/path/to/benchmarksql/run' # Path for storing the TPC-C test script benchmark-sql + cmd = "./runBenchmark.sh props.gs" # Customize a benchmark-sql test configuration file named props.gs. + + def run(remote_server, local_host): + # Switch to the TPC-C script directory, clear historical error logs, and run the test command. + # You are advised to wait for several seconds because the benchmark-sql test script generates the final test report through a shell script. The entire process may be delayed. + # To ensure that the final tpmC value report can be obtained, wait for 3 seconds. + stdout, stderr = remote_server.exec_command_sync(['cd %s' % path, 'rm -rf benchmarksql-error.log', cmd, 'sleep 3']) + # If there is data in the standard error stream, an exception is reported and the system exits abnormally. + if len(stderr) > 0: + raise ExecutionError(stderr) + + # Find the final tpmC result. + tpmC = None + split_string = stdout.split() # Split the standard output stream result. + for i, st in enumerate(split_string): + # In the benchmark-sql of version 5.0, the value of tpmC is the last two digits of the keyword (NewOrders). In normal cases, the value of tpmC is returned after the keyword is found. + if "(NewOrders)" in st: + tpmC = split_string[i + 2] + break + stdout, stderr = remote_server.exec_command_sync( + "cat %s/benchmarksql-error.log" % path) + nb_err = stdout.count("ERROR:") # Check whether errors occur during the benchmark running and record the number of errors. + return float(tpmC) - 10 * nb_err # The number of errors is used as a penalty item, and the penalty coefficient is 10. A higher penalty coefficient indicates a larger number of errors. + + ``` + +2. TPC-H driver script + + ```bash + import time + + from tuner.exceptions import ExecutionError + + # WARN: You need to import data into the database and SQL statements in the following path will be executed. + # The program automatically collects the total execution duration of these SQL statements. + path = '/path/to/tpch/queries' # Directory for storing SQL scripts used for the TPC-H test + cmd = "gsql -U {user} -W {password} -d {db} -p {port} -f {file}" # The command for running the TPC-H test script. Generally, gsql -f script file is used. + + def run(remote_server, local_host): + # Traverse all test case file names in the current directory. + find_file_cmd = "find . -type f -name '*.sql'" + stdout, stderr = remote_server.exec_command_sync(['cd %s' % path, find_file_cmd]) + if len(stderr) > 0: + raise ExecutionError(stderr) + files = stdout.strip().split('\n') + time_start = time.time() + for file in files: + # Replace {file} with the file variable and run the command. + perform_cmd = cmd.format(file=file) + stdout, stderr = remote_server.exec_command_sync(['cd %s' % path, perform_cmd]) + if len(stderr) > 0: + print(stderr) + # The cost is the total execution duration of all test cases. + cost = time.time() - time_start + # Use the inverse number to adapt to the definition of the run function. The larger the returned result is, the better the performance is. + return - cost + ``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md new file mode 100644 index 0000000000000000000000000000000000000000..311ccc689907c8df58c134dce8f0bda0b2fcae8d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-3-examples.md @@ -0,0 +1,151 @@ +--- +title: Examples +summary: Examples +author: Guo Huan +date: 2021-05-19 +--- + +# Examples + +X-Tuner supports three modes: recommend mode for obtaining parameter diagnosis reports, train mode for training reinforcement learning models, and tune mode for using optimization algorithms. The preceding three modes are distinguished by command line parameters, and the details are specified in the configuration file. + +## Configuring the Database Connection Information + +Configuration items for connecting to a database in the three modes are the same. You can enter the detailed connection information in the command line or in the JSON configuration file. Both methods are described as follows: + +1. Entering the connection information in the command line + + Input the following options: **-db-name -db-user -port -host -host-user**. The **-host-ssh-port** is optional. The following is an example: + + ``` + gs_xtuner recommend --db-name postgres --db-user omm --port 5678 --host 192.168.1.100 --host-user omm + ``` + +2. Entering the connection information in the JSON configuration file + + Assume that the file name is **connection.json**. The following is an example of the JSON configuration file: + + ``` + { + "db_name": "postgres", # Database name + "db_user": "dba", # Username for logging in to the database + "host": "127.0.0.1", # IP address of the database host + "host_user": "dba", # Username for logging in to the database host + "port": 5432, # Listening port number of the database + "ssh_port": 22 # SSH listening port number of the database host + } + ``` + + Input **-f connection.json**. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** To prevent password leakage, the configuration file and command line parameters do not contain password information by default. After you enter the preceding connection information, the program prompts you to enter the database password and the OS login password in interactive mode. + +## Example of Using recommend Mode + +The configuration item **scenario** takes effect for recommend mode. If the value is **auto**, the workload type is automatically detected. + +Run the following command to obtain the diagnosis result: + +``` + +gs_xtuner recommend -f connection.json + +``` + +The diagnosis report is generated as follows: + +**Figure 1** Report generated in recommend mode + +![report-generated-in-recommend-mode](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/examples-1.png) + +In the preceding report, the database parameter configuration in the environment is recommended, and a risk warning is provided. The report also generates the current workload features. The following features are for reference: + +- **temp_file_size**: number of generated temporary files. If the value is greater than 0, the system uses temporary files. If too many temporary files are used, the performance is poor. If possible, increase the value of **work_mem**. +- **cache_hit_rate**: cache hit ratio of **shared_buffer**, indicating the cache efficiency of the current workload. +- **read_write_ratio**: read/write ratio of database jobs. +- **search_modify_ratio**: ratio of data query to data modification of a database job. +- **ap_index**: AP index of the current workload. The value ranges from 0 to 10. A larger value indicates a higher preference for data analysis and retrieval. +- **workload_type**: workload type, which can be AP, TP, or HTAP based on database statistics. +- **checkpoint_avg_sync_time**: average duration for refreshing data to the disk each time when the database is at the checkpoint, in milliseconds. +- **load_average**: average load of each CPU core in 1 minute, 5 minutes, and 15 minutes. Generally, if the value is about 1, the current hardware matches the workload. If the value is about 3, the current workload is heavy. If the value is greater than 5, the current workload is too heavy. In this case, you are advised to reduce the load or upgrade the hardware. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** In recommend mode, information in the **pg_stat_database** and **pg_stat_bgwriter**system catalogs in the database is read. Therefore, the database login user must have sufficient permissions. (You are advised to own the administrator permission which can be granted to username using **alter user username sysadmin**.) Some system catalogs keep recording statistics, which may affect load feature identification. Therefore, you are advised to clear the statistics of some system catalogs, run the workload for a period of time, and then use recommend mode for diagnosis to obtain more accurate results. To clear the statistics, run the following command: select pg_stat_reset_shared('bgwriter'); select pg_stat_reset(); + +## Example of Using train Mode + +This mode is used to train the deep reinforcement learning model. The configuration items related to this mode are as follows: + +- **rl_algorithm**: algorithm used to train the reinforcement learning model. Currently, this parameter can be set to **ddpg**. + +- **rl_model_path**: path for storing the reinforcement learning model generated after training. + +- **rl_steps**: maximum number of training steps in the training process. + +- **max_episode_steps**: maximum number of steps in each episode. + +- **scenario**: specifies the workload type. If the value is **auto**, the system automatically determines the workload type. The recommended parameter tuning list varies according to the mode. + +- **tuning_list**: specifies the parameters to be tuned. If this parameter is not specified, the list of parameters to be tuned is automatically recommended based on the workload type. If this parameter is specified, **tuning_list**indicates the path of the tuning list file. The following is an example of the content of a tuning list configuration file. + + ``` + { + "work_mem": { + "default": 65536, + "min": 65536, + "max": 655360, + "type": "int", + "restart": false + }, + "shared_buffers": { + "default": 32000, + "min": 16000, + "max": 64000, + "type": "int", + "restart": true + }, + "random_page_cost": { + "default": 4.0, + "min": 1.0, + "max": 4.0, + "type": "float", + "restart": false + }, + "enable_nestloop": { + "default": true, + "type": "bool", + "restart": false + } + } + ``` + +After the preceding configuration items are configured, run the following command to start the training: + +``` + +gs_xtuner train -f connection.json + +``` + +After the training is complete, a model file is generated in the directory specified by the **rl_model_path**configuration item. + +## Example of Using tune Mode + +The tune mode supports a plurality of algorithms, including a DDPG algorithm based on reinforcement learning (RL), and a Bayesian optimization algorithm and a particle swarm algorithm (PSO) which are both based on a global optimization algorithm (GOP). + +The configuration items related to tune mode are as follows: + +- **tune_strategy**: specifies the algorithm to be used for optimization. The value can be **rl**(using the reinforcement learning model), **gop**(using the global optimization algorithm), or **auto**(automatic selection). If this parameter is set to **rl**, RL-related configuration items take effect. In addition to the preceding configuration items that take effect in train mode, the **test_episode**configuration item also takes effect. This configuration item indicates the maximum number of episodes in the tuning process. This parameter directly affects the execution time of the tuning process. Generally, a larger value indicates longer time consumption. +- **gop_algorithm**: specifies a global optimization algorithm. The value can be **bayes** or **pso**. +- **max_iterations**: specifies the maximum number of iterations. A larger value indicates a longer search time and better search effect. +- **particle_nums**: specifies the number of particles. This parameter is valid only for the PSO algorithm. +- For details about **scenario** and **tuning_list**, see the description of train mode. + +After the preceding items are configured, run the following command to start tuning: + +``` + +gs_xtuner tune -f connection.json + +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** Before using tune and train modes, you need to import the data required by the benchmark, check whether the benchmark can run properly, and back up the current database parameters. To query the current database parameters, run the following command: select name, setting from pg_settings; diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..12f32ef275085145a39b65474fc5e50fb90cb69e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-4-obtaining-help-information.md @@ -0,0 +1,51 @@ +--- +title: Obtaining Help Information +summary: Obtaining Help Information +author: Guo Huan +date: 2021-05-19 +--- + +# Obtaining Help Information + +Before starting the tuning program, run the following command to obtain help information: + +```bash +python main.py --help +``` + +The command output is as follows: + +```bash +usage: main.py [-h] [-m {train,tune}] [-f CONFIG_FILE] [--db-name DB_NAME] +[--db-user DB_USER] [--port PORT] [--host HOST] +[--host-user HOST_USER] [--host-ssh-port HOST_SSH_PORT] +[--scenario {ap,tp,htap}] [--benchmark BENCHMARK] +[--model-path MODEL_PATH] [-v] + +X-Tuner: a self-tuning toolkit for MogD. + +optional arguments: +-h, --help show this help message and exit +-m {train,tune}, --mode {train,tune} +train a reinforcement learning model or tune by your +trained model. +-f CONFIG_FILE, --config-file CONFIG_FILE +you can pass a config file path or you should manually +set database information. +--db-name DB_NAME database name. +--db-user DB_USER database user name. +--port PORT database connection port. +--host HOST where did your database install on? +--host-user HOST_USER +user name of the host where your database installed +on. +--host-ssh-port HOST_SSH_PORT +host ssh port. +--scenario {ap,tp,htap} +--benchmark BENCHMARK +--model-path MODEL_PATH +the place where you want to save model weights to or +load model weights from. +-v, --version +show version. +``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..b91dcea4663c9e5f376972215258148d1f932fc4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-5-command-reference.md @@ -0,0 +1,50 @@ +--- +title: Command Reference +summary: Command Reference +author: Guo Huan +date: 2021-05-19 +--- + +# Command Reference + +**Table 1** Command-line Parameter + +| Parameter | Description | Value Range | +| :--------------------- | :----------------------------------------------------------- | :--------------------- | +| mode | Specifies the running mode of the tuning program. | train, tune, recommend | +| -tuner-config-file, -x | Path of the core parameter configuration file of X-Tuner. The default path is **xtuner.conf** under the installation directory. | - | +| -db-config-file, -f | Path of the connection information configuration file used by the optimization program to log in to the database host. If the database connection information is configured in this file, the following database connection information can be omitted. | - | +| -db-name | Specifies the name of a database to be tuned. | - | +| -db-user | Specifies the user account used to log in to the tuned database. | - | +| -port | Specifies the database listening port. | - | +| -host | Specifies the host IP address of the database instance. | - | +| -host-user | Specifies the username for logging in to the host where the database instance is located. The database O&M tools, such as **gsql** and **gs_ctl**, can be found in the environment variables of the username. | - | +| -host-ssh-port | Specifies the SSH port number of the host where the database instance is located. This parameter is optional. The default value is **22**. | - | +| -help, -h | Returns the help information. | - | +| -version, -v | Returns the current tool version. | - | + +**Table 2** Parameters in the configuration file + +| Parameter | Description | Value Range | +| :-------------------- | :----------------- | :------------------- | +| logfile | Path for storing generated logs. | - | +| output_tuning_result | (Optional) Specifies the path for saving the tuning result. | - | +| verbose | Whether to print details. | on, off | +| recorder_file | Path for storing logs that record intermediate tuning information. | - | +| tune_strategy | Specifies a strategy used in tune mode. | rl, gop, auto | +| drop_cache | Whether to perform drop cache in each iteration. Drop cache can make the benchmark score more stable. If this parameter is enabled, add the login system user to the **/etc/sudoers** list and grant the NOPASSWD permission to the user. (You are advised to enable the NOPASSWD permission temporarily and disable it after the tuning is complete.) | on, off | +| used_mem_penalty_term | Penalty coefficient of the total memory used by the database. This parameter is used to prevent performance deterioration caused by unlimited memory usage. The greater the value is, the greater the penalty is. | Recommended value: 0 ~ 1 | +| rl_algorithm | Specifies the RL algorithm. | ddpg | +| rl_model_path | Path for saving or reading the RL model, including the save directory name and file name prefix. In train mode, this path is used to save the model. In tune mode, this path is used to read the model file. | - | +| rl_steps | Number of training steps of the deep reinforcement learning algorithm | - | +| max_episode_steps | Maximum number of training steps in each episode | - | +| test_episode | Number of episodes when the RL algorithm is used for optimization | - | +| gop_algorithm | Specifies a global optimization algorithm. | bayes, pso, auto | +| max_iterations | Maximum number of iterations of the global search algorithm. (The value is not fixed. Multiple iterations may be performed based on the actual requirements.) | - | +| particle_nums | Number of particles when the PSO algorithm is used | - | +| benchmark_script | Benchmark driver script. This parameter specifies the file with the same name in the benchmark path to be loaded. Typical benchmarks, such as TPC-C and TPC-H, are supported by default. | tpcc, tpch, tpcds, sysbench … | +| benchmark_path | Path for saving the benchmark script. If this parameter is not configured, the configuration in the benchmark drive script is used. | - | +| benchmark_cmd | Command for starting the benchmark script. If this parameter is not configured, the configuration in the benchmark drive script is used. | - | +| benchmark_period | This parameter is valid only for **period benchmark**. It indicates the test period of the entire benchmark. The unit is second. | - | +| scenario | Type of the workload specified by the user. | tp, ap, htap | +| tuning_list | List of parameters to be tuned. For details, see the **share/knobs.json.template** file. | - | diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..df96e0c7367c2bb6ec000afe1d48bfb28d021f51 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/1-x-tuner-parameter-optimization-and-diagnosis/1-6-Troubleshooting.md @@ -0,0 +1,14 @@ +--- +title: Troubleshooting +summary: Troubleshooting +author: Guo Huan +date: 2021-05-19 +--- + +# Troubleshooting + +- Failure of connection to the database instance: Check whether the database instance is faulty or the security permissions of configuration items in the **pg_hba.conf** file are incorrectly configured. +- Restart failure: Check the health status of the database instance and ensure that the database instance is running properly. +- Dependency installation failure: Upgrade the pip package management tool by running the **python -m pip install -upgrade pip** command. +- Poor performance of TPC-C jobs: In high-concurrency scenarios such as TPC-C, a large amount of data is modified during pressure tests. Each test is not idempotent, for example, the data volume in the TPC-C database increases, invalid tuples are not cleared using VACUUM FULL, checkpoints are not triggered in the database, and drop cache is not performed. Therefore, it is recommended that the benchmark data that is written with a large amount of data, such as TPC-C, be imported again at intervals (depending on the number of concurrent tasks and execution duration). A simple method is to back up the $PGDATA directory. +- When the TPC-C job is running, the TPC-C driver script reports the error "TypeError: float() argument must be a string or a number, not 'NoneType'" (**none** cannot be converted to the float type). This is because the TPC-C pressure test result is not obtained. There are many causes for this problem, manually check whether TPC-C can be successfully executed and whether the returned result can be obtained. If the preceding problem does not occur, you are advised to set the delay time of the **sleep** command in the command list in the TPC-C driver script to a larger value. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md new file mode 100644 index 0000000000000000000000000000000000000000..ea47de36d76b6a67f064224df74ff0668de2e638 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-1-single-query-index-recommendation.md @@ -0,0 +1,59 @@ +--- +title: Single-query Index Recommendation +summary: Single-query Index Recommendation +author: Guo Huan +date: 2021-05-19 +--- + +# Single-query Index Recommendation + +The single-query index recommendation function allows users to directly perform operations in the database. This function generates recommended indexes for a single query statement entered by users based on the semantic information of the query statement and the statistics of the database. This function involves the following interfaces: + +**Table 1** Single-query index recommendation interfaces + +| Function Name | Parameter | Description | +| :-------------- | :------------------- | :----------------------------------------------------------- | +| gs_index_advise | SQL statement string | Generates a recommendation index for a single query statement. | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - This function supports only a single SELECT statement and does not support other types of SQL statements. +> - Partitioned tables, column-store tables, segment-paged tables, common views, materialized views, global temporary tables, and encrypted databases are not supported. + +## Application Scenarios + +Use the preceding function to obtain the recommendation index generated for the query. The recommendation result consists of the table name and column name of the index. + +For example: + +```sql +mogdb=> select "table", "column" from gs_index_advise('SELECT c_discount from bmsql_customer where c_w_id = 10'); + table | column +----------------+---------- + bmsql_customer | (c_w_id) +(1 row) +``` + +The preceding information indicates that an index should be created on the **c_w_id** column of the **bmsql_customer** table. You can run the following SQL statement to create an index: + +```sql +CREATE INDEX idx on bmsql_customer(c_w_id); +``` + +Some SQL statements may also be recommended to create a join index, for example: + +```sql +mogdb=# select "table", "column" from gs_index_advise('select name, age, sex from t1 where age >= 18 and age < 35 and sex = ''f'';'); + table | column +-------+------------ + t1 | (age, sex) +(1 row) +``` + +The preceding statement indicates that a join index **(age, sex)** needs to be created in the **t1** table. You can run the following command to create a join index: + +```sql +CREATE INDEX idx1 on t1(age, sex); +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Parameters of the system function **gs_index_advise()** are of the text type. If the parameters contain special characters such as single quotation marks ('), you can use single quotation marks (') to escape the special characters. For details, see the preceding example. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md new file mode 100644 index 0000000000000000000000000000000000000000..4d6cd4168804fde4c3b17cedc26d968087243ba0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-2-virtual-index.md @@ -0,0 +1,123 @@ +--- +title: Virtual Index +summary: Virtual Index +author: Guo Huan +date: 2021-05-19 +--- + +# Virtual Index + +The virtual index function allows users to directly perform operations in the database. This function simulates the creation of a real index to avoid the time and space overhead required for creating a real index. Based on the virtual index, users can evaluate the impact of the index on the specified query statement by using the optimizer. + +This function involves the following interfaces: + +**Table 1** Virtual index function interfaces + +| Function Name | Parameter | Description | +| :------------------- | :------------------------------------------------------ | :----------------------------------------------------------- | +| hypopg_create_index | Character string of the statement for creating an index | Creates a virtual index. | +| hypopg_display_index | None | Displays information about all created virtual indexes. | +| hypopg_drop_index | OID of the index | Deletes a specified virtual index. | +| hypopg_reset_index | None | Clears all virtual indexes. | +| hypopg_estimate_size | OID of the index | Estimates the space required for creating a specified index. | + +This function involves the following GUC parameters: + +**Table 2** GUC parameters of the virtual index function + +| Parameter | Description | Default Value | +| :---------------- | :-------------------------------------------- | :------------ | +| enable_hypo_index | Whether to enable the virtual index function. | off | + +## Procedure + +1. Use the **hypopg_create_index** function to create a virtual index. For example: + + ```sql + mogdb=> select * from hypopg_create_index('create index on bmsql_customer(c_w_id)'); + indexrelid | indexname + ------------+------------------------------------- + 329726 | <329726>btree_bmsql_customer_c_w_id + (1 row) + ``` + +2. Enable the GUC parameter **enable_hypo_index**. This parameter controls whether the database optimizer considers the created virtual index when executing the EXPLAIN statement. By executing EXPLAIN on a specific query statement, you can evaluate whether the index can improve the execution efficiency of the query statement based on the execution plan provided by the optimizer. For example: + + ```sql + mogdb=> set enable_hypo_index = on; + SET + ``` + + Before enabling the GUC parameter, run **EXPLAIN** and the query statement. + + ```sql + mogdb=> explain SELECT c_discount from bmsql_customer where c_w_id = 10; + QUERY PLAN + ---------------------------------------------------------------------- + Seq Scan on bmsql_customer (cost=0.00..52963.06 rows=31224 width=4) + Filter: (c_w_id = 10) + (2 rows) + ``` + + After enabling the GUC parameter, run **EXPLAIN** and the query statement. + + ```sql + mogdb=> explain SELECT c_discount from bmsql_customer where c_w_id = 10; + QUERY PLAN + ------------------------------------------------------------------------------------------------------------------ + [Bypass] + Index Scan using <329726>btree_bmsql_customer_c_w_id on bmsql_customer (cost=0.00..39678.69 rows=31224 width=4) + Index Cond: (c_w_id = 10) + (3 rows) + ``` + + By comparing the two execution plans, you can find that the index may reduce the execution cost of the specified query statement. Then, you can consider creating a real index. + +3. (Optional) Use the **hypopg_display_index** function to display all created virtual indexes. For example: + + ```sql + mogdb=> select * from hypopg_display_index(); + indexname | indexrelid | table | column + --------------------------------------------+------------+----------------+------------------ + <329726>btree_bmsql_customer_c_w_id | 329726 | bmsql_customer | (c_w_id) + <329729>btree_bmsql_customer_c_d_id_c_w_id | 329729 | bmsql_customer | (c_d_id, c_w_id) + (2 rows) + ``` + +4. (Optional) Use the **hypopg_estimate_size** function to estimate the space (in bytes) required for creating a virtual index. For example: + + ```sql + mogdb=> select * from hypopg_estimate_size(329730); + hypopg_estimate_size + ---------------------- + 15687680 + (1 row) + ``` + +5. Delete the virtual index. + + Use the **hypopg_drop_index** function to delete the virtual index of a specified OID. For example: + + ```sql + mogdb=> select * from hypopg_drop_index(329726); + hypopg_drop_index + ------------------- + t + (1 row) + ``` + + Use the **hypopg_reset_index** function to clear all created virtual indexes at a time. For example: + + ```sql + mogdb=> select * from hypopg_reset_index(); + hypopg_reset_index + -------------------- + + (1 row) + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - Running **EXPLAIN ANALYZE** does not involve the virtual index function. +> - The created virtual index is at the database instance level and can be shared by sessions. After a session is closed, the virtual index still exists. However, the virtual index will be cleared after the database is restarted. +> - This function does not support common views, materialized views, and column-store tables. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md new file mode 100644 index 0000000000000000000000000000000000000000..409d1c1643699a00f5cef4b611b98d4d8c5e07cd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/2-index-advisor-index-recommendation/2-3-workload-level-index-recommendation.md @@ -0,0 +1,105 @@ +--- +title: Workload-level Index Recommendation +summary: Workload-level Index Recommendation +author: Guo Huan +date: 2021-05-19 +--- + +# Workload-level Index Recommendation + +For workload-level indexes, you can run scripts outside the database to use this function. This function uses the workload of multiple DML statements as the input to generate a batch of indexes that can optimize the overall workload execution performance. In addition, it provides the function of extracting service data SQL statements from logs. + +## Prerequisites + +- The database is normal, and the client can be connected properly. + +- The **gsql** tool has been installed by the current user, and the tool path has been added to the **PATH** environment variable. + +- The Python 3.6+ environment is available. + +- To use the service data extraction function, you need to set the GUC parameters of the node whose data is to be collected as follows: + + - log_min_duration_statement = 0 + + - log_statement= 'all' + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** After service data extraction is complete, you are advised to restore the preceding GUC parameters. Otherwise, log files may be expanded. + +## Procedure for Using the Service Data Extraction Script + +1. Set the GUC parameters according to instructions in the prerequisites. + +2. Run the Python script **extract_log.py**: + + ``` + ython extract_log.py [l LOG_DIRECTORY] [f OUTPUT_FILE] [-d DATABASE] [-U USERNAME][--start_time] [--sql_amount] [--statement] [--json] + ``` + + The input parameters are as follows: + + - **LOG_DIRECTORY**: directory for storing **pg_log**. + - **OUTPUT_PATH**: path for storing the output SQL statements, that is, path for storing the extracted service data. + - **DATABASE** (optional): database name. If this parameter is not specified, all databases are selected by default. + - **USERNAME** (optional): username. If this parameter is not specified, all users are selected by default. + - **start_time** (optional): start time for log collection. If this parameter is not specified, all files are collected by default. + - **sql_amount** (optional): maximum number of SQL statements to be collected. If this parameter is not specified, all SQL statements are collected by default. + - **statement** (optional): Collects the SQL statements starting with **statement** in **pg_log log**. If this parameter is not specified, the SQL statements are not collected by default. + - **json**: Specifies that the collected log files are stored in JSON format after SQL normalization. If the default format is not specified, each SQL statement occupies a line. + + An example is provided as follows. + + ``` + python extract_log.py $GAUSSLOG/pg_log/dn_6001 sql_log.txt -d postgres -U omm --start_time '2021-07-06 00:00:00' --statement + ``` + +3. Change the GUC parameter values set in step 1 to the values before the setting. + +## Procedure for Using the Index Recommendation Script + +1. Prepare a file that contains multiple DML statements as the input workload. Each statement in the file occupies a line. You can obtain historical service statements from the offline logs of the database. + +2. Run the Python script **index_advisor_workload.py**: + + ``` + python index_advisor_workload.py [p PORT] [d DATABASE] [f FILE] [--h HOST] [-U USERNAME] [-W PASSWORD][--schema SCHEMA][--max_index_num MAX_INDEX_NUM][--max_index_storage MAX_INDEX_STORAGE] [--multi_iter_mode] [--multi_node] [--json] [--driver] [--show_detail] + ``` + + The input parameters are as follows: + + - **PORT**: port number of the connected database. + - **DATABASE**: name of the connected database. + - **FILE**: file path that contains the workload statement. + - **HOST** (optional): ID of the host that connects to the database. + - **USERNAME** (optional): username for connecting to the database. + - **PASSWORD** (optional): password for connecting to the database. + - **SCHEMA**: schema name. + - **MAX_INDEX_NUM** (optional): maximum number of recommended indexes. + - **MAX_INDEX_STORAGE** (optional): maximum size of the index set space. + - **multi_node** (optional): specifies whether the current instance is a distributed database instance. + - **multi_iter_mode** (optional): algorithm mode. You can switch the algorithm mode by setting this parameter. + - **json** (optional): Specifies the file path format of the workload statement as JSON after SQL normalization. By default, each SQL statement occupies one line. + - **driver** (optional): Specifies whether to use the Python driver to connect to the database. By default, **gsql** is used for the connection. + - **show_detail** (optional): Specifies whether to display the detailed optimization information about the current recommended index set. + + Example: + + ``` + python index_advisor_workload.py 6001 postgres tpcc_log.txt --schema public --max_index_num 10 --multi_iter_mode + ``` + + The recommendation result is a batch of indexes, which are displayed on the screen in the format of multiple create index statements. The following is an example of the result. + + ```sql + create index ind0 on public.bmsql_stock(s_i_id,s_w_id); + create index ind1 on public.bmsql_customer(c_w_id,c_id,c_d_id); + create index ind2 on public.bmsql_order_line(ol_w_id,ol_o_id,ol_d_id); + create index ind3 on public.bmsql_item(i_id); + create index ind4 on public.bmsql_oorder(o_w_id,o_id,o_d_id); + create index ind5 on public.bmsql_new_order(no_w_id,no_d_id,no_o_id); + create index ind6 on public.bmsql_customer(c_w_id,c_d_id,c_last,c_first); + create index ind7 on public.bmsql_new_order(no_w_id); + create index ind8 on public.bmsql_oorder(o_w_id,o_c_id,o_d_id); + create index ind9 on public.bmsql_district(d_w_id); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The value of the **multi_node** parameter must be specified based on the current database architecture. Otherwise, the recommendation result is incomplete, or even no recommendation result is generated. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-1-overview.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-1-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..c848869159738f945f9ffe7a9c4b0b913a9329ee --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-1-overview.md @@ -0,0 +1,10 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2022-05-06 +--- + +# Overview + +Slow SQL statements have always been a pain point in data O&M. How to effectively diagnose the root causes of slow SQL statements is a big challenge. Based on the characteristics of MogDB and the slow SQL diagnosis experience of DBAs on the live network, this tool supports more than 15 root causes of slow SQL statements, outputs multiple root causes based on the possibility, and provides specific solutions. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..c9c04efc29ad425e9c0234440252d7ad0670a75e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-2-environment-deployment.md @@ -0,0 +1,11 @@ +--- +title: Environment Deployment +summary: Environment Deployment +author: Guo Huan +date: 2022-05-06 +--- + +# Environment Deployment + +- The database is working properly. +- The metric collection system is running properly. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..7eff71a5aef9d04ff3926432de0b95ef6db6f36b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-3-usage-guide.md @@ -0,0 +1,34 @@ +--- +title: Usage Guide +summary: Usage Guide +author: Guo Huan +date: 2022-05-06 +--- + +# Usage Guide + +Assume that the **confpath** configuration file directory has been initialized. + +- Run the following command to start only the slow SQL diagnosis function and output the top 3 root causes (for details, see the description of the **service** subcommand): + + ``` + gs_dbmind service start -c confpath --only-run slow_query_diagnosis + ``` + +- Run the following command to diagnose slow SQL statements in interactive mode: + + ``` + gs_dbmind component slow_query_diagnosis show -c confpath --query SQL --start-time timestamps0 --end-time timestamps1 + ``` + +- Run the following command to manually clear historical prediction results: + + ``` + gs_dbmind component slow_query_diagnosis clean -c confpath --retention-days DAYS + ``` + +- Run the following command to stop the services that have been started: + + ``` + gs_dbmind service stop -c confpath + ``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..fb76fac7621780933f175c5bbaffeb82740ebcf3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-4-obtaining-help-information.md @@ -0,0 +1,41 @@ +--- +title: Obtaining Help Information +summary: Obtaining Help Information +author: Guo Huan +date: 2022-05-06 +--- + +# Obtaining Help Information + +You can run the **–help** command to obtain the help information. For example: + +``` +gs_dbmind component slow_query_diagnosis --help +``` + +``` +usage: [-h] -c DIRECTORY [--query SLOW_QUERY] + [--start-time TIMESTAMP_IN_MICROSECONDS] + [--end-time TIMESTAMP_IN_MICROSECONDS] [--retention-days DAYS] + {show,clean} + +Slow Query Diagnosis: Analyse the root cause of slow query + +positional arguments: + {show,clean} choose a functionality to perform + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files + --query SLOW_QUERY set a slow query you want to retrieve + --start-time TIMESTAMP_IN_MICROSECONDS + set the start time of a slow SQL diagnosis result to + be retrieved + --end-time TIMESTAMP_IN_MICROSECONDS + set the end time of a slow SQL diagnosis result to be + retrieved + --retention-days DAYS + clear historical diagnosis results and set the maximum + number of days to retain data +``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..402f2b97b95b8f24b8815e64f847a0e29b0eb28d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-5-command-reference.md @@ -0,0 +1,20 @@ +--- +title: Command Reference +summary: Command Reference +author: Guo Huan +date: 2022-05-06 +--- + +# Command Reference + +**Table 1** gs_dbmind component slow_query_diagnosis parameters + +| Parameter | Description | Value Range | +| :--------------- | :-------------------------------------------- | :----------------------------------------------------- | +| -h, --help | Help information | - | +| action | Action parameter | - show: displays results.
- clean: clears results. | +| -c，--conf | Configuration directory | - | +| --query | Slow SQL text | * | +| --start-time | Timestamp of the start time, in milliseconds. | Non-negative integer | +| --end-time | Timestamp of the end time, in milliseconds. | Non-negative integer | +| --retention-days | Number of days retaining results | Non-negative real number | \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..407c339f03759e1027329aa3b034af088ab06ac1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/3-slow-query-diagnosis-root-cause-analysis-for-slow-sql-statements/3-6-troubleshooting.md @@ -0,0 +1,12 @@ +--- +title: Troubleshooting +summary: Troubleshooting +author: Guo Huan +date: 2022-05-06 +--- + +# Troubleshooting + +- If you run the interactive diagnosis command for a slow SQL statement that has not been executed, no diagnosis result is provided. +- If the exporter metric collection function is not enabled, the slow SQL diagnosis function is not available. +- After the parameters in the configuration file are reset, you need to restart the service process for the settings to take effect. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-1-overview.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-1-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..33df1fa40e3f83dc181222683c8304ec1026fb7a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-1-overview.md @@ -0,0 +1,10 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2022-05-06 +--- + +# Overview + +The trend prediction module predicts the future time series change trend based on historical time series data. The framework of this module has been decoupled to flexibly change prediction algorithms. This module can automatically select algorithms for different feature time series. The LR regression algorithm for linear feature time series prediction and the ARIMA algorithm for non-linear feature prediction are supported. At present, this module can cover the accurate prediction of linear time series, non-linear time series and periodic time series. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-2-environment-deployment.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-2-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..e370be78bafde7140d95a7e9ec3ffb77de4d6a94 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-2-environment-deployment.md @@ -0,0 +1,10 @@ +--- +title: Environment Deployment +summary: Environment Deployment +author: Guo Huan +date: 2022-05-06 +--- + +# Environment Deployment + +The metric collection system is running properly. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-3-usage-guide.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-3-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..ea8a1fb9107378f39cf2c8c427c98666ae08d373 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-3-usage-guide.md @@ -0,0 +1,34 @@ +--- +title: Usage Guide +summary: Usage Guide +author: Guo Huan +date: 2022-05-06 +--- + +# Usage Guide + +Assume that the **confpath** configuration file directory has been initialized. + +- To enable only the trend prediction function, run the following command (for details, see the description of the **service** subcommand): + + ``` + gs_dbmind service start -c confpath --only-run forecast + ``` + +- To enable trend prediction in interactive mode, run the following command and view the prediction results from **timestamps0** to **timestamps1**: + + ``` + gs_dbmind component forecast show -c confpath --start-time timestamps0 --end-time timestamps1 + ``` + +- Run the following command to manually clear historical prediction results: + + ``` + gs_dbmind component forecast clean -c confpath --retention-days DAYS + ``` + +- Run the following command to stop the services that have been started: + + ``` + gs_dbmind service stop -c confpath + ``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-4-obtaining-help-information.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-4-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..cfb64369f6f062cd519b78da1688f2fa4073a53b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-4-obtaining-help-information.md @@ -0,0 +1,38 @@ +--- +title: Obtaining Help Information +summary: Obtaining Help Information +author: Guo Huan +date: 2022-05-06 +--- + +# Obtaining Help Information + +You can run the **–help** command to obtain the help information. For example: + +``` +gs_dbmind component forecast --help +``` + +``` +usage: [-h] -c DIRECTORY [--metric-name METRIC_NAME] [--host HOST] [--start-time TIMESTAMP_IN_MICROSECONDS] [--end-time TIMESTAMP_IN_MICROSECONDS] [--retention-days DAYS] + {show,clean} + +Workload Forecasting: Forecast monitoring metrics + +positional arguments: + {show,clean} choose a functionality to perform + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files + --metric-name METRIC_NAME + set a metric name you want to retrieve + --host HOST set a host you want to retrieve + --start-time TIMESTAMP_IN_MICROSECONDS + set a start time of for retrieving + --end-time TIMESTAMP_IN_MICROSECONDS + set a end time of for retrieving + --retention-days DAYS + clear historical diagnosis results and set the maximum number of days to retain data +``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-5-command-reference.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-5-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..59e2e9da54b6a2d6cb80cea850a09c8a8ee065ad --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-5-command-reference.md @@ -0,0 +1,21 @@ +--- +title: Command Reference +summary: Command Reference +author: Guo Huan +date: 2022-05-06 +--- + +# Command Reference + +**Table 1** gs_dbmind component forecast parameters + +| Parameter | Description | Value Range | +| :-------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | +| -h, –help | Help information | - | +| action | Action parameter | - **show**: displays results.
- **clean**: clears results. | +| -c, –conf | Configuration directory | - | +| –metric-name | Specifies the metric name to be displayed, which is used for filtering. | - | +| –host | Specifies the service IP address and port number, which is used for filtering. | - | +| –start-time | Timestamp of the start time, in milliseconds. | Non-negative real number | +| –end-time | Timestamp of the end time, in milliseconds. | Non-negative real number | +| –retention-days | Number of days retaining results | Non-negative real number | diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-6-troubleshooting.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-6-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..46cf947bc059d7181b2f2921c8611c7368d95d05 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/4-forcast-trend-prediction/4-6-troubleshooting.md @@ -0,0 +1,11 @@ +--- +title: Troubleshooting +summary: Troubleshooting +author: Guo Huan +date: 2022-05-06 +--- + +# Troubleshooting + +- Considering the actual service and model prediction effect, you are advised to set the trend prediction duration to a value greater than 3600 seconds. (If the metric collection period is 15 seconds, the number of data records collected is 240.) Otherwise, the prediction effect will deteriorate, and the service will be abnormal when the data volume is extremely small. The default value is 3600 seconds. +- After the parameters in the configuration file are reset, you need to restart the service process for the settings to take effect. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..4a79133b3b0a30fba47083e66e6e02de9d13ab26 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-1-overview.md @@ -0,0 +1,16 @@ +--- +title: SQLdiag Slow SQL Discovery +summary: SQLdiag Slow SQL Discovery +author: Guo Huan +date: 2021-05-19 +--- + +# SQLdiag Slow SQL Discovery + +SQLdiag is a framework for predicting the execution duration of SQL statements in MogDB. The existing prediction technologies are mainly based on model prediction of execution plans. These prediction solutions are applicable only to jobs whose execution plans can be obtained in the OLAP scenarios, and are not useful for quick query such as OLTP or HTAP. Different from the preceding solutions, SQLdiag focuses on the historical SQL statements of databases. Because the execution duration of the database SQL statements in a short time does not vary greatly, SQLdiag can detect instruction sets similar to the entered instructions from the historical data, and predict the SQL statement execution duration based on the SQL vectorization technology and the time series prediction algorithm. This framework has the following benefits: + +1. Execution plans do not require instructions. This has no impact on database performance. +2. The framework is widely used, unlike many other well-targeted algorithms in the industry, for example, they may applicable only to OLTP or OLAP. +3. The framework is robust and easy to understand. Users can design their own prediction models by simply modifying the framework. + +SQLdiag is an SQL statement execution time prediction tool. It predicts the execution time of SQL statements based on the statement logic similarity and historical execution records without obtaining the SQL statement execution plan using a template or deep learning. Abnormal SQL statements can also be detected with this tool. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..743365946d2570b01d7fd5ce252e881b8af45454 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-2-usage-guide.md @@ -0,0 +1,96 @@ +--- +title: Usage Guide +summary: Usage Guide +author: Guo Huan +date: 2021-10-21 +--- + +# Usage Guide + +## Prerequisites + +- You have obtained training data. +- If you use the provided tool to collect training data, you need to enable the WDR function. The involved parameters are **track_stmt_stat_level** and **log_min_duration_statement**. For details, see the following sections. +- To ensure the prediction accuracy, the historical statement logs provided by users should be as comprehensive and representative as possible. + +## Collecting SQL Statements + +This tool requires users to prepare data in advance. Each sample is separated by a newline character. The training data format is as follows: + +``` +SQL,EXECUTION_TIME +``` + +The prediction data format is as follows: + +``` +SQL +``` + +**SQL** indicates the text of an SQL statement, and **EXECUTION_TIME** indicates the execution time of the SQL statement. For details about the sample data, see **train.csv** and **predict.csv** in **sample_data**. + +You can collect training data in the required format. The tool also provides the **load_sql_from_rd** script for automatic collection. The script obtains SQL information based on the WDR report. The involved parameters are **log_min_duration_statement** and **track_stmt_stat_level**: + +- **log_min_duration_statement** indicates the slow SQL threshold. If the value is **0**, full collection is performed. The unit is millisecond. +- **track_stmt_stat_level** indicates the information capture level. You are advised to set it to **'L0,L0'**. + +After this parameter is set, a certain amount of system resources may be occupied but the usage is generally low. In continuous high-concurrency scenarios, this may cause a performance loss less than 5%. If the database concurrency is low, the performance loss can be ignored. The following script is stored in the sqldiag root directory (*$GAUSSHOME***/bin/components/sqldiag**). + +``` +Use a script to obtain the training set: +load_sql_from_wdr.py [-h] --port PORT --start_time START_TIME + --finish_time FINISH_TIME [--save_path SAVE_PATH] +Example: + python load_sql_from_wdr.py --start_time "2021-04-25 00:00:00" --finish_time "2021-04-26 14:00:00" --port 5432 --save_path ./data.csv +``` + +## Procedure + +1. Provide historical logs for model training. + +2. Perform training and prediction. + + ``` + Template-based training and prediction: + gs_dbmind component sqldiag [train, predict] -f FILE --model template --model-path template_model_path + DNN-based training and prediction: + gs_dbmind component sqldiag [train, predict] -f FILE --model dnn --model-path dnn_model_path + ``` + +## Examples + +Use the provided test data to perform template-based training: + +``` +gs_dbmind component sqldiag train -f ./sample_data/train.csv --model template --model-path ./template +``` + +Use the provided test data for template-based prediction: + +``` +gs_dbmind component sqldiag predict -f ./sample_data/predict.csv --model template --model-path ./template --predicted-file ./result/t_result +``` + +Use the provided test data to update the template-based model: + +``` +gs_dbmind component sqldiag finetune -f ./sample_data/train.csv --model template --model-path ./template +``` + +Use the provided test data to perform DNN-based training: + +``` +gs_dbmind component sqldiag train -f ./sample_data/train.csv --model dnn --model-path ./dnn_model +``` + +Use the provided test data for DNN-based prediction: + +``` +gs_dbmind component sqldiag predict -f ./sample_data/predict.csv --model dnn --model-path ./dnn_model --predicted-file +``` + +Use the provided test data to update the DNN-based model: + +``` +gs_dbmind component sqldiag finetune -f ./sample_data/train.csv --model dnn --model-path ./dnn_model +``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..b863b043fe8298c7f4c2b8cc12657e162761db7f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-3-obtaining-help-information.md @@ -0,0 +1,49 @@ +--- +title: Obtaining Help Information +summary: Obtaining Help Information +author: Guo Huan +date: 2021-05-19 +--- + +# Obtaining Help Information + +Before using the SQLdiag tool, run the following command to obtain help information: + +``` +gs_dbmind component sqldiag --help +``` + +The command output is as follows: + +``` +usage: [-h] [-f CSV_FILE] [--predicted-file PREDICTED_FILE] + [--model {template,dnn}] --model-path MODEL_PATH + [--config-file CONFIG_FILE] + {train,predict,finetune} + +SQLdiag integrated by MogDB. + +positional arguments: + {train,predict,finetune} + The training mode is to perform feature extraction and + model training based on historical SQL statements. The + prediction mode is to predict the execution time of a + new SQL statement through the trained model. + +optional arguments: + -h, --help show this help message and exit + -f CSV_FILE, --csv-file CSV_FILE + The data set for training or prediction. The file + format is CSV. If it is two columns, the format is + (SQL statement, duration time). If it is three + columns, the format is (timestamp of SQL statement + execution time, SQL statement, duration time). + --predicted-file PREDICTED_FILE + The file path to save the predicted result. + --model {template,dnn} + Choose the model model to use. + --model-path MODEL_PATH + The storage path of the model file, used to read or + save the model file. + --config-file CONFIG_FILE +``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..533a05e9ee1a9edb9cda01d90007c6f3bb517faf --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-4-command-reference.md @@ -0,0 +1,17 @@ +--- +title: Command Reference +summary: Command Reference +author: Guo Huan +date: 2021-05-19 +--- + +# Command Reference + +**Table 1** Command-line options + +| Parameter | Description | Value Range | +| :-------------- | :----------------------------------- | :------------ | +| -f | Training or prediction file location | N/A | +| –predicted-file | Prediction result location | N/A | +| –model | Model selection | template, dnn | +| –model-path | Location of the training model | N/A | diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..442f8d1d07449da92ea7ca5ac3d6c860a438ac6b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai-sub-functions-of-the-dbmind/5-sqldiag-slow-sql-discovery/5-5-troubleshooting.md @@ -0,0 +1,11 @@ +--- +title: Troubleshooting +summary: Troubleshooting +author: Guo Huan +date: 2021-05-19 +--- + +# Troubleshooting + +- Failure in the training scenario: Check whether the file path of historical logs is correct and whether the file format meets the requirements. +- Failure in the prediction scenario: Check whether the model path is correct. Ensure that the format of the load file to be predicted is correct. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai4db-autonomous-database-o&m.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai4db-autonomous-database-o&m.md new file mode 100644 index 0000000000000000000000000000000000000000..7a1c0759b1b0da14c7bfb7f406dcb75c1924b238 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/ai4db-autonomous-database-o&m.md @@ -0,0 +1,34 @@ +--- +title: AI4DB Autonomous Database O&M +summary: AI4DB Autonomous Database O&M +author: Guo Huan +date: 2022-05-06 +--- + +# AI4DB Autonomous Database O&M + +As mentioned above, AI4DB is mainly used for autonomous O&M and management of databases, helping database O&M personnel reduce O&M workload. In implementation, the AI4DB framework of DBMind is monitored and service-oriented. It also provides instant AI toolkits and out-of-the-box AI O&M functions (such as index recommendation). AI4DB mainly uses the open-source Prometheus for monitoring. DBMind provides a monitoring data producer exporter, which can be interconnected with the Prometheus platform. The following figure shows the AI4DB service architecture of DBMind. + +**Figure 1** AI4DB service architecture of DBMind + +![DBMind-AI4DB服务架构](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/AI4DB-1.png) + +Key components in the figure are described as follows: + +- DBMind Service: DBMind background service, which can be used for periodic offline computing, including slow SQL root cause analysis and time series prediction. +- Prometheus-server: server for storing Prometheus monitoring metrics. +- metadatabase: After the offline computing is complete, DBMind stores the computing result. Databases such as MogDB and SQLite are supported. +- client: client used to read the DBMind offline computing results. Currently, only the CLI client is supported. If databases such as MogDB are used to store and calculate the DBMind computing results, you can configure visualization tools such as Grafana to visualize the results. +- openGauss-exporter: collects monitoring metrics from the MogDB database nodes for DBMind to calculate. +- node-exporter: exporter provided by Prometheus, which can be used to monitor system metrics of the node, such as CPU and memory usage. +- reprocessing-exporter: processes metrics collected by Prometheus, for example, calculating the CPU usage. + +## Environment Configuration + +DBMind must run on Python 3.6 or later. The required third-party dependency packages are recorded in the **requirements.txt** file (**requirements-x86.txt** or **requirements-arrch64.txt**, depending on the platform type) in the AI function root directory (*$GAUSSHOME***/bin/dbmind**). You can run the **pip install** command to install the dependencies. For example: + +``` +pip install requirements-x86.txt +``` + +If you do not install all required dependencies, the system will prompt you to install third-party dependencies when you run the **gs_dbmind** command. Note that this file provides the third-party dependencies required by DBMind. If a third-party package conflict exists in the user environment, you can handle the problem based on the actual situation. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md new file mode 100644 index 0000000000000000000000000000000000000000..88b4b69ef04099d24c9f334f82b7b2670ca5b588 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/components-that-support-dbmind.md @@ -0,0 +1,10 @@ +--- +title: Components that Support DBMind +summary: Components that Support DBMind +author: Guo Huan +date: 2022-05-06 +--- + +# Components that Support DBMind + +DBMind provides components to support the deployment and implementation of the entire service or solution. They are not AI functions, but are an important part of the entire service system. They are used to support the quick implementation of the entire autonomous O&M solution. For example, the exporter is used to collect database metrics. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..5a4d3bd8a801bbf15729268557ecb4f7991778d5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-command-reference.md @@ -0,0 +1,136 @@ +--- +title: Command Reference +summary: Command Reference +author: Guo Huan +date: 2022-05-06 +--- + +# Command Reference + +For details about how to use reprocessing-exporter, see the following help information: + +``` +gs_dbmind component reprocessing_exporter --help +usage: [-h] [--disable-https] [--ssl-keyfile SSL_KEYFILE] + [--ssl-certfile SSL_CERTFILE] + [--web.listen-address WEB.LISTEN_ADDRESS] + [--web.listen-port WEB.LISTEN_PORT] + [--collector.config COLLECTOR.CONFIG] [--log.filepath LOG.FILEPATH] + [--log.level {debug,info,warn,error,fatal}] [--version] + prometheus_host prometheus_port + +Reprocessing Exporter: A re-processing module for metrics stored in the +Prometheus server. + +positional arguments: + prometheus_host from which host to pull data + prometheus_port the port to connect to the Prometheus host + +optional arguments: + -h, --help show this help message and exit + --disable-https disable Https schema + --ssl-keyfile SSL_KEYFILE + set the path of ssl key file + --ssl-certfile SSL_CERTFILE + set the path of ssl certificate file + --web.listen-address WEB.LISTEN_ADDRESS + address on which to expose metrics and web interface + --web.listen-port WEB.LISTEN_PORT + listen port to expose metrics and web interface + --collector.config COLLECTOR.CONFIG + according to the content of the yaml file for metric + collection + --log.filepath LOG.FILEPATH + the path to log + --log.level {debug,info,warn,error,fatal} + only log messages with the given severity or above. + Valid levels: [debug, info, warn, error, fatal] + --version show program's version number and exit +``` + +**Table 1** reprocessing-exporter parameters + +| Parameter | Description | Value Range | +| :------------------ | :----------------------------------------------------------- | :------------------------------ | +| prometheus_host | Prometheus-server IP address | - | +| prometheus_port | Listening port of Prometheus-server | 1024-65535 | +| -h, –help | Help option | - | +| –disable-https | Disables HTTPS. | - | +| –ssl-keyfile | Path of the HTTPS certificate private key file | - | +| –ssl-certfile | Path of the HTTPS certificate file | - | +| –web.listen-address | IP address bound to the exporter service | - | +| –web.listen-port | Listening port of the exporter service | 1024-65535 | +| –collector.config | Path of the configuration file that explicitly specifies the metrics to be collected | - | +| –log.filepath | Path for storing log files. By default, log files are stored in the current directory. | - | +| –log.level | Printing level of the log file. The default level is **INFO**. | debug, info, warn, error, fatal | +| –version | Displays version information. | - | + +For details about how to use openGauss-exporter, see the following help information: + +``` +gs_dbmind component opengauss_exporter --help +usage: [-h] --url URL [--config CONFIG] [--constant-labels CONSTANT_LABELS] + [--web.listen-address WEB.LISTEN_ADDRESS] + [--web.listen-port WEB.LISTEN_PORT] + [--web.telemetry-path WEB.TELEMETRY_PATH] [--disable-cache] + [--disable-settings-metrics] [--disable-statement-history-metrics] + [--disable-https] [--ssl-keyfile SSL_KEYFILE] + [--ssl-certfile SSL_CERTFILE] [--parallel PARALLEL] + [--log.filepath LOG.FILEPATH] + [--log.level {debug,info,warn,error,fatal}] [--version] + +openGauss Exporter (DBMind): Monitoring for MogDB. + +optional arguments: + -h, --help show this help message and exit + --url URL MogDB database target url. + --config CONFIG path to config dir or file. + --constant-labels CONSTANT_LABELS + a list of label=value separated by comma(,). + --web.listen-address WEB.LISTEN_ADDRESS + address on which to expose metrics and web interface + --web.listen-port WEB.LISTEN_PORT + listen port to expose metrics and web interface + --web.telemetry-path WEB.TELEMETRY_PATH + path under which to expose metrics. + --disable-cache force not using cache. + --disable-settings-metrics + not collect pg_settings.yml metrics. + --disable-statement-history-metrics + not collect statement-history metrics (including slow + queries). + --disable-https disable Https schema + --ssl-keyfile SSL_KEYFILE + set the path of ssl key file + --ssl-certfile SSL_CERTFILE + set the path of ssl certificate file + --parallel PARALLEL not collect pg_settings.yml metrics. + --log.filepath LOG.FILEPATH + the path to log + --log.level {debug,info,warn,error,fatal} + only log messages with the given severity or above. + Valid levels: [debug, info, warn, error, fatal] + --version show program's version number and exit +``` + +**Table 2** openGauss-exporter parameters + +| Parameter | Description | Value Range | +| :--------------------------------- | :----------------------------------------------------------- | :------------------------------ | +| –url | URL of the MogDB server, for example, postgres://user:pwd@host:port/dbname. | - | +| –constant-labels | Constant labels, which are forcibly added to the collected metrics. | 1024-65535 | +| -h, –help | Help option | - | +| –disable-https | Disables HTTPS. | - | +| –ssl-keyfile | Path of the HTTPS certificate private key file | - | +| –ssl-certfile | Path of the HTTPS certificate file | - | +| –web.listen-address | IP address bound to the exporter service | - | +| –web.listen-port | Listening port of the exporter service | 1024-65535 | +| –web.telemetry-path | URI of the exporter for collecting metrics. The default value is **/metrics**. | - | +| –config | Path of the configuration file that explicitly specifies the metrics to be collected | - | +| –log.filepath | Path for storing log files. By default, log files are stored in the current directory. | - | +| –log.level | Printing level of the log file. The default level is **INFO**. | debug, info, warn, error, fatal | +| –version | Displays version information. | - | +| –disable-cache | Disables the cache. | - | +| –disable-settings-metrics | Disables the collection of metrics in the **pg_settings** table. | - | +| –disable-statement-history-metrics | Disables the collection of slow SQL statements in the **statement_history** table. | - | +| –parallel | Size of the database connection pool connected to MogDB. | Positive integer | \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..464288a0a96f313c0c8dd87f717d5a3919509cd6 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-environment-deployment.md @@ -0,0 +1,31 @@ +--- +title: 环境部署 +summary: 环境部署 +author: Guo Huan +date: 2022-05-06 +--- + +# Environment Deployment + +1. Start the exporter process and create a listening port number on the local host. +2. Modify the configuration file **prometheus.yml** on the Promethues server and add the information about the started exporter to the file. For example: + + ``` + scrape_configs: + ... + - job_name: 'opengauss_exporter' + static_configs: + - targets: ['127.0.0.1:9187'] + ... + ``` + +The exporter component uses HTTPS for communication by default. Therefore, you need to provide the SSL certificate and key file by default through **–ssl-keyfile** and **–ssl-certfile**. To disable HTTPS, you can run the **–disable-https** command. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **说明**: +> +> **NOTE:** The default communication protocol used by MogDB is incompatible with PostgreSQL. As a result, the PostgreSQL-based Python driver **psycopg2-binary** installed using the PyPI source cannot connect to the MogDB database. Therefore, you need to compile **psycopg2** or modify GUC parameters for adaptation. You can also download **psycopg2** compiled based on MogDB from the openGauss official website. (Typically, a version compiled for Python 3.6 is provided. User need to check the version number before using it.) +> +>- Download the openGauss Python driver from the official website: +> [https://opengauss.org/zh/download.html](https://opengauss.org/zh/download.html) +> - For details about the adaptation of the Python driver, see the openGauss operation guide at: +> [https://mp.weixin.qq.com/s/2TobUQKtw0N9sBpMZJr6zw](https://mp.weixin.qq.com/s/2TobUQKtw0N9sBpMZJr6zw) diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md new file mode 100644 index 0000000000000000000000000000000000000000..d13ec215707d66c9d08c9a9dbe50c70ac4cdeed8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-obtaining-help-information.md @@ -0,0 +1,15 @@ +--- +title: Obtaining Help Information +summary: Obtaining Help Information +author: Guo Huan +date: 2022-05-06 +--- + +# Obtaining Help Information + +You can run the **–help** command to obtain the help information. For example: + +``` +gs_dbmind component opengauss_exporter --help +gs_dbmind component reprocessing_exporter --help +``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..5e6c48bafd2b5e99182d6514cbf1bbf373c4fad5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-overview.md @@ -0,0 +1,12 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2022-05-06 +--- + +# Overview + +Prometheus is a popular open-source monitoring system in the industry. It is also a time series database. The collector of Prometheus is called exporter, which is used to collect metrics of monitored modules. To interconnect with the Prometheus platform, the AI tool provides two types of exporters: openGauss-exporter for collecting database metrics and reprocessing-exporter for reprocessing the collected metrics. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** Prometheus and exporter are popular monitoring and collection platforms in the industry. They are deployed on the intranet and do not expose interfaces to external systems. They are used only by internal monitoring platforms. Therefore, to enhance the security of the monitoring platform, users or O&M personnel need to configure firewalls to isolate external access and enhance the security of the monitoring platform. By default, the Prometheus platform adopts HTTP and has no security access restrictions. This is because the platform is generally deployed on the intranet and the attack risk is controllable. If you want to enhance security, you can modify the TLS configuration options of Prometheus. However, you are not advised to expose the access interface to external systems. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..372f2a26b5a1fdc63a5a3761448f3d887a4c8662 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-troubleshooting.md @@ -0,0 +1,16 @@ +--- +title: Troubleshooting +summary: Troubleshooting +author: Guo Huan +date: 2022-05-06 +--- + +# Troubleshooting + +1. A message is displayed, prompting the user to specify the **–ssl-keyfile** and **–ssl-certfile** options. + + By default, the exporter uses HTTPS for communication. Therefore, you must specify the path of the certificate and private key files. To use HTTP for communication, explicitly specify the **–disable-https** option to disable HTTPS. + +2. A message is displayed prompting users to enter the PEM pass phrase. + + In HTTPS communication mode, after specifying the path of the certificate and its key file (the key file have been encrypted), you need to enter the password of the encrypted private key file. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..c3c18124d87b15fd8944b43c4474866ab64abbd7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/components-that-support-dbmind/prometheus-exporter/prometheus-exporter-usage-guide.md @@ -0,0 +1,62 @@ +--- +title: Usage Guide +summary: Usage Guide +author: Guo Huan +date: 2022-05-06 +--- + +# Usage Guide + +You can run the **gs_dbmind** command to start an exporter. The following describes how to set up a complete Prometheus monitoring platform. + +1. Deploy openGauss-exporter. Run the following command to start openGauss-exporter, use the default listening port number 9187, set the listening IP address to 192.168.1.100, and disable HTTPS: + + ```bash + gs_dbmind component opengauss_exporter --url postgresql://user:password@ip:port/dbname --web.listen-address 192.168.1.100 --disable-https + ``` + +2. Deploy reprocessing-exporter. Run the following command to start reprocessing-exporter, use the default listening port number 8181, set the listening IP address to 192.168.1.101, the IP address and port number of the Prometheus server to 192.168.1.100 and 9090, and disable HTTPS: + + ```bash + gs_dbmind component reprocessing_exporter 192.168.1.100 9090 --web.listen-address 192.168.1.101 --ssl-keyfile server.key --ssl-certfile server.crt + ``` + +3. Deploy node-exporter. Generally, node-exporter needs to be deployed on the Prometheus monitoring platform to monitor the Linux OS. Some AI functions mentioned in the following sections also depend on node-exporter to collect Linux system metrics. Therefore, you need to deploy node-exporter. node-exporter is a built-in component of Prometheus. You can download it from [https://prometheus.io/download/#node_exporter](https://prometheus.io/download/#node_exporter). For details about how to use node-exporter, visit [https://prometheus.io/docs/guides/node-exporter/#installing-and-running-the-node-exporter](https://prometheus.io/docs/guides/node-exporter/#installing-and-running-the-node-exporter). You can decompress the package and run the node-exporter process. The default port number is 9100. The command for starting the process is as follows: + + ``` + ./node_exporter + ``` + +4. Add the following content to the **prometheus.yml** configuration file to configure Prometheus-server: + + ```yaml + scrape_configs: + ... + - job_name: 'opengauss_exporter' + static_configs: + - targets: ['192.168.1.100:9187'] + - job_name: 'reprocessing_exporter' + scheme: https + tls_config: + ca_file: xxx.crt + static_configs: + - targets: ['127.0.0.1:8181'] + - job_name: 'node_exporter' + static_configs: + - targets: ['127.0.0.1:9100'] + ... + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** +> +> - To connect to a database from openGauss-exporter, you must have the **monitor admin** permission or higher. Otherwise, some metrics cannot be collected. +> - openGauss-exporter samples slow SQL information from the **dbe_perf.statement_history** view. The slow SQL records in **dbe_perf.statement_history** are related to **log_min_duration_statement** and **track_stmnt_stat_level**. **log_min_duration_statement** indicates the slow SQL threshold, in milliseconds. You can set it to a specific value as required. **track_stmnt_stat_level** indicates the SQL record level. The default value is **'OFF,L0'**, indicating that only slow SQL statements are recorded. The level is L0. Exercise caution when modifying this parameter. +> - openGauss-exporter collects database information, including data in some system catalogs and views (for details, see the opengauss_exporter configuration file), node-exporter collects system metrics, mainly related to system disks and CPUs, and reprocessing_exporter performs secondary processing based on some metrics in prometheus-server (for details, see the reprocessing_exporter configuration file). Then, the processed data is provided to users. +> - Prometheus-server has a timeout mechanism when pulling exporter data. The timeout interval is controlled by **scrape_timeout** (10s by default). Therefore, when the exporter collects a large amount of data, you can increase the value of **scrape_timeout** as required to prevent timeout errors. Note that the value of **scrape_interval** cannot be smaller than that of **scrape_timeout**. Otherwise, an exception occurs. +> - Ensure that the time zone of the database is consistent with that of the system; otherwise, the time of time-related metrics may be inconsistent with the system time. +> - When HTTPS is used for communication, the tool checks the permission on the certificate and key file and the validity period of the certificate. If the file permission is greater than 600, an alarm is generated. If the certificate will expire within 90 days, an alarm is generated. +> - When metrics are repeatedly collected, an exception occurs on openGauss-exporter, and the exception information is recorded in logs. +> - When setting the **–config**, **–disable-settings-metrics** and **–disable-statement-history-metrics** parameters of openGauss-exporter, please note that: +> 1. If you do not specify any of them, the tool collects metrics in the three configuration files in the **yamls** directory at the same time. +> 2. If you explicitly specify **–config**, the tool does not collect metrics in the **default.yml** file in the **yamls** directory, but collects metrics in the specified configuration file. In addition, metrics in the **pg_settings.yml** and **statements.yml** files can be properly collected. In this case, ensure that the metrics in the specified configuration file and those in **pg_settings.yml** and **statements.yml** are not repeatedly collected. +> 3. If you explicitly specify **–disable-settings-metrics**, the tool does not collect metrics in **pg_settings.yml** in the **yamls** directory. If you explicitly specify **–disable-statement-history-metrics**, the tool does not collect metrics in **statements.yml** (related to slow SQL) in the **yamls** directory. diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/1-service.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/1-service.md new file mode 100644 index 0000000000000000000000000000000000000000..1bb4a7ba71d73d3af278308934ee74c36e97b6c0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/1-service.md @@ -0,0 +1,148 @@ +--- +title: service +summary: service +author: Guo Huan +date: 2022-05-06 +--- + +# service + +This subcommand can be used to initialize the configuration directory and start and stop background tasks. + +## Initializing the Configuration Directory + +You can run the **gs_dbmind service setup** subcommand to initialize the configuration directory. This directory may contain the configuration files and logs of the DBMind. Some files in the directory are described as follows: + +- **dbmind.conf**: DBMind parameter configuration file. You can modify it using the **gs_dbmind set** command or a text editor. +- **dynamic_config.db**: DBMind service metadata stored on the local node, including algorithm hyperparameters and monitoring thresholds. This file contains DBMind service metadata and cannot be configured by users. +- **metric_map.conf**: monitoring metric mapping table, which can be used to adapt to different collection platforms. For example, in DBMind, the monitored system CPU usage is named **os_cpu_usage**, but a user-defined collection tool may name the CPU usage **my_cpu_usage_rate**. In this case, if you want DBMind to represent the CPU usage **metric my_cpu_usage_rate**, you need to modify this configuration item. That is, add the **os_cpu_usage = my_cpu_usage_rate** configuration item for mapping. For common users, you are advised to use the collection component and solution of the DBMind. In this case, you do not need to modify the configuration file. +- **logs**: This directory stores logs generated by the DBMind service. + +You can initialize the configuration directory in interactive or non-interactive mode. For example, if the name of the configuration directory to be initialized is **confpath**, perform the following operations: + +**Interactive mode** + +``` +gs_dbmind service setup -c confpath --interactive +``` + +After running the preceding command, you can configure the configuration items in interactive mode through the CLI client. + +**Non-interactive mode** + +In non-interactive mode, the initialization consists of three steps: starting configuration, modifying configuration items, and initializing configuration. In the second step, you need to manually edit the configuration file by using the text editor. The procedure is as follows: + +1. Run the following command to start the configuration: + + ``` + gs_dbmind service setup -c confpath + ``` + +2. After the preceding command is executed, the **dbmind.conf** configuration file is generated in the **confpath** directory. You need to use the text editor to manually modify the file. The following parameters need to be modified: **[TSDB]**, **[METADATABASE]**, and **[SELF-MONITORING]**. + + ``` + [TSDB] # Information about the time series database, that is, information about the metric data source. + name = prometheus # Time series database type. Currently, only prometheus is supported. + host = # IP address of the time series database. + port = # Port number of the time series database. + + [METADATABASE] # Metadata database information and prediction result storage mode. + dbtype = # Metadata database type. The options are sqlite, MogDB, and postgresql. If the configuration item name is MogDB, pay attention to the Python driver. For details, see the following information about the Python driver. + host = #IP address of the metadata database. + port = # Port number of the metadata database. + username = # Metadata database user information. + password = # Metadata database password, which will be encrypted for storage. + database = # Metadata database name. If the database type is SQLite, you only need to set this parameter. The DBMind generates the SQLite database file with the corresponding file name in the configuration directory according to the value of the database configuration item. + + [SELF-MONITORING] # Self-monitoring parameter. + detection_interval = 600 # Interval for performing health check on MogDB (used only for slow SQL root cause diagnosis), in seconds. + last_detection_time = 600 # Duration of data monitoring for health check (used only for slow SQL root cause diagnosis), in seconds. + forecasting_future_time = 3600 # Duration of time series prediction (for time series prediction only) and scheduling period of the prediction function, in seconds. + # KPIs to be predicted (for time series prediction only) + golden_kpi = os_cpu_usage, os_mem_usage, gaussdb_qps_by_instance + + [LOG] # Log information + maxbytes = 10485760 # Maximum size of a log file. The default value is 10 MB. If the value of maxbytes is 0, the file size increases infinitely. It is recommended that you retain the default value. + backupcount = 1 # Maximum number of log files. + level = INFO # Log level. It can be set to DEBUG, INFO, WARNING, or ERROR. + ``` + +3. After manually modifying the preceding parameters, initialize the configuration items. In this phase, DBMind preliminarily checks the correctness of the configuration items, encrypts the plaintext passwords in the configuration items, and initializes the metadata database for storing the result data. + + ``` + gs_dbmind service setup --initialize -c confpath + ``` + +4. Start the DBMind background service based on the configuration directory. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> 1. The comments in the configuration file are used to prompt users in interactive mode. Do not manually modify or delete the comments. +> 2. Make sure that the value of the configuration item and the comment are separated by a space. Otherwise, the system regards the comment as the value of the configuration item. +> 3. If special characters in a configuration item need to be escaped, use the percent sign ( %) to escape the special characters. For example, if the password is **password %**, use the percent sign ( %) to escape the special characters, that is, **password %%**. + +## Starting a Service + +After the configuration directory is initialized, you can start the DBMind background service based on the configuration directory. For example, if the configuration directory is **confpath**, run the following command: + +``` +gs_dbmind service start -c confpath +``` + +After the preceding command is executed, the system displays a message indicating that the service has been started. If no additional parameter is specified, this command starts all background tasks by default. If you want to start only one background task, add the **–only-run** option. For example, if you only want to start the slow SQL root cause analysis service, run the following command: + +``` +gs_dbmind service start -c confpath --only-run slow_query_diagnosis +``` + +## Stopping a Service + +Similar to starting a service, stopping a service has a simpler command line structure. You only need to specify the address of the configuration directory. For example, if the configuration directory is **confpath**, run the following command: + +``` +gs_dbmind service stop -c confpath +``` + +The DBMind service automatically exits after the running task is complete in the background. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** +> +> - The metabase user in **[METADATABASE]** must have the permission to create tables and insert and update data in the database. Otherwise, an exception will occur during tool execution. +> - Currently, multiple services cannot be started separately using the same configuration file. +> - The tool provides the **requirement.txt** file. You can use this file to install required third-party dependencies. + +## Command Reference + +You can use the **–help** option to obtain the help information about this mode. For example: + +``` +gs_dbmind service --help +``` + +``` +usage: service [-h] -c DIRECTORY [--only-run {slow_query_diagnosis,forecast}] [--interactive | --initialize] {setup,start,stop} + +positional arguments: + {setup,start,stop} + perform an action for service + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files + --only-run {slow_query_diagnosis,forecast} + explicitly set a certain task running in the backend + --interactive configure and initialize with interactive mode + --initialize initialize and check configurations after configuring. +``` + +**Table 1** Parameters of the gs_dbmind service subcommand + +| Parameter | Description | Value Range | +| :------------ | :----------------------------------------- | :----------------------------------------------------------- | +| action | Action parameter | - setup: initializes configuration directory.
- start: starts a service.
- stop: stops a service. | +| -c，--conf | Configuration file directory | - | +| --initialize | Initializes configuration parameters. | - | +| --interactive | Configures parameters in interactive mode. | - | +| --only-run | Selects the module to be run only. | - forecast: prediction module.
- slow_query_diagnosis: root cause analysis module for slow SQL statements. | +| -h, --help | Help information | - | diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/2-component.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/2-component.md new file mode 100644 index 0000000000000000000000000000000000000000..ea7edcb2fb3f626a5e63e52af32c1be12d2bc4be --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/2-component.md @@ -0,0 +1,37 @@ +--- +title: component +summary: component +author: Guo Huan +date: 2022-05-06 +--- + +# component + +This subcommand can be used to start DBMind components, including the exporter for monitoring metrics and other AI functions. It forwards the commands passed by the user through the CLI client to the corresponding components. For details about the commands of different components, see the corresponding sections of the components. + +## Command Reference + +You can use the **–help** option to obtain the help information about this mode. For example: + +``` +gs_dbmind component --help +``` + +``` +usage: component [-h] COMPONENT_NAME ... + +positional arguments: + COMPONENT_NAME choice a component to start. ['extract_log', 'forecast', 'index_advisor', 'opengauss_exporter', 'reprocessing_exporter', 'slow_query_diagnosis', 'sqldiag', 'xtuner'] + ARGS arguments for the component to start + +optional arguments: + -h, --help show this help message and exit +``` + +**Table 1** Parameters of the gs_dbmind component subcommand + +| Parameter | Description | Value Range | +| :------------- | :------------------- | :----------------------------------------------------------- | +| COMPONENT_NAME | Component name | extract_log, forecast, index_advisor, opengauss_exporter, reprocessing_exporter, slow_query_diagnosis, sqldiag, xtuner | +| ARGS | Component parameters | Refer to the command description of the corresponding component. | +| -h, –help | Help information | - | \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/3-set.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/3-set.md new file mode 100644 index 0000000000000000000000000000000000000000..a2154eb23549de98db7cb380b4275a76e6287acc --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/3-set.md @@ -0,0 +1,54 @@ +--- +title: set +summary: set +author: Guo Huan +date: 2022-05-06 +--- + +# set + +This command is used to change the parameter values in the **dbmind.conf** configuration file. You can also manually modify the **dbmind.conf** configuration file. The two methods have no difference. For example, to change the value of **host** in the **TSDB** configuration item of the **dbmind.conf** file in the **confpath** directory to **127.0.0.1**, run the following command: + +``` +gs_dbmind set TSDB host 127.0.0.1 -c confpath +``` + +You can choose either of the methods to modify common parameters. However, the plaintext password cannot be encrypted by manually modifying the configuration file. If you want to change the password, run the following command: + +``` +gs_dbmind set METADATABASE password xxxxxx -c confpath +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** This command is case sensitive to character strings. If you enter an incorrect character string, an error may occur during the command execution. + +## Command Reference + +You can use the **–help** option to obtain the help information about this mode. For example: + +``` +gs_dbmind set --help +``` + +``` +usage: set [-h] -c DIRECTORY section option target + +positional arguments: + section which section (case sensitive) to set + option which option to set + target the parameter target to set + +optional arguments: + -h, --help show this help message and exit + -c DIRECTORY, --conf DIRECTORY + set the directory of configuration files +``` + +**Table 1** Parameters of the set subcommand: python dbmind/ set xxx + +| Parameter | Description | Value Range | +| :-------- | :---------------------------------------- | :---------- | +| -h, –help | Help information | - | +| -c, –conf | Configuration file directory **confpath** | - | +| section | Setting area | - | +| option | Configuration item | - | +| target | Set value | - | diff --git a/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/dbmind-mode.md b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/dbmind-mode.md new file mode 100644 index 0000000000000000000000000000000000000000..108623f4121e22e61a56ec13fdc92927b1bc6233 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/ai4db/dbmind-mode/dbmind-mode.md @@ -0,0 +1,47 @@ +--- +title: DBMind Mode +summary: DBMind Mode +author: Guo Huan +date: 2022-05-06 +--- + +# DBMind Mode + +You can run the **gs_dbmind** command to invoke all functions of AI4DB. This command provides the following basic functions: + +- Service functions: You can use the **service** subcommand to implement service related functions, including creating and initializing configuration directories, starting background services, and stopping background services. +- Invoking components: You can use the **component** subcommand to invoke components. AI4DB functions (such as index recommendation and parameter tuning) can be invoked in real time in this mode. +- Setting parameters: You can use the **set** subcommand to modify the configuration file in the configuration directory. The configuration file can also be modified by using the text editor. + +You can use the **–help** option to obtain the help information about the preceding modes. For example: + +``` +gs_dbmind --help +``` + +``` +usage: [-h] [--version] {service,set,component} ... + +MogDB DBMind: An autonomous platform for MogDB + +optional arguments: + -h, --help show this help message and exit + --version show program's version number and exit + +available subcommands: + {service,set,component} + type ' -h' for help on a specific subcommand + service send a command to DBMind to change the status of the service + set set a parameter + component pass command line arguments to each sub-component. +``` + +**Table 1** gs_dbmind options + +| Parameter | Description | Value Range | +| :--------- | :---------------------------------------------- | :---------- | +| -h, --help | Help information | - | +| --version | Version number | - | +| service | Subcommand related to service functions | - | +| component | Subcommand for invoking components | - | +| set | Subcommand for modifying the configuration file | - | \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/AI-features/db4ai/db4ai.md b/product/en/docs-mogdb/v3.1/AI-features/db4ai/db4ai.md new file mode 100644 index 0000000000000000000000000000000000000000..7ea0e6e50a2a6fbf8b1ae511e6bfca012b165886 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/db4ai/db4ai.md @@ -0,0 +1,10 @@ +--- +title: DB4AI Database-driven AI +summary: DB4AI Database-driven AI +author: Guo Huan +date: 2022-05-06 +--- + +# DB4AI Database-driven AI + +DB4AI uses database capabilities to drive AI tasks and implement data storage and technology stack isomorphism. By integrating AI algorithms into the database, MogDB supports the native AI computing engine, model management, AI operators, and native AI execution plan, providing users with inclusive AI technologies. Different from the traditional AI modeling process, DB4AI one-stop modeling eliminates repeated data flowing among different platforms, simplifies the development process, and plans the optimal execution path through the database, so that developers can focus on the tuning of specific services and models. It outcompetes similar products in ease-of-use and performance. diff --git a/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md b/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md new file mode 100644 index 0000000000000000000000000000000000000000..77adfa8c27b2bbed4ab104d42803f660b1b59e08 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/db4ai-snapshots-for-data-version-management.md @@ -0,0 +1,266 @@ +--- +title: DB4AI-Snapshots for Data Version Management +summary: DB4AI-Snapshots for Data Version Management +author: Guo Huan +date: 2022-05-06 +--- + +# DB4AI-Snapshots for Data Version Management + +DB4AI-Snapshots is used by the DB4AI module to manage dataset versions. With the DB4AI-Snapshots component, developers can easily and quickly perform data preprocessing operations such as feature filtering and type conversion. In addition, developers can perform version control on training datasets like Git. After a data table snapshot is created, it can be used as a view. However, once the data table snapshot is released, it is fixed as static data. To modify the content of this data table snapshot, you need to create a data table snapshot with a different version number. + +## Lifecycle of DB4AI-Snapshots + +DB4AI-Snapshots can be published, archived, or purged. After being published, DB4AI-Snapshots can be used. Archived DB4AI-Snapshots are in the archiving period and will not be used to train new models. Instead, old data is used to verify new models. Purged DB4AI-Snapshots will not be found in the database system. + +Note that the DB4AI-Snapshots function is used to provide unified training data for users. Team members can use the specified training data to retrain the machine learning models, facilitating collaboration between users. Therefore, the DB4AI-Snapshots feature is not supported in scenarios where user data conversion is not supported, such as **private users** and **separation of duty** (**enableSeparationOfDuty** set to **ON**). + +You can run the **CREATE SNAPSHOT** statement to create a data table snapshot. The created snapshot is in the published state by default. You can create a table snapshot in either **MSS** or **CSS** mode, which can be configured using the GUC parameter **db4ai_snapshot_mode**. For the MSS mode, it is realized by materialization algorithm where data entity of original datasets is stored. The CSS mode is implemented based on a relative calculation algorithm where incremental data information is stored. The metadata of the data table snapshot is stored in the system directory of DB4AI. You can view it in the **db4ai.snapshot** system catalog. + +You can run the **ARCHIVE SNAPSHOT** statement to mark a data table snapshot as archived, and run the **PUBLISH SNAPSHOT** statement to mark it as published again. The state of a data table snapshot is marked to help data scientists work together as a team. + +If a table snapshot is no longer useful, you can run the **PURGE SNAPSHOT** statement to permanently delete it and restore the storage space. + +## DB4AI-Snapshots Usage Guide + +1. Create a table and insert table data. + + If a data table exists in the database, you can create a data table snapshot based on the existing data table. To facilitate subsequent demonstration, create a data table named **t1** and insert test data into the table. + + ```sql + create table t1 (id int, name varchar); + insert into t1 values (1, 'zhangsan'); + insert into t1 values (2, 'lisi'); + insert into t1 values (3, 'wangwu'); + insert into t1 values (4, 'lisa'); + insert into t1 values (5, 'jack'); + ``` + + Run the following SQL statement to query the content of the collocation data table: + + ```sql + SELECT * FROM t1; + id | name + ----+---------- + 1 | zhangsan + 2 | lisi + 3 | wangwu + 4 | lisa + 5 | jack + (5 rows) + ``` + +2. Use DB4AI-Snapshots. + + - Create DB4AI-Snapshots. + + - Example 1: CREATE SNAPSHOT… AS + + In the following example, the default version separator is an at sign (@), and the default subversion separator is a period (.). You can set the separator by setting the **db4ai_snapshot_version_delimiter** and **db4ai_snapshot_version_separator** parameters. + + ```sql + create snapshot s1@1.0 comment is 'first version' as select * from t1; + schema | name + --------+-------- + public | s1@1.0 + (1 row) + ``` + + The command output indicates that a snapshot has been created for data table **s1** and the version number is **1.0**. A created data table snapshot can be queried in the same way as a common view, but cannot be updated using the **INSERT INTO** statement. For example, you can use any of the following statements to query the content of data table snapshot **s1** of version 1.0: + + ```sql + SELECT * FROM s1@1.0; + SELECT * FROM public.s1@1.0; + SELECT * FROM public . s1 @ 1.0; + id | name + ----+---------- + 1 | zhangsan + 2 | lisi + 3 | wangwu + 4 | lisa + 5 | jack + (5 rows) + ``` + + You can run the following SQL statement to modify the content of the **t1** data table: + + ```sql + UPDATE t1 SET name = 'tom' where id = 4; + insert into t1 values (6, 'john'); + insert into t1 values (7, 'tim'); + ``` + + When content of data table **t1** is retrieved again, it is found that although the content of data table **t1** has changed, the query result of data table snapshot **s1@1.0** does not change. Because data in data table **t1** has changed, to use content of the current data table as a version 2.0, you can create snapshot **s1@2.0** by using the following SQL statement: + + ```sql + create snapshot s1@2.0 as select * from t1; + ``` + + According to the foregoing example, it can be found that the data table snapshot can solidify content of a data table, to avoid instability during training of a machine learning model caused by data modification in the process, and can also avoid a lock conflict caused when multiple users access and modify the same table at the same time. + + - Example 2: CREATE SNAPSHOT… FROM + + You can run an SQL statement to inherit a created data table snapshot and generate a new data table snapshot based on the data modification. Example: + + ```sql + create snapshot s1@3.0 from @1.0 comment is 'inherits from @1.0' using (INSERT VALUES(6, 'john'), (7, 'tim'); DELETE WHERE id = 1); + schema | name + --------+-------- + public | s1@3.0 + (1 row) + ``` + + Where, @ is the data table snapshot version separator and the from clause is followed by the existing data table snapshot, in the format of @ + version number. You can add an operation keyword (such as **INSERT**, **UPDATE**, **DELETE**, or **ALTER**) after the **USING** keyword. In the **INSERT INTO** and **DELETE FROM** statements, clauses related to data table snapshot names, such as **INTO** and **FROM**, can be omitted. For details, see [AI Feature Functions](../../../reference-guide/functions-and-operators/28-ai-feature-functions.md). + + In the example, based on the **s1@1.0** snapshot, insert two pieces of data and delete one piece of data to generate a new snapshot **s1@3.0**. Then, retrieve **s1@3.0**. + + ```sql + SELECT * FROM s1@3.0; + id | name + ----+---------- + 2 | lisi + 3 | wangwu + 4 | lisa + 5 | jack + 6 | john + 7 | tim + (7 rows) + ``` + + - Delete the data table snapshot **SNAPSHOT**. + + ```sql + purge snapshot s1@3.0; + schema | name + --------+-------- + public | s1@3.0 + (1 row) + ``` + + At this time, no data can be retrieved from **s1@3.0**, and the records of the data table snapshot in the **db4ai.snapshot** view are cleared. Deleting the data table snapshot of this version does not affect the data table snapshots of other versions. + + - Sample from a data table snapshot. + + Example: Use the sampling rate 0.5 to extract data from snapshot **s1**. + + ```sql + sample snapshot s1@2.0 stratify by name as nick at ratio .5; + schema | name + --------+------------ + public | s1nick@2.0 + (1 row) + ``` + + You can use this function to create a training set and a test set. For example: + + ```sql + SAMPLE SNAPSHOT s1@2.0 STRATIFY BY name AS _test AT RATIO .2, AS _train AT RATIO .8 COMMENT IS 'training'; + schema | name + --------+---------------- + public | s1_test@2.0 + public | s1_train@2.0 + (2 rows) + ``` + + - Publish a data table snapshot. + + Run the following SQL statement to mark the data table snapshot **s1@2.0** as published: + + ```sql + publish snapshot s1@2.0; + schema | name + --------+-------- + public | s1@2.0 + (1 row) + ``` + + - Archive a data table snapshot. + + Run the following statement to mark the data table snapshot as archived: + + ```sql + archive snapshot s1@2.0; + schema | name + --------+-------- + public | s1@2.0 + (1 row) + ``` + + You can use the views provided by DB4AI-Snapshots to view the status of the current data table snapshot and other information. + + ```sql + select * from db4ai.snapshot; + id | parent_id | matrix_id | root_id | schema | name | owner | commands | comment | published | archived | created | row_count + ----+-----------+-----------+---------+--------+------------+--------+------------------------------------------+---------+-----------+----------+----------------------------+----------- + 1 | | | 1 | public | s1@2.0 | omm | {"select *","from t1 where id > 3",NULL} | | t | f | 2021-04-17 09:24:11.139868 | 2 + 2 | 1 | | 1 | public | s1nick@2.0 | omm | {"SAMPLE nick .5 {name}"} | | f | f | 2021-04-17 10:02:31.73923 | 0 + ``` + +3. Perform troubleshooting in case of exceptions. + + - The data table or DB4AI snapshot does not exist. + + ```sql + purge snapshot s1nick@2.0; + publish snapshot s1nick@2.0; + --------- + ERROR: snapshot public."s1nick@2.0" does not exist + CONTEXT: PL/pgSQL function db4ai.publish_snapshot(name,name) line 11 at assignment + + archive snapshot s1nick@2.0; + ---------- + ERROR: snapshot public."s1nick@2.0" does not exist + CONTEXT: PL/pgSQL function db4ai.archive_snapshot(name,name) line 11 at assignment + ``` + + - Before deleting a snapshot, ensure that other snapshots that depend on it have been deleted. + + ```sql + purge snapshot s1@1.0; + ERROR: cannot purge root snapshot 'public."s1@1.0"' having dependent snapshots + HINT: purge all dependent snapshots first + CONTEXT: referenced column: purge_snapshot_internal + SQL statement "SELECT db4ai.purge_snapshot_internal(i_schema, i_name)" + PL/pgSQL function db4ai.purge_snapshot(name,name) line 71 at PERFORM + ``` + +4. Set GUC parameters. + + - db4ai_snapshot_mode： + + There are two snapshot modes: MSS (materialized mode, used to store data entities) and CSS (computing mode, used to store incremental information). The snapshot mode can be switched between MSS and CSS. The default snapshot mode is MSS. + + - db4ai_snapshot_version_delimiter： + + Used to set the data table snapshot version separator. The at sign (@) is the default data table snapshot version separator. + + - db4ai_snapshot_version_separator + + Used to set the data table snapshot subversion separator. The period (.) is the default data table snapshot subversion separator. + +5. View the snapshot details of a data table in the DB4AI schema by using **db4ai.snapshot**. + + ```sql + MogDB=# \d db4ai.snapshot + Table "db4ai.snapshot" + Column | Type | Modifiers + -----------+-----------------------------+--------------------------- + id | bigint | + parent_id | bigint | + matrix_id | bigint | + root_id | bigint | + schema | name | not null + name | name | not null + owner | name | not null + commands | text[] | not null + comment | text | + published | boolean | not null default false + archived | boolean | not null default false + created | timestamp without time zone | default pg_systimestamp() + row_count | bigint | not null + Indexes: + "snapshot_pkey" PRIMARY KEY, btree (schema, name) TABLESPACE pg_default + "snapshot_id_key" UNIQUE CONSTRAINT, btree (id) TABLESPACE pg_default + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The DB4AI namespace is a private domain of this function. Functional indexes cannot be created in the DB4AI namespace. diff --git a/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/full-process-ai.md b/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/full-process-ai.md new file mode 100644 index 0000000000000000000000000000000000000000..df6e65012d8f72fb7c422580f10c8e9913e95839 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/full-process-ai.md @@ -0,0 +1,12 @@ +--- +title: Full-process AI +summary: Full-process AI +author: Guo Huan +date: 2022-05-06 +--- + +# Full-process AI + +A traditional AI task usually has multiple processes. For example, a data collection process includes data collection, data cleaning, and data storage, an algorithm training process includes data preprocessing, training, and model storage and management, and a model training process includes hyperparameter tuning. The entire lifecycle of such machine learning models can be integrated into the database. The model training, management, and optimization processes are performed closest to the data storage. The out-of-the-box AI lifecycle management function of SQL statements is provided on the database side, which is called full-process AI. + +MogDB implements some full-process AI functions, which will be described in detail in this section. diff --git a/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md b/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md new file mode 100644 index 0000000000000000000000000000000000000000..ecf7b7c72f14db3e231aa19b15642ef9dedfbb24 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/db4ai/full-process-ai/plpython-fenced-mode.md @@ -0,0 +1,135 @@ +--- +title: PLPython Fenced Mode +summary: PLPython Fenced Mode +author: Guo Huan +date: 2022-05-06 +--- + +# PLPython Fenced Mode + +PL/Python is added to the fenced mode, which is insecure. During database compilation, to integrate Python into the database, you can add the **–with-python** option to **configure**, or specify the Python path for installing PL/Python and add the **–with-includes='/python-dir=path'** option. + +Before starting the database, set the GUC parameter **unix_socket_directory** to specify the file address for communication between unix_socket processes. You need to create a folder in **user-set-dir-path** in advance and grant read, write, and execute permissions on the folder. + +``` +unix_socket_directory = '/user-set-dir-path' +``` + +After the configuration is complete, start the database. + +After PL/Python is added to the database compilation and the GUC parameter **unix_socket_directory** is set, the **fenced-Master** process is automatically created during database startup. If Python compilation is not performed for the database, you need to manually start the master process in fenced mode. After the GUC parameter is set, run the command to create the master process. + +Run the following command to start the **fenced-Master** process: + +``` +gaussdb --fenced -k /user-set-dir-path -D /user-set-dir-path & +``` + +After the fenced mode is configured, the UDF calculation is performed in the **fenced-worker** process for the PL/Python-fenced UDF database. + +## Usage Guide + +- Create an extension. + + - When the compiled PL/Python is Python 2: + + ``` + MogDB=# create extension plpythonu; + CREATE EXTENSION + ``` + + - When the compiled PL/Python is Python 3: + + ``` + MogDB=# create extension plpython3u; + CREATE EXTENSION + ``` + + The following uses Python 2 as an example. + +- Create a PL/Python-fenced UDF database. + + ``` + MogDB=# create or replace function pymax(a int, b int) + MogDB-# returns INT + MogDB-# language plpythonu fenced + MogDB-# as $$ + MogDB$# import numpy + MogDB$# if a > b: + MogDB$# return a; + MogDB$# else: + MogDB$# return b; + MogDB$# $$; + CREATE FUNCTION + ``` + +- View UDF information. + + ``` + MogDB=# select * from pg_proc where proname='pymax'; + -[ RECORD 1 ]----+-------------- + proname | pymax + pronamespace | 2200 + proowner | 10 + prolang | 16388 + procost | 100 + prorows | 0 + provariadic | 0 + protransform | - + proisagg | f + proiswindow | f + prosecdef | f + proleakproof | f + proisstrict | f + proretset | f + provolatile | v + pronargs | 2 + pronargdefaults | 0 + prorettype | 23 + proargtypes | 23 23 + proallargtypes | + proargmodes | + proargnames | {a,b} + proargdefaults | + prosrc | + | import numpy + | if a > b: + | return a; + | else: + | return b; + | + probin | + proconfig | + proacl | + prodefaultargpos | + fencedmode | t + proshippable | f + propackage | f + prokind | f + proargsrc | + ``` + +- Run the UDF. + + - Create a data table. + + ``` + MogDB=# create table temp (a int ,b int) ; + CREATE TABLE + MogDB=# insert into temp values (1,2),(2,3),(3,4),(4,5),(5,6); + INSERT 0 5 + ``` + + - Run the UDF. + + ``` + MogDB=# select pymax(a,b) from temp; + pymax + ------- + 2 + 3 + 4 + 5 + 6 + (5 rows) + ``` diff --git a/product/en/docs-mogdb/v3.1/AI-features/db4ai/native-db4ai-engine.md b/product/en/docs-mogdb/v3.1/AI-features/db4ai/native-db4ai-engine.md new file mode 100644 index 0000000000000000000000000000000000000000..fcca6bd62d436a3476d085e625832aac6aa2b0af --- /dev/null +++ b/product/en/docs-mogdb/v3.1/AI-features/db4ai/native-db4ai-engine.md @@ -0,0 +1,313 @@ +--- +title: Native DB4AI Engine +summary: Native DB4AI Engine +author: Guo Huan +date: 2022-05-06 +--- + +# Native DB4AI Engine + +The current version of openGauss supports the native DB4AI capability. By introducing native AI operators, openGauss simplifies the operation process and fully utilizes the optimization and execution capabilities of the database optimizer and executor to obtain the high-performance model training capability in the database. With a simpler model training and prediction process and higher performance, developers can focus on model tuning and data analysis in a shorter period of time, avoiding fragmented technology stacks and redundant code implementation. + +## Keyword Parsing + +**Table 1** DB4AI syntax and keywords + +| Name | | Description | +| :------ | :----------- | ---------------------------------------------------------- | +| Syntax | CREATE MODEL | Creates a model, trains it, and saves the model. | +| | PREDICT BY | Uses an existing model for prediction. | +| | DROP MODEL | Deletes a model. | +| Keyword | TARGET | Target column name of a training or prediction task. | +| | FEATURES | Data feature column name of a training or prediction task. | +| | MODEL | Model name of a training task. | + +## Usage Guide + +1. Introduce the algorithms supported in this version. + + The DB4AI of the current version supports the following new algorithms: + + **Table 2** Supported algorithms + + | Optimization Algorithm | Algorithm | + | :--------------------- | :------------------------------ | + | GD | logistic_regression | + | | linear_regression | + | | svm_classification | + | | PCA | + | | multiclass | + | Kmeans | kmeans | + | xgboost | xgboost_regression_logistic | + | | xgboost_binary_logistic | + | | xgboost_regression_squarederror | + | | xgboost_regression_gamma | + +2. Learn about the model training syntax. + + - CREATE MODEL + + You can run the **CREATE MODEL** statement to create and train a model. This SQL statement uses the public Iris dataset for model training. + + - The following uses multiclass as an example to describe how to train a model. Specify **sepal_length**, **sepal_width**, **petal_length**, and **petal_width** as feature columns in the **tb_iris** training set, and use the multiclass algorithm to create and save the **iris_classification_model** model. + + ``` + MogDB=# CREATE MODEL iris_classification_model USING xgboost_regression_logistic FEATURES sepal_length, sepal_width,petal_length,petal_width TARGET target_type < 2 FROM tb_iris_1 WITH nthread=4, max_depth=8; + MODEL CREATED. PROCESSED 1 + ``` + + In the preceding command: + + - The **CREATE MODEL** statement is used to train and save a model. + - **USING** specifies the algorithm name. + - **FEATURES** specifies the features of the training model and needs to be added based on the column name of the training data table. + - **TARGET** specifies the training target of the model. It can be the column name of the data table required for training or an expression, for example, **price > 10000**. + - **WITH** specifies the hyperparameters used for model training. When the hyperparameter is not set by the user, the framework uses the default value. + + The framework supports various hyperparameter combinations for different operators. + + **Table 3** Hyperparameters supported by operators + + | Operator | Hyperparameter | + | :----------------------------------------------------------- | :----------------------------------------------------------- | + | GD
(logistic_regression、linear_regression、svm_classification) | optimizer(char*), verbose(bool), max_iterations(int), max_seconds(double), batch_size(int), learning_rate(double), decay(double), and tolerance(double)* *SVM limits the hyperparameter **lambda(double)**.* | + | K-Means | max_iterations(int), num_centroids(int), tolerance(double), batch_size(int), num_features(int), distance_function(char), seeding_function(char*), verbose(int), and seed(int)* | + | GD(pca) | batch_size(int);max_iterations(int);max_seconds(int);tolerance(float8);verbose(bool);number_components(int);seed(int) | + | GD(multiclass) | classifier(char)
Note: Other hyperparameter types of multiclass depend on the categories in the selected classifier. | + | xgboost_regression_logistic、xgboost_binary_logistic、xgboost_regression_squarederror、xgboost_regression_gamma | batch_size(int);booster(char);tree_method(char);eval_metric(char);seed(int);nthread(int);max_depth(int);gamma(float8);eta(float8);min_child_weight(int);verbosity(int) | + + The default value and value range of each hyperparameter are as follows: + + **Table 4** Default values and value ranges of hyperparameters + + | Operator | Default Hyperparameter Value | Value Range | Hyperparameter Description | + | :----------------------------------------------------------- | :------------------------------------------------------ | :----------------------------------------------------------- | :----------------------------------------------------------- | + | GD:logistic_regression、linear_regression、svm_classification、pca | optimizer = gd (gradient descent) | gd/ngd (natural gradient descent) | Optimizer | + | | verbose = false | T/F | Log display | + | | max_iterations = 100 | (0, 10000] | Maximum iterations | + | | max_seconds = 0 (The running duration is not limited.) | [0,INT_MAX_VALUE] | Running duration | + | | batch_size = 1000 | (0, 1048575] | Number of data records selected per training | + | | learning_rate = 0.8 | (0, DOUBLE_MAX_VALUE] | Learning rate | + | | decay = 0.95 | (0, DOUBLE_MAX_VALUE] | Weight decay rate | + | | tolerance = 0.0005 | (0, DOUBLE_MAX_VALUE] | Tolerance | + | | seed = 0 (random value of **seed**) | [0, INT_MAX_VALUE] | Seed | + | | just for linear、SVM：kernel = “linear” | linear/gaussian/polynomial | Kernel function | + | | just for linear、SVM：components = MAX(2*features, 128) | [0, INT_MAX_VALUE] | Number of high-dimension space dimensions | + | | just for linear、SVM：gamma = 0.5 | (0, DOUBLE_MAX_VALUE] | Gaussian kernel function parameter | + | | just for linear、SVM：degree = 2 | [2, 9] | Polynomial kernel function parameter | + | | just for linear、SVM：coef0 = 1.0 | [0, DOUBLE_MAX_VALUE] | Polynomial kernel function parameter | + | | just for SVM：lambda = 0.01 | (0, DOUBLE_MAX_VALUE) | Regularization parameter | + | | just for pca： number_components | （0，INT_MAX_VALUE] | Target dimension after dimension reduction | + | GD:multiclass | classifier=“svm_classification” | svm_classification\logistic_regression | Classifier for multiclass tasks | + | Kmeans | max_iterations = 10 | [1, 10000] | Maximum iterations | + | | num_centroids = 10 | [1, 1000000] | Number of clusters | + | | tolerance = 0.00001 | (0,1] | Central point error | + | | batch_size = 10 | [1,1048575] | Number of data records selected per training | + | | num_features = 2 | [1, INT_MAX_VALUE] | Number of sample features | + | | distance_function = “L2_Squared” | L1\L2\L2_Squared\Linf | Regularization method | + | | seeding_function = “Random++” | “Random++”\“KMeans | Method for initializing seed points | + | | verbose = 0U | { 0, 1, 2 } | Verbose mode | + | | seed = 0U | [0, INT_MAX_VALUE] | Seed | + | xgboost:
xgboost_regression_logistic、xgboost_binary_logistic、xgboost_regression_gamma、xgboost_regression_squarederror | n_iter=10 | (0, 10000] | Iteration times | + | | batch_size=10000 | (0, 1048575] | Number of data records selected per training | + | | booster=“gbtree” | gbtree\gblinear\dart | Booster type | + | | tree_method=“auto” | auto\exact\approx\hist\gpu_hist
Note: To use the **gpu_hist** parameter, you must configure a GPU library. Otherwise, the DB4AI platform does not support this value. | Tree construction algorithm | + | | eval_metric=“rmse” | rmse\rmsle\map\mae\auc\aucpr | Data verification metric | + | | seed=0 | [0, 100] | Seed | + | | nthread=1 | (0, MAX_MEMORY_LIMIT] | Concurrency | + | | max_depth=5 | (0, MAX_MEMORY_LIMIT] | Maximum depth of the tree. This parameter is valid only for the tree booster. | + | | gamma=0.0 | [0, 1] | Minimum loss required for further partitioning on leaf nodes | + | | eta=0.3 | [0, 1] | Step used in the update to prevent overfitting | + | | min_child_weight=1 | [0, INT_MAX_VALUE] | Minimum sum of instance weights required by child nodes | + | | verbosity=1 | 0 (silent)\1 (warning)\2 (info)\3 (debug) | Printing level | + | MAX_MEMORY_LIMIT = Maximum number of tuples loaded in memory | | | | + | GS_MAX_COLS = Maximum number of attributes in a database table | | | | + + - If the model is saved successfully, the following information is returned: + + ``` + MODEL CREATED. PROCESSED x + ``` + +3. View the model information. + + After the training is complete, the model is stored in the **gs_model_warehouse** system catalog. You can view information about the model and training process in the **gs_model_warehouse** system catalog. + + The model details are stored in the system catalog in binary mode. You can use the **gs_explain_model** function to view the model details. The statement is as follows: + + ``` + MogDB=# select * from gs_explain_model("iris_classification_model"); + DB4AI MODEL + ------------------------------------------------------------- + Name: iris_classification_model + Algorithm: xgboost_regression_logistic + Query: CREATE MODEL iris_classification_model + USING xgboost_regression_logistic + FEATURES sepal_length, sepal_width,petal_length,petal_width + TARGET target_type < 2 + FROM tb_iris_1 + WITH nthread=4, max_depth=8; + Return type: Float64 + Pre-processing time: 0.000000 + Execution time: 0.001443 + Processed tuples: 78 + Discarded tuples: 0 + n_iter: 10 + batch_size: 10000 + max_depth: 8 + min_child_weight: 1 + gamma: 0.0000000000 + eta: 0.3000000000 + nthread: 4 + verbosity: 1 + seed: 0 + booster: gbtree + tree_method: auto + eval_metric: rmse + rmse: 0.2648450136 + model size: 4613 + ``` + +4. Use an existing model to perform a prediction task. + + Use the **SELECT** and **PREDICT BY** keywords to complete the prediction task based on the existing model. + + Query syntax: SELECT… PREDICT BY… (FEATURES…)… FROM…; + + ``` + MogDB=# SELECT id, PREDICT BY iris_classification (FEATURES sepal_length,sepal_width,petal_length,petal_width) as "PREDICT" FROM tb_iris limit 3; + + id | PREDICT + -----+--------- + 84 | 2 + 85 | 0 + 86 | 0 + (3 rows) + ``` + + For the same prediction task, the results of the same model are stable. In addition, models trained based on the same hyperparameter and training set are stable. AI model training is random (random gradient descent of data distribution each batch). Therefore, the computing performance and results of different models can vary slightly. + +5. View the execution plan. + + You can use the **EXPLAIN** statement to analyze the execution plan in the model training or prediction process of **CREATE MODEL** and **PREDICT BY**. The keyword **EXPLAIN** can be followed by a **CREATE MODEL** or **PREDICT BY** clause or an optional parameter. The supported parameters are as follows: + + **Table 5** Parameters supported by EXPLAIN + + | Parameter | Description | + | :-------- | :----------------------------------------------------------- | + | ANALYZE | Boolean variable, which is used to add description information such as the running time and number of loop times | + | VERBOSE | Boolean variable, which determines whether to output the training running information to the client | + | COSTS | Boolean variable | + | CPU | Boolean variable | + | DETAIL | Boolean variable, which is unavailable | + | NODES | Boolean variable, which is unavailable | + | NUM_NODES | Boolean variable, which is unavailable | + | BUFFERS | Boolean variable | + | TIMING | Boolean variable | + | PLAN | Boolean variable | + | FORMAT | Optional format type: TEXT, XML, JSON, and YAML | + + Example: + + ``` + MogDB=# Explain CREATE MODEL patient_logisitic_regression USING logistic_regression FEATURES second_attack, treatment TARGET trait_anxiety > 50 FROM patients WITH batch_size=10, learning_rate = 0.05; + QUERY PLAN + ------------------------------------------------------------------------- + Train Model - logistic_regression (cost=0.00..0.00 rows=0 width=0) + -> Materialize (cost=0.00..41.08 rows=1776 width=12) + -> Seq Scan on patients (cost=0.00..32.20 rows=1776 width=12) + (3 rows) + ``` + +6. Perform troubleshooting in case of exceptions. + + - Training phase + + - Scenario 1: When the value of the hyperparameter exceeds the value range, the model training fails and an error message is returned. For example: + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET trait_anxiety FROM patients WITH optimizer='aa'; + ERROR: Invalid hyperparameter value for optimizer. Valid values are: gd, ngd. + ``` + + - Scenario 2: If the model name already exists, the model fails to be saved, and an error message with the cause is displayed. For example: + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET trait_anxiety FROM patients; + ERROR: The model name "patient_linear_regression" already exists in gs_model_warehouse. + ``` + + - Scenario 3: If the value in the **FEATURE** or **TARGETS** column is *****, an error message with the cause is displayed. For example: + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES * TARGET trait_anxiety FROM patients; + ERROR: FEATURES clause cannot be * + ----------------------------------------------------------------------------------------------------------------------- + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment TARGET * FROM patients; + ERROR: TARGET clause cannot be * + ``` + + - Scenario 4: If the keyword **TARGET** is used in the unsupervised learning method or is not applicable to the supervised learning method, an error message with the cause is displayed. For example: + + ``` + MogDB=# CREATE MODEL patient_linear_regression USING linear_regression FEATURES second_attack,treatment FROM patients; + ERROR: Supervised ML algorithms require TARGET clause + ----------------------------------------------------------------------------------------------------------------------------- + CREATE MODEL patient_linear_regression USING linear_regression TARGET trait_anxiety FROM patients; + ERROR: Supervised ML algorithms require FEATURES clause + ``` + + - Scenario 5: If there is only one category in the **TARGET** column, an error message with the cause is displayed. For example: + + ``` + MogDB=# CREATE MODEL ecoli_svmc USING multiclass FEATURES f1, f2, f3, f4, f5, f6, f7 TARGET cat FROM (SELECT * FROM db4ai_ecoli WHERE cat='cp'); + ERROR: At least two categories are needed + ``` + + - Scenario 6: DB4AI filters out data that contains null values during training. When the model data involved in training is null, an error message with the cause is displayed. For example: + + ``` + MogDB=# create model iris_classification_model using xgboost_regression_logistic features message_regular target error_level from error_code; + ERROR: Training data is empty, please check the input data. + ``` + + - Scenario 7: The DB4AI algorithm has restrictions on the supported data types. If the data type is not in the whitelist, an error message is returned and the invalid OID is displayed. You can check the OID in **pg_type** to determine the invalid data type. For example: + + ``` + MogDB=# CREATE MODEL ecoli_svmc USING multiclass FEATURES f1, f2, f3, f4, f5, f6, f7, cat TARGET cat FROM db4ai_ecoli ; + ERROR: Oid type 1043 not yet supported + ``` + + - Scenario 8: If the GUC parameter **statement_timeout** is set, the statement that is executed due to training timeout will be terminated. In this case, execute the **CREATE MODEL** statement. Parameters such as the size of the training set, number of training rounds (**iteration**), early termination conditions (**tolerance** and **max_seconds**), and number of parallel threads (**nthread**) affect the training duration. When the duration exceeds the database limit, the statement execution is terminated and model training fails. + + - Model parsing + + - Scenario 9: If the model name cannot be found in the system catalog, an error message with the cause is displayed. For example: + + ``` + MogDB=# select gs_explain_model("ecoli_svmc"); + ERROR: column "ecoli_svmc" does not exist + ``` + + - Prediction phase + + - Scenario 10: If the model name cannot be found in the system catalog, an error message with the cause is displayed. For example: + + ``` + MogDB=# select id, PREDICT BY patient_logistic_regression (FEATURES second_attack,treatment) FROM patients; + ERROR: There is no model called "patient_logistic_regression". + ``` + + - Scenario 11: If the data dimension and data type of the **FEATURES** task are inconsistent with those of the training set, an error message with the cause is displayed. For example: + + ``` + MogDB=# select id, PREDICT BY patient_linear_regression (FEATURES second_attack) FROM patients; + ERROR: Invalid number of features for prediction, provided 1, expected 2 + CONTEXT: referenced column: patient_linear_regression_pred + ------------------------------------------------------------------------------------------------------------------------------------- + MogDB=# select id, PREDICT BY patient_linear_regression (FEATURES 1,second_attack,treatment) FROM patients; + ERROR: Invalid number of features for prediction, provided 3, expected 2 + CONTEXT: referenced column: patient_linear_regression_pre + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The DB4AI feature requires data access for computing and is not applicable to encrypted databases. diff --git a/product/en/docs-mogdb/v3.1/_index.md b/product/en/docs-mogdb/v3.1/_index.md new file mode 100644 index 0000000000000000000000000000000000000000..764f32f4fd9524992737a88e8c788b72acaeaf22 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/_index.md @@ -0,0 +1,90 @@ +--- +title: Introduction +summary: SELECT +author: Zhang Cuiping +date: 2022-08-07 +--- + + + + + +[MogDB Introduction](./overview.md) + +[Characteristic Description](./characteristic-description/characteristic-description-overview.md) + +[Release Notes](./about-mogdb/mogdb-new-feature/release-note.md) + + + + + +[MogDB Playground](./quick-start/mogdb-playground.md) + + + + + +[PTK-based Installation](./installation-guide/ptk-based-installation.md) + +[MogDB in Container](./installation-guide/docker-installation/docker-installation.md) + +[MogDB on Kubernetes](../../docs-mogdb-stack/v1.0/quick-start.md) + + + + + +[Development Based on JDBC](./developer-guide/dev/2-development-based-on-jdbc/3-development-process.md) + +[Data Import and Export](./administrator-guide/importing-and-exporting-data/importing-data/1-import-modes.md) + + + + + +[Routine Maintenance](./administrator-guide/routine-maintenance/0-starting-and-stopping-mogdb.md) + +[Backup and Restoration](./administrator-guide/br/1-1-br.md) + +[Upgrade Guide](./administrator-guide/upgrade-guide.md) + +[Common Fault Locating Methods](./common-faults-and-identification/common-fault-locating-methods.md) + +[MogDB Stack](../../docs-mogdb-stack/v1.0/overview.md) + +[MogHA](../../docs-mogha/v2.3/overview.md) + +[MogDB Tools](./reference-guide/tool-reference/tool-overview.md) + + + + + +[System Optimization](./performance-tuning/1-system/1-optimizing-os-parameters.md) + +[SQL Optimization](./performance-tuning/2-sql/1-query-execution-process.md) + +[WDR Snapshot](./performance-tuning/3-wdr/wdr-snapshot-schema.md) + +[TPCC Performance Tuning Guide](./performance-tuning/TPCC-performance-tuning-guide.md) + + + + + +[PTK - Provisioning Toolkit](../../docs-ptk/v0.4/overview.md) + +[MTK - Database Migration Toolkit](../../docs-mtk/v2.0/overview.md) + +[SCA - SQL Compatibility Analyzer](../../docs-sca/v5.1/overview.md) + +[MVD - MogDB Verify Data](../../docs-mvd/v2.4/overview.md) + +[MDB - MogDB Data Bridge](../../docs-mdb/v1.0/overview.md) + +[Mogeaver - Graphical Tool](./mogeaver/mogeaver-overview.md) + + + + diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/MogDB-compared-to-openGauss.md b/product/en/docs-mogdb/v3.1/about-mogdb/MogDB-compared-to-openGauss.md new file mode 100644 index 0000000000000000000000000000000000000000..6a6e8e81dccc5e15f2b511773e6065b7ba42ae51 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/MogDB-compared-to-openGauss.md @@ -0,0 +1,124 @@ +--- +title: Comparison Between MogDB and openGauss +summary: Comparison Between MogDB and openGauss +author: Guo Huan +date: 2021-04-02 +--- + +# Comparison Between MogDB and openGauss + +## Relationship Between MogDB and openGauss + +MogDB is an enterprise database that is packaged and improved on the basis of openGauss open source kernel and it is more friendly to enterprise applications. On the basis of openGauss kernel, MogDB adds the MogHA component for automatic management and high availability under the primary-standby architecture, which is crucial for enterprise applications. At the same time, MogDB Manager includes backup and recovery, monitoring, automatic installation and other components for enterprise-level ease of use requirements. + +MogDB is a commercial product that is sold according to an established license pricing system and is supported by Enmo's professional services. + +## Introduction to openGauss + +openGauss is an open source relational database management system. The kernel of openGauss is derived from PostgreSQL and is distributed under Mulan PSL v2. openGauss kernel is open source, anyone and any organization can download the source code for compilation and installation without any cost; The openGauss community also regularly releases compiled binary installation files, and the current release strategy is to release one stable frequently supported version per year (end of March each year) and one radical version with new features (end of September each year). + +openGauss is a standalone database. It has the basic features of relational databases as well as enhanced features. + +For more details, please visit openGauss official website: + +### Basic Features + +- Standard SQLs + + Supports SQL92, SQL99, SQL2003, and SQL2011 standards, GBK and UTF-8 character sets, SQL standard functions and analytic functions, and SQL Procedural Language. + +- Database storage management + + Supports tablespaces where different tables can be stored in different locations. + +- Primary/standby deployment + + Supports the ACID properties, single-node fault recoveries, primary/standby data synchronization, and primary/standby switchover. + +- APIs + + Supports standard JDBC 4.0 and ODBC 3.5. + +- Management tools + + Provides installation and deployment tools, instance start and stop tools, and backup and restoration tools. + +- Security management + + Supports SSL network connections, user permission management, password management, security auditing, and other functions, to ensure data security at the management, application, system, and network layers. + +### Enhanced Features + +- Data Partitioning + + Data partitioning is a general function for most database products. In openGauss, data is partitioned horizontally with a user-specified policy. This operation splits a table into multiple partitions that are not overlapped. + +- Vectorized Executor and Hybrid Row-Column Storage Engine + + In a wide table containing a huge amount of data, a query usually only involves certain columns. In this case, the query performance of the row-store engine is poor. For example, a single table containing the data of a meteorological agency has 200 to 800 columns. Among these columns, only ten of them are frequently accessed. In this case, a vectorized executor and column-store engine can significantly improve performance by saving storage space. + +- High Availability (HA) Transaction Processing + + openGauss manages transactions and guarantees the ACID properties. openGauss provides a primary/standby HA mechanism to reduce the service interruption time when the primary node is faulty. It protects key user programs and continuously provides external services, minimizing the impact of hardware, software, and human faults on services to ensure service continuity. + +- High Concurrency and High Performance + + openGauss supports 10,000 concurrent connections through server thread pools. It supports thread nucleophilicity and millions of tpmC using the NUMA-based kernel data structure, manages TB-level large memory buffers through efficient hot and cold data elimination, achieves multi-version access without read/write blocks using CSN-based snapshots, and avoids performance fluctuation caused by full-page writes using incremental checkpoints. + +- SQL Self-Diagnosis + + To locate performance issues of a query, you can use **EXPLAIN PERFORMANCE** to query its execution plan. However, this method produces many logs, requires to modify service logic, and depends on expertise to locate problems. SQL self-diagnosis enables users to locate performance issues more efficiently. + +- Equality Query in a Fully-encrypted Database + + The encrypted database allows the client to encrypt sensitive data within the client application. During the query period, the entire service data flow exists in the form of ciphertext during data processing. It has the following advantages: + + - Protects data privacy and security throughout the lifecycle on the cloud. + - Resolves trust issues by making the public cloud, consumer cloud, and development users keep their own keys. + - Enables partners to better comply with personal privacy protection laws and regulations with the help of the full encryption capability. + +- Memory Table + + With memory tables, all data access is lock-free and concurrent, optimizing data processing and meeting real-time requirements. + +- Primary/Standby Deployment + + The primary/standby deployment mode supports synchronous and asynchronous replication. Applications are deployed based on service scenarios. For synchronous replication, one primary node and two standby nodes are deployed. This ensures reliability but affects performance. For asynchronous replication, one primary node and one standby node are deployed. This has little impact on performance, but data may be lost when exceptions occur. openGauss supports automatic recovery of damaged pages. When a page on the primary node is damaged, the damaged page can be automatically recovered on the standby node. Besides, openGauss supports concurrent log recovery on the standby node to minimize the service unavailability time when the primary node is down. + + In addition, in primary/standby deployment mode, if the read function of the standby node is enabled, the standby node supports read operations instead of write operations (such as table creation, data insertion, and data deletion), reducing the pressure on the primary node. + +- AI Capabilities + + - Automatic parameter optimization + - Slow SQL discovery + - Index recommendation + - Time series prediction and exception detection + - DB4AI function + - SQL execution time prediction + - Database monitoring + +- Logical Log Replication + + In logical replication, the primary database is called the source database, and the standby database is called the target database. The source database parses the WAL file based on the specified logical parsing rules and parses the DML operations into certain logical change information (standard SQL statements). The source database sends standard SQL statements to the target database. After receiving the SQL statements, the target database applies them to implement data synchronization. Logical replication involves only DML operations. Logical replication can implement cross-version replication, heterogeneous database replication, dual-write database replication, and table-level replication. + +- Automatic WDR Performance Analysis Report + + Periodically and proactively analyzes run logs and WDR reports (which are automatically generated in the background and can be triggered by key indicator thresholds such as the CPU usage, memory usage, and long SQL statement proportions) and generates reports in HTML and PDF formats. The performance report can be automatically generated. The WDR generates a performance report between two different time points based on the system performance snapshot data at two different time points. The report is used to diagnose database kernel performance faults. + +- Incremental Backup and Restoration (beta) + + Supports full backup and incremental backup of the database, manages backup data, and views the backup status. Supports combination of incremental backups and deletion of expired backups. The database server dynamically tracks page changes, and when a relational page is updated, the page is marked for backup. The incremental backup function requires that the GUC parameter enable_cbm_tracking be enabled to allow the server to track the modification page. + +- Point-In-Time Recovery + + Point-in-time recovery (PITR) uses basic hot backup, write-ahead logs (WALs), and archived WALs for backup and recovery. Replaying a WAL record can be stopped at any point of time, so that there is a consistent snapshot of the database at any point of time. That is, you can restore the database to the state at any time since the backup starts. During recovery, you can specify a recovery stop point with a terminal ID (TID), time, and license serial number (LSN). + +## Advantages of MogDB + +openGauss is a standalone database where data is stored on a single physical node and data access tasks are pushed to service nodes. In this way, high concurrency of servers enables quick data processing. In addition, data can be copied to the standby server through log replication, ensuring high reliability and scalability. + +openGauss is a stand-alone database. To use openGauss in formal commercial projects, you need to build complete tool chain capabilities such as database monitoring and primary/standby switchover. + +At the product level, MogDB adds MogHA enterprise-class high availability components and feature-rich graphical management tool MogDB Manager to the original functions of openGauss, and continuously enhances the openGauss kernel along the established route. MogDB can maximize the high-availability deployment capabilities of multiple machine rooms, and can reach 2.5 million tpmC on 4-CPU server. MogDB Manager contains a variety of practical components, such as MogHA, MTK, PTK, MDB, SCA, MVD, etc., which greatly makes up the shortcomings of openGauss open source database and enriches various enterprise-class functions. + +At the service level, Enmo has decades of experience in database operation and maintenance, and can provide complete services to ensure a more stable database, smoother application transformation, and less risk, making up for the disadvantages of openGauss open source database human operation and maintenance shortage, while reducing maintenance costs. diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.0.md b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.0.md new file mode 100644 index 0000000000000000000000000000000000000000..962dfcb918fb84e068d8824ac60cd778e8421fe6 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.0.md @@ -0,0 +1,193 @@ +--- +title: MogDB 3.0.0 +summary: MogDB 3.0.0 +author: Guo Huan +date: 2022-05-30 +--- + +# MogDB 3.0.0 + +## 1. Version Description + +MogDB 3.0.0 was released on June 30, 2022. It is an enhanced version based on MogDB 2.1 and incorporated with new features of openGauss 3.0.0. + +
+ +## 2. New Features + +### 2.1 Incorporated with New Features of openGauss 3.0.0 + +- [Row-store execution to vectorized execution](../../characteristic-description/high-performance/12-row-store-execution-to-vectorized-execution.md) +- [Delay of entering the maximum availability mode](../../characteristic-description/high-availability/11-delaying-entering-the-maximum-availability-mode.md) +- [Parallel logical decoding](../../characteristic-description/high-availability/12-parallel-logical-decoding.md) +- [CM (Cluster Manager)](../../reference-guide/tool-reference/unified-database-management-tool.md) +- [Global SysCache](../../characteristic-description/high-availability/15-global-syscache.md) +- [Publication-Subscription](../../characteristic-description/enterprise-level-features/16-publication-subscription.md) +- [Foreign key lock enhancement](../../characteristic-description/enterprise-level-features/17-foreign-key-lock-enhancement.md) +- [Support for ANY permission management](../../reference-guide/sql-syntax/GRANT.md) +- [DBMind componentized](../../AI-features/ai4db/ai4db-autonomous-database-o&m.md) +- [Database AI algorithms supporting XGBoost, multiclass and PCA](../../AI-features/db4ai/native-db4ai-engine.md) +- [ShardingSphere used to build a distributed database](../../characteristic-description/middleware/distributed-database-capability.md) + +### 2.2 Asynchronous Commit of Autonomous Transactions + +- Transaction and transaction logging are executed in two threads. This will not make the session worker thread idle in the transaction wait phase. Therefore, the utilization of the session worker thread is improved, and the database performance is further improved on the premise of ensuring data integrity. +- Asynchronous commit of autonomous transaction makes the overall performance of the database improved by 20% to 50% and tpmC increased steadily by 36% in the TPCC compression test. + +**Related page**: [Transaction Asynchronous Submit](../../characteristic-description/enterprise-level-features/19-transaction-async-submit.md) + +### 2.3 Parallelism Control Achieved During Index Creation + +- MogDB allows you to set the parameter to control parallelism during index creation. It makes index creation efficient and flexible. + +**Related page**: [Index Creation Parallel Control](../../characteristic-description/enterprise-level-features/23-index-creation-parallel-control.md) + +### 2.4 Dynamic Partition Pruning + +- The dynamic partition pruning function is provided on the basis of static partition pruning. +- Partition pruning is realized in bind variable queries, subqueries, and queries of parameterized nested loops, thereby improving the partition query performance. + +**Related page**: [Dynamic Partition Pruning](../../characteristic-description/enterprise-level-features/21-dynamic-partition-pruning.md) + +### 2.5 Optimized COPY (Import) + +The SIMD (single instruction, multiple data) vector processing is used combined with the CPU SSE4.2 command set to improve the COPY (import) performance. + +**Related page**: [COPY Import Optimization](../../characteristic-description/enterprise-level-features/20-copy-import-optimization.md) + +### 2.6 Monitoring the Run Status of SQLs + +- Execution plans of session-level SQLs are collected and then execution operators are dynamically sampled. +- The query performance problems are efficiently located in slow SQL queries. + +**Related page**: [SQL Running Status Observation](../../characteristic-description/enterprise-level-features/22-sql-running-status-observation.md) + +### 2.7 BRIN Index + +- Compared with BTREE indexes, BRIN (Block Range Index) is very small and scanning the index adds little overhead but the query speed is fast. BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table. BRIN index features quick index creation, small overhead, and fast scanning in sequential order. +- This feature is from PostgreSQL. + +**Related page**: [BRIN Index](../../characteristic-description/enterprise-level-features/24-brin-index.md) + +### 2.8 Bloom Index + +- Bloom index is based on bloom filter. It is applied to scenarios where a table stores a lot of columns and can be queried using any combination of columns. Compared with the traditional index, such as B+ tree, multiple B+ tree indexes need to be created in those scenarios, but one bloom index is enough. +- This feature is from PostgreSQL. + +**Related page**: [BLOOM Index](../../characteristic-description/enterprise-level-features/25-bloom-index.md) + +### 2.9 PostGIS + +PostGIS is a spatial database system. It provides spatial information service capabilities such as spatial objects, spatial indexes, spatial manipulation functions. It can be used in: + +- Atmospheric sciences, marine sciences, geology, environmental sciences, traffic management, processing and analyzing of complex spatial data, and mapping +- Mobile communications, mobile positioning, and mobile Internet +- Urban management, disaster response, resource development, etc. + +**Related page**: [PostGIS](../../reference-guide/oracle-plugins/postgis-extension/postgis-overview.md) + +### 2.10 Oracle Compatibility Enhancement + +The whale plugin is incorporated into MogDB and supports the following Oracle functions and packages: + +**Functions** + +- INSTRB +- NLS_CHARSET_ID +- NLS_CHARSET_NAME +- NLS_LOWER +- NLS_UPPER +- ORA_HASH +- REMAINDER +- REPLACE +- SHOW +- SHOW_PARAMETER +- TO_TIMESTAMP +- TO_YMINTERVAL +- TZ_OFFSET +- NULLIF +- Ratio_to_report + +**Packages** + +- dbms_lock +- dbms_job +- dbms_utility +- dbms_random +- dbms_output +- dbms_application_info +- dbms_metadata + +**Related page**: [MogDB-Oracle Compatibility](../../characteristic-description/application-development-interfaces/MogDB-Oracle-compatibility.md) + +### 2.11 MySQL Compatibility Enhancement + +The dolphin plugin is incorporated into MogDB and supports the following syntax and function: + +- Timestamp on update +- SQL mode (sql_mode_strict and sql_mode_full_group) +- User lock +- Insert function + +**Related page**: [MogDB-MySQL Compatibility](../../characteristic-description/application-development-interfaces/MogDB-MySQL-compatibility.md) + +### 2.12 Enhanced OM Troubleshooting Capability + +This troubleshooting capability is enhanced by adding collection of system information, comparing parameters and test results, and adding debugging information, thereby helping users or debugging personnel locate problems efficiently. + +This feature can be achieved by enhancing the following tools: + +- gstrace: gets targeted execution paths by adding the component switch function, thereby improving the debugging efficiency. +- gs_check: saves and compares test results on the basis of scenario check. +- gs_watch: When MogDB fails, gs_watch collects OS information, log information, and configuration files to locate the problem. +- gs_gucquery: automatically collects, export, and compare MogDB GUC values. + +**Related page**: [Fault Diagnosis](../../characteristic-description/maintainability/fault-diagnosis.md) + +
+ +## 3. Fixed Defects + +### 3.1 Incorporate Fixed Defects in openGauss 3.0.0 + +- [I4VUXG](https://gitee.com/opengauss/openGauss-server/issues/I4VUXG?from=project-issue): Fixed the data loss issue of unlogged tables. +- [I4SF5P](https://gitee.com/opengauss/openGauss-server/issues/I4SF5P?from=project-issue): Fixed the core dump issue occurred by running create extension dblink after the database is compiled and installed in the release version, and the dblink module is compiled and installed. +- [I4S74D](https://gitee.com/opengauss/openGauss-server/issues/I4S74D?from=project-issue): Fixed the issue of failing to insert data (5⁄5) into a row-store compressed table using Jmeter when the data volume is greater than 1 GB. The compression type is set to **2**. +- [I4N81J](https://gitee.com/opengauss/openGauss-server/issues/I4N81J?from=project-issue): Fixed the issue of failing to synchronize the UPDATE and DELETE operations to subscribers. +- [I4YPJQ](https://gitee.com/opengauss/openGauss-server/issues/I4YPJQ?from=project-issue): Fixed the issue of failing to insert varchar constants into MOTs using JDBC. +- [I4PF6G](https://gitee.com/opengauss/openGauss-server/issues/I4PF6G?from=project-issue): Fixed the issue of TPCC execution failure during foreign key lock enhancement and gray upgrade from 2.0.0. to 2.2.0 (not committed). +- [I4WPD1](https://gitee.com/opengauss/openGauss-server/issues/I4WPD1?from=project-issue): Fixed the issue of failing to execute simplified installation because the **openGauss-2.1.0-CentOS-64bit.tar.bz2** file is missing in the decompressed installation package. +- [I4L268](https://gitee.com/opengauss/openGauss-server/issues/I4L268?from=project-issue): Fixed the issue of incorrect system catalog **pg_partition** after the partitioned table is truncated for multiple times and then the **vacuum freeze pg_partition** command is executed. +- [I3HZJN](https://gitee.com/opengauss/openGauss-server/issues/I3HZJN?from=project-issue): Fixed the issue of incorrect date format when the **copy** command is executed. +- [I4HUXD](https://gitee.com/opengauss/openGauss-server/issues/I4HUXD?from=project-issue): Fixed the issue of failing to query the JSONB type. +- [I4QDN9](https://gitee.com/opengauss/openGauss-server/issues/I4QDN9?from=project-issue): Fixed the issue of returning a value for **select 1.79E +308\*2,cume_dist() over(order by 1.0E128\*1.2)** out of range. +- [I4PAVO](https://gitee.com/opengauss/openGauss-server/issues/I4PAVO?from=project-issue): Fixed the issue of failing to identify the **start with connect by record** subquery. +- [I4UY9A](https://gitee.com/opengauss/openGauss-server/issues/I4UY9A?from=project-issue): Fixed the issue of failing to create the default partition during list partitioning. +- [I4W3UB](https://gitee.com/opengauss/openGauss-server/issues/I4W3UB?from=project-issue): Fixed the issue of failing to obtain the view definition when the view is created using a user-defined type and the user-defined type is renamed. +- [I4WRMX](https://gitee.com/opengauss/openGauss-server/issues/I4WRMX?from=project-issue): Fixed the issue of failing to clear data in the **statement_history** table. When the database restarts and the **enable_stmt_track** parameter is disabled, no record should be found in the **statement_history** table. +- [I4WOBH](https://gitee.com/opengauss/openGauss-server/issues/I4WOBH?from=project-issue): Fixed the issue of failing to restart the database by setting GUC parameter **pagewriter_sleep** from **360000** to **2000**. + +### 3.2 Fixed Defects in MogDB 3.0.0 + +- Updating a view may cause a core dump in some scenarios. +- Creating local index tables on second-level partition tables may cause a core dump in some scenarios. +- The CONNECT BY statement may cause a core dump in some scenarios. + +
+ +## 4. Compatibility + +This version supports the following operating system and CPU architecture combinations. + +| OS | CPU Architecture | +| ------------------ | -------------------------------------------------- | +| CentOS 7.x | X86_64 (Intel, AMD, Hygon, ZHAOXIN) | +| Redhat 7.x | X86_64 (Intel, AMD, Hygon, ZHAOXIN) | +| openEuler 20.03LTS | ARM (Kunpeng), X86_64 (Intel, AMD, Hygon, ZHAOXIN) | +| Kylin V10 | ARM (Kunpeng), X86_64 (Intel, AMD, Hygon, ZHAOXIN) | +| UOS V20-D / V20-E | ARM (Kunpeng), X86_64 (Intel, AMD, Hygon, ZHAOXIN) | +| UOS V20-A | X86_64 (Intel, AMD, Hygon, ZHAOXIN) | + +## 5. Known Issues + +In the scenario where a partition table is encrypted by PBE (password-based encryption), dynamic partition pruning may possibly cause a core dump. To solve this problem, set **enable_resource_track** to **off**. This parameter has been incorporated into Recommended Parameter Settings. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.1.md b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.1.md new file mode 100644 index 0000000000000000000000000000000000000000..61bce9e5abc656de4d248a0e244dadb476ee0207 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.1.md @@ -0,0 +1,86 @@ +--- +title: MogDB 3.0.1 +summary: MogDB 3.0.1 +author: Guo Huan +date: 2022-08-04 +--- + +# MogDB 3.0.1 + +## 1. Version Description + +MogDB 3.0.1 is a patch version of MogDB 3.0.0, which is released on July 30 of 2022. It mainly involves bug fixes based on MogDB 3.0.0. + +
+ +## 2. Modified Defects + +### 2.1 Kernel + +- The database is abnormal in PBE scenarios during dynamic partition pruning. +- The database is abnormal caused by gstrace_check_dump_usess. +- The result is incorrect during type conversion of a partition expression. +- The database is abnormal when **async_submit** is set to **on** in CM environment. +- The statement is incorrect when gs_dump is used to export a table created using a type. +- An error occurs when on update is processed by gs_dump. +- A table file cannot be opened when a global temporary table is merged in a new session. +- opengauss_exporter and MogDB are incompatible. +- The DDL of a table created based on the type cannot be obtained by the pg_get_tabledef function. +- An error occurs when the pg_get_tabledef function is used to obtain the partition table tablespace in a non-owner database of the table. +- An error occurs when the exception_init function in the package is used. +- Certain data fails to be inserted when **log_min_messages** is set to **debug5**. +- pg_repack fails to be executed when a bloom index is created in a secondary partition table. +- The test that MySQL is compatible with on update timestamp memcheck fails. +- The gs_async_submit_sessions_status view does not exist in dbe_perf. +- The on update timestamp feature becomes invalid when the database is disconnected and then connected. +- The database is abnormal when the memory usage view is queried. +- pg_get_tabledef does not generate the on update current_timestamp syntax. +- The libcjson dependency file is lacked in the **lib** directory of the tools package. + +### 2.2 CM + +- The AZ-related switchover operations (-a/-A/-z [az_name]) are supported in DCF mode. +- CM_CTL occasionally failed to set the primary mode of CMS to AUTO. +- The cm_agent's forced kill mechanism caused the MogDB process in the coredump state to fail to generate the complete corefile file. + +### 2.3 Extensions + +- [whale] The dbms_output package supports serveroutput. +- [whale] The value of `dbms_utility.db_version` is `inout`. +- [whale] An extension fails to be created when `behavior_compat_options` is set to `proc_outparam_override`. +- [whale] The output value of **get_line** and **get_lines** are incorrect. +- [dolphin] The out-of-bounds problem occurs when the insert function is used. +- [orafce] The package and view in orafce that conflict with whale are deleted. The package definition is added to the function definition to support function calling in pgplsql. +- [orafce] utl_file.fopen triggers the database core, and the file name becomes null pointer. +- [orafce] The default attribute is lost during pg_repack restructuring when a table contains a column with the on update timestamp attribute. +- [wal2json] The output is incorrect during event deletion when a table contains only unique index. + +
+ +## 3. Known Issues + +- In whale, the to_timestamp function cannot convert the date type to the timestamp type. + + Temporary solution: Add the to_timestamp function with the input parameter of timestamp without time zone and the return value and type the same as those of the input parameter. + + ```sql + CREATE or replace FUNCTION pg_catalog.to_timestamp(timestamp without time zone) + RETURNS timestamp without time zone + IMMUTABLE + AS $$ + select $1; + $$ LANGUAGE sql; + ``` + +- The output of the for reverse.. loop stored procedure is inconsistent with that in Oracle. + + Temporary solution: + + 1. If MTK is used to migrate the original app code, the data type can be converted. + 2. For new code, you need to manually switch the position of the start and end values. + +- When `wal_level` is set to `hot_standby` and `recovery-target-time` specified by gs_probackup is later than the last backup set time, PITR will fail. + +- In the scenario where `query_dop` is set to 1 which is not the default value, when millions of data is queried, "ERROR: bogus varno: 65001" will be reported. + +- The tuple in a compression table is not stored by specified order, the tuple sequence is not the expected one after running the cluster command (the data is not lost, but the data is stored in a different order). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.2.md b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.2.md new file mode 100644 index 0000000000000000000000000000000000000000..02a4983224602b36e3271749073526e4b6c899e7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/3.0.2.md @@ -0,0 +1,117 @@ +--- +title: MogDB 3.0.2 +summary: MogDB 3.0.2 +author: Guo Huan +date: 2022-09-19 +--- + +# MogDB 3.0.2 + +## 1. Version Description + +MogDB 3.0.2 is a patch version of MogDB 3.0.0, which is released on September 16 of 2022. It mainly involves bug fixes based on MogDB 3.0.1. + +
+ +## 2. Function Modifications + +- Version upgrade optimization: support direct upgrade from v2.0 to v3.0 (without the need to upgrade MogDB from v2.0 to v2.1) + +- The gsql '\l' command adds the function of displaying the database. + +
+ +## 3. Modified Defects + +- Fix the defect that the database frequently prints warning log after the undo_retention_time parameter is enabled. + +- Fix the defect that an error is reported once gs_global_chain cannot be found during vacuum or analyze execution after version upgrade. + +- Fix the defect that the data in the new tablespace of the primary server cannot be synchronized to the standby server in the primary/standby stream replication. + +- Fix the defect that an error is reported when log_min_messages is set to debug5 during the operation of CREATE or SELECT. + +- Fix the defect that "error parameter when release fake relation" is continuously reported when the fields of partition table is updated. + +- Fix the defect that the database installed using PTK fails to be started after dolphin is loaded. + +- Optimize the insertion of the xlog record in the x86 system. + +- Fix the defect that the ustore zone overflows even if the ustore table is not used when the pressure of the short link service is high. + +- Fix the defect that the tuples following the cluster command are disordered due to row-level compression. + +- Fix the defect that dolphin extension reports the function lookup error for the update varchar data type under on update current_timestamp syntax. + +- Modify the function of not using asynchronous commit for transactions in vacuum. + +- Fix the hang problem probably caused by enabling thread pool. + +- Fix the hang problem probably caused because no trace data is written during the gstrace stop operation. + +- Fix the defect of repeatedly parsing data after restarting an application when an LSN error is reported in multi-transaction scenarios after parallel decoding is enabled. + +- Fix the coredump probably caused by asynchronous submission. + +- Fix the database coredump caused by manually adding row lock to the view. + +- Fix the coredump caused by parallel replay or extreme recovery in page compression scenarios. + +- Fix the coredump caused by frequent creation and deletion of a database in the primary database and the database is connected by the standby database using gsql in the primary/standby deployment scenarios. + +- Fix the coredump caused by creating a domain with the same name after dropping table of type and using the field in a package. + +
+ +## 4. Synchronized with openGauss 3.0.1 Fixes + +- Resolve the issue that threshold has been cleaned but is not automatically cleaned during partition table autovacuum. +- Fix logical decode core. +- Fix deserialize func and view. +- Fix gs_dump. +- Fix ustore bug in publication. +- DFX: add exit log in PercentileMain +- Use RetryDataFileIdOpenFile to prevent the flags from being modified. +- Set compressByteConvert/compressDiffConvert to true when defElem is set to true. +- Bugfix: mdtruncate redo can not read compress_opt. +- Bugfix: backup failed when uncompressed page in compressed table. +- Fixed the bug of join (like...including reloptions) will success even if reloptions contains compresstype compresstype can be used with (semgent=off) reloption. +- Bugfix: poor index compression ratio when use default pglz compression top standby repair. +- Bugfix: poor index compression ratio when use default pglz compression. +- change fastcheck. +- Fix partition error. +- Fix record type issue caused by composite datum. +- Fix create index info. +- Fix generated issue. +- Fix the core problem caused by undetermined return value during hash_index playback. +- Fix the issue of man-in-the-middle attack in SSL authentication mode. +- Fix the lower bound integer rounding problem of array_set function. +- gs_ctl adds -C parameter to adapt to churn tolerance. +- Repair opengauss close failed issue. +- Repair GSC no cache issue. +- Repair gs_gsc_memory_detail memory info inaccurate issue. +- Repair wal_level=archive in switchover/failover get tuple concurrently updated issue. +- Repair gsc sub thread exit error issue. +- Solve the switchover problem of pagewriter exiting. +- Solve the switchover bugs caused by the wake_sub_threads. +- [Round Main] fix bug interval partition exchange failure. +- Fix the bug that the pgplsql invokes the syntax parser error. +- Fix publish subscription deadlock issue. +- Fix the bug that the DefElem type in the logical.h file is not declared when the wal2json plugin is compiled. +- smp bugfix. +- Fix the coredump caused when the switching between the primary server and standby server is frequent. +- Change log of alter compresstype\compress_chunk_size. +- Optimize the message for create table. +- Reserve 160kb where backup compressed file. + +
+ +## 5. Known Issues + +| No. | Problem Description | Impact Analysis | Evasion and Contingency Measures | +| ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| 1 | After the database is loaded with dolphin dynamic database, some SQL operations may cause the database to be down and unable to start. | **Risk: High (impact level: high; probability of occurrence: medium)**
Users who use some SQL operations after loading dolphin dynamic database will cause data to go down and not start. | **Workaround:** After loading dolphin and creating the dolphin extension, de-load the dolphin configuration and restart the database.
**Contingency measure:** NA | +| 2 | The coredump problem probably occurred in the standby database under the primary/standby pressure test | **Risk: High (impact level: high; probability of occurrence: low)**
When the primary database frequently creates database and drops database, it will cause the standby database to load the invalid data pages into the buffer pool when it plays back the data, thus causing a coredump. | **Workaround:** Avoiding frequent database creation and drop operations can reduce the probability of occurrence, but it cannot be completely avoided.
**Contingency measure:** NA | +| 3 | In the primary/standby pressure test, the standby database may have the problem that some modified data cannot be found. | **Risk: High (impact level: high; probability of occurrence: medium)**
In some cases, querying data from the standby database will not find the data that has just been updated, and the phenomenon is similar to data being deleted. | **Workaround:** When the standby database is queried, set the hot_standby_feedback parameter to avoid the data that needs to be queried from being reclaimed prematurely by the primary database.
**Contingency measure:** NA | +| 4 | After the version upgrade, operators such as >, >=, <, <=, =, etc. can be used for point types. In general, the point type cannot use these operators. | **Risk: low (impact level: low; probability of occurrence: high)**
After the upgrade, the operators that correspond to the point type works, but an error is still reported. | **Workaround:** This type is not an ordinary data type and is not usually used. If you do need to report an error, you can manually remove the operator metadata corresponding to the type to solve it.
**Contingency measure:** NA | +| 5 | After the upgrade, the Oracle dbms_job.user_export can return results when the parameters are illegal. In general, an illegal parameter error is reported. | **Risk: low (impact level: low; probability of occurrence: high)**
After the upgrade, the arm machine can return results when the parameter is illegal. | **Workaround:** It is considered normal. If you do need to report an error here, you need to reinstall the Oracle compatibility extension.
**Contingency measure:** NA | \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/release-note.md b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/release-note.md new file mode 100644 index 0000000000000000000000000000000000000000..ec8a6ead53d2aa21757a0f7a5ca061e014b158cb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/mogdb-new-feature/release-note.md @@ -0,0 +1,14 @@ +--- +title: Release Notes +summary: Release Notes +author: Guo Huan +date: 2022-09-27 +--- + +# Release Notes + +| Version | Release Date | Overview | +| ------------------- | ------------ | ------------------------------------------------------------ | +| [3.0.2](./3.0.2.md) | 2022.09.16 | MogDB 3.0.2 is a patch version of MogDB 3.0.0. Function modifications include version upgrade optimization (support direct upgrade from v2.0 to v3.0), and the gsql '\l' command adds the function of displaying the database. MogDB 3.0.2 fixed the known defects, and synchronize the fixes for openGauss 3.0.1. | +| [3.0.1](./3.0.1.md) | 2022.07.30 | MogDB 3.0.1 is a patch version of MogDB 3.0.0, which fixed bugs of database kernel, CM and extensions. | +| [3.0.0](./3.0.0.md) | 2022.06.30 | MogDB 3.0.0 is an enhanced version based on MogDB 2.1 and incorporated with new features of openGauss 3.0.0, such as row-store execution to vectorized execution, parallel logical decoding, publication-subscription, CM (Cluster Manager). At the same time, MogDB 3.0.0 support asynchronous commit of autonomous transactions, parallelism control achieved during index creation, dynamic partition pruning, optimized COPY (Import), monitoring the run status of SQLs, Brin Index, Bloom Index and other self-developed features. | \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/1-enhanced-opengauss-kernel.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/1-enhanced-opengauss-kernel.md new file mode 100644 index 0000000000000000000000000000000000000000..ac7a28ef8cfbb457901397299607e0261a0968bd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/1-enhanced-opengauss-kernel.md @@ -0,0 +1,12 @@ +--- +title: Enhanced openGauss Kernel +summary: Enhanced openGauss Kernel +author: Liuxu +date: 2021-06-09 +--- + +# Enhanced openGauss Kernel + +
+ +MogDB is an enterprise-level database which is more friendly to enterprise applications, encapsulated and improved on the basis of openGauss open source kernel. Based on the openGauss kernel, MogDB adds MogHA, TimeSeries (for IoT scenarios), Distributed (realize each data node as an openGauss kernel distributed environment), Oracle view compatibility (DBA and V$ views) and other plug-ins. The MogHA plug-in is used for automated HA management in the primary/standby architecture, which is crucial for enterprise applications. At the same time, MogDB Manager management software is developed, which includes backup and recovery, performance monitoring, health check toolkit, automated deployment, database migration, data synchronization, compatibility analysis, performance pressure test and data recovery from a corrupted database for a variety of enterprise-level usability requirements. diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/2-docker-based-mogdb.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/2-docker-based-mogdb.md new file mode 100644 index 0000000000000000000000000000000000000000..8a59bbdaa4ce4ea357defea71ce3d9e2f5dd5971 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/2-docker-based-mogdb.md @@ -0,0 +1,37 @@ +--- +title: Container-based MogDB +summary: Container-based MogDB +author: Liuxu +date: 2021-06-09 +--- + +# Container-based MogDB + +
+ +## Features + +- As the version of MogDB changes, release the image of the new version as soon as possible. +- The container version of the database image has built-in configuration of initialization parameters for best practices. +- The container version database supports both x86 and ARM CPU architectures. +- MogDB 2.1 container version supports the latest version of compat-tools and plugin functions. + +**Currently, x86-64 and ARM64 architectures are supported. Please get the corresponding container image according to the machine architecture of your host.** + +Starting from version 2.0 (including version 2.0) + +- MogDB of the x86-64 architecture is run on the [Ubuntu 18.04 operating system](https://ubuntu.com/). +- MogDB of the ARM64 architecture is run on the [Debian 10 operating system](https://www.debian.org/). + +Before version 1.1.0 (including version 1.1.0) + +- MogDB of the x86-64 architecture is run on the [CentOS7.6 operating system](https://www.centos.org/). +- MogDB of the ARM64 architecture is run on the [openEuler 20.03 LTS operating system](https://openeuler.org/en/). + +
+ +## How to Use an Image? + +For details, visit the following website: + +[Installation Guide - Container-based Installation](../../installation-guide/docker-installation/docker-installation.md) diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/5-wal2json-extention-for-mogdb&opengauss.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/5-wal2json-extention-for-mogdb&opengauss.md new file mode 100644 index 0000000000000000000000000000000000000000..88e2d0aa7a814ae9bebad1f11f63c1047674a9bd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/5-wal2json-extention-for-mogdb&opengauss.md @@ -0,0 +1,125 @@ +--- +title: wal2json Extention for MogDB&openGauss +summary: wal2json Extention for MogDB&openGauss +author: Guo Huan +date: 2021-06-02 +--- + +# wal2json Extention for MogDB&openGauss + +## How to Get the Component + + + +
+ +## Logical Decoding (wal2json) + +With the wal2json plug-in of MogDB and openGauss, you can export logical log files in the JSON format. + +
+ +## Prerequisites + +You have set **wal_level** to **logical**. + +
+ +## Background Information + +wal2json is a logical decoding plug-in. It can be used for accessing tuples generated by INSERT and UPDATE and parsing WAL logs. + +wal2json generates a JSON object in each transaction. JSON provides all new and old tuples as well as other optional properties, such as transaction timestamp, qualified schema, data type, and transaction ID. + +
+ +## Obtaining JSON Objects Using SQL + +1. Log in to the MogDB database. + +2. Run the following commands to create tables and initialize the plug-in. + + ```sql + --Open a new session and run the following: + pg_recvlogical -d mogdb --slot test_slot --create-slot -P wal2json + pg_recvlogical -d mogdb --slot test_slot --start -o pretty-print=1 -f - + --Perform the following basic DML operations: + CREATE TABLE test_table ( + id char(10) NOT NULL, + code char(10), + PRIMARY KEY (id) + ); + mogdb=# INSERT INTO test_table (id, code) VALUES('id1', 'code1'); + INSERT 0 1 + mogdb=# update test_table set code='code2' where id='id1'; + UPDATE 1 + mogdb=# delete from test_table where id='id1'; + DELETE 1 + ``` + + DML output: + + **INSERT** + + ```json + { + "change": [ + { + "kind": "insert", + "schema": "mdmv2", + "table": "test_table", + "columnnames": ["id", "code"], + "columntypes": ["character(10)", "character(10)"], + "columnvalues": ["id1 ", "code1 "] + } + ] + } + ``` + + **UPDATE** + + ```json + { + "change": [ + { + "kind": "update", + "schema": "mdmv2", + "table": "test_table", + "columnnames": ["id", "code"], + "columntypes": ["character(10)", "character(10)"], + "columnvalues": ["id1 ", "code2 "], + "oldkeys": { + "keynames": ["id"], + "keytypes": ["character(10)"], + "keyvalues": ["id1 "] + } + } + ] + } + ``` + + **DELETE** + + ```json + { + "change": [ + { + "kind": "delete", + "schema": "mdmv2", + "table": "test_table", + "oldkeys": { + "keynames": ["id"], + "keytypes": ["character(10)"], + "keyvalues": ["id1 "] + } + } + ] + } + ``` + + REPLICA IDENTITY can determine the details of the exported logical logs when the UPDATE and DELETE operations are performed on a table. + + - DEFAULT: A logical log includes the original values of the primary key columns when the value is updated or deleted. + - NOTHING: A logical log does not include any update or delete information. + - FULL: A logical log include original information of the whole row to which the value belongs when the value in a table is updated or deleted. + - USING INDEX: includes only original values of all columns in specified indexes. diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/DBMS-RANDOM.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/DBMS-RANDOM.md new file mode 100644 index 0000000000000000000000000000000000000000..9464e7f1128c56530ddf672e97f858240148046b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/DBMS-RANDOM.md @@ -0,0 +1,531 @@ +--- +title: DBMS_RANDOM - Generating Random Data (Numbers, Strings and Dates) in MogDB with compat-tools +summary: DBMS_RANDOM - Generating Random Data (Numbers, Strings and Dates) in MogDB with compat-tools +author: Zhang Cuiping +date: 2021-08-30 +--- + +# DBMS_RANDOM - Generating Random Data (Numbers, Strings and Dates) in MogDB with compat-tools + +
+ +## Introduction to compat-tools + +Compat-tools is a set of compatibility tools. It aims to provide compatibility for necessary functions and system views tha are created for OSs migrated from other asynchronous databases to MogDB, thereby facilitating the follow-up system maintenance and application modification. + +
+ +## compat-tools Download + +To install compat-tools, please download the tool of the latest version from [https://gitee.com/enmotech/compat-tools](https://gitee.com/enmotech/compat-tools). + +
+ +## Features of compat-tools + +1. runMe.sql: General scheduling script +2. Oracle_Views.sql: Compatible with Oracle database data dictionaries and views +3. Oracle_Functions.sql: Compatible with Oracle database functions +4. Oracle_Packages.sql: Compatible with Oracle database packages +5. MySQL_Views.sql: Compatible with MySQL database data dictionaries and views //TODO +6. MySQL_Functions.sql: Compatible with MySQL database functions //TODO + +
+ +## MogDB Versions Supported By compat-tools + +- MogDB 2.0 +- MogDB 1.1 + +
+ +## Installing and Using compat-tools + +1. Download compat-tools: +2. Store the downloaded files to a customized directory (**/opt/compat-tools-0902** is taken as an example in this article). + + ```bash + [root@mogdb-kernel-0005 compat-tools-0902]# pwd + /opt/compat-tools-0902 + [root@mogdb-kernel-0005 compat-tools-0902]# ls -l + total 228 + -rw-r--r-- 1 root root 9592 Sep 2 14:40 LICENSE + -rw-r--r-- 1 root root 0 Sep 2 14:40 MySQL_Functions.sql + -rw-r--r-- 1 root root 0 Sep 2 14:40 MySQL_Views.sql + -rw-r--r-- 1 root root 41652 Sep 2 14:40 Oracle_Functions.sql + -rw-r--r-- 1 root root 34852 Sep 2 14:40 Oracle_Packages.sql + -rw-r--r-- 1 root root 125799 Sep 2 14:40 Oracle_Views.sql + -rw-r--r-- 1 root root 4708 Sep 2 14:40 README.md + -rw-r--r-- 1 root root 420 Sep 2 14:40 runMe.sql + ``` + +3. Switch to user `omm`. + + ```bash + su - omm + ``` + +4. Run the following command (26000 is the port for connnecting the database). + + ```bash + gsql -d mogdb -p 26000 -f /opt/compat-tools-0902/runMe.sql + ``` + +
+ +## Testing DBMS_RANDOM + +### Log In to the mogdb Database + +```sql +[omm@mogdb-kernel-0005 ~]$ gsql -d mogdb -p 26000 +gsql ((MogDB x.x.x build 56189e20) compiled at 2022-01-07 18:47:53 commit 0 last mr ) +Non-SSL connection (SSL connection is recommended when requiring high-security) +Type "help" for help. + +mogdb=# +``` + +
+ +- [SEED](#seed) +- [VALUE](#value) +- [STRING](#string) +- [NORMAL](#normal) +- [RANDOM](#random) +- [Generating Random Dates](#generating-random-dates) +- [Generating Random Data](#generating-random-data) + +
+ +## SEED + +The `SEED` procedure allows you to seed the pseudo-random number generator, making it more random. `SEED` is limited to binary integers or strings up to 2000 characters. If you want to consistently generate the same set of pseudo-random numbers, always use the same seed. + +```sql +declare +BEGIN + DBMS_OUTPUT.put_line('Run 1 : seed=0'); + DBMS_RANDOM.seed (val => 0); + FOR i IN 1 ..5 LOOP + DBMS_OUTPUT.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10)); + END LOOP; + + DBMS_OUTPUT.put_line('Run 2 : seed=0'); + DBMS_RANDOM.seed (val => 0); + FOR i IN 1 ..5 LOOP + DBMS_OUTPUT.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10)); + END LOOP; + +END; +/ +NOTICE: Run 1 : seed=0 +CONTEXT: SQL statement "CALL dbms_output.put_line('Run 1 : seed=0')" +PL/pgSQL function inline_code_block line 3 at PERFORM +NOTICE: i=1 : value=2.53745232429355 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: i=2 : value=7.749117821455 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: i=3 : value=1.86734489817172 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: i=4 : value=8.83418704243377 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: i=5 : value=6.19573155790567 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: Run 2 : seed=0 +CONTEXT: SQL statement "CALL dbms_output.put_line('Run 2 : seed=0')" +PL/pgSQL function inline_code_block line 9 at PERFORM +NOTICE: i=1 : value=2.53745232429355 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: i=2 : value=7.749117821455 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: i=3 : value=1.86734489817172 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: i=4 : value=8.83418704243377 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: i=5 : value=6.19573155790567 +CONTEXT: SQL statement "CALL dbms_output.put_line('i=' || i || ' : value=' || DBMS_RANDOM.value(low => 1, high => 10))" +PL/pgSQL function inline_code_block line 12 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +
+ +## VALUE + +The `VALUE` function is used to produce random numbers with a specified range. When called without parameters it produce a number greater than or equal to 0 and less than 1, with 38 digit precision. + +```sql +DECLARE +BEGIN + FOR cur_rec IN 1 ..5 LOOP + DBMS_OUTPUT.put_line('value= ' || DBMS_RANDOM.value()); + END LOOP; +END; +/ +NOTICE: value= .785799258388579 +CONTEXT: SQL statement "CALL dbms_output.put_line('value= ' || DBMS_RANDOM.value())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value= .692194153089076 +CONTEXT: SQL statement "CALL dbms_output.put_line('value= ' || DBMS_RANDOM.value())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value= .368766269646585 +CONTEXT: SQL statement "CALL dbms_output.put_line('value= ' || DBMS_RANDOM.value())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value= .87390407640487 +CONTEXT: SQL statement "CALL dbms_output.put_line('value= ' || DBMS_RANDOM.value())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value= .745095098391175 +CONTEXT: SQL statement "CALL dbms_output.put_line('value= ' || DBMS_RANDOM.value())" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +``` + +If the parameters are used, the resulting number will be greater than or equal to the low value and less than the high value, with the precision restricted by the size of the high value. + +```sql +declare +BEGIN + FOR cur_rec IN 1 ..5 LOOP + DBMS_OUTPUT.put_line('value(1,100)= ' || DBMS_RANDOM.value(1,100)); + END LOOP; +END; +/ + +NOTICE: value(1,100)= 45.158544998616 +CONTEXT: SQL statement "CALL dbms_output.put_line('value(1,100)= ' || DBMS_RANDOM.value(1,100))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value(1,100)= 36.0190920610912 +CONTEXT: SQL statement "CALL dbms_output.put_line('value(1,100)= ' || DBMS_RANDOM.value(1,100))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value(1,100)= 73.5194435422309 +CONTEXT: SQL statement "CALL dbms_output.put_line('value(1,100)= ' || DBMS_RANDOM.value(1,100))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value(1,100)= 26.7619780991226 +CONTEXT: SQL statement "CALL dbms_output.put_line('value(1,100)= ' || DBMS_RANDOM.value(1,100))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: value(1,100)= 40.035083710216 +CONTEXT: SQL statement "CALL dbms_output.put_line('value(1,100)= ' || DBMS_RANDOM.value(1,100))" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +Use `TRUNC` or `ROUND` to alter the precision as required. For example, to produce random integer values between 1 and 10 truncate the output and add 1 to the upper boundary. + +```sql +mogdb=# select TRUNC(DBMS_RANDOM.value(1,11)) ; + + trunc +------- + + 6 + +(1 row) + +mogdb=# +``` + +
+ +## STRING + +The `STRING` function returns a string of random characters of the specified length. The `OPT` parameter determines the type of string produced as follows: + +- 'u', 'U' - uppercase alpha characters +- 'l', 'L' - lowercase alpha characters +- 'a', 'A' - mixed case alpha characters +- 'x', 'X' - uppercase alpha-numeric characters +- 'p', 'P' - any printable characters + +The `LEN` parameter, not surprisingly, specifies the length of the string returned. + +```sql +declare +BEGIN + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('string(''x'',10)= ' || DBMS_RANDOM.string('x',10)); + END LOOP; +END; +/ + +NOTICE: string('x',10)= i5S6XOZxrA +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''x'',10)= ' || DBMS_RANDOM.string('x',10))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('x',10)= HGvRm75w19 +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''x'',10)= ' || DBMS_RANDOM.string('x',10))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('x',10)= N9WsQGJl6l +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''x'',10)= ' || DBMS_RANDOM.string('x',10))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('x',10)= hDlPevVgRb +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''x'',10)= ' || DBMS_RANDOM.string('x',10))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('x',10)= ZdSd8x8RKx +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''x'',10)= ' || DBMS_RANDOM.string('x',10))" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +Combine the `STRING` and `VALUE` functions to get variable length strings. + +```sql +declare +BEGIN + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('string(''L'',?)= ' || DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,21)))); + END LOOP; +END; +/ + +NOTICE: string('L',?)= kcyzowdxqbyzu +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''L'',?)= ' || DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,21))))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('L',?)= ohzpljyatsplqtbbus +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''L'',?)= ' || DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,21))))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('L',?)= hbrjsfeevoi +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''L'',?)= ' || DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,21))))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('L',?)= lfsapmytdamvwcw +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''L'',?)= ' || DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,21))))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: string('L',?)= pcvtxnwzomkqwpfzes +CONTEXT: SQL statement "CALL dbms_output.put_line('string(''L'',?)= ' || DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,21))))" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +
+ +## NORMAL + +The `NORMAL` function returns random numbers in a normal distribution. + +```sql +declare +BEGIN + FOR cur_rec IN 1 ..5 LOOP + DBMS_OUTPUT.put_line('normal= ' || DBMS_RANDOM.normal()); + END LOOP; +END; +/ + +NOTICE: normal= .838851847718988 +CONTEXT: SQL statement "CALL dbms_output.put_line('normal= ' || DBMS_RANDOM.normal())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: normal= -.523612260373397 +CONTEXT: SQL statement "CALL dbms_output.put_line('normal= ' || DBMS_RANDOM.normal())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: normal= -.241931681458075 +CONTEXT: SQL statement "CALL dbms_output.put_line('normal= ' || DBMS_RANDOM.normal())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: normal= -.120847761874286 +CONTEXT: SQL statement "CALL dbms_output.put_line('normal= ' || DBMS_RANDOM.normal())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: normal= .360125112757284 +CONTEXT: SQL statement "CALL dbms_output.put_line('normal= ' || DBMS_RANDOM.normal())" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +
+ +## RANDOM + +```sql +declare +BEGIN + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('random= ' || DBMS_RANDOM.random()); + END LOOP; +END; +/ +NOTICE: This function is deprecated with Release 11gR1, although currently supported, it should not be used. +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: random= -1023930867 +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: This function is deprecated with Release 11gR1, although currently supported, it should not be used. +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: random= 1068572119 +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: This function is deprecated with Release 11gR1, although currently supported, it should not be used. +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: random= 95361253 +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: This function is deprecated with Release 11gR1, although currently supported, it should not be used. +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: random= -712638729 +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: This function is deprecated with Release 11gR1, although currently supported, it should not be used. +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: random= -1251059926 +CONTEXT: SQL statement "CALL dbms_output.put_line('random= ' || DBMS_RANDOM.random())" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +
+ +## Generating Random Dates + +There are no specific functions for generating random dates currently, but we can add random numbers to an existing date to make it random. The following example generates random dates over the next year. + +```sql +declare +BEGIN + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('date= ' || TRUNC(SYSDATE + DBMS_RANDOM.value(0,366))); + END LOOP; +END; +/ + +NOTICE: date= 2021-10-06 00:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('date= ' || TRUNC(SYSDATE + DBMS_RANDOM.value(0,366)))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: date= 2022-05-09 00:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('date= ' || TRUNC(SYSDATE + DBMS_RANDOM.value(0,366)))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: date= 2022-04-07 00:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('date= ' || TRUNC(SYSDATE + DBMS_RANDOM.value(0,366)))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: date= 2021-11-29 00:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('date= ' || TRUNC(SYSDATE + DBMS_RANDOM.value(0,366)))" +PL/pgSQL function inline_code_block line 4 at PERFORM +NOTICE: date= 2022-06-04 00:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('date= ' || TRUNC(SYSDATE + DBMS_RANDOM.value(0,366)))" +PL/pgSQL function inline_code_block line 4 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +By doing the correct divisions, we can add random numbers of hours, seconds or minutes to a date. + +```sql +DECLARE + l_hours_in_day NUMBER := 24; + l_mins_in_day NUMBER := 24*60; + l_secs_in_day NUMBER := 24*60*60; +BEGIN + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('hours= ' || (TRUNC(SYSDATE) + (TRUNC(DBMS_RANDOM.value(0,1000))/l_hours_in_day))); + END LOOP; + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('mins = ' || (TRUNC(SYSDATE) + (TRUNC(DBMS_RANDOM.value(0,1000))/l_mins_in_day))); + END LOOP; + FOR i IN 1 .. 5 LOOP + DBMS_OUTPUT.put_line('secs = ' || (TRUNC(SYSDATE) + (TRUNC(DBMS_RANDOM.value(0,1000))/l_secs_in_day))); + END LOOP; +END; +/ +NOTICE: hours= 2021-10-13 22:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('hours= ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_hours_in_day)))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: hours= 2021-10-10 00:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('hours= ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_hours_in_day)))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: hours= 2021-09-07 02:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('hours= ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_hours_in_day)))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: hours= 2021-09-26 11:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('hours= ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_hours_in_day)))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: hours= 2021-09-19 22:00:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('hours= ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_hours_in_day)))" +PL/pgSQL function inline_code_block line 6 at PERFORM +NOTICE: mins = 2021-09-04 00:01:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('mins = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_mins_in_day)))" +PL/pgSQL function inline_code_block line 9 at PERFORM +NOTICE: mins = 2021-09-04 11:56:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('mins = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_mins_in_day)))" +PL/pgSQL function inline_code_block line 9 at PERFORM +NOTICE: mins = 2021-09-04 00:53:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('mins = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_mins_in_day)))" +PL/pgSQL function inline_code_block line 9 at PERFORM +NOTICE: mins = 2021-09-04 00:21:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('mins = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_mins_in_day)))" +PL/pgSQL function inline_code_block line 9 at PERFORM +NOTICE: mins = 2021-09-04 12:38:00 +CONTEXT: SQL statement "CALL dbms_output.put_line('mins = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_mins_in_day)))" +PL/pgSQL function inline_code_block line 9 at PERFORM +NOTICE: secs = 2021-09-04 00:10:28 +CONTEXT: SQL statement "CALL dbms_output.put_line('secs = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_secs_in_day)))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: secs = 2021-09-04 00:15:31 +CONTEXT: SQL statement "CALL dbms_output.put_line('secs = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_secs_in_day)))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: secs = 2021-09-04 00:09:07 +CONTEXT: SQL statement "CALL dbms_output.put_line('secs = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_secs_in_day)))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: secs = 2021-09-04 00:06:54 +CONTEXT: SQL statement "CALL dbms_output.put_line('secs = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_secs_in_day)))" +PL/pgSQL function inline_code_block line 12 at PERFORM +NOTICE: secs = 2021-09-04 00:06:32 +CONTEXT: SQL statement "CALL dbms_output.put_line('secs = ' || (TRUNC(SYSDATE)+ (TRUNC(DBMS_RANDOM.value(0,1000))/l_secs_in_day)))" +PL/pgSQL function inline_code_block line 12 at PERFORM +ANONYMOUS BLOCK EXECUTE +mogdb=# +``` + +
+ +## Generating Random Data + +The `DBMS_RANDOM` package is useful for generating random test data. You can generate large amounts quickly by combining it into a query. + +```sql +mogdb=# CREATE TABLE random_data ( + id NUMBER, + small_number NUMBER(5), + big_number NUMBER, + short_string VARCHAR2(50), + long_string VARCHAR2(400), + created_date DATE, + CONSTRAINT random_data_pk PRIMARY KEY (id) +); +NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "random_data_pk" for table "random_data" +CREATE TABLE +mogdb=# +``` + +```sql +mogdb=# INSERT INTO random_data +SELECT generate_series(1,29999), + TRUNC(DBMS_RANDOM.value(1,5)) AS small_number, + TRUNC(DBMS_RANDOM.value(100,10000)) AS big_number, + DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(10,50))) AS short_string, + DBMS_RANDOM.string('L',TRUNC(DBMS_RANDOM.value(100,400))) AS long_string, + TRUNC(SYSDATE + DBMS_RANDOM.value(0,366)) AS created_date; +INSERT 0 29999 +mogdb=# +mogdb=# select count(*) from random_data; + count +------- + 29999 +(1 row) + +mogdb=# +``` diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/compat-tools.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/compat-tools.md new file mode 100644 index 0000000000000000000000000000000000000000..cf4fa7b631183ddd2b60d4792e6a59a636664fac --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/compat-tools.md @@ -0,0 +1,18 @@ +--- +title: compat-tools +summary: compat-tools +author: Zhang Cuiping +date: 2021-07-14 +--- + +# compat-tools + +This project is a set of compatibility tools. It aims to provide compatibility for necessary functions and system views created for OSs migrated from other asynchronous databases to MogDB, thereby facilitating the follow-up system maintenance and application modification. + +The script is executed based on the version information. When you execute the script, it will be executed in terms of the following three situations: + +1. If the object to be created does not exist in the target database, it will be directly created. +2. If the version of the object to be created is later than that of the object in the target database, the target database will be upgraded and has the object re-created. +3. If the version of the object to be created is earlier than that of the object in the target database, the creation operation will be skipped. + +Please refer to [compat-tools repository page](https://gitee.com/enmotech/compat-tools) for details on how to obtain and use component. diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mog_filedump.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mog_filedump.md new file mode 100644 index 0000000000000000000000000000000000000000..d7a26a02f2f469f644733f7c7c857d30fb084e50 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mog_filedump.md @@ -0,0 +1,106 @@ +--- +title: mog_filedump User Guide +summary: mog_filedump User Guide +author: Guo Huan +date: 2021-11-15 +--- + + + +# mog_filedump User Guide + +## Introduction + +mog_filedump is a tool for parsing data files ported to MogDB based on the improved compatibility of the pg_filedump tool, which is used to convert MogDB heap/index/control files into user-readable format content. This tool can parse part of the fields in the data columns as needed, and can also dump the data content directly in binary format. The tool can automatically determine the type of the file by the data in the blocks of the file. The **-c** option must be used to format the pg_control file. + +
+ +## Principle + +The implementation steps are divided into three main steps. + +1. Reads a data block from a data file. + +2. Parse the data of the corresponding type with the callback function of the corresponding data type. + +3. Call the output of the corresponding data type function to print the data content. + +
+ +## Enmo's Improvements + +1. Compatibility porting to MogDB. + +2. Fix official bug: parsing bug of data type **char**. + +3. Fix official bug: In the multi-field scenario, parsing the data file, the data type name will cause a data length mismatch bug. + +
+ +## Installation + +Visit [MogDB official website download page](https://www.mogdb.io/en/downloads/mogdb/) to download the corresponding version of the toolkit, and put the tool in the **bin** directory of the MogDB installation path. As shown below, toolkits-xxxxxx.tar.gz is the toolkit that contains mog_filedump. + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/open-source-components-4.png) + +
+ +## How to Use + +``` +mog_filedump [-abcdfhikxy] [-R startblock [endblock]] [-D attrlist] [-S blocksize] [-s segsize] [-n segnumber] file +``` + +Valid options for **heap** and **index** files are as follows: + +| options | function | +| ---- | ------------------------------------------------------------ | +| -a | show absolute path | +| -b | output a range of binary block images | +| -d | output file block content | +| -D | The data type of the table.
Currently supported data types are: bigint, bigserial, bool, charN, date, float, float4, float8, int, json, macaddr, name, oid, real, serial, smallint, smallserial, text, time, timestamp, timetz, uuid, varchar, varcharN, xid, xml, ~.
'~' means ignore all the following data types, for example, the tuple has 10 columns, `-D first three column data types, ~` means that only the first three columns of the table tuple are parsed. | +| -f | Output and parse the content of the data block | +| -h | Display instructions and help information | +| -i | Output and parse item details (including XMIN, XMAX, Block Id, linp Index, Attributes, Size, infomask) | +| -k | Verify the checksum of the data block | +| -R | Parse and output the data file contents for the specified LSN range, e.g. **-R startblock [endblock]**. If only has **startblock** and no **endblock**, only output a single data block content | +| -s | Set segment size | +| -n | Set the number of segments | +| -S | Set data block size | +| -x | Parse and output block items as index item format (included by default) | +| -y | Parse and output block items as heap item format (included by default) | + +The options available for the control file are as follows: + +| options | function | +| ------- | ---------------------------------------------- | +| -c | List of directories for parsing control files | +| -f | Output and parse the content of the data block | +| -S | Sets the block size that controls file parsing | + +You can combine the **-i** and **-f** parameters to get more effective data to help operation and maintenance personnel analyze and refer to. + +
+ +## Examples + +The test table basically covers the data types contained in mog_filedump. + +Here is a use case to show the data parsing function. Please add other parameters according to actual needs. + +```sql +-- Create table test: +create table test(serial serial, smallserial smallserial, bigserial bigserial, bigint bigint, bool bool, char char(3), date date, float float, float4 float4, float8 float8, int int, json json, macaddr macaddr, name name, oid oid, real real, smallint smallint, text text, time time, timestamp timestamp, timetz timetz, uuid uuid, varchar varchar(20), xid xid, xml xml); + +-- Insert data: +insert into test(bigint, bool, char, date, float, float4, float8, int, json, macaddr, name, oid, real, smallint, text, time, timestamp, timetz, uuid, varchar, xid, xml) values(123456789, true, 'abc', '2021-4-02 16:45:00', 3.1415926, 3.1415926, 3.14159269828412, 123456789, '{"a":1, "b":2, "c":3}'::json, '04-6C-59-99-AF-07', 'lvhui', 828243, 3.1415926, 12345, 'text', '2021-04-02 16:48:23', '2021-04-02 16:48:23', '2021-04-02 16:48:23', 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11', 'adsfghkjlzc', '9973::xid', 'Book0001'); + +-- The directory where the data files of the query table test are located. The data directory specified by gs_initdb here is db_p. So the table test data file is in db_p/base/15098/32904 +mogdb=# select pg_relation_filepath('test'); +base/15098/32904 (1 row) + +-- Use the mog_filedump tool to parse the data file content: +./mog_filedump -D serial,smallserial,bigserial,bigint,bool,charN,date,float,float4,float8,int,json,macaddr,name,oid,real,smallint,text,time,timestamp,timetz,uuid,varchar,xid,xml db_p/base/15098/32904 +``` + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/reference-guide/mog_filedump.png) diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mog_xlogdump.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mog_xlogdump.md new file mode 100644 index 0000000000000000000000000000000000000000..50669811ff83805617cccaf97ef5593fa4818645 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mog_xlogdump.md @@ -0,0 +1,319 @@ +--- +title: mog_xlogdump User Guide +summary: mog_xlogdump User Guide +author: Guo Huan +date: 2021-11-15 +--- + +# mog_xlogdump User Guide + +## Introduction + +mog_xlogdump is an offline parsing tool for wal logs independently developed by Enmo. It is mainly used in the active-standby cluster scenario, when the database is permanently down and cannot be recovered, reversely analyze the database that cannot be started, and then recover the data that is not synchronized at the end of the wal log in the cluster. + +
+ +## R&D Background + +In MogDB primary/standby high availability cluster with one primary database and multiple standby databases, using asynchronous logical replication scenario, when the primary shuts down and its transaction is committed, the data of the transaction operation has been written to the wal log. After the primary is down, the standby will generate incomplete data segment logs because it cannot send to the standby. Therefore, after the primary is down, there is data loss and no logical alignment between the standby and the primary. So there is a risk of data loss in the primary/standby cluster composed of its standby databases and the data in the actual business. + +During the recovery of the primary database, the cluster composed of the standby database will have business data written. At this time, when the primary database is restored immediately, the segment number, start and end position of the incomplete data segment at the end of the wal log of the primary database are the same as those in the standby database. The inconsistency will also cause the data lost when the primary database shuts down and cannot be restored to the standby database. + +
+ +## Scenario + +In a MogDB high-availability cluster, when the primary database is down, the walbuffer is triggered to write to the wal log when the walbuffer is filled with a certain percentage, or when the checkpoint or commit occurs. Due to the database downtime, the logically synchronized WalSender thread stops sending logs, and the standby database receives incomplete data segment wal logs. At this time, you need to use the flashback tool to read the data blocks in the wal log of the primary database, and decode the SQL statement corresponding to the data operation, so that the DBA can analyze whether the data is valuable and restore it to the standby database. + +
+ +## Principle + +The tool uses two mechanisms, one is the header parsing of the wal log, and the other is the logical replication mechanism. + +The implementation steps are mainly divided into three steps: + +1. Read the wal log file and parse the header. + +2. Read the data in turn, and de-decode the data. + +3. According to the different data types of the data, different types of function outputs are called back. + +
+ +## Supported Table Types for Parsing + +Partitioned and normal tables are currently supported. + +
+ +## Supported Data Types for Parsing + +bool, bytea, char, name, int8, int2, int, text, oid, tid, xid, cid, xid32, clob, float4, float8, money, inet, varchar, numeric, int4; + +> 注意：Note: Since mog_xlogdump is an offline wal parsing tool, it does not currently support large data types (clob, etc.) that require toast data. The next version will support offline parsing of toast table files. + +
+ +## Installation + +Visit [MogDB official website download page](https://www.mogdb.io/en/downloads/mogdb/) to download the corresponding version of the toolkit, and put the tool in the **bin** directory of the MogDB installation path. As shown below, toolkits-xxxxxx.tar.gz is the toolkit that contains mog_xlogdump. + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/about-mogdb/open-source-components-4.png) + +
+ +## Instructions for Use + +mog_xlogdump is a tool for parsing and displaying MogDB 2.1 or later WAL logs. Auxiliary tool designed to help DBAs analyze and debug database problems. + +The mog_xlogdump parsing tool currently does not support column table. (The research found that the column-stored table will generate two corresponding CUDesc and delta tables in cstore mode. CUDesc is the metadata of the column-stored table, and delta is the temporary table of the column-stored table, which is a row-stored table. These two tables (CUDesc, delta) will be written in the wal log. Although the wal log can parse the corresponding delta table, the table is controlled by the table attribute threshold deltarow_threshold. The default is 100, that is, less than 100 pieces of data will be written in the delta table. Write directly to the cu file) + +> Note: To write the delta table in the column-store table, you need to enable the parameter **enable_delta_store = on** in postgres.conf. + +
+ +## How to Use + +``` +mog_xlogdump [OPTION]... [STARTSEG [ENDSEG]] +``` + +
+ +## Options + +- -b, --bkp-details + + Details of output file blocks. (By default, the id of the block, rel, fork, blk, and lastlsn is displayed, and this parameter will display the Block Image) + +- -B, --bytea_output + + Specify the display format of bytea type decoding output, there are binary and character formats + +- -c --connectinfo + + Specify a connect string URL, such as postgres://user:password@ip:port/dbname + +- -e, --end=RECPTR + + Specify the end position for parsing the wal log, LSN number + +- -f, --follow + + Indicates that when the specified wal log is parsed to the end, continue parsing to the next file + +- -n, --limit=N + + Specify the number of output data records + +- -o, --oid=OID + + Specifies the OID of the inverse decoding table + +- -p, --path=PATH + + Specify the wal log storage directory + +- -R, --Rel=Relation + + Specifies the data type of the inverse decoding table + +- -r, --rmgr=RMGR + + Show only the contents of records generated by the explorer + +- -s, --start=RECPTR + + Specify the starting position for parsing the wal log, LSN number + +- -T, --CTimeZone_ENV + + Specify the time zone, the default is UTC. + +- -t, --timeline=TLI + + Specify the timeline to start reading the wal log + +- -V, --version + + show version number + +- -w, --write-FPW + + Display the information written on the full page, use with -b + +- -x, --xid=XID + + Output the record with the specified transaction ID + +- -z, --stats + + Output statistics of inserted records + +- -v, --verbose + + show verbose + +- -?, --help + + show help information and exit + +
+ +## Use Case 1 + +### Scenario + +When the primary database is down and cannot be recovered, the standby database can be connected normally. At this time, the wal log sent by the primary database may contain tens of thousands of table data operations, and the mog_xlogdump tool needs to start and end according to the specified -s, -e (starting and the ending lsn position), parse out all data operations of the table. + +### Instruction + +``` +mog_xlogdump -c -s -e +``` + +### Parse Settings + +Note: The main purpose is to record old data in the wal log, that is, the data tuple before the update operation is modified, and the data deleted by the delete operation. + +1. Set the **wal_level** in the database configuration file postgres.conf to the **logical** level. +2. Alter table: `alter table table_name replica identity full;` + +### Result + +Output the wal log data parsing result in json format. The tuple display format is `'column name':'data'` + +```json +{'table_name':'xxx','schema_name':'yyy','action':'insert','tuple':{'name':'xx','id':'ss'}} +``` + +### Example + +![fe1b12d080accfb9e54f857e79baebc](https://cdn-mogdb.enmotech.com/docs-media/mogdb/reference-guide/mog_xlogdump-1.png) + +The red box is the old data that will be parsed according to the parsing settings. If there is no setting, the old data of update and delete will not be parsed. + +The standby connect URL after -c is `postgres://test:Test123456@172.16.0.44:5003/postgres` + +- postgres:// + + connect string tag header + +- test + + connect username + +- Test123456 + + The password of the connect user + +- 172.16.0.44 + + The IP address of the standby node + +- 5003 + + Standby connect port + +- postgres + + The database name of the connect standby node + +- db_p/pg_xlog/000000010000000000000004 + + The wal log file of primary node + +
+ +## Use Case 2 + +### Scenario + +When the primary database is down and cannot be recovered, and the standby database can be connected normally, the user may only pay attention to a few tables (individual tables) in the database. The mog_xlogdump tool can parse the table data of the specified oid according to the parameters -o and -R. For example, -o specifies the oid of the table, and -R specifies the field type of the table. + +### Instruction + +Create a table, write data and modify it, and use the mog_xlogdump tool to parse the Wal log. + +```sql +create table t2(id int, money money,inet inet,bool bool,numeric numeric ,text text); +insert into t2 values(1, 24.241, '192.168.255.132', true, 3.1415926, 'ljfsodfo29892ifj'); +insert into t2 values(2, 928.8271, '10.255.132.101', false, 3.1415926, 'vzvzcxwf2424@'); +update t2 set id=111, money=982.371 where id =2; +delete from t2 where id=1; + +postgres=# select * from t2; + id | money | inet | bool | numeric | text +----+---------+-----------------+------+-----------+------------------ + 1 | $24.24 | 192.168.255.132 | t | 3.1415926 | ljfsodfo29892ifj + 2 | $928.83 | 10.255.132.101 | f | 3.1415926 | vzvzcxwf2424@ +(2 rows) + +postgres=# update t2 set id=111, money=982.371 where id =2; +Postgres=# delete from t2 where id=1; +postgres=# select * from t2; + id | money | inet | bool | numeric | text +-----+-------------+----------------+------+-----------+--------------- + 111 | $982,371.00 | 10.255.132.101 | f | 3.1415926 | vzvzcxwf2424@ + +(1 rows) +``` + +### Parse Settings + +Same as use case 1, set wal_level and alter table. + +### Result + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/reference-guide/mog_xlogdump-2.png) + +```json +./mog_xlogdump -o 16394 -R int,money,inet,bool,numeric,text ./db_p/pg_xlog/000000010000000000000004 +'insert','tuple':{'(null)':'1','(null)':'$24.24','(null)':'192.168.255.132','(null)':true,'(null)':'3.1415926','(null)':'ljfsodfo29892ifj'}} +'insert','tuple':{'(null)':'2','(null)':'$928.83','(null)':'10.255.132.101','(null)':false,'(null)':'3.1415926','(null)':'vzvzcxwf2424@'}} +'update','old_tuple':{'(null)':'2','(null)':'$928.83','(null)':'10.255.132.101','(null)':false,'(null)':'3.1415926','(null)':'vzvzcxwf2424@'},'new_tuple':{'(null)':'111','(null)':'$982,371.00','(null)':'10.255.132.101','(null)':false,'(null)':'3.1415926','(null)':'vzvzcxwf2424@'}} +'delete','tuple':{'(null)':'1','(null)':'$24.24','(null)':'192.168.255.132','(null)':true,'(null)':'3.1415926','(null)':'ljfsodfo29892ifj'}} +``` + +> Note: Due to the change of the output format, the table name, schema name and column name are queried on the standby node according to the -c connect string, but because the original -o, -R designation of the table oid and field type is completely offline, Therefore, information such as table name, schema name, and column name cannot be obtained, so use -o and -R to parse offline. The table name and schema name are not displayed, and the column name is displayed as null. + +``` +mog_xlogdump -o -R -s -e Wal log file +``` + +The tool also retains the original functionality of pg_xlogdump. + +
+ +## Use Case 3 + +### Scenario + +If you want to see the header data content of the wal log, or to count some related information of the wal log, please use the mog_xlogdump original function. + +### Instruction + +1. header information + + ``` + ./mog_xlogdump -n 10 + ``` + + -n 10 indicates that only 10 rows of data are displayed. + +2. Statistics + + ``` + ./mog_xlogdump -z + ``` + +### Results + +- Result 1 + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/reference-guide/mog_xlogdump-3.png) + +- Result 2 + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/reference-guide/mog_xlogdump-4.png) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mogdb-monitor.md b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mogdb-monitor.md new file mode 100644 index 0000000000000000000000000000000000000000..b5ae7c9fa118df1b1b12a725fc28e81870a3fb00 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/open-source-components/mogdb-monitor.md @@ -0,0 +1,27 @@ +--- +title: mogdb-monitor +summary: mogdb-monitor +author: Guo Huan +date: 2022-04-14 +--- + +# mogdb-monitor + +mogdb-monitor is a MogDB database cluster monitoring and deployment tool, with the help of the current very popular open source monitoring system prometheus framework, combined with the opengauss_exporter developed by Enmo database team, you can achieve a full range of detection of MogDB database. + +The core monitoring component opengauss_exporter has the following features. + +- Support all versions of MogDB/openGauss database +- Support for monitoring database clusters +- Support primary and standby database judgment within a cluster +- Support for automatic database discovery +- Support for custom query +- Supports online loading of configuration files +- Support for configuring the number of concurrent threads +- Support data collection information caching + +In terms of grafana display, Enmo also provide a complete set of dashboard, both an instance-level dashboard showing detailed information of each instance and a display big screen showing summary information of all instances, which, combined with the alertmanager component, can trigger alerts that meet the rules to relevant personnel in the first place. + +
+ +Please refer to [mogdb-monitor repository page](https://gitee.com/enmotech/mogdb-monitor) for details on how to obtain and use component. diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/terms-of-use.md b/product/en/docs-mogdb/v3.1/about-mogdb/terms-of-use.md new file mode 100644 index 0000000000000000000000000000000000000000..4d6b1b0e8cbf657dd9b9fad24d592c840788a0ce --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/terms-of-use.md @@ -0,0 +1,22 @@ +--- +title: Terms of Use +summary: Terms of Use +author: Guo Huan +date: 2021-06-01 +--- + +# Terms of Use + +**Copyright © 2009-2022 Yunhe Enmo (Beijing) Information Technology Co., Ltd. All rights reserved.** + +Your replication, use, modification, and distribution of this document are governed by the Creative Commons License Attribution-ShareAlike 4.0 International Public License (CC BY-SA 4.0). You can visit to view a human-readable summary of (and not a substitute for) CC BY-SA 4.0. For the complete CC BY-SA 4.0, visit . + +Certain document contents on this website are from the official openGauss website ()。 + +**Trademarks and Permissions** + +MogDB is a trademark of Yunhe Enmo (Beijing) Information Technology Co., Ltd. All other trademarks and registered trademarks mentioned in this document are the property of their respective holders. + +**Disclaimer** + +This document is used only as a guide. Unless otherwise specified by applicable laws or agreed by both parties in written form, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, including but not limited to non-infringement, timeliness, and specific purposes. diff --git a/product/en/docs-mogdb/v3.1/about-mogdb/usage-limitations.md b/product/en/docs-mogdb/v3.1/about-mogdb/usage-limitations.md new file mode 100644 index 0000000000000000000000000000000000000000..0dd3030919b12bbc5bd8c58df8c8e034fef55655 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/about-mogdb/usage-limitations.md @@ -0,0 +1,27 @@ +--- +title: Usage Limitations +summary: Usage Limitations +author: Guo Huan +date: 2021-06-01 +--- + +# Usage Limitations + +This document describes the common usage limitations of MogDB. + +| Item | Upper limit | +| ----------------------------------------- | ---------------------------------------------------- | +| Database capacity | Depend on operating systems and hardware | +| Size of a single table | 32 TB | +| Size of a single row | 1 GB | +| Size of a single field in a row | 1 GB | +| Number of rows in a single table | 281474976710656 (2^48^) | +| Number of columns in a single table | 250~1600 (Varies depending on different field types) | +| Number of indexes in a single table | Unlimited | +| Number of columns in a compound index | 32 | +| Number of constraints in a single table | Unlimited | +| Number of concurrent connections | 10000 | +| Number of partitions in a partition table | 1048575 | +| Size of a single partition | 32 TB | +| Number of rows in a single partition | 2^55^ | +| Maximum length of SQL text | 1048576 bytes (1MB) | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/br/1-1-br.md b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-1-br.md new file mode 100644 index 0000000000000000000000000000000000000000..3e55b93e5fd43eaf74278eb7430888d5e38e3c66 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-1-br.md @@ -0,0 +1,93 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2021-04-27 +--- + +# Overview + +For database security purposes, MogDB provides three backup and restoration types, multiple backup and restoration solutions, and data reliability assurance mechanisms during backup and restoration. + +Backup and restoration can be classified into logical backup and restoration, physical backup and restoration, and flashback. + +- Logical backup and restoration: backs up data by logically exporting data. This method can dump data that is backed up at a certain time point, and restore data only to this backup point. A logical backup does not back up data processed between failure occurrence and the last backup. It applies to scenarios where data rarely changes. Such data damaged due to misoperation can be quickly restored using a logical backup. To restore all the data in a database through logical backup, rebuild a database and import the backup data. Logical backup is not recommended for databases requiring high data availability because it takes a long time for data restoration. Logical backup is a major approach to migrate and transfer data because it can be performed on any platform. + +- Physical backup and restoration: copies physical file in the unit of disk blocks from the primary node to the standby node to back up a database. A database can be restored using backup files such as data files and archive log files. A physical backup is useful when you need to quickly back up and restore the complete database within a short period of time. Backup and restoration can be implemented at low costs through proper planning. + +- Flashback: This function is used to restore dropped tables from the recycle bin. Like in a Window OS, dropped table information is stored in the recycle bin of databases. The MVCC mechanism is used to restore data to a specified point in time or change sequence number (CSN). + + The three data backup and restoration solutions supported by MogDB are as follows. Methods for restoring data in case of an exception differ for different backup and restoration solutions. + + **Table 1** Comparison of three backup and restoration types + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Backup TypeApplication ScenarioMediaTool NameRecovery TimeAdvantage and Disadvantage
Logical backup and restorationSmall volume of data needs to be processed.
You can back up a single table, multiple tables, a single database, or all databases. The backup data needs to be restored using gsql or gs_restore. When the data volume is large, the restoration takes a long time.		 - Disk
- SSD		gs_dumpIt takes a long time to restore data in plain-text format. It takes a moderate time to restore data in archive format.This tool is used to export database information. Users can export a database or its objects (such as schemas, tables, and views). The database can be the default postgres database or a user-specified database. The exported file can be in plain-text format or archive format. Data in plain-text format can be restored only by using gsql, which takes a long time. Data in archive format can be restored only by using gs_restore. The restoration time is shorter than that of the plain-text format.
gs_dumpallLong data recovery timeThis tool is used to export all information of the openGauss database, including the data of the default postgres database, data of user-specified databases, and global objects of all openGauss databases.
Only data in plain-text format can be exported. The exported data can be restored only by using gsql, which takes a long time.
Physical backup and restorationHuge volume of data needs to be processed. It is mainly used for full backup and restoration as well as the backup of all WAL archive and run logs in the database.gs_backupSmall data volume and fast data recoveryThe OM tool for exporting database information can be used to export database parameter files and binary files. It helps the openGauss to back up and restore important data, and display help and version information. During the backup, you can select the type of the backup content. During the restoration, ensure that the backup file exists in the backup directory of each node. During cluster restoration, the cluster information in the static configuration file is used for restoration. It takes a short time to restore only parameter files.
gs_basebackupDuring the restoration, you can directly copy and replace the original files, or directly start the database on the backup database. The restoration takes a short time.This too is used to fully copy the binary files of the server database. Only the database at a certain time point can be backed up. With PITR, you can restore data to a time point after the full backup time point.
gs_probackupData can be directly restored to a backup point and the database can be started on the backup database. The restoration takes a short time.gs_probackup is a tool used to manage openGauss database backup and restoration. It periodically backs up openGauss instances. It supports the physical backup of a standalone database or a primary database node in a cluster. It supports the backup of contents in external directories, such as script files, configuration files, log files, and dump files. It supports incremental backup, periodic backup, and remote backup. The time required for incremental backup is shorter than that for full backup. You only need to back up the modified files. Currently, the data directory is backed up by default. If the tablespace is not in the data directory, you need to manually specify the tablespace directory to be backed up. Currently, data can be backed up only on the primary node.
FlashbackApplicable to:
1) A table is deleted by mistake.
2) Data in the tables needs to be restored to a specified time point or CSN.		NoneYou can restore a table to the status at a specified time point or before the table structure is deleted. The restoration takes a short time.Flashback can selectively and efficiently undo the impact of a committed transaction and recover from a human error. Before the flashback technology is used, the committed database modification can be retrieved only by means of restoring backup or PITR. The restoration takes several minutes or even hours. After the flashback technology is used, it takes only seconds to restore the committed data before the database is modified. The restoration time is irrelevant to the database size.
Flashback supports two recovery modes:
- Multi-version data restoration based on MVCC: applicable to the query and restoration of data that is deleted, updated, or inserted by mistake. You can configure the retention period of the old version and run the corresponding query or restoration command to query or restore data to a specified time point or CSN.
- Recovery based on the recycle bin (similar to that on Windows OS): This method is applicable to the recovery of tables that are dropped or truncated by mistake. You can configure the recycle bin switch and run the corresponding restoration command to restore the tables that are dropped or truncated by mistake.
+ +While backing up and restoring data, take the following aspects into consideration: + +- Whether the impact of data backup on services is acceptable + +- Database restoration efficiency + + To minimize the impact of database faults, try to minimize the restoration duration, achieving the highest restoration efficiency. + +- Data restorability + + Minimize data loss after the database is invalidated. + +- Database restoration cost + + There are many factors that need to be considered while you select a backup policy on the live network, such as backup objects, data volume, and network configuration. Table 2 lists available backup policies and applicable scenarios for each backup policy. + + **Table 2** Backup policies and scenarios + + | Backup Policy | Key Performance Factor | Typical Data Volume | Performance Specifications | + | :----------------------- | :----------------------------------------------------------- | :--------------------------------------------------------- | :----------------------------------------------------------- | + | Database instance backup | - Data amount
- Network configuration | Data volume: PB level
Object quantity: about 1 million | Backup:
- Data transfer rate on each host: 80 Mbit/s (NBU/EISOO+Disk)
- Disk I/O rate (SSD/HDD): about 90% | + | Table backup | - Schema where the table to be backed up resides
- Network configuration (NBU) | Data volume: 10 TB level | Backup: depends on query performance rate and I/O rate
NOTE:
For multi-table backup, the backup time is calculated as follows:
`Total time = Number of tables x Starting time + Total data volume/Data backup speed`
In the preceding information:
- The starting time of a disk is about 5s. The starting time of an NBU is longer than that of a disk (depending on the NBU deployment).
- The data backup speed is about 50 MB/s on a single node. (The speed is evaluated based on the backup of a 1 GB table from a physical host to a local disk.)
The smaller the table is, the lower the backup performance will be. | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/br/1-2-br.md b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-2-br.md new file mode 100644 index 0000000000000000000000000000000000000000..81b70cd8cbcf72149a75ab11dd9bfa9367513756 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-2-br.md @@ -0,0 +1,129 @@ +--- +title: Physical Backup and Restoration +summary: Physical Backup and Restoration +author: Guo Huan +date: 2021-04-27 +--- + +# Physical Backup and Restoration + +## gs_basebackup + +For details, see [gs_basebackup](../../reference-guide/tool-reference/tools-used-in-the-internal-system/3-gs_basebackup.md). + +## PITR Recovery + +### Background + +When a database breaks down or needs to be rolled back to a previous state, the point-in-time recovery (PITR) function of MogDB can be used to restore the database to any point in time after the backup and archive data is generated. + +**NOTE:** + +- PITR can only be restored to a point in time after the physical backup data is generated. +- Only the primary node can be restored using PITR. The standby node needs to be fully built to synchronize data with the primary node. + +### Prerequisites + +- Full data files have been physically backed up. +- WAL log files have been archived. + +### PITR Recovery Process + +1. Replace the target database directory with the physical backup files. +2. Delete all files in the database directory **pg_xlog/**. +3. the archived WAL log file to the **pg_xlog** file. (Or you can configure **restore_command** in the **recovery.conf** file to skip this step.) +4. Create the recovery command file **recovery.conf** in the database directory and specify the database recovery degree. +5. Start the database. +6. Connect to the database and check whether the database is recovered to the expected status. +7. If the expected status is reached, run the **pg_xlog_replay_resume()** command so that the primary node can provide services externally. + +### Configuring the recovery.conf File + +**Archive Recovery Configuration** + +- restore_command = string + + The **shell** command is used to obtain the archived WAL files among the WAL file series. Any %f in the string is replaced by the name of the file to retrieve from the archive, and any %p is replaced by the path name to it to on the server. Any %r is replaced by the name of the file containing the last valid restart point. + + For example: + + ```bash + restore_command = 'cp /mnt/server/archivedir/%f %p' + ``` + +- archive_cleanup_command = string + + This option parameter declares a **shell** command that is executed each time the system is restarted. **archive_cleanup_command** provides a mechanism for deleting unnecessary archived WAL files from the standby database. Any %r is replaced by the name of the file containing the last valid restart point. That is the earliest file that must be kept to allow recovery to be restartable, so all files older than %r can be safely removed. + + For example: + + ```bash + archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r' + ``` + + If multiple standby servers need to be recovered from the same archive path, ensure that WAL files are not deleted from any standby server before the recovery. + +- recovery_end_command = string + + This parameter is optional and is used to declare a **shell** command that is executed only when the recovery is complete. **recovery_end_command** provides a cleanup mechanism for future replication and recovery. + +**Recovery Object Configuration** + +- recovery_target_name = string + + This parameter declares that the name is recovered to a recovery point created using pg_create_restore_point(). + + For example: + + ```bash + recovery_target_name = 'restore_point_1' + ``` + +- recovery_target_time = timestamp + + This parameter declares that the name is recovered to a specified timestamp. + + For example: + + ```bash + recovery_target_time = '2020-01-01 12:00:00' + ``` + +- recovery_target_xid = string + + This parameter declares that the name is recovered to a transaction ID. + + For example: + + ```bash + recovery_target_xid = '3000' + ``` + +- recovery_target_lsn = string + + This parameter declares that the name is recovered to the LSN specified by log. + + For example: + + ```bash + recovery_target_lsn = '0/0FFFFFF' + ``` + +- recovery_target_inclusive = boolean + + This parameter declares whether to stop the recovery after the recovery target is specified (**true**) or before the recovery target is specified (**false**). This declaration supports only the recovery targets **recovery_target_time**, **recovery_target_xid**, and **recovery_target_lsn**. + + For example: + + ```bash + recovery_target_inclusive = true + ``` + +**NOTE:** + +- Only one of the four configuration items **recovery_target_name**, **recovery_target_time**, **recovery_target_xid**, and **recovery_target_lsn** can be used at a time. +- If no recovery targets are configured or the configured target does not exist, data is recovered to the latest WAL log point by default. + +## gs_probackup + +For details, see [gs_probackup](../../reference-guide/tool-reference/tools-used-in-the-internal-system/15-gs_probackup.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/br/1-3-br.md b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-3-br.md new file mode 100644 index 0000000000000000000000000000000000000000..6d6809a4c71b4626378f282466c99d9bd38d1cdf --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-3-br.md @@ -0,0 +1,24 @@ +--- +title: Logical Backup and Restoration +summary: Logical Backup and Restoration +author: Guo Huan +date: 2021-04-27 +--- + +# Logical Backup and Restoration + +## gs_dump + +For details, see [gs_dump](../../reference-guide/tool-reference/server-tools/5-gs_dump.md). + +## gs_dumpall + +For details, see [gs_dumpall](../../reference-guide/tool-reference/server-tools/6-gs_dumpall.md). + +## gs_restore + +For details, see [gs_restore](../../reference-guide/tool-reference/server-tools/9-gs_restore.md). + +## gs_backup + +For details, see [gs_backup](../../reference-guide/tool-reference/tools-used-in-the-internal-system/2-gs_backup.md). diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/br/1-4-br.md b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-4-br.md new file mode 100644 index 0000000000000000000000000000000000000000..bba2d64a84d5d1df160e76269562a48ad9ba62f7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/br/1-4-br.md @@ -0,0 +1,203 @@ +--- +title: Flashback Restoration +summary: Flashback Restoration +author: Guo Huan +date: 2021-10-12 +--- + +# Flashback Restoration + +Flashback restoration is a part of the database recovery technology. It can be used to selectively cancel the impact of a committed transaction and restore data from incorrect manual operations. Before the flashback technology is used, the committed database modification can be retrieved only by means of restoring backup and PITR. The restoration takes several minutes or even hours. After the flashback technology is used, it takes only seconds to restore the submitted data before the database is modified. The restoration time is irrelevant to the database size. + +> Note: The Astore engine does not support the flashback function. + +**Flashback supports two restoration modes:** + +- MVCC-based multi-version data restoration (only Ustore is supported): It is suitable for query and restoration of data deleted, updated and inserted by mistake. Users can configure the retention time of the old version and execute the corresponding query or restore command to query or restore data to Specified point in time or CSN point. +- Restoration based on database recycle bin (only Ustore is supported): It is suitable for the restoration of tables that are DROP and TRUNCATE by mistake. By configuring the recycle bin switch and executing the corresponding restore command, the user can retrieve the tables that were erroneously DROP and TRUNCATE. + +**Related parameters:** + +- [enable_default_ustore_table](../../reference-guide/guc-parameters/24-miscellaneous-parameters.md#enable_default_ustore_table)=on + + Enable default support for Ustore storage engine + +- [undo_retention_time](../../reference-guide/guc-parameters/37-flashback.md#undo_retention_time) + + Set the undo old version retention time. Equivalent to the time span for which flashback queries are allowed. Flashback queries beyond that time may report `restore point not found` error. + +- [enable_recyclebin](../../reference-guide/guc-parameters/37-flashback.md#enable_recyclebin)=on + + enable recycle bin + +- [recyclebin_retention_time](../../reference-guide/guc-parameters/37-flashback.md#recyclebin_retention_time)=15min + + Set the recycle bin object retention time, and the recycle bin objects that exceed this time will be automatically cleaned up + +
+ +## Flashback Query + +### Context + +Flashback query enables you to query a snapshot of a table at a certain time point in the past. This feature can be used to view and logically rebuild damaged data that is accidentally deleted or modified. The flashback query is based on the MVCC mechanism. You can retrieve and query the old version to obtain the data of the specified old version. + +### Prerequisites + +- The **undo_retention_time** parameter has been set for specifying the retention period of undo logs. + +### Syntax + +```ebnf+diagram +FlashBack ::= ( select_statement) TIMECAPSULE { TIMESTAMP | CSN } expression +``` + +In the syntax tree, **TIMECAPSULE {TIMESTAMP | CSN} expression** is a new expression for the flashback function. **TIMECAPSULE** indicates that the flashback function is used. **TIMESTAMP** and **CSN** indicate that the flashback function uses specific time point information or commit sequence number (CSN) information. + +### Parameter Description + +- TIMESTAMP + - Specifies a history time point of the table data to be queried. +- CSN + - Specifies a logical commit time point of the data in the entire database to be queried. Each CSN in the database represents a consistency point of the entire database. To query the data under a CSN means to query the data related to the consistency point in the database through SQL statements. + +### Examples + +- Example 1: + + ```sql + SELECT * FROM t1 TIMECAPSULE TIMESTAMP to_timestamp ('2020-02-11 10:13:22.724718', 'YYYY-MM-DD HH24:MI:SS.FF'); + ``` + +- Example 2: + + ```sql + SELECT * FROM t1 TIMECAPSULE CSN 9617; + ``` + +- Example 3: + + ```sql + SELECT * FROM t1 AS t TIMECAPSULE TIMESTAMP to_timestamp ('2020-02-11 10:13:22.724718', 'YYYY-MM-DD HH24:MI:SS.FF'); + ``` + +- Example 4: + + ```sql + SELECT * FROM t1 AS t TIMECAPSULE CSN 9617; + ``` + +## Flashback Table + +### Context + +Flashback table enables you to restore a table to a specific point in time. When only one table or a group of tables are logically damaged instead of the entire database, this feature can be used to quickly restore the table data. Based on the MVCC mechanism, the flashback table deletes incremental data at a specified time point and after the specified time point and retrieves the data deleted at the specified time point and the current time point to restore table-level data. + +### Prerequisites + +- The **undo_retention_time** parameter has been set for specifying the retention period of undo logs. + +### Syntax + +```ebnf+diagram +FlashBack ::= TIMECAPSULE TABLE table_name TO { TIMESTAMP | CSN } expression +``` + +### Examples + +```sql +TIMECAPSULE TABLE t1 TO TIMESTAMP to_timestamp ('2020-02-11 10:13:22.724718', 'YYYY-MM-DD HH24:MI:SS.FF'); +TIMECAPSULE TABLE t1 TO CSN 9617; +``` + +## Flashback DROP/TRUNCATE + +### Context + +Flashback drop enables you to restore tables that are dropped by mistake and their auxiliary structures, such as indexes and table constraints, from the recycle bin. Flashback drop is based on the recycle bin mechanism. You can restore physical table files recorded in the recycle bin to restore dropped tables. + +Flashback truncate enables you to restore tables that are truncated by mistake and restore the physical data of the truncated tables and indexes from the recycle bin. Flashback truncate is based on the recycle bin mechanism. You can restore physical table files recorded in the recycle bin to restore truncated tables. + +### Prerequisites + +- The **enable_recyclebin** parameter has been set for enabling the recycle bin. +- The **recyclebin_retention** parameter has been set for specifying the retention period of objects in the recycle bin. The objects will be automatically deleted after the retention period expires. + +### Syntax + +- Drop a table. + + ```ebnf+diagram + DropTable ::= DROP TABLE table_name [PURGE] + ``` + +- Purge objects in the recycle bin. + + ```ebnf+diagram + PurgeRecyclebin ::= PURGE { TABLE { table_name } + | INDEX { index_name } + | RECYCLEBIN + } + ``` + +- Flash back a dropped table. + + ```ebnf+diagram + TimecapsuleTable ::= TIMECAPSULE TABLE { table_name } TO BEFORE DROP [RENAME TO new_tablename] + ``` + +- Truncate a table. + + ```ebnf+diagram + TruncateTable ::= TRUNCATE TABLE { table_name } [ PURGE ] + ``` + +- Flash back a truncated table. + + ```ebnf+diagram + TimecapsuleTable ::= TIMECAPSULE TABLE { table_name } TO BEFORE TRUNCATE + ``` + +### Parameter Description + +- DROP/TRUNCATE TABLE table_name PURGE + - Purges table data in the recycle bin by default. +- PURGE RECYCLEBIN + - Purges objects in the recycle bin. +- **TO BEFORE DROP** + +Retrieves dropped tables and their subobjects from the recycle bin. + +You can specify either the original user-specified name of the table or the system-generated name assigned to the object when it was dropped. + +- System-generated recycle bin object names are unique. Therefore, if you specify the system-generated name, the database retrieves that specified object. To see the contents of your recycle bin, run **select \* from pg_recyclebin;**. +- If you specify the user-specified name and the recycle bin contains more than one object of that name, the database retrieves the object that was moved to the recycle bin most recently. If you want to retrieve an older version of the table, then do one of these things: + - Specify the system-generated recycle bin name of the table you want to retrieve. + - Run **TIMECAPSULE TABLE ... TO BEFORE DROP** statements until you retrieve the table you want. + - When a dropped table is restored, only the base table name is restored, and the names of other subobjects remain the same as those in the recycle bin. You can run the DDL command to manually change the names of subobjects as required. + - The recycle bin does not support write operations such as DML, DCL, and DDL, and does not support DQL query operations (supported in later versions). + - Between the flashback point and the current point, a statement has been executed to modify the table structure or to affect the physical structure. Therefore, the flashback fails. The error message "ERROR: The table definition of %s has been changed." is displayed when flashback is performed on a table where DDL operations have been performed. The error message "ERROR: recycle object %s desired does not exis" is displayed when flashback is performed on DDL operations, such as changing namespaces and table names. +- **RENAME TO** + +Specifies a new name for the table retrieved from the recycle bin. + +- **TO BEFORE TRUNCATE** + +Flashes back to the point in time before the TRUNCATE operation. + +### Syntax Example + +```sql +DROP TABLE t1 PURGE; + +PURGE TABLE t1; +PURGE TABLE "BIN$04LhcpndanfgMAAAAAANPw==$0"; +PURGE INDEX i1; +PURGE INDEX "BIN$04LhcpndanfgMAAAAAANPw==$0"; +PURGE RECYCLEBIN; + +TIMECAPSULE TABLE t1 TO BEFORE DROP; +TIMECAPSULE TABLE t1 TO BEFORE DROP RENAME TO new_t1; +TIMECAPSULE TABLE "BIN$04LhcpndanfgMAAAAAANPw==$0" TO BEFORE DROP; +TIMECAPSULE TABLE "BIN$04LhcpndanfgMAAAAAANPw==$0" TO BEFORE DROP RENAME TO new_t1; +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/column-store-tables-management.md b/product/en/docs-mogdb/v3.1/administrator-guide/column-store-tables-management.md new file mode 100644 index 0000000000000000000000000000000000000000..8e210216c483c34a7d97c478cc2b4d389ada9701 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/column-store-tables-management.md @@ -0,0 +1,386 @@ +--- +title: Column-store Tables Management +summary: Column-store Tables Management +author: Guo Huan +date: 2021-04-09 +--- + +# Column-store Tables Management + +## What is Column-store + +Row-store stores tables to disk partitions by row, and column-store stores tables to disk partitions by column. By default, a row-store table is created. For details about differences between row storage and column storage, see Figure 1. + +**Figure 1** Differences between row storage and column storage + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/column-store-tables-management.png) + +In the preceding figure, the upper left part is a row-store table, and the upper right part shows how the row-store table is stored on a disk; the lower left part is a column-store table, and the lower right part shows how the column-store table is stored on a disk. From the above figure, you can clearly see that the data of a row-store table are put together, but they are kept separately in column-store table. + +## Advantages and Disadvantages of Row-store and Column-store Tables and Their Usage Scenario + +Both storage models have benefits and drawbacks. + +| Storage Model | Benefit | Drawback | +| :------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | +| Row storage | Record data is stored together. Data can be easily inserted and updated. | All the columns of a record are read after the **SELECT** statement is executed even if only certain columns are required. | +| Column storage | Only the columns involved in a query are read. Projections are efficient. Any column can serve as an index. | The selected columns need to be reconstructed after the **SELECT** statement is executed. Data cannot be easily inserted or updated. | + +Generally, if a table contains many columns (called a wide table) and its query involves only a few columns, column storage is recommended. Row storage is recommended if a table contains only a few columns and a query involves most of the fields. + +| Storage Model | Application Scenarios | +| :------------- | :----------------------------------------------------------- | +| Row storage | Point queries (simple index-based queries that only return a few records)Scenarios requiring frequent addition, deletion, and modification | +| Column storage | Statistical analysis queries (requiring a large number of association and grouping operations)Ad hoc queries (using uncertain query conditions and unable to utilize indexes to scan row-store tables) | + +MogDB supports hybrid row storage and column storage. Each storage model applies to specific scenarios. Select an appropriate model when creating a table. Generally, MogDB is used for transactional processing databases. By default, row storage is used. Column storage is used only when complex queries in large data volume are performed. + +## Selecting a Storage Model + +- Update frequency + + If data is frequently updated, use a row-store table. + +- Data insertion frequency + + If a small amount of data is frequently inserted each time, use a row-store table. + +- Number of columns + + If a table is to contain many columns, use a column-store table. + +- Number of columns to be queried + + If only a small number of columns (less than 50% of the total) is queried each time, use a column-store table. + +- Compression ratio + + The compression ratio of a column-store table is higher than that of a row-store table. High compression ratio consumes more CPU resources. + +## Constraints of Column-store Table + +- The column-store table does not support arrays. +- The number of column-store tables is recommended to be no more than 1000. +- The table-level constraints of the column-store table only support **PARTIAL CLUSTER KEY**, and do not support table-level constraints such as primary and foreign keys. +- The field constraints of the column-store table only support **NULL**, **NOT NULL** and **DEFAULT** constant values. +- The column-store table does not support the **alter** command to modify field constraints. +- The column-store table supports the delta table, which is controlled by the parameter **enable_delta_store** whether to enable or not, and the threshold value for entering the delta table is controlled by the parameter **deltarow_threshold**. + +## Related Parameters + +- cstore_buffers + + The size of the shared buffer used by the column-store, the default value: 32768KB. + +- partition_mem_batch + + Specify the number of caches. In order to optimize the batch insertion of column-store partition tables, the data will be cached during the batch insertion process and then written to disk in batches. Default value: 256. + +- partition_max_cache_size + + Specify the size of the data buffer area. In order to optimize the batch insertion of column-store partition tables, the data will be cached during the batch insertion process and then written to disk in batches. Default value: 2GB. + +- enable_delta_store + + In order to enhance the performance of single data import in column-store and solve the problem of disk redundancy, whether it is necessary to enable the function of column-store delta table and use it in conjunction with the parameter **DELTAROW_THRESHOLD**. Default value: off. + +## Create Table Commands + +MogDB creates normal tables as uncompressed row-store tables by default. + +``` +mogdb=# \dt +No relations found. +mogdb=# create table test_t(id serial primary key ,col1 varchar(8),col2 decimal(6,2),create_time timestamptz not null default now()); +NOTICE: CREATE TABLE will create implicit sequence "test_t_id_seq" for serial column "test_t.id" +NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "test_t_pkey" for table "test_t" +CREATE TABLE +mogdb=# \dt+ + List of relations + Schema | Name | Type | Owner | Size | Storage | Description +--------+--------+-------+-------+---------+----------------------------------+------------- + public | test_t | table | omm | 0 bytes | {orientation=row,compression=no} | +(1 row) + +mogdb=# +``` + +To create a column-store table, you need to specify **orientation=column**, the default compression level is **low**. + +``` +mogdb=# create table column_t(id serial,col1 varchar(8),col2 decimal(6,2),create_time timestamptz not null default now()) with (orientation=column ); +NOTICE: CREATE TABLE will create implicit sequence "column_t_id_seq" for serial column "column_t.id" +CREATE TABLE +mogdb=# \dt+ + List of relations + Schema | Name | Type | Owner | Size | Storage | Description +--------+----------+-------+-------+---------+--------------------------------------+------------- + public | column_t | table | omm | 16 kB | {orientation=column,compression=low} | + public | test_t | table | omm | 0 bytes | {orientation=row,compression=no} | +(2 rows) + +mogdb=# \d+ column_t + Table "public.column_t" + Column | Type | Modifiers | Storage | Stats target | Description +-------------+--------------------------+-------------------------------------------------------+----------+--------------+------------- + id | integer | not null default nextval('column_t_id_seq'::regclass) | plain | | + col1 | character varying(8) | | extended | | + col2 | numeric(6,2) | | main | | + create_time | timestamp with time zone | not null default now() | plain | | +Has OIDs: no +Options: orientation=column, compression=low +``` + +Add partial clustered storage columns to the column-store table. + +``` +mogdb=# \d+ column_t + Table "public.column_t" + Column | Type | Modifiers | Storage | Stats target | Description +-------------+--------------------------+-------------------------------------------------------+----------+--------------+------------- + id | integer | not null default nextval('column_t_id_seq'::regclass) | plain | | + col1 | character varying(8) | | extended | | + col2 | numeric(6,2) | | main | | + create_time | timestamp with time zone | not null default now() | plain | | +Has OIDs: no +Options: orientation=column, compression=low + +mogdb=# alter table column_t add PARTIAL CLUSTER KEY(id); +ALTER TABLE +mogdb=# \d+ column_t + Table "public.column_t" + Column | Type | Modifiers | Storage | Stats target | Description +-------------+--------------------------+-------------------------------------------------------+----------+--------------+------------- + id | integer | not null default nextval('column_t_id_seq'::regclass) | plain | | + col1 | character varying(8) | | extended | | + col2 | numeric(6,2) | | main | | + create_time | timestamp with time zone | not null default now() | plain | | +Partial Cluster : + "column_t_cluster" PARTIAL CLUSTER KEY (id) +Has OIDs: no +Options: orientation=column, compression=low + +mogdb=# +``` + +Create column-store tables with partial clustered storage directly. + +``` +mogdb=# create table column_c(id serial,col1 varchar(8),col2 decimal(6,2),create_time timestamptz not null default now(),PARTIAL CLUSTER KEY(id)) with (orientation=column ); +NOTICE: CREATE TABLE will create implicit sequence "column_c_id_seq" for serial column "column_c.id" +CREATE TABLE +mogdb=# \d+ column_c + Table "public.column_c" + Column | Type | Modifiers | Storage | Stats target | Description +-------------+--------------------------+-------------------------------------------------------+----------+--------------+------------- + id | integer | not null default nextval('column_c_id_seq'::regclass) | plain | | + col1 | character varying(8) | | extended | | + col2 | numeric(6,2) | | main | | + create_time | timestamp with time zone | not null default now() | plain | | +Partial Cluster : + "column_c_cluster" PARTIAL CLUSTER KEY (id) +Has OIDs: no +Options: orientation=column, compression=low + +mogdb=# +``` + +Please refer to **Supported Data Types** > **Data Types Supported by Column-store Tables** under the **Reference Guide** for the data types supported by column-store tables. + +## Column-store versus Row-store + +**Used disk space** + +- The default size of the column-store table is 16K, the compression level is **low**. + +- The default size of the row-store table is 0bytes, the compression level is **no**. + +- Insert 1 million pieces of data into the two tables separately , and compare the occupied disk size. + + ``` + mogdb=# \dt+ + List of relations + Schema | Name | Type | Owner | Size | Storage | Description + --------+-----------+-------+-------+---------+-----------------------------------------+------------- + public | column_t | table | omm | 16 kB | {orientation=column,compression=low} | + public | column_th | table | omm | 16 kB | {orientation=column,compression=high} | + public | column_tm | table | omm | 16 kB | {orientation=column,compression=middle} | + public | row_tc | table | omm | 0 bytes | {orientation=row,compression=yes} | + public | test_t | table | omm | 0 bytes | {orientation=row,compression=no} | + (5 rows) + + mogdb=# insert into column_t select generate_series(1,1000000),left(md5(random()::text),8),random()::numeric(6,2); + INSERT 0 1000000 + Time: 11328.880 ms + mogdb=# insert into column_th select generate_series(1,1000000),left(md5(random()::text),8),random()::numeric(6,2); + INSERT 0 1000000 + Time: 10188.634 ms + mogdb=# insert into column_tm select generate_series(1,1000000),left(md5(random()::text),8),random()::numeric(6,2); + INSERT 0 1000000 + Time: 9802.739 ms + mogdb=# insert into test_t select generate_series(1,1000000),left(md5(random()::text),8),random()::numeric(6,2); + INSERT 0 1000000 + Time: 17404.945 ms + mogdb=# insert into row_tc select generate_series(1,1000000),left(md5(random()::text),8),random()::numeric(6,2); + INSERT 0 1000000 + Time: 12394.866 ms + mogdb=# \dt+ + List of relations + Schema | Name | Type | Owner | Size | Storage | Description + --------+-----------+-------+-------+----------+-----------------------------------------+------------- + public | column_t | table | omm | 12 MB | {orientation=column,compression=low} | + public | column_th | table | omm | 8304 kB | {orientation=column,compression=high} | + public | column_tm | table | omm | 10168 kB | {orientation=column,compression=middle} | + public | row_tc | table | omm | 58 MB | {orientation=row,compression=yes} | + public | test_t | table | omm | 58 MB | {orientation=row,compression=no} | + (5 rows) + + mogdb=# + ``` + +- The higher the compression level of the column-store table is, the less the disk space it uses. + +- After the row-store table is compressed, the size of the disk space dose not decrease significantly. + +- Column-store table take up nearly 6 times less disk space than row-store table. + +**DML Comparison** + +Search for a single column: + +``` +--- +---Search by range, column-store is nearly 20 times faster than row-store +--- +mogdb=# select col1 from test_t where id>=100010 and id<100020; + col1 +---------- + 4257a3f3 + 3d397284 + 64343438 + 6eb7bdb7 + d1c9073d + 6aeb037c + 1d424974 + 223235ab + 329de235 + 2f02adc1 +(10 rows) + +Time: 77.341 ms +mogdb=# select col1 from column_t where id>=100010 and id<100020; + col1 +---------- + d4837c30 + 87a46f7a + 2f42a9c9 + 4481c793 + 68800204 + 613b9205 + 9d8f4a0a + 5cc4ff9e + f948cd10 + f2775cee +(10 rows) + +Time: 3.884 ms + +--- +---Search Randomly, column-store is nearly 35 times faster than row-store +--- + +mogdb=# select col1 from test_t limit 10; + col1 +---------- + c2780d93 + 294be14d + 4e53b761 + 2c10f8a2 + ae776743 + 7d683c66 + b3b40054 + 7e56edf9 + a7b7336e + ea3d47d9 +(10 rows) + +Time: 249.887 ms +mogdb=# select col1 from column_t limit 10; + col1 +---------- + a745d77b + 4b6df494 + 76fed9c1 + 70c9664d + 3384de8a + 4158f3bf + 5d1c3b9f + 341876bb + f396f4ed + abfd78bb +(10 rows) + +Time: 7.738 ms +``` + +Search for all the data: + +``` +--- +---Row-store is 30% faster than column-store search +--- +mogdb=# select * from test_t limit 10; + id | col1 | col2 | create_time +----+----------+------+------------------------------- + 1 | c2780d93 | .37 | 2020-10-26 14:27:33.304108+08 + 2 | 294be14d | .57 | 2020-10-26 14:27:33.304108+08 + 3 | 4e53b761 | .98 | 2020-10-26 14:27:33.304108+08 + 4 | 2c10f8a2 | .27 | 2020-10-26 14:27:33.304108+08 + 5 | ae776743 | .97 | 2020-10-26 14:27:33.304108+08 + 6 | 7d683c66 | .58 | 2020-10-26 14:27:33.304108+08 + 7 | b3b40054 | .44 | 2020-10-26 14:27:33.304108+08 + 8 | 7e56edf9 | .43 | 2020-10-26 14:27:33.304108+08 + 9 | a7b7336e | .31 | 2020-10-26 14:27:33.304108+08 + 10 | ea3d47d9 | .42 | 2020-10-26 14:27:33.304108+08 +(10 rows) + +Time: 6.822 ms + +mogdb=# select * from column_t limit 10; + id | col1 | col2 | create_time +----+----------+------+------------------------------- + 1 | a745d77b | .33 | 2020-10-26 14:28:20.633253+08 + 2 | 4b6df494 | .42 | 2020-10-26 14:28:20.633253+08 + 3 | 76fed9c1 | .73 | 2020-10-26 14:28:20.633253+08 + 4 | 70c9664d | .74 | 2020-10-26 14:28:20.633253+08 + 5 | 3384de8a | .48 | 2020-10-26 14:28:20.633253+08 + 6 | 4158f3bf | .59 | 2020-10-26 14:28:20.633253+08 + 7 | 5d1c3b9f | .63 | 2020-10-26 14:28:20.633253+08 + 8 | 341876bb | .97 | 2020-10-26 14:28:20.633253+08 + 9 | f396f4ed | .73 | 2020-10-26 14:28:20.633253+08 + 10 | abfd78bb | .30 | 2020-10-26 14:28:20.633253+08 +(10 rows) + +Time: 9.982 ms +``` + +Update data: + +``` +--- +---Update a field directly, column-store is nearly 7 times faster than row-store +--- +mogdb=# update test_t set col1=col1; +UPDATE 1000000 +Time: 19779.978 ms +mogdb=# update column_t set col1=col1; +UPDATE 1000000 +Time: 2702.339 ms +``` + +## Conclusion + +1. The Column-store table saves nearly 6 times the disk space usage compared to the row-store table. +2. When searching for the specified field, the column-store table is about 20-35 times faster than the row-store table. +3. When searching for all the data, the column-store table is 30% slower than the row-store table. +4. When importing data in batches in the default compression mode, and column-store table is 40% faster than the row-store table. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/1-using-gs_dump-and-gs_dumpall-to-export-data-overview.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/1-using-gs_dump-and-gs_dumpall-to-export-data-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..e963ac79cb6083c0047058747571d80d3ee29280 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/1-using-gs_dump-and-gs_dumpall-to-export-data-overview.md @@ -0,0 +1,91 @@ +--- +title: Using gs_dump and gs_dumpall to Export Data Overview +summary: Using gs_dump and gs_dumpall to Export Data Overview +author: Guo Huan +date: 2021-03-04 +--- + +# Using gs_dump and gs_dumpall to Export Data Overview + +MogDB provides **gs_dump** and **gs_dumpall** to export required database objects and related information. You can use a tool to import the exported data to a target database for database migration. **gs_dump** exports a single database or its objects. **gs_dumpall** exports all databases or global objects in MogDB. For details, see [Table 1](#Scenarios). + +**Table 1** Scenarios + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Application ScenarioExport GranularityExport FormatImport Method
Exporting a single databaseDatabase-level export
+ - Export full information of a cluster.
You can use the exported information to create a database containing the same data as the current one.
+ - Export all object definitions of a database, including the definitions of the database, functions, schemas, tables, indexes, and stored procedures. +
You can use the exported object definitions to quickly create a database that is the same as the current one, except that the new database does not have data.
+ - Export data of a cluster.		 + - Plaintext
+ - Custom
+ - Directory
+ - .tar + 		+ - For details about how to import data files in text format, see Using a gsql Meta-Command to Import Data.
+ - For details about how to import data files in .tar, directory, or custom format, see Using gs_restore to Import Data.
Schema-level export
+ - Export full information of a schema.
+ - Export data of a schema.
+ - Export all object definitions of a schema, including the definitions of tables, stored procedures, and indexes. +
Table-level export
+ - Export full information of a table.
+ - Export data of a table.
+ - Export the definition of a table. +
Exporting all databasesDatabase-level export
+ - Export full information of a cluster.
You can use the exported information to create a host environment containing the same databases, global objects, and data as the current one.
+ - Export all object definitions of a database, including the definitions of tablespaces, databases, functions, schemas, tables, indexes, and stored procedures.
+ You can use the exported object definitions to quickly create a host environment that is the same as the current one, containing the same databases and tablespaces but no data.
+ - Export data only.		PlaintextFor details about how to import data files, see Using a gsql Meta-Command to Import Data.
Global object export
+- Export tablespaces.
+- Export roles.
+- Export tablespaces and roles.
+ +**gs_dump** and **gs_dumpall** use **-U** to specify the user that performs the export. If the specified user does not have the required permissions, data cannot be exported. In this case, you can set **-role** in the **gs_dump** or **gs_dumpall** command to the role that has the permissions. Then, **gs_dump** or **gs_dumpall** uses the specified role to export data. See Table 1 for application scenarios and [Data Export By a User Without Required Permissions](4-data-export-by-a-user-without-required-permissions.md) for operation details. + +**gs_dump** and **gs_dumpall** encrypt the exported data files. These files are decrypted before being imported. In this way, data disclosure is prevented, protecting database security. + +When **gs_dump** or **gs_dumpall** is used to export data from a cluster, other users can still access (read and write) databases in MogDB. + +**gs_dump** and **gs_dumpall** can export complete, consistent data. For example, if **gs_dump** is executed to export database A or **gs_dumpall** is executed to export all databases from MogDB at T1, data of database A or all databases in MogDB at that time point will be exported, and modifications on the databases after that time point will not be exported. + +**Precautions** + +- Do not modify an exported file or its content. Otherwise, restoration may fail. + +- If there are more than 500,000 objects (data tables, views, and indexes) in a database, you are advised to use **gs_guc** to set the following parameters for database nodes. This operation is not required if the parameter values are greater than the recommended ones. + + ```bash + gs_guc set -N all -I all -c 'max_prepared_transactions = 1000' + gs_guc set -N all -I all -c 'max_locks_per_transaction = 512' + ``` + +- For data consistency and integrity, **gs_dump** and **gs_dumpall** set a share lock for a table to dump. If a share lock has been set for the table in other transactions, **gs_dump** and **gs_dumpall** lock the table after it is released. If the table cannot be locked within the specified time, the dump fails. You can customize the timeout duration to wait for lock release by specifying the **-lock-wait-timeout** parameter. + +- During an export, **gs_dumpall** reads all tables in a database. Therefore, you need to connect to the database as a MogDB administrator to export a complete file. When you use **gsql** to execute SQL scripts, cluster administrator permissions are also required to add users and user groups, and create databases. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/2-exporting-a-single-database.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/2-exporting-a-single-database.md new file mode 100644 index 0000000000000000000000000000000000000000..d8ee43c8b2b2e2d0840c1e5b4909b826d1dcd9f7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/2-exporting-a-single-database.md @@ -0,0 +1,288 @@ +--- +title: Exporting a Single Database +summary: Exporting a Single Database +author: Guo Huan +date: 2021-03-04 +--- + +# Exporting a Single Database + +## Exporting a Database + +You can use **gs_dump** to export data and all object definitions of a database from MogDB. You can specify the information to export as follows: + +- Export full information of a database, including its data and all object definitions. + + You can use the exported information to create a database containing the same data as the current one. + +- Export all object definitions of a database, including the definitions of the database, functions, schemas, tables, indexes, and stored procedures. + + You can use the exported object definitions to quickly create a database that is the same as the current one, except that the new database does not have data. + +- Export data of a database. + +### Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Use **gs_dump** to export data of the **userdatabase** database. + + ```bash + gs_dump -U jack -f /home/omm/backup/userdatabase_backup.tar -p 8000 postgres -F t + Password: + ``` + + **Table 1** Common parameters + + | Parameter | Description | Example Value | + | :-------- | :----------------------------------------------------------- | :------------------------------------------ | + | -U | Username for database connection.
NOTE:
If the username is not specified, the initial system administrator created during installation is used for connection by default. | -U jack | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W abcd@123 | + | -f | Folder to store exported files. If this parameter is not specified, the exported files are stored in the standard output. | -f /home/omm/backup/**postgres**_backup.tar | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + | dbname | Name of the database to export. | postgres | + | -F | Select the format of file to export. The values of **-F** are as follows:
- **p**: plaintext
- **c**: custom
- **d**: directory
- **t**: .tar | -F t | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_dump](../../../reference-guide/tool-reference/server-tools/5-gs_dump.md)" in the **Reference Guide**. + +### Examples + +Example 1: Run **gs_dump** to export full information of the **postgres** database. The exported files are in .sql format. + +```bash +gs_dump -f /home/omm/backup/postgres_backup.sql -p 8000 postgres -F p +Password: +gs_dump[port='8000'][postgres][2017-07-21 15:36:13]: dump database postgres successfully +gs_dump[port='8000'][postgres][2017-07-21 15:36:13]: total time: 3793 ms +``` + +Example 2: Run **gs_dump** to export data of the **postgres** database, excluding object definitions. The exported files are in a custom format. + +```bash +gs_dump -f /home/omm/backup/postgres_data_backup.dmp -p 8000 postgres -a -F c +Password: +gs_dump[port='8000'][postgres][2017-07-21 15:36:13]: dump database postgres successfully +gs_dump[port='8000'][postgres][2017-07-21 15:36:13]: total time: 3793 ms +``` + +Example 3: Run **gs_dump** to export object definitions of the **postgres** database. The exported files are in .sql format. + +```bash +gs_dump -f /home/omm/backup/postgres_def_backup.sql -p 8000 postgres -s -F p +Password: +gs_dump[port='8000'][postgres][2017-07-20 15:04:14]: dump database postgres successfully +gs_dump[port='8000'][postgres][2017-07-20 15:04:14]: total time: 472 ms +``` + +Example 4: Run **gs_dump** to export object definitions of the **postgres** database. The exported files are in text format and are encrypted. + +```bash +gs_dump -f /home/omm/backup/postgres_def_backup.sql -p 8000 postgres --with-encryption AES128 --with-key 1234567812345678 -s -F p +Password: +gs_dump[port='8000'][postgres][2018-11-14 11:25:18]: dump database postgres successfully +gs_dump[port='8000'][postgres][2018-11-14 11:25:18]: total time: 1161 ms +``` + +## Exporting a Schema + +You can use **gs_dump** to export data and all object definitions of a schema from MogDB. You can export one or more specified schemas as needed. You can specify the information to export as follows: + +- Export full information of a schema, including its data and object definitions. +- Export data of a schema, excluding its object definitions. +- Export the object definitions of a schema, including the definitions of tables, stored procedures, and indexes. + +### Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run **gs_dump** to export the **hr** and **public** schemas. + + ```bash + gs_dump -W Bigdata@123 -U jack -f /home/omm/backup/MPPDB_schema_backup -p 8000 human_resource -n hr -n public -F d + ``` + + **Table 1** Common parameters + + | Parameter | Description | Example Value | + | :-------- | :----------------------------------------------------------- | :----------------------------------------------------------- | + | -U | Username for database connection. | -U jack | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W Bigdata@123 | + | -f | Folder to store exported files. If this parameter is not specified, the exported files are stored in the standard output. | -f /home/omm/backup/MPPDB*_*schema_backup | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + | dbname | Name of the database to export. | human_resource | + | -n | Names of schemas to export. Data of the specified schemas will also be exported.
- Single schema: Enter **-n** **schemaname**.
- Multiple schemas: Enter **-n** **schemaname** for each schema. | - Single schemas:**-n hr**
- Multiple schemas:**-n hr -n public** | + | -F | Select the format of file to export. The values of **-F** are as follows:
- **p**: plaintext
- **c**: custom
- **d**: directory
- **t**: .tar | -F d | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_dump](../../../reference-guide/tool-reference/server-tools/5-gs_dump.md)" in the **Reference Guide**. + +### Examples + +Example 1: Run **gs_dump** to export full information of the **hr** schema. The exported files stored in text format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_schema_backup.sql -p 8000 human_resource -n hr -F p +gs_dump[port='8000'][human_resource][2017-07-21 16:05:55]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 16:05:55]: total time: 2425 ms +``` + +Example 2: Run **gs_dump** to export data of the **hr** schema. The exported files are in .tar format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_schema_data_backup.tar -p 8000 human_resource -n hr -a -F t +gs_dump[port='8000'][human_resource][2018-11-14 15:07:16]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2018-11-14 15:07:16]: total time: 1865 ms +``` + +Example 3: Run **gs_dump** to export the object definitions of the **hr** schema. The exported files are stored in a directory. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_schema_def_backup -p 8000 human_resource -n hr -s -F d +gs_dump[port='8000'][human_resource][2018-11-14 15:11:34]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2018-11-14 15:11:34]: total time: 1652 ms +``` + +Example 4: Run **gs_dump** to export the **human_resource** database excluding the **hr** schema. The exported files are in a custom format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_schema_backup.dmp -p 8000 human_resource -N hr -F c +gs_dump[port='8000'][human_resource][2017-07-21 16:06:31]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 16:06:31]: total time: 2522 ms +``` + +Example 5: Run **gs_dump** to export the object definitions of the **hr** and **public** schemas. The exported files are in .tar format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_schema_backup1.tar -p 8000 human_resource -n hr -n public -s -F t +gs_dump[port='8000'][human_resource][2017-07-21 16:07:16]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 16:07:16]: total time: 2132 ms +``` + +Example 6: Run **gs_dump** to export the **human_resource** database excluding the **hr** and **public** schemas. The exported files are in a custom format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_schema_backup2.dmp -p 8000 human_resource -N hr -N public -F c +gs_dump[port='8000'][human_resource][2017-07-21 16:07:55]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 16:07:55]: total time: 2296 ms +``` + +Example 7: Run **gs_dump** to export all tables (views, sequences, and foreign tables are also included) in the **public** schema and the **staffs** table in the **hr** schema, including data and table definition. The exported files are in a custom format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_backup3.dmp -p 8000 human_resource -t public.* -t hr.staffs -F c +gs_dump[port='8000'][human_resource][2018-12-13 09:40:24]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2018-12-13 09:40:24]: total time: 896 ms +``` + +## Exporting a Table + +You can use **gs_dump** to export data and definition of a table-level object from MogDB. Views, sequences, and foreign tables are special tables. You can export one or more specified tables as needed. You can specify the information to export as follows: + +- Export full information of a table, including its data and definition. +- Export data of a table. +- Export the definition of a table. + +### Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run **gs_dump** to export the **hr.staffs** and **hr.employments** tables. + + ```bash + gs_dump -W Bigdata@123 -U jack -f /home/omm/backup/MPPDB_table_backup -p 8000 human_resource -t hr.staffs -t hr.employments -F d + ``` + + **Table 1** Common parameters + + | Parameter | Description | Example Value | + | :-------- | :----------------------------------------------------------- | :----------------------------------------------------------- | + | -U | Username for database connection. | -U jack | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W Bigdata@123 | + | -f | Folder to store exported files. If this parameter is not specified, the exported files are stored in the standard output. | -f /home/omm/backup/MPPDB_table_backup | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + | dbname | Name of the database to export. | human_resource | + | -t | Table (or view, sequence, foreign table) to export. You can specify multiple tables by listing them or using wildcard characters. When you use wildcard characters, quote wildcard patterns with single quotation marks (") to prevent the shell from expanding the wildcard characters.
- Single table: Enter **-t** **schema.table**.
- Multiple tables: Enter **-t** **schema.table** for each table. | - Single table: **-t hr.staffs**
- **Multiple tables:**-t hr.staffs -t hr.employments** | + | -F | Select the format of file to export. The values of **-F** are as follows:
- **p**: plaintext
- **c**: custom
- **d**: directory
- **t**: .tar | -F d | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_dump](../../../reference-guide/tool-reference/server-tools/5-gs_dump.md)" in the **Reference Guide**. + +### Examples + +Example 1: Run **gs_dump** to export full information of the **hr.staffs** table. The exported files are in text format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup.sql -p 8000 human_resource -t hr.staffs -F p +gs_dump[port='8000'][human_resource][2017-07-21 17:05:10]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 17:05:10]: total time: 3116 ms +``` + +Example 2: Run **gs_dump** to export data of the **hr.staffs** table. The exported files are in .tar format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_data_backup.tar -p 8000 human_resource -t hr.staffs -a -F t +gs_dump[port='8000'][human_resource][2017-07-21 17:04:26]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 17:04:26]: total time: 2570 ms +``` + +Example 3: Run **gs_dump** to export the definition of the **hr.staffs** table. The exported files are stored in a directory. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_def_backup -p 8000 human_resource -t hr.staffs -s -F d +gs_dump[port='8000'][human_resource][2017-07-21 17:03:09]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 17:03:09]: total time: 2297 ms +``` + +Example 4: Run **gs_dump** to export the **human_resource** database excluding the **hr.staffs** table. The exported files are in a custom format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup4.dmp -p 8000 human_resource -T hr.staffs -F c +gs_dump[port='8000'][human_resource][2017-07-21 17:14:11]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 17:14:11]: total time: 2450 ms +``` + +Example 5: Run **gs_dump** to export the **hr.staffs** and **hr.employments** tables. The exported files are in text format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup1.sql -p 8000 human_resource -t hr.staffs -t hr.employments -F p +gs_dump[port='8000'][human_resource][2017-07-21 17:19:42]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 17:19:42]: total time: 2414 ms +``` + +Example 6: Run **gs_dump** to export the **human_resource** database excluding the **hr.staffs** and **hr.employments** tables. The exported files are in text format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup2.sql -p 8000 human_resource -T hr.staffs -T hr.employments -F p +gs_dump[port='8000'][human_resource][2017-07-21 17:21:02]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 17:21:02]: total time: 3165 ms +``` + +Example 7: Run **gs_dump** to export data and definition of the **hr.staffs** table, and the definition of the **hr.employments** table. The exported files are in .tar format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup3.tar -p 8000 human_resource -t hr.staffs -t hr.employments --exclude-table-data hr.employments -F t +gs_dump[port='8000'][human_resource][2018-11-14 11:32:02]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2018-11-14 11:32:02]: total time: 1645 ms +``` + +Example 8: Run **gs_dump** to export data and definition of the **hr.staffs** table, encrypt the exported files, and store them in text format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup4.sql -p 8000 human_resource -t hr.staffs --with-encryption AES128 --with-key 1212121212121212 -F p +gs_dump[port='8000'][human_resource][2018-11-14 11:35:30]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2018-11-14 11:35:30]: total time: 6708 ms +``` + +Example 9: Run **gs_dump** to export all tables (views, sequences, and foreign tables are also included) in the **public** schema and the **staffs** table in the **hr** schema, including data and table definition. The exported files are in a custom format. + +```bash +gs_dump -W Bigdata@123 -f /home/omm/backup/MPPDB_table_backup5.dmp -p 8000 human_resource -t public.* -t hr.staffs -F c +gs_dump[port='8000'][human_resource][2018-12-13 09:40:24]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2018-12-13 09:40:24]: total time: 896 ms +``` + +Example 10: Run **gs_dump** to export the definition of the view referencing to the **test1** table in the **t1** schema. The exported files are in a custom format. + +```bash +gs_dump -W Bigdata@123 -U jack -f /home/omm/backup/MPPDB_view_backup6 -p 8000 human_resource -t t1.test1 --include-depend-objs --exclude-self -F d +gs_dump[port='8000'][jack][2018-11-14 17:21:18]: dump database human_resource successfully +gs_dump[port='8000'][jack][2018-11-14 17:21:23]: total time: 4239 ms +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/3-exporting-all-databases.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/3-exporting-all-databases.md new file mode 100644 index 0000000000000000000000000000000000000000..b5351d7d6bd515a79dce5d3d802a548d9ee752c6 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/3-exporting-all-databases.md @@ -0,0 +1,121 @@ +--- +title: Exporting All Databases +summary: Exporting All Databases +author: Guo Huan +date: 2021-03-04 +--- + +# Exporting All Databases + +## Exporting All Databases + +You can use **gs_dumpall** to export full information of all databases in MogDB, including information about each database and global objects in MogDB. You can specify the information to export as follows: + +- Export full information of all databases, including information about each database and global objects (such as roles and tablespaces) in MogDB. + + You can use the exported information to create a host environment containing the same databases, global objects, and data as the current one. + +- Export data of all databases, excluding all object definitions and global objects. + +- Export all object definitions of all databases, including the definitions of tablespaces, databases, functions, schemas, tables, indexes, and stored procedures. + + You can use the exported object definitions to quickly create a host environment that is the same as the current one, containing the same databases and tablespaces but no data. + +### Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run **gs_dumpall** to export full information of all databases. + + ```bash + gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_backup.sql -p 8000 + ``` + + **Table 1** Common parameters + + | Parameter | Description | Example Value | + | :-------- | :----------------------------------------------------------- | :----------------------------------- | + | -U | Username for database connection. The user must be an MogDB administrator. | -U omm | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W Bigdata@123 | + | -f | Folder to store exported files. If this parameter is not specified, the exported files are stored in the standard output. | -f /home/omm/backup/MPPDB_backup.sql | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_dumpall](../../../reference-guide/tool-reference/server-tools/6-gs_dumpall.md)" in the **Reference Guide**. + +### Examples + +Example 1: Run **gs_dumpall** as the cluster administrator **omm** to export full information of all databases in a cluster. After the command is executed, a large amount of output information will be displayed. **total time** will be displayed at the end of the information, indicating that the backup is successful. In this example, only relative output information is included. + +```bash +gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_backup.sql -p 8000 +gs_dumpall[port='8000'][2017-07-21 15:57:31]: dumpall operation successful +gs_dumpall[port='8000'][2017-07-21 15:57:31]: total time: 9627 ms +``` + +Example 2: Run **gs_dumpall** as the cluster administrator **omm** to export object definitions of all databases in a cluster. The exported files are in text format. After the command is executed, a large amount of output information will be displayed. **total time** will be displayed at the end of the information, indicating that the backup is successful. In this example, only relative output information is included. + +```bash +gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_backup.sql -p 8000 -s +gs_dumpall[port='8000'][2018-11-14 11:28:14]: dumpall operation successful +gs_dumpall[port='8000'][2018-11-14 11:28:14]: total time: 4147 ms +``` + +Example 3: Run **gs_dumpall** to export data of all databases in a cluster, encrypt the exported files, and store them in text format. After the command is executed, a large amount of output information will be displayed. **total time** will be displayed at the end of the information, indicating that the backup is successful. In this example, only relative output information is included. + +```bash +gs_dumpall -f /home/omm/backup/MPPDB_backup.sql -p 8000 -a --with-encryption AES128 --with-key 1234567812345678 +gs_dumpall[port='8000'][2018-11-14 11:32:26]: dumpall operation successful +gs_dumpall[port='8000'][2018-11-14 11:23:26]: total time: 4147 ms +``` + +## Exporting Global Objects + +You can use **gs_dumpall** to export global objects, including database users, user groups, tablespaces, and attributes (for example, global access permissions), from MogDB. + +### Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run **gs_dumpall** to export global tablespaces. + + ```bash + gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_tablespace.sql -p 8000 -t + ``` + + **Table 1** Common parameters + + | Parameter | Description | Example Value | + | :-------- | :----------------------------------------------------------- | :------------------------------------------- | + | -U | Username for database connection. The user must be an MogDB administrator. | -U omm | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W Bigdata@123 | + | -f | Folder to store exported files. If this parameter is not specified, the exported files are stored in the standard output. | -f /home/omm/backup/**MPPDB_tablespace**.sql | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + | -t | Dumps only tablespaces. You can also use **-tablespaces-only** alternatively. | - | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_dumpall](../../../reference-guide/tool-reference/server-tools/6-gs_dumpall.md)" in the **Reference Guide**. + +### Examples + +Example 1: Run **gs_dumpall** as the cluster administrator **omm** to export global tablespaces and users of all databases. The exported files are in text format. + +```bash +gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_globals.sql -p 8000 -g +gs_dumpall[port='8000'][2018-11-14 19:06:24]: dumpall operation successful +gs_dumpall[port='8000'][2018-11-14 19:06:24]: total time: 1150 ms +``` + +Example 2: Run **gs_dumpall** as the cluster administrator **omm** to export global tablespaces of all databases, encrypt the exported files, and store them in text format. + +```bash +gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_tablespace.sql -p 8000 -t --with-encryption AES128 --with-key 1212121212121212 +gs_dumpall[port='8000'][2018-11-14 19:00:58]: dumpall operation successful +gs_dumpall[port='8000'][2018-11-14 19:00:58]: total time: 186 ms +``` + +Example 3: Run **gs_dumpall** as the cluster administrator **omm** to export global users of all databases. The exported files are in text format. + +```bash +gs_dumpall -W Bigdata@123 -U omm -f /home/omm/backup/MPPDB_user.sql -p 8000 -r +gs_dumpall[port='8000'][2018-11-14 19:03:18]: dumpall operation successful +gs_dumpall[port='8000'][2018-11-14 19:03:18]: total time: 162 ms +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/4-data-export-by-a-user-without-required-permissions.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/4-data-export-by-a-user-without-required-permissions.md new file mode 100644 index 0000000000000000000000000000000000000000..7f469518fc4953f024df11bbe704b2323bbee683 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/exporting-data/4-data-export-by-a-user-without-required-permissions.md @@ -0,0 +1,82 @@ +--- +title: Data Export By a User Without Required Permissions +summary: Data Export By a User Without Required Permissions +author: Guo Huan +date: 2021-03-04 +--- + +# Data Export By a User Without Required Permissions + +**gs_dump** and **gs_dumpall** use **-U** to specify the user that performs the export. If the specified user does not have the required permissions, data cannot be exported. In this case, you need to assign the permission to a user who does not have the permission, and then set the **-role** parameter in the export command to specify the role with the permission. Then, **gs_dump** or **gs_dumpall** uses the **-role** parameter to specify a role to export data. + +## Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Use **gs_dump** to export data of the **human_resource** database. + + User **jack** does not have the permissions to export data of the **human_resource** database and the role **role1** has this permission. To export data of the **human_resource** database, you need to assign the permission of **role1** to **jack** and set **-role** to **role1** in the export command. The exported files are in .tar format. + + ```bash + gs_dump -U jack -f /home/omm/backup/MPPDB_backup.tar -p 8000 human_resource --role role1 --rolepassword abc@1234 -F t + Password: + ``` + + **Table 1** Common parameters + + | Parameter | Description | Example Value | + | :------------ | :----------------------------------------------------------- | :----------------------------------- | + | -U | Username for database connection. | -U jack | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W Bigdata@123 | + | -f | Folder to store exported files. If this parameter is not specified, the exported files are stored in the standard output. | -f /home/omm/backup/MPPDB_backup.tar | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + | dbname | Name of the database to export. | human_resource | + | -role | Role name for the export operation. After this parameter is set, the **SET ROLE** command will be issued after **gs_dump** or **gs_dumpall** connects to the database. It is useful when the user specified by **-U** does not have the permissions required by **gs_dump** or **gs_dumpall**. This parameter allows you to switch to a role with the required permissions. | -r role1 | + | -rolepassword | Role password. | -rolepassword abc@1234 | + | -F | Select the format of file to export. The values of **-F** are as follows:
- **p**: plaintext
- **c**: custom
- **d**: directory
- **t**: .tar | -F t | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_dump](../../../reference-guide/tool-reference/server-tools/5-gs_dump.md)" or "[gs_dumpall](../../../reference-guide/tool-reference/server-tools/6-gs_dumpall.md)" in the **Reference Guide**. + +## Examples + +Example 1: User **jack** does not have the permissions required to export data of the **human_resource** database using **gs_dump** and the role **role1** has the permissions. To export data of the **human_resource** database, you can set **-role** to **role1** in the **gs_dump** command. The exported files are in .tar format. + +```bash +$ human_resource=# CREATE USER jack IDENTIFIED BY "1234@abc"; +CREATE ROLE +human_resource=# GRANT role1 TO jack; +GRANT ROLE + +$ gs_dump -U jack -f /home/omm/backup/MPPDB_backup11.tar -p 8000 human_resource --role role1 --rolepassword abc@1234 -F t +Password: +gs_dump[port='8000'][human_resource][2017-07-21 16:21:10]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 16:21:10]: total time: 4239 ms +``` + +Example 2: User **jack** does not have the permissions required to export the **public** schema using **gs_dump** and the role **role1** has the permissions. To export the **public** schema, you can set **-role** to **role1** in the **gs_dump** command. The exported files are in .tar format. + +```bash +$ human_resource=# CREATE USER jack IDENTIFIED BY "1234@abc"; +CREATE ROLE +human_resource=# GRANT role1 TO jack; +GRANT ROLE + +$ gs_dump -U jack -f /home/omm/backup/MPPDB_backup12.tar -p 8000 human_resource -n public --role role1 --rolepassword abc@1234 -F t +Password: +gs_dump[port='8000'][human_resource][2017-07-21 16:21:10]: dump database human_resource successfully +gs_dump[port='8000'][human_resource][2017-07-21 16:21:10]: total time: 3278 ms +``` + +Example 3: User **jack** does not have the permissions required to export all databases in a cluster using **gs_dumpall** and the role **role1** (cluster administrator) has the permissions. To export all the databases, you can set **-role** to **role1** in the **gs_dumpall** command. The exported files are in text format. + +```bash +$ human_resource=# CREATE USER jack IDENTIFIED BY "1234@abc"; +CREATE ROLE +human_resource=# GRANT role1 TO jack; +GRANT ROLE + +$ gs_dumpall -U jack -f /home/omm/backup/MPPDB_backup.sql -p 8000 --role role1 --rolepassword abc@1234 +Password: +gs_dumpall[port='8000'][human_resource][2018-11-14 17:26:18]: dumpall operation successful +gs_dumpall[port='8000'][human_resource][2018-11-14 17:26:18]: total time: 6437 ms +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/1-import-modes.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/1-import-modes.md new file mode 100644 index 0000000000000000000000000000000000000000..db486ed377a2bcd89e66e9230e8246f825bea0f0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/1-import-modes.md @@ -0,0 +1,18 @@ +--- +title: Import Modes +summary: Import Modes +author: Guo Huan +date: 2021-03-04 +--- + +# Import Modes + +You can use **INSERT**, **COPY**, or **\copy** (a **gsql** meta-command) to import data to the MogDB database. The methods have different characteristics. For details, see Table 1. + +**Table 1** Import modes + +| Mode | Characteristics | +| :--------------------------------- | :----------------------------------------------------------- | +| INSERT | Insert one or more rows of data, or insert data from a specified table. | +| COPY | Run the **COPY FROM STDIN** statement to write data into the MogDB database.
Service data does not need to be stored in files when it is written from other databases to the MogDB database through the CopyManager interface driven by JDBC. | +| **\copy**, a **gsql** meta-command | Different from the SQL **COPY** statement, the **\copy** command can read data from or write data into only local files on a **gsql** client.
NOTE:
**\copy** applies only to small-scale data import in good format. It does not preprocess invalid characters or provide error tolerance. Therefore, **\copy** cannot be used in scenarios where abnormal data exists. **COPY** is preferred for data import. | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/10-managing-concurrent-write-operations.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/10-managing-concurrent-write-operations.md new file mode 100644 index 0000000000000000000000000000000000000000..5cd138eb7682d54bc9a971ad82b2a167dcd9eb39 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/10-managing-concurrent-write-operations.md @@ -0,0 +1,177 @@ +--- +title: Managing Concurrent Write Operations +summary: Managing Concurrent Write Operations +author: Guo Huan +date: 2021-03-04 +--- + +# Managing Concurrent Write Operations + +## Transaction Isolation + +MogDB manages transactions based on MVCC and two-phase locks, avoiding conflicts between read and write operations. SELECT is a read-only operation, whereas UPDATE and DELETE are read/write operations. + +- There is no conflict between read/write and read-only operations, or between read/write operations. Each concurrent transaction creates a snapshot when it starts. Concurrent transactions cannot detect updates made by each other. + - At the **READ COMMITTED** level, if transaction T1 is committed, transaction T2 can see changes made by T1. + - At the **REPEATABLE READ** level, if T2 starts before T1 is committed, T2 will not see changes made by T1 even after T1 is committed. The query results in a transaction are consistent and unaffected by other transactions. +- Read/Write operations use row-level locks. Different transactions can concurrently update the same table but not the same row. A row update transaction will start only after the previous one is committed. + - **READ COMMITTED**: At this level, a transaction can access only committed data. This is the default level. + - **REPEATABLE READ**: Only data committed before transaction start is read. Uncommitted data or data committed in other concurrent transactions cannot be read. + +## Write and Read/Write Operations + +Statements for write-only and read/write operations are as follows: + +- **INSERT**, used to insert one or more rows of data into a table +- **UPDATE**, used to modify existing data in a table +- **DELETE**, used to delete existing data from a table +- **COPY**, used to import data + +INSERT and COPY are write-only operations. Only one of them can be performed at a time. If INSERT or COPY of transaction T1 locks a table, INSERT or COPY of transaction T2 needs to wait until T1 unlocks the table. + +UPDATE and DELETE operations are read/write operations. They need to query for the target rows before modifying data. Concurrent transactions cannot see changes made by each other, and UPDATE and DELETE operations read snapshots of data committed before their transactions start. Write operations use row-level locks. If T2 starts after T1 and is to update the same row as T1 does, T2 waits for T1 to finish update. If T1 is not complete within the specified timeout duration, T2 will time out. If T1 and T2 update different rows in a table, they can be concurrently executed. + +## Potential Deadlocks During Concurrent Write + +Whenever transactions involve updates of more than one table, there is always the possibility that concurrently running transactions become deadlocked when they both try to write to the same set of tables. A transaction releases all of its locks at once when it either commits or rolls back; it does not relinquish locks one at a time. For example, transactions T1 and T2 start at roughly the same time. + +- If T1 starts writing to table A and T2 starts writing to table B, both transactions can proceed without conflict. However, if T1 finishes writing to table A and needs to start writing to the same rows as T2 does in table B, it will not be able to proceed because T2 still holds the lock on B. Conversely, if T2 finishes writing to table B and needs to start writing to the same rows as T1 does in table A, it will not be able to proceed either because T1 still holds the lock on A. In this case, a deadlock occurs. If T1 is committed and releases the lock within the lock timeout duration, subsequent update can proceed. If a lock times out, an error is reported and the corresponding transaction exits. +- If T1 updates rows 1 to 5 and T2 updates rows 6 to 10 in the same table, the two transactions do not conflict. However, if T1 finishes the update and proceeds to update rows 6 to 10, and T2 proceeds to update rows 1 to 5, neither of them can continue. If either of the transactions is committed and releases the lock within the lock timeout duration, subsequent update can proceed. If a lock times out, an error is reported and the corresponding transaction exits. + +## Concurrent Write Examples + +This section uses the **test** table as an example to describe how to perform concurrent **INSERT** and **DELETE** in the same table, concurrent **INSERT** in the same table, concurrent **UPDATE** in the same table, and concurrent import and queries. + +```sql +CREATE TABLE test(id int, name char(50), address varchar(255)); +``` + +### Concurrent INSERT and DELETE in the Same Table + +Transaction T1: + +```sql +START TRANSACTION; +INSERT INTO test VALUES(1,'test1','test123'); +COMMIT; +``` + +Transaction T2: + +```sql +START TRANSACTION; +DELETE test WHERE NAME='test1'; +COMMIT; +``` + +Scenario 1: + +T1 is started but not committed. At this time, T2 is started. After **INSERT** of T1 is complete, **DELETE** of T2 is performed. In this case, **DELETE 0** is displayed, because T1 is not committed and T2 cannot see the data inserted by T1. + +Scenario 2: + +- **READ COMMITTED** level + + T1 is started but not committed. At this time, T2 is started. After **INSERT** of T1 is complete, T1 is committed and **DELETE** of T2 is executed. In this case, **DELETE 1** is displayed, because T2 can see the data inserted by T1. + +- **REPEATABLE READ** level + + T1 is started but not committed. At this time, T2 is started. After **INSERT** of T1 is complete, T1 is committed and **DELETE** of T2 is executed. In this case, **DELETE 0** is displayed, because the data obtained in queries is consistent in a transaction. + +### Concurrent INSERT in the Same table + +Transaction T1: + +```sql +START TRANSACTION; +INSERT INTO test VALUES(2,'test2','test123'); +COMMIT; +``` + +Transaction T2: + +```sql +START TRANSACTION; +INSERT INTO test VALUES(3,'test3','test123'); +COMMIT; +``` + +Scenario 1: + +T1 is started but not committed. At this time, T2 is started. After **INSERT** of T1 is complete, **INSERT** of T2 is executed and succeeds. At the **READ COMMITTED** and **REPEATABLE READ** levels, the **SELECT** statement of T1 cannot see data inserted by T2, and a query in T2 cannot see data inserted by T1. + +Scenario 2: + +- **READ COMMITTED** level + + T1 is started but not committed. At this time, T2 is started. After **INSERT** of T1 is complete, T1 is committed. In T2, a query executed after **INSERT** can see the data inserted by T1. + +- **REPEATABLE READ** level + + T1 is started but not committed. At this time, T2 is started. After **INSERT** of T1 is complete, T1 is committed. In T2, a query executed after **INSERT** cannot see the data inserted by T1. + +### Concurrent UPDATE in the Same Table + +Transaction T1: + +```sql +START TRANSACTION; +UPDATE test SET address='test1234' WHERE name='test1'; +COMMIT; +``` + +Transaction T2: + +```sql +START TRANSACTION; +UPDATE test SET address='test1234' WHERE name='test2'; +COMMIT; +``` + +Transaction T3: + +```sql +START TRANSACTION; +UPDATE test SET address='test1234' WHERE name='test1'; +COMMIT; +``` + +Scenario 1: + +T1 is started but not committed. At this time, T2 is started. **UPDATE** of T1 and then T2 starts, and both of them succeed. This is because the **UPDATE** operations use row-level locks and do not conflict when they update different rows. + +Scenario 2: + +T1 is started but not committed. At this time, T3 is started. **UPDATE** of T1 and then T3 starts, and **UPDATE** of T1 succeeds. **UPDATE** of T3 times out. This is because T1 and T3 update the same row and the lock is held by T1 at the time of the update. + +### Concurrent Data Import and Queries + +Transaction T1: + +```sql +START TRANSACTION; +COPY test FROM '...'; +COMMIT; +``` + +Transaction T2: + +```sql +START TRANSACTION; +SELECT * FROM test; +COMMIT; +``` + +Scenario 1: + +T1 is started but not committed. At this time, T2 is started. **COPY** of T1 and then **SELECT** of T2 starts, and both of them succeed. In this case, T2 cannot see the data added by **COPY** of T1. + +Scenario 2: + +- **READ COMMITTED** level + + T1 is started but not committed. At this time, T2 is started. **COPY** of T1 is complete and T1 is committed. In this case, T2 can see the data added by **COPY** of T1. + +- **REPEATABLE READ** level + + T1 is started but not committed. At this time, T2 is started. **COPY** of T1 is complete and T1 is committed. In this case, T2 cannot see the data added by **COPY** of T1. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/2-running-the-INSERT-statement-to-insert-data.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/2-running-the-INSERT-statement-to-insert-data.md new file mode 100644 index 0000000000000000000000000000000000000000..9149247c9a26027cae83556801d74e8124cc959f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/2-running-the-INSERT-statement-to-insert-data.md @@ -0,0 +1,20 @@ +--- +title: Running the INSERT Statement to Insert Data +summary: Running the INSERT Statement to Insert Data +author: Guo Huan +date: 2021-03-04 +--- + +# Running the INSERT Statement to Insert Data + +Run the **INSERT** statement to write data into the MogDB database in either of the following ways: + +- Use the client tool provided by the MogDB database to write data into MogDB. + + For details, see Inserting Data to Tables. + +- Connect to the database using the JDBC or ODBC driver and run the **INSERT** statement to write data into the MogDB database. + + For details, see Connecting to a Database. + +You can add, modify, and delete database transactions for the MogDB database. **INSERT** is the simplest way to write data and applies to scenarios with small data volume and low concurrency. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/3-running-the-COPY-FROM-STDIN-statement-to-import-data.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/3-running-the-COPY-FROM-STDIN-statement-to-import-data.md new file mode 100644 index 0000000000000000000000000000000000000000..4cd4fbce8c8e8214ffc5d307f4425ef0868f7cde --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/3-running-the-COPY-FROM-STDIN-statement-to-import-data.md @@ -0,0 +1,318 @@ +--- +title: Running the COPY FROM STDIN Statement to Import Data +summary: Running the COPY FROM STDIN Statement to Import Data +author: Guo Huan +date: 2021-03-04 +--- + +# Running the COPY FROM STDIN Statement to Import Data + +
+ +## Data Import Using COPY FROM STDIN + +Run the **COPY FROM STDIN** statement to import data to MogDB in either of the following ways: + +- Write data into the MogDB database by typing. For details, see COPY. +- Import data from a file or database to MogDB through the CopyManager interface driven by JDBC. You can use any parameters in the **COPY** syntax. + +
+ +## Introduction to the CopyManager Class + +CopyManager is an API class provided by the JDBC driver in MogDB. It is used to import data to the MogDB database in batches. + +
+ +### Inheritance Relationship of CopyManager + +The CopyManager class is in the **org.opengauss.copy** package and inherits the java.lang.Object class. The declaration of the class is as follows: + +```java +public class CopyManager +extends Object +``` + +
+ +### Construction Method + +```java +public CopyManager(BaseConnection connection) +throws SQLException +``` + +
+ +### Common Methods + +**Table 1** Common methods of CopyManager + +| Return Value | Method | Description | throws | +| :----------- | :--------------------------------------------------- | :----------------------------------------------------------- | :----------------------- | +| CopyIn | copyIn(String sql) | - | SQLException | +| long | copyIn(String sql, InputStream from) | Uses **COPY FROM STDIN** to quickly import data to tables in a database from InputStream. | SQLException,IOException | +| long | copyIn(String sql, InputStream from, int bufferSize) | Uses **COPY FROM STDIN** to quickly import data to tables in a database from InputStream. | SQLException,IOException | +| long | copyIn(String sql, Reader from) | Uses **COPY FROM STDIN** to quickly import data to tables in a database from Reader. | SQLException,IOException | +| long | copyIn(String sql, Reader from, int bufferSize) | Uses **COPY FROM STDIN** to quickly import data to tables in a database from Reader. | SQLException,IOException | +| CopyOut | copyOut(String sql) | - | SQLException | +| long | copyOut(String sql, OutputStream to) | Sends the result set of **COPY TO STDOUT** from the database to the OutputStream class. | SQLException,IOException | +| long | copyOut(String sql, Writer to) | Sends the result set of **COPY TO STDOUT** from the database to the Writer class. | SQLException,IOException | + +
+ +## Handling Import Errors + +### Scenarios + +Handle errors that occurred during data import. + +### Querying Error Information + +Errors that occur when data is imported are divided into data format errors and non-data format errors. + +- Data format errors + + When creating a foreign table, specify **LOG INTO error_table_name**. Data format errors during data import will be written into the specified table. You can run the following SQL statement to query error details: + + ```sql + mogdb=# SELECT * FROM error_table_name; + ``` + + Table 1 lists the columns of the *error_table_name* table. + + **Table 1** Columns in the error information table + + | Column Name | Type | Description | + | :---------- | :----------------------- | :----------------------------------------------------------- | + | nodeid | integer | ID of the node where an error is reported | + | begintime | timestamp with time zone | Time when a data format error was reported | + | filename | character varying | Name of the source data file where a data format error occurs | + | rownum | numeric | Number of the row where a data format error occurs in a source data file | + | rawrecord | text | Raw record of a data format error in the source data file | + | detail | text | Error details | + +- Non-data format errors + + A non-data format error leads to the failure of an entire data import task. You can locate and troubleshoot a non-data format error based on the error message displayed during data import. + +### Handling Data Import Errors + +Troubleshoot data import errors based on obtained error information and descriptions in the following table. + +**Table 2** Handling data import errors + +| Error Message | Cause | Solution | +| :----------------------------------------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | +| missing data for column "r_reason_desc" | 1. The number of columns in the source data file is less than that in the foreign table.
2. In a TEXT-format source data file, an escape character (for example, ) leads to delimiter or quote mislocation.
**Example:** The target table contains three columns, and the following data is imported. The escape character () converts the delimiter (\|) into the value of the second column, causing the value of the third column to lose.
`BE|Belgium|1` | 1. If an error is reported due to missing columns, perform the following operations:
- Add the value of the **r_reason_desc** column to the source data file.
- When creating a foreign table, set the parameter **fill_missing_fields** to **on**. In this way, if the last column of a row in the source data file is missing, it will be set to **NULL** and no error will be reported.
2. Check whether the row where an error is reported contains the escape character (). If the row contains such a character, you are advised to set the parameter **noescaping** to **true** when creating a foreign table, indicating that the escape character () and the characters following it are not escaped. | +| extra data after last expected column | The number of columns in the source data file is greater than that in the foreign table. | - Delete extra columns from the source data file.
- When creating a foreign table, set the parameter **ignore_extra_data** to **on**. In this way, if the number of columns in the source data file is greater than that in the foreign table, the extra columns at the end of rows will not be imported. | +| invalid input syntax for type numeric: "a" | The data type is incorrect. | In the source data file, change the data type of the columns to import. If this error information is displayed, change the data type to **numeric**. | +| null value in column "staff_id" violates not-null constraint | The not-null constraint is violated. | In the source data file, add values to the specified columns. If this error information is displayed, add values to the **staff_id** column. | +| duplicate key value violates unique constraint "reg_id_pk" | The unique constraint is violated. | - Delete duplicate rows from the source data file.
- Run the **SELECT** statement with the **DISTINCT** keyword to ensure that all imported rows are unique.
`mogdb=# INSERT INTO reasons SELECT DISTINCT * FROM foreign_tpcds_reasons;` | +| value too long for type character varying(16) | The column length exceeds the upper limit. | In the source data file, change the column length. If this error information is displayed, reduce the column length to no greater than 16 bytes (VARCHAR2). | + +
+ +## Example 1: Importing and Exporting Data Through Local Files + +When the JAVA language is used for secondary development based on MogDB, you can use the CopyManager interface to export data from the database to a local file or import a local file to the database by streaming. The file can be in CSV or TEXT format. + +The sample program is as follows. Load the MogDB JDBC driver before executing it. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.io.IOException; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.sql.SQLException; +import org.opengauss.copy.CopyManager; +import org.opengauss.core.BaseConnection; + +public class Copy{ + + public static void main(String[] args) + { + String urls = new String("jdbc:opengauss://localhost:8000/postgres"); // URL of the database + String username = new String("username"); // Username + String password = new String("passwd"); // Password + String tablename = new String("migration_table"); // Table information + String tablename1 = new String("migration_table_1"); // Table information + String driver = "org.opengauss.Driver"; + Connection conn = null; + + try { + Class.forName(driver); + conn = DriverManager.getConnection(urls, username, password); + } catch (ClassNotFoundException e) { + e.printStackTrace(System.out); + } catch (SQLException e) { + e.printStackTrace(System.out); + } + + // Export data from the migration_table table to the d:/data.txt file. + try { + copyToFile(conn, "d:/data.txt", "(SELECT * FROM migration_table)"); + } catch (SQLException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (IOException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + // Import data from the d:/data.txt file to the migration_table_1 table. + try { + copyFromFile(conn, "d:/data.txt", tablename1); + } catch (SQLException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (IOException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + + // Export data from the migration_table_1 table to the d:/data1.txt file. + try { + copyToFile(conn, "d:/data1.txt", tablename1); + } catch (SQLException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (IOException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + } + + public static void copyFromFile(Connection connection, String filePath, String tableName) + throws SQLException, IOException { + + FileInputStream fileInputStream = null; + + try { + CopyManager copyManager = new CopyManager((BaseConnection)connection); + fileInputStream = new FileInputStream(filePath); + copyManager.copyIn("COPY " + tableName + " FROM STDIN with (" + "DELIMITER"+"'"+ delimiter + "'" + "ENCODING " + "'" + encoding + "')", fileInputStream); + } finally { + if (fileInputStream != null) { + try { + fileInputStream.close(); + } catch (IOException e) { + e.printStackTrace(); + } + } + } + } + public static void copyToFile(Connection connection, String filePath, String tableOrQuery) + throws SQLException, IOException { + + FileOutputStream fileOutputStream = null; + + try { + CopyManager copyManager = new CopyManager((BaseConnection)connection); + fileOutputStream = new FileOutputStream(filePath); + copyManager.copyOut("COPY " + tableOrQuery + " TO STDOUT", fileOutputStream); + } finally { + if (fileOutputStream != null) { + try { + fileOutputStream.close(); + } catch (IOException e) { + e.printStackTrace(); + } + } + } + } +} +``` + +
+ +## Example 2: Migrating Data from a MySQL Database to the MogDB Database + +The following example shows how to use CopyManager to migrate data from MySQL to the MogDB database. + +```java +import java.io.StringReader; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; + +import org.opengauss.copy.CopyManager; +import org.opengauss.core.BaseConnection; + +public class Migration{ + + public static void main(String[] args) { + String url = new String("jdbc:opengauss://localhost:8000/postgres"); // URL of the database + String user = new String("username"); // MogDB database user name + String pass = new String("passwd"); // MogDB database password + String tablename = new String("migration_table_1"); // Table information + String delimiter = new String("|"); // Delimiter + String encoding = new String("UTF8"); // Character set + String driver = "org.opengauss.Driver"; + StringBuffer buffer = new StringBuffer(); // Buffer to store formatted data + + try { + // Obtain the query result set of the source database. + ResultSet rs = getDataSet(); + + // Traverse the result set and obtain records row by row. + // The values of columns in each record are separated by the specified delimiter and end with a linefeed, forming strings. + // Add the strings to the buffer. + while (rs.next()) { + buffer.append(rs.getString(1) + delimiter + + rs.getString(2) + delimiter + + rs.getString(3) + delimiter + + rs.getString(4) + + "\n"); + } + rs.close(); + + try { + // Connect to the target database. + Class.forName(driver); + Connection conn = DriverManager.getConnection(url, user, pass); + BaseConnection baseConn = (BaseConnection) conn; + baseConn.setAutoCommit(false); + + // Initialize the table. + String sql = "Copy " + tablename + " from STDIN with (DELIMITER " + "'" + delimiter + "'" +","+ " ENCODING " + "'" + encoding + "'"); + + // Commit data in the buffer. + CopyManager cp = new CopyManager(baseConn); + StringReader reader = new StringReader(buffer.toString()); + cp.copyIn(sql, reader); + baseConn.commit(); + reader.close(); + baseConn.close(); + } catch (ClassNotFoundException e) { + e.printStackTrace(System.out); + } catch (SQLException e) { + e.printStackTrace(System.out); + } + + } catch (Exception e) { + e.printStackTrace(); + } + } + + //******************************** + // Return the query result set from the source database. + //********************************* + private static ResultSet getDataSet() { + ResultSet rs = null; + try { + Class.forName("com.MY.jdbc.Driver").newInstance(); + Connection conn = DriverManager.getConnection("jdbc:MY://10.119.179.227:3306/jack?useSSL=false&allowPublicKeyRetrieval=true", "jack", "Enmo@123"); + Statement stmt = conn.createStatement(); + rs = stmt.executeQuery("select * from migration_table"); + } catch (SQLException e) { + e.printStackTrace(); + } catch (Exception e) { + e.printStackTrace(); + } + return rs; + } +} +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/4-using-a-gsql-meta-command-to-import-data.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/4-using-a-gsql-meta-command-to-import-data.md new file mode 100644 index 0000000000000000000000000000000000000000..2215c6858e3499faa37d2eb157d1994e6f1ce453 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/4-using-a-gsql-meta-command-to-import-data.md @@ -0,0 +1,207 @@ +--- +title: Using a gsql Meta-Command to Import Data +summary: Using a gsql Meta-Command to Import Data +author: Guo Huan +date: 2021-03-04 +--- + +# Using a gsql Meta-Command to Import Data + +The **gsql** tool provides the **\copy** meta-command to import data. + +**\copy Command** + +## Syntax + +``` +\copy { table [ ( column_list ) ] | + +( query ) } { from | to } { filename | + +stdin | stdout | pstdin | pstdout } + +[ with ] [ binary ] [ delimiter + +[ as ] 'character' ] [ null [ as ] 'string' ] + +[ csv [ header ] [ quote [ as ] + +'character' ] [ escape [ as ] 'character' ] + +[ force quote column_list | * ] [ force + +not null column_list ] ] +``` + +You can run this command to import or export data after logging in to a database on any gsql client. Different from the **COPY** statement in SQL, this command performs read/write operations on local files rather than files on database servers. The accessibility and permissions of the local files are restricted to local users. + +> **NOTE:** +> +> **\copy** applies only to small-scale data import in good format. It does not preprocess invalid characters or provide error tolerance. Therefore, **\copy** cannot be used in scenarios where abnormal data exists. **GDS** or **COPY** is preferred for data import. + +**Parameter Description** + +- table + + Specifies the name (possibly schema-qualified) of an existing table. + + Value range: an existing table name + +- column_list + + Specifies an optional list of columns to be copied. + + Value range: any field in the table. If no column list is specified, all columns of the table will be copied. + +- query + + Specifies that the results are to be copied. + + Value range: a **SELECT** or **VALUES** command in parentheses + +- filename + + Specifies the absolute path of a file. To run the **COPY** command, the user must have the write permission for this path. + +- stdin + + Specifies that input comes from the standard input. + +- stdout + + Specifies that output goes to the standard output. + +- pstdin + + Specifies that input comes from the gsql client. + +- pstout + +- Specifies that output goes to the gsql client. + +- binary + + Specifies that data is stored and read in binary mode instead of text mode. In binary mode, you cannot declare **DELIMITER**, **NULL**, or **CSV**. After **binary** is specified, CSV, FIXED, and TEXT cannot be specified through **option** or **copy_option**. + +- delimiter [ as ] 'character' + + Specifies the character that separates columns within each row (line) of the file. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The value of **delimiter** cannot be **\r** or **\n**. + > - A delimiter cannot be the same as the null value. The delimiter for the CSV format cannot be same as the **quote** value. + > - The delimiter of TEXT data cannot contain any of the following characters: \\.abcdefghijklmnopqrstuvwxyz0123456789. + > - The data length of a single row should be less than 1 GB. A row that has many columns using long delimiters cannot contain much valid data. + > - You are advised to use multi-character delimiters or invisible delimiters. For example, you can use multi-characters (such as $^&) and invisible characters (such as 0x07, 0x08, and 0x1b). + + Value range: a multi-character delimiter within 10 bytes + + Default value: + + - A tab character in TEXT format + - A comma (,) in CSV format + - No delimiter in FIXED format + +- null [ as ] 'string' + + Specifies the string that represents a null value. + + Value range: + + - A null value cannot be **\\r** or **\\n**. The maximum length is 100 characters. + - A null value cannot be the same as the **delimiter** or **quote** value. + + Default value: + + - The default value for the CSV format is an empty string without quotation marks. + - The default value for the TEXT format is **\\N**. + +- header + + Specifies whether a file contains a header with the names of each column in the file. **header** is available only for CSV and FIXED files. + + When data is imported, if **header** is **on**, the first row of the data file will be identified as the header and ignored. If **header** is **off**, the first row will be identified as a data row. + + When data is exported, if header is **on**, **fileheader** must be specified. **fileheader** specifies the content in the header. If **header** is **off**, an exported file does not contain a header. + + Value range:**true/on** and **false/off** + + Default value: false + +- quote [ as ] 'character' + + Specifies a quoted character string for a CSV file. + + Default value: a double quotation mark (") + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The value of **quote** cannot be the same as that of the **delimiter** or null parameter. + > - The value of **quote** must be a single-byte character. + > - Invisible characters are recommended, such as 0x07, 0x08, and 0x1b. + +- escape [ as ] 'character' + + Specifies an escape character for a CSV file. The value must be a single-byte character. + + Default value: a double quotation mark (") If the value is the same as that of **quote**, it will be replaced by **\0**. + +- force quote column_list | * + + In **CSV COPY TO** mode, forces quotation marks to be used for all non-null values in each specified column. Null values are not quoted. + + Value range: an existing column name + +- force not null column_list + + Assigns a value to a specified column in **CSV COPY FROM** mode. + + Value range: an existing column name + +**Examples** + +1. Create a target table **a**. + + ```sql + mogdb=# CREATE TABLE a(a int); + ``` + +2. Import data. + + Copy data from **stdin** to table **a**. + + ```sql + mogdb=# \copy a from stdin; + ``` + + When the **>>** characters are displayed, enter data. To end your input, enter a backslash and a period (\.). + + ```sql + Enter data to be copied followed by a newline. + End with a backslash and a period on a line by itself. + >> 1 + >> 2 + >> \. + ``` + + Query data imported to table **a**. + + ```sql + mogdb=# SELECT * FROM a; + a + --- + 1 + 2 + (2 rows) + ``` + +3. Copy data from a local file to table **a**. The following assumes that the local file is **/home/omm/2.csv**. + + - Commas (,) are used as delimiters. + + - If the number of columns defined in a source data file is greater than that in a foreign table, extra columns will be ignored during import. + + ```sql + mogdb=# \copy a FROM '/home/omm/2.csv' WITH (delimiter',',IGNORE_EXTRA_DATA 'on'); + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/5-using-gs_restore-to-import-data.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/5-using-gs_restore-to-import-data.md new file mode 100644 index 0000000000000000000000000000000000000000..43ee79d09dce5c01d4283371f2907ad2f26caeda --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/5-using-gs_restore-to-import-data.md @@ -0,0 +1,263 @@ +--- +title: Using gs_restore to Import Data +summary: Using gs_restore to Import Data +author: Guo Huan +date: 2021-03-04 +--- + +# Using gs_restore to Import Data + +## Scenarios + +**gs_restore** is an import tool provided by the MogDB database. You can use **gs_restore** to import the files exported by **gs_dump** to a database. **gs_restore** can import the files in .tar, custom, or directory format. + +**gs_restore** can: + +- Import data to a database. + + If a database is specified, data is imported to the database. If multiple databases are specified, the password for connecting to each database also needs to be specified. + +- Import data to a script. + + If no database is specified, a script containing the SQL statement to recreate the database is created and written to a file or standard output. This script output is equivalent to the plain text output of **gs_dump**. + +You can specify and sort the data to import. + +## Procedure + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> **gs_restore** incrementally imports data by default. To prevent data exception caused by consecutive imports, use the **-e** and **-c** parameters for each import. **-c** indicates that existing data is deleted from the target database before each import. **-e** indicates that the system ignores the import task with an error (error message is displayed after the import process is complete) and proceeds with the next by default. Therefore, you need to exit the system if an error occurs when you send the SQL statement to the database. + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Use **gs_restore** to import all object definitions from the exported file of the entire **mogdb** database to the **backupdb** database. + + ```bash + $ gs_restore -U jack /home/omm/backup/MPPDB_backup.tar -p 8000 -d backupdb -s -e -c + Password: + ``` + + **Table 1** Common parameters + + | Parameters | Description | Example Value | + | :--------- | :----------------------------------------------------------- | :------------ | + | -U | Username for database connection. | -U jack | + | -W | User password for database connection.
- This parameter is not required for database administrators if the trust policy is used for authentication.
- If you connect to the database without specifying this parameter and you are not a database administrator, you will be prompted to enter the password. | -W abcd@123 | + | -d | Database to which data will be imported. | -d backupdb | + | -p | TCP port or local Unix-domain socket file extension on which the server is listening for connections. | -p 8000 | + | -e | Exits if an error occurs when you send the SQL statement to the database. Error messages are displayed after the import process is complete. | - | + | -c | Cleans existing objects from the target database before the import. | - | + | -s | Imports only object definitions in schemas and does not import data. Sequence values will also not be imported. | - | + + For details about other parameters, see "Tool Reference > Server Tools > [gs_restore](../../../reference-guide/tool-reference/server-tools/9-gs_restore.md)" in the **Reference Guide**. + +## Examples + +Example 1: Run **gs_restore** to import data and all object definitions of the **mogdb** database from the **MPPDB_backup.dmp** file (custom format). + +```bash +$ gs_restore backup/MPPDB_backup.dmp -p 8000 -d backupdb +Password: +gs_restore[2017-07-21 19:16:26]: restore operation successful +gs_restore: total time: 13053 ms +``` + +Example 2: Run **gs_restore** to import data and all object definitions of the **mogdb** database from the **MPPDB_backup.tar** file. + +```bash +$ gs_restore backup/MPPDB_backup.tar -p 8000 -d backupdb +gs_restore[2017-07-21 19:21:32]: restore operation successful +gs_restore[2017-07-21 19:21:32]: total time: 21203 ms +``` + +Example 3: Run **gs_restore** to import data and all object definitions of the **mogdb** database from the **MPPDB_backup** directory. + +```bash +$ gs_restore backup/MPPDB_backup -p 8000 -d backupdb +gs_restore[2017-07-21 19:26:46]: restore operation successful +gs_restore[2017-07-21 19:26:46]: total time: 21003 ms +``` + +Example 4: Run **gs_restore** to import all object definitions of the database from the **MPPDB_backup.tar** file to the **backupdb** database. Table data is not imported. + +```bash +$ gs_restore /home/omm/backup/MPPDB_backup.tar -p 8000 -d backupdb -s -e -c +Password: +gs_restore[2017-07-21 19:46:27]: restore operation successful +gs_restore[2017-07-21 19:46:27]: total time: 32993 ms +``` + +Example 5: Run **gs_restore** to import data and all definitions in the **PUBLIC** schema from the **MPPDB_backup.dmp** file. Existing objects are deleted from the target database before the import. If an existing object references to an object in another schema, manually delete the referenced object first. + +```bash +$ gs_restore backup/MPPDB_backup.dmp -p 8000 -d backupdb -e -c -n PUBLIC +gs_restore: [archiver (db)] Error while PROCESSING TOC: +gs_restore: [archiver (db)] Error from TOC entry 313; 1259 337399 TABLE table1 gaussdba +gs_restore: [archiver (db)] could not execute query: ERROR: cannot drop table table1 because other objects depend on it +DETAIL: view t1.v1 depends on table table1 +HINT: Use DROP ... CASCADE to drop the dependent objects too. +Command was: DROP TABLE public.table1; +``` + +Manually delete the referenced object and create it again after the import is complete. + +```bash +$ gs_restore backup/MPPDB_backup.dmp -p 8000 -d backupdb -e -c -n PUBLIC +gs_restore[2017-07-21 19:52:26]: restore operation successful +gs_restore[2017-07-21 19:52:26]: total time: 2203 ms +``` + +Example 6: Run **gs_restore** to import the definition of the **hr.staffs** table in the **hr** schema from the **MPPDB_backup.dmp** file. Before the import, the **hr.staffs** table does not exist. + +```bash +$ gs_restore backup/MPPDB_backup.dmp -p 8000 -d backupdb -e -c -s -n hr -t hr.staffs +gs_restore[2017-07-21 19:56:29]: restore operation successful +gs_restore[2017-07-21 19:56:29]: total time: 21000 ms +``` + +Example 7: Run **gs_restore** to import data of the **hr.staffs** table in **hr** schema from the **MPPDB_backup.dmp** file. Before the import, the **hr.staffs** table is empty. + +```bash +$ gs_restore backup/MPPDB_backup.dmp -p 8000 -d backupdb -e -a -n hr -t hr.staffs +gs_restore[2017-07-21 20:12:32]: restore operation successful +gs_restore[2017-07-21 20:12:32]: total time: 20203 ms +``` + +Example 8: Run **gs_restore** to import the definition of the **hr.staffs** table. Before the import, the **hr.staffs** table already exists. + +```sql +human_resource=# select * from hr.staffs; + staff_id | first_name | last_name | email | phone_number | hire_date | employment_id | salary | commission_pct | manager_id | section_id +----------+-------------+-------------+----------+--------------------+---------------------+---------------+----------+----------------+------------+------------ + 200 | Jennifer | Whalen | JWHALEN | 515.123.4444 | 1987-09-17 00:00:00 | AD_ASST | 4400.00 | | 101 | 10 + 201 | Michael | Hartstein | MHARTSTE | 515.123.5555 | 1996-02-17 00:00:00 | MK_MAN | 13000.00 | | 100 | 20 + +$ gsql -d human_resource -p 8000 + +gsql ((MogDB x.x.x build 56189e20) compiled at 2022-01-07 18:47:53 commit 0 last mr ) +Non-SSL connection (SSL connection is recommended when requiring high-security) +Type "help" for help. + +human_resource=# drop table hr.staffs CASCADE; +NOTICE: drop cascades to view hr.staff_details_view +DROP TABLE + +$ gs_restore /home/omm/backup/MPPDB_backup.tar -p 8000 -d human_resource -n hr -t staffs -s -e +restore operation successful +total time: 904 ms + +human_resource=# select * from hr.staffs; + staff_id | first_name | last_name | email | phone_number | hire_date | employment_id | salary | commission_pct | manager_id | section_id +----------+------------+-----------+-------+--------------+-----------+---------------+--------+----------------+------------+------------ +(0 rows) +``` + +Example 9: Run **gs_restore** to import data and definitions of the **staffs** and **areas** tables. Before the import, the **staffs** and **areas** tables do not exist. + +```sql +human_resource=# \d + List of relations + Schema | Name | Type | Owner | Storage +--------+--------------------+-------+----------+---------------------------------- + hr | employment_history | table | omm | {orientation=row,compression=no} + hr | employments | table | omm | {orientation=row,compression=no} + hr | places | table | omm | {orientation=row,compression=no} + hr | sections | table | omm | {orientation=row,compression=no} + hr | states | table | omm | {orientation=row,compression=no} +(5 rows) + +$ gs_restore /home/mogdb/backup/MPPDB_backup.tar -p 8000 -d human_resource -n hr -t staffs -n hr -t areas +restore operation successful +total time: 724 ms + +human_resource=# \d + List of relations + Schema | Name | Type | Owner | Storage +--------+--------------------+-------+----------+---------------------------------- + hr | areas | table | omm | {orientation=row,compression=no} + hr | employment_history | table | omm | {orientation=row,compression=no} + hr | employments | table | omm | {orientation=row,compression=no} + hr | places | table | omm | {orientation=row,compression=no} + hr | sections | table | omm | {orientation=row,compression=no} + hr | staffs | table | omm | {orientation=row,compression=no} + hr | states | table | omm | {orientation=row,compression=no} +(7 rows) + +human_resource=# select * from hr.areas; + area_id | area_name +---------+------------------------ + 4 | Middle East and Africa + 1 | Europe + 2 | Americas + 3 | Asia +(4 rows) +``` + +Example 10: Run **gs_restore** to import data and all object definitions in the **hr** schema. + +```bash +$ gs_restore /home/omm/backup/MPPDB_backup1.dmp 8000 -d backupdb -n hr -e -c +restore operation successful +total time: 702 ms +``` + +Example 11: Run **gs_restore** to import all object definitions in the **hr** and **hr1** schemas to the **backupdb** database. + +```bash +$ gs_restore /home/omm/backup/MPPDB_backup2.dmp -p 8000 -d backupdb -n hr -n hr1 -s +restore operation successful +total time: 665 ms +``` + +Example 12: Run **gs_restore** to decrypt the files exported from the **human_resource** database and import them to the **backupdb** database. + +```sql +mogdb=# create database backupdb; +CREATE DATABASE + +$ gs_restore /home/omm/backup/MPPDB_backup.tar -p 8000 -d backupdb --with-key=1234567812345678 +restore operation successful +total time: 23472 ms + +$ gsql -d backupdb -p 8000 -r + +gsql ((MogDB x.x.x build 56189e20) compiled at 2022-01-07 18:47:53 commit 0 last mr ) +Non-SSL connection (SSL connection is recommended when requiring high-security) +Type "help" for help. + +backupdb=# select * from hr.areas; + area_id | area_name +---------+------------------------ + 4 | Middle East and Africa + 1 | Europe + 2 | Americas + 3 | Asia +(4 rows) +``` + +Example 13: **user 1** does not have the permission to import data from an exported file to the **backupdb** database and **role1** has this permission. To import the exported data to the **backupdb** database, you can set **-role** to **role1** in the **gs_restore** command. + +```sql +human_resource=# CREATE USER user1 IDENTIFIED BY "1234@abc"; +CREATE ROLE role1 with SYSADMIN IDENTIFIED BY "abc@1234"; + +$ gs_restore -U user1 /home/omm/backup/MPPDB_backup.tar -p 8000 -d backupdb --role role1 --rolepassword abc@1234 +Password: +restore operation successful +total time: 554 ms + +$ gsql -d backupdb -p 8000 -r + +gsql ((MogDB x.x.x build 56189e20) compiled at 2022-01-07 18:47:53 commit 0 last mr ) +Non-SSL connection (SSL connection is recommended when requiring high-security) +Type "help" for help. + +backupdb=# select * from hr.areas; + area_id | area_name +---------+------------------------ + 4 | Middle East and Africa + 1 | Europe + 2 | Americas + 3 | Asia +(4 rows) +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/6-updating-data-in-a-table.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/6-updating-data-in-a-table.md new file mode 100644 index 0000000000000000000000000000000000000000..d1a64607026b303845f0533fe68e5b040ec2a16a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/6-updating-data-in-a-table.md @@ -0,0 +1,166 @@ +--- +title: Updating Data in a Table +summary: Updating Data in a Table +author: Guo Huan +date: 2021-03-04 +--- + +# Updating Data in a Table + +## Updating a Table by Using DML Statements + +In MogDB, you can update a table by running DML statements. + +### Procedure + +There is a table named **customer_t** and the table structure is as follows: + +```sql +CREATE TABLE customer_t +( c_customer_sk integer, + c_customer_id char(5), + c_first_name char(6), + c_last_name char(8) +) ; +``` + +You can run the following DML statements to update data in the table. + +- Run the **INSERT** statement to insert data into the table. + + - Insert a row to the **customer_t** table. + + ```sql + INSERT INTO customer_t (c_customer_sk, c_customer_id, c_first_name,c_last_name) VALUES (3769, 5, 'Grace','White'); + ``` + + - Insert multiple rows to the **customer_t** table. + + ```sql + INSERT INTO customer_t (c_customer_sk, c_customer_id, c_first_name,c_last_name) VALUES + (6885, 1, 'Joes', 'Hunter'), + (4321, 2, 'Lily','Carter'), + (9527, 3, 'James', 'Cook'), + (9500, 4, 'Lucy', 'Baker'); + ``` + + For details on how to use **INSERT**, see Inserting Data to Tables. + +- Run the **UPDATE** statement to update data in the table. Change the value of the **c_customer_id** column to **0**. + + ```sql + UPDATE customer_t SET c_customer_id = 0; + ``` + + For details on how to use **UPDATE**, see UPDATE. + +- Run the **DELETE** statement to delete rows from the table. + + You can use the **WHERE** clause to specify the rows whose data is to delete. If you do not specify it, all rows in the table are deleted and only the data structure is retained. + + ```sql + DELETE FROM customer_t WHERE c_last_name = 'Baker'; + ``` + + For details on how to use **DELETE**, see DELETE. + +- Run the **TRUNCATE** statement to delete all rows from the table. + + ```sql + TRUNCATE TABLE customer_t; + ``` + + For details on how to use **TRUNCATE**, see TRUNCATE. + + The **DELETE** statement deletes a row of data each time whereas the **TRUNCATE** statement deletes data by releasing the data page stored in the table. Therefore, data can be deleted more quickly by using **TRUNCATE** than using **DELETE**. + + **DELETE** deletes table data but does not release table storage space. **TRUNCATE** deletes table data and releases table storage space. + +## Updating and Inserting Data by Using the MERGE INTO Statement + +To add all or a large amount of data in a table to an existing table, you can run the **MERGE INTO** statement in MogDB to merge the two tables so that data can be quickly added to the existing table. + +The **MERGE INTO** statement matches data in a source table with that in a target table based on a join condition. If data matches, **UPDATE** will be executed on the target table. Otherwise, **INSERT** will be executed. This statement is a convenient way to combine multiple operations and avoids multiple **INSERT** or **UPDATE** statements. + +### Prerequisites + +You have the **INSERT** and **UPDATE** permissions for the target table and the **SELECT** permission for the source table. + +### Procedure + +1. Create a source table named **products** and insert data. + + ```sql + mogdb=# CREATE TABLE products + ( product_id INTEGER, + product_name VARCHAR2(60), + category VARCHAR2(60) + ); + + mogdb=# INSERT INTO products VALUES + (1502, 'olympus camera', 'electrncs'), + (1601, 'lamaze', 'toys'), + (1666, 'harry potter', 'toys'), + (1700, 'wait interface', 'books'); + ``` + +2. Create a target table named **newproducts** and insert data. + + ```sql + mogdb=# CREATE TABLE newproducts + ( product_id INTEGER, + product_name VARCHAR2(60), + category VARCHAR2(60) + ); + + mogdb=# INSERT INTO newproducts VALUES + (1501, 'vivitar 35mm', 'electrncs'), + (1502, 'olympus ', 'electrncs'), + (1600, 'play gym', 'toys'), + (1601, 'lamaze', 'toys'), + (1666, 'harry potter', 'dvd'); + ``` + +3. Run the **MERGE INTO** statement to merge data in the source table **products** into the target table **newproducts**. + + ```sql + MERGE INTO newproducts np + USING products p + ON (np.product_id = p.product_id ) + WHEN MATCHED THEN + UPDATE SET np.product_name = p.product_name, np.category = p.category + WHEN NOT MATCHED THEN + INSERT VALUES (p.product_id, p.product_name, p.category) ; + ``` + + For details on parameters in the statement, see [Table 1](#Parameters in the MERGE INTO statement). For more information, see MERGE INTO. + + **Table 1** Parameters in the MERGE INTO statement + + | Parameter | Description | Example Value | + | :-------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | + | **INTO** clause | Specifies a target table that is to be updated or has data to be inserted.
A table alias is supported. | Value: newproducts np
The table name is **newproducts** and the alias is **np**. | + | **USING** clause | Specifies a source table. A table alias is supported.
If the target table is a replication table, the source table must also be a replication table. | Value: products p
The table name is **products** and the alias is **p**. | + | **ON** clause | Specifies a join condition between a target table and a source table.
Columns in the join condition cannot be updated. | Value: np.product_id = p.product_id
The join condition is that the **product_id** column in the target table **newproducts** has equivalent values as the **product_id** column in the source table **products**. | + | **WHEN MATCHED** clause | Performs **UPDATE** if data in the source table matches that in the target table based on the condition.
- Only one **WHEN MATCHED** clause can be specified.
- The **WHEN MATCHED** clause can be omitted. If it is omitted, no operation will be performed on the rows that meet the condition in the **ON** clause.
- Columns involved in the distribution key of the target table cannot be updated. | Value: WHEN MATCHED THEN UPDATE SET np.product_name = p.product_name, np.category = p.category
When the condition in the **ON** clause is met, the values of the **product_name** and **category** columns in the target table **newproducts** are replaced with the values in the corresponding columns in the source table **products**. | + | **WHEN NOT MATCHED** clause | Performs **INSERT** if data in the source table does not match that in the target table based on the condition.
- Only one **WHEN NOT MATCHED** clause can be specified.
- The **WHEN NOT MATCHED** clause can be omitted.
- An **INSERT** clause can contain only one **VALUES**.
- The **WHEN MATCHED** and **WHEN NOT MATCHED** clauses can be exchanged in sequence. One of them can be omitted, but they cannot be omitted at the same time. | Value: WHEN NOT MATCHED THEN INSERT VALUES (p.product_id, p.product_name, p.category)
Insert rows in the source table **products** that do not meet the condition in the **ON** clause into the target table **newproducts**. | + +4. Query the target table **newproducts** after the merge. + + ```sql + SELECT * FROM newproducts; + ``` + + The command output is as follows: + + ```sql + product_id | product_name | category + ------------+----------------+----------- + 1501 | vivitar 35mm | electrncs + 1502 | olympus camera | electrncs + 1666 | harry potter | toys + 1600 | play gym | toys + 1601 | lamaze | toys + 1700 | wait interface | books + (6 rows) + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/7-deep-copy.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/7-deep-copy.md new file mode 100644 index 0000000000000000000000000000000000000000..15c0a82b53c13c648683e7e9b918441c7aa8f608 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/7-deep-copy.md @@ -0,0 +1,116 @@ +--- +title: Deep Copy +summary: Deep Copy +author: Guo Huan +date: 2021-03-04 +--- + +# Deep Copy + +After data is imported, you can perform a deep copy to modify a partition key, change a row-store table to a column-store table, or add a partial cluster key. A deep copy re-creates a table and batch inserts data into the table. + +MogDB provides three deep copy methods. + +## Performing a Deep Copy by Using the CREATE TABLE Statement + +Run the **CREATE TABLE** statement to create a copy of the original table, batch insert data of the original table into the copy, and rename the copy to the name of the original table. + +When creating the copy, you can specify table and column attributes, such as the primary key. + +**Procedure** + +Perform the following operations to carry out a deep copy for the **customer_t** table: + +1. Run the **CREATE TABLE** statement to create the copy **customer_t_copy** of the **customer_t** table. + + ```sql + CREATE TABLE customer_t_copy + ( c_customer_sk integer, + c_customer_id char(5), + c_first_name char(6), + c_last_name char(8) + ) ; + ``` + +2. Run the **INSERT INTO…SELECT** statement to batch insert data of the original table into the copy. + + ```sql + INSERT INTO customer_t_copy (SELECT * FROM customer_t); + ``` + +3. Delete the original table. + + ```sql + DROP TABLE customer_t; + ``` + +4. Run the **ALTER TABLE** statement to rename the copy to the name of the original table. + + ```sql + ALTER TABLE customer_t_copy RENAME TO customer_t; + ``` + +## Performing a Deep Copy by Using the CREATE TABLE LIKE Statement + +Run the **CREATE TABLE LIKE** statement to create a copy of the original table, batch insert data of the original table into the copy, and rename the copy to the name of the original table. This method does not inherit the primary key attributes of the original table. You can use the **ALTER TABLE** statement to add them. + +**Procedure** + +1. Run the **CREATE TABLE LIKE** statement to create the copy **customer_t_copy** of the **customer_t** table. + + ```sql + CREATE TABLE customer_t_copy (LIKE customer_t); + ``` + +2. Run the **INSERT INTO…SELECT** statement to batch insert data of the original table into the copy. + + ```sql + INSERT INTO customer_t_copy (SELECT * FROM customer_t); + ``` + +3. Delete the original table. + + ```sql + DROP TABLE customer_t; + ``` + +4. Run the **ALTER TABLE** statement to rename the copy to the name of the original table. + + ```sql + ALTER TABLE customer_t_copy RENAME TO customer_t; + ``` + +## Performing a Deep Copy by Creating a Temporary Table and Truncating the Original Table + +Run the **CREATE TABLE ….** **AS** statement to create a temporary table for the original table, truncate the original table, and batch insert data of the temporary data into the original table. + +When creating the temporary table, retain the primary key attributes of the original table. This method is recommended if the original table has dependency items. + +**Procedure** + +1. Run the **CREATE TABLE AS** statement to create a temporary table **customer_t_temp** for the **customer_t** table. + + ```sql + CREATE TEMP TABLE customer_t_temp AS SELECT * FROM customer_t; + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > Compared with the use of permanent tables, the use of temporary tables can improve performance but may incur data loss. A temporary table is automatically deleted at the end of the session where it is located. If data loss is unacceptable, use a permanent table. + +2. Truncate the original table **customer_t**. + + ```sql + TRUNCATE customer_t; + ``` + +3. Run the **INSERT INTO…SELECT** statement to batch insert data of the temporary table into the original table. + + ```sql + INSERT INTO customer_t (SELECT * FROM customer_t_temp); + ``` + +4. Delete the temporary table **customer_t_temp**. + + ```sql + DROP TABLE customer_t_temp; + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/8-ANALYZE-table.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/8-ANALYZE-table.md new file mode 100644 index 0000000000000000000000000000000000000000..c0f9a5e49616d29eb3d2d86ad9d719bf617788e0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/8-ANALYZE-table.md @@ -0,0 +1,46 @@ +--- +title: ANALYZE Table +summary: ANALYZE Table +author: Guo Huan +date: 2021-03-04 +--- + +# ANALYZE Table + +The execution plan generator needs to use table statistics to generate the most effective query execution plan to improve query performance. After data is imported, you are advised to run the **ANALYZE** statement to update table statistics. The statistics are stored in the system catalog **PG_STATISTIC**. + +## ANALYZE Table + +**ANALYZE** supports row-store and column-store tables. **ANALYZE** can also collect statistics about specified columns of a local table. For details on **ANALYZE**, see [ANALYZE | ANALYSE](../../../reference-guide/sql-syntax/ANALYZE-ANALYSE.md). + +Update table statistics. + +Do **ANALYZE** to the **product_info** table. + +```sql +ANALYZE product_info; +``` + +```sql +ANALYZE +``` + +## autoanalyze + +MogDB provides the GUC parameter autovacuum to specify whether to enable the autovacuum function of the database. + +If **autovacuum** is set to **on**, the system will start the autovacuum thread to automatically analyze tables when the data volume in the table reaches the threshold. This is the autoanalyze function. + +- For an empty table, when the number of rows inserted to it is greater than 50, **ANALYZE** is automatically triggered. +- For a table containing data, the threshold is 50 + 10% x **reltuples**, where **reltuples** indicates the total number of rows in the table. + +The autovacuum function also depends on the following two GUC parameters in addition to **autovacuum**: + +- track_counts: This parameter must be set to **on** to enable statistics collection about the database. +- autovacuum_max_workers: This parameter must be set to a value greater than **0** to specify the maximum number of concurrent autovacuum threads. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> +> - The autoanalyze function supports the default sampling mode but not percentage sampling. +> - The autoanalyze function does not collect multi-column statistics, which only supports percentage sampling. +> - The autoanalyze function supports row-store and column-store tables and does not support foreign tables, temporary tables, unlogged tables, and TOAST tables. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/9-doing-VACUUM-to-a-table.md b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/9-doing-VACUUM-to-a-table.md new file mode 100644 index 0000000000000000000000000000000000000000..06de9a0db25db95529fbef3049d539a5294a90e2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/importing-and-exporting-data/importing-data/9-doing-VACUUM-to-a-table.md @@ -0,0 +1,22 @@ +--- +title: Doing VACUUM to a Table +summary: Doing VACUUM to a Table +author: Guo Huan +date: 2021-03-04 +--- + +# Doing VACUUM to a Table + +If a large number of rows were updated or deleted during import, run **VACUUM FULL** before **ANALYZE**. A large number of UPDATE and DELETE operations generate huge disk page fragments, which reduces query efficiency. **VACUUM FULL** can restore disk page fragments and return them to the OS. + +Run the **VACUUM FULL** statement. + +Do **VACUUM FULL** to the **product_info** table. + +```sql +VACUUM FULL product_info +``` + +```sql +VACUUM +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/localization/character-set-support.md b/product/en/docs-mogdb/v3.1/administrator-guide/localization/character-set-support.md new file mode 100644 index 0000000000000000000000000000000000000000..43265a8f8e989b1cd4b09f54405160b2f220cb07 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/localization/character-set-support.md @@ -0,0 +1,206 @@ +--- +title: Character Set Support +summary: Character Set Support +author: Guo Huan +date: 2022-07-26 +--- + +# Character Set Support + +The character set support in MogDB allows you to store text in a variety of character sets (also called encodings), including single-byte character sets such as the ISO 8859 series and multiple-byte character sets such as EUC (Extended Unix Code), UTF-8, and Mule internal code. All supported character sets can be used transparently by clients, but a few are not supported for use within the server (that is, as a server-side encoding). The default character set is selected while initializing your MogDB database cluster using `gs_initdb`. It can be overridden when you create a database, so you can have multiple databases each with a different character set. + +An important restriction, however, is that each database's character set must be compatible with the database's `LC_CTYPE` (character classification) and `LC_COLLATE` (string sort order) locale settings. For `C` or `POSIX` locale, any character set is allowed, but for other locales there is only one character set that will work correctly. (On Windows, however, UTF-8 encoding can be used with any locale.) + +## Supported Character Sets + +Table1shows the character sets available for use in MogDB. + +**Table1 MogDB Character Sets** + +| Name | Description | Language | Server? | Bytes/Char | Aliases | +| ---------------- | --------------------------------- | ------------------------------ | ------- | ---------- | --------------------------------------------- | +| `BIG5` | Big Five | Traditional Chinese | No | 1-2 | `WIN950`, `Windows950` | +| `EUC_CN` | Extended UNIX Code-CN | Simplified Chinese | Yes | 1-3 | | +| `EUC_JP` | Extended UNIX Code-JP | Japanese | Yes | 1-3 | | +| `EUC_JIS_2004` | Extended UNIX Code-JP, JIS X 0213 | Japanese | Yes | 1-3 | | +| `EUC_KR` | Extended UNIX Code-KR | Korean | Yes | 1-3 | | +| `EUC_TW` | Extended UNIX Code-TW | Traditional Chinese, Taiwanese | Yes | 1-3 | | +| `GB18030` | National Standard | Chinese | Yes | 1-4 | | +| `GBK` | Extended National Standard | Simplified Chinese | Yes | 1-2 | `WIN936`, `Windows936` | +| `ISO_8859_5` | ISO 8859-5, ECMA 113 | Latin/Cyrillic | Yes | 1 | | +| `ISO_8859_6` | ISO 8859-6, ECMA 114 | Latin/Arabic | Yes | 1 | | +| `ISO_8859_7` | ISO 8859-7, ECMA 118 | Latin/Greek | Yes | 1 | | +| `ISO_8859_8` | ISO 8859-8, ECMA 121 | Latin/Hebrew | Yes | 1 | | +| `JOHAB` | JOHAB | Korean (Hangul) | No | 1-3 | | +| `KOI8R` | KOI8-R | Cyrillic (Russian) | Yes | 1 | `KOI8` | +| `KOI8U` | KOI8-U | Cyrillic (Ukrainian) | Yes | 1 | | +| `LATIN1` | ISO 8859-1, ECMA 94 | Western European | Yes | 1 | `ISO88591` | +| `LATIN2` | ISO 8859-2, ECMA 94 | Central European | Yes | 1 | `ISO88592` | +| `LATIN3` | ISO 8859-3, ECMA 94 | South European | Yes | 1 | `ISO88593` | +| `LATIN4` | ISO 8859-4, ECMA 94 | North European | Yes | 1 | `ISO88594` | +| `LATIN5` | ISO 8859-9, ECMA 128 | Turkish | Yes | 1 | `ISO88599` | +| `LATIN6` | ISO 8859-10, ECMA 144 | Nordic | Yes | 1 | `ISO885910` | +| `LATIN7` | ISO 8859-13 | Baltic | Yes | 1 | `ISO885913` | +| `LATIN8` | ISO 8859-14 | Celtic | Yes | 1 | `ISO885914` | +| `LATIN9` | ISO 8859-15 | LATIN1 with Euro and accents | Yes | 1 | `ISO885915` | +| `LATIN10` | ISO 8859-16, ASRO SR 14111 | Romanian | Yes | 1 | `ISO885916` | +| `MULE_INTERNAL` | Mule internal code | Multilingual Emacs | Yes | 1-4 | | +| `SJIS` | Shift JIS | Japanese | No | 1-2 | `Mskanji`, `ShiftJIS`, `WIN932`, `Windows932` | +| `SHIFT_JIS_2004` | Shift JIS, JIS X 0213 | Japanese | No | 1-2 | | +| `SQL_ASCII` | unspecified (see text) | any | Yes | 1 | | +| `UHC` | Unified Hangul Code | Korean | No | 1-2 | `WIN949`, `Windows949` | +| `UTF8` | Unicode, 8-bit | all | Yes | 1-4 | `Unicode` | +| `WIN866` | Windows CP866 | Cyrillic | Yes | 1 | `ALT` | +| `WIN874` | Windows CP874 | Thai | Yes | 1 | | +| `WIN1250` | Windows CP1250 | Central European | Yes | 1 | | +| `WIN1251` | Windows CP1251 | Cyrillic | Yes | 1 | `WIN` | +| `WIN1252` | Windows CP1252 | Western European | Yes | 1 | | +| `WIN1253` | Windows CP1253 | Greek | Yes | 1 | | +| `WIN1254` | Windows CP1254 | Turkish | Yes | 1 | | +| `WIN1255` | Windows CP1255 | Hebrew | Yes | 1 | | +| `WIN1256` | Windows CP1256 | Arabic | Yes | 1 | | +| `WIN1257` | Windows CP1257 | Baltic | Yes | 1 | | +| `WIN1258` | Windows CP1258 | Vietnamese | Yes | 1 | `ABC`, `TCVN`, `TCVN5712`, `VSCII` | + +Not all client APIs support all the listed character sets. The `SQL_ASCII` setting behaves considerably differently from the other settings. When the server character set is `SQL_ASCII`, the server interprets byte values 0-127 according to the ASCII standard, while byte values 128-255 are taken as uninterpreted characters. No encoding conversion will be done when the setting is `SQL_ASCII`. Thus, this setting is not so much a declaration that a specific encoding is in use, as a declaration of ignorance about the encoding. In most cases, if you are working with any non-ASCII data, it is unwise to use the `SQL_ASCII` setting because MogDB will be unable to help you by converting or validating non-ASCII characters. + +## Setting the Character Set + +`gs_initdb` defines the default character set (encoding) for a MogDB cluster. See [gs_initdb](../../reference-guide/tool-reference/tools-used-in-the-internal-system/5-gs_initdb.md) for details. + +You can specify a non-default encoding at database creation time use this SQL command, provided that the encoding is compatible with the selected locale: + +```sql +CREATE DATABASE chinese WITH ENCODING 'UTF8' LC_COLLATE='en_US.UTF8' LC_CTYPE='en_US.UTF8' TEMPLATE=template0; +``` + +Notice that the above commands specify copying the `template0` database. When copying any other database, the encoding and locale settings cannot be changed from those of the source database, because that might result in corrupt data. For more information see [CREATE DATABASE](../../reference-guide/sql-syntax/CREATE-DATABASE.md). + +The encoding for a database is stored in the system catalog `pg_database`. You can see it by using the `gsql` `-l` option or the `\l` command. + +```bash +$ gsql -l + List of databases + Name | Owner | Encoding | Collate | Ctype | Access privileges +-----------+-------+----------+-------------+-------------+------------------- + chinese | omm | UTF8 | en_US.UTF8 | en_US.UTF8 | + mogdb | omm | UTF8 | en_US.UTF-8 | en_US.UTF-8 | + mogila | omm | UTF8 | en_US.UTF-8 | en_US.UTF-8 | + postgres | omm | UTF8 | en_US.UTF-8 | en_US.UTF-8 | + template0 | omm | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/omm + + | | | | | omm=CTc/omm + template1 | omm | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/omm + + | | | | | omm=CTc/omm +(6 rows) +``` + +> **Important:** On most modern operating systems, MogDB can determine which character set is implied by the `LC_CTYPE` setting, and it will enforce that only the matching database encoding is used. On older systems it is your responsibility to ensure that you use the encoding expected by the locale you have selected. A mistake in this area is likely to lead to strange behavior of locale-dependent operations such as sorting. +> +> MogDB will allow superusers to create databases with `SQL_ASCII` encoding even when `LC_CTYPE` is not `C` or `POSIX`. As noted above, `SQL_ASCII` does not enforce that the data stored in the database has any particular encoding, and so this choice poses risks of locale-dependent misbehavior. Using this combination of settings is deprecated and may someday be forbidden altogether. + +## Automatic Character Set Conversion Between Server and Client + +MogDB supports automatic character set conversion between server and client for certain character set combinations. The conversion information is stored in the `pg_conversion` system catalog. MogDB comes with some predefined conversions, as shown in Table 2. + +**Table2 Client/Server Character Set Conversions** + +| Server Character Set | Available Client Character Sets | +| -------------------- | ------------------------------------------------------------ | +| `BIG5` | not supported as a server encoding | +| `EUC_CN` | EUC_CN, `MULE_INTERNAL`, `UTF8` | +| `EUC_JP` | EUC_JP, `MULE_INTERNAL`, `SJIS`, `UTF8` | +| `EUC_JIS_2004` | EUC_JIS_2004, `SHIFT_JIS_2004`, `UTF8` | +| `EUC_KR` | EUC_KR, `MULE_INTERNAL`, `UTF8` | +| `EUC_TW` | EUC_TW, `BIG5`, `MULE_INTERNAL`, `UTF8` | +| `GB18030` | not supported as a server encoding | +| `GBK` | not supported as a server encoding | +| `ISO_8859_5` | ISO_8859_5, `KOI8R`, `MULE_INTERNAL`, `UTF8`, `WIN866`, `WIN1251` | +| `ISO_8859_6` | ISO_8859_6, `UTF8` | +| `ISO_8859_7` | ISO_8859_7, `UTF8` | +| `ISO_8859_8` | ISO_8859_8, `UTF8` | +| `JOHAB` | not supported as a server encoding | +| `KOI8R` | KOI8R, `ISO_8859_5`, `MULE_INTERNAL`, `UTF8`, `WIN866`, `WIN1251` | +| `KOI8U` | KOI8U, `UTF8` | +| `LATIN1` | LATIN1, `MULE_INTERNAL`, `UTF8` | +| `LATIN2` | LATIN2, `MULE_INTERNAL`, `UTF8`, `WIN1250` | +| `LATIN3` | LATIN3, `MULE_INTERNAL`, `UTF8` | +| `LATIN4` | LATIN4, `MULE_INTERNAL`, `UTF8` | +| `LATIN5` | LATIN5, `UTF8` | +| `LATIN6` | LATIN6, `UTF8` | +| `LATIN7` | LATIN7, `UTF8` | +| `LATIN8` | LATIN8, `UTF8` | +| `LATIN9` | LATIN9, `UTF8` | +| `LATIN10` | LATIN10, `UTF8` | +| `MULE_INTERNAL` | MULE_INTERNAL, `BIG5`, `EUC_CN`, `EUC_JP`, `EUC_KR`, `EUC_TW`, `ISO_8859_5`, `KOI8R`, `LATIN1` to `LATIN4`, `SJIS`, `WIN866`, `WIN1250`, `WIN1251` | +| `SJIS` | not supported as a server encoding | +| `SHIFT_JIS_2004` | not supported as a server encoding | +| `SQL_ASCII` | any (no conversion will be performed) | +| `UHC` | not supported as a server encoding | +| `UTF8` | all supported encodings | +| `WIN866` | WIN866, `ISO_8859_5`, `KOI8R`, `MULE_INTERNAL`, `UTF8`, `WIN1251` | +| `WIN874` | WIN874, `UTF8` | +| `WIN1250` | WIN1250, `LATIN2`, `MULE_INTERNAL`, `UTF8` | +| `WIN1251` | WIN1251, `ISO_8859_5`, `KOI8R`, `MULE_INTERNAL`, `UTF8`, `WIN866` | +| `WIN1252` | WIN1252, `UTF8` | +| `WIN1253` | WIN1253, `UTF8` | +| `WIN1254` | WIN1254, `UTF8` | +| `WIN1255` | WIN1255, `UTF8` | +| `WIN1256` | WIN1256, `UTF8` | + +To enable automatic character set conversion, you have to tell MogDB the character set (encoding) you would like to use in the client. There are several ways to accomplish this: + +- Using the `\encoding` command in gsql. `\encoding` allows you to change client encoding on the fly. For example, to change the encoding to `SJIS`, type: + + ```sql + \encoding SJIS + ``` + +- [libpq](../../developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/1-database-connection-control-functions-overview.md) has functions to control the client encoding. + +- Using `SET client_encoding TO`. Setting the client encoding can be done with this SQL command: + + ```bash + SET CLIENT_ENCODING TO 'value'; + ``` + + Also you can use the standard SQL syntax `SET NAMES` for this purpose: + + ```bash + SET NAMES 'value'; + ``` + + To query the current client encoding: + + ```bash + SHOW client_encoding; + ``` + + To return to the default encoding: + + ```bash + RESET client_encoding; + ``` + +- Using `PGCLIENTENCODING`. If the environment variable `PGCLIENTENCODING` is defined in the client's environment, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.) + +- Using the configuration variable [client_encoding](../../reference-guide/guc-parameters/15-default-settings-of-client-connection/2-zone-and-formatting.md#client_encoding). If the `client_encoding` variable is set, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.) + +If the conversion of a particular character is not possible — suppose you chose `EUC_JP` for the server and `LATIN1` for the client, and some Japanese characters are returned that do not have a representation in `LATIN1` — an error is reported. + +If the client character set is defined as `SQL_ASCII`, encoding conversion is disabled, regardless of the server's character set. Just as for the server, use of `SQL_ASCII` is unwise unless you are working with all-ASCII data. + +## Further Reading + +These are good sources to start learning about various kinds of encoding systems. + +- CJKV Information Processing: Chinese, Japanese, Korean & Vietnamese Computing + + Contains detailed explanations of `EUC_JP`, `EUC_CN`, `EUC_KR`, `EUC_TW`. + +- [http://www.unicode.org/](http://www.unicode.org/) + + The web site of the Unicode Consortium. + +- RFC 3629 + + UTF-8 (8-bit UCS/Unicode Transformation Format) is defined here. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/localization/collation-support.md b/product/en/docs-mogdb/v3.1/administrator-guide/localization/collation-support.md new file mode 100644 index 0000000000000000000000000000000000000000..0c2303499b08d755560094945331ad588ba57fd0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/localization/collation-support.md @@ -0,0 +1,128 @@ +--- +title: Collation Support +summary: Collation Support +author: Zhang Cuiping +date: 2022-07-26 +--- + +# Collation Support + +The collation feature allows specifying the sort order and character classification behavior of data per-column, or even per-operation. This alleviates the restriction that the `LC_COLLATE` and `LC_CTYPE` settings of a database cannot be changed after its creation. + +## Concepts + +Conceptually, every expression of a collatable data type has a collation. (The built-in collatable data types are `text`, `varchar`, and `char`. User-defined base types can also be marked collatable, and of course a domain over a collatable data type is collatable.) If the expression is a column reference, the collation of the expression is the defined collation of the column. If the expression is a constant, the collation is the default collation of the data type of the constant. The collation of a more complex expression is derived from the collations of its inputs, as described below. + +The collation of an expression can be the “default” collation, which means the locale settings defined for the database. It is also possible for an expression's collation to be indeterminate. In such cases, ordering operations and other operations that need to know the collation will fail. + +When the database system has to perform an ordering or a character classification, it uses the collation of the input expression. This happens, for example, with `ORDER BY` clauses and function or operator calls such as `<`. The collation to apply for an `ORDER BY` clause is simply the collation of the sort key. The collation to apply for a function or operator call is derived from the arguments, as described below. In addition to comparison operators, collations are taken into account by functions that convert between lower and upper case letters, such as `lower`, `upper`, and `initcap`; by pattern matching operators; and by `to_char` and related functions. + +For a function or operator call, the collation that is derived by examining the argument collations is used at run time for performing the specified operation. If the result of the function or operator call is of a collatable data type, the collation is also used at parse time as the defined collation of the function or operator expression, in case there is a surrounding expression that requires knowledge of its collation. + +The *collation derivation* of an expression can be implicit or explicit. This distinction affects how collations are combined when multiple different collations appear in an expression. An explicit collation derivation occurs when a `COLLATE` clause is used; all other collation derivations are implicit. When multiple collations need to be combined, for example in a function call, the following rules are used: + +1. If any input expression has an explicit collation derivation, then all explicitly derived collations among the input expressions must be the same, otherwise an error is raised. If any explicitly derived collation is present, that is the result of the collation combination. +2. Otherwise, all input expressions must have the same implicit collation derivation or the default collation. If any non-default collation is present, that is the result of the collation combination. Otherwise, the result is the default collation. +3. If there are conflicting non-default implicit collations among the input expressions, then the combination is deemed to have indeterminate collation. This is not an error condition unless the particular function being invoked requires knowledge of the collation it should apply. If it does, an error will be raised at run-time. + +For example, consider this table definition: + +```sql +CREATE TABLE test1 ( + a text COLLATE "de_DE", + b text COLLATE "es_ES", + ... +); +``` + +Then in + +```sql +SELECT a < 'foo' FROM test1; +``` + +the `<` comparison is performed according to `de_DE` rules, because the expression combines an implicitly derived collation with the default collation. But in + +```sql +SELECT a < ('foo' COLLATE "fr_FR") FROM test1; +``` + +the comparison is performed using `fr_FR` rules, because the explicit collation derivation overrides the implicit one. Furthermore, given + +```sql +SELECT a < b FROM test1; +``` + +the parser cannot determine which collation to apply, since the `a` and `b` columns have conflicting implicit collations. Since the `<` operator does need to know which collation to use, this will result in an error. The error can be resolved by attaching an explicit collation specifier to either input expression, thus: + +```sql +SELECT a < b COLLATE "de_DE" FROM test1; +``` + +or equivalently + +```sql +SELECT a COLLATE "de_DE" < b FROM test1; +``` + +On the other hand, the structurally similar case + +```sql +SELECT a || b FROM test1; +``` + +does not result in an error, because the `||` operator does not care about collations: its result is the same regardless of the collation. + +The collation assigned to a function or operator's combined input expressions is also considered to apply to the function or operator's result, if the function or operator delivers a result of a collatable data type. So, in + +```sql +SELECT * FROM test1 ORDER BY a || 'foo'; +``` + +the ordering will be done according to `de_DE` rules. But this query: + +```sql +SELECT * FROM test1 ORDER BY a || b; +``` + +results in an error, because even though the `||` operator doesn't need to know a collation, the `ORDER BY` clause does. As before, the conflict can be resolved with an explicit collation specifier: + +```sql +SELECT * FROM test1 ORDER BY a || b COLLATE "fr_FR"; +``` + +## Managing Collations + +A collation is an SQL schema object that maps an SQL name to locales provided by libraries installed in the operating system. A collation definition has a *provider* that specifies which library supplies the locale data. One standard provider name is `libc`, which uses the locales provided by the operating system C library. These are the locales that most tools provided by the operating system use. Another provider is `icu`, which uses the external ICU library. ICU locales can only be used if support for ICU was configured when MogDB was built. + +A collation object provided by `libc` maps to a combination of `LC_COLLATE` and `LC_CTYPE` settings, as accepted by the `setlocale()` system library call. (As the name would suggest, the main purpose of a collation is to set `LC_COLLATE`, which controls the sort order. But it is rarely necessary in practice to have an `LC_CTYPE` setting that is different from `LC_COLLATE`, so it is more convenient to collect these under one concept than to create another infrastructure for setting `LC_CTYPE` per expression.) Also, a `libc` collation is tied to a character set encoding (see [Character Set Support](./character-set-support.md)). The same collation name may exist for different encodings. + +A collation object provided by `icu` maps to a named collator provided by the ICU library. ICU does not support separate “collate” and “ctype” settings, so they are always the same. Also, ICU collations are independent of the encoding, so there is always only one ICU collation of a given name in a database. + +### Standard Collations + +On all platforms, the collations named `default`, `C`, and `POSIX` are available. Additional collations may be available depending on operating system support. The `default` collation selects the `LC_COLLATE` and `LC_CTYPE` values specified at database creation time. The `C` and `POSIX` collations both specify “traditional C” behavior, in which only the ASCII letters “`A`” through “`Z`” are treated as letters, and sorting is done strictly by character code byte values. + +Additionally, the SQL standard collation name `ucs_basic` is available for encoding `UTF8`. It is equivalent to `C` and sorts by Unicode code point. + +### Predefined Collations + +If the operating system provides support for using multiple locales within a single program (`newlocale` and related functions), or if support for ICU is configured, then when a database cluster is initialized, `gs_initdb` populates the system catalog `pg_collation` with collations based on all the locales it finds in the operating system at the time. + +To inspect the currently available locales, use the query `SELECT * FROM pg_collation`, or the command `\dOS+` in gsql. + +#### libc Collations + +For example, the operating system might provide a locale named `de_DE.utf8`. `gs_initdb` would then create a collation named `de_DE.utf8` for encoding `UTF8` that has both `LC_COLLATE` and `LC_CTYPE` set to `de_DE.utf8`. It will also create a collation with the `.utf8` tag stripped off the name. So you could also use the collation under the name `de_DE`, which is less cumbersome to write and makes the name less encoding-dependent. Note that, nevertheless, the initial set of collation names is platform-dependent. + +The default set of collations provided by `libc` map directly to the locales installed in the operating system, which can be listed using the command `locale -a`. + +Within any particular database, only collations that use that database's encoding are of interest. Other entries in `pg_collation` are ignored. Thus, a stripped collation name such as `de_DE` can be considered unique within a given database even though it would not be unique globally. Use of the stripped collation names is recommended, since it will make one fewer thing you need to change if you decide to change to another database encoding. Note however that the `default`, `C`, and `POSIX` collations can be used regardless of the database encoding. + +MogDB considers distinct collation objects to be incompatible even when they have identical properties. Thus for example, + +```sql +SELECT a COLLATE "C" < b COLLATE "POSIX" FROM test1; +``` + +will draw an error even though the `C` and `POSIX` collations have identical behaviors. Mixing stripped and non-stripped collation names is therefore not recommended. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/localization/locale-support.md b/product/en/docs-mogdb/v3.1/administrator-guide/localization/locale-support.md new file mode 100644 index 0000000000000000000000000000000000000000..b3c1e01052409e78a6ae003b15dcc68189013b63 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/localization/locale-support.md @@ -0,0 +1,68 @@ +--- +title: Locale Support +summary: Locale Support +author: Zhang Cuiping +date: 2022-07-26 +--- + +# Locale Support + +Locale support refers to an application respecting cultural preferences regarding alphabets, sorting, number formatting, etc. MogDB uses the standard ISO C and POSIX locale facilities provided by the server operating system. For additional information refer to the documentation of your system. + +## Overview + +Locale support is automatically initialized when a database cluster is created using `gs_initdb`. `gs_initdb` will initialize the database cluster with the locale setting of its execution environment by default, so if your system is already set to use the locale that you want in your database cluster then there is nothing else you need to do. If you want to use a different locale (or you are not sure which locale your system is set to), you can instruct `gs_initdb` exactly which locale to use by specifying the `--locale` option. For example: + +``` +gs_initdb -D /opt/mogdb/data -w "XXXXXXXX" --nodename='data1' --locale=en_US +``` + +This example for Unix systems sets the locale to Swedish (`sv`) as spoken in Sweden (`SE`). Other possibilities might include `en_US` (U.S. English) and `fr_CA` (French Canadian). If more than one character set can be used for a locale then the specifications can take the form `language_territory.codeset`. For example, `fr_BE.UTF-8` represents the French language (fr) as spoken in Belgium (BE), with a UTF-8 character set encoding. + +What locales are available on your system under what names depends on what was provided by the operating system vendor and what was installed. On most Unix systems, the command `locale -a` will provide a list of available locales. Windows uses more verbose locale names, such as `German_Germany` or `Swedish_Sweden.1252`, but the principles are the same. + +Occasionally it is useful to mix rules from several locales, e.g., use English collation rules but Spanish messages. To support that, a set of locale subcategories exist that control only certain aspects of the localization rules: + +| `LC_COLLATE` | String Sort Order | +| ------------- | ------------------------------------------------------------ | +| `LC_CTYPE` | Character classification (What is a letter? Its upper-case equivalent?) | +| `LC_MESSAGES` | Language of messages | +| `LC_MONETARY` | Formatting of currency amounts | +| `LC_NUMERIC` | Formatting of numbers | +| `LC_TIME` | Formatting of dates and times | + +The category names translate into names of `gs_initdb` options to override the locale choice for a specific category. For instance, to set the locale to French Canadian, but use U.S. rules for formatting currency, use `gs_initdb -D /opt/mogdb/data -w "XXXXXXXX" --nodename='data1' --locale=fr_CA --lc-monetary=en_US`. + +If you want the system to behave as if it had no locale support, use the special locale name `C`, or equivalently `POSIX`. + +Some locale categories must have their values fixed when the database is created. You can use different settings for different databases, but once a database is created, you cannot change them for that database anymore. `LC_COLLATE` and `LC_CTYPE` are these categories. They affect the sort order of indexes, so they must be kept fixed, or indexes on text columns would become corrupt. (But you can alleviate this restriction using collations, as discussed in [Collation Support](./collation-support.md)) The default values for these categories are determined when `gs_initdb` is run, and those values are used when new databases are created, unless specified otherwise in the `CREATE DATABASE` command. + +The other locale categories can be changed whenever desired by setting the server configuration parameters that have the same name as the locale categories (see [Zone and Formatting](../../reference-guide/guc-parameters/15-default-settings-of-client-connection/2-zone-and-formatting.md) for details). The values that are chosen by `gs_initdb` are actually only written into the configuration file `postgresql.conf` to serve as defaults when the server is started. If you remove these assignments from `postgresql.conf` then the server will inherit the settings from its execution environment. + +Note that the locale behavior of the server is determined by the environment variables seen by the server, not by the environment of any client. Therefore, be careful to configure the correct locale settings before starting the server. A consequence of this is that if client and server are set up in different locales, messages might appear in different languages depending on where they originated. + +> **Note**: When we speak of inheriting the locale from the execution environment, this means the following on most operating systems: For a given locale category, say the collation, the following environment variables are consulted in this order until one is found to be set: `LC_ALL`, `LC_COLLATE` (or the variable corresponding to the respective category), `LANG`. If none of these environment variables are set then the locale defaults to `C`. +> +> Some message localization libraries also look at the environment variable `LANGUAGE` which overrides all other locale settings for the purpose of setting the language of messages. If in doubt, please refer to the documentation of your operating system, in particular the documentation about gettext. + +## Behavior + +The locale settings influence the following SQL features: + +- Sort order in queries using `ORDER BY` or the standard comparison operators on textual data +- The `upper`, `lower`, and `initcap` functions +- Pattern matching operators (`LIKE`, `SIMILAR TO`, and POSIX-style regular expressions); locales affect both case insensitive matching and the classification of characters by character-class regular expressions +- The `to_char` family of functions +- The ability to use indexes with `LIKE` clauses + +The drawback of using locales other than `C` or `POSIX` in MogDB is its performance impact. It slows character handling and prevents ordinary indexes from being used by `LIKE`. For this reason use locales only if you actually need them. + +As a workaround to allow MogDB to use indexes with `LIKE` clauses under a non-C locale, several custom operator classes exist. These allow the creation of an index that performs a strict character-by-character comparison, ignoring locale comparison rules. Another approach is to create indexes using the `C` collation, as discussed in [Collation Support](./collation-support.md). + +## Problems + +If locale support doesn't work according to the explanation above, check that the locale support in your operating system is correctly configured. To check what locales are installed on your system, you can use the command `locale -a` if your operating system provides it. + +MogDB `LC_COLLATE` and `LC_CTYPE` settings are determined when a database is created, and cannot be changed except by creating a new database. Other locale settings including `LC_MESSAGES` and `LC_MONETARY` are initially determined by the environment the server is started in, but can be changed on-the-fly. You can check the active locale settings using the `SHOW` command. + +Client applications that handle server-side errors by parsing the text of the error message will obviously have problems when the server's messages are in a different language. Authors of such applications are advised to make use of the error code scheme instead. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/1-mot-introduction.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/1-mot-introduction.md new file mode 100644 index 0000000000000000000000000000000000000000..6eba1aac0c1059819c174ebeb4b89735e24dcf9a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/1-mot-introduction.md @@ -0,0 +1,33 @@ +--- +title: MOT Introduction +summary: MOT Introduction +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Introduction + +MogDB introduces Memory-Optimized Tables (MOT) storage engine - a transactional row-based store (rowstore), that is optimized for many-core and large memory servers. MOT is a state-of-the-art production-grade feature (Beta release) of the MogDB database that provides greater performance for transactional workloads. MOT is fully ACID compliant and includes strict durability and high availability support. Businesses can leverage MOT for mission-critical, performance-sensitive Online Transaction Processing (OLTP) applications in order to achieve high performance, high throughput, low and predictable latency and high utilization of many-core servers. MOT is especially suited to leverage and scale-up when run on modern servers with multiple sockets and many-core processors, such as Huawei Taishan servers with ARM/Kunpeng processors and x86-based Dell or similar servers. + +**Figure 1** Memory-Optimized Storage Engine Within MogDB + +![memory-optimized-storage-engine-within-opengauss](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-introduction-2.png) + +[Figure 1](#memoryoptimized) presents the Memory-Optimized Storage Engine component (in green) of MogDB database and is responsible for managing MOT and transactions. + +MOT tables are created side-by-side regular disk-based tables. MOT's effective design enables almost full SQL coverage and support for a full database feature-set, such as stored procedures and user-defined functions (excluding the features listed in **MOT SQL Coverage and Limitations** section). + +With data and indexes stored totally in-memory, a Non-Uniform Memory Access (NUMA)-aware design, algorithms that eliminate lock and latch contention and query native compilation, MOT provides faster data access and more efficient transaction execution. + +MOT's effective almost lock-free design and highly tuned implementation enable exceptional near-linear throughput scale-up on many-core servers - probably the best in the industry. + +Memory-Optimized Tables are fully ACID compliant, as follows: + +- **Atomicity -** An atomic transaction is an indivisible series of database operations that either all occur or none occur after a transaction has been completed (committed or aborted, respectively). +- **Consistency -** Every transaction leaves the database in a consistent (data integrity) state. +- **Isolation -** Transactions cannot interfere with each other. MOT supports repeatable-reads and read-committed isolation levels. In the next release, MOT will also support serializable isolation. See the **MOT Isolation Levels** section for more information. +- **Durability -** The effects of successfully completed (committed) transactions must persist despite crashes and failures. MOT is fully integrated with the WAL-based logging of MogDB. Both synchronous and asynchronous logging options are supported. MOT also uniquely supports synchronous + group commit with NUMA-awareness optimization. See the **MOT Durability Concepts** section for more information. + +The MOT Engine was published in the VLDB 2020 (an International Conference on ‘Very Large Data Bases" or VLDB): + +**Industrial-Strength OLTP Using Main Memory and Many Cores**, VLDB 2020 vol. 13 - [Paper](http://www.vldb.org/pvldb/vol13/p3099-avni.pdf), [Video on youtube](https://www.modb.pro/video/6676?slink), [Video on bilibili](https://www.bilibili.com/video/BV1MA411n7ef?p=97). diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/2-mot-features-and-benefits.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/2-mot-features-and-benefits.md new file mode 100644 index 0000000000000000000000000000000000000000..1713c463ebe20e1f1b972747b3cee069dc6de4ec --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/2-mot-features-and-benefits.md @@ -0,0 +1,22 @@ +--- +title: MOT Features and Benefits +summary: MOT Features and Benefits +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Features and Benefits + +MOT provide users with significant benefits in performance (query and transaction latency), scalability (throughput and concurrency) and in some cases cost (high resource utilization) - + +- **Low Latency -** Provides fast query and transaction response time +- **High Throughput -** Supports spikes and constantly high user concurrency +- **High Resource Utilization -** Utilizes hardware to its full extent + +Using MOT, applications are able to achieve more 2.5 to 4 times (2.5x - 4x) higher throughput. For example, in our TPC-C benchmarks (interactive transactions and synchronous logging) performed both on Huawei Taishan Kunpeng-based (ARM) servers and on Dell x86 Intel Xeon-based servers, MOT provides throughput gains that vary from 2.5x on a 2-socket server to 3.7x on a 4-socket server, reaching 4.8M (million) tpmC on an ARM 4-socket 256-cores server. + +The lower latency provided by MOT reduces transaction speed by 3x to 5.5x, as observed in TPC-C benchmarks. + +Additionally, MOT enables extremely high utilization of server resources when running under high load and contention, which is a well-known problem for all leading industry databases. Using MOT, utilization reaches 99% on 4-socket server, compared with much lower utilization observed when testing other industry leading databases. + +This abilities are especially evident and important on modern many-core servers. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/3-mot-key-technologies.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/3-mot-key-technologies.md new file mode 100644 index 0000000000000000000000000000000000000000..3bf4108c17a46edc1a760b6734314a8939a4dea6 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/3-mot-key-technologies.md @@ -0,0 +1,24 @@ +--- +title: MOT Key Technologies +summary: MOT Key Technologies +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Key Technologies + +The following key MOT technologies enable its benefits: + +- **Memory Optimized Data Structures -** With the objective of achieving optimal high concurrent throughput and predictable low latency, all data and indexes are in memory, no intermediate page buffers are used and minimal, short-duration locks are used. Data structures and all algorithms have been specialized and optimized for in-memory design. +- **Lock-free Transaction Management -** The MOT storage engine applies an optimistic approach to achieving data integrity versus concurrency and high throughput. During a transaction, an MOT table does not place locks on any version of the data rows being updated, thus significantly reducing contention in some high-volume systems. Optimistic Concurrency Control (OCC) statements within a transaction are implemented without locks, and all data modifications are performed in a part of the memory that is dedicated to private transactions (also called *Private Transaction Memory*). This means that during a transaction, the relevant data is updated in the Private Transaction Memory, thus enabling lock-less reads and writes; and a very short duration lock is only placed at the Commit phase. For more details, see the **MOT Concurrency Control Mechanism** section. +- **Lock-free Index -** Because database data and indexes stored totally in-memory, having an efficient index data structure and algorithm is essential. The MOT Index is based on state-of-the-art Masstree a fast and scalable Key Value (KV) store for multi-core systems, implemented as a Trie of B+ trees. In this way, excellent performance is achieved on many-core servers and during high concurrent workloads. This index applies various advanced techniques in order to optimize performance, such as an optimistic lock approach, cache-line awareness and memory prefetching. +- **NUMA-aware Memory Management -** MOT memory access is designed with Non-Uniform Memory Access (NUMA) awareness. NUMA-aware algorithms enhance the performance of a data layout in memory so that threads access the memory that is physically attached to the core on which the thread is running. This is handled by the memory controller without requiring an extra hop by using an interconnect, such as Intel QPI. MOT's smart memory control module with pre-allocated memory pools for various memory objects improves performance, reduces locks and ensures stability. Allocation of a transaction's memory objects is always NUMA-local. Deallocated objects are returned to the pool. Minimal usage of OS malloc during transactions circumvents unnecessary locks. +- **Efficient Durability - Logging and Checkpoint -** Achieving disk persistence (also known as *durability*) is a crucial requirement for being ACID compliant (the **D** stands for Durability). All current disks (including the SSD and NVMe) are significantly slower than memory and thus are always the bottleneck of a memory-based database. As an in-memory storage engine with full durability support, MOT's durability design must implement a wide variety of algorithmic optimizations in order to ensure durability, while still achieving the speed and throughput objectives for which it was designed. These optimizations include - + - Parallel logging, which is also available in all MogDB disk tables + - Log buffering per transaction and lock-less transaction preparation + - Updating delta records, meaning only logging changes + - In addition to synchronous and asynchronous, innovative NUMA-aware group commit logging + - State-of-the-art database checkpoints (CALC) enable the lowest memory and computational overhead. +- **High SQL Coverage and Feature Set -** By extending and relying on the PostgreSQL Foreign Data Wrappers (FDW) + Index support, the entire range of SQL is covered, including stored procedures, user-defined functions and system function calls. You may refer to the **MOT SQL Coverage and Limitations** section for a list of the features that are not supported. +- **Queries Native Compilation using PREPARE Statements -** Queries and transaction statements can be executed in an interactive manner by using PREPARE client commands that have been precompiled into a native execution format (which are also known as *Code-Gen* or *Just-in-Time [JIT]* compilation). This achieves an average of 30% higher performance. Compilation and Lite Execution are applied when possible, and if not, applicable queries are processed using the standard execution path. A Cache Plan module (that has been optimized for OLTP) re-uses compilation results throughout an entire session (even using different bind settings), as well as across different sessions. +- **Seamless Integration of MOT and MogDB Database -** The MOT operates side by side the disk-based storage engine within an integrated envelope. MOT's main memory engine and disk-based storage engines co-exist side by side in order to support multiple application scenarios, while internally reusing database auxiliary services, such as a Write-Ahead Logging (WAL) Redo Log, Replication, Checkpointing, Recovery, High Availability and so on. Users benefit from the unified deployment, configuration and access of both disk-based tables and MOT tables. This provides a flexible and cost-efficient choice of which storage engine to use according to specific requirements. For example, to place highly performance-sensitive data that causes bottlenecks into memory. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/4-mot-usage-scenarios.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/4-mot-usage-scenarios.md new file mode 100644 index 0000000000000000000000000000000000000000..ef56b9ec8f19f87d2a511506afaed5bbae603f48 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/4-mot-usage-scenarios.md @@ -0,0 +1,22 @@ +--- +title: MOT Usage Scenarios +summary: MOT Usage Scenarios +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Usage Scenarios + +MOT can significantly speed up an application's overall performance, depending on the characteristics of the workload. MOT improves the performance of transaction processing by making data access and transaction execution more efficient and minimizing redirections by removing lock and latch contention between concurrently executing transactions. + +MOT's extreme speed stems from the fact that it is optimized around concurrent in-memory usage management (not just because it is in memory). Data storage, access and processing algorithms were designed from the ground up to take advantage of the latest state of the art enhancements in in-memory and high-concurrency computing. + +MogDB enables an application to use any combination of MOT tables and standard disk-based tables. MOT is especially beneficial for enabling your most active, high-contention and performance-sensitive application tables that have proven to be bottlenecks and for tables that require a predictable low-latency access and high throughput. + +MOT tables can be used for a variety of application use cases, which include: + +- **High-throughput Transactions Processing -** This is the primary scenario for using MOT, because it supports large transaction volume that requires consistently low latency for individual transactions. Examples of such applications are real-time decision systems, payment systems, financial instrument trading, sports betting, mobile gaming, ad delivery and so on. +- **Acceleration of Performance Bottlenecks -** High contention tables can significantly benefit from using MOT, even when other tables are on disk. The conversion of such tables (in addition to related tables and tables that are referenced together in queries and transactions) result in a significant performance boost as the result of lower latencies, less contention and locks, and increased server throughput ability. +- **Elimination of Mid-Tier Cache -** Cloud and Mobile applications tend to have periodic or spikes of massive workload. Additionally, many of these applications have 80% or above read-workload, with frequent repetitive queries. To sustain the workload spikes, as well to provide optimal user experience by low-latency response time, applications sometimes deploy a mid-tier caching layer. Such additional layers increase development complexity and time, and also increase operational costs. MOT provides a great alternative, simplifying the application architecture with a consistent and high performance data store, while shortening development cycles and reducing CAPEX and OPEX costs. +- **Large-scale Data Streaming and Data Ingestion -** MOT tables enables large-scale streamlined data processing in the Cloud (for Mobile, M2M and IoT), Transactional Processing (TP), Analytical Processing (AP) and Machine Learning (ML). MOT tables are especially good at consistently and quickly ingesting large volumes of data from many different sources at the same time. The data can be later processed, transformed and moved in slower disk-based tables. Alternatively, MOT enables the querying of consistent and up-date data that enable real-time conclusions. In IoT and cloud applications with many real-time data streams, it is common to have special data ingestion and processing triers. For instance, an Apache Kafka cluster can be used to ingest data of 100,000 events/sec with a 10msec latency. A periodic batch processing task enriches and converts the collected data into an alternative format to be placed into a relational database for further analysis. MOT can support such scenarios (while eliminating the separate data ingestion tier) by ingesting data streams directly into MOT relational tables, ready for analysis and decisions. This enables faster data collection and processing, MOT eliminates costly tiers and slow batch processing, increases consistency, increases freshness of analyzed data, as well as lowers Total Cost of Ownership (TCO). +- **Lower TCO -** Higher resource efficiency and mid-tier elimination can save 30% to 90%. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/5-mot-performance-benchmarks.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/5-mot-performance-benchmarks.md new file mode 100644 index 0000000000000000000000000000000000000000..5a113129038e098d2736fb5b33db0614799e3493 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/1-introducing-mot/5-mot-performance-benchmarks.md @@ -0,0 +1,189 @@ +--- +title: MOT Performance Benchmarks +summary: MOT Performance Benchmarks +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Performance Benchmarks + +Our performance tests are based on the TPC-C Benchmark that is commonly used both by industry and academia. + +Ours tests used BenchmarkSQL (see **MOT Sample TPC-C Benchmark**) and generates the workload using interactive SQL commands, as opposed to stored procedures. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Using the stored procedures approach may produce even higher performance results because it involves significantly less networking roundtrips and database envelope SQL processing cycles. + +All tests that evaluated the performance of MogDB MOT vs DISK used synchronous logging and its optimized **group-commit=on** version in MOT. + +Finally, we performed an additional test in order to evaluate MOT's ability to quickly and ingest massive quantities of data and to serve as an alternative to a mid-tier data ingestion solutions. + +All tests were performed in June 2020. + +The following shows various types of MOT performance benchmarks. + +## MOT Hardware + +The tests were performed on servers with the following configuration and with 10Gbe networking - + +- ARM64/Kunpeng 920-based 2-socket servers, model Taishan 2280 v2 (total 128 Cores), 800GB RAM, 1TB NVMe disk. For a detailed server specification, see - OS: openEuler + +- ARM64/Kunpeng 960-based 4-socket servers, model Taishan 2480 v2 (total 256 Cores), 512GB RAM, 1TB NVMe disk. For a detailed server specification, see - OS: openEuler + +- x86-based Dell servers, with 2-sockets of Intel Xeon Gold 6154 CPU @ 3GHz with 18 Cores (72 Cores, with hyper-threading=on), 1TB RAM, 1TB SSD OS: CentOS 7.6 + +- x86-based SuperMicro server, with 8-sockets of Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz 24 cores (total 384 Cores, with hyper-threading=on), 1TB RAM, 1.2TB SSD (Seagate 1200 SSD 200GB, SAS 12Gb/s). OS: Ubuntu 16.04.2 LTS + +- x86-based Huawei server, with 4-sockets of Intel(R) Xeon(R) CPU E7-8890 v4 2.2Ghz (total 96 Cores, with hyper-threading=on), 512GB RAM, SSD 2TB OS: CentOS 7.6 + +## MOT Results - Summary + +MOT provides higher performance than disk-tables by a factor of 2.5x to 4.1x and reaches 4.8 million tpmC on ARM/Kunpeng-based servers with 256 cores. The results clearly demonstrate MOT's exceptional ability to scale-up and utilize all hardware resources. Performance jumps as the quantity of CPU sockets and server cores increases. + +MOT delivers up to 30,000 tpmC/core on ARM/Kunpeng-based servers and up to 40,000 tpmC/core on x86-based servers. + +Due to a more efficient durability mechanism, in MOT the replication overhead of a Primary/Secondary High Availability scenario is 7% on ARM/Kunpeng and 2% on x86 servers, as opposed to the overhead in disk tables of 20% on ARM/Kunpeng and 15% on x86 servers. + +Finally, MOT delivers 2.5x lower latency, with TPC-C transaction response times of 2 to 7 times faster. + +## MOT High Throughput + +The following shows the results of various MOT table high throughput tests. + +### ARM/Kunpeng 2-Socket 128 Cores + +**Performance** + +The following figure shows the results of testing the TPC-C benchmark on a Huawei ARM/Kunpeng server that has two sockets and 128 cores. + +Four types of tests were performed - + +- Two tests were performed on MOT tables and another two tests were performed on MogDB disk-based tables. +- Two of the tests were performed on a Single node (without high availability), meaning that no replication was performed to a secondary node. The other two tests were performed on Primary/Secondary nodes (with high availability), meaning that data written to the primary node was replicated to a secondary node. + +MOT tables are represented in orange and disk-based tables are represented in blue. + +**Figure 1** ARM/Kunpeng 2-Socket 128 Cores - Performance Benchmarks + +![arm-kunpeng-2-socket-128-cores-performance-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-10.png) + +The results showed that: + +- As expected, the performance of MOT tables is significantly greater than of disk-based tables in all cases. +- For a Single Node - 3.8M tpmC for MOT tables versus 1.5M tpmC for disk-based tables +- For a Primary/Secondary Node - 3.5M tpmC for MOT tables versus 1.2M tpmC for disk-based tables +- For production grade (high-availability) servers (Primary/Secondary Node) that require replication, the benefit of using MOT tables is even more significant than for a Single Node (without high-availability, meaning no replication). +- The MOT replication overhead of a Primary/Secondary High Availability scenario is 7% on ARM/Kunpeng and 2% on x86 servers, as opposed to the overhead of disk tables of 20% on ARM/Kunpeng and 15% on x86 servers. + +**Performance per CPU core** + +The following figure shows the TPC-C benchmark performance/throughput results per core of the tests performed on a Huawei ARM/Kunpeng server that has two sockets and 128 cores. The same four types of tests were performed (as described above). + +**Figure 2** ARM/Kunpeng 2-Socket 128 Cores - Performance per Core Benchmarks + +![arm-kunpeng-2-socket-128-cores-performance-per-core-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-11.png) + +The results showed that as expected, the performance of MOT tables is significantly greater per core than of disk-based tables in all cases. It also shows that for production grade (high-availability) servers (Primary/Secondary Node) that require replication, the benefit of using MOT tables is even more significant than for a Single Node (without high-availability, meaning no replication). + +### ARM/Kunpeng 4-Socket 256 Cores + +The following demonstrates MOT's excellent concurrency control performance by showing the tpmC per quantity of connections. + +**Figure 3** ARM/Kunpeng 4-Socket 256 Cores - Performance Benchmarks + +![arm-kunpeng-4-socket-256-cores-performance-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-12.png) + +The results show that performance increases significantly even when there are many cores and that peak performance of 4.8M tpmC is achieved at 768 connections. + +### x86-based Servers + +- **8-Socket 384 Cores** + +The following demonstrates MOT’s excellent concurrency control performance by comparing the tpmC per quantity of connections between disk-based tables and MOT. This test was performed on an x86 server with eight sockets and 384 cores. The orange represents the results of the MOT table. + +**Figure 4** x86 8-Socket 384 Cores - Performance Benchmarks + +![x86-8-socket-384-cores-performance-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-13.png) + +The results show that MOT tables significantly outperform disk-based tables and have very highly efficient performance per core on a 386 core server, reaching over 3M tpmC / core. + +- **4-Socket 96 Cores** + +3.9 million tpmC was achieved by MOT on this 4-socket 96 cores server. The following figure shows a highly efficient MOT table performance per core reaching 40,000 tpmC / core. + +**Figure 5** 4-Socket 96 Cores - Performance Benchmarks + +![4-socket-96-cores-performance-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-14.png) + +## MOT Low Latency + +The following was measured on ARM/Kunpeng 2-socket server (128 cores). The numbers scale is milliseconds (ms). + +**Figure 1** Low Latency (90th%) - Performance Benchmarks + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-15.png) + +MOT's average transaction speed is 2.5x, with MOT latency of 10.5 ms, compared to 23-25ms for disk tables. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The average was calculated by taking into account all TPC-C 5 transaction percentage distributions. For more information, you may refer to the description of TPC-C transactions in the **MOT Sample TPC-C Benchmark** section. + +**Figure 2** Low Latency (90th%, Transaction Average) - Performance Benchmarks + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-16.png) + +## MOT RTO and Cold-Start Time + +### High Availability Recovery Time Objective (RTO) + +MOT is fully integrated into MogDB, including support for high-availability scenarios consisting of primary and secondary deployments. The WAL Redo Log's replication mechanism replicates changes into the secondary database node and uses it for replay. + +If a Failover event occurs, whether it is due to an unplanned primary node failure or due to a planned maintenance event, the secondary node quickly becomes active. The amount of time that it takes to recover and replay the WAL Redo Log and to enable connections is also referred to as the Recovery Time Objective (RTO). + +**The RTO of MogDB, including the MOT, is less than 10 seconds.** + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The Recovery Time Objective (RTO) is the duration of time and a service level within which a business process must be restored after a disaster in order to avoid unacceptable consequences associated with a break in continuity. In other words, the RTO is the answer to the question: "How much time did it take to recover after notification of a business process disruption?" + +In addition, as shown in the **MOT High Throughput** section in MOT the replication overhead of a Primary/Secondary High Availability scenario is only 7% on ARM/Kunpeng servers and 2% on x86 servers, as opposed to the replication overhead of disk-tables, which is 20% on ARM/Kunpeng and 15% on x86 servers. + +### Cold-Start Recovery Time + +Cold-start Recovery time is the amount of time it takes for a system to become fully operational after a stopped mode. In memory databases, this includes the loading of all data and indexes into memory, thus it depends on data size, hardware bandwidth, and on software algorithms to process it efficiently. + +Our MOT tests using ARM servers with NVMe disks demonstrate the ability to load **100 GB of database checkpoint in 40 seconds (2.5 GB/sec)**. Because MOT does not persist indexes and therefore they are created at cold-start, the actual size of the loaded data + indexes is approximately 50% more. Therefore, can be converted to **MOT cold-start time of Data + Index capacity of 150GB in 40 seconds,** or **225 GB per minute (3.75 GB/sec)**. + +The following figure demonstrates cold-start process and how long it takes to load data into a MOT table from the disk after a cold start. + +**Figure 1** Cold-Start Time - Performance Benchmarks + +![cold-start-time-performance-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-17.png) + +- **Database Size -** The total amount of time to load the entire database (in GB) is represented by the blue line and the **TIME (sec)** Y axis on the left. +- **Throughput -** The quantity of database GB throughput per second is represented by the orange line and the **Throughput GB/sec** Y axis on the right. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The performance demonstrated during the test is very close to the bandwidth of the SSD hardware. Therefore, it is feasible that higher (or lower) performance may be achieved on a different platform. + +## MOT Resource Utilization + +The following figure shows the resource utilization of the test performed on a x86 server with four sockets, 96 cores and 512GB RAM server. It demonstrates that a MOT table is able to efficiently and consistently consume almost all available CPU resources. For example, it shows that almost 100% CPU percentage utilization is achieved for 192 cores and 3.9M tpmC. + +- **tmpC -** Number of TPC-C transactions completed per minute is represented by the orange bar and the **tpmC** Y axis on the left. +- **CPU % Utilization -** The amount of CPU utilization is represented by the blue line and the **CPU %** Y axis on the right. + +**Figure 1** Resource Utilization - Performance Benchmarks + +![resource-utilization-performance-benchmarks](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-18.png) + +## MOT Data Ingestion Speed + +This test simulates realtime data streams arriving from massive IoT, cloud or mobile devices that need to be quickly and continuously ingested into the database on a massive scale. + +- The test involved ingesting large quantities of data, as follows - + + - 10 million rows were sent by 500 threads, 2000 rounds, 10 records (rows) in each insert command, each record was 200 bytes. + - The client and database were on different machines. Database server - x86 2-socket, 72 cores. + +- Performance Results + + - **Throughput - 10,000** Records/Core or **2** MB/Core. + - **Latency - 2.8ms per a 10 records** bulk insert (includes client-server networking) + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** We are projecting that multiple additional, and even significant, performance improvements will be made by MOT for this scenario. Click **MOT Usage Scenarios** for more information about large-scale data streaming and data ingestion. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/1-using-mot-overview.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/1-using-mot-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..a54764d553c93fc863a8b68b0005e8abd5bab3ce --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/1-using-mot-overview.md @@ -0,0 +1,18 @@ +--- +title: Using MOT Overview +summary: Using MOT Overview +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Using MOT Overview + +MOT is automatically deployed as part of openGauss. You may refer to the **MOT Preparation** section for a description of how to estimate and plan required memory and storage resources in order to sustain your workload. The **MOT Deployment** section describes all the configuration settings in MOT, as well as non-mandatory options for server optimization. + +Using MOT tables is quite simple. The syntax of all MOT commands is the same as for disk-based tables and includes support for most of standard PostgreSQL SQL, DDL and DML commands and features, such as Stored Procedures. Only the create and drop table statements in MOT differ from the statements for disk-based tables in openGauss. You may refer to the **MOT Usage** section for a description of these two simple commands, to learn how to convert a disk-based table into an MOT table, to get higher performance using Query Native Compilation and PREPARE statements and for a description of external tool support and the limitations of the MOT engine. + +The **MOT Administration** section describes how to perform database maintenance, monitoring and analysis of logs and reported errors. Lastly, the **MOT Sample TPC-C Benchmark** section describes how to perform a standard TPC-C benchmark. + +- Read the following topics to learn how to use MOT - + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/using-mot-overview-2.png) diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/2-mot-preparation.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/2-mot-preparation.md new file mode 100644 index 0000000000000000000000000000000000000000..3cceb8b2a61ef33fce7dd877b24daca2f1e19efb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/2-mot-preparation.md @@ -0,0 +1,206 @@ +--- +title: MOT Preparation +summary: MOT Preparation +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Preparation + +The following describes the prerequisites and the memory and storage planning to perform in order to prepare to use MOT. + +## MOT Prerequisites + +The following specifies the hardware and software prerequisites for using MogDB MOT. + +### Supported Hardware + +MOT can utilize state-of-the-art hardware, as well as support existing hardware platforms. Both x86 architecture and ARM by Huawei Kunpeng architecture are supported. + +MOT is fully aligned with the hardware supported by the MogDB database. For more information, see the *MogDB Installation Guide*. + +### CPU + +MOT delivers exceptional performance on many-core servers (scale-up). MOT significantly outperforms the competition in these environments and provides near-linear scaling and extremely high resource utilization. + +Even so, users can already start realizing MOT's performance benefits on both low-end, mid-range and high-end servers, starting from one or two CPU sockets, as well as four and even eight CPU sockets. Very high performance and resource utilization are also expected on very high-end servers that have 16 or even 32 sockets (for such cases, we recommend contacting Enmo support). + +### Memory + +MOT supports standard RAM/DRAM for its data and transaction management. All MOT tables’ data and indexes reside in-memory; therefore, the memory capacity must support the data capacity and still have space for further growth. For detailed information about memory requirements and planning, see the **MOT Memory and Storage Planning** section. + +### Storage IO + +MOT is a durable database and uses persistent storage (disk/SSD/NVMe drive[s]) for transaction log operations and periodic checkpoints. + +We recommend using a storage device with low latency, such as SSD with a RAID-1 configuration, NVMe or any enterprise-grade storage system. When appropriate hardware is used, the database transaction processing and contention are the bottleneck, not the IO. + +For detailed memory requirements and planning, see the **MOT Memory and Storage Planning** section. + +Supported Operating Systems + +MOT is fully aligned with the operating systems supported by MogDB. + +MOT supports both bare-metal and virtualized environments that run the following operating systems on a bare-metal server or virtual machine - + +- **x86 -** CentOS 7.6 and EulerOS 2.0 +- **ARM -** openEuler and EulerOS + +### OS Optimization + +MOT does not require any special modifications or the installation of new software. However, several optional optimizations can enhance performance. You may refer to the **MOT Server Optimization - x86** and **MOT Server Optimization - ARM Huawei Taishan 2P/4P** sections for a description of the optimizations that enable maximal performance. + +## MOT Memory and Storage Planning + +This section describes the considerations and guidelines for evaluating, estimating and planning the quantity of memory and storage capacity to suit your specific application needs. This section also describes the various data aspects that affect the quantity of required memory, such as the size of data and indexes for the planned tables, memory to sustain transaction management and how fast the data is growing. + +### MOT Memory Planning + +MOT belongs to the in-memory database class (IMDB) in which all tables and indexes reside entirely in memory. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Memory storage is volatile, meaning that it requires power to maintain the stored information. Disk storage is persistent, meaning that it is written to disk, which is non-volatile storage. MOT uses both, having all data in memory, while persisting (by WAL logging) transactional changes to disk with strict consistency (in synchronous logging mode). + +Sufficient physical memory must exist on the server in order to maintain the tables in their initial state, as well as to accommodate the related workload and growth of data. All this is in addition to the memory that is required for the traditional disk-based engine, tables and sessions that support the workload of disk-based tables. Therefore, planning ahead for enough memory to contain them all is essential. + +Even so, you can get started with whatever amount of memory you have and perform basic tasks and evaluation tests. Later, when you are ready for production, the following issues should be addressed. + +- **Memory Configuration Settings** + + Similar to standard PG , the memory of the MogDB database process is controlled by the upper limit in its max_process_memory setting, which is defined in the postgres.conf file. The MOT engine and all its components and threads, reside within the MogDB process. Therefore, the memory allocated to MOT also operates within the upper boundary defined by max_process_memory for the entire MogDB database process. + + The amount of memory that MOT can reserve for itself is defined as a portion of max_process_memory. It is either a percentage of it or an absolute value that is less than it. This portion is defined in the mot.conf configuration file by the _mot__memory settings. + + The portion of max_process_memory that can be used by MOT must still leave at least 2 GB available for the PG (MogDB) envelope. Therefore, in order to ensure this, MOT verifies the following during database startup - + + ``` + (max_mot_global_memory + max_mot_local_memory) + 2GB < max_process_memory + ``` + + If this limit is breached, then MOT memory internal limits are adjusted in order to provide the maximum possible within the limitations described above. This adjustment is performed during startup and calculates the value of MOT max memory accordingly. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** MOT max memory is a logically calculated value of either the configured settings or their adjusted values of (max_mot_global_memory + max_mot_local_memory). + + In this case, a warning is issued to the server log, as shown below - + + **Warning Examples** + + Two messages are reported - the problem and the solution. + + The following is an example of a warning message reporting the problem - + + ``` + [WARNING] MOT engine maximum memory definitions (global: 9830 MB, local: 1843 MB, session large store: 0 MB, total: 11673 MB) breach GaussDB maximum process memory restriction (12288 MB) and/or total system memory (64243 MB). MOT values shall be adjusted accordingly to preserve required gap (2048 MB). + ``` + + The following is an example of a warning message indicating that MOT is automatically adjusting the memory limits - + + ``` + [WARNING] Adjusting MOT memory limits: global = 8623 MB, local = 1617 MB, session large store = 0 MB, total = 10240 MB + ``` + + This is the only place that shows the new memory limits. + + Additionally, MOT does not allow the insertion of additional data when the total memory usage approaches the chosen memory limits. The threshold for determining when additional data insertions are no longer allowed, is defined as a percentage of MOT max memory (which is a calculated value, as described above). The default is 90, meaning 90%. Attempting to add additional data over this threshold returns an error to the user and is also registered in the database log file. + +- **Minimum and Maximum** + + In order to secure memory for future operations, MOT pre-allocates memory based on the minimum global and local settings. The database administrator should specify the minimum amount of memory required for the MOT tables and sessions to sustain their workload. This ensures that this minimal memory is allocated to MOT even if another excessive memory-consuming application runs on the same server as the database and competes with the database for memory resources. The maximum values are used to limit memory growth. + +- **Global and Local** + + The memory used by MOT is comprised of two parts - + + - **Global Memory -** Global memory is a long-term memory pool that contains the data and indexes of MOT tables. It is evenly distributed across NUMA-nodes and is shared by all CPU cores. + - **Local Memory -** Local memory is a memory pool used for short-term objects. Its primary consumers are sessions handling transactions. These sessions are storing data changes in the part of the memory dedicated to the relevant specific transaction (known as *transaction private memory*). Data changes are moved to the global memory at the commit phase. Memory object allocation is performed in NUMA-local manner in order to achieve the lowest possible latency. + + Deallocated objects are put back in the relevant memory pools. Minimal use of operating system memory allocation (malloc) functions during transactions circumvents unnecessary locks and latches. + + The allocation of these two memory parts is controlled by the dedicated **min/max_mot_global_memory** and **min/max_mot_local_memory** settings. If MOT global memory usage gets too close to this defined maximum, then MOT protects itself and does not accept new data. Attempts to allocate memory beyond this limit are denied and an error is reported to the user. + +- **Minimum Memory Requirements** + + To get started and perform a minimal evaluation of MOT performance, there are a few requirements. + + Make sure that the **max_process_memory** (as defined in **postgres.conf**) has sufficient capacity for MOT tables and sessions (configured by **mix/max_mot_global_memory** and **mix/max_mot_local_memory**), in addition to the disk tables buffer and extra memory. For simple tests, the default **mot.conf** settings can be used. + +- **Actual Memory Requirements During Production** + + In a typical OLTP workload, with 80:20 read:write ratio on average, MOT memory usage per table is 60% higher than in disk-based tables (this includes both the data and the indexes). This is due to the use of more optimal data structures and algorithms that enable faster access, with CPU-cache awareness and memory-prefetching. + + The actual memory requirement for a specific application depends on the quantity of data, the expected workload and especially on the data growth. + +- **Max Global Memory Planning - Data + Index Size** + + To plan for maximum global memory - + + 1. Determine the size of a specific disk table (including both its data and all its indexes). The following statistical query can be used to determine the data size of the **customer** table and the **customer_pkey** index size - + - **Data size -** select pg_relation_size(‘customer'); + - **Index -** select pg_relation_size('customer_pkey'); + 2. Add 60%, which is the common requirement in MOT relative to the current size of the disk-based data and index. + 3. Add an additional percentage for the expected growth of data. For example - + + 5% monthly growth = 80% yearly growth (1.05^12). Thus, in order to sustain a year's growth, allocate 80% more memory than is currently used by the tables. + + This completes the estimation and planning of the max_mot_global_memory value. The actual setting can be defined either as an absolute value or a percentage of the Postgres max_process_memory. The exact value is typically finetuned during deployment. + +- **Max Local Memory Planning - Concurrent Session Support** + + Local memory needs are primarily a function of the quantity of concurrent sessions. The typical OLTP workload of an average session uses up to 8 MB. This should be multiplied by the quantity of sessions and then a little bit extra should be added. + + A memory calculation can be performed in this manner and then finetuned, as follows - + + ``` + SESSION_COUNT * SESSION_SIZE (8 MB) + SOME_EXTRA (100MB should be enough) + ``` + + The default specifies 15% of Postgres's max_process_memory, which by default is 12 GB. This equals 1.8 GB, which is sufficient for 230 sessions, which is the requirement for the max_mot_local memory. The actual setting can be defined either in absolute values or as a percentage of the Postgres max_process_memory. The exact value is typically finetuned during deployment. + + **Unusually Large Transactions** + + Some transactions are unusually large because they apply changes to a large number of rows. This may increase a single session's local memory up to the maximum allowed limit, which is 1 GB. For example - + + ``` + delete from SOME_VERY_LARGE_TABLE; + ``` + + Take this scenario into consideration when configuring the max_mot_local_memory setting, as well as during application development. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** You may refer to the **MEMORY (MOT)** section for more information about configuration settings. + +### Storage IO + +MOT is a memory-optimized, persistent database storage engine. A disk drive(s) is required for storing the Redo Log (WAL) and a periodic checkpoint. + +It is recommended to use a storage device with low latency, such as SSD with a RAID-1 configuration, NVMe or any enterprise-grade storage system. When appropriate hardware is used, the database transaction processing and contention are the bottleneck, not the IO. + +Since the persistent storage is much slower than RAM memory, the IO operations (logging and checkpoint) can create a bottleneck for both an in-memory and memory-optimized databases. However, MOT has a highly efficient durability design and implementation that is optimized for modern hardware (such as SSD and NVMe). In addition, MOT has minimized and optimized writing points (for example, by using parallel logging, a single log record per transaction and NUMA-aware transaction group writing) and has minimized the data written to disk (for example, only logging the delta or updated columns of the changed records and only logging a transaction at the commit phase). + +### Required Capacity + +The required capacity is determined by the requirements of checkpointing and logging, as described below - + +- **Checkpointing** + + A checkpoint saves a snapshot of all the data to disk. + + Twice the size of all data should be allocated for checkpointing. There is no need to allocate space for the indexes for checkpointing + + Checkpointing = 2x the MOT Data Size (rows only, index is not persistent). + + Twice the size is required because a snapshot is saved to disk of the entire size of the data, and in addition, the same amount of space should be allocated for the checkpoint that is in progress. When a checkpoint process finishes, the previous checkpoint files are deleted. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** In the next MogDB release, MOT will have an incremental checkpoint feature, which will significantly reduce this storage capacity requirement. + +- **Logging** + + MOT table log records are written to the same database transaction log as the other records of disk-based tables. + + The size of the log depends on the transactional throughput, the size of the data changes and the time between checkpoints (at each time checkpoint the Redo Log is truncated and starts to expand again). + + MOT tables use less log bandwidth and have lower IO contention than disk-based tables. This is enabled by multiple mechanisms. + + For example, MOT does not log every operation before a transaction has been completed. It is only logged at the commit phase and only the updated delta record is logged (not full records like for disk-based tables). + + In order to ensure that the log IO device does not become a bottleneck, the log file must be placed on a drive that has low latency. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** You may refer to the **STORAGE (MOT)** section for more information about configuration settings. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/3-mot-deployment.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/3-mot-deployment.md new file mode 100644 index 0000000000000000000000000000000000000000..1bf1320b9f6b91492427e40529ee6ffe2eb1c133 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/3-mot-deployment.md @@ -0,0 +1,660 @@ +--- +title: MOT Deployment +summary: MOT Deployment +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Deployment + +The following sections describe various mandatory and optional settings for optimal deployment. + +## MOT Server Optimization - x86 + +Generally, databases are bounded by the following components - + +- **CPU -** A faster CPU speeds up any CPU-bound database. +- **Disk -** High-speed SSD/NVME speeds up any I/O-bound database. +- **Network -** A faster network speeds up any **SQL\\Net**-bound database. + +In addition to the above, the following general-purpose server settings are used by default and may significantly affect a database's performance. + +MOT performance tuning is a crucial step for ensuring fast application functionality and data retrieval. MOT can utilize state-of-the-art hardware, and therefore it is extremely important to tune each system in order to achieve maximum throughput. + +The following are optional settings for optimizing MOT database performance running on an Intel x86 server. These settings are optimal for high throughput workloads - + +### BIOS + +- Hyper Threading - ON + + Activation (HT=ON) is highly recommended. + + We recommend turning hyper threading ON while running OLTP workloads on MOT. When hyper-threading is used, some OLTP workloads demonstrate performance gains of up to40%. + +### OS Environment Settings + +- NUMA + + Disable NUMA balancing, as described below. MOT performs its own memory management with extremely efficient NUMA-awareness, much more than the default methods used by the operating system. + + ``` + echo 0 > /proc/sys/kernel/numa_balancing + ``` + +- Services + + Disable Services, as described below - + + ``` + service irqbalance stop # MANADATORY + service sysmonitor stop # OPTIONAL, performance + service rsyslog stop # OPTIONAL, performance + ``` + +- Tuned Service + + The following section is mandatory. + + The server must run the throughput-performance profile - + + ``` + [...]$ tuned-adm profile throughput-performance + ``` + + The **throughput-performance** profile is broadly applicable tuning that provides excellent performance across a variety of common server workloads. + + Other less suitable profiles for MogDB and MOT server that may affect MOT's overall performance are - balanced, desktop, latency-performance, network-latency, network-throughput and powersave. + +- Sysctl + + The following lists the recommended operating system settings for best performance. + + - Add the following settings to /etc/sysctl.conf and run sysctl -p + + ```bash + net.ipv4.ip_local_port_range = 9000 65535 + kernel.sysrq = 1 + kernel.panic_on_oops = 1 + kernel.panic = 5 + kernel.hung_task_timeout_secs = 3600 + kernel.hung_task_panic = 1 + vm.oom_dump_tasks = 1 + kernel.softlockup_panic = 1 + fs.file-max = 640000 + kernel.msgmnb = 7000000 + kernel.sched_min_granularity_ns = 10000000 + kernel.sched_wakeup_granularity_ns = 15000000 + kernel.numa_balancing=0 + vm.max_map_count = 1048576 + net.ipv4.tcp_max_tw_buckets = 10000 + net.ipv4.tcp_tw_reuse = 1 + net.ipv4.tcp_tw_recycle = 1 + net.ipv4.tcp_keepalive_time = 30 + net.ipv4.tcp_keepalive_probes = 9 + net.ipv4.tcp_keepalive_intvl = 30 + net.ipv4.tcp_retries2 = 80 + kernel.sem = 250 6400000 1000 25600 + net.core.wmem_max = 21299200 + net.core.rmem_max = 21299200 + net.core.wmem_default = 21299200 + net.core.rmem_default = 21299200 + #net.sctp.sctp_mem = 94500000 915000000 927000000 + #net.sctp.sctp_rmem = 8192 250000 16777216 + #net.sctp.sctp_wmem = 8192 250000 16777216 + net.ipv4.tcp_rmem = 8192 250000 16777216 + net.ipv4.tcp_wmem = 8192 250000 16777216 + net.core.somaxconn = 65535 + vm.min_free_kbytes = 26351629 + net.core.netdev_max_backlog = 65535 + net.ipv4.tcp_max_syn_backlog = 65535 + #net.sctp.addip_enable = 0 + net.ipv4.tcp_syncookies = 1 + vm.overcommit_memory = 0 + net.ipv4.tcp_retries1 = 5 + net.ipv4.tcp_syn_retries = 5 + ``` + + - Update the section of /etc/security/limits.conf to the following - + + ```bash + soft nofile 100000 + hard nofile 100000 + ``` + + The **soft** and a **hard** limit settings specify the quantity of files that a process may have opened at once. The soft limit may be changed by each process running these limits up to the hard limit value. + +- Disk/SSD + + The following describes how to ensure that disk R/W performance is suitable for database synchronous commit mode. + + To do so, test your disk bandwidth using the following + + ``` + [...]$ sync; dd if=/dev/zero of=testfile bs=1M count=1024; sync + 1024+0 records in + 1024+0 records out + 1073741824 bytes (1.1 GB) copied, 1.36034 s, 789 MB/s + ``` + + In case the disk bandwidth is significantly below the above number (789 MB/s), it may create a performance bottleneck for MogDB, and especially for MOT. + +### Network + +Use a 10Gbps network or higher. + +To verify, use iperf, as follows - + +``` +Server side: iperf -s +Client side: iperf -c +``` + +- rc.local - Network Card Tuning + + The following optional settings have a significant effect on performance - + + 1. Copy set_irq_affinity.sh from to /var/scripts/. + + 2. Put in /etc/rc.d/rc.local and run chmod in order to ensure that the following script is executed during boot - + + ```bash + chmod +x /etc/rc.d/rc.local + var/scripts/set_irq_affinity.sh -x all + ethtool -K gro off + ethtool -C adaptive-rx on adaptive-tx on + Replace with the network card, i.e. ens5f1 + ``` + +## MOT Server Optimization - ARM Huawei Taishan 2P/4P + +The following are optional settings for optimizing MOT database performance running on an ARM/Kunpeng-based Huawei Taishan 2280 v2 server powered by 2-sockets with a total of 256 Cores and Taishan 2480 v2 server powered by 4-sockets with a total of 256 Cores. + +Unless indicated otherwise, the following settings are for both client and server machines - + +### BIOS + +Modify related BIOS settings, as follows - + +1. Select **BIOS** - **Advanced** - **MISC Config**. Set **Support Smmu** to **Disabled**. + +2. Select **BIOS** - **Advanced** - **MISC Config**. Set **CPU Prefetching Configuration** to **Disabled**. + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-deployment-1.png) + +3. Select **BIOS** - **Advanced** - **Memory Config**. Set **Die Interleaving** to **Disabled**. + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-deployment-2.png) + +4. Select **BIOS** - **Advanced** - **Performance Config**. Set **Power Policy** to **Performance**. + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-deployment-3.png) + +### OS - Kernel and Boot + +- The following operating system kernel and boot parameters are usually configured by a sysadmin. + + Configure the kernel parameters, as follows - + + ```bash + net.ipv4.ip_local_port_range = 9000 65535 + kernel.sysrq = 1 + kernel.panic_on_oops = 1 + kernel.panic = 5 + kernel.hung_task_timeout_secs = 3600 + kernel.hung_task_panic = 1 + vm.oom_dump_tasks = 1 + kernel.softlockup_panic = 1 + fs.file-max = 640000 + kernel.msgmnb = 7000000 + kernel.sched_min_granularity_ns = 10000000 + kernel.sched_wakeup_granularity_ns = 15000000 + kernel.numa_balancing=0 + vm.max_map_count = 1048576 + net.ipv4.tcp_max_tw_buckets = 10000 + net.ipv4.tcp_tw_reuse = 1 + net.ipv4.tcp_tw_recycle = 1 + net.ipv4.tcp_keepalive_time = 30 + net.ipv4.tcp_keepalive_probes = 9 + net.ipv4.tcp_keepalive_intvl = 30 + net.ipv4.tcp_retries2 = 80 + kernel.sem = 32000 1024000000 500 32000 + kernel.shmall = 52805669 + kernel.shmmax = 18446744073692774399 + sys.fs.file-max = 6536438 + net.core.wmem_max = 21299200 + net.core.rmem_max = 21299200 + net.core.wmem_default = 21299200 + net.core.rmem_default = 21299200 + net.ipv4.tcp_rmem = 8192 250000 16777216 + net.ipv4.tcp_wmem = 8192 250000 16777216 + net.core.somaxconn = 65535 + vm.min_free_kbytes = 5270325 + net.core.netdev_max_backlog = 65535 + net.ipv4.tcp_max_syn_backlog = 65535 + net.ipv4.tcp_syncookies = 1 + vm.overcommit_memory = 0 + net.ipv4.tcp_retries1 = 5 + net.ipv4.tcp_syn_retries = 5 + ##NEW + kernel.sched_autogroup_enabled=0 + kernel.sched_min_granularity_ns=2000000 + kernel.sched_latency_ns=10000000 + kernel.sched_wakeup_granularity_ns=5000000 + kernel.sched_migration_cost_ns=500000 + vm.dirty_background_bytes=33554432 + kernel.shmmax=21474836480 + net.ipv4.tcp_timestamps = 0 + net.ipv6.conf.all.disable_ipv6=1 + net.ipv6.conf.default.disable_ipv6=1 + net.ipv4.tcp_keepalive_time=600 + net.ipv4.tcp_keepalive_probes=3 + kernel.core_uses_pid=1 + ``` + +- Tuned Service + + The following section is mandatory. + + The server must run a throughput-performance profile - + + ``` + [...]$ tuned-adm profile throughput-performance + ``` + + The **throughput-performance** profile is broadly applicable tuning that provides excellent performance across a variety of common server workloads. + + Other less suitable profiles for MogDB and MOT server that may affect MOT's overall performance are - balanced, desktop, latency-performance, network-latency, network-throughput and powersave. + +- Boot Tuning + + Add **iommu.passthrough=1** to the **kernel boot arguments**. + + When operating in **pass-through** mode, the adapter does require **DMA translation to the memory,** which improves performance. + +## MOT Configuration Settings + +MOT is provided preconfigured to creating working MOT Tables. For best results, it is recommended to customize the MOT configuration (defined in the file named mot.conf) according to your application's specific requirements and your preferences. + +This file is read-only upon server startup. If you edit this file while the system is running, then the server must be reloaded in order for the changes to take effect. + +The mot.conf file is located in the same folder as the postgres.conf configuration file. + +Read the **General Guidelines** section and then review and configure the following sections of the mot.conf file, as needed. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The topics listed above describe each of the setting sections in the mot.conf file. In addition to the above topics, for an overview of all the aspects of a specific MOT feature (such as Recovery), you may refer to the relevant topic of this user manual. For example, the mot.conf file has a Recovery section that contains settings that affect MOT recovery and this is described in the **MOT Recovery** section that is listed above. In addition, for a full description of all aspects of Recovery, you may refer to the **MOT Recovery** section of the Administration chapter of this user manual. Reference links are also provided in each relevant section of the descriptions below. + +The following topics describe each section in the mot.conf file and the settings that it contains, as well as the default value of each. + +### General Guidelines + +The following are general guidelines for editing the mot.conf file. + +- Each setting appears with its default value as follows - + + ``` + # name = value + ``` + +- Blank/white space is acceptable. + +- Comments are indicated by placing a number sign (#) anywhere on a line. + +- The default values of each setting appear as a comment throughout this file. + +- In case a parameter is uncommented and a new value is placed, the new setting is defined. + +- Changes to the mot.conf file are applied only at the start or reload of the database server. + +Memory Units are represented as follows - + +- KB - Kilobytes +- MB - Megabytes +- GB - Gigabytes +- TB - Terabytes + +If no memory units are specified, then bytes are assumed. + +Some memory units are represented as a percentage of the **max_process_memory** setting that is configured in **postgresql.conf**. For example - **20%**. + +Time units are represented as follows - + +- us - microseconds (or micros) +- ms - milliseconds (or millis) +- s - seconds (or secs) +- min - minutes (or mins) +- h - hours +- d - days + +If no time units are specified, then microseconds are assumed. + +### REDO LOG (MOT) + +- **enable_group_commit = false** + + Specifies whether to use group commit. + + This option is only relevant when MogDB is configured to use synchronous commit, meaning only when the synchronous_commit setting in postgresql.conf is configured to any value other than off. + +- **group_commit_size = 16** + +- **group_commit_timeout = 10 ms** + + This option is only relevant when the MOT engine has been configured to **Synchronous Group Commit** logging. This means that the synchronous_commit setting in postgresql.conf is configured to true and the enable_group_commit parameter in the mot.conf configuration file is configured to true. + + Defines which of the following determines when a group of transactions is recorded in the WAL Redo Log - + + **group_commit_size** - The quantity of committed transactions in a group. For example, **16** means that when 16 transactions in the same group have been committed by their client application, then an entry is written to disk in the WAL Redo Log for each of the 16 transactions. + + **group_commit_timeout** - A timeout period in ms. For example, **10** means that after 10 ms, an entry is written to disk in the WAL Redo Log for each of the transactions in the same group that have been committed by their client application in the lats 10 ms. + + A commit group is closed after either the configured number of transactions has arrived or after the configured timeout period since the group was opened. After the group is closed, all the transactions in the group wait for a group flush to complete execution and then notify the client that each transaction has ended. + + You may refer to **MOT Logging - WAL Redo Log** section for more information about the WAL Redo Log and synchronous group commit logging. + +### CHECKPOINT (MOT) + +- **checkpoint_dir =** + + Specifies the directory in which checkpoint data is to be stored. The default location is in the data folder of each data node. + +- **checkpoint_segsize = 16 MB** + + Specifies the segment size used during checkpoint. Checkpoint is performed in segments. When a segment is full, it is serialized to disk and a new segment is opened for the subsequent checkpoint data. + +- **checkpoint_workers = 3** + + Specifies the number of workers to use during checkpoint. + + Checkpoint is performed in parallel by several MOT engine workers. The quantity of workers may substantially affect the overall performance of the entire checkpoint operation, as well as the operation of other running transactions. To achieve a shorter checkpoint duration, a larger number of workers should be used, up to the optimal number (which varies based on the hardware and workload). However, be aware that if this number is too large, it may negatively impact the execution time of other running transactions. Keep this number as low as possible to minimize the effect on the runtime of other running transactions, but at the cost of longer checkpoint duration. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** You may refer to the **MOT Checkpoints** section for more information about configuration settings. + +### RECOVERY (MOT) + +- **checkpoint_recovery_workers = 3** + + Specifies the number of workers (threads) to use during checkpoint data recovery. Each MOT engine worker runs on its own core and can process a different table in parallel by reading it into memory. For example, while the default is three-course, you might prefer to set this parameter to the number of cores that are available for processing. After recovery these threads are stopped and killed. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** You may refer to the **MOT Recovery** section for more information about configuration settings. + +### STATISTICS (MOT) + +- **enable_stats = false** + + Configures periodic statistics for printing. + +- **print_stats_period = 10 minute** + + Configures the time period for printing a summary statistics report. + +- **print_full_stats_period = 1 hours** + + Configures the time period for printing a full statistics report. + + The following settings configure the various sections included in the periodic statistics report. If none of them are configured, then the statistics report is suppressed. + +- **enable_log_recovery_stats = false** + + Log recovery statistics contain various Redo Log recovery metrics. + +- **enable_db_session_stats = false** + + Database session statistics contain transaction events, such commits, rollbacks and so on. + +- **enable_network_stats = false** + + Network statistics contain connection/disconnection events. + +- **enable_log_stats = false** + + Log statistics contain details regarding the Redo Log. + +- **enable_memory_stats = false** + + Memory statistics contain memory-layer details. + +- **enable_process_stats = false** + + Process statistics contain total memory and CPU consumption for the current process. + +- **enable_system_stats = false** + + System statistics contain total memory and CPU consumption for the entire system. + +- **enable_jit_stats = false** + + JIT statistics contain information regarding JIT query compilation and execution. + +### ERROR LOG (MOT) + +- **log_level = INFO** + + Configures the log level of messages issued by the MOT engine and recorded in the Error log of the database server. Valid values are PANIC, ERROR, WARN, INFO, TRACE, DEBUG, DIAG1 and DIAG2. + +- **Log.COMPONENT.LOGGER.log_level=LOG_LEVEL** + + Configures specific loggers using the syntax described below. + + For example, to configure the TRACE log level for the ThreadIdPool logger in system component, use the following syntax - + + ``` + Log.System.ThreadIdPool.log_level=TRACE + ``` + + To configure the log level for all loggers under some component, use the following syntax - + + ``` + Log.COMPONENT.log_level=LOG_LEVEL + ``` + + For example - + + ``` + Log.System.log_level=DEBUG + ``` + +### MEMORY (MOT) + +- **enable_numa = true** + + Specifies whether to use NUMA-aware memory allocation. + + When disabled, all affinity configurations are disabled as well. + + MOT engine assumes that all the available NUMA nodes have memory. If the machine has some special configuration in which some of the NUMA nodes have no memory, then the MOT engine initialization and hence the database server startup will fail. In such machines, it is recommended that this configuration value be set to false, in order to prevent startup failures and let the MOT engine to function normally without using NUMA-aware memory allocation. + +- **affinity_mode = fill-physical-first** + + Configures the affinity mode of threads for the user session and internal MOT tasks. + + When a thread pool is used, this value is ignored for user sessions, as their affinity is governed by the thread pool. However, it is still used for internal MOT tasks. + + Valid values are **fill-socket-first**, **equal-per-socket**, **fill-physical-first** and **none** - + + - **Fill-socket-first** attaches threads to cores in the same socket until the socket is full and then moves to the next socket. + - **Equal-per-socket** spreads threads evenly among all sockets. + - **Fill-physical-first** attaches threads to physical cores in the same socket until all physical cores are employed and then moves to the next socket. When all physical cores are used, then the process begins again with hyper-threaded cores. + - **None** disables any affinity configuration and lets the system scheduler determine on which core each thread is scheduled to run. + +- **lazy_load_chunk_directory = true** + + Configures the chunk directory mode that is used for memory chunk lookup. + + **Lazy** mode configures the chunk directory to load parts of it on demand, thus reducing the initial memory footprint (from 1 GB to 1 MB approximately). However, this may result in minor performance penalties and errors in extreme conditions of memory distress. In contrast, using a **non-lazy** chunk directory allocates an additional 1 GB of initial memory, produces slightly higher performance and ensures that chunk directory errors are avoided during memory distress. + +- **reserve_memory_mode = virtual** + + Configures the memory reservation mode (either **physical** or **virtual**). + + Whenever memory is allocated from the kernel, this configuration value is consulted to determine whether the allocated memory is to be resident (**physical**) or not (**virtual**). This relates primarily to preallocation, but may also affect runtime allocations. For **physical** reservation mode, the entire allocated memory region is made resident by forcing page faults on all pages spanned by the memory region. Configuring **virtual** memory reservation may result in faster memory allocation (particularly during preallocation), but may result in page faults during the initial access (and thus may result in a slight performance hit) and more sever errors when physical memory is unavailable. In contrast, physical memory allocation is slower, but later access is both faster and guaranteed. + +- **store_memory_policy = compact** + + Configures the memory storage policy (**compact** or **expanding**). + + When **compact** policy is defined, unused memory is released back to the kernel, until the lower memory limit is reached (see **min_mot_memory** below). In **expanding** policy, unused memory is stored in the MOT engine for later reuse. A **compact** storage policy reduces the memory footprint of the MOT engine, but may occasionally result in minor performance degradation. In addition, it may result in unavailable memory during memory distress. In contrast, **expanding** mode uses more memory, but results in faster memory allocation and provides a greater guarantee that memory can be re-allocated after being de-allocated. + +- **chunk_alloc_policy = auto** + + Configures the chunk allocation policy for global memory. + + MOT memory is organized in chunks of 2 MB each. The source NUMA node and the memory layout of each chunk affect the spread of table data among NUMA nodes, and therefore can significantly affect the data access time. When allocating a chunk on a specific NUMA node, the allocation policy is consulted. + + Available values are **auto**, **local**, **page-interleaved**, **chunk-interleaved** and **native** - + + - **Auto** policy selects a chunk allocation policy based on the current hardware. + - **Local** policy allocates each chunk on its respective NUMA node. + - **Page-interleaved** policy allocates chunks that are composed of interleaved memory 4-kilobyte pages from all NUMA nodes. + - **Chunk-interleaved** policy allocates chunks in a round robin fashion from all NUMA nodes. + - **Native** policy allocates chunks by calling the native system memory allocator. + +- **chunk_prealloc_worker_count = 8** + + Configures the number of workers per NUMA node participating in memory preallocation. + +- **max_mot_global_memory = 80%** + + Configures the maximum memory limit for the global memory of the MOT engine. + + Specifying a percentage value relates to the total defined by **max_process_memory** configured in **postgresql.conf**. + + The MOT engine memory is divided into global (long-term) memory that is mainly used to store user data and local (short-term) memory that is mainly used by user sessions for local needs. + + Any attempt to allocate memory beyond this limit is denied and an error is reported to the user. Ensure that the sum of **max_mot_global_memory** and **max_mot_local_memory** do not exceed the **max_process_memory** configured in **postgresql.conf**. + +- **min_mot_global_memory = 0 MB** + + Configures the minimum memory limit for the global memory of the MOT engine. + + Specifying a percentage value relates to the total defined by the **max_process_memory** configured in **postgresql.conf**. + + This value is used for the preallocation of memory during startup, as well as to ensure that a minimum amount of memory is available for the MOT engine during its normal operation. When using **compact** storage policy (see **store_memory_policy** above), this value designates the lower limit under which memory is not released back to the kernel, but rather kept in the MOT engine for later reuse. + +- **max_mot_local_memory = 15%** + + Configures the maximum memory limit for the local memory of the MOT engine. + + Specifying a percentage value relates to the total defined by the **max_process_memory** configured in **postgresql.conf**. + + MOT engine memory is divided into global (long-term) memory that is mainly used to store user data and local (short-term) memory that is mainly used by user session for local needs. + + Any attempt to allocate memory beyond this limit is denied and an error is reported to the user. Ensure that the sum of **max_mot_global_memory** and **max_mot_local_memory** do not exceed the **max_process_memory** configured in **postgresql.conf**. + +- **min_mot_local_memory = 0 MB** + + Configures the minimum memory limit for the local memory of the MOT engine. + + Specifying a percentage value relates to the total defined by the **max_process_memory** configured in **postgresql.conf**. + + This value is used for preallocation of memory during startup, as well as to ensure that a minimum amount of memory is available for the MOT engine during its normal operation. When using compact storage policy (see **store_memory_policy** above), this value designates the lower limit under which memory is not released back to the kernel, but rather kept in the MOT engine for later reuse. + +- **max_mot_session_memory = 0 MB** + + Configures the maximum memory limit for a single session in the MOT engine. + + Typically, sessions in the MOT engine can allocate as much local memory as needed, so long as the local memory limit is not exceeded. To prevent a single session from taking too much memory, and thereby denying memory from other sessions, this configuration item is used to restrict small session-local memory allocations (up to 1,022 KB). + + Make sure that this configuration item does not affect large or huge session-local memory allocations. + + A value of zero denotes no restriction on any session-local small allocations per session, except for the restriction arising from the local memory allocation limit configured by **max_mot_local_memory**. + + Note: Percentage values cannot be set for this configuration item. + +- **min_mot_session_memory = 0 MB** + + Configures the minimum memory reservation for a single session in the MOT engine. + + This value is used to preallocate memory during session creation, as well as to ensure that a minimum amount of memory is available for the session to perform its normal operation. + + Note: Percentage values cannot be set for this configuration item. + +- **session_large_buffer_store_size = 0 MB** + + Configures the large buffer store for sessions. + + When a user session executes a query that requires a lot of memory (for example, when using many rows), the large buffer store is used to increase the certainty level that such memory is available and to serve this memory request more quickly. Any memory allocation for a session exceeding 1,022 KB is considered as a large memory allocation. If the large buffer store is not used or is depleted, such allocations are treated as huge allocations that are served directly from the kernel. + + Note: Percentage values cannot be set for this configuration item. + +- **session_large_buffer_store_max_object_size = 0 MB** + + Configures the maximum object size in the large allocation buffer store for sessions. + + Internally, the large buffer store is divided into objects of varying sizes. This value is used to set an upper limit on objects originating from the large buffer store, as well as to determine the internal division of the buffer store into objects of various size. + + This size cannot exceed 1⁄8 of the **session_large_buffer_store_size**. If it does, it is adjusted to the maximum possible. + + Note: Percentage values cannot be set for this configuration item. + +- **session_max_huge_object_size = 1 GB** + + Configures the maximum size of a single huge memory allocation made by a session. + + Huge allocations are served directly from the kernel and therefore are not guaranteed to succeed. + + This value also pertains to global (meaning not session-related) memory allocations. + + Note: Percentage values cannot be set for this configuration item. + +### GARBAGE COLLECTION (MOT) + +- **enable_gc = true** + + Specifies whether to use the Garbage Collector (GC). + +- **reclaim_threshold = 512 KB** + + Configures the memory threshold for the garbage collector. + + Each session manages its own list of to-be-reclaimed objects and performs its own garbage collection during transaction commitment. This value determines the total memory threshold of objects waiting to be reclaimed, above which garbage collection is triggered for a session. + + In general, the trade-off here is between un-reclaimed objects vs garbage collection frequency. Setting a low value keeps low levels of un-reclaimed memory, but causes frequent garbage collection that may affect performance. Setting a high value triggers garbage collection less frequently, but results in higher levels of un-reclaimed memory. This setting is dependent upon the overall workload. + +- **reclaim_batch_size = 8000** + + Configures the batch size for garbage collection. + + The garbage collector reclaims memory from objects in batches, in order to restrict the number of objects being reclaimed in a single garbage collection pass. The intent of this approach is to minimize the operation time of a single garbage collection pass. + +- **high_reclaim_threshold = 8 MB** + + Configures the high memory threshold for garbage collection. + + Because garbage collection works in batches, it is possible that a session may have many objects that can be reclaimed, but which were not. In such situations, in order to prevent garbage collection lists from becoming too bloated, this value is used to continue reclaiming objects within a single pass, even though that batch size limit has been reached, until the total size of the still-waiting-to-be-reclaimed objects is less than this threshold, or there are no more objects eligible for reclamation. + +### JIT (MOT) + +- **enable_mot_codegen = true** + + Specifies whether to use JIT query compilation and execution for planned queries. + + JIT query execution enables JIT-compiled code to be prepared for a prepared query during its planning phase. The resulting JIT-compiled function is executed whenever the prepared query is invoked. JIT compilation usually takes place in the form of LLVM. On platforms where LLVM is not natively supported, MOT provides a software-based fallback called Tiny Virtual Machine (TVM). + +- **force_mot_pseudo_codegen = false** + + Specifies whether to use TVM (pseudo-LLVM) even though LLVM is supported on the current platform. + + On platforms where LLVM is not natively supported, MOT automatically defaults to TVM. + + On platforms where LLVM is natively supported, LLVM is used by default. This configuration item enables the use of TVM for JIT compilation and execution on platforms on which LLVM is supported. + +- **enable_mot_codegen_print = false** + + Specifies whether to print emitted LLVM/TVM IR code for JIT-compiled queries. + +- **mot_codegen_limit = 100** + + Limits the number of JIT queries allowed per user session. + +### Default mot.conf + +The minimum settings and configuration specify to point the **postgresql.conf** file to the location of the **mot.conf** file - + +``` +postgresql.conf +mot_config_file = '/tmp/gauss/mot.conf' +``` + +Ensure that the value of the max_process_memory setting is sufficient to include the global (data and index) and local (sessions) memory of MOT tables. + +The default content of **mot.conf** is sufficient to get started. The settings can be optimized later. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/4-mot-usage.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/4-mot-usage.md new file mode 100644 index 0000000000000000000000000000000000000000..f3f4e60c6b9ede4d7b688958e4f2884a16316086 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/4-mot-usage.md @@ -0,0 +1,497 @@ +--- +title: MOT Usage +summary: MOT Usage +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Usage + +Using MOT tables is quite simple and is described in the few short sections below. + +MogDB enables an application to use of MOT tables and standard disk-based tables. You can use MOT tables for your most active, high-contention and throughput-sensitive application tables or you can use MOT tables for all your application's tables. + +The following commands describe how to create MOT tables and how to convert existing disk-based tables into MOT tables in order to accelerate an application's database-related performance. MOT is especially beneficial when applied to tables that have proven to be bottlenecks. + +The following is a simple overview of the tasks related to working with MOT tables: + +- Granting User Permissions +- Creating/Dropping an MOT Table +- Creating an Index for an MOT Table +- Converting a Disk Table into an MOT Table +- Query Native Compilation +- Retrying an Aborted Transaction +- MOT External Support Tools +- MOT SQL Coverage and Limitations + +## Granting User Permissions + +The following describes how to assign a database user permission to access the MOT storage engine. This is performed only once per database user, and is usually done during the initial configuration phase. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The granting of user permissions is required because MOT is integrated into the MogDB database by using and extending the Foreign Data Wrapper (FDW) mechanism, which requires granting user access permissions. + +To enable a specific user to create and access MOT tables (DDL, DML, SELECT) - + +Run the following statement only once - + +```sql +GRANT USAGE ON FOREIGN SERVER mot_server TO ; +``` + +All keywords are not case sensitive. + +## Creating/Dropping an MOT Table + +Creating a Memory Optimized Table (MOT) is very simple. Only the create and drop table statements in MOT differ from the statements for disk-based tables in MogDB. The syntax of **all other** commands for SELECT, DML and DDL are the same for MOT tables as for MogDB disk-based tables. + +- To create an MOT table - + + ```sql + create FOREIGN table test(x int) [server mot_server]; + ``` + +- Always use the FOREIGN keyword to refer to MOT tables. + +- The [server mot_server] part is optional when creating an MOT table because MOT is an integrated engine, not a separate server. + +- The above is an extremely simple example creating a table named **test** with a single integer column named **x**. In the next section (**Creating an Index**) a more realistic example is provided. + +- MOT tables cannot be created if incremental checkpoint is enabled in postgresql.conf. So please set enable_incremental_checkpoint to off before creating the MOT. + +- To drop an MOT table named test - + + ```sql + drop FOREIGN table test; + ``` + +For a description of the limitations of supported features for MOT tables, such as data types, see the **MOT SQL Coverage and Limitations** section. + +## Creating an Index for an MOT Table + +Standard PostgreSQL create and drop index statements are supported. + +For example - + +```sql +create index text_index1 on test(x) ; +``` + +The following is a complete example of creating an index for the ORDER table in a TPC-C workload - + +```sql +create FOREIGN table bmsql_oorder ( + o_w_id integer not null, + o_d_id integer not null, + o_id integer not null, + o_c_id integer not null, + o_carrier_id integer, + o_ol_cnt integer, + o_all_local integer, + o_entry_d timestamp, + primarykey (o_w_id, o_d_id, o_id) +); + +create index bmsql_oorder_index1 on bmsql_oorder(o_w_id, o_d_id, o_c_id, o_id) ; +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** There is no need to specify the **FOREIGN** keyword before the MOT table name, because it is only created for create and drop table commands. + +For MOT index limitations, see the Index subsection under the _SQL Coverage and Limitations_ section. + +## Converting a Disk Table into an MOT Table + +The direct conversion of disk tables into MOT tables is not yet possible, meaning that no ALTER TABLE statement yet exists that converts a disk-based table into an MOT table. + +The following describes how to manually perform a few steps in order to convert a disk-based table into an MOT table, as well as how the **gs_dump** tool is used to export data and the **gs_restore** tool is used to import data. + +### Prerequisite Check + +Check that the schema of the disk table to be converted into an MOT table contains all required columns. + +Check whether the schema contains any unsupported column data types, as described in the _Unsupported Data Types_ section. + +If a specific column is not supported, then it is recommended to first create a secondary disk table with an updated schema. This schema is the same as the original table, except that all the unsupported types have been converted into supported types. + +Afterwards, use the following script to export this secondary disk table and then import it into an MOT table. + +### Converting + +To covert a disk-based table into an MOT table, perform the following - + +1. Suspend application activity. +2. Use **gs_dump** tool to dump the table’s data into a physical file on disk. Make sure to use the **data only**. +3. Rename your original disk-based table. +4. Create an MOT table with the same table name and schema. Make sure to use the create FOREIGN keyword to specify that it will be an MOT table. +5. Use **gs_restore** to load/restore data from the disk file into the database table. +6. Visually/manually verify that all the original data was imported correctly into the new MOT table. An example is provided below. +7. Resume application activity. + +**IMPORTANT Note** **-** In this way, since the table name remains the same, application queries and relevant database stored-procedures will be able to access the new MOT table seamlessly without code changes. Please note that MOT does not currently support cross-engine multi-table queries (such as by using Join, Union and sub-query) and cross-engine multi-table transactions. Therefore, if an original table is accessed somewhere in a multi-table query, stored procedure or transaction, you must either convert all related disk-tables into MOT tables or alter the relevant code in the application or the database. + +### Conversion Example + +Let's say that you have a database name **benchmarksql** and a table named **customer** (which is a disk-based table) to be migrated it into an MOT table. + +To migrate the customer table into an MOT table, perform the following - + +1. Check your source table column types. Verify that all types are supported by MOT, refer to section *Unsupported Data Types*. + + ```sql + benchmarksql-# \d+ customer + Table "public.customer" + Column | Type | Modifiers | Storage | Stats target | Description + --------+---------+-----------+---------+--------------+------------- + x | integer | | plain | | + y | integer | | plain | | + Has OIDs: no + Options: orientation=row, compression=no + ``` + +2. Check your source table data. + + ```sql + benchmarksql=# select * from customer; + x | y + ---+--- + 1 | 2 + 3 | 4 + (2 rows) + ``` + +3. Dump table data only by using **gs_dump**. + + ```sql + $ gs_dump -Fc benchmarksql -a --table customer -f customer.dump + gs_dump[port='15500'][benchmarksql][2020-06-04 16:45:38]: dump database benchmarksql successfully + gs_dump[port='15500'][benchmarksql][2020-06-04 16:45:38]: total time: 332 ms + ``` + +4. Rename the source table name. + + ```sql + benchmarksql=# alter table customer rename to customer_bk; + ALTER TABLE + ``` + +5. Create the MOT table to be exactly the same as the source table. + + ```sql + benchmarksql=# create foreign table customer (x int, y int); + CREATE FOREIGN TABLE + benchmarksql=# select * from customer; + x | y + ---+--- + (0 rows) + ``` + +6. Import the source dump data into the new MOT table. + + ```sql + $ gs_restore -C -d benchmarksql customer.dump + restore operation successful + total time: 24 ms + Check that the data was imported successfully. + benchmarksql=# select * from customer; + x | y + ---+--- + 1 | 2 + 3 | 4 + (2 rows) + + benchmarksql=# \d + List of relations + Schema | Name | Type | Owner | Storage + --------+-------------+---------------+--------+---------------------------------- + public | customer | foreign table | aharon | + public | customer_bk | table | aharon | {orientation=row,compression=no} + (2 rows) + ``` + +## Query Native Compilation + +An additional feature of MOT is the ability to prepare and parse *pre-compiled full queries* in a native format (using a PREPARE statement) before they are needed for execution. + +This native format can later be executed (using an EXECUTE command) more efficiently. This type of execution is much quicker because the native format bypasses multiple database processing layers during execution and thus enables better performance. + +This division of labor avoids repetitive parse analysis operations. In this way, queries and transaction statements are executed in an interactive manner. This feature is sometimes called *Just-In-Time (JIT)* query compilation. + +### Query Compilation - PREPARE Statement + +To use MOT’s native query compilation, call the PREPARE client statement before the query is executed. This instructs MOT to pre-compile the query and/or to pre-load previously pre-compiled code from a cache. + +The following is an example of PREPARE syntax in SQL - + +```sql +PREPARE name [ ( data_type [, ...] ) ] AS statement +``` + +PREPARE creates a prepared statement in the database server, which is a server-side object that can be used to optimize performance. + +### Execute Command + +When an EXECUTE command is subsequently issued, the prepared statement is parsed, analyzed, rewritten and executed. This division of labor avoids repetitive parse analysis operations, while enabling the execution plan to depend on specific provided setting values. + +The following is an example of how to invoke a PREPARE and then an EXECUTE statement in a Java application. + +```sql +conn = DriverManager.getConnection(connectionUrl, connectionUser, connectionPassword); + +// Example 1: PREPARE without bind settings +String query = "SELECT * FROM getusers"; +PreparedStatement prepStmt1 = conn.prepareStatement(query); +ResultSet rs1 = pstatement.executeQuery()) +while (rs1.next()) {…} + +// Example 2: PREPARE with bind settings +String sqlStmt = "SELECT * FROM employees where first_name=? and last_name like ?"; +PreparedStatement prepStmt2 = conn.prepareStatement(sqlStmt); +prepStmt2.setString(1, "Mark"); // first name "Mark" +prepStmt2.setString(2, "%n%"); // last name contains a letter "n" +ResultSet rs2 = prepStmt2.executeQuery()) +while (rs2.next()) {…} +``` + +The following describes the supported and unsupported features of MOT compilation. + +### Supported Queries for Lite Execution + +The following query types are suitable for lite execution - + +- Simple point queries - + - SELECT (including SELECT for UPDATE) + - UPDATE + - DELETE +- INSERT query +- Range UPDATE queries that refer to a full prefix of the primary key +- Range SELECT queries that refer to a full prefix of the primary key +- JOIN queries where one or both parts collapse to a point query +- JOIN queries that refer to a full prefix of the primary key in each joined table + +### Unsupported Queries for Lite Execution + +Any special query attribute disqualifies it for Lite Execution. In particular, if any of the following conditions apply, then the query is declared as unsuitable for Lite Execution. You may refer to the Unsupported Queries for Native Compilation and Lite Execution section for more information. + +It is important to emphasize that in case a query statement does not fit + +native compilation and lite execution, no error is reported to the client and the query will still be executed in a normal and standard manner. + +For more information about MOT native compilation capabilities, see either the section about Query Native Compilation or a more detailed information in the Query Native Compilation (JIT) section. + +## Retrying an Aborted Transaction + +In Optimistic Concurrency Control (OCC) (such as the one used by MOT) during a transaction (using any isolation level) no locks are placed on a record until the COMMIT phase. This is a powerful advantage that significantly increases performance. Its drawback is that an update may fail if another session attempts to update the same record. This results in an entire transaction that must be aborted. These so called *Update Conflicts* are detected by MOT at the commit time by a version checking mechanism. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** A similar abort happens on engines using pessimistic concurrency control, such as standard PG and the MogDB disk-based tables, when SERIALIZABLE or REPEATABLE-READ isolation level are used. + +Such update conflicts are quite rare in common OLTP scenarios and are especially rare in our experience with MOT. However, because there is still a chance that they may happen, developers should consider resolving this issue using transaction retry code. + +The following describes how to retry a table command after multiple sessions attempt to update the same table simultaneously. You may refer to the OCC vs 2PL Differences by Example section for more detailed information. The following example is taken from TPC-C payment transaction. + +```java +int commitAborts = 0; + +while (commitAborts < RETRY_LIMIT) { + + try { + stmt =db.stmtPaymentUpdateDistrict; + stmt.setDouble(1, 100); + stmt.setInt(2, 1); + stmt.setInt(3, 1); + stmt.executeUpdate(); + + db.commit(); + + break; + } + catch (SQLException se) { + if(se != null && se.getMessage().contains("could not serialize access due to concurrent update")) { + log.error("commmit abort = " + se.getMessage()); + commitAborts++; + continue; + }else { + db.rollback(); + } + + break; + } +} +``` + +## MOT External Support Tools + +The following external MogDB tools have been modified in order to support MOT. Make sure to use the most recent version of each. An overview describing MOT-related usage is provided below. For a full description of these tools and their usage, refer to the MogDB Tools Reference document. + +### gs_ctl (Full and Incremental) + +This tool is used to create a standby server from a primary server, as well as to synchronize a server with another copy of the same server after their timelines have diverged. + +At the end of the operation, the latest MOT checkpoint is fetched by the tool, taking into consideration the **checkpoint_dir** configuration setting value. + +The checkpoint is fetched from the source server's **checkpoint_dir** to the destination server's **checkpoint_dir**. + +Currently, MOT does not support an incremental checkpoint. Therefore, the gs_ctl incremental build does not work in an incremental manner for MOT, but rather in FULL mode. The Postgres (disk-tables) incremental build can still be done incrementally. + +### gs_basebackup + +gs_basebackup is used to prepare base backups of a running server, without affecting other database clients. + +The MOT checkpoint is fetched at the end of the operation as well. However, the checkpoint's location is taken from **checkpoint_dir** in the source server and is transferred to the data directory of the source in order to back it up correctly. + +### gs_dump + +gs_dump is used to export the database schema and data to a file. It also supports MOT tables. + +### gs_restore + +gs_restore is used to import the database schema and data from a file. It also supports MOT tables. + +## MOT SQL Coverage and Limitations + +MOT design enables almost complete coverage of SQL and future feature sets. For example, standard Postgres SQL is mostly supported, as well common database features, such as stored procedures and user defined functions. + +The following describes the various types of SQL coverages and limitations - + +### Unsupported Features + +The following features are not supported by MOT - + +- Engine Interop - No cross-engine (Disk+MOT) queries, views or transactions. Planned for 2021. +- MVCC, Isolation - No snapshot/serializable isolation. Planned for 2021. +- Native Compilation (JIT) - Limited SQL coverage. Also, JIT compilation of stored procedures is not supported. +- LOCAL memory is limited to 1 GB. A transaction can only change data of less than 1 GB. +- Capacity (Data+Index) is limited to available memory. Anti-caching + Data Tiering will be available in the future. +- No full-text search index. +- Do not support Logical copy. + +In addition, the following are detailed lists of various general limitations of MOT tables, MOT indexes, Query and DML syntax and the features and limitations of Query Native Compilation. + +### MOT Table Limitations + +The following lists the functionality limitations of MOT tables - + +- Partition by range +- AES encryption +- Stream operations +- User-defined types +- Sub-transactions +- DML triggers +- DDL triggers +- Collations other than "C" or "POSIX" + +### Unsupported Table DDLs + +- Alter table +- Create table, like including +- Create table as select +- Partition by range +- Create table with no-logging clause +- DEFERRABLE primary key +- Reindex +- Tablespace +- Create schema with subcommands + +### Unsupported Data Types + +- UUID +- User-Defined Type (UDF) +- Array data type +- NVARCHAR2(n) +- Clob +- Name +- Blob +- Raw +- Path +- Circle +- Reltime +- Bit varying(10) +- Tsvector +- Tsquery +- JSON +- Box +- Text +- Line +- Point +- LSEG +- POLYGON +- INET +- CIDR +- MACADDR +- Smalldatetime +- BYTEA +- Bit +- Varbit +- OID +- Money +- Any unlimited varchar/character varying +- HSTORE +- XML + +### UnsupportedIndex DDLs and Index + +- Create index on decimal/numeric + +- Create index on nullable columns + +- Create index, index per table > 9 + +- Create index on key size > 256 + + The key size includes the column size in bytes + a column additional size, which is an overhead required to maintain the index. The below table lists the column additional size for different column types. + + Additionally, in case of non-unique indexes an extra 8 bytes is required. + + Thus, the following pseudo code calculates the **key size**: + + ```java + keySize =0; + + for each (column in index){ + keySize += (columnSize + columnAddSize); + } + if (index is non_unique) { + keySize += 8; + } + ``` + + | Column Type | Column Size | Column Additional Size | + | :---------- | :---------- | :--------------------- | + | varchar | N | 4 | + | tinyint | 1 | 1 | + | smallint | 2 | 1 | + | int | 4 | 1 | + | bigint | 8 | 1 | + | float | 4 | 2 | + | float8 | 8 | 3 | + + Types that are not specified in above table, the column additional size is zero (for instance timestamp). + +### Unsupported DMLs + +- Merge into +- Select into +- Lock table +- Copy from table +- Upsert + +### Unsupported Queries for Native Compilation and Lite Execution + +- The query refers to more than two tables +- The query has any one of the following attributes - + - Aggregation on non-primitive types + - Window functions + - Sub-query sub-links + - Distinct-ON modifier (distinct clause is from DISTINCT ON) + - Recursive (WITH RECURSIVE was specified) + - Modifying CTE (has INSERT/UPDATE/DELETE in WITH) + +In addition, the following clauses disqualify a query from lite execution - + +- Returning list +- Group By clause +- Grouping sets +- Having clause +- Windows clause +- Distinct clause +- Sort clause that does not conform to native index order +- Set operations +- Constraint dependencies diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/5-mot-administration.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/5-mot-administration.md new file mode 100644 index 0000000000000000000000000000000000000000..5b2171fe07ada57ec189d6b63e0a420bb9c4c604 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/5-mot-administration.md @@ -0,0 +1,419 @@ +--- +title: MOT Administration +summary: MOT Administration +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Administration + +The following describes various MOT administration topics. + +## MOT Durability + +Durability refers to long-term data protection (also known as *disk persistence*). Durability means that stored data does not suffer from any kind of degradation or corruption, so that data is never lost or compromised. Durability ensures that data and the MOT engine are restored to a consistent state after a planned shutdown (for example, for maintenance) or an unplanned crash (for example, a power failure). + +Memory storage is volatile, meaning that it requires power to maintain the stored information. Disk storage, on the other hand, is non-volatile, meaning that it does not require power to maintain stored information, thus, it can survive a power shutdown. MOT uses both types of storage - it has all data in memory, while persisting transactional changes to disk **MOT Durability** and by maintaining frequent periodic **MOT Checkpoints** in order to ensure data recovery in case of shutdown. + +The user must ensure sufficient disk space for the logging and Checkpointing operations. A separated drive can be used for the Checkpoint to improve performance by reducing disk I/O load. + +You may refer to the **MOT Key Technologies** section for an overview of how durability is implemented in the MOT engine. + +**To configure durability -** + +To ensure strict consistency, configure the synchronous_commit parameter to **On** in the postgres.conf configuration file. + +**MOTs WAL Redo Log and Checkpoints enable durability, as described below -** + +### MOT Logging - WAL Redo Log + +To ensure Durability, MOT is fully integrated with the MogDB's Write-Ahead Logging (WAL) mechanism, so that MOT persists data in WAL records using MogDB's XLOG interface. This means that every addition, update, and deletion to an MOT table’s record is recorded as an entry in the WAL. This ensures that the most current data state can be regenerated and recovered from this non-volatile log. For example, if three new rows were added to a table, two were deleted and one was updated, then six entries would be recorded in the log. + +MOT log records are written to the same WAL as the other records of MogDB disk-based tables. + +MOT only logs an operation at the transaction commit phase. + +MOT only logs the updated delta record in order to minimize the amount of data written to disk. + +During recovery, data is loaded from the last known or a specific Checkpoint; and then the WAL Redo log is used to complete the data changes that occur from that point forward. + +The WAL (Redo Log) retains all the table row modifications until a Checkpoint is performed (as described above). The log can then be truncated in order to reduce recovery time and to save disk space. + +**Note** - In order to ensure that the log IO device does not become a bottleneck, the log file must be placed on a drive that has low latency. + +### MOT Logging Types + +Two synchronous transaction logging options and one asynchronous transaction logging option are supported (these are also supported by the standard MogDB disk engine). MOT also supports synchronous Group Commit logging with NUMA-awareness optimization, as described below. + +According to your configuration, one of the following types of logging is implemented - + +- **Synchronous Redo Logging** + + The **Synchronous Redo Logging** option is the simplest and most strict redo logger. When a transaction is committed by a client application, the transaction redo entries are recorded in the WAL (Redo Log), as follows - + + 1. While a transaction is in progress, it is stored in the MOT's memory. + 2. After a transaction finishes and the client application sends a Commit command, the transaction is locked and then written to the WAL Redo Log on the disk. This means that while the transaction log entries are being written to the log, the client application is still waiting for a response. + 3. As soon as the transaction's entire buffer is written to the log, the changes to the data in memory take place and then the transaction is committed. After the transaction has been committed, the client application is notified that the transaction is complete. + + **Summary** + + The **Synchronous Redo Logging** option is the safest and most strict because it ensures total synchronization of the client application and the WAL Redo log entries for each transaction as it is committed; thus ensuring total durability and consistency with absolutely no data loss. This logging option prevents the situation where a client application might mark a transaction as successful, when it has not yet been persisted to disk. + + The downside of the **Synchronous Redo Logging** option is that it is the slowest logging mechanism of the three options. This is because a client application must wait until all data is written to disk and because of the frequent disk writes (which typically slow down the database). + +- **Group Synchronous Redo Logging** + + The **Group Synchronous Redo Logging** option is very similar to the **Synchronous Redo Logging** option, because it also ensures total durability with absolutely no data loss and total synchronization of the client application and the WAL (Redo Log) entries. The difference is that the **Group Synchronous Redo Logging** option writes _groups of transaction_r edo entries to the WAL Redo Log on the disk at the same time, instead of writing each and every transaction as it is committed. Using Group Synchronous Redo Logging reduces the amount of disk I/Os and thus improves performance, especially when running a heavy workload. + + The MOT engine performs synchronous Group Commit logging with Non-Uniform Memory Access (NUMA)-awareness optimization by automatically grouping transactions according to the NUMA socket of the core on which the transaction is running. + + You may refer to the **NUMA Awareness Allocation and Affinity** section for more information about NUMA-aware memory access. + + When a transaction commits, a group of entries are recorded in the WAL Redo Log, as follows - + + 1. While a transaction is in progress, it is stored in the memory. The MOT engine groups transactions in buckets according to the NUMA socket of the core on which the transaction is running. This means that all the transactions running on the same socket are grouped together and that multiple groups will be filling in parallel according to the core on which the transaction is running. + + Writing transactions to the WAL is more efficient in this manner because all the buffers from the same socket are written to disk together. + + **Note** - Each thread runs on a single core/CPU which belongs to a single socket and each thread only writes to the socket of the core on which it is running. + + 2. After a transaction finishes and the client application sends a Commit command, the transaction redo log entries are serialized together with other transactions that belong to the same group. + + 3. After the configured criteria are fulfilled for a specific group of transactions (quantity of committed transactions or timeout period as describes in the **REDO LOG (MOT)** section), the transactions in this group are written to the WAL on the disk. This means that while these log entries are being written to the log, the client applications that issued the commit are waiting for a response. + + 4. As soon as all the transaction buffers in the NUMA-aware group have been written to the log, all the transactions in the group are performing the necessary changes to the memory store and the clients are notified that these transactions are complete. + + **Summary** + + The **Group Synchronous Redo Logging** option is a an extremely safe and strict logging option because it ensures total synchronization of the client application and the WAL Redo log entries; thus ensuring total durability and consistency with absolutely no data loss. This logging option prevents the situation where a client application might mark a transaction as successful, when it has not yet been persisted to disk. + + On one hand this option has fewer disk writes than the **Synchronous Redo Logging** option, which may mean that it is faster. The downside is that transactions are locked for longer, meaning that they are locked until after all the transactions in the same NUMA memory have been written to the WAL Redo Log on the disk. + + The benefits of using this option depend on the type of transactional workload. For example, this option benefits systems that have many transactions (and less so for systems that have few transactions, because there are few disk writes anyway). + +- **Asynchronous Redo Logging** + + The **Asynchronous Redo Logging** option is the fastest logging method, However, it does not ensure no data loss, meaning that some data that is still in the buffer and was not yet written to disk may get lost upon a power failure or database crash. When a transaction is committed by a client application, the transaction redo entries are recorded in internal buffers and written to disk at preconfigured intervals. The client application does not wait for the data being written to disk. It continues to the next transaction. This is what makes asynchronous redo logging the fastest logging method. + + When a transaction is committed by a client application, the transaction redo entries are recorded in the WAL Redo Log, as follows - + + 1. While a transaction is in progress, it is stored in the MOT's memory. + 2. After a transaction finishes and the client application sends a Commit command, the transaction redo entries are written to internal buffers, but are not yet written to disk. Then changes to the MOT data memory take place and the client application is notified that the transaction is committed. + 3. At a preconfigured interval, a redo log thread running in the background collects all the buffered redo log entries and writes them to disk. + + **Summary** + + The Asynchronous Redo Logging option is the fastest logging option because it does not require the client application to wait for data being written to disk. In addition, it groups many transactions redo entries and writes them together, thus reducing the amount of disk I/Os that slow down the MOT engine. + + The downside of the Asynchronous Redo Logging option is that it does not ensure that data will not get lost upon a crash or failure. Data that was committed, but was not yet written to disk, is not durable on commit and thus cannot be recovered in case of a failure. The Asynchronous Redo Logging option is most relevant for applications that are willing to sacrifice data recovery (consistency) over performance. + +### Configuring Logging + +Two synchronous transaction logging options and one asynchronous transaction logging option are supported by the standard MogDB disk engine. + +To configure logging - + +1. The determination of whether synchronous or asynchronous transaction logging is performed is configured in the synchronous_commit **(On = Synchronous)** parameters in the postgres.conf configuration file. + +If a synchronous mode of transaction logging has been selected (synchronous_commit = **On**, as described above), then the enable_group_commit parameter in the mot.conf configuration file determines whether the **Group Synchronous Redo Logging** option or the **Synchronous Redo Logging** option is used. For **Group Synchronous Redo Logging**, you must also define in the mot.conf file which of the following thresholds determine when a group of transactions is recorded in the WAL + +- group_commit_size **-** The quantity of committed transactions in a group. For example, **16** means that when 16 transactions in the same group have been committed by a client application, then an entry is written to disk in the WAL Redo Log for all 16 transactions. + +- group_commit_timeout - A timeout period in ms. For example, **10** means that after 10 ms, an entry is written to disk in the WAL Redo Log for each of the transactions in the same group that have been committed by their client application in the last 10 ms. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** You may refer to the **REDO LOG (MOT)** for more information about configuration settings. + +### MOT Checkpoints + +A Checkpoint is the point in time at which all the data of a table's rows is saved in files on persistent storage in order to create a full durable database image. It is a snapshot of the data at a specific point in time. + +A Checkpoint is required in order to reduce a database's recovery time by shortening the quantity of WAL (Redo Log) entries that must be replayed in order to ensure durability. Checkpoint's also reduce the storage space required to keep all the log entries. + +If there were no Checkpoints, then in order to recover a database, all the WAL redo entries would have to be replayed from the beginning of time, which could take days/weeks depending on the quantity of records in the database. Checkpoints record the current state of the database and enable old redo entries to be discarded. + +Checkpoints are essential during recovery scenarios (especially for a cold start). First, the data is loaded from the last known or a specific Checkpoint; and then the WAL is used to complete the data changes that occurred since then. + +For example - If the same table row is modified 100 times, then 100 entries are recorded in the log. When Checkpoints are used, then even if a specific table row was modified 100 times, it is recorded in the Checkpoint a single time. After the recording of a Checkpoint, recovery can be performed on the basis of that Checkpoint and only the WAL Redo Log entries that occurred since the Checkpoint need be played. + +## MOT Recovery + +The main objective of MOT Recovery is to restore the data and the MOT engine to a consistent state after a planned shutdown (for example, for maintenance) or an unplanned crash (for example, after a power failure). + +MOT recovery is performed automatically with the recovery of the rest of the MogDB database and is fully integrated into MogDB recovery process (also called a *Cold Start*). + +MOT recovery consists of two stages - + +**Checkpoint Recovery -** First, data must be recovered from the latest Checkpoint file on disk by loading it into memory rows and creating indexes. + +**WAL Redo Log Recovery -** Afterwards, the recent data (which was not captured in the Checkpoint) must be recovered from the WAL Redo Log by replaying records that were added to the log since the Checkpoint that was used in the Checkpoint Recovery (described above). + +The WAL Redo Log recovery is managed and triggered by MogDB. + +- To configure recovery. + +- While WAL recovery is performed in a serial manner, the Checkpoint recovery can be configured to run in a multi-threaded manner (meaning in parallel by multiple workers). + +- Configure the **Checkpoint_recovery_workers** parameter in the **mot.conf** file, which is described in the **RECOVERY (MOT)** section. + +## MOT Replication and High Availability + +Since MOT is integrated into MogDB and uses/supports its replication and high availability, both synchronous and asynchronous replication are supported out of the box. + +The MogDB gs_ctl tool is used for availability control and to operate the cluster. This includes gs_ctl switchover, gs_ctl failover, gs_ctl build and so on. + +You may refer to the MogDB Tools Reference document for more information. + +- To configure replication and high availability. +- Refer to the relevant MogDB documentation. + +## MOT Memory Management + +For planning and finetuning, see the **MOT Memory and Storage Planning** and **MOT Configuration Settings** sections. + +## MOT Vacuum + +Use VACUUM for garbage collection and optionally to analyze a database, , as follows - + +- [PG] + + In Postgress (PG), the VACUUM reclaims storage occupied by dead tuples. In normal PG operation, tuples that are deleted or that are made obsolete by an update are not physically removed from their table. They remain present until a VACUUM is done. Therefore, it is necessary to perform a VACUUM periodically, especially on frequently updated tables. + +- [MOT Extension] + + MOT tables do not need a periodic VACUUM operation, since dead/empty tuples are re-used by new ones. MOT tables require VACUUM operations only when their size is significantly reduced and they do not expect to grow to their original size in the near future. + + For example, an application that periodically (for example, once in a week) performs a large deletion of a table/tables data while inserting new data takes days and does not necessarily require the same quantity of rows. In such cases, it makes sense to activate the VACUUM. + + The VACUUM operation on MOT tables is always transformed into a VACUUM FULL with an exclusive table lock. + +- Supported Syntax and Limitations + + Activation of the VACUUM operation is performed in a standard manner. + + ```sql + VACUUM [FULL | ANALYZE] [ table ]; + ``` + + Only the FULL and ANALYZE VACUUM options are supported. The VACUUM operation can only be performed on an entire MOT table. + + The following PG vacuum options are not supported: + + - FREEZE + - VERBOSE + - Column specification + - LAZY mode (partial table scan) + + Additionally, the following functionality is not supported + + - AUTOVACUUM + +## MOT Statistics + +Statistics are intended for performance analysis or debugging. It is uncommon to turn them ON in a production environment (by default, they are OFF). Statistics are primarily used by database developers and to a lesser degree by database users. + +There is some impact on performance, particularly on the server. Impact on the user is negligible. + +The statistics are saved in the database server log. The log is located in the data folder and named **postgresql-DATE-TIME.log**. + +Refer to **STATISTICS (MOT)** for detailed configuration options. + +## MOT Monitoring + +All syntax for monitoring of PG-based FDW tables is supported. This includes Table or Index sizes (as described below). In addition, special functions exist for monitoring MOT memory consumption, including MOT Global Memory, MOT Local Memory and a single client session. + +### Table and Index Sizes + +The size of tables and indexes can be monitored by querying pg_relation_size. + +For example + +**Data Size** + +```sql +select pg_relation_size('customer'); +``` + +**Index** + +```sql +select pg_relation_size('customer_pkey'); +``` + +### MOT GLOBAL Memory Details + +Check the size of MOT global memory, which includes primarily the data and indexes. + +```sql +select * from mot_global_memory_detail(); +``` + +Result - + +```sql +numa_node | reserved_size | used_size +----------------+----------------+------------- +-1 | 194716368896 | 25908215808 +0 | 446693376 | 446693376 +1 | 452984832 | 452984832 +2 | 452984832 | 452984832 +3 | 452984832 | 452984832 +4 | 452984832 | 452984832 +5 | 364904448 | 364904448 +6 | 301989888 | 301989888 +7 | 301989888 | 301989888 +``` + +Where - + +- -1 is the total memory. +- 0..7 are NUMA memory nodes. + +### MOT LOCAL Memory Details + +Check the size of MOT local memory, which includes session memory. + +```sql +select * from mot_local_memory_detail(); +``` + +Result - + +```sql +numa_node | reserved_size | used_size +----------------+----------------+------------- +-1 | 144703488 | 144703488 +0 | 25165824 | 25165824 +1 | 25165824 | 25165824 +2 | 18874368 | 18874368 +3 | 18874368 | 18874368 +4 | 18874368 | 18874368 +5 | 12582912 | 12582912 +6 | 12582912 | 12582912 +7 | 12582912 | 12582912 +``` + +Where - + +- -1 is the total memory. +- 0..7 are NUMA memory nodes. + +### Session Memory + +Memory for session management is taken from the MOT local memory. + +Memory usage by all active sessions (connections) is possible using the following query - + +```sql +select * from mot_session_memory_detail(); +``` + +Result - + +```sql +sessid | total_size | free_size | used_size +----------------------------------------+-----------+----------+---------- +1591175063.139755603855104 | 6291456 | 1800704 | 4490752 + +``` + +Legend - + +- **total_size -** is allocated for the session +- **free_size -** not in use +- **used_size -** In actual use + +The following query enables a DBA to determine the state of local memory used by the current session - + +```sql +select * from mot_session_memory_detail() + where sessid = pg_current_sessionid(); +``` + +Result - + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-administration-1.png) + +## MOT Error Messages + +Errors may be caused by a variety of scenarios. All errors are logged in the database server log file. In addition, user-related errors are returned to the user as part of the response to the query, transaction or stored procedure execution or to database administration action. + +- Errors reported in the Server log include - Function, Entity, Context, Error message, Error description and Severity. +- Errors reported to users are translated into standard PostgreSQL error codes and may consist of an MOT-specific message and description. + +The following lists the error messages, error descriptions and error codes. The error code is actually an internal code and not logged or returned to users. + +### Errors Written the Log File + +All errors are logged in the database server log file. The following lists the errors that are written to the database server log file and are **not** returned to the user. The log is located in the data folder and named **postgresql-DATE-TIME.log**. + +**Table 1** Errors Written Only to the Log File + +| Message in the Log | Error Internal Code | +| :---------------------------------- | :------------------------------- | +| Error code denoting success | MOT_NO_ERROR 0 | +| Out of memory | MOT_ERROR_OOM 1 | +| Invalid configuration | MOT_ERROR_INVALID_CFG 2 | +| Invalid argument passed to function | MOT_ERROR_INVALID_ARG 3 | +| System call failed | MOT_ERROR_SYSTEM_FAILURE 4 | +| Resource limit reached | MOT_ERROR_RESOURCE_LIMIT 5 | +| Internal logic error | MOT_ERROR_INTERNAL 6 | +| Resource unavailable | MOT_ERROR_RESOURCE_UNAVAILABLE 7 | +| Unique violation | MOT_ERROR_UNIQUE_VIOLATION 8 | +| Invalid memory allocation size | MOT_ERROR_INVALID_MEMORY_SIZE 9 | +| Index out of range | MOT_ERROR_INDEX_OUT_OF_RANGE 10 | +| Error code unknown | MOT_ERROR_INVALID_STATE 11 | + +### Errors Returned to the User + +The following lists the errors that are written to the database server log file and are returned to the user. + +MOT returns PG standard error codes to the envelope using a Return Code (RC). Some RCs cause the generation of an error message to the user who is interacting with the database. + +The PG code (described below) is returned internally by MOT to the database envelope, which reacts to it according to standard PG behavior. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** %s, %u and %lu in the message are replaced by relevant error information, such as query, table name or another information. - %s - String - %u - Number - %lu - Number + +**Table 2** Errors Returned to the User and Logged to the Log File + +| Short and Long Description Returned to the User | PG Code | Internal Error Code | +| :---------------------------------------------------- | :------------------------------ | :------------------------------ | +| Success.Denotes success | ERRCODE_SUCCESSFUL_COMPLETIONCOMPLETION | RC_OK = 0 | +| FailureUnknown error has occurred. | ERRCODE_FDW_ERROR | RC_ERROR = 1 | +| Unknown error has occurred.Denotes aborted operation. | ERRCODE_FDW_ERROR | RC_ABORT | +| Column definition of %s is not supported.Column type %s is not supported yet. | ERRCODE_INVALID_COLUMN_DEFINITION | RC_UNSUPPORTED_COL_TYPE | +| Column definition of %s is not supported.Column type Array of %s is not supported yet. | ERRCODE_INVALID_COLUMN_DEFINITION | RC_UNSUPPORTED_COL_TYPE_ARR | +| Column size %d exceeds max tuple size %u.Column definition of %s is not supported. | ERRCODE_FEATURE_NOT_SUPPORTED | RC_EXCEEDS_MAX_ROW_SIZE | +| Column name %s exceeds max name size %u.Column definition of %s is not supported. | ERRCODE_INVALID_COLUMN_DEFINITION | RC_COL_NAME_EXCEEDS_MAX_SIZE | +| Column size %d exceeds max size %u.Column definition of %s is not supported. | ERRCODE_INVALID_COLUMN_DEFINITION | RC_COL_SIZE_INVLALID | +| Cannot create table.Cannot add column %s; as the number of declared columns exceeds the maximum declared columns. | ERRCODE_FEATURE_NOT_SUPPORTED | RC_TABLE_EXCEEDS_MAX_DECLARED_COLS | +| Cannot create index.Total column size is greater than maximum index size %u. | ERRCODE_FDW_KEY_SIZE_EXCEEDS_MAX_ALLOWED | RC_INDEX_EXCEEDS_MAX_SIZE | +| Cannot create index.Total number of indexes for table %s is greater than the maximum number of indexes allowed %u. | ERRCODE_FDW_TOO_MANY_INDEXES | RC_TABLE_EXCEEDS_MAX_INDEXES | +| Cannot execute statement.Maximum number of DDLs per transaction reached the maximum %u. | ERRCODE_FDW_TOO_MANY_DDL_CHANGES_IN_TRANSACTION_NOT_ALLOWED | RC_TXN_EXCEEDS_MAX_DDLS | +| Unique constraint violationDuplicate key value violates unique constraint \"%s\"".Key %s already exists. | ERRCODE_UNIQUE_VIOLATION | RC_UNIQUE_VIOLATION | +| Table \"%s\" does not exist. | ERRCODE_UNDEFINED_TABLE | RC_TABLE_NOT_FOUND | +| Index \"%s\" does not exist. | ERRCODE_UNDEFINED_TABLE | RC_INDEX_NOT_FOUND | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_LOCAL_ROW_FOUND | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_LOCAL_ROW_NOT_FOUND | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_LOCAL_ROW_DELETED | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_INSERT_ON_EXIST | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_INDEX_RETRY_INSERT | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_INDEX_DELETE | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_LOCAL_ROW_NOT_VISIBLE | +| Memory is temporarily unavailable. | ERRCODE_OUT_OF_LOGICAL_MEMORY | RC_MEMORY_ALLOCATION_ERROR | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_ILLEGAL_ROW_STATE | +| Null constraint violated.NULL value cannot be inserted into non-null column %s at table %s. | ERRCODE_FDW_ERROR | RC_NULL_VIOLATION | +| Critical error.Critical error: %s. | ERRCODE_FDW_ERROR | RC_PANIC | +| A checkpoint is in progress - cannot truncate table. | ERRCODE_FDW_OPERATION_NOT_SUPPORTED | RC_NA | +| Unknown error has occurred. | ERRCODE_FDW_ERROR | RC_MAX_VALUE | +| <recovery message> | - | ERRCODE_CONFIG_FILE_ERROR | +| <recovery message> | - | ERRCODE_INVALID_TABLE_DEFINITION | +| Memory engine - Failed to perform commit prepared. | - | ERRCODE_INVALID_TRANSACTION_STATE | +| Invalid option <option name> | - | ERRCODE_FDW_INVALID_OPTION_NAME | +| Invalid memory allocation request size. | - | ERRCODE_INVALID_PARAMETER_VALUE | +| Memory is temporarily unavailable. | - | ERRCODE_OUT_OF_LOGICAL_MEMORY | +| Could not serialize access due to concurrent update. | - | ERRCODE_T_R_SERIALIZATION_FAILURE | +| Alter table operation is not supported for memory table.Cannot create MOT tables while incremental checkpoint is enabled.Re-index is not supported for memory tables. | - | ERRCODE_FDW_OPERATION_NOT_SUPPORTED | +| Allocation of table metadata failed. | - | ERRCODE_OUT_OF_MEMORY | +| Database with OID %u does not exist. | - | ERRCODE_UNDEFINED_DATABASE | +| Value exceeds maximum precision: %d. | - | ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE | +| You have reached a maximum logical capacity %lu of allowed %lu. | - | ERRCODE_OUT_OF_LOGICAL_MEMORY | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/6-mot-sample-tpcc-benchmark.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/6-mot-sample-tpcc-benchmark.md new file mode 100644 index 0000000000000000000000000000000000000000..acee5fc193b6b7910debc21e0e70ad3ea39a2ebe --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/2-using-mot/6-mot-sample-tpcc-benchmark.md @@ -0,0 +1,116 @@ +--- +title: MOT Sample TPC-C Benchmark +summary: MOT Sample TPC-C Benchmark +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Sample TPC-C Benchmark + +## TPC-C Introduction + +The TPC-C Benchmark is an industry standard benchmark for measuring the performance of Online Transaction Processing (OLTP) systems. It is based on a complex database and a number of different transaction types that are executed on it. TPC-C is both a hardware-independent and a software-independent benchmark and can thus be run on every test platform. An official overview of the benchmark model can be found at the tpc.org website here - . + +The database consists of nine tables of various structures and thus also nine types of data records. The size and quantity of the data records varies per table. A mix of five concurrent transactions of varying types and complexities is executed on the database, which are largely online or in part queued for deferred batch processing. Because these tables compete for limited system resources, many system components are stressed and data changes are executed in a variety of ways. + +**Table 1** TPC-C Database Structure + +| Table | Number of Entries | +| :--------- | :--------------------------------------- | +| Warehouse | n | +| Item | 100,000 | +| Stock | n x 100,000 | +| District | n x 10 | +| Customer | 3,000 per district, 30,000 per warehouse | +| Order | Number of customers (initial value) | +| New order | 30% of the orders (initial value) | +| Order line | ~ 10 per order | +| History | Number of customers (initial value) | + +The transaction mix represents the complete business processing of an order - from its entry through to its delivery. More specifically, the provided mix is designed to produce an equal number of new-order transactions and payment transactions and to produce a single delivery transaction, a single order-status transaction and a single stock-level transaction for every ten new-order transactions. + +**Table 2** TPC-C Transactions Ratio + +| Transaction Level ≥ 4% | Share of All Transactions | +| :--------------------- | :------------------------ | +| TPC-C New order | ≤ 45% | +| Payment | ≥ 43% | +| Order status | ≥ 4% | +| Delivery | ≥ 4% (batch) | +| Stock level | ≥ 4% | + +There are two ways to execute the transactions - **as stored procedures** (which allow higher throughput) and in **standard interactive SQL mode**. + +**Performance Metric - tpm-C** + +The tpm-C metric is the number of new-order transactions executed per minute. Given the required mix and a wide range of complexity and types among the transactions, this metric most closely simulates a comprehensive business activity, not just one or two transactions or computer operations. For this reason, the tpm-C metric is considered to be a measure of business throughput. + +The tpm-C unit of measure is expressed as transactions-per-minute-C, whereas "C" stands for TPC-C specific benchmark. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The official TPC-C Benchmark specification can be accessed at - . Some of the rules of this specification are generally not fulfilled in the industry, because they are too strict for industry reality. For example, Scaling rules - (a) tpm-C / Warehouse must be >9 and <12.86 (implying that a very high warehouses rate is required in order to achieve a high tpm-C rate, which also means that an extremely large database and memory capacity are required); and (b) 10x terminals x Warehouses (implying a huge quantity of simulated clients). + +## System-Level Optimization + +Follow the instructions in the **MOT Server Optimization - x86** section. The following section describes the key system-level optimizations for deploying the MogDB database on a Huawei Taishan server and on a Euler 2.8 operating system for ultimate performance. + +## BenchmarkSQL - An Open-Source TPC-C Tool + +For example, to test TPCC, the **BenchmarkSQL** can be used, as follows - + +- Download **benchmarksql** from the following link - +- The schema creation scripts in the **benchmarksql** tool need to be adjusted to MOT syntax and unsupported DDLs need to be avoided. The adjusted scripts can be directly downloaded from the following link - . The contents of this tar file includes sql.common.mogdb.mot folder and jTPCCTData.java file as well as a sample configuration file postgresql.conf and a TPCC properties file props.mot for reference. +- Place the sql.common.mogdb.mot folder in the same level as sql.common under run folder and replace the file src/client/jTPCCTData.java with the downloaded java file. +- Edit the file runDatabaseBuild.sh under run folder to remove **extraHistID** from **AFTER_LOAD** list to avoid unsupported alter table DDL. +- Replace the JDBC driver under lib/postgres folder with the MogDB JDBC driver available from the following link - . + +The only change done in the downloaded java file (compared to the original one) was to comment the error log printing for serialization and duplicate key errors. These errors are normal in case of MOT, since it uses Optimistic Concurrency Control (OCC) mechanism. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The benchmark test is executed using a standard interactive SQL mode without stored procedures. + +## Running the Benchmark + +Anyone can run the benchmark by starting up the server and running the **benchmarksql** scripts. + +To run the benchmark - + +1. Go to the **benchmarksql** run folder and rename sql.common to sql.common.orig. +2. Create a link sql.common to sql.common.mogdb.mot in order to test MOT. +3. Start up the database server. +4. Configure the props.pg file in the client. +5. Run the benchmark. + +## Results Report + +- Results in CLI + + BenchmarkSQL results should appear as follows - + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-sample-tpcc-benchmark-1.jpg) + + Over time, the benchmark measures and averages the committed transactions. The example above benchmarks for two minutes. + + The score is **2.71M tmp-C** (new-orders per-minute), which is 45% of the total committed transactions, meaning the **tpmTOTAL**. + +- Detailed Result Report + + The following is an example of a detailed result report - + + **Figure 1** Detailed Result Report + + ![detailed-result-report](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-sample-tpcc-benchmark-2.png) + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-sample-tpcc-benchmark-3.png) + + BenchmarkSQL collects detailed performance statistics and operating system performance data (if configured). + + This information can show the latency of the queries, and thus expose bottlenecks related to storage/network/CPU. + +- Results of TPC-C of MOT on Huawei Taishan 2480 + + Our TPC-C benchmark dated 01-May-2020 with an MogDB database installed on Taishan 2480 server (a 4-socket ARM/Kunpeng server), achieved a throughput of 4.79M tpm-C. + + A near linear scalability was demonstrated, as shown below - + + **Figure 2** Results of TPC-C of MOT on Huawei Taishan 2480 + + ![results-of-tpc-c-of-mot-on-huawei-taishan-2480](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-sample-tpcc-benchmark-4.png) diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-1.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-1.md new file mode 100644 index 0000000000000000000000000000000000000000..3b0a39a439eda8d332f6527826490c6da449b2f3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-1.md @@ -0,0 +1,90 @@ +--- +title: MOT Scale-up Architecture +summary: MOT Scale-up Architecture +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Scale-up Architecture + +To **scale up** means to add additional cores to the *same machine* in order to add computing power. To scale up refers to the most common traditional form of adding computing power in a machine that has a single pair of controllers and multiple cores. Scale-up architecture is limited by the scalability limits of a machine’s controller. + +## Technical Requirements + +MOT has been designed to achieve the following - + +- **Linear Scale-up -** MOT delivers a transactional storage engine that utilizes all the cores of a single NUMA architecture server in order to provide near-linear scale-up performance. This means that MOT is targeted to achieve a direct, near-linear relationship between the quantity of cores in a machine and the multiples of performance increase. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The near-linear scale-up results achieved by MOT significantly outperform all other existing solutions, and come as close as possible to achieving optimal results, which are limited by the physical restrictions and limitations of hardware, such as wires. + +- **No Maximum Number of Cores Limitation -** MOT does not place any limits on the maximum quantity of cores. This means that MOT is scalable from a single core up to 1,000s of cores, with minimal degradation per additional core, even when crossing NUMA socket boundaries. + +- **Extremely High Transactional Throughout -** MOT delivers a transactional storage engine that can achieve extremely high transactional throughout compared with any other OLTP vendor on the market. + +- **Extremely Low Transactional Latency -** MOT delivers a transactional storage engine that can reach extremely low transactional latency compared with any other OLTP vendor on the market. + +- **Seamless Integration and Leveraging with/of MogDB -** MOT integrates its transactional engine in a standard and seamless manner with the MogDB product. In this way, MOT reuses maximum functionality from the MogDB layers that are situated on top of its transactional storage engine. + +## Design Principles + +To achieve the requirements described above (especially in an environment with many-cores), our storage engine's architecture implements the following techniques and strategies - + +- **Data and indexes only reside in memory**. +- **Data and indexes are not laid out with physical partitions** (because these might achieve lower performance for certain types of applications). +- Transaction concurrency control is based on **Optimistic Concurrency Control (OCC)** without any centralized contention points. See the **MOT Concurrency Control Mechanism** section for more information about OCC. +- **Parallel Redo Logs (ultimately per core)** are used to efficiently avoid a central locking point. +- **Indexes are lock-free**. See the **MOT Indexes** section for more information about lock-free indexes. +- **NUMA-awareness memory allocation** is used to avoid cross-socket access, especially for session lifecycle objects. See the **NUMA Awareness Allocation and Affinity** section for more information about NUMA-awareness. +- **A customized MOT memory management allocator** with pre-cached object pools is used to avoid expensive runtime allocation and extra points of contention. This dedicated MOT memory allocator makes memory allocation more efficient by pre-accessing relatively large chunks of memory from the operation system as needed and then divvying it out to the MOT as needed. + +## Integration using Foreign Data Wrappers (FDW) + +MOT complies with and leverages MogDB's standard extensibility mechanism - Foreign Data Wrapper (FDW), as shown in the following diagram. + +The PostgreSQL Foreign Data Wrapper (FDW) feature enables the creation of foreign tables in an MOT database that are proxies for some other data source, such as Oracle, MySQL, PostgreSQL and so on. When a query is made on a foreign table, the FDW queries the external data source and returns the results, as if they were coming from a table in your database. + +MogDB relies on the PostgreSQL Foreign Data Wrappers (FDW) and Index support so that SQL is entirely covered, including stored procedures, user defined functions, system functions calls. + +**Figure 1** MOT Architecture + +![mot-architecture](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-scale-up-architecture-2.png) + +In the diagram above, the MOT engine is represented in green, while the existing MogDB (based on Postgres) components are represented in the top part of this diagram in blue. As you can see, the Foreign Data Wrapper (FDW) mediates between the MOT engine and the MogDB components. + +**MOT-Related FDW Customizations** + +Integrating MOT through FDW enables the reuse of the most upper layer MogDB functionality and therefore significantly shortened MOT's time-to-market without compromising SQL coverage. + +However, the original FDW mechanism in MogDB was not designed for storage engine extensions, and therefore lacks the following essential functionalities - + +- Index awareness of foreign tables to be calculated in the query planning phase +- Complete DDL interfaces +- Complete transaction lifecycle interfaces +- Checkpoint interfaces +- Redo Log interface +- Recovery interfaces +- Vacuum interfaces + +In order to support all the missing functionalities, the SQL layer and FDW interface layer were extended to provide the necessary infrastructure in order to enable the plugging in of the MOT transactional storage engine. + +## Result - Linear Scale-up + +The following shows the results achieved by the MOT design principles and implementation described above. + +To the best of our knowledge, MOT outperforms all existing industry-grade OLTP databases in transactional throughput of ACID-compliant workloads. + +MogDB and MOT have been tested on the following many-core systems with excellent performance scalability results. The tests were performed both on x86 Intel-based and ARM/Kunpeng-based many-core servers. You may refer to the **MOT Performance Benchmarks** section for more detailed performance review. + +Our TPC-C benchmark dated June 2020 tested an MogDB MOT database on a Taishan 2480 server. A 4-socket ARM/Kunpeng server, achieved throughput of 4.8 M tpmC. The following graph shows the near-linear nature of the results, meaning that it shows a significant increase in performance correlating to the increase of the quantity of cores - + +**Figure 2** TPC-C on ARM (256 Cores) + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-12.png) + +The following is an additional example that shows a test on an x86-based server also showing CPU utilization. + +**Figure 3** tpmC vs CPU Usage + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-performance-benchmarks-18.png) + +The chart shows that MOT demonstrates a significant performance increase correlation with an increase of the quantity of cores. MOT consumes more and more of the CPU correlating to the increase of the quantity of cores. Other industry solutions do not increase and sometimes show slightly degraded performance, which is a well-known problem in the database industry that affects customers’ CAPEX and OPEX expenses and operational efficiency. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-2.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-2.md new file mode 100644 index 0000000000000000000000000000000000000000..aab31c2a03baeef5b1ebf3c8cc0b8fbbe8e77365 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-2.md @@ -0,0 +1,179 @@ +--- +title: MOT Concurrency Control Mechanism +summary: MOT Concurrency Control Mechanism +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Concurrency Control Mechanism + +After investing extensive research to find the best concurrency control mechanism, we concluded that SILO based on OCC is the best ACID-compliant OCC algorithm for MOT. SILO provides the best foundation for MOT's challenging requirements. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** MOT is fully Atomicity, Consistency, Isolation, Durability (ACID)-compliant, as described in the **MOT Introduction** section. + +The following topics describe MOT's concurrency control mechanism - + +## MOT Local and Global Memory + +SILO manages both a local memory and a global memory, as shown in Figure 1. + +- **Global** memory is long-term shared memory is shared by all cores and is used primarily to store all the table data and indexes +- **Local** memory is short-term memory that is used primarily by sessions for handling transactions and store data changes in a primate to transaction memory until the commit phase. + +When a transaction change is required, SILO handles the copying of all that transaction's data from the global memory into the local memory. Minimal locks are placed on the global memory according to the OCC approach, so that the contention time in the global shared memory is extremely minimal. After the transaction’ change has been completed, this data is pushed back from the local memory to the global memory. + +The basic interactive transactional flow with our SILO-enhanced concurrency control is shown in the figure below - + +**Figure 1** Private (Local) Memory (for each transaction) and a Global Memory (for all the transactions of all the cores) + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-concurrency-control-mechanism-2.png) + +For more details, refer to the Industrial-Strength OLTP Using Main Memory and Many-cores document [**Comparison - Disk vs. MOT**]. + +## MOT SILO Enhancements + +SILO in its basic algorithm flow outperformed many other ACID-compliant OCCs that we tested in our research experiments. However, in order to make it a product-grade mechanism, we had to enhance it with many essential functionalities that were missing in the original design, such as - + +- Added support for interactive mode transactions, where transactions are running SQL by SQL from the client side and not as a single step on the server side +- Added optimistic inserts +- Added support for non-unique indexes +- Added support for read-after-write in transactions so that users can see their own changes before they are committed +- Added support for lockless cooperative garbage collection +- Added support for lockless checkpoints +- Added support for fast recovery +- Added support for two-phase commit in a distributed deployment + +Adding these enhancements without breaking the scalable characteristic of the original SILO was very challenging. + +## MOT Isolation Levels + +Even though MOT is fully ACID-compliant (as described in the section), not all isolation levels are supported in MogDB 2.1. The following table describes all isolation levels, as well as what is and what is not supported by MOT. + +**Table 1** Isolation Levels + +| Isolation Level | Description | +| :--------------- | :----------------------------------------------------------- | +| READ UNCOMMITTED | **Not supported by MOT.** | +| READ COMMITTED | **Supported by MOT.**
The READ COMMITTED isolation level that guarantees that any data that is read was already committed when it was read. It simply restricts the reader from seeing any intermediate, uncommitted or dirty reads. Data is free to be changed after it has been read so that READ COMMITTED does not guarantee that if the transaction re-issues the read, that the same data will be found. | +| SNAPSHOT | **Not supported by MOT.**
The SNAPSHOT isolation level makes the same guarantees as SERIALIZABLE, except that concurrent transactions can modify the data. Instead, it forces every reader to see its own version of the world (its own snapshot). This makes it very easy to program, plus it is very scalable, because it does not block concurrent updates. However, in many implementations this isolation level requires higher server resources. | +| REPEATABLE READ | **Supported by MOT.**
REPEATABLE READ is a higher isolation level that (in addition to the guarantees of the READ COMMITTED isolation level) guarantees that any data that is read cannot change. If a transaction reads the same data again, it will find the same previously read data in place, unchanged and available to be read.
Because of the optimistic model, concurrent transactions are not prevented from updating rows read by this transaction. Instead, at commit time this transaction validates that the REPEATABLE READ isolation level has not been violated. If it has, this transaction is rolled back and must be retried. | +| SERIALIZABLE | **Not supported by MOT**.
Serializable isolation makes an even stronger guarantee. In addition to everything that the REPEATABLE READ isolation level guarantees, it also guarantees that no new data can be seen by a subsequent read.
It is named SERIALIZABLE because the isolation is so strict that it is almost a bit like having the transactions run in series rather than concurrently. | + +The following table shows the concurrency side effects enabled by the different isolation levels. + +**Table 2** Concurrency Side Effects Enabled by Isolation Levels + +| Isolation Level | Description | Non-repeatable Read | Phantom | +| :--------------- | :---------- | :------------------ | :------ | +| READ UNCOMMITTED | Yes | Yes | Yes | +| READ COMMITTED | No | Yes | Yes | +| REPEATABLE READ | No | No | Yes | +| SNAPSHOT | No | No | No | +| SERIALIZABLE | No | No | No | + +In the near future release, MogDB MOT will also support both SNAPSHOT and SERIALIZABLE isolation levels. + +## MOT Optimistic Concurrency Control + +The Concurrency Control Module (CC Module for short) provides all the transactional requirements for the Main Memory Engine. The primary objective of the CC Module is to provide the Main Memory Engine with support for various isolation levels. + +### Optimistic OCC vs. Pessimistic 2PL + +The functional differences of Pessimistic 2PL (2-Phase Locking) vs. Optimistic Concurrency Control (OCC) involve pessimistic versus optimistic approaches to transaction integrity. + +Disk-based tables use a pessimistic approach, which is the most commonly used database method. The MOT Engine use an optimistic approach. + +The primary functional difference between the pessimistic approach and the optimistic approach is that if a conflict occurs - + +- The pessimistic approach causes the client to wait. +- The optimistic approach causes one of the transactions to fail, so that the failed transaction must be retried by the client. + +**Optimistic Concurrency Control Approach (Used by MOT)** + +The **Optimistic Concurrency Control (OCC)** approach detects conflicts as they occur, and performs validation checks at commit time. + +The optimistic approach has less overhead and is usually more efficient, partly because transaction conflicts are uncommon in most applications. + +The functional differences between optimistic and pessimistic approaches is larger when the REPEATABLE READ isolation level is enforced and is largest for the SERIALIZABLE isolation level. + +**Pessimistic Approaches (Not used by MOT)** + +The **Pessimistic Concurrency Control** (2PL or 2-Phase Locking) approach uses locks to block potential conflicts before they occur. A lock is applied when a statement is executed and released when the transaction is committed. Disk-based row-stores use this approach (with the addition of Multi-version Concurrency Control [MVCC]). + +In 2PL algorithms, while a transaction is writing a row, no other transaction can access it; and while a row is being read, no other transaction can overwrite it. Each row is locked at access time for both reading and writing; and the lock is released at commit time. These algorithms require a scheme for handling and avoiding deadlock. Deadlock can be detected by calculating cycles in a wait-for graph. Deadlock can be avoided by keeping time ordering using TSO or by some kind of back-off scheme. + +**Encounter Time Locking (ETL)** + +Another approach is Encounter Time Locking (ETL), where reads are handled in an optimistic manner, but writes lock the data that they access. As a result, writes from different ETL transactions are aware of each other and can decide to abort. It has been empirically verified that ETL improves the performance of OCC in two ways - + +- First, ETL detects conflicts early on and often increases transaction throughput. This is because transactions do not perform useless operations, because conflicts discovered at commit time (in general) cannot be solved without aborting at least one transaction. +- Second, encounter-time locking Reads-After-Writes (RAW) are handled efficiently without requiring expensive or complex mechanisms. + +**Conclusion** + +OCC is the fastest option for most workloads. This finding has also been observed in our preliminary research phase. + +One of the reasons is that when every core executes multiple threads, a lock is likely to be held by a swapped thread, especially in interactive mode. Another reason is that pessimistic algorithms involve deadlock detection (which introduces overhead) and usually uses read-write locks (which are less efficient than standard spin-locks). + +We have chosen Silo because it was simpler than other existing options, such as TicToc, while maintaining the same performance for most workloads. ETL is sometimes faster than OCC, but it introduces spurious aborts which may confuse a user, in contrast to OCC which aborts only at commit. + +### OCC vs 2PL Differences by Example + +The following shows the differences between two user experiences - Pessimistic (for disk-based tables) and Optimistic (MOT tables) when sessions update the same table simultaneously. + +In this example, the following table test command is run - + +``` +table "TEST" - create table test (x int, y int, z int, primary key(x)); +``` + +This example describes two aspects of the same test - user experience (operations in the example) and retry requirements. + +**Example Pessimistic Approach - Used in Disk-based Tables** + +The following is an example of the Pessimistic approach (which is not Mot). Any Isolation Level may apply. + +The following two sessions perform a transaction that attempts to update a single table. + +A WAIT LOCK action occurs and the client experience is that session #2 is *stuck* until Session #1 has completed a COMMIT. Only afterwards, is Session #2 able to progress. + +However, when this approach is used, both sessions succeed and no abort occurs (unless SERIALIZABLE or REPEATABLE-READ isolation level is applied), which results in the entire transaction needing to be retried. + +**Table 1** Pessimistic Approach Code Example + +| | Session 1 | Session 2 | +| :--- | :------------------------------- | :----------------------------------------------------------- | +| t0 | Begin | Begin | +| t1 | update test set y=200 where x=1; | | +| t2 | y=200 | Update test set y=300 where x=1; - Wait on lock | +| t4 | Commit | | +| | | Unlock | +| | | Commit(in READ-COMMITTED this will succeed, in SERIALIZABLE it will fail) | +| | | y = 300 | + +**Example Optimistic Approach - Used in MOT** + +The following is an example of the Optimistic approach. + +It describes the situation of creating an MOT table and then having two concurrent sessions updating that same MOT table simultaneously - + +``` +create foreign table test (x int, y int, z int, primary key(x)); +``` + +- The advantage of OCC is that there are no locks until COMMIT. +- The disadvantage of using OCC is that the update may fail if another session updates the same record. If the update fails (in all supported isolation levels), an entire SESSION #2 transaction must be retried. +- Update conflicts are detected by the kernel at commit time by using a version checking mechanism. +- SESSION #2 will not wait in its update operation and will be aborted because of conflict detection at commit phase. + +**Table 2** Optimistic Approach Code Example - Used in MOT + +| | Session 1 | Session 2 | +| :--- | :------------------------------- | :------------------------------- | +| t0 | Begin | Begin | +| t1 | update test set y=200 where x=1; | | +| t2 | y=200 | Update test set y=300 where x=1; | +| t4 | Commit | y = 300 | +| | | Commit | +| | | ABORT | +| | | y = 200 | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-3.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-3.md new file mode 100644 index 0000000000000000000000000000000000000000..3af831038f8bfda7238c36d37d0f6be5afea3f97 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-3.md @@ -0,0 +1,59 @@ +--- +title: Extended FDW and Other MogDB Features +summary: Extended FDW and Other MogDB Features +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Extended FDW and Other MogDB Features + +MogDB is based on PostgreSQL, which does not have a built-in storage engine adapter, such as MySQL handlerton. To enable the integration of the MOT storage engine into MogDB, we have leveraged and extended the existing Foreign Data Wrapper (FDW) mechanism. With the introduction of FDW into PostgreSQL 9.1, externally managed databases can now be accessed in a way that presents these foreign tables and data sources as united, locally accessible relations. + +In contrast, the MOT storage engine is embedded inside MogDB and its tables are managed by it. Access to tables is controlled by the MogDB planner and executor. MOT gets logging and checkpointing services from MogDB and participates in the MogDB recovery process in addition to other processes. + +We refer to all the components that are in use or are accessing the MOT storage engine as the *Envelope*. + +The following figure shows how the MOT storage engine is embedded inside MogDB and its bi-directional access to database functionality. + +**Figure 1** MOT Storage Engine Embedded inside MogDB - FDW Access to External Databases + +![mot-architecture](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-scale-up-architecture-2.png) + +We have extended the capabilities of FDW by extending and modifying the FdwRoutine structure in order to introduce features and calls that were not required before the introduction of MOT. For example, support for The following new features was added - Add Index, Drop Index/Table, Truncate, Vacuum and Table/Index Memory Statistics. A significant emphasis was put on integration with MogDB logging, replication and checkpointing mechanisms in order to provide consistency for cross-table transactions through failures. In this case, the MOT itself sometimes initiates calls to MogDB functionality through the FDW layer. + +## Creating Tables and Indexes + +In order to support the creation of MOT tables, standard FDW syntax was reused. + +For example, create FOREIGN table. + +The MOT FDW mechanism passes the instruction to the MOT storage engine for actual table creation. Similarly, we support index creation (create index …). This feature was not previously available in FDW, because it was not needed since its tables are managed externally. + +To support both in MOT FDW, the **ValidateTableDef** function actually creates the specified table. It also handles the index creation of that relation, as well as DROP TABLE and DROP INDEX, in addition to VACUUM and ALTER TABLE, which were not previously supported in FDW. + +## Index Usage for Planning and Execution + +A query has two phases - **Planning** and **Execution**. During the Planning phase (which may take place once per multiple executions), the best index for the scan is chosen. This choice is made based on the matching query's WHERE clauses, JOIN clauses and ORDER BY conditions. During execution, a query iterates over the relevant table rows and performs various tasks, such as update or delete, per iteration. An insert is a special case where the table adds the row to all indexes and no scanning is required. + +- **Planner -** In standard FDW, a query is passed for execution to a foreign data source. This means that index filtering and the actual planning (such as the choice of indexes) is not performed locally in the database, rather it is performed in the external data source. Internally, the FDW returns a general plan to the database planner. MOT tables are handled in a similar manner as disk tables. This means that relevant MOT indexes are filtered and matched, and the indexes that minimize the set of traversed rows are selected and are added to the plan. +- **Executor -** The Query Executor uses the chosen MOT index in order to iterate over the relevant rows of the table. Each row is inspected by the MogDB envelope, and according to the query conditions, an update or delete is called to handle the relevant row. + +## Durability, Replication and High Availability + +A storage engine is responsible for storing, reading, updating and deleting data in the underlying memory and storage systems. The logging, checkpointing and recovery are not handled by the storage engine, especially because some transactions encompass multiple tables with different storage engines. Therefore, in order to persist and replicate data, the high-availability facilities from the MogDB envelope are used as follows - + +- **Durability -** In order to ensure Durability, the MOT engine persists data by Write-Ahead Logging (WAL) records using the MogDB's XLOG interface. This also provides the benefits of MogDB's replication capabilities that use the same APIs. You may refer to the **MOT Durability Concepts** for more information. +- **Checkpointing -** A MOT Checkpoint is enabled by registering a callback to the MogDB Checkpointer. Whenever a general database Checkpoint is performed, the MOT Checkpoint process is called as well. MOT keeps the Checkpoint's Log Sequence Number (LSN) in order to be aligned with MogDB recovery. The MOT Checkpointing algorithm is highly optimized and asynchronous and does not stop concurrent transactions. You may refer to the **MOT Checkpoint Concepts** for more information. +- **Recovery -** Upon startup, MogDB first calls an MOT callback that recovers the MOT Checkpoint by loading into memory rows and creating indexes, followed by the execution of the WAL recovery by replaying records according to the Checkpoint's LSN. The MOT Checkpoint is recovered in parallel using multiple threads - each thread reads a different data segment. This makes MOT Checkpoint recovery quite fast on many-core hardware, though it is still potentially slower compared to disk-based tables where only WAL records are replayed. You may refer to the **MOT Recovery Concepts** for more information. + +## VACUUM and DROP + +In order to maximize MOT functionality, we added support for VACUUM, DROP TABLE and DROP INDEX. All three execute with an exclusive table lock, meaning without allowing concurrent transactions on the table. The system VACUUM calls a new FDW function to perform the MOT vacuuming, while DROP was added to the ValidateTableDef() function. + +## Deleting Memory Pools + +Each index and table tracks all the memory pools that it uses. A DROP INDEX command is used to remove metadata. Memory pools are deleted as a single consecutive block. The MOT VACUUM only compacts used memory, because memory reclamation is performed continuously in the background by the epoch-based Garbage Collector (GC). In order to perform the compaction, we switch the index or the table to new memory pools, traverse all the live data, delete each row and insert it using the new pools and finally delete the pools as is done for a drop. + +## Query Native Compilation (JIT) + +The FDW adapter to MOT engine also contains a lite execution path that employs Just-In-Time (JIT) compiled query execution using the LLVM compiler. More information about MOT Query Native Compilation can be found in the **Query Native Compilation (JIT)** section. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-4.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-4.md new file mode 100644 index 0000000000000000000000000000000000000000..0194a6b5c0ca52d4ddac1cb6ecc1fff8bd75f2b5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-4.md @@ -0,0 +1,22 @@ +--- +title: NUMA Awareness Allocation and Affinity +summary: NUMA Awareness Allocation and Affinity +author: Zhang Cuiping +date: 2021-03-04 +--- + +# NUMA Awareness Allocation and Affinity + +Non-Uniform Memory Access (NUMA) is a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can take advantage of NUMA by preferring to access its own local memory (which is faster), rather than accessing non-local memory (meaning that it will prefer **not** to access the local memory of another processor or memory shared between processors). + +MOT memory access has been designed with NUMA awareness. This means that MOT is aware that memory is not uniform and achieves best performance by accessing the quickest and most local memory. + +The benefits of NUMA are limited to certain types of workloads, particularly on servers where the data is often strongly associated with certain tasks or users. + +In-memory database systems running on NUMA platforms face several issues, such as the increased latency and the decreased bandwidth when accessing remote main memory. To cope with these NUMA-related issues, NUMA awareness must be considered as a major design principle for the fundamental architecture of a database system. + +To facilitate quick operation and make efficient use of NUMA nodes, MOT allocates a designated memory pool for rows per table and for nodes per index. Each memory pool is composed from 2 MB chunks. A designated API allocates these chunks from a local NUMA node, from pages coming from all nodes or in a round-robin fashion, where each chunk is allocated on the next node. By default, pools of shared data are allocated in a round robin fashion in order to balance access, while not splitting rows between different NUMA nodes. However, thread private memory is allocated from a local node. It must also be verified that a thread always operates in the same NUMA node. + +**Summary** + +MOT has a smart memory control module that has preallocated memory pools intended for various types of memory objects. This smart memory control improves performance, reduces locks and ensures stability. The allocation of the memory objects of a transaction is always NUMA-local, ensuring optimal performance for CPU memory access and resulting in low latency and reduced contention. Deallocated objects go back to the memory pool. Minimized use of OS malloc functions during transactions circumvents unnecessary locks. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-5.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-5.md new file mode 100644 index 0000000000000000000000000000000000000000..016a23dd0c18a64b8c7ced456d9edc851899fa95 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-5.md @@ -0,0 +1,41 @@ +--- +title: MOT Indexes +summary: MOT Indexes +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Indexes + +MOT Index is a lock-free index based on state-of-the-art Masstree, which is a fast and scalable Key Value (KV) store for multicore systems, implemented as tries of B+ trees. It achieves excellent performance on many-core servers and high concurrent workloads. It uses various advanced techniques, such as an optimistic lock approach, cache-awareness and memory prefetching. + +After comparing various state-of-the-art solutions, we chose Masstree for the index because it demonstrated the best overall performance for point queries, iterations and modifications. Masstree is a combination of tries and a B+ tree that is implemented to carefully exploit caching, prefetching, optimistic navigation and fine-grained locking. It is optimized for high contention and adds various optimizations to its predecessors, such as OLFIT. However, the downside of a Masstree index is its higher memory consumption. While row data consumes the same memory size, the memory per row per each index (primary or secondary) is higher on average by 16 bytes - 29 bytes in the lock-based B-Tree used in disk-based tables vs. 45 bytes in MOT's Masstree. + +Our empirical experiments showed that the combination of the mature lock-free Masstree implementation and our robust improvements to Silo have provided exactly what we needed in that regard. + +Another challenge was making an optimistic insertion into a table with multiple indexes. + +The Masstree index is at the core of MOT memory layout for data and index management. Our team enhanced and significantly improved Masstree and submitted some of the key contributions to the Masstree open source. These improvements include - + +- Dedicated memory pools per index - Efficient allocation and fast index drop +- Global GC for Masstree - Fast, on-demand memory reclamation +- Masstree iterator implementation with access to an insertion key +- ARM architecture support + +We contributed our Masstree index improvements to the Masstree open-source implementation, which can be found here - . + +MOT's main innovation was to enhance the original Masstree data structure and algorithm, which did not support Non-Unique Indexes (as a Secondary index). You may refer to the **Non-unique Indexes** section for the design details. + +MOT supports both Primary, Secondary and Keyless indexes (subject to the limitations specified in the **Unsupported Index DDLs and Index**section). + +## Non-unique Indexes + +A non-unique index may contain multiple rows with the same key. Non-unique indexes are used solely to improve query performance by maintaining a sorted order of data values that are used frequently. For example, a database may use a non-unique index to group all people from the same family. However, the Masstree data structure implementation does not allow the mapping of multiple objects to the same key. Our solution for enabling the creation of non-unique indexes (as shown in the figure below) is to add a symmetry-breaking suffix to the key, which maps the row. This added suffix is the pointer to the row itself, which has a constant size of 8 bytes and a value that is unique to the row. When inserting into a non-unique index, the insertion of the sentinel always succeeds, which enables the row allocated by the executing transaction to be used. This approach also enable MOT to have a fast, reliable, order-based iterator for a non-unique index. + +**Figure 1** Non-unique Indexes + +![non-unique-indexes](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-indexes-2.png) + +The structure of an MOT table T that has three rows and two indexes is depicted in the figure above. The rectangles represent data rows, and the indexes point to sentinels (the elliptic shapes) which point to the rows. The sentinels are inserted into unique indexes with a key and into non-unique indexes with a key + a suffix. The sentinels facilitate maintenance operations so that the rows can be replaced without touching the index data structure. In addition, there are various flags and a reference count embedded in the sentinel in order to facilitate optimistic inserts. + +When searching a non-unique secondary index, the required key (for example, the family name) is used. The fully concatenated key is only used for insert and delete operations. Insert and delete operations always get a row as a parameter, thereby making it possible to create the entire key and to use it in the execution of the deletion or the insertion of the specific row for the index. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-6.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-6.md new file mode 100644 index 0000000000000000000000000000000000000000..9f4b3ea27c7c53f0a5f2c8c27275ee0a142c853d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-6.md @@ -0,0 +1,204 @@ +--- +title: MOT Durability Concepts +summary: MOT Durability Concepts +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Durability Concepts + +Durability refers to long-term data protection (also known as *disk persistence*). Durability means that stored data does not suffer from any kind of degradation or corruption, so that data is never lost or compromised. Durability ensures that data and the MOT engine are restored to a consistent state after a planned shutdown (for example, for maintenance) or an unplanned crash (for example, a power failure). + +Memory storage is volatile, meaning that it requires power to maintain the stored information. Disk storage, on the other hand, is non-volatile, meaning that it does not require power to maintain stored information, thus, it can survive a power shutdown. MOT uses both types of storage - it has all data in memory, while persisting transactional changes to disk **MOT Durability** and by maintaining frequent periodic **MOT Checkpoints** in order to ensure data recovery in case of shutdown. + +The user must ensure sufficient disk space for the logging and Checkpointing operations. A separated drive can be used for the Checkpoint to improve performance by reducing disk I/O load. + +You may refer to **MOT Key Technologies** section__for an overview of how durability is implemented in the MOT engine. + +MOTs WAL Redo Log and checkpoints enabled durability, as described below - + +- **MOT Logging - WAL Redo Log Concepts** +- **MOT Checkpoint Concepts** + +## MOT Logging - WAL Redo Log Concepts + +### Overview + +Write-Ahead Logging (WAL) is a standard method for ensuring data durability. The main concept of WAL is that changes to data files (where tables and indexes reside) are only written after those changes have been logged, meaning only after the log records that describe the changes have been flushed to permanent storage. + +The MOT is fully integrated with the MogDB envelope logging facilities. In addition to durability, another benefit of this method is the ability to use the WAL for replication purposes. + +Three logging methods are supported, two standard Synchronous and Asynchronous, which are also supported by the standard MogDB disk-engine. In addition, in the MOT a Group-Commit option is provided with special NUMA-Awareness optimization. The Group-Commit provides the top performance while maintaining ACID properties. + +To ensure Durability, MOT is fully integrated with the MogDB's Write-Ahead Logging (WAL) mechanism, so that MOT persists data in WAL records using MogDB's XLOG interface. This means that every addition, update, and deletion to an MOT table's record is recorded as an entry in the WAL. This ensures that the most current data state can be regenerated and recovered from this non-volatile log. For example, if three new rows were added to a table, two were deleted and one was updated, then six entries would be recorded in the log. + +- MOT log records are written to the same WAL as the other records of MogDB disk-based tables. + +- MOT only logs an operation at the transaction commit phase. + +- MOT only logs the updated delta record in order to minimize the amount of data written to disk. + +- During recovery, data is loaded from the last known or a specific Checkpoint; and then the WAL Redo log is used to complete the data changes that occur from that point forward. + +- The WAL (Redo Log) retains all the table row modifications until a Checkpoint is performed (as described above). The log can then be truncated in order to reduce recovery time and to save disk space. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** In order to ensure that the log IO device does not become a bottleneck, the log file must be placed on a drive that has low latency. + +### Logging Types + +Two synchronous transaction logging options and one asynchronous transaction logging option are supported (these are also supported by the standard MogDB disk engine). MOT also supports synchronous Group Commit logging with NUMA-awareness optimization, as described below. + +According to your configuration, one of the following types of logging is implemented: + +- **Synchronous Redo Logging** + + The **Synchronous Redo Logging** option is the simplest and most strict redo logger. When a transaction is committed by a client application, the transaction redo entries are recorded in the WAL (Redo Log), as follows - + + 1. While a transaction is in progress, it is stored in the MOT’s memory. + 2. After a transaction finishes and the client application sends a **Commit** command, the transaction is locked and then written to the WAL Redo Log on the disk. This means that while the transaction log entries are being written to the log, the client application is still waiting for a response. + 3. As soon as the transaction's entire buffer is written to the log, the changes to the data in memory take place and then the transaction is committed. After the transaction has been committed, the client application is notified that the transaction is complete. + +- **Technical Description** + + When a transaction ends, the SynchronousRedoLogHandler serializes its transaction buffer and write it to the XLOG iLogger implementation. + + **Figure 1** Synchronous Logging + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-durability-concepts-6.png) + + **Summary** + + The **Synchronous Redo Logging** option is the safest and most strict because it ensures total synchronization of the client application and the WAL Redo log entries for each transaction as it is committed; thus ensuring total durability and consistency with absolutely no data loss. This logging option prevents the situation where a client application might mark a transaction as successful, when it has not yet been persisted to disk. + + The downside of the **Synchronous Redo Logging** option is that it is the slowest logging mechanism of the three options. This is because a client application must wait until all data is written to disk and because of the frequent disk writes (which typically slow down the database). + +- **Group Synchronous Redo Logging** + + The **Group Synchronous Redo Logging** option is very similar to the **Synchronous Redo Logging** option, because it also ensures total durability with absolutely no data loss and total synchronization of the client application and the WAL (Redo Log) entries. The difference is that the **Group Synchronous Redo Logging** option writes _groups of transaction_redo entries to the WAL Redo Log on the disk at the same time, instead of writing each and every transaction as it is committed. Using Group Synchronous Redo Logging reduces the amount of disk I/Os and thus improves performance, especially when running a heavy workload. + + The MOT engine performs synchronous Group Commit logging with Non-Uniform Memory Access (NUMA)-awareness optimization by automatically grouping transactions according to the NUMA socket of the core on which the transaction is running. + + You may refer to the **NUMA Awareness Allocation and Affinity** section for more information about NUMA-aware memory access. + + When a transaction commits, a group of entries are recorded in the WAL Redo Log, as follows - + + 1. While a transaction is in progress, it is stored in the memory. The MOT engine groups transactions in buckets according to the NUMA socket of the core on which the transaction is running. This means that all the transactions running on the same socket are grouped together and that multiple groups will be filling in parallel according to the core on which the transaction is running. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > Each thread runs on a single core/CPU which belongs to a single socket and each thread only writes to the socket of the core on which it is running. + + 2. After a transaction finishes and the client application sends a Commit command, the transaction redo log entries are serialized together with other transactions that belong to the same group. + + 3. After the configured criteria are fulfilled for a specific group of transactions (quantity of committed transactions or timeout period as describes in the **REDO LOG (MOT)** section), the transactions in this group are written to the WAL on the disk. This means that while these log entries are being written to the log, the client applications that issued the commit are waiting for a response. + + 4. As soon as all the transaction buffers in the NUMA-aware group have been written to the log, all the transactions in the group are performing the necessary changes to the memory store and the clients are notified that these transactions are complete. + + Writing transactions to the WAL is more efficient in this manner because all the buffers from the same socket are written to disk together. + + **Technical Description** + + The four colors represent 4 NUMA nodes. Thus each NUMA node has its own memory log enabling a group commit of multiple connections. + + **Figure 2** Group Commit - with NUMA-awareness + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-durability-concepts-7.png) + + **Summary** + + The **Group Synchronous Redo Logging** option is a an extremely safe and strict logging option because it ensures total synchronization of the client application and the WAL Redo log entries; thus ensuring total durability and consistency with absolutely no data loss. This logging option prevents the situation where a client application might mark a transaction as successful, when it has not yet been persisted to disk. + + On one hand this option has fewer disk writes than the **Synchronous Redo Logging** option, which may mean that it is faster. The downside is that transactions are locked for longer, meaning that they are locked until after all the transactions in the same NUMA memory have been written to the WAL Redo Log on the disk. + + The benefits of using this option depend on the type of transactional workload. For example, this option benefits systems that have many transactions (and less so for systems that have few transactions, because there are few disk writes anyway). + +- **Asynchronous Redo Logging** + + The **Asynchronous Redo Logging** option is the fastest logging method, However, it does not ensure no data loss, meaning that some data that is still in the buffer and was not yet written to disk may get lost upon a power failure or database crash. When a transaction is committed by a client application, the transaction redo entries are recorded in internal buffers and written to disk at preconfigured intervals. The client application does not wait for the data being written to disk. It continues to the next transaction. This is what makes asynchronous redo logging the fastest logging method. + + When a transaction is committed by a client application, the transaction redo entries are recorded in the WAL Redo Log, as follows - + + 1. While a transaction is in progress, it is stored in the MOT's memory. + 2. After a transaction finishes and the client application sends a Commit command, the transaction redo entries are written to internal buffers, but are not yet written to disk. Then changes to the MOT data memory take place and the client application is notified that the transaction is committed. + 3. At a preconfigured interval, a redo log thread running in the background collects all the buffered redo log entries and writes them to disk. + + **Technical Description** + + Upon transaction commit, the transaction buffer is moved (pointer assignment - not a data copy) to a centralized buffer and a new transaction buffer is allocated for the transaction. The transaction is released as soon as its buffer is moved to the centralized buffer and the transaction thread is not blocked. The actual write to the log uses the Postgres walwriter thread. When the walwriter timer elapses, it first calls the AsynchronousRedoLogHandler (via registered callback) to write its buffers and then continues with its logic and flushes the data to the XLOG. + + **Figure 3** Asynchronous Logging + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-durability-concepts-8.png) + + **Summary** + + The Asynchronous Redo Logging option is the fastest logging option because it does not require the client application to wait for data being written to disk. In addition, it groups many transactions redo entries and writes them together, thus reducing the amount of disk I/Os that slow down the MOT engine. + + The downside of the Asynchronous Redo Logging option is that it does not ensure that data will not get lost upon a crash or failure. Data that was committed, but was not yet written to disk, is not durable on commit and thus cannot be recovered in case of a failure. The Asynchronous Redo Logging option is most relevant for applications that are willing to sacrifice data recovery (consistency) over performance. + + Logging Design Details + + The following describes the design details of each persistence-related component in the In-Memory Engine Module. + + **Figure 4** Three Logging Options + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-durability-concepts-9.png) + + The RedoLog component is used by both by backend threads that use the In-Memory Engine and by the WAL writer in order to persist their data. Checkpoints are performed using the Checkpoint Manager, which is triggered by the Postgres checkpointer. + +- **Logging Design Overview** + + Write-Ahead Logging (WAL) is a standard method for ensuring data durability. WAL's central concept is that changes to data files (where tables and indexes reside) are only written after those changes have been logged, meaning after the log records that describe these changes have been flushed to permanent storage. + + The MOT Engine uses the existing MogDB logging facilities, enabling it also to participate in the replication process. + +- **Per-transaction Logging** + + In the In-Memory Engine, the transaction log records are stored in a transaction buffer which is part of the transaction object (TXN). The transaction buffer is logged during the calls to addToLog() - if the buffer exceeds a threshold it is then flushed and reused. When a transaction commits and passes the validation phase (OCC SILO**[Comparison - Disk vs. MOT] validation)** or aborts for some reason, the appropriate message is saved in the log as well in order to make it possible to determine the transaction's state during a recovery. + + **Figure 5** Per-transaction Logging + + ![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/mot-durability-concepts-10.png) + + Parallel Logging is performed both by MOT and disk engines. However, the MOT engine enhances this design with a log-buffer per transaction, lockless preparation and a single log record. + +- **Exception Handling** + + The persistence module handles exceptions by using the Postgres error reporting infrastructure (ereport). An error message is recorded in the system log for each error condition. In addition, the error is reported to the envelope using Postgres’s built-in error reporting infrastructure. + + The following exceptions are reported by this module - + + **Table 1** Exception Handling + + | Exception Condition | Exception Code | Scenario | Resulting Outcome | + | :----------------------------------- | :----------------------------- | :----------------------------------------------------------- | :--------------------- | + | WAL write failure | ERRCODE_FDW_ERROR | On any case the WAL write fails | Transaction terminates | + | File IO error: write, open and so on | ERRCODE_IO_ERROR | Checkpoint - Called on any file access error | FATAL - process exists | + | Out of Memory | ERRCODE_INSUFFICIENT_RESOURCES | Checkpoint - Local memory allocation failures | FATAL - process exists | + | Logic, DB errors | ERRCODE_INTERNAL_ERROR | Checkpoint: algorithm fails or failure to retrieve table data or indexes. | FATAL - process exists | + +## MOT Checkpoint Concepts + +In MogDB, a Checkpoints is a snapshot of a point in the sequence of transactions at which it is guaranteed that the heap and index data files have been updated with all information written before the checkpoint. + +At the time of a Checkpoint, all dirty data pages are flushed to disk and a special checkpoint record is written to the log file. + +The data is stored directly in memory. The MOT does not store its data it the same way as MogDB so that the concept of dirty pages does not exist. + +For this reason, we have researched and implemented the CALC algorithm, which is described in the paper named Low-Overhead Asynchronous Checkpointing in Main-Memory Database Systems, SIGMOND 2016 from Yale University. + +Low-overhead asynchronous checkpointing in main-memory database systems. + +### CALC Checkpoint Algorithm - Low Overhead in Memory and Compute + +The checkpoint algorithm provides the following benefits - + +- **Reduced Memory Usage -** At most two copies of each record are stored at any time. Memory usage is minimized by only storing a single physical copy of a record while it is live and stable versions are equal or when no checkpoint is actively being recorded. +- **Low Overhead -** CALC's overhead is smaller than other asynchronous checkpointing algorithms. +- **Uses Virtual Points of Consistency -** CALC does not require quiescing of the database in order to achieve a physical point of consistency. + +### Checkpoint Activation + +MOT checkpoints are integrated into MogDB's envelope's Checkpoint mechanism. The Checkpoint process can be triggered manually by executing the **CHECKPOINT;** command or automatically according to the envelope's Checkpoint triggering settings (time/size). + +Checkpoint configuration is performed in the mot.conf file - see the **CHECKPOINT (MOT)** section. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-7.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-7.md new file mode 100644 index 0000000000000000000000000000000000000000..1cb8bcb94e9f27a28c384fe15562565c8595b614 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-7.md @@ -0,0 +1,24 @@ +--- +title: MOT Recovery Concepts +summary: MOT Recovery Concepts +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Recovery Concepts + +The MOT Recovery Module provides all the required functionality for recovering the MOT tables data. The main objective of the Recovery module is to restore the data and the MOT engine to a consistent state after a planned (maintenance for example) shut down or an unplanned (power failure for example) crash. + +MogDB database recovery, which is also sometimes called a *Cold Start*, includes MOT tables and is performed automatically with the recovery of the rest of the database. The MOT Recovery Module is seamlessly and fully integrated into the MogDB recovery process. + +MOT recovery has two main stages - Checkpoint Recovery and WAL Recovery (Redo Log). + +MOT checkpoint recovery is performed before the envelope's recovery takes place. This is done only at cold-start events (start of a PG process). It recovers the metadata first (schema) and then inserts all the rows from the current valid checkpoint, which is done in parallel by checkpoint_recovery_workers, each working on a different table. The indexes are created during the insert process. + +When checkpointing a table, it is divided into 16MB chunks, so that multiple recovery workers can recover the table in parallel. This is done in order to speed-up the checkpoint recovery, it is implemented as a multi-threaded procedure where each thread is responsible for recovering a different segment. There are no dependencies between different segments therefore there is no contention between the threads and there is no need to use locks when updating table or inserting new rows. + +WAL records are recovered as part of the envelope's WAL recovery. MogDB envelope iterates through the XLOG and performs the necessary operation based on the xlog record type. In case of entry with record type MOT, the envelope forwards it to MOT RecoveryManager for handling. The xlog entry will be ignored by MOT recovery, if it is 'too old' - its LSN is older than the checkpoint's LSN (Log Sequence Number). + +In an active-standby deployment, the standby server is always in a Recovery state for an automatic WAL recovery process. + +The MOT recovery parameters are set in the mot.conf file explained in the **[MOT Recovery](../../../administrator-guide/mot-engine/2-using-mot/5-mot-administration.md#mot-recovery)** section. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-8.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-8.md new file mode 100644 index 0000000000000000000000000000000000000000..661cac1f9337f39acb9c6093215b9d02c18de05e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-8.md @@ -0,0 +1,73 @@ +--- +title: MOT Query Native Compilation (JIT) +summary: MOT Query Native Compilation (JIT) +author: Zhang Cuiping +date: 2021-03-04 +--- + +# MOT Query Native Compilation (JIT) + +MOT enables you to prepare and parse *pre-compiled full queries* in a native format (using a **PREPARE** statement) before they are needed for execution. + +This native format can later be executed (using an **EXECUTE** command) more efficiently. This type of execution is much more efficient because during execution the native format bypasses multiple database processing layers. This division of labor avoids repetitive parse analysis operations. The Lite Executor module is responsible for executing **prepared** queries and has a much faster execution path than the regular generic plan performed by the envelope. This is achieved using Just-In-Time (JIT) compilation via LLVM. In addition, a similar solution that has potentially similar performance is provided in the form of pseudo-LLVM. + +The following is an example of a **PREPARE** syntax in SQL: + +``` +PREPARE name [ ( data_type [, ...] ) ] AS statement +``` + +The following is an example of how to invoke a PREPARE and then an EXECUTE statement in a Java application - + +``` +conn = DriverManager.getConnection(connectionUrl, connectionUser, connectionPassword); + +// Example 1: PREPARE without bind settings +String query = "SELECT * FROM getusers"; +PreparedStatement prepStmt1 = conn.prepareStatement(query); +ResultSet rs1 = pstatement.executeQuery()) +while (rs1.next()) {…} + +// Example 2: PREPARE with bind settings +String sqlStmt = "SELECT * FROM employees where first_name=? and last_name like ?"; +PreparedStatement prepStmt2 = conn.prepareStatement(sqlStmt); +prepStmt2.setString(1, "Mark"); // first name "Mark" +prepStmt2.setString(2, "%n%"); // last name contains a letter "n" +ResultSet rs2 = prepStmt2.executeQuery()) +while (rs2.next()) {…} +``` + +## Prepare + +**PREPARE** creates a prepared statement. A prepared statement is a server-side object that can be used to optimize performance. When the **PREPARE** statement is executed, the specified statement is parsed, analyzed and rewritten. + +If the tables mentioned in the query statement are MOT tables, the MOT compilation takes charge of the object preparation and performs a special optimization by compiling the query into IR byte code based on LLVM. + +Whenever a new query compilation is required, the query is analyzed and a proper tailored IR byte code is generated for the query using the utility GsCodeGen object and standard LLVM JIT API (IRBuilder). After byte-code generation is completed, the code is JIT-compiled into a separate LLVM module. The compiled code results in a C function pointer that can later be invoked for direct execution. Note that this C function can be invoked concurrently by many threads, as long as each thread provides a distinct execution context (details are provided below). Each such execution context is referred to as *JIT Context*. + +To improve performance further, MOT JIT applies a caching policy for its LLVM code results, enabling them to be reused for the same queries across different sessions. + +## Execute + +When an EXECUTE command is issued, the prepared statement (described above) is planned and executed. This division of labor avoids repetitive parse analysis work, while enabling the execution plan to depend on the specific setting values supplied. + +When the resulting execute query command reaches the database, it uses the corresponding IR byte code which is executed directly and more efficiently within the MOT engine. This is referred to as *Lite Execution*. + +In addition, for availability, the Lite Executor maintains a preallocated pool of JIT sources. Each session preallocates its own session-local pool of JIT context objects (used for repeated executions of precompiled queries). + +For more details you may refer to the Supported Queries for Lite Execution and Unsupported Queries for Lite Execution sections. + +## JIT Compilation Comparison - MogDB Disk-based vs. MOT Tables + +Currently, MogDB contains two main forms of JIT / CodeGen query optimizations for its disk-based tables - + +- Accelerating expression evaluation, such as in WHERE clauses, target lists, aggregates and projections. +- Inlining small function invocations. + +These optimizations are partial (in the sense they do not optimize the entire interpreted operator tree or replace it altogether) and are targeted mostly at CPU-bound complex queries, typically seen in OLAP use cases. The execution of queries is performed in a pull-model (Volcano-style processing) using an interpreted operator tree. When activated, the compilation is performed at each query execution. At the moment, caching of the generated LLVM code and its reuse across sessions and queries is not yet provided. + +In contrast, MOT JIT optimization provides LLVM code for entire queries that qualify for JIT optimization by MOT. The resulting code is used for direct execution over MOT tables, while the interpreted operator model is abandoned completely. The result is *practically* handwritten LLVM code that has been generated for an entire specific query execution. + +Another significant conceptual difference is that MOT LLVM code is only generated for prepared queries during the PREPARE phase of the query, rather than at query execution. This is especially important for OLTP scenarios due to the rather short runtime of OLTP queries, which cannot allow for code generation and relatively long query compilation time to be performed during each query execution. + +Finally, in PostgreSQL the activation of a PREPARE implies the reuse of the resulting plan across executions with different parameters in the same session. Similarly, the MOT JIT applies a caching policy for its LLVM code results, and extends it for reuse across different sessions. Thus, a single query may be compiled just once and its LLVM code may be reused across many sessions, which again is beneficial for OLTP scenarios. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-9.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-9.md new file mode 100644 index 0000000000000000000000000000000000000000..6aac4a5c5884d754728cbc0b9b253ee0aeae0d2e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/3-concepts-of-mot/3-9.md @@ -0,0 +1,69 @@ +--- +title: Comparison - Disk vs. MOT +summary: Comparison - Disk vs. MOT +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Comparison - Disk vs. MOT + +The following table briefly compares the various features of the MogDB disk-based storage engine and the MOT storage engine. + +**Table 1** Comparison - Disk-based vs. MOT + +| Feature | MogDB Disk Store | MogDB MOT Engine | +| :--------------------------- | :------------------ | :---------------------------- | +| Intel x86 + Kunpeng ARM | Yes | Yes | +| SQL and Feature-set Coverage | 100% | 98% | +| Scale-up (Many-cores, NUMA) | Low Efficiency | High Efficiency | +| Throughput | High | Extremely High | +| Latency | Low | Extremely Low | +| Distributed (Cluster Mode) | Yes | Yes | +| Isolation Levels | RC+SIRRSerializable | RCRRRC+SI (in V2 release) | +| Concurrency Control | Pessimistic | Optimistic | +| Data Capacity (Data + Index) | Unlimited | Limited to DRAM | +| Native Compilation | No | Yes | +| Replication, Recovery | Yes | Yes | +| Replication Options | 2 (sync, async) | 3 (sync, async, group-commit) | + +**Legend -** + +- RR = Repeatable Reads +- RC = Read Committed +- SI = Snapshot Isolation + +## Appendices + +## References + +[1] Y. Mao, E. Kohler, and R. T. Morris. Cache craftiness for fast multicore key-value storage. In Proc. 7th ACM European Conference on Computer Systems (EuroSys), Apr. 2012. + +[2] K. Ren, T. Diamond, D. J. Abadi, and A. Thomson. Low-overhead asynchronous checkpointing in main-memory database systems. In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data, 2016. + +[3] . + +[4] . + +[5] Tu, S., Zheng, W., Kohler, E., Liskov, B., and Madden, S. Speedy transactions in multicore in-memory databases. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (New York, NY, USA, 2013), SOSP ’13, ACM, pp. 18-32. + +[6] H. Avni at al. Industrial-Strength OLTP Using Main Memory and Many-cores, VLDB 2020. + +[7] Bernstein, P. A., and Goodman, N. Concurrency control in distributed database systems. ACM Comput. Surv. 13, 2 (1981), 185-221. + +[8] Felber, P., Fetzer, C., and Riegel, T. Dynamic performance tuning of word-based software transactional memory. In Proceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPOPP 2008, Salt Lake City, UT, USA, February 20-23, 2008 (2008), + +pp. 237-246. + +[9] Appuswamy, R., Anadiotis, A., Porobic, D., Iman, M., and Ailamaki, A. Analyzing the impact of system architecture on the scalability of OLTP engines for high-contention workloads. PVLDB 11, 2 (2017), + +121-134. + +[10] R. Sherkat, C. Florendo, M. Andrei, R. Blanco, A. Dragusanu, A. Pathak, P. Khadilkar, N. Kulkarni, C. Lemke, S. Seifert, S. Iyer, S. Gottapu, R. Schulze, C. Gottipati, N. Basak, Y. Wang, V. Kandiyanallur, S. Pendap, D. Gala, R. Almeida, and P. Ghosh. Native store extension for SAP HANA. PVLDB, 12(12): + +2047-2058, 2019. + +[11] X. Yu, A. Pavlo, D. Sanchez, and S. Devadas. Tictoc: Time traveling optimistic concurrency control. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 1629-1642, 2016. + +[12] V. Leis, A. Kemper, and T. Neumann. The adaptive radix tree: Artful indexing for main-memory databases. In C. S. Jensen, C. M. Jermaine, and X. Zhou, editors, 29th IEEE International Conference on Data Engineering, ICDE 2013, Brisbane, Australia, April 8-12, 2013, pages 38-49. IEEE Computer Society, 2013. + +[13] S. K. Cha, S. Hwang, K. Kim, and K. Kwon. Cache-conscious concurrency control of main-memory indexes on shared-memory multiprocessor systems. In P. M. G. Apers, P. Atzeni, S. Ceri, S. Paraboschi, K. Ramamohanarao, and R. T. Snodgrass, editors, VLDB 2001, Proceedings of 27th International Conference on Very Large Data Bases, September 11-14, 2001, Roma, Italy, pages 181-190. Morga Kaufmann, 2001. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/4-appendix/1-references.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/4-appendix/1-references.md new file mode 100644 index 0000000000000000000000000000000000000000..4055b8f1162ff4f4b79927bdb5d3fcb2172f3c85 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/4-appendix/1-references.md @@ -0,0 +1,40 @@ +--- +title: References +summary: References +author: Zhang Cuiping +date: 2021-05-18 +--- + +# References + +[1] Y. Mao, E. Kohler, and R. T. Morris. Cache craftiness for fast multicore key-value storage. In Proc. 7th ACM European Conference on Computer Systems (EuroSys), Apr. 2012. + +[2] K. Ren, T. Diamond, D. J. Abadi, and A. Thomson. Low-overhead asynchronous checkpointing in main-memory database systems. In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data, 2016. + +[3] . + +[4] . + +[5] Tu, S., Zheng, W., Kohler, E., Liskov, B., and Madden, S. Speedy transactions in multicore in-memory databases. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (New York, NY, USA, 2013), SOSP ’13, ACM, pp. 18-32. + +[6] H. Avni at al. Industrial-Strength OLTP Using Main Memory and Many-cores, VLDB 2020. + +[7] Bernstein, P. A., and Goodman, N. Concurrency control in distributed database systems. ACM Comput. Surv. 13, 2 (1981), 185-221. + +[8] Felber, P., Fetzer, C., and Riegel, T. Dynamic performance tuning of word-based software transactional memory. In Proceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPOPP 2008, Salt Lake City, UT, USA, February 20-23, 2008 (2008), + +pp. 237-246. + +[9] Appuswamy, R., Anadiotis, A., Porobic, D., Iman, M., and Ailamaki, A. Analyzing the impact of system architecture on the scalability of OLTP engines for high-contention workloads. PVLDB 11, 2 (2017), + +121-134. + +[10] R. Sherkat, C. Florendo, M. Andrei, R. Blanco, A. Dragusanu, A. Pathak, P. Khadilkar, N. Kulkarni, C. Lemke, S. Seifert, S. Iyer, S. Gottapu, R. Schulze, C. Gottipati, N. Basak, Y. Wang, V. Kandiyanallur, S. Pendap, D. Gala, R. Almeida, and P. Ghosh. Native store extension for SAP HANA. PVLDB, 12(12): + +2047-2058, 2019. + +[11] X. Yu, A. Pavlo, D. Sanchez, and S. Devadas. Tictoc: Time traveling optimistic concurrency control. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 1629-1642, 2016. + +[12] V. Leis, A. Kemper, and T. Neumann. The adaptive radix tree: Artful indexing for main-memory databases. In C. S. Jensen, C. M. Jermaine, and X. Zhou, editors, 29th IEEE International Conference on Data Engineering, ICDE 2013, Brisbane, Australia, April 8-12, 2013, pages 38-49. IEEE Computer Society, 2013. + +[13] S. K. Cha, S. Hwang, K. Kim, and K. Kwon. Cache-conscious concurrency control of main-memory indexes on shared-memory multiprocessor systems. In P. M. G. Apers, P. Atzeni, S. Ceri, S. Paraboschi, K. Ramamohanarao, and R. T. Snodgrass, editors, VLDB 2001, Proceedings of 27th International Conference on Very Large Data Bases, September 11-14, 2001, Roma, Italy, pages 181-190. Morga Kaufmann, 2001. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/4-appendix/2-glossary.md b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/4-appendix/2-glossary.md new file mode 100644 index 0000000000000000000000000000000000000000..f2d1576038d266439c9d9238fcf119133a52ce34 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/mot-engine/4-appendix/2-glossary.md @@ -0,0 +1,59 @@ +--- +title: Glossary +summary: Glossary +author: Zhang Cuiping +date: 2021-05-18 +--- + +# Glossary + +| Acronym | Definition/Description | +| :------ | :----------------------------------------------------------- | +| 2PL | 2-Phase Locking | +| ACID | Atomicity, Consistency, Isolation, Durability | +| AP | Analytical Processing | +| ARM | Advanced RISC Machine, a hardware architecture alternative to x86 | +| CC | Concurrency Control | +| CPU | Central Processing Unit | +| DB | Database | +| DBA | Database Administrator | +| DBMS | Database Management System | +| DDL | Data Definition Language. Database Schema management language | +| DML | Data Modification Language | +| ETL | Extract, Transform, Load or Encounter Time Locking | +| FDW | Foreign Data Wrapper | +| GC | Garbage Collector | +| HA | High Availability | +| HTAP | Hybrid Transactional-Analytical Processing | +| IoT | Internet of Things | +| IM | In-Memory | +| IMDB | In-Memory Database | +| IR | Intermediate Representation of a source code, used in compilation and optimization | +| JIT | Just In Time | +| JSON | JavaScript Object Notation | +| KV | Key Value | +| LLVM | Low-Level Virtual Machine, refers to a compilation code or queries to IR | +| M2M | Machine-to-Machine | +| ML | Machine Learning | +| MM | Main Memory | +| MO | Memory Optimized | +| MOT | Memory Optimized Tables storage engine (SE), pronounced as /em/ /oh/ /tee/ | +| MVCC | Multi-Version Concurrency Control | +| NUMA | Non-Uniform Memory Access | +| OCC | Optimistic Concurrency Control | +| OLTP | Online Transaction Processing | +| PG | PostgreSQL | +| RAW | Reads-After-Writes | +| RC | Return Code | +| RTO | Recovery Time Objective | +| SE | Storage Engine | +| SQL | Structured Query Language | +| TCO | Total Cost of Ownership | +| TP | Transactional Processing | +| TPC-C | An On-Line Transaction Processing Benchmark | +| Tpm-C | Transactions-per-minute-C. A performance metric for TPC-C benchmark that counts new-order transactions. | +| TVM | Tiny Virtual Machine | +| TSO | Time Sharing Option | +| UDT | User-Defined Type | +| WAL | Write Ahead Log | +| XLOG | A PostgreSQL implementation of transaction logging (WAL - described above) | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/primary-and-standby-management.md b/product/en/docs-mogdb/v3.1/administrator-guide/primary-and-standby-management.md new file mode 100644 index 0000000000000000000000000000000000000000..f628f197f2dc43a861cef7c61d97097e9bed5fc2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/primary-and-standby-management.md @@ -0,0 +1,126 @@ +--- +title: Primary and Standby Management +summary: Primary and Standby Management +author: Guo Huan +date: 2021-03-11 +--- + +# Primary and Standby Management + +## Scenarios + +During MogDB database running, the database administrator needs to manually perform an primary/standby switchover on the database node. For example, after a primary/standby database node failover, you need to restore the original primary/standby roles, or you need to manually perform a primary/standby switchover due to a hardware fault. A cascaded standby server cannot be directly switched to a primary server. You must perform a switchover or failover to change the cascaded standby server to a standby server, and then to a primary server. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - The primary/standby switchover is a maintenance operation. Ensure that the MogDB database is normal and perform the switchover after all services are complete. +> - When the ultimate RTO is enabled, cascaded standby servers are not supported. The standby server cannot be connected when the ultimate RTO is enabled. As a result, the cascaded standby server cannot synchronize data. + +## Procedure + +1. Log in to any database node as the OS user **omm** and run the following command to check the primary/standby status: + + ```bash + gs_om -t status --detail + ``` + +2. Log in to the standby node to be switched to the primary node as the OS user **omm** and run the following command: + + ```bash + gs_ctl switchover -D /home/omm/cluster/dn1/ + ``` + + **/home/omm/cluster/dn1/** is the data directory of the standby database node. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** For the same database, you cannot perform a new primary/standby switchover if the previous switchover has not completed. If a switchover is performed when the host thread is processing services, the thread cannot stop, and switchover timeout will be reported. Actually, the switchover is ongoing in the background and will complete after the thread finishes service processing and stops. For example, when a host is deleting a large partitioned table, it may fail to respond to the switchover request. + +3. After the switchover is successful, run the following command to record the information about the current primary and standby nodes: + + ```bash + gs_om -t refreshconf + ``` + +## Examples + +Run the following command to switch the standby database instance to the primary database instance: + +1. Queries database status. + + ```bash + $ gs_om -t status --detail + [ Cluster State ] + + cluster_state : Normal + redistributing : No + current_az : AZ_ALL + + [ Datanode State ] + + node node_ip port instance state + -------------------------------------------------------------------------------------------------- + 1 pekpopgsci00235 10.244.62.204 5432 6001 /home/omm/cluster/dn1/ P Primary Normal + 2 pekpopgsci00238 10.244.61.81 5432 6002 /home/omm/cluster/dn1/ S Standby Normal + ``` + +2. Log in to the standby node and perform a primary/standby switchover. In addition, after a cascaded standby node is switched over, the cascaded standby server becomes a standby server, and the original standby server becomes a cascaded standby server. + + ```bash + $ gs_ctl switchover -D /home/omm/cluster/dn1/ + [2020-06-17 14:28:01.730][24438][][gs_ctl]: gs_ctl switchover ,datadir is -D "/home/omm/cluster/dn1" + [2020-06-17 14:28:01.730][24438][][gs_ctl]: switchover term (1) + [2020-06-17 14:28:01.768][24438][][gs_ctl]: waiting for server to switchover............ + [2020-06-17 14:28:11.175][24438][][gs_ctl]: done + [2020-06-17 14:28:11.175][24438][][gs_ctl]: switchover completed (/home/omm/cluster/dn1) + ``` + +3. Save the information about the primary and standby nodes in the database. + + ```bash + $ gs_om -t refreshconf + Generating dynamic configuration file for all nodes. + Successfully generated dynamic configuration file. + ``` + +## Troubleshooting + +If a switchover fails, troubleshoot the problem according to the log information. For details, see [Log Reference](../administrator-guide/routine-maintenance/11-log-reference.md). + +## Exception Handling + +Exception handling rules are as follows: + +- A switchover takes a long time under high service loads. In this case, no further operation is required. + +- When standby nodes are being built, a primary node can be demoted to a standby node only after sending logs to one of the standby nodes. As a result, the primary/standby switchover takes a long time. In this case, no further operation is required. However, you are not advised to perform a primary/standby switchover during the build process. + +- During a switchover, due to network faults and high disk usage, it is possible that the primary and standby instances are disconnected, or two primary nodes exist in a single pair. In this case, perform the following steps: + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-warning.gif) **WARNING:** After two primary nodes appear, perform the following steps to restore the normal primary/standby state: Otherwise, data loss may occur. + +1. Run the following commands to query the current instance status of the database: + + ```bash + gs_om -t status --detail + ``` + + The query result shows that the status of two instances is **Primary**, which is abnormal. + +2. Determine the node that functions as the standby node and run the following command on the node to stop the service: + + ```bash + gs_ctl stop -D /home/omm/cluster/dn1/ + ``` + +3. Run the following command to start the standby node in standby mode: + + ```bash + gs_ctl start -D /home/omm/cluster/dn1/ -M standby + ``` + +4. Save the information about the primary and standby nodes in the database. + + ```bash + gs_om -t refreshconf + ``` + +5. Check the database status and ensure that the instance status is restored. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/0-starting-and-stopping-mogdb.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/0-starting-and-stopping-mogdb.md new file mode 100644 index 0000000000000000000000000000000000000000..8715c411706c2c4d4d35a92e97227d0820f015d3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/0-starting-and-stopping-mogdb.md @@ -0,0 +1,218 @@ +--- +title: Starting and Stopping MogDB +summary: Starting and Stopping MogDB +author: Guo Huan +date: 2021-06-24 +--- + +# Starting and Stopping MogDB + +## Operating By OM + +### Starting MogDB + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to start MogDB: + + ```bash + gs_om -t start + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** An HA cluster must be started in HA mode. If the cluster is started in standalone mode, you need to restore the HA relationship by running the **gs_ctl build** command. For details about how to use the **gs_ctl** tool, see [gs_ctl](../../reference-guide/tool-reference/tools-used-in-the-internal-system/4-gs_ctl.md). + +### Stopping MogDB + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to stop MogDB: + + ```bash + gs_om -t stop + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** For details about how to start and stop nodes and availability zones (AZs), see [gs_om](../../reference-guide/tool-reference/server-tools/8-gs_om.md). + +### Examples + +Start MogDB: + +```bash +gs_om -t start +Starting cluster. +========================================= +========================================= +Successfully started. +``` + +Stop MogDB: + +```bash +gs_om -t stop +Stopping cluster. +========================================= +Successfully stopped cluster. +========================================= +End stop cluster. +``` + +### Troubleshooting + +If starting or stopping MogDB fails, troubleshoot the problem based on log information. For details, see [Log Reference](11-log-reference.md). + +If the startup fails due to timeout, you can run the following command to set the startup timeout interval, which is 300s by default: + +```bash +gs_om -t start --time-out=300 +``` + +
+ +## Operating By PTK + +### Start the Cluster + +> The following uses the `cluster_slirist` cluster as an example. + +After the database cluster is installed, PTK will start the database cluster by default. + +If the `--skip-launch-db` parameter is specified during installation, the database is in the stopped status. + +You can run the `cluster start` command to start the database cluster. You need to specify the cluster name in the command. + +Example: + +```shell +# ptk cluster -n cluster_slirist start +INFO[2022-08-02T11:40:48.728] Operating: Starting. +INFO[2022-08-02T11:40:48.728] ========================================= +INFO[2022-08-02T11:40:48.784] starting host 192.168.122.101 +INFO[2022-08-02T11:40:54.097] starting host 192.168.122.101 successfully +INFO[2022-08-02T11:40:54.097] starting host 192.168.122.102 +INFO[2022-08-02T11:40:56.329] starting host 192.168.122.102 successfully +INFO[2022-08-02T11:40:56.613] waiting for check cluster state... +INFO[2022-08-02T11:41:01.861] ========================================= +INFO[2022-08-02T11:41:01.861] Successfully started. +INFO[2022-08-02T11:41:01.861] Operation succeeded: Start. +``` + +PTK starts all instances in the cluster by default. It also supports the starting of a single instance. + +```shell +# ptk cluster -n cluster_slirist start -H 192.168.122.101 +INFO[2022-08-02T11:50:04.442] Operating: Starting. +INFO[2022-08-02T11:50:04.442] ========================================= +INFO[2022-08-02T11:50:06.692] starting host 192.168.122.101 successfully +``` + +For more parameters, see the help information. + +```shell +# ptk cluster start -h +Start a database instance or cluster. + +Usage: + ptk cluster start [flags] + +Flags: + -h, --help help for start + -H, --host string Specifies the IP address of an instance. + -n, --name string Specifies the cluster name. + --security-mode string Specifies whether to start a database in safe mode. + The value can be on and off. + --time-out duration Specifies the start timeout duration. The default value is 10 minutes. +``` + +### Stop the Cluster + +> The following uses the `cluster_slirist` cluster as an example. + +You can run the `cluster stop` command to stop a database cluster. PTK will stop all instances in a cluster by default. + +```shell +# ptk cluster -n cluster_slirist stop +INFO[2022-08-02T11:49:40.685] Operating: Stopping. +INFO[2022-08-02T11:49:40.685] ========================================= +INFO[2022-08-02T11:49:40.891] stopping host 192.168.122.102 +INFO[2022-08-02T11:49:41.946] stopping host 192.168.122.102 successfully +INFO[2022-08-02T11:49:41.946] stopping host 192.168.122.101 +INFO[2022-08-02T11:49:43.004] stopping host 192.168.122.101 successfully +INFO[2022-08-02T11:49:43.004] ========================================= +INFO[2022-08-02T11:49:43.004] Successfully stoped. +INFO[2022-08-02T11:49:43.004] Operation succeeded: Stop. +``` + +You can use the `-H` parameter to specify a instance to stop it. + +```shell +# ptk cluster -n cluster_slirist stop -H 192.168.122.101 +INFO[2022-08-02T11:56:32.880] Operating: Stopping. +INFO[2022-08-02T11:56:32.881] ========================================= +INFO[2022-08-02T11:56:34.154] stopping host 192.168.122.101 successfully +``` + +For more parameters, see the help information. + +```shell +# ptk cluster stop -h +Stop a database instance or cluster. + +Usage: + ptk cluster stop [flags] + +Flags: + -h, --help help for stop + -H, --host string Specifies the IP address of an instance. + -n, --name string Specifies the cluster name. + --time-out duration Specifies the stop timeout duration. The default value is 10 minutes. +``` + +### Restart the Cluster + +> The following uses the `cluster_slirist` cluster as an example. + +Restarting a cluster is actually to stop the database first and then start the database. + +You can run the `cluster restart` command to restart the cluster. + +```shell +# ptk cluster -n cluster_slirist restart +INFO[2022-08-02T11:59:31.037] Operating: Stopping. +INFO[2022-08-02T11:59:31.037] ========================================= +INFO[2022-08-02T11:59:31.217] stopping host 192.168.122.102 +INFO[2022-08-02T11:59:32.269] stopping host 192.168.122.102 successfully +INFO[2022-08-02T11:59:32.269] stopping host 192.168.122.101 +INFO[2022-08-02T11:59:33.309] stopping host 192.168.122.101 successfully +INFO[2022-08-02T11:59:33.309] ========================================= +INFO[2022-08-02T11:59:33.309] Successfully stoped. +INFO[2022-08-02T11:59:33.309] Operation succeeded: Stop. + +INFO[2022-08-02T11:59:33.310] Operating: Starting. +INFO[2022-08-02T11:59:33.310] ========================================= +INFO[2022-08-02T11:59:33.376] starting host 192.168.122.101 +INFO[2022-08-02T11:59:35.583] starting host 192.168.122.101 successfully +INFO[2022-08-02T11:59:35.583] starting host 192.168.122.102 +INFO[2022-08-02T11:59:36.787] starting host 192.168.122.102 successfully +INFO[2022-08-02T11:59:36.995] waiting for check cluster state... +INFO[2022-08-02T11:59:42.247] ========================================= +INFO[2022-08-02T11:59:42.247] Successfully started. +INFO[2022-08-02T11:59:42.247] Operation succeeded: Start. +``` + +For more parameters, see the help information. + +```shell +# ptk cluster restart -h +Restart a database instance or cluster. + +Usage: + ptk cluster restart [flags] + +Flags: + -h, --help help for restart + -H, --host string Specifies the IP address of an instance. + -n, --name string Specifies the cluster name. + --security-mode string Specifies whether to start a database in safe mode. + The value can be on and off. + --time-out duration Specifies the start timeout duration. The default value is 10 minutes. +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/1-routine-maintenance-check-items.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/1-routine-maintenance-check-items.md new file mode 100644 index 0000000000000000000000000000000000000000..b27725fd83edf81be0ba615b314dfcf8d0108479 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/1-routine-maintenance-check-items.md @@ -0,0 +1,164 @@ +--- +title: Routine Maintenance Check Items +summary: Routine Maintenance Check Items +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Routine Maintenance Check Items + +## Checking MogDB Status + +MogDB provides tools to check database and instance status, ensuring that databases and instances are running properly to provide data services. + +- Check instance status. + + ```bash + gs_check -U omm -i CheckClusterState + ``` + +- Check parameters. + + ```sql + mogdb=# SHOW parameter_name; + ``` + +- Modify parameters. + + ```bash + gs_guc reload -D /mogdb/data/dbnode -c "paraname=value" + ``` + +## Checking Lock Information + +The lock mechanism is an important method to ensure data consistency. Information check helps learn database transactions and database running status. + +- Query lock information in the database. + + ```sql + mogdb=# SELECT * FROM pg_locks; + ``` + +- Query the status of threads waiting to acquire locks. + + ```sql + mogdb=# SELECT * FROM pg_thread_wait_status WHERE wait_status = 'acquire lock'; + ``` + +- Kill a system process. + + Search for a system process that is running and run the following command to end the process: + + ``` + ps ux + kill -9 pid + ``` + +## Collecting Event Statistics + +Long-time running of SQL statements will occupy a lot of system resources. You can check event occurrence time and occupied memory to learn about database running status. + +- Query the time points about an event. + + Run the following command to query the thread start time, transaction start time, SQL start time, and status change time of the event: + + ```sql + mogdb=# SELECT backend_start,xact_start,query_start,state_change FROM pg_stat_activity; + ``` + +- Query the number of sessions on the current server. + + ```sql + mogdb=# SELECT count(*) FROM pg_stat_activity; + ``` + +- Query system-level statistics. + + Run the following command to query information about the session that uses the maximum memory: + + ```sql + mogdb=# SELECT * FROM pv_session_memory_detail() ORDER BY usedsize desc limit 10; + ``` + +## Checking Objects + +Tables, indexes, partitions, and constraints are key storage objects of a database. A database administrator needs to routinely maintain key information and these objects. + +- View table details. + + ```sql + mogdb=# \d+ table_name + ``` + +- Query table statistics. + + ```sql + mogdb=# SELECT * FROM pg_statistic; + ``` + +- View index details. + + ```sql + mogdb=# \d+ index_name + ``` + +- Query partitioned table information. + + ```sql + mogdb=# SELECT * FROM pg_partition; + ``` + +- Collect statistics. + + Run the **ANALYZE** statement to collect related statistics on the database. + + Run the **VACUUM** statement to reclaim space and update statistics. + +- Query constraint information. + + ```sql + mogdb=# SELECT * FROM pg_constraint; + ``` + +## Checking an SQL Report + +Run the **EXPLAIN** statement to view execution plans. + +## Backing Up Data + +Never forget to back up data. During the routine work, the backup execution and backup data validity need to be checked to ensure data security and encryption security. + +- Export a specified user. + + ```bash + gs_dump dbname -p port -f out.sql -U user_name -W password + ``` + +- Export a schema. + + ```bash + gs_dump dbname -p port -n schema_name -f out.sql + ``` + +- Export a table. + + ```bash + gs_dump dbname -p port -t table_name -f out.sql + ``` + +## Checking Basic Information + +Basic information includes versions, components, and patches. Periodic database information checks and records are important for database life cycle management. + +- Check version information. + + ```sql + mogdb=# SELECT version(); + ``` + +- Check table size and database size. + + ```sql + mogdb=# SELECT pg_table_size('table_name'); + mogdb=# SELECT pg_database_size('database_name'); + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/10-data-security-maintenance-suggestions.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/10-data-security-maintenance-suggestions.md new file mode 100644 index 0000000000000000000000000000000000000000..9b041693f88c4877b5cdf645abc0fd555d3d60bb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/10-data-security-maintenance-suggestions.md @@ -0,0 +1,29 @@ +--- +title: 日常运维 +summary: 日常运维 +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Data Security Maintenance Suggestions + +To ensure data security in MogDB Kernel and prevent accidents, such as data loss and illegal data access, read this section carefully. + +**Preventing Data Loss** + +You are advised to plan routine physical backup and store backup files in a reliable medium. If a serious error occurs in the system, you can use the backup files to restore the system to the state at the backup point. + +**Preventing Illegal Data Access** + +- You are advised to manage database users based on their permission hierarchies. A database administrator creates users and grants permissions to the users based on service requirements to ensure users properly access the database. +- You are advised to deploy MogDB Kernel servers and clients (or applications developed based on the client library) in trusted internal networks. If the servers and clients must be deployed in an untrusted network, enable SSL encryption before services are started to ensure data transmission security. Note that enabling the SSL encryption function compromises database performance. + +**Preventing System Logs from Leaking Personal Data** + +- Delete personal data before sending debug logs to others for analysis. + + **NOTE:** The log level **log_min_messages** is set to **DEBUG**x (*x* indicates the debug level and the value ranges from 1 to 5). The information recorded in debug logs may contain personal data. + +- Delete personal data before sending system logs to others for analysis. If the execution of a SQL statement fails, the error SQL statement will be recorded in a system log by default. SQL statements may contain personal data. + +- Set **log_min_error_statement** to **PANIC** to prevent error SQL statements from being recorded in system logs. If this function is disabled, it is difficult to locate fault causes when a fault occurs. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/11-log-reference.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/11-log-reference.md new file mode 100644 index 0000000000000000000000000000000000000000..0fc3d231b205d59ca55e883ee9164cc546662b5d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/11-log-reference.md @@ -0,0 +1,152 @@ +--- +title: Log Reference +summary: Log Reference +author: Guo Huan +date: 2021-06-24 +--- + +# Log Reference + +## Log Overview + +During database running, a large number of logs are generated, including write-ahead logs (WALs, also called Xlogs) for ensuring database security and reliability and run logs and operation logs for daily database maintenance. If the database is faulty, you can refer to these logs to locate the fault and restore the database. + +**Log Type** + +The following table describes details about log types. + +**Table 1** Log types + +| Type | Description | +| :-------------- | :----------------------------------------------------------- | +| System log | Logs generated during database running. They are used to record abnormal process information. | +| Operation log | Logs generated when a client tool (such as **gs_guc**) is operating databases. | +| Trace log | Logs generated after the database debug switch is enabled. They are used to analyze database exceptions. | +| Black box log | Logs generated when the database system breaks down. You can analyze the process context when the fault occurs based on the heap and stack information in the logs to facilitate fault locating. A black box dumps stack, heap, and register information about processes and threads when a system breaks down. | +| Audit log | Logs used to record some of the database user operations after the database audit function is enabled. | +| WAL | Logs used to restore a damaged database. They are also called redo logs. You are advised to routinely back up WALs. | +| Performance log | Logs used to record the status of physical resources and the performance of access to external resources (such as disks, OBS and Hadoop clusters). | + +## System Logs + +System logs include those generated by database nodes when MogDB is running, and those generated when MogDB is deployed. If an error occurs during MogDB running, you can locate the cause and troubleshoot it based on system logs. + +**Log Storage Directory** + +Run logs of database nodes are stored in the corresponding folders in the **/var/log/mogdb/username/pg_log** directory. + +Logs generated during OM MogDB installation and uninstallation are stored in the **/var/log/mogdb/username/om** directory. + +**Log Naming Rules** + +- The name format of database node run logs is: + +postgresql-creation time.log + +By default, a new log file is generated at 0:00 every day, or when the size of the latest log file exceeds 16 MB or a database instance (database node) is restarted. + +- The name formats of CM run logs are: + - cm_agent logs: cm_agent-creation time.log, cm_agent-creation time-current.log, system_call-creation time.log, and system_call-creation time-current.log + - cm_server logs: cm_server-creation time.log, cm_server creation time-current.log, key_event-creation time.log, and key_event-creation time-current.log + - om_monitor logs: om_monitor-creation time.log and om_monitor-creation time-current.log. + +Logs whose names do not contain **current** are historical log files. Logs whose names contain **current** are current log files. When a process is invoked for the first time, a log file whose name contains **current** is created. If the size of this file exceeds 16 MB, the file is renamed in the historical log file name format, and a new log file is generated at the current time point. + +**Log Content Description** + +- Content of a line in a database node log: + +Date+Time+Time zone+Username+Database name+Session ID+Log level+Log content. + +- By default, a line in a cm_agent, cm_server, om_monitor log is arranged in the following format: + +Time+Time zone+Session ID+Log content + +The **SYSTEM_CALL** log records tool commands invoked by cm_agent. + +By default, a line in a key_event log is arranged in the following format: Time+Thread ID+Thread name:Key event type+Arbitration object instance ID+Arbitration details. + +## Operation Logs + +Operation logs are generated when database tools are used by a database administrator or invoked by a cluster. If the cluster is faulty, you can backtrack user operations on the database and reproduce the fault based on the operation logs. + +**Log Storage Directory** + +The default path is **$GAUSSLOG/bin**. If the environmental variable **$GAUSSLOG** does not exist or its value is empty, the log information generated for a tool will be displayed, but not recorded in the log file of the tool. + +The default value of **$GAUSSLOG** is **/var/log/mogdb/username**. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** If a database is deployed using the OM script, the log path is **/var/log/mogdb/username**. + +**Log Naming Rules** + +The log file name format is as follows: + +- **tool name-log creation time.log** +- **tool name-log creation time-current.log** + +**tool name-log creation time.log** is a historical log file, and **tool name-log creation time-current.log** is a current log file. + +If the size of a log file exceeds 16 MB, the next time the tool is invoked, the log file is renamed in the historical log file name format, and a new log file is generated at the current time point. + +For example, **gs_guc-2015-01-16_183728-current.log** is renamed as **gs_guc-2015-01-16_183728.log**, and **gs_guc-2015-01-17_142216-current.log** is generated. + +**Maintenance Suggestions** + +You are advised to dump expired logs periodically to save disk space and prevent important logs from being lost. + +## Audit Logs + +After the audit function is enabled, a large number of audit logs will be generated, which occupy large storage space. You can customize an audit log maintenance policy based on the size of available storage space. + +For details, see “Configuring Database Audit > Maintaining Audit Logs” in the *Security Guide*. + +## WALs + +In a system using write-ahead logs (WALs or Xlogs), all data file modifications are written to a log before they are applied. That is, the corresponding log must be written into a permanent memory before a data file is modified. You can use WALs to restore the cluster if the system crashes. + +**Log Storage Directory** + +Take a DN as an example. Its WALs are stored in the **/mogdb/data/data_dn/pg_xlog** directory. + +**/mogdb/data/data_dn** is the data directory of a node in the cluster. + +**Log Naming Rules** + +Log files are saved as segment files. Each segment is 16 MB and is divided into multiple 8 KB pages. The name of a WAL file consists of 24 hexadecimal characters. Each name has three parts, with each part having eight hexadecimal characters. The first part indicates the time line, the second part indicates the log file identifier, and the third part indicates the file segment identifier. A time line starts from 1, and a log file identifier and a file segment identifier start from 0. + +For example, the name of the first transaction log is **000000010000000000000000**. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The numbers in each part are used in ascending order in succession. Exhausting all available numbers takes a long time, and the numbers will start from zero again after they reach the maximum. + +**Log Content Description** + +The content of WALs depends on the types of recorded transactions. WALs can be used to restore a system after the system breaks down. + +By default, MogDB reads WALs for system restoration during each startup. + +**Maintenance Suggestions** + +WALs are important for database restoration. You are advised to routinely back up WALs. + +## Performance Logs + +Performance logs focus on the access performance of external resources. Performance logs are used to record the status of physical resources and the performance of access to external resources (such as disks, OBS and Hadoop clusters). When a performance issue occurs, you can locate the cause using performance logs, which greatly improves troubleshooting efficiency. + +**Log Storage Directory** + +The performance logs of database are stored in the directories under **$GAUSSLOG/gs_profile**. + +**Log Naming Rules** + +The name format ofdatabase performance logs is: + +**postgresql-creation time.prf** + +By default, a new log file is generated at 0:00 every day, or when the latest log file exceeds 20 MB or a database instance (CN or DN) is restarted. + +**Log Content Description** + +Content of a line in a database log: + +**Host name+Date+Time+Instance name+Thread number+Log content** diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/2-checking-os-parameters.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/2-checking-os-parameters.md new file mode 100644 index 0000000000000000000000000000000000000000..55c064d641e3facab04d9eee92125264a9034fe1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/2-checking-os-parameters.md @@ -0,0 +1,178 @@ +--- +title: 日常运维 +summary: 日常运维 +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Checking OS Parameters + +## Check Method + +Use the **gs_checkos** tool provided by MogDB to check the OS status. + +**Prerequisites** + +- The hardware and network are working properly. +- The trust relationship of user **root** among the hosts is normal. +- Only user **root** is authorized to run the **gs_checkos** command. + +**Procedure** + +1. Log in to a server as user **root**. + +2. Run the following command to check OS parameters of servers where the MogDB nodes are deployed: + + ``` + gs_checkos -i A + ``` + + The purpose of checking the OS parameters is to ensure that MogDB is preinstalled properly and can efficiently operate after it is installed. + +**Examples** + +Before running the **gs_checkos** command, execute pre-processing scripts by running **gs_preinstall** to prepare the environment. The following uses parameter **A** as an example: + +``` +gs_checkos -i A +Checking items: + A1. [ OS version status ] : Normal + A2. [ Kernel version status ] : Normal + A3. [ Unicode status ] : Normal + A4. [ Time zone status ] : Normal + A5. [ Swap memory status ] : Normal + A6. [ System control parameters status ] : Normal + A7. [ File system configuration status ] : Normal + A8. [ Disk configuration status ] : Normal + A9. [ Pre-read block size status ] : Normal + A10.[ IO scheduler status ] : Normal + A11.[ Network card configuration status ] : Normal + A12.[ Time consistency status ] : Warning + A13.[ Firewall service status ] : Normal + A14.[ THP service status ] : Normal +Total numbers:14. Abnormal numbers:0. Warning number:1. +``` + +The following uses parameter **B** as an example: + +``` +gs_checkos -i B +Setting items: + B1. [ Set system control parameters ] : Normal + B2. [ Set file system configuration value ] : Normal + B3. [ Set pre-read block size value ] : Normal + B4. [ Set IO scheduler value ] : Normal + B5. [ Set network card configuration value ] : Normal + B6. [ Set THP service ] : Normal + B7. [ Set RemoveIPC value ] : Normal + B8. [ Set Session Process ] : Normal +Total numbers:6. Abnormal numbers:0. Warning number:0. +``` + +## Exception Handling + +If you use the **gs_checkos** tool to check the OS and the command output shows **Abnormal**, run the following command to view detailed error information: + +``` +gs_checkos -i A --detail +``` + +The **Abnormal** state cannot be ignored because the OS in this state affects cluster installation. The **Warning** state does not affect cluster installation and thereby can be ignored. + +- If the check result for OS version status (**A1**) is **Abnormal**, replace OSs out of the mixed programming scope with those within the scope. + +- If the check result for kernel version status (**A2**) is **Warning**, the platform kernel versions in the cluster are inconsistent. + +- If the check result for Unicode status (**A3**) is **Abnormal**, set the same character set for all the hosts. You can add **export LANG=***unicode* to the **/etc/profile** file. + + ``` + vim /etc/profile + ``` + +- If the check result for time zone status (**A4**) is **Abnormal**, set the same time zone for all the hosts. You can copy the time zone file in the **/usr/share/zoneinfo/** directory as the **/etc/localtime** file. + + ``` + cp /usr/share/zoneinfo/$primary time zone/$secondary time zone /etc/localtime + ``` + +- If the check result for swap memory status (**A5**) is **Abnormal**, a possible cause is that the swap memory is larger than the physical memory. You can troubleshoot this issue by reducing the swap memory or increasing the physical memory. + +- If the check result for system control parameter status (**A6**) is **Abnormal**, troubleshoot this issue in either of the following two ways: + + - Run the following command: + + ``` + gs_checkos -i B1 + ``` + + - Modify the **/etc/sysctl.conf** file based on the error message and run **sysctl -p** to make it take effect. + + ``` + vim /etc/sysctl.conf + ``` + +- If the check result for file system configuration status (**A7**) is **Abnormal**, run the following command to troubleshoot this issue: + + ``` + gs_checkos -i B2 + ``` + +- If the check result for disk configuration status (**A8**) is **Abnormal**, set the disk mounting format to **rw,noatime,inode64,allocsize=16m**. + + Run the **man mount** command to mount the XFS parameter: + + ``` + rw,noatime,inode64,allocsize=16m + ``` + + You can also set the XFS parameter in the **/etc/fstab** file. For example: + + ``` + /dev/data /data xfs rw,noatime,inode64,allocsize=16m 0 0 + ``` + +- If the check result for pre-read block size status (**A9**) is **Abnormal**, run the following command to troubleshoot this issue: + + ``` + gs_checkos -i B3 + ``` + +- If the check result for I/O scheduling status (**A10**) is **Abnormal**, run the following command to troubleshoot this issue: + + ``` + gs_checkos -i B4 + ``` + +- If the check result for NIC configuration status (**A11**) is **Warning**, run the following command to troubleshoot this issue: + + ``` + gs_checkos -i B5 + ``` + +- If the check result for time consistency status (**A12**) is **Abnormal**, verify that the NTP service has been installed and started and has synchronized time from the NTP clock. + +- If the check result for firewall status (**A13**) is **Abnormal**, disable the firewall. Run the following commands: + + - SUSE: + + ``` + SuSEfirewall2 stop + ``` + + - RedHat7: + + ``` + systemctl disable firewalld + ``` + + - RedHat6: + + ``` + service iptables stop + ``` + +- If the check result for THP service status (**A14**) is **Abnormal**, run the following command to troubleshoot this issue: + + ``` + gs_checkos -i B6 + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/3-checking-mogdb-health-status.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/3-checking-mogdb-health-status.md new file mode 100644 index 0000000000000000000000000000000000000000..73f6a5f0f3f17b46c9e6c7065a32374adf73e0c7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/3-checking-mogdb-health-status.md @@ -0,0 +1,645 @@ +--- +title: Checking MogDB Health Status +summary: Checking MogDB Health Status +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Checking MogDB Health Status + +## Check Method + +Use the **gs_check** tool provided by MogDB to check the MogDB health status. + +**Precautions** + +- Only user **root** is authorized to check new nodes added during cluster scale-out. In other cases, the check can be performed only by user **omm**. +- Parameter **-i** or **-e** must be set. **-i** specifies a single item to be checked, and **-e** specifies an inspection scenario where multiple items will be checked. +- If **-i** is not set to a **root** item or no such items are contained in the check item list of the scenario specified by **-e**, you do not need to enter the name or password of user **root**. +- You can run **-skip-root-items** to skip **root** items. +- Check the consistency between the new node and existing nodes. Run the **gs_check** command on an existing node and specify the **-hosts** parameter. The IP address of the new node needs to be written into the **hosts** file. + +**Procedure** + +Method 1: + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to check the MogDB database status: + + ```bash + gs_check -i CheckClusterState + ``` + + In the command, **-i** indicates the check item and is case-sensitive. The format is **-i CheckClusterState**, **-i CheckCPU** or **-i CheckClusterState,CheckCPU**. + + Checkable items are listed in "Table 1 MogDB status checklist" in "Tool Reference > Server Tools > [gs_check](../../reference-guide/tool-reference/server-tools/1-gs_check.md)". You can create a check item as needed. + +Method 2: + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to check the MogDB database health status: + + ```bash + gs_check -e inspect + ``` + + In the command, **-e** indicates the inspection scenario and is case-sensitive. The format is **-e inspect** or **-e upgrade**. + + The inspection scenarios include **inspect** (routine inspection), **upgrade** (inspection before upgrade), **Install** (install inspection ), **binary_upgrade** (inspection before in-place upgrade), **slow_node** (node inspection), **longtime** (time-consuming inspection) and **health** (health inspection). You can create an inspection scenario as needed. + +The MogDB inspection is performed to check MogDB status during MogDB running or to check the environment and conditions before critical operations, such as upgrade or scale-out. For details about the inspection items and scenarios, see "Server Tools > gs_check > MogDB status checks" in the *MogDB Tool Reference*. + +**Examples** + +Check result of a single item: + +```bash +perfadm@lfgp000700749:/opt/huawei/perfadm/tool/script> gs_check -i CheckCPU +Parsing the check items config file successfully +Distribute the context file to remote hosts successfully +Start to health check for the cluster. Total Items:1 Nodes:3 + +Checking... [=========================] 1/1 +Start to analysis the check result +CheckCPU....................................OK +The item run on 3 nodes. success: 3 + +Analysis the check result successfully +Success. All check items run completed. Total:1 Success:1 Failed:0 +For more information please refer to /opt/mogdb/tools/script/gspylib/inspection/output/CheckReport_201902193704661604.tar.gz +``` + +Local execution result: + +```bash +perfadm@lfgp000700749:/opt/huawei/perfadm/tool/script> gs_check -i CheckCPU -L + +2017-12-29 17:09:29 [NAM] CheckCPU +2017-12-29 17:09:29 [STD] Check the CPU usage of the host. If the value of idle is greater than 30% and the value of iowait is less than 30%, this item passes the check. Otherwise, this item fails the check. +2017-12-29 17:09:29 [RST] OK + +2017-12-29 17:09:29 [RAW] +Linux 4.4.21-69-default (lfgp000700749) 12/29/17 _x86_64_ + +17:09:24 CPU %user %nice %system %iowait %steal %idle +17:09:25 all 0.25 0.00 0.25 0.00 0.00 99.50 +17:09:26 all 0.25 0.00 0.13 0.00 0.00 99.62 +17:09:27 all 0.25 0.00 0.25 0.13 0.00 99.37 +17:09:28 all 0.38 0.00 0.25 0.00 0.13 99.25 +17:09:29 all 1.00 0.00 0.88 0.00 0.00 98.12 +Average: all 0.43 0.00 0.35 0.03 0.03 99.17 +``` + +Check result of a scenario: + +```bash +[perfadm@SIA1000131072 Check]$ gs_check -e inspect +Parsing the check items config file successfully +The below items require root privileges to execute:[CheckBlockdev CheckIOrequestqueue CheckIOConfigure CheckCheckMultiQueue CheckFirewall CheckSshdService CheckSshdConfig CheckCrondService CheckBootItems CheckFilehandle CheckNICModel CheckDropCache] +Please enter root privileges user[root]:root +Please enter password for user[root]: +Please enter password for user[root] on the node[10.244.57.240]: +Check root password connection successfully +Distribute the context file to remote hosts successfully +Start to health check for the cluster. Total Items:57 Nodes:2 + +Checking... [ ] 21/57 +Checking... [=========================] 57/57 +Start to analysis the check result +CheckClusterState...........................OK +The item run on 2 nodes. success: 2 + +CheckDBParams...............................OK +The item run on 1 nodes. success: 1 + +CheckDebugSwitch............................OK +The item run on 2 nodes. success: 2 + +CheckDirPermissions.........................OK +The item run on 2 nodes. success: 2 + +CheckReadonlyMode...........................OK +The item run on 1 nodes. success: 1 + +CheckEnvProfile.............................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +GAUSSHOME /usr1/mogdb/app +LD_LIBRARY_PATH /usr1/mogdb/app/lib +PATH /usr1/mogdb/app/bin + + +CheckBlockdev...............................OK +The item run on 2 nodes. success: 2 + +CheckCurConnCount...........................OK +The item run on 1 nodes. success: 1 + +CheckCursorNum..............................OK +The item run on 1 nodes. success: 1 + +CheckPgxcgroup..............................OK +The item run on 1 nodes. success: 1 + +CheckDiskFormat.............................OK +The item run on 2 nodes. success: 2 + +CheckSpaceUsage.............................OK +The item run on 2 nodes. success: 2 + +CheckInodeUsage.............................OK +The item run on 2 nodes. success: 2 + +CheckSwapMemory.............................OK +The item run on 2 nodes. success: 2 + +CheckLogicalBlock...........................OK +The item run on 2 nodes. success: 2 + +CheckIOrequestqueue.....................WARNING +The item run on 2 nodes. warning: 2 +The warning[host240,host157] value: +On device (vdb) 'IO Request' RealValue '256' ExpectedValue '32768' +On device (vda) 'IO Request' RealValue '256' ExpectedValue '32768' + +CheckMaxAsyIOrequests.......................OK +The item run on 2 nodes. success: 2 + +CheckIOConfigure............................OK +The item run on 2 nodes. success: 2 + +CheckMTU....................................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +1500 + +CheckPing...................................OK +The item run on 2 nodes. success: 2 + +CheckRXTX...................................NG +The item run on 2 nodes. ng: 2 +The ng[host240,host157] value: +NetWork[eth0] +RX: 256 +TX: 256 + + +CheckNetWorkDrop............................OK +The item run on 2 nodes. success: 2 + +CheckMultiQueue.............................OK +The item run on 2 nodes. success: 2 + +CheckEncoding...............................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +LANG=en_US.UTF-8 + +CheckFirewall...............................OK +The item run on 2 nodes. success: 2 + +CheckKernelVer..............................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +3.10.0-957.el7.x86_64 + +CheckMaxHandle..............................OK +The item run on 2 nodes. success: 2 + +CheckNTPD...................................OK +host240: NTPD service is running, 2020-06-02 17:00:28 +host157: NTPD service is running, 2020-06-02 17:00:06 + + +CheckOSVer..................................OK +host240: The current OS is centos 7.6 64bit. +host157: The current OS is centos 7.6 64bit. + +CheckSysParams..........................WARNING +The item run on 2 nodes. warning: 2 +The warning[host240,host157] value: +Warning reason: variable 'net.ipv4.tcp_retries1' RealValue '3' ExpectedValue '5'. +Warning reason: variable 'net.ipv4.tcp_syn_retries' RealValue '6' ExpectedValue '5'. + +CheckTHP....................................OK +The item run on 2 nodes. success: 2 + +CheckTimeZone...............................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: ++0800 + +CheckCPU....................................OK +The item run on 2 nodes. success: 2 + +CheckSshdService............................OK +The item run on 2 nodes. success: 2 + +Warning reason: UseDNS parameter is not set; expected: no + +CheckCrondService...........................OK +The item run on 2 nodes. success: 2 + +CheckStack..................................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +8192 + +CheckSysPortRange...........................OK +The item run on 2 nodes. success: 2 + +CheckMemInfo................................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +totalMem: 31.260929107666016G + +CheckHyperThread............................OK +The item run on 2 nodes. success: 2 + +CheckTableSpace.............................OK +The item run on 1 nodes. success: 1 + +CheckSysadminUser...........................OK +The item run on 1 nodes. success: 1 + + +CheckGUCConsistent..........................OK +All DN instance guc value is consistent. + +CheckMaxProcMemory..........................OK +The item run on 1 nodes. success: 1 + +CheckBootItems..............................OK +The item run on 2 nodes. success: 2 + +CheckHashIndex..............................OK +The item run on 1 nodes. success: 1 + +CheckPgxcRedistb............................OK +The item run on 1 nodes. success: 1 + +CheckNodeGroupName..........................OK +The item run on 1 nodes. success: 1 + +CheckTDDate.................................OK +The item run on 1 nodes. success: 1 + +CheckDilateSysTab...........................OK +The item run on 1 nodes. success: 1 + +CheckKeyProAdj..............................OK +The item run on 2 nodes. success: 2 + +CheckProStartTime.......................WARNING +host157: +STARTED COMMAND +Tue Jun 2 16:57:18 2020 /usr1/dmuser/dmserver/metricdb1/server/bin/mogdb --single_node -D /usr1/dmuser/dmb1/data -p 22204 +Mon Jun 1 16:15:15 2020 /usr1/mogdb/app/bin/mogdb -D /usr1/mogdb/data/dn1 -M standby + + +CheckFilehandle.............................OK +The item run on 2 nodes. success: 2 + +CheckRouting................................OK +The item run on 2 nodes. success: 2 + +CheckNICModel...............................OK +The item run on 2 nodes. success: 2 (consistent) +The success on all nodes value: +version: 1.0.1 +model: Red Hat, Inc. Virtio network device + + +CheckDropCache..........................WARNING +The item run on 2 nodes. warning: 2 +The warning[host240,host157] value: +No DropCache process is running + +CheckMpprcFile..............................NG +The item run on 2 nodes. ng: 2 +The ng[host240,host157] value: +There is no mpprc file + +Analysis the check result successfully +Failed. All check items run completed. Total:57 Success:50 Warning:5 NG:2 +For more information please refer to /usr1/mogdb/tool/script/gspylib/inspection/output/CheckReport_inspect611.tar.gz +``` + +## Exception Handling + +Troubleshoot exceptions detected in the inspection by following instructions in this section. + +**Table 1** Check of MogDB running status + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Check Item Abnormal Status Solution
CheckClusterState (Checks the MogDB status.) MogDB or MogDB instances are not started. Run the following command to start MogDB and instances:

gs_om -t start
The status of MogDB or MogDB instances is abnormal. Check the status of hosts and instances. Troubleshoot this issue based on the status information.
gs_check -i CheckClusterState
CheckDBParams (Checks database parameters.) Database parameters have incorrect values. Use the gs_guc tool to set the parameters to specified values.
CheckDebugSwitch (Checks debug logs.) The log level is incorrect. Use the gs_guc tool to set log_min_messages to specified content.
CheckDirPermissions (Checks directory permissions.) The permission for a directory is incorrect. Change the directory permission to a specified value (750 or 700).
chmod 750 DIR
CheckReadonlyMode (Checks the read-only mode.) The read-only mode is enabled. Verify that the usage of the disk where database nodes are located does not exceed the threshold (60% by default) and no other O&M operations are performed.
gs_check -i CheckDataDiskUsage ps ux
Use the gs_guc tool to disable the read-only mode of MogDB.
gs_guc reload -N all -I all -c 'default_transaction_read_only = off'
CheckEnvProfile (Checks environment variables.) Environment variables are inconsistent. Update the environment variable information.
CheckBlockdev (Checks pre-read blocks.) The size of a pre-read block is not 16384 KB. Use the gs_checkos tool to set the size of the pre-read block to 16384 KB and write the setting into the auto-startup file.
gs_checkos -i B3
CheckCursorNum (Checks the number of cursors.) The number of cursors fails to be checked. Check whether the database is properly connected and whether the MogDB status is normal.
CheckPgxcgroup (Checks the data redistribution status.) There are pgxc_group tables that have not been redistributed. Proceed with the redistribution.
gs_expand、gs_shrink
CheckDiskFormat (Checks disk configurations.) Disk configurations are inconsistent between nodes. Configure disk specifications to be consistent between nodes.
CheckSpaceUsage (Checks the disk space usage.) Disk space is insufficient. Clear or expand the disk for the directory.
CheckInodeUsage (Checks the disk index usage.) Disk indexes are insufficient. Clear or expand the disk for the directory.
CheckSwapMemory (Checks the swap memory.) The swap memory is greater than the physical memory. Reduce or disable the swap memory.
CheckLogicalBlock (Checks logical blocks.) The size of a logical block is not 512 KB. Use the gs_checkos tool to set the size of the logical block to 512 KB and write the setting into the auto-startup file.
gs_checkos -i B4
CheckIOrequestqueue (Checks I/O requests.) The requested I/O is not 32768. Use the gs_checkos tool to set the requested I/O to 32768 and write the setting into the auto-startup file.
gs_checkos -i B4
CheckCurConnCount (Checks the number of current connections.) The number of current connections exceeds 90% of the allowed maximum number of connections. Break idle primary database node connections.
CheckMaxAsyIOrequests (Checks the maximum number of asynchronous requests.) The maximum number of asynchronous requests is less than 104857600 or (Number of database instances on the current node x 1048576). Use the gs_checkos tool to set the maximum number of asynchronous requests to the larger one between 104857600 and (Number of database instances on the current node x 1048576).
gs_checkos -i B4
CheckMTU (Checks MTU values.) MTU values are inconsistent between nodes. Set the MTU value on each node to 1500 or 8192.
ifconfig eth* MTU 1500
CheckIOConfigure (Checks I/O configurations.) The I/O mode is not deadline. Use the gs_checkos tool to set the I/O mode to deadline and write the setting into the auto-startup file.
gs_checkos -i B4
CheckRXTX (Checks the RX/TX value.) The NIC RX/TX value is not 4096. Use the checkos tool to set the NIC RX/TX value to 4096 for MogDB.
gs_checkos -i B5
CheckPing (Checks whether the network connection is normal.) There are MogDB IP addresses that cannot be pinged. Check the network settings, network status, and firewall status between the abnormal IP addresses.
CheckNetWorkDrop (Checks the network packet loss rate.) The network packet loss rate is greater than 1%. Check the network load and status between the corresponding IP addresses.
CheckMultiQueue (Checks the NIC multi-queue function.) Multiqueue is not enabled for the NIC, and NIC interruptions are not bound to different CPU cores. Enable multiqueue for the NIC, and bind NIC interruptions to different CPU cores.
CheckEncoding (Checks the encoding format.) Encoding formats are inconsistent between nodes. Write the same encoding format into /etc/profile for each node.
echo "export LANG=XXX" >> /etc/profile
CheckActQryCount (Checks the archiving mode.) The archiving mode is enabled, and the archiving directory is not under the primary database node directory. Disable archiving mode or set the archiving directory to be under the primary database node directory.
CheckFirewall (Checks the firewall.) The firewall is enabled. Disable the firewall.
systemctl disable firewalld.service
CheckKernelVer (Checks kernel versions.) Kernel versions are inconsistent between nodes.
CheckMaxHandle (Checks the maximum number of file handles.) The maximum number of handles is less than 1000000. Set the soft and hard limits in the 91-nofile.conf or 90-nofile.conf file to 1000000.
gs_checkos -i B2
CheckNTPD (Checks the time synchronization service.) The NTPD service is disabled or the time difference is greater than 1 minute. Enable the NTPD service and set the time to be consistent.
CheckSysParams (Checks OS parameters.) OS parameter settings do not meet requirements. Use the gs_checkos tool or manually set parameters to values meeting requirements.
gs_checkos -i B1 vim /etc/sysctl.conf
CheckTHP (Checks the THP service.) The THP service is disabled. Use the gs_checkos to enable the THP service.
gs_checkos -i B6
CheckTimeZone (Checks time zones.) Time zones are inconsistent between nodes. Set time zones to be consistent between nodes.
cp /usr/share/zoneinfo/\$primary time zone/$secondary time zone\ /etc/localtime
CheckCPU (Checks the CPU.) CPU usage is high or I/O waiting time is too long. Upgrade CPUs or improve disk performance.
CheckSshdService (Checks the SSHD service.) The SSHD service is disabled. Enable the SSHD service and write the setting into the auto-startup file.
service sshd start echo "server sshd start" >> initFile
CheckSshdConfig (Checks SSHD configurations.) The SSHD service is incorrectly configured. Reconfigure the SSHD service.
PasswordAuthentication=no; MaxStartups=1000; UseDNS=yes; ClientAliveInterval=10800/ClientAliveInterval=0
Restart the service.
server sshd start
CheckCrondService (Checks the Crond service.) The Crond service is disabled. Install and enable the Crond service.
CheckStack (Checks the stack size.) The stack size is less than 3072. Use the gs_checkos tool to set the stack size to 3072 and restart the processes with a smaller stack size.
gs_checkos -i B2
CheckSysPortRange (Checks OS port configurations.) OS IP ports are not within the required port range or MogDB ports are within the OS IP port range. Set the OS IP ports within 26000 to 65535 and set the MogDB ports beyond the OS IP port range.
vim /etc/sysctl.conf
CheckMemInfo (Checks the memory information.) Memory sizes are inconsistent between nodes. Use physical memory of the same specifications between nodes.
CheckHyperThread (Checks the hyper-threading.) The CPU hyper-threading is disabled. Enable the CPU hyper-threading.
CheckTableSpace (Checks tablespaces.) The tablespace path is nested with the MogDB path or nested with the path of another tablespace. Migrate tablespace data to the tablespace with a valid path.
+ +## Querying Status + +### Background + +MogDB allows you to view the status of the entire MogDB. The query result shows whether the database or a single host is running properly. + +### Prerequisites + +The database has started. + +### Procedure + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to query the database status: + + ```bash + $ gs_om -t status --detail + ``` + + Table 1 describes parameters in the query result. + + To query the instance status on a host, add **-h** to the command. For example: + + ```bash + $ gs_om -t status -h plat2 + ``` + + **plat2** indicates the name of the host to be queried. + +### Parameter Description + +**Table 1** Node role description + +| Field | Description | Value | +| :------------ | :----------------------------------------------------------- | :----------------------------------------------------------- | +| cluster_state | The database status, which indicates whether the entire database is running properly. | **Normal**: The database is available and the data has redundancy backup. All the processes are running and the primary/standby relationship is normal.**Unavailable**: The database is unavailable.**Degraded**: The database is available and faulty database nodes and primary database nodes exist. | +| node | Host name. | Specifies the name of the host where the instance is located. If multiple AZs exist, the AZ IDs will be displayed. | +| node_ip | Host IP Address. | Specifies the IP address of the host where the instance is located. | +| instance | Instance ID. | Specifies the instance ID. | +| state | Instance role | **Normal**: a single host instance.**Primary**: The instance is a primary instance.**Standby**: The instance is a standby instance.**Cascade Standby**: The instance is a cascaded standby instance.**Secondary**: The instance is a secondary instance.**Pending**: The instance is in the quorum phase.**Unknown**: The instance status is unknown.**Down**: The instance is down.**Abnormal**: The node is abnormal.**Manually stopped**: The node has been manually stopped. | + +Each role has different states, such as startup and connection. The states are described as follows: + +**Table 2** Node state description + +| State | Description | +| :------------- | :----------------------------------------------------------- | +| Normal | The node starts up normally. | +| Need repair | The node needs to be restored. | +| Starting | The node is starting up. | +| Wait promoting | The node is waiting for upgrade. For example, after the standby node sends an upgrade request to the primary node, the standby node is waiting for the response from the primary node. | +| Promoting | The standby node is being upgraded to the primary node. | +| Demoting | The node is being downgraded, for example, the primary node is being downgraded to the standby node. | +| Building | The standby node fails to be started and needs to be rebuilt. | +| Catchup | The standby node is catching up with the primary node. | +| Coredump | The node program breaks down. | +| Unknown | The node status is unknown. | + +If a node is in **Need repair** state, you need to rebuild the node to restore it. Generally, the reasons for rebuilding a node are as follows: + +**Table 3** Node rebuilding causes + +| State | Description | +| :-------------------- | :----------------------------------------------------------- | +| Normal | The node starts up normally. | +| WAL segment removed | WALs of the primary node do not exist, and logs of the standby node are later than those of the primary node. | +| Disconnect | Standby node cannot be connected to the primary node. | +| Version not matched | The binary versions of the primary and standby nodes are inconsistent. | +| Mode not matched | Nodes do not match the primary and standby roles. For example, two standby nodes are connected. | +| System id not matched | The database system IDs of the primary and standby nodes are inconsistent. The system IDs of the primary and standby nodes must be the same. | +| Timeline not matched | The log timelines are inconsistent. | +| Unknown | Unknown cause. | + +## Examples + +View the database status details, including instance status. + +```bash +$ gs_om -t status --detail +[ Cluster State ] + +cluster_state : Normal +redistributing : No +current_az : AZ_ALL + +[ Datanode State ] + + node node_ip port instance state +----------------------------------------------------------------------------------------------------- +1 pekpopgsci00235 10.244.62.204 5432 6001 /opt/mogdb/cluster/data/dn1 P Primary Normal +2 pekpopgsci00238 10.244.61.81 5432 6002 /opt/mogdb/cluster/data/dn1 S Standby Normal +``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/4-checking-database-performance.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/4-checking-database-performance.md new file mode 100644 index 0000000000000000000000000000000000000000..9d03147e3d06e426a6ce2b8a3993e6bcdebae233 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/4-checking-database-performance.md @@ -0,0 +1,83 @@ +--- +title: 日常运维 +summary: 日常运维 +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Checking Database Performance + +## Check Method + +Use the **gs_checkperf** tool provided by MogDB to check hardware performance. + +**Prerequisites** + +- MogDB is running properly. +- Services are running properly on the database. + +**Procedure** + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to check the MogDB database performance: + + ``` + gs_checkperf + ``` + +For details about performance statistical items, see "Table 1 Performance check items" in "Tool Reference > Server Tools > [gs_checkperf](../../reference-guide/tool-reference/server-tools/3-gs_checkperf.md)". + +**Examples** + +Simple performance statistical result is displayed on the screen as follows: + +``` +gs_checkperf -i pmk -U omm +Cluster statistics information: + Host CPU busy time ratio : 1.43 % + MPPDB CPU time % in busy time : 1.88 % + Shared Buffer Hit ratio : 99.96 % + In-memory sort ratio : 100.00 % + Physical Reads : 4 + Physical Writes : 25 + DB size : 70 MB + Total Physical writes : 25 + Active SQL count : 2 + Session count : 3 +``` + +## Exception Handling + +After you use the **gs_checkperf** tool to check the cluster performance, if the performance is abnormal, troubleshoot the issue by following instructions in this section. + +**Table 1** Cluster-level performance status + +| Abnormal Status | Solution | +| ---------------------------------- | ------------------------------------------------------------ | +| High CPU usage of hosts | 1. Add high-performance CPUs, or replace current CPUs with them.2. Run the **top** command to check which system processes cause high CPU usage, and run the **kill** command to stop unused processes.
`top` | +| High CPU usage of MogDB Kernel | 1. Add high-performance CPUs, or replace current CPUs with them.
2. Run the **top** command to check which database processes cause high CPU usage, and run the **kill** command to stop unused processes.
`top`
3. Use the **gs_expand** tool to add new hosts to lower the CPU usage. | +| Low hit ratio of the shared memory | 1. Expand the memory.
2. Run the following command to check the OS configuration file **/etc/sysctl.conf** and increase the value of **kernel.shmmax**.
`vim /etc/sysctl.conf` | +| Low in-memory sort ratio | Expand the memory. | +| High I/O and disk usage | 1. Replace current disks with high-performance ones.
2. Adjust the data layout to evenly distribute I/O requests to all the physical disks.
3. Run **VACUUM FULL** for the entire database.
`vacuum full;`
4. Clean up the disk space.
5. Reduce the number of concurrent connections. | +| Transaction statistics | Query the **pg_stat_activity** system catalog and disconnect unnecessary connections. (Log in to the database and run the **mogdb=# \d+ pg_stat_activity;** command.) | + +**Table 2** Node-level performance status + +| Abnormal Status | Solution | +| ----------------- | ------------------------------------------------------------ | +| High CPU usage | 1. Add high-performance CPUs, or replace current CPUs with them.
2. Run the **top** command to check which system processes cause high CPU usage, and run the **kill** command to stop unused processes.
`top` | +| High memory usage | Expand or clean up the memory. | +| High I/O usage | 1. Replace current disks with high-performance ones.
2. Clean up the disk space.
3. Use memory read/write to replace as much disk I/O as possible, putting frequently accessed files or data in the memory. | + +**Table 3** Session/process-level performance status + +| Abnormal Status | Solution | +| ------------------------------- | ------------------------------------------------------------ | +| High CPU, memory, and I/O usage | Check which processes cause high CPU, memory, or I/O usage. If they are unnecessary processes, kill them; otherwise, analyze the specific cause of high usage. For example, if SQL statement execution occupies much memory, check whether the SQL statements need optimization. | + +**Table 4** SSD performance status + +| Abnormal Status | Solution | +| -------------------- | ------------------------------------------------------------ | +| SSD read/write fault | Run the following command to check whether SSD is faulty. If yes, analyze the specific cause.
`gs_checkperf -i SSD -U omm` | diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/5-checking-and-deleting-logs.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/5-checking-and-deleting-logs.md new file mode 100644 index 0000000000000000000000000000000000000000..92358a20f3da94e394a7aa98ac21cd2daf2bb51d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/5-checking-and-deleting-logs.md @@ -0,0 +1,160 @@ +--- +title: Checking and Deleting Logs +summary: Checking and Deleting Logs +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Checking and Deleting Logs + +You are advised to check OS logs and database run logs monthly for monitoring system status and troubleshooting, and to delete database run logs monthly for saving disk space. + +## Checking OS Logs + +You are advised to monthly check OS logs to detect and prevent potential OS problems. + +**Procedure** + +Run the following command to check OS log files: + +``` +vim /var/log/messages +``` + +(Pay attention to words like **kernel**, **error**, and **fatal** in logs generated within the last month and handle the problems based on the alarm information.) + +## Checking MogDB Run Logs + +A database can still run when errors occur during the execution of some operations. However, data may be inconsistent before and after the error occurrences. Therefore, you are advised to monthly check MogDB run logs to detect potential problems in time. + +**Prerequisites** + +- The host used for collecting logs is running properly, and the network connection is normal. Database installation users trust each other. +- An OS tool (for example, **gstack**) that the log collection tool requires has been installed. If it is not installed, an error message is displayed, and this collection item is skipped. + +**Procedure** + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to collect database logs: + + ``` + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" + ``` + + In the command, **20160616 01:01** indicates the start time of the log and **20160616 23:59** indicates the end time of the log. + +3. Based on command output in [2](#2), access the related log collection directory, decompress collected database logs, and check these logs. + + Assume that collected logs are stored in **/opt/mogdb/tmp/gaussdba_mppdb/collector_20160726_105158.tar.gz**. + + ``` + tar -xvzf /opt/mogdb/tmp/gaussdba_mppdb/collector_20160726_105158.tar.gz + cd /opt/mogdb/tmp/gaussdba_mppdb/collector_20160726_105158 + ``` + +**Examples** + +- Run the **gs_collector** command together with parameters **-begin-time** and **-end-time**: + + ```bash + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" + ``` + + If information similar to the following is displayed, the logs have been archived: + + ``` + Successfully collected files + All results are stored in /tmp/gaussdba_mppdb/collector_20160616_175615.tar.gz. + ``` + +- Run the **gs_collector** command together with parameters **-begin-time**, **-end-time**, and **-h**: + + ```bash + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" -h plat2 + ``` + + If information similar to the following is displayed, the logs have been archived: + + ``` + Successfully collected files + All results are stored in /tmp/gaussdba_mppdb/collector_20160616_190225.tar.gz. + ``` + +- Run the **gs_collector** command together with parameters **-begin-time**, **-end-time**, and **-f**: + + ```bash + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" -f /opt/software/mogdb/output + ``` + + If information similar to the following is displayed, the logs have been archived: + + ``` + Successfully collected files + All results are stored in /opt/software/mogdb/output/collector_20160616_190511.tar.gz. + ``` + +- Run the **gs_collector** command together with parameters **-begin-time**, **-end-time**, and **-keyword**: + + ```bash + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" --keyword="os" + ``` + + If information similar to the following is displayed, the logs have been archived: + + ``` + Successfully collected files. + All results are stored in /tmp/gaussdba_mppdb/collector_20160616_190836.tar.gz. + ``` + +- Run the **gs_collector** command together with parameters **-begin-time**, **-end-time**, and **-o**: + + ```bash + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" -o /opt/software/mogdb/output + ``` + + If information similar to the following is displayed, the logs have been archived: + + ``` + Successfully collected files. + All results are stored in /opt/software/mogdb/output/collector_20160726_113711.tar.gz. + ``` + +- Run the **gs_collector** command together with parameters **-begin-time**, **-end-time**, and **-l** (the file name extension must be .log): + + ```bash + gs_collector --begin-time="20160616 01:01" --end-time="20160616 23:59" -l /opt/software/mogdb/logfile.log + ``` + + If information similar to the following is displayed, the logs have been archived: + + ``` + Successfully collected files. + All results are stored in /opt/software/mogdb/output/collector_20160726_113711.tar.gz. + ``` + +## Cleaning Run Logs + +A large number of run logs will be generated during database running and occupy huge disk space. You are advised to delete expired run logs and retain logs generated within one month. + +**Procedure** + +1. Log in as the OS user **omm** to any host in the MogDB Kernel cluster. + +2. Clean logs. + + a. Back up logs generated over one month ago to other disks. + + b. Access the directory where logs are stored. + + ``` + cd $GAUSSLOG + ``` + + c. Access the corresponding sub-directory and run the following command to delete logs generated one month ago: + + ``` + rm log name + ``` + + The naming convention of a log file is **mogdb-**year*-*month*-*day_**HHMMSS**. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/6-checking-time-consistency.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/6-checking-time-consistency.md new file mode 100644 index 0000000000000000000000000000000000000000..52090c929f44951f2426a79667402e65da2c4e05 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/6-checking-time-consistency.md @@ -0,0 +1,52 @@ +--- +title: Checking Time Consistency +summary: Checking Time Consistency +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Checking Time Consistency + +Database transaction consistency is guaranteed by a logical clock and is not affected by OS time. However, OS time inconsistency will lead to problems, such as abnormal backend O&M and monitoring functions. Therefore, you are advised to monthly check time consistency among nodes. + +**Procedure** + +1. Log in as the OS user **omm** to any host in the MogDB Kernel cluster. + +2. Create a configuration file for recording each cluster node. (You can specify the *mpphosts* file directory randomly. It is recommended that the file be stored in the **/tmp** directory.) + + ```bash + vim /tmp/mpphosts + ``` + + Add the host name of each node. + + ``` + plat1 + plat2 + plat3 + ``` + +3. Save the configuration file. + + ``` + :wq! + ``` + +4. Run the following command and write the time on each node into the **/tmp/sys_ctl-os1.log** file: + + ``` + for ihost in `cat /tmp/mpphosts`; do ssh -n -q $ihost "hostname;date"; done > /tmp/sys_ctl-os1.log + ``` + +5. Check time consistency between the nodes based on the command output. The time difference should not exceed 30s. + + ``` + cat /tmp/sys_ctl-os1.log + plat1 + Thu Feb 9 16:46:38 CST 2017 + plat2 + Thu Feb 9 16:46:49 CST 2017 + plat3 + Thu Feb 9 16:46:14 CST 2017 + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/7-checking-the-number-of-application-connections.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/7-checking-the-number-of-application-connections.md new file mode 100644 index 0000000000000000000000000000000000000000..f5f77f016e8644dfe1729ae93bf5b9de1bab1ba8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/7-checking-the-number-of-application-connections.md @@ -0,0 +1,130 @@ +--- +title: 日常运维 +summary: 日常运维 +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Checking the Number of Application Connections + +If the number of connections between applications and the database exceeds the maximum value, new connections cannot be established. You are advised to daily check the number of connections, release idle connections in time, or increase the allowed maximum number of connections. + +**Procedure** + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Run the following command to connect to the database: + + ``` + gsql -d mogdb -p 8000 + ``` + + **mogdb** is the name of the database to be connected, and **8000** is the port number of the database primary node. + + If information similar to the following is displayed, the connection succeeds: + + ``` + gsql ((MogDB x.x.x build 56189e20) compiled at 2022-01-07 18:47:53 commit 0 last mr ) + Non-SSL connection (SSL connection is recommended when requiring high-security) + Type "help" for help. + + mogdb=# + ``` + +3. Run the following SQL statement to check the number of connections: + + ``` + mogdb=# SELECT count(*) FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s; + ``` + + Information similar to the following is displayed. **2** indicates that two applications are connected to the database. + + ``` + count + ------- + 2 + (1 row) + ``` + +4. View the allowed maximum connections. + + ``` + mogdb=# SHOW max_connections; + ``` + + Information similar to the following is displayed. **200** indicates the currently allowed maximum number of connections. + + ``` + max_connections + ----------------- + 200 + (1 row) + ``` + +## Exception Handling + +If the number of connections in the command output is close to the value of **max_connections** of the database, delete existing connections or change the upper limit based on site requirements. + +1. Run the following SQL statement to view information about connections whose **state** is set to **idle**, and **state_change** column is not updated for a long time. + + ``` + mogdb=# SELECT * FROM pg_stat_activity where state='idle' order by state_change; + ``` + + Information similar to the following is displayed: + + ``` + datid | datname | pid | usesysid | usename | application_name | client_addr + | client_hostname | client_port | backend_start | xact_start | quer + y_start | state_change | waiting | enqueue | state | resource_pool + | query + -------+----------+-----------------+----------+----------+------------------+--------------- + -+-----------------+-------------+-------------------------------+------------+-------------- + -----------------+-------------------------------+---------+---------+-------+--------------- + +---------------------------------------------- + 13626 | mogdb | 140390162233104 | 10 | gaussdba | | + | | -1 | 2016-07-15 14:08:59.474118+08 | | 2016-07-15 14 + :09:04.496769+08 | 2016-07-15 14:09:04.496975+08 | f | | idle | default_pool + | select count(group_name) from pgxc_group; + 13626 | mogdb | 140390132872976 | 10 | gaussdba | cn_5002 | 10.180.123.163 + | | 48614 | 2016-07-15 14:11:16.014871+08 | | 2016-07-15 14 + :21:17.346045+08 | 2016-07-15 14:21:17.346095+08 | f | | idle | default_pool + | SET SESSION AUTHORIZATION DEFAULT;RESET ALL; + (2 rows) + ``` + +2. Release idle connections. + + Check each connection and release them after obtaining approval from the users of the connections. Run the following SQL command to release a connection using **pid** obtained in the previous step: + + ``` + mogdb=# SELECT pg_terminate_backend(140390132872976); + ``` + + Information similar to the following is displayed: + + ``` + mogdb=# SELECT pg_terminate_backend(140390132872976); + pg_terminate_backend + ---------------------- + t + (1 row) + ``` + + If no connections can be released, go to the next step. + +3. Increase the maximum number of connections. + + ``` + gs_guc set -D /mogdb/data/dbnode -c "max_connections= 800" + ``` + + **800** is the new maximum value. + +4. Restart database services to make the new settings take effect. + + > **NOTE:** The restart results in operation interruption. Properly plan the restart to avoid affecting users. + + ``` + gs_om -t stop && gs_om -t start + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/8-routinely-maintaining-tables.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/8-routinely-maintaining-tables.md new file mode 100644 index 0000000000000000000000000000000000000000..e99aeb3acb4adde43e4ac875e1a4a08978bd6b58 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/8-routinely-maintaining-tables.md @@ -0,0 +1,111 @@ +--- +title: 日常运维 +summary: 日常运维 +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Routinely Maintaining Tables + +To ensure proper database running, after insert and delete operations, you need to routinely run **VACUUM FULL** and **ANALYZE** as appropriate for customer scenarios and update statistics to obtain better performance. + +**Related Concepts** + +You need to routinely run **VACUUM**, **VACUUM FULL**, and **ANALYZE** to maintain tables, because: + +- **VACUUM FULL** can be used to reclaim disk space occupied by updated or deleted data and combine small-size data files. +- **VACUUM** can be used to maintain a visualized mapping for each table to track pages that contain arrays visible to other active transactions. A common index scan uses the mapping to obtain the corresponding arrays and check whether the arrays are visible to the current transaction. If the arrays cannot be obtained, capture a batch of arrays to check the visibility. Therefore, updating the visualized mapping of a table can accelerate unique index scans. +- Running **VACUUM** can avoid original data loss caused by duplicate transaction IDs when the number of executed transactions exceeds the database threshold. +- **ANALYZE** can be used to collect statistics on tables in databases. The statistics are stored in the system catalog **PG_STATISTIC**. Then the query optimizer uses the statistics to work out the most efficient execution plan. + +**Procedure** + +1. Run the **VACUUM** or **VACUUM FULL** command to reclaim disk space. + + - **VACUUM**: + + Run **VACUUM** for a table. + + ``` + mogdb=# VACUUM customer; + ``` + + ``` + VACUUM + ``` + + This statement can be concurrently executed with database operation commands, including **SELECT**, **INSERT**, **UPDATE**, and **DELETE**; excluding **ALTER TABLE**. + + Run **VACUUM** for the table partition. + + ``` + mogdb=# VACUUM customer_par PARTITION ( P1 ); + ``` + + ``` + VACUUM + ``` + + - **VACUUM FULL**: + + ``` + mogdb=# VACUUM FULL customer; + ``` + + ``` + VACUUM + ``` + + During the command running, exclusive locks need to be added to the table and all other database operations need to be suspended. + +2. Run **ANALYZE** to update statistics. + + ``` + mogdb=# ANALYZE customer; + ``` + + ``` + ANALYZE + ``` + + Run **ANALYZE VERBOSE** to update statistics and display table information. + + ``` + mogdb=# ANALYZE VERBOSE customer; + ``` + + ``` + ANALYZE + ``` + + You can run **VACUUM ANALYZE** at the same time to optimize the query. + + ``` + mogdb=# VACUUM ANALYZE customer; + ``` + + ``` + VACUUM + ``` + + > **NOTE:** **VACUUM** and **ANALYZE** cause a substantial increase in I/O traffic, which may affect other active sessions. Therefore, you are advised to set the cost-based vacuum delay feature by specifying the **vacuum_cost_delay** parameter. For details, see "GUC Parameters > Resource Consumption > Cost-based Vacuum Delay" in the *Developer Guide*. + +3. Delete a table. + + ``` + mogdb=# DROP TABLE customer; + mogdb=# DROP TABLE customer_par; + mogdb=# DROP TABLE part; + ``` + + If the following information is displayed, the tables have been deleted: + + ``` + DROP TABLE + ``` + +**Maintenance Suggestions** + +- Routinely run **VACUUM FULL** for large tables. If the database performance deteriorates, run **VACUUM FULL** for the entire database. If the database performance is stable, you are advised to run **VACUUM FULL** monthly. +- Routinely run **VACUUM FULL** on system catalogs, especially **PG_ATTRIBUTE**. +- Enable automatic vacuum processes (**AUTOVACUUM**) in the system. The processes automatically run the **VACUUM** and **ANALYZE** statements to reclaim the record space marked as the deleted state and update statistics in the table. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/9-routinely-recreating-an-index.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/9-routinely-recreating-an-index.md new file mode 100644 index 0000000000000000000000000000000000000000..0a8c2b31941943f233f3e4f397ce12790613e460 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/9-routinely-recreating-an-index.md @@ -0,0 +1,68 @@ +--- +title: 日常运维 +summary: 日常运维 +author: Zhang Cuiping +date: 2021-03-04 +--- + +# Routinely Recreating an Index + +## **Background** + +When data deletion is repeatedly performed in the database, index keys will be deleted from the index pages, resulting in index bloat. Recreating an index routinely improves query efficiency. + +The database supports B-tree indexes. Recreating a B-tree index routinely helps improve query efficiency. + +- If a large amount of data is deleted, index keys on the index pages will be deleted. As a result, the number of index pages reduces and index bloat occurs. Recreating an index helps reclaim wasted space. +- In a newly created index, pages with adjacent logical structures tend to have adjacent physical structures. Therefore, a new index achieves a higher access speed than an index that has been updated for multiple times. + +**Methods** + +Use either of the following two methods to recreate an index: + +- Run the **DROP INDEX** statement to delete the index and then run the **CREATE INDEX** statement to create an index. + + When you delete an index, a temporary exclusive lock is added in the parent table to block related read/write operations. During index creation, the write operation is locked, whereas the read operation is not locked and can use only sequential scans. + +- Run **REINDEX** to recreate an index. + + - When you run the **REINDEX TABLE** statement to recreate an index, an exclusive lock is added to block related read/write operations. + - When you run the **REINDEX INTERNAL TABLE** statement to recreate an index for a **desc** table (such as column-store **cudesc** table), an exclusive lock is added to block related read/write operations on the table. + +**Procedure** + +Assume the ordinary index **areaS_idx** exists in the **area_id** column of the imported table **areaS**. Use either of the following two methods to recreate an index: + +- Run the **DROP INDEX** statement to delete the index and run the **CREATE INDEX** statement to create an index. + + 1. Delete the index. + + ``` + mogdb=# DROP INDEX areaS_idx; + DROP INDEX + ``` + + 2. Create an index + + ``` + mogdb=# CREATE INDEX areaS_idx ON areaS (area_id); + CREATE INDEX + ``` + +- Run **REINDEX** to recreate an index. + + - Run **REINDEX TABLE** to recreate an index. + + ``` + mogdb=# REINDEX TABLE areaS; + REINDEX + ``` + + - Run **REINDEX INTERNAL TABLE** to recreate an index for a **desc** table (such as column-store **cudesc** table). + + ``` + mogdb=# REINDEX INTERNAL TABLE areaS; + REINDEX + ``` + +> **NOTE:** Before you recreate an index, you can increase the values of **maintenance_work_mem** and **psort_work_mem** to accelerate the index recreation. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/slow-sql-diagnosis.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/slow-sql-diagnosis.md new file mode 100644 index 0000000000000000000000000000000000000000..1e416ae9d1b43f80869bc549ee83fb735afdbe3b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/slow-sql-diagnosis.md @@ -0,0 +1,29 @@ +--- +title: Slow SQL Diagnosis +summary: Slow SQL Diagnosis +author: Guo Huan +date: 2022-04-13 +--- + +# Slow SQL Diagnosis + +## Background + +If the SQL statement execution performance does not meet expectations, you can view the SQL statement execution information to analyze the behavior and diagnose problems that occur during the execution. + +## Prerequisites + +- The database instance is running properly. + +- The GUC parameter **track_stmt_stat_level** is properly set for querying the SQL statement information. + +- Only the system administrator and monitor administrator can perform this operation. + + ```sql + Run the following command to check the execution information about the SQL statements in the database instance: + gsql> select * from dbe_perf.get_global_full_sql_by_timestamp(start_timestamp, end_timestamp); + Run the following command to check the execution information about the slow SQL statements in the database instance: + gsql> select * from dbe_perf.get_global_slow_sql_by_timestamp(start_timestamp, end_timestamp); + Check the execution information about the SQL statement on the current node. + gsql> select * from statement_history; + ``` diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/using-the-gsql-client-for-connection.md b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/using-the-gsql-client-for-connection.md new file mode 100644 index 0000000000000000000000000000000000000000..586d50cd8e8c5b92d3db1f8d613df0f36d39649c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/routine-maintenance/using-the-gsql-client-for-connection.md @@ -0,0 +1,212 @@ +--- +title: Using the gsql Client for Connection +summary: Using the gsql Client for Connection +author: Zhang Cuiping +date: 2021-04-14 +--- + +# Using the gsql Client for Connection + +## Confirming Connection Information + +You can use a client tool to connect a database through the primary node of the database. Before the connection, obtain the IP address of the primary node of the database and the port number of the server where the primary node of the database is deployed. + +1. Log in to the primary node of the database as the OS user **omm**. + +2. Run the **gs_om -t status --detail** command to query instances in the MogDB cluster. + + ```bash + gs_om -t status --detail + + [ Datanode State ] + + node node_ip instance state + --------------------------------------------------------------------------------- + 1 mogdb-kernel-0005 172.16.0.176 6001 /mogdb/data/db1 P Primary Normal + ``` + + For example, the server IP address where the primary node of the database is deployed is 172.16.0.176. The data path of the primary node of the database is **/mogdb/data/db1**. + +3. Confirm the port number of the primary node of the database. + + View the port number in the **postgresql.conf** file in the data path of the primary database node obtained in step 2. The command is as follows: + + ```bash + cat /mogdb/data/db1/postgresql.conf | grep port + + port = 26000 # (change requires restart) + #comm_sctp_port = 1024 # Assigned by installation (change requires restart) + #comm_control_port = 10001 # Assigned by installation (change requires restart) + # supported by the operating system: + # e.g. 'localhost=10.145.130.2 localport=12211 remotehost=10.145.130.3 remoteport=12212, localhost=10.145.133.2 localport=12213 remotehost=10.145.133.3 remoteport=12214' + # e.g. 'localhost=10.145.130.2 localport=12311 remotehost=10.145.130.4 remoteport=12312, localhost=10.145.133.2 localport=12313 remotehost=10.145.133.4 remoteport=12314' + # %r = remote host and port + alarm_report_interval = 10 + support_extended_features=true + ``` + + **26000** in the first line is the port number of the primary database node. + +
+ +### Installing the gsql Client + +On the host, upload the client tool package and configure environment variables for the **gsql** client. + +1. Log in to the host where the client resides as any user. + +2. Run the following command to create the **/opt/mogdb/tools** directory: + + ```bash + mkdir /opt/mogdb/tools + ``` + +3. Obtain the file **MogDB-x.x.x-openEuler-64bit-tools.tar.gz** from the software installation package and upload it to the **/opt/mogdb/tools** directory. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The software package is located where you put it before installation. Set it based on site requirements. + > - The tool package name may vary in different OSs. Select the tool package suitable for your OS. + +4. Run the following commands to decompress the package: + + ```bash + cd /opt/mogdb/tools + tar -zxvf MogDB-x.x.x-openEuler-64bit-tools.tar.gz + ``` + +5. Set environment variables. + + Run the following command to open the **~/.bashrc** file: + + ```bash + vi ~/.bashrc + ``` + + Enter the following content and run **:wq!** to save and exit. + + ```bash + export PATH=/opt/mogdb/tools/bin:$PATH + export LD_LIBRARY_PATH=/opt/mogdb/tools/lib:$LD_LIBRARY_PATH + ``` + +6. Run the following command to make the environment variables take effect: + + ```bash + source ~/.bashrc + ``` + +
+ +## Connecting to a Database Using gsql + +
+ +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - By default, if a client is idle state after connecting to a database, the client automatically disconnects from the database in the duration specified by **session_timeout**. To disable the timeout setting, set **session_timeout** to **0**. + +
+ +### Connecting to a Database Locally + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Connect to a database. + + After the database is installed, a database named **postgres** is generated by default. When connecting to a database for the first time, you can connect to this database. + + Run the following command to connect to the **postgres** database: + + ```bash + gsql -d postgres -p 26000 + ``` + + **postgres** is the name of the database to be connected, and **26000** is the port number of the database primary node. Replace the values as required. + + If information similar to the following is displayed, the connection succeeds: + + ```sql + gsql ((MogDB x.x.x build 56189e20) compiled at 2022-01-07 18:47:53 commit 0 last mr ) + Non-SSL connection (SSL connection is recommended when requiring high-security) + Type "help" for help. + + postgres=# + ``` + + User **omm** is the administrator, and **postgres=#** is displayed. If you log in to and connect to the database as a common user, **postgres=>** is displayed. + + **Non-SSL connection** indicates that the database is not connected in SSL mode. If high security is required, connect to the database in SSL mode. + +3. Exit the database. + + ```sql + postgres=# \q + ``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - When connecting to the database locally as user **omm**, no password is required. This is due to the default setting in the **pg_hba.conf** file that allows the local machine to connect in the **trust** way. +> - For details about the client authentication methods, see the [Client Access Authentication](../../security-guide/security/1-client-access-authentication.md) chapter. + +
+ +## Connecting to a Database Remotely + +
+ +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - Due to security restrictions, you can not remotely connect to the database as user **omm**. + +
+ +### Configuring a Whitelist Using gs_guc (Update pg_hba.conf) + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Configure the client authentication mode and enable the client to connect to the host as user **jack**. User **omm** cannot be used for remote connection. + + Assume you are to allow the client whose IP address is **172.16.0.245** to access the current host. + + ```sql + gs_guc set -N all -I all -h "host all jack 172.16.0.245/24 sha256" + ``` + + **NOTICE:** + + - Before using user **jack**, connect to the database locally and run the following command in the database to create user **jack**: + + ```sql + postgres=# CREATE USER jack PASSWORD 'Test@123'; + ``` + + - **-N all** indicates all hosts in MogDB. + + - **-I all** indicates all instances on the host. + + - **-h** specifies statements that need to be added in the **pg_hba.conf** file. + + - **all** indicates that a client can connect to any database. + + - **jack** indicates the user that accesses the database. + + - **172.16.0.245/24** indicates that only the client whose IP address is **172.16.0.245** can connect to the host. The specified IP address must be different from those used in MogDB. **24** indicates that there are 24 bits whose value is 1 in the subnet mask. That is, the subnet mask is 255.255.255.0. + + - **sha256** indicates that the password of user **jack** is encrypted using the SHA-256 algorithm. + + This command adds a rule to the **pg_hba.conf** file corresponds to the primary node of the database. The rule is used to authenticate clients that access primary node. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > + For details about the client authentication methods, see the [Client Access Authentication](../../security-guide/security/1-client-access-authentication.md) chapter. + +3. Connect to a database. + + After the database is installed, a database named **postgres** is generated by default. When connecting to a database for the first time, you can connect to this database. + + ```bash + gsql -d postgres -h 172.16.0.176 -U jack -p 26000 -W Test@123 + ``` + + **postgres** is the name of the database, **172.16.0.176** is the IP address of the server where the primary node of the database resides, **jack** is the user of the database, **26000** is the port number of the CN, and **Test@123** is the password of user **jack**. diff --git a/product/en/docs-mogdb/v3.1/administrator-guide/upgrade-guide.md b/product/en/docs-mogdb/v3.1/administrator-guide/upgrade-guide.md new file mode 100644 index 0000000000000000000000000000000000000000..ce2d18712662a2f6b42f90a717d7e00777ac2129 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/administrator-guide/upgrade-guide.md @@ -0,0 +1,375 @@ +--- +title: Upgrade Guide +summary: Upgrade Guide +author: Zhang Cuiping +date: 2021-09-27 +--- + +# Upgrade Guide + +## Overview + +This document provides guidance on version upgrade and rollback process. It also offers common problem resolving and troubleshooting methods. + +## Intended Audience + +This document is mainly intended for upgrade operators. They must have the following experience and skills: + +- Be familiar with the networking of the current network and versions of related NEs (network elements). +- Have maintenance experience of the related devices and be familiar with their operation and maintenance methods. + +## Upgrade Scheme + +This section provides guidance on selection of the upgrade modes. + +The user determines whether to upgrade the current system according to the new features of MogDB and database situations. + +The supported upgrade modes include in-place upgrade and gray upgrade. The upgrade strategies include major upgrade and minor upgrade. + +After the upgrade mode is determined, the system will automatically determine and choose the suitable upgrade strategy. + +* In-place upgrade: All services need to be stopped during the upgrade. All nodes are upgraded at a time. + +* Gray upgrade: supports full-service operations. All nodes are also upgraded at a time. (Currently, only the gray upgrade from version 1.1.0 to 2.0 and above is supported.) + +## Version Requirements Before the Upgrade (Upgrade Path) + +[Table 1](#biaoyi) lists the MogDB upgrade version requirements. + +**Table 1** Version requirements before the upgrade (upgrade path) + +| Version | Description | +| --------- | ------------------------------------------------------- | +| MogDB 2.0 | Need to upgrade to MogDB 2.1, then upgrade to MogDB 3.0 | +| MogDB 2.1 | Upgradeable to MogDB 3.0 | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** You can run the following command to check the version before the upgrade: +> +> ```bash +> gsql -V | --version +> ``` + +## Impact and Constraints + +The following precautions need to be considered during the upgrade: + +- The upgrade cannot be performed with capacity expansion and reduction concurrently. +- VIP (virtual IP) is not supported. +- During the upgrade, you are not allowed to modify the **wal_level**, **max_connections**, **max_prepared_transactions**, and **max_locks_per_transaction** GUC parameters. Otherwise, the instance will be started abnormally after rollback. +- It is recommended that the upgrade is performed when the database system is under the light workload. You can determine the off-peak hours according to your experience, such as holidays and festivals. +- Before the upgrade, make sure that the database is normal. You can run the **gs_om -t status** command to check the database status. If the returned value of **cluster_state** is **Normal**, the database is normal. +- Before the upgrade, make sure that mutual trust is established between database nodes. You can run the **ssh hostname** command on any node to connect to another node to verify whether the mutual trust has been established. If mutual connection between any two nodes does not require a password, the mutual trust is normal. (Generally, when the database status is normal, mutual trust is normal.) +- Before and after the upgrade, the database deployment mode must be kept consistent. Before the upgrade, the database deployment mode will be verified. If it is changed after the upgrade, an error will occur. +- Before the upgrade, make sure that the OS is normal. You can check the OS status using the **gs_checkos** tool. +- In-place upgrade requires stopping of services. Gray upgrade supports full-service operations. +- The database is running normally and the data of the primary domain name (DN) is fully synchronized to the standby DN. +- During the upgrade, the kerberos is not allowed to be enabled. +- You are not allowed to modify the **version.cfg** file decompressed from the installation package. +- During the upgrade, if an error causes upgrade failure, you need to perform rollback operations manually. The next upgrade can be performed only after the rollback is successful. +- After the rollback, if the next upgrade is successful, GUC parameters set before the upgrade is submitted will become invalid. +- During the upgrade, you are not allowed to set GUC parameters manually. +- During the gray upgrade, service interruption will occur and lasts less than 10s. +- During the upgrade, OM operations can be performed only when the kernel and OM versions are consistent. This consistency refers that the kernel code and OM code are from the same software package. If the pre-installation script of the installation package is executed but the upgrade is not performed, or the pre-installation script of the baseline package after the rollback is not performed, the kernel code will be inconsistent with the OM code. +- During the upgrade, if new fields are added to a system table but they cannot be found by running the **\d** command after the upgrade, you can run the **select** command to check the new fields. +- The upgrade is allowed only when the value of **enable_stream_replication** is **on**. +- During the gray upgrade, the number of concurrent read/write services must be less than 200. +- If the MOT is used in MogDB 1.1.0, MogDB 1.1.0 cannot be upgraded to MogDB 2.0. + +## Upgrade Process + +This section describes the upgrade process. + +**Figure 1** Upgrade process + +![21](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/upgrade-guide-2.png) + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif)**NOTE:** The time listed in the following table is for reference only. The actual time required depends on the upgrade environment. + +**Table 2** Estimated upgrade efficiency + +
Procedure Recommended Start Time Time Required (Day/Hour/Minute) Service Interruption Time Remarks
Perform the pre-upgrade preparations and check operations. One day before the upgrade About 2 to 3 hours No impact on services Pre-upgrade check, data backup, and software package verification
Perform the upgrade. Off-peak hours The time is mainly spent in starting and stopping the database and modifying the system table of each database. The upgrade usually takes less than 30 minutes. The service interruption time is the same as the upgrade time. Generally, the time taken is not greater than 30 minutes. Performed based on the Upgrade Guide
Verify the upgrade. Off-peak hours About 30 minutes The service interruption time is the same as the upgrade verification time, about 30 minutes. -
Submit the upgrade. Off-peak hours The upgrade submission usually takes less than 10 minutes. The service interruption time is the same as the upgrade submission time. Generally, the time taken is not greater than 10 minutes. -
Roll back the upgrade. Off-peak hours The rollback usually takes less than 30 minutes. The service interruption time is the same as the rollback time. Generally, the time taken is not greater than 30 minutes. -
+ +## Pre-Upgrade Preparations and Check + +### Pre-Upgrade Preparations and Checklist + +**Table 3** Pre-upgrade preparations and checklist + +
No. Item to Be Prepared for the Upgrade Preparation Content Recommended Start Time Time Required (Day/Hour/Minute)
1 Collect node information. Obtain the name, IP address, and passwords of users root and omm of related database nodes One day before the upgrade 1 hour
2 Set remote login as user root. Set the configuration file that allows remote login as user root One day before the upgrade 2 hours
3 Back up data. For details, see the Backup and Restoration section in the Administrator Guide. One day before the upgrade The time taken varies depends on the volume of data to be backed up and the backup strategy.
4 Obtain and verify the installation package. Obtain the installation package and verify the package integrity. One day before the upgrade 0.5 hour
5 Perform the health check. Check the OS status using the gs_checkos tool One day before the upgrade 0.5 hour
6 Check the disk usage of each database node. Check the disk usage by running the df command. One day before the upgrade 0.5 hour
7 Check the database status. Check the database status using the gs_om tool. One day before the upgrade 0.5 hour
+ +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif)**NOTE:** **Time Required** varies depends on the environment, including data volume, server performance, and other factors. + +### Collecting Node Information + +You can contact the system administrator to obtain the environment information, such as name, IP address, and passwords of users **root** and **omm** of the database node. + +**Table 4** Node information + + + + + + + + + + + + + + + + + + +
No. Node Name IP Address of the Node +Password of User root Password of User omm Remarks
1 +- +- - - -
+ +### Backing Up Data + +Once the upgrade fails, services will be affected. Therefore, you need to back up data in advance so that services can be quickly restored once the risk occurs. + +For details about data backup, see the **Backup and Restoration** section in the *Administrator Guide*. + +### Obtaining the Installation Package + +You can obtain the installation package from [this website](https://www.mogdb.io/en/downloads/mogdb/). + +### Performing the Health Check + +The **gs_checkos** tool can be used to check the OS status. + +**Prerequisites** + +- The current hardware and network environment is normal. +- The mutual trust between the **root** users of all hosts is normal. +- The **gs_checkos** command can be executed only as user **root**. + +**Procedure** + +1. Log in to the primary database node as user **root**. + +2. Run the following command to check the server OS parameters: + + ``` + # gs_checkos -i A + ``` + + Checking the OS parameters aims to ensure that the database can be pre-installed normally and can be run safely and efficiently after being installed. + +#### Checking the Disk Usage of the Database Node + +It is recommended that the upgrade is performed when the disk usage of the database node is less than 80%. + +#### Checking the Database Status + +This section introduces how to check the database status. + +**Procedure** + +1. Log in to the primary database node as user **omm** and run the **source** command to reload environment variables. + + ``` + # su - omm + $ source /home/omm/.bashrc + ``` + +2. Run the following command to check the database status: + + ```bash + gs_om -t status + ``` + +3. Ensure that the database status is normal. + +## Upgrade Procedure + +This section introduces details about in-place upgrade and gray upgrade. + +**Procedure** + +1. Log in to the primary database node as user **root**. + +2. Create a directory for storing the new package. + + ``` + # mkdir -p /opt/software/mogdb_upgrade + ``` + +3. Upload the new package to the **/opt/software/mogdb_upgrade** directory and decompress the package. + +4. Found the **script** file. + + ``` + # cd /opt/software/mogdb_upgrade/script + ``` + +5. Create **static** folder, put the plugin package into **script/static** folder. (This step is a necessary operation, because the current version does not determine whether the user has used plugins, so the default database in the upgrade script to install the use of plugins, later versions will be a separate part of the plugin split out.) + + For example: + + ```sql + mkdir static + cd static/ + wget https://cdn-mogdb.enmotech.com/mogdb-media/3.0.1/Plugins-3.0.1-CentOS-x86_64.tar.gz + ``` + +6. Before the in-place or gray upgrade, execute the pre-installation script by running the **gs_preinstall** command. + + ``` + # ./gs_preinstall -U omm -G dbgrp -X /opt/software/mogdb/clusterconfig.xml + ``` + +7. Switch to user **omm**. + + ``` + # su - omm + ``` + +8. After ensuring that the database status is normal, run the required command to perform the in-place upgrade or gray upgrade. + + Example one: Execute the **gs_upgradectl** script to perform the in-place upgrade. + + ```bash + gs_upgradectl -t auto-upgrade -X /opt/software/mogdb/clusterconfig.xml + ``` + + Example two: Execute the **gs_upgradectl** script to perform the gray upgrade. + + ```bash + gs_upgradectl -t auto-upgrade -X /opt/software/mogdb/clusterconfig.xml --grey + ``` + +## Upgrade Verification + +This section introduces upgrade verification and provides detailed use cases and operations. + +### Verifying the Checklist of the Project + +**Table 5** Verification item checklist + +
No. Verification Item Check Standard Check Result
1 Version check Check whether the version is correct after the upgrade. -
2 Health check Use the gs_checkos tool to check the OS status. -
3 Database status Use the gs_om tool to check the database status. -
+ +### Querying the Upgrade Version + +This section introduces how to check the version. + +**Procedure** + +1. Log in to the primary database node as user **omm** and run the **source** command to reload environment variables. + + ``` + # su - omm + $ source /home/omm/.bashrc + ``` + +2. Run the following command to check the version information of all nodes: + + ```bash + gs_ssh -c "gsql -V" + ``` + +### Checking the Database Status + +This section introduces how to check the database status. + +**Procedure** + +1. Log in to the primary database node as user **omm**. + + ``` + # su - omm + ``` + +2. Run the following command to check the database status: + + ```bash + gs_om -t status + ``` + + If the value of **cluster_state** is **Normal**, the database is normal. + +## Upgrade Submission + +After the upgrade, if the verification is successful, the subsequent operation is to submit the upgrade. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Once the upgrade is submitted, it cannot be rolled back. + +**Procedure** + +1. Log in to the primary database node as user **omm**. + + ``` + # su - omm + ``` + +2. Run the following command to submit the upgrade: + + ```bash + gs_upgradectl -t commit-upgrade -X /opt/software/mogdb/clusterconfig.xml + ``` + +3. Reset the control file format to be compatible with the new ustore storage engine added in version 2.1.0 (2.0.1 upgrade to 2.1 only). + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION**: + > + > - This operation is not reversible and cannot be downgraded back to version 2.0.1 after execution. + > - Before performing this operation, it is recommended to make a full data backup by referring to [Logical Backup Recovery](../administrator-guide/br/1-3-br.md). + + ``` + pg_resetxlog -f $PGDATA + ``` + + The echo appear as follow: + + ``` + Transaction log reset + ``` + +## Version Rollback + +This section introduces how to roll back the upgrade. + +**Procedure** + +1. Log in to the primary database node as user **omm**. + + ``` + # su - omm + ``` + +2. Run the following command to perform the rollback operation (rolling back the kernel code). After the rollback, if you need to keep the versions of the kernel code and OM code consistent, execute the pre-installation script in the old package. (For details, see the [execute the pre-installtion script](#qianzhijiaoben) step.) + + ```bash + gs_upgradectl -t auto-rollback -X /opt/software/mogdb/clusterconfig.xml + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** If the database is abnormal, run the following command to perform the forcible rollback operation: + > + > ```bash + > gs_upgradectl -t auto-rollback -X /opt/software/mogdb/clusterconfig.xml --force + > ``` + +3. Check the version after the rollback. + + ```bash + gs_om -V | --version + ``` + + If the upgrade fails, perform the following operations to resolve the issue: + + a. Check whether the environment is abnormal. + + For example, the disk is fully occupied, the network is faulty, or the installation package or upgrade version is incorrect. After the problem is located and resolved, try to perform the upgrade again. + + b. If no environment issue is found or the upgrade fails again, collect related logs and contact technical engineers. + + Run the following command to collect logs: + + ```bash + gs_collector -begin-time='20200724 00:00' -end-time='20200725 00:00' + ``` + + If permitted, you are advised to retain the environment. diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai-in-db/predictor-ai-query-time-forecasting.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai-in-db/predictor-ai-query-time-forecasting.md new file mode 100644 index 0000000000000000000000000000000000000000..916aadf4bc06c66b08e0aca7876a30782af7815e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai-in-db/predictor-ai-query-time-forecasting.md @@ -0,0 +1,48 @@ +--- +title: Predictor AI Query Time Forecasting +summary: Predictor AI Query Time Forecasting +author: Guo Huan +date: 2022-05-10 +--- + +# Predictor: AI Query Time Forecasting + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Predictor is a query time forecasting tool that leverages machine learning and has online learning capability. Predictor can predict the execution time of a plan in the database kernel by continuously learning the historical execution information collected in the database. + +## Benefits + +This feature predicts the SQL statement execution time based on the AI model before the SQL statement is executed. Based on the estimated SQL time, you can detect possible abnormal SQL statements and optimize or schedule them in advance to prevent system running from being affected. + +## Description + +The prediction of AI query time depends on the collection of local query plans. During query execution, you need to collect the actual query plan (including the plan structure, operator type, related data source, and filter criteria), actual execution time of each operator node, cost estimated by the optimizer, number of rows returned by the optimizer, number of rows estimated by the optimizer, and number of parallel rows. These records are stored in data tables and managed persistently, and expired data will be cleared periodically. + +After the database kernel automatically collects historical data, the administrator encodes the data and sends a request to Python through the CURL API to configure the model, send data, and trigger training. You can call the TensorBoard API to monitor the training process. The model returns the prediction accuracy of each project and saves the final model. The database updates the system tables related to the model information based on the result returned by the AI engine. + +This feature is triggered when **explain (analyze on, predictor ) SELECT… “** is executed and the model is converged. The database determines whether the current model meets the requirements based on the plan encoding. If the model meets the requirements, the model loading request is sent to Python (the model needs to be loaded only once after the model training is complete). After the model is loaded, the plan encoding file is inferred and the prediction result of each node is returned. + +## Enhancements + +None. + +## Constraints + +- The database system is normal. The user successfully logs in to the database through identity authentication and accesses the authorized data. +- The SQL syntax is correct and no error is reported. +- SQL statements executed by users do not cause database exceptions. +- In the historical performance data window, the cluster concurrency is stable, the cluster scale, table structure, and table quantity remain unchanged, the data volume does not change abruptly, and the GUC parameters related to query performance remain unchanged. If the preceding conditions are damaged, the model becomes invalid. In this case, you need to invalidate all historical performance data, collect data again, and retrain the model. +- When a model is loaded, the structure, parameters, and training information of the original model are completely saved. If the original model information is lost, the model cannot be loaded and needs to be trained again. +- Historical performance data can be properly collected and encoded, and no error is reported when the encoded data is properly parsed. +- You can install the following software by using the provided installation scripts or by yourself: Python=3.6.4, configparser== 3.8.1, Flask== 0.12.2, Keras== 2.2.4, numpy== 1.16.4, scikit-learn== 0.19.1, pandas== 0.25.1, tensorboard== 1.14.0, and tensorflow-gpu== 1.14.0 or tensorflow== 1.14.0. +- OpenSSL has been installed in the user environment, and the certificate has been generated using a script or based on the usage description. +- Currently, administrators need to manually synchronize models across database namespaces. Automatic synchronization is not supported. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/1-database-metric-collection-forecast-and-exception-detection.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/1-database-metric-collection-forecast-and-exception-detection.md new file mode 100644 index 0000000000000000000000000000000000000000..312539e38aa3afad81c75bce4736af0f5e014033 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/1-database-metric-collection-forecast-and-exception-detection.md @@ -0,0 +1,41 @@ +--- +title: Database Metric Collection, Forecast, and Exception Detection +summary: Database Metric Collection, Forecast, and Exception Detection +author: Guo Huan +date: 2022-05-10 +--- + +# Database Metric Collection, Forecast, and Exception Detection + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +This feature serves as an AI tool integrated into MogDB and can be used to collect and forecast database metrics, as well as monitor and diagnose exceptions. It is a component in the DBMind suite. Currently, this feature is compatible with the Prometheus platform to collect database system metrics. Prometheus exporters are provided to collect and process database monitoring metrics. By monitoring the time series data of metrics, you can forecast the future load trend and diagnose problems. + +## Benefits + +- This feature greatly simplifies the work of O&M personnel, releases a large number of labor resources, and reduces costs for the company. +- You can use the metric collection, monitoring, and forecast functions to detect problems in advance, preventing database exceptions from causing greater loss. + +## Description + +Prometheus is a popular open-source monitoring system in the industry. It is also a time series database. The collector of Prometheus is called exporter, which is used to collect metrics of monitored modules. To interconnect with the Prometheus platform, DBMind provides two types of exporters: openGauss-exporter for collecting database metrics and reprocessing-exporter for reprocessing the collected metrics. + +This feature supports forecast of collected metrics. You can specify key performance indicators (KPIs) to be forecasted by modifying configuration files. This helps you find metric trends and perform O&M operations in a timely manner. For example, you can forecast the memory usage to detect memory leakage and forecast the disk usage to expand the capacity at a proper time. + +## Enhancements + +This feature is greatly improved in MogDB 3.0.0 and is compatible with the Prometheus platform. Two exporters are used to connect to Prometheus. + +## Constraints + +- The database is normal, and the data directory has been written into environment variables. +- The Python version must be 3.6 or later. +- The Prometheus monitoring platform is configured and the Prometheus service is started so that monitoring data can be collected. + +## Dependencies + +Prometheus \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/2-root-cause-analysis-for-slow-sql-statements.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/2-root-cause-analysis-for-slow-sql-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..63358c777f63c5ea71745803c6810f78e2fb3c47 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/2-root-cause-analysis-for-slow-sql-statements.md @@ -0,0 +1,38 @@ +--- +title: Root Cause Analysis for Slow SQL Statements +summary: Root Cause Analysis for Slow SQL Statements +author: Guo Huan +date: 2022-05-10 +--- + +# Root Cause Analysis for Slow SQL Statements + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Slow SQL statements have always been a pain point in data O&M. How to effectively diagnose the root causes of slow SQL statements is a big challenge. Based on the characteristics of MogDB and the slow SQL statement diagnosis experience of DBAs on the live network, this tool supports more than 15 root causes of slow SQL statements, outputs multiple root causes based on the possibility, and provides specific solutions. + +## Benefits + +This feature provides customers with fast and reliable slow SQL statement discovery and root cause analysis functions, greatly simplifying the work of O&M personnel. + +## Description + +Based on the Prometheus data collection solution, data required for root cause analysis for slow SQL statements is collected, including system resource information (CPU usage, memory usage, and I/O), load information (QPS), large process information (including external large processes and scheduled database tasks), slow SQL statement text information, start time and end time of slow SQL statement execution, slow SQL statement execution plan, temporary file information, and so on. Then, this feature calculates the most matched root cause of slow SQL statements based on the AI algorithm, and provides suggestions and confidence. + +## Enhancements + +None. + +## Constraints + +- The database is normal, and the client can be connected properly. +- An environment running Python 3.6 or later is available. +- The information about slow SQL statements is obtained from the workload diagnosis report (WDR). In the database WDR, slow SQL statements are marked. The GUC parameter **track_stmt_stat_level** is enabled by default. Otherwise, you need to manually enable it. Generally, the **track_stmt_stat_level** is set to **'off, L0'**. Higher levels will affect the performance. Data collection is implemented by the Prometheus solution. Therefore, you need to configure the Prometheus data collection platform. This feature focuses on algorithms and obtains metric sequence information from Prometheus. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/3-index-recommendation.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/3-index-recommendation.md new file mode 100644 index 0000000000000000000000000000000000000000..9cdf7a7f1fa7d9e862ac643c940d9a667ce4f77c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/3-index-recommendation.md @@ -0,0 +1,40 @@ +--- +title: Index Recommendation +summary: Index Recommendation +author: Guo Huan +date: 2022-05-10 +--- + +# Index Recommendation + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +This feature serves as an intelligent database index recommendation tool that covers multiple task levels and application scenarios. It provides the single-query index recommendation function, virtual index function, and workload-level index recommendation function to provide reliable index recommendations for users. + +## Benefits + +This feature provides the quick and reliable index recommendation function, greatly simplifying the work of O&M personnel. + +## Description + +The single-query index recommendation function allows users to directly perform operations in the database. This feature generates recommended indexes for a single query statement entered by users based on the semantic information of the query statement and the statistics of the database. The virtual index function allows users to directly perform operations in the database. This feature simulates the creation of a real index to avoid the time and space overhead required for creating a real index. Based on the virtual index, users can evaluate the impact of the index on the specified query statement by using the optimizer. The workload-level index recommendation can be used by running scripts outside the database. This feature uses the workload of multiple DML statements as the input to generate a batch of indexes that can optimize the overall workload execution performance. + +## Enhancements + +None. + +## Constraints + +The database is normal, and the client can be connected properly. + +The gsql tool has been installed by the current user, and the tool path has been added to the "_PATH_" environment variable. + +An environment running Python 3.6 or later is available. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/4-parameter-tuning-and-diagnosis.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/4-parameter-tuning-and-diagnosis.md new file mode 100644 index 0000000000000000000000000000000000000000..a732c908a52df314e4e4ec39aea7470d63bd421c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/4-parameter-tuning-and-diagnosis.md @@ -0,0 +1,46 @@ +--- +title: Parameter Tuning and Diagnosis +summary: Parameter Tuning and Diagnosis +author: Guo Huan +date: 2022-05-10 +--- + +# Parameter Tuning and Diagnosis + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +This feature servers as a parameter tuning tool integrated into databases. It uses AI technologies such as deep reinforcement learning and global search algorithms to obtain the optimal database parameter settings without manual intervention. It is not forcibly deployed with the database environment. It can be independently deployed and run without the database installation environment. + +## Benefits + +This tool can quickly provide the parameter adjustment configuration of the current load in any scenario, reducing database administrator's manual intervention, improving the O&M effect, and meeting customer expectations. + +## Description + +The tuning program can run in any of the following modes: + +- **recommend**: Log in to the database using the specified username, obtain the feature information about the running workload, and generate a parameter recommendation report based on the feature information. Report improper parameter settings and potential risks in the current database. Output the currently running workload behavior and characteristics. Output the recommended parameter settings. In this mode, the database does not need to be restarted. In other modes, the database may need to be restarted repeatedly. +- **train**: Modify parameters and execute the benchmark based on the benchmark information provided by users. The reinforcement learning model is trained through repeated iteration so that you can load the model in **tune** mode for optimization. +- **tune**: Use an optimization algorithm to tune database parameters. Currently, two types of algorithms are supported: deep reinforcement learning and global search algorithm (global optimization algorithm). The deep reinforcement learning mode requires **train** mode to generate the optimized model after training. However, the global search algorithm does not need to be trained in advance and can be directly used for search and optimization. + +## Enhancements + +None. + +## Constraints + +- The database is normal, the client can be properly connected, and data can be imported to the database. As a result, the optimization program can perform the benchmark test for optimization effect. +- To use this tool, you need to specify the user who logs in to the database. The user who logs in to the database must have sufficient permissions to obtain sufficient database status information. +- If you log in to the database host as a Linux user, add *$GAUSSHOME***/bin** to the "_PATH_" environment variable so that you can directly run database O&M tools, such as gsql, gs_guc, and gs_ctl. +- The recommended Python version is Python 3.6 or later. The required dependency has been installed in the operating environment, and the optimization program can be started properly. You can install a Python 3.6+ environment independently without setting it as a global environment variable. You are not advised to install the tool as the root user. If you install the tool as the root user and run the tool as another user, ensure that you have the read permission on the configuration file. +- This tool can run in three modes. In **tune** and **train** modes, you need to configure the benchmark running environment and import data. This tool will iteratively run the benchmark to check whether the performance is improved after the parameters are modified. +- In **recommend** mode, you are advised to run the command when the database is executing the workload to obtain more accurate real-time workload information. +- By default, this tool provides benchmark running script samples of TPC-C, TPC-H, TPC-DS, and sysbench. If you use the benchmarks to perform pressure tests on the database system, you can modify or configure the preceding configuration files. To adapt to your own service scenarios, you need to compile the script file that drives your customized benchmark based on the **template.py** file in the **benchmark** directory. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/5-slow-sql-statement-discovery.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/5-slow-sql-statement-discovery.md new file mode 100644 index 0000000000000000000000000000000000000000..315e51b8e84a60ccd220da825659bcdb72b97f8d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/ai4db-autonomous-database-o-m/5-slow-sql-statement-discovery.md @@ -0,0 +1,39 @@ +--- +title: Slow SQL Statement Discovery +summary: Slow SQL Statement Discovery +author: Guo Huan +date: 2022-05-10 +--- + +# Slow SQL Statement Discovery + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +This feature servers as an SQL statement execution time forecast tool. It forecasts the execution time of SQL statements based on the statement logic similarity and historical execution records without obtaining the SQL statement execution plan using a template. + +## Benefits + +- This feature does not require users to provide SQL execution plans. Therefore, the database performance is not affected. +- Different from other algorithms in the industry that are limited to OLAP or OLTP, this feature is more widely used. + +## Description + +The SQLdiag focuses on the historical SQL statements of the database, summarizes the execution performance of the historical SQL statements, and then uses the historical SQL statements to infer unknown services. The execution duration of SQL statements in the database does not differ greatly in a short period of time. SQLdiag can detect the statement result set similar to the executed SQL statements from historical data and predict the execution duration of SQL statements based on the SQL vectorization technology and template-based method. + +## Enhancements + +None. + +## Constraints + +- The historical logs and the format of the workload to be predicted meet the requirements. You can use the GUC parameter of the database to enable the collection or use the monitoring tool to collect logs. +- To ensure the prediction accuracy, the historical statement logs provided by users should be as comprehensive and representative as possible. +- The Python environment has been configured as required. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/db4ai-database-driven-ai.md b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/db4ai-database-driven-ai.md new file mode 100644 index 0000000000000000000000000000000000000000..d9fb6a83aa392245f88e873b7541c5e5978b10f2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/ai-capabilities/db4ai-database-driven-ai.md @@ -0,0 +1,39 @@ +--- +title: DB4AI Database-driven AI +summary: DB4AI Database-driven AI +author: Guo Huan +date: 2022-05-10 +--- + +# DB4AI: Database-driven AI + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +DB4AI uses database capabilities to drive AI tasks and implement data storage and technology stack isomorphism. By integrating AI algorithms into the database, MogDB supports the native AI computing engine, model management, AI operators, and native AI execution plan, providing users with inclusive AI technologies. Different from the traditional AI modeling process, DB4AI one-stop modeling eliminates repeated data flowing among different platforms, simplifies the development process, and plans the optimal execution path through the database, so that developers can focus on the tuning of specific services and models. It outcompetes similar products in ease-of-use and performance. + +## Benefits + +- With this feature, you do not need to manually compile AI model code. Instead, you can use out-of-the-box SQL statements to train and forecast machine learning models, reducing the learning and usage costs. +- Extra overhead that is caused by fragmented data storage and repeated data migration can be avoided. +- A higher execution efficiency can be achieved. With this feature, the AI model training efficiency is high. Compared with manual model training, the performance is improved by several times. +- Stricter security protection prevents data leakage during AI model training. + +## Description + +MogDB supports the native DB4AI capability. By introducing native AI operators, MogDB simplifies the operation process and fully utilizes the optimization and execution capabilities of the database optimizer and executor to obtain the high-performance model training capability in the database. With a simpler model training and forecast process and higher performance, developers can focus on model tuning and data analysis in a shorter period of time, avoiding fragmented technology stacks and redundant code implementation. + +## Enhancements + +More algorithms are supported in MogDB 3.0.0. + +## Constraints + +- The database is running properly. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/1-standard-sql.md b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/1-standard-sql.md new file mode 100644 index 0000000000000000000000000000000000000000..8681364bfd058814bd3e9d5b8afe8f759f4af595 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/1-standard-sql.md @@ -0,0 +1,38 @@ +--- +title: Standard SQL +summary: Standard SQL +author: Guo Huan +date: 2022-05-07 +--- + +# Standard SQL + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +SQL is a standard computer language used to control the access to databases and manage data in databases. SQL standards are classified into core features and optional features. Most databases do not fully support SQL standards. + +MogDB supports most of the core features of SQL:2011 and some optional features, providing a unified SQL interface for users. + +## Benefits + +All database vendors can use a unified SQL interface, reducing the costs of learning languages and migrating applications. + +## Description + +For details, see “SQL Syntax” in the *Reference Guide*. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/2-standard-development-interfaces.md b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/2-standard-development-interfaces.md new file mode 100644 index 0000000000000000000000000000000000000000..7bc6d2b68ba0a32d5956515d31ea7ba3b2c2961d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/2-standard-development-interfaces.md @@ -0,0 +1,36 @@ +--- +title: Standard Development Interfaces +summary: Standard Development Interfaces +author: Guo Huan +date: 2022-05-07 +--- + +# Standard Development Interfaces + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Standard ODBC 3.5 and JDBC 4.0 APIs are supported. + +## Benefits + +Standard ODBC and JDBC interfaces are provided to ensure quick migration of user services to MogDB. + +## Description + +Currently, the standard ODBC 3.5 and JDBC 4.0 APIs are supported. The ODBC interface supports SUSE Linux, Windows 32-bit, and Windows 64-bit platforms. The JDBC API supports all platforms. + +## Enhancements + +The function of connecting JDBC to a third-party log framework is added. JDBC can interconnect with a third-party log framework to meet users' log management and control requirements. + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/3-postgresql-api-compatibility.md b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/3-postgresql-api-compatibility.md new file mode 100644 index 0000000000000000000000000000000000000000..1a6a25dfd426f275b0896b28da5bf694c4044532 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/3-postgresql-api-compatibility.md @@ -0,0 +1,36 @@ +--- +title: PostgreSQL API Compatibility +summary: PostgreSQL API Compatibility +author: Guo Huan +date: 2022-05-07 +--- + +# PostgreSQL API Compatibility + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Compatible with PostgreSQL clients and standard APIs. + +## Benefits + +Compatible with the PostgreSQL clients and standard APIs, and can be seamlessly interconnected with PostgreSQL ecosystem tools. + +## Description + +Compatible with PostgreSQL clients and standard APIs. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/4-pl-java.md b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/4-pl-java.md new file mode 100644 index 0000000000000000000000000000000000000000..00d676b1668059ce510574b7f60b850e4f3c47e0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/4-pl-java.md @@ -0,0 +1,66 @@ +--- +title: PL/Java +summary: PL/Java +author: Guo Huan +date: 2022-05-07 +--- + +# PL/Java + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The Java user-defined field (UDF) is supported. + +## Benefits + +A development environment is provided for multiple functions. + +## Description + +With the MogDB PL/Java function, you can choose your favorite Java IDE to write Java methods and install the JAR files containing these methods into MogDB before invoking them. MogDB PL/Java is developed based on open-source tada PL/Java 1.5.2. PL/Java uses Huawei JDK V100R001C00SPC190B003-b09. + +## Enhancements + +None + +## Constraints + +- Java UDFs can implement simple Java computing. However, do not encapsulate services in Java UDFs. + +- Do not connect to a database in any way (for example, JDBC) in Java functions. + +- Use Huawei JDK V100R001C00SPC190B003-b09 to compile Java methods and JAR files. + +- Currently, only data types listed in Table 1 are supported. Other data types, such as user-defined data types and complex data types (for example, Java array and its derived types) are not supported. + +- Currently, UDAF and UDTF are not supported. + + **Table 1** PL/Java mapping for default data types + + | MogDB | **Java** | + | :---------- | :------------------------------------------------- | + | BOOLEAN | boolean | + | “char” | byte | + | bytea | byte[] | + | SMALLINT | short | + | INTEGER | int | + | BIGINT | long | + | FLOAT4 | float | + | FLOAT8 | double | + | CHAR | java.lang.String | + | VARCHAR | java.lang.String | + | TEXT | java.lang.String | + | name | java.lang.String | + | DATE | java.sql.Timestamp | + | TIME | java.sql.Time (stored value treated as local time) | + | TIMETZ | java.sql.Time | + | TIMESTAMP | java.sql.Timestamp | + | TIMESTAMPTZ | java.sql.Timestamp | + +## Dependencies + +PL/Java depends on the Java Development Kit (JDK) environment. Currently, JDK is included in MogDB and installation is not required. If you have installed the same or different versions of JDK, no conflict will occur. MogDB uses Huawei JDK V100R001C00SPC190B003-b09 to run PL/Java. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/MogDB-MySQL-compatibility.md b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/MogDB-MySQL-compatibility.md new file mode 100644 index 0000000000000000000000000000000000000000..f9c898ae9029100b9fead8bfa5fe1a070bad3b36 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/MogDB-MySQL-compatibility.md @@ -0,0 +1,30 @@ +--- +title: MogDB-MySQL Compatibility +summary: MogDB-MySQL Compatibility +author: Zhang Cuiping +date: 2022-06-21 +--- + +# MogDB-MySQL Compatibility + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +This feature mainly enhances the compatibility of MogDB with MySQL in the following three aspects. At the same time, the `insert` syntax is supported, and `insert into` can be abbreviated as `insert`. + +User lock allows users to add custom locks through SQL, which allows multiple programs to complete the lock-related interaction process, making the client access from any location to get a consistent lock view. + +When data is inserted into a table to be created, the current time is inserted by default. During data update, if the update time is not specified, the time when the data is updated is displayed by default. + +Session-level SQL mode can be set, allowing change in running, global change, and intra-session change. + +## Benefits + +By setting user locks, data, data structures or certain strings are protected from interfering with each other between sessions, ensuring consistency and security of information. It solves the problem of recording the timestamp of users' operation when their business data is written and modified. By setting SQL mode, it can solve the compatibility between the legacy problems of earlier versions and later versions. + +## Related Pages + +[CREATE TABLE](../../reference-guide/sql-syntax/CREATE-TABLE.md), [ALTER TABLE](../../reference-guide/sql-syntax/ALTER-TABLE.md), [INSERT](../../reference-guide/sql-syntax/INSERT.md), [Advisory Lock Functions](../../reference-guide/functions-and-operators/24-system-management-functions/7-advisory-lock-functions.md), [SQL Mode](../../reference-guide/guc-parameters/SQL-mode.md), [dolphin](../../reference-guide/oracle-plugins/dolphin.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/MogDB-Oracle-compatibility.md b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/MogDB-Oracle-compatibility.md new file mode 100644 index 0000000000000000000000000000000000000000..82fd61e115b29ffc4c5ac64a14b02b1c4e5f5690 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/application-development-interfaces/MogDB-Oracle-compatibility.md @@ -0,0 +1,30 @@ +--- +title: MogDB-Oracle Compatibility +summary: MogDB-Oracle Compatibility +author: Zhang Cuiping +date: 2022-06-17 +--- + +# MogDB-Oracle Compatibility + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +MogDB is compatible with Oracle's related functions and package functions using whale extension. + +In the function part, it mainly adds instrb, nls_charset_id, nls_charset_name, nls_lower, nls_upper, ora_hash, remainder, replace, show, show_parameter, to_timestamp, to_yminterval, tz_offset, nullif, ratio_to_report, etc. + +packages are generally only used in stored procedures, and according to ORACLE data package rules, new packages are placed under the corresponding schema. The supported Oracle management packages are dbms_random, dbms_output, dbms_lock, dbms_application_info, dbms_metadata, dbms_job, dbms_utility. + +For more information about the functions and the packages, please see the [whale](. /... /reference-guide/oracle-plugins/whale.md). + +## Benefits + +MogDB's compatibility with Oracle is enhanced by using the whale extension to enhance MogDB functions. + +## Related Pages + +[whale](../../reference-guide/oracle-plugins/whale.md), [Character Processing Functions and Operators](../../reference-guide/functions-and-operators/3-character-processing-functions-and-operators.md), [Mathematical Functions and Operators](../../reference-guide/functions-and-operators/7-mathematical-functions-and-operators.md), [Date and Time Processing Functions and Operators](../../reference-guide/functions-and-operators/8-date-and-time-processing-functions-and-operators.md), [HLL Functions and Operators](../../reference-guide/functions-and-operators/13.1-hll-functions-and-operators.md), [Window Functions](../../reference-guide/functions-and-operators/18-window-functions.md), [System Information Functions](../../reference-guide/functions-and-operators/23-system-information-functions.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/characteristic-description-overview.md b/product/en/docs-mogdb/v3.1/characteristic-description/characteristic-description-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..8dda8972612f3d7e0b9ca5c1d9a18bb5fdfe5f3e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/characteristic-description-overview.md @@ -0,0 +1,112 @@ +--- +title: Characteristic Description +summary: Characteristic Description +author: Guo Huan +date: 2022-09-01 +--- + +# Characteristic Description + +MogDB 3.0 has the following characteristics. + ++ High Performance + + [CBO Optimizer](./high-performance/1-cbo-optimizer.md) + + [LLVM](./high-performance/2-llvm.md) + + [Vectorized Engine](./high-performance/3-vectorized-engine.md) + + [Hybrid Row-Column Store](./high-performance/4-hybrid-row-column-store.md) + + [Adaptive Compression](./high-performance/5-adaptive-compression.md) + + [SQL by pass](./high-performance/6-sql-by-pass.md) + + [Kunpeng NUMA Architecture Optimization](./high-performance/7-kunpeng-numa-architecture-optimization.md) + + [High Concurrency of Thread Pools](./high-performance/8-high-concurrency-of-thread-pools.md) + + [SMP for Parallel Execution](./high-performance/9-smp-for-parallel-execution.md) + + [Xlog no Lock Flush](./high-performance/10-xlog-no-lock-flush.md) + + [Parallel Page-based Redo For Ustore](./high-performance/11-parallel-page-based-redo-for-ustore.md) + + [Row-Store Execution to Vectorized Execution](./high-performance/12-row-store-execution-to-vectorized-execution.md) ++ High Availability (HA) + + [Primary/Standby](./high-availability/1-primary-standby.md) + + [Logical Replication](./high-availability/2-logical-replication.md) + + [Online Node Replacement](./high-availability/3-online-node-replacement.md) + + [Logical Backup](./high-availability/4-logical-backup.md) + + [Physical Backup](./high-availability/5-physical-backup.md) + + [Automatic Job Retry upon Failure](./high-availability/6-automatic-job-retry-upon-failure.md) + + [Ultimate RTO](./high-availability/7-ultimate-rto.md) + + [Cascaded Standby Server](./high-availability/8-cascaded-standby-server.md) + + [Delayed Replay](./high-availability/9-delayed-replay.md) + + [Adding or Deleting a Standby Server](./high-availability/10-adding-or-deleting-a-standby-server.md) + + [Delaying Entering the Maximum Availability Mode](./high-availability/11-delaying-entering-the-maximum-availability-mode.md) + + [Parallel Logical Decoding](./high-availability/12-parallel-logical-decoding.md) + + [DCF](./high-availability/13-dcf.md) + + [CM](./high-availability/14-cm.md) + + [Global SysCache](./high-availability/15-global-syscache.md) + + [Using a Standby Node to Build a Standby Node](./high-availability/16-using-a-standby-node-to-build-a-standby-node.md) ++ Maintainability + + [Gray Upgrade](./maintainability/1-gray-upgrade.md) + + [Workload Diagnosis Report (WDR)](./maintainability/2-workload-diagnosis-report.md) + + [Slow SQL Diagnosis](./maintainability/3-slow-sql-diagnosis.md) + + [Session Performance Diagnosis](./maintainability/4-session-performance-diagnosis.md) + + [System KPI-aided Diagnosis](./maintainability/5-system-kpi-aided-diagnosis.md) + + [Fault Diagnosis](./maintainability/fault-diagnosis.md) ++ Database Security + + [Access Control Model](./database-security/1-access-control-model.md) + + [Separation of Control and Access Permissions](./database-security/2-separation-of-control-and-access-permissions.md) + + [Database Encryption Authentication](./database-security/3-database-encryption-authentication.md) + + [Data Encryption and Storage](./database-security/4-data-encryption-and-storage.md) + + [Database Audit](./database-security/5-database-audit.md) + + [Network Communication Security](./database-security/6-network-communication-security.md) + + [Resource Label](./database-security/7-resource-label.md) + + [Unified Audit](./database-security/8-unified-audit.md) + + [Dynamic Data Anonymization](./database-security/9-dynamic-data-anonymization.md) + + [Row-Level Access Control](./database-security/10-row-level-access-control.md) + + [Password Strength Verification](./database-security/11-password-strength-verification.md) + + [Equality Query in a Fully-encrypted Database](./database-security/12-equality-query-in-a-fully-encrypted-database.md) + + [Ledger Database Mechanism](./database-security/13-ledger-database-mechanism.md) + + [Transparent Data Encryption](./database-security/14-transparent-data-encryption.md) ++ Enterprise-Level Features + + [Support for Functions and Stored Procedures](./enterprise-level-features/1-support-for-functions-and-stored-procedures.md) + + [SQL Hints](./enterprise-level-features/2-sql-hints.md) + + [Full-Text Indexing](./enterprise-level-features/3-full-text-indexing.md) + + [Copy Interface for Error Tolerance](./enterprise-level-features/4-copy-interface-for-error-tolerance.md) + + [Partitioning](./enterprise-level-features/5-partitioning.md) + + [Support for Advanced Analysis Functions](./enterprise-level-features/6-support-for-advanced-analysis-functions.md) + + [Materialized View](./enterprise-level-features/7-materialized-view.md) + + [HyperLogLog](./enterprise-level-features/8-hyperloglog.md) + + [Creating an Index Online](./enterprise-level-features/9-creating-an-index-online.md) + + [Autonomous Transaction](./enterprise-level-features/10-autonomous-transaction.md) + + [Global Temporary Table](./enterprise-level-features/11-global-temporary-table.md) + + [Pseudocolumn ROWNUM](./enterprise-level-features/12-pseudocolumn-rownum.md) + + [Stored Procedure Debugging](./enterprise-level-features/13-stored-procedure-debugging.md) + + [JDBC Client Load Balancing and Read/Write Isolation](./enterprise-level-features/14-jdbc-client-load-balancing-and-readwrite-isolation.md) + + [In-place Update Storage Engine](./enterprise-level-features/15-in-place-update-storage-engine.md) + + [Publication-Subscription](./enterprise-level-features/16-publication-subscription.md) + + [Foreign Key Lock Enhancement](./enterprise-level-features/17-foreign-key-lock-enhancement.md) + + [Data Compression in OLTP Scenarios](./enterprise-level-features/18-data-compression-in-oltp-scenarios.md) + + [Transaction Async Submit](./enterprise-level-features/19-transaction-async-submit.md) + + [Index Creation Parallel Control](./enterprise-level-features/23-index-creation-parallel-control.md) + + [Dynamic Partition Pruning](./enterprise-level-features/21-dynamic-partition-pruning.md) + + [COPY Import Optimization](./enterprise-level-features/20-copy-import-optimization.md) + + [SQL Running Status Observation](./enterprise-level-features/22-sql-running-status-observation.md) + + [BRIN Index](./enterprise-level-features/24-brin-index.md) + + [BLOOM Index](./enterprise-level-features/25-bloom-index.md) ++ Application Development Interfaces + + [Standard SQL](./application-development-interfaces/1-standard-sql.md) + + [Standard Development Interfaces](./application-development-interfaces/2-standard-development-interfaces.md) + + [PostgreSQL API Compatibility](./application-development-interfaces/3-postgresql-api-compatibility.md) + + [PL/Java](./application-development-interfaces/4-pl-java.md) + + [MogDB-Oracle Compatibility](./application-development-interfaces/MogDB-Oracle-compatibility.md) + + [MogDB-MySQL Compatibility](./application-development-interfaces/MogDB-MySQL-compatibility.md) ++ AI Capabilities + + AI4DB: Autonomous Database O&M + + [Database Metric Collection, Forecast, and Exception Detection](./ai-capabilities/ai4db-autonomous-database-o-m/1-database-metric-collection-forecast-and-exception-detection.md) + + [Root Cause Analysis for Slow SQL Statements](./ai-capabilities/ai4db-autonomous-database-o-m/2-root-cause-analysis-for-slow-sql-statements.md) + + [Index Recommendation](./ai-capabilities/ai4db-autonomous-database-o-m/3-index-recommendation.md) + + [Parameter Tuning and Diagnosis](./ai-capabilities/ai4db-autonomous-database-o-m/4-parameter-tuning-and-diagnosis.md) + + [Slow SQL Statement Discovery](./ai-capabilities/ai4db-autonomous-database-o-m/5-slow-sql-statement-discovery.md) + + [DB4AI: Database-driven AI](./ai-capabilities/db4ai-database-driven-ai.md) + + AI in DB + + [Predictor: AI Query Time Forecasting](./ai-capabilities/ai-in-db/predictor-ai-query-time-forecasting.md) ++ Tool Chain + + [MySQL to openGauss Migration Tool Chameleon](./tool-chain/mysql-to-mogdb-migration-tool-chameleon.md) + + [OpenGauss Client Tool DataStudio](./tool-chain/mogdb-client-tool-datastudio.md) ++ Middleware + + [Distributed Database Capability](./middleware/distributed-database-capability.md) + + [Deploying a Distributed Database Using Kubernetes](./middleware/deploying-a-distributed-database-using-kubernetes.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/1-access-control-model.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/1-access-control-model.md new file mode 100644 index 0000000000000000000000000000000000000000..0486ccd1043150f443f4eec59d919d190beb0cc0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/1-access-control-model.md @@ -0,0 +1,42 @@ +--- +title: Access Control Model +summary: Access Control Model +author: Guo Huan +date: 2022-05-07 +--- + +# Access Control Model + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +The access control model can be used to manage users' access permissions and grant them the minimum permissions required for completing a task. + +## Benefits + +You can create users and grant permissions to them as needed to minimize risks. + +## Description + +The database provides a role-based access control model and an access control model based on the separation of duties. In the role-based access control model, database roles are classified into system administrator, monitoring administrator, O&M administrator, security policy administrator, and common user. The security administrator creates roles or user groups and grant permissions to roles. The monitoring administrator views the monitoring views or functions in **dbe_perf** mode. The O&M administrator uses the Roach tool to back up and restore the database. The security policy administrator creates resource labels, anonymization policies, and unified audit policies. A user who is assigned a role has the role's permissions. + +In the access control model based on the separation of duties, database roles are classified into system administrator, security administrator, audit administrator, monitoring administrator, O&M administrator, security policy administrator, and common user. The security administrator creates users, the system administrator grants permissions to users, and the audit administrator audits all user behavior. + +By default, the role-based access control model is used. To switch to another mode, set the GUC parameter **enableSeparationOfDuty** to **on**. + +## Enhancements + +None + +## Constraints + +The permissions of the system administrator are controlled by the GUC parameter **enableSeparationOfDuty**. + +The database needs to be restarted when the separation of duties is enabled, disabled or switched. In addition, improper user permissions in the new model cannot be automatically identified. The database administrator needs to manually identify and rectify the fault. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/10-row-level-access-control.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/10-row-level-access-control.md new file mode 100644 index 0000000000000000000000000000000000000000..431275c2410598e35ea8eb1c10cc1f9a05078e65 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/10-row-level-access-control.md @@ -0,0 +1,43 @@ +--- +title: Row-Level Access Control +summary: Row-Level Access Control +author: Guo Huan +date: 2022-05-07 +--- + +# Row-Level Access Control + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +The row-level access control feature enables database access control to be accurate to each row of data tables. When different users perform the same SQL query operation, the read results may be different. + +## Benefits + +When different users perform the same SQL query operation, the read results may be different. + +## Description + +You can create a row-level access control policy for a data table. The policy defines an expression that takes effect only for specific database users and SQL operations. When a database user accesses the data table, if a SQL statement meets the specified row-level access control policy of the data table, the expressions that meet the specified condition will be combined by using **AND** or **OR** based on the attribute type (**PERMISSIVE** or **RESTRICTIVE**) and applied to the execution plan in the query optimization phase. + +Row-level access control is used to control the visibility of row-level data in tables. By predefining filters for data tables, the expressions that meet the specified condition can be applied to execution plans in the query optimization phase, which will affect the final execution result. Currently, the SQL statements that can be affected include **SELECT**, **UPDATE**, and **DELETE**. + +## Enhancements + +None + +## Constraints + +- Row-level access control policies can be applied only to **SELECT**, **UPDATE**, and **DELETE** operations and cannot be applied to **INSERT** and **MERGE** operations. +- Row-level access control policies can be defined for row-store tables, row-store partitioned tables, column-store tables, column-store partitioned tables, replication tables, unlogged tables, and hash tables. Row-level access control policies cannot be defined for HDFS tables, foreign tables, and temporary tables. +- Row-level access control policies cannot be defined for views. +- A maximum of 100 row-level access control policies can be defined for a table. +- Initial users and system administrators are not affected by row-level access control policies. +- If a dynamic data anonymization policy is configured for a table that has the row-level access control policies defined, grant the trigger permission of the table to other users with caution to prevent other users from using the trigger to bypass the anonymization policy. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/11-password-strength-verification.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/11-password-strength-verification.md new file mode 100644 index 0000000000000000000000000000000000000000..acbf34b9fdf2cb7ce866e611c80f9fd6039137f0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/11-password-strength-verification.md @@ -0,0 +1,71 @@ +--- +title: Password Strength Verification +summary: Password Strength Verification +author: Guo Huan +date: 2022-05-07 +--- + +# Password Strength Verification + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +Verifies the password strength when users access the database. + +## Benefits + +Users cannot set passwords with low strength to enhance customer data security. + +## Description + +You need to specify a password when initializing a database, creating a user, or modifying a user. The password must meet the strength requirements. Otherwise, the system prompts you to enter the password again. Password complexity requirements: + +- Minimum number of uppercase letters (A-Z) (**password_min_uppercase**) +- Minimum number of lowercase letters (a-z) (**password_min_lowercase**) +- Minimum number of digits (0-9) (**password_min_digital**) +- Minimum number of special characters (**password_min_special**) +- Minimum password length (**password_min_length**) +- Maximum password length (**password_max_length**) +- A password must contain at least three types of the characters (uppercase letters, lowercase letters, digits, and special characters). +- A password is case insensitive and cannot be the username or the username spelled backwards. +- A new password cannot be the current password and the current password spelled backwards. +- It must be a strong password. + - Weak passwords are weak passwords that are easy to crack. The definition of weak passwords may vary with users or user groups. Users can define their own weak passwords. + +If parameter **password_policy** is set to **1**, the default password complexity rule is used to check passwords. + +Passwords in the weak password dictionary are stored in the **gs_global_config** system catalog (the record whose name field is **weak_password** is the stored weak password). When a user is created or modified, the password set by the user is compared with the password stored in the weak password dictionary. If the password is matched, a message is displayed, indicating that the password is weak and the password fails to be set. + +The weak password dictionary is empty by default. You can add or delete weak passwords using the following syntax: + +```sql +CREATE WEAK PASSWORD DICTIONARY WITH VALUES ('password1'), ('password2'); + +DROP WEAK PASSWORD DICTIONARY; +``` + +In the preceding statement, **password1** and **password2** are weak passwords prepared by users. After the statement is executed successfully, the passwords are saved to the weak password system catalog. + +When a user attempts to run the CREATE WEAK PASSWORD DICTIONARY statement to insert a weak password that already exists in the table, only one weak password is retained in the table. + +The DROP WEAK PASSWORD DICTIONARY statement clears weak passwords in the entire system catalog. + +The gs_global_config system catalog does not have a unique index. You are not advised to use the COPY FROM statement to copy the same data to the gs_global_config system catalog. + +To audit weak password operations, set the third bit of the value of the **audit_system_object** parameter to **1**. + +## Enhancements + +In MogDB, the weak password dictionary function is implemented. + +## Constraints + +- Initial users, system administrators, and security administrators can view, add, and delete weak password dictionaries. +- Common users can view but cannot add or delete weak password dictionaries. + +## Dependencies + +None diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/12-equality-query-in-a-fully-encrypted-database.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/12-equality-query-in-a-fully-encrypted-database.md new file mode 100644 index 0000000000000000000000000000000000000000..b3bb341e195a2b995d7d3f8b09218c26ab986ad3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/12-equality-query-in-a-fully-encrypted-database.md @@ -0,0 +1,88 @@ +--- +title: Equality Query in a Fully-encrypted Database +summary: Equality Query in a Fully-encrypted Database +author: Guo Huan +date: 2022-05-07 +--- + +# Equality Query in a Fully-encrypted Database + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The encrypted database aims to protect privacy throughout the data lifecycle. In this way, data is always in ciphertext during transmission, computing, and storage regardless of the service scenario and environment. After the data owner encrypts data on the client and sends the encrypted data to the server, no attacker can obtain valuable information even if the attacker steals user data by exploiting system vulnerabilities. In this way, data privacy is protected. + +## Benefits + +The entire service data flow is in ciphertext during data processing, so the following can be implemented by using a fully-encrypted database: + +1. Protect data privacy and security throughout the lifecycle on the cloud. Attackers cannot obtain valid information from the database server regardless of the data status. +2. Help cloud service providers obtain third-party trust. Users, including service administrators and O&M administrators in enterprise service scenarios and application developers in consumer cloud services, can keep keys by themselves so that users with high permissions cannot obtain valid data. +3. Enable cloud databases to better comply with personal privacy protection laws and regulations with the help of the fully-encrypted database. + +## Description + +From the perspective of users, the encrypted equality query functions are divided into three parts, which are implemented by the newly added KeyTool and the enhanced MogDB gsql client. + +First, this feature provides the client key management function. Users can use KeyTool to generate, destroy, and update CMKs, and import and export keys. With the import and export functions of KeyTool, CMKs can be transmitted between different clients. In addition, the KeyTool implements key management on a single client. By configuring management files, you can store and update keys. + +In addition, this feature provides the key creation and encrypted table creation functions. The SQL syntax CREATE CLINET MASTER KEY and CREATE COLUMN ENCRYPTION KEY are added to record and manage CMK and CEK metadata in the database. The CMK and CEK information is recorded in the new system catalog. The CREATE TABLE syntax is extended to specify a column encryption key and encryption algorithm for each sensitive information column in a table, facilitating subsequent ciphertext data storage. + +This feature supports the encrypted equality query function, which is the core of the entire feature. Although users are unaware of the ciphertext query, the query of sensitive data is restricted by the specifications of the current encrypted equality query. + +From the overall perspective, this feature is used to store and manage data based on sensitive data protection requirements and implement query tasks based on ciphertext data. + +## Enhancements + +None. + +## Constraints + +- Data is encrypted at the column level, and encryption policies cannot be differentiated by row level. +- Except the RENAME operation, the ALTER TABLE syntax cannot be used to change columns in an encrypted table (including the conversion between encrypted and unencrypted columns). The ADD and DROP operations can be used to add and delete encrypted columns, respectively. +- The CHECK(COLUMN IS NOT NULL) syntax can be used, but most check constraint syntax cannot be set for encrypted columns. +- When **support_extended_features** is set to **off**, primary key and unique cannot be used for encrypted columns. When **support_extended_features** is set to **on**, only primary key and unique can be used for encrypted columns. +- Different data types cannot be implicitly converted. +- The set operation cannot be performed between ciphertexts of different data types. +- Range partitioning cannot be created for encrypted columns. +- Only the repeat and empty_blob() functions can be used to encrypt columns. +- The current version supports only gsql and JDBC (deployed on a Linux OS) clients. Other clients such as ODBC do not support encrypted equality query. +- Data can only be imported to the encrypted table by running **copy from stdin**, **\copy**, or **insert into values (…)** on the client. +- Copying an encrypted table to a file is not supported. +- The system does not support encrypted queries, such as sorting, range query, and fuzzy query, except equality query. +- The encrypted syntax of stored procedures for some functions is supported. For details about the constraints, see “Encrypted Functions and Stored Procedures” in the *Developer Guide*. +- Non-encrypted table data cannot be inserted into encrypted table data using the **INSERT INTO… SELECT…** or **MERGE INTO** syntax. +- For a request in connection state, the CEK information change on the server can be detected only after the cache update operation is triggered (for example, the user is changed or the encrypted column fails to be decrypted) and the connection is re-established. +- Encrypted equality query is not supported on columns encrypted using the random encryption algorithm. +- An error is reported if the two attribute conditions used for comparison in the encrypted equality query use different data encryption keys. +- Encrypted equality query is not supported in time series tables and foreign tables. The ustore storage engine is not supported. +- If the database service configuration (such as the pg_settings system catalog, permission, key, and encrypted column) is changed, you need to re-establish a JDBC connection to make the configuration take effect. +- Multiple SQL statements cannot be executed at the same time. This constraint does not apply to the scenario where the INSERT INTO statement is executed in multiple batches. +- Encrypted equality query supports the following data types: + +| Category | Type | Description | +| -------------------- | ------------------ | ------------------------------------------------------------ | +| Integer types | tinyint/tinyint(n) | Tiny integer, which is the same as int1. | +| | smallint | Small integer, which is the same as int2. | +| | int4 | Common integer. | +| | binary_integer | Oracle compatibility type. Generally, the value is an integer. | +| | bigint/bigint(n) | Big integer, which is the same as int8. | +| Numeric data types | numeric(p,s) | A number with the precision **p**. | +| | number | Oracle compatibility type, which is the same as numeric(p,s). | +| Floating point types | float4 | Single-precision floating point. | +| | float8 | Double-precision floating point. | +| | double precision | Double-precision floating point. | +| Character data types | char/char(n) | Fixed-length character string. If the length is insufficient, add spaces. The default precision is **1**. | +| | varchar(n) | Variable-length character string, where **n** indicates the maximum number of bytes. | +| | text | Text type. | +| | varchar2(n) | Oracle compatibility type, which is the same as varchar(n). | +| | clob | Character large object. | +| Binary data types | bytea | Variable-length binary string. | +| | blob | Binary large object. | + +## Dependencies + +None. diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/13-ledger-database-mechanism.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/13-ledger-database-mechanism.md new file mode 100644 index 0000000000000000000000000000000000000000..f1316f08884c8386f1114a468c9ab00f5f1498e3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/13-ledger-database-mechanism.md @@ -0,0 +1,43 @@ +--- +title: Ledger Database Mechanism +summary: Ledger Database Mechanism +author: Guo Huan +date: 2022-05-07 +--- + +# Ledger Database Mechanism + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +The ledger database feature includes adding the verification information to a tamper-proof table specified by a user and recording the user's data operation history. The consistency between the data and operation history is checked to ensure that the user data cannot be maliciously tampered with. When a user performs DML operations on a tamper-proof table, the system adds a small amount of additional row-level verification information to the table and records the SQL statements and data change history. The feature provides a verification API for users to check whether the data in the tamper-proof table is consistent with the operation information recorded by the system. + +## Benefits + +The ledger database provides user data operation records, historical data change records, and easy-to-use consistency verification API to help users check whether sensitive information in the database is maliciously tampered with at any time, effectively improving the tamper-proof capability of the database. + +## Description + +The ledger database uses the ledger schema to isolate common tables from tamper-proof user tables. If a row-store table created in the ledger schema has the tamper-proof attribute, it is a tamper-proof user table. When data is inserted into a tamper-proof user table, the system automatically generates a small amount of row-level verification information. When a user executes DML, the system records user operations in the global blockchain table (GS_GLOBAL_CHAIN) and records data changes in the historical table corresponding to the user table. The data in operation records, data change records, and the user table must be the same. The ledger database provides a high-performance verification API for users to verify data consistency. If the consistency verification fails, the data may be tampered with. In this case, contact the audit administrator to trace the operation history. + +## Enhancements + +None. + +## Constraints + +- In tamper-proof schema, row-store tables are tamper-proofing, whereas temporary tables, unlogged tables, column-store tables, and time series tables are not. +- The structure of the tamper-proof user table cannot be modified. The tamper-proof tables cannot be truncated. The tamper-proof user table cannot be switched to a common schema. The non-tamper-proof table cannot be switched to the tamper-proof schema. +- If the tamper-proof table is a partitioned table, operations such as exchange partition, drop partition and truncate partition are not supported. +- Functions and triggers cannot be used to modify data in a tamper-proof user table. +- When a tamper-proof user table is created, the column named **hash** cannot exist. +- Common users can call the tampering verification API to verify only tables that they have the permission to query. +- Only the audit administrator and initial user can query the global blockchain table and tables in BLOCKCHAIN schema. Common users do not have the permission to access and all users do not have the permission to modify the tables. +- According to the naming rules of historical tables, if the name of the schema or table to be created ends or starts with an underscore (_), the name of the corresponding historical table may conflict with that of an existing table. In this case, you need to rename the table. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/14-transparent-data-encryption.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/14-transparent-data-encryption.md new file mode 100644 index 0000000000000000000000000000000000000000..9fadc9bfd04b1d1b471bbc2e24e880027777bf65 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/14-transparent-data-encryption.md @@ -0,0 +1,53 @@ +--- +title: Transparent Data Encryption +summary: Transparent Data Encryption +author: Guo Huan +date: 2022-05-07 +--- + +# Transparent Data Encryption + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +Transparent data encryption (TDE) encrypts data when the database writes the data to the storage medium and automatically decrypts the data when reading the data from the storage medium. This prevents attackers from reading data in the data file without database authentication, solving the static data leakage problem. This function is almost transparent to the application layer. You can determine whether to enable the transparent data encryption function as required. + +## Benefits + +To prevent attackers from reading data files without authentication, you can use the transparent data encryption function to encrypt data files in the database. This ensures that users can read decrypted data only after starting and connecting to the database properly. + +## Description + +The three-layer key structure is used to implement the key management mechanism, including the root key (RK), cluster master key (CMK), and data encryption key (DEK). CMKs are encrypted and protected by RKs, and DEKs are encrypted and protected by CMKs. DEKs are used to encrypt and decrypt user data. Each table corresponds to a DEK. + +Table-level encryption is supported. When creating a table, you can specify whether to encrypt the table and the encryption algorithm to be used. The encryption algorithm can be AES_128_CTR or SM4_CTR, which cannot be changed once specified. If an encrypted table is created, the database automatically applies for a DEK for the table and saves the encryption algorithm, key ciphertext, and corresponding CMK ID in the **reloptions** column of the pg_class system catalog in keyword=value format. + +You can switch an encrypted table to a non-encrypted table or switch a non-encrypted table to an encrypted table. If the encryption function is not enabled when a table is created, the table cannot be switched to an encrypted table. + +For encrypted tables, DEK rotation is supported. After the key rotation, the data encrypted using the old key is decrypted using the old key, and the newly written data is encrypted using the new key. The encryption algorithm is not changed during key rotation. + +## Enhancements + +None. + +## Constraints + +The current version interconnects with HUAWEI CLOUD KMS to support table-level key storage and row-store table encryption. The specifications are as follows: + +- Encryption of a row-store table stored as a heap is supported. +- Column-store encryption, materialized view encryption, and ustore storage engine encryption are not supported. +- Indexes, sequences, Xlogs, MOTs, and system catalogs cannot be encrypted. +- You can specify an encryption algorithm when creating a table. Once specified, the encryption algorithm cannot be changed. If **enable_tde** is set to **on** but the encryption algorithm **encrypt_algo** is not specified when a table is created, the AES-128-CTR encryption algorithm is used by default. +- If the encryption function is not enabled or the encryption algorithm is not specified when a table is created, the table cannot be switched to an encrypted table. +- For a table that has been assigned an encryption key, switching between the encrypted and unencrypted states of the table does not change the key or encryption algorithm. +- Data key rotation is supported only when the table encryption function is enabled. +- Cross-region primary/standby synchronization of multiple copies in a single cluster is not supported. Cross-region scaling of a single cluster is not supported. Cross-region backup and restoration, cluster DR, and data migration are not supported. +- In hybrid cloud scenarios, if the HUAWEI CLOUD KMS and management plane functions are used, transparent data encryption is supported. Other KMS services are not supported if their APIs are incompatible. +- The query performance of encrypted tables is lower than that of non-encrypted tables. If high performance is required, exercise caution when enabling the encryption function. + +## Dependencies + +The key management service is provided by the external KMS. The current version can interconnect with HUAWEI CLOUD KMS. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/2-separation-of-control-and-access-permissions.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/2-separation-of-control-and-access-permissions.md new file mode 100644 index 0000000000000000000000000000000000000000..0cd785df93e7670f404a905bda566881b57ead75 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/2-separation-of-control-and-access-permissions.md @@ -0,0 +1,46 @@ +--- +title: Separation of Control and Access Permissions +summary: Separation of Control and Access Permissions +author: Guo Huan +date: 2022-05-07 +--- + +# Separation of Control and Access Permissions + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +The control permissions and the access permissions can be separated. + +## Benefits + +The control permissions of database administrators for tables need to be isolated from their access permissions to improve the data security of common users. + +## Description + +If multiple business departments use different database users to perform service operations and a database maintenance department at the same level uses database administrators to perform O&M operations, the business departments may require that database administrators can only perform control operations (**DROP**, **ALTER**, and **TRUNCATE**) and cannot perform access operations (**INSERT**, **DELETE**, **UPDATE**, **SELECT**, and **COPY**) without authorization. That is, the control permissions of database administrators for tables need to be isolated from their access permissions to improve the data security of common users. + +In separation-of-duties mode, a database administrator does not have permissions for the tables in schemas of other users. In this case, database administrators have neither control permissions nor access permissions. This does not meet the requirements of the business departments mentioned above. Therefore, MogDB provides private users to solve the problem. That is, create private users with the **INDEPENDENT** attribute in non-separation-of-duties mode. Users with the CREATEROLE permission or the system administrator permission can create private users or change the attributes of common users to private users. Common users can also change their own attributes to private users. + +```sql +MogDB=# CREATE USER user_independent WITH INDEPENDENT IDENTIFIED BY "1234@abc"; +``` + +System administrators can manage (**DROP**, **ALTER**, and **TRUNCATE**) table objects of private users but cannot access (**INSERT**, **DELETE**, **SELECT**, **UPDATE**, **COPY**, **GRANT**, **REVOKE**, and **ALTER OWNER**) the objects before being authorized. + +## Enhancements + +None + +## Constraints + +For a table owned by a private user, grant the trigger permission of the table to other users with caution to prevent other users from using the trigger to view the data of the private user. + +If permissions related to private user tables are granted to non-private users, the system administrator will obtain the same permissions. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/3-database-encryption-authentication.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/3-database-encryption-authentication.md new file mode 100644 index 0000000000000000000000000000000000000000..112eaf3884d9952ac9fe6b06e9c6f1db995fdc5d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/3-database-encryption-authentication.md @@ -0,0 +1,36 @@ +--- +title: Database Encryption Authentication +summary: Database Encryption Authentication +author: Guo Huan +date: 2022-05-07 +--- + +# Database Encryption Authentication + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The client/server (C/S) mode-based client connection authentication mechanism is provided. + +## Benefits + +The unidirectional, irreversible hash encryption algorithm PBKDF2 is used for encryption and authentication, effectively defending against rainbow attacks. + +## Description + +MogDB uses a basic client connection authentication mechanism. After a client initiates a connection request, the server verifies the information and sends the information required for authentication to the client based on the verification result. The authentication information includes the salt, token, and server signature. The client responds to the request and sends the authentication information to the server. The server calls the authentication module to authenticate the client authentication information. The user password is encrypted and stored in the memory. During the entire authentication process, passwords are encrypted for storage and transmission. When the user logs in to the system next time, the hash value is calculated and compared with the key value stored on the server to verify the correctness. + +## Enhancements + +The message processing flow in the unified encryption and authentication process effectively prevents attackers from cracking the username or password by capturing packets. + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/4-data-encryption-and-storage.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/4-data-encryption-and-storage.md new file mode 100644 index 0000000000000000000000000000000000000000..27efdbfc71d21e4d407c0b226eb8b44784f16932 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/4-data-encryption-and-storage.md @@ -0,0 +1,48 @@ +--- +title: Data Encryption and Storage +summary: Data Encryption and Storage +author: Guo Huan +date: 2022-05-07 +--- + +# Data Encryption and Storage + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Imported data is encrypted before stored. + +## Benefits + +You can use encrypted import interfaces to encrypt sensitive information and store it in a table. + +## Description + +MogDB provides the encryption functions **gs_encrypt_aes128()** and **gs_encrypt()**, and decryption functions **gs_decrypt_aes128()** and **gs_decrypt()**. Before you import data to a certain column in a table, you can use this function to encrypt the data. The function can be called using a statement in the following format: + +```shell +gs_encrypt_aes128(column, key), gs_encrypt (decryptstr, keystr, decrypttype) +``` + +In the preceding command, **key** indicates the initial password specified by the user, which is used to derive the encryption key. To encrypt an entire table, you need to write an encryption function for each column. + +If a user with the required permission wants to view specific data, the user can decrypt required columns using the decryption function interface **gs_decrypt_aes128(***column***, ***key\***)**. To invoke the interface, run the following command: + +```shell +gs_decrypt_aes128(column, key), gs_decrypt(decryptstr, keystr, decrypttype) +``` + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/5-database-audit.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/5-database-audit.md new file mode 100644 index 0000000000000000000000000000000000000000..8ee310d00881cd797d30856c749fbddc933a2a8b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/5-database-audit.md @@ -0,0 +1,36 @@ +--- +title: Database Audit +summary: Database Audit +author: Guo Huan +date: 2022-05-07 +--- + +# Database Audit + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +Audit logs record user operations performed on database startup and stopping, as well as connection, DDL, DML, and DCL operations. + +## Benefits + +The audit log mechanism enhances the database capability of tracing unauthorized operations and collecting evidence. + +## Description + +Database security is essential for a database system. MogDB writes all user operations in the database into audit logs. Database security administrators can use the audit logs to reproduce a series of events that cause faults in the database and identify unauthorized users, unauthorized operations, and the time when these operations are performed. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/6-network-communication-security.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/6-network-communication-security.md new file mode 100644 index 0000000000000000000000000000000000000000..113013ad8b1022c9b23648a9dbfee046ac69b5c2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/6-network-communication-security.md @@ -0,0 +1,44 @@ +--- +title: Network Communication Security +summary: Network Communication Security +author: Guo Huan +date: 2022-05-07 +--- + +# Network Communication Security + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +To secure the transmission of sensitive data on the Internet, MogDB encrypts communication between the server and the client using the Secure Socket Layer (SSL) protocol. + +## Benefits + +The communication between your client and the server can be secured. + +## Description + +MogDB supports the SSL protocol. The SSL protocol is an application-layer communication protocol with high security, which is mainly used for secure web transmission. SSL contains a record layer and a transport layer. The record-layer protocol determines the encapsulation format of the transport-layer data. The transport-layer security protocol uses X.509 for authentication. The SSL protocol uses asymmetric encryption algorithms to authenticate the identities of communicating parties, and then the two parties exchange symmetric keys as communication keys. The SSL protocol effectively ensures the confidentiality and reliability of the communication between two applications and prevents the communication between a client and a server from being eavesdropped by attackers. + +MogDB also supports the TLS 1.2 protocol. TLS 1.2 is a transport-layer communication protocol with high security. It consists of the TLS Record and TLS Handshake protocols. Each protocol suit has information in multiple formats. The TLS protocol is independent of application-layer protocols. Upper-layer protocols can be transparently distributed on the TLS protocol. The TLS protocol ensures the data confidentiality and integrity for both communication parties. + +## Enhancements + +Checking the strength of certificate signature algorithms: For low-strength signature algorithms, alarms are reported, reminding you to replace the certificate with another certificate containing a high-strength signature algorithm. + +Checking the certificate validity period: If a certificate is about to expire in less than seven days, an alarm is reported, reminding you to replace the certificate on the client. + +Checking certificate permissions: The certificate permissions are verified at the connection setup stage. + +## Constraints + +The formal certificates and keys for servers and clients shall be obtained from the Certificate Authority (CA). Assume the private key and certificate for a server are **server.key** and **server.crt**, the private key and certificate for the client are **client.key** and **client.crt**, and the CA root certificate is **cacert.pem**. + +You need to enable the SSL protocol and configure the certificate and connection mode. + +## Dependencies + +OpenSSL \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/7-resource-label.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/7-resource-label.md new file mode 100644 index 0000000000000000000000000000000000000000..7e2c55241fa419163216b9e256a08dcd08549896 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/7-resource-label.md @@ -0,0 +1,49 @@ +--- +title: Resource Label +summary: Resource Label +author: Guo Huan +date: 2022-05-07 +--- + +# Resource Label + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +Database resources refer to database objects, such as databases, schemas, tables, columns, views, and triggers. The more the database objects are, the more complex the classification management of database resources is. The resource label mechanism is a technology that classifies and labels database resources with certain features to implement resource classification management. After adding labels to some resources in a database, administrators can perform operations such as data audit or anonymization using the labels to implement security management on labeled database resources. + +## Benefits + +Proper resource labels can be used to effectively classify data objects, improve management efficiency, and simplify security policy configuration. To perform unified audit or data anonymization on a group of database resources, the administrator can allocate a resource label to these resources first. The label indicates that the database resources have a certain feature or require unified configuration of a certain policy. The administrator can directly perform operations on the resource label, which greatly reduces the complexity of policy configuration and information redundancy as well as improves management efficiency. + +## Description + +The resource label mechanism selectively classifies resources in the current database. Administrators can use the following SQL syntax to create a resource label and add the label to a group of database resources: + +```sql +CREATE RESOURCE LABEL schm_lb ADD SCHEMA(schema_for_label); +CREATE RESOURCE LABEL tb_lb ADD TABLE(schema_for_label.table_for_label); +CREATE RESOURCE LABEL col_lb ADD COLUMN(schema_for_label.table_for_label.column_for_label); +CREATE RESOURCE LABEL multi_lb ADD SCHEMA(schema_for_label), TABLE(table_for_label); +``` + +**schema_for_label**, **table_for_label**, and **column_for_label** indicate the schema, table, and column to be labeled, respectively. The **schm_lb** label is added to schema **schm_for_label**, **tb_lb** is added to table **table_for_label**, **col_lb** is added to column **column_for_label**, and **multi_lb** is added to schema **schm_for_label** and table **table_for_label**. You can perform unified audit or dynamic data anonymization using the configured resource labels, that is, manage all labeled database resources. + +Currently, resource labels support the following database resource types: schema, table, column, view, and function. + +## Enhancements + +None + +## Constraints + +- Resource labels can be created only by a user with the **POLADMIN** and **SYSADMIN** attributes or an initial user. +- Resource labels cannot be created for temporary tables. +- Columns in the same basic table can belong to only one resource tag. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/8-unified-audit.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/8-unified-audit.md new file mode 100644 index 0000000000000000000000000000000000000000..ce0ffa0f9931f78de94172d305d8ba653de0d412 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/8-unified-audit.md @@ -0,0 +1,74 @@ +--- +title: Unified Audit +summary: Unified Audit +author: Guo Huan +date: 2022-05-07 +--- + +# Unified Audit + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The audit mechanism is a security management solution that can effectively deal with the attackers' repudiation. The larger the audit scope is, the more operations can be monitored and the more audit logs are generated, affecting the actual audit efficiency. The unified audit mechanism is a technology that implements efficient security audit management by customizing audit policies. After the administrator defines the audit object and audit behaviors, if the task executed by a user is associated with an audit policy, the corresponding audit behavior is generated and the audit log is recorded. Customized audit policies can cover common user management activities, as well as DDL and DML operations, meeting routine audit requirements. + +## Benefits + +Audit is indispensable for routine security management. When a traditional audit mechanism is used to audit an operation, such as **SELECT**, a large number of audit logs are generated, increasing the I/O of the entire system and affecting the system performance and audit efficiency of administrators. The unified audit mechanism allows you to customize policies for generating audit logs. For example, only the operation that database account **A** queries table **a** is audited. Customized audit greatly reduces the number of generated audit logs, ensuring audit behaviors and reducing the impact on system performance. In addition, customized audit policies can improve the audit efficiency of administrators. + +## Description + +The unified audit mechanism customizes audit behaviors based on resource labels and classifies the supported audit behaviors into the **ACCESS** and **PRIVILEGES** classes. The SQL syntax for creating a complete audit policy is as follows: + +```sql +CREATE RESOURCE LABEL auditlabel add table(table_for_audit1, table_for_audit2); +CREATE AUDIT POLICY audit_select_policy ACCESS SELECT ON LABEL(auditlabel) FILTER ON ROLES(usera); +CREATE AUDIT POLICY audit_admin_policy PRIVILEGES ALTER, DROP ON LABEL(auditlabel) FILTER ON IP(local); +``` + +**auditlabel** indicates the resource label in the current audit, which contains two table objects. **audit_select_policy** defines the audit policy for user **usera** to audit the **SELECT** operation on the objects with the **auditlabel** label, regardless of the access source. **audit_admin_policy** defines a local audit policy for **ALTER** and **DROP** operations on the objects with the **auditlabel** label, regardless of the user. If **ACCESS** and **PRIVILEGES** are not specified, all DDL and DML operations on objects with a resource label are audited. If no audit objects are specified, operations on all objects are audited. The addition, deletion, and modification of unified audit policies are also recorded in unified audit logs. + +Currently, unified audit supports the following audit behaviors: + +| **SQL Type** | Supported operations and object types | +| ------------ | ------------------------------------------------------------ | +| DDL | Operations: ALL, ALTER, ANALYZE, COMMENT, CREATE, DROP, GRANT, and REVOKE
SET SHOW
Objects: DATABASE, SCHEMA, FUNCTION, TRIGGER, TABLE, SEQUENCE, FOREIGN_SERVER, FOREIGN_TABLE, TABLESPACE, ROLE/USER, INDEX, VIEW, and DATA_SOURCE | +| DML | Operations: ALL, COPY, DEALLOCATE, DELETE, EXECUTE, REINDEX, INSERT, PREPARE, SELECT, TRUNCATE, and UPDATE | + +## Enhancements + +None. + +## Constraints + +- The unified audit policy must be created by a user with the **POLADMIN** or **SYSADMIN** attribute, or by the initial user. Common users do not have the permission to access the security policy system catalog and system view. + +- The syntax of a unified audit policy applies to either DDL or DML operations. DDL operations and DML operations are mutually exclusive in an audit policy. A maximum of 98 unified audit policies can be configured. + +- Unified audit monitors the SQL statements executed by users on the clients, but does not record the internal SQL statements of databases. + +- In the same audit policy, the same resource tag can be bound to different audit behaviors, and the same behavior can be bound to different resource tags. The ALL operation type includes all operations supported by DDL or DML. + +- A resource label can be associated with different unified audit policies. Unified audit outputs audit information in sequence based on the policies matched by SQL statements. + +- Audit logs of unified audit policies are recorded separately. Currently, no visualized query interfaces are provided. Audit logs depend on the OS service Rsyslog and are archived through the service configuration. + +- In cloud service scenarios, logs need to be stored in the OBS. In hybrid cloud scenarios, you can deploy Elasticsearch to collect CN logs and perform visualized processing. + +- It is recommended that **APP** in **FILTER** be set to applications in the same trusted domain. Since a client may be forged, a security mechanism must be formed on the client when **APP** is used to reduce misuse risks. Generally, you are not advised to set **APP**. If it is set, pay attention to the risk of client spoofing. + +- Taking an IPv4 address as an example, the following formats are supported: + + | IP Address Format | Example | + | -------------------- | ------------------------ | + | Single IP address | 127.0.0.1 | + | IP address with mask | 127.0.0.1\|255.255.255.0 | + | CIDR IP address | 127.0.0.1⁄24 | + | IP address segment | 127.0.0.1-127.0.0.5 | + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/database-security/9-dynamic-data-anonymization.md b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/9-dynamic-data-anonymization.md new file mode 100644 index 0000000000000000000000000000000000000000..7410dba29c203d7989881f4869c3296c7e7c6f46 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/database-security/9-dynamic-data-anonymization.md @@ -0,0 +1,106 @@ +--- +title: Dynamic Data Masking +summary: Dynamic Data Masking +author: Guo Huan +date: 2022-05-07 +--- + +# Dynamic Data Masking + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Data masking is an effective database privacy protection solution, which can prevent attackers from snooping on private data. The dynamic data masking mechanism is a technology that protects privacy data by customizing masking policies. It can effectively prevent unauthorized users from accessing sensitive information while retaining original data. After the administrator specifies the object to be anonymized and customizes a data masking policy, if the database resources queried by a user are associated with a masking policy, data is anonymized based on the user identity and masking policy to restrict attackers' access to privacy data. + +## Benefits + +Data privacy protection is one of the required database security capabilities. It can restrict attackers' access to privacy data, ensuring privacy data security. The dynamic data masking mechanism can protect the privacy of specified database resources by configuring masking policies. In addition, the masking policy configuration is flexible and can implement targeted privacy protection in specific user scenarios. + +## Description + +The dynamic data masking mechanism customizes masking policies based on resource labels. It can select masking modes based on the site requirements or customize masking policies for specific users. The SQL syntax for creating a complete masking policy is as follows: + +```sql +CREATE RESOURCE LABEL label_for_creditcard ADD COLUMN(user1.table1.creditcard); +CREATE RESOURCE LABEL label_for_name ADD COLUMN(user1.table1.name); +CREATE MASKING POLICY msk_creditcard creditcardmasking ON LABEL(label_for_creditcard); +CREATE MASKING POLICY msk_name randommasking ON LABEL(label_for_name) FILTER ON IP(local), ROLES(dev); +``` + +**label_for_creditcard** and **msk_name** are the resource labels for masking, and each label is allocated to two column objects. **creditcardmasking** and **randommasking** are preset masking functions. **msk_creditcard** specifies that the masking policy **creditcardmasking** will be applied when any user accesses resources with **label_for_creditcard**, regardless of the access source. **msk_name** specifies that the masking policy **randommasking** will be applied when local user **dev** accesses resources with **label_for_name**. If **FILTER** is not specified, the setting takes effect for all users. Otherwise, the setting takes effect only for specified users. + +The following table shows the preset masking functions: + +| **Masking Function** | **Example** | +| -------------------- | ------------------------------------------------------------ | +| creditcardmasking | '4880-9898-4545-2525' will be anonymized as 'xxxx-xxxx-xxxx-2525'. This function anonymizes digits except the last four digits. | +| basicemailmasking | 'abcd@gmail.com' will be anonymized as 'xxxx@gmail.com'. This function anonymizes text before the first @. | +| fullemailmasking | 'abcd@gmail.com' will be anonymized as 'xxxx@xxxxx.com'. This function anonymizes text before the first dot (.) (except @). | +| alldigitsmasking | 'alex123alex' will be anonymized as 'alex000alex'. This function anonymizes only digits in the text. | +| shufflemasking | 'hello word' will be anonymized as 'hlwoeor dl'. This weak masking function is implemented through character dislocation. You are not advised to use this function to anonymize strings with strong semantics. | +| randommasking | 'hello word' will be anonymized as 'ad5f5ghdf5'. This function randomly anonymizes text by character. | +| maskall | '4880-9898-4545-2525' will be anonymized as 'xxxxxxxxxxxxxxxxxxx'. | + +The data types supported by each masking function are as follows: + +| **Masking Function** | **Supported Data Types** | +| -------------------- | ------------------------------------------------------------ | +| creditcardmasking | BPCHAR, VARCHAR, NVARCHAR, TEXT (character data in credit card format only) | +| basicemailmasking | BPCHAR, VARCHAR, NVARCHAR, TEXT (character data in email format only) | +| fullemailmasking | BPCHAR, VARCHAR, NVARCHAR, TEXT (character data in email format only) | +| alldigitsmasking | BPCHAR, VARCHAR, NVARCHAR, TEXT (character data containing digits only) | +| shufflemasking | BPCHAR, VARCHAR, NVARCHAR, TEXT (text data only) | +| randommasking | BPCHAR, VARCHAR, NVARCHAR, TEXT (text data only) | +| maskall | BOOL, RELTIME, TIME, TIMETZ, INTERVAL, TIMESTAMP, TIMESTAMPTZ, SMALLDATETIME, ABSTIME, TEXT, BPCHAR, VARCHAR, NVARCHAR2, NAME, INT8, INT4, INT2, INT1, NUMRIC, FLOAT4, FLOAT8, CASH | + +For unsupported data types, the **maskall** function is used for data masking by default. The data of the BOOL type is masked as **'0'**. The RELTIME type is masked as **'1970'**. The TIME, TIMETZ, and INTERVAL types are masked as **'00:00:00.0000+00'**. The TIMESTAMP, TIMESTAMPTZ, SMALLDATETIME, and ABSTIME types are masked as **'1970-01-01 00:00:00.0000'**. The TEXT, CHAR, BPCHAR, VARCHAR, NVARCHAR2, and NAME type are masked as **'x'**. The INT8, INT4, INT2, INT1, NUMERIC, FLOAT4, FLOAT8 types are masked as **'0'**. If the data type is not supported by **maskall**, the masking policy cannot be created. If implicit conversion is involved in the masking column, the data type after implicit conversion is used for masking. In addition, if the masking policy is applied to a data column and takes effect, operations on the data in the column are performed based on the masking result. + +Dynamic data masking applies to scenarios closely related to actual services. It provides users with proper masking query APIs and error handling logic based on service requirements to prevent raw data from being obtained through credential stuffing. + +## Enhancements + +None. + +## Constraints + +- The dynamic data masking policy must be created by a user with the **POLADMIN** or **SYSADMIN** attribute, or by the initial user. Common users do not have the permission to access the security policy system catalog and system view. + +- Dynamic data masking takes effect only on data tables for which masking policies are configured. Audit logs are not within the effective scope of the masking policies. + +- In a masking policy, only one masking mode can be specified for a resource label. + +- Multiple masking policies cannot be used to anonymize the same resource label, except when **FILTER** is used to specify user scenarios where the policies take effect and there is no intersection between user scenarios of different masking policies that contain the same resource label. In this case, you can identify the policy that a resource label is anonymized by based on the user scenario. + +- It is recommended that **APP** in **FILTER** be set to applications in the same trusted domain. Since a client may be forged, a security mechanism must be formed on the client when **APP** is used to reduce misuse risks. Generally, you are not advised to set **APP**. If it is set, pay attention to the risk of client spoofing. + +- For INSERT or MERGE INTO operations with the query clause, if the source table contains anonymized columns, the inserted or updated result in the preceding two operations is the anonymized value and cannot be restored. + +- When the built-in security policy is enabled, the ALTER TABLE EXCHANGE PARTITION statement fails to be executed if the source table is in the anonymized column. + +- If a dynamic data masking policy is configured for a table, grant the trigger permission of the table to other users with caution to prevent other users from using the trigger to bypass the masking policy. + +- A maximum of 98 dynamic data masking policies can be created. + +- Only the preceding seven preset masking policies can be used. + +- Only data with the resource labels containing the **COLUMN** attribute can be anonymized. + +- Only columns in base tables can be anonymized. + +- Only the data queried using **SELECT** can be anonymized. + +- Taking an IPv4 address as an example, the following formats are supported: + + | IP Address Format | Example | + | -------------------- | ------------------------ | + | Single IP address | 127.0.0.1 | + | IP address with mask | 127.0.0.1\|255.255.255.0 | + | CIDR IP address | 127.0.0.1⁄24 | + | IP address segment | 127.0.0.1-127.0.0.5 | + +## Dependencies + +None. diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/1-support-for-functions-and-stored-procedures.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/1-support-for-functions-and-stored-procedures.md new file mode 100644 index 0000000000000000000000000000000000000000..83307662b1ef48a691f980ecc6bd3248729608f7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/1-support-for-functions-and-stored-procedures.md @@ -0,0 +1,38 @@ +--- +title: Support for Functions and Stored Procedures +summary: Support for Functions and Stored Procedures +author: Guo Huan +date: 2022-05-07 +--- + +# Support for Functions and Stored Procedures + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Functions and stored procedures are important database objects. They encapsulate SQL statement sets used for certain functions so that the statements can be easily invoked. + +## Benefits + +1. Allows customers to modularize program design and encapsulate SQL statement sets, easy to invoke. +2. Caches the compilation results of stored procedures to accelerate SQL statement set execution. +3. Allows system administrators to restrict the permission for executing a specific stored procedure and controls access to the corresponding type of data. This prevents access from unauthorized users and ensures data security. + +## Description + +MogDB supports functions and stored procedures compliant with the SQL standard. The stored procedures are compatible with certain mainstream stored procedure syntax, improving their usability. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/10-autonomous-transaction.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/10-autonomous-transaction.md new file mode 100644 index 0000000000000000000000000000000000000000..e51b7c6fc3863f79b1e8c2144185d9b7952bf1f8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/10-autonomous-transaction.md @@ -0,0 +1,44 @@ +--- +title: Autonomous Transaction +summary: Autonomous Transaction +author: Guo Huan +date: 2022-05-07 +--- + +# Autonomous Transaction + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +An autonomous transaction is a type of transaction in which the commit of a sub-transaction is not affected by the commit or rollback of the main transaction. + +## Benefits + +This feature meets diversified application scenarios. + +## Description + +In an autonomous transaction, a specified type of SQL statements are executed in an independent transaction context during the execution of the main transaction. The commit and rollback operations of an autonomous transaction are not affected by the commit and rollback operations of the main transaction. + +User-defined functions and stored procedures support autonomous transactions. + +A typical application scenario is as follows: A table is used to record the operation information during the main transaction execution. When the main transaction fails to be rolled back, the operation information recorded in the table cannot be rolled back. + +## Enhancements + +None + +## Constraints + +- A trigger function does not support autonomous transactions. +- In the autonomous transaction block of a function or stored procedure, static SQL statements do not support variable transfer. +- Autonomous transactions do not support nesting. +- A function containing an autonomous transaction does not support the return value of parameter transfer. +- A stored procedure or function that contains an autonomous transaction does not support exception handling. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/11-global-temporary-table.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/11-global-temporary-table.md new file mode 100644 index 0000000000000000000000000000000000000000..35ba96555d029e04c76afd34d6e2706e20169947 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/11-global-temporary-table.md @@ -0,0 +1,45 @@ +--- +title: Global Temporary Table +summary: Global Temporary Table +author: Guo Huan +date: 2022-05-07 +--- + +# Global Temporary Table + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +A temporary table does not guarantee persistency. Its life cycle is usually bound to a session or transaction, which can be used to store temporary data during processing and accelerate query. + +## Benefits + +This feature improves the expression capability and usability of temporary tables. + +## Description + +The metadata of the global temporary table is visible to all sessions. After the sessions end, the metadata still exists. The user data, indexes, and statistics of a session are isolated from those of another session. Each session can only view and modify the data submitted by itself. + +Global temporary tables have two schemas: ON COMMIT PRESERVE ROWS and ON COMMIT PRESERVE ROWS. In session-based ON COMMIT PRESERVE ROWS schema, user data is automatically cleared when a session ends. In transaction-based ON COMMIT DELETE ROWS schema, user data is automatically cleared when the commit or rollback operation is performed. If the **ON COMMIT** option is not specified during table creation, the session level is used by default. Different from local temporary tables, you can specify a schema that does not start with **pg_temp_** when creating a global temporary table. + +## Enhancements + +The processing of the global temporary table is added based on the local temporary table. + +## Constraints + +- Parallel scanning is not supported. +- Temp tablespace is not supported. +- Partitions are not supported. +- GIST indexes are not supported. +- The user-defined statistics **pg_statistic_ext** is not supported. +- ON COMMIT DROP is not supported. +- Hash bucket cluster storage is not supported. +- Row store is not supported. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/12-pseudocolumn-rownum.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/12-pseudocolumn-rownum.md new file mode 100644 index 0000000000000000000000000000000000000000..5e84a561ad059bdbe9a882d213e62168cd3f1153 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/12-pseudocolumn-rownum.md @@ -0,0 +1,43 @@ +--- +title: Pseudocolumn ROWNUM +summary: Pseudocolumn ROWNUM +author: Guo Huan +date: 2022-05-07 +--- + +# Pseudocolumn ROWNUM + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +ROWNUM is a sequence number generated for each record in the query result. The sequence number starts from 1 and is unique. + +## Benefits + +- This feature is compatible with Oracle features, facilitating database migration. +- Similar to the LIMIT feature, this feature can filter out the first *n* records in the result set. + +## Description + +ROWNUM (pseudocolumn), which is used to label the records that meet conditions in the SQL query in sequence. In the query result, the value of **ROWNUM** in the first line is **1**, the value of **ROWNUM** in the second line is **2**, and so on. The value of **ROWNUM** in the _n_th line is *n*. This feature is used to filter the first *n* rows of data in the query result set, which is similar to the LIMIT function in MogDB. + +## Enhancements + +During internal execution, the optimizer rewrites ROWNUM into LIMIT to accelerate the execution speed. + +## Constraints + +- Do not use the pseudocolumn ROWNUM as an alias to avoid ambiguity in SQL statements. +- Do not use ROWNUM when creating an index. Bad example: **create index index_name on table(rownum);** +- Do not use ROWNUM as the default value when creating a table. Bad example: **create table table_name(id int default rownum);** +- Do not use ROWNUM as an alias in the WHERE clause. Bad example: **select rownum rn from table where rn < 5;** +- Do not use ROWNUM when inserting data. Bad example: **insert into table values (rownum,'blue')** +- Do not use ROWNUM in a table-less query. Bad example: **select \* from (values(rownum,1)), x(a,b);** +- If the HAVING clause contains ROWNUM (and is not in the aggregate function), the GROUP BY clause must contain ROWNUM (and is not in the aggregate function). + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/13-stored-procedure-debugging.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/13-stored-procedure-debugging.md new file mode 100644 index 0000000000000000000000000000000000000000..05d299fa0e63e8f128cbb1ddfb97545f4164e51e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/13-stored-procedure-debugging.md @@ -0,0 +1,38 @@ +--- +title: Stored Procedure Debugging +summary: Stored Procedure Debugging +author: Guo Huan +date: 2022-05-07 +--- + +# Stored Procedure Debugging + +## Availability + +This feature was introduced in MogDB 1.1.0. After the third-party library code directory structure was adjusted, this feature was temporarily deleted and is now available since MogDB 1.1.0. + +## Introduction + +This feature provides a group of APIs for debugging stored procedures, such as breakpoint debugging and variable printing. + +## Benefits + +This feature improves user experience in developing stored procedures based on MogDB. + +## Description + +Stored procedures are important database objects. They encapsulate SQL statement sets used for certain functions so that the statements can be easily invoked. A stored procedure usually contains many SQL statements and procedural execution structures, depending on the service scale. However, writing a large stored procedure is usually accompanied by logic bugs. It is difficult or even impossible to find the bugs by only executing the stored procedure. Therefore, a debugging tool is required. + +The stored procedure debugging tool provides a group of debugging APIs to enable the stored procedure to be executed step by step. During the execution, you can set breakpoints and print variables so that SQL developers can detect and correct errors in time and develop functions more efficiently and with high quality. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/14-jdbc-client-load-balancing-and-readwrite-isolation.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/14-jdbc-client-load-balancing-and-readwrite-isolation.md new file mode 100644 index 0000000000000000000000000000000000000000..38de3c0a431308ffc035e4b64acc1f468d90c0a1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/14-jdbc-client-load-balancing-and-readwrite-isolation.md @@ -0,0 +1,36 @@ +--- +title: JDBC Client Load Balancing and Read/Write Isolation +summary: JDBC Client Load Balancing and Read/Write Isolation +author: Guo Huan +date: 2022-05-07 +--- + +# JDBC Client Load Balancing and Read/Write Isolation + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +The JDBC client provides load balancing and read/write isolation capabilities. + +## Benefits + +Load balancing and read/write isolation can be configured on the JDBC client. + +## Description + +The IP addresses and port numbers of multiple nodes on the client are configured to adapt to HA switchover between multiple AZs and remote DR switchover. The connection-level read/write isolation configuration is supported. Preferentially connecting to read-only nodes is supported. Multiple read-only nodes are evenly distributed. + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/15-in-place-update-storage-engine.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/15-in-place-update-storage-engine.md new file mode 100644 index 0000000000000000000000000000000000000000..462c963bfafb3986d4b1ce6e2cbd30adc13a1b03 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/15-in-place-update-storage-engine.md @@ -0,0 +1,36 @@ +--- +title: In-place Update Storage Engine +summary: In-place Update Storage Engine +author: Guo Huan +date: 2022-05-07 +--- + +# In-place Update Storage Engine + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +The in-place update storage engine is a new storage mode added to MogDB. The row storage engine used by the earlier versions of MogDB is in append update mode. The append update has good performance in addition, deletion, and HOT (Heap Only Tuple) update (that is, update on the same page) in the service. However, in a non-HOT UPDATE scenario across data pages, garbage collection is not efficient. The Ustore storage engine can solve this problem. + +## Benefits + +The in-place update storage engine can effectively reduce storage space occupation after tuples are updated for multiple times. + +## Description + +The in-place update storage engine solves the problems of space expansion and large tuples of the Append update storage engine. The design of efficient rollback segments is the basis of the in-place update storage engine. + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/16-publication-subscription.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/16-publication-subscription.md new file mode 100644 index 0000000000000000000000000000000000000000..edf4b4adb1b281d7f417005ec1d9ace4b28e5853 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/16-publication-subscription.md @@ -0,0 +1,52 @@ +--- +title: Publication-Subscription +summary: Publication-Subscription +author: Guo Huan +date: 2022-05-10 +--- + +# Publication-Subscription + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Publication-subscription is implemented based on logical replication, with one or more subscribers subscribing to one or more publications on a publisher node. The subscriber pulls data from the publications they subscribe to. Data across database clusters can be synchronized in real time. + +## Benefits + +The typical application scenarios of publication-subscription are as follows: + +- Sending incremental changes in a database or a subset of a database to subscribers as they occur +- Firing triggers when changes reach subscribers +- Consolidating multiple databases into a single one (for example, for analysis purposes) + +## Description + +Changes on the publisher are sent to the subscriber as they occur in real time. The subscriber applies the published data in the same order as the publisher, so that transactional consistency is guaranteed for publications within a single subscription. This method of data replication is sometimes called transactional replication. + +The subscriber database behaves in the same way as any other MogDB instance and can be used as a publisher for other databases by defining its own publications. When the subscriber is treated as read-only by an application, there will be no conflicts in a single subscription. On the other side, conflicts may occur if other write operations are performed by the application or by other subscribers in the same set of tables. + +## Enhancements + +None. + +## Constraints + +Publication-subscription is implemented based on logical replication and inherits all restrictions of logical replication. In addition, publication-subscription has the following additional restrictions or missing functions. + +- Database schemas and DDL commands are not replicated. Initial schemas can be manually copied by using **pg_dump –schema-only**. Subsequent schema changes need to be manually synchronized. +- Sequence data is not replicated. The data in serial or identifier columns backed by the sequence in the background will be replicated as part of the table, but the sequence itself will still display the start value on the subscriber. If the subscriber is used as a read-only database, this is usually not a problem. However, if some kind of switchover or failover to the subscriber database is intended, the sequence needs to be updated to the latest value, either by copying the current data from the publisher (perhaps using **pg_dump**) or by determining a sufficiently large value from the tables themselves. +- Only tables, including partitioned tables, can be replicated. Attempts to replicate other types of relations, such as views, materialized views, or foreign tables, will result in errors. +- Multiple subscriptions in the same database cannot subscribe to the same publication (that is, the same published table). Otherwise, duplicate data or primary key conflicts may occur. +- If a published table contains data types that do not support B-tree or hash indexes (such as the geography types), the table must have a primary key so that UPDATE and DELETE operations can be successfully replicated to the subscription side. Otherwise, the replication will fail, and the message “FATAL: could not identify an equality operator for type xx” will be displayed on the subscription side. + +## Dependencies + +Publication-subscription depends on the logical replication function. + +## Related Pages + +[CREATE PUBLICATION](../../reference-guide/sql-syntax/CREATE-PUBLICATION.md)、[CREATE SUBSCRIPTION](../../reference-guide/sql-syntax/CREATE-SUBSCRIPTION.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/17-foreign-key-lock-enhancement.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/17-foreign-key-lock-enhancement.md new file mode 100644 index 0000000000000000000000000000000000000000..3f0c1d611de985edbc69f02e34e0bef8006ab90f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/17-foreign-key-lock-enhancement.md @@ -0,0 +1,45 @@ +--- +title: Foreign Key Lock Enhancement +summary: Foreign Key Lock Enhancement +author: Guo Huan +date: 2022-05-10 +--- + +# Foreign Key Lock Enhancement + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Two types of row locks are added, which are extended from share and update locks to key share, share, no key update, and update locks. A non-primary key update obtains a no key update lock, and a row lock obtained by a foreign key trigger is a key share lock. The two types of locks do not conflict with each other, thereby improving concurrency of foreign key locks. + +## Benefits + +Most table update operations are non-primary key updates. This feature effectively reduces the blocking of concurrent updates in scenarios with foreign key constraints and improves efficiency. + +## Description + +When the non-primary key column of a tuple in the parent table is updated, the no key update lock is obtained. When the corresponding tuple in the child table is updated or inserted, the foreign key trigger is triggered to obtain the key share lock of the tuple in the parent table. They do not block each other. + +Because row locks that do not conflict with each other are added, multiple transactions are not composed of only share locks. Instead, there are multiple combinations of different row locks according to the following conflict table. + +| Lock Mode | key share | share | no key update | update | +| ------------- | --------- | ----- | ------------- | ------ | +| key share | | | | X | +| share | | | X | X | +| no key update | | X | X | X | +| update | X | X | X | X | + +## Enhancements + +None. + +## Constraints + +- The new row lock does not support the Ustore table. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/18-data-compression-in-oltp-scenarios.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/18-data-compression-in-oltp-scenarios.md new file mode 100644 index 0000000000000000000000000000000000000000..c735bccb15b52b4d621a17a9bc2adfece535a7f5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/18-data-compression-in-oltp-scenarios.md @@ -0,0 +1,29 @@ +--- +title: Data Compression in OLTP Scenarios +summary: Data Compression in OLTP Scenarios +author: Guo Huan +date: 2022-05-10 +--- + +# Data Compression in OLTP Scenarios + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +The feature supports row-store data compression in OLTP scenarios, provides a general compression algorithm, and implements transparent compression of data pages and maintenance of page storage locations to achieve high compression and high performance. Disk persistence is implemented using two types of files: compressed address file (with the file name extension .pca) and compressed data file (with the file name extension .pcd). + +## Benefits + +Typically, it is applicable where the database disk space needs to be reduced. + +## Description + +Data compression in OLTP scenarios can reduce the disk storage space of row tables and index data and improve performance in I/O-intensive database systems. + +## Constraints + +- Only row-store tables and BTree indexes are supported. The feature cannot be used on Ustore and segment-page storage engines. +- Compressed table index files are accessed using mmap. **max_map_count** must be set properly based on the number of compressed table files. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/19-transaction-async-submit.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/19-transaction-async-submit.md new file mode 100644 index 0000000000000000000000000000000000000000..6fedce89945240b4d3f3d684debc168c3f7c9412 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/19-transaction-async-submit.md @@ -0,0 +1,30 @@ +--- +title: Transaction Asynchronous Submit +summary: Transaction Asynchronous Submit +author: Guo Huan +date: 2022-06-13 +--- + +# Transaction Asynchronous Submit + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +In the thread pool mode of MogDB, a committer thread is added to each thread group, and the session is hung while waiting for the logs to be flushed during the transaction commit process, and the committer thread will continue to commit the transaction after the logs are flushed, while the original worker thread can go to handle other pending sessions after the session is hung. + +## Benefits + +In TP scenario, there is a high demand on the performance of transaction processing under high concurrency. However, the transaction commit process in MogDB needs to wait for the log to drop synchronously, and the worker threads are idle during this period and cannot be used to process other transactions. Although there is an implementation of "synchronous_commit=off" in MogDB, it does not guarantee the integrity of data after the database crashes in the middle. + +This feature can realize the real sense of transaction asynchronous commit, which can make full use of CPU and improve the transaction processing ability in high concurrency scenario, especially in the small query adding, deleting and changing operations. + +## Constraints + +- This feature is valid only in threadpool mode, non-threadpool mode does not support transaction asynchronous commit. That is, set "enable_thread_pool = on" and "synchronous_commit" not to "off". + +## Related Pages + +[GS_ASYNC_SUBMIT_SESSIONS_STATUS](../../reference-guide/system-catalogs-and-system-views/system-views/GS_ASYNC_SUBMIT_SESSIONS_STATUS.md), [async_submit](../../reference-guide/guc-parameters/20-MogDB-transaction.md#async_submit), [LOCAL_THREADPOOL_STATUS](../../reference-guide/schema/DBE_PERF/session-thread/LOCAL_THREADPOOL_STATUS.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/2-sql-hints.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/2-sql-hints.md new file mode 100644 index 0000000000000000000000000000000000000000..14acb5b576e0a3e61348b9bdfb051debc178f81b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/2-sql-hints.md @@ -0,0 +1,36 @@ +--- +title: SQL Hints +summary: SQL Hints +author: Guo Huan +date: 2022-05-07 +--- + +# SQL Hints + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +SQL hints can be used to override execution plans. + +## Benefits + +Improves SQL query performance. + +## Description + +In plan hints, you can specify a join order; join, stream, and scan operations, the number of rows in a result, and redistribution skew information to tune an execution plan, improving query performance. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/20-copy-import-optimization.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/20-copy-import-optimization.md new file mode 100644 index 0000000000000000000000000000000000000000..9fc45df1139ec541d68c88a62f798691e45f5c10 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/20-copy-import-optimization.md @@ -0,0 +1,93 @@ +--- +title: COPY Import Optimization +summary: COPY Import Optimization +author: Guo Huan +date: 2022-06-16 +--- + +# COPY Import Optimization + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +COPY is the most used way to import user table data. This feature improves the performance of COPY in the parsing stage by using the SIMD feature of modern CPUs to improve the performance of COPY and the import speed. + +When COPY imports data from a file, it is theoretically a string comparison operation during the parsing phase to find the separator and to determine whether data parsed by CSV/TEXT is legal or not. The SIMD feature supports comparison of multiple strings at one time, thereby reducing the number of branch judgments and then improving performance. + +## Benefits + +The row or column separator is optimized in lookup during COPY parsing using SIMD command. The end users of this feature are general customers, such as database DBAs, software developers, etc. The performance of COPY is increased by 10% to 30%. + +| Number of Data Records | **100000000** | +| ----------------------------------- | ------------- | +| **Total Data Size** | **24 GB** | +| **Average Performance Improvement** | **12.29%** | + +The test results are as follows. + +| Test Sequence | Time Spent with SIMD feature Unused (Second) | Time Spent with SIMD Feature Used (Second) | +| ------------------ | -------------------------------------------- | ------------------------------------------ | +| 1 | 761.01 | 671.05 | +| 2 | 747.06 | 662.60 | +| 3 | 770.22 | 663.03 | +| 4 | 747.940 | 674.03 | +| 5 | 787.22 | 674.13 | +| Average time spent | 762.69 | 668.97 | + +## Constraints + +- Only machines with the x86 architecture, only text and csv files are supported. The following are not supported: escape characters, escape and quote, null value substitution and custom column separators. + +- Because this string comparison instruction value is only supported since SSE4.2, only x86 that supports SSE4.2 can use this optimization. + +The following commands can be used to determine if the machine supports the SSE4.2 command set (logging in as either root or omm user). + +```shell +[omm3@hostname ~]$ grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported" +SSE 4.2 supported + +[xxxx@hostname ~]$ grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported" +SSE 4.2 not supported +``` + +The enable_sse42 feature can be enabled or disabled using the following command. + +Log in to the database. + +```shell +[omm3@hostname ~]$ gsql -d postgres -p18000 +gsql ((MogDB 3.0.0 build 945141ad) compiled at 2022-05-28 16:14:02 commit 0 last mr ) +Non-SSL connection (SSL connection is recommended when requiring high-security) +Type "help" for help. +``` + +Enable the enable_sse42 feature. + +```sql +MogDB=# set enable_sse42 to on; +SET +MogDB=# show enable_sse42; +enable_sse42 +-------------- + on +(1 row) +``` + +Disable the enable_sse42 feature. + +```sql +MogDB=# set enable_sse42 to off; +SET +MogDB=# show enable_sse42; +enable_sse42 +-------------- +off +(1 row) +``` + +## Related Pages + +[COPY](../../reference-guide/sql-syntax/COPY.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/21-dynamic-partition-pruning.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/21-dynamic-partition-pruning.md new file mode 100644 index 0000000000000000000000000000000000000000..4ba2d1d8eac111c03c29eea88a96d2bdb1b5e74c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/21-dynamic-partition-pruning.md @@ -0,0 +1,32 @@ +--- +title: Dynamic Partition Pruning +summary: Dynamic Partition Pruning +author: Guo Huan +date: 2022-06-17 +--- + +# Dynamic Partition Pruning + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +This feature mainly implements the following functions. + +1. Support NEST LOOP for partition pruning. + +2. Support bind variables for partition pruning. + +3. Support subqueries for partition pruning. + +4. Support using EXPLAIN ANALYZE to view the dynamic partition pruning results. + +## Benefits + +This feature mainly optimizes the partition pruning feature, introduces dynamic partition pruning, and supports viewing partition pruning results by EXPLAIN ANALYZE. This feature improves the query performance of partitioned tables by pruning off the unneeded partitions and then scanning the partitioned tables during SQL execution. + +## Related Pages + +[Static Partition Pruning](../../developer-guide/partition-management/partition-pruning/static-partition-pruning.md), [Dynamic Partition Pruning](../../developer-guide/partition-management/partition-pruning/dynamic-partition-pruning.md) diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/22-sql-running-status-observation.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/22-sql-running-status-observation.md new file mode 100644 index 0000000000000000000000000000000000000000..0a79f45630ab589245a34a72fd8ba0731c0bcceb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/22-sql-running-status-observation.md @@ -0,0 +1,105 @@ +--- +title: SQL Running Status Observation +summary: SQL Running Status Observation +author: Guo Huan +date: 2022-06-17 +--- + +# SQL Running Status Observation + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +The sampling thread is sampled once in 1s by default, which can be controlled by the GUC parameter asp_sample_interval, and up to 100000 lines of data are sampled in memory, which is controlled by asp_sample_num, and will be flushed to a new disk when the upper limit is reached. + +## Benefits + +When a performance bottleneck is found in a SQL statement and the execution of each operator of that SQL cannot be queried in the sampled view, you can locate the performance problem by plan_node_id. + +Operator: the specific action of each step in the execution of the SQL statement (e.g. SELECT, SUM, WHERE, Group By, Having, Order By, Limit, etc.) + +## Description + +A new column plan_node_id is added to [dbe_perf.local_active_session](../../reference-guide/schema/DBE_PERF/session-thread/LOCAL_ACTIVE_SESSION.md) and [GS_ASP](../../reference-guide/system-catalogs-and-system-views/system-catalogs/GS_ASP.md) to record the execution of each operator of the SQL statement. + +The existing monitoring level is defined by the GUC parameter [resource_track_level](../../reference-guide/guc-parameters/13-load-management.md#resource_track_level), which has three values according to the level, namely + +- **none**: Resources are not monitored. +- **query**: Resources used at the query level are monitored. +- **operator**: Resources used at query and operator levels are monitored. + +So each operator of the SQL statement is sampled only if the resource_track_level is set to operator. + +MogDB will start a background worker sampling thread after being started. In order to avoid wasting resources, this sampling thread will not sample all the time, but sample MogDB every one sampling period, collect the snapshot of MogDB running at that time and save it in memory. dbe_perf.local_active_session can query the real-time sampling information. The sampling period is defined by the GUC parameter [asp_sample_interval](../../reference-guide/guc-parameters/27-system-performance-snapshot.md#asp_sample_interval), and the default sample period is 1s. MogDB will flush the sampled data in memory to the GS_ASP table for historical query when 100000 rows (controlled by guc parameter [asp_sample_num](../../reference-guide/guc-parameters/27-system-performance-snapshot.md#asp_sample_num)) are sampled in memory every time. Only when the statement execution time is greater than the sampling time, the running information will be collected by the sampling thread. + +## Scenarios + +1. Create the table test in session1 and perform the insert operation. + + ```sql + MogDB=# create table test(c1 int); + CREATE TABLE + MogDB=# insert into test select generate_series(1, 1000000000); + ``` + +2. In session2, look up the query_id of the SQL from the active session view + + ```sql + MogDB=# select query,query_id from pg_stat_activity where query like 'insert into test select%'; + query | query_id + -----------------------------------------------------------+----------------- + insert into test select generate_series(1, 100000000000); | 562949953421368 + (1 row) + ``` + +3. In session2, according to the query_id from the active job management view to query the statement with plan_node_id execution plan (the statement execution cost needs to be greater than the GUC value **resource_track_cost** to be recorded in the view, the default value of the GUC parameter is 100000, session level can be updated, so in order to facilitate testing, you can change the value to 10 in the test) + + Set resource_track_cost=10; + + ```sql + MogDB=# select query_plan from dbe_perf.statement_complex_runtime where queryid = 562949953421368; + query_plan + ---------------------------------------------------------------------------- + Coordinator Name: datanode1 + + 1 | Insert on test (cost=0.00..17.51 rows=1000 width=8) + + 2 | -> Subquery Scan on "*SELECT*" (cost=0.00..17.51 rows=1000 width=8) + + 3 | -> Result (cost=0.00..5.01 rows=1000 width=0) + + + + (1 row) + ``` + +4. In session2, the sampling of the statement is queried from the sampling view dbe_perf.local_active_session based on the query_id, and the performance analysis is done in conjunction with the execution plan of the above query. + + ```sql + MogDB=# select plan_node_id, count(plan_node_id) from dbe_perf.local_active_session where query_id = 562949953421368 group by plan_node_id; + plan_node_id | count + --------------+------- + 3 | 12 + 1 | 366 + 2 | 2 + (3 rows) + ``` + +5. In session2, when the memory data reaches the upper limit (controlled by the GUC parameter **asp_sample_num**), the existing memory sampling data will be flushed to the gs_asp table, and the data sampled using the statement can be queried from the gs_asp table after the flush. + + ```sql + MogDB=# select plan_node_id, count(plan_node_id) from gs_asp where query_id = 562949953421368 group by plan_node_id; + plan_node_id | count + --------------+------- + 3 | 19 + 1 | 582 + 2 | 3 + + (3 rows) + ``` + +## Conclusion + +When it is found that there is a performance bottleneck in `insert into test select generate_series(1, 1000000000)`, the above steps locate that the insert operation is sampled with the highest value (plan_node_id =1 , count=366) during the whole SQL statement execution, which can be optimized. + +## Related Pages + +[GS_ASP](../../reference-guide/system-catalogs-and-system-views/system-catalogs/GS_ASP.md), [LOCAL_ACTIVE_SESSION](../../reference-guide/schema/DBE_PERF/session-thread/LOCAL_ACTIVE_SESSION.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/23-index-creation-parallel-control.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/23-index-creation-parallel-control.md new file mode 100644 index 0000000000000000000000000000000000000000..ed1426596db4e95d6f0fc4e3ee3e8d829d1aab76 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/23-index-creation-parallel-control.md @@ -0,0 +1,28 @@ +--- +title: Index Creation Parallel Control +summary: Index Creation Parallel Control +author: Guo Huan +date: 2022-06-17 +--- + +# Index Creation Parallel Control + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +This feature supports specifying the degree of parallelism directly in the index creation statement, making more efficient use of resources while increasing usage flexibility. + +## Benefits + +This feature adds the option to specify the degree of parallelism in the index creation syntax, so that DBAs and delivery testers can control the degree of concurrency of index creation from the syntax level to achieve optimal execution. + +## Description + +Add parallel syntax when creating indexes to control the parallelism of index creation. + +## Related Pages + +[CREATE INDEX](../../reference-guide/sql-syntax/CREATE-INDEX.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/24-brin-index.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/24-brin-index.md new file mode 100644 index 0000000000000000000000000000000000000000..f64e69fbc89b2d7f0a3076884f4a9a58cf15a3fe --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/24-brin-index.md @@ -0,0 +1,257 @@ +--- +title: BRIN Index +summary: BRIN Index +author: Guo Huan +date: 2022-06-20 +--- + +# BRIN Index + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +MogDB 3.0.0 adds BRIN index feature. + +A BRIN index is a block range index. Unlike other indexes, the BRIN index allows you to quickly exclude rows that do not satisfy the query criteria. + +## Benefits + +- Sequential scans will be fast, then statistical SQL performance against large tables will be dramatically improved. +- Creating indexes is very fast. +- Indexes take up very little space. + +## Description + +Block Range INdex is short for BRIN index. Unlike other indexes, the idea of a BRIN index is to quickly exclude rows that do not meet the query criteria, rather than quickly find matching rows. + +The way a BRIN index works: The block of a table is divided into some intervals, and the index stores summary information (usually min and max information, and some other information for spatial data) for each interval. If the value of the column to be queried does not fall into the summary information of this interval, then the interval can be skipped. Otherwise, all rows of this interval need to be scanned. + +The BRIN index is suitable for columns where the data is correlated with the physical location. The optimizer uses this value to make a choice between an index scan and a bitmap scan. We can also use it to estimate how well the BRIN index fits. The closer the column position correlation is to 1, the more correlated the column data is and the more suitable for building a BRIN index. Also BRIN indexes are mainly designed for large tables. Compared with indexes like btree, BRIN indexes have less data volume. + +## Scenarios + +### Create a BRIN Index + +The syntax is the same as that for creating B-trees, hash, GiST, SP-GiST, and GINs. + +The parameters pages_per_range and autosummarize can be specified when creating a BRIN index. + +pages_per_range: specifies how many pages are contained in each range in the BRIN index. The range is 1 ~ 131072. if it is not specified, the default value is 128. + +autosummarize: specifies whether to automatically create indexes for unindexed data pages in the table. The default value is **off**. + +```sql +CREATE INDEX brinidx ON tbbrin USING brin(i1,i2,i3) WITH (pages_per_range=64, autosummarize=off); +--Online Create +CREATE INDEX CONCURRENTLY brinidx ON tbbrin USING brin(i1,i2,i3) WITH (pages_per_range=64); +``` + +### Recreate a BRIN Index + +The syntax is the same as that for recreating B-tree, hash, GiST, SP-GiST, and GINs. + +```sql +REINDEX INDEX brinidx ; +--Online reindex +REINDEX INDEX CONCURRENTLY brinidx ; +``` + +### Alter a BRIN Index + +BRIN supports modifying the pages_per_range and autosummarize parameters. After the alter command is executed, only the metadata is updated, and the parameters set by the reindex command need to be executed to take effect. + +Example: + +```sql +alter index idx set(pages_per_range=64); +reindex index idx; +``` + +### View the Execution Plan of the BRIN Index + +Example: + +```sql +MogDB=# explain select * from example where id = 100; + QUERY PLAN +--------------------------------------------------------- + Bitmap Heap Scan on example (cost=15.88..486.21 rows=500 width=4) + Recheck Cond:(id = 100) + -> Bitmap Index Scan on idx (cost=0.00..15.75 rows=500 width=0) + Index Cond:(id =100) +(4 rows) +``` + +### Manually Update a BRIN Index + +It may happen that some data pages do not appear in the index during the use of the BRIN index. You can update the BRIN index manually by the following two ways. + +- Perform a vacuum operation on a table. + +- Execute the brin_summarize_new_values(oid) function. The input is the ID of the BRIN index. If the return value is 0, then the index is not updated. If it returns 1, then the index has been updated. + +Example: + +```sql +SELECT brin_summarize_new_values((select oid from pg_class where relname='brinidx')::oid); +``` + +The index can be checked for updates by the following actions. + +- View the total block size of the table from pg_class. + +- View the page numbers of the indexes that have been created based on brin_revmap_data. + +- Calculate if the difference between the previous 2 items is greater than pages_per_range. If it is, the index needs to be updated. + +### autosummarize for BRIN index + +autosummarize is a switch for whether to automatically create indexes for data pages in the table that do not appear in the index. + +You can test if autosummarize is invalid by following these steps. + +1. Update the table data. + +2. Query the table for updating the last_autovacuum field via pg_stat_user_table, e.g. + + `````sql + MogDB=# select relname,last_vacuum,last_autovacuum from pg_stat_user_tables where relname = 'person'; + relname | last_vacuum | last_autovacuum + ---------+-------------------------------+----------------- + person | 2022-06-20 19:21:58.201214+08 | + (1 row) | + ````` + + The following two commands can be executed to speed up the autovacuum frequency of the table. + + ```sql + ALTER TABLE example SET (autovacuum_vacuum_scale_factor = 0.0); + ALTER TABLE example SET (autovacuum_vacuum_threshold = 100); + ``` + +3. After the pg_stat_user_table table is updated, you can observe the automatic update of index data. + + ```sql + MogDB=# select relname,last_vacuum,last_autovacuum from pg_stat_user_tables where relname = 'person'; + relname | last_vacuum | last_autovacuum + ---------+-------------------------------+----------------- + person | 2022-06-20 19:23:58.201214+08 | 2022-06-20 19:24:59.201214+08 + (1 row) + ``` + +### View BRIN Index Via pageinspect + +For the BRIN index of the table, you can query the page data by pageinspect. + +The pageinspect module provides functions that allow viewing the contents of database data pages or index pages, which can be useful for debugging or locating problems. pageinspect tool is installed in the same way, by executing the following command during installation. + +````sql +create extension pageinspect; +```` + +pageinspect only supports non-partitioned tables for now. + +The BRIN index provides three functions for querying data in the meta page, revmap_page and regular page respectively. + +- brin_metapage_info(page bytea) returns record + + Returns information about the classification of the BRIN index metapage, where the second parameter is fixed to 0. e.g. + + ```sql + MogDB=# SELECT * FROM brin_metapage_info(get_raw_page('brinidx', 0)); + magic | version | pagesperrange | lastrevmappage + ------------+---------+---------------+---------------- + 0xA8109CFA | 1 | 4 | 2 + ``` + +- brin_revmap_data(page bytea) returns setof tid + + Returns the list of tuple identifiers in the BRIN index range mapping page. e.g. + + ```sql + MogDB=# SELECT * FROM brin_revmap_data(get_raw_page('brinidx', 2)) limit 5; + pages + --------- + (6,137) + (6,138) + (6,139) + (6,140) + (6,141) + ``` + +- brin_page_items(page bytea, index oid) returns setof record + + Returns the data stored in the BRIN data page. e.g. + + ```sql + MogDB=# SELECT * FROM brin_page_items(get_raw_page('brinidx', 5), + 'brinidx') + ORDER BY blknum, attnum LIMIT 6; + itemoffset | blknum | attnum | allnulls | hasnulls | placeholder | value + ------------+--------+--------+----------+----------+-------------+-------------- + 137 | 0 | 1 | t | f | f | + 137 | 0 | 2 | f | f | f | {1 .. 88} + 138 | 4 | 1 | t | f | f | + 138 | 4 | 2 | f | f | f | {89 .. 176} + 139 | 8 | 1 | t | f | f | + 139 | 8 | 2 | f | f | f | {177 .. 264} + ``` + +## Examples + +```sql +# Create a test table +MogDB=# CREATE TABLE testtab (id int NOT NULL PRIMARY KEY,date TIMESTAMP NOT NULL, level INTEGER, msg TEXT); +NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "testtab_pkey" for table "testtab" +CREATE TABLE +# Insert test data +MogDB=# INSERT INTO testtab (id, date, level, msg) SELECT g, CURRENT_TIMESTAMP + ( g || 'minute' ) :: interval, random() * 6, md5(g::text) FROM generate_series(1,8000000) as g; +INSERT 0 8000000 +# If you look at the execution plan of a query statement without creating an index, you can see that the execution plan uses seq scan +MogDB=# explain analyze select * from public.testtab where date between '2019-08-08 14:40:47.974791' and '2019-08-08 14:50:47.974791'; + QUERY PLAN +--------------------------------------------------------------------------------------------------------------------------------------------- + Seq Scan on testtab (cost=0.00..212024.20 rows=43183 width=48) (actual time=46620.314..46620.314 rows=0 loops=1) + Filter: (("date" >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND ("date" <= '2019-08-08 14:50:47.974791'::timestamp without time zone)) + Rows Removed by Filter: 8000000 + Total runtime: 46620.580 ms +(4 rows) +# Create a brin index on the table +MogDB=# create index testtab_date_brin_idx on testtab using brin (date); +CREATE INDEX +# Looking at the index information of the brin index, you can see that the size of the brin index is about 64 kB +MogDB=# \di+ testtab_date_brin_idx  + List of relations + Schema | Name | Type | Owner | Table | Size | Storage | Description +--------+------------------------+-------+--------+---------+-------+---------+------------- + public | testtab_date_brin_idx  | index | wusong | testtab | 64 kB | | +(1 row) +# Looking at the execution plan of the same query statement, you can see that the query uses the brin index +MogDB=# explain analyze select * from public.testtab where date between '2019-08-08 14:40:47.974791' and '2019-08-08 14:50:47.974791'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Bitmap Heap Scan on testtab (cost=20.00..24.02 rows=1 width=49) (actual time=1.121..1.121 rows=0 loops=1) + Recheck Cond: (("date" >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND ("date" <= '2019-08-08 14:50:47.974791'::timestamp without time zone)) + -> Bitmap Index Scan on "testtab_date_brin_idx " (cost=0.00..20.00 rows=1 width=0) (actual time=1.119..1.119 rows=0 loops=1) + Index Cond: (("date" >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND ("date" <= '2019-08-08 14:50:47.974791'::timestamp without time zone)) + Total runtime: 1.281 ms +(5 rows) +# Create a btree index on the date column of the table +MogDB=# create index testtab_date_idx  on testtab(date); +CREATE INDEX +# Looking at the size of the btree index, we can see that the size of the btree index is about 172 MB, much larger than the 64 KB of the brin index +MogDB=# \di+ testtab_date_idx  + List of relations + Schema | Name | Type | Owner | Table | Size | Storage | Description +--------+-------------------+-------+--------+---------+--------+---------+------------- + public | testtab_date_idx  | index | wusong | testtab | 172 MB | | +(1 row) + +``` + +## Related Pages + +[CREATE INDEX](../../reference-guide/sql-syntax/CREATE-INDEX.md), [DROP INDEX](../../reference-guide/sql-syntax/DROP-INDEX.md), [ALTER INDEX](../../reference-guide/sql-syntax/ALTER-INDEX.md), [VACUUM](../../reference-guide/sql-syntax/VACUUM.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/25-bloom-index.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/25-bloom-index.md new file mode 100644 index 0000000000000000000000000000000000000000..5e60686b1b77549f6fd0abdd33be7404bfe4bc91 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/25-bloom-index.md @@ -0,0 +1,211 @@ +--- +title: BLOOM Index +summary: BLOOM Index +author: Guo Huan +date: 2022-06-21 +--- + +# BLOOM Index + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Bloom provides a Bloom filter-based approach to index access. A Bloom filter is a spatially efficient data structure that is used to test whether an element is a member of a collection. In the case of the index access method, it can quickly exclude non-matching meta-ancestors by sizing the signature decided at index creation. + +## Benefits + +Bloom indexes are suitable for scenarios where there are many columns in a table and the query can use any combination of columns. For traditional indexes, such as Btree in this scenario may need to build many indexes to cover possible query conditions, which may lead to a large index space and affect the performance of insertion and update, while Bloom indexes only need to build one to cope with it. + +## Description + +Users can query through Bloom index in the applicable scenario of Bloom index, and also reduce the size of index space occupied compared with Btree index. + +## Scenarios + +Blooms indexes are most useful when the table has many attributes and the query may test any combination of them. Traditional btree indexes will be faster than Bloom indexes, but many btree indexes are needed to support all possible queries, while for Bloom only one is needed. + +### Create Bloom Index + +#### Constraints + +- Bloom index only supports being created on row-stored table. + +- The column type for index creation can only be 4-byte-length int type or variable-length string type. 4-byte-length int type in MogDB can be int, int4 and integer, and variable-length string type can be varchar, text and clob. + +- Bloom does not support creating unique indexes. + +- null values cannot be indexed and null values are skipped. +- Only equal-value queries are supported. +- Partitioned table Global indexes are not supported. + +#### Example + +```sql +CREATE INDEX ON USING bloom(col1,col2...) with (length=80,col1=2,col2=4); +``` + +#### Parameters + +**length** + +Specifies how many bits are used to represent the signature generated in an index. The default value is 80, and the user-specified value is internally rounded up to an integer multiple of 16 (transparent to the user), with a minimum value of 1 and a maximum value of 4096. If the set value is not within the limit, the command execution reports an error and indicates the correct range of the value. + +**col1-col32** + +Specifies how many bits are used to represent each index column. The default value is 2, the minimum value is 1, and the maximum value is 4095. If the set value is not within the limit, the command execution reports an error and prompts for the correct range of values. + +If Bloom index is created for non-row-stored tables, an error is reported, and it is prompted that it cannot be created for non-row-stored tables. + +If a unique index is created for Bloom, an error is reported indicating that the unique index cannot be created. + +If a Global index for a partitioned table with index type Bloom is created, an error is reported, indicating that the partitioned table Global index can only be of type Btree. + +### Delete Bloom Index + +Example: + +```sql +MogDB=# DROP INDEX bloom_idx; +DROP INDEX +``` + +### Reindex Bloom Index + +This feature supports rebuilding a Bloom index that already exists. The user issues the command to rebuild Bloom index through the client or database driver, the index is finally rebuilt successfully, the index metadata is correct, and the index can be used normally (e.g., if the original execution plan goes to Bloom index, the execution plan can still use the rebuilt Bloom index after the index is rebuilt). + +Example: + +```sql +MogDB=# REINDEX INDEX bloom_idx; +REINDEX +``` + +### Alter Bloom Index + +This feature supports modifying the attributes of an existing Bloom index. The constraints related to the index attributes are consistent with those at the time of index creation, e.g. the modified length of a Bloom index needs to be in the range of [1,4096]. + +Key points: + +- Modified index attributes need to conform to indexing rules and constraints, and if they do not, an error needs to be reported, returning the corresponding prompt message. +- Renaming the index and verifying whether the index metadata is correct in pg_class. +- Modify the length or col attribute of Bloom index. If you want to verify whether the attribute is effective, you can simply judge by the change of the value of the relpages field in pg_class, or by SELECT pg_size_pretty(pg_relation_size('blidx ')) and other commands to see the change in index size. +- To modify tablespace, you can confirm whether it takes effect by viewing the data file on disk. + +Example: + +```sql +MogDB=# ALTER INDEX IF EXISTS bloom_idx RENAME TO newbloom_idx; +ALTER INDEX +MogDB=# SELECT oid,relname,relfilenode,relpages FROM pg_class WHERE relname = 'newbloom_idx'; + oid | relname | relfilenode | relpages +-------+--------------+-------------+---------- + 41159 | newbloom_idx | 41160 | 30 +(1 row) +MogDB=# ALTER INDEX IF EXISTS newbloom_idx SET (length=160); +ALTER INDEX +MogDB=# REINDEX INDEX newbloom_idx; +REINDEX +MogDB=# SELECT oid,relname,relfilenode,relpages FROM pg_class WHERE relname = 'newbloom_idx'; + oid | relname | relfilenode | relpages +-------+--------------+-------------+---------- + 41159 | newbloom_idx | 41162 | 49 +(1 row) +``` + +### Querying with Bloom Index + +When a Bloom index exists on the table and the query conditions match the applicable conditions of the Bloom index, the execution plan will show that the Bloom index has been used. + +Example: + +```sql +CREATE TABLE tbloom AS + SELECT + (random() * 1000000)::int as i1, + (random() * 1000000)::int as i2, + (random() * 1000000)::int as i3, + (random() * 1000000)::int as i4, + (random() * 1000000)::int as i5, + (random() * 1000000)::int as i6 + FROM + generate_series(1,10000000); + +CREATE INDEX btreeidx ON tbloom (i1, i2, i3, i4, i5, i6); +CREATE INDEX bloomidx ON tbloom USING bloom (i1, i2, i3, i4, i5, i6); + +MogDB=# EXPLAIN ANALYZE SELECT * FROM tbloom where i3 = 100 AND i5 = 1000; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------ + Bitmap Heap Scan on tbloom (cost=178436.31..179393.23 rows=250 width=24) (actual time=138.209..138.209 rows=0 loops=1) + Recheck Cond: ((i3 = 100) AND (i5 = 1000)) + Rows Removed by Index Recheck: 21936 + Heap Blocks: exact=18673 + -> Bitmap Index Scan on bloomidx (cost=0.00..178436.25 rows=250 width=0) (actual time=85.681..85.681 rows=21936 loops=1) + Index Cond: ((i3 = 100) AND (i5 = 1000)) + Total runtime: 138.412 ms +(7 rows) + +MogDB=# EXPLAIN ANALYZE SELECT * FROM tbloom where i1 = 100 AND i2 = 1000; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------- + [Bypass] + Index Only Scan using btreeidx on tbloom (cost=0.00..8.27 rows=1 width=24) (actual time=0.084..0.084 rows=0 loops=1) + Index Cond: ((i1 = 100) AND (i2 = 1000)) + Heap Fetches: 0 + Total runtime: 0.134 ms +(5 rows) +``` + +### Update Bloom Index + +Example: + +```sql +MogDB=# select i2,i3,i4 from tbloom where rownum <=5; + i2 | i3 | i4 +--------+--------+-------- + 778090 | 624067 | 948170 + 927435 | 800792 | 904419 + 325217 | 726778 | 834407 + 925272 | 221411 | 826500 + 93906 | 992575 | 997677 + +UPDATE tbloom SET i1 = 10 + WHERE i2 = 325217 AND + i3 = 726778 AND + i4 = 834407; + +MogDB=# select * from tbloom where i2 = 325217 and i3 = 726778; + i1 | i2 | i3 | i4 | i5 | i6 +----+--------+--------+--------+--------+-------- + 10 | 325217 | 726778 | 834407 | 702579 | 525581 +(1 row) + +MogDB=# explain select * from tbloom where i2 = 325217 and i3 = 726778; + QUERY PLAN +---------------------------------------------------------------------------- + Bitmap Heap Scan on tbloom (cost=178439.48..178443.50 rows=1 width=24) + Recheck Cond: ((i2 = 325217) AND (i3 = 726778)) + -> Bitmap Index Scan on bloomidx (cost=0.00..178439.48 rows=1 width=0) + Index Cond: ((i2 = 325217) AND (i3 = 726778)) +(4 rows) + +DELETE FROM tbloom WHERE i2 = 1000 AND i3 = 789678 AND i4 = 311551; +select * from tbloom where i2 = 1000 and i3 = 789678; + i1 | i2 | i3 | i4 | i5 | i6 +----+----+----+----+----+---- +(0 rows) +explain select * from tbloom where i2 = 1000 and i3 = 789678; + Bitmap Heap Scan on tbloom (cost=178440.26..178444.28 rows=1 width=24) + Recheck Cond: ((i2 = 1000) AND (i3 = 789678)) + -> Bitmap Index Scan on tbloomidx (cost=0.00..178440.26 rows=1 width=0) + Index Cond: ((i2 = 1000) AND (i3 = 789678)) +(4 rows) +``` + +## Related Pages + +[CREATE INDEX](../../reference-guide/sql-syntax/CREATE-INDEX.md), [DROP INDEX](../../reference-guide/sql-syntax/DROP-INDEX.md), [ALTER INDEX](../../reference-guide/sql-syntax/ALTER-INDEX.md) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/26-backend-compression.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/26-backend-compression.md new file mode 100644 index 0000000000000000000000000000000000000000..837d05c79653de27794f4393db14243e7eb59c63 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/26-backend-compression.md @@ -0,0 +1,28 @@ +--- +title: Backend Compression +summary: Backend Compression +author: Guo Huan +date: 2022-06-22 +--- + +# Backend Compression + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +MogDB 3.0.0 supports data compression in Astore row-store scenarios. The foreground service thread adds or modifies data in a table by executing DML statements. Data operations are only performed in the buffer cache. The foreground service does not participate in the actual compression logic. + +## Benefits + +It reduces customer storage cost and stores data on persistent media in the form of data compression. When compression is enabled, the performance of foreground services is not affected, and the impact of foreground compression length on performance is eliminated. + +## Constraints + +The compression function is realized by specifying the compression attribute when a table is created. The basic SQL syntax is listed below: + +```sql +create table (......) with (compression=yes); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/3-full-text-indexing.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/3-full-text-indexing.md new file mode 100644 index 0000000000000000000000000000000000000000..e79775d97227c75ad52d56aaca477113cc878ead --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/3-full-text-indexing.md @@ -0,0 +1,53 @@ +--- +title: Full-Text Indexing +summary: Full-Text Indexing +author: Guo Huan +date: 2022-05-07 +--- + +# Full-Text Indexing + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +MogDB full-text indexing allows documents to be preprocessed and facilitates subsequent search. + +## Benefits + +MogDB full-text indexing provides the capability to identify natural-language documents that satisfy a query and sort them by relevance. + +## Description + +The preprocessing process of creating a full-text index includes: + +- Parsing documents into tokens + + It is useful to identify various classes of tokens, for example, numbers, words, compound words, and email addresses, so that they can be processed differently. In principle, token classes depend on the specific application, but for most purposes it is adequate to use a predefined set of classes. + +- Converting tokens into lexemes + + A lexeme is a string, just like a token, but it has been normalized so that different forms of the same word are made alike. For example, normalization almost always includes folding upper-case letters to lower-case, and often involves removal of suffixes (such as **s** or **es** in English). This allows searches to find variant forms of the same word, without entering all the possible variants. Also, this step typically eliminates stop words, which are so common and usually useless for searching. (In short, tokens are raw fragments of the document text, while lexemes are words that are believed useful for indexing and searching.) MogDB uses dictionaries to perform this step and provides various standard dictionaries. + +- Storing preprocessed documents optimized for searching + + For example, each document can be represented as a sorted array of normalized lexemes. Along with the lexemes, it is often desirable to store positional information for proximity ranking. Therefore, a document that contains a more “dense” area of query words is assigned with a higher rank than the one with scattered query words. Dictionaries allow fine-grained control over how tokens are normalized. With appropriate dictionaries, you can define stop words that should not be indexed. + +## Enhancements + +None + +## Constraints + +The current limitations of MogDB's text search features are: + +- The length of each lexeme must be less than 2 KB. +- The length of a **tsvector** (lexemes + positions) must be less than 1 MB. +- Position values in **tsvector** must be greater than 0 and less than or equal to 16383. +- No more than 256 positions per lexeme. Excessive positions, if any, will be discarded. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/4-copy-interface-for-error-tolerance.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/4-copy-interface-for-error-tolerance.md new file mode 100644 index 0000000000000000000000000000000000000000..d13add1b3b0b39e457e6a3bb610046497c6c4337 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/4-copy-interface-for-error-tolerance.md @@ -0,0 +1,36 @@ +--- +title: Copy Interface for Error Tolerance +summary: Copy Interface for Error Tolerance +author: Guo Huan +date: 2022-05-07 +--- + +# Copy Interface for Error Tolerance + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Certain errors that occur during the copy process are imported to a specified error table without interrupting the process. + +## Benefits + +Refine the copy function and improve the tolerance and robustness to common errors such as invalid formats. + +## Description + +MogDB provides the encapsulated copy error tables for creating functions and allows users to specify error tolerance options when using the **Copy From** statement. In this way, errors related to parsing, data format, and character set during the execution of the **Copy From** statement are recorded in the error table instead of being reported and interrupted. Even if a small amount of data in the target file of **Copy From** is incorrect, the data can be imported to the database. You can locate and rectify the fault in the error table later. + +## Enhancements + +None + +## Constraints + +For details, see “Importing Data > Running the COPY FROM STDIN Statement to Import Data > [Handling Import Errors](../../administrator-guide/importing-and-exporting-data/importing-data/3-running-the-COPY-FROM-STDIN-statement-to-import-data.md#handling-import-errors)” in the *Administrator Guide*. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/5-partitioning.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/5-partitioning.md new file mode 100644 index 0000000000000000000000000000000000000000..f12f6bcca059cec254126cf13a1baabbb5c275d5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/5-partitioning.md @@ -0,0 +1,60 @@ +--- +title: Partitioning +summary: Partitioning +author: Guo Huan +date: 2022-05-07 +--- + +# Partitioning + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Data is partitioned horizontally on a node using a specified policy. This operation splits a table into multiple partitions that are not overlapped. + +## Benefits + +In common scenarios, a partitioned table has the following advantages over a common table: + +- High query performance: You can specify partitions when querying partitioned tables, improving query efficiency. +- High availability: If a certain partition in a partitioned table is faulty, data in the other partitions is still available. +- Balanced I/O: Partitions can be mapped to different disks to balance I/O and improve the overall system performance. + +## Description + +Currently, the MogDB database supports range partitioned tables, list partitioned tables, hash partitioned tables, interval partitioned tables, and level-2 partitioned tables. + +- In a range partitioned table, data within a certain range is mapped to each partition. The range is determined by the partition key specified when the partitioned table is created. This partitioning mode is most commonly used. + + With the range partitioning function, the database divides a record, which is to be inserted into a table, into multiple ranges using one or multiple columns and creates a partition for each range to store data. Partition ranges do no overlap. + +- In a list partitioned table, data is mapped to each partition based on the key values contained in each partition. The key values contained in a partition are specified when the partition is created. + + The list partitioning function divides the key values in the records to be inserted into a table into multiple lists (the lists do not overlap in different partitions) based on a column of the table, and then creates a partition for each list to store the corresponding data. + +- In a hash partitioned table, data is mapped to each partition using the hash algorithm, and each partition stores records with the same hash value. + + The hash partitioning function uses the internal hash algorithm to divide records to be inserted into a table into partitions based on a column of the table. + +- Interval partitioning is a special type of range partitioning. Compared with range partitioning, interval partitioning defines the interval value. When no matching partition can be found for an inserted record, a partition can be automatically created based on the interval value. + +- A level-2 partitioned table is based on level-1 partition. Its partitioning scheme is a combination of two level-1 partitioning schemes. Currently, the level-2 partitioned table supports nine partitioning policies combining range partitioning, list partitioning, and hash partitioning. + +If you specify the **PARTITION** parameter when running the **CREATE TABLE** statement, data in the table will be partitioned. Users can modify partition keys as needed during table creation to make the query result stored in the same or least partitions (called partition pruning), obtaining consecutive I/O to improve the query performance. + +In actual services, time is often used to filter query objects. Therefore, you can select the time column as the partition key. The key value range can be adjusted based on the total data volume and the data volume queried at a time. + +## Enhancements + +Range partitioned tables can be combined. + +## Constraints + +None. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/6-support-for-advanced-analysis-functions.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/6-support-for-advanced-analysis-functions.md new file mode 100644 index 0000000000000000000000000000000000000000..ab3316e25a632f1db379b2a1a6dd7454c71f9012 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/6-support-for-advanced-analysis-functions.md @@ -0,0 +1,57 @@ +--- +title: Support for Advanced Analysis Functions +summary: Support for Advanced Analysis Functions +author: Guo Huan +date: 2022-05-07 +--- + +# Support for Advanced Analysis Functions + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +None + +## Benefits + +Window functions are provided for advanced data analysis and processing. The window function groups the data in a table in advance. Each row belongs to a specific group. Then, a series of association analysis calculations are performed on the group. In this way, some attributes of each tuple in the set and association information with other tuples can be mined. + +## Description + +The following uses an example to describe the window analysis function: Compare the salary of each person in a department with the average salary of the department. + +```sql +SELECT depname, empno, salary, avg(salary) OVER (PARTITION BY depname) FROM empsalary; +depname | empno | salary | avg +-----------+-------+--------+----------------------- +develop | 11 | 5200 | 5020.0000000000000000 +develop | 7 | 4200 | 5020.0000000000000000 +develop | 9 | 4500 | 5020.0000000000000000 +develop | 8 | 6000 | 5020.0000000000000000 +develop | 10 | 5200 | 5020.0000000000000000 +personnel | 5 | 3500 | 3700.0000000000000000 +personnel | 2 | 3900 | 3700.0000000000000000 +sales | 3 | 4800 | 4866.6666666666666667 +sales | 1 | 5000 | 4866.6666666666666667 +sales | 4 | 4800 | 4866.6666666666666667 +(10 rows) +``` + +The analysis function **avg(salary) OVER (PARTITION BY depname)** easily calculates each employee's salary and the average salary of the department. + +Currently, the system supports the following analysis functions: **row_number()**, **rank()**, **dense_rank()**, **percent_rank()**, **cume_dist()**, **ntile()**, **lag()**, **lead()**, **first_value()**, **last_value()**, and **nth_value()**. For details about functions and statements, see “Functions and Operators > [Window Functions](../../reference-guide/functions-and-operators/18-window-functions.md)” in the *Reference Guide*. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/7-materialized-view.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/7-materialized-view.md new file mode 100644 index 0000000000000000000000000000000000000000..84724672818d3bd7e2fd8fcc7d4495eae8e977e0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/7-materialized-view.md @@ -0,0 +1,36 @@ +--- +title: Materialized View +summary: Materialized View +author: Guo Huan +date: 2022-05-07 +--- + +# Materialized View + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +A materialized view is a special physical table, which is relative to a common view. A common view is a virtual table and has many application limitations. Any query on a view is actually converted into a query on an SQL statement, and performance is not actually improved. The materialized view actually stores the results of the statements executed by the SQL statement, and is used to cache the results. + +## Benefits + +The materialized view function is used to improve query efficiency. + +## Description + +Full materialized views and incremental materialized views are supported. Full materialized views can only be updated in full mode. Incremental materialized views can be updated asynchronously. You can run statements to update new data to materialized views. + +## Enhancements + +None + +## Constraints + +Only simple filter queries and UNION ALL statements are supported for base tables. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/8-hyperloglog.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/8-hyperloglog.md new file mode 100644 index 0000000000000000000000000000000000000000..c91321866d2cffb34d3e7829f4398592e4420b78 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/8-hyperloglog.md @@ -0,0 +1,38 @@ +--- +title: HyperLogLog +summary: HyperLogLog +author: Guo Huan +date: 2022-05-07 +--- + +# HyperLogLog + +## Availability + +This feature is available as of MogDB 1.1.0. + +## Introduction + +HyperLoglog (HLL) is used to count the number of distinct values. + +## Benefits + +Improves AP/TP query performance. + +## Description + +HLL is an approximation algorithm for efficiently counting the number of distinct values in a dataset. It features faster computing and lower space usage. You only need to store HLL data structures instead of datasets. When new data is added to a dataset, make hash calculation on the data and insert the result to an HLL. Then, you can obtain the final result based on the HLL. + +HLL has advantages over others in the computing speed and storage space requirement. In terms of time complexity, the Sort algorithm needs to sort at least O(n log n) time. Although the Hash algorithm can obtain the result by scanning the entire table O(n) time, the storage space is as follows: Both the Sort and Hash algorithms need to store the original data before collecting statistics, which consumes a large amount of storage space. For the HLL, the original data does not need to be stored, and only the HLL data structure needs to be maintained. Therefore, the occupied space is always at the 1280-byte constant level. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/9-creating-an-index-online.md b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/9-creating-an-index-online.md new file mode 100644 index 0000000000000000000000000000000000000000..d0d2b33a9e548ce33b1fe6a5571dd5a6d05ed151 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/enterprise-level-features/9-creating-an-index-online.md @@ -0,0 +1,40 @@ +--- +title: Creating an Index Online +summary: Creating an Index Online +author: Guo Huan +date: 2022-05-07 +--- + +# Creating an Index Online + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Uses the CREATE INDEX CONCURRENTLY syntax to create indexes online without blocking DML. + +## Benefits + +When creating an index, you can specify the CONCURRENTLY keyword to ensure that the DML and online services are not blocked during the index creation. + +## Description + +A normal CREATE INDEX acquires exclusive lock on the table on which the index depends, blocking other accesses until the index drop can be completed. If the CONCURRENTLY keyword is specified, the ShareUpdateExclusiveLock lock is added to the table so that DML is not blocked during the creation. + +This keyword is specified when an index is created online. The entire table needs to be scanned twice and built. When the table is scanned for the first time, an index is created and the read and write operations are not blocked. During the second scan, changes that have occurred since the first scan are merged and updated. The table needs to be scanned and built twice, and all existing transactions that may modify the table must be completed. This means that the creation of the index takes a longer time than normal. In addition, the CPU and I/O consumption also affects other services. + +## Enhancements + +None + +## Constraints + +- Only one index name can be specified when an index is created online. +- The CREATE INDEX statement can be run within a transaction, but CREATE INDEX CONCURRENTLY cannot. +- Column-store tables, partitioned tables, and temporary tables do not support CREATE INDEX CONCURRENTLY. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/1-primary-standby.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/1-primary-standby.md new file mode 100644 index 0000000000000000000000000000000000000000..52212e756bca8891bbc4430fe66fc13a37332ef0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/1-primary-standby.md @@ -0,0 +1,42 @@ +--- +title: Primary/Standby +summary: Primary/Standby +author: Guo Huan +date: 2022-05-07 +--- + +# Primary/Standby + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +To ensure that a fault can be rectified, data needs to be written into multiple copies. Multiple copies are configured for the primary and standby nodes, and logs are used for data synchronization. In this way, MogDB has no data lost when a node is faulty or the system restarts after a stop, meeting the ACID feature requirements. + +## Benefits + +Services can be switched to the standby node when the primary node is faulty. Therefore, data is not lost and services can be quickly restored. + +## Description + +You can deploy the one-primary-multiple-standby mode. In the one-primary-multiple-standby mode, all standby nodes need to redo logs and can be promoted to the primary. The one-primary-multiple-standby mode provides higher DR capabilities and is more suitable for the OLTP system that processes a large number of transactions. + +The **switchover** command can be used to trigger a switchover between the primary and standby nodes. If the primary node is faulty, the **failover** command can be used to promote the standby node to the primary. + +In scenarios such as initial installation or backup and restoration, data on the standby node needs to be rebuilt based on the primary node. In this case, the build function is required to send the data and WALs of the primary node to the standby node. When the primary node is faulty and joins again as a standby node, the build function needs to be used to synchronize data and WALs with those of the new primary node. In addition, in online capacity expansion scenarios, you need to use build to synchronize metadata to instances on new nodes. Build includes full build and incremental build. Full build depends on primary node data for rebuild. The amount of data to be copied is large and the time required is long. Incremental build copies only differential files. The amount of data to be copied is small and the time required is short. Generally, the incremental build is preferred for fault recovery. If the incremental build fails, the full build continues until the fault is rectified. + +To implement HA DR for all instances, in addition to the preceding primary/standby multi-copy replication configured for DNs, MogDB also provides other primary/standby DR capabilities, such as CM server (one primary and multiple standbys) and ETCD (one primary and multiple standbys). In this way, instances can be recovered as soon as possible without interrupting services, minimizing the impact of hardware, software, and human errors on services and ensuring service continuity. + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/10-adding-or-deleting-a-standby-server.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/10-adding-or-deleting-a-standby-server.md new file mode 100644 index 0000000000000000000000000000000000000000..6f4ddffc9c1cbd11b8c334d13e27a2493945d6a4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/10-adding-or-deleting-a-standby-server.md @@ -0,0 +1,58 @@ +--- +title: Adding or Deleting a Standby Node +summary: Adding or Deleting a Standby Node +author: Guo Huan +date: 2022-05-07 +--- + +# Adding or Deleting a Standby Node + +## Availability + +This feature is available since MogDB 2.0.0. + +## Introduction + +Standby nodes can be added and deleted. + +## Benefits + +If the read pressure of the primary node is high or you want to improve the disaster recovery capability of the database, you need to add a standby node. If some standby nodes in a cluster are faulty and cannot be recovered within a short period of time, you can delete the faulty nodes to ensure that the cluster is running properly. + +## Description + +MogDB can be scaled out from a single node or one primary and multiple standbys to one primary and eight standbys. Cascaded standby nodes can be added. Standby nodes can be added when a faulty standby node exists in the cluster. One primary and multiple standbys can be scaled in to a single node. A faulty standby node can be deleted. + +Standby nodes can be added or deleted online without affecting the primary node. + +## Enhancements + +None. + +## Constraints + +For adding a standby node: + +- Ensure that the MogDB image package exists on the primary node. +- Ensure that the same users and user groups as those on the primary node have been created on the new standby node. +- Ensure that the mutual trust of user **root** and the database management user has been established between the existing database nodes and the new nodes. +- Ensure that the XML file has been properly configured and information about the standby node to be scaled has been added to the installed database configuration file. +- Ensure that only user **root** is authorized to run the scale-out command. +- Do not run the **gs_dropnode** command on the primary node to delete other standby nodes at the same time. +- Ensure that the environment variables of the primary node have been imported before the scale-out command is run. +- Ensure that the operating system of the new standby node is the same as that of the primary node. +- Do not perform an primary/standby switchover or failover on other standby nodes at the same time. + +For deleting a standby node: + +- Delete the standby node only on the primary node. +- Do not perform an primary/standby switchover or failover on other standby nodes at the same time. +- Do not run the **gs_expansion** command on the primary node for scale-out at the same time. +- Do not run the **gs_dropnode** command twice at the same time. +- Before deletion, ensure that the database management user trust relationship has been established between the primary and standby nodes. +- Run this command as a database administrator. +- Before running commands, run the **source** command to import environment variables of the primary node. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/11-delaying-entering-the-maximum-availability-mode.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/11-delaying-entering-the-maximum-availability-mode.md new file mode 100644 index 0000000000000000000000000000000000000000..73ae55a481fea2db2fd65a075d929bc0f748c07c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/11-delaying-entering-the-maximum-availability-mode.md @@ -0,0 +1,42 @@ +--- +title: Delaying Entering the Maximum Availability Mode +summary: Delaying Entering the Maximum Availability Mode +author: Guo Huan +date: 2022-05-10 +--- + +# Delaying Entering the Maximum Availability Mode + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +The primary node can be delayed to enter the maximum availability mode. + +## Benefits + +When the primary node detects that the standby node exits due to network instability or other reasons and the maximum availability mode is enabled on the primary node, the primary node remains in the maximum protection mode within a specified time window. After the time window expires, the primary node enters the maximum availability mode. + +This prevents the primary node from frequently switching between the maximum protection mode and maximum availability mode due to factors such as network jitter and intermittent process disconnection. + +## Description + +If **most_available_sync** is set to **on**, when synchronous standby nodes are faulty in primary/standby scenarios and the number of configured synchronous standby nodes is insufficient (for details, see the meaning of **synchonous_standby_name**), setting **keep_sync_window** will retain the maximum protection mode within the time window specified by **keep_sync_window**. That is, committing transactions on the primary node is blocked, delaying the primary node to enter the maximum availability mode. + +If synchronous standby nodes recover from faults and the number of synchronous standby nodes meets the configuration requirements, transactions are not blocked. + +## Enhancements + +None. + +## Constraints + +- This feature takes effect only when the maximum availability mode is enabled. +- Enabling this feature may affect the RPO. If the primary node is faulty within the configured timeout window, its transactions are committed locally but not synchronized to the faulty synchronous standby nodes. +- This feature does not apply to cascaded standby nodes. + +## Dependencies + +This feature depends on the maximum availability mode. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/12-parallel-logical-decoding.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/12-parallel-logical-decoding.md new file mode 100644 index 0000000000000000000000000000000000000000..072a7fa7ba2c9ea5176807e3239298deb232fb67 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/12-parallel-logical-decoding.md @@ -0,0 +1,53 @@ +--- +title: Parallel Logical Decoding +summary: Parallel Logical Decoding +author: Guo Huan +date: 2022-05-10 +--- + +# Parallel Logical Decoding + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Multi-thread parallel decoding is supported. + +## Benefits + +The logical decoding performance is greatly improved and the decoding speed can be improved from 3 to 5 Mbit/s to 100 Mbit/s in standard scenarios (16-core CPU, 128 GB memory, network bandwidth > 200 Mbit/s, 10 to 100 columns in a table, 0.1 KB to 1 KB data volume in a single row, insert as the main DML operation, no flush transaction is involved, that is, number of statements in a single transaction < 4096). + +## Description + +When JDBC or pg_recvlogical is used for decoding, you can set **parallel-decode-num** to a value greater than 1 and less than or equal to 20 to enable parallel decoding. In this way, one read thread, multiple decoding threads, and one sending thread are used to perform logical decoding, significantly improving the decoding speed. + +## Enhancements + +None. + +## Constraints + +1. The hardware and network environment must be normal. The size of a logical log is twice that of an Xlog. To ensure that the speed of the Xlog reaches 100 Mbit/s, the I/O bandwidth must be at least 200 Mbit/s. Resources need to be reserved for the reader, decoder, and sender threads. The number of CPU cores to be reserved must be the number of concurrent threads plus 2. For example, if there are four concurrent threads, six CPU cores need to be reserved. In actual scenarios, decoding on the standby node can meet the requirements, and no special resource reservation planning is required. To ensure that the decoding performance meets the requirements and minimize the impact on services, you are advised to set up only one parallel decoding connection on a standby node to ensure that the CPU, memory, and bandwidth resources are sufficient. +2. The value of GUC parameter **wal_level** of the log level is **logical**. +3. The GUC parameter **max_replication_slots** is set to a value greater than or equal to the number of physical streaming replication slots, backup slots, and logical replication slots required by each DN. +4. The value of the decoding configuration option **parallel-decode-num** is greater than 1 and less than or equal to 20, specifying the number of concurrent decoding threads. +5. Decoding into DDL statements is not supported. +6. Decoding is not supported for column-store data and data page replication. +7. Distributed transactions cannot be decoded. Currently, decoding is performed by DN, which cannot ensure consistency in distributed transaction decoding. +8. The size of a single tuple cannot exceed 1 GB. Decoding results may be larger than inserted data. Therefore, it is recommended that the size of a single tuple be less than or equal to 500 MB. +9. Decoding compressed tables into DML statements is not supported. +10. MogDB supports the following data types for decoding: **INTEGER**, **BIGINT**, **SMALLILNT**, **TINYINT**, **SERIAL**, **SMALLSERIAL**, **BIGSERIAL**, **FLOAT**, **DOUBLE PRECISION**, **DATE**, **TIME[WITHOUT TIME ZONE]**, **TIMESTAMP[WITHOUT TIME ZONE]**, **CHAR(***n***)**, **VARCHAR(***n***)**, and **TEXT**. +11. If the SSL connection is required, the GUC parameter **ssl** must be set to **on**. +12. Interval partitioned tables cannot be replicated. +13. After a DDL statement is executed in a transaction, the DDL statement and subsequent statements are not decoded. +14. To perform decoding on the standby node, set the GUC parameter **enable_slot_log** to **on** on the corresponding host. +15. Currently, ultra-large CLOB decoding is not supported. +16. The same replication slot for decoding cannot be used between the primary node and standby node or between different standby nodes at the same time. Otherwise, data inconsistency will occur. +17. Do not perform operations on the replication slot on other nodes when the logical replication slot is in use. To delete a replication slot, stop decoding in the replication slot first. +18. Only the built-in 'j' style export is supported, and the wal2json extension is not supported. + +## Dependencies + +Decoding is performed on the standby node. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/13-dcf.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/13-dcf.md new file mode 100644 index 0000000000000000000000000000000000000000..50f7b054c082b9e997ba2d58a02c97ea5d5f6544 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/13-dcf.md @@ -0,0 +1,45 @@ +--- +title: DCF +summary: DCF +author: Guo Huan +date: 2022-05-10 +--- + +# DCF + +## Availability + +This feature is available since MogDB 2.0.0. + +## Introduction + +Distributed consensus framework (DCF) implements data synchronization based on the Paxos algorithm. After the DCF mode is enabled, DNs support Paxos-based replication and quorum capabilities. + +## Benefits + +Primary DN selection and log replication are performed based on Paxos. Compression and flow control are supported during the replication to prevent high bandwidth usage. DCF provides capabilities such as log replication and cluster HA. DCF supports automatic leader election, forcible minority startup, and dynamic traffic adjustment for log replication. In addition, multiple Paxos-based roles are provided and can be adjusted. You can query the instance status in the current database. + +DCF is a high-performance, highly mature, reliable, scalable, and easy-to-use independent basic library. Other systems can easily interconnect with DCF through interfaces to obtain the strong consistency, high availability, and automatic disaster recovery capabilities provided by the Paxos algorithm. + +## Description + +- During log replication, DCF supports log compression before transmission to reduce network bandwidth usage. +- DCF supports SSL, including TLS 1.2 and TLS 1.3. When SSL is enabled, the DN configures DCF as TLS 1.2 by default. +- DCF supports five TLS 1.3 cipher suites: TLS13-AES-256-GCM-SHA384, TLS13-CHACHA20-POLY1305-SHA256, TLS13-AES-128-GCM-SHA256, TLS13-AES-128-CCM-8-SHA256 and TLS13-AES-128-CCM-SHA256. +- DCF supports passive nodes, which do not participate in leader election and only synchronize and replay logs. When this type of nodes is heavily loaded during log synchronization, flow control is performed. +- DCF supports logger nodes, which participate in leader election and voting. However, the logger nodes only replicate DCF logs. Xlogs are not replicated and redo is not performed. +- DCF supports online switchover between follower and passive nodes. That is, without interrupting services, a node can switch over between the follower and passive roles. +- DCF supports forcible startup of the minority of DNs. In case that the majority of database instances are faulty, a minority of standby DNs are selected and forcibly promoted to primary, and other normal standby DNs replicate logs from the primary DNs. +- DCF supports automatic leader election. When the original primary DN is faulty, a new primary DN is automatically selected from the remaining standby DNs on the premise that data consistency is ensured. + +## Enhancements + +None. + +## Constraints + +To use this function, you need to enable DCF during installation and deployment. In DCF mode, the majority election is used. During the installation, if the number of faulty nodes and build nodes reaches the majority, the cluster installation will fail. For example, when one primary node and two standby nodes are installed, one node fails to be installed due to insufficient memory, but the other two nodes can be started normally. A standby node will be built again. In this case, the number of build nodes and faulty nodes is 2, which is the majority, and the cluster installation will fail. During the installation, check whether the memory and disk resources are sufficient. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/14-cm.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/14-cm.md new file mode 100644 index 0000000000000000000000000000000000000000..90d4a15f86fef51054f6a280f2a80a385b3e7cfd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/14-cm.md @@ -0,0 +1,62 @@ +--- +title: CM +summary: CM +author: Guo Huan +date: 2022-05-10 +--- + +# CM + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Cluster manager (CM) is a database management software, which consists of cm_server, cm_agent, om_monitor, cm_ctl. Its core features are as follows. + +- cm_server + + Send commands (such as start, stop, status query, switchover and failover of database instances) to cm_agent on each node and receive responses; + + Receive the database instance status information reported by cm_agent on each node; + + Highly available arbitration of cm_server itself on each node; + + Highly available arbitration of database instances on each node. + +- cm_agent + + Receive and execute commands from cm_server, such as start, stop, status query, switchover and failover of database instances; + + Monitor the status of the database instance running on the current node and report it to cm_server. + +- cm_monitor + + Monitor the cm_agent service running on the current node to ensure its availability. + +- cm_ctl + + Client tool for CM. + +## Benefits + +It manages and monitors the running status of functional units and physical resources in a database system, ensuring stable running of the system. + +## Description + +It supports customized resource monitoring and provides capabilities such as monitoring of the primary/standby database status, network communication faults, file system faults, and automatic primary/standby switchover upon faults. It also provides various database management capabilities, such as starting and stopping nodes and instances, querying database instance status, performing primary/standby switchover, and managing logs. + +## Enhancements + +- CM Highly Available Self-Arbitration with embedded arbitration algorithms and logic that does not rely on third-party components to achieve arbitration; + +- Customized Resource Monitoring. + +## Constraints + +In scenarios where there are one primary node and one standby node, CM supports only basic capabilities, such as installation, startup, stop, and detection. + +## Dependencies + +Highly available arbitration relies on DCF, DCC, and CBB components. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/15-global-syscache.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/15-global-syscache.md new file mode 100644 index 0000000000000000000000000000000000000000..d7f8243001af307e4d2ffe8196c3457ca617b67a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/15-global-syscache.md @@ -0,0 +1,45 @@ +--- +title: Global SysCache +summary: Global SysCache +author: Guo Huan +date: 2022-05-10 +--- + +# Global SysCache + +## 可获得性 + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Global SysCache is the global and local caches of system catalog data. Figure 1 illustrates the principle. + +**Figure 1** Global SysCache principle + + ![Global-SysCache原理图](https://cdn-mogdb.enmotech.com/docs-media/mogdb/characteristic-description/Global-SysCache.png) + +## Benefits + +Global SysCache reduces the cache memory usage of database processes and improves the concurrent expansion capability of a database. + +## Description + +Global SysCache decouples the system cache from sessions and binds them to threads to reduce the memory usage together with the thread pool feature. In addition, it is used to improve the cache hit rate and ensure stable performance. + +## Enhancements + +Supports more concurrent queries. + +## Constraints + +- Set **enable_global_syscache** to **on**. You are advised to set **enable_thread_pool** to **on**. +- When the number of databases is large and the value of **global_syscache_threshold** is small, memory control cannot work properly and the performance deteriorates. +- Distributed time series tasks are not supported. The memory control and performance of these tasks are not affected by the GSC feature. +- If **wal_level** is set to **minimal** or **archive**, the query performance of the standby server deteriorates and short connections are used. + +## Dependencies + +The memory reduction capability of this feature depends on the thread pool feature. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/16-using-a-standby-node-to-build-a-standby-node.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/16-using-a-standby-node-to-build-a-standby-node.md new file mode 100644 index 0000000000000000000000000000000000000000..9604e9a78bd0233e6c64912083eb01d1d506f94e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/16-using-a-standby-node-to-build-a-standby-node.md @@ -0,0 +1,36 @@ +--- +title: Using a Standby Node to Build a Standby Node +summary: Using a Standby Node to Build a Standby Node +author: Guo Huan +date: 2022-05-10 +--- + +# Using a Standby Node to Build a Standby Node + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +A standby node can be built by another standby node to accelerate standby node recovery from faults. The I/O and bandwidth pressure of the primary node can be reduced. + +## Benefits + +When the service load is heavy, building a standby node by using the primary node increases the resource burden on the primary node. As a result, the performance of the primary node deteriorates and the build becomes slow. Building a standby node by using a standby node does not affect services on the primary node. + +## Description + +You can run the **gs_ctl** command to specify a standby node to build another standby node to be repaired. For details, see “Tools Used in the Internal System > [gs_ctl](../../reference-guide/tool-reference/tools-used-in-the-internal-system/4-gs_ctl.md)” in *Tool Reference*. + +## Enhancements + +None. + +## Constraints + +A standby node can only be used to build another standby node. You can only use a specified IP address and port number to build the standby node. Before building the standby node, ensure that the logs of the standby node to be repaired are later than those of the standby node that sends data. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/2-logical-replication.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/2-logical-replication.md new file mode 100644 index 0000000000000000000000000000000000000000..7aed4a90f19181b2918b293545713b7e41320faa --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/2-logical-replication.md @@ -0,0 +1,37 @@ +--- +title: Logical Replication +summary: Logical Replication +author: Guo Huan +date: 2022-05-07 +--- + +# Logical Replication + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +MogDB provides the logical decoding function to reversely parse physical logs to logical logs. Logical replication tools such as DRS convert logical logs to SQL statements and replay the SQL statements in the peer database. In this way, data can be synchronized between heterogeneous databases. Currently, unidirectional and bidirectional logical replication between the MogDB database and the MySQL or Oracle database is supported. + +## Benefits + +Logical replication is applicable to real-time database data migration, dual-database active-active system, and rolling upgrades. + +## Description + +DNs reversely parse physical logs to logical logs. Logical replication tools such as DRS extract logical logs from DNs, convert the logs to SQL statements, and replay the SQL statements in MySQL. Logical replication tools also extract logical logs from a MySQL database, reversely parse the logs to SQL statements, and replay the SQL statements in MogDB. In this way, data can be synchronized between heterogeneous databases. + +## Enhancements + +- MogDB 1.1.0 logic decoding supports the extraction of logs from full and incremental logs. +- MogDB 1.1.0 supports logical decoding on a standby node. + +## Constraints + +Column-store replication and DDL replication are not supported. + +## Dependencies + +It depends on logical replication tools that decode logical logs. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/3-online-node-replacement.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/3-online-node-replacement.md new file mode 100644 index 0000000000000000000000000000000000000000..785730b077ac788a9fc65781e82490edb7171a1b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/3-online-node-replacement.md @@ -0,0 +1,38 @@ +--- +title: Online Node Replacement +summary: Online Node Replacement +author: Guo Huan +date: 2022-05-07 +--- + +# Online Node Replacement + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +If a node in a database is unavailable or the instance status is abnormal due to a hardware fault and the database is not locked, you can replace the node or rectify the instance fault to restore the database. During the restoration, DML operations are supported. DDL operations are supported in limited scenarios only. + +## Benefits + +Currently, the scale of enterprise data is increasing, the number of nodes increases sharply, and the probability of hardware damage increases accordingly. The traditional offline node replacement mode cannot meet customer requirements for uninterrupted services. During routine O&M, frequent service interruption will bring great loss to customers. However, the current database products in the industry cannot meet the requirements of physical node replacement in large-scale data scenarios without service interruption. Services need to be interrupted, or only some operations are allowed when services are not interrupted. + +## Description + +If a node in a database is unavailable or the instance status is abnormal due to a hardware fault, you can replace the node or rectify the instance fault to restore the database. During the restoration, DML operations are supported. DDL operations are supported in limited scenarios only. + +## Enhancements + +None + +## Constraints + +Currently, online DDL operations are supported during node replacement. + +- During node replacement, DML operations are supported and DDL operations are supported in certain scenarios. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/4-logical-backup.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/4-logical-backup.md new file mode 100644 index 0000000000000000000000000000000000000000..58d7b2aa5d463e018d1abc85ab8d7d3ead6427b1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/4-logical-backup.md @@ -0,0 +1,40 @@ +--- +title: Logical Backup +summary: Logical Backup +author: Guo Huan +date: 2022-05-07 +--- + +# Logical Backup + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Data in user tables in the database is backed up to a specified storage medium in a general format. + +## Benefits + +Through logical backup, you can achieve the following purposes: + +- Back up user data to a reliable storage medium to secure data. +- Support cross-version recovery and heterogeneous recovery using a general data format. +- Archive cold data. + +## Description + +MogDB provides the logical backup capability to back up data in user tables to local disk files in text or CSV format and restore the data in homogeneous or heterogeneous databases. + +## Enhancements + +None + +## Constraints + +For details about the restrictions on logical backup, see [gs_dump](../../reference-guide/tool-reference/server-tools/5-gs_dump.md) in the *Tool Reference*. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/5-physical-backup.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/5-physical-backup.md new file mode 100644 index 0000000000000000000000000000000000000000..9eefbcee1dee0b07f96dd8b8794cefbf028c701f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/5-physical-backup.md @@ -0,0 +1,52 @@ +--- +title: Physical Backup +summary: Physical Backup +author: Guo Huan +date: 2022-05-07 +--- + +# Physical Backup + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Data in the entire database is backed up to a specified storage medium in an internal format. + +## Benefits + +Through physical backup, you can achieve the following purposes: + +- Back up data of the entire database to a reliable storage medium, improving system reliability. +- Improve backup and restoration performance using an internal data format. +- Archive cold data. + +The typical physical backup policy and application scenario are as follows: + +- On Monday, perform a full backup of the database. +- On Tuesday, perform an incremental backup based on the full backup on Monday. +- On Wednesday, perform an incremental backup based on the incremental backup on Tuesday. +- … +- On Sunday, perform an incremental backup based on the incremental backup on Saturday. + +The preceding backup operations are executed every week. + +## Description + +MogDB 1.1.0 provides the physical backup capability to back up data of the entire database to local disk files, OBS objects, NBU objects, or EISOO objects in the internal database format, and restore data of the entire database in a homogeneous database. In addition to the preceding functions, it also provides advanced functions such as compression, flow control, and resumable backup. + +Physical backup is classified into full backup and incremental backup. The difference is as follows: Full backup includes the full data of the database at the backup time point. The time required for full backup is long (in direct proportion to the total data volume of the database), and a complete database can be restored. An incremental backup involves only incremental data modified after a specified time point. It takes a short period of time (in direct proportion to the incremental data volume and irrelevant to the total data volume). However, a complete database can be restored only after the incremental backup and full backup are performed. + +## Enhancements + +Supports full backup and incremental backup simultaneously. + +## Constraints + +For details about the constraints of physical backup, see Backup and Restoration > [Physical Backup and Restoration](../../administrator-guide/br/1-2-br.md) in *Administrator Guide*. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/6-automatic-job-retry-upon-failure.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/6-automatic-job-retry-upon-failure.md new file mode 100644 index 0000000000000000000000000000000000000000..ad3bdd07d78e2e5c14daa72110b2270816521e30 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/6-automatic-job-retry-upon-failure.md @@ -0,0 +1,87 @@ +--- +title: Automatic Job Retry upon Failure +summary: Automatic Job Retry upon Failure +author: Guo Huan +date: 2022-05-07 +--- + +# Automatic Job Retry upon Failure + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +If an error occurs in batch processing jobs due to network exceptions or deadlocks, failed jobs are automatically retried. + +## Benefits + +In common fault scenarios, such as network exception and deadlock, queries retry automatically in case of failure to improve database usability. + +## Description + +MogDB provides the job retry mechanism: gsql Retry. + +- The gsql retry mechanism uses a unique error code (SQL STATE) to identify an error that requires a retry. The function of the client tool gsql is enhanced. The error code configuration file **retry_errcodes.conf** is used to configure the list of errors that require a retry. The file is stored in the installation directory at the same level as gsql. **gsql** provides the **\set RETRY** [*number*] command to enable or disable the retry function. The number of retry times ranges from 5 to 10, and the default value is **5**. When this function is enabled, **gsql** reads the preceding configuration file. The error retry controller records the error code list through the container. If an error occurs in the configuration file after the function is enabled, the controller sends the cached query statement to the server for retry until the query is successful or an error is reported when the number of retry times exceeds the maximum. + +## Enhancements + +None + +## Constraints + +- Functionality constraints: + + - Retrying increases execution success rate but does not guarantee success. + +- Error type constraints: + + Only the error types in Table 1 are supported. + + **Table 1** Supported error types + + | Error Type | Error Code | Remarks | + | :-------------------------------- | :--------- | :----------------------------------------------------------- | + | CONNECTION_RESET_BY_PEER | YY001 | TCP communication error. Print information: “Connection reset by peer” | + | STREAM_CONNECTION_RESET_BY_PEER | YY002 | TCP communication error. Print information: “Stream connection reset by peer” (communication between DNs) | + | LOCK_WAIT_TIMEOUT | YY003 | Lock wait timeout. Print information: “Lock wait timeout” | + | CONNECTION_TIMED_OUT | YY004 | TCP communication error. Print information: “Connection timed out” | + | SET_QUERY_ERROR | YY005 | Failed to deliver the **SET** command. Print information: “Set query error” | + | OUT_OF_LOGICAL_MEMORY | YY006 | Failed to apply for memory. Print information: “Out of logical memory” | + | SCTP_MEMORY_ALLOC | YY007 | SCTP communication error. Print information: “Memory allocate error” | + | SCTP_NO_DATA_IN_BUFFER | YY008 | SCTP communication error. Print information: “SCTP no data in buffer” | + | SCTP_RELEASE_MEMORY_CLOSE | YY009 | SCTP communication error. Print information: “Release memory close” | + | SCTP_TCP_DISCONNECT | YY010 | SCTP and TCP communication error. Print information: “SCTP, TCP disconnect” | + | SCTP_DISCONNECT | YY011 | SCTP communication error. Print information: “SCTP disconnect” | + | SCTP_REMOTE_CLOSE | YY012 | SCTP communication error. Print information: “Stream closed by remote” | + | SCTP_WAIT_POLL_UNKNOW | YY013 | Waiting for an unknown poll. Print information: “SCTP wait poll unknow” | + | SNAPSHOT_INVALID | YY014 | Invalid snapshot. Print information: “Snapshot invalid” | + | ERRCODE_CONNECTION_RECEIVE_WRONG | YY015 | Failed to receive a connection. Print information: “Connection receive wrong” | + | OUT_OF_MEMORY | 53200 | Out of memory. Print information: “Out of memory” | + | CONNECTION_EXCEPTION | 08000 | Failed to communicate with DNs due to connection errors. Print information: “Connection exception” | + | ADMIN_SHUTDOWN | 57P01 | System shutdown by the administrator. Print information: “Admin shutdown” | + | STREAM_REMOTE_CLOSE_SOCKET | XX003 | Remote socket disabled. Print information: “Stream remote close socket” | + | ERRCODE_STREAM_DUPLICATE_QUERY_ID | XX009 | Duplicate query. Print information: “Duplicate query id” | + | ERRCODE_STREAM_CONCURRENT_UPDATE | YY016 | Concurrent stream query and update. Print information: “Stream concurrent update” | + +- Statement type constraints: + + Support single-statement stored procedures, functions, and anonymous blocks. Statements in transaction blocks are not supported. + +- Statement constraints of a stored procedure: + + - If an error occurs during the execution of a stored procedure containing EXCEPTION (including statement block execution and statement execution in EXCEPTION), the stored procedure can be retried. If the error is captured by EXCEPTION, the stored procedure cannot be retried. + - Advanced packages that use global variables are not supported. + - DBE_TASK is not supported. + - PKG_UTIL file operation is not supported. + +- Data import constraints: + + - The **COPY FROM STDIN** statement is not supported. + - The **gsql \copy from** metacommand is not supported. + - Data cannot be imported using **JDBC CopyManager copyIn**. + +## Dependencies + +Valid only if the **gsql** tool works normally and the error list is correctly configured. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/7-ultimate-rto.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/7-ultimate-rto.md new file mode 100644 index 0000000000000000000000000000000000000000..b843cff877e9974a8adf829c132af06160ee7a54 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/7-ultimate-rto.md @@ -0,0 +1,39 @@ +--- +title: Ultimate RTO +summary: Ultimate RTO +author: Guo Huan +date: 2022-05-07 +--- + +# Ultimate RTO + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +- The database host can be quickly restored after being restarted. +- Logs can be synchronized between the primary and standby nodes to accelerate playback on the standby node. + +## Benefits + +When the service load is heavy, the playback speed of the standby node cannot catch up with that of the primary node. After the system runs for a long time, logs are accumulated on the standby node. If a host is faulty, data restoration takes a long time and the database is unavailable, which severely affects system availability. + +The ultimate recovery time object (RTO) is enabled to reduce the data recovery time after a host fault occurs and improve availability. + +## Description + +After the ultimate RTO function is enabled, multi-level pipelines are established for Xlog log playback to improve the concurrency and log playback speed. + +## Enhancements + +None. + +## Constraints + +The ultimate RTO focuses only on whether the RTO of the standby node meets the requirements. The ultimate RTO removes the built-in flow control and uses the **recovery_time_target** parameter for flow control. This feature does not support the read operation on the standby node. If you query the standby node, a core dump may occur on the standby node. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/8-cascaded-standby-server.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/8-cascaded-standby-server.md new file mode 100644 index 0000000000000000000000000000000000000000..51a0cec5d74d3851b856ee353a1267200d3b0f10 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/8-cascaded-standby-server.md @@ -0,0 +1,45 @@ +--- +title: Cascaded Standby Server +summary: Cascaded Standby Server +author: Guo Huan +date: 2022-05-07 +--- + +# Cascaded Standby Server + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +A cascaded standby server can be connected to a standby server based on the one-primary-multiple-standby architecture. + +## Benefits + +The one-primary-multiple-standby architecture cannot support a flexible structure in feature service scenarios. The multi-equipment room deployment cannot meet requirements of the complete structure in the HA switchover scenario (three instances in the primary and standby equipment rooms and two or three instances in the secondary equipment room). If the number of standby servers increases, the primary server may be overloaded. Queries that have low real-time requirements can be implemented on cascaded standby servers. Therefore, the cascading backup capability is required. + +## Description + +The primary server replicates logs to the standby server in synchronous or asynchronous mode. The standby server replicates logs to the cascaded standby server only in asynchronous mode. + +In the current one-primary-multiple-standby architecture, the primary server uses the WAL sender process (walsender) to replicate logs to the standby server. The standby server uses the WAL receiver process (walreceiver) to receive and then flushes logs to local disks. The standby server reads redo logs to complete data replication between the primary and standby servers. There is a one-to-one mapping between walsender and walreceiver on the primary and standby servers. Logs are sent between the standby and cascaded standby servers in asynchronous mode using walsender and walreceiver, reducing the streaming replication pressure on the primary server. + +## Enhancements + +None + +## Constraints + +- A cascaded standby server can only replicate data from a standby server and cannot directly replicate data from the primary server. +- A cascaded standby server does not support data build from a standby server. Currently, data can be built only from the primary server. If the standby server is fully built, the cascaded standby server needs to be fully built. +- The cascaded standby node is in asynchronous replication mode. +- The cascaded standby server cannot be promoted. +- The cascaded standby server cannot be notified. +- Currently, the overall architecture of the primary-standby-cascaded standby cluster cannot be queried. You need to find the standby server through the primary server and then find the cascaded standby server based on the standby server. +- A cascaded standby server cannot own another cascaded standby server. +- When the ultimate RTO is enabled, no cascaded standby server is supported. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/9-delayed-replay.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/9-delayed-replay.md new file mode 100644 index 0000000000000000000000000000000000000000..a9e134dc965a589f8f37bd6b2b366747affb7f40 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-availability/9-delayed-replay.md @@ -0,0 +1,46 @@ +--- +title: Delayed Replay +summary: Delayed Replay +author: Guo Huan +date: 2022-05-07 +--- + +# Delayed Replay + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The time for a standby node to replay can be delayed. + +## Benefits + +By default, the standby server restores the Xlog records from the primary server as soon as possible. This function allows you to delay the time for a standby node to replay Xlog records. In this case, you can query a copy that records data before a period of time, which helps correct errors such as misoperations. + +## Description + +The GUC parameter **recovery_min_apply_delay** can be used to set the delay time so that a standby server can replay Xlog records from the primary server after a delay time. + +Value range: an integer ranging from 0 to INT_MAX. The unit is ms. + +Default value: **0** (no delay) + +## Enhancements + +None + +## Constraints + +- The **recovery_min_apply_delay** parameter is invalid on the primary node. It must be set on the standby node to be delayed. +- The delay time is calculated based on the timestamp of transaction commit on the primary server and the current time on the standby server. Therefore, ensure that the clocks of the primary and standby servers are the same. +- Operations without transactions are not delayed. +- After the primary/standby switchover, if the original primary server needs to be delayed, you need to manually set this parameter. +- When **synchronous_commit** is set to **remote_apply**, synchronous replication is affected by the delay. Each commit message is returned only after the replay on the standby server is complete. +- Using this feature also delays **hot_standby_feedback**, which may cause the primary server to bloat, so be careful when using both. +- If a DDL operation (such as DROP or TRUNCATE) that holds an AccessExclusive lock is performed on the primary server, the query operation on the operation object on the standby server will be returned only after the lock is released during the delayed replay of the record on the standby server. + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/1-cbo-optimizer.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/1-cbo-optimizer.md new file mode 100644 index 0000000000000000000000000000000000000000..b81a4b1542db35ebe3a0edf4dea5d0f3b67aed5b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/1-cbo-optimizer.md @@ -0,0 +1,36 @@ +--- +title: CBO Optimizer +summary: CBO Optimizer +author: Guo Huan +date: 2022-05-07 +--- + +# CBO Optimizer + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The MogDB optimizer is cost-based optimization (CBO). + +## Benefits + +The MogDB CBO optimizer can select the most efficient execution plan among multiple plans based on the cost to meet customer service requirements to the maximum extent. + +## Description + +By using CBO, the database calculates the number of tuples and the execution cost for each step under each execution plan based on the number of table tuples, column width, null record ratio, and characteristic values, such as distinct, MCV, and HB values, and certain cost calculation methods. The database then selects the execution plan that takes the lowest cost for the overall execution or for the return of the first tuple. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/10-xlog-no-lock-flush.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/10-xlog-no-lock-flush.md new file mode 100644 index 0000000000000000000000000000000000000000..bd7cc515fd4bd95211083ebc30a35aeb03dcfedb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/10-xlog-no-lock-flush.md @@ -0,0 +1,36 @@ +--- +title: Xlog no Lock Flush +summary: Xlog no Lock Flush +author: Guo Huan +date: 2022-05-07 +--- + +# Xlog no Lock Flush + +## Availability + +This feature is available since MogDB 2.0.0. + +## Introduction + +Canceled the WalInsertLock contention and dedicated WalWriter disk write thread. + +## Benefits + +The system performance is further improved on the basis that the original Xlog functions remain unchanged. + +## Description + +This feature optimizes the WalInsertLock mechanism by using log sequence numbers (LSNs) and log record counts (LRCs) to record the copy progress of each backend. The backend can directly copy logs to the WalBuffer without contending for the WalInsertLock. In addition, a dedicated WALWriter thread is used to write logs, and the backend thread does not need to ensure the Xlog flushing. + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/11-parallel-page-based-redo-for-ustore.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/11-parallel-page-based-redo-for-ustore.md new file mode 100644 index 0000000000000000000000000000000000000000..886a45a0a3e58fcd4eb68183690cf1d96b2f676d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/11-parallel-page-based-redo-for-ustore.md @@ -0,0 +1,36 @@ +--- +title: Parallel Page-based Redo For Ustore +summary: Parallel Page-based Redo For Ustore +author: Guo Huan +date: 2022-05-07 +--- + +# Parallel Page-based Redo For Ustore + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +Optimized Ustore inplace update WAL write and improved the degree of parallelism for Ustore DML operation replay. + +## Benefits + +The WAL space used by the update operation is reduced, and the degree of parallelism for Ustore DML operation replay is improved. + +## Description + +Prefixes and suffixes are used to reduce the write times of WAL update. Replay threads are classified to solve the problem that most Ustore DML WALs are replayed on multiple pages. In addition, Ustore data pages are replayed based on **blkno**. + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +This feature depends on the Ustore engine. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/12-row-store-execution-to-vectorized-execution.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/12-row-store-execution-to-vectorized-execution.md new file mode 100644 index 0000000000000000000000000000000000000000..55fb69e4998dc1b3464fa59cc81886511eb531c3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/12-row-store-execution-to-vectorized-execution.md @@ -0,0 +1,40 @@ +--- +title: Row-Store Execution to Vectorized Execution +summary: Row-Store Execution to Vectorized Execution +author: Guo Huan +date: 2022-05-10 +--- + +# Row-Store Execution to Vectorized Execution + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Row-store table queries are converted into vectorized execution plans for execution, improving the execution performance of complex queries. + +## Benefits + +When processing row-store table queries, the executor has poor performance when executing complex queries that include many expressions or join operations, while the vectorized executor has excellent performance when executing complex queries. Therefore, query performance of complex queries can be effectively improved by converting row-store table execution into vectorized execution. + +## Description + +This feature adds a RowToVec operation to scan operators. After the data in the row-store table is converted into the vectorized format in the memory, the upper-layer operators can be converted into vectorized operators. In this way, the vectorized executor can be used for calculation. Scan operators that support this conversion include SeqScan, IndexOnlyscan, IndexScan, BitmapScan, FunctionScan, ValueScan and TidScan. + +## Enhancements + +None. + +## Constraints + +- Vectorization is not supported in the following scenarios: + - The targetList contains a function that returns set. + - The targetList or qual contains expressions that do not support vectorization, such as array expressions, multi-subquery expressions, field expressions, and system catalog columns. + - The following types do not support vectorization: POINTOID, LSEGOID, BOXOID, LINEOID, CIRCLEOID, POLYGONOID, PATHOID, and user-defined types. +- MOTs do not support vectorization. + +## Dependencies + +Vectorized executor \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/2-llvm.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/2-llvm.md new file mode 100644 index 0000000000000000000000000000000000000000..0fb2bc2bd01b313d23ecdd6a2c74d775eb334fd4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/2-llvm.md @@ -0,0 +1,36 @@ +--- +title: LLVM +summary: LLVM +author: Guo Huan +date: 2022-05-07 +--- + +# LLVM + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +MogDB provides the Low Level Virtual Machine (LLVM) technology to query dynamic compilation execution. + +## Benefits + +The requery performance is greatly improved by dynamically building and executing queries. + +## Description + +Based on the query execution plan tree, with the library functions provided by the LLVM, MogDB moves the process of determining the actual execution path from the executor phase to the execution initialization phase. In this way, problems such as function calling, logic condition branch determination, and a large amount of data reading that are related to the original query execution are avoided, to improve the query performance. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +It depends on the LLVM open-source component. Currently, the open-source version 10.0.0 is used. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/3-vectorized-engine.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/3-vectorized-engine.md new file mode 100644 index 0000000000000000000000000000000000000000..8cd4b289edde15ad60f0370c13772c2befb76f79 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/3-vectorized-engine.md @@ -0,0 +1,43 @@ +--- +title: Vectorized Engine +summary: Vectorized Engine +author: Guo Huan +date: 2022-05-07 +--- + +# Vectorized Engine + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The vectorized execution engine, provided by MogDB, is usually used in OLAP data warehouse systems because analytical systems are usually data-intensive and access most data in a table in a sequential manner, perform calculation, and finally output a calculation result to an end user. + +## Benefits + +Batch calculation greatly improves the performance of complex query. + +## Description + +The traditional database query execution uses the tuple-based pipeline execution mode. In most time, the CPU is not used to actually process data, but to traverse the query operation tree. As a result, the effective utilization of the CPU is not high. This also results in low instruction cache performance and frequent jumps. Worse still, this approach does not take advantage of the new capabilities of the new hardware to speed up the execution of queries. In the execution engine, another solution is to change a tuple to a column at a time. This is also the basis of our vectorized execution engine. + +The vectorized engine is bound to the column-store technology, because data of each column is stored together, and it may be considered that the data is stored in an array manner. Based on such a feature, when a same operation needs to be performed on the column data, calculation of each value of the data block may be efficiently completed by using a cycle. + +The advantages of the vectorized execution engine are as follows: + +- This reduces inter-node scheduling and improves CPU usage. +- Because the same type of data is put together, it is easier to leverage the new optimization features of hardware and compilation. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +It depends on column store. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/4-hybrid-row-column-store.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/4-hybrid-row-column-store.md new file mode 100644 index 0000000000000000000000000000000000000000..962a56f5079bd16e8afc3852a2d1633bfdd49bc0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/4-hybrid-row-column-store.md @@ -0,0 +1,72 @@ +--- +title: Hybrid Row-Column Store +summary: Hybrid Row-Column Store +author: Guo Huan +date: 2022-05-07 +--- + +# Hybrid Row-Column Store + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +MogDB supports both row-store and column-store models. Choose a row-store or column-store table as needed. + +Column-store is recommended if a table contains many columns (called a wide table) but its query involves only a few columns. Figure 1 shows the column-store model. Row store is recommended if a table contains only a few columns and a query involves most of the fields. + +**Figure 1** Column-store + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/characteristic-description/hybrid-row-column-store-2.png) + +## Benefits + +In a wide table containing a huge amount of data, a query usually only includes certain columns. In this case, the query performance of the row-store engine is poor. For example, a single table containing the data of a meteorological agency has 200 to 800 columns. Among these columns, only 10 are frequently accessed. In this case, a vectorized execution and column-store engine can significantly improve performance by saving storage space. + +## Description + +Tables are categorized into row-store and column-store tables. Each storage model applies to specific scenarios. Select an appropriate model when creating a table. + +- Row-store table + + Row-store tables are created by default. Data is stored by row. Row-store supports adding, deleting, modifying, and querying data of a complete row. Therefore, this storage model applies to scenarios where data needs to be updated frequently. + +- Column-store table + + Data is stored by column. The I/O of data query in a single column is small, and column-store tables occupy less storage space than row-store tables. This storage model applies to scenarios where data is inserted in batches, less updated, and queried for statistical analysis. The performance of single point query and single record insertion in a column-store table is poor. + +- Selecting a storage model + + - Update frequency + + If data is frequently updated, use a row-store table. + + - Data insertion frequency + + If a small amount of data is frequently inserted each time, use a row-store table. If a large amount of data is inserted at a time, use a column-store table. + + - Number of columns + + If a table is to contain many columns, use a column-store table. + + - Number of columns to be queried + + If only a small number of columns (less than 50% of the total) is queried each time, use a column-store table. + + - Compression ratio + + The compression ratio of a column-store table is higher than that of a row-store table. High compression ratio consumes more CPU resources. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/5-adaptive-compression.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/5-adaptive-compression.md new file mode 100644 index 0000000000000000000000000000000000000000..c3b375036a15849d499c033fa923f42f2d291fe9 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/5-adaptive-compression.md @@ -0,0 +1,42 @@ +--- +title: Adaptive Compression +summary: Adaptive Compression +author: Guo Huan +date: 2022-05-07 +--- + +# Adaptive Compression + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Data compression is the major technology used in current databases. Various compression algorithms are used for different data types. If pieces of data of the same type have different characteristics, their compression algorithms and results will also be different. Adaptive compression chooses the suitable compression algorithm for data based on the data type and characteristics, achieving high performance in compression ratio, import, and query. + +## Benefits + +Importing and frequently querying a huge amount of data are the main application scenarios. When you import data, adaptive compression greatly reduces the data volume, increases I/O operation efficiency several times, and clusters data before storage, achieving fast data import. In this way, only a small number of I/O operations is required and data is quickly decompressed in a query. Data can be quickly retrieved and the query result is quickly returned. + +## Description + +Currently, the database has implemented various compression algorithms on column store, including RLE, DELTA, BYTEPACK/BITPACK, LZ4, ZLIB, and LOCAL DICTIONARY. The following table lists data types and the compression algorithms suitable for them. + +| - | RLE | DELTA | BITPACK/BYTEPACK | LZ4 | ZLIB | LOCAL DICTIONARY | +| :----------------------------------------------------------- | :--- | :---- | :--------------- | :--- | :--- | :--------------- | +| Smallint/Int/Bigint/Oid
Decimal/Real/Double
Money/Time/Date/Timestamp | √ | √ | √ | √ | √ | - | +| Tinterval/Interval/Time with time zone/ | - | - | - | - | √ | - | +| Numeric/Char/Varchar/Text/Nvarchar2
and other supported data types | √ | √ | √ | √ | √ | √ | + +## Enhancements + +The compression level of compression algorithms can be adjusted. + +## Constraints + +None + +## Dependencies + +It depends on LZ4 or ZLIB. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/6-sql-by-pass.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/6-sql-by-pass.md new file mode 100644 index 0000000000000000000000000000000000000000..d391b8d0e87a32649b4d638cfdd12af2c10f25c0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/6-sql-by-pass.md @@ -0,0 +1,36 @@ +--- +title: SQL by pass +summary: SQL by pass +author: Guo Huan +date: 2022-05-07 +--- + +# SQL by pass + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Query performance is improved by customizing an execution scheme for typical queries in the TP scenario. + +## Benefits + +The TP query performance is improved. + +## Description + +In a typical OLTP scenario, simple queries account for a large proportion. This type of queries involves only single tables and simple expressions. To accelerate such query, the SQL bypass framework is proposed. After simple mode judgment is performed on such query at the parse layer, the query enters a special execution path and skips the classic execution framework, including operator initialization and execution, expression, and projection. Instead, it directly rewrites a set of simple execution paths and directly invokes storage interfaces, greatly accelerating the execution of simple queries. + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/7-kunpeng-numa-architecture-optimization.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/7-kunpeng-numa-architecture-optimization.md new file mode 100644 index 0000000000000000000000000000000000000000..602782aef350485589b3e9e58053eaf3cb836bfe --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/7-kunpeng-numa-architecture-optimization.md @@ -0,0 +1,40 @@ +--- +title: Kunpeng NUMA Architecture Optimization +summary: Kunpeng NUMA Architecture Optimization +author: Guo Huan +date: 2022-05-07 +--- + +# Kunpeng NUMA Architecture Optimization + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Kunpeng NUMA architecture optimization mainly focuses on Kunpeng processor architecture features and ARMv8 instruction set, and optimizes the system from multiple layers, including OS, software architecture, lock concurrency, logs, atomic operations, and cache access. This greatly improves the MogDB performance on the Kunpeng platform. + +## Benefits + +Transactions per minute (TPM) is a key performance indicator of the database competitiveness. Under the same hardware costs, a higher database performance means the database can process more services, thereby reducing the usage cost of customers. + +## Description + +- MogDB optimizes the Kunpeng NUMA architecture based on the architecture characteristics. This reduces cross-core memory access latency and maximizes multi-core Kunpeng computing capabilities. The key technologies include redo log batch insertion, NUMA distribution of hotspot data, and CLog partitions, greatly improving the TP system performance. +- Based on the ARMv8.1 architecture used by the Kunpeng chip, MogDB uses the LSE instruction set to implement efficient atomic operations, effectively improving the CPU usage, multi-thread synchronization performance, and Xlog write performance. +- Based on the wider L3 cache line provided by the Kunpeng chip, MogDB optimizes hotspot data access, effectively improving the cache access hit ratio, reducing the cache consistency maintenance overhead, and greatly improving the overall data access performance of the system. +- Kunpeng 920, 2P server (64 cores x 2, memory: 768 GB), 10 GE network, I/O: 4 NVMe PCIe SSDs, TPC-C: 1000 warehouses, performance: 1,500,000 tpmC. + +## Enhancements + +- Batch redo log insertion and CLog partition are supported, improving the database performance on the Kunpeng platform. +- Efficient atomic operations using the LSE instruction set are supported, improving multi-thread synchronization performance. + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/8-high-concurrency-of-thread-pools.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/8-high-concurrency-of-thread-pools.md new file mode 100644 index 0000000000000000000000000000000000000000..8a2b5303b6c1adcc39cfa05678e06a2142085159 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/8-high-concurrency-of-thread-pools.md @@ -0,0 +1,38 @@ +--- +title: High Concurrency of Thread Pools +summary: High Concurrency of Thread Pools +author: Guo Huan +date: 2022-05-07 +--- + +# High Concurrency of Thread Pools + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The thread pooling technology is used to support stable running of databases at high concurrency. + +## Benefits + +The overall system throughput is stable in case of a large number of concurrent requests. + +## Description + +The overall design idea of the thread pool technology is to pool thread resources and reuse them among different connections. After the system is started, a fixed number of working threads are started based on the current number of cores or user configuration. A working thread serves one or more connection sessions. In this way, the session and thread are decoupled. The number of worker threads is fixed. Therefore, frequent thread switchover does not occur in case of high concurrency. The database layer schedules and manages sessions. + +## Enhancements + +This feature is available since MogDB 1.1.0. + +In MogDB 1.1.0, thread pools can be dynamically scaled in or out. + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/9-smp-for-parallel-execution.md b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/9-smp-for-parallel-execution.md new file mode 100644 index 0000000000000000000000000000000000000000..131b424d9061109359d239f1a2f724b68eb20714 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/high-performance/9-smp-for-parallel-execution.md @@ -0,0 +1,44 @@ +--- +title: SMP for Parallel Execution +summary: SMP for Parallel Execution +author: Guo Huan +date: 2022-05-07 +--- + +# SMP for Parallel Execution + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The Symmetric Multi-Processing (SMP) technology of MogDB uses the multi-core CPU architecture of a computer to implement multi-thread parallel computing, fully using CPU resources to improve query performance. + +## Benefits + +Fully utilizes the system multi-core capability to improve requery performance. + +## Description + +In complex query scenarios, a single query takes long time and the system concurrency is low. Therefore, the SMP technology is used to implement operator-level parallel execution, which effectively reduces the query time and improves the query performance and resource utilization. The overall implementation of the SMP technology is as follows: For query operators that can be executed in parallel, data is sliced, multiple working threads are started for computation, and then the results are summarized and returned to the frontend. The data interaction operator **Stream** is added to the SMP architecture to implement data interaction between multiple working threads, ensuring the correctness and integrity of the query. + +## Enhancements + +None. + +## Constraints + +- Index scanning cannot be executed in parallel. +- MergeJoin cannot be executed in parallel. +- WindowAgg order by cannot be executed in parallel. +- The cursor cannot be executed in parallel. +- Queries in stored procedures and functions cannot be executed in parallel. +- Subplans and initplans cannot be queried in parallel, and operators that contain subqueries cannot be executed in parallel, either. +- Query statements that contain the median operation cannot be executed in parallel. +- Queries with global temporary tables cannot be executed in parallel. +- Updating materialized views cannot be executed in parallel. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/1-gray-upgrade.md b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/1-gray-upgrade.md new file mode 100644 index 0000000000000000000000000000000000000000..2608eae939f3d51f7783ed7c141f632492e17e57 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/1-gray-upgrade.md @@ -0,0 +1,36 @@ +--- +title: Gray Upgrade +summary: Gray Upgrade +author: Guo Huan +date: 2022-05-07 +--- + +# Gray Upgrade + +## Availability + +This feature is available since MogDB 2.0.0. + +## Introduction + +Gray upgrade supports full-service operations. All nodes can be upgraded at a time. + +## Benefits + +Gray upgrade provides an online upgrade mode to ensure that all nodes are upgraded without interrupting services. + +## Description + +Gray upgrade is an online upgrade mode that upgrades all nodes. If only the binary files of the database need to be replaced during the gray upgrade, to minimize the impact on services, the two sets of binary files exist on the same node at the same time, and the soft connection switchover mode is used to switch the process version (one intermittent disconnection within 10 seconds). + +## Enhancements + +None. + +## Constraints + +For details about the gray upgrade constraints, see [Upgrade Guide](../../administrator-guide/upgrade-guide.md) in *Administrator Guide*. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/2-workload-diagnosis-report.md b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/2-workload-diagnosis-report.md new file mode 100644 index 0000000000000000000000000000000000000000..57b53b7fa2e96906f7dbd9e4d679f55943d17f19 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/2-workload-diagnosis-report.md @@ -0,0 +1,72 @@ +--- +title: WDR +summary: WDR +author: Guo Huan +date: 2022-05-07 +--- + +# WDR + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +The workload diagnosis report (WDR) provides database performance diagnosis reports based on the baseline performance and incremental data that reflects performance changes. + +## Benefits + +- The WDR is the main method for diagnosing long-term performance problems. Based on the performance baseline of a snapshot, performance analysis is performed from multiple dimensions, helping DBAs understand the system load, performance of each component, and performance bottlenecks. +- Snapshots are also an important data source for self-diagnosis and self-optimization suggestions on subsequent performance problems. + +## Description + +The WDR generates a performance report between two different time points based on the system performance snapshot data at these time points. The report is used to diagnose database kernel performance faults. + +You can use generate_wdr_report(…) to generate a performance report based on two performance snapshots. + +The WDR depends on the following two components: + +- Snapshot: The performance snapshot can be configured to collect a certain amount of performance data from the kernel at a specified interval and store the data in the user tablespace. Any snapshot can be used as a performance baseline for comparison with other snapshots. +- WDR Reporter: This tool analyzes the overall system performance based on two snapshots, calculates the changes of more specific performance indicators between the two time points, and generates summarized and detailed performance data. For details, see Table 1 and Table 2. + +**Table 1** Summarized diagnosis report + +| Diagnosis Type | Description | +| :------------------------------ | :----------------------------------------------------------- | +| Database Stat | Evaluates the load and I/O status of the current database. Load and I/O are the most important indicators of the TP system.
The statistics include the number of sessions connected to the database, number of committed and rolled back transactions, number of read disk blocks, number of disk blocks found in the cache, number of rows returned, captured, inserted, updated, and deleted through database query, number of conflicts and deadlocks, usage of temporary files, and I/O read/write time. | +| Load Profile | Evaluates the current system load from the time, I/O, transaction, and SQL dimensions.
The statistics include the job running elapse time, CPU time, daily transaction quality, logical and physical read volume, read and write I/O times and size, login and logout times, SQL, transaction execution volume, and SQL P80 and P95 response time. | +| Instance Efficiency Percentages | Evaluates the cache efficiency of the current system.
The statistics include the database cache hit ratio. | +| Events | Evaluates the performance of key system kernel resources and key events.
The statistics include the number of times that the key events of the database kernel occur and the waiting time. | +| Wait Classes | Evaluates the performance of key events in the system.
The statistics include the release of the data kernel in the main types of wait events, such as **STATUS**, **LWLOCK_EVENT**, **LOCK_EVENT**, and **IO_EVENT**. | +| CPU | Includes time release of the CPU in user mode, kernel mode, I/O wait mode, or idle mode. | +| IO Profile | Includes the number of database I/O times, database I/O data volume, number of redo I/O times, and redo I/O volume. | +| Memory Statistics | Includes maximum process memory, used process memory, maximum shared memory, and used shared memory. | + +**Table 2** Detailed diagnosis report + +| Diagnosis Type | Description | +| :--------------------- | :----------------------------------------------------------- | +| Time Model | Evaluates the performance of the current system in the time dimension.
The statistics include time consumed by the system in each phase, including the kernel time, CPU time, execution time, parsing time, compilation time, query rewriting time, plan generation time, network time, and I/O time. | +| SQL Statistics | Diagnoses SQL statement performance problems.
The statistics include normalized SQL performance indicators in multiple dimensions: elapsed time, CPU time, rows returned, tuple reads, executions, physical reads, and logical reads. The indicators can be classified into execution time, number of execution times, row activity, and cache I/O. | +| Wait Events | Diagnoses performance of key system resources and key time in detail.
The statistics include the performance of all key events in a period of time, including the number of events and the time consumed. | +| Cache IO Stats | Diagnoses the performance of user tables and indexes.
The statistics include read and write operations on all user tables and indexes, and the cache hit ratio. | +| Utility status | Diagnoses the background task performance.
The statistics include the performance of background tasks such as replication. | +| Object stats | Diagnoses the performance of database objects.
The statistics include user tables, tables on indexes, index scan activities, as well as insert, update, and delete activities, number of valid rows, and table maintenance status. | +| Configuration settings | Determines whether the configuration is changed.
It is a snapshot that contains all current configuration parameters. | +| SQL detail | Displays information about unique query text. | + +## Enhancements + +None. + +## Constraints + +- The WDR snapshot collects performance data of different databases. If there are a large number of databases or tables in the database instance, it takes a long time to create a WDR snapshot. +- If WDR snapshot is performed when a large number of DDL statements are executed, WDR snapshot may fail. +- When the database is dropped, WDR snapshot may fail. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/3-slow-sql-diagnosis.md b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/3-slow-sql-diagnosis.md new file mode 100644 index 0000000000000000000000000000000000000000..32ea05ff01a5e5f5259ac9a28f5f26b16fb10a00 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/3-slow-sql-diagnosis.md @@ -0,0 +1,137 @@ +--- +title: Slow SQL Diagnosis +summary: Slow SQL Diagnosis +author: Guo Huan +date: 2022-05-07 +--- + +# Slow SQL Diagnosis + +## Availability + +This feature is available since MogDB 1.1.0. The following slow SQL views have been discarded before reconstruction: dbe_perf.gs_slow_query_info, dbe_perf.gs_slow_query_history, dbe_perf.global_slow_query_hisotry, and dbe_perf.global_slow_query_info. + +## Introduction + +Slow SQL diagnosis provides necessary information for diagnosing slow SQL statements, helping developers backtrack SQL statements whose execution time exceeds the threshold and diagnose SQL performance bottlenecks. + +## Benefits + +Slow SQL provides detailed information required for slow SQL diagnosis. You can diagnose performance problems of specific slow SQL statements offline without reproducing the problem. The table-based and function-based APIs help users collect statistics on slow SQL indicators and connect to third-party platforms. + +## Description + +Slow SQL diagnosis records information about all jobs whose execution time exceeds the threshold **log_min_duration_statement**. + +Slow SQL provides table-based and function-based query APIs. You can query the execution plan, start time, end time, query statement, row activity, kernel time, CPU time, execution time, parsing time, compilation time, query rewriting time, plan generation time, network time, I/O time, network overhead, and lock overhead. All information is anonymized. + +## Enhancements + +Optimized slow SQL indicators, security (anonymization), execution plans, and query interfaces. + +```sql +Run the following command to check the execution information about the SQL statements in the database instance: +gsql> select * from dbe_perf.get_global_full_sql_by_timestamp(start_timestamp, end_timestamp); +For example: +MogDB=# select * from DBE_PERF.get_global_full_sql_by_timestamp('2020-12-01 09:25:22', '2020-12-31 23:54:41'); +-[ RECORD 1 ]--------+--------------------------------------------------------------------------------------------------------------- +--------------------------------------------------------------- +node_name | dn_6001_6002_6003 +db_name | postgres +schema_name | "$user",public +origin_node | 1938253334 +user_name | user_dj +application_name | gsql +client_addr | +client_port | -1 +unique_query_id | 3671179229 +debug_query_id | 72339069014839210 +query | select name, setting from pg_settings where name in (?) +start_time | 2020-12-19 16:19:51.216818+08 +finish_time | 2020-12-19 16:19:51.224513+08 +slow_sql_threshold | 1800000000 +transaction_id | 0 +thread_id | 139884662093568 +session_id | 139884662093568 +n_soft_parse | 0 +n_hard_parse | 1 +query_plan | Datanode Name: dn_6001_6002_6003 + | Function Scan on pg_show_all_settings a (cost=0.00..12.50 rows=5 width=64) + | Filter: (name = '***'::text) +... + +Run the following command to check the execution information about the slow SQL statements in the database instance: +gsql> select * from dbe_perf.get_global_slow_sql_by_timestamp(start_timestamp, end_timestamp); +MogDB=# select * from DBE_PERF.get_global_slow_sql_by_timestamp('2020-12-01 09:25:22', '2020-12-31 23:54:41'); +-[ RECORD 1 ]--------+--------------------------------------------------------------------------------------------------- +node_name | dn_6001_6002_6003 +db_name | postgres +schema_name | "$user",public +origin_node | 1938253334 +user_name | user_dj +application_name | gsql +client_addr | +client_port | -1 +unique_query_id | 2165004317 +debug_query_id | 72339069014839319 +query | select * from DBE_PERF.get_global_slow_sql_by_timestamp(?, ?); +start_time | 2020-12-19 16:23:20.738491+08 +finish_time | 2020-12-19 16:23:20.773714+08 +slow_sql_threshold | 10000 +transaction_id | 0 +thread_id | 139884662093568 +session_id | 139884662093568 +n_soft_parse | 10 +n_hard_parse | 8 +query_plan | Datanode Name: dn_6001_6002_6003 + | Result (cost=1.01..1.02 rows=1 width=0) + | InitPlan 1 (returns $0) + | -> Seq Scan on pgxc_node (cost=0.00..1.01 rows=1 width=64) + | Filter: (nodeis_active AND ((node_type = '***'::"char") OR (node_type = '***'::"char"))) +... + +Check the execution information about the SQL statement on the current node. +gsql> select * from statement_history; +For example: +MogDB=# select * from statement_history; +-[ RECORD 1 ]--------+--------------------------------------------------------------------------------------------------------------- +--------------------------------------------------------------- +db_name | postgres +schema_name | "$user",public +origin_node | 1938253334 +user_name | user_dj +application_name | gsql +client_addr | +client_port | -1 +unique_query_id | 3671179229 +debug_query_id | 72339069014839210 +query | select name, setting from pg_settings where name in (?) +start_time | 2020-12-19 16:19:51.216818+08 +finish_time | 2020-12-19 16:19:51.224513+08 +slow_sql_threshold | 1800000000 +transaction_id | 0 +thread_id | 139884662093568 +session_id | 139884662093568 +n_soft_parse | 0 +n_hard_parse | 1 +query_plan | Datanode Name: dn_6001_6002_6003 + | Function Scan on pg_show_all_settings a (cost=0.00..12.50 rows=5 width=64) + | Filter: (name = '***'::text) +``` + +## Constraints + +- The SQL tracing information is based on the normal execution logic. The tracing information may inaccurate if SQL statements fail to be executed. +- Restarting a node may cause data loss on the node. +- If you exit a session immediately after SQL statements are executed, the session data that is not updated to the system catalog may be lost. +- The number of SQL statements to be collected is specified by a GUC parameter. If the number of SQL statements exceeds the threshold, new SQL statement execution information will not be collected. +- The maximum number of bytes of lock event details collected by a single SQL statement is specified by a GUC parameter. If the number of bytes exceeds the threshold, new lock event details will not be collected. +- The SQL statement information is updated in asynchronous mode. Therefore, after a query statement is executed, the related view function result is slightly delayed. +- Certain indicator information (such as row activities, cache I/O, and time distribution) depends on the dbe_perf.statement view. If the number of records in the view exceeds the preset size (depending on GUC:instr_unique_sql_count), related indicators may not be collected. +- Functions related to the statement_history table and the details column in the view are in binary format. To parse the detailed information, use the **pg_catalog.statement_detail_decode(details, 'plaintext', true)** function. +- The statement_history table can be queried only in the postgres database. The data in other databases is empty. +- The content of the statement_history table is controlled by track_stmt_stat_level. The default value is **'OFF,L0'**. The first part of the parameter indicates the full SQL statement, and the second part indicates the slow SQL statement. Slow SQL statements are recorded in the statement_history table only when the SQL statement execution time exceeds the value of **log_min_duration_statement**. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/4-session-performance-diagnosis.md b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/4-session-performance-diagnosis.md new file mode 100644 index 0000000000000000000000000000000000000000..cce1a2e377e5be67ceb7daf70fe2eb23ac21c634 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/4-session-performance-diagnosis.md @@ -0,0 +1,102 @@ +--- +title: Session Performance Diagnosis +summary: Session Performance Diagnosis +author: Guo Huan +date: 2022-05-07 +--- + +# Session Performance Diagnosis + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +Session performance diagnosis targets session-level performance faults. + +## Benefits + +- Display the latest events that consume the most resources of user sessions. +- Check the wait events that occupy the most resource-consuming SQL statements. +- Check the wait events that occupy the most resource-consuming sessions. +- Check information about the most resource-consuming users. +- Check the waiting relationship between blocked sessions. + +## Description + +The session performance diagnosis function diagnoses performance of all active sessions in the system. As real-time collection of indicators of all active sessions has a greater impact on user load, the session snapshot technology is used to sample indicators of active sessions, and collect statistics on active sessions from the sampling. The statistics reflect the basic information, status, and resources of active sessions from the dimensions of client information, execution start time, execution end time, SQL text, wait events, and current database objects. The active session information collected based on the probability can help users diagnose which sessions consume more CPU and memory resources, which database objects are hot objects, and which SQL statements consume more key event resources in the system. In this way, users can locate faulty sessions, SQL statements, and database designs. + +Session sampling data is classified into two levels, as shown in Figure 1. + +1. The first level is real-time information stored in the memory. The active session information in the latest several minutes is displayed, which has the highest precision. +2. The second level is the persistent historical information stored in disk files. It displays the historical active session information in a long period of time and is sampled from the memory data. This level is suitable for long-run statistics and analysis. + +**Figure 1** Session performance diagnosis principle + + ![Session性能诊断原理](https://cdn-mogdb.enmotech.com/docs-media/mogdb/characteristic-description/session-performance-diagnosis-1.png) + +Some application scenarios are as follows: + +1. Check the blocking relationship between sessions. + + ```sql + select sessionid, block_sessionid from pg_thread_wait_status; + ``` + +2. Sample information about blocked sessions. + + ```sql + select sessionid, block_sessionid from DBE_PERF.local_active_session; + ``` + +3. Display the final blocked session. + + ```sql + select sessionid, block_sessionid, final_block_sessionid from DBE_PERF.local_active_session; + ``` + +4. Check the wait event that consumes the most resources. + + ```sql + SELECT s.type, s.event, t.count + FROM dbe_perf.wait_events s, ( + SELECT event, COUNT (*) + FROM dbe_perf.local_active_session + WHERE sample_time > now() - 5 / (24 * 60) + GROUP BY event)t WHERE s.event = t.event ORDER BY count DESC; + ``` + +5. Check the events that consume the most session resources in the last five minutes. + + ```sql + SELECT sessionid, start_time, event, count + FROM ( + SELECT sessionid, start_time, event, COUNT(*) + FROM dbe_perf.local_active_session + WHERE sample_time > now() - 5 / (24 * 60) + GROUP BY sessionid, start_time, event) as t ORDER BY SUM(t.count) OVER \(PARTITION BY t. sessionid, start_time)DESC, t.event; + ``` + +6. Check the events that consume the most resources in the last five minutes. + + ```sql + SELECT query_id, event, count + FROM ( + SELECT query_id, event, COUNT(*) + FROM dbe_perf.local_active_session + WHERE sample_time > now() - 5 / (24 * 60) + GROUP BY query_id, event) t ORDER BY SUM (t.count) OVER (PARTITION BY t.query_id ) DESC, t.event DESC; + ``` + +## Enhancements + +None + +## Constraints + +None + +## Dependencies + +None \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/5-system-kpi-aided-diagnosis.md b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/5-system-kpi-aided-diagnosis.md new file mode 100644 index 0000000000000000000000000000000000000000..b91817bc78838ff9855b8ddc17ac1f664053bf15 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/5-system-kpi-aided-diagnosis.md @@ -0,0 +1,66 @@ +--- +title: System KPI-aided Diagnosis +summary: System KPI-aided Diagnosis +author: Guo Huan +date: 2022-05-07 +--- + +# System KPI-aided Diagnosis + +## Availability + +This feature is available since MogDB 1.1.0. + +## Introduction + +KPIs are views of key performance indicators for kernel components or the entire system. Based on KPIs, users can learn about the real-time and historical running status of the system. + +## Benefits + +- Summarized system load diagnosis + + Precise alarms for system load exceptions (overload, stall, and SLA exceptions) and precise system load profile + +- Summarized system time model diagnosis + + Instance-level and query-level time model segmentation, diagnosing the root causes of instance and query performance problems + +- Query performance diagnosis + + Database-level query summary, including top SQL, SQL CPU usage, I/O consumption, execution plan, and excessive hard parsing + +- Diagnosis of disk I/O, index, and buffer performance problems + +- Diagnosis of connection and thread pool problems + +- Diagnosis of checkpoint and redo (RTO) performance problems + +- Diagnosis of system I/O, LWLock, and wait performance problems + + Diagnosis of over 60 modules and over 240 key operation performance problems + +- Function-level performance monitoring and diagnosis (by GSTRACE) + + Tracing of over 50 functions at the storage and execution layers + +## Description + +MogDB provides KPIs of 11 categories and 26 sub-categories, covering instances, files, objects, workload, communication, sessions, threads, cache I/O, locks, wait events, and clusters. + +Figure 1 shows the distribution of kernel KPIs. + +**Figure 1** Distribution of kernel KPIs +![distribution-of-kernel-kpis](https://cdn-mogdb.enmotech.com/docs-media/mogdb/characteristic-description/system-kpi-aided-diagnosis-2.png) + +## Enhancements + +None. + +## Constraints + +- Utility statements do not support normalization. Non-DML statements, such as CREATE, DROP, COPY, and VACUUM, are not supported. +- Currently, only the top-level normalized SQL statements are recorded. SQL statements in a stored procedure are not normalized, and only the SQL statements that call the stored procedure are recorded. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/fault-diagnosis.md b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/fault-diagnosis.md new file mode 100644 index 0000000000000000000000000000000000000000..ad017be29691bff6a1bb64b69dec9eb052d5934c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/maintainability/fault-diagnosis.md @@ -0,0 +1,34 @@ +--- +title: Fault Diagnosis +summary: Fault Diagnosis +author: Zhang Cuiping +date: 2022-06-17 +--- + +# Fault Diagnosis + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +In order to quickly locate faults, collect system fault information, export fault data and then rectify faults, MogDB 3.0 has enhanced OM functions and gstrace diagnostic capabilities. + +## Benefits + +The enhanced fault diagnosis capability can facilitate R&D personnel to rectify faults in time and ensure the normal operation of the system. + +## Description + +The gs_check tool can compare the difference of scenario check results and output a difference analysis report to help users locate the problem quickly. + +The gs_watch tool can monitor MogDB processes and automatically call gs_collector to collect the system status when a process crash is found for later analysis. + +The gs_gucquery tool can automatically collect, organize, and export GUC values, and compare the changes of GUC values at different moment. + +gstrace diagnostic capability is enhanced. It supports opening the trace item of one or more component (module) and function by module name and function name. It enhances the number of gstrace points in the code and the ability to express gstrace output information. It supports export of new key data structures PGPROC and user session data. It realizes fault injection, including system call error report simulation and variable content saving write. + +## Related Pages + +[gs_check](../../reference-guide/tool-reference/server-tools/1-gs_check.md), [gs_gucquery](../../reference-guide/tool-reference/server-tools/gs_gucquery.md), [gstrace](../../reference-guide/tool-reference/tools-used-in-the-internal-system/16-gstrace.md), [gs_watch](../../reference-guide/tool-reference/server-tools/gs_watch.md) diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/middleware/deploying-a-distributed-database-using-kubernetes.md b/product/en/docs-mogdb/v3.1/characteristic-description/middleware/deploying-a-distributed-database-using-kubernetes.md new file mode 100644 index 0000000000000000000000000000000000000000..3d7d35e3a6a472d23cd808a30e8e3875c39e5921 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/middleware/deploying-a-distributed-database-using-kubernetes.md @@ -0,0 +1,36 @@ +--- +title: Deploying a Distributed Database Using Kubernetes +summary: Deploying a Distributed Database Using Kubernetes +author: Guo Huan +date: 2022-05-10 +--- + +# Deploying a Distributed Database Using Kubernetes + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +Deploys a distributed database in one-click mode. + +## Benefits + +Quickly builds a distributed database, and verifies and uses the distributed capability. + +## Description + +Patroni is used to implement planned switchover and automatic failover in case of faults. HAProxy is used to implement read and write load balancing between the primary and standby MogDB nodes. ShardingSphere is used to implement distributed capabilities. All functions are packaged into images and one-click deployment scripts are provided. + +## Enhancements + +None. + +## Constraints + +Only CentOS and openEuler are supported. + +## Dependencies + +ShardingSphere, Patroni, HAProxy \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/middleware/distributed-database-capability.md b/product/en/docs-mogdb/v3.1/characteristic-description/middleware/distributed-database-capability.md new file mode 100644 index 0000000000000000000000000000000000000000..81a028ec59c126405a5687c35e3a4cab85e8d0d9 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/middleware/distributed-database-capability.md @@ -0,0 +1,36 @@ +--- +title: Distributed Database Capability +summary: Distributed Database Capability +author: Guo Huan +date: 2022-05-10 +--- + +# Distributed Database Capability + +## Availability + +This feature is available since MogDB 2.1.0. + +## Introduction + +This feature uses the distributed middleware shardingsphere to provide MogDB the distributed database capability. When 16 Kunpeng 920 (128 cores) nodes are used for networking (1 x shardingsphere-proxy, 7 x shardingsphere-jdbc, 8 x MogDB), the perfect sharding performance is greater than 10 million transactions per minute C (tpmC). + +## Benefits + +A distributed database that logically has no resource restriction can be built over the middleware. + +## Description + +With the sharding capability of shardingsphere, multiple MogDB databases can logically form a larger database with distributed transactions and elastic scaling capabilities. The usage method is the same as that of an MogDB database. + +## Enhancements + +None. + +## Constraints + +None. + +## Dependencies + +Shardingsphere middleware \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/tool-chain/mogdb-client-tool-datastudio.md b/product/en/docs-mogdb/v3.1/characteristic-description/tool-chain/mogdb-client-tool-datastudio.md new file mode 100644 index 0000000000000000000000000000000000000000..612c7742bb38783a8c3a112d31fc0220435aa2cc --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/tool-chain/mogdb-client-tool-datastudio.md @@ -0,0 +1,57 @@ +--- +title: MogDB Client Tool DataStudio +summary: MogDB Client Tool DataStudio +author: Guo Huan +date: 2022-05-10 +--- + +# MogDB Client Tool DataStudio + +## Availability + +You can obtain the installation package of DataStudio from the openGauss official website or obtain the source code from Gitee and compile by yourself. + +## Introduction + +DataStudio is a universal and integrated development environment for developers and database administrators. It simplifies the development and management of openGauss/MogDB databases. + +DataStudio allows database developers to: + +- Manage and create database objects. +- Run SQL statements or SQL scripts. +- Edit and run PL/SQL statements. +- Import and export table data. + +## Benefits + +As a client software used with openGauss/MogDB databases (currently, only openGauss/MogDB is supported), DataStudio helps database developers easily develop applications and provides key database features on the GUI. The tool makes database inclusive and improves user loyalty. + +## Description + +Start DataStudio on Windows and you can remotely connect to openGauss/MogDB and perform the following operations: + +- Connect to the openGauss/MogDB database over an integrated GUI-based development environment. +- Efficiently develop SQL. +- Manage or create database objects (databases, schemas, functions, stored procedures, tables, sequences, columns, indexes, constraints, views, users, roles, and tablespaces). +- Run SQL statements or SQL scripts. +- Create and execute a stored procedure. +- Add, delete, modify, and query table data. +- Import and export table data. +- Display and export DDL data. +- Import and export connection information. +- Format SQL statements. +- View SQL execution history. +- Display the execution plan and ER diagram. + +## Enhancements + +None. + +## Constraints + +- Supports Windows OS. +- Supports JDK 11 or later versions. + +## Dependencies + +Currently, only openGauss/MogDB is supported. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/characteristic-description/tool-chain/mysql-to-mogdb-migration-tool-chameleon.md b/product/en/docs-mogdb/v3.1/characteristic-description/tool-chain/mysql-to-mogdb-migration-tool-chameleon.md new file mode 100644 index 0000000000000000000000000000000000000000..e49f850515c9d9b97af9260a254ddb63386ff618 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/characteristic-description/tool-chain/mysql-to-mogdb-migration-tool-chameleon.md @@ -0,0 +1,37 @@ +--- +title: MySQL to MogDB Migration Tool Chameleon +summary: MySQL to MogDB Migration Tool Chameleon +author: Guo Huan +date: 2022-05-10 +--- + +# MySQL to MogDB Migration Tool Chameleon + +## Availability + +This feature is available since MogDB 3.0.0. + +## Introduction + +Chameleon is Python-based. It supports real-time data replication from MySQL to MogDB. The tool can replicate initial full data and incremental data in real time to migrate them from MySQL to MogDB. It supports migration of various data types in MySQL, and maintains the precision of floating-point data, including decimal, dec, numeric, float, float4, float8, real, double, double precision and fixed after migration. + +## Benefits + +You can use Chameleon to migrate data from the MySQL database to the MogDB database. + +## Description + +Chameleon provides full and incremental data replication functions to migrate data from the MySQL database to the MogDB database. Chameleon stores the mappings between MySQL and MogDB data types and supports migration of various data types in MySQL. Specifically, floating-point data in MySQL, including decimal, dec, numeric, float, float4, float8, real, double, double precision and fixed, will be converted to the numeric[p, s] type in MogDB if the precision is specified or contained by default. If the precision is not specified, the data will be converted to the numeric data type in MogDB. In this way, the data precision remains unchanged after offline and online migration. + +## Enhancements + +None. + +## Constraints + +- MySQL 5.7 is supported. +- If incompatibility still exists after data type mapping, table data migration fails, but the subsequent offline data migration process does not terminate. + +## Dependencies + +None. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/1-core-dump-occurs-due-to-full-disk-space.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/1-core-dump-occurs-due-to-full-disk-space.md new file mode 100644 index 0000000000000000000000000000000000000000..abd30c482ce2a2bdca8c4fa5d27d1c13ca2b654f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/1-core-dump-occurs-due-to-full-disk-space.md @@ -0,0 +1,22 @@ +--- +title: Core Dump Occurs due to Full Disk Space +summary: Core Dump Occurs due to Full Disk Space +author: Guo Huan +date: 2021-05-24 +--- + +# Core Dump Occurs due to Full Disk Space + +## Symptom + +When TPC-C is running, the disk space is full during injection. As a result, a core dump occurs on the MogDB process, as shown in the following figure. + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/core-dump-occurs-due-to-full-disk-space.png) + +## Cause Analysis + +When the disk is full, Xlog logs cannot be written. The program exits through the panic log. + +## Procedure + +Externally monitor the disk usage and periodically clean up the disk. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/2-core-dump-occurs-due-to-incorrect-settings-of-guc-parameter-log-directory.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/2-core-dump-occurs-due-to-incorrect-settings-of-guc-parameter-log-directory.md new file mode 100644 index 0000000000000000000000000000000000000000..e27a2ea52a6019e510b68e0e06c2d203202406ad --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/2-core-dump-occurs-due-to-incorrect-settings-of-guc-parameter-log-directory.md @@ -0,0 +1,20 @@ +--- +title: Core Dump Occurs Due to Incorrect Settings of GUC Parameter log_directory +summary: Core Dump Occurs Due to Incorrect Settings of GUC Parameter log_directory +author: Guo Huan +date: 2021-05-24 +--- + +# Core Dump Occurs Due to Incorrect Settings of GUC Parameter log_directory + +## Symptom + +After the database process is started, a core dump occurs and no log is recorded. + +## Cause Analysis + +The directory specified by GUC parameter **log_directory** cannot be read or you do not have permissions to access this directory. As a result, the verification fails during the database startup, and the program exits through the panic log. + +## Procedure + +Set **log_directory** to a valid directory. For details, see [log_directory](../../../reference-guide/guc-parameters/10-error-reporting-and-logging/1-logging-destination.md#log_directory). diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/3-core-dump-occurs-when-removeipc-is-enabled.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/3-core-dump-occurs-when-removeipc-is-enabled.md new file mode 100644 index 0000000000000000000000000000000000000000..033ec2ad2b35b5419075d440e5b19add3f7056e0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/3-core-dump-occurs-when-removeipc-is-enabled.md @@ -0,0 +1,24 @@ +--- +title: Core Dump Occurs when RemoveIPC Is Enabled +summary: Core Dump Occurs when RemoveIPC Is Enabled +author: Guo Huan +date: 2021-05-24 +--- + +# Core Dump Occurs when RemoveIPC Is Enabled + +## Symptom + +The **RemoveIPC** parameter in the OS configuration is set to **yes**. The database breaks down during running, and the following log information is displayed: + +``` +FATAL: semctl(1463124609, 3, SETVAL, 0) failed: Invalid argument +``` + +## Cause Analysis + +If **RemoveIPC** is set to **yes**, the OS deletes the IPC resources (shared memory and semaphore) when the corresponding user exits. As a result, the IPC resources used by the MogDB server are cleared, causing the database to break down. + +## Procedure + +Set **RemoveIPC** to **no**. For details, see [Operating System Configuration](../../../installation-guide/installation-preparation/os-configuration.md) in the *Installation Guide*. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/4-core-dump-occurs-after-installation-on-x86.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/4-core-dump-occurs-after-installation-on-x86.md new file mode 100644 index 0000000000000000000000000000000000000000..0258da542e7bfc06585c3b580c02d35d6b3e2a8d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/1-core-fault-locating/4-core-dump-occurs-after-installation-on-x86.md @@ -0,0 +1,26 @@ +--- +title: Core Dump Occurs After Installation on x86 +summary: Core Dump Occurs After Installation on x86 +author: Guo Huan +date: 2021-12-09 +--- + +# Core Dump Occurs After Installation on x86 + +## Symptom + +The core dump occurs after the installation of MogDB on x86 architecture machine is completed, and the error in the following figure displays + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/core-dump-occurs-after-installation-on-x86-1.png) + +## Cause Analysis + +The x86 architecture does not include the rdtscp instruction set, and the deployment of MogDB fails. This problem is common in the case of virtualized installation of Linux server on local windows system, but the virtualization version is too low. + +## Procedure + +Run the `lscpu | grep rdtscp` command to see if the rdtscp instruction set is supported. + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/administrator-guide/core-dump-occurs-after-installation-on-x86-2.png) + +Support for this instruction set can be enabled through the host's admin side settings. Set the cloud host CPU mode to **host-passthrough**, and then reboot. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/10-disk-space-usage-reaches-the-threshold.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/10-disk-space-usage-reaches-the-threshold.md new file mode 100644 index 0000000000000000000000000000000000000000..0e7ce312879f0bd51cc8211964064716025083a8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/10-disk-space-usage-reaches-the-threshold.md @@ -0,0 +1,58 @@ +--- +title: Disk Space Usage Reaches the Threshold and the Database Becomes Read-only +summary: Disk Space Usage Reaches the Threshold and the Database Becomes Read-only +author: Guo Huan +date: 2021-05-24 +--- + +# Disk Space Usage Reaches the Threshold and the Database Becomes Read-only + +## Symptom + +The following error is reported when a non-read-only SQL statement is executed: + +``` +ERROR: cannot execute %s in a read-only transaction. +``` + +An error is reported when some non-read-only SQL statements (such as insert, update, create table as, create index, alter table, and copy from) are executed: + +``` +canceling statement due to default_transaction_read_only is on. +``` + +## Cause Analysis + +After the disk space usage reaches the threshold, the database enters the read-only mode. In this mode, only read-only statements can be executed. + +## Procedure + +1. Use either of the following methods to connect to the database in maintenance mode: + + - Method 1 + + ``` + gsql -d mogdb -p 8000 -r -m + ``` + + - Method 2 + + ``` + gsql -d mogdb -p 8000 -r + ``` + + After the connection is successful, run the following command: + + ``` + set xc_maintenance_mode=on; + ``` + +2. Run the **DROP** or **TRUNCATE** statement to delete user tables that are no longer used until the disk space usage falls below the threshold. + + Deleting user tables can only temporarily relieve the insufficient disk space. To permanently solve the problem, expand the disk space. + +3. Disable the read-only mode of the database as user **omm**. + + ``` + gs_guc reload -D /mogdb/data/dbnode -c "default_transaction_read_only=off" + ``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/11-slow-response-to-a-query-statement.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/11-slow-response-to-a-query-statement.md new file mode 100644 index 0000000000000000000000000000000000000000..ae47939076b67d01f3829ad93691cc4bc3d87820 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/11-slow-response-to-a-query-statement.md @@ -0,0 +1,54 @@ +--- +title: Slow Response to a Query Statement +summary: Slow Response to a Query Statement +author: Guo Huan +date: 2021-05-24 +--- + +# Slow Response to a Query Statement + +## Symptom + +After a query statement has been executed, no response is returned for a long time. + +## Cause Analysis + +- The query statement is complex and requires a long time for execution. +- The query statement is blocked. + +## Procedure + +1. Log in to the host as the OS user **omm**. + +2. Run the following command to connect to the database: + + ``` + gsql -d mogdb -p 8000 + ``` + + **mogdb** is the name of the database, and **8000** is the port number. + +3. Check for the query statements that are executed for a long time in the system. + + ```sql + SELECT EXTRACT(DAY FROM (current_timestamp - query_start)) * 24 * 60 + EXTRACT(HOUR FROM (current_timestamp - query_start)) * 60 + EXTRACT(MINUTE FROM (current_timestamp - query_start)) AS runtime, datname, usename, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY 1 desc; + + -- In B-compatible mode, run the following statement: + SELECT timestampdiff(minutes, query_start, current_timestamp) AS runtime, datname, usename, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY 1 desc; + ``` + + Query statements are returned, sorted by execution time length in descending order. The first record is the query statement that takes the long time for execution. + + Alternatively, you can use the [TIMESTAMPDIFF](../../reference-guide/functions-and-operators/8-date-and-time-processing-functions-and-operators.md) function to set **current_timestamp** and **query_start** to be greater than a threshold to identify query statements that are executed for a duration longer than this threshold. The first parameter of **timestampdiff** is the time difference unit. For example, execute the following statement to query the statements whose execution lasts more than 2 minutes: + + ```sql + SELECT query FROM pg_stat_activity WHERE (EXTRACT(DAY FROM (current_timestamp - query_start)) * 24 * 60 + EXTRACT(HOUR FROM (current_timestamp - query_start)) * 60 + EXTRACT(MINUTE FROM (current_timestamp - query_start))) > 2; + + -- In B-compatible mode, run the following statement: + SELECT query FROM pg_stat_activity WHERE timestampdiff(minutes, query_start, current_timestamp) > 2; + ``` + +4. Analyze the status of the query statements that were run for a long time. + + - If the query statement is normal, wait until the execution of the query statement is complete. + - If the query statement is blocked, rectify the fault by referring to [Analyzing Whether a Query Statement Is Blocked](14-analyzing-whether-a-query-statement-is-blocked.md). diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/12-analyzing-the-status-of-a-query-statement.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/12-analyzing-the-status-of-a-query-statement.md new file mode 100644 index 0000000000000000000000000000000000000000..08a8c1b897a07b5d2bdb0eb4650356f8df914a10 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/12-analyzing-the-status-of-a-query-statement.md @@ -0,0 +1,57 @@ +--- +title: Analyzing the Status of a Query Statement +summary: Analyzing the Status of a Query Statement +author: Guo Huan +date: 2021-05-24 +--- + +# Analyzing the Status of a Query Statement + +## Symptom + +Some query statements are executed for an excessively long time in the system. You need to analyze the status of the query statements. + +## Procedure + +1. Log in to the host as the OS user **omm**. + +2. Run the following command to connect to the database: + + ``` + gsql -d mogdb -p 8000 + ``` + + **mogdb** is the name of the database, and **8000** is the port number. + +3. Set the parameter **track_activities** to **on**. + + ``` + SET track_activities = on; + ``` + + The database collects the running information about active queries only if the parameter is set to **on**. + +4. View the running query statements. The **pg_stat_activity** view is used as an example here. + + ``` + SELECT datname, usename, state, query FROM pg_stat_activity; + datname | usename | state | query + ----------+---------+--------+------- + mogdb | omm | idle | + mogdb | omm | active | + (2 rows) + ``` + + If the **state** column is **idle**, the connection is idle and requires a user to enter a command. To identify only active query statements, run the following command: + + ``` + SELECT datname, usename, state, query FROM pg_stat_activity WHERE state != 'idle'; + ``` + +5. Analyze whether a query statement is in the active or blocked state. Run the following command to view a query statement in the block state: + + ``` + SELECT datname, usename, state, query FROM pg_stat_activity WHERE waiting = true; + ``` + + The query statement is displayed. It is requesting a lock resource that may be held by another session, and is waiting for the lock resource to be released by the session. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/13-forcibly-terminating-a-session.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/13-forcibly-terminating-a-session.md new file mode 100644 index 0000000000000000000000000000000000000000..6b04991aafec8b6a6c13c082500bde2b7abc7fee --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/13-forcibly-terminating-a-session.md @@ -0,0 +1,62 @@ +--- +title: Forcibly Terminating a Session +summary: Forcibly Terminating a Session +author: Guo Huan +date: 2021-05-24 +--- + +# Forcibly Terminating a Session + +## Symptom + +In some cases, the administrator must forcibly terminate abnormal sessions to keep the system healthy. + +## Procedure + +1. Log in to the host as the OS user **omm**. + +2. Run the following command to connect to the database: + + ``` + gsql -d mogdb -p 8000 + ``` + + **mogdb** is the name of the database, and **8000** is the port number. + +3. Find the thread ID of the faulty session from the current active session view. + + ``` + SELECT datid, pid, state, query FROM pg_stat_activity; + ``` + + A command output similar to the following is displayed, where the pid value indicates the thread ID of the session. + + ``` + datid | pid | state | query + -------+-----------------+--------+------ + 13205 | 139834762094352 | active | + 13205 | 139834759993104 | idle | + (2 rows) + ``` + +4. Terminate the session using its thread ID. + + ``` + SELECT pg_terminate_backend(139834762094352); + ``` + + If information similar to the following is displayed, the session is successfully terminated: + + ``` + pg_terminate_backend + --------------------- + t + (1 row) + ``` + + If a command output similar to the following is displayed, a user is attempting to terminate the session, and the session will be reconnected rather than being terminated. + + ``` + FATAL: terminating connection due to administrator command + FATAL: terminating connection due to administrator command The connection to the server was lost. Attempting reset: Succeeded. + ``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/14-analyzing-whether-a-query-statement-is-blocked.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/14-analyzing-whether-a-query-statement-is-blocked.md new file mode 100644 index 0000000000000000000000000000000000000000..226fa3ff060dd542d60b36b79093a86c2d24054f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/14-analyzing-whether-a-query-statement-is-blocked.md @@ -0,0 +1,56 @@ +--- +title: Analyzing Whether a Query Statement Is Blocked +summary: Analyzing Whether a Query Statement Is Blocked +author: Guo Huan +date: 2021-05-24 +--- + +# Analyzing Whether a Query Statement Is Blocked + +## Symptom + +During database running, query statements are blocked in some service scenarios. As a result, the query statements are executed for an excessively long time. + +## Cause Analysis + +A query statement uses a lock to protect the data objects that it wants to access. If the data objects have been locked by another session, the query statement will be blocked and wait for the session to complete operation and release the lock resource. The data objects requiring locks include tables and tuples. + +## Procedure + +1. Log in to the host as the OS user **omm**. + +2. Run the following command to connect to the database: + + ``` + gsql -d mogdb -p 8000 + ``` + + **mogdb** is the name of the database, and **8000** is the port number. + +3. Find the thread ID of the faulty session from the current active session view. + + ``` + SELECT w.query AS waiting_query, w.pid AS w_pid, w.usename AS w_user, l.query AS locking_query, l.pid AS l_pid, l.usename AS l_user, t.schemaname || '.' || t.relname AS tablename FROM pg_stat_activity w JOIN pg_locks l1 ON w.pid = l1.pid AND NOT l1.granted JOIN pg_locks l2 ON l1.relation = l2.relation AND l2.granted JOIN pg_stat_activity l ON l2.pid = l.pid JOIN pg_stat_user_tables t ON l1.relation = t.relid WHERE w.waiting = true; + ``` + +4. Terminate the session using its thread ID. + + ``` + SELECT pg_terminate_backend(139834762094352); + ``` + + If information similar to the following is displayed, the session is successfully terminated: + + ``` + pg_terminate_backend + --------------------- + t + (1 row) + ``` + + If a command output similar to the following is displayed, a user is attempting to terminate the session, and the session will be reconnected rather than being terminated. + + ``` + FATAL: terminating connection due to administrator command + FATAL: terminating connection due to administrator command The connection to the server was lost. Attempting reset: Succeeded. + ``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/15-low-query-efficiency.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/15-low-query-efficiency.md new file mode 100644 index 0000000000000000000000000000000000000000..479c9e35dc1ba78b9fe3a2238afa62cdb4a1c7e1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/15-low-query-efficiency.md @@ -0,0 +1,32 @@ +--- +title: Low Query Efficiency +summary: Low Query Efficiency +author: Guo Huan +date: 2021-05-24 +--- + +# Low Query Efficiency + +## Symptom + +A query task that used to take a few milliseconds to complete is now requiring several seconds, and that used to take several seconds is now requiring even half an hour. + +## Procedure + +Perform the following procedure to locate the cause. + +1. Run the **analyze** command to analyze the database. + + Run the **analyze** command to update statistics such as data sizes and attributes in all tables. You are advised to perform the operation with light job load. If the query efficiency is improved or restored after the command execution, the **autovacuum** process does not function well that requires further analysis. + +2. Check whether the query statement returns unnecessary information. + + For example, if a query statement is used to query all records in a table with the first 10 records being used, then it is quick to query a table with 50 records. However, if a table contains 50000 records, the query efficiency decreases. If an application requires only a part of data information but the query statement returns all information, add a LIMIT clause to the query statement to restrict the number of returned records. In this way, the database optimizer can optimize space and improve query efficiency. + +3. Check whether the query statement still has a low response even when it is solely executed. + + Run the query statement when there are no or only a few other query requests in the database, and observe the query efficiency. If the efficiency is high, the previous issue is possibly caused by a heavily loaded host in the database system or + +4. Check the same query statement repeatedly to check the query efficiency. + + One major cause of low query efficiency is that the required information is not cached in the memory or is replaced by other query requests due to insufficient memory resources. This can be verified by running the same query statement repeatedly and the query efficiency increases gradually. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/16-lock-wait-timeout-is-displayed.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/16-lock-wait-timeout-is-displayed.md new file mode 100644 index 0000000000000000000000000000000000000000..1944729224a8780c1a0ac7da777588ae5fc7d96a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/16-lock-wait-timeout-is-displayed.md @@ -0,0 +1,25 @@ +--- +title: Lock wait timeout Is Displayed When a User Executes an SQL Statement +summary: Lock wait timeout Is Displayed When a User Executes an SQL Statement +author: Guo Huan +date: 2021-05-24 +--- + +# "Lock wait timeout" Is Displayed When a User Executes an SQL Statement + +## Symptom + +"Lock wait timeout" is displayed when a user executes an SQL statement. + +``` +ERROR: Lock wait timeout: thread 140533638080272 waiting for ShareLock on relation 16409 of database 13218 after 1200000.122 ms ERROR: Lock wait timeout: thread 140533638080272 waiting for AccessExclusiveLock on relation 16409 of database 13218 after 1200000.193 ms +``` + +## Cause Analysis + +Lock waiting times out in the database. + +## Procedure + +- After detecting such errors, the database automatically retries the SQL statements. The number of retries is controlled by **max_query_retry_times**. +- To analyze the cause of the lock wait timeout, find the SQL statements that time out in the **pg_locks** and **pg_stat_activity** system catalogs. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/17-table-size-does-not-change.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/17-table-size-does-not-change.md new file mode 100644 index 0000000000000000000000000000000000000000..ce32f744d1816aee73e45f868e0120127061d697 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/17-table-size-does-not-change.md @@ -0,0 +1,39 @@ +--- +title: Table Size Does not Change After VACUUM FULL Is Executed on the Table +summary: Table Size Does not Change After VACUUM FULL Is Executed on the Table +author: Guo Huan +date: 2021-05-24 +--- + +# Table Size Does not Change After VACUUM FULL Is Executed on the Table + +## Symptom + +A user runs the **VACUUM FULL** command to clear a table, the table size does not change. + +## Cause Analysis + +Assume the table is named **table_name**. Possible causes are as follows: + +- No data has been deleted from the **table_name** table. Therefore, the execution of **VACUUM FULL table_name** does not cause the table size to change. +- Concurrent transactions exist during the execution of **VACUUM FULL table_name**. As a result, recently deleted data may be skipped when clearing the table. + +## Procedure + +For the second possible cause, use either of the following methods: + +- Wait until all concurrent transactions are complete, and run the **VACUUM FULL table_name** command again. + +- If the table size still does not change, ensure no service operations are performed on the table, and then execute the following SQL statements to query the active transaction list status: + + ``` + select txid_current(); + ``` + + The current XID is obtained. Then, run the following command to check the active transaction list: + + ``` + select txid_current_snapshot(); + ``` + + If any XID in the active transaction list is smaller than the current transaction XID, stop the database and then start it. Run **VACUUM FULL** to clear the table again. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/18-an-error-is-reported-when-the-table-partition-is-modified.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/18-an-error-is-reported-when-the-table-partition-is-modified.md new file mode 100644 index 0000000000000000000000000000000000000000..c5710586803c4dbb5d8c1ff210fb0c31e7b1c3ae --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/18-an-error-is-reported-when-the-table-partition-is-modified.md @@ -0,0 +1,46 @@ +--- +title: An Error Is Reported When the Table Partition Is Modified +summary: An Error Is Reported When the Table Partition Is Modified +author: Guo Huan +date: 2021-05-24 +--- + +# An Error Is Reported When the Table Partition Is Modified + +## Symptom + +When **ALTER TABLE PARTITION** is performed, the following error message is displayed: + +``` +ERROR:start value of partition "XX" NOT EQUAL up-boundary of last partition. +``` + +## Cause Analysis + +If the **ALTER TABLE PARTITION** statement involves both the DROP PARTITION operation and the ADD PARTITION operation, MogDB always performs the DROP PARTITION operation before the ADD PARTITION operation regardless of their orders. However, performing DROP PARTITION before ADD PARTITION causes a partition gap. As a result, an error is reported. + +## Procedure + +To prevent partition gaps, set **END** in DROP PARTITION to the value of **START** in ADD PARTITION. The following is an example: + +``` +-- Create a partitioned table partitiontest. +mogdb=# CREATE TABLE partitiontest +( +c_int integer, +c_time TIMESTAMP WITHOUT TIME ZONE +) +PARTITION BY range (c_int) +( +partition p1 start(100)end(108), +partition p2 start(108)end(120) +); +-- An error is reported when the following statements are used: +mogdb=# ALTER TABLE partitiontest ADD PARTITION p3 start(120)end(130), DROP PARTITION p2; +ERROR: start value of partition "p3" NOT EQUAL up-boundary of last partition. +mogdb=# ALTER TABLE partitiontest DROP PARTITION p2,ADD PARTITION p3 start(120)end(130); +ERROR: start value of partition "p3" NOT EQUAL up-boundary of last partition. +-- Change them as follows: +mogdb=# ALTER TABLE partitiontest ADD PARTITION p3 start(108)end(130), DROP PARTITION p2; +mogdb=# ALTER TABLE partitiontest DROP PARTITION p2,ADD PARTITION p3 start(108)end(130); +``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/19-different-data-is-displayed.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/19-different-data-is-displayed.md new file mode 100644 index 0000000000000000000000000000000000000000..2a916055ec0340fb50fb590742c8e98e5b69eafb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/19-different-data-is-displayed.md @@ -0,0 +1,32 @@ +--- +title: Different Data Is Displayed for the Same Table Queried By Multiple Users +summary: Different Data Is Displayed for the Same Table Queried By Multiple Users +author: Guo Huan +date: 2021-05-24 +--- + +# Different Data Is Displayed for the Same Table Queried By Multiple Users + +## Symptom + +Two users log in to the same database human_resource and run the following statement separately to query the areas table, but obtain different results. + +``` +select count(*) from areas; +``` + +## Cause Analysis + +1. Check whether tables with same names are really the same table. In a relational database, a table is identified by three elements: **database**, **schema**, and **table**. In this issue, **database** is **human_resource** and **table** is **areas**. +2. Check whether schemas of tables with the same name are consistent. Log in as users **omm** and **user01** separately. It is found that **search_path** is **public** and **$user**, respectively. As **omm** is the cluster administrator, a schema having the same name as user **omm** will not be created by default. That is, all tables will be created in **public** if no schema is specified. However, when a common user, such as **user01**, is created, the same-name schema (**user01**) is created by default. That is, all tables are created in **user01** if the schema is not specified. +3. If different users access different data in the same table, check whether objects in the table have different access policies for different users. + +## Procedure + +- For the query of tables with the same name in different schemas, add the schema reference to the queried table. The format is as follows: + + ``` + schema.table + ``` + +- If different access policies result in different query results of the same table, you can query the **pg_rlspolicy** system catalog to determine the specific access rules. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/2-when-the-tpcc-is-running.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/2-when-the-tpcc-is-running.md new file mode 100644 index 0000000000000000000000000000000000000000..bd1a3f9de1fcf92473c9d87a263152cf3f727fe3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/2-when-the-tpcc-is-running.md @@ -0,0 +1,20 @@ +--- +title: When the TPC-C is running and a disk to be injected is full, the TPC-C stops responding. +summary: When the TPC-C is running and a disk to be injected is full, the TPC-C stops responding. +author: Guo Huan +date: 2021-05-24 +--- + +# When the TPC-C is running and a disk to be injected is full, the TPC-C stops responding + +## Symptom + +When the TPC-C is running and a disk to be injected is full, the TPC-C stops responding. After the fault is rectified, the TPC-C automatically continues to run. + +## Cause Analysis + +When the disk where the performance log (**gs_profile**) is located is full, the database cannot write data and enters the infinite waiting state. As a result, the TPC-C stops responding. After the disk space insufficiency fault is rectified, performance logs can be properly written, and the TPC-C is restored. + +## Procedure + +Externally monitor the disk usage and periodically clean up the disk. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/20-when-a-user-specifies-only-an-index-name.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/20-when-a-user-specifies-only-an-index-name.md new file mode 100644 index 0000000000000000000000000000000000000000..58f381dd29aaef3d3b2e076f04919f108f66d6ec --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/20-when-a-user-specifies-only-an-index-name.md @@ -0,0 +1,47 @@ +--- +title: When a User Specifies Only an Index Name to Modify the Index, A Message Indicating That the Index Does Not Exist Is Displayed +summary: When a User Specifies Only an Index Name to Modify the Index, A Message Indicating That the Index Does Not Exist Is Displayed +author: Guo Huan +date: 2021-05-24 +--- + +# When a User Specifies Only an Index Name to Modify the Index, A Message Indicating That the Index Does Not Exist Is Displayed + +## Symptom + +When a User Specifies Only an Index Name to Modify the Index, A Message Indicating That the Index Does Not Exist Is Displayed The following provides an example: + +``` +-- Create a partitioned table index HR_staffS_p1_index1, without specifying index partitions. +CREATE INDEX HR_staffS_p1_index1 ON HR.staffS_p1 (staff_ID) LOCAL; +-- Create a partitioned table index HR_staffS_p1_index2, with index partitions specified. +CREATE INDEX HR_staffS_p1_index2 ON HR.staffS_p1 (staff_ID) LOCAL +( +PARTITION staff_ID1_index, +PARTITION staff_ID2_index TABLESPACE example3, +PARTITION staff_ID3_index TABLESPACE example4 +) TABLESPACE example; +-- Change the tablespace of index partition staff_ID2_index to example1. A message is displayed, indicating that the index does not exist. +ALTER INDEX HR_staffS_p1_index2 MOVE PARTITION staff_ID2_index TABLESPACE example1; +``` + +## Cause Analysis + +The possible reason is that the user is in the public schema instead of the hr schema. + +``` +-- Run the following command to validate the inference. It is found that the calling is successful. +ALTER INDEX hr.HR_staffS_p1_index2 MOVE PARTITION staff_ID2_index TABLESPACE example1; +-- Change the schema of the current session to hr. +ALTER SESSION SET CURRENT_SCHEMA TO hr; +-- Run the following command to modify the index: +ALTER INDEX HR_staffS_p1_index2 MOVE PARTITION staff_ID2_index TABLESPACE example1; +``` + +## Procedure + +Add a schema reference to a table, index, or view. The format is as follows: + +``` +schema.table +``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/21-reindexing-fails.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/21-reindexing-fails.md new file mode 100644 index 0000000000000000000000000000000000000000..d8920008ea20e994d71b262654841aa64d36719f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/21-reindexing-fails.md @@ -0,0 +1,29 @@ +--- +title: Reindexing Fails +summary: Reindexing Fails +author: Guo Huan +date: 2021-05-24 +--- + +# Reindexing Fails + +## Symptom + +When an index of the desc table is damaged, a series of operations cannot be performed. The error information may be as follows: + +``` +index \"%s\" contains corrupted page at block + %u" ,RelationGetRelationName(rel),BufferGetBlockNumber(buf), please reindex it. +``` + +## Cause Analysis + +In actual operations, indexes may break down due to software or hardware faults. For example, if disk space is insufficient or pages are damaged after indexes are split, the indexes may be damaged. + +## Procedure + +If the table is a column-store table named **pg_cudesc_xxxxx_index**, the desc index table is damaged. Find the OID and table corresponding to the primary table based on the desc index table name, and run the following statement to recreate the cudesc index. + +``` +REINDEX INTERNAL TABLE name; +``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/22-an-error-occurs-during-integer-conversion.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/22-an-error-occurs-during-integer-conversion.md new file mode 100644 index 0000000000000000000000000000000000000000..e353708426e46193c8907c0aee48a18e075beac9 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/22-an-error-occurs-during-integer-conversion.md @@ -0,0 +1,24 @@ +--- +title: An Error Occurs During Integer Conversion +summary: An Error Occurs During Integer Conversion +author: Guo Huan +date: 2021-05-24 +--- + +# An Error Occurs During Integer Conversion + +## Symptom + +The following error is reported during integer conversion: + +``` +Invalid input syntax for integer: "13." +``` + +## Cause Analysis + +Some data types cannot be converted to the target data type. + +## Procedure + +Gradually narrow down the range of SQL statements to determine the data types that cannot be converted. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/23-too-many-clients-already.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/23-too-many-clients-already.md new file mode 100644 index 0000000000000000000000000000000000000000..59e5b1d1fdd39c625a12458087e634f1266b8ef6 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/23-too-many-clients-already.md @@ -0,0 +1,50 @@ +--- +title: too many clients already Is Reported or Threads Failed To Be Created in High Concurrency Scenarios +summary: too many clients already Is Reported or Threads Failed To Be Created in High Concurrency Scenarios +author: Guo Huan +date: 2021-05-24 +--- + +# "too many clients already" Is Reported or Threads Failed To Be Created in High Concurrency Scenarios + +## Symptom + +When a large number of SQL statements are concurrently executed, the error message "sorry, too many clients already" is displayed or an error is reported, indicating that threads cannot be created or processes cannot be forked. + +## Cause Analysis + +These errors are caused by insufficient OS threads. Check **ulimit -u** in the OS. If the value is too small (for example, less than 32768), the errors are caused by the OS limitation. + +## Procedure + +Run **ulimit -u** to obtain the value of **max user processes** in the OS. + +``` +[root@MogDB36 mnt]# ulimit -u +unlimited +``` + +Use the following formula to calculate the minimum value: + +``` +value=max (32768, number of instances x 8192) +``` + +The number of instances refers to the total number of instances on the node. + +To set the minimum value, add the following two lines to the **/etc/security/limits.conf** file: + +``` +* hard nproc [value] +* soft nproc [value] +``` + +The file to be modified varies based on the OS. For versions later than CentOS6, modify the **/etc/security/limits.d/90-nofile.conf** file in the same way. + +Alternatively, you can run the following command to change the value. However, the change becomes invalid upon OS restart. To solve this problem, you can add **ulimit -u [value]** to the global environment variable file **/etc/profile**. + +``` +ulimit -u [values] +``` + +In high concurrency mode, enable the thread pool to control thread resources in the database. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/24-b-tree-index-faults.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/24-b-tree-index-faults.md new file mode 100644 index 0000000000000000000000000000000000000000..c04a36d92e7b877ab9f758e8b740cc85e5f90ac1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/24-b-tree-index-faults.md @@ -0,0 +1,68 @@ +--- +title: B-tree Index Faults +summary: B-tree Index Faults +author: Guo Huan +date: 2021-05-24 +--- + +# B-tree Index Faults + +## Symptom + +The following error message is displayed, indicating that the index is lost occasionally. + +``` +ERROR: index 'xxxx_index' contains unexpected zero page +Or +ERROR: index 'pg_xxxx_index' contains unexpected zero page +Or +ERROR: compressed data is corrupt +``` + +## Cause Analysis + +This type of error is caused by the index fault. The possible causes are as follows: + +- The index is unavailable due to software bugs or hardware faults. +- The index contains many empty pages or almost empty pages. +- During concurrent DDL execution, the network is intermittently disconnected. +- The index failed to be created when indexes are concurrently created. +- A network fault occurs when a DDL or DML operation is performed. + +## Procedure + +Run the REINDEX command to rebuild the index. + +1. Log in to the host as the OS user **omm**. + +2. Run the following command to connect to the database: + + ``` + gsql -d mogdb -p 8000 -r + ``` + +3. Rebuild the index. + + - During DDL or DML operations, if index problems occur due to software or hardware faults, run the following command to rebuild the index: + + ``` + REINDEX TABLE tablename; + ``` + + - If the error message contains **xxxx_index**, the index of a user table is faulty. **xxxx** indicates the name of the user table. Run either of the following commands to rebuild the index: + + ``` + REINDEX INDEX indexname; + ``` + + Or + + ``` + REINDEX TABLE tablename; + ``` + + - If the error message contains **pg_xxxx_index**, the index of the system catalog is faulty. Run the following command to rebuild the index: + + ``` + REINDEX SYSTEM databasename; + ``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/3-standby-node-in-the-need-repair-state.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/3-standby-node-in-the-need-repair-state.md new file mode 100644 index 0000000000000000000000000000000000000000..9f1206cc63528f8acf3d647a7ce3b1de01632e63 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/3-standby-node-in-the-need-repair-state.md @@ -0,0 +1,20 @@ +--- +title: Standby Node in the Need Repair (WAL) State +summary: Standby Node in the Need Repair (WAL) State +author: Guo Huan +date: 2021-05-24 +--- + +# Standby Node in the **Need Repair (WAL)** State + +## Symptom + +The **Need Repair (WAL)** fault occurs on a standby node of the MogDB. + +## Cause Analysis + +The primary and standby DB instances are disconnected due to network faults or insufficient disk space. As a result, logs are not synchronized between the primary and standby DB instances, and the database cluster fails to start. + +## Procedure + +Run the **gs_ctl build -D** command to rebuild the faulty node. For details, see the [build parameter](../../reference-guide/tool-reference/tools-used-in-the-internal-system/4-gs_ctl.md#6) in the *MogDB Tool Reference*. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/4-insufficient-memory.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/4-insufficient-memory.md new file mode 100644 index 0000000000000000000000000000000000000000..1bdc8300c9e4d89c03ef3de875625d05f261ee43 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/4-insufficient-memory.md @@ -0,0 +1,20 @@ +--- +title: Insufficient Memory +summary: Insufficient Memory +author: Guo Huan +date: 2021-05-24 +--- + +# Insufficient Memory + +## Symptom + +The client or logs contain the error message **memory usage reach the max_dynamic_memory**. + +## Cause Analysis + +The possible cause is that the value of the GUC parameter **max_process_memory** is too small. This parameter limits the maximum memory that can be used by an MogDB instance. + +## Procedure + +Use the **gs_guc** tool to adjust the value of **max_process_memory**. Note that you need to restart the instance for the modification to take effect. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/5-service-startup-failure.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/5-service-startup-failure.md new file mode 100644 index 0000000000000000000000000000000000000000..ee94999f57c8d23d931e9cd8b3994580ce93337d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/5-service-startup-failure.md @@ -0,0 +1,88 @@ +--- +title: Service Startup Failure +summary: Service Startup Failure +author: Guo Huan +date: 2021-05-24 +--- + +# Service Startup Failure + +## Symptom + +The service startup failed. + +## Cause Analysis + +- Parameters are set to improper values, resulting in insufficient system resources in the database cluster, or parameter settings do not meet the internal restrictions in the cluster. +- The status of some DNs is abnormal. +- Permissions to modify directories are insufficient. For example, users do not have sufficient permissions for the **/tmp** directory or the data directory in the cluster. +- The configured port has been occupied. +- The system firewall is enabled. +- The trust relationship between servers of the database in the cluster is abnormal. + +## Procedure + +- Check whether the parameter configurations are improper or meet internal constraints. + + - Log in to the node that cannot be started. Check the run logs and check whether the resources are insufficient or whether the parameter configurations meet internal constraints. For example, if the message "Out of memory" or the following error information is displayed, the resources are insufficient, the startup fails, or the configuration parameters do not meet the internal constraints. + + ``` + FATAL: hot standby is not possible because max_connections = 10 is a lower setting than on the master server (its value was 100) + ``` + + - Check whether the GUC parameters are set to proper values. For example, check parameters, such as **shared_buffers**, **effective_cache_size**, and **bulk_write_ring_size** that consume much resources, or parameter **max_connections** that cannot be easily set to a value that is less than its last value. For details about how to view and set GUC parameters, see Configuring Running Parameters. + +- Check whether the status of some DNs is abnormal. Check the status of each primary and standby instances in the current cluster using **gs_om -t status -detail**. + + - If the status of all the instances on a host is abnormal, replace the host. + + - If the status of an instance is **Unknown**, **Pending**, or **Down**, log in to the node where the instance resides as a cluster user to view the instance log and identify the cause. For example: + + ``` + 2014-11-27 14:10:07.022 CST 140720185366288 FATAL: database "postgres" does not exist 2014-11-27 14:10:07.022 CST 140720185366288 DETAIL: The database subdirectory "base/ 13252" is missing. + ``` + + If the preceding information is displayed in a log, files stored in the data directory where the DN resides are damaged, and the instance cannot be queried. You cannot execute normal queries to this instance. + +- Check whether users have sufficient directory permissions. For example, users do not have sufficient permissions for the **/tmp** directory or the data directory in the cluster. + + - Determine the directory for which users have insufficient permissions. + - Run the **chmod** command to modify directory permissions as required. The database user must have read/write permissions for the **/tmp** directory. To modify permissions for data directories, refer to the settings for data directories with sufficient permissions. + +- Check whether the configured ports have been occupied. + + - Log in to the node that cannot be started and check whether the instance process exists. + + - If the instance process does not exist, view the instance log to check the exception reasons. For example: + + ``` + 2014-10-17 19:38:23.637 CST 139875904172320 LOG: could not bind IPv4 socket at the 0 time: Address already in use 2014-10-17 19:38:23.637 CST 139875904172320 HINT: Is another postmaster already running on port 40005? If not, wait a few seconds and retry. + ``` + + If the preceding information is displayed in a log, the TCP port on the DN has been occupied, and the instance cannot be started. + + ``` + 2015-06-10 10:01:50 CST 140329975478400 [SCTP MODE] WARNING: (sctp bind) bind(socket=9, [addr:0.0.0.0,port:1024]):Address already in use -- attempt 10/10 2015-06-10 10:01:50 CST 140329975478400 [SCTP MODE] ERROR: (sctp bind) Maximum bind() attempts. Die now... + ``` + + If the preceding information is displayed in a log, the SCTP port on the DN has been occupied, and the instance cannot be started. + +- Run **sysctl -a** to view the **net.ipv4.ip_local_port_range** parameter. If this port configured for this instance is within the range of the port number randomly occupied by the system, modify the value of **net.ipv4.ip_local_port_range**, ensuring that all the instance port numbers in the XML file are beyond this range. Check whether a port has been occupied: + + ``` + netstat -anop | grep Port number + ``` + + The following is an example: + + ``` + [root@MogDB36 ~]# netstat -anop | grep 15970 + tcp 0 0 127.0.0.1:15970 0.0.0.0:* LISTEN 3920251/mogdb off (0.00/0/0) + tcp6 0 0 ::1:15970 :::* LISTEN 3920251/mogdb off (0.00/0/0) + unix 2 [ ACC ] STREAM LISTENING 197399441 3920251/mogdb /tmp/.s.PGSQL.15970 + unix 3 [ ] STREAM CONNECTED 197461142 3920251/mogdb /tmp/.s.PGSQL.15970 + ``` + +- Check whether the system firewall is enabled. + +- Check whether the mutual trust relationship is abnormal. Reconfigure the mutual trust relationship between servers in the cluster. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/6-error-no-space-left-on-device-is-displayed.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/6-error-no-space-left-on-device-is-displayed.md new file mode 100644 index 0000000000000000000000000000000000000000..e87b496e6a84ebfce718b53fc29c53f8f1753a5a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/6-error-no-space-left-on-device-is-displayed.md @@ -0,0 +1,64 @@ +--- +title: Error:No space left on device Is Displayed +summary: Error:No space left on device Is Displayed +author: Guo Huan +date: 2021-05-24 +--- + +# "Error:No space left on device" Is Displayed + +## Symptom + +The following error message is displayed when the cluster is being used: + +``` +Error:No space left on device +``` + +## Cause Analysis + +The disk space is insufficient. + +## Procedure + +- Run the following command to check the disk usage. The **Avail** column indicates the available disk space, and the **Use%** column indicates the percentage of disk space that has been used. + + ``` + [root@openeuler123 mnt]# df -h + Filesystem Size Used Avail Use% Mounted on + devtmpfs 255G 0 255G 0% /dev + tmpfs 255G 35M 255G 1% /dev/shm + tmpfs 255G 57M 255G 1% /run + tmpfs 255G 0 255G 0% /sys/fs/cgroup + /dev/mapper/openeuler-root 196G 8.8G 178G 5% / + tmpfs 255G 1.0M 255G 1% /tmp + /dev/sda2 9.8G 144M 9.2G 2% /boot + /dev/sda1 10G 5.8M 10G 1% /boot/efi + ``` + + The demand for remaining disk space depends on the increase in service data. Suggestions: + + - Check the disk space usage status, ensuring that the remaining space is sufficient for the growth of disk space for over one year. + - If the disk space usage exceeds 60%, you must clear or expand the disk space. + +- Run the following command to check the size of the data directory. + + ``` + du --max-depth=1 -h /mnt/ + ``` + + The following information is displayed. The first column shows the sizes of directories or files, and the second column shows all the sub-directories or files under the **/mnt/** directory. + + ``` + [root@MogDB36 mnt]# du --max-depth=1 -h /mnt + 83G /mnt/data3 + 71G /mnt/data2 + 365G /mnt/data1 + 518G /mnt + ``` + +- Clean up the disk space. You are advised to periodically back up audit logs to other storage devices. The recommended log retention period is one month. **pg_log** stores database process run logs which help database administrators locate faults. You can delete error logs if you view them every day and handle errors in time. + +- Delete useless data. Back up data that is not used frequently or used for a certain period of time to storage media with lower costs, and clean the backup data to free up disk space. + +- If the disk space is still insufficient, expand the disk capacity. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/7-after-you-run-the-du-command.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/7-after-you-run-the-du-command.md new file mode 100644 index 0000000000000000000000000000000000000000..e2e8d4fabff6f1c8be852b6b410f681780b2f1b3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/7-after-you-run-the-du-command.md @@ -0,0 +1,32 @@ +--- +title: After You Run the du Command to Query Data File Size In the XFS File System, the Query Result Is Greater than the Actual File Size +summary: After You Run the du Command to Query Data File Size In the XFS File System, the Query Result Is Greater than the Actual File Size +author: Guo Huan +date: 2021-05-24 +--- + +# After You Run the du Command to Query Data File Size In the XFS File System, the Query Result Is Greater than the Actual File Size + +## Symptom + +After you run the **du** command to query data file size in the cluster, the query result is probably greater than the actual file size. + +``` + du -sh file +``` + +## Cause Analysis + +The XFS file system has a pre-assignment mechanism. The file size is determined by the **allocsize** parameter. The file size displayed by the **du** command includes the pre-assigned disk space. + +## Procedure + +- Select the default value (64 KB) for the XFS file system mount parameter allocsize to eliminate the problem. + +- Add the **-apparent-size** parameter when using the **du** command to query the actual file size. + + ``` + du -sh file --apparent-size + ``` + +- If the XFS file system reclaims the pre-assigned space of a file, the **du** command displays the actual file size. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/8-file-is-damaged-in-the-xfs-file-system.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/8-file-is-damaged-in-the-xfs-file-system.md new file mode 100644 index 0000000000000000000000000000000000000000..8b1f081fc8cfd54c85e9604e83e75f1aa420ec57 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/8-file-is-damaged-in-the-xfs-file-system.md @@ -0,0 +1,22 @@ +--- +title: File Is Damaged in the XFS File System +summary: File Is Damaged in the XFS File System +author: Guo Huan +date: 2021-05-24 +--- + +# File Is Damaged in the XFS File System + +## Symptom + +When a cluster is in use, error reports such as an input/output error or the structure needs cleaning generally do not occur in the XFS file system. + +## Cause Analysis + +The XFS file system is abnormal. + +## Procedure + +Try to mount or unmount the file system to check whether the problem can be solved. + +If the problem recurs, refer to the file system document, such as **xfs_repair**, and ask the system administrator to restore the file system. After the file system is repaired, run the **gs_ctl build** command to restore the damaged DNs. diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/9-primary-node-is-hung-in-demoting.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/9-primary-node-is-hung-in-demoting.md new file mode 100644 index 0000000000000000000000000000000000000000..36409ffe18838895b4fc9ae61c93f7622c57b37b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-cases/9-primary-node-is-hung-in-demoting.md @@ -0,0 +1,24 @@ +--- +title: Primary Node Is Hung in Demoting During a Switchover +summary: Primary Node Is Hung in Demoting During a Switchover +author: Guo Huan +date: 2021-05-24 +--- + +# Primary Node Is Hung in Demoting During a Switchover + +## Symptom + +In a cluster deployed with one primary and multiple standby DNs, if system resources are insufficient and a switchover occurs, a node is hung in demoting. + +## Cause Analysis + +If system resources are insufficient, the third-party management thread cannot be created. As a result, the managed sub-threads cannot exit and the primary node is hung in demoting. + +## Procedure + +Run the following command to stop the process of the primary node so that the standby node can be promoted to primary: Perform the following operations only in the preceding scenario. + +``` + kill -9 PID +``` diff --git a/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-methods.md b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-methods.md new file mode 100644 index 0000000000000000000000000000000000000000..e970ef0de77963343dcf34c7094280c698d4afd5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/common-faults-and-identification/common-fault-locating-methods.md @@ -0,0 +1,283 @@ +--- +title: Common Fault Locating Methods +summary: Common Fault Locating Methods +author: Guo Huan +date: 2021-05-24 +--- + +# Common Fault Locating Methods + +## **Locating OS Faults** + +If all instances on a node are abnormal, an OS fault may have occurred. + +Use one of the following methods to check whether any OS fault occurs: + +- Log in to the node using SSH or other remote login tools. If the login fails, run the **ping** command to check the network status. + + - If no response is returned, the server is down or being restarted, or its network connection is abnormal. + + The restart takes a long time (about 20 minutes) if the system crashes due to an OS kernel panic. Try to connect the host every 5 minutes. If the connection failed 20 minutes later, the server is down or the network connection is abnormal. In this case, contact the administrator to locate the fault on site. + + - If ping operations succeed but SSH login fails or commands cannot be executed, the server does not respond to external connections possibly because system resources are insufficient (for example, CPU or I/O resources are overloaded). In this case, try again. If the fault persists within 5 minutes, contact the administrator for further fault locating on site. + +- If login is successful but responses are slow, check the system running status, such as collecting system information as well as checking system version, hardware, parameter setting, and login users. The following are common commands for reference: + + - Use the **who** command to check online users. + + ``` + [root@MogDB36 ~]# who + root pts/0 2020-11-07 16:32 (10.70.223.238) + wyc pts/1 2020-11-10 09:54 (10.70.223.222) + root pts/2 2020-10-10 14:20 (10.70.223.238) + root pts/4 2020-10-09 10:14 (10.70.223.233) + root pts/5 2020-10-09 10:14 (10.70.223.233) + root pts/7 2020-10-31 17:03 (10.70.223.222) + root pts/9 2020-10-20 10:03 (10.70.220.85) + ``` + + - Use the **cat /etc/openEuler-release** and **uname -a** commands to check the system version and kernel information. + + ``` + [root@MogDB36 ~]# cat /etc/openEuler-release + openEuler release 20.03 (LTS) + [root@MogDB36 ~]# uname -a + Linux MogDB36 4.19.90-2003.4.0.0036.oe1.aarch64 #1 SMP Mon Mar 23 19:06:43 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux + [root@MogDB36 ~]# + ``` + + - Use the **sysctl -a** (run this command as user **root**) and **cat /etc/sysctl.conf** commands to obtain system parameter information. + + - Use the **cat /proc/cpuinfo** and **cat /proc/meminfo** commands to obtain CPU and memory information. + + ``` + [root@MogDB36 ~]# cat /proc/cpuinfo + processor : 0 + BogoMIPS : 200.00 + Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm + CPU implementer : 0x48 + CPU architecture: 8 + CPU variant : 0x1 + CPU part : 0xd01 + CPU revision : 0 + [root@MogDB36 ~]# cat /proc/meminfo + MemTotal: 534622272 kB + MemFree: 253322816 kB + MemAvailable: 369537344 kB + Buffers: 2429504 kB + Cached: 253063168 kB + SwapCached: 0 kB + Active: 88570624 kB + Inactive: 171801920 kB + Active(anon): 4914880 kB + Inactive(anon): 67011456 kB + Active(file): 83655744 kB + Inactive(file): 104790464 kB + ``` + + - Use the **top -H** command to query the CPU usage and check whether the CPU usage is high due to a specific process. If it is, use the **gdb** or **gstack** command to print the stack trace of this process and check whether this process is in an infinite loop. + + - Use the **iostat -x 1 3** command to query the I/O usage and check whether the I/O usage of the current disk is full. View the ongoing jobs to determine whether to handle the jobs with high I/O usage. + + - Use the **vmstat 1 3** command to query the memory usage in the current system and use the **top** command to obtain the processes with unexpectedly high memory usage. + + - View the OS logs (**/var/log/messages**) or dmseg information as user **root** to check whether errors have occurred in the OS. + + - The watchdog of an OS is a mechanism to ensure that the OS runs properly or exits from the infinite loop or deadlock state. If the watchdog times out (the default value is 60s), the system resets. + +## **Locating Network Faults** + +When the database runs normally, the network layer is transparent to upper-layer users. However, during the long-term operation of a database cluster, network exceptions or errors may occur. Common exceptions caused by network faults are as follows: + +- Network error reported due to database startup failure. +- Abnormal status, for example, all instances on a host are in the **UnKnown** state, or all services are switched over to standby instances. +- Network connection failure. +- Network disconnection reported during database sql query. +- Process response failures during database connection or query execution. When a network fault occurs in a database, locate and analyze the fault by using network-related Linux command tools (such as **ping**, **ifconfig**, **netstat**, and **lsof**) and process stack viewers (such as **gdb** and **gstack**) based on database log information. This section lists common network faults and describes how to analyze and locate faults. + +Common faults are as follows: + +- Network error reported due to a startup failure + + **Symptom 1**: The log contains the following error information. The port may be listened on by another process. + + ``` + LOG: could not bind socket at the 10 time, is another postmaster already running on port 54000? + ``` + + **Solution**: Run the following command to check the process that listens on the port. Replace the port number with the actual one. + + ``` + [root@MogDB36 ~]# netstat -anop | grep 15970 + tcp 0 0 127.0.0.1:15970 0.0.0.0:* LISTEN 3920251/mogdb off (0.00/0/0) + tcp6 0 0 ::1:15970 :::* LISTEN 3920251/mogdb off (0.00/0/0) + unix 2 [ ACC ] STREAM LISTENING 197399441 3920251/mogdb /tmp/.s.PGSQL.15970 + unix 3 [ ] STREAM CONNECTED 197461142 3920251/mogdb /tmp/.s.PGSQL.15970 + + ``` + + Forcibly stop the process that is occupying the port or change the listening port of the database based on the query result. + + **Symptom 2**: When the **gs_om -t status -detail** command is used to query status, the command output shows that the connection between the primary and standby nodes is not established. + + **Solution**: In openEuler, run the **systemctl status firewalld.service** command to check whether the firewall is enabled on this node. If it is enabled, run the **systemctl stop firewalld.service** command to disable it. + + ``` + [root@MogDB36 mnt]# systemctl status firewalld.service + ●firewalld.service - firewalld - dynamic firewall daemon + Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) + Active: inactive (dead) + Docs: man:firewalld(1) + ``` + + The command varies according to the operating system. You can run the corresponding command to view and modify the configuration. + +- The database is abnormal. + + **Symptom**: The following problems occur on a node: + + - All instances are in the **Unknown** state. + - All primary instances are switched to standby instances. + - Errors "Connection reset by peer" and "Connection timed out" are frequently displayed. + + **Solution** + + - If you cannot connect to the faulty server through SSH, run the **ping** command on other servers to send data packages to the faulty server. If the ping operation succeeds, connection fails because resources such as memory, CPUs, and disks, on the faulty server are used up. + + - Connect to the faulty server through through SSH and run the **/sbin/ifconfig eth ?** command every other second (replace the question mark (?) with the number indicating the position of the NIC). Check value changes of **dropped** and **errors**. If they increase rapidly, the NIC or NIC driver may be faulty. + + ``` + [root@MogDB36 ~]# ifconfig enp125s0f0 + enp125s0f0: flags=4163 mtu 1500 + inet 10.90.56.36 netmask 255.255.255.0 broadcast 10.90.56.255 + inet6 fe80::7be7:8038:f3dc:f916 prefixlen 64 scopeid 0x20 + ether 44:67:47:7d:e6:84 txqueuelen 1000 (Ethernet) + RX packets 129344246 bytes 228050833914 (212.3 GiB) + RX errors 0 dropped 647228 overruns 0 frame 0 + TX packets 96689431 bytes 97279775245 (90.5 GiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 + ``` + + - Check whether the following parameters are correctly configured: + + ``` + net.ipv4.tcp_retries1 = 3 + net.ipv4.tcp_retries2 = 15 + ``` + +- Network connection failure. + + **Symptom 1**: A node fails to connect to other nodes, and the "Connection refused" error is reported in the log. + + **Solution** + + - Check whether the port is incorrectly configured, resulting in that the port used for connection is not the listening port of the peer end. Check whether the port number recorded in the **postgresql.conf** configuration file of the faulty node is the same as the listening port number of the peer end. + - Check whether the peer listening port is normal (for example, by running the **netstat -anp** command). + - Check whether the peer process exists. + + **Symptom 2**: When SQL operations are performed on the database, the connection descriptor fails to be obtained. The following error information is displayed: + + ``` + WARNING: 29483313: incomplete message from client:4905,9 + WARNING: 29483313: failed to receive connDefs at the time:1. + ERROR: 29483313: failed to get pooled connections + ``` + + In logs, locate and view the log content before the preceding error messages, which are generated due to incorrect active and standby information. Error details are displayed as follows. + + ``` + FATAL: dn_6001_6002: can not accept connection in pending mode. + FATAL: dn_6001_6002: the database system is starting up + FATAL: dn_6009_6010: can not accept connection in standby mode. + ``` + + **Solution** + + - Run the **gs_om -t status -detail** command to query the status and check whether an primary/standby switchover has occurred. Reset the instance status. + - In addition, check whether a core dump or restart occurs on the node that fails to be connected. In the om log, check whether restart occurs. + +- Network disconnection reported during database sql query. + + **Symptom 1**: The query fails, and the following error information is displayed: + + ``` + ERROR: dn_6065_6066: Failed to read response from Datanodes. Detail: Connection reset by peer. Local: dn_6065_6066 Remote: dn_6023_6024 + ERROR: Failed to read response from Datanodes Detail: Remote close socket unexpectedly + ERROR: dn_6155_6156: dn_6151_6152: Failed to read vector response from Datanodes + ``` + + If the connection fails, the error information may be as follows: + + ``` + ERROR: Distribute Query unable to connect 10.145.120.79:14600 [Detail:stream connect connect() fail: Connection timed out + ERROR: Distribute Query unable to connect 10.144.192.214:12600 [Detail:receive accept response fail: Connection timed out + ``` + + **Solution** + + 1. Use **gs_check** to check whether the network configuration meets requirements. For network check, see "Tool Reference > Server Tools > [gs_check](../reference-guide/tool-reference/server-tools/1-gs_check.md)" in the *Reference Guide*. + 2. Check whether a process core dump, restart, or switchover occurs. + 3. If problems still exist, contact network technical engineers. + +## **Locating Disk Faults** + +Common disk faults include insufficient disk space, bad blocks of disks, and unmounted disks. Disk faults such as unmount of disks damage the file system. The cluster management mechanism identifies this kind of faults and stops the instance, and the instance status is **Unknown**. However, disk faults such as insufficient disk space do not damage the file system. The cluster management mechanism cannot identify this kind of faults and service processes exit abnormally when accessing a faulty disk. Failures cover database startup, checksum verification, page read and write operation, and page verification. + +- For faults that result in file system damages, the instance status is **Unknown** when you view the host status. Perform the following operations to locate the disk fault: + + - Check the logs. If the logs contain information similar to "data path disc writable test failed", the file system is damaged. + + - The possible cause of file system damage may be unmounted disks. Run the **ls -l** command and you can view that the disk directory permission is abnormal, as shown in the following: + + - Another possible cause is that the disk has bad blocks. In this case, the OS rejects read and write operations to protect the file system. You can use a bad block check tool, for example, **badblocks**, to check whether bad blocks exist. + + ``` + [root@openeuler123 mnt]# badblocks /dev/sdb1 -s -v + Checking blocks 0 to 2147482623 + Checking for bad blocks (read-only test): done + Pass completed, 0 bad blocks found. (0/0/0 errors) + ``` + +- For faults that do not damage the file system, the service process will report an exception and exit when it accesses the faulty disk. Perform the following operations to locate the disk fault: + + View logs. The log contains read and write errors, such as "No space left on device" and "invalid page header n block 122838 of relation base/16385/152715". Run the **df -h** command to check the disk space. If the disk usage is 100% as shown below, the read and write errors are caused by insufficient disk space: + + ``` + [root@openeuler123 mnt]# df -h + Filesystem Size Used Avail Use% Mounted on + devtmpfs 255G 0 255G 0% /dev + tmpfs 255G 35M 255G 1% /dev/shm + tmpfs 255G 57M 255G 1% /run + tmpfs 255G 0 255G 0% /sys/fs/cgroup + /dev/mapper/openeuler-root 196G 8.8G 178G 5% / + tmpfs 255G 1.0M 255G 1% /tmp + /dev/sda2 9.8G 144M 9.2G 2% /boot + /dev/sda1 10G 5.8M 10G 1% /boot/efi + /dev/mapper/openeuler-home 1.5T 69G 1.4T 5% /home + tmpfs 51G 0 51G 0% /run/user/0 + tmpfs 51G 0 51G 0% /run/user/1004 + /dev/sdb1 2.0T 169G 1.9T 9% /data + ``` + +## **Locating Database Faults** + +- Logs. Database logs record the operations (starting, running, and stopping) on servers. Database users can view logs to quickly locate fault causes and rectify the faults accordingly. + +- View. A database provides different views to display its internal status. When locating a fault, you can use: + + - **pg_stat_activity**: shows the status of each session on the current instance. + - **pg_thread_wait_status**: shows the wait events of each thread on the current instance. + - **pg_locks**: shows the status of locks on the current instance. + +- Core files. Abnormal termination of a database process will trigger a core dump. A core dump file helps locate faults and determine fault causes. Once a core dump occurs during process running, collect the core file immediately for further analyzing and locating the fault. + + - The OS performance is affected, especially when errors occur frequently. + + - The OS disk space will be occupied by core files. Therefore, after core files are discovered, locate and rectify the errors as soon as possible. The OS is delivered with a core dump mechanism. If this mechanism is enabled, core files are generated for each core dump, which has an impact on the OS performance and disk space. + + - Set the path for generating core files. Modify the **/proc/sys/kernel/core_pattern** file. + + ``` + [root@openeuler123 mnt]# cat /proc/sys/kernel/core_pattern + /data/jenkins/workspace/MogDBInstall/dbinstall/cluster/corefile/core-%e-%p-%t + ``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/1-1-stored-procedure.md b/product/en/docs-mogdb/v3.1/developer-guide/1-1-stored-procedure.md new file mode 100644 index 0000000000000000000000000000000000000000..3c3d400a7a62e95cd9847c2beb0ca7b2689bbed8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/1-1-stored-procedure.md @@ -0,0 +1,16 @@ +--- +title: Stored Procedure +summary: Stored Procedure +author: Guo Huan +date: 2021-03-04 +--- + +# Stored Procedure + +In MogDB, business rules and logics are saved as stored procedures. + +A stored procedure is a combination of SQL and PL/pgSQL. Stored procedures can move the code that executes business rules from applications to databases. Therefore, the code storage can be used by multiple programs at a time. + +For details about how to create and call a stored procedure, see [CREATE PROCEDURE](../reference-guide/sql-syntax/CREATE-PROCEDURE.md). + +The application methods for PL/pgSQL functions mentioned in [User-defined Functions](user-defined-functions.md) are similar to those for stored procedures. For details, please refer to [PL/pgSQL-SQL Procedural Language](../developer-guide/plpgsql/1-1-plpgsql-overview.md) section, unless otherwise specified, the contents apply to stored procedures and user-defined functions. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/1-introduction-to-autonomous-transaction.md b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/1-introduction-to-autonomous-transaction.md new file mode 100644 index 0000000000000000000000000000000000000000..604bb593648d89fd718f9a4728b7afbd1e85d924 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/1-introduction-to-autonomous-transaction.md @@ -0,0 +1,12 @@ +--- +title: Introduction +summary: Introduction +author: Zhang Cuiping +date: 2021-05-10 +--- + +# Introduction + +An autonomous transaction is an independent transaction that is started during the execution of a primary transaction. Committing and rolling back an autonomous transaction does not affect the data that has been committed by the primary transaction. In addition, an autonomous transaction is not affected by the primary transaction. + +Autonomous transactions are defined in stored procedures, functions, and anonymous blocks, and are declared using the **PRAGMA AUTONOMOUS_TRANSACTION** keyword. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/2-function-supporting-autonomous-transaction.md b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/2-function-supporting-autonomous-transaction.md new file mode 100644 index 0000000000000000000000000000000000000000..659d052d0557999b37c661e4ac1f6af6a1192dd8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/2-function-supporting-autonomous-transaction.md @@ -0,0 +1,43 @@ +--- +title: Function Supporting Autonomous Transaction +summary: Function Supporting Autonomous Transaction +author: Zhang Cuiping +date: 2021-05-10 +--- + +# Function Supporting Autonomous Transaction + +An autonomous transaction can be defined in a function. The identifier of an autonomous transaction is **PRAGMA AUTONOMOUS_TRANSACTION**. The syntax of an autonomous transaction is the same as that of creating a function. The following is an example. + +```sql +create table t4(a int, b int, c text); + +CREATE OR REPLACE function autonomous_32(a int ,b int ,c text) RETURN int AS +DECLARE + PRAGMA AUTONOMOUS_TRANSACTION; +BEGIN + insert into t4 values(a, b, c); + return 1; +END; +/ +CREATE OR REPLACE function autonomous_33(num1 int) RETURN int AS +DECLARE + num3 int := 220; + tmp int; + PRAGMA AUTONOMOUS_TRANSACTION; +BEGIN + num3 := num3/num1; + return num3; +EXCEPTION + WHEN division_by_zero THEN + select autonomous_32(num3, num1, sqlerrm) into tmp; + return 0; +END; +/ + +select autonomous_33(0); + +select * from t4; +``` + +In the preceding example, a function containing an autonomous transaction is finally executed before a transaction block to be rolled back, which directly illustrates a characteristic of the autonomous transaction, that is, rollback of the primary transaction does not affect content that has been committed by the autonomous transaction. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/3-stored-procedure-supporting-autonomous-transaction.md b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/3-stored-procedure-supporting-autonomous-transaction.md new file mode 100644 index 0000000000000000000000000000000000000000..65adcbc347df7a23f6afa890b621f08bce920aa2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/3-stored-procedure-supporting-autonomous-transaction.md @@ -0,0 +1,45 @@ +--- +title: Stored Procedure Supporting Autonomous Transaction +summary: Stored Procedure Supporting Autonomous Transaction +author: Zhang Cuiping +date: 2021-05-10 +--- + +# Stored Procedure Supporting Autonomous Transaction + +An autonomous transaction can be defined in a stored procedure. The identifier of an autonomous transaction is **PRAGMA AUTONOMOUS_TRANSACTION**. The syntax of an autonomous transaction is the same as that of creating a stored procedure. The following is an example. + +```sql +-- Create a table. +create table t2(a int, b int); +insert into t2 values(1,2); +select * from t2; + +-- Create a stored procedure that contains an autonomous transaction. +CREATE OR REPLACE PROCEDURE autonomous_4(a int, b int) AS +DECLARE + num3 int := a; + num4 int := b; + PRAGMA AUTONOMOUS_TRANSACTION; +BEGIN + insert into t2 values(num3, num4); + dbe_output.print_line('just use call.'); +END; +/ +-- Create a common stored procedure that invokes an autonomous transaction stored procedure. +CREATE OR REPLACE PROCEDURE autonomous_5(a int, b int) AS +DECLARE +BEGIN + dbe_output.print_line('just no use call.'); + insert into t2 values(666, 666); + autonomous_4(a,b); + rollback; +END; +/ +-- Invoke a common stored procedure. +select autonomous_5(11,22); +-- View the table result. +select * from t2 order by a; +``` + +In the preceding example, a stored procedure containing an autonomous transaction is finally executed in a transaction block to be rolled back, which directly illustrates a characteristic of the autonomous transaction, that is, rollback of the primary transaction does not affect content that has been committed by the autonomous transaction. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/4-restrictions.md b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/4-restrictions.md new file mode 100644 index 0000000000000000000000000000000000000000..e15ba978707d998d53f7ddfb0ce7c59797211bc7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/4-restrictions.md @@ -0,0 +1,73 @@ +--- +title: Restrictions +summary: Restrictions +author: Zhang Cuiping +date: 2021-05-10 +--- + +# Restrictions + +- A trigger function does not support autonomous transactions. + +- In the autonomous transaction block of a function or stored procedure, static SQL statements do not support variable transfer. + + ```sql + -- Autonomous transactions do not support the execution of the following functions. The SQL statement contains the variable i. + CREATE OR REPLACE FUNCTION autonomous_easy_2(i int) RETURNS integer + LANGUAGE plpgsql + AS $$ + DECLARE + PRAGMA AUTONOMOUS_TRANSACTION; + BEGIN + START TRANSACTION; + INSERT INTO test1 VALUES (i, 'test'); + COMMIT; + RETURN 42; + END; + $$; + -- To use the parameter transfer, use the dynamic statement EXECUTE to replace variables. The following is an example: + CREATE OR REPLACE FUNCTION autonomous_easy(i int) RETURNS integer + LANGUAGE plpgsql + AS $$ + DECLARE + PRAGMA AUTONOMOUS_TRANSACTION; + BEGIN + START TRANSACTION; + EXECUTE 'INSERT INTO test1 VALUES (' || i::integer || ', ''test'')'; + COMMIT; + RETURN 42; + END; + $$; + ``` + +- Autonomous transactions do not support nesting. + + > **NOTICE:** In a function that contains an autonomous transaction, it is not allowed to explicitly execute another function or stored procedure that contains an autonomous transaction through **PERFORM**, **SELECT**, or **CALL**. However, another function or stored procedure that contains an autonomous transaction can be explicitly called in the last **RETURN**. + +- A function containing an autonomous transaction does not support the return value of parameter transfer. + + ```sql + -- In the following example, the return value ret is not transferred and only null is returned. + create or replace function at_test2(i int) returns text + LANGUAGE plpgsql + as $$ + declare + ret text; + pragma autonomous_transaction; + begin + START TRANSACTION; + insert into at_tb2 values(1, 'before s1'); + if i > 10 then + rollback; + else + commit; + end if; + select val into ret from at_tb2 where id=1; + return ret; + end; + $$; + ``` + +- A stored procedure or function that contains an autonomous transaction does not support exception handling. + +- A trigger function does not support autonomous transactions. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/anonymous-block-supporting-autonomous-transaction.md b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/anonymous-block-supporting-autonomous-transaction.md new file mode 100644 index 0000000000000000000000000000000000000000..483a9b1af82abb9e85b1310e9f05434aba1d5fb2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/autonomous-transaction/anonymous-block-supporting-autonomous-transaction.md @@ -0,0 +1,36 @@ +--- +title: Anonymous Block Supporting Autonomous Transaction +summary: Anonymous Block Supporting Autonomous Transaction +author: Guo Huan +date: 2021-10-15 +--- + +# Anonymous Block Supporting Autonomous Transaction + +An autonomous transaction can be defined in an anonymous block. The identifier of an autonomous transaction is **PRAGMA AUTONOMOUS_TRANSACTION**. The syntax of an autonomous transaction is the same as that of creating an anonymous block. The following is an example. + +```sql +MogDB=# create table t1(a int ,b text); +CREATE TABLE + +START TRANSACTION; +DECLARE + PRAGMA AUTONOMOUS_TRANSACTION; +BEGIN + + insert into t1 values(1,'you are so cute,will commit!'); +END; +/ +MogDB=# insert into t1 values(1,'you will rollback!'); +INSERT 0 1 +MogDB=# rollback; +ROLLBACK + +MogDB=# select * from t1; +a | b +---+------------------------------ +1 | you are so cute,will commit! +(1 row) +``` + +In the preceding example, an anonymous block containing an autonomous transaction is finally executed before a transaction block to be rolled back, which directly illustrates a characteristic of the autonomous transaction, that is, rollback of the primary transaction does not affect content that has been committed by the autonomous transaction. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/1-development-specifications.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/1-development-specifications.md new file mode 100644 index 0000000000000000000000000000000000000000..f10d7ebf4d54d87124c8d11b450c53461fc2e2c3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/1-development-specifications.md @@ -0,0 +1,737 @@ +--- +title: Development Specifications +summary: Development Specifications +author: Guo Huan +date: 2022-04-20 +--- + +# Development Specifications + +If the connection pool mechanism is used during application development, comply with the following specifications: + +- If GUC parameters are set in the connection, run **SET SESSION AUTHORIZATION DEFAULT;RESET ALL;** to clear the connection status before you return the connection to the connection pool. +- If a temporary table is used, delete the temporary table before you return the connection to the connection pool. + +If you do not do so, the connection in the connection pool will be stateful, which affects subsequent operations on the connection pool. + +Compatibility: + +- The new driver is forward compatible with the database. To use the new features added to the driver and database, you must upgrade the database. +- Setting **behavior_compat_options** to **'proc_outparam_override'** is applicable only in A-compatible mode. + +## Overview + +### Introduction + +Although ISO has issued SQL-92, SQL:1999, SQL:2006, and other standards for SQL, due to the characteristics of different databases, the same functions are not the same in the implementation of their products, which also makes the relevant grammatical rules different. Therefore, when formulating specific development specifications, it is necessary to write corresponding specifications for different databases. + +This specification emphasizes practicability and operability. According to the common problems and mistakes easily made by developers in the coding process, detailed and clear specifications and constraints are carried out on all aspects of code writing. It mainly includes the following content: + +- Naming specification +- Design specification +- Syntax specification +- Optimization-related specification +- PG compatibility + +In addition, specific examples are given for each detailed rule of the specification. + +### Application Scope + +This specification applies to MogDB 1.1.0 and later versions. + +## Naming Specification + +### Unified Object Naming Specification + +The unified standards for naming database objects, such as database, schema, table, column, view, index, constraint, sequence, function, trigger, etc. are as follows: + +- The length cannot exceed 63 characters. +- It is advised to use meaningful English vocabularies. + +- It is advised to use a combination of lowercase letters, numbers, and underscores. +- It is not advised to use double quotation marks (") unless it must contain special characters such as uppercase letters or spaces. +- It is not advised to start with PG, GS (to avoid confusion with the system DB object), and it is not advised to start with a number. +- It is forbidden to use [reserved words](../../reference-guide/sql-reference/2-keywords.md). Refer to official documents for reserved keywords. +- The number of columns that a table can contain varies from 250 to 1600 depending on the field type. + +### Temporary and Backup Object Naming + +- It is recommended to add a date to the names of temporary or backup database objects (such as table), for example, `dba.trade_record_1970_01_01`(where dba is the DBA-specific schema, trade_record is the table name, and 1970_01_01 is the backup date). + +### Tablespace Naming + +- The user tablespace of the database is represented by **ts_\**, where the **tablespace name** contains the following two categories: + 1. Data space: For the user's default tablespace, it is represented by **default**. For other tablespaces, it is represented according to the category of the tables hosted on the tablespace. For example, the table that stores code is represented by **code**. The table that stores customer information is represented by **customer**. Try to use one tablespace to host the tables of that category. If a table is particularly large, consider using a separate tablespace. + 2. Index space: add **idx_** in front of the name of the corresponding data tablespace. For example, the index space for the user's default tablespace is represented by **ts_idx_default**. For index tablespace of code table, use **ts_idx_code**. + +### Index Naming + +- Index object naming rules: **table_column_idx**, such as **student_name_idx**, the index naming method is the default naming method when the index name is not explicitly specified when an index is created for the MogDB database. + + Therefore, it is advised to create indexes without naming them explicitly, but using DBMS defaults. + +```sql +create unique index on departments(department_id); +CREATE INDEX + \di ++----------+-------------------------------+--------+---------+ +| Schema | Name | Type | Owner | +|----------+-------------------------------+--------+---------| +| mogdb | departments_department_id_idx | index | mogdb | ++----------+-------------------------------+--------+---------+ +SELECT 1 +``` + +### Variables Naming + +- English words should be used for naming, and pinyin should be avoided, especially pinyin abbreviations should not be used. Chinese or special characters are not allowed in the naming. + +- If no complicated operations are involved, simple applications such as counting are always defined by number. + +### Partitioned Table Naming + +- The name of the partitioned table follows the naming rules of ordinary tables. + +- A table is partitioned by time range (one partition per month), and the partition name is **PART_YYYYMM**. + + For example, PART_201901 and PART_201902 + +### Function Naming + +- The name should be consistent with its actual function. A verb should be used as a prefix command to cause an action to take place. + + Example: The following naming conforms to the specification: + + ``` + func_addgroups (Add multiple groups) + func_addgroup (Add one group) + ``` + +## Design Specification + +### Database Design + +- It is recommended to name the database after the business function, which is simple and intuitive. + +- The database is preferentially created using the PG compatibility type. + +- The recommended database encoding is utf8. + +### Tablespace Design + +- The frequently used tables and indexes are stored in a separate tablespace, which should be created on a disk with good performance. + +- Tables and indexes that are dominated by historical data or are less active can be stored in tablespaces with poor disk performance. + +- Tables and indexes can be stored separately in different tablespaces. + +- Tablespaces can also be divided by database, by schema, or by business function. + +- Each database/schema corresponds to a tablespace and a corresponding index tablespace. + +### Schema Design + +- When you perform a user creation under a database, a schema with the same name will be created under that database by default. +- It is not recommended to create database objects under the default public schema. +- Create a schema that is different from the username for the business to use. + +### Table Design + +- When designing the table structure, it should be planned to avoid adding fields frequently or modifying field types or lengths. + +- Comment information must be added to the table, with the table name matching the comment information. + +- The use of the unlogged/ temp/temporary keyword to create business tables is prohibited. + +- The data type must be strictly consistent for the fields that are used as join relationships between tables to avoid indexes not working properly. + +- It is forbidden to use VARCHAR or other character types to store date values, and if used, operations cannot be done on this field and need to be strictly defined in the data specification. + +- For astore tables with frequent updates, it is recommended to specify the table fillfactor=85 when building the table to reserve space for HOT. + +- Tables used for frequent updates should be placed separately in a tablespace with good storage performance. + +- It is recommended to consider partitioning for tables with data volume over billion or occupying more than 10GB on disk. + +- The data types defined in the fields in the table structure are consistent with those defined in the application, and the field proofreading rules are consistent between tables to avoid error reporting or the inability to use indexes. + + > Note: For example, the data type of the **user_id** field of table A is defined as **varchar**, but the SQL statement is **where user_id=1234;** + +### Partitioned Table Design + +- The number of partitioned tables is not recommended to exceed 1000. + +- Partitioned tables can be selected with different tablespaces by frequency of use. + +- The primary key or unique index must contain partitioned keys. + +- For tables with larger data volume, partition according to the attributes of table data to get better performance. + +- To convert a normal table into a partitioned table, you need to create a new partitioned table, and then import the data from the normal table into the newly created partitioned table. Therefore, when you initially design the table, please plan in advance whether to use partitioned tables according to your business. + +- It is recommended that for businesses with regular historical data deletion needs, the tables are partitioned by time, and when deleting, do not use the DELETE operation, but DROP or TRUNCATE the corresponding table. + +- It is not recommended to use global indexes in partitioned tables, because doing partition maintenance operations may cause global indexes to fail, making it difficult to maintain. + +#### Use of Partitioned Table + +Operate on the range partitioned table as follows. + +- Create a tablespace + +```sql +MogDB=# CREATE TABLESPACE example1 RELATIVE LOCATION 'tablespace1/tablespace_1'; +MogDB=# CREATE TABLESPACE example2 RELATIVE LOCATION 'tablespace2/tablespace_2'; +MogDB=# CREATE TABLESPACE example3 RELATIVE LOCATION 'tablespace3/tablespace_3'; +MogDB=# CREATE TABLESPACE example4 RELATIVE LOCATION 'tablespace4/tablespace_4'; +``` + +When the following information is displayed, it means the creation is successful. + +```sql +CREATE TABLESPACE +``` + +- Create a partitioned table + +```sql +MogDB=# CREATE TABLE mogdb_usr.customer_address +( + ca_address_sk integer NOT NULL , + ca_address_id character(16) NOT NULL , + ca_street_number character(10) , + ca_street_name character varying(60) , + ca_street_type character(15) , + ca_suite_number character(10) , + ca_city character varying(60) , + ca_county character varying(30) , + ca_state character(2) , + ca_zip character(10) , + ca_country character varying(20) , + ca_gmt_offset numeric(5,2) , + ca_location_type character(20) +) +TABLESPACE example1 + +PARTITION BY RANGE (ca_address_sk) +( + PARTITION P1 VALUES LESS THAN(5000), + PARTITION P2 VALUES LESS THAN(10000), + PARTITION P3 VALUES LESS THAN(15000), + PARTITION P4 VALUES LESS THAN(20000), + PARTITION P5 VALUES LESS THAN(25000), + PARTITION P6 VALUES LESS THAN(30000), + PARTITION P7 VALUES LESS THAN(40000), + PARTITION P8 VALUES LESS THAN(MAXVALUE) TABLESPACE example2 +) +ENABLE ROW MOVEMENT; +``` + +When the following information is displayed, it means the creation is successful. + +```sql +CREATE TABLE +``` + + ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) It is recommended that the number of column-based partitioned tables does not exceed 1,000. + +- Insert data + +Insert data from table **mogdb_usr.customer_address** into table **mogdb_usr.customer_address_bak**. For example, you have created a backup table **mogdb_usr.customer_address_bak** of table **mogdb_usr.customer_address** in the database, and now you need to insert the data from table **mogdb_usr.customer_address** into table **mogdb_usr. customer_address_bak**, then you can run the following command. + +```sql +MogDB=# CREATE TABLE mogdb_usr.customer_address_bak +( + ca_address_sk integer NOT NULL , + ca_address_id character(16) NOT NULL , + ca_street_number character(10) , + ca_street_name character varying(60) , + ca_street_type character(15) , + ca_suite_number character(10) , + ca_city character varying(60) , + ca_county character varying(30) , + ca_state character(2) , + ca_zip character(10) , + ca_country character varying(20) , + ca_gmt_offset numeric(5,2) , + ca_location_type character(20) +) +TABLESPACE example1 +PARTITION BY RANGE (ca_address_sk) +( + PARTITION P1 VALUES LESS THAN(5000), + PARTITION P2 VALUES LESS THAN(10000), + PARTITION P3 VALUES LESS THAN(15000), + PARTITION P4 VALUES LESS THAN(20000), + PARTITION P5 VALUES LESS THAN(25000), + PARTITION P6 VALUES LESS THAN(30000), + PARTITION P7 VALUES LESS THAN(40000), + PARTITION P8 VALUES LESS THAN(MAXVALUE) TABLESPACE example2 +) +ENABLE ROW MOVEMENT; +CREATE TABLE +MogDB=# INSERT INTO mogdb_usr.customer_address_bak SELECT * FROM mogdb_usr.customer_address; +INSERT 0 0 +``` + +- Alter partitioned table row movement properties + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak DISABLE ROW MOVEMENT; +ALTER TABLE +``` + +- Delete a partition + +Delete partition P8。 + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak DROP PARTITION P8; +ALTER TABLE +``` + +- Add a partition + +Add partition P8. The range is 40000<= P8<=MAXVALUE. + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak ADD PARTITION P8 VALUES LESS THAN (MAXVALUE); +ALTER TABLE +``` + +- Rename a partition + +Rename partition P8 as P_9. + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak RENAME PARTITION P8 TO P_9; +ALTER TABLE +``` + +Rename partition P_9 as P8. + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak RENAME PARTITION FOR (40000) TO P8; +ALTER TABLE +``` + +- Alter the tablespace of partition + +Alter the tablespace of partition P6 to example3. + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak MOVE PARTITION P6 TABLESPACE example3; +ALTER TABLE +``` + +Alter the tablespace of partition P4 to example4. + +```sql +MogDB=# ALTER TABLE mogdb_usr.customer_address_bak MOVE PARTITION P4 TABLESPACE example4; +ALTER TABLE +``` + +- Query a partition + +Query partition P6. + +```sql +MogDB=# SELECT * FROM mogdb_usr.customer_address_bak PARTITION (P6); +MogDB=# SELECT * FROM mogdb_usr.customer_address_bak PARTITION FOR (35888); +``` + +- Delete a partitioned table and tablespace + +```sql +MogDB=# DROP TABLE mogdb_usr.customer_address_bak; +DROP TABLE +MogDB=# DROP TABLESPACE example1; +MogDB=# DROP TABLESPACE example2; +MogDB=# DROP TABLESPACE example3; +MogDB=# DROP TABLESPACE example4; +DROP TABLESPACE +``` + +### Column Design + +- Avoid duplication of column names with system tables. + +- Field meanings and data types should be consistent with the program code design. + +- All fields must have comment information added. + +- Do not use character types when you can use numeric types. + +- It is forbidden to store date data in character types. + +- Use timestamptz for time type fields. + +- Try to require not null for fields and provide default values for fields. + +- MogDB new database is compatible with oracle by default, not null constraint does not allow to pass empty string, empty string will be converted to null by default, compatible with PG mode database will not have this problem. + +### Sequence Design + +- Manual creation of table-related sequences is prohibited and should be specified in the serial/bingserial type way. + +- It is recommended to set the step size of the sequence to 1. + +- It is not recommended to set minvalue and maxvalue. + +- It is not recommended to set cache, the serial number is not consecutive after setting cache. + +- It is prohibited to turn on cycle. + +- Serial should be consistent with the type and range of variable definition in the code to prevent the inability to insert data. + +### Constraint Design + +#### Primary Key Constraint + +- Each table must include a primary key. + +- It is not recommended that the name of the primary key has the service meaning, such as identification certificate or country name although the name is unique. + +- It is recommended that a primary key is written as `id serial primary key` or `id bigserial primary key`. + +- It is recommended that the primary key in a large-sized table can be written as follows, which is easy to maintain later. + + ```sql + create table test(id serial not null ); + create unique index CONCURRENTLY ON test (id); + ``` + +#### Unique Constraint + +Apart from the primary key, unique constraint is needed. You can create a unique index with **uk_** as the prefix to create unique constraint. + +#### Foreign Key Constraint + +- You'd better create foreign key constraints for a table with foreign key relationship. +- The use of foreign keys is not recommended for systems with high performance requirements and security under your control. +- When using the foreign key, you must set the action of the foreign key, such as cascade, set null, or set default. + +#### Non-Empty Column + +- All non-null columns must have the not null constraint added + +#### Check Constraint + +- For fields with the check constraint, it is required to specify the check rules, such as the gender and status fields. + +### Index Design + +- The number of table indexes for frequent DML operations is not recommended to exceed 5. +- Add concurrently parameter when create/drop index. +- Virtual indexes can be used to determine the validity of indexes before actually creating them. +- Create indexes for fields that frequently appear after the keywords order by, group by, and distinguish. +- Fields that are often used as query selections to create indexes. +- Indexes on attributes that are often used as table joins. +- The number of fields in a composite index is not recommended to exceed 3. +- Composite indexes should have one field that is a common search condition. +- The first field of a composite index should not have a single-field index. +- For tables where data is rarely updated and only a few of the fields are frequently queried, consider using index overrides. +- Do not create indexes on fields that have a large number of identical fetch values. +- It is recommended to use unique index instead of unique constraints for subsequent maintenance. +- It is recommended to build compound indexes with multiple fields for high frequency queries with multiple fields and conditions in where, with reference to the data distribution. +- Useless indexes and duplicate indexes should be deleted to avoid negative impact on the execution plan and database performance. + +### View Design + +- You'd better use simple views and use less complex views. + + Simple view: Data comes from a single table and a simple view does not contain groups of data and functions. + + Complex view: Data comes from multiple tables, or a complex view contains groups of data or functions. A complex view can contain three tables at most. + +- You'd better not use nested views. If nested views have to be used, it is advised to have two levels of nesting at most. + +### Function Design + +- A function must retrieve database table records or other database objects, or even modify database information, such as Insert, Delete, Update, Drop, or Create. +- If a function does not relate to a database, it cannot be realized using a database function. +- It is not advised to use DML or DDL statements in a function. + +## Syntax Specification + +### About NULL + +- Description: `NULL` judgment: `IS NULL`, `IS NOT NULL`. +- Description: Beware of `boolean` types taking the values `true`, `false`, `NULL`. + +- Description: Beware of `NOT IN` collections with `NULL` elements. + +- Recommendation: Use `count(1)` or `count(*)` to count rows, but not `count(col)` to count rows, because `NULL` values will not be counted. + +- Rule: When `count(multi-column names)`, the multi-column names must be enclosed in parentheses, e.g. `count( (col1,col2,col3) )`. + +- Note: With multi-column `count`, the row is counted even if all columns are `NULL`, so the effect is the same as `count(*)`. + +- Note: `count(distingu col)` counts the number of non-`NULL` non-repeats of a column, `NULL` is not counted; `count(distingu (col1,col2,...) )` counts the unique values of multiple columns, `NULL` is counted, while `NULL` and `NULL` are considered the same. + +- Note: Distinction between count and sum of NULL + + ```sql + select count(1), count(a), sum(a) from (SELECT * FROM (VALUES (NULL), (2) ) v(a)) as foo where a is NULL; + count | count | sum + -------+-------+----- + 1 | 0 | + (1 row) + ``` + +- Check whether two values are the same (NULL is considered as the same value). + + ```sql + select null is distinct from null; + ?column? + \--------- + f + (1 row) + + select null is distinct from 1; + ?column? + \--------- + t + (1 row) + + select null is not distinct from null; + ?column? + \--------- + t + (1 row) + + select null is not distinct from 1; + ?column? + \--------- + f + (1 row) + ``` + +### About Invalid Indexes + +- During SQL statement writing, functions and expressions are usually used in query operations. It is not recommended that functions and expressions are used in condition columns. Using a function or expression in a condition column will make indexes of the condition column unused, thereby affecting the SQL query efficiency. It is recommended that functions or expressions are used in condition values. For example, + + ```sql + select name from tab where id+100>1000; + ``` + + This statement can be changed to the following: + + ```sql + select name from tab where id>1000-100; + ``` + +- Do not use left fuzzy query. For example, + + ```sql + select id from tab where name like '%ly'; + ``` + +- Do not use the negative query, such as not in/like. For example, + + ```sql + select id from tab where name not in ('ly','ty'); + ``` + +### Ensuring That All Variables and Parameters Are Used + +- Declare-variable also generates certain system overhead and makes code look loose. If some variables are not used in compilation, they will report alarms. Make sure that no any alarm is reported. + +## Query Operations + +### DDL Operation + +- Any DDL operations on existing tables are prohibited during peak business periods + +- All production DDL operations must be verified by the development test environment + +- Concurrently should be used when maintaining indexes + +- pg_repack should be used instead of vacuum full to rebuild the table + +- When adding fields with default values to a large table, it should be split into three parts: adding fields, filling default values and adding non-null constraints, such as breaking `alter table t add column col datatype not null default xxx;` into the following, to avoid too long a table lock caused by filling `default` values + + ```sql + alter table t add column col datatype ； + alter table t alter column col set default xxx； + update table t set column= DEFAULT where id in ( select id from t where column is null limit + 1000 ) ; \watch 3 + alter table t alter column col set not null； + ``` + +### DML Operation + +- The SQL statement for updating data is prohibited to appear `where 1=1` +- The amount of data operated by a single DML statement should not exceed 100,000 + +- When clearing the data in the table, `truncate` should be used + +- For risky operations, you should open the transaction and confirm it before committing. + +- The SQL logic in the transaction should be as simple as possible, and the operation should be submitted in time after execution to avoid `idle in transaction` status. + +- Use `copy` instead of `insert` when importing a large amount of data. + +- Consider deleting indexes before importing data, and rebuild them after importing. + +### DQL Operation + +- Prohibit the use of `select *`, apply the specific required field substitution +- Prohibit the use of `where 1=1` to avoid full table scan or Cartesian product + +- The search condition value should be consistent with the field type to prevent not going to the index + +- Fields to the left of the equal sign should be consistent with the index, especially conditional or functional indexes + +- Pay attention to the execution plan of slow SQL, if it is not consistent with the expectation, change it as soon as possible + +- Use `count(*)` or `count(1)` to count rows, `count(column)` will not count `null` rows + +- Limit the number of `join`, no more than 3 are recommended + +- Recursive queries need to be limited to prevent infinite loops + +- For `or` operations, you should use `union all` or `union` instead + +### Data Import + +- When a large amount of data needs to be stored in a table, it is recommended that COPY is used rather than INSERT. This can improve the data write speed. +- Before data is imported, delete related indexes. After the import is complete, recreate indexes. This can improve the data import speed. + +### Transaction Operation + +- Make sure that the SQL logic in a transaction is simple, the granularity of each transaction is small, less resources are locked, lock and deadlock can be avoided, and transaction can be committed in a timely manner after being executed. +- For DDL operations, especially multiple DDL operations, including CRAETE, DROP, and ALTER, do not explicitly start a transaction because the lock mode value is very high and deadlock easily occurs. +- If the state of the master node is idle in transaction, related resources will be locked, thereby leading to lock, even deadlock. If the state of the slave node is idle in transaction, synchronization between the master and slave nodes will be suspended. + +### Others + +- For instances running in SSDs, it is recommended that the value of **random_page_cost** (default value: **4**) is set to a value ranging from 1.0 to 2.0. This can make the query planner preferably use the index to perform scanning. +- In the scenario where EXPLAIN ANALYZE needs to be used to view the actual execution plan and time, if a write query is to be performed, it is strongly recommended that a transaction is started first and then rollback is performed. +- For tables frequently updated and with the data size largely increased, table reorganization should be performed in appropriate time to lower the high water mark. + +## PostgreSQL Compatibility + +### Database Creation Specifications + +During MogDB database creation, the following PG compatibility mode is used: + +```sql +create database dbnam DBCOMPATIBILITY='PG' encoding=’utf8’; +``` + +### Data Type + +#### Value Type + +During development and usage, MogDB supports only the smallint, integer, bigint, numeric[(p[,s])], serial, and bigserial value types. + +| Type | PostgreSQL | MogDB | Storage Length | Remarks | +| :--------------- | :--------- | :-------- | :------------- | :----------------------------------------------------------- | +| tinyint | / | Supported | 1 byte | 0 to 255 | +| smallint | Supported | Supported | 2 bytes | -32,768 to +32,767 | +| integer | Supported | Supported | 4 bytes | -2,147,483,648 to +2,147,483,647 | +| binary_integer | / | Supported | / | integer alias | +| bigint | Supported | Supported | 8 bytes | -9,223,372,036,854,775,808 to +9,223,372,036,854,775,807 | +| decimal[(p[,s])] | Supported | Supported | Variable byte | A maximum of 131072 before the decimal point and 16383 after the decimal point | +| numeric[(p[,s])] | Supported | Supported | Variable byte | A maximum of 131072 before the decimal point and 16383 after the decimal point | +| number[(p[,s])] | / | Supported | / | Numeric alias | +| real | Supported | Supported | 4 bytes | Accurate to six decimal digits | +| float4 | / | Supported | 4 bytes | Accurate to six decimal digits | +| double precision | Supported | Supported | 8 bytes | Accurate to fifteen decimal digits | +| binary_double | / | Supported | 8 bytes | Double precision alias | +| float8 | / | Supported | 8 bytes | Accurate to fifteen decimal digits | +| float[(p )] | / | Supported | 4 or 8 bytes | | +| dec[(p,[s])] | / | Supported | / | A maximum of 131072 before the decimal point and 16383 after the decimal point | +| integer[(p,[s])] | / | Supported | / | A maximum of 131072 before the decimal point and 16383 after the decimal point | +| smallserial | Supported | Supported | 2 bytes | 1 to 32,767 | +| serial | Supported | Supported | 4 bytes | 1 to 2,147,483,647 | +| bigserial | Supported | Supported | 8 bytes | 1 to 9,223,372,036,854,775,807 | +| tinyint | / | Supported | 1 byte | 0 to 255 | + +#### Character Type + +During the development, MogDB supports only the char(n), varchar(n), and text character types. + +| Type | PostgreSQL | MogDB | Storage Length | Remarks | +| :----------- | :--------- | :-------- | :----------------------------------------------------------- | :----------------------------------------------------------- | +| char(n) | Supported | Supported | A maximum of 1 GB in postgreSQL
A maximum of 10 MB in MogDB | In postgreSQL, *n* indicates the number of characters.
In MogDB, *n* indicates the number of bytes.
In the compatibility PG mode, *n* indicates the number of characters. | +| nchar(n) | / | Supported | A maximum of 10 MB | *n* indicates the number of bytes.
In the compatibility PG mode, *n* indicates the number of characters. | +| varchar(n) | Supported | Supported | A maximum of 1 GB in postgreSQL
A maximum of 10 MB in MogDB | In postgreSQL, *n* indicates the number of characters.
In MogDB, *n* indicates the number of bytes.
In the compatibility PG mode, *n* indicates the number of characters. | +| varchar2(n) | / | Supported | A maximum of 10 MB | varchar(n) alias | +| nvarchar2(n) | / | Supported | A maximum of 10 MB | *n* indicates the number of characters. | +| text | Supported | Supported | 1 GB - 1 | | +| clob | / | Supported | 1 GB - 1 | text alias | + +#### Time Type + +During the development, MogDB supports only the timestamp[(p )][with time zone] and date time types. + +| Type | PostgreSQL | MogDB | Storage Length | Remarks | +| :--------------------------------- | :--------- | :-------- | :------------- | :----------------------------------------------------------- | +| timestamp[(p )][without time zone] | Supported | Supported | 8 bytes | 4713 BC to 294276 AD | +| timestamp[(p )][with time zone] | Supported | Supported | 8 bytes | 4713 BC to 294276 AD | +| date | Supported | Supported | 4 bytes | 4713 BC to 5874897 AD (The actual storage size is 8 bytes in MogDB) | +| time[(p )][without time zone] | Supported | Supported | 8 bytes | 00:00:00 to 24:00:00 | +| time[(p )][with time zone] | Supported | Supported | 12 bytes | 00:00:00+1459 to 24:00:00-1459 | +| interval[fields][(p )] | Supported | Supported | 16 bytes | -178000000 to 178000000 years | +| smalldatetime | / | Supported | 8 bytes | Date and time without timezone, accurating to the minute, 30s equaling one minute | +| interval day(1) to second(p ) | / | Supported | 16 bytes | | +| reltime | / | Supported | 4 bytes | | + +#### JSON Type + +MogDB supports only the JSON type. + +| Type | PostgreSQL | MogDB | Storage Length | Remarks | +| :---- | :--------- | :-------------------------- | :------------- | :------ | +| json | Supported | Supported | / | | +| jsonb | Supported | Supported since version 2.1 | / | | + +### Keywords + +In the following table, **Reserved** indicates that keywords in a database are reserved and cannot be customized. **Non-reserved** or **N/A** indicates that keywords can be customized. + +| Keyword | MogDB | PostgreSQL | +| :------------ | :----------------------------------------------- | :----------------------------------------------- | +| AUTHID | Reserved | N/A | +| BUCKETS | Reserved | N/A | +| COMPACT | Reserved | N/A | +| DATE | Non-reserved (function or type is not supported) | | +| DELTAMERGE | Reserved | N/A | +| EXCLUDED | Reserved | N/A | +| FENCED | Reserved | N/A | +| GROUPING | | Non-reserved (function or type is not supported) | +| HDFSDIRECTORY | Reserved | N/A | +| IS | Reserved | Reserved (function or type is supported) | +| ISNULL | Non-reserved | Reserved (function or type is supported) | +| LATERAL | | Reserved | +| LESS | Reserved | N/A | +| MAXVALUE | Reserved | Non-reserved | +| MINUS | Reserved | N/A | +| MODIFY | Reserved | N/A | +| NLSSORT | Reserved | N/A | +| NUMBER | Non-reserved (function or type is not supported) | | +| PERFORMANCE | Reserved | N/A | +| PROCEDURE | Reserved | Non-reserved | +| REJECT | Reserved | N/A | +| ROWNUM | Reserved | N/A | +| SYSDATE | Reserved | N/A | +| VERIFY | Reserved | N/A | + +### Implicit Conversion Comparison Table + +| Input Type | Target Type | MogDB | PG | +| :---------- | :----------------------------------------------------------- | :------------------------ | ------------------------------------------------------------ | +| bool | int2, int4, int8 | Supported | int4 is not supported, others are the same | +| int2 | bool, text, bpchar, varchar,interval | Supported (except bpchar) | NA | +| int4 | bool, int2, text, bpchar, varchar, interval | Supported (except bpchar) | bool is not supported, int2 is in assignment, others are the same | +| int8 | bool, text, bpchar, varchar | Supported (except bpchar) | NA | +| text | int8, int4, int2, float4, float8, date, timestamp, nemeric | Supported | NA | +| float4 | int8, int4, int2, text, bpchar, varchar | Supported (except bpchar) | First three are in assignment, others are the same | +| float8 | int8, int4, int2, text, float4, bpchar, varchar, interval, numeric | Supported (except bpchar) | int8, int4, int2, float4, numeric are in assignment, others are the same | +| bpchar | int8, int4, date, timestamp, numeric | | | +| date | text, bpchar, varchar | Supported (except bpchar) | NA | +| timestamp | text, varchar | Supported | NA | +| timestamptz | text | Supported | NA | +| numeric | int8, int4, int2, text, bpchar, varchar, interval | Supported (except bpchar) | First three are in assignment, others are the same | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/1-development-based-on-jdbc-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/1-development-based-on-jdbc-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..74f232a2f0df7b72d9bf6687e4b9f077ae9f84bd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/1-development-based-on-jdbc-overview.md @@ -0,0 +1,10 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2021-04-26 +--- + +# Overview + +Java Database Connectivity (JDBC) is a Java API for running SQL statements. It provides unified access interfaces for different relational databases, based on which applications process data. MogDB supports JDBC 4.0 and requires JDK 1.8 for code compiling. It does not support JDBC-ODBC bridge. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/10-example-common-operations.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/10-example-common-operations.md new file mode 100644 index 0000000000000000000000000000000000000000..bada212f27fbb34906a14f65b96dc0b15035faa7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/10-example-common-operations.md @@ -0,0 +1,280 @@ +--- +title: Example Common Operations +summary: Example Common Operations +author: Guo Huan +date: 2021-04-26 +--- + +# Example: Common Operations + +**Example 1:** + +The following illustrates how to develop applications based on MogDB JDBC interfaces. + +```java +//DBtest.java +// This example illustrates the main processes of JDBC-based development, covering database connection creation, table creation, and data insertion. + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.SQLException; +import java.sql.Statement; +import java.sql.CallableStatement; +import java.sql.Types; + +public class DBTest { + + // Create a database connection. + public static Connection GetConnection(String username, String passwd) { + String driver = "org.opengauss.Driver"; + String sourceURL = "jdbc:opengauss://localhost:8000/postgres"; + Connection conn = null; + try { + // Load the database driver. + Class.forName(driver).newInstance(); + } catch (Exception e) { + e.printStackTrace(); + return null; + } + + try { + // Create a database connection. + conn = DriverManager.getConnection(sourceURL, username, passwd); + System.out.println("Connection succeed!"); + } catch (Exception e) { + e.printStackTrace(); + return null; + } + + return conn; + }; + + // Run a common SQL statement to create table customer_t1. + public static void CreateTable(Connection conn) { + Statement stmt = null; + try { + stmt = conn.createStatement(); + + // Run a common SQL statement. + int rc = stmt + .executeUpdate("CREATE TABLE customer_t1(c_customer_sk INTEGER, c_customer_name VARCHAR(32));"); + + stmt.close(); + } catch (SQLException e) { + if (stmt != null) { + try { + stmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + // Run a prepared statement to insert data in batches. + public static void BatchInsertData(Connection conn) { + PreparedStatement pst = null; + + try { + // Generate a prepared statement. + pst = conn.prepareStatement("INSERT INTO customer_t1 VALUES (?,?)"); + for (int i = 0; i < 3; i++) { + // Add parameters. + pst.setInt(1, i); + pst.setString(2, "data " + i); + pst.addBatch(); + } + // Perform batch processing. + pst.executeBatch(); + pst.close(); + } catch (SQLException e) { + if (pst != null) { + try { + pst.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + // Run a prepared statement to update data. + public static void ExecPreparedSQL(Connection conn) { + PreparedStatement pstmt = null; + try { + pstmt = conn + .prepareStatement("UPDATE customer_t1 SET c_customer_name = ? WHERE c_customer_sk = 1"); + pstmt.setString(1, "new Data"); + int rowcount = pstmt.executeUpdate(); + pstmt.close(); + } catch (SQLException e) { + if (pstmt != null) { + try { + pstmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + +// Run a stored procedure. + public static void ExecCallableSQL(Connection conn) { + CallableStatement cstmt = null; + try { + // 存储过程TESTPROC需提前创建。 + cstmt=conn.prepareCall("{? = CALL TESTPROC(?,?,?)}"); + cstmt.setInt(2, 50); + cstmt.setInt(1, 20); + cstmt.setInt(3, 90); + cstmt.registerOutParameter(4, Types.INTEGER); // Register an OUT parameter of the integer type. + cstmt.execute(); + int out = cstmt.getInt(4); // Obtain the OUT parameter. + System.out.println("The CallableStatment TESTPROC returns:"+out); + cstmt.close(); + } catch (SQLException e) { + if (cstmt != null) { + try { + cstmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + + /** + *Main process. Call static methods one by one. + * @param args + */ + public static void main(String[] args) { + // Create a database connection. + Connection conn = GetConnection("tester", "Password1234"); + + // Create a table. + CreateTable(conn); + + // Insert data in batches. + BatchInsertData(conn); + + // Run a prepared statement to update data. + ExecPreparedSQL(conn); + + // Run a stored procedure. + ExecCallableSQL(conn); + + // Close the connection to the database. + try { + conn.close(); + } catch (SQLException e) { + e.printStackTrace(); + } + + } + +} +``` + +**Example 2 High Client Memory Usage** + +In this example, **setFetchSize** adjusts the memory usage of the client by using the database cursor to obtain server data in batches. It may increase network interaction and deteriorate some performance. + +The cursor is valid within a transaction. Therefore, disable automatic commit and then manually commit the code. + +```java +// Disable automatic commit. +conn.setAutoCommit(false); +Statement st = conn.createStatement(); + +// Open the cursor and obtain 50 lines of data each time. +st.setFetchSize(50); +ResultSet rs = st.executeQuery("SELECT * FROM mytable"); + +while (rs.next()) +{ + System.out.print("a row was returned."); +} +conn.commit(); +rs.close(); + +// Disable the server cursor. +st.setFetchSize(0); +rs = st.executeQuery("SELECT * FROM mytable"); + +while (rs.next()) +{ + System.out.print("many rows were returned."); +} +conn.commit(); +rs.close(); + +// Close the statement. +st.close(); +conn.close(); +``` + +Run the following command to enable automatic commit: + +```java +conn.setAutoCommit(true); +``` + +**Example 3 Example of Common Data Types** + +```java +//Example of the bit type. Note that the value range of the bit type is [0,1]. +Statement st = conn.createStatement(); +String sqlstr = "create or replace function fun_1()\n" + + "returns bit AS $$\n" + + "select col_bit from t_bit limit 1;\n" + + "$$\n" + + "LANGUAGE SQL;"; +st.execute(sqlstr); +CallableStatement c = conn.prepareCall("{ ? = call fun_1() }"); +//Register the output type, which is a bit string. +c.registerOutParameter(1, Types.BIT); +c.execute(); +//Use the Boolean type to obtain the result. +System.out.println(c.getBoolean(1)); + +// Example of using the money type +// Example of using a column of the money type in the table structure. +st.execute("create table t_money(col1 money)"); +PreparedStatement pstm = conn.prepareStatement("insert into t_money values(?)"); +// Use PGobject to assign a value. The value range is [-92233720368547758.08,92233720368547758.07]. +PGobject minMoney = new PGobject(); +minMoney.setType("money"); +minMoney.setValue("-92233720368547758.08"); +pstm.setObject(1, minMoney); +pstm.execute(); +// Use PGMoney to assign a value. The value range is [-9999999.99,9999999.99]. +pstm.setObject(1,new PGmoney(9999999.99)); +pstm.execute(); + +// Example of using the function whose return value is money. +st.execute("create or replace function func_money() " + + "return money " + + "as declare " + + "var1 money; " + + "begin " + + " select col1 into var1 from t_money limit 1; " + + " return var1; " + + "end;"); +CallableStatement cs = conn.prepareCall("{? = call func_money()}"); +cs.registerOutParameter(1,Types.DOUBLE); +cs.execute(); +cs.getObject(1); +``` + +**Example 4 Obtaining the Driver Version** + +```java +Driver.getGSVersion(); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/11-example-retrying-sql-queries-for-applications.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/11-example-retrying-sql-queries-for-applications.md new file mode 100644 index 0000000000000000000000000000000000000000..b89460135ab002fd7eb587819d44c22d0c3591ef --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/11-example-retrying-sql-queries-for-applications.md @@ -0,0 +1,205 @@ +--- +title: Example Retrying SQL Queries for Applications +summary: Example Retrying SQL Queries for Applications +author: Guo Huan +date: 2021-04-26 +--- + +# Example Retrying SQL Queries for Applications + +If the primary database node is faulty and cannot be restored within 10s, the standby database node automatically switches to the active state to ensure the normal running of MogDB. During the switchover, jobs that are running will fail and those start running after the switchover are not affected. To prevent upper-layer services from being affected by the failover, refer to the following example to construct an SQL retry mechanism at the service layer. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; + +class ExitHandler extends Thread { + private Statement cancel_stmt = null; + + public ExitHandler(Statement stmt) { + super("Exit Handler"); + this.cancel_stmt = stmt; + } + public void run() { + System.out.println("exit handle"); + try { + this.cancel_stmt.cancel(); + } catch (SQLException e) { + System.out.println("cancel query failed."); + e.printStackTrace(); + } + } +} + +public class SQLRetry { + // Create a database connection. + public static Connection GetConnection(String username, String passwd) { + String driver = "org.opengauss.Driver"; + String sourceURL = "jdbc:opengauss://10.131.72.136:8000/postgres"; + Connection conn = null; + try { + // Load the database driver. + Class.forName(driver).newInstance(); + } catch (Exception e) { + e.printStackTrace(); + return null; + } + + try { + // Create a database connection. + conn = DriverManager.getConnection(sourceURL, username, passwd); + System.out.println("Connection succeed!"); + } catch (Exception e) { + e.printStackTrace(); + return null; + } + + return conn; +} + + // Run a common SQL statement. Create the jdbc_test1 table. + public static void CreateTable(Connection conn) { + Statement stmt = null; + try { + stmt = conn.createStatement(); + + + Runtime.getRuntime().addShutdownHook(new ExitHandler(stmt)); + + // Run a common SQL statement. + int rc2 = stmt + .executeUpdate("DROP TABLE if exists jdbc_test1;"); + + int rc1 = stmt + .executeUpdate("CREATE TABLE jdbc_test1(col1 INTEGER, col2 VARCHAR(10));"); + + stmt.close(); + } catch (SQLException e) { + if (stmt != null) { + try { + stmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + // Run a prepared statement to insert data in batches. + public static void BatchInsertData(Connection conn) { + PreparedStatement pst = null; + + try { + // Generate a prepared statement. + pst = conn.prepareStatement("INSERT INTO jdbc_test1 VALUES (?,?)"); + for (int i = 0; i < 100; i++) { + // Add parameters. + pst.setInt(1, i); + pst.setString(2, "data " + i); + pst.addBatch(); + } + // Perform batch processing. + pst.executeBatch(); + pst.close(); + } catch (SQLException e) { + if (pst != null) { + try { + pst.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + // Run a prepared statement to update data. + private static boolean QueryRedo(Connection conn){ + PreparedStatement pstmt = null; + boolean retValue = false; + try { + pstmt = conn + .prepareStatement("SELECT col1 FROM jdbc_test1 WHERE col2 = ?"); + + pstmt.setString(1, "data 10"); + ResultSet rs = pstmt.executeQuery(); + + while (rs.next()) { + System.out.println("col1 = " + rs.getString("col1")); + } + rs.close(); + + pstmt.close(); + retValue = true; + } catch (SQLException e) { + System.out.println("catch...... retValue " + retValue); + if (pstmt != null) { + try { + pstmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + + System.out.println("finesh......"); + return retValue; + } + + // Configure the number of retry attempts for the retry of a query statement upon a failure. + public static void ExecPreparedSQL(Connection conn) throws InterruptedException { + int maxRetryTime = 50; + int time = 0; + String result = null; + do { + time++; + try { + System.out.println("time:" + time); + boolean ret = QueryRedo(conn); + if(ret == false){ + System.out.println("retry, time:" + time); + Thread.sleep(10000); + QueryRedo(conn); + } + } catch (Exception e) { + e.printStackTrace(); + } + } while (null == result && time < maxRetryTime); + + } + + /** + *Main process. Call static methods one by one. + * @param args + * @throws InterruptedException + */ + public static void main(String[] args) throws InterruptedException { + // Create a database connection. + Connection conn = GetConnection("testuser", "test@123"); + + // Create a table. + CreateTable(conn); + + // Insert data in batches. + BatchInsertData(conn); + + // Run a prepared statement to update data. + ExecPreparedSQL(conn); + + // Close the connection to the database. + try { + conn.close(); + } catch (SQLException e) { + e.printStackTrace(); + } + + } + + } +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/12-example-importing-and-exporting-data-through-local-files.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/12-example-importing-and-exporting-data-through-local-files.md new file mode 100644 index 0000000000000000000000000000000000000000..fd8e06372473236bf822f90e364fc556925d851b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/12-example-importing-and-exporting-data-through-local-files.md @@ -0,0 +1,119 @@ +--- +title: Example Importing and Exporting Data Through Local Files +summary: Example Importing and Exporting Data Through Local Files +author: Guo Huan +date: 2021-04-26 +--- + +# Example Importing and Exporting Data Through Local Files + +When Java is used for secondary development based on MogDB, you can use the CopyManager interface to export data from the database to a local file or import a local file to the database by streaming. The file can be in CSV or TEXT format. + +The sample program is as follows. Load the MogDB JDBC driver before running it. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.io.IOException; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.sql.SQLException; +import org.opengauss.copy.CopyManager; +import org.opengauss.core.BaseConnection; + +public class Copy{ + + public static void main(String[] args) + { + String urls = new String("jdbc:opengauss://10.180.155.74:8000/postgres"); // Database URL + String username = new String("jack"); // Username + String password = new String("Enmo@123"); // Password + String tablename = new String("migration_table"); // Table information + String tablename1 = new String("migration_table_1"); // Table information + String driver = "org.opengauss.Driver"; + Connection conn = null; + + try { + Class.forName(driver); + conn = DriverManager.getConnection(urls, username, password); + } catch (ClassNotFoundException e) { + e.printStackTrace(System.out); + } catch (SQLException e) { + e.printStackTrace(System.out); + } + + // Export the query result of SELECT * FROM migration_table to the local file d:/data.txt. + try { + copyToFile(conn, "d:/data.txt", "(SELECT * FROM migration_table)"); + } catch (SQLException e) { + k + e.printStackTrace(); + } catch (IOException e) { + + e.printStackTrace(); + } + // Import data from the d:/data.txt file to the migration_table_1 table. + try { + copyFromFile(conn, "d:/data.txt", tablename1); + } catch (SQLException e) { + e.printStackTrace(); + } catch (IOException e) { + + e.printStackTrace(); + } + + // Export the data from the migration_table_1 table to the d:/data1.txt file. + try { + copyToFile(conn, "d:/data1.txt", tablename1); + } catch (SQLException e) { + + e.printStackTrace(); + } catch (IOException e) { + + + e.printStackTrace(); + } + } + // Use copyIn to import data from a file to the database. + public static void copyFromFile(Connection connection, String filePath, String tableName) + throws SQLException, IOException { + + FileInputStream fileInputStream = null; + + try { + CopyManager copyManager = new CopyManager((BaseConnection)connection); + fileInputStream = new FileInputStream(filePath); + copyManager.copyIn("COPY " + tableName + " FROM STDIN", fileInputStream); + } finally { + if (fileInputStream != null) { + try { + fileInputStream.close(); + } catch (IOException e) { + e.printStackTrace(); + } + } + } + } + + // Use copyOut to export data from the database to a file. + public static void copyToFile(Connection connection, String filePath, String tableOrQuery) + throws SQLException, IOException { + + FileOutputStream fileOutputStream = null; + + try { + CopyManager copyManager = new CopyManager((BaseConnection)connection); + fileOutputStream = new FileOutputStream(filePath); + copyManager.copyOut("COPY " + tableOrQuery + " TO STDOUT", fileOutputStream); + } finally { + if (fileOutputStream != null) { + try { + fileOutputStream.close(); + } catch (IOException e) { + e.printStackTrace(); + } + } + } + } +} +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/13-example-2-migrating-data-from-a-my-database-to-mogdb.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/13-example-2-migrating-data-from-a-my-database-to-mogdb.md new file mode 100644 index 0000000000000000000000000000000000000000..9916d2e6f8dc1ae0037c72e6ffea59c5ce8ca9d5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/13-example-2-migrating-data-from-a-my-database-to-mogdb.md @@ -0,0 +1,97 @@ +--- +title: Example 2 Migrating Data from a MY Database to MogDB +summary: Example 2 Migrating Data from a MY Database to MogDB +author: Guo Huan +date: 2021-04-26 +--- + +# Example 2 Migrating Data from a MY Database to MogDB + +The following example shows how to use CopyManager to migrate data from MY to MogDB. + +```java +import java.io.StringReader; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; + +import org.opengauss.copy.CopyManager; +import org.opengauss.core.BaseConnection; + +public class Migration{ + + public static void main(String[] args) { + String url = new String("jdbc:opengauss://10.180.155.74:8000/postgres"); // Database URL + String user = new String("jack"); // MogDB username + String pass = new String("Enmo@123"); // MogDB password + String tablename = new String("migration_table"); // Table information + String delimiter = new String("|"); // Delimiter + String encoding = new String("UTF8"); // Character set + String driver = "org.opengauss.Driver"; + StringBuffer buffer = new StringBuffer(); // Buffer to store formatted data + + try { + // Obtain the query result set of the source database. + ResultSet rs = getDataSet(); + + // Traverse the result set and obtain records row by row. + // The values of columns in each record are separated by the specified delimiter and end with a linefeed, forming strings. + // Add the strings to the buffer. + while (rs.next()) { + buffer.append(rs.getString(1) + delimiter + + rs.getString(2) + delimiter + + rs.getString(3) + delimiter + + rs.getString(4) + + "\n"); + } + rs.close(); + + try { + // Connect to the target database. + Class.forName(driver); + Connection conn = DriverManager.getConnection(url, user, pass); + BaseConnection baseConn = (BaseConnection) conn; + baseConn.setAutoCommit(false); + + // Initialize the table. + String sql = "Copy " + tablename + " from STDIN DELIMITER " + "'" + delimiter + "'" + " ENCODING " + "'" + encoding + "'"; + + // Commit data in the buffer. + CopyManager cp = new CopyManager(baseConn); + StringReader reader = new StringReader(buffer.toString()); + cp.copyIn(sql, reader); + baseConn.commit(); + reader.close(); + baseConn.close(); + } catch (ClassNotFoundException e) { + e.printStackTrace(System.out); + } catch (SQLException e) { + e.printStackTrace(System.out); + } + + } catch (Exception e) { + e.printStackTrace(); + } + } + + //******************************** + // Return the query result set from the source database. + //********************************* + private static ResultSet getDataSet() { + ResultSet rs = null; + try { + Class.forName("com.MY.jdbc.Driver").newInstance(); + Connection conn = DriverManager.getConnection("jdbc:MY://10.119.179.227:3306/jack?useSSL=false&allowPublicKeyRetrieval=true", "jack", "Enmo@123"); + Statement stmt = conn.createStatement(); + rs = stmt.executeQuery("select * from migration_table"); + } catch (SQLException e) { + e.printStackTrace(); + } catch (Exception e) { + e.printStackTrace(); + } + return rs; + } +} +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/14-example-logic-replication-code.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/14-example-logic-replication-code.md new file mode 100644 index 0000000000000000000000000000000000000000..0568351e7048c07f1fdc2b0ddadedc02f3bc00f7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/14-example-logic-replication-code.md @@ -0,0 +1,166 @@ +--- +title: Example Logic Replication Code +summary: Example Logic Replication Code +author: Guo Huan +date: 2021-04-26 +--- + +# Example Logic Replication Code + +The following example demonstrates how to use the logical replication function through the JDBC APIs. + +For logical replication, in addition to the configuration items described in [Logical Decoding](../../logical-replication/logical-decoding/1-logical-decoding.md), the following configuration items are added for JDBC stream decoding: + +1. Decoding thread concurrency + + Set **parallel-decode-num** to specify the number of decoder threads for parallel decoding. The value is an integer ranging from 1 to 20. The value **1** indicates that decoding is performed based on the original serial logic. Other values indicate that parallel decoding is enabled. The default value is **1**. When this item is set to **1**, the decoding format **decode-style** cannot be configured. + +2. Decoding format + + Configure **decode-style** to specify the decoding format. The value can be **'j'**, **'t'** or **'b'** of the char type, indicating the JSON, text, or binary format, respectively. The default value is **'b'**, indicating binary decoding. This option is set only when parallel decoding is allowed and the binary decoding is supported only in the parallel decoding scenario. + +> **NOTE:** +> The binary encoding rules are as follows: +> +> 1. The first four bytes represent the total number of bytes of the decoding result that follows the statement-level delimiter letter P (excluded) or the batch end character F (excluded) If the value is **0**, the decoding of this batch ends. +> 2. The next eight bytes (uint64) indicate the LSN. +> 3. The following 1-byte letter can be **B**, **C**, **I**, **U**, or **D**, representing BEGIN, COMMIT, INSERT, UPDATE, or DELETE. +> 4. If the letter described in step 3 is **B**: +> a. The following eight bytes (uint64) indicate the CSN. +> b. The following eight bytes (uint64) indicate commit_lsn. +> c. (Optional) If the next 1-byte letter is **T**, the following four bytes (uint32) indicate the timestamp length for committing the transaction. The following characters with the same length are the timestamp character string. +> d. Because there may still be a decoding statement subsequently, a 1-byte letter **P** or **F** is used as a separator between statements. **P** indicates that there are still decoded statements in this batch, and **F** indicates that this batch is completed. +> 5. If the letter described in 3 is **C**: +> a. (Optional) If the next 1-byte letter is **X**, the following eight bytes (uint64) indicate XID. +> b. (Optional) If the next 1-byte letter is **T**, the following four bytes (uint32) indicate the timestamp length. The following characters with the same length are the timestamp character string. +> c. When logs are sent in batches, decoding results of other transactions may still exist after a COMMIT log is decoded. If the next 1-byte letter is **P**, the batch still needs to be decoded. If the letter is **F**, the batch decoding ends. +> 6. If the letter described in 3 is **I**, **U**, or **D**: +> a. The following two bytes (uint16) indicate the length of the schema name. +> b. The schema name is read based on the preceding length. +> c. The following two bytes (uint16) indicate the length of the table name. +> d. The table name is read based on the preceding length. +> e. (Optional) If the next 1-byte letter is **N**, it indicates a new tuple. If the letter is **O**, it indicates an old tuple. In this case, the new tuple is sent first. +> i. The following two bytes (uint16) indicate the number of columns to be decoded for the tuple, which is recorded as **attrnum**. +> ii. The following procedure is repeated for *attrnum* times. +> 1). The next two bytes (uint16) indicate the length of the column name. +> 2). The column name is read based on the preceding length. +> 3). The following four bytes (uint32) indicate the OID of the current column type. +> 4). The next four bytes (uint32) indicate the length of the value (stored in the character string format) in the current column. If the value is **0xFFFFFFFF**, it indicates null. If the value is **0**, it indicates a character string whose length is 0. +> 5). The column value is read based on the preceding length. +> f. Because there may still be a decoding statement after, if the next one-byte letter is **P**, it indicates that the batch still needs to be decoded, and if the next one-byte letter is **F**, it indicates that decoding of the batch ends. + +1. Decoding only on the standby node + + Configure the **standby-connection** option to specify whether to perform decoding only on the standby node. The value is of the Boolean type (**0** or **1**). The value **true** (or **1**) indicates that only the standby node can be connected for decoding. When the primary node is connected for decoding, an error is reported and the system exits. The value **false** (or **0**) indicates that there is no restriction. The default value is **false** (**0**). + +2. Batch sending + + Configure **batch-sending** to specify whether to send results in batches. The value is an integer ranging from 0 to 1. The value **0** indicates that decoding results are sent one by one. The value **1** indicates that decoding results are sent in batches when the accumulated size of decoding results reaches 1 MB. The default value is **0**. This parameter can be set only during parallel decoding. In the scenario where batch sending is enabled, if the decoding format is 'j' or 't', before each original decoding statement, a uint32 type is added indicating the length of the decoding result (excluding the current uint32 type), and a uint64 type is added, indicating the LSN corresponding to the current decoding result. + +The decoding performance (Xlog consumption) is greater than or equal to 100 MB/s in the following standard parallel decoding scenario: 16-core CPU, 128 GB memory, network bandwidth > 200 MB/s, 10 to 100 columns in a table, 0.1 KB to 1 KB data volume in a single row, DML operations are mainly INSERT operations, the number of statements in a single transaction is less than 4096, **parallel_decode_num** is set to **8**, the decoding format is **'b'**, and the batch sending function is enabled. To ensure that the decoding performance meets the requirements and minimize the impact on services, you are advised to set up only one parallel decoding connection on a standby node to ensure that the CPU, memory, and bandwidth resources are sufficient. + +```java +//Logical replication function example: file name, LogicalReplicationDemo.java +//Prerequisite: Add the IP address of the JDBC user machine to the database whitelist. Add the following content to pg_hba.conf: +//Assume that the IP address of the JDBC user machine is 10.10.10.10. +//host all all 10.10.10.10/32 sha256 +//host replication all 10.10.10.10/32 sha256 + +import org.postgresql.PGProperty; +import org.postgresql.jdbc.PgConnection; +import org.postgresql.replication.LogSequenceNumber; +import org.postgresql.replication.PGReplicationStream; + +import java.nio.ByteBuffer; +import java.sql.DriverManager; +import java.util.Properties; +import java.util.concurrent.TimeUnit; + +public class LogicalReplicationDemo { + public static void main(String[] args) { + String driver = "org.postgresql.Driver"; + // Configure the IP address and haPort number of the database. By default, the port number is the port number of the connected DN plus 1. + String sourceURL = "jdbc:postgresql://$ip:$port/postgres"; + PgConnection conn = null; + //The default name of the logical replication slot is replication_slot. + //Test mode: Create a logical replication slot. + int TEST_MODE_CREATE_SLOT = 1; + //Test mode: Enable logical replication (The prerequisite is that the logical replication slot already exists). + int TEST_MODE_START_REPL = 2; + //Test mode: Delete a logical replication slot. + int TEST_MODE_DROP_SLOT = 3; + //Enable different test modes. + int testMode = TEST_MODE_START_REPL; + + try { + Class.forName(driver); + } catch (Exception e) { + e.printStackTrace(); + return; + } + + try { + Properties properties = new Properties(); + PGProperty.USER.set(properties, "user"); + PGProperty.PASSWORD.set(properties, "passwd"); + //For logical replication, the following three attributes are mandatory. + PGProperty.ASSUME_MIN_SERVER_VERSION.set(properties, "9.4"); + PGProperty.REPLICATION.set(properties, "database"); + PGProperty.PREFER_QUERY_MODE.set(properties, "simple"); + conn = (PgConnection) DriverManager.getConnection(sourceURL, properties); + System.out.println("connection success!"); + + if(testMode == TEST_MODE_CREATE_SLOT){ + conn.getReplicationAPI() + .createReplicationSlot() + .logical() + .withSlotName("replication_slot") + .withOutputPlugin("mppdb_decoding") + .make(); + }else if(testMode == TEST_MODE_START_REPL) { + //Create a replication slot before enabling this mode. + LogSequenceNumber waitLSN = LogSequenceNumber.valueOf("6F/E3C53568"); + PGReplicationStream stream = conn + .getReplicationAPI() + .replicationStream() + .logical() + .withSlotName("replication_slot") + .withSlotOption("include-xids", false) + .withSlotOption("skip-empty-xacts", true) + .withStartPosition(waitLSN) + .withSlotOption("parallel-decode-num", 10) // Decoding thread concurrency + .withSlotOption("white-table-list", "public.t1,public.t2") // Whitelist + .withSlotOption("standby-connection", true) // Forcible standby decoding + .withSlotOption("decode-style", "t") // Decoding format + .withSlotOption("sending-bacth", 1) // Sending decoding results in batches + .start(); + while (true) { + ByteBuffer byteBuffer = stream.readPending(); + + if (byteBuffer == null) { + TimeUnit.MILLISECONDS.sleep(10L); + continue; + } + + int offset = byteBuffer.arrayOffset(); + byte[] source = byteBuffer.array(); + int length = source.length - offset; + System.out.println(new String(source, offset, length)); + + //If the LSN needs to be flushed, call the following APIs based on the service requirements: + //LogSequenceNumber lastRecv = stream.getLastReceiveLSN(); + //stream.setFlushedLSN(lastRecv); + //stream.forceUpdateStatus(); + + } + }else if(testMode == TEST_MODE_DROP_SLOT){ + conn.getReplicationAPI() + .dropReplicationSlot("replication_slot"); + } + } catch (Exception e) { + e.printStackTrace(); + return; + } + } +} +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/14.1-example-parameters-for-connecting-to-the-database-in-different-scenarios.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/14.1-example-parameters-for-connecting-to-the-database-in-different-scenarios.md new file mode 100644 index 0000000000000000000000000000000000000000..a1b72501a1ef1b30e20048e9e4fae10e474b363e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/14.1-example-parameters-for-connecting-to-the-database-in-different-scenarios.md @@ -0,0 +1,79 @@ +--- +title: Parameters for Connecting to the Database in Different Scenarios +summary: Parameters for Connecting to the Database in Different Scenarios +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Example: Parameters for Connecting to the Database in Different Scenarios + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** In the following example, **host:port** represents a node, where **host** indicates the name or IP address of the server where the database resides, and **port** indicates the port number of the server where the database resides. + +## DR + +A customer has two database instances. Database instance A is the production database instance, and database instance B is the DR database instance. When the customer performs a DR switchover, database instance A is demoted to the DR database instance, and database instance B is promoted the production database instance. In this case, to avoid application restart or re-release caused by modifications on the configuration file, the customer can write database instances A and B to the connection string when initializing the configuration file. If the primary database instance cannot be connected, the driver attempts to connect to the DR database instance. For example, database instance A consists of *node1*, *node2*, and *node3*, and database instance B consists of *node4*, *node5*, and *node6*. + +The URL can be configured as follows: + +``` +jdbc:opengauss://node1,node2,node3,node4,node5,node6/database?priorityServers=3 +``` + +## Load Balancing + +A customer has a centralized database instance that consists of one primary node and two standby nodes, that is, *node1*, *node2*, and *node3*. *node1* is the primary node, and *node2* and *node3* are the standby nodes. + +If the customer wants to evenly distribute the connections established on the same application to three nodes, the URL can be configured as follows: + +``` +jdbc:opengauss://node1,node2,node3/database?loadBalanceHosts=true +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** When **loadBalanceHosts** is used, if the connection is established on the standby DN, write operations cannot be performed. If read and write operations are required, do not set this parameter. + +## Log Diagnosis + +If a customer encounters slow data import or some errors that are difficult to analyze, the trace log function can be enabled for diagnosis. The URL can be configured as follows: + +``` +jdbc:opengauss://node1/database?loggerLevel=trace&loggerFile=jdbc.log +``` + +## High Performance + +A customer may execute the same SQL statement for multiple times with different input parameters. To improve the execution efficiency, the **prepareThreshold** parameter can be enabled to avoid repeatedly generating execution plans. The URL can be configured as follows: + +``` +jdbc:opengauss://node1/database?prepareThreshold=5 +``` + +A customer queries 10 million data records at a time. To prevent memory overflow caused by simultaneous return of the data records, the **defaultRowFetchSize** parameter can be used. The URL can be configured as follows: + +``` +jdbc:opengauss://node1/database?defaultRowFetchSize=50000 +``` + +A customer needs to insert 10 million data records in batches. To improve efficiency, the **batchMode** parameter can be used. The URL can be configured as follows: + +``` +jdbc:opengauss://node1/database?batchMode=true +``` + +## Case Conversion + +In the Oracle database, metadata is stored in uppercase letters by default. In the GaussDB, metadata is stored in lowercase letters by default. Therefore, after the metadata is migrated from Oracle to GaussDB, the uppercase letters changes to lowercase letters. If the original service involves the processing of uppercase metadata, you can enable this parameter. However, you are advised to modify the service code instead of using this method to solve the problem. If you have to use this function, ensure that the metadata in the current database is in lowercase to avoid problems. + +```java +jdbc:postgresql://node1/database?uppercaseAttributeName=true +``` + +The APIs involved in DatabaseMetaData can be directly invoked based on input parameters. The methods of using the APIs involved in ResultSetMetaData are as follows: + +```java +Statement stmt = conn.createStatement(); +ResultSet rs = stmt.executeQuery("select * from test_supper"); +ResultSetMetaData rsmd = rs.getMetaData(); +for (int i = 1; i <= rsmd.getColumnCount(); i++) { + System.out.println(rsmd.getColumnLabel(i) + " " + rsmd.getColumnName(i)); +} +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/1-java-sql-Connection.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/1-java-sql-Connection.md new file mode 100644 index 0000000000000000000000000000000000000000..0598e5efa995bda8f5c6978c869fcc29a70a63f8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/1-java-sql-Connection.md @@ -0,0 +1,67 @@ +--- +title: java.sql.Connection +summary: java.sql.Connection +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.Connection + +This section describes **java.sql.Connection**, the interface for connecting to a database. + +**Table 1** Support status for java.sql.Connection + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :----------------------------------------------------------- | :------------------------------- | :------------------------- | +| abort(Executor executor) | void | Yes | +| clearWarnings() | void | Yes | +| close() | void | Yes | +| commit() | void | Yes | +| createArrayOf(String typeName, Object[] elements) | Array | Yes | +| createBlob() | Blob | Yes | +| createClob() | Clob | Yes | +| createSQLXML() | SQLXML | Yes | +| createStatement() | Statement | Yes | +| createStatement(int resultSetType, int resultSetConcurrency) | Statement | Yes | +| createStatement(int resultSetType, int resultSetConcurrency, int resultSetHoldability) | Statement | Yes | +| getAutoCommit() | Boolean | Yes | +| getCatalog() | String | Yes | +| getClientInfo() | Properties | Yes | +| getClientInfo(String name) | String | Yes | +| getHoldability() | int | Yes | +| getMetaData() | DatabaseMetaData | Yes | +| getNetworkTimeout() | int | Yes | +| getSchema() | String | Yes | +| getTransactionIsolation() | int | Yes | +| getTypeMap() | Map<String,Class<?>> | Yes | +| getWarnings() | SQLWarning | Yes | +| isClosed() | Boolean | Yes | +| isReadOnly() | Boolean | Yes | +| isValid(int timeout) | boolean | Yes | +| nativeSQL(String sql) | String | Yes | +| prepareCall(String sql) | CallableStatement | Yes | +| prepareCall(String sql, int resultSetType, int resultSetConcurrency) | CallableStatement | Yes | +| prepareCall(String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) | CallableStatement | Yes | +| prepareStatement(String sql) | PreparedStatement | Yes | +| prepareStatement(String sql, int autoGeneratedKeys) | PreparedStatement | Yes | +| prepareStatement(String sql, int[] columnIndexes) | PreparedStatement | Yes | +| prepareStatement(String sql, int resultSetType, int resultSetConcurrency) | PreparedStatement | Yes | +| prepareStatement(String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) | PreparedStatement | Yes | +| prepareStatement(String sql, String[] columnNames) | PreparedStatement | Yes | +| releaseSavepoint(Savepoint savepoint) | void | Yes | +| rollback() | void | Yes | +| rollback(Savepoint savepoint) | void | Yes | +| setAutoCommit(boolean autoCommit) | void | Yes | +| setClientInfo(Properties properties) | void | Yes | +| setClientInfo(String name,String value) | void | Yes | +| setHoldability(int holdability) | void | Yes | +| setNetworkTimeout(Executor executor, int milliseconds) | void | Yes | +| setReadOnly(boolean readOnly) | void | Yes | +| setSavepoint() | Savepoint | Yes | +| setSavepoint(String name) | Savepoint | Yes | +| setSchema(String schema) | void | Yes | +| setTransactionIsolation(int level) | void | Yes | +| setTypeMap(Map<String,Class<?>> map) | void | Yes | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> The AutoCommit mode is used by default within the interface. If you disable it by running **setAutoCommit(false)**, all the statements executed later will be packaged in explicit transactions, and you cannot execute statements that cannot be executed within transactions. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/10-javax-sql-DataSource.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/10-javax-sql-DataSource.md new file mode 100644 index 0000000000000000000000000000000000000000..0aeccd036e7e7b833f90f25bc62ea06c3abed656 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/10-javax-sql-DataSource.md @@ -0,0 +1,21 @@ +--- +title: javax.sql.DataSource +summary: javax.sql.DataSource +author: Guo Huan +date: 2021-05-17 +--- + +# javax.sql.DataSource + +This section describes **javax.sql.DataSource**, the interface for data sources. + +**Table 1** Support status for javax.sql.DataSource + +| Method Name | Return Type | Support JDBC 4 | +| :--------------------------------------------- | :---------- | :------------- | +| getConneciton() | Connection | Yes | +| getConnection(String username,String password) | Connection | Yes | +| getLoginTimeout() | int | Yes | +| getLogWriter() | PrintWriter | Yes | +| setLoginTimeout(int seconds) | void | Yes | +| setLogWriter(PrintWriter out) | void | Yes | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/11-javax-sql-PooledConnection.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/11-javax-sql-PooledConnection.md new file mode 100644 index 0000000000000000000000000000000000000000..b1f4004dc0fcf426efcfc75bf4d586cac2d2ebb3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/11-javax-sql-PooledConnection.md @@ -0,0 +1,19 @@ +--- +title: javax.sql.PooledConnection +summary: javax.sql.PooledConnection +author: Guo Huan +date: 2021-05-17 +--- + +# javax.sql.PooledConnection + +This section describes **javax.sql.PooledConnection**, the connection interface created by a connection pool. + +**Table 1** Support status for javax.sql.PooledConnection + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :----------------------------------------------------------- | :---------- | :------------------------- | +| addConnectionEventListener (ConnectionEventListener listener) | void | Yes | +| close() | void | Yes | +| getConnection() | Connection | Yes | +| removeConnectionEventListener (ConnectionEventListener listener) | void | Yes | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/12-javax-naming-Context.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/12-javax-naming-Context.md new file mode 100644 index 0000000000000000000000000000000000000000..26da1f5a0b5d258855b70ea3967d650844c19c6c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/12-javax-naming-Context.md @@ -0,0 +1,25 @@ +--- +title: javax.naming.Context +summary: javax.naming.Context +author: Guo Huan +date: 2021-05-17 +--- + +# javax.naming.Context + +This section describes **javax.naming.Context**, the context interface for connection configuration. + +**Table 1** Support status for javax.naming.Context + +| Method Name | Return Type | Support JDBC 4 | +| :------------------------------------- | :---------- | :------------- | +| bind(Name name, Object obj) | void | Yes | +| bind(String name, Object obj) | void | Yes | +| lookup(Name name) | Object | Yes | +| lookup(String name) | Object | Yes | +| rebind(Name name, Object obj) | void | Yes | +| rebind(String name, Object obj) | void | Yes | +| rename(Name oldName, Name newName) | void | Yes | +| rename(String oldName, String newName) | void | Yes | +| unbind(Name name) | void | Yes | +| unbind(String name) | void | Yes | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/13-javax-naming-spi-InitialContextFactory.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/13-javax-naming-spi-InitialContextFactory.md new file mode 100644 index 0000000000000000000000000000000000000000..ef15dfccd645fedeb9a5175528da0703ebf2a40a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/13-javax-naming-spi-InitialContextFactory.md @@ -0,0 +1,16 @@ +--- +title: javax.naming.spi.InitialContextFactory +summary: javax.naming.spi.InitialContextFactory +author: Guo Huan +date: 2021-05-17 +--- + +# javax.naming.spi.InitialContextFactory + +This section describes **javax.naming.spi.InitialContextFactory**, the initial context factory interface. + +**Table 1** Support status for javax.naming.spi.InitialContextFactory + +| Method Name | Return Type | Support JDBC 4 | +| :-------------------------------------------------- | :---------- | :------------- | +| getInitialContext(Hashtable<?,?> environment) | Context | Yes | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/14-CopyManager.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/14-CopyManager.md new file mode 100644 index 0000000000000000000000000000000000000000..50a3214953a4294cab75dd257d66492c4a3550e5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/14-CopyManager.md @@ -0,0 +1,40 @@ +--- +title: CopyManager +summary: CopyManager +author: Guo Huan +date: 2021-05-17 +--- + +# CopyManager + +CopyManager is an API class provided by the JDBC driver in MogDB. It is used to import data to MogDB in batches. + +## Inheritance Relationship of CopyManager + +The CopyManager class is in the **org.opengauss.copy** package and inherits the java.lang.Object class. The declaration of the class is as follows: + +```java +public class CopyManager +extends Object +``` + +## Construction Method + +public CopyManager(BaseConnection connection) + +throws SQLException + +## Common Methods + +**Table 1** Common methods of CopyManager + +| Return Value | Method | Description | throws | +| :----------------------- | :------------------- | :------------------- | :------------------- | +| CopyIn | copyIn(String sql) | - | SQLException | +| long | copyIn(String sql, InputStream from) | Uses **COPY FROM STDIN** to quickly load data to tables in the database from InputStream. | SQLException,IOException | +| long | copyIn(String sql, InputStream from, int bufferSize) | Uses **COPY FROM STDIN** to quickly load data to tables in the database from InputStream. | SQLException,IOException | +| long | copyIn(String sql, Reader from) | Uses **COPY FROM STDIN** to quickly load data to tables in the database from Reader. | SQLException,IOException | +| long | copyIn(String sql, Reader from, int bufferSize) | Uses **COPY FROM STDIN** to quickly load data to tables in the database from Reader. | SQLException,IOException | +| CopyOut | copyOut(String sql) | - | SQLException | +| long | copyOut(String sql, OutputStream to) | Sends the result set of **COPY TO STDOUT** from the database to the OutputStream class. | SQLException,IOException | +| long | copyOut(String sql, Writer to) | Sends the result set of **COPY TO STDOUT** from the database to the Writer class. | SQLException,IOException | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/2-java-sql-CallableStatement.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/2-java-sql-CallableStatement.md new file mode 100644 index 0000000000000000000000000000000000000000..dc493d97a52c8b9ae60e13d2c77609c5e530a108 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/2-java-sql-CallableStatement.md @@ -0,0 +1,46 @@ +--- +title: java.sql.CallableStatement +summary: java.sql.CallableStatement +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.CallableStatement + +This section describes **java.sql.CallableStatement**, the interface for executing the stored procedure. + +**Table 1** Support status for java.sql.CallableStatement + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :------------------------------------------------- | :---------- | :------------------------- | +| getArray(int parameterIndex) | Array | Yes | +| getBigDecimal(int parameterIndex) | BigDecimal | Yes | +| getBlob(int parameterIndex) | Blob | Yes | +| getBoolean(int parameterIndex) | boolean | Yes | +| getByte(int parameterIndex) | byte | Yes | +| getBytes(int parameterIndex) | byte[] | Yes | +| getClob(int parameterIndex) | Clob | Yes | +| getDate(int parameterIndex) | Date | Yes | +| getDate(int parameterIndex, Calendar cal) | Date | Yes | +| getDouble(int parameterIndex) | double | Yes | +| getFloat(int parameterIndex) | float | Yes | +| getInt(int parameterIndex) | int | Yes | +| getLong(int parameterIndex) | long | Yes | +| getObject(int parameterIndex) | Object | Yes | +| getObject(int parameterIndex, Class<T> type) | Object | Yes | +| getShort(int parameterIndex) | short | Yes | +| getSQLXML(int parameterIndex) | SQLXML | Yes | +| getString(int parameterIndex) | String | Yes | +| getNString(int parameterIndex) | String | Yes | +| getTime(int parameterIndex) | Time | Yes | +| getTime(int parameterIndex, Calendar cal) | Time | Yes | +| getTimestamp(int parameterIndex) | Timestamp | Yes | +| getTimestamp(int parameterIndex, Calendar cal) | Timestamp | Yes | +| registerOutParameter(int parameterIndex, int type) | void | Yes | +| wasNull() | Boolean | Yes | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - The batch operation of statements containing OUT parameter is not allowed. +> - The following methods are inherited from java.sql.Statement: close, execute, executeQuery, executeUpdate, getConnection, getResultSet, getUpdateCount, isClosed, setMaxRows, and setFetchSize. +> - The following methods are inherited from java.sql.PreparedStatement: addBatch, clearParameters, execute, executeQuery, executeUpdate, getMetaData, setBigDecimal, setBoolean, setByte, setBytes, setDate, setDouble, setFloat, setInt, setLong, setNull, setObject, setString, setTime, and setTimestamp. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/3-java-sql-DatabaseMetaData.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/3-java-sql-DatabaseMetaData.md new file mode 100644 index 0000000000000000000000000000000000000000..be9b8811e8f2d06ebaca8476e18f17186509e678 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/3-java-sql-DatabaseMetaData.md @@ -0,0 +1,213 @@ +--- +title: java.sql.DatabaseMetaData +summary: java.sql.DatabaseMetaData +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.DatabaseMetaData + +This section describes **java.sql.DatabaseMetaData**, the interface for defining database objects. + +**Table 1** Support status for java.sql.DatabaseMetaData + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :----------------------------------------------------------- | :----------- | :------------------------- | +| allProceduresAreCallable() | boolean | Yes | +| allTablesAreSelectable() | boolean | Yes | +| autoCommitFailureClosesAllResultSets() | boolean | Yes | +| dataDefinitionCausesTransactionCommit() | boolean | Yes | +| dataDefinitionIgnoredInTransactions() | boolean | Yes | +| deletesAreDetected(int type) | boolean | Yes | +| doesMaxRowSizeIncludeBlobs() | boolean | Yes | +| generatedKeyAlwaysReturned() | boolean | Yes | +| getBestRowIdentifier(String catalog, String schema, String table, int scope, boolean nullable) | ResultSet | Yes | +| getCatalogs() | ResultSet | Yes | +| getCatalogSeparator() | String | Yes | +| getCatalogTerm() | String | Yes | +| getClientInfoProperties() | ResultSet | Yes | +| getColumnPrivileges(String catalog, String schema, String table, String columnNamePattern) | ResultSet | Yes | +| getConnection() | Connection | Yes | +| getCrossReference(String parentCatalog, String parentSchema, String parentTable, String foreignCatalog, String foreignSchema, String foreignTable) | ResultSet | Yes | +| getDefaultTransactionIsolation() | int | Yes | +| getExportedKeys(String catalog, String schema, String table) | ResultSet | Yes | +| getExtraNameCharacters() | String | Yes | +| getFunctionColumns(String catalog, String schemaPattern, String functionNamePattern, String columnNamePattern) | ResultSet | Yes | +| getFunctions(String catalog, String schemaPattern, String functionNamePattern) | ResultSet | Yes | +| getIdentifierQuoteString() | String | Yes | +| getImportedKeys(String catalog, String schema, String table) | ResultSet | Yes | +| getIndexInfo(String catalog, String schema, String table, boolean unique, boolean approximate) | ResultSet | Yes | +| getMaxBinaryLiteralLength() | int | Yes | +| getMaxCatalogNameLength() | int | Yes | +| getMaxCharLiteralLength() | int | Yes | +| getMaxColumnNameLength() | int | Yes | +| getMaxColumnsInGroupBy() | int | Yes | +| getMaxColumnsInIndex() | int | Yes | +| getMaxColumnsInOrderBy() | int | Yes | +| getMaxColumnsInSelect() | int | Yes | +| getMaxColumnsInTable() | int | Yes | +| getMaxConnections() | int | Yes | +| getMaxCursorNameLength() | int | Yes | +| getMaxIndexLength() | int | Yes | +| getMaxLogicalLobSize() | default long | Yes | +| getMaxProcedureNameLength() | int | Yes | +| getMaxRowSize() | int | Yes | +| getMaxSchemaNameLength() | int | Yes | +| getMaxStatementLength() | int | Yes | +| getMaxStatements() | int | Yes | +| getMaxTableNameLength() | int | Yes | +| getMaxTablesInSelect() | int | Yes | +| getMaxUserNameLength() | int | Yes | +| getNumericFunctions() | String | Yes | +| getPrimaryKeys(String catalog, String schema, String table) | ResultSet | Yes | +| getPartitionTablePrimaryKeys(String catalog, String schema, String table) | ResultSet | Yes | +| getProcedureColumns(String catalog, String schemaPattern, String procedureNamePattern, String columnNamePattern) | ResultSet | Yes | +| getProcedures(String catalog, String schemaPattern, String procedureNamePattern) | ResultSet | Yes | +| getProcedureTerm() | String | Yes | +| getSchemas() | ResultSet | Yes | +| getSchemas(String catalog, String schemaPattern) | ResultSet | Yes | +| getSchemaTerm() | String | Yes | +| getSearchStringEscape() | String | Yes | +| getSQLKeywords() | String | Yes | +| getSQLStateType() | int | Yes | +| getStringFunctions() | String | Yes | +| getSystemFunctions() | String | Yes | +| getTablePrivileges(String catalog, String schemaPattern, String tableNamePattern) | ResultSet | Yes | +| getTimeDateFunctions() | String | Yes | +| getTypeInfo() | ResultSet | Yes | +| getUDTs(String catalog, String schemaPattern, String typeNamePattern, int[] types) | ResultSet | Yes | +| getURL() | String | Yes | +| getVersionColumns(String catalog, String schema, String table) | ResultSet | Yes | +| insertsAreDetected(int type) | boolean | Yes | +| locatorsUpdateCopy() | boolean | Yes | +| othersDeletesAreVisible(int type) | boolean | Yes | +| othersInsertsAreVisible(int type) | boolean | Yes | +| othersUpdatesAreVisible(int type) | boolean | Yes | +| ownDeletesAreVisible(int type) | boolean | Yes | +| ownInsertsAreVisible(int type) | boolean | Yes | +| ownUpdatesAreVisible(int type) | boolean | Yes | +| storesLowerCaseIdentifiers() | boolean | Yes | +| storesMixedCaseIdentifiers() | boolean | Yes | +| storesUpperCaseIdentifiers() | boolean | Yes | +| supportsBatchUpdates() | boolean | Yes | +| supportsCatalogsInDataManipulation() | boolean | Yes | +| supportsCatalogsInIndexDefinitions() | boolean | Yes | +| supportsCatalogsInPrivilegeDefinitions() | boolean | Yes | +| supportsCatalogsInProcedureCalls() | boolean | Yes | +| supportsCatalogsInTableDefinitions() | boolean | Yes | +| supportsCorrelatedSubqueries() | boolean | Yes | +| supportsDataDefinitionAndDataManipulationTransactions() | boolean | Yes | +| supportsDataManipulationTransactionsOnly() | boolean | Yes | +| supportsGetGeneratedKeys() | boolean | Yes | +| supportsMixedCaseIdentifiers() | boolean | Yes | +| supportsMultipleOpenResults() | boolean | Yes | +| supportsNamedParameters() | boolean | Yes | +| supportsOpenCursorsAcrossCommit() | boolean | Yes | +| supportsOpenCursorsAcrossRollback() | boolean | Yes | +| supportsOpenStatementsAcrossCommit() | boolean | Yes | +| supportsOpenStatementsAcrossRollback() | boolean | Yes | +| supportsPositionedDelete() | boolean | Yes | +| supportsPositionedUpdate() | boolean | Yes | +| supportsRefCursors() | boolean | Yes | +| supportsResultSetConcurrency(int type, int concurrency) | boolean | Yes | +| supportsResultSetType(int type) | boolean | Yes | +| supportsSchemasInIndexDefinitions() | boolean | Yes | +| supportsSchemasInPrivilegeDefinitions() | boolean | Yes | +| supportsSchemasInProcedureCalls() | boolean | Yes | +| supportsSchemasInTableDefinitions() | boolean | Yes | +| supportsSelectForUpdate() | boolean | Yes | +| supportsStatementPooling() | boolean | Yes | +| supportsStoredFunctionsUsingCallSyntax() | boolean | Yes | +| supportsStoredProcedures() | boolean | Yes | +| supportsSubqueriesInComparisons() | boolean | Yes | +| supportsSubqueriesInExists() | boolean | Yes | +| supportsSubqueriesInIns() | boolean | Yes | +| supportsSubqueriesInQuantifieds() | boolean | Yes | +| supportsTransactionIsolationLevel(int level) | boolean | Yes | +| supportsTransactions() | boolean | Yes | +| supportsUnion() | boolean | Yes | +| supportsUnionAll() | boolean | Yes | +| updatesAreDetected(int type) | boolean | Yes | +| getTables(String catalog, String schemaPattern, String tableNamePattern, String[] types) | ResultSet | Yes | +| getColumns(String catalog, String schemaPattern, String tableNamePattern, String columnNamePattern) | ResultSet | Yes | +| getTableTypes() | ResultSet | Yes | +| getUserName() | String | Yes | +| isReadOnly() | boolean | Yes | +| nullsAreSortedHigh() | boolean | Yes | +| nullsAreSortedLow() | boolean | Yes | +| nullsAreSortedAtStart() | boolean | Yes | +| nullsAreSortedAtEnd() | boolean | Yes | +| getDatabaseProductName() | String | Yes | +| getDatabaseProductVersion() | String | Yes | +| getDriverName() | String | Yes | +| getDriverVersion() | String | Yes | +| getDriverMajorVersion() | int | Yes | +| getDriverMinorVersion() | int | Yes | +| usesLocalFiles() | boolean | Yes | +| usesLocalFilePerTable() | boolean | Yes | +| supportsMixedCaseIdentifiers() | boolean | Yes | +| storesUpperCaseIdentifiers() | boolean | Yes | +| storesLowerCaseIdentifiers() | boolean | Yes | +| supportsMixedCaseQuotedIdentifiers() | boolean | Yes | +| storesUpperCaseQuotedIdentifiers() | boolean | Yes | +| storesLowerCaseQuotedIdentifiers() | boolean | Yes | +| storesMixedCaseQuotedIdentifiers() | boolean | Yes | +| supportsAlterTableWithAddColumn() | boolean | Yes | +| supportsAlterTableWithDropColumn() | boolean | Yes | +| supportsColumnAliasing() | boolean | Yes | +| nullPlusNonNullIsNull() | boolean | Yes | +| supportsConvert() | boolean | Yes | +| supportsConvert(int fromType, int toType) | boolean | Yes | +| supportsTableCorrelationNames() | boolean | Yes | +| supportsDifferentTableCorrelationNames() | boolean | Yes | +| supportsExpressionsInOrderBy() | boolean | Yes | +| supportsOrderByUnrelated() | boolean | Yes | +| supportsGroupBy() | boolean | Yes | +| supportsGroupByUnrelated() | boolean | Yes | +| supportsGroupByBeyondSelect() | boolean | Yes | +| supportsLikeEscapeClause() | boolean | Yes | +| supportsMultipleResultSets() | boolean | Yes | +| supportsMultipleTransactions() | boolean | Yes | +| supportsNonNullableColumns() | boolean | Yes | +| supportsMinimumSQLGrammar() | boolean | Yes | +| supportsCoreSQLGrammar() | boolean | Yes | +| supportsExtendedSQLGrammar() | boolean | Yes | +| supportsANSI92EntryLevelSQL() | boolean | Yes | +| supportsANSI92IntermediateSQL() | boolean | Yes | +| supportsANSI92FullSQL() | boolean | Yes | +| supportsIntegrityEnhancementFacility() | boolean | Yes | +| supportsOuterJoins() | boolean | Yes | +| supportsFullOuterJoins() | boolean | Yes | +| supportsLimitedOuterJoins() | boolean | Yes | +| isCatalogAtStart() | boolean | Yes | +| supportsSchemasInDataManipulation() | boolean | Yes | +| supportsSavepoints() | boolean | Yes | +| supportsResultSetHoldability(int holdability) | boolean | Yes | +| getResultSetHoldability() | int | Yes | +| getDatabaseMajorVersion() | int | Yes | +| getDatabaseMinorVersion() | int | Yes | +| getJDBCMajorVersion() | int | Yes | +| getJDBCMinorVersion() | int | Yes | + +> **NOTE:** If the value of **uppercaseAttributeName** is **true**, the following APIs convert the query result to uppercase letters. The conversion range is the same as that of the **toUpperCase** method in Java. +> +> - public ResultSet getProcedures(String catalog, String schemaPattern, String procedureNamePattern) +> - public ResultSet getProcedureColumns(String catalog, String schemaPattern, String procedureNamePattern, String columnNamePattern) +> - public ResultSet getTables(String catalog, String schemaPattern, String tableNamePattern, String[] types) +> - public ResultSet getSchemas(String catalog, String schemaPattern) +> - public ResultSet getColumns(String catalog, String schemaPattern, String tableNamePattern, String columnNamePattern) +> - public ResultSet getColumnPrivileges(String catalog, String schema, String table, String columnNamePattern) +> - public ResultSet getTablePrivileges(String catalog, String schemaPattern, String tableNamePattern) +> - public ResultSet getBestRowIdentifier(String catalog, String schema, String table, int scope, boolean nullable) +> - public ResultSet getPrimaryKeys(String catalog, String schema, String table) +> - protected ResultSet getImportedExportedKeys(String primaryCatalog, String primarySchema, String primaryTable, String foreignCatalog, String foreignSchema, String foreignTable) +> - public ResultSet getIndexInfo(String catalog, String schema, String tableName, boolean unique, boolean approximate) +> - public ResultSet getUDTs(String catalog, String schemaPattern, String typeNamePattern, int[] types) +> - public ResultSet getFunctions(String catalog, String schemaPattern, String functionNamePattern) + +> **CAUTION:** The **getPartitionTablePrimaryKeys(String catalog, String schema, String table)** API is used to obtain the primary key column of a partitioned table that contains global indexes. The following is an example: +> +> ``` +> PgDatabaseMetaData dbmd = (PgDatabaseMetaData)conn.getMetaData(); +> dbmd.getPartitionTablePrimaryKeys("catalogName", "schemaName", "tableName"); +> ``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/4-java-sql-Driver.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/4-java-sql-Driver.md new file mode 100644 index 0000000000000000000000000000000000000000..cb99815436e25f30ee33cab95079467f9aab0a14 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/4-java-sql-Driver.md @@ -0,0 +1,22 @@ +--- +title: java.sql.Driver +summary: java.sql.Driver +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.Driver + +This section describes **java.sql.Driver**, the database driver interface. + +**Table 1** Support status for java.sql.Driver + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :------------------------------------------- | :------------------- | :------------------------- | +| acceptsURL(String url) | Boolean | Yes | +| connect(String url, Properties info) | Connection | Yes | +| jdbcCompliant() | Boolean | Yes | +| getMajorVersion() | int | Yes | +| getMinorVersion() | int | Yes | +| getParentLogger() | Logger | Yes | +| getPropertyInfo(String url, Properties info) | DriverPropertyInfo[] | Yes | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/5-java-sql-PreparedStatement.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/5-java-sql-PreparedStatement.md new file mode 100644 index 0000000000000000000000000000000000000000..50091489a626e20b507a6220852ed0e968204686 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/5-java-sql-PreparedStatement.md @@ -0,0 +1,70 @@ +--- +title: java.sql.PreparedStatement +summary: java.sql.PreparedStatement +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.PreparedStatement + +This section describes **java.sql.PreparedStatement**, the interface for preparing statements. + +**Table 1** Support status for java.sql.PreparedStatement + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :----------------------------------------------------------- | :---------------- | :------------------------- | +| clearParameters() | void | Yes | +| execute() | Boolean | Yes | +| executeQuery() | ResultSet | Yes | +| excuteUpdate() | int | Yes | +| executeLargeUpdate() | long | No | +| getMetaData() | ResultSetMetaData | Yes | +| getParameterMetaData() | ParameterMetaData | Yes | +| setArray(int parameterIndex, Array x) | void | Yes | +| setAsciiStream(int parameterIndex, InputStream x, int length) | void | Yes | +| setBinaryStream(int parameterIndex, InputStream x) | void | Yes | +| setBinaryStream(int parameterIndex, InputStream x, int length) | void | Yes | +| setBinaryStream(int parameterIndex, InputStream x, long length) | void | Yes | +| setBlob(int parameterIndex, InputStream inputStream) | void | Yes | +| setBlob(int parameterIndex, InputStream inputStream, long length) | void | Yes | +| setBlob(int parameterIndex, Blob x) | void | Yes | +| setCharacterStream(int parameterIndex, Reader reader) | void | Yes | +| setCharacterStream(int parameterIndex, Reader reader, int length) | void | Yes | +| setClob(int parameterIndex, Reader reader) | void | Yes | +| setClob(int parameterIndex, Reader reader, long length) | void | Yes | +| setClob(int parameterIndex, Clob x) | void | Yes | +| setDate(int parameterIndex, Date x, Calendar cal) | void | Yes | +| setNull(int parameterIndex, int sqlType) | void | Yes | +| setNull(int parameterIndex, int sqlType, String typeName) | void | Yes | +| setObject(int parameterIndex, Object x) | void | Yes | +| setObject(int parameterIndex, Object x, int targetSqlType) | void | Yes | +| setObject(int parameterIndex, Object x, int targetSqlType, int scaleOrLength) | void | Yes | +| setSQLXML(int parameterIndex, SQLXML xmlObject) | void | Yes | +| setTime(int parameterIndex, Time x) | void | Yes | +| setTime(int parameterIndex, Time x, Calendar cal) | void | Yes | +| setTimestamp(int parameterIndex, Timestamp x) | void | Yes | +| setTimestamp(int parameterIndex, Timestamp x, Calendar cal) | void | Yes | +| setUnicodeStream(int parameterIndex, InputStream x, int length) | void | Yes | +| setURL(int parameterIndex, URL x) | void | Yes | +| setBoolean(int parameterIndex, boolean x) | void | Yes | +| setBigDecimal(int parameterIndex, BigDecimal x) | void | Yes | +| setByte(int parameterIndex, byte x) | void | Yes | +| setBytes(int parameterIndex, byte[] x) | void | Yes | +| setDate(int parameterIndex, Date x) | void | Yes | +| setDouble(int parameterIndex, double x) | void | Yes | +| setFloat(int parameterIndex, float x) | void | Yes | +| setInt(int parameterIndex, int x) | void | Yes | +| setLong(int parameterIndex, long x) | void | Yes | +| setShort(int parameterIndex, short x) | void | Yes | +| setString(int parameterIndex, String x) | void | Yes | +| setNString(int parameterIndex, String x) | void | Yes | +| addBatch() | void | Yes | +| executeBatch() | int[] | Yes | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - Execute addBatch() and execute() only after running clearBatch(). +> - Batch is not cleared by calling executeBatch(). Clear batch by explicitly calling clearBatch(). +> - After bounded variables of a batch are added, if you want to reuse these values (add a batch again), set*() is not necessary. +> - The following methods are inherited from java.sql.Statement: close, execute, executeQuery, executeUpdate, getConnection, getResultSet, getUpdateCount, isClosed, setMaxRows, and setFetchSize. +> - The **executeLargeUpdate()** method can only be used in JDBC 4.2 or later. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/6-java-sql-ResultSet.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/6-java-sql-ResultSet.md new file mode 100644 index 0000000000000000000000000000000000000000..390cb97bc47eca24ad83ed1a653e4f4f7602d0a1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/6-java-sql-ResultSet.md @@ -0,0 +1,154 @@ +--- +title: java.sql.ResultSet +summary: java.sql.ResultSet +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.ResultSet + +This section describes **java.sql.ResultSet**, the interface for execution result sets. + +**Table 1** Support status for java.sql.ResultSet + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :----------------------------------------------------------- | :---------------- | :------------------------- | +| absolute(int row) | Boolean | Yes | +| afterLast() | void | Yes | +| beforeFirst() | void | Yes | +| cancelRowUpdates() | void | Yes | +| clearWarnings() | void | Yes | +| close() | void | Yes | +| deleteRow() | void | Yes | +| findColumn(String columnLabel) | int | Yes | +| first() | Boolean | Yes | +| getArray(int columnIndex) | Array | Yes | +| getArray(String columnLabel) | Array | Yes | +| getAsciiStream(int columnIndex) | InputStream | Yes | +| getAsciiStream(String columnLabel) | InputStream | Yes | +| getBigDecimal(int columnIndex) | BigDecimal | Yes | +| getBigDecimal(String columnLabel) | BigDecimal | Yes | +| getBinaryStream(int columnIndex) | InputStream | Yes | +| getBinaryStream(String columnLabel) | InputStream | Yes | +| getBlob(int columnIndex) | Blob | Yes | +| getBlob(String columnLabel) | Blob | Yes | +| getBoolean(int columnIndex) | Boolean | Yes | +| getBoolean(String columnLabel) | Boolean | Yes | +| getByte(int columnIndex) | byte | Yes | +| getBytes(int columnIndex) | byte[] | Yes | +| getByte(String columnLabel) | byte | Yes | +| getBytes(String columnLabel) | byte[] | Yes | +| getCharacterStream(int columnIndex) | Reader | Yes | +| getCharacterStream(String columnLabel) | Reader | Yes | +| getClob(int columnIndex) | Clob | Yes | +| getClob(String columnLabel) | Clob | Yes | +| getConcurrency() | int | Yes | +| getCursorName() | String | Yes | +| getDate(int columnIndex) | Date | Yes | +| getDate(int columnIndex, Calendar cal) | Date | Yes | +| getDate(String columnLabel) | Date | Yes | +| getDate(String columnLabel, Calendar cal) | Date | Yes | +| getDouble(int columnIndex) | double | Yes | +| getDouble(String columnLabel) | double | Yes | +| getFetchDirection() | int | Yes | +| getFetchSize() | int | Yes | +| getFloat(int columnIndex) | float | Yes | +| getFloat(String columnLabel) | float | Yes | +| getInt(int columnIndex) | int | Yes | +| getInt(String columnLabel) | int | Yes | +| getLong(int columnIndex) | long | Yes | +| getLong(String columnLabel) | long | Yes | +| getMetaData() | ResultSetMetaData | Yes | +| getObject(int columnIndex) | Object | Yes | +| getObject(int columnIndex, Class<T> type) | <T> T | Yes | +| getObject(int columnIndex, Map<String,Class<?>> map) | Object | Yes | +| getObject(String columnLabel) | Object | Yes | +| getObject(String columnLabel, Class<T> type) | <T> T | Yes | +| getObject(String columnLabel, Map<String,Class<?>> map) | Object | Yes | +| getRow() | int | Yes | +| getShort(int columnIndex) | short | Yes | +| getShort(String columnLabel) | short | Yes | +| getSQLXML(int columnIndex) | SQLXML | Yes | +| getSQLXML(String columnLabel) | SQLXML | Yes | +| getStatement() | Statement | Yes | +| getString(int columnIndex) | String | Yes | +| getString(String columnLabel) | String | Yes | +| getNString(int columnIndex) | String | Yes | +| getNString(String columnLabel) | String | Yes | +| getTime(int columnIndex) | Time | Yes | +| getTime(int columnIndex, Calendar cal) | Time | Yes | +| getTime(String columnLabel) | Time | Yes | +| getTime(String columnLabel, Calendar cal) | Time | Yes | +| getTimestamp(int columnIndex) | Timestamp | Yes | +| getTimestamp(int columnIndex, Calendar cal) | Timestamp | Yes | +| getTimestamp(String columnLabel) | Timestamp | Yes | +| getTimestamp(String columnLabel, Calendar cal) | Timestamp | Yes | +| getType() | int | Yes | +| getWarnings() | SQLWarning | Yes | +| insertRow() | void | Yes | +| isAfterLast() | Boolean | Yes | +| isBeforeFirst() | Boolean | Yes | +| isClosed() | Boolean | Yes | +| isFirst() | Boolean | Yes | +| isLast() | Boolean | Yes | +| last() | Boolean | Yes | +| moveToCurrentRow() | void | Yes | +| moveToInsertRow() | void | Yes | +| next() | Boolean | Yes | +| previous() | Boolean | Yes | +| refreshRow() | void | Yes | +| relative(int rows) | Boolean | Yes | +| rowDeleted() | Boolean | Yes | +| rowInserted() | Boolean | Yes | +| rowUpdated() | Boolean | Yes | +| setFetchDirection(int direction) | void | Yes | +| setFetchSize(int rows) | void | Yes | +| updateArray(int columnIndex, Array x) | void | Yes | +| updateArray(String columnLabel, Array x) | void | Yes | +| updateAsciiStream(int columnIndex, InputStream x, int length) | void | Yes | +| updateAsciiStream(String columnLabel, InputStream x, int length) | void | Yes | +| updateBigDecimal(int columnIndex, BigDecimal x) | void | Yes | +| updateBigDecimal(String columnLabel, BigDecimal x) | void | Yes | +| updateBinaryStream(int columnIndex, InputStream x, int length) | void | Yes | +| updateBinaryStream(String columnLabel, InputStream x, int length) | void | Yes | +| updateBoolean(int columnIndex, boolean x) | void | Yes | +| updateBoolean(String columnLabel, boolean x) | void | Yes | +| updateByte(int columnIndex, byte x) | void | Yes | +| updateByte(String columnLabel, byte x) | void | Yes | +| updateBytes(int columnIndex, byte[] x) | void | Yes | +| updateBytes(String columnLabel, byte[] x) | void | Yes | +| updateCharacterStream(int columnIndex, Reader x, int length) | void | Yes | +| updateCharacterStream(String columnLabel, Reader reader, int length) | void | Yes | +| updateDate(int columnIndex, Date x) | void | Yes | +| updateDate(String columnLabel, Date x) | void | Yes | +| updateDouble(int columnIndex, double x) | void | Yes | +| updateDouble(String columnLabel, double x) | void | Yes | +| updateFloat(int columnIndex, float x) | void | Yes | +| updateFloat(String columnLabel, float x) | void | Yes | +| updateInt(int columnIndex, int x) | void | Yes | +| updateInt(String columnLabel, int x) | void | Yes | +| updateLong(int columnIndex, long x) | void | Yes | +| updateLong(String columnLabel, long x) | void | Yes | +| updateNull(int columnIndex) | void | Yes | +| updateNull(String columnLabel) | void | Yes | +| updateObject(int columnIndex, Object x) | void | Yes | +| updateObject(int columnIndex, Object x, int scaleOrLength) | void | Yes | +| updateObject(String columnLabel, Object x) | void | Yes | +| updateObject(String columnLabel, Object x, int scaleOrLength) | void | Yes | +| updateRow() | void | Yes | +| updateShort(int columnIndex, short x) | void | Yes | +| updateShort(String columnLabel, short x) | void | Yes | +| updateSQLXML(int columnIndex, SQLXML xmlObject) | void | Yes | +| updateSQLXML(String columnLabel, SQLXML xmlObject) | void | Yes | +| updateString(int columnIndex, String x) | void | Yes | +| updateString(String columnLabel, String x) | void | Yes | +| updateTime(int columnIndex, Time x) | void | Yes | +| updateTime(String columnLabel, Time x) | void | Yes | +| updateTimestamp(int columnIndex, Timestamp x) | void | Yes | +| updateTimestamp(String columnLabel, Timestamp x) | void | Yes | +| wasNull() | Boolean | Yes | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - One Statement cannot have multiple open ResultSets. +> - The cursor that is used for traversing the ResultSet cannot be open after being committed. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/7-java-sql-ResultSetMetaData.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/7-java-sql-ResultSetMetaData.md new file mode 100644 index 0000000000000000000000000000000000000000..5b7664db9446bb610ec8bb8d8637c74649df887e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/7-java-sql-ResultSetMetaData.md @@ -0,0 +1,41 @@ +--- +title: java.sql.ResultSetMetaData +summary: java.sql.ResultSetMetaData +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.ResultSetMetaData + +This section describes **java.sql.ResultSetMetaData**, which provides details about ResultSet object information. + +**Table 1** Support status for java.sql.ResultSetMetaData + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :------------------------------- | :---------- | :------------------------- | +| getCatalogName(int column) | String | Yes | +| getColumnClassName(int column) | String | Yes | +| getColumnCount() | int | Yes | +| getColumnDisplaySize(int column) | int | Yes | +| getColumnLabel(int column) | String | Yes | +| getColumnName(int column) | String | Yes | +| getColumnType(int column) | int | Yes | +| getColumnTypeName(int column) | String | Yes | +| getPrecision(int column) | int | Yes | +| getScale(int column) | int | Yes | +| getSchemaName(int column) | String | Yes | +| getTableName(int column) | String | Yes | +| isAutoIncrement(int column) | boolean | Yes | +| isCaseSensitive(int column) | boolean | Yes | +| isCurrency(int column) | boolean | Yes | +| isDefinitelyWritable(int column) | boolean | Yes | +| isNullable(int column) | int | Yes | +| isReadOnly(int column) | boolean | Yes | +| isSearchable(int column) | boolean | Yes | +| isSigned(int column) | boolean | Yes | +| isWritable(int column) | boolean | Yes | + +> **NOTE:** When **uppercaseAttributeName** is set to **true**, the following APIs convert the query result to uppercase letters. The conversion range is 26 English letters. +> +> - public String getColumnName(int column) +> - public String getColumnLabel(int column) \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/8-java-sql-Statement.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/8-java-sql-Statement.md new file mode 100644 index 0000000000000000000000000000000000000000..d3333ee2e427a91f87c980033b042a552cfe339e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/8-java-sql-Statement.md @@ -0,0 +1,69 @@ +--- +title: java.sql.Statement +summary: java.sql.Statement +author: Guo Huan +date: 2021-05-17 +--- + +# java.sql.Statement + +This section describes **java.sql.Statement**, the interface for executing SQL statements. + +**Table 1** Support status for java.sql.Statement + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :---------------------------------------------------- | :---------- | :------------------------- | +| addBatch(String sql) | void | Yes | +| clearBatch() | void | Yes | +| clearWarnings() | void | Yes | +| close() | void | Yes | +| closeOnCompletion() | void | Yes | +| execute(String sql) | Boolean | Yes | +| execute(String sql, int autoGeneratedKeys) | Boolean | Yes | +| execute(String sql, int[] columnIndexes) | Boolean | Yes | +| execute(String sql, String[] columnNames) | Boolean | Yes | +| executeBatch() | Boolean | Yes | +| executeQuery(String sql) | ResultSet | Yes | +| executeUpdate(String sql) | int | Yes | +| executeUpdate(String sql, int autoGeneratedKeys) | int | Yes | +| executeUpdate(String sql, int[] columnIndexes) | int | Yes | +| executeUpdate(String sql, String[] columnNames) | int | Yes | +| getConnection() | Connection | Yes | +| getFetchDirection() | int | Yes | +| getFetchSize() | int | Yes | +| getGeneratedKeys() | ResultSet | Yes | +| getMaxFieldSize() | int | Yes | +| getMaxRows() | int | Yes | +| getMoreResults() | boolean | Yes | +| getMoreResults(int current) | boolean | Yes | +| getResultSet() | ResultSet | Yes | +| getResultSetConcurrency() | int | Yes | +| getResultSetHoldability() | int | Yes | +| getResultSetType() | int | Yes | +| getQueryTimeout() | int | Yes | +| getUpdateCount() | int | Yes | +| getWarnings() | SQLWarning | Yes | +| isClosed() | Boolean | Yes | +| isCloseOnCompletion() | Boolean | Yes | +| isPoolable() | Boolean | Yes | +| setCursorName(String name) | void | Yes | +| setEscapeProcessing(boolean enable) | void | Yes | +| setFetchDirection(int direction) | void | Yes | +| setMaxFieldSize(int max) | void | Yes | +| setMaxRows(int max) | void | Yes | +| setPoolable(boolean poolable) | void | Yes | +| setQueryTimeout(int seconds) | void | Yes | +| setFetchSize(int rows) | void | Yes | +| cancel() | void | Yes | +| executeLargeUpdate(String sql) | long | No | +| getLargeUpdateCount() | long | No | +| executeLargeBatch() | long | No | +| executeLargeUpdate(String sql, int autoGeneratedKeys) | long | No | +| executeLargeUpdate(String sql, int[] columnIndexes) | long | No | +| executeLargeUpdate(String sql, String[] columnNames) | long | No | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - Using setFetchSize can reduce the memory occupied by result sets on the client. Result sets are packaged into cursors and segmented for processing, which will increase the communication traffic between the database and the client, affecting performance. +> - Database cursors are valid only within their transactions. If **setFetchSize** is set, set **setAutoCommit(false)** and commit transactions on the connection to flush service data to a database. +> - **LargeUpdate** methods can only be used in JDBC 4.2 or later. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/9-javax-sql-ConnectionPoolDataSource.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/9-javax-sql-ConnectionPoolDataSource.md new file mode 100644 index 0000000000000000000000000000000000000000..73d1518fb6b9ee1849a7d1fba13d185e3d87b1d1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/15-JDBC/9-javax-sql-ConnectionPoolDataSource.md @@ -0,0 +1,17 @@ +--- +title: javax.sql.ConnectionPoolDataSource +summary: javax.sql.ConnectionPoolDataSource +author: Guo Huan +date: 2021-05-17 +--- + +# javax.sql.ConnectionPoolDataSource + +This section describes **javax.sql.ConnectionPoolDataSource**, the interface for data source connection pools. + +**Table 1** Support status for javax.sql.ConnectionPoolDataSource + +| Method Name | Return Type | JDBC 4 Is Supported Or Not | +| :----------------------------------------------- | :--------------- | :------------------------- | +| getPooledConnection() | PooledConnection | Yes | +| getPooledConnection(String user,String password) | PooledConnection | Yes | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/2-jdbc-package-driver-class-and-environment-class.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/2-jdbc-package-driver-class-and-environment-class.md new file mode 100644 index 0000000000000000000000000000000000000000..3613b2ee3cc1fda565e05b3e4d0ed374681e7ffa --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/2-jdbc-package-driver-class-and-environment-class.md @@ -0,0 +1,51 @@ +--- +title: JDBC Package, Driver Class, and Environment Class +summary: JDBC Package, Driver Class, and Environment Class +author: Guo Huan +date: 2021-04-26 +--- + +# JDBC Package, Driver Class, and Environment Class + +**JDBC Package** + +Run **build.sh** in the source code directory on Linux OS to obtain the driver JAR package **postgresql.jar**, which is stored in the source code directory. Obtain the package from the release package named [**openGauss-x.x.x-JDBC.tar.gz**](https://opengauss.org/en/download.html). + +The driver package is compatible with PostgreSQL. The class name and structure in the driver are the same as those in the PostgreSQL driver. All applications running on PostgreSQL can be smoothly migrated to the current system. + +**Driver Class** + +Before establishing a database connection, load the **org.opengauss.Driver** database driver class. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - MogDB is compatible with PostgreSQL in the use of JDBC. Therefore, the org.postgresql.Driver class will conflict when the JDBC driver packages for PostgreSQL and openGauss are loaded in the same JVM virtual machine at the same time. +> +> - Compared with the PostgreSQL driver, the openGauss JDBC driver has the following enhanced features: +> - The SHA256 encryption mode is supported for login. +> - The third-party log framework that implements the sf4j API can be connected. +> - DR failover is supported. + +**Environment Class** + +JDK 1.8 must be configured on the client. The configuration method is as follows: + +1. In the MS-DOS window, run **java -version** to check the JDK version. Ensure that the version is JDK 1.8. If JDK is not installed, download the installation package from the official website and install it. If the system environment JDK version is lower than 1.8, please refer to [Use WebSphere to Configure MogDB Data Sources](../../../quick-start/mogdb-access/use-middleware-to-access-mogdb/websphere-configures-mogdb-data-source-reference.md). + +2. Configure system environment variables. + + 1. Right-click **My computer** and choose **Properties**. + + 2. In the navigation pane, choose **Advanced system settings**. + + 3. In the **System Properties** dialog box, click **Environment Variables** on the **Advanced** tab page. + + 4. In the **System variables** area of the **Environment Variables** dialog box, click **New** or **Edit** to configure system variables. For details, see [Table 1](#Description). + + **Table 1** Description + + | Variable | Operation | Variable Value | + | :-------- | :----------------------------------------------------------- | :----------------------------------------------------------- | + | JAVA_HOME | - If the variable exists, click **Edit**.
- If the variable does not exist, click **New**. | Specifies the Java installation directory.
Example: C:\Program Files\Java\jdk1.8.0_131 | + | Path | Edit | - If JAVA_HOME is configured, add **%JAVA_HOME%\bin** before the variable value.
- If JAVA_HOME is not configured, add the full Java installation path before the variable value:
C:\Program Files\Java\jdk1.8.0_131\bin; | + | CLASSPATH | New | .;%JAVA_HOME%\lib;%JAVA_H | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/3-development-process.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/3-development-process.md new file mode 100644 index 0000000000000000000000000000000000000000..e73100d573bff2b2082b342988768e7879df88f2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/3-development-process.md @@ -0,0 +1,12 @@ +--- +title: Development Process +summary: Development Process +author: Guo Huan +date: 2021-04-26 +--- + +# Development Process + +**Figure 1** Application development process based on JDBC + +![application-development-process-based-on-jdbc](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/development-process-2.png) diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/4-loading-the-driver.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/4-loading-the-driver.md new file mode 100644 index 0000000000000000000000000000000000000000..de6094ed057d041276534974753d148e9bebd03b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/4-loading-the-driver.md @@ -0,0 +1,19 @@ +--- +title: Loading the Driver +summary: Loading the Driver +author: Guo Huan +date: 2021-04-26 +--- + +# Loading the Driver + +Load the database driver before creating a database connection. + +You can load the driver in the following ways: + +- Before creating a connection, implicitly load the driver in the code:**Class.forName("org.opengauss.Driver")** + +- During the JVM startup, transfer the driver as a parameter to the JVM:**java -Djdbc.drivers=org.opengauss.Driver jdbctest** + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > **jdbctest** is the name of a test application. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/5-connecting-to-a-database.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/5-connecting-to-a-database.md new file mode 100644 index 0000000000000000000000000000000000000000..8242b9209f88d1644c5267ce94163bbc22bde541 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/5-connecting-to-a-database.md @@ -0,0 +1,105 @@ +--- +title: Connecting to a Database +summary: Connecting to a Database +author: Guo Huan +date: 2021-04-26 +--- + +# Connecting to a Database + +After a database is connected, you can use JDBC to run SQL statements to operate data. + +**Function Prototype** + +JDBC provides the following three database connection methods: + +- DriverManager.getConnection(String url); +- DriverManager.getConnection(String url, Properties info); +- DriverManager.getConnection(String url, String user, String password); + +**Parameters** + +**Table 1** Database connection parameters + +| Parameter | Description | +| --------- | ------------------------------------------------------------ | +| url | **postgresql.jar** database connection descriptor. The format is as follows:
- `jdbc:opengauss:database`
- `jdbc:opengauss://host/database`
- `jdbc:opengauss://host:port/database`
- `jdbc:opengauss://host:port/database?param1=value1¶m2=value2`
- `jdbc:opengauss://host1:port1,host2:port2/database?param1=value1¶m2=value2`
NOTE:
- **database** indicates the name of the database to connect.
- **host** indicates the name or IP address of the database server.
If a machine connected to MogDB is not in the same network segment as MogDB, the IP address specified by **host** should be the value of **coo.cooListenIp2** (application access IP address) set in Manager.
For security purposes, the primary database node forbids access from other nodes in MogDB without authentication. To access the primary database node from inside MogDB, deploy the JDBC program on the host where the primary database node is located and set **host** to **127.0.0.1**. Otherwise, the error message "FATAL: Forbid remote connection with trust method!" may be displayed.
It is recommended that the service system be deployed outside MogDB. If it is deployed inside, database performance may be affected.
By default, the local host is used to connect to the server.
- **port** indicates the port number of the database server.
By default, the database on port 5432 of the local host is connected.
- **param** indicates a database connection attribute.
The parameter can be configured in the URL. The URL starts with a question mark (?), uses an equal sign (=) to assign a value to the parameter, and uses an ampersand (&) to separate parameters. You can also use the attributes of the **info** object for configuration. For details, see the example below.
- **value** indicates the database connection attribute values.
The **connectTimeout** and **socketTimeout** parameters must be set for connection. If they are not set, the default value **0** is used, indicating that the connection will not time out. When the network between the DN and client is faulty, the client does not receive the ACK packet from the DN. In this case, the client starts the timeout retransmission mechanism to continuously retransmit packets. A timeout error is reported only when the timeout interval reaches the default value **600s**. As a result, the RTO is high. | +| info | Database connection attributes (all attributes are case sensitive). Common attributes are described as follows:
- **PGDBNAME**: string type. This parameter specifies the database name. (This parameter does not need to be set in the URL. The system automatically parses the URL to obtain its value.)
- **PGHOST**: string type. This parameter specifies the host IP address. For details, see the example below.
- **PGPORT**: integer type. This parameter specifies the host port number. For details, see the example below.
- **user**: string type. This parameter specifies the database user who creates the connection.
- **password**: string type. This parameter specifies the password of the database user.
- **enable_ce**: string type. If **enable_ce** is set to **1**, JDBC supports encrypted equality query.
- **loggerLevel**: string type. The following log levels are supported:**OFF**, **DEBUG**, and **TRACE**. The value **OFF** indicates that the log function is disabled. **DEBUG** and **TRACE** logs record information of different levels.
- **loggerFile**: string type. This parameter specifies the name of a log file. You can specify a directory for storing logs. If no directory is specified, logs are stored in the directory where the client program is running.
- **allowEncodingChanges**: Boolean type. If this parameter is set to **true**, the character set type can be changed. This parameter is used together with **characterEncoding=CHARSET** to set the character set. The two parameters are separated by ampersands (&). The value of **characterEncoding** can be **UTF8**, **GBK**, or **LATIN1**.
- **currentSchema**: string type. This parameter specifies the schema to be set in **search-path**.
- **hostRecheckSeconds**: integer type. After JDBC attempts to connect to a host, the host status is saved: connection success or connection failure. This status is trusted within the duration specified by **hostRecheckSeconds**. After the duration expires, the status becomes invalid. The default value is 10 seconds.
- **ssl**: Boolean type. This parameter specifies a connection in SSL mode. When **ssl** is set to **true**, the NonValidatingFactory channel and certificate mode are supported.
1. For the NonValidatingFactory channel, configure the username and password and set **SSL** to **true**.
2. In certification mode, configure the client certificate, key, and root certificate, and set **SSL** to **true**.
- **sslmode**: string type. This parameter specifies the SSL authentication mode. The value can be **require**, **verify-ca**, or **verify-full**.
- **require**: The system attempts to set up an SSL connection. If there is a CA file, the system performs verification as if the parameter was set to **verify-ca**.
- **verify-ca**: The system attempts to set up an SSL connection and checks whether the server certificate is issued by a trusted CA.
- **verify-full**: The system attempts to set up an SSL connection, checks whether the server certificate is issued by a trusted CA, and checks whether the host name of the server is the same as that in the certificate.
- **sslcert**: string type. This parameter specifies the complete path of the certificate file. The type of the client and server certificates is **End Entity**.
- **sslkey**: string type. This parameter specifies the complete path of the key file. You must run the following command to convert the client certificate to the DER format:
`openssl pkcs8 -topk8 -outform DER -in client.key -out client.key.pk8 -nocrypt`
- **sslrootcert**: string type. This parameter specifies the name of the SSL root certificate. The root certificate type is CA.
- **sslpassword**: string type. This parameter is provided for ConsoleCallbackHandler.
- **sslpasswordcallback**: string type. This parameter specifies the class name of the SSL password provider. The default value is **org.opengauss.ssl.jdbc4.LibPQFactory.ConsoleCallbackHandler**.
- **sslfactory**: string type. This parameter specifies the class name used by SSLSocketFactory to establish an SSL connection.
- **sslfactoryarg**: string type. The value is an optional parameter of the constructor function of the **sslfactory** class and is not recommended.
- **sslhostnameverifier**: string type. This parameter specifies the class name of the host name verifier. The interface must implement javax.net.ssl.HostnameVerifier. The default value is **org.opengauss.ssl.PGjdbcHostnameVerifier**.
- **loginTimeout**: integer type. This parameter specifies the waiting time for establishing the database connection, in seconds.**connectTimeout**: integer type. This parameter specifies the timeout duration for connecting to a server, in seconds. If the time taken to connect to a server exceeds the value specified, the connection is interrupted. If the value is **0**, the timeout mechanism is disabled.
- **socketTimeout**: integer type. This parameter specifies the timeout duration for a socket read operation, in seconds. If the time taken to read data from a server exceeds the value specified, the connection is closed. If the value is **0**, the timeout mechanism is disabled.
- **cancelSignalTimeout**: integer type. Cancel messages may cause a block. This parameter controls **connectTimeout** and **socketTimeout** in a cancel message, in seconds. The default value is 10 seconds.
- **tcpKeepAlive**: Boolean type. This parameter is used to enable or disable TCP keepalive detection. The default value is **false**.
- **logUnclosedConnections**: Boolean type. The client may leak a connection object because it does not call the connection object's close() method. These objects will be collected as garbage and finalized using the finalize() method. If the caller ignores this operation, this method closes the connection.
- **assumeMinServerVersion**: string type. The client sends a request to set a floating point. This parameter specifies the version of the server to connect, for example, **assumeMinServerVersion=9.0**. This parameter can reduce the number of packets to send during connection setup.
- **ApplicationName**: string type. This parameter specifies the name of the JDBC driver that is being connected. You can query the **pg_stat_activity** table on the primary database node to view information about the client that is being connected. The JDBC driver name is displayed in the **application_name** column. The default value is **PostgreSQL JDBC Driver**.
- **connectionExtraInfo**: Boolean type. This parameter specifies whether the JDBC driver reports the driver deployment path and process owner to the database.The value can be **true** or **false**. The default value is **false**. If **connectionExtraInfo** is set to **true**, the JDBC driver reports the driver deployment path and process owner, and URL connection configuration information to the database and displays the information in the **connection_info** parameter. In this case, you can query the information from **PG_STAT_ACTIVITY**.
- **autosave**: string type. The value can be **always**, **never**, or **conservative**. The default value is **never**. This parameter specifies the action that the driver should perform upon a query failure. If **autosave** is set to **always**, the JDBC driver sets a savepoint before each query and rolls back to the savepoint if the query fails. If **autosave** is set to **never**, there is no savepoint. If **autosave** is set to **conservative**, a savepoint is set for each query. However, the system rolls back and retries only when there is an invalid statement.
- **protocolVersion**: integer type. This parameter specifies the connection protocol version. Only version 3 is supported. Note: MD5 encryption is used when this parameter is set. You must use the following command to change the database encryption mode:**gs_guc set -N all -I all -c "password_encryption_type=1"**. After MogDB is restarted, create a user that uses MD5 encryption to encrypt passwords. You must also change the client connection mode to **md5** in the **pg_hba.conf** file. Log in as the new user (not recommended).
NOTE:
The MD5 encryption algorithm has lower security and poses security risks. Therefore, you are advised to use a more secure encryption algorithm.
- **prepareThreshold**: integer type. This parameter specifies the time when the parse statement is sent. The default value is **5**. It takes a long time to parse an SQL statement for the first time, but a short time to parse SQL statements later because of cache. If a session runs an SQL statement multiple consecutive times and the number of execution times exceeds the value of **prepareThreshold**, JDBC does not send the parse command to the SQL statement.
- **preparedStatementCacheQueries**: integer type. This parameter specifies the number of queries cached in each connection. The default value is **256**. If more than 256 different queries are used in the prepareStatement() call, the least recently used query cache will be discarded. The value **0** indicates that the cache function is disabled.
- **preparedStatementCacheSizeMiB**: integer type. This parameter specifies the maximum cache size of each connection, in MB. The default value is **5**. If the size of the cached queries exceeds 5 MB, the least recently used query cache will be discarded. The value **0** indicates that the cache function is disabled.
- **databaseMetadataCacheFields**: integer type. The default value is **65536**. This parameter specifies the maximum cache size of each connection. The value **0** indicates that the cache function is disabled.
- **databaseMetadataCacheFieldsMiB**: integer type. The default value is **5**. This parameter specifies the maximum cache size of each connection, in MB. The value **0** indicates that the cache function is disabled.
- **stringtype**: string type. The value can be **false**, **unspecified**, or **varchar**. The default value is **varchar**. This parameter specifies the type of the **PreparedStatement** parameter used by the setString() method. If **stringtype** is set to **varchar**, these parameters are sent to the server as varchar parameters. If **stringtype** is set to **unspecified**, these parameters are sent to the server as an untyped value, and the server attempts to infer their appropriate type.
- **batchMode**: Boolean type. This parameter specifies whether to connect the database in batch mode. The default value is **on**, indicating that the batch mode is enabled.
- **fetchsize**: integer type. This parameter specifies the default fetchsize for statements in the created connection. The default value is **0**, indicating that all results are obtained at a time.
- **reWriteBatchedInserts**: Boolean type. During batch import, set this parameter to **true** to combine **N** insertion statements into one: insert into TABLE_NAME values(values1, …, valuesN), …, (values1, …, valuesN). To use this parameter, set **batchMode** to **off**.
- **unknownLength**: integer type. The default value is **Integer.MAX\_VALUE**. This parameter specifies the length of the unknown length type when the data of some postgresql types (such as TEXT) is returned by functions such as ResultSetMetaData.getColumnDisplaySize and ResultSetMetaData.getPrecision.
- **defaultRowFetchSize**: integer type. This parameter specifies the number of rows read by fetch in ResultSet at a time. Limiting the number of rows read each time in a database access request can avoid unnecessary memory consumption, thereby avoiding out of memory exception. The default value is **0**, indicating that all rows are obtained at a time in ResultSet. There is no negative value.
- **binaryTransfer**: Boolean type. This parameter specifies whether data is sent and received in binary format. The default value is **false**.
- **binaryTransferEnable**: string type. This parameter specifies the type for which binary transmission is enabled. Every two types are separated by commas (,). You can select either the OID or name, for example, binaryTransferEnable=Integer4_ARRAY,Integer8_ARRAY.
For example, if the OID name is **BLOB** and the OID number is 88, you can configure the OID as follows:
**binaryTransferEnable=BLOB or binaryTransferEnable=88**
- **binaryTransferDisEnable**: string type. This parameter specifies the type for which binary transmission is disabled. Every two types are separated by commas (,). You can select either the OID or name. The value of this parameter overwrites the value of **binaryTransferEnable**.
- **blobMode**: string type. This parameter sets the setBinaryStream method to assign values to different types of data. The value **on** indicates that values are assigned to blob data. The value **off** indicates that values are assigned to bytea data. The default value is **on**.
- **socketFactory**: string type. This parameter specifies the name of the class used to create a socket connection with the server. This class must implement the **javax.net.SocketFactory** interface and define a constructor with no parameter or a single string parameter.
- **socketFactoryArg**: string type. The value is an optional parameter of the constructor function of the socketFactory class and is not recommended.
- **receiveBufferSize**: integer type. This parameter is used to set **SO\_RCVBUF** on the connection stream.
- **sendBufferSize**: integer type. This parameter is used to set **SO\_SNDBUF** on the connection stream.
- **preferQueryMode**: string type. The value can be **extended**, **extendedForPrepared**, **extendedCacheEverything**, or **simple**. This parameter specifies the query mode. In **simple** mode, the query is executed without parsing or binding. In **extended** mode, the query is executed and bound. The **extendedForPrepared** mode is used for prepared statement extension. In **extendedCacheEverything** mode, each statement is cached.
- **targetServerType**: string type. The difference between primary DN and standby DN is whether the DN allows the write operation in the URL connection string. The default value is **any**. The value can be **any**, **master**, **slave**, or **preferSlave**.
- **master**: attempts to connect to a primary DN in the URL connection string. If the primary DN cannot be found, an exception is thrown.
- **slave**: attempts to connect to a standby DN in the URL connection string. If the primary DN cannot be found, an exception is thrown.
- **preferSlave** attempts to connect to a standby DN (if available) in the URL connection string. Otherwise, it connects to the primary DN.
- **any** attempts to connect to any DN in the URL connection string.
- **priorityServers**: integer type. This value is used to specify the first **n** nodes configured in the URL as the primary database instance to be connected preferentially. The default value is **null**. The value is a number greater than 0 and less than the number of DNs configured in the URL.
For example, `jdbc:opengauss://host1:port1,host2:port2,host3:port3,host4:port4,/database?priorityServers=2`. That is, **host1** and **host2** are primary database instance nodes, and **host3** and **host4** are DR database instance nodes.
- **forceTargetServerSlave**: Boolean type. This parameter specifies whether to enable the function of forcibly connecting to the standby node and forbid the existing connections to be used on the standby node that is promoted to primary during the primary/standby switchover of the database instance. The default value is **false**, indicating that the function of forcibly connecting to the standby node is disabled. **true**: The function of forcibly connecting to the standby node is enabled. | +| user | Database user. | +| password | Password of the database user. | + +**Examples** + +```java +// The following code encapsulates database connection operations into an interface. The database can then be connected using an authorized username and a password. +public static Connection getConnect(String username, String passwd) + { + // Driver class. + String driver = "org.opengauss.Driver"; + // Database connection descriptor. + String sourceURL = "jdbc:opengauss://10.10.0.13:8000/postgres"; + Connection conn = null; + + try + { + // Load the driver. + Class.forName(driver); + } + catch( Exception e ) + { + e.printStackTrace(); + return null; + } + + try + { + // Create a connection. + conn = DriverManager.getConnection(sourceURL, username, passwd); + System.out.println("Connection succeed!"); + } + catch(Exception e) + { + e.printStackTrace(); + return null; + } + + return conn; + }; +// The following code uses the Properties object as a parameter to establish a connection. +public static Connection getConnectUseProp(String username, String passwd) + { + // Driver class. + String driver = "org.opengauss.Driver"; + // Database connection descriptor. + String sourceURL = "jdbc:opengauss://10.10.0.13:8000/postgres?"; + Connection conn = null; + Properties info = new Properties(); + + try + { + // Load the driver. + Class.forName(driver); + } + catch( Exception e ) + { + e.printStackTrace(); + return null; + } + + try + { + info.setProperty("user", username); + info.setProperty("password", passwd); + // Create a connection. + conn = DriverManager.getConnection(sourceURL, info); + System.out.println("Connection succeed!"); + } + catch(Exception e) + { + e.printStackTrace(); + return null; + } + + return conn; + }; +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/6-connecting-to-a-database-using-ssl.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/6-connecting-to-a-database-using-ssl.md new file mode 100644 index 0000000000000000000000000000000000000000..d9f02a546d7eba7213ef370995a7003cd97f4787 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/6-connecting-to-a-database-using-ssl.md @@ -0,0 +1,146 @@ +--- +title: Connecting to the Database (Using SSL) +summary: Connecting to the Database (Using SSL) +author: Guo Huan +date: 2021-04-26 +--- + +# Connecting to the Database (Using SSL) + +When establishing connections to the MogDB server using JDBC, you can enable SSL connections to encrypt client and server communications for security of sensitive data transmission on the Internet. This section describes how applications establish an SSL connection to MogDB using JDBC. To start the SSL mode, you must have the server certificate, client certificate, and private key files. For details on how to obtain these files, see related documents and commands of OpenSSL. + +**Configuring the Server** + +The SSL mode requires a root certificate, a server certificate, and a private key. + +Perform the following operations (assuming that the license files are saved in the data directory **/mogdb/data/datanode** and the default file names are used): + +1. Log in as the OS user **omm** to the primary node of the database. + +2. Generate and import a certificate. + + Generate an SSL certificate. For details, see [Generating Certificates](../../../security-guide/security/1-client-access-authentication.md#Generating Certificates). Copy the generated **server.crt**, **server.key**, and **cacert.pem** files to the data directory on the server. + + Run the following command to query the data directory of the database node. The instance column indicates the data directory. + + ```bash + gs_om -t status --detail + ``` + + In the Unix OS, **server.crt** and **server.key** must deny the access from the external or any group. Run the following command to set this permission: + + ```bash + chmod 0600 server.key + ``` + +3. Enable the SSL authentication mode. + + ```bash + gs_guc set -D /mogdb/data/datanode -c "ssl=on" + ``` + +4. Set client access authentication parameters. The IP address is the IP address of the host to be connected. + + ```bash + gs_guc reload -D /mogdb/data/datanode -h "hostssl all all 127.0.0.1/32 cert" + gs_guc reload -D /mogdb/data/datanode -h "hostssl all all IP/32 cert" + ``` + + Clients on the **127.0.0.1⁄32** network segment can connect to MogDB servers in SSL mode. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** + > + > - If **METHOD** is set to **cert** in the **pg_hba.conf** file of the server, the client must use the username (common name) configured in the license file (**client.crt**) for the database connection. If **METHOD** is set to **md5**, **sm3** or **sha256**, there is no such a restriction. + > - The MD5 encryption algorithm has lower security and poses security risks. Therefore, you are advised to use a more secure encryption algorithm. + +5. Configure the digital certificate parameters related to SSL authentication. + + The information following each command indicates operation success. + + ```bash + gs_guc set -D /mogdb/data/datanode -c "ssl_cert_file='server.crt'" + gs_guc set: ssl_cert_file='server.crt' + ``` + + ```bash + gs_guc set -D /mogdb/data/datanode -c "ssl_key_file='server.key'" + gs_guc set: ssl_key_file='server.key' + ``` + + ```bash + gs_guc set -D /mogdb/data/datanode -c "ssl_ca_file='cacert.pem'" + gs_guc set: ssl_ca_file='cacert.pem' + ``` + +6. Restart the database. + + ```bash + gs_om -t stop && gs_om -t start + ``` + +**Configuring the Client** + +To configure the client, perform the following steps: + +Upload the certificate files **client.key.pk8**, **client.crt**, and **cacert.pem** generated in **Configuring the Server** to the client. + +**Example** + +Note: Choose one of example 1 and example 2. + +```java +public class SSL{ + public static void main(String[] args) { + Properties urlProps = new Properties(); + String urls = "jdbc:opengauss://10.29.37.136:8000/postgres"; + + /** + * ================== Example 1: Use the NonValidatingFactory channel. + */ + urlProps.setProperty("sslfactory","org.opengauss.ssl.NonValidatingFactory"); + urlProps.setProperty("user", "world"); + urlProps.setProperty("password", "test@123"); + urlProps.setProperty("ssl", "true"); + /** + * ================== Examples 2: Use a certificate. + */ + urlProps.setProperty("sslcert", "client.crt"); + urlProps.setProperty("sslkey", "client.key.pk8"); + urlProps.setProperty("sslrootcert", "cacert.pem"); + urlProps.setProperty("user", "world"); + urlProps.setProperty("ssl", "true"); + /* sslmode can be set to require, verify-ca, or verify-full. Select one from the following three examples.*/ + /* ================== Example 2.1: Set sslmode to require to use the certificate for authentication. */ + urlProps.setProperty("sslmode", "require"); + /* ================== Example 2.2: Set sslmode to verify-ca to use the certificate for authentication. */ + urlProps.setProperty("sslmode", "verify-ca"); + /* ================== Example 2.3: Set sslmode to verify-full to use the certificate (in the Linux OS) for authentication. */ + urls = "jdbc:opengauss://world:8000/postgres"; + urlProps.setProperty("sslmode", "verify-full"); + try { + Class.forName("org.opengauss.Driver").newInstance(); + } catch (Exception e) { + e.printStackTrace(); + } + try { + Connection conn; + conn = DriverManager.getConnection(urls,urlProps); + conn.close(); + } catch (Exception e) { + e.printStackTrace(); + } + } +} +/** + * Note: Convert the client key to the DER format. + * openssl pkcs8 -topk8 -outform DER -in client.key -out client.key.pk8 -nocrypt + * openssl pkcs8 -topk8 -inform PEM -in client.key -outform DER -out client.key.der -v1 PBE-MD5-DES + * openssl pkcs8 -topk8 -inform PEM -in client.key -outform DER -out client.key.der -v1 PBE-SHA1-3DES + * The preceding algorithms are not recommended due to their low security. + * If the customer needs to use a higher-level private key encryption algorithm, the following private key encryption algorithms can be used after the BouncyCastle or a third-party private key is used to decrypt the password package: + * openssl pkcs8 -in client.key -topk8 -outform DER -out client.key.der -v2 AES128 + * openssl pkcs8 -in client.key -topk8 -outform DER -out client.key.der -v2 aes-256-cbc -iter 1000000 + * openssl pkcs8 -in client.key -topk8 -out client.key.der -outform Der -v2 aes-256-cbc -v2prf hmacWithSHA512 + * Enable BouncyCastle: Introduce the bcpkix-jdk15on.jar package for projects that use JDBC. The recommended version is 1.65 or later. + */ +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/7-running-sql-statements.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/7-running-sql-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..553f856737b2120abed0e7cc0e30e44ad8686b63 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/7-running-sql-statements.md @@ -0,0 +1,241 @@ +--- +title: Running SQL Statements +summary: Running SQL Statements +author: Guo Huan +date: 2021-04-26 +--- + +# Running SQL Statements + +**Running a Common SQL Statement** + +To enable an application to operate data in the database by running SQL statements (statements that do not need to transfer parameters), perform the following operations: + +1. Create a statement object by calling the **createStatement** method in **Connection**. + + ```bash + Connection conn = DriverManager.getConnection("url","user","password"); + Statement stmt = conn.createStatement(); + ``` + +2. Run the SQL statement by calling the **executeUpdate** method in **Statement**. + + ```bash + int rc = stmt.executeUpdate("CREATE TABLE customer_t1(c_customer_sk INTEGER, c_customer_name VARCHAR(32));"); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - If an execution request (not in a transaction block) received in the database contains multiple statements, the request is packed into a transaction. **VACUUM** is not supported in a transaction block. If one of the statements fails, the entire request will be rolled back. + > - Use semicolons (;) to separate statements. Stored procedures, functions, and anonymous blocks do not support multi-statement execution. + > - The slash (/) can be used as the terminator for creating a single stored procedure, function, or anonymous block. + +3. Close the statement object. + + ``` + stmt.close(); + ``` + +**Running a Prepared SQL Statement** + +Prepared statements are complied and optimized once but can be used in different scenarios by assigning multiple values. Using prepared statements improves execution efficiency. If you want to run a statement for several times, use a precompiled statement. Perform the following operations: + +1. Create a prepared statement object by calling the prepareStatement method in Connection. + + ```json + PreparedStatement pstmt = con.prepareStatement("UPDATE customer_t1 SET c_customer_name = ? WHERE c_customer_sk = 1"); + ``` + +2. Set parameters by calling the setShort method in PreparedStatement. + + ```json + pstmt.setShort(1, (short)2); + ``` + +3. Run the prepared statement by calling the executeUpdate method in PreparedStatement. + + ```json + int rowcount = pstmt.executeUpdate(); + ``` + +4. Close the prepared statement object by calling the close method in PreparedStatement. + + ```json + pstmt.close(); + ``` + +**Calling a Stored Procedure** + +To call an existing stored procedure through JDBC in MogDB, perform the following operations: + +1. Create a call statement object by calling the **prepareCall** method in **Connection**. + + ```bash + Connection myConn = DriverManager.getConnection("url","user","password"); + CallableStatement cstmt = myConn.prepareCall("{? = CALL TESTPROC(?,?,?)}"); + ``` + +2. Set parameters by calling the **setInt** method in **CallableStatement**. + + ``` + cstmt.setInt(2, 50); + cstmt.setInt(1, 20); + cstmt.setInt(3, 90); + ``` + +3. Register an output parameter by calling the **registerOutParameter** method in **CallableStatement**. + + ``` + cstmt.registerOutParameter(4, Types.INTEGER); // Register an OUT parameter of the integer type. + ``` + +4. Call the stored procedure by calling the **execute** method in **CallableStatement**. + + ``` + cstmt.execute(); + ``` + +5. Obtain the output parameter by calling the **getInt** method in **CallableStatement**. + + ``` + int out = cstmt.getInt(4); // Obtain the OUT parameter. + ``` + + Example: + + ``` + // The following stored procedure (containing the OUT parameter) has been created: + create or replace procedure testproc + ( + psv_in1 in integer, + psv_in2 in integer, + psv_inout in out integer + ) + as + begin + psv_inout := psv_in1 + psv_in2 + psv_inout; + end; + / + ``` + +6. Close the call statement by calling the **close** method in **CallableStatement**. + + ``` + cstmt.close(); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - Many database classes such as Connection, Statement, and ResultSet have a close() method. Close these classes after using their objects. Closing Connection will close all the related Statements, and closing a Statement will close its ResultSet. + > - Some JDBC drivers support named parameters, which can be used to set parameters by name rather than sequence. If a parameter has the default value, you do not need to specify any parameter value but can use the default value directly. Even though the parameter sequence changes during a stored procedure, the application does not need to be modified. Currently, the MogDB JDBC driver does not support this method. + > - MogDB does not support functions containing OUT parameters, or stored procedures and function parameters containing default values. + > + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE**: + > + > - If JDBC is used to call a stored procedure whose returned value is a cursor, the returned cursor cannot be used. + > - A stored procedure and an SQL statement must be run separately. + +**Calling a Stored Procedure When Overloading Is Enabled in Oracle Compatibility Mode** + +After the **behavior_compat_options='proc_outparam_override'** parameter is enabled, perform the following steps to call the stored procedure: + +1. Create a call statement object by calling the **prepareCall** method in **Connection**. + + ```bash + Connection conn = DriverManager.getConnection("url","user","password"); + CallableStatement cs = conn.prepareCall("{ CALL TEST_PROC(?,?,?) }"); + ``` + +2. Set parameters by calling the **setInt** method in **CallableStatement**. + + ```bash + PGobject pGobject = new PGobject(); + pGobject.setType("public.compfoo"); // Set the composite type name. The format is "schema.typename". + pGobject.setValue("(1,demo)"); //: Bind the value of the composite type. The format is "(value1,value2)". + cs.setObject(1, pGobject); + ``` + +3. Register an output parameter by calling the **registerOutParameter** method in **CallableStatement**. + + ```bash + //Register an out parameter of the composite type. The format is "schema.typename". + cs.registerOutParameter(2, Types.STRUCT, "public.compfoo"); + ``` + +4. Call the stored procedure by calling the **execute** method in **CallableStatement**. + + ```bash + cs.execute(); + ``` + +5. Obtain the output parameter by calling the **getObject** method in **CallableStatement**. + + ```bash + PGobject result = (PGobject)cs.getObject(2); // Obtain the out parameter. + result.getValue(); // Obtain the string value of the composite type. + result.getArrayValue(); // Obtain the array values of the composite type and sort the values according to the sequence of columns of the composite type. + result.getStruct(); // Obtain the subtype names of the composite type and sort them according to the creation sequence. + ``` + +6. Close the call statement by calling the **close** method in **CallableStatement**. + + ```bash + cs.close(); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - After the Oracle compatibility mode is enabled, you must use the **{call proc_name(?,?,?)}** format to call a stored procedure and use the **{? = call func_name(?,?)}** format to call a function. The question mark (?) on the left of the equal mark is the placeholder for the return value of the function and is used to register the return value of the function. + > - After **behavior_compat_options** is set to **'proc_outparam_override'**, the service needs to re-establish a connection. Otherwise, the stored procedures and functions cannot be correctly called. + > - If a function or stored procedure contains a composite type, bind and register parameters in the schema.typename format. + +Example: + +``` +//Create a composite data type in the database. +CREATE TYPE compfoo AS (f1 int, f3 text); +// The following stored procedure (containing the OUT parameter) has been created: +create or replace procedure test_proc +( + psv_in in compfoo, + psv_out out compfoo +) +as +begin + psv_out := psv_in; +end; +/ +``` + +**Batch Processing** + +When a prepared statement processes multiple pieces of similar data, the database creates only one execution plan. This improves compilation and optimization efficiency. Perform the following operations: + +1. Create a prepared statement object by calling the prepareStatement method in Connection. + + ``` + Connection conn = DriverManager.getConnection("url","user","password"); + PreparedStatement pstmt = conn.prepareStatement("INSERT INTO customer_t1 VALUES (?)"); + ``` + +2. Call the setShort parameter for each piece of data, and call addBatch to confirm that the setting is complete. + + ``` + pstmt.setShort(1, (short)2); + pstmt.addBatch(); + ``` + +3. Perform batch processing by calling the executeBatch method in PreparedStatement. + + ``` + int[] rowcount = pstmt.executeBatch(); + ``` + +4. Close the prepared statement object by calling the close method in PreparedStatement. + + ``` + pstmt.close(); + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > Do not terminate a batch processing action when it is ongoing; otherwise, database performance will deteriorate. Therefore, disable automatic commit during batch processing. Manually commit several rows at a time. The statement for disabling automatic commit is **conn.setAutoCommit(false);**. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/8-processing-data-in-a-result-set.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/8-processing-data-in-a-result-set.md new file mode 100644 index 0000000000000000000000000000000000000000..d45f9df5ee3e1d155fa4e5f44a1b70f2c2a59cb2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/8-processing-data-in-a-result-set.md @@ -0,0 +1,76 @@ +--- +title: Processing Data in a Result Set +summary: Processing Data in a Result Set +author: Guo Huan +date: 2021-04-26 +--- + +# Processing Data in a Result Set + +**Setting a Result Set Type** + +Different types of result sets apply to different application scenarios. Applications select proper types of result sets based on requirements. Before running an SQL statement, you must create a statement object. Some methods of creating statement objects can set the type of a result set. [Table 1](#Result set types) lists result set parameters. The related Connection methods are as follows: + +``` +// Create a Statement object. This object will generate a ResultSet object with a specified type and concurrency. +createStatement(int resultSetType, int resultSetConcurrency); + +// Create a PreparedStatement object. This object will generate a ResultSet object with a specified type and concurrency. +prepareStatement(String sql, int resultSetType, int resultSetConcurrency); + +// Create a CallableStatement object. This object will generate a ResultSet object with a specified type and concurrency. +prepareCall(String sql, int resultSetType, int resultSetConcurrency); +``` + +**Table 1** Result set types + +| Parameter | Description | +| :------------------- | :----------------------------------------------------------- | +| resultSetType | Type of a result set. There are three types of result sets:
- **ResultSet.TYPE_FORWARD_ONLY**: The ResultSet object can only be navigated forward. It is the default value.
- **ResultSet.TYPE_SCROLL_SENSITIVE**: You can view the modified result by scrolling to the modified row.
- **ResultSet.TYPE_SCROLL_INSENSITIVE**: The ResultSet object is insensitive to changes in the underlying data source.
NOTE:
After a result set has obtained data from the database, the result set is insensitive to data changes made by other transactions, even if the result set type is **ResultSet.TYPE_SCROLL_SENSITIVE**. To obtain up-to-date data of the record pointed by the cursor from the database, call the refreshRow() method in a ResultSet object. | +| resultSetConcurrency | Concurrency type of a result set. There are two types of concurrency.
- **ResultSet.CONCUR_READ_ONLY**: Data in a result set cannot be updated except that an updated statement has been created in the result set data.
- **ResultSet.CONCUR_UPDATEABLE**: changeable result set. The concurrency type for a result set object can be updated if the result set is scrollable. | + +**Positioning a Cursor in a Result Set** + +ResultSet objects include a cursor pointing to the current data row. The cursor is initially positioned before the first row. The next method moves the cursor to the next row from its current position. When a ResultSet object does not have a next row, a call to the next method returns **false**. Therefore, this method is used in the while loop for result set iteration. However, the JDBC driver provides more cursor positioning methods for scrollable result sets, which allows positioning cursor in the specified row. [Table 2](#Methods for positioning) describes these methods. + +**Table 2** Methods for positioning a cursor in a result set + +| Method | Description | +| :------------ | :----------------------------------------------------------- | +| next() | Moves cursor to the next row from its current position. | +| previous() | Moves cursor to the previous row from its current position. | +| beforeFirst() | Places cursor before the first row. | +| afterLast() | Places cursor after the last row. | +| first() | Places cursor to the first row. | +| last() | Places cursor to the last row. | +| absolute(int) | Places cursor to a specified row. | +| relative(int) | Moves the row specified by the forward parameter (that is, the value of is 1, which is equivalent to next()) or backward (that is, the value of is -1, which is equivalent to previous()). | + +**Obtaining the Cursor Position from a Result Set** + +This cursor positioning method will be used to change the cursor position for a scrollable result set. The JDBC driver provides a method to obtain the cursor position in a result set. [Table 3](#Methods for obtaining) describes these methods. + +**Table 3** Methods for obtaining a cursor position in a result set + +| Method | Description | +| :-------------- | :------------------------------------------------- | +| isFirst() | Checks whether the cursor is in the first row. | +| isLast() | Checks whether the cursor is in the last row. | +| isBeforeFirst() | Checks whether the cursor is before the first row. | +| isAfterLast() | Checks whether the cursor is after the last row. | +| getRow() | Gets the current row number of the cursor. | + +**Obtaining Data from a Result Set** + +ResultSet objects provide a variety of methods to obtain data from a result set. [Table 4](#Common methods for obtaining) describes the common methods for obtaining data. If you want to know more about other methods, see JDK official documents. + +**Table 4** Common methods for obtaining data from a result set + +| Method | Description | +| :----------------------------------- | :----------------------------------------------------------- | +| int getInt(int columnIndex) | Retrieves the value of the column designated by a column index in the current row as an integer. | +| int getInt(String columnLabel) | Retrieves the value of the column designated by a column label in the current row as an integer. | +| String getString(int columnIndex) | Retrieves the value of the column designated by a column index in the current row as a string. | +| String getString(String columnLabel) | Retrieves the value of the column designated by a column label in the current row as a string. | +| Date getDate(int columnIndex) | Retrieves the value of the column designated by a column index in the current row as a date. | +| Date getDate(String columnLabel) | Retrieves the value of the column designated by a column name in the current row as a date. | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/8.1-log-management.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/8.1-log-management.md new file mode 100644 index 0000000000000000000000000000000000000000..af93fb3d79cca25fcd1caffdb9384c5a6de389c5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/8.1-log-management.md @@ -0,0 +1,118 @@ +--- +title: Log Management +summary: Log Management +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Log Management + +The MogDB JDBC driver uses log records to help solve problems when the MogDB JDBC driver is used in applications. MogDB JDBC supports the following log management methods: + +1. Use the SLF4J log framework for interconnecting with applications. +2. Use the JdkLogger log framework for interconnecting with applications. + +SLF4J and JdkLogger are mainstream frameworks for Java application log management in the industry. For details about how to use these frameworks, see the official documents. + +- SLF4J: +- JdkLogger: + +Method 1: Use the SLF4J log framework for interconnecting with applications. + +When a connection is set up, **logger=Slf4JLogger** is configured in the URL. + +The SLF4J may be implemented by using Log4j or Log4j2. When the Log4j is used to implement the SLF4J, the following JAR packages need to be added: **log4j-\*.jar**, **slf4j-api-\*.jar**, and **slf4j-log4\*-\*.jar** (\* varies according to versions), and configuration file **log4j.properties**. If the Log4j2 is used to implement the SLF4J, you need to add the following JAR packages: **log4j-api-\*.jar**, **log4j-core-\*.jar**, **log4j-slf4j18-impl-\*.jar**, and **slf4j-api-\*-alpha1.jar** (\* varies according to versions), and configuration file **log4j2.xml**. + +This method supports log management and control. The SLF4J can implement powerful log management and control functions through related configurations in files. This method is recommended. + +Example: + +``` +public static Connection GetConnection(String username, String passwd){ + + String sourceURL = "jdbc:opengauss://10.10.0.13:8000/postgres?logger=Slf4JLogger"; + Connection conn = null; + + try{ +// Create a connection. + conn = DriverManager.getConnection(sourceURL,username,passwd); + System.out.println("Connection succeed!"); + }catch (Exception e){ + e.printStackTrace(); + return null; + } + return conn; +} +``` + +The following is an example of the **log4j.properties** file: + +``` +log4j.logger.org.opengauss=ALL, log_gsjdbc + +# Default file output configuration +log4j.appender.log_gsjdbc=org.apache.log4j.RollingFileAppender +log4j.appender.log_gsjdbc.Append=true +log4j.appender.log_gsjdbc.File=gsjdbc.log +log4j.appender.log_gsjdbc.Threshold=TRACE +log4j.appender.log_gsjdbc.MaxFileSize=10MB +log4j.appender.log_gsjdbc.MaxBackupIndex=5 +log4j.appender.log_gsjdbc.layout=org.apache.log4j.PatternLayout +log4j.appender.log_gsjdbc.layout.ConversionPattern=%d %p %t %c - %m%n +log4j.appender.log_gsjdbc.File.Encoding = UTF-8 +``` + +The following is an example of the **log4j2.xml** file: + +``` + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +Method 3: Use the JdkLogger log framework for interconnecting with applications. + +The default Java logging framework stores its configurations in a file named **logging.properties**. Java installs the global configuration file in the folder in the Java installation directory. The **logging.properties** file can also be created and stored with a single project. + +Configuration example of **logging.properties**: + +``` +# Specify the processing program as a file. +handlers= java.util.logging.FileHandler + +# Specify the default global log level. +.level= ALL + +# Specify the log output control standard. +java.util.logging.FileHandler.level=ALL +java.util.logging.FileHandler.pattern = gsjdbc.log +java.util.logging.FileHandler.limit = 500000 +java.util.logging.FileHandler.count = 30 +java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter +java.util.logging.FileHandler.append=false +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/9-closing-a-connection.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/9-closing-a-connection.md new file mode 100644 index 0000000000000000000000000000000000000000..2d6064b55066b95ada6813b88c3e95542cf5df75 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/2-development-based-on-jdbc/9-closing-a-connection.md @@ -0,0 +1,17 @@ +--- +title: Closing a Connection +summary: Closing a Connection +author: Guo Huan +date: 2021-04-26 +--- + +# Closing a Connection + +After you complete required data operations in the database, close the database connection. + +Call the close method to close the connection, for example, + +```text +Connection conn = DriverManager.getConnection\("url","user","password"\) ; +conn.close\(\); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/1-development-based-on-odbc-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/1-development-based-on-odbc-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..f4a1710b8df4539903ca8fcf0220da9980ffcfdb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/1-development-based-on-odbc-overview.md @@ -0,0 +1,34 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2021-04-26 +--- + +# Overview + +Open Database Connectivity (ODBC) is a Microsoft API for accessing databases based on the X/OPEN CLI. Applications interact with the database through the APIs provided by ODBC, which enhances their portability, scalability, and maintainability. + +[Figure 1](#ODBC) shows the system structure of ODBC. + +**Figure 1** ODBC system structure + +![odbc-system-structure](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/development-based-on-odbc-overview-2.png) + +MogDB supports ODBC 3.5 in the following environments. + +**Table 1** OSs Supported by ODBC + +| OS | Platform | +| :------------------------------------------------- | :------- | +| CentOS 6.4/6.5/6.6/6.7/6.8/6.9/7.0/7.1/7.2/7.3/7.4 | x86_64 | +| CentOS 7.6 | ARM64 | +| EulerOS 2.0 SP2/SP3 | x86_64 | +| EulerOS 2.0 SP8 | ARM64 | + +The ODBC Driver Manager running on UNIX or Linux can be unixODBC or iODBC. unixODBC-2.3.0 is used as the component for connecting the database. + +Windows has a native ODBC Driver Manager. You can locate **Data Sources (ODBC)** by choosing **Control Panel** > **Administrative Tools**. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> The current database ODBC driver is based on an open-source version and may be incompatible with data types tinyint, smalldatetime, and nvarchar2. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/2-odbc-packages-dependent-libraries-and-header-files.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/2-odbc-packages-dependent-libraries-and-header-files.md new file mode 100644 index 0000000000000000000000000000000000000000..16764e9dddb25c43a26673eda71911a855ee0c1b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/2-odbc-packages-dependent-libraries-and-header-files.md @@ -0,0 +1,12 @@ +--- +title: ODBC Packages, Dependent Libraries, and Header Files +summary: ODBC Packages, Dependent Libraries, and Header Files +author: Guo Huan +date: 2021-04-26 +--- + +# ODBC Packages, Dependent Libraries, and Header Files + +**ODBC Packages for the Linux OS** + +Obtain the [**openGauss-x.x.x-ODBC.tar.gz**](https://opengauss.org/en/download.html) package from the release package. In the Linux OS, header files (including **sql.h** and **sqlext.h**) and library (**libodbc.so**) are required in application development. These header files and library can be obtained from the **unixODBC-2.3.0** installation package. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/3-configuring-a-data-source-in-the-linux-os.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/3-configuring-a-data-source-in-the-linux-os.md new file mode 100644 index 0000000000000000000000000000000000000000..a5be42b5a6cfd6904af787b12d839298c5c9c180 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/3-configuring-a-data-source-in-the-linux-os.md @@ -0,0 +1,328 @@ +--- +title: Configuring a Data Source in the Linux OS +summary: Configuring a Data Source in the Linux OS +author: Guo Huan +date: 2021-04-26 +--- + +# Configuring a Data Source in the Linux OS + +The ODBC driver (psqlodbcw.so) provided by MogDB can be used after it has been configured in a data source. To configure a data source, you must configure the **odbc.ini** and **odbcinst.ini** files on the server. The two files are generated during the unixODBC compilation and installation, and are saved in the **/usr/local/etc** directory by default. + +**Procedure** + +1. Obtain the source code package of unixODBC by following link: + + + + After the download, validate the integrity based on the integrity validation algorithm provided by the community. + +2. Install unixODBC. It does not matter if unixODBC of another version has been installed. + + Currently, unixODBC-2.2.1 is not supported. For example, to install unixODBC-2.3.0, run the commands below. unixODBC is installed in the **/usr/local** directory by default. The data source file is generated in the **/usr/local/etc** directory, and the library file is generated in the **/usr/local/lib** directory. + + ```bash + tar zxvf unixODBC-2.3.0.tar.gz + cd unixODBC-2.3.0 + #Modify the configure file. (If it does not exist, modify the configure.ac file.) Find LIB_VERSION. + #Change the value of LIB_VERSION to 1:0:0 to compile a *.so.1 dynamic library with the same dependency on psqlodbcw.so. + vim configure + + ./configure --enable-gui=no #To perform compilation on a Kunpeng server, add the configure parameter --build=aarch64-unknown-linux-gnu. + make + #The installation may require root permissions. + make install + ``` + +3. Replace the openGauss driver on the client. + + a. Decompress the **openGauss-1.1.0-ODBC.tar.gz** package. After the decompression, the **lib** and **odbc** folders are generated. The **odbc** folder contains another **lib** folder. Copy the **psqlodbca.la**, **psqlodbca.so**, **psqlodbcw.la**, and **psqlodbcw.so** files from **/odbc/lib** to **/usr/local/lib**. + + b. Copy the library in the **lib** directory obtained after decompressing **openGauss-1.1.0-ODBC.tar.gz** to the **/usr/local/lib** directory. + +4. Configure a data source. + + a. Configure the ODBC driver file. + + Add the following content to the **/usr/local/etc/odbcinst.ini** file: + + ``` + [GaussMPP] + Driver64=/usr/local/lib/psqlodbcw.so + setup=/usr/local/lib/psqlodbcw.so + ``` + + For descriptions of the parameters in the **odbcinst.ini** file, see [Table 1](#odbcinst.ini). + + **Table 1** odbcinst.ini configuration parameters + + | **Parameter** | **Description** | **Example** | + | ------------- | ------------------------------------------------------------ | ------------------------------------ | + | [DriverName] | Driver name, corresponding to the driver in DSN. | [DRIVER_N] | + | Driver64 | Path of the dynamic driver library. | Driver64=/usr/local/lib/psqlodbcw.so | + | setup | Driver installation path, which is the same as the dynamic library path in Driver64. | setup=/usr/local/lib/psqlodbcw.so | + + b. Configure the data source file. + + Add the following content to the **/usr/local/etc/odbc.ini** file: + + ```bash + [MPPODBC] + Driver=GaussMPP + Servername=10.145.130.26 (IP address of the server where the database resides) + Database=postgres (database name) + Username=omm (database username) + Password= (user password of the database) + Port=8000 (listening port of the database) + Sslmode=allow + ``` + + For descriptions of the parameters in the **odbc.ini** file, see [Table 2](#odbc.ini). + + **Table 2** odbc.ini configuration parameters + + | **Parameter** | **Description** | **Example** | + | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | + | [DSN] | Data source name | [MPPODBC] | + | Driver | Driver name, corresponding to DriverName in **odbcinst.ini** | Driver=DRIVER_N | + | Servername | Server IP address. Multiple IP addresses can be configured. | Servername=10.145.130.26 | + | Database | Name of the database to connect to | Database=postgres | + | Username | Database username | Username=omm | + | Password | Database user password | Password=
NOTE:
After a user established a connection, the ODBC driver automatically clears their password stored in memory.
However, if this parameter is configured, UnixODBC will cache data source files, which may cause the password to be stored in the memory for a long time.
When you connect to an application, you are advised to send your password through an API instead of writing it in a data source configuration file. After the connection has been established, immediately clear the memory segment where your password is stored. | + | Port | Port number of the server | Port=8000 | + | Sslmode | Whether to enable SSL | Sslmode=allow | + | Debug | If this parameter is set to **1**, the **mylog** file of the PostgreSQL ODBC driver will be printed. The directory generated for storing logs is **/tmp/**. If this parameter is set to **0**, no directory is generated. | Debug=1 | + | UseServerSidePrepare | Whether to enable the extended query protocol for the database.
The value can be **0** or **1**. The default value is **1**, indicating that the extended query protocol is enabled. | UseServerSidePrepare=1 | + | UseBatchProtocol | Whether to enable the batch query protocol. If it is enabled, DML performance can be improved. The value can be **0** or **1**. The default value is **1**.
If this parameter is set to **0**, the batch query protocol is disabled (mainly for communication with earlier database versions).
If this parameter is set to **1** and **support_batch_bind** is set to **on**, the batch query protocol is enabled. | UseBatchProtocol=1 | + | ForExtensionConnector | This parameter specifies whether the savepoint is sent. | ForExtensionConnector=1 | + | UnamedPrepStmtThreshold | Each time **SQLFreeHandle** is invoked to release statements, ODBC sends a **Deallocate plan_name** statement to the server. A large number of such a statement exist in the service. To reduce the number of the statements to be sent, **stmt->plan_name** is left empty so that the database can identify them as unnamed statements. This parameter is added to control the threshold for unnamed statements. | UnamedPrepStmtThreshold=100 | + | ConnectionExtraInfo | Whether to display the driver deployment path and process owner in the **connection_info** parameter mentioned in [connection_info](../../../reference-guide/guc-parameters/3-connection-and-authentication/1-connection-settings.md#connection_info). | ConnectionExtraInfo=1NOTE:The default value is **0**. If this parameter is set to **1**, the ODBC driver reports the driver deployment path and process owner to the database and displays the information in the **connection_info** parameter (see [connection_info](../../../reference-guide/guc-parameters/3-connection-and-authentication/1-connection-settings.md#connection_info)). In this case, you can query the information from [PG_STAT_ACTIVITY](../../../reference-guide/system-catalogs-and-system-views/system-views/PG_STAT_ACTIVITY.md). | + | BoolAsChar | If this parameter is set to **Yes**, the Boolean value is mapped to the SQL_CHAR type. If this parameter is not set, the value is mapped to the SQL_BIT type. | BoolsAsChar = Yes | + | RowVersioning | When an attempt is made to update a row of data, setting this parameter to **Yes** allows the application to detect whether the data has been modified by other users. | RowVersioning = Yes | + | ShowSystemTables | By default, the driver regards the system table as a common SQL table. | ShowSystemTables = Yes | + + The valid values of **Sslmode** are as follows: + + **Table 3** Sslmode options + + | sslmode | Whether SSL Encryption Is Enabled | Description | + | ----------- | --------------------------------- | ------------------------------------------------------------ | + | disable | No | SSL connection is not enabled. | + | allow | Possible | If the database server requires SSL connection, SSL connection can be enabled. However, authenticity of the database server will not be verified. | + | prefer | Possible | If the database supports SSL connection, SSL connection is recommended. However, authenticity of the database server will not be verified. | + | require | Yes | SSL connection is required and data is encrypted. However, authenticity of the database server will not be verified. | + | verify-ca | Yes | SSL connection is required and whether the database has a trusted certificate will be verified. | + | verify-full | Yes | SSL connection is required. In addition to the check scope specified by **verify-ca**, the system checks whether the name of the host where the database resides is the same as that in the certificate. MogDB does not support this mode. | + +5. (Optional) Generate an SSL certificate. For details, see [Generating Certificates](../../../security-guide/security/1-client-access-authentication.md#Generating Certificates).This step and step 6 need to be performed when the server and the client are connected via ssl. It can be skipped in case of non-ssl connection. + +6. (Optional) Replace an SSL certificate. For details, see [Replacing Certificates](../../../security-guide/security/1-client-access-authentication.md#Replacing Certificates). + +7. Enable SSL mode. + + Declare the following environment variables and ensure that the permission for the **client.key\*** series files is set to **600**. + + ``` + Go back to the root directory, create the .postgresql directory, and save root.crt, client.crt, client.key, client.key.cipher, client.key.rand, client.req, server.crt, server.key, server.key.cipher, server.key.rand, and server.req to the .postgresql directory. + In the Unix OS, server.crt and server.key must deny the access from the external or any group. Run the following command to set this permission: + chmod 0600 server.key + Copy the certificate files whose names start with root.crt and server to the install/data directory of the database (the directory is the same as that of the postgresql.conf file). + Modify the postgresql.conf file. + ssl = on + ssl_cert_file = 'server.crt' + ssl_key_file = 'server.key' + ssl_ca_file = 'root.crt' + After modifying the parameters, restart the database. + Set the sslmode parameter to require or verify-ca in the odbc.ini file. + ``` + +8. Configure the database server. + + a. Log in as the OS user **omm** to the primary node of the database. + + b. Run the following command to add NIC IP addresses or host names, with values separated by commas (,). The NICs and hosts are used to provide external services. In the following command, *NodeName* specifies the name of the current node. + + ``` + gs_guc reload -N NodeName -I all -c "listen_addresses='localhost,192.168.0.100,10.11.12.13'" + ``` + + If direct routing of LVS is used, add the virtual IP address (10.11.12.13) of LVS to the server listening list. + + You can also set **listen_addresses** to **\*** or **0.0.0.0** to listen to all NICs, but this incurs security risks and is not recommended. + + c. Run the following command to add an authentication rule to the configuration file of the primary database node. In this example, the IP address (10.11.12.13) of the client is the remote host IP address. + + ``` + gs_guc reload -N all -I all -h "host all jack 10.11.12.13/32 sha256" + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - **-N all** indicates all hosts in MogDB. + > - **-I all** indicates all instances of the host. + > - **-h** specifies statements that need to be added in the **pg_hba.conf** file. + > - **all** indicates that a client can connect to any database. + > - **jack** indicates the user that accesses the database. + > - **10.11.12.13/32** indicates hosts whose IP address is 10.11.12.13 can be connected. Configure the parameter based on your network conditions. **32** indicates that there are 32 bits whose value is 1 in the subnet mask. That is, the subnet mask is 255.255.255.255. + > - **sha256** indicates that the password of user **jack** is encrypted using the SHA-256 algorithm. + + If the ODBC client and the primary database node to connect are deployed on the same machine, you can use the local trust authentication mode. Run the following command: + + ``` + local all all trust + ``` + + If the ODBC client and the primary database node to connect are deployed on different machines, use the SHA-256 authentication mode. Run the following command: + + ``` + host all all xxx.xxx.xxx.xxx/32 sha256 + ``` + + d. Restart MogDB. + + ``` + gs_om -t stop + gs_om -t start + ``` + +9. Configure environment variables on the client. + + ``` + vim ~/.bashrc + ``` + + Add the following information to the configuration file: + + ```bash + export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH + export ODBCSYSINI=/usr/local/etc + export ODBCINI=/usr/local/etc/odbc.ini + ``` + +10. Run the following command to validate the addition: + + ``` + source ~/.bashrc + ``` + +**Verifying the Data Source Configuration** + +Run the **./isql-v** **MPPODBC** command (**MPPODBC** is the data source name). + +- If the following information is displayed, the configuration is correct and the connection succeeds. + + ``` + +---------------------------------------+ + | Connected! | + | | + | sql-statement | + | help [tablename] | + | quit | + | | + +---------------------------------------+ + SQL> + ``` + +- If error information is displayed, the configuration is incorrect. Check the configuration. + +**FAQs** + +- [UnixODBC]Can't open lib 'xxx/xxx/psqlodbcw.so' : file not found. + + Possible causes: + + - The path configured in the **odbcinst.ini** file is incorrect. + + Run **ls** to check the path in the error information, and ensure that the **psqlodbcw.so** file exists and you have execute permissions on it. + + - The dependent library of **psqlodbcw.so** does not exist or is not in system environment variables. + + Run **ldd** to check the path in the error information. If **libodbc.so.1** or other UnixODBC libraries do not exist, configure UnixODBC again following the procedure provided in this section, and add the **lib** directory under its installation directory to **LD_LIBRARY_PATH**. If other libraries do not exist, add the **lib** directory under the ODBC driver package to **LD_LIBRARY_PATH**. + +- [UnixODBC]connect to server failed: no such file or directory + + Possible causes: + + - An incorrect or unreachable database IP address or port number was configured. + + Check the **Servername** and **Port** configuration items in data sources. + + - Server monitoring is improper. + + If **Servername** and **Port** are correctly configured, ensure the proper network adapter and port are monitored by following the database server configurations in the procedure in this section. + + - Firewall and network gatekeeper settings are improper. + + Check firewall settings, and ensure that the database communication port is trusted. + + Check to ensure network gatekeeper settings are proper (if any). + +- [unixODBC]The password-stored method is not supported. + + Possible causes: + + The **sslmode** configuration item is not configured in the data sources. + + Solution: + + Set the configuration item to **allow** or a higher level. For details, see [Table 3](#sslmode). + +- Server common name "xxxx" does not match host name "xxxxx" + + Possible causes: + + When **verify-full** is used for SSL encryption, the driver checks whether the host name in certificates is the same as the actual one. + + Solution: + + To solve this problem, use **verify-ca** to stop checking host names, or generate a set of CA certificates containing the actual host names. + +- Driver's SQLAllocHandle on SQL_HANDLE_DBC failed + + Possible causes: + + The executable file (such as the **isql** tool of unixODBC) and the database driver (**psqlodbcw.so**) depend on different library versions of ODBC, such as **libodbc.so.1** and **libodbc.so.2**. You can verify this problem by using the following method: + + ``` + ldd `which isql` | grep odbc + ldd psqlodbcw.so | grep odbc + ``` + + If the suffix digits of the outputs **libodbc.so** are different or indicate different physical disk files, this problem exists. Both **isql** and **psqlodbcw.so** load **libodbc.so**. If different physical files are loaded, different ODBC libraries with the same function list conflict with each other in a visible domain. As a result, the database driver cannot be loaded. + + Solution: + + Uninstall the unnecessary unixODBC, such as libodbc.so.2, and create a soft link with the same name and the .so.2 suffix for the remaining libodbc.so.1 library. + +- FATAL: Forbid remote connection with trust method! + + For security purposes, the primary database node forbids access from other nodes in MogDB without authentication. + + To access the primary database node from inside MogDB, deploy the ODBC program on the host where the primary database node is located and set the server address to **127.0.0.1**. It is recommended that the service system be deployed outside MogDB. If it is deployed inside, database performance may be affected. + +- [unixODBC]Invalid attribute value + + This problem occurs when you use SQL on other MogDB. The possible cause is that the unixODBC version is not the recommended one. You are advised to run the **odbcinst -version** command to check the unixODBC version. + +- authentication method 10 not supported. + + If this error occurs on an open-source client, the cause may be: + + The database stores only the SHA-256 hash of the password, but the open-source client supports only MD5 hashes. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The database stores the hashes of user passwords instead of actual passwords. + > - If a password is updated or a user is created, both types of hashes will be stored, compatible with open-source authentication protocols. + > - An MD5 hash can only be generated using the original password, but the password cannot be obtained by reversing its SHA-256 hash. Passwords in the old version will only have SHA-256 hashes and not support MD5 authentication. + > - The MD5 encryption algorithm has lower security and poses security risks. Therefore, you are advised to use a more secure encryption algorithm. + + To solve this problem, you can update the user password (see [ALTER USER](../../../reference-guide/sql-syntax/ALTER-USER.md)) or create a user (see [CREATE USER](../../../reference-guide/sql-syntax/CREATE-USER.md)) having the same permissions as the faulty user. + +- unsupported frontend protocol 3.51: server supports 1.0 to 3.0 + + The database version is too early or the database is an open-source database. Use the driver of the required version to connect to the database. + +- FATAL: GSS authentication method is not allowed because XXXX user password is not disabled. + + In **pg_hba.conf** of the target primary database node, the authentication mode is set to **gss** for authenticating the IP address of the current client. However, this authentication algorithm cannot authenticate clients. Change the authentication algorithm to **sha256** and try again. For details, see [8](#8). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/4-development-process.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/4-development-process.md new file mode 100644 index 0000000000000000000000000000000000000000..40fee82eddce2a3f5c590e13f27d7a1baee5b495 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/4-development-process.md @@ -0,0 +1,42 @@ +--- +title: Development Process +summary: Development Process +author: Guo Huan +date: 2021-04-26 +--- + +# Development Process + +**Figure 1** ODBC-based application development process + +![odbc-based-application-development-process](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/development-process-4.png) + +**APIs Involved in the Development Process** + +**Table 1** API description + +| **Function** | **API** | +| :-------------------------------------------------------- | :----------------------------------------------------------- | +| Allocate a handle | SQLAllocHandle is a generic function for allocating a handle. It can replace the following functions:
- SQLAllocEnv: allocate an environment handle
- SQLAllocConnect: allocate a connection handle
- SQLAllocStmt: allocate a statement handle | +| Set environment attributes | SQLSetEnvAttr | +| Set connection attributes | SQLSetConnectAttr | +| Set statement attributes | SQLSetStmtAttr | +| Connect to a data source | SQLConnect | +| Bind a buffer to a column in the result set | SQLBindCol | +| Bind the parameter marker of an SQL statement to a buffer | SQLBindParameter | +| Return the error message of the last operation | SQLGetDiagRec | +| Prepare an SQL statement for execution | SQLPrepare | +| Run a prepared SQL statement | SQLExecute | +| Run an SQL statement directly | SQLExecDirect | +| Fetch the next row (or rows) from the result set | SQLFetch | +| Return data in a column of the result set | SQLGetData | +| Get the column information from a result set | SQLColAttribute | +| Disconnect from a data source | SQLDisconnect | +| Release a handle | SQLFreeHandle is a generic function for releasing a handle. It can replace the following functions:
- SQLFreeEnv: release an environment handle
- SQLFreeConnect: release a connection handle
- SQLFreeStmt: release a statement handle | + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> If an execution request (not in a transaction block) received in the database contains multiple statements, the request is packed into a transaction. If one of the statements fails, the entire request will be rolled back. +> +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif)**NOTICE:** +> +> ODBC is the central layer of the application program and the database. It is responsible for transmitting the SQL instructions issued by the application program to the database, and does not parse the SQL syntax by itself. Therefore, when an SQL statement with confidential information (such as a plaintext password) is written in an application, the confidential information will be exposed in the driver log. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/5-example-common-functions-and-batch-binding.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/5-example-common-functions-and-batch-binding.md new file mode 100644 index 0000000000000000000000000000000000000000..1ac90e259c052a619e127d1ac9cc2ef29998dff9 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/5-example-common-functions-and-batch-binding.md @@ -0,0 +1,440 @@ +--- +title: Common Functions and Batch Binding +summary: Common Functions and Batch Binding +author: Guo Huan +date: 2021-04-26 +--- + +# Example: Common Functions and Batch Binding + +## Code for Common Functions + +``` +//The following example shows how to obtain data from MogDB through the ODBC interface. +// DBtest.c (compile with: libodbc.so) +#include +#include +#include +#ifdef WIN32 +#include +#endif +SQLHENV V_OD_Env; // Handle ODBC environment +SQLHSTMT V_OD_hstmt; // Handle statement +SQLHDBC V_OD_hdbc; // Handle connection +char typename[100]; +SQLINTEGER value = 100; +SQLINTEGER V_OD_erg,V_OD_buffer,V_OD_err,V_OD_id; +int main(int argc,char *argv[]) +{ + // 1. Allocate an environment handle. + V_OD_erg = SQLAllocHandle(SQL_HANDLE_ENV,SQL_NULL_HANDLE,&V_OD_Env); + if ((V_OD_erg != SQL_SUCCESS) && (V_OD_erg != SQL_SUCCESS_WITH_INFO)) + { + printf("Error AllocHandle\n"); + exit(0); + } + // 2. Set environment attributes (version information). + SQLSetEnvAttr(V_OD_Env, SQL_ATTR_ODBC_VERSION, (void*)SQL_OV_ODBC3, 0); + // 3. Allocate a connection handle. + V_OD_erg = SQLAllocHandle(SQL_HANDLE_DBC, V_OD_Env, &V_OD_hdbc); + if ((V_OD_erg != SQL_SUCCESS) && (V_OD_erg != SQL_SUCCESS_WITH_INFO)) + { + SQLFreeHandle(SQL_HANDLE_ENV, V_OD_Env); + exit(0); + } + // 4. Set connection attributes. + SQLSetConnectAttr(V_OD_hdbc, SQL_ATTR_AUTOCOMMIT, SQL_AUTOCOMMIT_ON, 0); + // 5. Connect to the data source. userName and password indicate the username and password for connecting to the database. Set them as needed. + // If the username and password have been set in the odbc.ini file, you do not need to set userName or password here, retaining "" for them. However, you are not advised to do so because the username and password will be disclosed if the permission for odbc.ini is abused. + V_OD_erg = SQLConnect(V_OD_hdbc, (SQLCHAR*) "gaussdb", SQL_NTS, + (SQLCHAR*) "userName", SQL_NTS, (SQLCHAR*) "password", SQL_NTS); + if ((V_OD_erg != SQL_SUCCESS) && (V_OD_erg != SQL_SUCCESS_WITH_INFO)) + { + printf("Error SQLConnect %d\n",V_OD_erg); + SQLFreeHandle(SQL_HANDLE_ENV, V_OD_Env); + exit(0); + } + printf("Connected !\n"); + // 6. Set statement attributes. + SQLSetStmtAttr(V_OD_hstmt,SQL_ATTR_QUERY_TIMEOUT,(SQLPOINTER *)3,0); + // 7. Allocate a statement handle. + SQLAllocHandle(SQL_HANDLE_STMT, V_OD_hdbc, &V_OD_hstmt); + // 8. Run SQL statements. + SQLExecDirect(V_OD_hstmt,"drop table IF EXISTS customer_t1",SQL_NTS); + SQLExecDirect(V_OD_hstmt,"CREATE TABLE customer_t1(c_customer_sk INTEGER, c_customer_name VARCHAR(32));",SQL_NTS); + SQLExecDirect(V_OD_hstmt,"insert into customer_t1 values(25,li)",SQL_NTS); + // 9. Prepare for execution. + SQLPrepare(V_OD_hstmt,"insert into customer_t1 values(?)",SQL_NTS); + // 10. Bind parameters. + SQLBindParameter(V_OD_hstmt,1,SQL_PARAM_INPUT,SQL_C_SLONG,SQL_INTEGER,0,0, + &value,0,NULL); + // 11. Run prepared statements. + SQLExecute(V_OD_hstmt); + SQLExecDirect(V_OD_hstmt,"select id from testtable",SQL_NTS); + // 12. Obtain attributes of a specific column in the result set. + SQLColAttribute(V_OD_hstmt,1,SQL_DESC_TYPE,typename,100,NULL,NULL); + printf("SQLColAtrribute %s\n",typename); + // 13. Bind the result set. + SQLBindCol(V_OD_hstmt,1,SQL_C_SLONG, (SQLPOINTER)&V_OD_buffer,150, + (SQLLEN *)&V_OD_err); + // 14. Obtain data in the result set by executing SQLFetch. + V_OD_erg=SQLFetch(V_OD_hstmt); + // 15. Obtain and return data by executing SQLGetData. + while(V_OD_erg != SQL_NO_DATA) + { + SQLGetData(V_OD_hstmt,1,SQL_C_SLONG,(SQLPOINTER)&V_OD_id,0,NULL); + printf("SQLGetData ----ID = %d\n",V_OD_id); + V_OD_erg=SQLFetch(V_OD_hstmt); + }; + printf("Done !\n"); + // 16. Disconnect data source connections and release handles. + SQLFreeHandle(SQL_HANDLE_STMT,V_OD_hstmt); + SQLDisconnect(V_OD_hdbc); + SQLFreeHandle(SQL_HANDLE_DBC,V_OD_hdbc); + SQLFreeHandle(SQL_HANDLE_ENV, V_OD_Env); + return(0); + } +``` + +## Code for Batch Processing + +``` +/********************************************************************** +* Enable UseBatchProtocol in the data source and set the database parameter support_batch_bind +* to on. +* The CHECK_ERROR command is used to check and print error information. +* This example is used to interactively obtain the DSN, data volume to be processed, and volume of ignored data from users, and insert required data into the test_odbc_batch_insert table. +***********************************************************************/ +#include +#include +#include +#include +#include + +void Exec(SQLHDBC hdbc, SQLCHAR* sql) +{ + SQLRETURN retcode; // Return status + SQLHSTMT hstmt = SQL_NULL_HSTMT; // Statement handle + SQLCHAR loginfo[2048]; + + // Allocate Statement Handle + retcode = SQLAllocHandle(SQL_HANDLE_STMT, hdbc, &hstmt); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLAllocHandle(SQL_HANDLE_STMT) failed"); + return; + } + + // Prepare Statement + retcode = SQLPrepare(hstmt, (SQLCHAR*) sql, SQL_NTS); + sprintf((char*)loginfo, "SQLPrepare log: %s", (char*)sql); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLPrepare(hstmt, (SQLCHAR*) sql, SQL_NTS) failed"); + return; + } + + // Execute Statement + retcode = SQLExecute(hstmt); + sprintf((char*)loginfo, "SQLExecute stmt log: %s", (char*)sql); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute(hstmt) failed"); + return; + } + // Free Handle + retcode = SQLFreeHandle(SQL_HANDLE_STMT, hstmt); + sprintf((char*)loginfo, "SQLFreeHandle stmt log: %s", (char*)sql); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLFreeHandle(SQL_HANDLE_STMT, hstmt) failed"); + return; + } +} + +int main () +{ + SQLHENV henv = SQL_NULL_HENV; + SQLHDBC hdbc = SQL_NULL_HDBC; + int batchCount = 1000; // Amount of data that is bound in batches + SQLLEN rowsCount = 0; + int ignoreCount = 0; // Amount of data that is not imported to the database among the data that is bound in batches + + SQLRETURN retcode; + SQLCHAR dsn[1024] = {'\0'}; + SQLCHAR loginfo[2048]; + + do + { + if (ignoreCount > batchCount) + { + printf("ignoreCount(%d) should be less than batchCount(%d)\n", ignoreCount, batchCount); + } + }while(ignoreCount > batchCount); + + retcode = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &henv); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLAllocHandle failed"); + goto exit; + } + + // Set ODBC Verion + retcode = SQLSetEnvAttr(henv, SQL_ATTR_ODBC_VERSION, + (SQLPOINTER*)SQL_OV_ODBC3, 0); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetEnvAttr failed"); + goto exit; + } + + // Allocate Connection + retcode = SQLAllocHandle(SQL_HANDLE_DBC, henv, &hdbc); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLAllocHandle failed"); + goto exit; + } + + + // Set Login Timeout + retcode = SQLSetConnectAttr(hdbc, SQL_LOGIN_TIMEOUT, (SQLPOINTER)5, 0); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetConnectAttr failed"); + goto exit; + } + + + // Set Auto Commit + retcode = SQLSetConnectAttr(hdbc, SQL_ATTR_AUTOCOMMIT, + (SQLPOINTER)(1), 0); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetConnectAttr failed"); + goto exit; + } + + + // Connect to DSN + // gaussdb indicates the name of the data source used by users. + sprintf(loginfo, "SQLConnect(DSN:%s)", dsn); + retcode = SQLConnect(hdbc, (SQLCHAR*) "gaussdb", SQL_NTS, + (SQLCHAR*) NULL, 0, NULL, 0); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLConnect failed"); + goto exit; + } + + // init table info. + Exec(hdbc, "drop table if exists test_odbc_batch_insert"); + Exec(hdbc, "create table test_odbc_batch_insert(id int primary key, col varchar2(50))"); + + // The following code constructs the data to be inserted based on the data volume entered by users: + { + SQLRETURN retcode; + SQLHSTMT hstmtinesrt = SQL_NULL_HSTMT; + int i; + SQLCHAR *sql = NULL; + SQLINTEGER *ids = NULL; + SQLCHAR *cols = NULL; + SQLLEN *bufLenIds = NULL; + SQLLEN *bufLenCols = NULL; + SQLUSMALLINT *operptr = NULL; + SQLUSMALLINT *statusptr = NULL; + SQLULEN process = 0; + + // Data is constructed by column. Each column is stored continuously. + ids = (SQLINTEGER*)malloc(sizeof(ids[0]) * batchCount); + cols = (SQLCHAR*)malloc(sizeof(cols[0]) * batchCount * 50); + // Data size in each row for a column + bufLenIds = (SQLLEN*)malloc(sizeof(bufLenIds[0]) * batchCount); + bufLenCols = (SQLLEN*)malloc(sizeof(bufLenCols[0]) * batchCount); + // Whether this row needs to be processed. The value is SQL_PARAM_IGNORE or SQL_PARAM_PROCEED. + operptr = (SQLUSMALLINT*)malloc(sizeof(operptr[0]) * batchCount); + memset(operptr, 0, sizeof(operptr[0]) * batchCount); + // Processing result of the row + // Note: In the database, a statement belongs to one transaction. Therefore, data is processed as a unit. Either all data is inserted successfully or all data fails to be inserted. + statusptr = (SQLUSMALLINT*)malloc(sizeof(statusptr[0]) * batchCount); + memset(statusptr, 88, sizeof(statusptr[0]) * batchCount); + + if (NULL == ids || NULL == cols || NULL == bufLenCols || NULL == bufLenIds) + { + fprintf(stderr, "FAILED:\tmalloc data memory failed\n"); + goto exit; + } + + for (int i = 0; i < batchCount; i++) + { + ids[i] = i; + sprintf(cols + 50 * i, "column test value %d", i); + bufLenIds[i] = sizeof(ids[i]); + bufLenCols[i] = strlen(cols + 50 * i); + operptr[i] = (i < ignoreCount) ? SQL_PARAM_IGNORE : SQL_PARAM_PROCEED; + } + + // Allocate Statement Handle + retcode = SQLAllocHandle(SQL_HANDLE_STMT, hdbc, &hstmtinesrt); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLAllocHandle failed"); + goto exit; + } + + // Prepare Statement + sql = (SQLCHAR*)"insert into test_odbc_batch_insert values(?, ?)"; + retcode = SQLPrepare(hstmtinesrt, (SQLCHAR*) sql, SQL_NTS); + sprintf((char*)loginfo, "SQLPrepare log: %s", (char*)sql); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLPrepare failed"); + goto exit; + } + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAMSET_SIZE, (SQLPOINTER)batchCount, sizeof(batchCount)); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetStmtAttr failed"); + goto exit; + } + + retcode = SQLBindParameter(hstmtinesrt, 1, SQL_PARAM_INPUT, SQL_C_SLONG, SQL_INTEGER, sizeof(ids[0]), 0,&(ids[0]), 0, bufLenIds); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLBindParameter failed"); + goto exit; + } + + retcode = SQLBindParameter(hstmtinesrt, 2, SQL_PARAM_INPUT, SQL_C_CHAR, SQL_CHAR, 50, 50, cols, 50, bufLenCols); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLBindParameter failed"); + goto exit; + } + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAMS_PROCESSED_PTR, (SQLPOINTER)&process, sizeof(process)); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetStmtAttr failed"); + goto exit; + } + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAM_STATUS_PTR, (SQLPOINTER)statusptr, sizeof(statusptr[0]) * batchCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetStmtAttr failed"); + goto exit; + } + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAM_OPERATION_PTR, (SQLPOINTER)operptr, sizeof(operptr[0]) * batchCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLSetStmtAttr failed"); + goto exit; + } + + retcode = SQLExecute(hstmtinesrt); + sprintf((char*)loginfo, "SQLExecute stmt log: %s", (char*)sql); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute(hstmtinesrt) failed"); + goto exit; + + retcode = SQLRowCount(hstmtinesrt, &rowsCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLRowCount failed"); + goto exit; + } + + if (rowsCount != (batchCount - ignoreCount)) + { + sprintf(loginfo, "(batchCount - ignoreCount)(%d) != rowsCount(%d)", (batchCount - ignoreCount), rowsCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute failed"); + goto exit; + } + } + else + { + sprintf(loginfo, "(batchCount - ignoreCount)(%d) == rowsCount(%d)", (batchCount - ignoreCount), rowsCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute failed"); + goto exit; + } + } + + // check row number returned + if (rowsCount != process) + { + sprintf(loginfo, "process(%d) != rowsCount(%d)", process, rowsCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute failed"); + goto exit; + } + } + else + { + sprintf(loginfo, "process(%d) == rowsCount(%d)", process, rowsCount); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute failed"); + goto exit; + } + } + + for (int i = 0; i < batchCount; i++) + { + if (i < ignoreCount) + { + if (statusptr[i] != SQL_PARAM_UNUSED) + { + sprintf(loginfo, "statusptr[%d](%d) != SQL_PARAM_UNUSED", i, statusptr[i]); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute failed"); + goto exit; + } + } + } + else if (statusptr[i] != SQL_PARAM_SUCCESS) + { + sprintf(loginfo, "statusptr[%d](%d) != SQL_PARAM_SUCCESS", i, statusptr[i]); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLExecute failed"); + goto exit; + } + } + } + + retcode = SQLFreeHandle(SQL_HANDLE_STMT, hstmtinesrt); + sprintf((char*)loginfo, "SQLFreeHandle hstmtinesrt"); + + if (!SQL_SUCCEEDED(retcode)) { + printf("SQLFreeHandle failed"); + goto exit; + } + } + + +exit: + (void) printf ("\nComplete.\n"); + + // Connection + if (hdbc != SQL_NULL_HDBC) { + SQLDisconnect(hdbc); + SQLFreeHandle(SQL_HANDLE_DBC, hdbc); + } + + // Environment + if (henv != SQL_NULL_HENV) + SQLFreeHandle(SQL_HANDLE_ENV, henv); + + return 0; +} +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/5.1-typical-application-scenarios-and-configurations.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/5.1-typical-application-scenarios-and-configurations.md new file mode 100644 index 0000000000000000000000000000000000000000..7017deb1846b5e1fab85f9a0981ded031b1d75cd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/5.1-typical-application-scenarios-and-configurations.md @@ -0,0 +1,496 @@ +--- +title: Typical Application Scenarios and Configurations +summary: Typical Application Scenarios and Configurations +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Typical Application Scenarios and Configurations + +## Log Diagnosis Scenario + +ODBC logs are classified into unixODBC driver manager logs and psqlODBC driver logs. The former is used to trace whether the application API is successfully executed, and the latter is used to locate problems based on DFX logs generated during underlying implementation. + +The unixODBC log needs to be configured in the **odbcinst.ini** file: + +```bash +[ODBC] +Trace=Yes +TraceFile=/path/to/odbctrace.log + +[GaussMPP] +Driver64=/usr/local/lib/psqlodbcw.so +setup=/usr/local/lib/psqlodbcw.so +``` + +You only need to add the following information to the **odbc.ini** file: + +```bash +[gaussdb] +Driver=GaussMPP +Servername=10.10.0.13 (database server IP address) +... +Debug=1 (Enable the debug log function of the driver.) +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** The unixODBC logs are generated in the path configured by **TraceFile**. The psqlODBC generates the **mylog_***xxx***.log** file in the **/tmp/** directory. + +## High Performance + +If a large amount of data needs to be inserted, you are advised to perform the following operations: + +- You need to set **UseBatchProtocol** to **1** in the **odbc.ini** file and **support_batch_bind** to **on** in the database. +- The ODBC program binding type must be the same as that in the database. +- The character set of the client is the same as that of the database. +- The transaction is committed manually. + +**odbc.ini** configuration file: + +```bash +[gaussdb] +Driver=GaussMPP +Servername=10.10.0.13 (database server IP address) +... +UseBatchProtocol=1 (enabled by default) +ConnSettings=set client_encoding=UTF8 (Set the character code on the client to be the same as that on the server.) +``` + +Binding type case: + +```c +#include +#include +#include +#include +#include +#include + +#define MESSAGE_BUFFER_LEN 128 +SQLHANDLE h_env = NULL; +SQLHANDLE h_conn = NULL; +SQLHANDLE h_stmt = NULL; +void print_error() +{ + SQLCHAR Sqlstate[SQL_SQLSTATE_SIZE+1]; + SQLINTEGER NativeError; + SQLCHAR MessageText[MESSAGE_BUFFER_LEN]; + SQLSMALLINT TextLength; + SQLRETURN ret = SQL_ERROR; + + ret = SQLGetDiagRec(SQL_HANDLE_STMT, h_stmt, 1, Sqlstate, &NativeError, MessageText, MESSAGE_BUFFER_LEN, &TextLength); + if ( SQL_SUCCESS == ret) + { + printf("\n STMT ERROR-%05d %s", NativeError, MessageText); + return; + } + + ret = SQLGetDiagRec(SQL_HANDLE_DBC, h_conn, 1, Sqlstate, &NativeError, MessageText, MESSAGE_BUFFER_LEN, &TextLength); + if ( SQL_SUCCESS == ret) + { + printf("\n CONN ERROR-%05d %s", NativeError, MessageText); + return; + } + + ret = SQLGetDiagRec(SQL_HANDLE_ENV, h_env, 1, Sqlstate, &NativeError, MessageText, MESSAGE_BUFFER_LEN, &TextLength); + if ( SQL_SUCCESS == ret) + { + printf("\n ENV ERROR-%05d %s", NativeError, MessageText); + return; + } + + return; +} + +/* Expect the function to return SQL_SUCCESS. */ +#define RETURN_IF_NOT_SUCCESS(func) \ +{\ + SQLRETURN ret_value = (func);\ + if (SQL_SUCCESS != ret_value)\ + {\ + print_error();\ + printf("\n failed line = %u: expect SQL_SUCCESS, but ret = %d", __LINE__, ret_value);\ + return SQL_ERROR; \ + }\ +} + +/* Expect the function to return SQL_SUCCESS. */ +#define RETURN_IF_NOT_SUCCESS_I(i, func) \ +{\ + SQLRETURN ret_value = (func);\ + if (SQL_SUCCESS != ret_value)\ + {\ + print_error();\ + printf("\n failed line = %u (i=%d): : expect SQL_SUCCESS, but ret = %d", __LINE__, (i), ret_value);\ + return SQL_ERROR; \ + }\ +} + +/* Expect the function to return SQL_SUCCESS_WITH_INFO. */ +#define RETURN_IF_NOT_SUCCESS_INFO(func) \ +{\ + SQLRETURN ret_value = (func);\ + if (SQL_SUCCESS_WITH_INFO != ret_value)\ + {\ + print_error();\ + printf("\n failed line = %u: expect SQL_SUCCESS_WITH_INFO, but ret = %d", __LINE__, ret_value);\ + return SQL_ERROR; \ + }\ +} + +/* Expect the values are the same. */ +#define RETURN_IF_NOT(expect, value) \ +if ((expect) != (value))\ +{\ + printf("\n failed line = %u: expect = %u, but value = %u", __LINE__, (expect), (value)); \ + return SQL_ERROR;\ +} + +/* Expect the character strings are the same. */ +#define RETURN_IF_NOT_STRCMP_I(i, expect, value) \ +if (( NULL == (expect) ) || (NULL == (value)))\ +{\ + printf("\n failed line = %u (i=%u): input NULL pointer !", __LINE__, (i)); \ + return SQL_ERROR; \ +}\ +else if (0 != strcmp((expect), (value)))\ +{\ + printf("\n failed line = %u (i=%u): expect = %s, but value = %s", __LINE__, (i), (expect), (value)); \ + return SQL_ERROR;\ +} + + +// prepare + execute SQL statement +int execute_cmd(SQLCHAR *sql) +{ + if ( NULL == sql ) + { + return SQL_ERROR; + } + + if ( SQL_SUCCESS != SQLPrepare(h_stmt, sql, SQL_NTS)) + { + return SQL_ERROR; + } + + if ( SQL_SUCCESS != SQLExecute(h_stmt)) + { + return SQL_ERROR; + } + + return SQL_SUCCESS; +} +// execute + commit handle +int commit_exec() +{ + if ( SQL_SUCCESS != SQLExecute(h_stmt)) + { + return SQL_ERROR; + } + + // Manual committing + if ( SQL_SUCCESS != SQLEndTran(SQL_HANDLE_DBC, h_conn, SQL_COMMIT)) + { + return SQL_ERROR; + } + + return SQL_SUCCESS; +} + +int begin_unit_test() +{ + SQLINTEGER ret; + + /* Allocate an environment handle. */ + ret = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &h_env); + if ((SQL_SUCCESS != ret) && (SQL_SUCCESS_WITH_INFO != ret)) + { + printf("\n begin_unit_test::SQLAllocHandle SQL_HANDLE_ENV failed ! ret = %d", ret); + return SQL_ERROR; + } + + /* Set the version number before connection. */ + if (SQL_SUCCESS != SQLSetEnvAttr(h_env, SQL_ATTR_ODBC_VERSION, (SQLPOINTER)SQL_OV_ODBC3, 0)) + { + print_error(); + printf("\n begin_unit_test::SQLSetEnvAttr SQL_ATTR_ODBC_VERSION failed ! ret = %d", ret); + SQLFreeHandle(SQL_HANDLE_ENV, h_env); + return SQL_ERROR; + } + + /* Allocate a connection handle. */ + ret = SQLAllocHandle(SQL_HANDLE_DBC, h_env, &h_conn); + if (SQL_SUCCESS != ret) + { + print_error(); + printf("\n begin_unit_test::SQLAllocHandle SQL_HANDLE_DBC failed ! ret = %d", ret); + SQLFreeHandle(SQL_HANDLE_ENV, h_env); + return SQL_ERROR; + } + + /* Establish a connection. */ + ret = SQLConnect(h_conn, (SQLCHAR*) "gaussdb", SQL_NTS, + (SQLCHAR*) NULL, 0, NULL, 0); + if (SQL_SUCCESS != ret) + { + print_error(); + printf("\n begin_unit_test::SQLConnect failed ! ret = %d", ret); + SQLFreeHandle(SQL_HANDLE_DBC, h_conn); + SQLFreeHandle(SQL_HANDLE_ENV, h_env); + return SQL_ERROR; + } + + /* Allocate a statement handle. */ + ret = SQLAllocHandle(SQL_HANDLE_STMT, h_conn, &h_stmt); + if (SQL_SUCCESS != ret) + { + print_error(); + printf("\n begin_unit_test::SQLAllocHandle SQL_HANDLE_STMT failed ! ret = %d", ret); + SQLFreeHandle(SQL_HANDLE_DBC, h_conn); + SQLFreeHandle(SQL_HANDLE_ENV, h_env); + return SQL_ERROR; + } + + return SQL_SUCCESS; +} + +void end_unit_test() +{ + /* Release a statement handle. */ + if (NULL != h_stmt) + { + SQLFreeHandle(SQL_HANDLE_STMT, h_stmt); + } + + /* Release a connection handle. */ + if (NULL != h_conn) + { + SQLDisconnect(h_conn); + SQLFreeHandle(SQL_HANDLE_DBC, h_conn); + } + + /* Release an environment handle. */ + if (NULL != h_env) + { + SQLFreeHandle(SQL_HANDLE_ENV, h_env); + } + + return; +} + +int main() +{ + // begin test + if (begin_unit_test() != SQL_SUCCESS) + { + printf("\n begin_test_unit failed."); + return SQL_ERROR; + } + // The handle configuration is the same as that in the preceding case + int i = 0; + SQLCHAR* sql_drop = "drop table if exists test_bindnumber_001"; + SQLCHAR* sql_create = "create table test_bindnumber_001(" + "f4 number, f5 number(10, 2)" + ")"; + SQLCHAR* sql_insert = "insert into test_bindnumber_001 values(?, ?)"; + SQLCHAR* sql_select = "select * from test_bindnumber_001"; + SQLLEN RowCount; + SQL_NUMERIC_STRUCT st_number; + SQLCHAR getValue[2][MESSAGE_BUFFER_LEN]; + + /* Step 1. Create a table. */ + RETURN_IF_NOT_SUCCESS(execute_cmd(sql_drop)); + RETURN_IF_NOT_SUCCESS(execute_cmd(sql_create)); + + /* Step 2.1 Bind parameters using the SQL_NUMERIC_STRUCT structure. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + + // First line: 1234.5678 + memset(st_number.val, 0, SQL_MAX_NUMERIC_LEN); + st_number.precision = 8; + st_number.scale = 4; + st_number.sign = 1; + st_number.val[0] = 0x4E; + st_number.val[1] = 0x61; + st_number.val[2] = 0xBC; + + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_NUMERIC, SQL_NUMERIC, sizeof(SQL_NUMERIC_STRUCT), 4, &st_number, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_NUMERIC, SQL_NUMERIC, sizeof(SQL_NUMERIC_STRUCT), 4, &st_number, 0, NULL)); + + // Disable the automatic commit function. + SQLSetConnectAttr(h_conn, SQL_ATTR_AUTOCOMMIT, (SQLPOINTER)SQL_AUTOCOMMIT_OFF, 0); + + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + // Second line: 12345678 + memset(st_number.val, 0, SQL_MAX_NUMERIC_LEN); + st_number.precision = 8; + st_number.scale = 0; + st_number.sign = 1; + st_number.val[0] = 0x4E; + st_number.val[1] = 0x61; + st_number.val[2] = 0xBC; + + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_NUMERIC, SQL_NUMERIC, sizeof(SQL_NUMERIC_STRUCT), 0, &st_number, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_NUMERIC, SQL_NUMERIC, sizeof(SQL_NUMERIC_STRUCT), 0, &st_number, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + // Third line: 12345678 + memset(st_number.val, 0, SQL_MAX_NUMERIC_LEN); + st_number.precision = 0; + st_number.scale = 4; + st_number.sign = 1; + st_number.val[0] = 0x4E; + st_number.val[1] = 0x61; + st_number.val[2] = 0xBC; + + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_NUMERIC, SQL_NUMERIC, sizeof(SQL_NUMERIC_STRUCT), 4, &st_number, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_NUMERIC, SQL_NUMERIC, sizeof(SQL_NUMERIC_STRUCT), 4, &st_number, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + + /* Step 2.2 Bind parameters by using the SQL_C_CHAR character string in the fourth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + SQLCHAR* szNumber = "1234.5678"; + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_CHAR, SQL_NUMERIC, strlen(szNumber), 0, szNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_CHAR, SQL_NUMERIC, strlen(szNumber), 0, szNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Step 2.3 Bind parameters by using SQL_C_FLOAT in the fifth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + SQLREAL fNumber = 1234.5678; + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_FLOAT, SQL_NUMERIC, sizeof(fNumber), 4, &fNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_FLOAT, SQL_NUMERIC, sizeof(fNumber), 4, &fNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Step 2.4 Bind parameters by using SQL_C_DOUBLE in the sixth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + SQLDOUBLE dNumber = 1234.5678; + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_DOUBLE, SQL_NUMERIC, sizeof(dNumber), 4, &dNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_DOUBLE, SQL_NUMERIC, sizeof(dNumber), 4, &dNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + SQLBIGINT bNumber1 = 0xFFFFFFFFFFFFFFFF; + SQLBIGINT bNumber2 = 12345; + + /* Step 2.5 Bind parameters by using SQL_C_SBIGINT in the seventh line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_SBIGINT, SQL_NUMERIC, sizeof(bNumber1), 4, &bNumber1, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_SBIGINT, SQL_NUMERIC, sizeof(bNumber2), 4, &bNumber2, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Step 2.6 Bind parameters by using SQL_C_UBIGINT in the eighth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_UBIGINT, SQL_NUMERIC, sizeof(bNumber1), 4, &bNumber1, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_UBIGINT, SQL_NUMERIC, sizeof(bNumber2), 4, &bNumber2, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + SQLLEN lNumber1 = 0xFFFFFFFFFFFFFFFF; + SQLLEN lNumber2 = 12345; + + /* Step 2.7 Bind parameters by using SQL_C_LONG in the ninth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_LONG, SQL_NUMERIC, sizeof(lNumber1), 0, &lNumber1, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_LONG, SQL_NUMERIC, sizeof(lNumber2), 0, &lNumber2, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Step 2.8 Bind parameters by using SQL_C_ULONG in the tenth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_ULONG, SQL_NUMERIC, sizeof(lNumber1), 0, &lNumber1, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_ULONG, SQL_NUMERIC, sizeof(lNumber2), 0, &lNumber2, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + SQLSMALLINT sNumber = 0xFFFF; + + /* Step 2.9 Bind parameters by using SQL_C_SHORT in the eleventh line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_SHORT, SQL_NUMERIC, sizeof(sNumber), 0, &sNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_SHORT, SQL_NUMERIC, sizeof(sNumber), 0, &sNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Step 2.10 Bind parameters by using SQL_C_USHORT in the twelfth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_USHORT, SQL_NUMERIC, sizeof(sNumber), 0, &sNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_USHORT, SQL_NUMERIC, sizeof(sNumber), 0, &sNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + SQLCHAR cNumber = 0xFF; + + /* Step 2.11 Bind parameters by using SQL_C_TINYINT in the thirteenth line. */ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_TINYINT, SQL_NUMERIC, sizeof(cNumber), 0, &cNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_TINYINT, SQL_NUMERIC, sizeof(cNumber), 0, &cNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Step 2.12 Bind parameters by using SQL_C_UTINYINT in the fourteenth line.*/ + RETURN_IF_NOT_SUCCESS(SQLPrepare(h_stmt, sql_insert, SQL_NTS)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 1, SQL_PARAM_INPUT, SQL_C_UTINYINT, SQL_NUMERIC, sizeof(cNumber), 0, &cNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(SQLBindParameter(h_stmt, 2, SQL_PARAM_INPUT, SQL_C_UTINYINT, SQL_NUMERIC, sizeof(cNumber), 0, &cNumber, 0, NULL)); + RETURN_IF_NOT_SUCCESS(commit_exec()); + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(1, RowCount); + + /* Use the character string type to unify the expectation. */ + SQLCHAR* expectValue[14][2] = {{"1234.5678", "1234.57"}, + {"12345678", "12345678"}, + {"0", "0"}, + {"1234.5678", "1234.57"}, + {"1234.5677", "1234.57"}, + {"1234.5678", "1234.57"}, + {"-1", "12345"}, + {"18446744073709551615", "12345"}, + {"-1", "12345"}, + {"4294967295", "12345"}, + {"-1", "-1"}, + {"65535", "65535"}, + {"-1", "-1"}, + {"255", "255"}, + }; + + RETURN_IF_NOT_SUCCESS(execute_cmd(sql_select)); + while ( SQL_NO_DATA != SQLFetch(h_stmt)) + { + RETURN_IF_NOT_SUCCESS_I(i, SQLGetData(h_stmt, 1, SQL_C_CHAR, &getValue[0], MESSAGE_BUFFER_LEN, NULL)); + RETURN_IF_NOT_SUCCESS_I(i, SQLGetData(h_stmt, 2, SQL_C_CHAR, &getValue[1], MESSAGE_BUFFER_LEN, NULL)); + + //RETURN_IF_NOT_STRCMP_I(i, expectValue[i][0], getValue[0]); + //RETURN_IF_NOT_STRCMP_I(i, expectValue[i][1], getValue[1]); + i++; + } + + RETURN_IF_NOT_SUCCESS(SQLRowCount(h_stmt, &RowCount)); + RETURN_IF_NOT(i, RowCount); + SQLCloseCursor(h_stmt); + /* Final step. Delete the table and restore the environment. */ + RETURN_IF_NOT_SUCCESS(execute_cmd(sql_drop)); + + end_unit_test(); +} +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** In the preceding example, the number column is defined. When the **SQLBindParameter** API is called, the performance of binding SQL_NUMERIC is higher than that of SQL_LONG. If char is used, the data type needs to be converted when data is inserted to the database server, causing a performance bottleneck. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-0-odbc-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-0-odbc-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..2afef8791be5a7e9a2cdb15fe82800702d437a08 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-0-odbc-overview.md @@ -0,0 +1,10 @@ +--- +title: Description +summary: Description +author: Guo Huan +date: 2021-05-17 +--- + +# Description + +The ODBC interface is a set of API functions provided to users. This chapter describes its common interfaces. For details on other interfaces, see "ODBC Programmer's Reference" at MSDN (). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-1-SQLAllocEnv.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-1-SQLAllocEnv.md new file mode 100644 index 0000000000000000000000000000000000000000..69cde828e5f701728bc3375c296d23da46828b58 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-1-SQLAllocEnv.md @@ -0,0 +1,10 @@ +--- +title: SQLAllocEnv +summary: SQLAllocEnv +author: Guo Huan +date: 2021-05-17 +--- + +# SQLAllocEnv + +In ODBC 3.x, SQLAllocEnv (an ODBC 2.x function) was deprecated and replaced by SQLAllocHandle. For details, see [SQLAllocHandle](2-3-SQLAllocHandle.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-10-SQLExecDirect.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-10-SQLExecDirect.md new file mode 100644 index 0000000000000000000000000000000000000000..b939ecb386049d39c0d3b8086331e745aab302e8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-10-SQLExecDirect.md @@ -0,0 +1,48 @@ +--- +title: SQLExecDirect +summary: SQLExecDirect +author: Guo Huan +date: 2021-05-17 +--- + +# SQLExecDirect + +## Function + +SQLExecDirect is used to execute a prepared SQL statement specified in this parameter. This is the fastest method for executing only one SQL statement at a time. + +## Prototype + +``` +SQLRETURN SQLExecDirect(SQLHSTMT StatementHandle, + SQLCHAR *StatementText, + SQLINTEGER TextLength); +``` + +## Parameter + +**Table 1** SQLExecDirect parameters + +| **Keyword** | **Parameter Description** | +| :-------------- | :----------------------------------------------------------- | +| StatementHandle | Statement handle, obtained from SQLAllocHandle. | +| StatementText | SQL statement to be executed. One SQL statement can be executed at a time. | +| TextLength | Length of **StatementText**. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_NEED_DATA** indicates that parameters provided before executing the SQL statement are insufficient. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. +- **SQL_STILL_EXECUTING** indicates that the statement is being executed. +- **SQL_NO_DATA** indicates that the SQL statement does not return a result set. + +## Precautions + +If SQLExecDirect returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-11-SQLExecute.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-11-SQLExecute.md new file mode 100644 index 0000000000000000000000000000000000000000..378e20e48a984a8c4819e4a184b6e74e1f0f8115 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-11-SQLExecute.md @@ -0,0 +1,44 @@ +--- +title: SQLExecute +summary: SQLExecute +author: Guo Huan +date: 2021-05-17 +--- + +# SQLExecute + +## Function + +SQLExecute is used to execute a prepared SQL statement using SQLPrepare. The statement is executed using the current value of any application variables that were bound to parameter markers by SQLBindParameter. + +## Prototype + +``` +SQLRETURN SQLExecute(SQLHSTMT StatementHandle); +``` + +## Parameter + +**Table 1** SQLExecute parameters + +| **Keyword** | **Parameter Description** | +| :-------------- | :------------------------------- | +| StatementHandle | Statement handle to be executed. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_NEED_DATA** indicates that parameters provided before executing the SQL statement are insufficient. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_NO_DATA** indicates that the SQL statement does not return a result set. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. +- **SQL_STILL_EXECUTING** indicates that the statement is being executed. + +## Precautions + +If SQLExecute returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-12-SQLFetch.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-12-SQLFetch.md new file mode 100644 index 0000000000000000000000000000000000000000..f32a9a602920afebed5019f965ff755de70890fc --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-12-SQLFetch.md @@ -0,0 +1,43 @@ +--- +title: SQLFetch +summary: SQLFetch +author: Guo Huan +date: 2021-05-17 +--- + +# SQLFetch + +## Function + +SQLFetch is used to advance the cursor to the next row of the result set and retrieve any bound columns. + +## Prototype + +``` +CSQLRETURN SQLFetch(SQLHSTMT StatementHandle); +``` + +## Parameter + +**Table 1** SQLFetch parameters + +| **Keyword** | **Parameter Description** | +| :-------------- | :---------------------------------------------- | +| StatementHandle | Statement handle, obtained from SQLAllocHandle. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_NO_DATA** indicates that the SQL statement does not return a result set. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. +- **SQL_STILL_EXECUTING** indicates that the statement is being executed. + +## Precautions + +If SQLFetch returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-13-SQLFreeStmt.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-13-SQLFreeStmt.md new file mode 100644 index 0000000000000000000000000000000000000000..b0c549f34400c419c86bb40ce91c3fd9ea69e79d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-13-SQLFreeStmt.md @@ -0,0 +1,10 @@ +--- +title: SQLFreeStmt +summary: SQLFreeStmt +author: Guo Huan +date: 2021-05-17 +--- + +# SQLFreeStmt + +In ODBC 3.x, SQLFreeStmt (an ODBC 2.x function) was deprecated and replaced by SQLFreeHandle. For details, see [SQLFreeHandle](2-15-SQLFreeHandle.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-14-SQLFreeConnect.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-14-SQLFreeConnect.md new file mode 100644 index 0000000000000000000000000000000000000000..654509c75f7569cf5a21f97e398179f3d219a7a4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-14-SQLFreeConnect.md @@ -0,0 +1,10 @@ +--- +title: SQLFreeConnect +summary: SQLFreeConnect +author: Guo Huan +date: 2021-05-17 +--- + +# SQLFreeConnect + +In ODBC 3.x, SQLFreeConnect (an ODBC 2.x function) was deprecated and replaced by SQLFreeHandle. For details, see [SQLFreeHandle](2-15-SQLFreeHandle.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-15-SQLFreeHandle.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-15-SQLFreeHandle.md new file mode 100644 index 0000000000000000000000000000000000000000..e689e0ce0640ec3abcc027372d7598f2e6c99017 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-15-SQLFreeHandle.md @@ -0,0 +1,43 @@ +--- +title: SQLFreeHandle +summary: SQLFreeHandle +author: Guo Huan +date: 2021-05-17 +--- + +# SQLFreeHandle + +## Function + +SQLFreeHandle is used to release resources associated with a specific environment, connection, or statement handle. It replaces the ODBC 2.x functions: SQLFreeEnv, SQLFreeConnect, and SQLFreeStmt. + +## Prototype + +``` +SQLRETURN SQLFreeHandle(SQLSMALLINT HandleType, + SQLHANDLE Handle); +``` + +## Parameter + +**Table 1** SQLFreeHandle parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| HandleType | Type of handle to be freed by SQLFreeHandle. The value must be one of the following:
- SQL_HANDLE_ENV
- SQL_HANDLE_DBC
- SQL_HANDLE_STMT
- SQL_HANDLE_DESC
If **HandleType** is not one of the preceding values, SQLFreeHandle returns **SQL_INVALID_HANDLE**. | +| Handle | Name of the handle to be freed. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLFreeHandle returns **SQL_ERROR**, the handle is still valid. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-16-SQLFreeEnv.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-16-SQLFreeEnv.md new file mode 100644 index 0000000000000000000000000000000000000000..b5f7ce560230b9bb32a7a58aa8cb1ce92da50457 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-16-SQLFreeEnv.md @@ -0,0 +1,10 @@ +--- +title: SQLFreeEnv +summary: SQLFreeEnv +author: Guo Huan +date: 2021-05-17 +--- + +# SQLFreeEnv + +In ODBC 3.x, SQLFreeEnv (an ODBC 2.x function) was deprecated and replaced by SQLFreeHandle. For details, see [SQLFreeHandle](2-15-SQLFreeHandle.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-17-SQLPrepare.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-17-SQLPrepare.md new file mode 100644 index 0000000000000000000000000000000000000000..b2b6f1a0a8b8ca9458900567682bbd8a0d43a708 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-17-SQLPrepare.md @@ -0,0 +1,46 @@ +--- +title: SQLPrepare +summary: SQLPrepare +author: Guo Huan +date: 2021-05-17 +--- + +# SQLPrepare + +## Function + +SQLPrepare is used to prepare an SQL statement to be executed. + +## Prototype + +``` +SQLRETURN SQLPrepare(SQLHSTMT StatementHandle, + SQLCHAR *StatementText, + SQLINTEGER TextLength); +``` + +## Parameter + +**Table 1** SQLPrepare parameters + +| **Keyword** | **Parameter Description** | +| :-------------- | :--------------------------- | +| StatementHandle | Statement handle. | +| StatementText | SQL text string. | +| TextLength | Length of **StatementText**. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. +- **SQL_STILL_EXECUTING** indicates that the statement is being executed. + +## Precautions + +If SQLPrepare returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-18-SQLGetData.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-18-SQLGetData.md new file mode 100644 index 0000000000000000000000000000000000000000..22db826d5d48391a405f28322266a6108f3fc533 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-18-SQLGetData.md @@ -0,0 +1,53 @@ +--- +title: SQLGetData +summary: SQLGetData +author: Guo Huan +date: 2021-05-17 +--- + +# SQLGetData + +## Function + +SQLGetData is used to retrieve data for a single column in the result set. It can be called for many times to retrieve data of variable lengths. + +## Prototype + +``` +SQLRETURN SQLGetData(SQLHSTMT StatementHandle, + SQLUSMALLINT Col_or_Param_Num, + SQLSMALLINT TargetType, + SQLPOINTER TargetValuePtr, + SQLLEN BufferLength, + SQLLEN *StrLen_or_IndPtr); +``` + +## Parameter + +**Table 1** SQLGetData parameters + +| **Keyword** | **Parameter Description** | +| :--------------- | :----------------------------------------------------------- | +| StatementHandle | Statement handle, obtained from SQLAllocHandle. | +| Col_or_Param_Num | Column number for which the data retrieval is requested. The column number starts with 1 and increases in ascending order. The number of the bookmark column is 0. | +| TargetType | C data type in the TargetValuePtr buffer. If **TargetType** is **SQL_ARD_TYPE**, the driver uses the data type of the **SQL_DESC_CONCISE_TYPE** field in ARD. If **TargetType** is **SQL_C_DEFAULT**, the driver selects a default data type according to the source SQL data type. | +| TargetValuePtr | **Output parameter**: pointer to the pointer that points to the buffer where the data is located. | +| BufferLength | Size of the buffer pointed to by **TargetValuePtr**. | +| StrLen_or_IndPtr | **Output parameter**: pointer to the buffer where the length or identifier value is returned. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_NO_DATA** indicates that the SQL statement does not return a result set. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. +- **SQL_STILL_EXECUTING** indicates that the statement is being executed. + +## Precautions + +If SQLGetData returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-19-SQLGetDiagRec.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-19-SQLGetDiagRec.md new file mode 100644 index 0000000000000000000000000000000000000000..291300b0a14c2903d12b3d1ad98f1081f2eb3a9d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-19-SQLGetDiagRec.md @@ -0,0 +1,74 @@ +--- +title: SQLGetDiagRec +summary: SQLGetDiagRec +author: Guo Huan +date: 2021-05-17 +--- + +# SQLGetDiagRec + +## Function + +SQLGetDiagRec is used to return the current values of multiple fields in a diagnostic record that contains error, warning, and status information. + +## Prototype + +``` +SQLRETURN SQLGetDiagRec(SQLSMALLINT HandleType + SQLHANDLE Handle, + SQLSMALLINT RecNumber, + SQLCHAR *SQLState, + SQLINTEGER *NativeErrorPtr, + SQLCHAR *MessageText, + SQLSMALLINT BufferLength + SQLSMALLINT *TextLengthPtr); +``` + +## Parameter + +**Table 1** SQLGetDiagRec parameters + +| **Keyword** | **Parameter Description** | +| :------------- | :----------------------------------------------------------- | +| HandleType | A handle-type identifier that describes the type of handle for which diagnostics are desired. The value must be one of the following:
- SQL_HANDLE_ENV
- SQL_HANDLE_DBC
- SQL_HANDLE_STMT
- SQL_HANDLE_DESC | +| Handle | A handle for the diagnostic data structure. Its type is indicated by **HandleType**. If **HandleType** is **SQL_HANDLE_ENV**, **Handle** may be a shared or non-shared environment handle. | +| RecNumber | Status record from which the application seeks information. **RecNumber** starts with 1. | +| SQLState | **Output parameter**: pointer to a buffer that saves the 5-character **SQLSTATE** code pertaining to **RecNumber**. | +| NativeErrorPtr | **Output parameter**: pointer to a buffer that saves the native error code. | +| MessageText | Pointer to a buffer that saves text strings of diagnostic information. | +| BufferLength | Length of **MessageText**. | +| TextLengthPtr | **Output parameter**: pointer to the buffer, the total number of bytes in the returned **MessageText**. If the number of bytes available to return is greater than **BufferLength**, then the diagnostics information text in **MessageText** is truncated to **BufferLength** minus the length of the null termination character. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +SQLGetDiagRec does not release diagnostic records for itself. It uses the following return values to report execution results: + +- **SQL_SUCCESS** indicates that the function successfully returns diagnostic information. +- **SQL_SUCCESS_WITH_INFO** indicates that the **MessageText** buffer is too small to hold the requested diagnostic information. No diagnostic records are generated. +- **SQL_INVALID_HANDLE** indicates that the handle indicated by **HandType** and **Handle** is an invalid handle. +- **SQL_ERROR** indicates that **RecNumber** is less than or equal to 0 or that **BufferLength** is smaller than 0. + +If an ODBC function returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec to obtain the **SQLSTATE** value. The possible **SQLSTATE** values are listed as follows: + +**Table 2** SQLSTATE values + +| SQLSATATE | Error | Description | +| :-------- | :------------------------------------ | :----------------------------------------------------------- | +| HY000 | General error. | An error occurred for which there is no specific SQLSTATE. | +| HY001 | Memory allocation error. | The driver is unable to allocate memory required to support execution or completion of the function. | +| HY008 | Operation canceled. | SQLCancel is called to terminate the statement execution, but the StatementHandle function is still called. | +| HY010 | Function sequence error. | The function is called prior to sending data to data parameters or columns being executed. | +| HY013 | Memory management error. | The function fails to be called. The error may be caused by low memory conditions. | +| HYT01 | Connection timeout. | The timeout period expired before the application was able to connect to the data source. | +| IM001 | Function not supported by the driver. | The called function is not supported by the StatementHandle driver. | + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-2-SQLAllocConnect.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-2-SQLAllocConnect.md new file mode 100644 index 0000000000000000000000000000000000000000..cd0c16f4eb4d3e0643fe2159da971e36c256144a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-2-SQLAllocConnect.md @@ -0,0 +1,10 @@ +--- +title: SQLAllocConnect +summary: SQLAllocConnect +author: Guo Huan +date: 2021-05-17 +--- + +# SQLAllocConnect + +In ODBC 3.x, SQLAllocConnect (an ODBC 2.x function) was deprecated and replaced by SQLAllocHandle. For details, see [SQLAllocHandle](2-3-SQLAllocHandle.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-20-SQLSetConnectAttr.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-20-SQLSetConnectAttr.md new file mode 100644 index 0000000000000000000000000000000000000000..2e5dc4b43e5ac32ab426df63f4f6eabeb873f102 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-20-SQLSetConnectAttr.md @@ -0,0 +1,47 @@ +--- +title: SQLSetConnectAttr +summary: SQLSetConnectAttr +author: Guo Huan +date: 2021-05-17 +--- + +# SQLSetConnectAttr + +## Function + +SQLSetConnectAttr is used to set connection attributes. + +## Prototype + +``` +SQLRETURN SQLSetConnectAttr(SQLHDBC ConnectionHandle + SQLINTEGER Attribute, + SQLPOINTER ValuePtr, + SQLINTEGER StringLength); +``` + +## Parameter + +**Table 1** SQLSetConnectAttr parameters + +| **Keyword** | **Parameter Description** | +| :--------------- | :----------------------------------------------------------- | +| ConnectionHandle | Connection handle. | +| Attribute | Attribute to set. | +| ValuePtr | Pointer to the **Attribute** value. **ValuePtr** depends on the **Attribute** value, and can be a 32-bit unsigned integer value or a null-terminated string. If the **ValuePtr** parameter is a driver-specific value, it may be a signed integer. | +| StringLength | If **ValuePtr** points to a string or a binary buffer, **StringLength** is the length of ***ValuePtr**. If **ValuePtr** points to an integer, **StringLength** is ignored. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLSetConnectAttr returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_DBC** and **ConnectionHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-21-SQLSetEnvAttr.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-21-SQLSetEnvAttr.md new file mode 100644 index 0000000000000000000000000000000000000000..9a30d73e683ff99e32f6aac7b28765902582ec04 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-21-SQLSetEnvAttr.md @@ -0,0 +1,47 @@ +--- +title: SQLSetEnvAttr +summary: SQLSetEnvAttr +author: Guo Huan +date: 2021-05-17 +--- + +# SQLSetEnvAttr + +## Function + +SQLSetEnvAttr is used to set environment attributes. + +## Prototype + +``` +SQLRETURN SQLSetEnvAttr(SQLHENV EnvironmentHandle + SQLINTEGER Attribute, + SQLPOINTER ValuePtr, + SQLINTEGER StringLength); +``` + +## Parameter + +**Table 1** SQLSetEnvAttr parameters + +| **Keyword** | **Parameter Description** | +| :---------------- | :----------------------------------------------------------- | +| EnvironmentHandle | Environment handle. | +| Attribute | Environment attribute to be set. The value must be one of the following:
- **SQL_ATTR_ODBC_VERSION**: ODBC version
- **SQL_CONNECTION_POOLING**: connection pool attribute
- **SQL_OUTPUT_NTS**: string type returned by the driver | +| ValuePtr | Pointer to the **Attribute** value. **ValuePtr** depends on the **Attribute** value, and can be a 32-bit integer value or a null-terminated string. | +| StringLength | If **ValuePtr** points to a string or a binary buffer, **StringLength** is the length of ***ValuePtr**. If **ValuePtr** points to an integer, **StringLength** is ignored. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLSetEnvAttr returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, set **HandleType** and **Handle** to **SQL_HANDLE_ENV** and **EnvironmentHandle**, and obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-22-SQLSetStmtAttr.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-22-SQLSetStmtAttr.md new file mode 100644 index 0000000000000000000000000000000000000000..8da851db92e6cd63813a9bf4e5d1c6e6693ffa01 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-22-SQLSetStmtAttr.md @@ -0,0 +1,47 @@ +--- +title: SQLSetStmtAttr +summary: SQLSetStmtAttr +author: Guo Huan +date: 2021-05-17 +--- + +# SQLSetStmtAttr + +## Function + +SQLSetStmtAttr is used to set attributes related to a statement. + +## Prototype + +``` +SQLRETURN SQLSetStmtAttr(SQLHSTMT StatementHandle + SQLINTEGER Attribute, + SQLPOINTER ValuePtr, + SQLINTEGER StringLength); +``` + +## Parameter + +**Table 1** SQLSetStmtAttr parameters + +| **Keyword** | **Parameter Description** | +| :-------------- | :----------------------------------------------------------- | +| StatementHandle | Statement handle. | +| Attribute | Attribute to set. | +| ValuePtr | Pointer to the **Attribute** value. **ValuePtr** depends on the **Attribute** value, and can be a 32-bit unsigned integer value or a pointer to a null-terminated string, a binary buffer, or a driver-specified value. If the **ValuePtr** parameter is a driver-specific value, it may be a signed integer. | +| StringLength | If **ValuePtr** points to a string or a binary buffer, **StringLength** is the length of ***ValuePtr**. If **ValuePtr** points to an integer, **StringLength** is ignored. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLSetStmtAttr returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-23-Examples.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-23-Examples.md new file mode 100644 index 0000000000000000000000000000000000000000..ef42c9cba275467f59dce375514878ec989f1ad3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-23-Examples.md @@ -0,0 +1,345 @@ +--- +title: Examples +summary: Examples +author: Guo Huan +date: 2021-05-17 +--- + +# Examples + +## Code for Common Functions + +```c +// The following example shows how to obtain data from MogDB through the ODBC interface. +// DBtest.c (compile with: libodbc.so) +#include +#include +#include +#ifdef WIN32 +#include +#endif +SQLHENV V_OD_Env; // Handle ODBC environment +SQLHSTMT V_OD_hstmt; // Handle statement +SQLHDBC V_OD_hdbc; // Handle connection +char typename[100]; +SQLINTEGER value = 100; +SQLINTEGER V_OD_erg,V_OD_buffer,V_OD_err,V_OD_id; +int main(int argc,char *argv[]) +{ + // 1. Allocate an environment handle. + V_OD_erg = SQLAllocHandle(SQL_HANDLE_ENV,SQL_NULL_HANDLE,&V_OD_Env); + if ((V_OD_erg != SQL_SUCCESS) && (V_OD_erg != SQL_SUCCESS_WITH_INFO)) + { + printf("Error AllocHandle\n"); + exit(0); + } + // 2. Set environment attributes (version information). + SQLSetEnvAttr(V_OD_Env, SQL_ATTR_ODBC_VERSION, (void*)SQL_OV_ODBC3, 0); + // 3. Allocate a connection handle. + V_OD_erg = SQLAllocHandle(SQL_HANDLE_DBC, V_OD_Env, &V_OD_hdbc); + if ((V_OD_erg != SQL_SUCCESS) && (V_OD_erg != SQL_SUCCESS_WITH_INFO)) + { + SQLFreeHandle(SQL_HANDLE_ENV, V_OD_Env); + exit(0); + } + // 4. Set connection attributes. + SQLSetConnectAttr(V_OD_hdbc, SQL_ATTR_AUTOCOMMIT, SQL_AUTOCOMMIT_ON, 0); + // 5. Connect to the data source. userName and password indicate the username and password for connecting to the database. Set them as needed. + // If the username and password have been set in the odbc.ini file, you do not need to set userName or password here, retaining "" for them. However, you are not advised to do so because the username and password will be disclosed if the permission for odbc.ini is abused. + V_OD_erg = SQLConnect(V_OD_hdbc, (SQLCHAR*) "gaussdb", SQL_NTS, + (SQLCHAR*) "userName", SQL_NTS, (SQLCHAR*) "password", SQL_NTS); + if ((V_OD_erg != SQL_SUCCESS) && (V_OD_erg != SQL_SUCCESS_WITH_INFO)) + { + printf("Error SQLConnect %d\n",V_OD_erg); + SQLFreeHandle(SQL_HANDLE_ENV, V_OD_Env); + exit(0); + } + printf("Connected !\n"); + // 6. Set statement attributes. + SQLSetStmtAttr(V_OD_hstmt,SQL_ATTR_QUERY_TIMEOUT,(SQLPOINTER *)3,0); + // 7. Allocate a statement handle. + SQLAllocHandle(SQL_HANDLE_STMT, V_OD_hdbc, &V_OD_hstmt); + // 8. Run SQL statements. + SQLExecDirect(V_OD_hstmt,"drop table IF EXISTS customer_t1",SQL_NTS); + SQLExecDirect(V_OD_hstmt,"CREATE TABLE customer_t1(c_customer_sk INTEGER, c_customer_name VARCHAR(32));",SQL_NTS); + SQLExecDirect(V_OD_hstmt,"insert into customer_t1 values(25,li)",SQL_NTS); + // 9. Prepare for execution. + SQLPrepare(V_OD_hstmt,"insert into customer_t1 values(?)",SQL_NTS); + // 10. Bind parameters. + SQLBindParameter(V_OD_hstmt,1,SQL_PARAM_INPUT,SQL_C_SLONG,SQL_INTEGER,0,0, + &value,0,NULL); + // 11. Run prepared statements. + SQLExecute(V_OD_hstmt); + SQLExecDirect(V_OD_hstmt,"select id from testtable",SQL_NTS); + // 12. Obtain attributes of a specific column in the result set. + SQLColAttribute(V_OD_hstmt,1,SQL_DESC_TYPE,typename,100,NULL,NULL); + printf("SQLColAtrribute %s\n",typename); + // 13. Bind the result set. + SQLBindCol(V_OD_hstmt,1,SQL_C_SLONG, (SQLPOINTER)&V_OD_buffer,150, + (SQLLEN *)&V_OD_err); + // 14. Obtain data in the result set by executing SQLFetch. + V_OD_erg=SQLFetch(V_OD_hstmt); + // 15. Obtain and return data by executing SQLGetData. + while(V_OD_erg != SQL_NO_DATA) + { + SQLGetData(V_OD_hstmt,1,SQL_C_SLONG,(SQLPOINTER)&V_OD_id,0,NULL); + printf("SQLGetData ----ID = %d\n",V_OD_id); + V_OD_erg=SQLFetch(V_OD_hstmt); + }; + printf("Done !\n"); + // 16. Disconnect data source connections and release handles. + SQLFreeHandle(SQL_HANDLE_STMT,V_OD_hstmt); + SQLDisconnect(V_OD_hdbc); + SQLFreeHandle(SQL_HANDLE_DBC,V_OD_hdbc); + SQLFreeHandle(SQL_HANDLE_ENV, V_OD_Env); + return(0); + } +``` + +## Code for Batch Processing + +```c +/********************************************************************** +*Set UseBatchProtocol to 1 in the data source and set the database parameter support_batch_bind +*to on. +*The CHECK_ERROR command is used to check and print error information. +*This example is used to interactively obtain the DSN, data volume to be processed, and volume of ignored data from users, and insert required data into the test_odbc_batch_insert table. +***********************************************************************/ +#include +#include +#include +#include +#include + +#include "util.c" + +void Exec(SQLHDBC hdbc, SQLCHAR* sql) +{ + SQLRETURN retcode; // Return status + SQLHSTMT hstmt = SQL_NULL_HSTMT; // Statement handle + SQLCHAR loginfo[2048]; + + // Allocate Statement Handle + retcode = SQLAllocHandle(SQL_HANDLE_STMT, hdbc, &hstmt); + CHECK_ERROR(retcode, "SQLAllocHandle(SQL_HANDLE_STMT)", + hstmt, SQL_HANDLE_STMT); + + // Prepare Statement + retcode = SQLPrepare(hstmt, (SQLCHAR*) sql, SQL_NTS); + sprintf((char*)loginfo, "SQLPrepare log: %s", (char*)sql); + CHECK_ERROR(retcode, loginfo, hstmt, SQL_HANDLE_STMT); + + // Execute Statement + retcode = SQLExecute(hstmt); + sprintf((char*)loginfo, "SQLExecute stmt log: %s", (char*)sql); + CHECK_ERROR(retcode, loginfo, hstmt, SQL_HANDLE_STMT); + + // Free Handle + retcode = SQLFreeHandle(SQL_HANDLE_STMT, hstmt); + sprintf((char*)loginfo, "SQLFreeHandle stmt log: %s", (char*)sql); + CHECK_ERROR(retcode, loginfo, hstmt, SQL_HANDLE_STMT); +} + +int main () +{ + SQLHENV henv = SQL_NULL_HENV; + SQLHDBC hdbc = SQL_NULL_HDBC; + int batchCount = 1000; + SQLLEN rowsCount = 0; + int ignoreCount = 0; + + SQLRETURN retcode; + SQLCHAR dsn[1024] = {'\0'}; + SQLCHAR loginfo[2048]; + + // Interactively obtain data source names. + getStr("Please input your DSN", (char*)dsn, sizeof(dsn), 'N'); + // Interactively obtain the volume of data to be batch processed. + getInt("batchCount", &batchCount, 'N', 1); + do + { + // Interactively obtain the volume of batch processing data that is not inserted into the database. + getInt("ignoreCount", &ignoreCount, 'N', 1); + if (ignoreCount > batchCount) + { + printf("ignoreCount(%d) should be less than batchCount(%d)\n", ignoreCount, batchCount); + } + }while(ignoreCount > batchCount); + + retcode = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &henv); + CHECK_ERROR(retcode, "SQLAllocHandle(SQL_HANDLE_ENV)", + henv, SQL_HANDLE_ENV); + + // Set ODBC Verion + retcode = SQLSetEnvAttr(henv, SQL_ATTR_ODBC_VERSION, + (SQLPOINTER*)SQL_OV_ODBC3, 0); + CHECK_ERROR(retcode, "SQLSetEnvAttr(SQL_ATTR_ODBC_VERSION)", + henv, SQL_HANDLE_ENV); + + // Allocate Connection + retcode = SQLAllocHandle(SQL_HANDLE_DBC, henv, &hdbc); + CHECK_ERROR(retcode, "SQLAllocHandle(SQL_HANDLE_DBC)", + henv, SQL_HANDLE_DBC); + + // Set Login Timeout + retcode = SQLSetConnectAttr(hdbc, SQL_LOGIN_TIMEOUT, (SQLPOINTER)5, 0); + CHECK_ERROR(retcode, "SQLSetConnectAttr(SQL_LOGIN_TIMEOUT)", + hdbc, SQL_HANDLE_DBC); + + // Set Auto Commit + retcode = SQLSetConnectAttr(hdbc, SQL_ATTR_AUTOCOMMIT, + (SQLPOINTER)(1), 0); + CHECK_ERROR(retcode, "SQLSetConnectAttr(SQL_ATTR_AUTOCOMMIT)", + hdbc, SQL_HANDLE_DBC); + + // Connect to DSN + sprintf(loginfo, "SQLConnect(DSN:%s)", dsn); + retcode = SQLConnect(hdbc, (SQLCHAR*) dsn, SQL_NTS, + (SQLCHAR*) NULL, 0, NULL, 0); + CHECK_ERROR(retcode, loginfo, hdbc, SQL_HANDLE_DBC); + + // init table info. + Exec(hdbc, "drop table if exists test_odbc_batch_insert"); + Exec(hdbc, "create table test_odbc_batch_insert(id int primary key, col varchar2(50))"); + + // The following code constructs the data to be inserted based on the data volume entered by users: + { + SQLRETURN retcode; + SQLHSTMT hstmtinesrt = SQL_NULL_HSTMT; + int i; + SQLCHAR *sql = NULL; + SQLINTEGER *ids = NULL; + SQLCHAR *cols = NULL; + SQLLEN *bufLenIds = NULL; + SQLLEN *bufLenCols = NULL; + SQLUSMALLINT *operptr = NULL; + SQLUSMALLINT *statusptr = NULL; + SQLULEN process = 0; + + // Data is constructed by column. Each column is stored continuously. + ids = (SQLINTEGER*)malloc(sizeof(ids[0]) * batchCount); + cols = (SQLCHAR*)malloc(sizeof(cols[0]) * batchCount * 50); + // Data size in each row for a column + bufLenIds = (SQLLEN*)malloc(sizeof(bufLenIds[0]) * batchCount); + bufLenCols = (SQLLEN*)malloc(sizeof(bufLenCols[0]) * batchCount); + // Whether this row needs to be processed. The value is SQL_PARAM_IGNORE or SQL_PARAM_PROCEED. + operptr = (SQLUSMALLINT*)malloc(sizeof(operptr[0]) * batchCount); + memset(operptr, 0, sizeof(operptr[0]) * batchCount); + // Processing result of the row + // Note: In the database, a statement belongs to one transaction. Therefore, data is processed as a unit. Either all data is inserted successfully or all data fails to be inserted. + statusptr = (SQLUSMALLINT*)malloc(sizeof(statusptr[0]) * batchCount); + memset(statusptr, 88, sizeof(statusptr[0]) * batchCount); + + if (NULL == ids || NULL == cols || NULL == bufLenCols || NULL == bufLenIds) + { + fprintf(stderr, "FAILED:\tmalloc data memory failed\n"); + goto exit; + } + + for (int i = 0; i < batchCount; i++) + { + ids[i] = i; + sprintf(cols + 50 * i, "column test value %d", i); + bufLenIds[i] = sizeof(ids[i]); + bufLenCols[i] = strlen(cols + 50 * i); + operptr[i] = (i < ignoreCount) ? SQL_PARAM_IGNORE : SQL_PARAM_PROCEED; + } + + // Allocate Statement Handle + retcode = SQLAllocHandle(SQL_HANDLE_STMT, hdbc, &hstmtinesrt); + CHECK_ERROR(retcode, "SQLAllocHandle(SQL_HANDLE_STMT)", + hstmtinesrt, SQL_HANDLE_STMT); + + // Prepare Statement + sql = (SQLCHAR*)"insert into test_odbc_batch_insert values(?, ?)"; + retcode = SQLPrepare(hstmtinesrt, (SQLCHAR*) sql, SQL_NTS); + sprintf((char*)loginfo, "SQLPrepare log: %s", (char*)sql); + CHECK_ERROR(retcode, loginfo, hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAMSET_SIZE, (SQLPOINTER)batchCount, sizeof(batchCount)); + CHECK_ERROR(retcode, "SQLSetStmtAttr", hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLBindParameter(hstmtinesrt, 1, SQL_PARAM_INPUT, SQL_C_SLONG, SQL_INTEGER, sizeof(ids[0]), 0,&(ids[0]), 0, bufLenIds); + CHECK_ERROR(retcode, "SQLBindParameter for id", hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLBindParameter(hstmtinesrt, 2, SQL_PARAM_INPUT, SQL_C_CHAR, SQL_CHAR, 50, 50, cols, 50, bufLenCols); + CHECK_ERROR(retcode, "SQLBindParameter for cols", hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAMS_PROCESSED_PTR, (SQLPOINTER)&process, sizeof(process)); + CHECK_ERROR(retcode, "SQLSetStmtAttr for SQL_ATTR_PARAMS_PROCESSED_PTR", hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAM_STATUS_PTR, (SQLPOINTER)statusptr, sizeof(statusptr[0]) * batchCount); + CHECK_ERROR(retcode, "SQLSetStmtAttr for SQL_ATTR_PARAM_STATUS_PTR", hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLSetStmtAttr(hstmtinesrt, SQL_ATTR_PARAM_OPERATION_PTR, (SQLPOINTER)operptr, sizeof(operptr[0]) * batchCount); + CHECK_ERROR(retcode, "SQLSetStmtAttr for SQL_ATTR_PARAM_OPERATION_PTR", hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLExecute(hstmtinesrt); + sprintf((char*)loginfo, "SQLExecute stmt log: %s", (char*)sql); + CHECK_ERROR(retcode, loginfo, hstmtinesrt, SQL_HANDLE_STMT); + + retcode = SQLRowCount(hstmtinesrt, &rowsCount); + CHECK_ERROR(retcode, "SQLRowCount execution", hstmtinesrt, SQL_HANDLE_STMT); + + if (rowsCount != (batchCount - ignoreCount)) + { + sprintf(loginfo, "(batchCount - ignoreCount)(%d) != rowsCount(%d)", (batchCount - ignoreCount), rowsCount); + CHECK_ERROR(SQL_ERROR, loginfo, NULL, SQL_HANDLE_STMT); + } + else + { + sprintf(loginfo, "(batchCount - ignoreCount)(%d) == rowsCount(%d)", (batchCount - ignoreCount), rowsCount); + CHECK_ERROR(SQL_SUCCESS, loginfo, NULL, SQL_HANDLE_STMT); + } + + // check row number returned + if (rowsCount != process) + { + sprintf(loginfo, "process(%d) != rowsCount(%d)", process, rowsCount); + CHECK_ERROR(SQL_ERROR, loginfo, NULL, SQL_HANDLE_STMT); + } + else + { + sprintf(loginfo, "process(%d) == rowsCount(%d)", process, rowsCount); + CHECK_ERROR(SQL_SUCCESS, loginfo, NULL, SQL_HANDLE_STMT); + } + + for (int i = 0; i < batchCount; i++) + { + if (i < ignoreCount) + { + if (statusptr[i] != SQL_PARAM_UNUSED) + { + sprintf(loginfo, "statusptr[%d](%d) != SQL_PARAM_UNUSED", i, statusptr[i]); + CHECK_ERROR(SQL_ERROR, loginfo, NULL, SQL_HANDLE_STMT); + } + } + else if (statusptr[i] != SQL_PARAM_SUCCESS) + { + sprintf(loginfo, "statusptr[%d](%d) != SQL_PARAM_SUCCESS", i, statusptr[i]); + CHECK_ERROR(SQL_ERROR, loginfo, NULL, SQL_HANDLE_STMT); + } + } + + retcode = SQLFreeHandle(SQL_HANDLE_STMT, hstmtinesrt); + sprintf((char*)loginfo, "SQLFreeHandle hstmtinesrt"); + CHECK_ERROR(retcode, loginfo, hstmtinesrt, SQL_HANDLE_STMT); + } + + +exit: + printf ("\nComplete.\n"); + + // Connection + if (hdbc != SQL_NULL_HDBC) { + SQLDisconnect(hdbc); + SQLFreeHandle(SQL_HANDLE_DBC, hdbc); + } + + // Environment + if (henv != SQL_NULL_HENV) + SQLFreeHandle(SQL_HANDLE_ENV, henv); + + return 0; +} +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-3-SQLAllocHandle.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-3-SQLAllocHandle.md new file mode 100644 index 0000000000000000000000000000000000000000..e27295b993ee0feaa997bd5fbc2e71084168d4be --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-3-SQLAllocHandle.md @@ -0,0 +1,45 @@ +--- +title: SQLAllocHandle +summary: SQLAllocHandle +author: Guo Huan +date: 2021-05-17 +--- + +# SQLAllocHandle + +## Function + +SQLAllocHandle is used to allocate environment, connection, statement, or descriptor handles. This function replaces the deprecated ODBC 2.x functions SQLAllocEnv, SQLAllocConnect, and SQLAllocStmt. + +## Prototype + +``` +SQLRETURN SQLAllocHandle(SQLSMALLINT HandleType, + SQLHANDLE InputHandle, + SQLHANDLE *OutputHandlePtr); +``` + +## Parameter + +**Table 1** SQLAllocHandle parameters + +| **Keyword** | **Parameter Description** | +| :-------------- | :----------------------------------------------------------- | +| HandleType | Type of handle to be allocated by SQLAllocHandle. The value must be one of the following:
- SQL_HANDLE_ENV (environment handle)
- SQL_HANDLE_DBC (connection handle)
- SQL_HANDLE_STMT (statement handle)
- SQL_HANDLE_DESC (descriptor handle)
The handle application sequence is: **SQL_HANDLE_ENV** > **SQL_HANDLE_DBC** > **SQL_HANDLE_STMT**. The handle applied later depends on the handle applied prior to it. | +| InputHandle | Existing handle to use as a context for the new handle being allocated.
- If **HandleType** is **SQL_HANDLE_ENV**, this parameter is set to **SQL_NULL_HANDLE**.
- If **HandleType** is **SQL_HANDLE_DBC**, this parameter value must be an environment handle.
- If **HandleType** is **SQL_HANDLE_STMT** or **SQL_HANDLE_DESC**, this parameter value must be a connection handle. | +| OutputHandlePtr | **Output parameter**: Pointer to a buffer that stores the returned handle in the newly allocated data structure. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLAllocHandle returns **SQL_ERROR** when it is used to allocate a non-environment handle, it sets **OutputHandlePtr** to **SQL_NULL_HDBC**, **SQL_NULL_HSTMT**, or **SQL_NULL_HDESC**. The application can then call SQLGetDiagRec, with **HandleType** and **Handle** set to the value of **IntputHandle**, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-4-SQLAllocStmt.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-4-SQLAllocStmt.md new file mode 100644 index 0000000000000000000000000000000000000000..f23e49c5df0f0469c76346c4d3db1759c52ac3a9 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-4-SQLAllocStmt.md @@ -0,0 +1,10 @@ +--- +title: SQLAllocStmt +summary: SQLAllocStmt +author: Guo Huan +date: 2021-05-17 +--- + +# SQLAllocStmt + +In ODBC 3.x, SQLAllocStmt was deprecated and replaced by SQLAllocHandle. For details, see [SQLAllocHandle](2-3-SQLAllocHandle.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-5-SQLBindCol.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-5-SQLBindCol.md new file mode 100644 index 0000000000000000000000000000000000000000..bd4dc0cab1815d720641b4c4a21b2bfc335402cb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-5-SQLBindCol.md @@ -0,0 +1,51 @@ +--- +title: SQLBindCol +summary: SQLBindCol +author: Guo Huan +date: 2021-05-17 +--- + +# SQLBindCol + +## Function + +SQLBindCol is used to bind columns in a result set to an application data buffer. + +## Prototype + +``` +SQLRETURN SQLBindCol(SQLHSTMT StatementHandle, + SQLUSMALLINT ColumnNumber, + SQLSMALLINT TargetType, + SQLPOINTER TargetValuePtr, + SQLLEN BufferLength, + SQLLEN *StrLen_or_IndPtr); +``` + +## Parameters + +**Table 1** SQLBindCol parameters + +| **Keyword** | **Parameter Description** | +| :--------------- | :----------------------------------------------------------- | +| StatementHandle | Statement handle. | +| ColumnNumber | Number of the column to be bound. The column number starts with 0 and increases in ascending order. Column 0 is the bookmark column. If no bookmark column is set, column numbers start with 1. | +| TargetType | C data type in the buffer. | +| TargetValuePtr | **Output parameter**: pointer to the buffer bound with the column. The SQLFetch function returns data in the buffer. If **TargetValuePtr** is null, **StrLen_or_IndPtr** is a valid value. | +| BufferLength | Length of the **TargetValuePtr** buffer in bytes. | +| StrLen_or_IndPtr | **Output parameter**: pointer to the length or indicator of the buffer. If **StrLen_or_IndPtr** is null, no length or indicator is used. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLBindCol returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-6-SQLBindParameter.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-6-SQLBindParameter.md new file mode 100644 index 0000000000000000000000000000000000000000..d8b5f924ccc502db6b0da3ce68c1427fa8af2022 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-6-SQLBindParameter.md @@ -0,0 +1,59 @@ +--- +title: SQLBindParameter +summary: SQLBindParameter +author: Guo Huan +date: 2021-05-17 +--- + +# SQLBindParameter + +## Function + +SQLBindParameter is used to bind parameter markers in an SQL statement to a buffer. + +## Prototype + +``` +SQLRETURN SQLBindParameter(SQLHSTMT StatementHandle, + SQLUSMALLINT ParameterNumber, + SQLSMALLINT InputOutputType, + SQLSMALLINT ValuetType, + SQLSMALLINT ParameterType, + SQLULEN ColumnSize, + SQLSMALLINT DecimalDigits, + SQLPOINTER ParameterValuePtr, + SQLLEN BufferLength, + SQLLEN *StrLen_or_IndPtr); +``` + +## Parameters + +**Table 1** SQLBindParameter + +| **Keyword** | **Parameter Description** | +| :---------------- | :----------------------------------------------------------- | +| StatementHandle | Statement handle. | +| ParameterNumber | Parameter marker number, starting with 1 and increasing in ascending order. | +| InputOutputType | Input/output type of the parameter. | +| ValueType | C data type of the parameter. | +| ParameterType | SQL data type of the parameter. | +| ColumnSize | Size of the column or expression of the corresponding parameter marker. | +| DecimalDigits | Decimal digit of the column or the expression of the corresponding parameter marker. | +| ParameterValuePtr | Pointer to the storage parameter buffer. | +| BufferLength | Length of the **ParameterValuePtr** buffer in bytes. | +| StrLen_or_IndPtr | Pointer to the length or indicator of the buffer. If **StrLen_or_IndPtr** is null, no length or indicator is used. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLBindParameter returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-7-SQLColAttribute.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-7-SQLColAttribute.md new file mode 100644 index 0000000000000000000000000000000000000000..fef3d7d5dd20935ae313d516cf81162145778810 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-7-SQLColAttribute.md @@ -0,0 +1,53 @@ +--- +title: SQLColAttribute +summary: SQLColAttribute +author: Guo Huan +date: 2021-05-17 +--- + +# SQLColAttribute + +## Function + +SQLColAttribute is used to return the descriptor information about a column in the result set. + +## Prototype + +``` +SQLRETURN SQLColAttibute(SQLHSTMT StatementHandle, + SQLUSMALLINT ColumnNumber, + SQLUSMALLINT FieldIdentifier, + SQLPOINTER CharacterAtrriburePtr, + SQLSMALLINT BufferLength, + SQLSMALLINT *StringLengthPtr, + SQLLEN *NumericAttributePtr); +``` + +## Parameters + +**Table 1** SQLColAttribute parameters + +| **Keyword** | **Parameter Description** | +| :-------------------- | :----------------------------------------------------------- | +| StatementHandle | Statement handle. | +| ColumnNumber | Column number of the field to be queried, starting with 1 and increasing in ascending order. | +| FieldIdentifier | Field identifier of **ColumnNumber** in IRD. | +| CharacterAttributePtr | **Output parameter**: pointer to the buffer that returns the **FieldIdentifier** value. | +| BufferLength | - **BufferLength** indicates the length of the buffer if **FieldIdentifier** is an ODBC-defined field and **CharacterAttributePtr** points to a string or a binary buffer.
- Ignore this parameter if **FieldIdentifier** is an ODBC-defined field and **CharacterAttributePtr** points to an integer. | +| StringLengthPtr | **Output parameter**: pointer to a buffer in which the total number of valid bytes (for string data) is stored in ***CharacterAttributePtr**. Ignore the value of **BufferLength** if the data is not a string. | +| NumericAttributePtr | **Output parameter**: pointer to an integer buffer in which the value of **FieldIdentifier** in the **ColumnNumber** row of the IRD is returned. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLColAttribute returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_STMT** and **StatementHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-8-SQLConnect.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-8-SQLConnect.md new file mode 100644 index 0000000000000000000000000000000000000000..e61f96adf52a9b18a7efc1f2a158da3839d85aef --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-8-SQLConnect.md @@ -0,0 +1,54 @@ +--- +title: SQLConnect +summary: SQLConnect +author: Guo Huan +date: 2021-05-17 +--- + +# SQLConnect + +## Function + +SQLConnect is used to establish a connection between a driver and a data source. After the connection is established, the connection handle can be used to access all information about the data source, including its application operating status, transaction processing status, and error information. + +## Prototype + +``` +SQLRETURN SQLConnect(SQLHDBC ConnectionHandle, + SQLCHAR *ServerName, + SQLSMALLINT NameLength1, + SQLCHAR *UserName, + SQLSMALLINT NameLength2, + SQLCHAR *Authentication, + SQLSMALLINT NameLength3); +``` + +## Parameter + +**Table 1** SQLConnect parameters + +| **Keyword** | **Parameter Description** | +| :--------------- | :------------------------------------------------ | +| ConnectionHandle | Connection handle, obtained from SQLAllocHandle. | +| ServerName | Name of the data source to connect. | +| NameLength1 | Length of **ServerName**. | +| UserName | Username of the database in the data source. | +| NameLength2 | Length of **UserName**. | +| Authentication | User password of the database in the data source. | +| NameLength3 | Length of **Authentication**. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. +- **SQL_STILL_EXECUTING** indicates that the statement is being executed. + +## Precautions + +If SQLConnect returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_DBC** and **ConnectionHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-9-SQLDisconnect.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-9-SQLDisconnect.md new file mode 100644 index 0000000000000000000000000000000000000000..a4eab78a4b29cdf5e99bc7d99a3a77f0737d08a1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/3-development-based-on-odbc/6-ODBC/2-9-SQLDisconnect.md @@ -0,0 +1,41 @@ +--- +title: SQLDisconnect +summary: SQLDisconnect +author: Guo Huan +date: 2021-05-17 +--- + +# SQLDisconnect + +## Function + +SQLDisconnect is used to close the connection associated with a database connection handle. + +## Prototype + +``` +SQLRETURN SQLDisconnect(SQLHDBC ConnectionHandle); +``` + +## Parameter + +**Table 1** SQLDisconnect parameters + +| **Keyword** | **Parameter Description** | +| :--------------- | :----------------------------------------------- | +| ConnectionHandle | Connection handle, obtained from SQLAllocHandle. | + +## Return Value + +- **SQL_SUCCESS** indicates that the call succeeded. +- **SQL_SUCCESS_WITH_INFO** indicates that some warning information is displayed. +- **SQL_ERROR** indicates major errors, such as memory allocation and connection failures. +- **SQL_INVALID_HANDLE** indicates that invalid handles were called. This value may also be returned by other APIs. + +## Precautions + +If SQLDisconnect returns **SQL_ERROR** or **SQL_SUCCESS_WITH_INFO**, the application can call SQLGetDiagRec, with **HandleType** and **Handle** set to **SQL_HANDLE_DBC** and **ConnectionHandle**, respectively, to obtain the **SQLSTATE** value. The **SQLSTATE** value provides the detailed function calling information. + +## Example + +See [Examples](2-23-Examples.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/1-database-connection-control-functions-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/1-database-connection-control-functions-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..67d1a774f4281eca8e67a29e54ac58145c29c7ac --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/1-database-connection-control-functions-overview.md @@ -0,0 +1,10 @@ +--- +title: Description +summary: Description +author: Guo Huan +date: 2021-05-17 +--- + +# Description + +Database connection control functions control the connections to MogDB servers. An application can connect to multiple servers at a time. For example, a client connects to multiple databases. Each connection is represented by a PGconn object, which is obtained from the function PQconnectdb, PQconnectdbParams, or PQsetdbLogin. Note that these functions will always return a non-null object pointer, unless there is too little memory to allocate the PGconn object. The interface for establishing a connection is stored in the PGconn object. The PQstatus function can be called to check the return value for a successful connection. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/10-PQstatus.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/10-PQstatus.md new file mode 100644 index 0000000000000000000000000000000000000000..0e891c2d7245699af759307990393d75db5acccd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/10-PQstatus.md @@ -0,0 +1,64 @@ +--- +title: PQstatus +summary: PQstatus +author: Guo Huan +date: 2021-05-17 +--- + +# PQstatus + +## Function + +PQstatus is used to return the connection status. + +## Prototype + +``` +ConnStatusType PQstatus(const PGconn *conn); +``` + +## Parameter + +**Table 1** PQ status parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | + +## Return Value + +**ConnStatusType** indicates the connection status. The enumerated values are as follows: + +``` +CONNECTION_STARTED +Waiting for the connection to be established. + +CONNECTION_MADE +Connection succeeded; waiting to send + +CONNECTION_AWAITING_RESPONSE +Waiting for a response from the server. + +CONNECTION_AUTH_OK +Authentication received; waiting for backend startup to complete. + +CONNECTION_SSL_STARTUP +Negotiating SSL encryption. + +CONNECTION_SETENV +Negotiating environment-driven parameter settings. + +CONNECTION_OK +Normal connection. + +CONNECTION_BAD +Failed connection. +``` + +## Precautions + +The connection status can be one of the preceding values. After the asynchronous connection procedure is complete, only two of them, **CONNECTION_OK** and **CONNECTION_BAD**, can return. **CONNECTION_OK** indicates that the connection to the database is normal. **CONNECTION_BAD** indicates that the connection attempt fails. Generally, the **CONNECTION_OK** state remains until PQfinish is called. However, a communication failure may cause the connection status to become to **CONNECTION_BAD** before the connection procedure is complete. In this case, the application can attempt to call PQreset to restore the communication. + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/2-PQconnectdbParams.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/2-PQconnectdbParams.md new file mode 100644 index 0000000000000000000000000000000000000000..644e629bd96b85b7f2d48543ebe6a66032ef5b9b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/2-PQconnectdbParams.md @@ -0,0 +1,42 @@ +--- +title: PQconnectdbParams +summary: PQconnectdbParams +author: Guo Huan +date: 2021-05-17 +--- + +# PQconnectdbParams + +## Function + +PQconnectdbParams is used to establish a new connection with the database server. + +## Prototype + +``` +PGconn *PQconnectdbParams(const char * const *keywords, + const char * const *values, + int expand_dbname); +``` + +## Parameter + +**Table 1** PQconnectdbParams parameters + +| **Keyword** | **Parameter Description** | +| :------------ | :----------------------------------------------------------- | +| keywords | An array of strings, each of which is a keyword. | +| values | Value assigned to each keyword. | +| expand_dbname | When **expand\_dbname** is non-zero, the **dbname** keyword value can be recognized as a connection string. Only **dbname** that first appears is expanded in this way, and any subsequent **dbname** value is treated as a database name. | + +## Return Value + +**PGconn \*** points to the object pointer that contains a connection. The memory is applied for by the function internally. + +## Precautions + +This function establishes a new database connection using the parameters taken from two NULL-terminated arrays. Unlike PQsetdbLogin, the parameter set can be extended without changing the function signature. Therefore, use of this function (or its non-blocking analogs PQconnectStartParams and PQconnectPoll) is preferred for new application programming. + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/3-PQconnectdb.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/3-PQconnectdb.md new file mode 100644 index 0000000000000000000000000000000000000000..ba689d78337e29ca7e502dd48a3104683b84075b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/3-PQconnectdb.md @@ -0,0 +1,39 @@ +--- +title: PQconnectdb +summary: PQconnectdb +author: Guo Huan +date: 2021-05-17 +--- + +# PQconnectdb + +## Function + +PQconnectdb is used to establish a new connection with the database server. + +## Prototype + +``` +PGconn *PQconnectdb(const char *conninfo); +``` + +## Parameter + +**Table 1** PQconnectdb parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conninfo | Connection string. For details about the fields in the string, see Connection Characters. | + +## Return Value + +**PGconn \*** points to the object pointer that contains a connection. The memory is applied for by the function internally. + +## Precautions + +- This function establishes a new database connection using the parameters taken from the string **conninfo**. +- The input parameter can be empty, indicating that all default parameters can be used. It can contain one or more values separated by spaces or contain a URL. + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/4-PQconninfoParse.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/4-PQconninfoParse.md new file mode 100644 index 0000000000000000000000000000000000000000..12ba279198b8437b40f3c448ab83d04285651e0f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/4-PQconninfoParse.md @@ -0,0 +1,31 @@ +--- +title: PQconninfoParse +summary: PQconninfoParse +author: Guo Huan +date: 2021-05-17 +--- + +# PQconninfoParse + +## Function + +PQconninfoParse is used to return parsed connection options based on the connection. + +## Prototype + +``` +PQconninfoOption* PQconninfoParse(const char* conninfo, char** errmsg); +``` + +## Parameters + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conninfo | Passed string. This parameter can be left empty. In this case, the default value is used. It can contain one or more values separated by spaces or contain a URL. | +| errmsg | Error information. | + +## Return Value + +PQconninfoOption pointers diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/5-PQconnectStart.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/5-PQconnectStart.md new file mode 100644 index 0000000000000000000000000000000000000000..d4214eb50a08c4cc8db762d582a2863b870e1063 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/5-PQconnectStart.md @@ -0,0 +1,30 @@ +--- +title: PQconnectStart +summary: PQconnectStart +author: Guo Huan +date: 2021-05-17 +--- + +# PQconnectStart + +## Function + +PQconnectStart is used to establish a non-blocking connection with the database server. + +## Prototype + +``` +PGconn* PQconnectStart(const char* conninfo); +``` + +## Parameters + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conninfo | String of connection information. This parameter can be left empty. In this case, the default value is used. It can contain one or more values separated by spaces or contain a URL. | + +## Return Value + +PGconn pointers diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/6-PQerrorMessage.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/6-PQerrorMessage.md new file mode 100644 index 0000000000000000000000000000000000000000..f729e2b88028d2e51e4bcdb413f050d71402bcb7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/6-PQerrorMessage.md @@ -0,0 +1,34 @@ +--- +title: PQerrorMessage +summary: PQerrorMessage +author: Guo Huan +date: 2021-05-17 +--- + +# PQerrorMessage + +## Function + +PQerrorMessage is used to return error information on a connection. + +## Prototype + +``` +char* PQerrorMessage(const PGconn* conn); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :---------- | :------------------------ | +| conn | Connection handle. | + +## Return Value + +char pointers + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/7-PQsetdbLogin.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/7-PQsetdbLogin.md new file mode 100644 index 0000000000000000000000000000000000000000..2e064f37cafd6dc525212894088d682d11b7fee0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/7-PQsetdbLogin.md @@ -0,0 +1,51 @@ +--- +title: PQsetdbLogin +summary: PQsetdbLogin +author: Guo Huan +date: 2021-05-17 +--- + +# PQsetdbLogin + +## Function + +PQsetdbLogin is used to establish a new connection with the database server. + +## Prototype + +``` +PGconn *PQsetdbLogin(const char *pghost, + const char *pgport, + const char *pgoptions, + const char *pgtty, + const char *dbName, + const char *login, + const char *pwd); +``` + +## Parameter + +**Table 1** PQsetdbLogin parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| pghost | Name of the host to be connected. For details, see the **host** field described in Connection Characters. | +| pgport | Port number of the host server. For details, see the **port** field described in Connection Characters. | +| pgoptions | Command-line options to be sent to the server during running. For details, see the **options** field described in Connection Characters. | +| pgtty | This field can be ignored. (Previously, this field declares the output direction of server logs.) | +| dbName | Name of the database to be connected. For details, see the **dbname** field described in Connection Characters. | +| login | Username for connection. For details, see the **user** field described in Connection Characters. | +| pwd | Password used for authentication during connection. For details, see the **password** field described in Connection Characters. | + +## Return Value + +**PGconn \*** points to the object pointer that contains a connection. The memory is applied for by the function internally. + +## Precautions + +- This function is the predecessor of PQconnectdb with a fixed set of parameters. When an undefined parameter is called, its default value is used. Write NULL or an empty string for any one of the fixed parameters that is to be defaulted. +- If the **dbName** value contains an = sign or a valid prefix in the connection URL, it is taken as a conninfo string and passed to PQconnectdb, and the remaining parameters are consistent with PQconnectdbParams parameters. + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/8-PQfinish.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/8-PQfinish.md new file mode 100644 index 0000000000000000000000000000000000000000..234d94f61b31bcfbc17d7c02dece81b2b24bb769 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/8-PQfinish.md @@ -0,0 +1,34 @@ +--- +title: PQfinish +summary: PQfinish +author: Guo Huan +date: 2021-05-17 +--- + +# PQfinish + +## Function + +PQfinish is used to close the connection to the server and release the memory used by the PGconn object. + +## Prototype + +``` +void PQfinish(PGconn *conn); +``` + +## Parameter + +**Table 1** PQfinish parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | + +## Precautions + +If the server connection attempt fails (as indicated by PQstatus), the application should call PQfinish to release the memory used by the PGconn object. The PGconn pointer must not be used again after PQfinish has been called. + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/9-PQreset.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/9-PQreset.md new file mode 100644 index 0000000000000000000000000000000000000000..5360ef5e6c5ec550bd9f10d9550849634a884776 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/1-database-connection-control-functions/9-PQreset.md @@ -0,0 +1,34 @@ +--- +title: PQreset +summary: PQreset +author: Guo Huan +date: 2021-05-17 +--- + +# PQreset + +## Function + +PQreset is used to reset the communication port to the server. + +## Prototype + +``` +void PQreset(PGconn *conn); +``` + +## Parameter + +**Table 1** PQreset parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | + +## Precautions + +This function will close the connection to the server and attempt to establish a new connection to the same server by using all the parameters previously used. This function is applicable to fault recovery after a connection exception occurs. + +## Example + +For details, see [Example](../../libpq-example.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/1-PQclear.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/1-PQclear.md new file mode 100644 index 0000000000000000000000000000000000000000..d31b1cec3b23dcfba12db140a4c9ed4dbf350aaf --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/1-PQclear.md @@ -0,0 +1,34 @@ +--- +title: PQclear +summary: PQclear +author: Guo Huan +date: 2021-05-17 +--- + +# PQclear + +## Function + +PQclear is used to release the storage associated with PGresult. Any query result should be released by PQclear when it is no longer needed. + +## Prototype + +``` +void PQclear(PGresult *res); +``` + +## Parameters + +**Table 1** PQclear parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :--------------------------------------------- | +| res | Object pointer that contains the query result. | + +## Precautions + +PGresult is not automatically released. That is, it does not disappear when a new query is submitted or even if you close the connection. To delete it, you must call PQclear. Otherwise, memory leakage occurs. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/10-PQntuples.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/10-PQntuples.md new file mode 100644 index 0000000000000000000000000000000000000000..e3100a76e980ca08dd6ad832623893b1e9b0c5a7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/10-PQntuples.md @@ -0,0 +1,34 @@ +--- +title: PQntuples +summary: PQntuples +author: Guo Huan +date: 2021-05-17 +--- + +# PQntuples + +## Function + +PQntuples is used to return the number of rows (tuples) in the query result. An overflow may occur if the return value is out of the value range allowed in a 32-bit OS. + +## Prototype + +``` +int PQntuples(const PGresult *res); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :---------- | :------------------------ | +| res | Operation result handle. | + +## Return Value + +Value of the int type + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/11-PQprepare.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/11-PQprepare.md new file mode 100644 index 0000000000000000000000000000000000000000..27ec13ad93ccd18e00ae688c3147de9300532f2e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/11-PQprepare.md @@ -0,0 +1,50 @@ +--- +title: PQprepare +summary: PQprepare +author: Guo Huan +date: 2021-05-17 +--- + +# PQprepare + +## Function + +PQprepare is used to submit a request to create a prepared statement with given parameters and wait for completion. + +## Prototype + +``` +PGresult *PQprepare(PGconn *conn, + const char *stmtName, + const char *query, + int nParams, + const Oid *paramTypes); +``` + +## Parameters + +**Table 1** PQprepare parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | +| stmtName | Name of **stmt** to be executed. | +| query | Query string to be executed. | +| nParams | Parameter quantity. | +| paramTypes | Array of the parameter type. | + +## Return Value + +**PGresult** indicates the object pointer that contains the query result. + +## Precautions + +- PQprepare creates a prepared statement for later execution with PQexecPrepared. This function allows commands to be repeatedly executed, without being parsed and planned each time they are executed. PQprepare is supported only in protocol 3.0 or later. It will fail when protocol 2.0 is used. +- This function creates a prepared statement named **stmtName** from the query string, which must contain an SQL command. **stmtName** can be **""** to create an unnamed statement. In this case, any pre-existing unnamed statement will be automatically replaced. Otherwise, this is an error if the statement name has been defined in the current session. If any parameters are used, they are referred to in the query as $1, $2, and so on. **nParams** is the number of parameters for which types are pre-specified in the array paramTypes[]. (The array pointer can be **NULL** when **nParams** is **0**.) paramTypes[] specifies the data types to be assigned to the parameter symbols by OID. If **paramTypes** is **NULL**, or any element in the array is **0**, the server assigns a data type to the parameter symbol in the same way as it does for an untyped literal string. In addition, the query can use parameter symbols whose numbers are greater than **nParams**. Data types of these symbols will also be inferred. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> You can also execute the **SQLPREPARE** statement to create a prepared statement that is used with PQexecPrepared. Although there is no libpq function of deleting a prepared statement, the **SQL DEALLOCATE** statement can be used for this purpose. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/12-PQresultStatus.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/12-PQresultStatus.md new file mode 100644 index 0000000000000000000000000000000000000000..4a2958410cda31749cd23f455a0b3522066a2f20 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/12-PQresultStatus.md @@ -0,0 +1,72 @@ +--- +title: PQresultStatus +summary: PQresultStatus +author: Guo Huan +date: 2021-05-17 +--- + +# PQresultStatus + +## Function + +PQresultStatus is used to return the result status of a command. + +## Prototype + +``` +ExecStatusType PQresultStatus(const PGresult *res); +``` + +## Parameter + +**Table 1** PQresultStatus parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :--------------------------------------------- | +| res | Object pointer that contains the query result. | + +## Return Value + +**PQresultStatus** indicates the command execution status. The enumerated values are as follows: + +``` +PQresultStatus can return one of the following values: +PGRES_EMPTY_QUERY +The string sent to the server was empty. + +PGRES_COMMAND_OK +A command that does not return data was successfully executed. + +PGRES_TUPLES_OK +A query (such as SELECT or SHOW) that returns data was successfully executed. + +PGRES_COPY_OUT +Copy Out (from the server) data transfer started. + +PGRES_COPY_IN +Copy In (to the server) data transfer started. + +PGRES_BAD_RESPONSE +The response from the server cannot be understood. + +PGRES_NONFATAL_ERROR +A non-fatal error (notification or warning) occurred. + +PGRES_FATAL_ERROR +A fatal error occurred. + +PGRES_COPY_BOTH +Copy In/Out (to and from the server) data transfer started. This state occurs only in streaming replication. + +PGRES_SINGLE_TUPLE +PGresult contains a result tuple from the current command. This state occurs in a single-row query. +``` + +## Precautions + +- Note that the SELECT command that happens to retrieve zero rows still returns **PGRES_TUPLES_OK**. **PGRES_COMMAND_OK** is used for commands that can never return rows (such as INSERT or UPDATE, without return clauses). The result status **PGRES_EMPTY_QUERY** might indicate a bug in the client software. +- The result status **PGRES_NONFATAL_ERROR** will never be returned directly by PQexec or other query execution functions. Instead, such results will be passed to the notice processor. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/2-PQexec.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/2-PQexec.md new file mode 100644 index 0000000000000000000000000000000000000000..c752c487dfdf9a72784249550ad4ac912b0262fa --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/2-PQexec.md @@ -0,0 +1,42 @@ +--- +title: PQexec +summary: PQexec +author: Guo Huan +date: 2021-05-17 +--- + +# PQexec + +## Function + +PQexec is used to commit a command to the server and wait for the result. + +## Prototype + +``` +PGresult *PQexec(PGconn *conn, const char *command); +``` + +## Parameter + +**Table 1** PQexec parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | +| command | Query string to be executed. | + +## Return Value + +**PGresult** indicates the object pointer that contains the query result. + +## Precautions + +The PQresultStatus function should be called to check the return value for any errors (including the value of a null pointer, in which **PGRES_FATAL_ERROR** will be returned). The PQerrorMessage function can be called to obtain more information about such errors. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> The command string can contain multiple SQL commands separated by semicolons (;). Multiple queries sent in a PQexec call are processed in one transaction, unless there are specific BEGIN/COMMIT commands in the query string to divide the string into multiple transactions. Note that the returned PGresult structure describes only the result of the last command executed from the string. If a command fails, the string processing stops and the returned PGresult describes the error condition. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/3-PQexecParams.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/3-PQexecParams.md new file mode 100644 index 0000000000000000000000000000000000000000..f8c2120cfb02b653d1cae499a403cb59f366cee0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/3-PQexecParams.md @@ -0,0 +1,44 @@ +--- +title: PQexecParams +summary: PQexecParams +author: Guo Huan +date: 2021-05-17 +--- + +# PQexecParams + +## Function + +PQexecParams is used to run a command to bind one or more parameters. + +## Prototype + +``` +PGresult* PQexecParams(PGconn* conn, + const char* command, + int nParams, + const Oid* paramTypes, + const char* const* paramValues, + const int* paramLengths, + const int* paramFormats, + int resultFormat); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :----------- | :---------------------------------- | +| conn | Connection handle. | +| command | SQL text string. | +| nParams | Number of parameters to be bound. | +| paramTypes | Types of parameters to be bound. | +| paramValues | Values of parameters to be bound. | +| paramLengths | Parameter lengths. | +| paramFormats | Parameter formats (text or binary). | +| resultFormat | Result format (text or binary). | + +## Return Value + +PGresult pointers diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/4-PQexecParamsBatch.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/4-PQexecParamsBatch.md new file mode 100644 index 0000000000000000000000000000000000000000..f7e4b3c0daab95406454d0c8ebe6a710b86b4813 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/4-PQexecParamsBatch.md @@ -0,0 +1,46 @@ +--- +title: PQexecParamsBatch +summary: PQexecParamsBatch +author: Guo Huan +date: 2021-05-17 +--- + +# PQexecParamsBatch + +## Function + +PQexecParamsBatch is used to run a command to bind batches of parameters. + +## Prototype + +``` +PGresult* PQexecParamsBatch(PGconn* conn, + const char* command, + int nParams, + int nBatch, + const Oid* paramTypes, + const char* const* paramValues, + const int* paramLengths, + const int* paramFormats, + int resultFormat); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :----------- | :---------------------------------- | +| conn | Connection handle. | +| command | SQL text string. | +| nParams | Number of parameters to be bound. | +| nBatch | Number of batch operations. | +| paramTypes | Types of parameters to be bound. | +| paramValues | Values of parameters to be bound. | +| paramLengths | Parameter lengths. | +| paramFormats | Parameter formats (text or binary). | +| resultFormat | Result format (text or binary). | + +## Return Value + +PGresult pointers diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/5-PQexecPrepared.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/5-PQexecPrepared.md new file mode 100644 index 0000000000000000000000000000000000000000..b2eab81bdce3baedbb35350692afc3f52e353e77 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/5-PQexecPrepared.md @@ -0,0 +1,42 @@ +--- +title: PQexecPrepared +summary: PQexecPrepared +author: Guo Huan +date: 2021-05-17 +--- + +# PQexecPrepared + +## Function + +PQexecPrepared is used to send a request to execute a prepared statement with given parameters and wait for the result. + +## Prototype + +``` +PGresult* PQexecPrepared(PGconn* conn, + const char* stmtName, + int nParams, + const char* const* paramValues, + const int* paramLengths, + const int* paramFormats, + int resultFormat); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :----------- | :----------------------------------------------------------- | +| conn | Connection handle. | +| stmtName | **stmt** name, which can be set to "" or NULL to reference an unnamed statement. Otherwise, it must be the name of an existing prepared statement. | +| nParams | Parameter quantity. | +| paramValues | Actual values of parameters. | +| paramLengths | Actual data lengths of parameters. | +| paramFormats | Parameter formats (text or binary). | +| resultFormat | Return result format (text or binary). | + +## Return Value + +PGresult pointers diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/6-PQexecPreparedBatch.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/6-PQexecPreparedBatch.md new file mode 100644 index 0000000000000000000000000000000000000000..23ed2c96854988ad26cf1519fdcc2f7e77519d51 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/6-PQexecPreparedBatch.md @@ -0,0 +1,44 @@ +--- +title: PQexecPreparedBatch +summary: PQexecPreparedBatch +author: Guo Huan +date: 2021-05-17 +--- + +# PQexecPreparedBatch + +## Function + +PQexecPreparedBatch is used to send a request to execute a prepared statement with batches of given parameters and wait for the result. + +## Prototype + +``` +PGresult* PQexecPreparedBatch(PGconn* conn, + const char* stmtName, + int nParams, + int nBatchCount, + const char* const* paramValues, + const int* paramLengths, + const int* paramFormats, + int resultFormat); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :----------- | :----------------------------------------------------------- | +| conn | Connection handle. | +| stmtName | **stmt** name, which can be set to "" or NULL to reference an unnamed statement. Otherwise, it must be the name of an existing prepared statement. | +| nParams | Parameter quantity. | +| nBatchCount | Number of batches. | +| paramValues | Actual values of parameters. | +| paramLengths | Actual data lengths of parameters. | +| paramFormats | Parameter formats (text or binary). | +| resultFormat | Return result format (text or binary). | + +## Return Value + +PGresult pointers diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/7-PQfname.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/7-PQfname.md new file mode 100644 index 0000000000000000000000000000000000000000..f62035545dacd5991ec91d386f339a0ad52acbc8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/7-PQfname.md @@ -0,0 +1,36 @@ +--- +title: PQfname +summary: PQfname +author: Guo Huan +date: 2021-05-17 +--- + +# PQfname + +## Function + +PQfname is used to return the column name associated with the given column number. Column numbers start from 0. The caller should not release the result directly. The result will be released when the associated PGresult handle is passed to PQclear. + +## Prototype + +``` +char *PQfname(const PGresult *res, + int column_number); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :------------ | :------------------------ | +| res | Operation result handle. | +| column_number | Number of columns. | + +## Return Value + +char pointers + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/8-PQgetvalue.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/8-PQgetvalue.md new file mode 100644 index 0000000000000000000000000000000000000000..86a7fa972a42441e7a1e0d7847afaf391e21c14d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/8-PQgetvalue.md @@ -0,0 +1,42 @@ +--- +title: PQgetvalue +summary: PQgetvalue +author: Guo Huan +date: 2021-05-17 +--- + +# PQgetvalue + +## Function + +PQgetvalue is used to return a single field value of one row of a PGresult. Row and column numbers start from 0. The caller should not release the result directly. The result will be released when the associated PGresult handle is passed to PQclear. + +## Prototype + +``` +char *PQgetvalue(const PGresult *res, + int row_number, + int column_number); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :------------ | :------------------------ | +| res | Operation result handle. | +| row_number | Number of rows. | +| column_number | Number of columns. | + +## Return Value + +For data in text format, the value returned by PQgetvalue is a null-terminated string representation of the field value. + +For binary data, the value is a binary representation determined by the typsend and typreceive functions of the data type. + +If this field is left blank, an empty string is returned. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/9-PQnfields.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/9-PQnfields.md new file mode 100644 index 0000000000000000000000000000000000000000..459ecaa8aa32218edcb98e0c5e3dbe3b495fd18d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/2-database-statement-execution-functions/9-PQnfields.md @@ -0,0 +1,34 @@ +--- +title: PQnfields +summary: PQnfields +author: Guo Huan +date: 2021-05-17 +--- + +# PQnfields + +## Function + +PQnfields is used to return the number of columns (fields) in each row of the query result. + +## Prototype + +``` +int PQnfields(const PGresult *res); +``` + +## Parameter + +**Table 1** + +| **Keyword** | **Parameter Description** | +| :---------- | :------------------------ | +| res | Operation result handle. | + +## Return Value + +Value of the int type + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/1-functions-for-asynchronous-command-processing-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/1-functions-for-asynchronous-command-processing-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..df0fbcb1114ea986a47ee14e8da2aa4194ab269f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/1-functions-for-asynchronous-command-processing-overview.md @@ -0,0 +1,17 @@ +--- +title: Description +summary: Description +author: Guo Huan +date: 2021-05-17 +--- + +# Description + +The PQexec function is adequate for committing commands in common, synchronous applications. However, it has several defects, which may be important to some users: + +- PQexec waits for the end of the command, but the application may have other work to do (for example, maintaining a user interface). In this case, PQexec would not want to be blocked to wait for the response. +- As the client application is suspended while waiting for the result, it is difficult for the application to determine whether to cancel the ongoing command. +- PQexec can return only one PGresult structure. If the committed command string contains multiple SQL commands, all the PGresult structures except the last PGresult are discarded by PQexec. +- PQexec always collects the entire result of the command and caches it in a PGresult. Although this mode simplifies the error handling logic for applications, it is impractical for results that contain multiple rows. + +Applications that do not want to be restricted by these limitations can use the following functions that PQexec is built from: PQsendQuery and PQgetResult. The functions PQsendQueryParams, PQsendPrepare, and PQsendQueryPrepared can also be used with PQgetResult. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/2-PQsendQuery.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/2-PQsendQuery.md new file mode 100644 index 0000000000000000000000000000000000000000..c8237c713138fe440129269872011057a531536a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/2-PQsendQuery.md @@ -0,0 +1,39 @@ +--- +title: PQsendQuery +summary: PQsendQuery +author: Guo Huan +date: 2021-05-17 +--- + +# PQsendQuery + +## Function + +PQsendQuery is used to commit a command to the server without waiting for the result. If the query is successful, **1** is returned. Otherwise, **0** is returned. + +## Prototype + +```c +int PQsendQuery(PGconn *conn, const char *command); +``` + +## Parameter + +**Table 1** PQsendQuery parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | +| command | Query string to be executed. | + +## Return Value + +**int** indicates the execution result. **1** indicates successful execution and **0** indicates an execution failure. The failure cause is stored in **conn->errorMessage**. + +## Precautions + +After PQsendQuery is successfully called, call PQgetResult one or more times to obtain the results. PQsendQuery cannot be called again (on the same connection) until PQgetResult returns a null pointer, indicating that the command execution is complete. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/3-PQsendQueryParams.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/3-PQsendQueryParams.md new file mode 100644 index 0000000000000000000000000000000000000000..d8e5c99f9f3437c389dce7cade907d7ee417f12e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/3-PQsendQueryParams.md @@ -0,0 +1,52 @@ +--- +title: PQsendQueryParams +summary: PQsendQueryParams +author: Guo Huan +date: 2021-05-17 +--- + +# PQsendQueryParams + +## Function + +PQsendQueryParams is used to commit a command and separate parameters to the server without waiting for the result. + +## Prototype + +```c +int PQsendQueryParams(PGconn *conn, + const char *command, + int nParams, + const Oid *paramTypes, + const char * const *paramValues, + const int *paramLengths, + const int *paramFormats, + int resultFormat); +``` + +## Parameter + +**Table 1** PQsendQueryParams parameters + +| **Keyword** | **Parameter Description** | +| :----------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | +| command | Query string to be executed. | +| nParams | Parameter quantity. | +| paramTypes | Parameter type. | +| paramValues | Parameter value. | +| paramLengths | Parameter length. | +| paramFormats | Parameter format. | +| resultFormat | Result format. | + +## Return Value + +**int** indicates the execution result. **1** indicates successful execution and **0** indicates an execution failure. The failure cause is stored in **conn->errorMessage**. + +## Precautions + +PQsendQueryParams is equivalent to PQsendQuery. The only difference is that query parameters can be specified separately from the query string. PQsendQueryParams parameters are handled in the same way as PQexecParams parameters. Like PQexecParams, PQsendQueryParams cannot work on connections using protocol 2.0 and it allows only one command in the query string. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/4-PQsendPrepare.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/4-PQsendPrepare.md new file mode 100644 index 0000000000000000000000000000000000000000..550c4f1ea98aa02fd94db396a62bc9272aef3aa8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/4-PQsendPrepare.md @@ -0,0 +1,46 @@ +--- +title: PQsendPrepare +summary: PQsendPrepare +author: Guo Huan +date: 2021-05-17 +--- + +# PQsendPrepare + +## Function + +PQsendPrepare is used to send a request to create a prepared statement with given parameters, without waiting for completion. + +## Prototype + +```c +int PQsendPrepare(PGconn *conn, + const char *stmtName, + const char *query, + int nParams, + const Oid *paramTypes); +``` + +## Parameters + +**Table 1** PQsendPrepare parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | +| stmtName | Name of **stmt** to be executed. | +| query | Query string to be executed. | +| nParams | Parameter quantity. | +| paramTypes | Array of the parameter type. | + +## Return Value + +**int** indicates the execution result. **1** indicates successful execution and **0** indicates an execution failure. The failure cause is stored in **conn->errorMessage**. + +## Precautions + +PQsendPrepare is an asynchronous version of PQprepare. If it can dispatch a request, **1** is returned. Otherwise, **0** is returned. After a successful calling of PQsendPrepare, call PQgetResult to check whether the server successfully created the prepared statement. PQsendPrepare parameters are handled in the same way as PQprepare parameters. Like PQprepare, PQsendPrepare cannot work on connections using protocol 2.0. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/5-PQsendQueryPrepared.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/5-PQsendQueryPrepared.md new file mode 100644 index 0000000000000000000000000000000000000000..4ce131cc997b91c09897c106ec204977bf8fbac5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/5-PQsendQueryPrepared.md @@ -0,0 +1,50 @@ +--- +title: PQsendQueryPrepared +summary: PQsendQueryPrepared +author: Guo Huan +date: 2021-05-17 +--- + +# PQsendQueryPrepared + +## Function + +PQsendQueryPrepared is used to send a request to execute a prepared statement with given parameters, without waiting for the result. + +## Prototype + +```c +int PQsendQueryPrepared(PGconn *conn, + const char *stmtName, + int nParams, + const char * const *paramValues, + const int *paramLengths, + const int *paramFormats, + int resultFormat); +``` + +## Parameters + +**Table 1** PQsendQueryPrepared parameters + +| **Keyword** | **Parameter Description** | +| :----------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | +| stmtName | Name of **stmt** to be executed. | +| nParams | Parameter quantity. | +| paramValues | Parameter value. | +| paramLengths | Parameter length. | +| paramFormats | Parameter format. | +| resultFormat | Result format. | + +## Return Value + +**int** indicates the execution result. **1** indicates successful execution and **0** indicates an execution failure. The failure cause is stored in **conn->errorMessage**. + +## Precautions + +PQsendQueryPrepared is similar to PQsendQueryParams, but the command to be executed is specified by naming a previously-prepared statement, instead of providing a query string. PQsendQueryPrepared parameters are handled in the same way as PQexecPrepared parameters. Like PQexecPrepared, PQsendQueryPrepared cannot work on connections using protocol 2.0. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/6-PQflush.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/6-PQflush.md new file mode 100644 index 0000000000000000000000000000000000000000..485875afc9bba8f3479e9a60ebc3eb8c0eb0ed2b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/3-functions-for-asynchronous-command-processing/6-PQflush.md @@ -0,0 +1,38 @@ +--- +title: PQflush +summary: PQflush +author: Guo Huan +date: 2021-05-17 +--- + +# PQflush + +## Function + +PQflush is used to try to flush any queued output data to the server. + +## Prototype + +```c +Cint PQflush(PGconn *conn); +``` + +## Parameter + +**Table 1** PQflush parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | + +## Return Value + +**int** indicates the execution result. If the operation is successful (or the send queue is empty), **0** is returned. If the operation fails, **-1** is returned. If all data in the send queue fails to be sent, **1** is returned. (This case occurs only when the connection is non-blocking.) The failure cause is stored in **conn->error_message**. + +## Precautions + +Call PQflush after sending any command or data over a non-blocking connection. If **1** is returned, wait for the socket to become read- or write-ready. If the socket becomes write-ready, call PQflush again. If the socket becomes read-ready, call PQconsumeInput and then call PQflush again. Repeat the operation until the value **0** is returned for PQflush. (It is necessary to check for read-ready and drain the input using PQconsumeInput. This is because the server can block trying to send us data, for example, notification messages, and will not read our data until we read it.) Once PQflush returns **0**, wait for the socket to be read-ready and then read the response as described above. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/1-PQgetCancel.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/1-PQgetCancel.md new file mode 100644 index 0000000000000000000000000000000000000000..234b1051b81532f0a6a84885251b9599579d3346 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/1-PQgetCancel.md @@ -0,0 +1,38 @@ +--- +title: PQgetCancel +summary: PQgetCancel +author: Guo Huan +date: 2021-05-17 +--- + +# PQgetCancel + +## Function + +PQgetCancel is used to create a data structure that contains the information required to cancel a command issued through a specific database connection. + +## Prototype + +```c +PGcancel *PQgetCancel(PGconn *conn); +``` + +## Parameter + +**Table 1** PQgetCancel parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| conn | Points to the object pointer that contains the connection information. | + +## Return Value + +**PGcancel** points to the object pointer that contains the cancel information. + +## Precautions + +PQgetCancel creates a PGcancel object for a given PGconn connection object. If the given connection object (**conn**) is NULL or an invalid connection, PQgetCancel will return NULL. The PGcancel object is an opaque structure that cannot be directly accessed by applications. It can be transferred only to PQcancel or PQfreeCancel. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/2-PQfreeCancel.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/2-PQfreeCancel.md new file mode 100644 index 0000000000000000000000000000000000000000..3df29bc62ea1c9870179d967dcb595f3bc19a543 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/2-PQfreeCancel.md @@ -0,0 +1,34 @@ +--- +title: PQfreeCancel +summary: PQfreeCancel +author: Guo Huan +date: 2021-05-17 +--- + +# PQfreeCancel + +## Function + +PQfreeCancel is used to release the data structure created by PQgetCancel. + +## Prototype + +```c +void PQfreeCancel(PGcancel *cancel); +``` + +## Parameter + +**Table 1** PQfreeCancel parameter + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| cancel | Points to the object pointer that contains the cancel information. | + +## Precautions + +PQfreeCancel releases a data object previously created by PQgetCancel. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/3-PQcancel.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/3-PQcancel.md new file mode 100644 index 0000000000000000000000000000000000000000..1a71f6b718054f756db1770cfb69807ecd846436 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/2-libpq/4-functions-for-canceling-queries-in-progress/3-PQcancel.md @@ -0,0 +1,41 @@ +--- +title: PQcancel +summary: PQcancel +author: Guo Huan +date: 2021-05-17 +--- + +# PQcancel + +## Function + +PQcancel is used to request the server to abandon processing of the current command. + +## Prototype + +```c +int PQcancel(PGcancel *cancel, char *errbuf, int errbufsize); +``` + +## Parameter + +**Table 1** PQcancel parameters + +| **Keyword** | **Parameter Description** | +| :---------- | :----------------------------------------------------------- | +| cancel | Points to the object pointer that contains the cancel information. | +| errbuf | Buffer for storing error information. | +| errbufsize | Size of the buffer for storing error information. | + +## Return Value + +**int** indicates the execution result. **1** indicates successful execution and **0** indicates an execution failure. The failure cause is stored in **errbuf**. + +## Precautions + +- Successful sending does not guarantee that the request will have any effect. If the cancellation is valid, the current command is terminated early and an error is returned. If the cancellation fails (for example, because the server has processed the command), no result is returned. +- If **errbuf** is a local variable in a signal handler, you can safely call PQcancel from the signal handler. For PQcancel, the PGcancel object is read-only, so it can also be called from a thread that is separate from the thread that is operating the PGconn object. + +## Example + +For details, see Example. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/dependent-header-files-of-libpq.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/dependent-header-files-of-libpq.md new file mode 100644 index 0000000000000000000000000000000000000000..d417a52670361b8ac3b504616d0b552e3911f907 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/dependent-header-files-of-libpq.md @@ -0,0 +1,10 @@ +--- +title: Dependent Header Files of libpq +summary: Dependent Header Files of libpq +author: Guo Huan +date: 2022-04-26 +--- + +# Dependent Header Files of libpq + +Client programs that use **libpq** must include the header file **libpq-fe.h** and must link with the libpq library. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/development-process.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/development-process.md new file mode 100644 index 0000000000000000000000000000000000000000..6e1a4a5c0219de822d114d8ef4997ccf6c52fcd0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/development-process.md @@ -0,0 +1,34 @@ +--- +title: Development Process +summary: Development Process +author: Guo Huan +date: 2022-04-26 +--- + +# Development Process + +To compile and connect to a libpq source program, perform the following operations: + +1. Decompress the release package (for example, `MogDB-3.0.1-CentOS-64bit-Libpq.tar.gz`). The required header file is stored in the **include** folder, and the **lib** folder contains the required libpq library file. + + > **NOTE:** In addition to **libpq-fe.h**, the **include** folder contains the header files **postgres_ext.h**, **gs_thread.h**, and **gs_threadlocal.h** by default. These three header files are the dependency files of **libpq-fe.h**. + +2. Include the **libpq-fe.h** header file. + + ``` + #include + ``` + +3. Provide the **-I** *directory* option to provide the installation location of the header files. (Sometimes the compiler looks for the default directory, so this option can be ignored.) Example: + + ``` + gcc -I (Directory where the header files are located) -L (Directory where the libpq library is located) testprog.c -lpq + ``` + +4. If the makefile is used, add the following options to variables *CPPFLAGS*, *LDFLAGS*, and *LIBS*: + + ``` + CPPFLAGS += -I (Directory where the header files are located) + LDFLAGS += -L (Directory where the libpq library is located) + LIBS += -lpq + ``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/libpq-example.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/libpq-example.md new file mode 100644 index 0000000000000000000000000000000000000000..e6954672945991d6b8cfdcb166dd490f64f6c28d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/libpq-example.md @@ -0,0 +1,282 @@ +--- +title: Example +summary: Example +author: Guo Huan +date: 2022-04-26 +--- + +# Example + +## Code for Common Functions + +Example 1: + +```c + +/* + * testlibpq.c + */ +#include +#include +#include + +static void +exit_nicely(PGconn *conn) +{ + PQfinish(conn); + exit(1); +} + +int +main(int argc, char **argv) +{ + const char *conninfo; + PGconn *conn; + PGresult *res; + int nFields; + int i,j; + + /* + * This value is used when the user provides the value of the conninfo character string in the command line. + * Otherwise, the environment variables or the default values + * are used for all other connection parameters. + */ + if (argc > 1) + conninfo = argv[1]; + else + conninfo = "dbname=postgres port=42121 host='10.44.133.171' application_name=test connect_timeout=5 sslmode=allow user='test' password='test_1234'"; + + /* Connect to the database. */ + conn = PQconnectdb(conninfo); + + /* Check whether the backend connection has been successfully established. */ + if (PQstatus(conn) != CONNECTION_OK) + { + fprintf(stderr, "Connection to database failed: %s", + PQerrorMessage(conn)); + exit_nicely(conn); + } + + /* + * Since a cursor is used in the test case, a transaction block is required. + * Put all data in one "select * from pg_database" + * PQexec() is too simple and is not recommended. + */ + + /* Start a transaction block. */ + res = PQexec(conn, "BEGIN"); + if (PQresultStatus(res) != PGRES_COMMAND_OK) + { + fprintf(stderr, "BEGIN command failed: %s", PQerrorMessage(conn)); + PQclear(res); + exit_nicely(conn); + } + + /* + * PQclear PGresult should be executed when it is no longer needed, to avoid memory leakage. + */ + PQclear(res); + + /* + * Fetch data from the pg_database system catalog. + */ + res = PQexec(conn, "DECLARE myportal CURSOR FOR select * from pg_database"); + if (PQresultStatus(res) != PGRES_COMMAND_OK) + { + fprintf(stderr, "DECLARE CURSOR failed: %s", PQerrorMessage(conn)); + PQclear(res); + exit_nicely(conn); + } + PQclear(res); + + res = PQexec(conn, "FETCH ALL in myportal"); + if (PQresultStatus(res) != PGRES_TUPLES_OK) + { + fprintf(stderr, "FETCH ALL failed: %s", PQerrorMessage(conn)); + PQclear(res); + exit_nicely(conn); + } + + /* First, print out the attribute name. */ + nFields = PQnfields(res); + for (i = 0; i < nFields; i++) + printf("%-15s", PQfname(res, i)); + printf("\n\n"); + + /* Print lines. */ + for (i = 0; i < PQntuples(res); i++) + { + for (j = 0; j < nFields; j++) + printf("%-15s", PQgetvalue(res, i, j)); + printf("\n"); + } + + PQclear(res); + + /* Close the portal. We do not need to check for errors. */ + res = PQexec(conn, "CLOSE myportal"); + PQclear(res); + + /* End the transaction. */ + res = PQexec(conn, "END"); + PQclear(res); + + /* Close the database connection and clean up the database. */ + PQfinish(conn); + + return 0; +} +``` + +Example 2: + +```c + +/* + * testlibpq2.c + * Test out-of-line parameters and binary I/Os. + * + * Before running this example, run the following command to populate a database: + * + * + * CREATE TABLE test1 (i int4, t text); + * + * INSERT INTO test1 values (2, 'ho there'); + * + * The expected output is as follows: + * + * + * tuple 0: got + * i = (4 bytes) 2 + * t = (8 bytes) 'ho there' + * + */ +#include +#include +#include +#include +#include + +/* for ntohl/htonl */ +#include +#include + +static void +exit_nicely(PGconn *conn) +{ + PQfinish(conn); + exit(1); +} + +/* + * This function is used to print out the query results. The results are in binary format +* and fetched from the table created in the comment above. + */ +static void +show_binary_results(PGresult *res) +{ + int i; + int i_fnum, + t_fnum; + + /* Use PQfnumber to avoid assumptions about field order in the result. */ + i_fnum = PQfnumber(res, "i"); + t_fnum = PQfnumber(res, "t"); + + for (i = 0; i < PQntuples(res); i++) + { + char *iptr; + char *tptr; + int ival; + + /* Obtain the field value. (Ignore the possibility that they may be null). */ + iptr = PQgetvalue(res, i, i_fnum); + tptr = PQgetvalue(res, i, t_fnum); + + /* + * The binary representation of INT4 is the network byte order, + * which is better to be replaced with the local byte order. + */ + ival = ntohl(*((uint32_t *) iptr)); + + /* + * The binary representation of TEXT is text. Since libpq can append a zero byte to it, + * and think of it as a C string. + * + */ + + printf("tuple %d: got\n", i); + printf(" i = (%d bytes) %d\n", + PQgetlength(res, i, i_fnum), ival); + printf(" t = (%d bytes) '%s'\n", + PQgetlength(res, i, t_fnum), tptr); + printf("\n\n"); + } +} + +int +main(int argc, char **argv) +{ + const char *conninfo; + PGconn *conn; + PGresult *res; + const char *paramValues[1]; + int paramLengths[1]; + int paramFormats[1]; + uint32_t binaryIntVal; + + /* + * If the user provides a parameter on the command line, + * The value of this parameter is a conninfo character string. Otherwise, + * Use environment variables or default values. + */ + if (argc > 1) + conninfo = argv[1]; + else + conninfo = "dbname=postgres port=42121 host='10.44.133.171' application_name=test connect_timeout=5 sslmode=allow user='test' password='test_1234'"; + + /* Connect to the database. */ + conn = PQconnectdb(conninfo); + + /* Check whether the connection to the server was successfully established. */ + if (PQstatus(conn) != CONNECTION_OK) + { + fprintf(stderr, "Connection to database failed: %s", + PQerrorMessage(conn)); + exit_nicely(conn); + } + + /* Convert the integer value "2" to the network byte order. */ + binaryIntVal = htonl((uint32_t) 2); + + /* Set the parameter array for PQexecParams. */ + paramValues[0] = (char *) &binaryIntVal; + paramLengths[0] = sizeof(binaryIntVal); + paramFormats[0] = 1; /* Binary */ + + res = PQexecParams(conn, + "SELECT * FROM test1 WHERE i = $1::int4", + 1, /* One parameter */ + NULL, /* Enable the backend to deduce the parameter type. */ + paramValues, + paramLengths, + paramFormats, + 1); /* require binary results. */ + + if (PQresultStatus(res) != PGRES_TUPLES_OK) + { + fprintf(stderr, "SELECT failed: %s", PQerrorMessage(conn)); + PQclear(res); + exit_nicely(conn); + } + + show_binary_results(res); + + PQclear(res); + + /* Close the database connection and clean up the database. */ + PQfinish(conn); + + return 0; +} +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/link-parameters.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/link-parameters.md new file mode 100644 index 0000000000000000000000000000000000000000..1f84feecce89f933e34f98f1fe9b1d3dd37439bd --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4-development-based-on-libpq/link-parameters.md @@ -0,0 +1,52 @@ +--- +title: Link Parameters +summary: Link Parameters +author: Guo Huan +date: 2022-04-26 +--- + +# Link Parameters + +**Table 1** Link parameters + +| Parameter | Description | +| :------------------------ | :----------------------------------------------------------- | +| host | Name of the host to connect to. If the host name starts with a slash (/), Unix-domain socket communications instead of TCP/IP communications are used. The value is the directory where the socket file is stored. If **host** is not specified, the default behavior is to connect to the Unix-domain socket in the **/tmp** directory (or the socket directory specified during database installation). On a machine without a Unix-domain socket, the default behavior is to connect to **localhost**.
You can specify multiple host names by using a character string separated by commas (,). Multiple host names can be specified. | +| hostaddr | IP address of the host to connect to. The value is in standard IPv4 address format, for example, 172.28.40.9. If the machine supports IPv6, IPv6 address can also be used. If a non-null string is specified, TCP/IP communications are used.
You can specify multiple IP addresses by using a character string separated by commas (,). Multiple IP addresses can be specified.
Replacing **host** with **hostaddr** can prevent applications from querying host names, which may be important for applications with time constraints. However, a host name is required for GSSAPI or SSPI authentication methods. Therefore, the following rules are used:
1. If **host** is specified but **hostaddr** is not, a query for the host name will be executed.
2. If **hostaddr** is specified but **host** is not, the value of **hostaddr** is the server network address. If the host name is required by authentication, the connection attempt fails.
3. If both **host** and **hostaddr** are specified, the value of **hostaddr** is the server network address. The value of **host** is ignored unless it is required by authentication, in which case it is used as the host name.
NOTICE:
- If **host** is not the server name in the network address specified by **hostaddr**, the authentication may fail.
- If neither **host** nor **hostaddr** is specified, libpq will use a local Unix-domain socket for connection. If the machine does not have a Unix-domain socket, it will attempt to connect to **localhost**. | +| port | Port number of the host server, or the socket file name extension for Unix-domain connections.
You can specify multiple port numbers by using a character string separated by commas (,). Multiple port numbers can be specified. | +| user | Name of the user to connect as. By default, the username is the same as the operating system name of the user running the application. | +| dbname | Database name. The default value is the same as the username. | +| password | Password to be used if the server requires password authentication. | +| connect_timeout | Maximum timeout period of the connection, in seconds (in decimal integer string). The value **0** or null indicates infinity. You are not advised to set the connection timeout period to a value less than 2 seconds. | +| client_encoding | Client encoding for the connection. In addition to the values accepted by the corresponding server options, you can use **auto** to determine the correct encoding from the current environment in the client (the *LC_CTYPE* environment variable in the Unix system). | +| tty | This parameter can be ignored. (This parameter was used to specify the location to which the debugging output of the server was sent). | +| options | Adds command-line options to send to the server at runtime. | +| application_name | Current user identity. | +| fallback_application_name | Specifies a backup value for the **application_name** parameter. This value is used if no value is set for **application_name** through a connection parameter or the *PGAPPNAME* environment variable. It is useful to specify a backup value in a common tool program that wants to set a default application name but does not want it to be overwritten by the user. | +| keepalives | Whether TCP keepalive is enabled on the client side. The default value is **1**, indicating that the function is enabled. The value **0** indicates that the function is disabled. Ignore this parameter for Unix-domain connections. | +| keepalives_idle | The number of seconds of inactivity after which TCP should send a keepalive message to the server. The value **0** indicates that the default value is used. Ignore this parameter for Unix-domain connections or if keep-alive is disabled. | +| keepalives_interval | The number of seconds after which a TCP keepalive message that is not acknowledged by the server should be retransmitted. The value **0** indicates that the default value is used. Ignore this parameter for Unix-domain connections or if keep-alive is disabled. | +| keepalives_count | Adds command-line options to send to the server at runtime. For example, adding **-c comm_debug_mode=off** to set the value of the GUC parameter **comm_debug_mode** to **off**. | +| rw_timeout | Sets the read and write timeout interval of the client connection. | +| sslmode | Specifies whether to enable SSL encryption.
- **disable**: SSL connection is disabled.
- **allow**: If the database server requires SSL connection, SSL connection can be enabled. However, authenticity of the database server will not be verified.
- **prefer**: If the database supports SSL connection, SSL connection is preferred. However, authenticity of the database server will not be verified.
- **require**: SSL connection is required and data is encrypted. However, authenticity of the database server will not be verified.
- **verify-ca**: SSL connection is required. Currently, Windows ODBC does not support cert authentication.
- **verify-full**: SSL connection is required. Currently, Windows ODBC does not support cert authentication. | +| sslcompression | If this parameter is set to **1** (default value), the data transmitted over the SSL connection is compressed (this requires that the OpenSSL version be 0.9.8 or later). If this parameter is set to **0**, compression will be disabled (this requires OpenSSL 1.0.0 or later). If a connection without SSL is established, this parameter is ignored. If the OpenSSL version in use does not support this parameter, it will also be ignored. Compression takes up CPU time, but it increases throughput when the bottleneck is the network. If CPU performance is a limiting factor, disabling compression can improve response time and throughput. | +| sslcert | This parameter specifies the file name of the client SSL certificate. It replaces the default **~/.postgresql/postgresql.crt**. If no SSL connection is established, this parameter is ignored. | +| sslkey | This parameter specifies the location of the key used for the client certificate. It can specify the name of a file used to replace the default **~/.postgresql/postgresql.key**, or specify a key obtained from an external “engine” that is a loadable module of OpenSSL. The description of an external engine should consist of a colon-separated engine name and an engine-related key identifier. If no SSL connection is established, this parameter is ignored. | +| sslrootcert | This parameter specifies the name of a file that contains the SSL Certificate Authority (CA) certificate. If the file exists, the system authenticates the server certificate issued by one of these authorities. The default value is **~/.postgresql/root.crt**. | +| sslcrl | This parameter specifies the file name of the SSL Certificate Revocation List (CRL). If a certificate listed in this file exists, the server certificate authentication will be rejected. The default value is **~/.postgresql/root.crl**. | +| requirepeer | This parameter specifies the OS user of the server, for example, **requirepeer=postgres**. When a Unix-domain socket connection is established, if this parameter is set, the client checks whether the server process is running under the specified username at the beginning of the connection. If not, the connection will be interrupted by an error. This parameter can be used to provide server authentication similar to that of the SSL certificate on TCP/IP connections. (Note that if the Unix domain socket is in **/tmp** or another public writable location, any user can start a server for listening to the location. Use this parameter to ensure that your device is connected to a server that is run by a trusted user.) This option is supported only on platforms that implement the peer authentication method. | +| krbsrvname | This parameter specifies the Kerberos service name used for GSSAPI authentication. For successful Kerberos authentication, this value must match the service name specified in the server configuration. | +| gsslib | This parameter specifies the GSS library used for GSSAPI authentication. It is used only in the Windows OS. If this parameter is set to **gssapi**, **libpq** is forced to use the GSSAPI library to replace the default SSPI for authentication. | +| service | This parameter specifies the name of the service for which the additional parameter is used. It specifies a service name in **pg_service.conf** that holds the additional connection parameters. This allows the application to specify only one service name so that the connection parameters can be centrally maintained. | +| authtype | **authtype** is no longer used, so it is marked as a parameter not to be displayed. It is retained in an array so as not to reject the **conninfo** string from old applications that might still try to set it. | +| remote_nodename | Specifies the name of the remote node connected to the local node. | +| localhost | Specifies the local host in a connection channel. | +| localport | Specifies the local port in a connection channel. | +| fencedUdfRPCMode | Specifies whether the fenced udf RPC protocol uses UNIX domain sockets or special socket file names. The default value is **0**, indicating that the UNIX domain socket mode is used and the file type is .s.PGSQL.%d. To use the fenced UDF mode, set this parameter to **1**. In this case, the file type is .s.fencedMaster_unixdomain. | +| replication | Specifies whether the connection should use replication protocols instead of common protocols. Protocols with this parameter configured are internal protocols used for PostgreSQL replication connections and tools such as **pg_basebackup**, while they can also be used by third-party applications. The following values, which are case-insensitive, are supported:
- **true**, **on**, **yes**, and **1**: Specify that the physical replication mode is connected.
- **database**: Specifies that the logical replication mode and the database specified by **dbname** are connected.
- **false**, **off**, **no**, and **0**: Specify that the connection is a regular connection, which is the default behavior.
In physical or logical replication mode, only simple query protocols can be used. | +| backend_version | Specifies the backend version to be passed to the remote end. | +| prototype | Sets the current protocol level. The default value is **PROTO_TCP**. | +| enable_ce | Specifies whether a client is allowed to connect to a fully encrypted database. The default value is **0**. To enable this function, change the value to **1**. | +| connection_info | The value of `connection_info` is a JSON character string consisting of `driver_name`, `driver_version`, `driver_path`, and `os_user`.
If the value is not null, use `connection_info` and ignore `connectionExtraInf`.
If the value is null, a connection information string related to `libpq` is generated. When `connectionExtraInf` is set to `false`, the value of `connection_info` consists of only `driver_name` and `driver_version`. | +| connectionExtraInf | Specifies whether the value of **connection_info** contains extension information. The default value is **0**. If the value contains other information, set this parameter to **1**. | +| target_session_attrs | Specifies the type of the host to be connected. The connection is successful only when the host type is the same as the configured value. The rules for setting **target_session_attrs** are as follows:
- **any** (default value): All types of hosts can be connected.
- **read-write**: The connection is set up only when the connected host is readable and writable.
- **read-only**: Only readable hosts can be connected.
- **primary**: Only the primary server in the primary/standby systems can be connected.
- **standby**: Only the standby server in the primary/standby systems can be connected.
- **prefer-standby**: The system first attempts to find a standby node for connection. If all hosts in the **hosts** list fail to be connected, try the **any** mode. | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/1-psycopg-based-development.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/1-psycopg-based-development.md new file mode 100644 index 0000000000000000000000000000000000000000..c07cea4588bd9f3b2d607276334f91717e225c46 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/1-psycopg-based-development.md @@ -0,0 +1,19 @@ +--- +title: Psycopg-Based Development +summary: Psycopg-Based Development +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Psycopg-Based Development + +Psycopg is a Python API used to execute SQL statements and provides a unified access API for PostgreSQL and MogDB. Applications can perform data operations based on psycopg. Psycopg2 is an encapsulation of libpq and is implemented using the C language, which is efficient and secure. It provides cursors on both clients and servers, asynchronous communication and notification, and the COPY TO and COPY FROM functions. Psycopg2 supports multiple types of Python out-of-the-box and adapts to PostgreSQL data types. Through the flexible object adaptation system, you can extend and customize the adaptation. Psycopg2 is compatible with Unicode and Python 3. + +MogDB supports the psycopg2 feature and allows psycopg2 to be connected in SSL mode. + +**Table 1** Platforms supported by Psycopg + +| OS | Platform | +| :---------- | :------- | +| EulerOS 2.5 | x86_64 | +| EulerOS 2.8 | ARM64 | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md new file mode 100644 index 0000000000000000000000000000000000000000..8a8e166444a133d5de682a126890b1ee435983ec --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md @@ -0,0 +1,243 @@ +--- +title: Common Operations +summary: Common Operations +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Examples: Common Operations + +## Example 1 + +The following illustrates how to develop applications based on MogDB JDBC interfaces. + +``` +//DBtest.java +// This example illustrates the main processes of JDBC-based development, covering database connection creation, table creation, and data insertion. + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.SQLException; +import java.sql.Statement; +import java.sql.CallableStatement; + +public class DBTest { + + // Create a database connection. + public static Connection GetConnection(String username, String passwd) { + String driver = "org.opengauss.Driver"; + String sourceURL = "jdbc:opengauss://localhost:8000/postgres"; + Connection conn = null; + try { + // Load the database driver. + Class.forName(driver).newInstance(); + } catch (Exception e) { + e.printStackTrace(); + return null; + } + + try { + // Create a database connection. + conn = DriverManager.getConnection(sourceURL, username, passwd); + System.out.println("Connection succeed!"); + } catch (Exception e) { + e.printStackTrace(); + return null; + } + + return conn; + }; + + // Run a common SQL statement to create table customer_t1. + public static void CreateTable(Connection conn) { + Statement stmt = null; + try { + stmt = conn.createStatement(); + + // Run a common SQL statement. + int rc = stmt + .executeUpdate("CREATE TABLE customer_t1(c_customer_sk INTEGER, c_customer_name VARCHAR(32));"); + + stmt.close(); + } catch (SQLException e) { + if (stmt != null) { + try { + stmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + // Run a prepared statement to insert data in batches. + public static void BatchInsertData(Connection conn) { + PreparedStatement pst = null; + + try { + // Generate a prepared statement. + pst = conn.prepareStatement("INSERT INTO customer_t1 VALUES (?,?)"); + for (int i = 0; i < 3; i++) { + // Add parameters. + pst.setInt(1, i); + pst.setString(2, "data " + i); + pst.addBatch(); + } + // Perform batch processing. + pst.executeBatch(); + pst.close(); + } catch (SQLException e) { + if (pst != null) { + try { + pst.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + // Run a prepared statement to update data. + public static void ExecPreparedSQL(Connection conn) { + PreparedStatement pstmt = null; + try { + pstmt = conn + .prepareStatement("UPDATE customer_t1 SET c_customer_name = ? WHERE c_customer_sk = 1"); + pstmt.setString(1, "new Data"); + int rowcount = pstmt.executeUpdate(); + pstmt.close(); + } catch (SQLException e) { + if (pstmt != null) { + try { + pstmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + +// Run a stored procedure. + public static void ExecCallableSQL(Connection conn) { + CallableStatement cstmt = null; + try { + + cstmt=conn.prepareCall("{? = CALL TESTPROC(?,?,?)}"); + cstmt.setInt(2, 50); + cstmt.setInt(1, 20); + cstmt.setInt(3, 90); + cstmt.registerOutParameter(4, Types.INTEGER); // Register an OUT parameter of the integer type. + cstmt.execute(); + int out = cstmt.getInt(4); // Obtain the OUT parameter. + System.out.println("The CallableStatment TESTPROC returns:"+out); + cstmt.close(); + } catch (SQLException e) { + if (cstmt != null) { + try { + cstmt.close(); + } catch (SQLException e1) { + e1.printStackTrace(); + } + } + e.printStackTrace(); + } + } + + + /** + * Main process. Call static methods one by one. + * @param args + */ + public static void main(String[] args) { + // Create a database connection. + Connection conn = GetConnection("tester", "Password1234"); + + // Create a table. + CreateTable(conn); + + // Insert data in batches. + BatchInsertData(conn); + + // Run a prepared statement to update data. + ExecPreparedSQL(conn); + + // Run a stored procedure. + ExecCallableSQL(conn); + + // Close the connection to the database. + try { + conn.close(); + } catch (SQLException e) { + e.printStackTrace(); + } + + } + +} +``` + +## Example 2: High Client Memory Usage + +In this example, **setFetchSize** adjusts the memory usage of the client by using the database cursor to obtain server data in batches. It may increase network interaction and damage some performance. + +The cursor is valid within a transaction. Therefore, disable automatic commit and then manually commit the code. + +``` +// Disable automatic commit. +conn.setAutoCommit(false); +Statement st = conn.createStatement(); + +// Open the cursor and obtain 50 lines of data each time. +st.setFetchSize(50); +ResultSet rs = st.executeQuery("SELECT * FROM mytable"); +conn.commit(); +while (rs.next()) +{ + System.out.print("a row was returned."); +} +rs.close(); + +// Disable the server cursor. +st.setFetchSize(0); +rs = st.executeQuery("SELECT * FROM mytable"); +conn.commit(); +while (rs.next()) +{ + System.out.print("many rows were returned."); +} +rs.close(); + +// Close the statement. +st.close(); +conn.close(); +``` + +Run the following command to enable automatic commit: + +``` +conn.setAutoCommit(true); +``` + +## Example 3: Example of Common Data Types + +``` +//Example of the bit type. Note that the value range of the bit type is [0,1]. +Statement st = conn.createStatement(); +String sqlstr = "create or replace function fun_1()\n" + + "returns bit AS $$\n" + + "select col_bit from t_bit limit 1;\n" + + "$$\n" + + "LANGUAGE SQL;"; +st.execute(sqlstr); +CallableStatement c = conn.prepareCall("{ ? = call fun_1() }"); +//Register the output type, which is a bit string. +c.registerOutParameter(1, Types.BIT); +c.execute(); +//Use the Boolean type to obtain the result. +System.out.println(c.getBoolean(1)); +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/1-psycopg2-connect.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/1-psycopg2-connect.md new file mode 100644 index 0000000000000000000000000000000000000000..c251e496ed095da0a75e8ae7f043bfbdd6604df0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/1-psycopg2-connect.md @@ -0,0 +1,42 @@ +--- +title: psycopg2.connect() +summary: psycopg2.connect() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# psycopg2.connect() + +## Function + +This method creates a database session and returns a new connection object. + +## Prototype + +``` +conn=psycopg2.connect(dbname="test",user="postgres",password="secret",host="127.0.0.1",port="5432") +``` + +## Parameter + +**Table 1** psycopg2.connect parameters + +| **Keyword** | **Description** | +| :---------- | :----------------------------------------------------------- | +| dbname | Database name. | +| user | Username. | +| password | Password. | +| host | Database IP address. The default type is UNIX socket. | +| port | Connection port number. The default value is **5432**. | +| sslmode | SSL mode, which is used for SSL connection. | +| sslcert | Path of the client certificate, which is used for SSL connection. | +| sslkey | Path of the client key, which is used for SSL connection. | +| sslrootcert | Path of the root certificate, which is used for SSL connection. | + +## Return Value + +Connection object (for connecting to the openGauss DB instance) + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/10-connection-close.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/10-connection-close.md new file mode 100644 index 0000000000000000000000000000000000000000..579401dee9535e7c9cef6e484ace585a77c845b2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/10-connection-close.md @@ -0,0 +1,32 @@ +--- +title: connection.close() +summary: connection.close() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# connection.close() + +## Function + +This method closes the database connection. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** This method closes the database connection and does not automatically call **commit()**. If you just close the database connection without calling **commit()** first, changes will be lost. + +## Prototype + +``` +connection.close() +``` + +## Parameter + +None + +## Return Value + +None + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/2-connection-cursor.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/2-connection-cursor.md new file mode 100644 index 0000000000000000000000000000000000000000..df03d7acf54a19a21164e4d082e0b25a87a63f3f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/2-connection-cursor.md @@ -0,0 +1,37 @@ +--- +title: connection.cursor() +summary: connection.cursor() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# connection.cursor() + +## Function + +This method returns a new cursor object. + +## Prototype + +``` +cursor(name=None, cursor_factory=None, scrollable=None, withhold=False) +``` + +## Parameter + +**Table 1** connection.cursor parameters + +| **Keyword** | **Description** | +| :------------- | :----------------------------------------------------------- | +| name | Cursor name. The default value is **None**. | +| cursor_factory | Creates a non-standard cursor. The default value is **None**. | +| scrollable | Sets the SCROLL option. The default value is **None**. | +| withhold | Sets the HOLD option. The default value is **False**. | + +## Return Value + +Cursor object (used for cusors that are programmed using Python in the entire database) + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/3-cursor-execute-query-vars-list.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/3-cursor-execute-query-vars-list.md new file mode 100644 index 0000000000000000000000000000000000000000..f2bae52d63e263591b3d4795bfb3ef5140d75153 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/3-cursor-execute-query-vars-list.md @@ -0,0 +1,35 @@ +--- +title: cursor.execute(query,vars_list) +summary: cursor.execute(query,vars_list) +author: Zhang Cuiping +date: 2021-10-11 +--- + +# cursor.execute(query,vars_list) + +## Function + +This method executes the parameterized SQL statements (that is, placeholders instead of SQL literals). The psycopg2 module supports placeholders marked with **%s**. + +## Prototype + +``` +curosr.execute(query,vars_list) +``` + +## Parameters + +**Table 1** curosr.execute parameters + +| **Keyword** | **Description** | +| :---------- | :----------------------------------------------------------- | +| query | SQL statement to be executed. | +| vars_list | Variable list, which matches the **%s** placeholder in the query. | + +## Return Value + +None + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/4-curosr-executemany-query-vars-list.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/4-curosr-executemany-query-vars-list.md new file mode 100644 index 0000000000000000000000000000000000000000..39b665204d13066b04920f8ef79cf4325de0a22f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/4-curosr-executemany-query-vars-list.md @@ -0,0 +1,35 @@ +--- +title: curosr.executemany(query,vars_list) +summary: curosr.executemany(query,vars_list) +author: Zhang Cuiping +date: 2021-10-11 +--- + +# curosr.executemany(query,vars_list) + +## Function + +This method executes an SQL command against all parameter sequences or mappings found in the sequence SQL. + +## Prototype + +``` +curosr.executemany(query,vars_list) +``` + +## Parameter + +**Table 1** curosr.executemany parameters + +| **Keyword** | **Description** | +| :---------- | :----------------------------------------------------------- | +| query | SQL statement that you want to execute. | +| vars_list | Variable list, which matches the **%s** placeholder in the query. | + +## Return Value + +None + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/5-connection-commit.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/5-connection-commit.md new file mode 100644 index 0000000000000000000000000000000000000000..6d058394e4d0b637532fe1d3dbcb3ad48e881b9f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/5-connection-commit.md @@ -0,0 +1,32 @@ +--- +title: connection.commit() +summary: connection.commit() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# connection.commit() + +## Function + +This method commits the currently pending transaction to the database. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** By default, Psycopg opens a transaction before executing the first command. If **commit()** is not called, the effect of any data operation will be lost. + +## Prototype + +``` +connection.commit() +``` + +## Parameter + +None + +## Return Value + +None + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/6-connection-rollback.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/6-connection-rollback.md new file mode 100644 index 0000000000000000000000000000000000000000..764c9869c799e804735f3dd9be1ccffd55a0389c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/6-connection-rollback.md @@ -0,0 +1,32 @@ +--- +title: connection.rollback() +summary: connection.rollback() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# connection.rollback() + +## Function + +This method rolls back the current pending transaction. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** If you close the connection using **close()** but do not commit the change using **commit()**, an implicit rollback will be performed. + +## Prototype + +``` +connection.rollback() +``` + +## Parameter + +None + +## Return Value + +None + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/7-cursor-fetchone.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/7-cursor-fetchone.md new file mode 100644 index 0000000000000000000000000000000000000000..acb0411f8bdd2a315ec5f6115418b4a7f3cfe3ef --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/7-cursor-fetchone.md @@ -0,0 +1,30 @@ +--- +title: cursor.fetchone() +summary: cursor.fetchone() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# cursor.fetchone() + +## Function + +This method extracts the next row of the query result set and returns a tuple. + +## Prototype + +``` +cursor.fetchone() +``` + +## Parameter + +None + +## Return Value + +A single tuple is the first result in the result set. If no more data is available, **None** is returned. + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/8-cursor-fetchall.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/8-cursor-fetchall.md new file mode 100644 index 0000000000000000000000000000000000000000..9933d6904bb9c6be80b8bcc790776e372007aa61 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/8-cursor-fetchall.md @@ -0,0 +1,30 @@ +--- +title: cursor.fetchall() +summary: cursor.fetchall() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# cursor.fetchall() + +## Function + +This method obtains all the (remaining) rows of the query result and returns them as a list of tuples. + +## Prototype + +``` +cursor.fetchall() +``` + +## Parameter + +None + +## Return Value + +Tuple list, which contains all results of the result set. An empty list is returned when no rows are available. + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/9-cursor-close.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/9-cursor-close.md new file mode 100644 index 0000000000000000000000000000000000000000..8c55d5f2722381e665280f10f6ff8a62a623614d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/11-psycopg-api-reference/9-cursor-close.md @@ -0,0 +1,30 @@ +--- +title: cursor.close() +summary: cursor.close() +author: Zhang Cuiping +date: 2021-10-11 +--- + +# cursor.close() + +## Function + +This method closes the cursor of the current connection. + +## Prototype + +``` +cursor.close() +``` + +## Parameter + +None + +## Return Value + +None + +## Examples + +For details, see [Example: Common Operations](../../../../developer-guide/dev/4.1-psycopg-based-development/10.1-example-common-operations.md). \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/2-psycopg-package.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/2-psycopg-package.md new file mode 100644 index 0000000000000000000000000000000000000000..dfddfb1edec68411c7055312a7d1fe81bbc2a212 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/2-psycopg-package.md @@ -0,0 +1,17 @@ +--- +title: Psycopg Package +summary: Psycopg Package +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Psycopg Package + +The psycopg package is obtained from the [openGauss Download](https://opengauss.org/zh/download.html) page. + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/psycopg-package-en.png) + +After the decompression, the following folders are generated: + +- **psycopg2**: **psycopg2** library file +- **lib**: **lib** library file \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/3.1-development-process.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/3.1-development-process.md new file mode 100644 index 0000000000000000000000000000000000000000..9e2f3d5b39a81a023d1db4536d204d6ce9a0a8b3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/3.1-development-process.md @@ -0,0 +1,12 @@ +--- +title: Development Process +summary: Development Process +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Development Process + +**Figure 1** Application development process based on psycopg2 + +![application-development-process-based-on-psycopg2](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/development-process-2.png) diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/4-loading-a-driver.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/4-loading-a-driver.md new file mode 100644 index 0000000000000000000000000000000000000000..952882e00dc0e038b500f40914acd2e53a1cb72a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/4-loading-a-driver.md @@ -0,0 +1,21 @@ +--- +title: Loading a Driver +summary: Loading a Driver +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Loading a Driver + +- Before using the driver, perform the following operations: + + 1. Decompress the driver package of the corresponding version and copy psycopg2 to the **site-packages** folder in the Python installation directory as the **root** user. + 2. Change the **psycopg2** directory permission to **755**. + 3. Add the **psycopg2** directory to the environment variable *$PYTHONPATH* and validate it. + 4. For non-database users, configure the **lib** directory in *LD_LIBRARY_PATH* after decompression. + +- Load a database driver before creating a database connection: + + ```bash + import psycopg2 + ``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/5.1-connecting-to-a-database.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/5.1-connecting-to-a-database.md new file mode 100644 index 0000000000000000000000000000000000000000..556bf4391713b2f183702b5fe62c4dbd64e5252f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/5.1-connecting-to-a-database.md @@ -0,0 +1,12 @@ +--- +title: Connecting to a Database +summary: Connecting to a Database +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Connecting to a Database + +1. Use the .ini file (the **configparser** package of Python can parse this type of configuration file) to save the configuration information about the database connection. +2. Use the **psycopg2.connect** function to obtain the connection object. +3. Use the connection object to create a cursor object. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/6-executing-sql-statements.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/6-executing-sql-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..78313f70fa3ec3772f63aeeb1417c2436a4a268e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/6-executing-sql-statements.md @@ -0,0 +1,11 @@ +--- +title: Executing SQL Statements +summary: Executing SQL Statements +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Executing SQL Statements + +1. Construct an operation statement and use **%s** as a placeholder. During execution, psycopg2 will replace the placeholder with the parameter value. You can add the RETURNING clause to obtain the automatically generated column values. +2. The **cursor.execute** method is used to perform operations on one row, and the **cursor.executemany** method is used to perform operations on multiple rows. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/7-processing-the-result-set.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/7-processing-the-result-set.md new file mode 100644 index 0000000000000000000000000000000000000000..c24096ab1b5dd0318a1bbd8ce2984dc552945fe4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/7-processing-the-result-set.md @@ -0,0 +1,11 @@ +--- +title: Processing the Result Set +summary: Processing the Result Set +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Processing the Result Set + +1. **cursor.fetchone()**: Fetches the next row in a query result set and returns a sequence. If no data is available, null is returned. +2. **cursor.fetchall()**: Fetches all remaining rows in a query result and returns a list. An empty list is returned when no rows are available. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/8-closing-the-connection.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/8-closing-the-connection.md new file mode 100644 index 0000000000000000000000000000000000000000..6e0d0c06b4b563815b4ede3fc4691063ac85666d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/8-closing-the-connection.md @@ -0,0 +1,12 @@ +--- +title: Closing the Connection +summary: Closing the Connection +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Closing the Connection + +After you complete required data operations in a database, close the database connection. Call the close method such as **connection.close()** to close the connection. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-caution.gif) **CAUTION:** This method closes the database connection and does not automatically call **commit()**. If you just close the database connection without calling **commit()** first, changes will be lost. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/9-connecting-to-the-database-using-ssl.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/9-connecting-to-the-database-using-ssl.md new file mode 100644 index 0000000000000000000000000000000000000000..17dffd022da56403d52b2402997109d7afdc900f --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/4.1-psycopg-based-development/9-connecting-to-the-database-using-ssl.md @@ -0,0 +1,30 @@ +--- +title: Connecting to the Database +summary: Connecting to the Database +author: Zhang Cuiping +date: 2021-10-11 +--- + +# Connecting to the Database (Using SSL) + +When you use psycopy2 to connect to the MogDB server, you can enable SSL to encrypt the communication between the client and server. To enable SSL, you must have the server certificate, client certificate, and private key files. For details on how to obtain these files, see related documents and commands of OpenSSL. + +1. Use the .ini file (the **configparser** package of Python can parse this type of configuration file) to save the configuration information about the database connection. +2. Add SSL connection parameters **sslmode**, **sslcert**, **sslkey**, and **sslrootcert** to the connection options. + 1. **sslmode**: [Table 1](#table1.1) + 2. **sslcert**: client certificate path + 3. **sslkey**: client key path + 4. **sslrootcert**: root certificate path +3. Use the **psycopg2.connect** function to obtain the connection object. +4. Use the connection object to create a cursor object. + +**Table 1** sslmode options + +| sslmode | Whether SSL Encryption Is Enabled | Description | +| :---------- | :-------------------------------- | :----------------------------------------------------------- | +| disable | No | SSL connection is not enabled. | +| allow | Possible | If the database server requires SSL connection, SSL connection can be enabled. However, authenticity of the database server will not be verified. | +| prefer | Possible | If the database supports SSL connection, SSL connection is preferred. However, authenticity of the database server will not be verified. | +| require | Yes | SSL connection is required and data is encrypted. However, authenticity of the database server will not be verified. | +| verify-ca | Yes | The SSL connection must be enabled. | +| verify-full | Yes | The SSL connection must be enabled, which is not supported by MogDB currently. | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/5-commissioning.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/5-commissioning.md new file mode 100644 index 0000000000000000000000000000000000000000..3fc96e9b6ca3b7f65f22c4c9db22fdd15ca13c58 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/5-commissioning.md @@ -0,0 +1,40 @@ +--- +title: Commissioning +summary: Commissioning +author: Guo Huan +date: 2021-04-27 +--- + +# Commissioning + +To control the output of log files and better understand the operating status of the database, modify specific configuration parameters in the **postgresql.conf** file in the instance data directory. + +[Table 1](#Configuration parameters) describes the adjustable configuration parameters. + +**Table 1** Configuration parameters + +| Parameter | Description | Value Range | Remarks | +| ------------------------------------------------------------ | ------------------------------------- | ------------------------------------- | ------------------------------------- | +| client_min_messages | Level of messages to be sent to clients. | - DEBUG5
- DEBUG4
- DEBUG3
- DEBUG2
- DEBUG1
- LOG
- NOTICE
WARNING
- ERROR
- FATAL
- PANIC
Default value: NOTICE | Messages of the set level or lower will be sent to clients. The lower the level is, the fewer the messages will be sent. | +| log_min_messages | Level of messages to be recorded in server logs. | - DEBUG5
- DEBUG4
- DEBUG3
- DEBUG2
- DEBUG1
- INFO
- NOTICE
- WARNING
- ERROR
- LOG
- FATAL
- PANIC
Default value: WARNING | Messages higher than the set level will be recorded in logs. The higher the level is, the fewer the server logs will be recorded. | +| log_min_error_statement | Level of SQL error statements to be recorded in server logs. | - DEBUG5
- DEBUG4
- DEBUG3
- DEBUG2
- DEBUG1
- INFO
- NOTICE
- WARNING
- ERROR
- FATAL
- PANIC
Default value: ERROR。 | SQL error statements of the set level or higher will be recorded in server logs.Only a system administrator is allowed to modify this parameter. | +| log_min_duration_statement | Minimum execution duration of a statement. If the execution duration of a statement is equal to or longer than the set milliseconds, the statement and its duration will be recorded in logs. Enabling this function can help you track the query attempts to be optimized. | INT type
Default value: 30min
Unit: millisecond | The default value (-1) indicates that the function is disabled.Only a system administrator is allowed to modify this parameter. | +| log_connections/log_disconnections | Whether to record a server log message when each session is connected or disconnected. | - **on**: The system records a log server when each session is connected or disconnected.
- **off**: The system does not record a log server when each session is connected or disconnected.
Default value: off | - | +| log_duration | Whether to record the duration of each executed statement. | - **on**: The system records the duration of each executed statement.
- **off**: The system does not record the duration of each executed statement.
Default value: on | Only a system administrator is allowed to modify this parameter. | +| log_statement | SQL statements to be recorded in logs. | - **none**: The system does not record any SQL statements.
- **ddl**: The system records data definition statements.
- **mod**: The system records data definition statements and data operation statements.
- **all**: The system records all statements.
Default value: none | Only a system administrator is allowed to modify this parameter. | +| log_hostname | Whether to record host names. | - **on**: The system records host names.
- **off**: The system does not record host names.
Default value: off | By default, connection logs only record the IP addresses of connected hosts. With this function, the host names will also be recorded.This parameter affects parameters in **Querying Audit Results**, GS_WLM_SESSION_HISTORY, PG_STAT_ACTIVITY, and **log_line_prefix**. | + +[Table 2](#description) describes the preceding parameter levels. + +**Table 2** Description of log level parameters + +| Level | Description | +| ---------- | ------------------------------------------------------------ | +| DEBUG[1-5] | Provides information that can be used by developers. Level 1 is the lowest level whereas level 5 is the highest level. | +| INFO | Provides information about users' hidden requests, for example, information about the VACUUM VERBOSE process. | +| NOTICE | Provides information that may be important to users, for example, truncations of long identifiers or indexes created as a part of a primary key. | +| WARNING | Provides warning information for users, for example, COMMIT out of transaction blocks. | +| ERROR | Reports an error that causes a command to terminate. | +| LOG | Reports information that administrators may be interested in, for example, the activity levels of check points. | +| FATAL | Reports the reason that causes a session to terminate. | +| PANIC | Reports the reason that causes all sessions to terminate. | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/dev/6-appendix.md b/product/en/docs-mogdb/v3.1/developer-guide/dev/6-appendix.md new file mode 100644 index 0000000000000000000000000000000000000000..72f0ec2618d90ff64ef5685fe94e4376aa871684 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/dev/6-appendix.md @@ -0,0 +1,107 @@ +--- +title: Appendices +summary: Appendices +author: Guo Huan +date: 2021-04-27 +--- + +# Appendices + +## **Table 1** PG_STAT_ACTIVITY Columns + +| Name | Type | Description | +| ---------------- | ------------------------ | ------------------------------------------------------------ | +| datid | oid | OID of the database that the user session connects to in the backend | +| datname | name | Name of the database that the user session connects to in the backend | +| pid | bigint | Thread ID of the backend | +| sessionid | bigint | Session ID | +| usesysid | oid | OID of the user logged in to the backend | +| usename | name | Name of the user logged in to the backend | +| application_name | text | Name of the application connected to the backend | +| client_addr | inet | IP address of the client connected to the backend. If this column is null, it indicates either the client is connected via a Unix socket on the server or this is an internal process, such as autovacuum. | +| client_hostname | text | Host name of the connected client, as reported by a reverse DNS lookup of `client_addr`. This column will be non-null only for IP connections and only when **log_hostname** is enabled. | +| client_port | integer | TCP port number that the client uses for communication with this backend (**-1** if a Unix socket is used) | +| backend_start | timestamp with time zone | Time when this process was started, that is, when the client connected to the server | +| xact_start | timestamp with time zone | Time when current transaction was started (null if no transaction is active) If the current query is the first of its transaction, the value of this column is the same as that of the **query_start** column. | +| query_start | timestamp with time zone | Time when the currently active query was started, or if **state** is not **active**, when the last query was started | +| state_change | timestamp with time zone | Time when the **state** was last changed | +| waiting | Boolean | Whether the backend is currently waiting on a lock. If yes, the value is **true**. | +| enqueue | text | Unsupported currently | +| state | text | Overall status of this backend. The value must be one of the following:
- **active**: The backend is executing a query.
- **idle**: The backend is waiting for a new client command.
- **idle in transaction**: The backend is in a transaction, but there is no statement being executed in the transaction.
- **idle in transaction** (aborted): The backend is in a transaction, but there are statements failed in the transaction.
- **fastpath function call**: The backend is executing a fast-path function.
- **disabled**: This state is reported if **track_activities** is disabled in this backend.
NOTE:
Common users can view their own session status only. The state information of other accounts is empty. For example, after user **judy** is connected to the database, the state information of user **joe** and the initial user omm in **pg_stat_activity** is empty.
`SELECT datname, usename, usesysid, state,pid FROM pg_stat_activity;`
`datname | usename | usesysid | state | pid ----+---+----+---+------ postgres | omm | 10 | | 139968752121616 postgres | omm | 10 | | 139968903116560 db_tpcc | judy | 16398 | active | 139968391403280 postgres | omm | 10 | | 139968643069712 postgres | omm | 10 | | 139968680818448 postgres | joe | 16390 | | 139968563377936 (6 rows)` | +| resource_pool | name | Resource pool used by the user | +| query_id | bigint | ID of a query | +| query | text | Text of this backend's most recent query. If **state** is **active**, this column shows the ongoing query. In all other states, it shows the last query that was executed. | +| connection_info | text | A string in JSON format recording the driver type, driver version, driver deployment path, and process owner of the connected database. For details, see **connection_info**. | + +## **Table 2** GS_WLM_SESSION_HISTORY Columns + +| Name | Type | Description | +| ------------------------ | ---------------------- | ---------------------- | +| datid | oid | OID of the database that the backend is connected to | +| dbname | text | Name of the database that the backend is connected to | +| schemaname | text | Schema name | +| nodename | text | Name of the database node where the statement is executed | +| username | text | Username used for connecting to the backend | +| application_name | text | Name of the application connected to the backend | +| client_addr | inet | IP address of the client connected to the backend. If this column is null, it indicates either the client is connected via a Unix socket on the server or this is an internal process, such as autovacuum. | +| client_hostname | text | Host name of the connected client, as reported by a reverse DNS lookup of `client_addr`. This column will be non-null only for IP connections and only when **log_hostname** is enabled. | +| client_port | integer | TCP port number that the client uses for communication with this backend (**-1** if a Unix socket is used) | +| query_band | text | Job type, which is specified by the GUC parameter **query_band**. The default value is a null string. | +| block_time | bigint | Duration that the statement is blocked before being executed, including the statement parsing and optimization duration (unit: ms) | +| start_time | timestamp with time zone | Time when the statement starts to be executed | +| finish_time | timestamp with time zone | Time when the statement execution ends | +| duration | bigint | Execution time of the statement, in ms | +| estimate_total_time | bigint | Estimated execution time of the statement, in ms | +| status | text | Final statement execution status, which can be **finished** (normal) or **aborted** (abnormal). | +| abort_info | text | Exception information displayed if the final statement execution status is **aborted** | +| resource_pool | text | Resource pool used by the user | +| control_group | text | The function is not supported currently. | +| estimate_memory | integer | Estimated memory size of the statement. | +| min_peak_memory | integer | Minimum memory peak of the statement across the database nodes, in MB | +| max_peak_memory | integer | Maximum memory peak of the statement across the database nodes, in MB | +| average_peak_memory | integer | Average memory usage during statement execution (unit: MB) | +| memory_skew_percent | integer | Memory usage skew of the statement among the database nodes | +| spill_info | text | Information about statement spill to the database nodes
- **None**: The statement has not been spilled to disks on the database nodes.
- **All**: The statement has been spilled to disks on the database nodes.
- **[a:b]**: The statement has been spilled to disks on *a* of *b* database nodes. | +| min_spill_size | integer | Minimum spilled data among database nodes when a spill occurs, in MB (default value:**0**) | +| max_spill_size | integer | Maximum spilled data among database nodes when a spill occurs, in MB (default value:**0**) | +| average_spill_size | integer | Average spilled data among database nodes when a spill occurs, in MB (default value:**0**) | +| spill_skew_percent | integer | database node spill skew when a spill occurs | +| min_dn_time | bigint | Minimum execution time of the statement across the database nodes, in ms | +| max_dn_time | bigint | Maximum execution time of the statement across the database nodes, in ms | +| average_dn_time | bigint | Average execution time of the statement across the database nodes, in ms | +| dntime_skew_percent | integer | Execution time skew of the statement among the database nodes | +| min_cpu_time | bigint | Minimum CPU time of the statement across the database nodes, in ms | +| max_cpu_time | bigint | Maximum CPU time of the statement across the database nodes, in ms | +| total_cpu_time | bigint | Total CPU time of the statement across the database nodes, in ms | +| cpu_skew_percent | integer | CPU time skew of the statement among database nodes | +| min_peak_iops | integer | Minimum IOPS peak of the statement across the database nodes. It is counted by ones in a column-store table and by ten thousands in a row-store table. | +| max_peak_iops | integer | Maximum IOPS peak of the statement across the database nodes. It is counted by ones in a column-store table and by ten thousands in a row-store table. | +| average_peak_iops | integer | Average IOPS peak of the statement across the database nodes. It is counted by ones in a column-store table and by ten thousands in a row-store table. | +| iops_skew_percent | integer | I/O skew across database nodes | +| warning | text | Warning. The following warnings are displayed:
- Spill file size large than 256 MB
- Broadcast size large than 100 MB
- Early spill
- Spill times is greater than 3
- Spill on memory adaptive
- Hash table conflict | +| queryid | bigint | Internal query ID used for statement execution | +| query | text | Statement executed | +| query_plan | text | Execution plan of the statement | +| node_group | text | Unsupported currently | +| cpu_top1_node_name | text | Name of the current database node | +| cpu_top2_node_name | text | Unsupported currently | +| cpu_top3_node_name | text | Unsupported currently | +| cpu_top4_node_name | text | Unsupported currently | +| cpu_top5_node_name | text | Unsupported currently | +| mem_top1_node_name | text | Current database name | +| mem_top2_node_name | text | Unsupported currently | +| mem_top3_node_name | text | Unsupported currently | +| mem_top4_node_name | text | Unsupported currently | +| mem_top5_node_name | text | Unsupported currently | +| cpu_top1_value | bigint | CPU usage of the current database node | +| cpu_top2_value | bigint | Unsupported currently | +| cpu_top3_value | bigint | Unsupported currently | +| cpu_top4_value | bigint | Unsupported currently | +| cpu_top5_value | bigint | Unsupported currently | +| mem_top1_value | bigint | Memory usage of the current database node | +| mem_top2_value | bigint | Unsupported currently | +| mem_top3_value | bigint | Unsupported currently | +| mem_top4_value | bigint | Unsupported currently | +| mem_top5_value | bigint | Unsupported currently | +| top_mem_dn | text | Memory usage information of the current database node | +| top_cpu_dn | text | CPU usage information of the current database node | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/1-oracle_fdw.md b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/1-oracle_fdw.md new file mode 100644 index 0000000000000000000000000000000000000000..db54e0fe688f0ea9e4dcaa0867c6e355a28a2576 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/1-oracle_fdw.md @@ -0,0 +1,49 @@ +--- +title: oracle_fdw +summary: oracle_fdw +author: Zhang Cuiping +date: 2021-05-17 +--- + +# oracle_fdw + +oracle_fdw is an open-source plug-in. MogDB is developed and adapted based on the open-source oracle_fdw Release 2.2.0. + +To compile and use oracle_fdw, the Oracle development packages must be included in the environment. Therefore, MogDB does not compile oracle_fdw by default. The following describes how to compile and use oracle_fdw. + +## Compiling oracle_fdw + +To compile oracle_fdw, install the Oracle development library and header files from the [Oracle official website](https://www.oracle.com/database/technologies/instant-client/downloads.html). + +Select a proper running environment and version, download **Basic Package** and **SDK Package**, and install them. In addition, **SQLPlus Package** is a client tool of the Oracle server. You can install it as required to connect to the Oracle server for testing. + +After installing the development packages, start oracle_fdw compilation. Add the `--enable-oracle-fdw` option when running the **configure** command. Perform compilation using the common MogDB compilation method. + +After the compilation is complete, the **oracle_fdw.so** file is generated in `lib/postgresql/` in the installation directory. SQL files and control files related to oracle_fdw are stored in `share/postgresql/extension/` in the installation directory. + +If the `--enable-oracle-fdw` option is not added during compilation and installation, compile oracle_fdw again after MogDB is installed, and then manually place the `oracle\_fdw.so` file to `lib/postgresql/` in the installation directory, and place `oracle\_fdw-1.0-1.1.sql`, `oracle\_fdw-1.1.sql`, and `oracle\_fdw.control` to `share/postgresql/extension/` in the installation directory. + +## Using oracle_fdw + +- To use oracle_fdw, install and connect to the Oracle server. +- Load the oracle_fdw extension using **CREATE EXTENSION oracle_fdw**. +- Create a server object using **CREATE SERVER**. +- Create a user mapping using **CREATE USER MAPPING**. +- Create a foreign table using **CREATE FOREIGN TABLE**. The structure of the foreign table must be the same as that of the Oracle table. The first column in the table on the Oracle server must be unique, for example, **PRIMARY KEY** and **UNIQUE**. +- Perform normal operations on the foreign table, such as **INSERT**, **UPDATE**, **DELETE**, **SELECT**, **EXPLAIN**, **ANALYZE** and **COPY**. +- Drop a foreign table using **DROP FOREIGN TABLE**. +- Drop a user mapping using **DROP USER MAPPING**. +- Drop a server object using **DROP SERVER**. +- Drop an extension using **DROP EXTENSION oracle_fdw**. + +## Common Issues + +- When a foreign table is created on the MogDB, the table is not created on the Oracle server. You need to use the Oracle client to connect to the Oracle server to create a table. +- The Oracle user used for executing **CREATE USER MAPPING** must have the permission to remotely connect to the Oracle server and perform operations on tables. Before using a foreign table, you can use the Oracle client on the machine where the MogDB server is located and use the corresponding user name and password to check whether the Oracle server can be successfully connected and operations can be performed. +- When **CREATE EXTENSION oracle_fdw;** is executed, the message `libclntsh.so: cannot open shared object file: No such file or directory` is displayed. The reason is that the Oracle development library **libclntsh.so** is not in the related path of the system. You can find the specific path of **libclntsh.so**, and then add the folder where the **libclntsh.so** file is located to **/etc/ld.so.conf**. For example, if the path of **libclntsh.so** is **/usr/lib/oracle/11.2/client64/lib/libclntsh.so.11.1**, add **/usr/lib/oracle/11.2/client64/lib/** to the end of **/etc/ld.so.conf**. Run the **ldconfig** command for the modification to take effect. Note that this operation requires the **root** permission. + +## Precautions + +- **SELECT JOIN** between two Oracle foreign tables cannot be pushed down to the Oracle server for execution. Instead, **SELECT JOIN** is divided into two SQL statements and transferred to the Oracle server for execution. Then the processing result is summarized in the MogDB. +- The **IMPORT FOREIGN SCHEMA** syntax is not supported. +- **CREATE TRIGGER** cannot be executed for foreign tables. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/2-mysql_fdw.md b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/2-mysql_fdw.md new file mode 100644 index 0000000000000000000000000000000000000000..52aeab78d46ce744885930f8dceef8d46a08de0c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/2-mysql_fdw.md @@ -0,0 +1,49 @@ +--- +title: mysql_fdw +summary: mysql_fdw +author: Zhang Cuiping +date: 2021-05-17 +--- + +# mysql_fdw + +mysql_fdw is an open-source plug-in. MogDB is developed and adapted based on the open-source [mysql_fdw Release 2.5.3](https://github.com/EnterpriseDB/mysql_fdw/archive/REL-2_5_3.tar.gz). + +To compile and use mysql_fdw, the MariaDB development packages must be included in the environment. Therefore, MogDB does not compile mysql_fdw by default. The following describes how to compile and use mysql_fdw. + +## Compiling mysql_fdw + +To compile mysql_fdw, install the development library and header file of MariaDB. You are advised to use the official MariaDB repositories. For details about how to select a repository, visit [http://downloads.mariadb.org/mariadb/repositories/](http://downloads.mariadb.org/mariadb/repositories). + +After the repository is configured, run the **yum install MariaDB-devel MariaDB-shared** command to install the related development libraries. In addition, **MariaDB-client** is a client tool of the MariaDB. You can install it as required to connect to the MariaDB for testing. + +After installing the development packages, start mysql_fdw compilation. Add the `--enable-mysql-fdw` option when running the **configure** command. Perform compilation using the common MogDB compilation method. + +After the compilation is complete, the **mysql_fdw.so** file is generated in **lib/postgresql/** in the installation directory. SQL files and control files related to mysql_fdw are stored in **share/postgresql/extension/** in the installation directory. + +If the `--enable-mysql-fdw` option is not added during compilation and installation, compile mysql_fdw again after MogDB is installed, and then manually place the `mysql\_fdw.so` file to **lib/postgresql/** in the installation directory, and place **mysql\_fdw-1.0-1.1.sql**, **mysql\_fdw-1.1.sql**, and **mysql\_fdw.control** to **share/postgresql/extension/** in the installation directory. + +## Using mysql_fdw + +- To use mysql_fdw, install and connect to MariaDB or MySQL server. +- Load the mysql_fdw extension using **CREATE EXTENSION mysql_fdw;**. +- Create a server object using **CREATE SERVER**. +- Create a user mapping using **CREATE USER MAPPING**. +- Create a foreign table using **CREATE FOREIGN TABLE**. The structure of the foreign table must be the same as that of the MySQL or MariaDB table. The first column in the table on the MySQL or MariaDB must be unique, for example, **PRIMARY KEY** and **UNIQUE**. +- Perform normal operations on the foreign table, such as **INSERT**, **UPDATE**, **DELETE**, **SELECT**, **EXPLAIN**, **ANALYZE** and **COPY**. +- Drop a foreign table using **DROP FOREIGN TABLE**. +- Drop a user mapping using **DROP USER MAPPING**. +- Drop a server object using **DROP SERVER**. +- Drop an extension using **DROP EXTENSION mysql_fdw;**. + +## Common Issues + +- When a foreign table is created on the MogDB, the table is not created on the MariaDB or MySQL server. You need to use the MariaDB or MySQL server client to connect to the MariaDB or MySQL server to create a table. +- The MariaDB or MySQL server user used for creating **USER MAPPING** must have the permission to remotely connect to the MariaDB or MySQL server and perform operations on tables. Before using a foreign table, you can use the MariaDB or MySQL server client on the machine where the MogDB server is located and use the corresponding user name and password to check whether the MariaDB or MySQL server can be successfully connected and operations can be performed. +- The **Can't initialize character set SQL_ASCII (path: compiled_in)** error occurs when the DML operation is performed on a foreign table. MariaDB does not support the **SQL_ASCII** encoding format. Currently, this problem can be resolved only by modifying the encoding format of the MogDB database. Change the database encoding format to **update pg_database set encoding = pg_char_to_encoding('UTF-8') where datname = 'postgres';**. Set **datname** based on the actual requirements. After the encoding format is changed, start a gsql session again so that mysql_fdw can use the updated parameters. You can also use `-locale=LOCALE` when running **gs_initdb** to set the default encoding format to non-SQL_ASCII. + +## Precautions + +- **SELECT JOIN** between two MySQL foreign tables cannot be pushed down to the MariaDB or MySQL server for execution. Instead, **SELECT JOIN** is divided into two SQL statements and transferred to the MariaDB or MySQL server for execution. Then the processing result is summarized in the MogDB. +- The **IMPORT FOREIGN SCHEMA** syntax is not supported. +- **CREATE TRIGGER** cannot be executed for foreign tables. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/3-postgres_fdw.md b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/3-postgres_fdw.md new file mode 100644 index 0000000000000000000000000000000000000000..50864c9e199e3024164a19597cd6f9f662f09489 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/3-postgres_fdw.md @@ -0,0 +1,37 @@ +--- +title: postgres_fdw +summary: postgres_fdw +author: Zhang Cuiping +date: 2021-05-17 +--- + +# postgres_fdw + +postgres_fdw is an open-source plug-in. Its code is released with the PostgreSQL source code. MogDB is developed and adapted based on the open-source postgres_fdw source code () in PostgreSQL 9.4.26. + +The postgres_fdw plug-in is involved in compilation by default. After installing MogDB using the installation package, you can directly use postgres_fdw without performing other operations. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** Currently, postgres_fdw supports only connection between MogDB databases. + +## Using postgres_fdw + +- Load the postgres_fdw extension using **CREATE EXTENSION postgres_fdw;**. +- Create a server object using **CREATE SERVER**. +- Create a user mapping using **CREATE USER MAPPING**. +- Create a foreign table using **CREATE FOREIGN TABLE**. The structure of the foreign table must be the same as that of the remote MogDB table. +- Perform normal operations on the foreign table, such as **INSERT**, **UPDATE**, **DELETE**, **SELECT**, **EXPLAIN**, **ANALYZE** and **COPY**. +- Drop a foreign table using **DROP FOREIGN TABLE**. +- Drop a user mapping using **DROP USER MAPPING**. +- Drop a server object using **DROP SERVER**. +- Drop an extension using **DROP EXTENSION postgres_fdw;**. + +## Common Issues + +- When a foreign table is created on the MogDB, the table is not created on the remote MogDB database. You need to use the Oracle client to connect to the remote MogDB database to create a table. +- The MogDB user used for executing **CREATE USER MAPPING** must have the permission to remotely connect to the MogDB database and perform operations on tables. Before using a foreign table, you can use the gsql client on the local machine and use the corresponding user name and password to check whether the remote MogDB database can be successfully connected and operations can be performed. + +## Precautions + +- **SELECT JOIN** between two postgres_fdw foreign tables cannot be pushed down to the remote MogDB database for execution. Instead, **SELECT JOIN** is divided into two SQL statements and transferred to the remote MogDB database for execution. Then the processing result is summarized locally. +- The **IMPORT FOREIGN SCHEMA** syntax is not supported. +- **CREATE TRIGGER** cannot be executed for foreign tables. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/dblink.md b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/dblink.md new file mode 100644 index 0000000000000000000000000000000000000000..a0018a1092bad2141d4b9dacbfee8f664d308e96 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/dblink.md @@ -0,0 +1,80 @@ +--- +title: dblink +summary: dblink +author: Guo Huan +date: 2021-11-22 +--- + +# dblink + +dblink is a tool that can connect to other MogDB databases in an MogDB database session. The connection parameters supported by dblink are the same as those supported by libpq. For details, see [Link Parameters](../../developer-guide/dev/4-development-based-on-libpq/link-parameters.md). By default, MogDB does not compile dblink. The following describes how to compile and use dblink. + +## Compiling dblink + +Currently, the source code of dblink is stored in the [contrib/dblink](https://gitee.com/opengauss/openGauss-server/tree/master/contrib/dblink) directory. After the MogDB database is compiled and installed, if you need to use the dblink, go to the preceding directory and run the following command to compile and install the dblink: + +```bash +make +make install +``` + +## Common dblink Functions + +- Load the dblink extension. + + ```sql + CREATE EXTENSION dblink; + ``` + +- Open a persistent connection to a remote database. + + ```sql + SELECT dblink_connect(text connstr); + ``` + +- Close a persistent connection to a remote database. + + ```sql + SELECT dblink_disconnect(); + ``` + +- Query data in a remote database. + + ```sql + SELECT * FROM dblink(text connstr, text sql); + ``` + +- Execute commands in a remote database. + + ```sql + SELECT dblink_exec(text connstr, text sql); + ``` + +- Return the names of all opened dblinks. + + ```sql + SELECT dblink_get_connections(); + ``` + +- Send an asynchronous query to a remote database. + + ```sql + SELECT dblink_send_query(text connname, text sql); + ``` + +- Check whether the connection is busy with an asynchronous query. + + ```sql + SELECT dblink_is_busy(text connname); + ``` + +- Delete the extension. + + ```sql + DROP EXTENSION dblink; + ``` + +## Precautions + +- Currently, dblink allows only the MogDB database to access another MogDB database and does not allow the MogDB database to access a PostgreSQL database. +- Currently, dblink does not support the thread pool mode. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/fdw-introduction.md b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/fdw-introduction.md new file mode 100644 index 0000000000000000000000000000000000000000..d1d655fbafa870f9a6b002f80a28d0d3a95bce1d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/fdw-introduction.md @@ -0,0 +1,10 @@ +--- +title: Introduction +summary: Introduction +author: Zhang Cuiping +date: 2021-05-17 +--- + +# Introduction + +The foreign data wrapper (FDW) of the MogDB can implement cross-database operations between MogDB databases and remote databases. Currently, the following remote servers are supported: Oracle, MySQL(MariaDB), PostgreSQL/openGauss/MogDB(postgres_fdw), file_fdw, dblink. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/file_fdw.md b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/file_fdw.md new file mode 100644 index 0000000000000000000000000000000000000000..14660817e53ed10eb6482155387b9be8a2783019 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/foreign-data-wrapper/file_fdw.md @@ -0,0 +1,81 @@ +--- +title: file_fdw +summary: file_fdw +author: Guo Huan +date: 2021-10-19 +--- + +# file_fdw + +The file_fdw module provides the external data wrapper file_fdw, which can be used to access data files in the file system of a server. The format of the data files must be readable by the **COPY FROM** command. For details, see [COPY](../../reference-guide/sql-syntax/COPY.md). file_fdw is only used to access readable data files, but cannot write data to the data files. + +By default, the file_fdw is compiled in MogDB. During database initialization, the plug-in is created in the **pg_catalog** schema. + +When you create a foreign table using file_fdw, you can add the following options: + +- filename + + File to be read. This parameter is mandatory and must be an absolute path. + +- format + + File format of the remote server, which is the same as the **FORMAT** option in the **COPY** statement. The value can be **text**, **csv**, **binary**, or **fixed**. + +- header + + Whether the specified file has a header, which is the same as the **HEADER** option of the **COPY** statement. + +- delimiter + + File delimiter, which is the same as the **DELIMITER** option of the **COPY** statement. + +- quote + + Quote character of a file, which is the same as the **QUOTE** option of the **COPY** statement. + +- escape + + Escape character of a file, which is the same as the **ESCAPE** option of the **COPY** statement. + +- null + + Null string of a file, which is the same as the **NULL** option of the **COPY** statement. + +- encoding + + Encoding of a file, which is the same as the **ENCODING** option of the **COPY** statement. + +- force_not_null + + File-level null option, which is a Boolean option. If it is true, the value of the declared field cannot be an empty string. This option is the same as the **FORCE_NOT_NULL** option of the **COPY** statement. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - file_fdw does not support the **OIDS** and **FORCE\_QUOTE** options of the **COPY** statement. +> - These options can only be declared for a foreign table or the columns of the foreign table, not for the file_fdw itself, nor for the server or user mapping that uses file_fdw. +> - To modify table-level options, you must obtain the system administrator role permissions. For security reasons, only the system administrator can determine the files to be read. +> - For an external table that uses file_fdw, **EXPLAIN** displays the name and size (in bytes) of the file to be read. If the keyword **COSTS OFF** is specified, the file size is not displayed. + +## Using file_fdw + +- To create a server object, use **CREATE SERVER**. + +- To create a user mapping, use **CREATE USER MAPPING**. + +- To create a foreign table, use **CREATE FOREIGN TABLE**. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > + > - The structure of the foreign table must be consistent with the data in the specified file. + > - When a foreign table is queried, no write operation is allowed. + +- To drop a foreign table, use **DROP FOREIGN TABLE**. + +- To drop a user mapping, use **DROP USER MAPPING**. + +- To drop a server object, use **DROP SERVER**. + +## Precautions + +- To use file_fdw, you need to specify the file to be read. Prepare the file and grant the read permission on the file for the database to access the file. +- **DROP EXTENSION file_fdw** is not supported. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/logical-decoding/1-logical-decoding.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/logical-decoding/1-logical-decoding.md new file mode 100644 index 0000000000000000000000000000000000000000..4dacfb06005dd71f819f24d118ffcd6e13ce4dd2 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/logical-decoding/1-logical-decoding.md @@ -0,0 +1,56 @@ +--- +title: Overview +summary: Overview +author: Zhang Cuiping +date: 2021-05-10 +--- + +# Overview + +## Function + +The data replication capabilities supported by MogDB are as follows: + +Data is periodically synchronized to heterogeneous databases (such as Oracle databases) using a data migration tool. Real-time data replication is not supported. Therefore, the requirements for real-time data synchronization to heterogeneous databases are not satisfied. + +MogDB provides the logical decoding function to generate logical logs by decoding Xlogs. A target database parses logical logs to replicate data in real time. For details, see Figure 1. Logical replication reduces the restrictions on target databases, allowing for data synchronization between heterogeneous databases and homogeneous databases with different forms. It allows data to be read and written during data synchronization on a target database, reducing the data synchronization latency. + +**Figure 1** Logical replication + +![image-20210512181021060](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/logical-decoding-2.png) + +Logical replication consists of logical decoding and data replication. Logical decoding outputs logical logs by transaction. The database service or middleware parses the logical logs to implement data replication. Currently, MogDB supports only logical decoding. Therefore, this section involves only logical decoding. + +Logical decoding provides basic transaction decoding capabilities for logical replication. MogDB uses SQL functions for logical decoding. This method features easy function calling, requires no tools to obtain logical logs, and provides specific interfaces for interconnecting with external replay tools, saving the need of additional adaptation. + +Logical logs are output only after transactions are committed because they use transactions as the unit and logical decoding is driven by users. Therefore, to prevent Xlogs from being reclaimed by the system when transactions start and prevent required transaction information from being reclaimed by **VACUUM**, MogDB introduces logical replication slots to block Xlog reclaiming. + +A logical replication slot means a stream of changes that can be replayed in other databases in the order they were generated in the original database. Each owner of logical logs maintains one logical replication slot. + +## Precautions + +- DDL statement decoding is not supported. When a specific DDL statement (for example, to truncate an ordinary table or exchange a partitioned table) is executed, decoded data may be lost. +- Decoding for column-store data and data page replication is not supported. +- Logical decoding is not supported on the cascaded standby node. +- After a DDL statement (for example, **ALTER TABLE**) is executed, the physical logs that are not decoded before the DDL statement execution may be lost. +- The size of a single tuple cannot exceed 1 GB, and decoded data may be larger than inserted data. Therefore, it is recommended that the size of a single tuple be less than or equal to 500 MB. +- openGauss supports the following data types for decoding: **INTEGER**, **BIGINT**, **SMALLINT**, **TINYINT**, **SERIAL**, **SMALLSERIAL**, **BIGSERIAL**, **FLOAT**, **DOUBLE PRECISION**, **DATE**, **TIME[WITHOUT TIME ZONE]**, **TIMESTAMP[WITHOUT TIME ZONE]**, **CHAR(***n***)**, **VARCHAR(***n***)**, and **TEXT**. +- If SSL connections are required, make sure that the GUC parameter **ssl** is set to **on**. +- The logical replication slot name must contain fewer than 64 characters and contain only one or more types of the following characters: lowercase letters, digits, and underscores (_). +- Currently, logical replication does not support the MOT feature. +- After the database where a logical replication slot resides is deleted, the replication slot becomes unavailable and needs to be manually deleted. +- Only the UTF-8 character set is supported. +- To decode multiple databases, you need to create a stream replication slot in each database and start decoding. Logs need to be scanned for decoding of each database. +- Forcible startup is not supported. After forcible startup, you need to export all data again. +- During decoding on the standby node, the decoded data may increase during switchover and failover, which needs to be manually filtered out. When the quorum protocol is used, switchover and failover should be performed on the standby node that is to be promoted to primary, and logs must be synchronized from the primary node to the standby node. +- The same replication slot for decoding cannot be used between the primary node and standby node or between different standby nodes at the same time. Otherwise, data inconsistency occurs. +- Replication slots can only be created or deleted on hosts. +- After the database is restarted due to a fault or the logical replication process is restarted, duplicate decoded data exists. You need to filter out the duplicate data. +- If the computer kernel is faulty, garbled characters are displayed during decoding which need to be manually or automatically filtered out. +- Currently, the logical decoding on the standby node does not support enabling the ultimate RTO. +- Ensure that the long transaction is not started during the creation of the logical replication slot. If the long transaction is started, the creation of the logical replication slot will be blocked. +- Interval partitioned tables cannot be replicated. +- Global temporary tables are not supported. +- After a DDL statement is executed in a transaction, the DDL statement and subsequent statements are not decoded. +- To perform decoding on the standby node, set the GUC parameter **enable_slot_log** to **on** on the corresponding host. +- Do not perform operations on the replication slot on other nodes when the logical replication slot is in use. To delete a replication slot, stop decoding in the replication slot first. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/logical-decoding/2-logical-decoding-by-sql-function-interfaces.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/logical-decoding/2-logical-decoding-by-sql-function-interfaces.md new file mode 100644 index 0000000000000000000000000000000000000000..d3cbfc087c8b08c5c589660d3bbc1fdd0dcfea8c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/logical-decoding/2-logical-decoding-by-sql-function-interfaces.md @@ -0,0 +1,85 @@ +--- +title: Logical Decoding by SQL Function Interfaces +summary: Logical Decoding by SQL Function Interfaces +author: Zhang Cuiping +date: 2021-05-10 +--- + +# Logical Decoding by SQL Function Interfaces + +In MogDB, you can call SQL functions to create, delete, and push logical replication slots, as well as obtain decoded transaction logs. + +## Prerequisites + +- Currently, logical logs are extracted from host nodes. Since SSL connections are disabled by default, to perform logical replication, set the GUC parameter [ssl](../../../reference-guide/guc-parameters/3-connection-and-authentication/2-security-and-authentication.md#ssl) to **on** on host nodes. + + > **NOTE:** For security purposes, ensure that SSL connections are enabled. + +- The GUC parameter [wal_level](../../../reference-guide/guc-parameters/6-write-ahead-log/1-settings.md#wal_level) is set to **logical**. + +- The GUC parameter [max_replication_slots](../../../reference-guide/guc-parameters/7-ha-replication/1-sending-server.md#max_replication_slots) is set to a value greater than the number of physical replication slots and logical replication slots required by each node. + + Physical replication slots provide an automatic method to ensure that Xlogs are not removed from a primary node before they are received by all the standby nodes and secondary nodes. That is, physical replication slots are used to support HA clusters. The number of physical replication slots required by a cluster is equal to the ratio of standby and secondary nodes to the primary node. For example, if an HA cluster has 1 primary node, 1 standby node, and 1 secondary node, the number of required physical replication slots will be 2. If an HA cluster has 1 primary node and 3 standby nodes, the number of required physical replication slots will be 3. + + Plan the number of logical replication slots as follows: + + - A logical replication slot can carry changes of only one database for decoding. If multiple databases are involved, create multiple logical replication slots. + - If logical replication is needed by multiple target databases, create multiple logical replication slots in the source database. Each logical replication slot corresponds to one logical replication link. + +- Only initial users and users with the **REPLICATION** permission can perform this operation. When separation of duties is disabled, database administrators can perform logical replication operations. When separation of duties is enabled, database administrators are not allowed to perform logical replication operations. + +- Currently, primary/standby/secondary deployment is not supported by default. + +## Procedure + +1. Log in to the primary node of the MogDB cluster as the cluster installation user. + +2. Run the following command to connect to the default database **postgres**: + + ```bash + gsql -d postgres -p 16000 -r + ``` + + In this command, **16000** is the database port number. It can be replaced by an actual port number. + +3. Create a logical replication slot named **slot1**. + + ``` + MogDB=# SELECT * FROM pg_create_logical_replication_slot('slot1', 'mppdb_decoding'); + slotname | xlog_position + ----------+--------------- + slot1 | 0/601C150 + (1 row) + ``` + +4. Create a table **t** in the database and insert data into it. + + ``` + MogDB=# CREATE TABLE t(a int PRIMARY KEY, b int); + MogDB=# INSERT INTO t VALUES(3,3); + ``` + +5. Read the decoding result of **slot1**. The number of decoded records is 4096. + + ``` + MogDB=# SELECT * FROM pg_logical_slot_peek_changes('slot1', NULL, 4096); + location | xid | data + -----------+-------+------------------------------------------------------------------------------------------------------------------------------------------------- + ------------------------------------------- + 0/601C188 | 1010023 | BEGIN 1010023 + 0/601ED60 | 1010023 | COMMIT 1010023 CSN 1010022 + 0/601ED60 | 1010024 | BEGIN 1010024 + 0/601ED60 | 1010024 | {"table_name":"public.t","op_type":"INSERT","columns_name":["a","b"],"columns_type":["integer","integer"],"columns_val":["3","3"],"old_keys_name":[],"old_keys_type":[],"old_keys_val":[]} + 0/601EED8 | 1010024 | COMMIT 1010024 CSN 1010023 + (5 rows) + ``` + +6. Delete the logical replication slot **slot1**. + + ``` + MogDB=# SELECT * FROM pg_drop_replication_slot('slot1'); + pg_drop_replication_slot + -------------------------- + + (1 row) + ``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/architecture.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/architecture.md new file mode 100644 index 0000000000000000000000000000000000000000..7aeb8ed1e655f91a1c602265b760b8846f665c87 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/architecture.md @@ -0,0 +1,16 @@ +--- +title: Architecture +summary: Architecture +author: Guo Huan +date: 2022-04-29 +--- + +# Architecture + +Changes on publishers are sent to subscribers in real time as they occur. The subscriber applies data in the order in which it is committed on the publisher to ensure transactional consistency of publications in any single subscription. + +Logical replication is built with an architecture similar to physical streaming replication. It is implemented by the walsender and apply processes. The walsender process starts logical decoding of the WAL and loads the standard logical decoding plug-in (pgoutput). The plug-in transforms the changes read from the WAL into a logical replication protocol and filters the data according to the publication specifications. The data is then continuously transferred to the apply worker using the streaming replication protocol, and the apply worker maps the data to the local table and applies the changes they receive in the correct transactional order. + +The apply process in the subscriber database always runs with **session_replication_role** set to **replica**, which produces the usual effects on triggers and constraints. + +The logical replication apply process currently fires only row triggers, not statement triggers. However, the initial table synchronization is implemented through methods similar to **COPY** command execution, and therefore, row and statement triggers for INSERT are fired. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/configuration-settings.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/configuration-settings.md new file mode 100644 index 0000000000000000000000000000000000000000..5cd73eddd4e34c7975eabc6cba29297995dd3945 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/configuration-settings.md @@ -0,0 +1,13 @@ +--- +title: Configuration Settings +summary: Configuration Settings +author: Guo Huan +date: 2022-04-29 +--- + +# Configuration Settings + +Publication-subscription requires some configuration options to be set. + +- On the publisher, **wal_level** must be set to **logical**, and the value of **max_replication_slots** must be at least the minimum number of subscriptions expected to be connected. **max_wal_senders** should be set to at least the value of **max_replication_slots** plus the number of physical replicas that are connected at the same time. +- **max_replication_slots** must also be set on the subscriber. It must be set to at least the number of subscriptions that will be added to the subscriber. **max_logical_replication_workers** must be set to at least the minimum number of subscriptions. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/conflicts.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/conflicts.md new file mode 100644 index 0000000000000000000000000000000000000000..d06c4d6b8edd2d9a2da762d292c55e0484f0fc4b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/conflicts.md @@ -0,0 +1,14 @@ +--- +title: Conflicts +summary: Conflicts +author: Guo Huan +date: 2022-04-29 +--- + +# Conflicts + +Logical replication behaves similarly to common DML operations. Even if the data is modified locally on the subscriber node, logical replication updates the data based on the received changes. If the incoming data violates any constraints, the replication will stop. This situation is called a conflict. When UPDATE or DELETE operations are replicated, the missing data will not cause conflicts and such operations will be simply skipped. + +A conflict will cause errors and stop the replication, which must be resolved manually by the user. Details about the conflict can be found in the subscriber's server log. + +The conflict can be resolved either by changing the data on the subscriber (so that the data does not conflict with incoming data) or by skipping the transaction that conflicts with the existing data. The transaction can be skipped by calling the **pg_replication_origin_advance()** function with **node_name** corresponding to the subscription name, and an Xlog LSN. The current position of the replication source can be seen in the **pg_replication_origin_status system** view. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/monitoring.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/monitoring.md new file mode 100644 index 0000000000000000000000000000000000000000..701e70b4ce13fd3827bc3be889073cb15098c81d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/monitoring.md @@ -0,0 +1,14 @@ +--- +title: Monitoring +summary: Monitoring +author: Guo Huan +date: 2022-04-29 +--- + +# Monitoring + +Because logical replication is based on an architecture similar to physical streaming replication, monitoring on a publication node is similar to monitoring on a primary physical replication node. + +Monitoring information about subscriptions is available in the **pg_stat_subscription** view. This view contains one row for every subscription worker. A subscription can have zero or more active subscription workers depending on its state. + +Normally, a single apply process runs for an enabled subscription. A disabled or crashed subscription does not have rows in this view. If data synchronization of any table is in progress, there will be additional workers for the tables being synchronized. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/publication-subscription.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/publication-subscription.md new file mode 100644 index 0000000000000000000000000000000000000000..815a8920ae07f8d1e2cf598ad4c3e91ebb2d9dcb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/publication-subscription.md @@ -0,0 +1,20 @@ +--- +title: Publication-Subscription +summary: Publication-Subscription +author: Guo Huan +date: 2022-04-29 +--- + +# Publication-Subscription + +Publication-subscription is implemented based on logical replication, with one or more subscribers subscribing to one or more publications on a publisher node. The subscriber pulls data from the publication they subscribe to. + +Changes on the publisher are sent to the subscriber as they occur in real time. The subscriber applies the data in the same order as the publisher, so that transactional consistency is guaranteed for publications within a single subscription. This method of data replication is sometimes called transactional replication. + +The typical usage of publication-subscription is as follows: + +- Sending incremental changes in a database or a subset of a database to subscribers as they occur +- Firing triggers when changes reach subscribers +- Consolidating multiple databases into a single one (for example, for analysis purposes) + +The subscriber database behaves in the same way as any other MogDB instance and can be used as a publisher for other databases by defining its own publications. When the subscriber is treated as read-only by an application, there will be no conflicts in a single subscription. On the other side, conflicts may occur if other write operations are performed by the application or by other subscribers in the same set of tables. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/publications.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/publications.md new file mode 100644 index 0000000000000000000000000000000000000000..62524bc7920ae7c047abc5f0513d41bd3639902c --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/publications.md @@ -0,0 +1,22 @@ +--- +title: Publications +summary: Publications +author: Guo Huan +date: 2022-04-29 +--- + +# Publications + +A publication can be defined on any primary physical replication server. The node where a publication is defined is called the publisher. A publication is a set of changes generated from a table or a group of tables. It can also be described as a change set or replication set. Each publication exists in only one database. + +Publications are different from schemas and do not affect how tables are accessed. Each table can be added to multiple publications if needed. Currently, publications can contain only tables. Objects must be explicitly added to a publication, unless the publication is created by running **ALL TABLES**. + +Publications can choose to limit the changes they produce to any combination of **INSERT**, **UPDATE**, and **DELETE**, which is similar to how triggers are fired by particular events. By default, all types of operations are replicated. + +A published table must be configured with a “replication identifier” in order to be able to replicate UPDATE and DELETE operations, so that appropriate rows to be updated or deleted can be identified on the subscriber side. By default, the replication identifier is the primary key (if any). You can also set another unique index (with certain additional requirements) to be the replication identifier. If the table does not have any suitable key, you can set the replication identifier to “full”, which indicates that the entire row becomes the key. However, this is very inefficient and should be used only when there are no other solutions. If a replication identifier other than “full” is set on the publisher side, a replication identifier comprising the same or fewer columns must also be set on the subscriber side. If a table without a replication identifier is added to a publication that replicates UPDATE or DELETE operations, subsequent UPDATE or DELETE operations on the subscriber side will cause an error. INSERT operations can proceed regardless of any replication identifier. + +Each publication can have multiple subscribers. + +A publication is created by running the **CREATE PUBLICATION** command and can be altered or dropped by running the corresponding commands. + +Tables can be added or removed dynamically by running the **ALTER PUBLICATION** command. ADD TABLE and DROP TABLE operations are transactional. Therefore, once the transaction is committed, table replication will start or stop using a proper snapshot. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/quick-setup.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/quick-setup.md new file mode 100644 index 0000000000000000000000000000000000000000..268590ec7b4c48cba7c0a8ac69dc954c0a8280ae --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/quick-setup.md @@ -0,0 +1,36 @@ +--- +title: Quick Setup +summary: Quick Setup +author: Guo Huan +date: 2022-04-29 +--- + +# Quick Setup + +Set the following configuration items in the **postgresql.conf** file: + +```bash +wal_level = logical +``` + +For a basic setup, retain the default values for the other necessary configuration items. + +You need to adjust the **pg_hba.conf** file to allow replication (the value depends on the actual network configuration and the user used for connection). + +```bash +host all repuser 0.0.0.0/0 sha256 +``` + +In the publisher database: + +```sql +CREATE PUBLICATION mypub FOR TABLE users, departments; +``` + +In the subscriber database: + +```sql +CREATE SUBSCRIPTION mysub CONNECTION 'dbname=foo host=bar user=repuser' PUBLICATION mypub; +``` + +The above statements start the replication process, replicating incremental changes to those tables. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/restrictions.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/restrictions.md new file mode 100644 index 0000000000000000000000000000000000000000..1c08c31a80fb2ca7b249b1b45b21682a8fffb797 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/restrictions.md @@ -0,0 +1,17 @@ +--- +title: Restrictions +summary: Restrictions +author: Guo Huan +date: 2022-04-29 +--- + +# Restrictions + +Publication-subscription is implemented based on logical replication and inherits all restrictions of logical replication. In addition, publication-subscription has the following additional restrictions or missing functions. + +- Database schemas and DDL commands are not replicated. Initial schemas can be manually copied by using **pg_dump –schema-only**. Subsequent schema changes need to be manually synchronized. +- Sequence data is not replicated. The data in serial or identifier columns backed by the sequence in the background will be replicated as part of the table, but the sequence itself will still display the start value on the subscriber. If the subscriber is used as a read-only database, this is usually not a problem. However, if some kind of switchover or failover to the subscriber database is intended, the sequence needs to be updated to the latest value, either by copying the current data from the publisher (perhaps using **pg_dump**) or by determining a sufficiently large value from the tables themselves. +- Only tables, including partitioned tables, can be replicated. Attempts to replicate other types of relations, such as views, materialized views, or foreign tables, will result in errors. +- Multiple subscriptions in the same database cannot subscribe to the same publication (that is, the same published table). Otherwise, duplicate data or primary key conflicts may occur. +- If a published table contains data types that do not support B-tree or hash indexes (such as the geography types), the table must have a primary key so that UPDATE and DELETE operations can be successfully replicated to the subscription side. Otherwise, the replication will fail, and the message “FATAL: could not identify an equality operator for type xx” will be displayed on the subscription side. +- When gs_probackup is used to back up the publisher, the original publication-subscription relationship cannot be established because no logical replication slot exists after the backup and restoration. gs_probackup does not support the backup of the logical replication slot. You are advised to use gs_basebackup to back up the publisher. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/security.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/security.md new file mode 100644 index 0000000000000000000000000000000000000000..675aea851665fd722a6fb6de4ae42656b88b6798 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/security.md @@ -0,0 +1,20 @@ +--- +title: Security +summary: Security +author: Guo Huan +date: 2022-04-29 +--- + +# Security + +The role used for the replication connection must have the **REPLICATION** attribute (or be a user with the **SYSADMIN** permission). If the role lacks **SUPERUSER** and **BYPASSRLS**, the publisher's row security policies can be executed. The access permission of the role must be configured in the **pg_hba.conf** file and the role must have the **LOGIN** attribute. + +To create a publication, the user must have the **CREATE** permission on the database. + +To add tables to a publication, the user must have ownership of the table. To create a publication that automatically publishes all tables, the user must be a user with **SYSADMIN** permission. + +To create a subscription, the user must be a user with the **SYSADMIN** permission. + +The subscription apply process will run in the local database with the privileges of a user with the **SYSADMIN** permission. + +Privileges are only checked once at the start of the replication connection. They are not re-checked when each change record is read from the publisher and when each change is applied. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/subscriptions.md b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/subscriptions.md new file mode 100644 index 0000000000000000000000000000000000000000..ed37a45dff13c6143e4e9a488f1b8e22542d3612 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/logical-replication/publication-subscription/subscriptions.md @@ -0,0 +1,28 @@ +--- +title: Subscriptions +summary: Subscriptions +author: Guo Huan +date: 2022-04-29 +--- + +# Subscriptions + +A subscription is the downstream side of logical replication. The node where a subscription is defined is called the subscriber. A subscription defines the connection to another database and the set of publications (one or more) that it wants to subscribe to. + +The subscriber database behaves in the same way as any other MogDB instance and can be used as a publisher for other databases by defining its own publications. + +A subscriber node can have multiple subscriptions if needed. You can define multiple subscriptions between a pair of publishers and subscribers, in which case you need to ensure that the published objects do not overlap. + +Each subscription will receive changes through a replication slot. Currently, initial data in pre-existing tables cannot be synchronized. + +If the current user is a user with the **SYSADMIN** permission, subscriptions are dumped by **pg_dump**. Otherwise, the subscriptions are skipped and a warning is written because users without the **SYSADMIN** permission cannot read all subscription information from the **pg_subscription** directory. + +You can use **CREATE SUBSCRIPTION** to add a subscription, **ALTER SUBSCRIPTION** to alter a subscription, and **DROP SUBSCRIPTION** to drop a subscription. + +When a subscription is dropped and recreated, the synchronized information is lost. This means that the data must be resynchronized. + +The schema definitions are not replicated, and the published tables must exist on the subscriber. Only regular tables can be replicated. For example, a view cannot be replicated. + +The tables are matched between the publisher and the subscriber using fully qualified table names. Replication to differently-named tables on the subscriber is not supported. + +Columns of a table are also matched by name. The order of columns in the subscribed table does not need to be the same as that in the published table. The data types of the columns do not need to be the same, as long as the text representation of the data can be converted to the target type. For example, you can replicate from a column of the integer type to a column of the bigint type. The target table can also have additional columns that do not exist in the published table. The additional columns will be filled with the default values specified in the definition of the target table. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/1-materialized-view-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/1-materialized-view-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..c5063b86a32975ba56ea224df710ec4e8050b221 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/1-materialized-view-overview.md @@ -0,0 +1,12 @@ +--- +title: Materialized View Overview +summary: Materialized View Overview +author: Guo Huan +date: 2021-05-21 +--- + +# Materialized View Overview + +A materialized view is a special physical table, which is relative to a common view. A common view is a virtual table and has many application limitations. Any query on a view is actually converted into a query on an SQL statement, and performance is not actually improved. The materialized view actually stores the results of the statements executed by the SQL statement, and is used to cache the results. + +Currently, the Ustore engine does not support creating and using materialized views. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/1-full-materialized-view-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/1-full-materialized-view-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..721d69a42595935535c885dd0a4e700b934aca3a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/1-full-materialized-view-overview.md @@ -0,0 +1,10 @@ +--- +title: Overview +summary: Overview +author: Liuxu +date: 2021-05-21 +--- + +# Overview + +Full materialized views can be fully refreshed only. The syntax for creating a full materialized view is similar to the CREATE TABLE AS syntax. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/2-full-materialized-view-usage.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/2-full-materialized-view-usage.md new file mode 100644 index 0000000000000000000000000000000000000000..b4a0ce512dec1b93b983bdfbc5f4b7eceb5f3f2e --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/2-full-materialized-view-usage.md @@ -0,0 +1,69 @@ +--- +title: Usage +summary: Usage +author: Guo Huan +date: 2021-05-21 +--- + +# Usage + +## Syntax + +- Create a full materialized view. + + ``` + CREATE MATERIALIZED VIEW [ view_name ] AS { query_block }; + ``` + +- Fullly refresh a materialized view. + + ``` + REFRESH MATERIALIZED VIEW [ view_name ]; + ``` + +- Delete a materialized view. + + ``` + DROP MATERIALIZED VIEW [ view_name ]; + ``` + +- Query a materialized view. + + ``` + SELECT * FROM [ view_name ]; + ``` + +## Examples + +``` +-- Prepare data. +MogDB=# CREATE TABLE t1(c1 int, c2 int); +MogDB=# INSERT INTO t1 VALUES(1, 1); +MogDB=# INSERT INTO t1 VALUES(2, 2); + +-- Create a full materialized view. +MogDB=# CREATE MATERIALIZED VIEW mv AS select count(*) from t1; + +-- Query the materialized view result. +MogDB=# SELECT * FROM mv; + count +------- + 2 +(1 row) + +-- Insert data into the base table in the materialized view. +MogDB=# INSERT INTO t1 VALUES(3, 3); + +-- Fully refresh a full materialized view. +MogDB=# REFRESH MATERIALIZED VIEW mv; + +-- Query the materialized view result. +MogDB=# SELECT * FROM mv; + count +------- + 3 +(1 row) + +-- Delete a materialized view. +MogDB=# DROP MATERIALIZED VIEW mv; +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/3-full-materialized-view-support-and-constraints.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/3-full-materialized-view-support-and-constraints.md new file mode 100644 index 0000000000000000000000000000000000000000..2a672cc12c3cc49ba5e5205fe351742e254358c4 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/2-full-materialized-view/3-full-materialized-view-support-and-constraints.md @@ -0,0 +1,22 @@ +--- +title: Support and Constraints +summary: Support and Constraints +author: Liuxu +date: 2021-05-21 +--- + +# Support and Constraints + +## Supported Scenarios + +- Supports the same query scope as the CREATE TABLE AS statement does. +- Supports index creation in full materialized views. +- Supports ANALYZE and EXPLAIN. + +## Unsupported Scenarios + +Materialized views cannot be added, deleted, or modified. They support only query statements. + +## Constraints + +A high-level lock is added to the base table during the process of refreshing or deleting a full materialized view. If a materialized view involves multiple tables, pay attention to the service logic to avoid deadlocks. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/1-incremental-materialized-view-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/1-incremental-materialized-view-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..8da9c3df584795fc5ebd719cbfb74bcdb7a67037 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/1-incremental-materialized-view-overview.md @@ -0,0 +1,10 @@ +--- +title: Overview +summary: Overview +author: Guo Huan +date: 2021-05-21 +--- + +# Overview + +Incremental materialized views can be incrementally refreshed. You need to manually execute statements to incrementally refresh materialized views in a period of time. The difference between the incremental and the full materialized views is that the incremental materialized view supports only a small number of scenarios. Currently, only base table scanning statements or UNION ALL can be used to create materialized views. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/2-incremental-materialized-view-usage.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/2-incremental-materialized-view-usage.md new file mode 100644 index 0000000000000000000000000000000000000000..eae768d415750cf5cdeaf211bc6e1e4e9e2f30e1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/2-incremental-materialized-view-usage.md @@ -0,0 +1,92 @@ +--- +title: Usage +summary: Usage +author: Guo Huan +date: 2021-05-21 +--- + +# Usage + +## Syntax + +- Create a incremental materialized view. + + ``` + CREATE INCREMENTAL MATERIALIZED VIEW [ view_name ] AS { query_block }; + ``` + +- Fully refresh a materialized view. + + ``` + REFRESH MATERIALIZED VIEW [ view_name ]; + ``` + +- Incrementally refresh a materialized view. + + ``` + REFRESH INCREMENTAL MATERIALIZED VIEW [ view_name ]; + ``` + +- Delete a materialized view. + + ``` + DROP MATERIALIZED VIEW [ view_name ]; + ``` + +- Query a materialized view. + + ``` + SELECT * FROM [ view_name ]; + ``` + +## Examples + +``` +-- Prepare data. +MogDB=# CREATE TABLE t1(c1 int, c2 int); +MogDB=# INSERT INTO t1 VALUES(1, 1); +MogDB=# INSERT INTO t1 VALUES(2, 2); + +-- Create an incremental materialized view. +MogDB=# CREATE INCREMENTAL MATERIALIZED VIEW mv AS SELECT * FROM t1; +CREATE MATERIALIZED VIEW + +-- Insert data. +MogDB=# INSERT INTO t1 VALUES(3, 3); +INSERT 0 1 + +-- Incrementally refresh a materialized view. +MogDB=# REFRESH INCREMENTAL MATERIALIZED VIEW mv; +REFRESH MATERIALIZED VIEW + +-- Query the materialized view result. +MogDB=# SELECT * FROM mv; + c1 | c2 +----+---- + 1 | 1 + 2 | 2 + 3 | 3 +(3 rows) + +-- Insert data. +MogDB=# INSERT INTO t1 VALUES(4, 4); +INSERT 0 1 + +-- Fullly refresh a materialized view. +MogDB=# REFRESH MATERIALIZED VIEW mv; +REFRESH MATERIALIZED VIEW + +-- Query the materialized view result. +MogDB=# select * from mv; + c1 | c2 +----+---- + 1 | 1 + 2 | 2 + 3 | 3 + 4 | 4 +(4 rows) + +-- Delete a materialized view. +MogDB=# DROP MATERIALIZED VIEW mv; +DROP MATERIALIZED VIEW +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/3-incremental-materialized-view-support-and-constraints.md b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/3-incremental-materialized-view-support-and-constraints.md new file mode 100644 index 0000000000000000000000000000000000000000..f50aa595994125457b59d12250f67296fb9dcca3 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/materialized-view/3-incremental-materialized-view/3-incremental-materialized-view-support-and-constraints.md @@ -0,0 +1,31 @@ +--- +title: Support and Constraints +summary: Support and Constraints +author: Guo Huan +date: 2021-05-21 +--- + +# Support and Constraints + +## Supported Scenarios + +- Supports statements for querying a single table. +- Supports UNION ALL for querying multiple single tables. +- Supports index creation in materialized views. +- Supports the Analyze operation in materialized views. + +## Unsupported Scenarios + +- Multi-table join plans and subquery plans are not supported in materialized views. +- Except for a few ALTER operations, most DDL operations cannot be performed on base tables in materialized views. +- Materialized views cannot be added, deleted, or modified. They support only query statements. +- The temporary table, hashbucket, unlog, or partitioned table cannot be used to create materialized views. +- Materialized views cannot be created in nested mode (that is, a materialized view cannot be created in another materialized view). +- The column-store tables are not supported. Only row-store tables are supported. +- Materialized views of the UNLOGGED type are not supported, and the WITH syntax is not supported. + +## Constraints + +If the materialized view definition is UNION ALL, each subquery needs to use a different base table. + +A high-level lock is added to the base table during the creation, full refresh, and deletion of an incremental materialized view. If the materialized view is defined as a UNION ALL, pay attention to the service logic to avoid deadlocks. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/benefits-of-partition-pruning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/benefits-of-partition-pruning.md new file mode 100644 index 0000000000000000000000000000000000000000..dc42b875dbe1657dcfdd1626140e14a0a45c93af --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/benefits-of-partition-pruning.md @@ -0,0 +1,66 @@ +--- +title: Benefits of Partition Pruning +summary: Benefits of Partition Pruning +author: Guo Huan +date: 2022-06-14 +--- + +# Benefits of Partition Pruning + +Partition pruning is a common performance optimization method in partitioned tables. Before scanning a partition table, you can check partition constraints and the definition of each partition to exclude partitions that do not need to be scanned in advance, which can greatly improve scan performance. In the query planning stage, if the partition constraint is a definite expression, the partition that does not need to be scanned can be cut off according to the partition constraint expression in the query planning stage. This partition pruning method is generally called static partition pruning. Static partition pruning result can be seen from the EXPLAIN VERBOSE output, as shown below. + +explain verbose select * from prune_tt01 where a<8; + +```sql +MogDB=# drop table if exists prune_tt01; +DROP TABLE +MogDB=# CREATE TABLE prune_tt01(a int, b int) +PARTITION BY RANGE(a) +( + PARTITION prune_tt01_p1 VALUES LESS THAN(5), + PARTITION prune_tt01_p2 VALUES LESS THAN(10), + PARTITION prune_tt01_p3 VALUES LESS THAN(15), + PARTITION prune_tt01_p4 VALUES LESS THAN(MAXVALUE) +); +CREATE TABLE +MogDB=# INSERT INTO prune_tt01 VALUES (generate_series(1, 20), generate_series(1,20)); +INSERT 0 20 +MogDB=# CREATE INDEX index_prune_tt01 ON prune_tt01 USING btree(a) LOCAL; +CREATE INDEX +MogDB=# explain verbose select * from prune_tt01 where a<8 ; + QUERY PLAN +---------------------------------------------------------------------------------------------------- + Partition Iterator (cost=13.80..27.75 rows=716 width=8) + Output: a, b + Iterations: 2 + Selected Partitions: 1..2 + -> Partitioned Bitmap Heap Scan on public.prune_tt01 (cost=13.80..27.75 rows=716 width=8) + Output: a, b + Recheck Cond: (prune_tt01.a < 8) + -> Partitioned Bitmap Index Scan on index_prune_tt01 (cost=0.00..13.62 rows=716 width=0) + Index Cond: (prune_tt01.a < 8) +(9 rows) +``` + +However, in many scenarios such as the prepare-execute execution mode and the scenario in which the partition constraint expression contains subqueries, the partition constraint expression is uncertain or contains unknown parameters in the query planning phase, and cannot be tailored in the query planning phase. Partitioning expression is determined by external parameters and the results of subqueries for pruning, and usually pruning in the execution phase is called dynamic partitioning pruning. Dynamic partition pruning allows you to see pruning information from the execution through Explain Verbose (selected partitions: PART) + +explain verbose select * from prune_tt01 where a < (select 8); + +```sql +MogDB=# explain verbose select * from prune_tt01 where a < (select 8); + QUERY PLAN +-------------------------------------------------------------------------------------- + Partition Iterator (cost=0.01..36.87 rows=716 width=8) + Output: prune_tt01.a, prune_tt01.b + Iterations: PART + Selected Partitions: PART + InitPlan 1 (returns $0) + -> Result (cost=0.00..0.01 rows=1 width=0) + Output: 8 + -> Partitioned Seq Scan on public.prune_tt01 (cost=0.00..36.86 rows=716 width=8) + Output: prune_tt01.a, prune_tt01.b + Filter: (prune_tt01.a < $0) +(10 rows) +``` + +MogDB 3.0 introduces dynamic pruning of partitioned tables, which greatly reduces the amount of data retrieved from disk and reduces processing time, thus improving query performance and optimizing resource utilization. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/dynamic-partition-pruning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/dynamic-partition-pruning.md new file mode 100644 index 0000000000000000000000000000000000000000..927a994f7f05347b899f853da80ffe0c3baf24ef --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/dynamic-partition-pruning.md @@ -0,0 +1,260 @@ +--- +title: Dynamic Partition Pruning +summary: Dynamic Partition Pruning +author: Guo Huan +date: 2022-06-14 +--- + +# Dynamic Partition Pruning + +Dynamic partition pruning supports the following three types: + +1. The prepare-execute (bind variable) execution mode and dynamic pruning expression are similar to static pruning. The only difference is that one side of the static pruning expression is Const. One side of the dynamic pruning expression is external parameters or an expression composed of external parameters. + + ```sql + prepare ps(int) as select * from prune_tt01 where a > $1; + explain verbose execute ps(12); + ``` + +2. A query statement contains subqueries, which are divided into two methods: pruning partitioned tables in the main query and pruning partitioned tables in the subquery: + + - The partition table in the main query is pruned. The subquery is generally an unrelated subquery. The subquery works as one side of the partition pruning expression in the main query, and the other side is the partitioning key. + + ```sql + select * from prune_tt01 where a > (select a from t2 limit 1); + ``` + + - The partition table in the subquery is pruned. The subquery is generally a correlation subquery, and the subquery does not support promotion. Note: The related subquery involves pruning of multiple partitioned table, the result of the last pruning is shown in Explain Analyze. + any subquery: + + ```sql + explain analyze select * from t2 where a > any(select a from prune_tt01 where prune_tt01.a = t2.a); + ``` + + exists subquery: + + ```sql + explain analyze select * from t2 where exists (select a from prune_tt01 where prune_tt01.a = t2.a limit 1); + ``` + +3. NestLoop parameterized query, that is, the query plan use the nestLoop+indexscan query method. The partition key of a partitioned table must be the join key of two tables and an index must be created on the partition key. + +The following is the dynamic partition pruning example. + +```sql +drop table if exists prune_tt01; +CREATE TABLE prune_tt01(a int, b int) +PARTITION BY RANGE(a) +( + PARTITION prune_tt01_p1 VALUES LESS THAN(5), + PARTITION prune_tt01_p2 VALUES LESS THAN(10), + PARTITION prune_tt01_p3 VALUES LESS THAN(15), + PARTITION prune_tt01_p4 VALUES LESS THAN(MAXVALUE) +); +INSERT INTO prune_tt01 VALUES (generate_series(1, 20), generate_series(1,20)); +CREATE INDEX index_prune_tt01 ON prune_tt01 USING btree(a) LOCAL; + +drop table if exists tt02; +create table tt02(a int, b int); +INSERT INTO tt02 VALUES (generate_series(1, 20), generate_series(1,20)); +``` + +**prepare-execute (bind variable) scenario** + +```sql +MogDB=# prepare ps(int) as select * from prune_tt01 where a > $1; +PREPARE +MogDB=# explain verbose execute ps(12); + QUERY PLAN +-------------------------------------------------------------------------------------- + Partition Iterator (cost=0.00..36.86 rows=716 width=8) + Output: a, b + Iterations: PART + Selected Partitions: PART + -> Partitioned Seq Scan on public.prune_tt01 (cost=0.00..36.86 rows=716 width=8) + Output: a, b + Filter: (prune_tt01.a > $1) +(7 rows) + +MogDB=# explain analyze execute ps(12); + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------- + Partition Iterator (cost=0.00..36.86 rows=716 width=8) (actual time=0.099..0.110 rows=8 loops=1) + Iterations: 2 + Selected Partitions: 3..4 + -> Partitioned Seq Scan on prune_tt01 (cost=0.00..36.86 rows=716 width=8) (actual time=0.031..0.034 rows=8 loops=2) + Filter: (a > $1) + Rows Removed by Filter: 3 + Total runtime: 0.218 ms +(7 rows) +``` + +**Subquery scenario** + +- Prune the partition table in the main query + +```sql +MogDB=# explain verbose select * from prune_tt01 where a > (select a from t2 where a > 12 limit 1); + QUERY PLAN +-------------------------------------------------------------------------------------- + Partition Iterator (cost=0.04..36.90 rows=716 width=8) + Output: prune_tt01.a, prune_tt01.b + Iterations: PART + Selected Partitions: PART + InitPlan 1 (returns $0) + -> Limit (cost=0.00..0.04 rows=1 width=4) + Output: t2.a + -> Seq Scan on public.t2 (cost=0.00..1.75 rows=49 width=4) + Output: t2.a + Filter: (t2.a > 12) + -> Partitioned Seq Scan on public.prune_tt01 (cost=0.00..36.86 rows=716 width=8) + Output: prune_tt01.a, prune_tt01.b + Filter: (prune_tt01.a > $0) +(13 rows) + +MogDB=# explain analyze select * from prune_tt01 where a > (select a from t2 where a > 12 limit 1); + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------- + Partition Iterator (cost=0.04..36.90 rows=716 width=8) (actual time=0.172..0.180 rows=7 loops=1) + Iterations: 2 + Selected Partitions: 3..4 + InitPlan 1 (returns $0) + -> Limit (cost=0.00..0.04 rows=1 width=4) (actual time=0.093..0.093 rows=1 loops=1) + -> Seq Scan on t2 (cost=0.00..1.75 rows=49 width=4) (actual time=0.091..0.091 rows=1 loops=1) + Filter: (a > 12) + Rows Removed by Filter: 12 + -> Partitioned Seq Scan on prune_tt01 (cost=0.00..36.86 rows=716 width=8) (actual time=0.020..0.020 rows=7 loops=2) + Filter: (a > $0) + Rows Removed by Filter: 4 + Total runtime: 0.301 ms +(12 rows) +``` + +- Prune the partition table in the subquery: + + any subquery: + +```sql +MogDB=# explain verbose select * from t2 where a > any(select a from prune_tt01 where prune_tt01.a = t2.a); + QUERY PLAN +---------------------------------------------------------------------------------------------------------- + Seq Scan on public.t2 (cost=0.00..582.83 rows=30 width=8) + Output: t2.a, t2.b + Filter: (SubPlan 1) + SubPlan 1 + -> Partition Iterator (cost=4.34..15.01 rows=11 width=4) + Output: prune_tt01.a + Iterations: PART + Selected Partitions: PART + -> Partitioned Bitmap Heap Scan on public.prune_tt01 (cost=4.34..15.01 rows=11 width=4) + Output: prune_tt01.a + Recheck Cond: (prune_tt01.a = t2.a) + -> Partitioned Bitmap Index Scan on index_prune_tt01 (cost=0.00..4.33 rows=11 width=0) + Index Cond: (prune_tt01.a = t2.a) +(13 rows) + +MogDB=# explain analyze select * from t2 where a > any(select a from prune_tt01 where prune_tt01.a = t2.a); + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------ + Seq Scan on t2 (cost=0.00..582.83 rows=30 width=8) (actual time=2.130..2.130 rows=0 loops=1) + Filter: (SubPlan 1) + Rows Removed by Filter: 60 + SubPlan 1 + -> Partition Iterator (cost=4.34..15.01 rows=11 width=4) (actual time=1.860..1.877 rows=20 loops=60) + Iterations: 1 + Selected Partitions: 4 + -> Partitioned Bitmap Heap Scan on prune_tt01 (cost=4.34..15.01 rows=11 width=4) (actual time=0.566..0.576 rows=20 loops=60) + Recheck Cond: (a = t2.a) + Heap Blocks: exact=20 + -> Partitioned Bitmap Index Scan on index_prune_tt01 (cost=0.00..4.33 rows=11 width=0) (actual time=0.482..0.482 rows=20 loops=60) + Index Cond: (a = t2.a) + Total runtime: 2.600 ms +(13 rows) +``` + + exists subquery: + +```sql +MogDB=# explain verbose select * from t2 where exists (select a from prune_tt01 where prune_tt01.a = t2.a limit 1); + QUERY PLAN +---------------------------------------------------------------------------------------------------------------- + Seq Scan on public.t2 (cost=0.00..319.92 rows=30 width=8) + Output: t2.a, t2.b + Filter: (SubPlan 1) + SubPlan 1 + -> Limit (cost=4.34..5.31 rows=1 width=4) + Output: prune_tt01.a + -> Partition Iterator (cost=4.34..15.01 rows=11 width=4) + Output: prune_tt01.a + Iterations: PART + Selected Partitions: PART + -> Partitioned Bitmap Heap Scan on public.prune_tt01 (cost=4.34..15.01 rows=11 width=4) + Output: prune_tt01.a + Recheck Cond: (prune_tt01.a = t2.a) + -> Partitioned Bitmap Index Scan on index_prune_tt01 (cost=0.00..4.33 rows=11 width=0) + Index Cond: (prune_tt01.a = t2.a) +(15 rows) + +MogDB=# explain analyze select * from t2 where exists (select a from prune_tt01 where prune_tt01.a = t2.a limit 1); + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------ + Seq Scan on t2 (cost=0.00..319.92 rows=30 width=8) (actual time=0.058..0.875 rows=20 loops=1) + Filter: (SubPlan 1) + Rows Removed by Filter: 40 + SubPlan 1 + -> Limit (cost=4.34..5.31 rows=1 width=4) (actual time=0.826..0.826 rows=20 loops=60) + -> Partition Iterator (cost=4.34..15.01 rows=11 width=4) (actual time=0.789..0.789 rows=20 loops=60) + Iterations: 1 + Selected Partitions: 4 + -> Partitioned Bitmap Heap Scan on prune_tt01 (cost=4.34..15.01 rows=11 width=4) (actual time=0.162..0.162 rows=20 loops=60) + Recheck Cond: (a = t2.a) + Heap Blocks: exact=20 + -> Partitioned Bitmap Index Scan on index_prune_tt01 (cost=0.00..4.33 rows=11 width=0) (actual time=0.123..0.123 rows=20 loops=60) + Index Cond: (a = t2.a) + Total runtime: 1.151 ms +(14 rows) +``` + +**nestloop scenario:** + +The following parameters are set only to simulate the use of nestloop in SQL statements (normally the optimizer will choose the best access path based on the amount of data in the table) + +```sql +MogDB=# SET enable_material = OFF; +SET +MogDB=# SET enable_mergejoin = OFF; +SET +MogDB=# SET enable_hashjoin = OFF; +SET +``` + +```sql +MogDB=# explain verbose select * from prune_tt01 inner join tt02 on prune_tt01.a = tt02.a; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------- + Nested Loop (cost=0.00..1501.91 rows=23091 width=16) + Output: prune_tt01.a, prune_tt01.b, tt02.a, tt02.b + -> Seq Scan on public.tt02 (cost=0.00..31.49 rows=2149 width=8) + Output: tt02.a, tt02.b + -> Partition Iterator (cost=0.00..0.57 rows=11 width=8) + Output: prune_tt01.a, prune_tt01.b + Iterations: PART + Selected Partitions: PART + -> Partitioned Index Scan using index_prune_tt01 on public.prune_tt01 (cost=0.00..0.57 rows=11 width=8) + Output: prune_tt01.a, prune_tt01.b + Index Cond: (prune_tt01.a = tt02.a) +(11 rows) + +MogDB=# explain analyze select * from prune_tt01 inner join tt02 on prune_tt01.a = tt02.a; + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------- + Nested Loop (cost=0.00..1501.91 rows=23091 width=16) (actual time=0.078..0.535 rows=20 loops=1) + -> Seq Scan on tt02 (cost=0.00..31.49 rows=2149 width=8) (actual time=0.023..0.030 rows=20 loops=1) + -> Partition Iterator (cost=0.00..0.57 rows=11 width=8) (actual time=0.441..0.462 rows=20 loops=20) + Iterations: 1 + Selected Partitions: 4 + -> Partitioned Index Scan using index_prune_tt01 on prune_tt01 (cost=0.00..0.57 rows=11 width=8) (actual time=0.146..0.158 rows=20 loops=20) + Index Cond: (a = tt02.a) + Total runtime: 0.770 ms +(8 rows) +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/how-to-identify-whether-partition-pruning-has-been-used.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/how-to-identify-whether-partition-pruning-has-been-used.md new file mode 100644 index 0000000000000000000000000000000000000000..41dc28d2165defbd40a9d69ca12fa63bbf9585d7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/how-to-identify-whether-partition-pruning-has-been-used.md @@ -0,0 +1,12 @@ +--- +title: How to Judge that Parition Pruning Is Used +summary: How to Judge that Parition Pruning Is Used +author: Guo Huan +date: 2022-06-22 +--- + +### How to Judge that Parition Pruning Is Used + +Whether MogDB uses partition pruning is reflected from the execution plan of SQLs. You can check it using EXPLAIN VERBOSE or EXPLAIN ANALYZE. + +The partition pruning information is reflected in the execution plan column Iterations and Selected Partitions. \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/information-that-can-be-used-for-partition-pruning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/information-that-can-be-used-for-partition-pruning.md new file mode 100644 index 0000000000000000000000000000000000000000..792d42e40708e4a187affd5508e8406805113085 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/information-that-can-be-used-for-partition-pruning.md @@ -0,0 +1,33 @@ +--- +title: Information that Can Be Pruned +summary: Information that Can Be Pruned +author: Guo Huan +date: 2022-06-14 +--- + +# Information that Can Be Pruned + +Partition pruning can be performed on partitioned columns. + +MogDB prunes partitions when you use range, `LIKE`, equality, and `IN` list predicates on range or list partition columns, and equality and `IN` list predicates on hash partition columns. + +In example 1, table prune_tt01 is partitioned by range on column A. + +MogDB uses predicates on partition columns to perform partition pruning, as shown below. + +**Example 1 Createing a table with partition pruning** + +```sql +CREATE TABLE prune_tt01(a int, b int) +PARTITION BY RANGE(a) +( + PARTITION prune_tt01_p1 VALUES LESS THAN(5), + PARTITION prune_tt01_p2 VALUES LESS THAN(10), + PARTITION prune_tt01_p3 VALUES LESS THAN(15), + PARTITION prune_tt01_p4 VALUES LESS THAN(MAXVALUE) +); +INSERT INTO prune_tt01 VALUES (generate_series(1, 20), generate_series(1,20)); +CREATE INDEX index_prune_tt01 ON prune_tt01 USING btree(a) LOCAL; + +select * from prune_tt01 where a > 8 ; +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/static-partition-pruning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/static-partition-pruning.md new file mode 100644 index 0000000000000000000000000000000000000000..393eb564bf857fada1a63b4b951e0780aadc0ea8 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/partition-pruning/static-partition-pruning.md @@ -0,0 +1,60 @@ +--- +title: Static Partition Pruning +summary: Static Partition Pruning +author: Guo Huan +date: 2022-06-14 +--- + +# Static Partition Pruning + +MogDB determines when static pruning is used primarily based on static predicates. + +If MogDB can identify which consecutive set of partitions was accessed at parsing time, the **Selected Partitions** column in the execution plan shows the start and end values of the partition being accessed. For any other partition pruning case (including dynamic partition pruning), MogDB displays **Selected Partitions**. + +Static partition pruning is available via the [EXPLAIN VERBOSE](../../../reference-guide/sql-syntax/EXPLAIN.md) statement to view the execution plan. In the following example, there are four partitions in the prune_tt01 table. The query plan shows that partitions 3 and 4 are scanned by SeqScan. Therefore, it is judged that partitions 1 and 2 have been cut out. + +```sql +MogDB=# \d+ prune_tt01 + Table "public.prune_tt01" + Column | Type | Modifiers | Storage | Stats target | Description +--------+---------+-----------+---------+--------------+------------- + a | integer | | plain | | + b | integer | | plain | | +Indexes: + "index_prune_tt01" btree (a) LOCAL TABLESPACE pg_default +Partition By RANGE(a) +Number of partitions: 4 (View pg_partition to check each partition range.) +Has OIDs: no +Options: orientation=row, compression=no +``` + +```sql +MogDB=# explain verbose select * from prune_tt01 where a>12; + QUERY PLAN +---------------------------------------------------------------------------------------------------- + Partition Iterator (cost=13.80..27.75 rows=716 width=8) + Output: a, b + Iterations: 2 + Selected Partitions: 3..4 + -> Partitioned Bitmap Heap Scan on public.prune_tt01 (cost=13.80..27.75 rows=716 width=8) + Output: a, b + Recheck Cond: (prune_tt01.a > 12) + -> Partitioned Bitmap Index Scan on index_prune_tt01 (cost=0.00..13.62 rows=716 width=0) + Index Cond: (prune_tt01.a > 12) +(9 rows) +``` + +
+ +The detailed information for static partition pruning is shown in the following table. + +**Table 1** Detailed information for static partition pruning + +| No. | Constraint Name | Constraint Range | +| ---- | ------------------------------ | ------------------------------------------------------------ | +| 1 | Partition table type | Range partition, list partition, and hash partition | +| 2 | Partition expression type | - A partition constraint can be a single expression, such as `a >12`
- A partition constraint can be a BOOL expression, such as `a > 2 and a < 12`
- A partition constraint can be an array, such as `a in (2, 3)`
- A partition constraint can be a constant expression, such as `1 = 1`
- A partition constraint can be the `Is (NOT)NULL` expression, such as `a IS NULL` | +| 3 | Partition expression operator | - Range partition table supports five operators, including `=`, `>`, `>=`, `<`, and `<=`.
- List partition and hash partition supports only the `=` operator. | +| 4 | Partition expression parameter | One side is partition key, and the other side is constance, such as `a > 12`. | +| 5 | Second-level partition | Combination of range, list, and hash partition tables, such as Range-List partition table. | +| 6 | Partition pruning result | Explain verbose shows the pruned partition list. | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-hash-partitioning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-hash-partitioning.md new file mode 100644 index 0000000000000000000000000000000000000000..d94969417a1fa307348978ccae8af7a97b82e111 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-hash-partitioning.md @@ -0,0 +1,41 @@ +--- +title: When to Use Hash Partitioning +summary: When to Use Hash Partitioning +author: Guo Huan +date: 2022-06-14 +--- + +# When to Use Hash Partitioning + +Hash partitioning is useful for randomly distributing data between partitions based on hash algorithms rather than grouping similar data. + +In some cases, it is not obvious which partition the partition data should be located in, although the partition key can be determined. In some cases, you don't want to group similar data like range partitioning, but want the distribution of data to be inconsistent with its business or logical view. Hash partitioning puts a row into a partition based on the result of passing the keyword to the hash algorithm. + +With this approach, data is distributed randomly across partitions rather than grouped. This is a good approach for some data, but it may not be appropriate for managing historical data. However, hash partitioning shares some of the same performance characteristics as range partitioning. For example, partitioning pruning is limited to equality predicates. You can also use partitioned joins, parallel index access, and parallel DML. + +The advantage of hash partitioning is that the distribution of data is almost random, so the distribution is relatively uniform, which can avoid hot spot problems to some extent. + +The disadvantages of hash partitioning are as follows: + +- You cannot perform a range query without storing additional data. +- Since each node requires a corresponding hash value when nodes are added or deleted, the addition of nodes requires modification of the hash function, which causes much of the existing data to be remapped, causing massive data movement. In the meantime, the system may not continue to work. + +Example 1 Four hash partitions are created for table **Sales_hash** using the s_productid column as the partitioning keyword. Parallel joins with the products table can take advantage of partial or full partitioned joins. At this time, partitioning pruning is conducive to the query that only accesses the sales data of a single product or part of the product. + +If you specify a hash partition number instead of an explicit partition name, MogDB automatically generates an internal name for the partition. + +**Example 1 Creating a hash-partitioned table** + +```sql +CREATE TABLE sales_hash + (s_productid NUMBER, + s_saledate DATE, + s_custid NUMBER, + s_totalprice NUMBER) +PARTITION BY HASH(s_productid) +( PARTITION p1 , + PARTITION p2, + PARTITION p3 , + PARTITION p4 +); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-list-partitioning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-list-partitioning.md new file mode 100644 index 0000000000000000000000000000000000000000..2f786730a21378aba1f791aef843215cad6252eb --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-list-partitioning.md @@ -0,0 +1,33 @@ +--- +title: When to Use List Partitioning +summary: When to Use List Partitioning +author: Guo Huan +date: 2022-06-14 +--- + +# When to Use List Partitions + +List partitioning explicitly map rows to partitions based on discrete values. + +In example 1, account managers who analyze accounts by region can make effective use of partition pruning. + +**Example 1 Creating a list-partitioned table** + +```sql +CREATE TABLE accounts +( id NUMBER +, account_number NUMBER +, customer_id NUMBER +, branch_id NUMBER +, region VARCHAR(2) +, status VARCHAR2(1) +) +PARTITION BY LIST (region) +( PARTITION p_northwest VALUES ('OR', 'WA') +, PARTITION p_southwest VALUES ('AZ', 'UT', 'NM') +, PARTITION p_northeast VALUES ('NY', 'VM', 'NJ') +, PARTITION p_southeast VALUES ('FL', 'GA') +, PARTITION p_northcentral VALUES ('SD', 'WI') +, PARTITION p_southcentral VALUES ('OK', 'TX') +); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-range-partitioning.md b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-range-partitioning.md new file mode 100644 index 0000000000000000000000000000000000000000..61419a4e4d1121e570ed6b7e74c29f9bee6ee0e0 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/partition-management/recommendations-for-choosing-a-partitioning-strategy/when-to-use-range-partitioning.md @@ -0,0 +1,36 @@ +--- +title: When to Use Range Partitioning +summary: When to Use Range Partitioning +author: Guo Huan +date: 2022-06-14 +--- + +# When to Use Range Partitioning + +Range partitioning is useful for organizing similar data, especially date and time data. + +Range partitioning is a convenient way to partition historical data. The boundaries of a range partition define the order of partitions in a table or index + +Most SQL statements that access range partitions focus on time ranges. For example, SQL statements select data from a specific time period. If each partition represents a month of data, the amount of data scanned is reduced to a fraction of the total. This optimization method is called partition pruning. + +Range partitions are also useful when you need to periodically load new data and clean up old data, because range partitions are easy to add or remove. For example, the system typically keeps a scrolling data window that keeps data online for the last 36 months. Range partitioning simplifies this process. + +In summary, you can consider using range partitioning when: + +- It is often necessary to perform range predicate scans on easily partitioned columns in large tables. +- You want to maintain a scrolling data window. +- Large tables cannot be managed within a specified time range, such as backup and restoration, but can be divided into smaller logical blocks based on partition range columns. + +**Example 1 Creating a range-partitioned table** + +```sql +CREATE TABLE employees ( + id INT NOT NULL, + fname VARCHAR(30), + lname VARCHAR(30), + hired DATE NOT NULL DEFAULT '1970-01-01', + separated DATE DEFAULT '9999-12-31', + job_code INT, + store_id INT NOT NULL +); +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-1-plpgsql-overview.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-1-plpgsql-overview.md new file mode 100644 index 0000000000000000000000000000000000000000..2e19bda3747159d690ade3a33a22ef567d9a0cb7 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-1-plpgsql-overview.md @@ -0,0 +1,24 @@ +--- +title: Overview of PL/pgSQL Functions +summary: Overview of PL/pgSQL Functions +author: Guo Huan +date: 2021-11-10 +--- + +# Overview of PL/pgSQL Functions + +PL/pgSQL is a loadable procedural language. + +The functions created using PL/pgSQL can be used in any place where you can use built-in functions. For example, you can create calculation functions with complex conditions and use them to define operators or use them for index expressions. + +SQL is used by most databases as a query language. It is portable and easy to learn. Each SQL statement must be executed independently by a database server. + +In this case, when a client application sends a query to the server, it must wait for it to be processed, receive and process the results, and then perform some calculation before sending more queries to the server. If the client and server are not on the same machine, all these operations will cause inter-process communication and increase network loads. + +PL/pgSQL enables a whole computing part and a series of queries to be grouped inside a database server. This makes procedural language available and SQL easier to use. In addition, the client/server communication cost is reduced. + +- Extra round-trip communication between clients and servers is eliminated. +- Intermediate results that are not required by clients do not need to be sorted or transmitted between the clients and servers. +- Parsing can be skipped in multiple rounds of queries. + +PL/pgSQL can use all data types, operators, and functions in SQL. There are some common functions, such as gs_extend_library. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-10-other-statements.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-10-other-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..cc2a571d77a39bdb9c8ab711691347c38658d69b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-10-other-statements.md @@ -0,0 +1,20 @@ +--- +title: Other Statements +summary: Other Statements +author: Guo Huan +date: 2021-03-04 +--- + +# Other Statements + +## Lock Operations + +MogDB provides multiple lock modes to control concurrent accesses to table data. These modes are used when Multi-Version Concurrency Control (MVCC) cannot give expected behaviors. Alike, most MogDB commands automatically apply appropriate locks to ensure that called tables are not deleted or modified in an incompatible manner during command execution. For example, when concurrent operations exist, **ALTER TABLE** cannot be executed on the same table. + +## Cursor Operations + +MogDB provides cursors as a data buffer for users to store execution results of SQL statements. Each cursor region has a name. Users can use SQL statements to obtain records one by one from cursors and grant the records to master variables, then being processed further by host languages. + +Cursor operations include cursor definition, open, fetch, and close operations. + +For the complete example of cursor operations, see [Explicit Cursor](1-11-cursors.md#explicit-cursor). diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-11-cursors.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-11-cursors.md new file mode 100644 index 0000000000000000000000000000000000000000..04c8397e067e1863b0e0ee940b7c367dd7132a85 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-11-cursors.md @@ -0,0 +1,180 @@ +--- +title: Cursors +summary: Cursors +author: Guo Huan +date: 2021-03-04 +--- + +# Cursors + +## Overview + +To process SQL statements, the stored procedure process assigns a memory segment to store context association. Cursors are handles or pointers pointing to context regions. With cursors, stored procedures can control alterations in context regions. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> If JDBC is used to call a stored procedure whose returned value is a cursor, the returned cursor cannot be used. + +Cursors are classified into explicit cursors and implicit cursors. [Table 1](#Table 1) shows the usage conditions of explicit and implicit cursors for different SQL statements. + +**Table 1** Cursor usage conditions + +| SQL Statement | Cursor | +| :---------------------------------------- | :------------------- | +| Non-query statements | Implicit | +| Query statements with single-line results | Implicit or explicit | +| Query statements with multi-line results | Explicit | + +## Explicit Cursor + +An explicit cursor is used to process query statements, particularly when query results are multiple records. + +**Procedure** + +An explicit cursor performs the following six PL/SQL steps to process query statements: + +1. **Define a static cursor:** Define a cursor name and its corresponding **SELECT** statement. + + [Figure 1](#static_cursor_define) shows the syntax diagram for defining a static cursor. + + **Figure 1** static_cursor_define::= + + ![static_cursor_define](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-1.jpg) + + Parameter description: + + - **cursor_name**: defines a cursor name. + + - **parameter**: specifies cursor parameters. Only input parameters are allowed in the following format: + + ``` + parameter_name datatype + ``` + + - **select_statement**: specifies a query statement. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > The system automatically determines whether the cursor can be used for backward fetches based on the execution plan. + + **Define a dynamic cursor:** Define a **ref** cursor, which means that the cursor can be opened dynamically by a set of static SQL statements. First define the type of the **ref** cursor first and then the cursor variable of this cursor type. Dynamically bind a **SELECT** statement through **OPEN FOR** when the cursor is opened. + + [Figure 2](#cursor_typename) and [Figure 3](#dynamic_cursor_define) show the syntax diagrams for defining a dynamic cursor. + + **Figure 2** cursor_typename::= + + ![cursor_typename](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-2.png) + + **Figure 3** dynamic_cursor_define::= + + ![dynamic_cursor_define](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-3.png) + +2. **Open the static cursor:** Execute the **SELECT** statement corresponding to the cursor. The query result is placed in the workspace and the pointer directs to the head of the workspace to identify the cursor result set. If the cursor query statement carries the **FOR UPDATE** option, the **OPEN** statement locks the data row corresponding to the cursor result set in the database table. + + [Figure 4](#open_static_cursor) shows the syntax diagram for opening a static cursor. + + **Figure 4** open_static_cursor::= + + ![open_static_cursor](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-4.png) + + **Open the dynamic cursor:** Use the **OPEN FOR** statement to open the dynamic cursor and the SQL statement is dynamically bound. + + [Figure 5](#open_dynamic_cursor) shows the syntax diagrams for opening a dynamic cursor. + + **Figure 5** open_dynamic_cursor::= + + ![open_dynamic_cursor](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-5.png) + + A PL/SQL program cannot use the OPEN statement to repeatedly open a cursor. + +3. **Fetch cursor data**: Retrieve data rows in the result set and place them in specified output variables. + + [Figure 6](#fetch_cursor) shows the syntax diagrams for fetching cursor data. + + **Figure 6** fetch_cursor::= + + ![fetch_cursor](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-6.png) + +4. Process the record. + +5. Continue to process until the active set has no record. + +6. **Close the cursor**: When fetching and finishing the data in the cursor result set, close the cursor immediately to release system resources used by the cursor and invalidate the workspace of the cursor so that the **FETCH** statement cannot be used to fetch data any more. A closed cursor can be reopened by an OPEN statement. + + [Figure 7](#close_cursor) shows the syntax diagram for closing a cursor. + + **Figure 7** close_cursor::= + + ![close_cursor](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-7.jpg) + +**Attribute** + +Cursor attributes are used to control program procedures or know program status. When a DML statement is executed, the PL/SQL opens a built-in cursor and processes its result. A cursor is a memory segment for maintaining query results. It is opened when a DML statement is executed and closed when the execution is finished. An explicit cursor has the following attributes: + +- **%FOUND** attribute: returns **TRUE** if the last fetch returns a row. +- **%NOTFOUND** attribute: works opposite to the **%FOUND** attribute. +- **%ISOPEN** attribute: returns **TRUE** if the cursor has been opened. +- **%ROWCOUNT** attribute: returns the number of records fetched from the cursor. + +## Implicit Cursor + +Implicit cursors are automatically set by the system for non-query statements such as modify or delete operations, along with their workspace. Implicit cursors are named **SQL**, which is defined by the system. + +**Overview** + +Implicit cursor operations, such as definition, open, value-grant, and close operations, are automatically performed by the system and do not need users to process. Users can use only attributes related to implicit cursors to complete operations. In workspace of implicit cursors, the data of the latest SQL statement is stored and is not related to explicit cursors defined by users. + +Format call:**SQL%** + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - **INSERT**, **UPDATE**, **DELETE**, and **SELECT** statements do not need defined cursors. +> - In O-compatible mode, if the GUC parameter **behavior_compat_options** is set to **compat_cursor**, implicit cursors are valid across stored procedures. + +**Attributes** + +An implicit cursor has the following attributes: + +- **SQL%FOUND**: Boolean attribute, which returns **TRUE** if the last fetch returns a row. +- **SQL%NOTFOUND**: Boolean attribute, which works opposite to the **SQL%FOUND** attribute. +- **SQL%ROWCOUNT**: numeric attribute, which returns the number of records fetched from the cursor. +- **SQL%ISOPEN**: Boolean attribute, whose value is always **FALSE**. Close implicit cursors immediately after an SQL statement is run. + +**Examples** + +```sql +-- Delete all employees in a department from the hr.staffs table. If the department has no employees, delete the department from the hr.sections table. +CREATE OR REPLACE PROCEDURE proc_cursor3() +AS + DECLARE + V_DEPTNO NUMBER(4) := 100; + BEGIN + DELETE FROM hr.staffs WHERE section_ID = V_DEPTNO; + -- Proceed based on cursor status. + IF SQL%NOTFOUND THEN + DELETE FROM hr.sections WHERE section_ID = V_DEPTNO; + END IF; + END; +/ + +CALL proc_cursor3(); + +-- Delete the stored procedure and the temporary table. +DROP PROCEDURE proc_cursor3; +``` + +## Cursor Loop + +Use of cursors in WHILE and LOOP statements is called a cursor loop. Generally, OPEN, FETCH, and CLOSE statements are involved in this kind of loop. The following describes a loop that simplifies a cursor loop without the need for these operations. This kind of loop is applicable to a static cursor loop, without executing four steps about a static cursor. + +**Syntax** + +[Figure 8](#FOR_AS_loop) shows the syntax diagram of the **FOR AS** loop. + +**Figure 8** FOR_AS_loop::= + +![for_as_loop](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/cursors-8.png) + +**Precautions** + +- The **UPDATE** operation for the queried table is not allowed in the loop statement. +- The variable loop_name is automatically defined and is valid only in this loop. Its type is the same as that in the select_statement query result. The value of **loop_name** is the query result of **select_statement**. +- The **%FOUND**, **%NOTFOUND**, and **%ROWCOUNT** attributes access the same internal variable in MogDB. Transactions and the anonymous block do not support multiple cursor accesses at the same time. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-12-retry-management.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-12-retry-management.md new file mode 100644 index 0000000000000000000000000000000000000000..7800bd1ddba0c4b6350d5f488e158227fc97d72b --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-12-retry-management.md @@ -0,0 +1,26 @@ +--- +title: Retry Management +summary: Retry Management +author: Guo Huan +date: 2021-03-04 +--- + +# Retry Management + +Retry is a process in which the database executes a SQL statement or stored procedure (including anonymous block) again in the case of execution failure, improving the execution success rate and user experience. The database checks the error code and retry configuration to determine whether to retry. + +- If the execution fails, the system rolls back the executed statements and executes the stored procedure again. + + Example: + + ```sql + MogDB=# CREATE OR REPLACE PROCEDURE retry_basic ( IN x INT) + AS + BEGIN + INSERT INTO t1 (a) VALUES (x); + INSERT INTO t1 (a) VALUES (x+1); + END; + / + + MogDB=# CALL retry_basic(1); + ``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-13-debugging.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-13-debugging.md new file mode 100644 index 0000000000000000000000000000000000000000..54568dc7f309abda9b764aed391c7fd2779be638 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-13-debugging.md @@ -0,0 +1,173 @@ +--- +title: Debugging +summary: Debugging +author: Guo Huan +date: 2021-03-04 +--- + +# Debugging + +## Syntax + +### RAISE语法 + +RAISE has the following five syntax formats: + +**Figure 1** raise_format::= + +![raise_format](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/debugging-1.png) + +**Figure 2** raise_condition::= + +![raise_condition](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/debugging-2.png) + +**Figure 3** raise_sqlstate::= + +![raise_sqlstate](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/debugging-3.png) + +**Figure 4** raise_option::= + +![raise_option](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/debugging-4.png) + +**Figure 5** raise::= + +![raise](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/debugging-5.png) + +**Parameter description**: + +- The level option is used to specify the error level, that is, **DEBUG**, **LOG**, **INFO**, **NOTICE**, **WARNING**, or **EXCEPTION** (default). **EXCEPTION** throws an error that normally terminates the current transaction and the others only generate information at their levels. The [log_min_messages](../../reference-guide/guc-parameters/10-error-reporting-and-logging/2-logging-time.md#log_min_messages) and [client_min_messages](../../reference-guide/guc-parameters/10-error-reporting-and-logging/2-logging-time.md#client_min_messages) parameters control whether the error messages of specific levels are reported to the client and are written to the server log. + +- **format**: specifies the error message text to be reported, a format string. The format string can be appended with an expression for insertion to the message text. In a format string, **%** is replaced by the parameter value attached to format and **%%** is used to print **%**. For example: + + ``` + --v_job_id replaces % in the string. + RAISE NOTICE 'Calling cs_create_job(%)',v_job_id; + ``` + +- **option = expression**: inserts additional information to an error report. The keyword option can be **MESSAGE**, **DETAIL**, **HINT**, or **ERRCODE**, and each expression can be any string. + + - **MESSAGE**: specifies the error message text. This option cannot be used in a **RAISE** statement that contains a format character string in front of **USING**. + - **DETAIL**: specifies detailed information of an error. + - **HINT**: prints hint information. + - **ERRCODE**: designates an error code (SQLSTATE) to a report. A condition name or a five-character SQLSTATE error code can be used. + +- **condition_name**: specifies the condition name corresponding to the error code. + +- **sqlstate**: specifies the error code. + +If neither a condition name nor an **SQLSTATE** is designated in a **RAISE EXCEPTION** command, the **RAISE EXCEPTION (P0001)** is used by default. If no message text is designated, the condition name or SQLSTATE is used as the message text by default. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> +> - If the **SQLSTATE** designates an error code, the error code is not limited to a defined error code. It can be any error code containing five digits or ASCII uppercase rather than **00000**. Do not use an error code ended with three zeros because such error codes are category codes and can be captured by the whole category. +> - In O-compatible mode, SQLCODE is equivalent to SQLSTATE. +> +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> The syntax described in [Figure 5](#raise) does not append any parameter. This form is used only for the **EXCEPTION** statement in a **BEGIN** block so that the error can be re-processed. + +**EXCEPTION_INIT** + +In O-compatible mode, EXCEPTION_INIT can be used to define the SQLCODE error code. The syntax is as follows: + +**Figure 6** exception_init::= + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/debugging-6.png) + +**Parameter description**: + +- **exception_name** indicates the name of the exception declared by the user. The **EXCEPTION_INIT** syntax must follow the declared exception. +- **sqlcode** is a customized SQL code, which must be a negative integer ranging from –2147483647 to –1. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** When **EXCEPTION_INIT** is used to customize an SQL code, SQLSTATE is equivalent to SQLCODE, and SQLERRM is in the format of *xxx*: **non-GaussDB Exception**. For example, if the customized SQL code is **–1**, SQLSTATE is **–1** and SQLERRM is **1: non-GaussDB Exception**. + +**Example** + +Display error and hint information when a transaction terminates: + +```sql +CREATE OR REPLACE PROCEDURE proc_raise1(user_id in integer) +AS +BEGIN +RAISE EXCEPTION 'Noexistence ID --> %',user_id USING HINT = 'Please check your user ID'; +END; +/ + +call proc_raise1(300011); + +-- Execution result: +ERROR: Noexistence ID --> 300011 +HINT: Please check your user ID +``` + +Two methods are available for setting **SQLSTATE**: + +```sql +CREATE OR REPLACE PROCEDURE proc_raise2(user_id in integer) +AS +BEGIN +RAISE 'Duplicate user ID: %',user_id USING ERRCODE = 'unique_violation'; +END; +/ + +\set VERBOSITY verbose +call proc_raise2(300011); + +-- Execution result: +ERROR: Duplicate user ID: 300011 +SQLSTATE: 23505 +``` + +If the main parameter is a condition name or **SQLSTATE**, the following applies: + +RAISE division_by_zero; + +RAISE SQLSTATE '22012'; + +For example: + +```sql +CREATE OR REPLACE PROCEDURE division(div in integer, dividend in integer) +AS +DECLARE +res int; + BEGIN + IF dividend=0 THEN + RAISE division_by_zero; + RETURN; + ELSE + res := div/dividend; + RAISE INFO 'division result: %', res; + RETURN; + END IF; + END; +/ +call division(3,0); + +-- Execution result: +ERROR: division_by_zero +``` + +Alternatively: + +``` +RAISE unique_violation USING MESSAGE = 'Duplicate user ID: ' || user_id; +``` + +In O-compatible mode, EXCEPTION_INIT can be used to customize error codes SQLCODE. + +``` +declare + deadlock_detected exception; + pragma exception_init(deadlock_detected, -1); +begin + if 1 > 0 then + raise deadlock_detected; + end if; +exception + when deadlock_detected then + raise notice 'sqlcode:%,sqlstate:%,sqlerrm:%',sqlcode,sqlstate,sqlerrm; +end; +/ +-- Execution result: +NOTICE: sqlcode:-1,sqlstate:-1,sqlerrm: 1: non-GaussDB Exception +``` \ No newline at end of file diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-14-package.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-14-package.md new file mode 100644 index 0000000000000000000000000000000000000000..56a0041187b7a928467f55c6f9c8ec2f17ebd2b5 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-14-package.md @@ -0,0 +1,21 @@ +--- +title: Package +summary: Package +author: Guo Huan +date: 2022-04-27 +--- + +# Package + +A package is a combination of PL/SQL programs, such as stored procedures, functions, variables, constants, and cursors. It is object-oriented and can encapsulate PL/SQL program design elements. Functions in a package are created, deleted, and modified in a unified manner. + +A package contains two parts: package specifications and package body. The declaration contained in the package specifications can be accessed by external functions and anonymous blocks. The declaration contained in the package body cannot be accessed by external functions or anonymous blocks, but can be accessed only by functions and stored procedures in the package body. + +For details about how to create a package, see [CREATE PACKAGE](../../reference-guide/sql-syntax/CREATE-PACKAGE.md). + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> +> - Cross-package variables cannot be used as control variables in the for loops. +> - Types defined in a package cannot be deleted or modified, and cannot be used to define tables. +> - Cursor variables cannot be referenced in SCHEMA.PACKAGE.CUROSR mode. +> - A cursor with parameters can be opened only in the current package. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-2-data-types.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-2-data-types.md new file mode 100644 index 0000000000000000000000000000000000000000..e43bc6ea5ee4c8e9239cdbb57171b4d25418ae1d --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-2-data-types.md @@ -0,0 +1,10 @@ +--- +title: Data Types +summary: Data Types +author: Guo Huan +date: 2021-03-04 +--- + +# Data Types + +A data type refers to a value set and an operation set defined on the value set. The MogDB database consists of tables, each of which is defined by its own columns. Each column corresponds to a data type. The MogDB uses corresponding functions to perform operations on data based on data types. For example, the MogDB can perform addition, subtraction, multiplication, and division operations on data of numeric values. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-3-data-type-conversion.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-3-data-type-conversion.md new file mode 100644 index 0000000000000000000000000000000000000000..f37d0f9bf0c1e8eb4d4ed5d76990ed1cb36dcc21 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-3-data-type-conversion.md @@ -0,0 +1,41 @@ +--- +title: Data Type Conversion +summary: Data Type Conversion +author: Guo Huan +date: 2021-03-04 +--- + +# Data Type Conversion + +Certain data types in the database support implicit data type conversions, such as assignments and parameters called by functions. For other data types, you can use the type conversion functions provided by MogDB, such as the **CAST** function, to forcibly convert them. + +MogDB lists common implicit data type conversions in [Table 1](#Implicit data type conversions). + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> The valid value range of **DATE** supported by MogDB is from 4713 B.C. to 294276 A.D. + +**Table 1** Implicit data type conversions + +| Raw Data Type | Target Data Type | **Remarks** | +| :------------ | :--------------- | :------------------------------------------- | +| CHAR | VARCHAR2 | - | +| CHAR | NUMBER | Raw data must consist of digits. | +| CHAR | DATE | Raw data cannot exceed the valid date range. | +| CHAR | RAW | - | +| CHAR | CLOB | - | +| VARCHAR2 | CHAR | - | +| VARCHAR2 | NUMBER | Raw data must consist of digits. | +| VARCHAR2 | DATE | Raw data cannot exceed the valid date range. | +| VARCHAR2 | CLOB | - | +| NUMBER | CHAR | - | +| NUMBER | VARCHAR2 | - | +| DATE | CHAR | - | +| DATE | VARCHAR2 | - | +| RAW | CHAR | - | +| RAW | VARCHAR2 | - | +| CLOB | CHAR | - | +| CLOB | VARCHAR2 | - | +| CLOB | NUMBER | Raw data must consist of digits. | +| INT4 | CHAR | - | +| INT4 | BOOLEAN | - | +| BOOLEAN | INT4 | - | diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-4-arrays-and-records.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-4-arrays-and-records.md new file mode 100644 index 0000000000000000000000000000000000000000..44b5c175a26a89f418b22cce708d177bd2c6aa5a --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-4-arrays-and-records.md @@ -0,0 +1,195 @@ +--- +title: Arrays and Records +summary: Arrays and Records +author: Guo Huan +date: 2021-03-04 +--- + +# Arrays and Records + +## Arrays + +**Use of Array Types** + +Before the use of arrays, an array type needs to be defined: + +Define an array type immediately after the **AS** keyword in a stored procedure. The method is as follows: + +``` +TYPE array_type IS VARRAY(size) OF data_type; +``` + +In the preceding information: + +- **array_type**: indicates the name of the array type to be defined. +- **VARRAY**: indicates the array type to be defined. +- **size**: indicates the maximum number of members in the array type to be defined. The value is a positive integer. +- **data_type**: indicates the types of members in the array type to be created. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - In MogDB, an array automatically increases. If an access violation occurs, a null value is returned, and no error message is reported. +> - The scope of an array type defined in a stored procedure takes effect only in this storage process. +> - It is recommended that you use one of the preceding methods to define an array type. If both methods are used to define the same array type, MogDB prefers the array type defined in a stored procedure to declare array variables. +> - **data_type** can also be a **record** type defined in a stored procedure (anonymous blocks are not supported), but cannot be an array or collection type defined in a stored procedure. + +MogDB supports access to array elements by using parentheses, and it also supports the **extend**, **count**, **first**, **last**, **prior**, **exists**, **trim**, **next**, and **delete** functions. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - If a stored procedure contains a DML statement (such as SELECT, UPDATE, INSERT, and DELETE), you are advised to use square brackets to access array elements. Using parentheses will access arrays by default. If no array exists, function expressions will be identified. +> - When the CLOB size is greater than 1 GB, the table of type, record type, and CLOB cannot be used in the input or output parameter, cursor, or raise info in a stored procedure. + +## Sets + +## Use of Set Types + +Before the use of sets, a set type needs to be defined. + +Define a set type immediately after the **AS** keyword in a stored procedure. The definition method is as follows: + +![syntax-of-the-record-type](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/arrays-and-records-2.png) + +In the preceding information: + +- **table_type**: indicates the name of the set type to be defined. +- **TABLE**: indicates the set type to be defined. +- **data_type**: indicates the types of members in the set to be created. +- **indexby_type**: indicates the type of the set index to be created. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - In MogDB, a set automatically increases. If an access violation occurs, a null value is returned, and no error message is reported. +> - The scope of a set type defined in a stored procedure takes effect only in this stored procedure. +> - The index can only be of the integer or varchar type. The length of the varchar type is not restricted. +> - **NOT NULL** has no function but only takes effect in the syntax. +> - **data_type** can also be the record type or set type defined in a stored procedure (anonymous blocks are not supported), but cannot be the array type. +> - Variables of the nested set type cannot be used across packages. +> - Variables of the **TABLE OF** **index by** type cannot be nested in a record as the input and output parameters of a stored procedure. +> - Variables of the **TABLE OF** **index by** type cannot be used as input and output parameters of functions. +> - The **RAISE INFO** command cannot be used to print the entire nested **TABLE OF** variable. +> - The **TABLE OF** variable cannot be transferred across autonomous transactions. +> - The input and output parameters of a stored procedure cannot be defined as the nested **TABLE OF** type. + +MogDB supports access to set elements by using parentheses, and it also supports the **extend**, **count**, **first**, **last**, **prior**, **next**, and **delete** functions. + +The set functions support **multiset union**, **intersect**, **except all**, and **distinct**. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** An expression can contain only one variable of the **TABLE OF** **index by** type. + +## record + +**record Variables** + +Perform the following operations to create a record variable: + +Define a record type and use this type to declare a variable. + +**Syntax** + +For the syntax of the record type, see [Figure 1](#Syntax of the record type). + +**Figure 1** Syntax of the record type + +![syntax-of-the-record-type](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/arrays-and-records-1.png) + +The above syntax diagram is explained as follows: + +- **record_type**: record name +- **field**: record columns +- **datatype**: record data type +- **expression**: expression for setting a default value + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> In MogDB: +> +> - When assigning values to record variables, you can: +> - Declare a record type and define member variables of this type when you declare a function or stored procedure. +> - Assign the value of a record variable to another record variable. +> - Use **SELECT INTO** or **FETCH** to assign values to a record type. +> - Assign the **NULL** value to a record variable. +> - The **INSERT** and **UPDATE** statements cannot use a record variable to insert or update data. +> - Just like a variable, a record column of the compound type does not have a default value in the declaration. +> - **date_type** can also be the **record** type, array type, and collection type defined in the stored procedure (anonymous blocks are not supported). + +**Example** + +```sql +The table used in the following example is defined as follows: +MogDB=# \d emp_rec + Table "public.emp_rec" + Column | Type | Modifiers +----------+--------------------------------+----------- + empno | numeric(4,0) | not null + ename | character varying(10) | + job | character varying(9) | + mgr | numeric(4,0) | + hiredate | timestamp(0) without time zone | + sal | numeric(7,2) | + comm | numeric(7,2) | + deptno | numeric(2,0) | + +-- Perform array operations in the function. +MogDB=# CREATE OR REPLACE FUNCTION regress_record(p_w VARCHAR2) +RETURNS +VARCHAR2 AS $$ +DECLARE + + -- Declare a record type. + type rec_type is record (name varchar2(100), epno int); + employer rec_type; + + -- Use %type to declare the record type. + type rec_type1 is record (name emp_rec.ename%type, epno int not null :=10); + employer1 rec_type1; + + -- Declare a record type with a default value. + type rec_type2 is record ( + name varchar2 not null := 'SCOTT', + epno int not null :=10); + employer2 rec_type2; + CURSOR C1 IS select ename,empno from emp_rec order by 1 limit 1; + +BEGIN + -- Assign a value to a member record variable. + employer.name := 'WARD'; + employer.epno = 18; + raise info 'employer name: % , epno:%', employer.name, employer.epno; + + -- Assign the value of a record variable to another variable. + employer1 := employer; + raise info 'employer1 name: % , epno: %',employer1.name, employer1.epno; + + -- Assign the NULL value to a record variable. + employer1 := NULL; + raise info 'employer1 name: % , epno: %',employer1.name, employer1.epno; + + -- Obtain the default value of a record variable. + raise info 'employer2 name: % ,epno: %', employer2.name, employer2.epno; + + -- Use a record variable in the FOR loop. + for employer in select ename,empno from emp_rec order by 1 limit 1 + loop + raise info 'employer name: % , epno: %', employer.name, employer.epno; + end loop; + + -- Use a record variable in the SELECT INTO statement. + select ename,empno into employer2 from emp_rec order by 1 limit 1; + raise info 'employer name: % , epno: %', employer2.name, employer2.epno; + + -- Use a record variable in a cursor. + OPEN C1; + FETCH C1 INTO employer2; + raise info 'employer name: % , epno: %', employer2.name, employer2.epno; + CLOSE C1; + RETURN employer.name; +END; +$$ +LANGUAGE plpgsql; + +-- Invoke the function. +MogDB=# CALL regress_record('abc'); + +-- Delete the function. +MogDB=# DROP FUNCTION regress_record; +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-5-declare-syntax.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-5-declare-syntax.md new file mode 100644 index 0000000000000000000000000000000000000000..0ae3fb792494249ae60c66515e851e1bb0362118 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-5-declare-syntax.md @@ -0,0 +1,82 @@ +--- +title: DECLARE Syntax +summary: DECLARE Syntax +author: Guo Huan +date: 2021-03-04 +--- + +# DECLARE Syntax + +## Basic Structure + +**Structure** + +A PL/SQL block can contain a sub-block which can be placed in any section. The following describes the architecture of a PL/SQL block: + +- **DECLARE**: declares variables, types, cursors, and regional stored procedures and functions used in the PL/SQL block. + + ```sql + DECLARE + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** + > This part is optional if no variable needs to be declared. + > + > - An anonymous block may omit the **DECLARE** keyword if no variable needs to be declared. + > - For a stored procedure, **AS** is used, which is equivalent to **DECLARE**. The **AS** keyword must be reserved even if there is no variable declaration part. + +- **EXECUTION**: specifies procedure and SQL statements. It is the main part of a program. Mandatory. + + ```sql + BEGIN + ``` + +- Exception part: processes errors. Optional. + + ```sql + EXCEPTION + ``` + +- End + + ``` + END; + / + ``` + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** + > You are not allowed to use consecutive tabs in the PL/SQL block because they may result in an exception when the **gsql** tool is executed with the **-r** parameter specified. + +**Category** + +PL/SQL blocks are classified into the following types: + +- Anonymous block: a dynamic block that can be executed only for once. For details about the syntax, see [Figure 1](#anonymous_block::=). +- Subprogram: a stored procedure, function, operator, or packages stored in a database. A subprogram created in a database can be called by other programs. + +## Anonymous Blocks + +An anonymous block applies to a script infrequently executed or a one-off activity. An anonymous block is executed in a session and is not stored. + +**Syntax** + +[Figure 1](#anonymous_block::=) shows the syntax diagrams for an anonymous block. + +**Figure 1** anonymous_block::= + +![anonymous_block](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/declare-syntax-1.png) + +Details about the syntax diagram are as follows: + +- The execute part of an anonymous block starts with a **BEGIN** statement, has a break with an **END** statement, and ends with a semicolon (;). Type a slash (/) and press **Enter** to execute the statement. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** + > The terminator "/" must be written in an independent row. + +- The declaration section includes the variable definition, type, and cursor definition. + +- A simplest anonymous block does not execute any commands. At least one statement, even a **NULL** statement, must be presented in any implementation blocks. + +## Subprogram + +A subprogram stores stored procedures, functions, operators, and advanced packages. A subprogram created in a database can be called by other programs. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-6-basic-statements.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-6-basic-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..5e88299700d5b66995d80ba73044db606ecc0eb1 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-6-basic-statements.md @@ -0,0 +1,215 @@ +--- +title: Basic Statements +summary: Basic Statements +author: Guo Huan +date: 2021-03-04 +--- + +# Basic Statements + +During PL/SQL programming, you may define some variables, assign values to variables, and call other stored procedures. This chapter describes basic PL/SQL statements, including variable definition statements, value assignment statements, call statements, and return statements. + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> You are advised not to call the SQL statements containing passwords in the stored procedures because authorized users may view the stored procedure file in the database and password information is leaked. If a stored procedure contains other sensitive information, permission to access this procedure must be configured, preventing information leakage. + +## Variable Definition Statements + +This section describes the declaration of variables in the PL/SQL and the scope of this variable in codes. + +**Variable Declaration** + +For details about the variable declaration syntax, see [Figure 1](#declare_variable::=). + +**Figure 1** declare_variable::= + +![declare_variable](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/basic-statements-1.png) + +The above syntax diagram is explained as follows: + +- **variable_name** indicates the name of a variable. +- **type** indicates the type of a variable. +- **value** indicates the initial value of the variable. (If the initial value is not given, NULL is taken as the initial value.) **value** can also be an expression. + +**Examples** + +```sql +MogDB=# DECLARE + emp_id INTEGER := 7788; -- Define a variable and assign a value to it. +BEGIN + emp_id := 5*7784; -- Assign a value to the variable. +END; +/ +``` + +In addition to the declaration of basic variable types, **%TYPE** and **%ROWTYPE** can be used to declare variables related to table columns or table structures. + +**%TYPE Attribute** + +**%TYPE** declares a variable to be of the same data type as a previously declared variable (for example, a column in a table). For example, if you want to define a *my_name* variable whose data type is the same as the data type of the **firstname** column in the **employee** table, you can define the variable as follows: + +``` +my_name employee.firstname%TYPE +``` + +In this way, you can declare *my_name* without the need of knowing the data type of **firstname** in **employee**, and the data type of **my_name** can be automatically updated when the data type of **firstname** changes. + +``` +TYPE employee_record is record (id INTEGER, firstname VARCHAR2(20)); +my_employee employee_record; +my_id my_employee.id%TYPE; +my_id_copy my_id%TYPE; +``` + +**%ROWTYPE Attribute** + +**%ROWTYPE** declares data types of a set of data. It stores a row of table data or results fetched from a cursor. For example, if you want to define a set of data with the same column names and column data types as the **employee** table, you can define the data as follows: + +``` +my_employee employee%ROWTYPE +``` + +The attribute can also be used on the cursor. The column names and column data types of this set of data are the same as those of the **employee** table. For the cursor in a package, **%ROWTYPE** can be omitted. **%TYPE** can also reference the type of a column in the cursor. You can define the data as follows: + +``` +cursor cur is select * from employee; +my_employee cur%ROWTYPE +my_name cur.firstname%TYPE +my_employee2 cur -- For the cursor defined in a package, %ROWTYPE can be omitted. +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> +> - **%TYPE** cannot reference the type of a composite variable or a record variable, a column type of the record type, a column type of a variable of the cross-package composite type, or a column type of a cursor variable of the cross-package type. +> - **%ROWTYPE** cannot reference the type of a composite variable or a record variable and the type of a cross-package cursor. + +**Scope of a Variable** + +The scope of a variable indicates the accessibility and availability of the variable in code block. In other words, a variable takes effect only within its scope. + +- To define a function scope, a variable must declare and create a **BEGIN-END** block in the declaration section. The necessity of such declaration is also determined by block structure, which requires that a variable has different scopes and lifetime during a process. +- A variable can be defined multiple times in different scopes, and inner definition can cover outer one. +- A variable defined in an outer block can also be used in a nested block. However, the outer block cannot access variables in the nested block. + +## Assignment Statements + +**Syntax** + +[Figure 2](#assignment_value) shows the syntax diagram for assigning a value to a variable. + +**Figure 2** assignment_value::= + +![assignment_value](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/basic-statements-2.png) + +The above syntax diagram is explained as follows: + +- **variable_name** indicates the name of a variable. +- **value** can be a value or an expression. The type of **value** must be compatible with the type of **variable_name**. + +**Example** + +```sql +MogDB=# DECLARE + emp_id INTEGER := 7788; --Assignment +BEGIN + emp_id := 5; --Assignment + emp_id := 5*7784; +END; +/ +``` + +**Nested Value Assignment** + +[Figure 3](#nested_assignment_value) shows the syntax diagram for assigning a nested value to a variable. + +**Figure 3** nested_assignment_value::= + +![nested_assignment_value](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/basic-statements-3.png) + +The syntax in Figure 3 is described as follows: + +- **variable_name**: variable name +- **col_name**: column name +- **subscript**: subscript, which is used for an array variable. The value can be a value or an expression and must be of the int type. +- **value**: value or expression. The type of **value** must be compatible with the type of **variable_name**. + +**Example** + +```sql +MogDB=# CREATE TYPE o1 as (a int, b int); +MogDB=# DECLARE + TYPE r1 is VARRAY(10) of o1; + emp_id r1; +BEGIN + emp_id(1).a := 5;-- Assign a value. + emp_id(1).b := 5*7784; +END; +/ +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> +> - In INTO mode, values can be assigned only to the columns at the first layer. Two-dimensional or above arrays are not supported. +> - When a nested column value is referenced, if an array subscript exists, only one parenthesis can exist in the first three layers of columns. You are advised to use square brackets to reference the subscript. + +## INTO/BULK COLLECT INTO + +**INTO** and **BULK COLLECT INTO** store values returned by statements in a stored procedure to variables. **BULK COLLECT INTO** allows some or all returned values to be temporarily stored in an array. + +**Example** + +``` +MogDB=# DECLARE + my_id integer; +BEGIN + select id into my_id from customers limit 1; -- Assign a value. +END; +/ + +MogDB=# DECLARE + type id_list is varray(6) of customers.id%type; + id_arr id_list; +BEGIN + select id bulk collect into id_arr from customers order by id DESC limit 20; -- Assign values in batches. +END; +/ +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** **BULK COLLECT INTO** can only assign values to arrays in batches. Use **LIMIT** properly to prevent performance deterioration caused by excessive operations on data. + +## Call Statement + +**Syntax** + +[Figure 4](#call_clause) shows the syntax diagram for calling a clause. + +**Figure 4** call_clause::= + +![call_clause](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/basic-statements-4.png) + +The above syntax diagram is explained as follows: + +- **procedure_name** specifies the name of a stored procedure. +- **parameter** specifies the parameters for the stored procedure. You can set no parameter or multiple parameters. + +**Example** + +```sql +-- Create the stored procedure proc_staffs: +MogDB=# CREATE OR REPLACE PROCEDURE proc_staffs +( +section NUMBER(6), +salary_sum out NUMBER(8,2), +staffs_count out INTEGER +) +IS +BEGIN +SELECT sum(salary), count(*) INTO salary_sum, staffs_count FROM hr.staffs where section_id = section; +END; +/ + +-- Invoke a stored procedure proc_return: +MogDB=# CALL proc_staffs(2,8,6); + +-- Delete a stored procedure: +MogDB=# DROP PROCEDURE proc_staffs; +``` diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-7-dynamic-statements.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-7-dynamic-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..5e116ccad3383d83e3af29714dbafe8ab440a762 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-7-dynamic-statements.md @@ -0,0 +1,166 @@ +--- +title: Dynamic Statements +summary: Dynamic Statements +author: Guo Huan +date: 2021-03-04 +--- + +# Dynamic Statements + +## Executing Dynamic Query Statements + +You can perform dynamic queries MogDB provides two modes: EXECUTE IMMEDIATE and OPEN FOR. **EXECUTE IMMEDIATE** dynamically executes **SELECT** statements and **OPEN FOR** combines use of cursors. If you need to store query results in a data set, use **OPEN FOR**. + +**EXECUTE IMMEDIATE** + +[Figure 1](#EXECUTE IMMEDIATE) shows the syntax diagram. + +**Figure 1** EXECUTE IMMEDIATE dynamic_select_clause::= + +![execute-immediate-dynamic_select_clause](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-1.png) + +[Figure 2](#using_clause) shows the syntax diagram for **using_clause**. + +**Figure 2** using_clause::= + +![using_clause](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-2.png) + +The above syntax diagram is explained as follows: + +- **define_variable**: specifies variables to store single-line query results. + +- **USING IN bind_argument**: specifies where the variable passed to the dynamic SQL value is stored, that is, in the dynamic placeholder of **dynamic_select_string**. + +- **USING OUT bind_argument**: specifies where the dynamic SQL returns the value of the variable. + + > ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** + > + > - In query statements, **INTO** and **OUT** cannot coexist. + > - A placeholder name starts with a colon (:) followed by digits, characters, or strings, corresponding to **bind_argument** in the **USING** clause. + > - **bind_argument** can only be a value, variable, or expression. It cannot be a database object such as a table name, column name, and data type. That is, **bind_argument** cannot be used to transfer schema objects for dynamic SQL statements. If a stored procedure needs to transfer database objects through **bind_argument** to construct dynamic SQL statements (generally, DDL statements), you are advised to use double vertical bars (||) to concatenate **dynamic_select_clause** with a database object. + > - A dynamic PL/SQL block allows duplicate placeholders. That is, a placeholder can correspond to only one **bind_argument** in the **USING** clause. + +**OPEN FOR** + +Dynamic query statements can be executed by using **OPEN FOR** to open dynamic cursors. + +[Figure 3](#open_for) shows the syntax diagram. + +**Figure 3** open_for::= + +![open_for](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-3.png) + +Parameter description: + +- **cursor_name**: specifies the name of the cursor to be opened. +- **dynamic_string**: specifies the dynamic query statement. +- **USING value**: applies when a placeholder exists in dynamic_string. + +For use of cursors, see [Cursors](1-11-cursors.md). + +## Executing Dynamic Non-query Statements + +**Syntax** + +[Figure 4](#noselect) shows the syntax diagram. + +**Figure 4** noselect::= + +![noselect](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-4.png) + +[Figure 5](#using_clause::=) shows the syntax diagram for **using_clause**. + +**Figure 5** using_clause::= + +![using_clause-0](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-5.png) + +The above syntax diagram is explained as follows: + +**USING IN bind_argument** is used to specify the variable whose value is passed to the dynamic SQL statement. The variable is used when a placeholder exists in **dynamic_noselect_string**. That is, a placeholder is replaced by the corresponding **bind_argument** when a dynamic SQL statement is executed. Note that **bind_argument** can only be a value, variable, or expression, and cannot be a database object such as a table name, column name, and data type. If a stored procedure needs to transfer database objects through **bind_argument** to construct dynamic SQL statements (generally, DDL statements), you are advised to use double vertical bars (||) to concatenate **dynamic_select_clause** with a database object. In addition, a dynamic PL/SQL block allows duplicate placeholders. That is, a placeholder can correspond to only one **bind_argument**. + +**Example** + +```sql +-- Create a table: +MogDB=# CREATE TABLE sections_t1 +( + section NUMBER(4) , + section_name VARCHAR2(30), + manager_id NUMBER(6), + place_id NUMBER(4) +); + +-- Declare a variable: +MogDB=# DECLARE + section NUMBER(4) := 280; + section_name VARCHAR2(30) := 'Info support'; + manager_id NUMBER(6) := 103; + place_id NUMBER(4) := 1400; + new_colname VARCHAR2(10) := 'sec_name'; +BEGIN +-- Execute the query: + EXECUTE IMMEDIATE 'insert into sections_t1 values(:1, :2, :3, :4)' + USING section, section_name, manager_id,place_id; +-- Execute the query (duplicate placeholders): + EXECUTE IMMEDIATE 'insert into sections_t1 values(:1, :2, :3, :1)' + USING section, section_name, manager_id; +-- Run the ALTER statement. (You are advised to use double vertical bars (||) to concatenate the dynamic DDL statement with a database object.) + EXECUTE IMMEDIATE 'alter table sections_t1 rename section_name to ' || new_colname; +END; +/ + +-- Query data: +MogDB=# SELECT * FROM sections_t1; + +--Delete the table. +MogDB=# DROP TABLE sections_t1; +``` + +## Dynamically Calling Stored Procedures + +This section describes how to dynamically call store procedures. You must use anonymous statement blocks to package stored procedures or statement blocks and append **IN** and **OUT** behind the **EXECUTE IMMEDIATE…USING** statement to input and output parameters. + +**Syntax** + +[Figure 6](#call_procedure) shows the syntax diagram. + +**Figure 6** call_procedure::= + +![call_procedure](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-6.png) + +[Figure 7](#Figure 2) shows the syntax diagram for **using_clause**. + +**Figure 7** using_clause::= + +![using_clause-1](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-7.png) + +The above syntax diagram is explained as follows: + +- **CALL procedure_name**: calls the stored procedure. +- **[:placeholder1,:placeholder2,…]**: specifies the placeholder list of the stored procedure parameters. The numbers of the placeholders and parameters are the same. +- **USING [IN|OUT|IN OUT]bind_argument**: specifies where the variable passed to the stored procedure parameter value is stored. The modifiers in front of **bind_argument** and of the corresponding parameter are the same. + +## Dynamically Calling Anonymous Blocks + +This section describes how to execute anonymous blocks in dynamic statements. Append **IN** and **OUT** behind the **EXECUTE IMMEDIATE…USING** statement to input and output parameters. + +**Syntax** + +[Figure 8](#call_anonymous_block) shows the syntax diagram. + +**Figure 8** call_anonymous_block::= + +![call_anonymous_block](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-8.png) + +[Figure 9](#Figure 2using_clause) shows the syntax diagram for **using_clause**. + +**Figure 9** using_clause::= + +![using_clause-2](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/dynamic-statements-9.png) + +The above syntax diagram is explained as follows: + +- The execute part of an anonymous block starts with a **BEGIN** statement, has a break with an **END** statement, and ends with a semicolon (;). +- **USING [IN|OUT|IN OUT]bind_argument**: specifies where the variable passed to the stored procedure parameter value is stored. The modifiers in front of **bind_argument** and of the corresponding parameter are the same. +- The input and output parameters in the middle of an anonymous block are designated by placeholders. The numbers of the placeholders and parameters are the same. The sequences of the parameters corresponding to the placeholders and the USING parameters are the same. +- Currently in MogDB, when dynamic statements call anonymous blocks, placeholders cannot be used to pass input and output parameters in an **EXCEPTION** statement. diff --git a/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-8-control-statements.md b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-8-control-statements.md new file mode 100644 index 0000000000000000000000000000000000000000..126dbaf9bd581c05aa6f56dc9a927b109dfd7c03 --- /dev/null +++ b/product/en/docs-mogdb/v3.1/developer-guide/plpgsql/1-8-control-statements.md @@ -0,0 +1,649 @@ +--- +title: Control Statements +summary: Control Statements +author: Guo Huan +date: 2021-03-04 +--- + +# Control Statements + +## RETURN Statements + +In MogDB, data can be returned in either of the following ways:**RETURN**, **RETURN NEXT**, or **RETURN QUERY**. **RETURN NEXT** and **RETURN QUERY** are used only for functions and cannot be used for stored procedures. + +### RETURN + +**Syntax** + +[Figure 1](#return_clause::=) shows the syntax diagram for a return statement. + +**Figure 1** return_clause::= + +![return_clause](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-1.jpg) + +The above syntax diagram is explained as follows: + +This statement returns control from a stored procedure or function to a caller. + +**Examples** + +See [Example](1-6-basic-statements.md#call-statement) for call statement examples. + +### RETURN NEXT and RETURN QUERY + +**Syntax** + +When creating a function, specify **SETOF datatype** for the return values. + +return_next_clause::= + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-2.png) + +return_query_clause::= + +![img](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-3.png) + +The above syntax diagram is explained as follows: + +If a function needs to return a result set, use **RETURN NEXT** or **RETURN QUERY** to add results to the result set, and then continue to execute the next statement of the function. As the **RETURN NEXT** or **RETURN QUERY** statement is executed repeatedly, more and more results will be added to the result set. After the function is executed, all results are returned. + +**RETURN NEXT** can be used for scalar and compound data types. + +**RETURN QUERY** has a variant **RETURN QUERY EXECUTE**. You can add dynamic queries and add parameters to the queries by **USING**. + +**Examples** + +```sql +MogDB=# CREATE TABLE t1(a int); +MogDB=# INSERT INTO t1 VALUES(1),(10); + +--RETURN NEXT +MogDB=# CREATE OR REPLACE FUNCTION fun_for_return_next() RETURNS SETOF t1 AS $$ +DECLARE + r t1%ROWTYPE; +BEGIN + FOR r IN select * from t1 + LOOP + RETURN NEXT r; + END LOOP; + RETURN; +END; +$$ LANGUAGE PLPGSQL; +MogDB=# call fun_for_return_next(); + a +--- + 1 + 10 +(2 rows) + +-- RETURN QUERY +MogDB=# CREATE OR REPLACE FUNCTION fun_for_return_query() RETURNS SETOF t1 AS $$ +DECLARE + r t1%ROWTYPE; +BEGIN + RETURN QUERY select * from t1; +END; +$$ +language plpgsql; +MogDB=# call fun_for_return_query(); + a +--- + 1 + 10 +(2 rows) +``` + +## Conditional Statements + +Conditional statements are used to decide whether given conditions are met. Operations are executed based on the decisions made. + +MogDB supports five usages of **IF**: + +- IF_THEN + + **Figure 2** IF_THEN::= + + ![if_then](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-4.jpg) + + **IF_THEN** is the simplest form of **IF**. If the condition is true, statements are executed. If it is false, they are skipped. + + Example: + + ```sql + MogDB=# IF v_user_id <> 0 THEN + UPDATE users SET email = v_email WHERE user_id = v_user_id; + END IF; + ``` + +- IF_THEN_ELSE + + **Figure 3** IF_THEN_ELSE::= + + ![if_then_else](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-5.jpg) + + **IF-THEN-ELSE** statements add **ELSE** branches and can be executed if the condition is false. + + Example: + + ```sql + MogDB=# IF parentid IS NULL OR parentid = '' + THEN + RETURN; + ELSE + hp_true_filename(parentid); -- Call the stored procedure. + END IF; + ``` + +- IF_THEN_ELSE IF + + **IF** statements can be nested in the following way: + + ```sql + MogDB=# IF sex = 'm' THEN + pretty_sex := 'man'; + ELSE + IF sex = 'f' THEN + pretty_sex := 'woman'; + END IF; + END IF; + ``` + + Actually, this is a way of an **IF** statement nesting in the **ELSE** part of another **IF** statement. Therefore, an **END IF** statement is required for each nesting **IF** statement and another **END IF** statement is required to end the parent **IF-ELSE** statement. To set multiple options, use the following form: + +- IF_THEN_ELSIF_ELSE + + **Figure 4** IF_THEN_ELSIF_ELSE::= + + ![if_then_elsif_else](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-6.png) + + Example: + + ```sql + IF number_tmp = 0 THEN + result := 'zero'; + ELSIF number_tmp > 0 THEN + result := 'positive'; + ELSIF number_tmp < 0 THEN + result := 'negative'; + ELSE + result := 'NULL'; + END IF; + ``` + +- IF_THEN_ELSEIF_ELSE + + **ELSEIF** is an alias of **ELSIF**. + + Example: + + ```sql + CREATE OR REPLACE PROCEDURE proc_control_structure(i in integer) + AS + BEGIN + IF i > 0 THEN + raise info 'i:% is greater than 0. ',i; + ELSIF i < 0 THEN + raise info 'i:% is smaller than 0. ',i; + ELSE + raise info 'i:% is equal to 0. ',i; + END IF; + RETURN; + END; + / + + CALL proc_control_structure(3); + + -- Delete the stored procedure. + DROP PROCEDURE proc_control_structure; + ``` + +## Loop Statements + +**Simple LOOP Statements** + +The syntax diagram is as follows: + +**Figure 5** loop::= + +![loop](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-7.png) + +**Example** + +```sql +CREATE OR REPLACE PROCEDURE proc_loop(i in integer, count out integer) +AS + BEGIN + count:=0; + LOOP + IF count > i THEN + raise info 'count is %. ', count; + EXIT; + ELSE + count:=count+1; + END IF; + END LOOP; + END; +/ + +CALL proc_loop(10,5); +``` + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-notice.gif) **NOTICE:** +> The loop must be exploited together with **EXIT**; otherwise, a dead loop occurs. + +**WHILE-LOOP Statements** + +**Syntax diagram** + +**Figure 6** while_loop::= + +![while_loop](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-8.png) + +If the conditional expression is true, a series of statements in the WHILE statement are repeatedly executed and the condition is decided each time the loop body is executed. + +**Example** + +```sql +CREATE TABLE integertable(c1 integer) ; +CREATE OR REPLACE PROCEDURE proc_while_loop(maxval in integer) +AS + DECLARE + i int :=1; + BEGIN + WHILE i < maxval LOOP + INSERT INTO integertable VALUES(i); + i:=i+1; + END LOOP; + END; +/ + +-- Invoke a function: +CALL proc_while_loop(10); + +-- Delete the stored procedure and table. +DROP PROCEDURE proc_while_loop; +DROP TABLE integertable; +``` + +**FOR_LOOP (Integer variable) Statement** + +**Syntax diagram** + +**Figure 7** for_loop::= + +![for_loop](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-9.png) + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - The variable **name** is automatically defined as the **integer** type and exists only in this loop. The variable name falls between lower_bound and upper_bound. +> - When the keyword **REVERSE** is used, the lower bound must be greater than or equal to the upper bound; otherwise, the loop body is not executed. + +**FOR_LOOP Query Statements** + +**Syntax diagram** + +**Figure 8** for_loop_query::= + +![for_loop_query](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-10.png) + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> The variable **target** is automatically defined, its type is the same as that in the **query** result, and it is valid only in this loop. The target value is the query result. + +**FORALL Batch Query Statements** + +**Syntax diagram** + +**Figure 9** forall::= + +![forall](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-11.png) + +> ![img](https://cdn-mogdb.enmotech.com/docs-media/icon/icon-note.gif) **NOTE:** +> +> - The variable *index* is automatically defined as the integer type and exists only in this loop. The value of *index* falls between the value of **low_bound** and the value of **upper_bound**. +> - If **SAVE EXCEPTIONS** is specified, exceptions occurred during DML execution in the loop body are saved in **SQL&BULK_EXCEPTIONS** and an exception is thrown after the execution is complete. If there is no abnormal execution result in the loop, the loop will not be rolled back in the current subtransaction. + +**Example** + +```sql +CREATE TABLE hdfs_t1 ( + title NUMBER(6), + did VARCHAR2(20), + data_peroid VARCHAR2(25), + kind VARCHAR2(25), + interval VARCHAR2(20), + time DATE, + isModified VARCHAR2(10) +); + +INSERT INTO hdfs_t1 VALUES( 8, 'Donald', 'OConnell', 'DOCONNEL', '650.507.9833', to_date('21-06-1999', 'dd-mm-yyyy'), 'SH_CLERK' ); + +CREATE OR REPLACE PROCEDURE proc_forall() +AS +BEGIN + FORALL i IN 100..120 + update hdfs_t1 set title = title + 100*i; +END; +/ + +-- Invoke a function: +CALL proc_forall(); + +-Query the invocation result of the stored procedure. +SELECT * FROM hdfs_t1 WHERE title BETWEEN 100 AND 120; + +-- Delete the stored procedure and table. +DROP PROCEDURE proc_forall; +DROP TABLE hdfs_t1; +``` + +## Branch Statements + +**Syntax** + +[Figure 10](#case_when) shows the syntax diagram for a branch statement. + +**Figure 10** case_when::= + +![case_when](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-12.png) + +[Figure 11](#when_clause) shows the syntax diagram for **when_clause**. + +**Figure 11** when_clause::= + +![when_clause](https://cdn-mogdb.enmotech.com/docs-media/mogdb/developer-guide/control-statements-13.png) + +Parameter description: + +- **case_expression**: specifies the variable or expression. +- **when_expression**: specifies the constant or conditional expression. +- **statement**: specifies the statement to be executed. + +**Examples** + +```sql +CREATE OR REPLACE PROCEDURE proc_case_branch(pi_result in integer, pi_return out integer) +AS + BEGIN + CASE pi_result + WHEN 1 THEN + pi_return := 111; + WHEN 2 THEN + pi_return := 222; + WHEN 3 THEN + pi_return := 333; + WHEN 6 THEN + pi_return := 444; + WHEN 7 THEN + pi_return := 555; + WHEN 8 THEN + pi_return := 666; + WHEN 9 THEN + pi_return := 777; + WHEN 10 THEN + pi_return := 888; + ELSE + pi_return := 999; + END CASE; + raise info 'pi_return : %',pi_return ; +END; +/ + +CALL proc_case_branch(3,0); + +-- Delete the stored procedure. +DROP PROCEDURE proc_case_branch; +``` + +## NULL Statements + +In PL/SQL programs, **NULL** statements are used to indicate "nothing should be done", equal to placeholders. They grant meanings to some statements and improve program readability. + +**Syntax** + +The following shows example use of **NULL** statements. + +```sql +DECLARE + ... +BEGIN + ... + IF v_num IS NULL THEN + NULL; --No data needs to be processed. + END IF; +END; +/ +``` + +## Error Trapping Statements + +By default, any error occurring in a PL/SQL function aborts execution of the function, and indeed of the surrounding transaction as well. You can trap errors and restore from them by using a **BEGIN** block with an **EXCEPTION** clause. The syntax is an extension of the normal syntax for a **BEGIN** block: + +```sql +[<