diff --git a/docs/api_cpp/source_en/api.md b/docs/api_cpp/source_en/api.md index e0c9aead9a5e5a1a083f5a37611c3436fd7ee6f0..2300ce56b644533b8baa526d4fdd07084c593c76 100644 --- a/docs/api_cpp/source_en/api.md +++ b/docs/api_cpp/source_en/api.md @@ -1,80 +1,328 @@ # mindspore::api - + -## Context +## Context(for mindspore) \#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/context.h)> The Context class is used to store environment variables during execution. +// to-do + +## Context(for mindspore-lite) + +\#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/context.h)> + +\#include <[lite_context.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/lite_context.h)> + +The Context class is used to store environment variables during execution. + ### Static Public Member Function -#### Instance +#### Clear ```cpp -static Context &Instance(); +static void Clear(const std::shared_ptr &contxet); ``` -Obtains the MindSpore Context instance object. +Clears the variables of a Context object. -### Public Member Functions +- Parameters + + - `context`: the Context object to clear. + +#### SetAsDefault + +```cpp +static void SetAsDefault(const std::shared_ptr &contxet); +``` + +Sets the variables of a Context object as default. + +- Parameters + + - `context`: the Context object to modify. + +#### SetVendorName + +```cpp +static void SetVendorName(const std::shared_ptr &contxet, const std::string &name); +``` + +Sets the vendor name of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `name`: the vendor name to set. + +#### GetVendorName + +```cpp +static const std::string GetVendorName(const std::shared_ptr &contxet); +``` + +Gets the vendor name of a Context object. + +- Parameters + + - `context`: the Context object to read. + +- Returns + - The vendor name of the Context object. + +#### SetThreadNum + +```cpp +static void SetThreadNum(const std::shared_ptr &contxet, int num); +``` + +Sets the thread number of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `num`: the thread number to set. + +#### GetThreadNum + +```cpp +static int GetThreadNum(const std::shared_ptr &contxet); +``` + +Gets the thread number of a Context object. + +- Parameters + + - `context`: the Context object to read. + +- Returns + - The thread number of the Context object. + +#### SetAllocator + +```cpp +static void SetAllocator(const std::shared_ptr &contxet, std::shared_ptr alloc); +``` + +Sets the Allocator of a Context object. + +- Parameters -#### GetDeviceTarget + - `context`: the Context object to modify. + - `alloc`: the Allocator to set. + +#### GetAllocator ```cpp -const std::string &GetDeviceTarget() const; +static std::shared_ptr GetAllocator(const std::shared_ptr &contxet); ``` -Obtains the target device type. +Gets the Allocator of a Context object. + +- Parameters + + - `context`: the Context object to read. - Returns + - The Allocator of a Context object. + +#### ConfigCPU + +```cpp +static void ConfigCPU(const std::shared_ptr &contxet, bool config); +``` + +Sets the switch of the CPU of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `config`: the boolean value to set. + +#### IfCPUEnabled + +```cpp +static bool IfCPUEnabled(const std::shared_ptr &contxet); +``` + +Gets the boolean value that indicates whether the CPU is enabled of a Context object. + +- Parameters + + - `context`: the Context object to read. + +- Returns + - The boolean value that indicates whether the CPU is enabled. + +#### ConfigCPUFp16 + +```cpp +static void ConfigCPUFp16(const std::shared_ptr &contxet, bool config); +``` + +Sets the switch of the half-precision float feature of the CPU of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `config`: the boolean value to set. + +#### IfCPUFp16Enabled + +```cpp +static bool IfCPUFp16Enabled(const std::shared_ptr &contxet); +``` + +Gets the boolean value that indicates whether the half-precision float feature of the CPU is enabled of a Context object. + +- Parameters + + - `context`: the Context object to read. + +- Returns + - The boolean value that indicates whether the half-precision float feature of the CPU is enabled. + +#### SetCPUBindMode + +```cpp +static void SetCPUBindMode(const std::shared_ptr &contxet, lite::CpuBindMode mode); +``` + +Sets the bind mode to CPU cores of working threads of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `mode`: the bind mode to CPU cores of working threads to set，options are `CpuBindMode::NO_BIND`, `CpuBindMode::MID_CPU`, `CpuBindMode::HIGHER_CPU`. + +#### GetCPUBindMode + +```cpp +static lite::CpuBindMode GetCPUBindMode(const std::shared_ptr &contxet); +``` + +Gets the bind mode to CPU cores of working threads of a Context object. + +- Parameters + + - `context`: the Context object to read. + +- Returns + - The bind mode to CPU cores of working threads. + +#### ConfigGPU + +```cpp +static void ConfigGPU(const std::shared_ptr &contxet, bool config); +``` + +Sets the switch of the GPU of a Context object. + +- Parameters - Current DeviceTarget type. + - `context`: the Context object to modify. + - `config`: the boolean value to set. -#### GetDeviceID +#### IfGPUEnabled ```cpp -uint32_t GetDeviceID() const; +static bool IfGPUEnabled(const std::shared_ptr &contxet); ``` -Obtains the device ID. +Gets the boolean value that indicates whether the GPU is enabled of a Context object. + +- Parameters + + - `context`: the Context object to read. - Returns + - The boolean value that indicates whether the GPU is enabled. + +#### ConfigGPUFp16 - Current device ID. +```cpp +static void ConfigGPUFp16(const std::shared_ptr &contxet, bool config); +``` + +Sets the switch of the half-precision float feature of the GPU of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `config`: the boolean value to set. -#### SetDeviceTarget +#### IfGPUFp16Enabled ```cpp -Context &SetDeviceTarget(const std::string &device_target); +static bool IfGPUFp16Enabled(const std::shared_ptr &contxet); ``` -Configures the target device. +Gets the boolean value that indicates whether the half-precision float feature of the GPU is enabled of a Context object. - Parameters - - `device_target`: target device to be configured. The options are `kDeviceTypeAscend310` and `kDeviceTypeAscend910`. + - `context`: the Context object to read. - Returns + - The boolean value that indicates whether the half-precision float feature of the GPU is enabled. - MindSpore Context instance object. +#### ConfigNPU -#### SetDeviceID +```cpp +static void ConfigNPU(const std::shared_ptr &contxet, bool config); +``` + +Sets the switch of the NPU of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `config`: the boolean value to set. + +#### IfNPUEnabled ```cpp -Context &SetDeviceID(uint32_t device_id); +static bool IfNPUEnabled(const std::shared_ptr &contxet); ``` -Obtains the device ID. +Gets the boolean value that indicates whether the NPU is enabled of a Context object. - Parameters - - `device_id`: device ID to be configured. + - `context`: the Context object to read. - Returns + - The boolean value that indicates whether the NPU is enabled. + +#### SetNPUFrequency - MindSpore Context instance object. +```cpp +static void SetNPUFrequency(const std::shared_ptr &contxet, int freq); +``` + +Sets the working frequency of the NPU of a Context object. + +- Parameters + + - `context`: the Context object to modify. + - `freq`: the frequency to set, ranges from 1 to 3, 4 is optional on `Kirin9000`. Validity is judged during runtime. + +#### GetNPUFrequency + +```cpp +static int GetNPUFrequency(const std::shared_ptr &contxet); +``` + +Gets the working frequency of the NPU of a Context object. + +- Parameters + + - `context`: the Context object to read. + +- Returns + - The working frequency of the NPU. ## Serialization @@ -86,53 +334,65 @@ The Serialization class is used to summarize methods for reading and writing mod #### LoadModel +```cpp +static Graph LoadModel(const std::string &file, ModelType model_type); +``` + +Loads model file from path. Unavailable in mindspore-lite. + +- Parameters + + - `file`: the path of model file. + - `model_type`: the Type of model file, options are `ModelType::kMindIR`, `ModelType::kOM`. + +- Returns + + An instance of `Graph`, used for storing graph data. + +```cpp +static Graph LoadModel(const void *model_data, size_t data_size, ModelType model_type); +``` + +Loads model file from memory buffer. + - Parameters - - `file`: model file path. - - `model_type`: model file type. The options are `ModelType::kMindIR` and `ModelType::kOM`. + - `model_data`: a buffer filled by model file. + - `data_size`: the size of the buffer. + - `model_type`: the Type of model file, options are `ModelType::kMindIR`, `ModelType::kOM`. - Returns - Object for storing graph data. + An instance of `Graph`, used for storing graph data. ## Model \#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/model.h)> -A Model class is used to define a MindSpore model, facilitating computational graph management. +The Model class is used to define a MindSpore model, facilitating computational graph management. ### Constructor and Destructor ```cpp -Model(const GraphCell &graph); +explicit Model(const GraphCell &graph, const std::shared_ptr &model_context); +explicit Model(const std::vector &network, const std::shared_ptr &model_context); ~Model(); ``` -`GraphCell` is a derivative of `Cell`. `Cell` is not open for use currently. `GraphCell` can be constructed from `Graph`, for example, `Model model(GraphCell(graph))`. +`GraphCell` is a derivative of `Cell`. `Cell` is not open for use currently.`GraphCell` can be constructed from `Graph`, for example, `Model model(GraphCell(graph))`。 + +`Context` is used to store the environment variables during execution. ### Public Member Functions #### Build ```cpp -Status Build(const std::map &options); +Status Build(); ``` Builds a model so that it can run on a device. -- Parameters - - - `options`: model build options. In the following table, Key indicates the option name, and Value indicates the corresponding option. - -| Key | Value | -| --- | --- | -| kModelOptionInsertOpCfgPath | [AIPP](https://support.huaweicloud.com/intl/en-us/adevg-ms-atlas200dkappc32/atlasadm_01_0023.html) configuration file path. | -| kModelOptionInputFormat | Manually specifies the model input format. The options are `"NCHW"` and `"NHWC"`. | -| kModelOptionInputShape | Manually specifies the model input shape, for example, `"input_op_name1: n1,c2,h3,w4;input_op_name2: n4,c3,h2,w1"` | -| kModelOptionOutputType | Manually specifies the model output type, for example, `"FP16"` or `"UINT8"`. The default value is `"FP32"`. | -| kModelOptionPrecisionMode | Model precision mode. The options are `"force_fp16"`, `"allow_fp32_to_fp16"`, `"must_keep_origin_dtype"`, and `"allow_mix_precision"`. The default value is `"force_fp16"`. | -| kModelOptionOpSelectImplMode | Operator selection mode. The options are `"high_performance"` and `"high_precision"`. The default value is `"high_performance"`. | - - Returns Status code. @@ -140,7 +400,7 @@ Builds a model so that it can run on a device. #### Predict ```cpp -Status Predict(const std::vector &inputs, std::vector *outputs); +Status Predict(const std::vector &inputs, std::vector *outputs); ``` Inference model. @@ -154,159 +414,186 @@ Inference model. Status code. -#### GetInputsInfo +#### GetInputs + +```cpp +std::vector GetInputs(); +``` + +Obtains all input tensors of the model. + +- Returns + + The vector that includes all input tensors. + +#### GetOutputs + +```cpp +std::vector GetOutputs(); +``` + +Obtains all output tensors of the model. + +- Returns + + - A `vector` that includes all output tensors. + + +#### Resize ```cpp -Status GetInputsInfo(std::vector *names, std::vector> *shapes, std::vector *data_types, std::vector *mem_sizes) const; +Status Resize(const std::vector &inputs, const std::vector> &dims); ``` -Obtains the model input information. +Resize the shapes of inputs. - Parameters - - `names`: optional output parameter, which is the pointer to a `vector` where model inputs are arranged in sequence. The input names are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. - - `shapes`: optional output parameter, which is the pointer to a `vector` where model inputs are arranged in sequence. The input shapes are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. - - `data_types`: optional output parameter, which is the pointer to a `vector` where model inputs are arranged in sequence. The input data types are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. - - `mem_sizes`: optional output parameter, which is the pointer to a `vector` where model inputs are arranged in sequence. The input memory lengths (in bytes) are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. + - `inputs`: a `vector` that includes all input tensors in order. + - `dims`: defines the new shapes of inputs, should be consistent with `inputs`. + - Returns Status code. -#### GetOutputsInfo +#### CheckModelSupport ```cpp -Status GetOutputsInfo(std::vector *names, std::vector> *shapes, std::vector *data_types, std::vector *mem_sizes) const; +static bool CheckModelSupport(const std::string &device_type, ModelType model_type); ``` -Obtains the model output information. +Check whether the type of device supports the type of model. Not available in mindspore-lite. - Parameters - - `names`: optional output parameter, which is the pointer to a `vector` where model outputs are arranged in sequence. The output names are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. - - `shapes`: optional output parameter, which is the pointer to a `vector` where model outputs are arranged in sequence. The output shapes are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. - - `data_types`: optional output parameter, which is the pointer to a `vector` where model outputs are arranged in sequence. The output data types are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. - - `mem_sizes`: optional output parameter, which is the pointer to a `vector` where model outputs are arranged in sequence. The output memory lengths (in bytes) are filled in the container in sequence. If `nullptr` is input, the attribute is not obtained. + - `device_type`: device type，options are `Ascend310`, `Ascend910`. + - `model_type`: the Type of model file, options are `ModelType::kMindIR`, `ModelType::kOM`. - Returns Status code. -## Tensor +## MSTensor \#include <[types.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/types.h)> +The MSTensor class defines a tensor in MindSpore. + ### Constructor and Destructor ```cpp -Tensor(); -Tensor(const std::string &name, DataType type, const std::vector &shape, const void *data, size_t data_len); -~Tensor(); +MSTensor(); +explicit MSTensor(const std::shared_ptr &impl); +MSTensor(const std::string &name, DataType type, const std::vector &shape, const void *data, size_t data_len); +~MSTensor(); ``` ### Static Public Member Function -#### GetTypeSize +#### CreateTensor ```cpp -static int GetTypeSize(api::DataType type); +static MSTensor CreateTensor(const std::string &name, DataType type, const std::vector &shape, + const void *data, size_t data_len) noexcept; ``` -Obtains the memory length of a data type, in bytes. +Create a MSTensor object, whose data need to be copied before accessing by `Model`. - Parameters - - `type`: data type. + - `name`: the name of the `MSTensor`. + - `type`: the data type of the `MSTensor`. + - `shape`: the shape of the `MSTensor`. + - `data`: the data pointer that points to allocated memory. + - `data`: the length of the memory, in bytes. - Returns - Memory length, in bytes. + An instance of `MStensor`. -### Public Member Functions - -#### Name +#### CreateRefTensor ```cpp -const std::string &Name() const; +static MSTensor CreateRefTensor(const std::string &name, DataType type, const std::vector &shape, void *data, + size_t data_len) noexcept; ``` -Obtains the name of a tensor. - -- Returns - - Tensor name. - -#### DataType +Create a MSTensor object, whose data can be directly accessed by `Model`. -```cpp -api::DataType DataType() const; -``` +- Parameters -Obtains the data type of a tensor. + - `name`: the name of the `MSTensor`. + - `type`: the data type of the `MSTensor`. + - `shape`: the shape of the `MSTensor`. + - `data`: the data pointer that points to allocated memory. + - `data`: the length of the memory, in bytes. - Returns - Tensor data type. + An instance of `MStensor`. -#### Shape +### Public Member Functions + +#### Name ```cpp -const std::vector &Shape() const; +const std::string &Name() const; ``` -Obtains the shape of a tensor. +Obtains the name of the `MSTensor`. - Returns - Tensor shape. + The name of the `MSTensor`. -#### SetName +#### DataType ```cpp -void SetName(const std::string &name); +enum DataType DataType() const; ``` -Sets the name of a tensor. +Obtains the data type of the `MSTensor`. -- Parameters +- Returns - - `name`: name to be set. + The data type of the `MSTensor`. -#### SetDataType +#### Shape ```cpp -void SetDataType(api::DataType type); +const std::vector &Shape() const; ``` -Sets the data type of a tensor. +Obtains the shape of the `MSTensor`. -- Parameters +- Returns - - `type`: type to be set. + A `vector` that contains the shape of the `MSTensor`. -#### SetShape +#### ElementNum ```cpp -void SetShape(const std::vector &shape); +int64_t ElementNum() const; ``` -Sets the shape of a tensor. +Obtains the number of elements of the `MSTensor`. -- Parameters +- Returns - - `shape`: shape to be set. + The number of elements of the `MSTensor`. #### Data ```cpp -const void *Data() const; +std::shared_ptr Data() const; ``` -Obtains the constant pointer to the tensor data. +Obtains a shared pointer to the copy of data of the `MSTensor`. - Returns - Constant pointer to the tensor data. + A shared pointer to the copy of data of the `MSTensor`. #### MutableData @@ -314,11 +601,11 @@ Obtains the constant pointer to the tensor data. void *MutableData(); ``` -Obtains the pointer to the tensor data. +Obtains the pointer to the data of the `MSTensor`. - Returns - Pointer to the tensor data. + The pointer to the data of the `MSTensor`. #### DataSize @@ -326,65 +613,46 @@ Obtains the pointer to the tensor data. size_t DataSize() const; ``` -Obtains the memory length (in bytes) of the tensor data. +Obtains the length of the data of the `MSTensor`, in bytes. - Returns - Memory length of the tensor data, in bytes. + The length of the data of the `MSTensor`, in bytes. -#### ResizeData +#### IsDevice ```cpp -bool ResizeData(size_t data_len); +bool IsDevice() const; ``` -Adjusts the memory size of the tensor. - -- Parameters - - - `data_len`: number of bytes in the memory after adjustment. +Get the boolean value that indicates whether the `MSTensor` is on device. - Returns - A value of bool indicates whether the operation is successful. + The boolean value that indicates whether the `MSTensor` is on device. -#### SetData +#### Clone ```cpp -bool SetData(const void *data, size_t data_len); +Tensor Clone() const; ``` -Adjusts the memory data of the tensor. - -- Parameters - - - `data`: memory address of the source data. - - `data_len`: length of the source data memory. +Get a deep copy of the `MSTensor`. Not available in mindspore-lite. - Returns - A value of bool indicates whether the operation is successful. + A deep copy of the `MSTensor`. -#### ElementNum +#### operator==(std::nullptr_t) ```cpp -int64_t ElementNum() const; +bool operator==(std::nullptr_t) const; ``` -Obtains the number of elements in a tensor. +Get the boolean value that indicates whether the `MSTensor` is valid. - Returns - Number of elements in a tensor. + The boolean value that indicates whether the `MSTensor` is valid. -#### Clone - -```cpp -Tensor Clone() const; -``` - -Performs a self copy. - -- Returns - A deep copy. \ No newline at end of file diff --git a/docs/api_cpp/source_en/class_list.md b/docs/api_cpp/source_en/class_list.md index d6f4cd6216606ff0b95bc67f0cc18ebbbb8e4f9d..4b77f2e1fa8982fae208491e5e78a3da863765f8 100644 --- a/docs/api_cpp/source_en/class_list.md +++ b/docs/api_cpp/source_en/class_list.md @@ -16,3 +16,12 @@ Here is a list of all classes with links to the namespace documentation for each | mindspore::session | [LiteSession](https://www.mindspore.cn/doc/api_cpp/en/master/session.html#litesession) | LiteSession defines sessions in MindSpore Lite for compiling Model and forwarding model. | | mindspore::tensor | [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/master/tensor.html#mstensor) | MSTensor defines tensor in MindSpore Lite. | | mindspore::dataset | [LiteMat](https://www.mindspore.cn/doc/api_cpp/en/master/dataset.html#litemat) |LiteMat is a class used to process images. | + +The definitions and namespaces of classes in MindSpore: + +| 命名空间 | 类 | 描述 | +| --------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| mindspore | [Context](https://www.mindspore.cn/doc/api_cpp/en/master/api.html#context) | The Context class is used to store environment variables during execution. | +| mindspore | [Serialization](https://www.mindspore.cn/doc/api_cpp/en/master/api.html#serialization) | The Serialization class is used to summarize methods for reading and writing model files. | +| mindspore | [Model](https://www.mindspore.cn/doc/api_cpp/en/master/api.html#model) | The Model class is used to define a MindSpore model, facilitating computational graph management. | +| mindspore | [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/master/api.html#mstensor) | The MSTensor class defines a tensor in MindSpore. | \ No newline at end of file diff --git a/docs/api_cpp/source_zh_cn/api.md b/docs/api_cpp/source_zh_cn/api.md index ed98393806abbb1699deb49af6f96b9ca2229973..d1e75012a3eeab0311b17cac274f649036d1b78f 100644 --- a/docs/api_cpp/source_zh_cn/api.md +++ b/docs/api_cpp/source_zh_cn/api.md @@ -1,369 +1,779 @@ -# mindspore::api +# mindspore -## Context +## Context(for mindspore) \#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/context.h)> -Context类用于保存执行中的环境变量。 +Context类用于保存执行中的环境变量。包含GlobalContext与ModelContext两个派生类。 + +## GlobalContext : Context(for mindspore) + +GlobalContext定义了执行时的全局变量。 ### 静态公有成员函数 -#### Instance +#### GetGlobalContext ```cpp -static Context &Instance(); +static std::shared_ptr GetGlobalContext(); ``` -获取MindSpore Context实例对象。 +返回GlobalContext单例。 -### 公有成员函数 +- 返回值 + + - 指向GlobalContext单例的智能指针。 + +#### SetGlobalDeviceTarget + +```cpp +static void SetGlobalDeviceTarget(const std::string &device_target); +``` + +配置目标Device。 + +- 参数 + - `device_target`: 将要配置的目标Device，可选有`kDeviceTypeAscend310`、`kDeviceTypeAscend910`。 -#### GetDeviceTarget +#### GetGlobalDeviceTarget ```cpp -const std::string &GetDeviceTarget() const; +static std::string GetGlobalDeviceTarget(); ``` -获取当前目标Device类型。 +获取已配置的Device。 - 返回值 - 当前DeviceTarget的类型。 + 已配置的目标Device。 + +#### SetGlobalDeviceID + +```cpp +static void SetGlobalDeviceID(const unit32_t &device_id); +``` + +配置Device ID。 -#### GetDeviceID +- 参数 + - `device_id`: 将要配置的Device ID。 + +#### GetGlobalDeviceID ```cpp -uint32_t GetDeviceID() const; +static uint32_t GetGlobalDeviceID(); ``` -获取当前Device ID。 +获取已配置的Device ID。 - 返回值 - 当前Device ID。 + 已配置的Device ID。 + +## ModelContext : Context(for mindspore) + +ModelContext定义了Model执行时的环境变量。 + +### 静态公有成员函数 -#### SetDeviceTarget +#### SetInsertOpConfigPath ```cpp -Context &SetDeviceTarget(const std::string &device_target); +static void SetInsertOpConfigPath(const std::shared_ptr &context, const std::string &cfg_path); ``` -配置目标Device。 +。 - 参数 - - `device_target`: 将要配置的目标Device，可选有`kDeviceTypeAscend310`、`kDeviceTypeAscend910`。 + - `context`：将要修改的Context。 + - `cfg_path`：。 -- 返回值 +#### GetInsertOpConfigPath + +```cpp +static std::string GetInsertOpConfigPath(const std::shared_ptr &context); +``` + +。 - 该MindSpore Context实例对象。 +- 参数 + - `context`: 将要读取的Context。 -#### SetDeviceID +#### SetInputFormat ```cpp -Context &SetDeviceID(uint32_t device_id); +static void SetInputFormat(const std::shared_ptr &context, const std::string &format); ``` -获取当前Device ID。 +设置输入格式。 - 参数 + - `context`：将要修改的Context。 + - `format`：输入格式。 - - `device_id`: 将要配置的Device ID。 +#### GetInputFormat -- 返回值 +```cpp +static std::string GetInputFormat(const std::shared_ptr &context); +``` - 该MindSpore Context实例对象。 +获取已配置的输入格式。 -## Serialization +- 参数 + - `context`: 将要读取的Context。 -\#include <[serialization.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/serialization.h)> +#### SetInputShape -Serialization类汇总了模型文件读写的方法。 +```cpp +static void SetInputShape(const std::shared_ptr &context, const std::string &shape); +``` + +设置输入形状。 + +- 参数 + - `context`：将要修改的Context。 + - `format`：输入形状。 + +#### GetInputShape + +```cpp +static std::string GetInputShape(const std::shared_ptr &context); +``` + +获取已配置的输入形状。 + +- 参数 + - `context`: 将要读取的Context。 + +#### SetOutputType + +```cpp +static void SetOutputType(const std::shared_ptr &context, enum DataType output_type); +``` + +设置输出类型。 + +- 参数 + - `context`：将要修改的Context。 + - `output_type`：。 + +#### GetOutputType + +```cpp +static std::string GetOutputType(const std::shared_ptr &context); +``` + +获取已配置的输出类型。 + +- 参数 + - `context`: 将要读取的Context。 + +#### SetPrecisionMode + +```cpp +static void SetPrecisionMode(const std::shared_ptr &context, const std::string &format); +``` + +。 + +- 参数 + - `context`：将要修改的Context。 + - `format`：。 + +#### GetPrecisionMode + +```cpp +static std::string SetGlobalDeviceID(const std::shared_ptr &context); +``` + +。 + +- 参数 + - `context`: 将要读取的Context。 + +#### SetOpSelectImplMode + +```cpp +static void SetOpSelectImplMode(const std::shared_ptr &context, const std::string &precision_mode); +``` + +。 + +- 参数 + - `context`：将要修改的Context。 + - `precision_mode`：。 + +#### GetOpSelectImplMode + +```cpp +static std::string GetOpSelectImplMode(const std::shared_ptr &context); +``` + +。 + +- 参数 + - `context`: 将要读取的Context。 + +## Context(for mindspore-lite) + +\#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/context.h)> + +\#include <[lite_context.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/lite_context.h)> + +Context类用于保存执行时的环境变量。 ### 静态公有成员函数 -#### LoadModel +#### Clear + +```cpp +static void Clear(const std::shared_ptr &contxet); +``` + +清空已配置的环境变量。 - 参数 - - `file`: 模型文件路径。 - - `model_type`：模型文件类型，可选有`ModelType::kMindIR`、`ModelType::kOM`。 + - `context`: 将要清空的Context。 -- 返回值 +#### SetAsDefault - 保存图数据的对象。 +```cpp +static void SetAsDefault(const std::shared_ptr &contxet); +``` -## Model +设置默认的环境变量，通常是默认CPU配置。 -\#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/model.h)> +- 参数 -Model定义了MindSpore中的模型，便于计算图管理。 + - `context`: 将要修改的Context。 -### 构造函数和析构函数 +#### SetVendorName ```cpp -Model(const GraphCell &graph); -~Model(); +static void SetVendorName(const std::shared_ptr &contxet, const std::string &name); ``` -`GraphCell`是`Cell`的一个派生，`Cell`目前没有开放使用。`GraphCell`可以由`Graph`构造，如`Model model(GraphCell(graph))`。 +设置芯片厂商名称。 -### 公有成员函数 +- 参数 -#### Build + - `context`: 将要修改的Context。 + - `name`: 将要设置的芯片厂商名称。 + +#### GetVendorName ```cpp -Status Build(const std::map &options); +static std::string GetVendorName(const std::shared_ptr &contxet); ``` -将模型编译至可在Device上运行的状态。 +获取已配置的芯片厂商名称。 - 参数 - - `options`: 模型编译选项，key为选项名，value为对应选项，支持的options有： + - `context`: 将要读取的Context。 -| Key | Value | -| --- | --- | -| kModelOptionInsertOpCfgPath | [AIPP](https://support.huaweicloud.com/adevg-ms-atlas200dkappc32/atlasadm_01_0023.html)配置文件路径 | -| kModelOptionInputFormat | 手动指定模型输入format，可选有`"NCHW"`，`"NHWC"`等 | -| kModelOptionInputShape | 手动指定模型输入shape，如`"input_op_name1: n1,c2,h3,w4;input_op_name2: n4,c3,h2,w1"` | -| kModelOptionOutputType | 手动指定模型输出type，如`"FP16"`，`"UINT8"`等，默认为`"FP32"` | -| kModelOptionPrecisionMode | 模型精度模式，可选有`"force_fp16"`，`"allow_fp32_to_fp16"`，`"must_keep_origin_dtype"`或者`"allow_mix_precision"`，默认为`"force_fp16"` | -| kModelOptionOpSelectImplMode | 算子选择模式，可选有`"high_performance"`和`"high_precision"`，默认为`"high_performance"` | +- 返回值 + - 芯片厂商名称 + +#### SetThreadNum + +```cpp +static void SetThreadNum(const std::shared_ptr &contxet, int num); +``` + +设置线程数。 + +- 参数 + + - `context`: 将要修改的Context。 + - `num`: 将要设置的线程数。 + +#### GetThreadNum + +```cpp +static int GetThreadNum(const std::shared_ptr &contxet); +``` + +获取已配置的线程数。 + +- 参数 + + - `context`: 将要读取的Context。 - 返回值 + - 线程数 - 状态码。 +#### SetAllocator -#### Predict +```cpp +static void SetAllocator(const std::shared_ptr &contxet, std::shared_ptr alloc); +``` + +设置内存管理器。 + +- 参数 + + - `context`: 将要修改的Context。 + - `alloc`: 将要设置的内存管理器。 + +#### GetAllocator ```cpp -Status Predict(const std::vector &inputs, std::vector *outputs); +static std::shared_ptr GetAllocator(const std::shared_ptr &contxet); ``` -推理模型。 +获取已配置的内存管理器。 - 参数 - - `inputs`: 模型输入按顺序排列的`vector`。 - - `outputs`: 输出参数，按顺序排列的`vector`的指针，模型输出会按顺序填入该容器。 + - `context`: 将要读取的Context。 - 返回值 + - 内存管理器的智能指针 - 状态码。 +#### ConfigCPU + +```cpp +static void ConfigCPU(const std::shared_ptr &contxet, bool config); +``` + +设置是否启用CPU设备。 + +- 参数 -#### GetInputsInfo + - `context`: 将要修改的Context。 + - `config`: 将要设置的bool值。 + +#### IfCPUEnabled ```cpp -Status GetInputsInfo(std::vector *names, std::vector> *shapes, std::vector *data_types, std::vector *mem_sizes) const; +static bool IfCPUEnabled(const std::shared_ptr &contxet); ``` -获取模型输入信息。 +判断CPU是否已启用。 - 参数 - - `names`: 可选输出参数，模型输入按顺序排列的`vector`的指针，模型输入的name会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 - - `shapes`: 可选输出参数，模型输入按顺序排列的`vector`的指针，模型输入的shape会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 - - `data_types`: 可选输出参数，模型输入按顺序排列的`vector`的指针，模型输入的数据类型会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 - - `mem_sizes`: 可选输出参数，模型输入按顺序排列的`vector`的指针，模型输入的以字节为单位的内存长度会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 + - `context`: 将要读取的Context。 - 返回值 + - CPU是否已启用 - 状态码。 +#### ConfigCPUFp16 + +```cpp +static void ConfigCPUFp16(const std::shared_ptr &contxet, bool config); +``` -#### GetOutputsInfo +设置是否启用CPU的FP16算子。 + +- 参数 + + - `context`: 将要修改的Context。 + - `config`: 将要设置的bool值。 + +#### IfCPUFp16Enabled ```cpp -Status GetOutputsInfo(std::vector *names, std::vector> *shapes, std::vector *data_types, std::vector *mem_sizes) const; +static bool IfCPUFp16Enabled(const std::shared_ptr &contxet); ``` -获取模型输出信息。 +判断CPU的FP16算子是否已启用。 - 参数 - - `names`: 可选输出参数，模型输出按顺序排列的`vector`的指针，模型输出的name会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 - - `shapes`: 可选输出参数，模型输出按顺序排列的`vector`的指针，模型输出的shape会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 - - `data_types`: 可选输出参数，模型输出按顺序排列的`vector`的指针，模型输出的数据类型会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 - - `mem_sizes`: 可选输出参数，模型输出按顺序排列的`vector`的指针，模型输出的以字节为单位的内存长度会按顺序填入该容器，传入`nullptr`则表示不获取该属性。 + - `context`: 将要读取的Context。 - 返回值 + - CPU的FP16算子是否已启用 - 状态码。 +#### SetCPUBindMode + +```cpp +static void SetCPUBindMode(const std::shared_ptr &contxet, lite::CpuBindMode mode); +``` -## Tensor +设置CPU线程绑定模式。 -\#include <[types.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/types.h)> +- 参数 -### 构造函数和析构函数 + - `context`: 将要修改的Context。 + - `mode`: 将要设置的线程绑定模式，可以是`CpuBindMode::NO_BIND`，`CpuBindMode::MID_CPU`，`CpuBindMode::HIGHER_CPU`。 + +#### GetCPUBindMode ```cpp -Tensor(); -Tensor(const std::string &name, DataType type, const std::vector &shape, const void *data, size_t data_len); -~Tensor(); +static lite::CpuBindMode GetCPUBindMode(const std::shared_ptr &contxet); ``` -### 静态公有成员函数 +获取已配置的CPU线程绑定模式。 + +- 参数 + + - `context`: 将要读取的Context。 + +- 返回值 + - CPU线程绑定模式 -#### GetTypeSize +#### ConfigGPU ```cpp -static int GetTypeSize(api::DataType type); +static void ConfigGPU(const std::shared_ptr &contxet, bool config); ``` -获取数据类型的内存长度，以字节为单位。 +设置是否启用GPU设备。 - 参数 - - `type`: 数据类型。 + - `context`: 将要修改的Context。 + - `config`: 将要设置的bool值。 + +#### IfGPUEnabled + +```cpp +static bool IfGPUEnabled(const std::shared_ptr &contxet); +``` + +判断GPU是否已启用。 + +- 参数 + + - `context`: 将要读取的Context。 - 返回值 + - GPU是否已启用 - 内存长度，单位是字节。 +#### ConfigGPUFp16 -### 公有成员函数 +```cpp +static void ConfigGPUFp16(const std::shared_ptr &contxet, bool config); +``` -#### Name +设置是否启用GPU的FP16算子。 + +- 参数 + + - `context`: 将要修改的Context。 + - `config`: 将要设置的bool值。 + +#### IfGPUFp16Enabled ```cpp -const std::string &Name() const; +static bool IfGPUFp16Enabled(const std::shared_ptr &contxet); ``` -获取Tensor的名字。 +判断GPU的FP16算子是否已启用。 + +- 参数 + + - `context`: 将要读取的Context。 - 返回值 + - GPU的FP16算子是否已启用 - Tensor的名字。 +#### ConfigNPU -#### DataType +```cpp +static void ConfigNPU(const std::shared_ptr &contxet, bool config); +``` + +设置是否启用NPU设备。 + +- 参数 + + - `context`: 将要修改的Context。 + - `config`: 将要设置的bool值。 + +#### IfNPUEnabled ```cpp -api::DataType DataType() const; +static bool IfNPUEnabled(const std::shared_ptr &contxet); ``` -获取Tensor的数据类型。 +判断NPU是否已启用。 + +- 参数 + + - `context`: 将要读取的Context。 - 返回值 + - NPU是否已启用 - Tensor的数据类型。 +#### SetNPUFrequency -#### Shape +```cpp +static void SetNPUFrequency(const std::shared_ptr &contxet, int freq); +``` + +设置NPU工作频率。 + +- 参数 + + - `context`: 将要修改的Context。 + - `freq`: 将要设置的频率，通常取值范围为1~3，`麒麟9000`上为1~4。只能在设备运行时判断合法性。 + +#### GetNPUFrequency ```cpp -const std::vector &Shape() const; +static int GetNPUFrequency(const std::shared_ptr &contxet); ``` -获取Tensor的Shape。 +获取已设置的NPU工作频率。 + +- 参数 + + - `context`: 将要读取的Context。 - 返回值 + - NPU工作频率 + +## Serialization - Tensor的Shape。 +\#include <[serialization.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/serialization.h)> -#### SetName +Serialization类汇总了模型文件读写的方法。 + +### 静态公有成员函数 + +#### LoadModel ```cpp -void SetName(const std::string &name); +static Graph LoadModel(const std::string &file, ModelType model_type); ``` -设置Tensor的名字。 +从文件加载模型。Mindspore-lite未提供此功能。 - 参数 - - `name`: 将要设置的name。 + - `file`：模型文件路径。 + - `model_type`：模型文件类型，可选有`ModelType::kMindIR`，`ModelType::kOM`。 + +- 返回值 -#### SetDataType + 保存图数据的`Graph`实例。 ```cpp -void SetDataType(api::DataType type); +static Graph LoadModel(const void *model_data, size_t data_size, ModelType model_type); ``` -设置Tensor的数据类型。 +从内存缓冲区加载模型。 - 参数 - - `type`: 将要设置的type。 + - `model_data`：已读取模型文件的缓存区。 + - `data_size`：缓存区大小。 + - `model_type`：模型文件类型，可选有`ModelType::kMindIR`、`ModelType::kOM`。 -#### SetShape +- 返回值 + + 保存图数据的`Graph`实例。 + +## Model + +\#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/model.h)> + +Model定义了MindSpore中的模型，便于计算图管理。 + +### 构造函数和析构函数 + +```cpp +explicit Model(const GraphCell &graph, const std::shared_ptr &model_context); +explicit Model(const std::vector &network, const std::shared_ptr &model_context); +~Model(); +``` + +`GraphCell`是`Cell`的一个派生，`Cell`目前没有开放使用。`GraphCell`可以由`Graph`构造，如`Model model(GraphCell(graph))`。 + +`Context`表示运行时的环境变量。 + +### 公有成员函数 + +#### Build + +```cpp +Status Build(); +``` + +将模型编译至可在Device上运行的状态。 + +- 返回值 + + 状态码类`Status`对象，可以使用其公有函数`StatusCode`或`ToString`函数来获取具体错误码及错误信息。 + +#### Predict ```cpp -void SetShape(const std::vector &shape); +Status Predict(const std::vector &inputs, std::vector *outputs); ``` -设置Tensor的Shape。 +执行推理。 - 参数 - - `shape`: 将要设置的shape。 + - `inputs`: 模型输入按顺序排列的`vector`。 + - `outputs`: 输出参数，按顺序排列的`vector`的指针，模型输出会按顺序填入该容器。 -#### Data +- 返回值 + + 状态码类`Status`对象，可以使用其公有函数`StatusCode`或`ToString`函数来获取具体错误码及错误信息 + +#### GetInputs ```cpp -const void *Data() const; +std::vector GetInputs(); ``` -获取Tensor中的数据的const指针。 +获取模型所有输入张量。 - 返回值 - 指向Tensor中的数据的const指针。 + 包含模型所有输入张量的容器类型变量。 -#### MutableData +#### GetOutputs ```cpp -void *MutableData(); +std::vector GetOutputs(); ``` -获取Tensor中的数据的指针。 +获取模型所有输出张量。 - 返回值 - 指向Tensor中的数据的指针。 + 包含模型所有输出张量的容器类型变量。 -#### DataSize +#### Resize ```cpp -size_t DataSize() const; +Status Resize(const std::vector &inputs, const std::vector> &dims); +``` + +调整已编译模型的输入形状。 + +- 参数 + + - `inputs`: 模型输入按顺序排列的`vector`。 + - `dims`: 输入形状，按输入顺序排列的由形状组成的`vector`，模型会按顺序依次调整张量形状。 + + +- 返回值 + + 状态码类`Status`对象，可以使用其公有函数`StatusCode`或`ToString`函数来获取具体错误码及错误信息。 + +#### CheckModelSupport + +```cpp +static bool CheckModelSupport(const std::string &device_type, ModelType model_type); ``` -获取Tensor中的数据的以字节为单位的内存长度。 +检查设备是否支持该模型。Mindspore-lite未提供此功能。 + +- 参数 + + - `device_type`: 设备名称，例如`Ascend310`。 + - `model_type`: 模型类型，例如`MindIR`。 - 返回值 - Tensor中的数据的以字节为单位的内存长度。 + 状态码。 + +## MSTensor + +\#include <[types.h](https://gitee.com/mindspore/mindspore/blob/master/include/api/types.h)> + +### 构造函数和析构函数 + +```cpp +MSTensor(); +explicit MSTensor(const std::shared_ptr &impl); +MSTensor(const std::string &name, DataType type, const std::vector &shape, const void *data, size_t data_len); +~MSTensor(); +``` + +### 静态公有成员函数 -#### ResizeData +#### CreateTensor ```cpp -bool ResizeData(size_t data_len); +static MSTensor CreateTensor(const std::string &name, DataType type, const std::vector &shape, + const void *data, size_t data_len) noexcept; ``` -重新调整Tensor的内存大小。 +创建一个`MSTensor`对象，其数据需复制后才能由`Model`访问。 - 参数 - - `data_len`: 调整后的内存字节数。 + - `name`: 名称。 + - `type`：数据类型。 + - `shape`：形状。 + - `data`：数据指针，指向一段已开辟的内存。 + - `data`：数据长度，以字节为单位。 - 返回值 - bool值表示是否成功。 + `MStensor`实例。 -#### SetData +#### CreateRefTensor ```cpp -bool SetData(const void *data, size_t data_len); +static MSTensor CreateRefTensor(const std::string &name, DataType type, const std::vector &shape, void *data, + size_t data_len) noexcept; ``` -重新调整Tensor的内存数据。 +创建一个`MSTensor`对象，其数据可以直接由`Model`访问。 - 参数 - - `data`: 源数据内存地址。 - - `data_len`: 源数据内存长度。 + - `name`: 名称。 + - `type`：数据类型。 + - `shape`：形状。 + - `data`：数据指针，指向一段已开辟的内存。 + - `data`：数据长度，以字节为单位。 + +- 返回值 + + `MStensor`实例。 + +### 公有成员函数 + +#### Name + +```cpp +const std::string &Name() const; +``` + +获取`MSTensor`的名字。 + +- 返回值 + + `MSTensor`的名字。 + +#### DataType + +```cpp +enum DataType DataType() const; +``` + +获取`MSTensor`的数据类型。 + +- 返回值 + + `MSTensor`的数据类型。 + +#### Shape + +```cpp +const std::vector &Shape() const; +``` + +获取`MSTensor`的Shape。 - 返回值 - bool值表示是否成功。 + `MSTensor`的Shape。 #### ElementNum @@ -371,11 +781,59 @@ bool SetData(const void *data, size_t data_len); int64_t ElementNum() const; ``` -获取Tensor中元素的个数。 +获取`MSTensor`的元素个数。 - 返回值 - Tensor中的元素个数 + `MSTensor`的元素个数。 + +#### Data + +```cpp +std::shared_ptr Data() const; +``` + +获取指向`MSTensor`中的数据拷贝的智能指针。 + +- 返回值 + + 指向`MSTensor`中的数据拷贝的智能指针。 + +#### MutableData + +```cpp +void *MutableData(); +``` + +获取`MSTensor`中的数据的指针。 + +- 返回值 + + 指向`MSTensor`中的数据的指针。 + +#### DataSize + +```cpp +size_t DataSize() const; +``` + +获取`MSTensor`中的数据的以字节为单位的内存长度。 + +- 返回值 + + `MSTensor`中的数据的以字节为单位的内存长度。 + +#### IsDevice + +```cpp +bool IsDevice() const; +``` + +判断`MSTensor`是否在设备上。 + +- 返回值 + + `MSTensor`是否在设备上。 #### Clone @@ -383,8 +841,21 @@ int64_t ElementNum() const; Tensor Clone() const; ``` -拷贝一份自身的副本。 +拷贝一份自身的副本。Mindspore-lite未提供此功能。 - 返回值 深拷贝的副本。 + +#### operator==(std::nullptr_t) + +```cpp +bool operator==(std::nullptr_t) const; +``` + +判断`MSTensor`是否合法。 + +- 返回值 + + `MSTensor`是否合法。 + diff --git a/docs/api_cpp/source_zh_cn/class_list.md b/docs/api_cpp/source_zh_cn/class_list.md index 2934485fc023b42c63b20fca44a70cfe310e96c3..e6e0a6932bb14b9b772eaf04d996cd7c442d8323 100644 --- a/docs/api_cpp/source_zh_cn/class_list.md +++ b/docs/api_cpp/source_zh_cn/class_list.md @@ -21,8 +21,7 @@ MindSpore中的类定义及其所属命名空间和描述： | 命名空间 | 类 | 描述 | | --- | --- | --- | -| mindspore::api | [Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context) | Context用于保存执行期间的环境变量。 | -| mindspore::api | [Serialization](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#serialization) | Serialization汇总了模型文件读写的方法。 | -| mindspore::api | [Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model) | Model定义了MindSpore中的模型，便于计算图管理。 | -| mindspore::api | [Tensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#tensor) | Tensor定义了MindSpore中的张量。 | -| mindspore::api | [Buffer](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#buffer) | Buffer管理了一段内存空间。 | +| mindspore | [Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context) | Context用于保存执行期间的环境变量。 | +| mindspore | [Serialization](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#serialization) | Serialization汇总了模型文件读写的方法。 | +| mindspore | [Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model) | Model定义了MindSpore中的模型，便于计算图管理。 | +| mindspore | [MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#mstensor) | Tensor定义了MindSpore中的张量。 | diff --git a/tutorials/lite/source_en/use/runtime.rst b/tutorials/lite/source_en/use/runtime.rst index 49466637b51350fa689d28927a760aa4b73b859b..f7e68728d57993fe6eeb5e4b16fa1315d8972966 100644 --- a/tutorials/lite/source_en/use/runtime.rst +++ b/tutorials/lite/source_en/use/runtime.rst @@ -4,5 +4,6 @@ Executing Model Inference .. toctree:: :maxdepth: 1 + runtime_api runtime_cpp runtime_java \ No newline at end of file diff --git a/tutorials/lite/source_en/use/runtime_api.md b/tutorials/lite/source_en/use/runtime_api.md new file mode 100644 index 0000000000000000000000000000000000000000..b3440db73ef7d7ef6465249b44420f3e87070fc2 --- /dev/null +++ b/tutorials/lite/source_en/use/runtime_api.md @@ -0,0 +1,250 @@ +# Using Unified API for Model Inference（C++） + +`Windows` `Linux` `Android` `推理应用` `模型加载` `数据准备` `中级` `高级` + + + +- [Using Unified API for Model Inference（C++）](#using-unified-api-for-model-inferencec) + - [Overview](#overview) + - [Reading Model Files](#reading-model-files) + - [Session Creation](#session-creation) + - [Creating Contexts](#creating-contexts) + - [Creating Models](#creating-models) + - [Example](#example) + - [Graph Compilation](#graph-compilation) + - [Variable Dimension](#variable-dimension) + - [Example](#example-1) + - [Compiling Graphs](#compiling-graphs) + - [Example](#example-2) + - [Data Input (via model)](#data-input-via-model) + - [Obtaining Input Tensors](#obtaining-input-tensors) + - [Copying Data](#copying-data) + - [Example](#example-3) + - [Data Input (directly)](#data-input-directly) + - [Creating Input Tensors](#creating-input-tensors) + - [Graph Execution](#graph-execution) + - [Executing Sessions](#executing-sessions) + - [Example](#example-4) + - [Obtaining Outputs](#obtaining-outputs) + - [Obtaining Output Tensors](#obtaining-output-tensors) + - [Model parallel launch](#model-parallel-launch) + + + + + +## Overview + +MindSpore provides an unified API, which aims at “Write once, deploy everywhere". You will be able to migrate a Mindspore inference program to Mindspore Lite at a very low cost, vice versa. + +## Reading Model Files + +In MindSpore Lite, a model file is a MS file that is converted by the converter of Mindspore Lite, which holds model data such as weight data and operator attributes. During model inference, the model needs to be loaded from the file system and parsed. Related operations are mainly implemented in the [Serialization](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#serializarion) component. + +Firstly we call `LoadModel`, a static member function of [Serialization](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#serializarion), to load model from a memory buffer, which returns a `Graph` instance. Then we use the `Graph` instance to construct a `GraphCell` instance via the construct method of `GraphCell`. A `GraphCell` instance wraps the imformation of model file, and can be used to build a inference model. + +## Session Creation + +When MindSpore Lite is used for inference, [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model) is the main entrance of inference. You can compile and execute graphs through [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model). + +### Creating Contexts + +[Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#context) saves basic configuration parameters required by [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model) to guide graph compilation and execution. The definition of [Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#context) is as follows: + +MindSpore Lite supports heterogeneous inference. The preferred backend for inference is specified by the configuration in [Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/lite.html#context), where CPU is by default and always enabled. During graph compilation, operator selection and scheduling are performed based on backend configuration. At present, only two heterogeneous configurations are supported, CPU and GPU or CPU and NPU. When GPU or NPU is enabled, they are prior to be chosed. + +> For now, it is not supported to enable CPU, GPU and NPU at the same time. + +MindSpore Lite has a built-in thread pool shared by processes. During inference, the maximum number of threads in the thread pool could be set via `SetThreadnum`. The default maximum number is 2. It is recommended that the maximum number should be no more than 4. Otherwise, the performance may be affected. + +MindSpore Lite defines a inner class `Allocator`, which supports dynamic memory allocation and release. If [Allocator](https://www.mindspore.cn/doc/api_cpp/en/r1.1/lite.html#allocator) is not specified, a default [Allocator](https://www.mindspore.cn/doc/api_cpp/en/r1.1/lite.html#allocator) instance is generated and wrote back to `Context` after a model is compiled. You can share the memory allocator between `Model` by sharing `Context` or expicitly set the `Context`. + +If users create the [Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/lite.html#context) by using, it should be released by using `delete` once it’s not required. Usually the [Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/lite.html#context) is released after finishing the session creation is finished. + +We suggest to create shared pointers of [Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#context), which could be shared between [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model). After a `Mdoel` is compiled, the modification of [Context](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#context) will not take effect to the `Model`. + +### Creating Models + +There are two methods to create `Model`： + +- Create from `Network` via construct method, only available in midspore. +- Create from `GraphCell` and `Context` via construct method. + +### Example + +The following sample code demonstrates how to create a [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model): + +```cpp +#include +#include +#include +#include +// create context +std::shared_ptr context = std::make_shared(); +// set up context +Context::SetAsDefault(context); +// load graph +Graph graph = Serialization::LoadModel(model_buffer_from_file, model_buffer_size, ModelType::kMindIR); +// create model +Model model(GraphCell(graph), context); +``` + +Note: + +- The model input `model_buffer_from_file` is read from disks. After it is copied to model input tensors, you need to release it. + +## Graph Compilation + +### Variable Dimension + +When using MindSpore Lite for inference, after the session creation and the graph compilation, you can resize the shape of the input tensor by calling the `Resize` metohd of [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model), if needed. + +> Not all models support variable dimensions. For example, when there is a MatMul operator in the model whose input Tensor is a weight tensor and an input tensor, calling the variable dimension interface will cause the shape of the input tensor and the weight tensor being unmatched. + +### Example + +The following code demonstrates how to resize an input in MindSpore Lite: + +```cpp +// Assume we have created a Model instance named model +std::vector inputs = model.GetInputs(); +std::vector resize_shape = {1, 128, 128, 3}; +// Assume the model has only one input,resize input shape to [1, 128, 128, 3] +std::vector> new_shapes; +new_shapes.push_back(resize_shape); +model.Resize(inputs, new_shapes); +``` + +### Compiling Graphs + +Before graph execution, you need to call the [Build](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#build) method of [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model) to compile graphs and further parse the `Graph` instance loaded from the file, mainly for subgraph split and operator selection and scheduling. [Build](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#build) can be called only once, who takes a long time. After compilation, a part of memory in `Graph` is released, and compilations from the same `Graph` instance will fail by then. + +### Example + +The following code demonstrates how to compile a graph in MindSpore Lite: + +```cpp +// Assume we have created a Model instance named model +Status ret = model.Build(); +if (ret != kSuccess) { + std::cerr << "Build model failed." << std::endl; + return -1; +} +``` + +## Data Input (via model) + +### Obtaining Input Tensors + +Before graph execution, you can copy the input data to the input [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) of [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model). + +MindSpore Lite provides `GetInputs` method to obtain all input [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/tensor.html#mstensor) of a [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model). + +```cpp +std::vector inputs = model.GetInputs(); +``` + +### Copying Data + +After model input tensors are obtained, you need to enter data into the tensors. Use the `DataSize` method of [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) to obtain the size of the data to be entered into tensors, use the `DataType` method to obtain the data type of tensors, and use the `MutableData` method of [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) to obtain the writable pointer. + +### Example + +The following sample code shows how to obtain the entire graph input [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) from [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model) and enter the input data to [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor). + +```cpp +std::vector inputs = model.GetInputs(); +// Assume that the model has only one input tensor. +MSTensor in_tensor = inputs.front(); +if (in_tensor == nullptr) { + std::cerr << "Input tensor is null." << std::endl; + return -1; +} +if (in_tensor.DataSize() != data_size_expected) { + std::cerr << "Input data size is not suit for model input." << std::endl; + return -1; +} +void *in_data = in_tensor->MutableData(); +if (in_data == nullptr) { + std::cerr << "Data of in_tensor is nullptr" << std::endl; + return -1; +} +memcpy(in_data, input_buf, data_size); +``` + +Note: + +- The data layout in the input [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) must be NHWC. +- The `vector` returned by the `GetInputs` method does not need to be released by users. + +## Data Input (directly) + +### Creating Input Tensors + +MindSpore Lite provides `CreateTensor` method and `CreateRefTensor` method to create [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor). The data in [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) created by `CreateTensor` will be copied once, but directly used in [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) created by `CreateRefTensor`. + +You need to ensure the order of input [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/tensor.html#mstensor) in the `vector`, and the `DataSize` of them for both. The shapes will keep the same as the last time the `Build` method or the `Resize` method is called. + +```cpp +std::vector inputs; +// if you are not sure about tensor's name, data type, shape and size, use GetInputs +MSTensor tensor_a("input_0", kNumberTypeInt8, {1, 4, 4, 3}, data_a, 48); +if (tensor_a == nullptr) { + std::cerr << "Create tensor A failed" << std::endl; + return -1; +} +inputs.push_back(tensor_a); + +MSTensor tensor_b = CreateTensor("input_1", kNumberTypeInt8, {1, 4, 4, 3}, data_b, 48); +if (tensor_b == nullptr) { + std::cerr << "Create tensor B failed" << std::endl; + return -1; +} +inputs.push_back(tensor_b); + +MSTensor tensor_c = CreateRefTensor("input_2", kNumberTypeInt8, {1, 4, 4, 3}, data_c, 48); +if (tensor_c == nullptr) { + std::cerr << "Create tensor C failed" << std::endl; + return -1; +} +inputs.push_back(tensor_c); +``` + +Note: + +- The data layout in the input [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) must be NHWC. + +- The memory of input data in [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#mstensor) should be released by its owner. + +## Graph Execution + +### Executing Sessions + +After a MindSpore Lite model performs graph compilation, you can use the `Predict` method of [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model) for inference. + +### Example + +```cpp +// init outputs +std::vector outputs; +// run +model.Predict(input, &outputs); +``` + +## Obtaining Outputs + +### Obtaining Output Tensors + +After performing inference, the result could also be obtained manually. + +MindSpore Lite provides `GetOutputs` method to obtain the output [MSTensor](https://www.mindspore.cn/doc/api_cpp/en/r1.1/tensor.html#mstensor). + +```c++ +std::vector outputs = model.GetOutputs(); +``` + +The `vector` returned by the `GetOutputs` method does not need to be released by users. + +## Model parallel launch + +Please refer to [LiteSession](https://www.mindspore.cn/tutorial/lite/en/r1.1/use/runtime_cpp.html#session-parallel-launch), with which [Model](https://www.mindspore.cn/doc/api_cpp/en/r1.1/api.html#model) has the same parallel feature. diff --git a/tutorials/lite/source_zh_cn/use/runtime.rst b/tutorials/lite/source_zh_cn/use/runtime.rst index f221f4421703d63341b064fd72d773bbe0a80cb1..cce3f3cfa79e9b4a75028449c82761b16fb11f50 100644 --- a/tutorials/lite/source_zh_cn/use/runtime.rst +++ b/tutorials/lite/source_zh_cn/use/runtime.rst @@ -4,5 +4,6 @@ .. toctree:: :maxdepth: 1 + runtime_api runtime_cpp - runtime_java \ No newline at end of file + runtime_java diff --git a/tutorials/lite/source_zh_cn/use/runtime_api.md b/tutorials/lite/source_zh_cn/use/runtime_api.md new file mode 100644 index 0000000000000000000000000000000000000000..c638ff3a032a4b1d4ba9bce34484e9263c443b60 --- /dev/null +++ b/tutorials/lite/source_zh_cn/use/runtime_api.md @@ -0,0 +1,243 @@ +# 使用统一API执行推理（C++） + +`Windows` `Linux` `Android` `推理应用` `模型加载` `数据准备` `中级` `高级` + + + +- [使用统一API执行推理（C++）](#使用统一api执行推理c) + - [概述](#概述) + - [读取模型](#读取模型) + - [创建会话](#创建会话) + - [创建上下文](#创建上下文) + - [创建会话](#创建会话-1) + - [使用示例](#使用示例) + - [图编译](#图编译) + - [可变维度](#可变维度) + - [使用示例](#使用示例-1) + - [图编译](#图编译-1) + - [使用示例](#使用示例-2) + - [输入数据（由模型输入）](#输入数据由模型输入) + - [获取输入](#获取输入) + - [数据拷贝](#数据拷贝) + - [使用示例](#使用示例-3) + - [输入数据（直接输入）](#输入数据直接输入) + - [创建输入](#创建输入) + - [图执行](#图执行) + - [执行会话](#执行会话) + - [使用示例](#使用示例-4) + - [获取输出](#获取输出) + - [获取输出Tensor](#获取输出tensor) + - [Model并行](#model并行) + + + + + + +## 概述 + +MindSpore Lite提供的端云统一API，是一套旨在实现“一次编码，多次部署”的应用程序编程接口，您将可以以极小的代价将MindSpore推理代码迁移至MindSpore Lite，反之亦然。 + +## 读取模型 + +在MindSpore Lite中，模型文件是从模型转换工具转换得到的`.ms`文件。模型文件持有权重数据、算子属性等数据。进行推理时，需要从文件系统加载模型，并进行模型解析，这部分操作主要在[Serialization](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#serializarion)中实现。 + +我们首先通过[Serialization](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#serializarion)类的静态方法`LoadModel`从内存数据中读取模型，得到一个`Graph`实例。然后调用`GraphCell`的构造方法由刚刚得到的`Graph`对象创建一个`GraphCell`实例。`GraphCell`实例封装了模型信息，并且可用于构造推理模型。 + +## 创建会话 + +使用MindSpore Lite执行推理时，[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)是推理的主入口，通过[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)我们可以进行图编译、图执行。 + +### 创建上下文 + +上下文会保存会话所需的一些基本配置参数，用于指导图编译和图执行，其定义如下： + +MindSpore Lite支持异构推理，推理时的后端配置由[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context)中的配置信息指定，默认且强制启用CPU。在进行图编译时，会根据[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context)中不同的后端配置信息进行算子选型调度。目前仅支持两种异构，CPU和GPU异构或者CPU和NPU异构。 当启用GPU或NPU时，优先使用GPU或NPU推理。 + +> 暂时不支持同时启用CPU, GPU和NPU三种设备。三者同时启用则会报错。 + +MindSpore Lite内置一个进程共享的线程池，推理时通过`SetThreadNum`指定线程池的最大线程数，默认为2线程，推荐最多不超过4个线程，否则可能会影响性能。 + +MindSpore Lite内部定义了一个内存管理类[Allocator](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#allocator)，支持动态内存分配和释放，如果没有指定[Allocator](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#allocator)，模型`Build`后会生成一个默认的[Allocator](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#allocator)并写回[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#context)中，可以通过共用[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#context)指针或显式指定[Allocator](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#allocator)的方法共享内存分配器。 + +推荐使用智能指针创建[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context)，注意多个[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)可以共享一个[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context)，但在`Build`后，[Context](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#context)的修改将不再对[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)生效。 + +### 创建会话 + +有两种方式可以创建会话： + +- 第一种方法是从`Network`创建，目前未开放支持。 +- 第二种方法是从上面得到的`GraphCell`与`Context`创建。 + +### 使用示例 + +下面示例代码演示了[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)的创建： + +```cpp +#include +#include +#include +#include +// create context +std::shared_ptr context = std::make_shared(); +// set up context +Context::SetAsDefault(context); +// load graph +Graph graph = Serialization::LoadModel(model_buffer_from_file, model_buffer_size, ModelType::kMindIR); +// create model +Model model(GraphCell(graph), context); +``` + +## 图编译 + +### 可变维度 + +使用MindSpore Lite进行推理时，在已完成会话创建与图编译之后，如果需要对输入进行`Resize`，则可以通过调整输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#mstensor)的形状，然后调用[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)的`Resize`接口。 + +> 某些网络是不支持可变维度，会提示错误信息后异常退出，比如，模型中有MatMul算子，并且MatMul的一个输入Tensor是权重，另一个输入Tensor是输入时，调用可变维度接口会导致输入Tensor和权重Tensor的Shape不匹配，最终导致推理失败。 + +### 使用示例 + +下面代码演示如何对MindSpore Lite的输入进行[Resize](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#resize)： + +```cpp +// Assume we have created a Model instance named model +std::vector inputs = model.GetInputs(); +std::vector resize_shape = {1, 128, 128, 3}; +// Assume the model has only one input,resize input shape to [1, 128, 128, 3] +std::vector> new_shapes; +new_shapes.push_back(resize_shape); +model.Resize(inputs, new_shapes); +``` + +### 图编译 + +在图执行前，需要调用[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)的[Build](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#build)接口进行图编译，进一步解析从文件中加载的`Graph`实例，主要进行子图切分、算子选型调度。[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)只能编译一次，且这部分会耗费较多时间。在编译之后，`Graph`中的一部分内存将会被自动释放，届时再次由这个`Graph`编译[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)时会失败。 + +### 使用示例 + +下面代码演示如何进行图编译: + +```cpp +// Assume we have created a Model instance named model +Status ret = model.Build(); +if (ret != kSuccess) { + std::cerr << "Build model failed." << std::endl; + return -1; +} +``` + +## 输入数据（由模型输入） + +### 获取输入 + +在图执行前，可以将输入数据拷贝到模型的输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)。MindSpore Lite提供`GetInputs`来直接获取所有的模型输入。 + +```cpp +std::vector inputs = model.GetInputs(); +``` + +### 数据拷贝 + +当获取到模型的输入，就需要向[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)中填入数据。通过[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)的`DataSize`方法来获取应该填入的数据大小，通过`DataType`方法来获取数据类型，通过的`MutableData`方法来获取可写的指针。 + +### 使用示例 + +下面示例代码演示了从[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)中获取整图输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)，并且向其中灌入模型输入数据的过程： + +```cpp +std::vector inputs = model.GetInputs(); +// Assume that the model has only one input tensor. +MSTensor in_tensor = inputs.front(); +if (in_tensor == nullptr) { + std::cerr << "Input tensor is null." << std::endl; + return -1; +} +if (in_tensor.DataSize() != data_size_expected) { + std::cerr << "Input data size is not suit for model input." << std::endl; + return -1; +} +void *in_data = in_tensor->MutableData(); +if (in_data == nullptr) { + std::cerr << "Data of in_tensor is nullptr" << std::endl; + return -1; +} +memcpy(in_data, input_buf, data_size); +``` + +需要注意的是： + +- MindSpore Lite的模型输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)中的数据排布必须是`NHWC`。 +- 模型的输入`input_buf`是用户从磁盘读取的，当拷贝给模型输入以后，用户需要自行释放`input_buf`。 +- `GetInputs`方法返回的`vector`不需要用户释放。 + +## 输入数据（直接输入） + +### 创建输入 + +MindSpore Lite提供`CreateTensor`与`CreateRefTensor`来直接创建[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)，注意使用`CreateTensor`创建的[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)推理时会发生一次数据拷贝，`CreateRefTensor`则不会。 + +需要注意通过此方法进行推理时，需要保证输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)的顺序。同时每个[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)需要与输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)有相同的`DataSize`，形状将会默认保持最后一次`Build`或`Resize`之后的形状。 + +```cpp +std::vector inputs; +// if you are not sure about tensor's name, data type, shape and size, use GetInputs +MSTensor tensor_a("input_0", kNumberTypeInt8, {1, 4, 4, 3}, data_a, 48); +if (tensor_a == nullptr) { + std::cerr << "Create tensor A failed" << std::endl; + return -1; +} +inputs.push_back(tensor_a); + +MSTensor tensor_b = CreateTensor("input_1", kNumberTypeInt8, {1, 4, 4, 3}, data_b, 48); +if (tensor_b == nullptr) { + std::cerr << "Create tensor B failed" << std::endl; + return -1; +} +inputs.push_back(tensor_b); + +MSTensor tensor_c = CreateRefTensor("input_2", kNumberTypeInt8, {1, 4, 4, 3}, data_c, 48); +if (tensor_c == nullptr) { + std::cerr << "Create tensor C failed" << std::endl; + return -1; +} +inputs.push_back(tensor_c); +``` + +需要注意的是： + +- MindSpore Lite的模型输入[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)中的数据排布必须是`NHWC`。 +- [MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)的输入`data`在推理结束后由用户自行释放。 + +## 图执行 + +### 执行会话 + +MindSpore Lite会话在进行图编译以后，即可使用[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)的`Predict`方法进行模型推理。 + +### 使用示例 + +```cpp +// init outputs +std::vector outputs; +// run +model.Predict(input, &outputs); +``` + +## 获取输出 + +### 获取输出Tensor + +MindSpore Lite在执行完推理后，也可以手动获取模型的推理结果。 + +MindSpore Lite提供`GetOutputs`方法，直接获取所有的模型输出[MSTensor](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/tensor.html#mstensor)。 + +```c++ +std::vector outputs = model.GetOutputs(); +``` + +`GetOutputs`方法返回的`vector`不需要用户释放。 + +## Model并行 + +请参考[LiteSession](https://gitee.com/mindspore/docs/blob/r1.1/tutorials/lite/source_zh_cn/use/runtime_cpp.md#session%E5%B9%B6%E8%A1%8C)的说明，[Model](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/api.html#model)与其具有相同的并行特性。