diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..071c1984a3148714b0b4595f694d84fba87f1c32 Binary files /dev/null and b/.DS_Store differ diff --git a/contrib/.DS_Store b/contrib/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..7f59e4ce0eb7ed70e203fcbe9a0ab6ba116d6851 Binary files /dev/null and b/contrib/.DS_Store differ diff --git a/contrib/PoseEstNet/README.md b/contrib/PoseEstNet/README.md new file mode 100755 index 0000000000000000000000000000000000000000..ae0e59a333252a1e65e2901138bb8fe155ff0cd8 --- /dev/null +++ b/contrib/PoseEstNet/README.md @@ -0,0 +1,282 @@ +# MindXSDK 车辆姿态识别 + +## 1 简介 +本开发样例基于MindX SDK实现了姿态估计网络(PoseEstNet)，用于检测并预测车辆36个关键点坐标以及生成热点图。 + +## 2 目录结构 +本工程名称为PoseEstNet，工程目录如下图所示： + +``` +PoseEstNet +|---- models +| | |---- aipp_nv12.cfg +| | |---- coco.names +| | |---- PoseEstNet.om +| | |---- yolov3.cfg +| | |---- yolov3.om +| | |---- aipp_hrnet_256_256.aippconfig +|---- pipeline // 流水线配置文件夹 +| | |---- eval_PoseEstNet.pipeline +| | |---- PoseEstNet.pipeline +|---- plugins // 插件文件夹 +|---- data +|---- data_eval +| | |---- images +| | |---- labels +|---- output // 结果保存文件夹 +|---- output_eval +|---- main.py +|---- eval.py +|---- README.md +``` + +## 3 依赖 +| 软件名称 | 版本 | +| :--------: | :------: | +|ubuntu 18.04|18.04.1 LTS | +|CANN|5.0.4| +|MindX SDK|2.0.4| +|Python| 3.9.12| +|numpy | 1.22.4 | +|opencv_python|4.6.0| +|cmake|3.5+| +请注意MindX SDK使用python版本为3.9.12，如出现无法找到python对应lib库请在root下安装python3.9开发库 +``` +apt-get install libpython3.9 +``` + +## 4 模型转换 +车辆姿态识别先采用了yolov3模型将图片中的车辆检测出来，然后利用PoseEstNet模型预测车辆36个关键点坐标。 + +4.1 yolov3的模型转换： + +**步骤1** 获取yolov3的原始模型(.pb文件)和相应的配置文件(.cfg文件) +      [原始模型下载链接](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/mindxsdk-referenceapps%20/contrib/PoseEstNet/yolov3_tensorflow_1.5.pb) +      [配置文件下载链接](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/mindxsdk-referenceapps%20/contrib/PoseEstNet/aipp_nv12.cfg) + +**步骤2** 将获取到的yolov3模型.pb文件和.cfg文件存放至：“项目所在目录/models” + +**步骤3** .om模型转换 +以下操作均在“项目所在目录/models”路径下进行： +- 设置环境变量（请确认install_path路径是否正确） +``` +export install_path=/usr/local/Ascend/ascend-toolkit/latest + +export PATH=/usr/local/python3.9.2/bin:${install_path}/atc/ccec_compiler/bin:${install_path}/atc/bin:$PATH +export PYTHONPATH=${install_path}/atc/python/site-packages:$PYTHONPATH +export LD_LIBRARY_PATH=${install_path}/atc/lib64:${install_path}/acllib/lib64:$LD_LIBRARY_PATH +export ASCEND_OPP_PATH=${install_path}/opp +export ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest/ +``` +- 使用ATC将.pb文件转成为.om文件 +``` +atc --model=yolov3_tensorflow_1.5.pb --framework=3 --output=yolov3 --output_type=FP32 --soc_version=Ascend310 --input_shape="input:1,416,416,3" --out_nodes="yolov3/yolov3_head/Conv_6/BiasAdd:0;yolov3/yolov3_head/Conv_14/BiasAdd:0;yolov3/yolov3_head/Conv_22/BiasAdd:0" --log=info --insert_op_conf=aipp_nv12.cfg +``` +- 执行完模型转换脚本后，若提示如下信息说明模型转换成功，可以在该路径下找到名为yolov3.om模型文件。 +（可以通过修改output参数来重命名这个.om文件） +``` +ATC run success, welcome to the next use. +``` + +4.2 PoseEstNet的模型转换 + +4.2.1 模型概述 +      [PoseEstNet论文地址](https://arxiv.org/pdf/2005.00673.pdf) +      [PoseEstNet代码地址](https://github.com/NVlabs/PAMTRI/tree/master/PoseEstNet) + +4.2.2 模型转换环境需求 +``` +- 框架需求 + CANN == 5.0.4 + torch == 1.5.0 + torchvision == 0.6.0 + onnx == 1.11.1 + +- python第三方库 + numpy == 1.22.4 + opencv-python == 4.6.0 + Pillow == 8.2.0 + yacs == 0.1.8 + pytorch-ignite == 0.4.5 +``` + +4.2.3 模型转换步骤 + +**步骤1** .pth模型转.onnx模型 + +***1*** 获取.pth权重文件：[下载链接](https://docs.google.com/uc?export=download&id=1vD08fh-za3mgTJ9UkK1ASCTJAqypW0RL) + + models/veri/pose_hrnet/w32_256x256_adam_lr1e-3/model_best.pth + +***2*** 获取PoseEstNet_pth2onnx.py +  下载PoseEstNet源码并创建项目，将该脚本放在“项目所在目录/models”路径下，执行下列命令，生成.onnx模型文件 +``` +python3 tools/PoseEstNet_pth2onnx.py --cfg experiments/veri/hrnet/w32_256x256_adam_lr1e-3.yaml TEST.MODEL_FILE models/veri/pose_hrnet/w32_256x256_adam_lr1e-3/model_best.pth +``` +> 注意目前ATC支持的onnx算子版本为11 + +此时在“项目所在目录/models”路径下会出现PoseEstNet.onnx模型，到此步骤1已完成 +如果在线环境中无法安装pytorch，你可以在本地环境中进行上述.pth模型转.onnx模型操作，然后将得到的.onnx模型放在“项目所在目录/models”即可 + +本项目提供onnx模型：[Huawei Cloud下载链接](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/mindxsdk-referenceapps%20/contrib/PoseEstNet/PoseEstNet.onnx) + +**步骤2** .onnx模型转.om模型 + +***1*** 设置环境变量，进入根目录，执行如下命令后返回原目录 +``` +source usr/local/Ascend/ascend-toolkit/set_env.sh +``` + +***2*** 进入.onnx文件所在目录，使用ATC将.onnx文件转成为.om文件(注意文件路径) +``` +atc --framework=5 --model=PoseEstNet.onnx --output=PoseEstNet --input_format=NCHW --input_shape="image:1,3,256,256" --insert_op_conf=aipp_hrnet_256_256.aippconfig --log=debug --soc_version=Ascend310 +``` +- 执行完模型转换脚本后，若提示如下信息说明模型转换成功（同样的，可以通过修改output参数来重命名这个.om文件） +``` +ATC run success, welcome to the next use. +``` + +经过上述操作，可以在“项目所在目录/models”找到yolov3.om模型和PoseEstNet.om模型，模型转换操作已全部完成 + +4.3 参考链接 +> 模型转换使用了ATC工具，如需更多信息请参考：[ATC工具使用指南-快速入门](https://support.huaweicloud.com/tg-cannApplicationDev330/atlasatc_16_0005.html) + +## 5 数据集 +5.1 原始VeRi数据集 + +      [Github下载链接](https://vehiclereid.github.io/VeRi/) +原数据集images文件夹下面分为images_train和images_test，需要自己将这两个文件夹里的图片复制到data_eval/images文件夹下面 +目录结构如下： +``` +├── data_eval + ├── images + | ├── 0010_c014_00034990_0.jpg + | ├── 0010_c017_00034050_1.jpg + ├── labels + | ├── label_test.csv +``` +其中data/labels中的csv文件：[Github下载链接](https://github.com/NVlabs/PAMTRI/tree/master/PoseEstNet/data/veri/annot) + +---------------------------------------------------- +## 6 测试 + +6.1 配置环境变量 + +运行cann和sdk的set_env.sh脚本 +``` +export MX_SDK_HOME=${SDK安装路径}/mxVision + +export LD_LIBRARY_PATH=${MX_SDK_HOME}/lib:${MX_SDK_HOME}/opensource/lib:${MX_SDK_HOME}/opensource/lib64:${MX_SDK_HOME}/opensource/lib64:/usr/local/Ascend/ascend-toolkit/latest/acllib/lib64:/usr/local/Ascend/driver/lib64/ + +export PYTHONPATH=${MX_SDK_HOME}/python + +export GST_PLUGIN_SCANNER=${MX_SDK_HOME}/opensource/libexec/gstreamer-1.0/gst-plugin-scanner + +export GST_PLUGIN_PATH=${MX_SDK_HOME}/opensource/lib/gstreamer-1.0:${MX_SDK_HOME}/lib/plugins +``` +6.2 获取om模型 +``` +步骤详见4： 模型转换 +``` +6.3 准备数据集 +``` +步骤详见5： 数据集 +``` +6.4 安装插件编译所需要的NumCpp库 +``` +cd plugins +git clone https://github.com/dpilger26/NumCpp +mkdir include +cp -r NumCpp/include/NumCpp ./include/ +``` +6.5 编译插件 +``` +bash build.sh +``` +6.6 切换到插件目录更新权限 +``` +cd ${MX_SDK_HOME}/lib/plugins +chmod 640 libmxpi_pnetpostprocessplugin.so +chmod 640 libmxpi_pnetpreprocessplugin.so +``` +6.7 切换回根目录，创建文件夹 +``` +cd (项目根目录) +mkdir data +mkdir data_eval +mkdir output +mkdir output_eval +``` +6.8 配置pipeline +根据所需场景，配置pipeline文件，调整路径参数等。 + +PoseEstNet.pipeline: +``` + # 配置mxpi_tensorinfer插件的yolov3.om模型加载路径（lines 26-33） + lines 26-33: + "mxpi_tensorinfer0": { + "props": { + "dataSource": "mxpi_imageresize0", + "modelPath": "models/yolov3.om(这里根据你的命名或路径进行更改)" + }, + "factory": "mxpi_tensorinfer", + "next": "mxpi_objectpostprocessor0" + }, + # 配置mxpi_objectpostprocessor插件的yolov3.cfg配置文件加载路径以及SDN的安装路径（lines 34-43） + lines 34-43: + "mxpi_objectpostprocessor0": { + "props": { + "dataSource": "mxpi_tensorinfer0", + "postProcessConfigPath": "models/yolov3.cfg(这里根据你的命名或路径进行更改)", + "labelPath": "models/coco.names", + "postProcessLibPath": "${SDK安装路径}/lib/modelpostprocessors/libyolov3postprocess.so" + }, + "factory": "mxpi_objectpostprocessor", + "next": "mxpi_imagecrop0" + }, + # 配置mxpi_tensorinfer插件的PoseEstNet.om模型加载路径（lines 68-75 以及 92-99） + lines 68-75： + "mxpi_tensorinfer2":{ + "props": { + "dataSource": "mxpi_preprocess1", + "modelPath": "models/PoseEstNet.om(这里根据你的命名或路径进行更改)" + }, + "factory":"mxpi_tensorinfer", + "next":"mxpi_postprocess1" + }, + +``` +eval_PoseEstNet.pipeline: +``` + # 配置mxpi_tensorinfer插件的PoseEstNet.om模型加载路径（lines 28-35） + lines 28-35： + "mxpi_tensorinfer2":{ + "props": { + "dataSource": "mxpi_preprocess1", + "modelPath": "models/PoseEstNet.om(这里根据你的命名或路径进行更改)" + }, + "factory":"mxpi_tensorinfer", + "next":"mxpi_postprocess1" + }, +``` + +6.9 执行 + +业务代码main.py结果在output文件夹 +``` +python3 main.py --inputPath data +``` +评估代码的具体结果在output_eval文件夹 +``` +python3 eval.py --inputPath data_eval/images/ --labelPath data_eval/labels/label_test.csv +``` + + +## 7 精度对比 + +项目精度： +![项目精度](image/output_eval.png) + +目标精度： +![目标精度](image/oral_eval.png) + diff --git a/contrib/PoseEstNet/eval.py b/contrib/PoseEstNet/eval.py new file mode 100755 index 0000000000000000000000000000000000000000..dfef46d4196b3f9022e509797e61ada75dff10f0 --- /dev/null +++ b/contrib/PoseEstNet/eval.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python +# coding=utf-8 + +""" +Copyright(C) Huawei Technologies Co.,Ltd. 2012-2022 All rights reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +""" + +import argparse +import os +import stat +import numpy as np +import csv +import math + +import MxpiDataType_pb2 as MxpiDataType +from StreamManagerApi import StreamManagerApi, MxDataInput, StringVector + +YUV_BYTES_NU = 3 +YUV_BYTES_DE = 2 + +POSEESTNET_STREAM_NAME = b'PoseEstNetProcess' +IN_PLUGIN_ID = 0 + + +def initialize_stream(): + """ + Initialize stream + :arg: + None + :return: + Stream api + """ + stream_api = StreamManagerApi() + stream_state = stream_api.InitManager() + if stream_state != 0: + error_message = "Failed to init Stream manager, stream_state=%s" % str(stream_state) + print(error_message) + exit() + + # creating stream based on json strings in the pipeline file: 'ReID.pipeline' + with open("pipeline/eval_PoseEstNet.pipeline", 'rb') as f: + pipeline = f.read().replace(b'\r', b'').replace(b'\n', b'') + pipeline_string = pipeline + + stream_state = stream_api.CreateMultipleStreams(pipeline_string) + if stream_state != 0: + error_message = "Failed to create Stream, stream_state=%s" % str(stream_state) + print(error_message) + exit() + + return stream_api + + +def get_result(input_path, label_path, stream_api): + # constructing the results returned by the stream + plugin_names = [b"mxpi_postprocess1"] + plugin_name_vector = StringVector() + for key in plugin_names: + plugin_name_vector.push_back(key) + + # check the file + if os.path.exists(input_path) != 1: + error_message = 'The file of input images does not exist.' + print(error_message) + exit() + if len(os.listdir(input_path)) == 0: + error_message = 'The file of input images is empty.' + print(error_message) + exit() + + label_csv = open(label_path) + reader = csv.reader(label_csv, delimiter=',') + hash_annot = {} + for row in reader: + img_name = row[0] + width = int(row[1]) + height = int(row[2]) + joints = [] + for j in range(36): + joint = [int(row[j * 3 + 3]), int(row[j * 3 + 4]), int(row[j * 3 + 5])] + joints.append(joint) + hash_annot[img_name] = (width, height, joints) + all_preds = np.zeros((len(hash_annot), 36, 3), dtype=np.float32) + batch_count = 0 + idx = 0 + image_names = [] + + for k in sorted(hash_annot.keys()): + image_name = k + image_names.append(image_name) + if not image_name.lower().endswith((".jpg")): + print('Input image only support jpg') + exit() + img_path = os.path.join(input_path, image_name) + queryDataInput = MxDataInput() + with open(img_path, 'rb') as f: + queryDataInput.data = f.read() + + # send the prepared data to the stream + unique_id = stream_api.SendData(POSEESTNET_STREAM_NAME, IN_PLUGIN_ID, queryDataInput) + if unique_id < 0: + error_message = 'Failed to send data to queryImageProcess stream.' + print(error_message) + exit() + + # get infer result + infer_result = stream_api.GetProtobuf(POSEESTNET_STREAM_NAME, IN_PLUGIN_ID, plugin_name_vector) + # checking whether the infer results is valid or not + if infer_result.size() == 0: + error_message = 'Unable to get effective infer results, please check the stream log for details' + print(error_message) + exit() + if infer_result[0].errorCode != 0: + error_message = "GetProtobuf error. errorCode=%d, error_message=%s" % (infer_result[0].errorCode, + infer_result[0].messageName) + print(error_message) + exit() + + # get the output + object_list = MxpiDataType.MxpiObjectList() + object_list.ParseFromString(infer_result[0].messageBuf) + for index in range(len(object_list.objectVec)): + x = object_list.objectVec[index].x0 + y = object_list.objectVec[index].y0 + vision = object_list.objectVec[index].x1 + all_preds[idx + batch_count, index, 0] = x + all_preds[idx + batch_count, index, 1] = y + all_preds[idx + batch_count, index, 2] = vision + + if batch_count == 31: + print(f'-------- Test: [{int((idx+1)/32 + 1)}/{int(len(hash_annot)/32)}] ---------') + idx += batch_count + 1 + batch_count = 0 + else: + batch_count += 1 + + # output pose in CSV format + preds_length = len(all_preds) + output_pose = os.open('output_eval/pose_test.csv', os.O_RDWR | os.O_CREAT, stat.S_IRWXU | stat.S_IRGRP) + for p in range(preds_length): + os.write(output_pose, ("%s," % (image_names[p])).encode()) + key_point_num = len(all_preds[p]) + for k in range(key_point_num-1): + os.write(output_pose, ("%.3f,%.3f,%.3f," % (all_preds[p][k][0], + all_preds[p][k][1], + all_preds[p][k][2])).encode()) + os.write(output_pose, ("%.3f,%.3f,%.3f\n" % (all_preds[p][len(all_preds[p])-1][0], + all_preds[p][len(all_preds[p])-1][1], + all_preds[p][len(all_preds[p])-1][2])).encode()) + os.close(output_pose) + + +def evaluate(label_path): + sc_bias = 0.25 + threshold = 0.5 + + preds_read = [] + with open('output_eval/pose_test.csv') as annot_file: + reader = csv.reader(annot_file, delimiter=',') + for row in reader: + joints = [] + for j in range(36): + joint = [float(row[j * 3 + 1]), float(row[j * 3 + 2]), float(row[j * 3 + 3])] + joints.append(joint) + preds_read.append(joints) + gts = [] + viss = [] + area_sqrts = [] + with open(label_path) as annot_file: + reader = csv.reader(annot_file, delimiter=',') + for row in reader: + joints = [] + vis = [] + top_lft = btm_rgt = [int(row[3]), int(row[4])] + for j in range(36): + joint = [int(row[j * 3 + 3]), int(row[j * 3 + 4]), int(row[j * 3 + 5])] + joints.append(joint) + vis.append(joint[2]) + if joint[0] < top_lft[0]: + top_lft[0] = joint[0] + if joint[1] < top_lft[1]: + top_lft[1] = joint[1] + if joint[0] > btm_rgt[0]: + btm_rgt[0] = joint[0] + if joint[1] > btm_rgt[1]: + btm_rgt[1] = joint[1] + gts.append(joints) + viss.append(vis) + area_sqrts.append(math.sqrt((btm_rgt[0] - top_lft[0] + 1) * (btm_rgt[1] - top_lft[1] + 1))) + + jnt_visible = np.array(viss, dtype=int) + jnt_visible = np.transpose(jnt_visible) + pos_pred_src = np.transpose(preds_read, [1, 2, 0]) + pos_gt_src = np.transpose(gts, [1, 2, 0]) + uv_error = pos_pred_src - pos_gt_src + uv_err = np.linalg.norm(uv_error, axis=1) + area_sqrts = np.linalg.norm(area_sqrts, axis=0) + area_sqrts *= sc_bias + scale = np.multiply(area_sqrts, np.ones((len(uv_err), 1))) + scaled_uv_err = np.divide(uv_err, scale) + scaled_uv_err = np.multiply(scaled_uv_err, jnt_visible) + jnt_count = np.sum(jnt_visible, axis=1) + less_than_threshold = np.multiply((scaled_uv_err <= threshold), + jnt_visible) + PCKh = np.divide(100. * np.sum(less_than_threshold, axis=1), jnt_count) + + # save + rng = np.arange(0, 0.5 + 0.01, 0.01) + pck_all = np.zeros((len(rng), 36)) + + length_rng = len(rng) + for r in range(length_rng): + threshold = rng[r] + less_than_threshold = np.multiply((scaled_uv_err <= threshold), + jnt_visible) + pck_all[r, :] = np.divide(100. * np.sum(less_than_threshold, axis=1), + jnt_count) + + PCKh = np.ma.array(PCKh, mask=False) + jnt_count = np.ma.array(jnt_count, mask=False) + jnt_ratio = jnt_count / np.sum(jnt_count).astype(np.float64) + + name_value = { + 'Wheel': (1.0 / 4.0) * (PCKh[0] + PCKh[1] + PCKh[18] + PCKh[19]), + 'Fender': (1.0 / 16.0) * (PCKh[2] + PCKh[3] + PCKh[4] + PCKh[5] + PCKh[6] + PCKh[7] + PCKh[8] + + PCKh[9] + PCKh[20] + PCKh[21] + PCKh[22] + PCKh[23] + PCKh[24] + + PCKh[25] + PCKh[26] + PCKh[27]), + 'Back': (1.0 / 4.0) * (PCKh[10] + PCKh[11] + PCKh[28] + PCKh[29]), + 'Front': (1.0 / 4.0) * (PCKh[16] + PCKh[17] + PCKh[34] + PCKh[35]), + 'WindshieldBack': (1.0 / 4.0) * (PCKh[12] + PCKh[13] + PCKh[30] + PCKh[31]), + 'WindshieldFront': (1.0 / 4.0) * (PCKh[14] + PCKh[15] + PCKh[32] + PCKh[33]), + 'Mean': np.sum(PCKh * jnt_ratio), + 'Mean@0.1': np.sum(pck_all[11, :] * jnt_ratio) + } + + _print_name_value(name_value, 'PoseEstNet') + + +def _print_name_value(name_value, full_arch_name): + ''' print accuracy ''' + names = name_value.keys() + values = name_value.values() + print( + '| Arch ' + + ' '.join(['| {}'.format(name) for name in names]) + + ' |' + ) + if len(full_arch_name) > 15: + full_arch_name = full_arch_name[:8] + '...' + print( + '| ' + full_arch_name + ' ' + + ' '.join(['| {:.3f}'.format(value) for value in values]) + + ' |' + ) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--inputPath', type=str, default='data_eval/images', help="Query File Path") + parser.add_argument('--labelPath', type=str, default='data_eval/labels/label_test.csv', help="Gallery File Path") + opt = parser.parse_args() + streamManagerApi = initialize_stream() + get_result(opt.inputPath, opt.labelPath, streamManagerApi) + evaluate(opt.labelPath) \ No newline at end of file diff --git a/contrib/PoseEstNet/image/oral_eval.png b/contrib/PoseEstNet/image/oral_eval.png new file mode 100644 index 0000000000000000000000000000000000000000..f54cda403a99cdd2b43d809446b548477ac664f9 Binary files /dev/null and b/contrib/PoseEstNet/image/oral_eval.png differ diff --git a/contrib/PoseEstNet/image/output_eval.png b/contrib/PoseEstNet/image/output_eval.png new file mode 100644 index 0000000000000000000000000000000000000000..dec5b150a9af122d2cba39d17cda7f1df21059cd Binary files /dev/null and b/contrib/PoseEstNet/image/output_eval.png differ diff --git a/contrib/PoseEstNet/image/plugin_tree.png b/contrib/PoseEstNet/image/plugin_tree.png new file mode 100644 index 0000000000000000000000000000000000000000..6993314f8183b3a0e55a392a651c2a1cb1241741 Binary files /dev/null and b/contrib/PoseEstNet/image/plugin_tree.png differ diff --git a/contrib/PoseEstNet/image/project_tree.png b/contrib/PoseEstNet/image/project_tree.png new file mode 100644 index 0000000000000000000000000000000000000000..0155d3138768bd12cdf2579f4c425fb605a46278 Binary files /dev/null and b/contrib/PoseEstNet/image/project_tree.png differ diff --git a/contrib/PoseEstNet/main.py b/contrib/PoseEstNet/main.py new file mode 100755 index 0000000000000000000000000000000000000000..3043ea10aa4ca252966f1a0e11353b267a601313 --- /dev/null +++ b/contrib/PoseEstNet/main.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python +# coding=utf-8 + +""" +Copyright(C) Huawei Technologies Co.,Ltd. 2012-2022 All rights reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +""" + +import argparse +import os +import cv2 + +import MxpiDataType_pb2 as MxpiDataType +from StreamManagerApi import StreamManagerApi, MxDataInput, StringVector + +YUV_BYTES_NU = 3 +YUV_BYTES_DE = 2 + +POSTESTNET_STREAM_NAME = b'PoseEstNetProcess' +IN_PLUGIN_ID = 0 + + +def initialize_stream(): + """ + Initialize stream + + :arg: + None + + :return: + Stream api + """ + stream_api = StreamManagerApi() + stream_state = stream_api.InitManager() + if stream_state != 0: + error_message = "Failed to init Stream manager, stream_state=%s" % str(stream_state) + print(error_message) + exit() + + # creating stream based on json strings in the pipeline file: 'PoseEstNet.pipeline' + with open("pipeline/PoseEstNet.pipeline", 'rb') as f: + pipeline = f.read().replace(b'\r', b'').replace(b'\n', b'') + pipeline_string = pipeline + + stream_state = stream_api.CreateMultipleStreams(pipeline_string) + if stream_state != 0: + error_message = "Failed to create Stream, stream_state=%s" % str(stream_state) + print(error_message) + exit() + + return stream_api + + +def process(input_path, stream_api): + """ + :arg: + queryPath: the directory of query images + streamApi: stream api + + :return: + queryFeatures: the vectors of queryFeatures + queryPid: the vectors of queryPid + """ + + # constructing the results returned by the queryImageProcess stream + plugin_names = [b"mxpi_distributor0_0", b"mxpi_postprocess1"] + + plugin_name_vector = StringVector() + for key in plugin_names: + plugin_name_vector.push_back(key) + + # check the query file + if os.path.exists(input_path) != 1: + error_message = 'The file of input images does not exist.' + print(error_message) + exit() + if len(os.listdir(input_path)) == 0: + error_message = 'The file of input images is empty.' + print(error_message) + exit() + + # extract the features for all images in query file + for root, dirs, files in os.walk(input_path): + for file in files: + if not file.endswith('.jpg'): + print('Input image only support jpg') + exit() + + query_data_input = MxDataInput() + file_path = os.path.join(root, file) + with open(file_path, 'rb') as f: + query_data_input.data = f.read() + + # send the prepared data to the stream + unique_id = stream_api.SendData(POSTESTNET_STREAM_NAME, IN_PLUGIN_ID, query_data_input) + if unique_id < 0: + error_message = 'Failed to send data to queryImageProcess stream.' + print(error_message) + exit() + + # get infer result + infer_result = stream_api.GetProtobuf(POSTESTNET_STREAM_NAME, IN_PLUGIN_ID, plugin_name_vector) + + # checking whether the infer results is valid or not + if infer_result.size() == 0: + error_message = 'Unable to get effective infer results, please check the stream log for details' + print(error_message) + exit() + if infer_result[0].errorCode != 0: + error_message = "GetProtobuf error. errorCode=%d, error_message=%s" % (infer_result[0].errorCode, + infer_result[0].messageName) + print(error_message) + exit() + + # get the output information of "mxpi_objectpostprocessor0" plugin + car_object_list = MxpiDataType.MxpiObjectList() + car_object_list.ParseFromString(infer_result[0].messageBuf) + car_num = len(car_object_list.objectVec) + + # get the output information of "mxpi_objectpostprocessor0" plugin + keypoint_object_list = MxpiDataType.MxpiObjectList() + keypoint_object_list.ParseFromString(infer_result[1].messageBuf) + keypoint_num = len(keypoint_object_list.objectVec) + if keypoint_num // car_num != 36: + error_message = 'Failed to map the inferred key points to the detected cars.' + print(error_message) + exit() + + original_img = cv2.imread(file_path) + for index in range(len(keypoint_object_list.objectVec)): + original_x = int(car_object_list.objectVec[index // 36].x0) + keypoint_object_list.objectVec[index].x0 + original_y = int(car_object_list.objectVec[index // 36].y0) + keypoint_object_list.objectVec[index].y0 + visible = keypoint_object_list.objectVec[index].x1 + if round(visible): + cv2.circle(original_img, (int(original_x), int(original_y)), 2, [255, 0, 0], 2) + cv2.imwrite("output/result_{}".format(str(file)), original_img) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--inputPath', type=str, default='data') + opt = parser.parse_args() + streamManagerApi = initialize_stream() + process(opt.inputPath, streamManagerApi) \ No newline at end of file diff --git a/contrib/PoseEstNet/models/aipp_hrnet_256_256.aippconfig b/contrib/PoseEstNet/models/aipp_hrnet_256_256.aippconfig new file mode 100644 index 0000000000000000000000000000000000000000..7806914defc33b6242169dad9aea924a45bbf0ac --- /dev/null +++ b/contrib/PoseEstNet/models/aipp_hrnet_256_256.aippconfig @@ -0,0 +1,32 @@ +aipp_op { + aipp_mode: static + related_input_rank: 0 + src_image_size_w: 256 + src_image_size_h: 256 + crop: false + input_format: RGB888_U8 + csc_switch: false + rbuv_swap_switch: false + matrix_r0c0 : 256 + matrix_r0c1 : 0 + matrix_r0c2 : 359 + matrix_r1c0 : 256 + matrix_r1c1 : -88 + matrix_r1c2 : -183 + matrix_r2c0 : 256 + matrix_r2c1 : 454 + matrix_r2c2 : 0 + input_bias_0 : 0 + input_bias_1 : 128 + input_bias_2 : 128 + # 归一化和标准化 + mean_chn_0: 124 + mean_chn_1: 116 + mean_chn_2: 104 + min_chn_0: 0.0 + min_chn_1: 0.0 + min_chn_2: 0.0 + var_reci_chn_0: 0.0171 + var_reci_chn_1: 0.0175 + var_reci_chn_2: 0.0174 +} diff --git a/contrib/PoseEstNet/models/aipp_nv12.cfg b/contrib/PoseEstNet/models/aipp_nv12.cfg new file mode 100644 index 0000000000000000000000000000000000000000..0ed9a631496c1a4b32789890c259af6769312122 --- /dev/null +++ b/contrib/PoseEstNet/models/aipp_nv12.cfg @@ -0,0 +1,38 @@ +aipp_op{ +aipp_mode:static +crop:false +rbuv_swap_switch:true +input_format : YUV420SP_U8 +load_start_pos_h : 0 +load_start_pos_w : 0 +src_image_size_w : 416 +src_image_size_h : 416 + +csc_switch : true + +matrix_r0c0 : 298 +matrix_r0c1 : 516 +matrix_r0c2 : 0 +matrix_r1c0 : 298 +matrix_r1c1 : -100 +matrix_r1c2 : -208 +matrix_r2c0 : 298 +matrix_r2c1 : 0 +matrix_r2c2 : 409 +input_bias_0 : 16 +input_bias_1 : 128 +input_bias_2 : 128 +mean_chn_0 : 0 +mean_chn_1 : 0 +mean_chn_2 : 0 +min_chn_0 : 0.0 +min_chn_1 : 0.0 +min_chn_2 : 0.0 + +var_reci_chn_0 :0.003921568627451 +var_reci_chn_1 :0.003921568627451 +var_reci_chn_2 :0.003921568627451 + + +} + diff --git a/contrib/PoseEstNet/models/coco.names b/contrib/PoseEstNet/models/coco.names new file mode 100644 index 0000000000000000000000000000000000000000..8f143661cf3397afac6b1513c25c9211778fb6d1 --- /dev/null +++ b/contrib/PoseEstNet/models/coco.names @@ -0,0 +1,81 @@ +# This file is originally from https://github.com/pjreddie/darknet/blob/master/data/coco.names +person +bicycle +car +motorbike +aeroplane +bus +train +truck +boat +traffic light +fire hydrant +stop sign +parking meter +bench +bird +cat +dog +horse +sheep +cow +elephant +bear +zebra +giraffe +backpack +umbrella +handbag +tie +suitcase +frisbee +skis +snowboard +sports ball +kite +baseball bat +baseball glove +skateboard +surfboard +tennis racket +bottle +wine glass +cup +fork +knife +spoon +bowl +banana +apple +sandwich +orange +broccoli +carrot +hot dog +pizza +donut +cake +chair +sofa +pottedplant +bed +diningtable +toilet +tvmonitor +laptop +mouse +remote +keyboard +cell phone +microwave +oven +toaster +sink +refrigerator +book +clock +vase +scissors +teddy bear +hair drier +toothbrush \ No newline at end of file diff --git a/contrib/PoseEstNet/models/yolov3.cfg b/contrib/PoseEstNet/models/yolov3.cfg new file mode 100644 index 0000000000000000000000000000000000000000..e6ad98df26124dea5ce3255697e61e1537533dc3 --- /dev/null +++ b/contrib/PoseEstNet/models/yolov3.cfg @@ -0,0 +1,10 @@ +CLASS_NUM=80 +BIASES_NUM=18 +BIASES=10,13,16,30,33,23,30,61,62,45,59,119,116,90,156,198,373,326 +SCORE_THRESH=0.4 +OBJECTNESS_THRESH=0.8 +IOU_THRESH=0.5 +YOLO_TYPE=3 +ANCHOR_DIM=3 +MODEL_TYPE=0 +RESIZE_FLAG=0 diff --git a/contrib/PoseEstNet/pipeline/PoseEstNet.pipeline b/contrib/PoseEstNet/pipeline/PoseEstNet.pipeline new file mode 100755 index 0000000000000000000000000000000000000000..c4cb28aeae912e053e35c1daf28180614291ed1d --- /dev/null +++ b/contrib/PoseEstNet/pipeline/PoseEstNet.pipeline @@ -0,0 +1,92 @@ +{ + "PoseEstNetProcess": { + "stream_config": { + "deviceId": "0" + }, + "appsrc0": { + "props": { + "blocksize": "409600" + }, + "factory": "appsrc", + "next": "mxpi_imagedecoder0" + }, + "mxpi_imagedecoder0": { + "factory": "mxpi_imagedecoder", + "next": "mxpi_imageresize0" + }, + "mxpi_imageresize0": { + "props": { + "dataSource": "mxpi_imagedecoder0", + "resizeHeight": "416", + "resizeWidth": "416" + }, + "factory": "mxpi_imageresize", + "next": "mxpi_tensorinfer0" + }, + "mxpi_tensorinfer0": { + "props": { + "dataSource": "mxpi_imageresize0", + "modelPath": "models/yolov3.om(这里根据你的命名或路径进行更改)" + }, + "factory": "mxpi_tensorinfer", + "next": "mxpi_objectpostprocessor0" + }, + "mxpi_objectpostprocessor0": { + "props": { + "dataSource": "mxpi_tensorinfer0", + "postProcessConfigPath": "models/yolov3.cfg(这里根据你的命名或路径进行更改)", + "labelPath": "models/coco.names", + "postProcessLibPath": "\${SDK安装路径}/lib/modelpostprocessors/libyolov3postprocess.so" + }, + "factory": "mxpi_objectpostprocessor", + "next": "mxpi_distributor0" + }, + "mxpi_distributor0": { + "props": { + "dataSource": "mxpi_objectpostprocessor0", + "classIds": "2" + }, + "factory": "mxpi_distributor", + "next": "mxpi_imagecrop0" + }, + "mxpi_imagecrop0": { + "props": { + "dataSource": "mxpi_distributor0_0", + "dataSourceImage": "mxpi_imagedecoder0" + }, + "factory": "mxpi_imagecrop", + "next": "mxpi_preprocess1" + }, + "mxpi_preprocess1": { + "props":{ + "dataSource":"mxpi_imagecrop0", + "outputFormat": "RGB" + }, + "factory": "mxpi_pnetpreprocessplugin", + "next": "mxpi_tensorinfer2" + }, + "mxpi_tensorinfer2":{ + "props": { + "dataSource": "mxpi_preprocess1", + "modelPath": "models/PoseEstNet.om(这里根据你的命名或路径进行更改)" + }, + "factory":"mxpi_tensorinfer", + "next":"mxpi_postprocess1" + }, + "mxpi_postprocess1":{ + "props": { + "dataSource":"mxpi_tensorinfer2", + "InfoSource":"mxpi_imagecrop0", + "descriptionMessage": "Don’t panic!" + }, + "factory":"mxpi_pnetpostprocessplugin", + "next":"appsink0" + }, + "appsink0": { + "props": { + "blocksize": "4096000" + }, + "factory": "appsink" + } + } +} \ No newline at end of file diff --git a/contrib/PoseEstNet/pipeline/eval_PoseEstNet.pipeline b/contrib/PoseEstNet/pipeline/eval_PoseEstNet.pipeline new file mode 100755 index 0000000000000000000000000000000000000000..f269b20edb932b34f1510061a40f0e4d67e0cb10 --- /dev/null +++ b/contrib/PoseEstNet/pipeline/eval_PoseEstNet.pipeline @@ -0,0 +1,52 @@ +{ + "PoseEstNetProcess": { + "stream_config": { + "deviceId": "0" + }, + "appsrc0": { + "props": { + "blocksize": "409600" + }, + "factory": "appsrc", + "next": "mxpi_imagedecoder0" + }, + "mxpi_imagedecoder0": { + "props": { + "deviceId": "0" + }, + "factory": "mxpi_imagedecoder", + "next": "mxpi_preprocess1" + }, + "mxpi_preprocess1": { + "props":{ + "dataSource":"mxpi_imagedecoder0", + "outputFormat": "RGB" + }, + "factory": "mxpi_pnetpreprocessplugin", + "next": "mxpi_tensorinfer2" + }, + "mxpi_tensorinfer2":{ + "props": { + "dataSource": "mxpi_preprocess1", + "modelPath": "models/PoseEstNet.om(这里根据你的命名或路径进行更改)" + }, + "factory":"mxpi_tensorinfer", + "next":"mxpi_postprocess1" + }, + "mxpi_postprocess1":{ + "props": { + "dataSource":"mxpi_tensorinfer2", + "InfoSource":"mxpi_imagedecoder0", + "descriptionMessage": "Don’t panic!" + }, + "factory":"mxpi_pnetpostprocessplugin", + "next":"appsink0" + }, + "appsink0": { + "props": { + "blocksize": "4096000" + }, + "factory": "appsink" + } + } +} \ No newline at end of file diff --git a/contrib/PoseEstNet/plugins/._postprocess b/contrib/PoseEstNet/plugins/._postprocess new file mode 100644 index 0000000000000000000000000000000000000000..e28521c38eac08999d8964d32fdb2bc4189a61dc Binary files /dev/null and b/contrib/PoseEstNet/plugins/._postprocess differ diff --git a/contrib/PoseEstNet/plugins/._preprocess b/contrib/PoseEstNet/plugins/._preprocess new file mode 100644 index 0000000000000000000000000000000000000000..e28521c38eac08999d8964d32fdb2bc4189a61dc Binary files /dev/null and b/contrib/PoseEstNet/plugins/._preprocess differ diff --git a/contrib/PoseEstNet/plugins/build.sh b/contrib/PoseEstNet/plugins/build.sh new file mode 100755 index 0000000000000000000000000000000000000000..f444148ba7ca28d8ff821e7ec7c4c2602b8adf20 --- /dev/null +++ b/contrib/PoseEstNet/plugins/build.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Copyright 2022 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e +current_folder="$( cd "$(dirname "$0")" ;pwd -P )" + +SAMPLE_FOLDER=( + /postprocess/ + /preprocess/ +) + + +err_flag=0 +for sample in ${SAMPLE_FOLDER[@]};do + cd ${current_folder}/${sample} + bash build.sh || { + echo -e "Failed to build ${sample}" + err_flag=1 + } +done + + +if [ ${err_flag} -eq 1 ]; then + exit 1 +fi +exit 0 \ No newline at end of file diff --git a/contrib/PoseEstNet/plugins/postprocess/CMakeLists.txt b/contrib/PoseEstNet/plugins/postprocess/CMakeLists.txt new file mode 100755 index 0000000000000000000000000000000000000000..2021046045d58982f6006af7dc2d6fb9371a63e7 --- /dev/null +++ b/contrib/PoseEstNet/plugins/postprocess/CMakeLists.txt @@ -0,0 +1,35 @@ +cmake_minimum_required(VERSION 3.5.2) +project(MxpiPNetPostprocess) + +add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0) +add_definitions(-Dgoogle=mindxsdk_private) +add_definitions(-DNUMCPP_NO_USE_BOOST) + +set(PLUGIN_NAME "mxpi_pnetpostprocessplugin") +set(TARGET_LIBRARY ${PLUGIN_NAME}) +set(MX_SDK_HOME $ENV{MX_SDK_HOME}) +set(LIBRARY_OUTPUT_PATH ${MX_SDK_HOME}/lib/plugins) +set(CMAKE_CXX_STANDARD 14) + +include_directories(${MX_SDK_HOME}/include) +include_directories(${MX_SDK_HOME}/opensource/include) +include_directories(${MX_SDK_HOME}/opensource/include/gstreamer-1.0) +include_directories(${MX_SDK_HOME}/opensource/include/glib-2.0) +include_directories(${MX_SDK_HOME}/opensource/lib/glib-2.0/include) +include_directories(${MX_SDK_HOME}/opensource/include/opencv4) +include_directories(../include) + +link_directories(${MX_SDK_HOME}/lib) +link_directories(${MX_SDK_HOME}/opensource/lib) + +add_compile_options(-std=c++11 -fPIC -fstack-protector-all -pie -Wno-deprecated-declarations -g) +add_compile_options("-DPLUGIN_NAME=${PLUGIN_NAME}") + +add_definitions(-DENABLE_DVPP_INTERFACE) + +add_library(${TARGET_LIBRARY} SHARED MxpiPNetPostprocess.cpp) +target_link_libraries(${TARGET_LIBRARY} glib-2.0 gstreamer-1.0 gobject-2.0 gstbase-1.0 gmodule-2.0 glog) + + +target_link_libraries(${TARGET_LIBRARY} mxpidatatype plugintoolkit mxbase mindxsdk_protobuf) +target_link_libraries(${TARGET_LIBRARY} -Wl,-z,relro,-z,now,-z,noexecstack -s) diff --git a/contrib/PoseEstNet/plugins/postprocess/MxpiPNetPostprocess.cpp b/contrib/PoseEstNet/plugins/postprocess/MxpiPNetPostprocess.cpp new file mode 100755 index 0000000000000000000000000000000000000000..f76cd27cb6becc81a6be2c2e18aff0d230205b8d --- /dev/null +++ b/contrib/PoseEstNet/plugins/postprocess/MxpiPNetPostprocess.cpp @@ -0,0 +1,382 @@ +/* + * Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "MxpiPNetPostprocess.h" +#include "MxBase/Log/Log.h" +#include "MxBase/Tensor/TensorBase/TensorBase.h" + + +using namespace MxBase; +using namespace MxPlugins; +using namespace MxTools; +using namespace std; +namespace { + const string SAMPLE_KEY = "MxpiTensorPackageList"; + const string INFO_KEY = "MxpiVisionList"; + const int BIN_WIDTH_IN_DEGREES = 3; + const int ER = 2; + const int SAN = 3; + const float LING_DIAN_ER_WU = 0.25; + const float LING_DIAN_WU = 0.5; +} + +// Unpack MxpiTensorPackageList +void GetTensors(const MxTools::MxpiTensorPackageList tensorPackageList, + std::vector &tensors) { + for (int i = 0; i < tensorPackageList.tensorpackagevec_size(); ++i) { + for (int j = 0; j < tensorPackageList.tensorpackagevec(i).tensorvec_size(); j++) { + MxBase::MemoryData memoryData = {}; + memoryData.deviceId = tensorPackageList.tensorpackagevec(i).tensorvec(j).deviceid(); + memoryData.type = (MxBase::MemoryData::MemoryType)tensorPackageList. + tensorpackagevec(i).tensorvec(j).memtype(); + memoryData.size = (uint32_t) tensorPackageList. + tensorpackagevec(i).tensorvec(j).tensordatasize(); + memoryData.ptrData = (void *) tensorPackageList. + tensorpackagevec(i).tensorvec(j).tensordataptr(); + std::vector outputShape = {}; + for (int k = 0; k < tensorPackageList. + tensorpackagevec(i).tensorvec(j).tensorshape_size(); ++k) { + outputShape.push_back((uint32_t) tensorPackageList. + tensorpackagevec(i).tensorvec(j).tensorshape(k)); + } + MxBase::TensorBase tmpTensor(memoryData, true, outputShape, + (MxBase::TensorDataType)tensorPackageList. + tensorpackagevec(i).tensorvec(j).tensordatatype()); + tensors.push_back(tmpTensor); + } + } +} + +APP_ERROR MxpiPNetPostprocess::Init(std::map>& configParamMap) { + LogInfo << "MxpiPNetPostprocess::Init start."; + APP_ERROR ret = APP_ERR_OK; + // Get the property values by key + std::shared_ptr parentNamePropSptr = std::static_pointer_cast(configParamMap["dataSource"]); + parentName_ = *parentNamePropSptr.get(); + std::shared_ptr infoPlugPropSptr = std::static_pointer_cast(configParamMap["InfoSource"]); + infoPlugName_ = *infoPlugPropSptr.get(); + std::shared_ptr descriptionMessageProSptr = + std::static_pointer_cast(configParamMap["descriptionMessage"]); + descriptionMessage_ = *descriptionMessageProSptr.get(); + LogInfo << "MxpiPNetPostprocess::Init complete."; + return APP_ERR_OK; +} + +APP_ERROR MxpiPNetPostprocess::DeInit() { + LogInfo << "MxpiPNetPostprocess::DeInit start."; + LogInfo << "MxpiPNetPostprocess::DeInit complete."; + return APP_ERR_OK; +} + +APP_ERROR MxpiPNetPostprocess::SetMxpiErrorInfo(MxpiBuffer& buffer, const std::string pluginName, + const MxpiErrorInfo mxpiErrorInfo) { + APP_ERROR ret = APP_ERR_OK; + // Define an object of MxpiMetadataManager + MxpiMetadataManager mxpiMetadataManager(buffer); + ret = mxpiMetadataManager.AddErrorInfo(pluginName, mxpiErrorInfo); + if (ret != APP_ERR_OK) { + LogError << "Failed to AddErrorInfo."; + return ret; + } + ret = SendData(0, buffer); + return ret; +} + +APP_ERROR MxpiPNetPostprocess::GenerateHeadPoseInfo(const MxpiTensorPackageList srcMxpiTensorPackage, + const MxpiVisionList srcMxpiVisionPackage, + MxpiObjectList& dstMxpiObjectListSptr) { + // Get Tensors + std::vector tensors = {}; + GetTensors(srcMxpiTensorPackage, tensors); + + auto tensor_size = tensors.size(); + for (int i = 0; i < tensor_size; i++){ + auto dataPtr = (float *)tensors[i].GetBuffer(); + auto tensor_shape = tensors[i].GetShape(); + + auto orignal_width = srcMxpiVisionPackage.visionvec()[i].visioninfo().width(); + auto orignal_height = srcMxpiVisionPackage.visionvec()[i].visioninfo().height(); + int keypoint_nums, hm_h, hm_w; + keypoint_nums = tensor_shape[1]; + hm_h = tensor_shape[ER]; + hm_w = tensor_shape[SAN]; + + nc::NdArray coordinate; + coordinate = nc::zeros(keypoint_nums, ER); + nc::NdArray maxval; + maxval = nc::zeros(keypoint_nums, 1); + + for (auto j = 0; j < keypoint_nums; j++){ + nc::NdArray heatmap; + heatmap = nc::zeros(1, hm_h*hm_w); + for (auto k = 0; k < hm_h*hm_w; k++){ + heatmap(0, k) = dataPtr[j*hm_h*hm_w + k]; + } + auto index = nc::argmax(heatmap); + auto max_value = nc::max(heatmap); + auto hm_x = index(0, 0) % hm_w; + auto hm_y = floor(index(0, 0) / hm_w); + coordinate(j, 0) = hm_x; + coordinate(j, 1) = hm_y; + maxval(j, 0) = max_value(0, 0); + + auto px = floor(coordinate(j, 0) + 0.5); + auto py = floor(coordinate(j, 1) + 0.5); + heatmap = heatmap.reshape(hm_h, hm_w); + + if ((1 < px < hm_w - 1) and (1 < py < hm_h - 1)){ + auto diff_x = heatmap(py, px + 1) - heatmap(py, px - 1); + auto diff_y = heatmap(py + 1, px) - heatmap(py - 1, px); + if( diff_x > 0) { + coordinate(j, 0) = coordinate(j, 0) + LING_DIAN_ER_WU; + } else if( diff_x < 0 ) { + coordinate(j, 0) = coordinate(j, 0) - LING_DIAN_ER_WU; + } else { + coordinate(j, 0) = coordinate(j, 0); + } + if ( diff_y > 0) { + coordinate(j, 1) = coordinate(j, 1) + LING_DIAN_ER_WU; + } else if ( diff_y < 0 ) { + coordinate(j, 1) = coordinate(j, 1) - LING_DIAN_ER_WU; + } else { + coordinate(j, 1) = coordinate(j, 1); + } + } + } + + auto pixel_std = 200; + nc::NdArray center = {float(orignal_width / 2.0), float(orignal_height / 2.0)}; + float scale = MAX(orignal_width, orignal_height) * 1.25 / pixel_std; + + int w_h[2] = { hm_w, hm_h }; + nc::NdArray final_coordinate = transform_preds(coordinate, center, scale, w_h); + + for (auto it = 0; it < keypoint_nums; it++){ + auto dstMxpiHObjectInfoPtr = dstMxpiObjectListSptr.add_objectvec(); + MxpiMetaHeader* dstMxpiMetaHeaderList = dstMxpiHObjectInfoPtr->add_headervec(); + dstMxpiMetaHeaderList->set_datasource(parentName_); + dstMxpiMetaHeaderList->set_memberid(0); + dstMxpiHObjectInfoPtr->set_x0(final_coordinate(it, 0)); + dstMxpiHObjectInfoPtr->set_y0(final_coordinate(it, 1)); + dstMxpiHObjectInfoPtr->set_x1(maxval(it, 0)); + dstMxpiHObjectInfoPtr->set_y1(0); + } + } + return APP_ERR_OK; +} + + +nc::NdArray MxpiPNetPostprocess::transform_preds(nc::NdArray coords, nc::NdArray center, + float scale, int output_size[2]) { + nc::NdArray target_coords = nc::zeros(coords.shape()); + nc::NdArray target_coords_temp; + + cv::Mat trans = get_affine_transform(center, scale, 0, output_size, {0, 0}, 1); + nc::NdArray trans_NdArray = nc::zeros(trans.rows, trans.cols); + double* ptr_data = (double*)trans.data; + for (int i = 0; i < trans.rows; i++) { + for (int j = 0; j < trans.cols; j++) { + trans_NdArray(i, j) = (float)ptr_data[i * trans.cols + j]; + } + } + for (int p = 0; p < coords.shape().rows; p++) { + target_coords_temp = nc::copy(affine_transform(coords(p, {0, 2}), trans_NdArray)); + for (int q = 0; q < ER; q++) { + target_coords(p, q) = target_coords_temp(q, 0); + } + } + return target_coords; +} + +nc::NdArray MxpiPNetPostprocess::affine_transform(nc::NdArray pt, nc::NdArray t) { + nc::NdArray new_pt = {pt(0, 0), pt(0, 1), 1.0}; + new_pt = new_pt.transpose(); + nc::NdArray new_pt_dot = nc::dot(t, new_pt); + nc::NdArray my_pt_dot = new_pt_dot({0, 2}, 0); + return my_pt_dot; +} + +cv::Mat MxpiPNetPostprocess::get_affine_transform(nc::NdArray center, float scale, float rot, + int output_size[2], nc::NdArray shift = {0, 0}, int inv = 0) { + nc::NdArray scales = { scale * 200, scale * 200 }; + nc::NdArray scale_tmp = scales; + float src_w = scale_tmp[0]; + int dst_w = output_size[0]; + int dst_h = output_size[1]; + float rot_rad = nc::constants::pi * rot / 180; + nc::NdArray src_dir = get_dir(nc::NdArray { 0, float(src_w * -0.5) }, rot_rad); + nc::NdArray dst_dir = { 0, float(dst_w * -0.5) }; + nc::NdArray src = nc::zeros(3, 2); + nc::NdArray dst = nc::zeros(3, 2); + nc::NdArray temp; + + temp = center + scale_tmp * shift; + src(0, 0) = temp(0, 0); + src(0, 1) = temp(0, 1); + temp = center + src_dir + scale_tmp * shift; + src(1, 0) = temp(0, 0); + src(1, 1) = temp(0, 1); + temp = dst_w * LING_DIAN_WU, dst_h * LING_DIAN_WU; + dst(0, 0) = temp(0, 0); + dst(0, 1) = temp(0, 1); + temp = nc::NdArray {float(dst_w * 0.5), float(dst_h * 0.5)} + dst_dir; + dst(1, 0) = temp(0, 0); + dst(1, 1) = temp(0, 1); + temp = get_3rd_point(src(0, src.cSlice()), src(1, src.cSlice())); + src(ER, 0) = temp(0, 0); + src(ER, 1) = temp(0, 1); + temp = get_3rd_point(dst(0, dst.cSlice()), dst(1, dst.cSlice())); + dst(ER, 0) = temp(0, 0); + dst(ER, 1) = temp(0, 1); + + cv::Mat trans; + cv::Point2f SRC[3]; + cv::Point2f DST[3]; + SRC[0] = cv::Point2f(src(0, 0), src(0, 1)); + SRC[1] = cv::Point2f(src(1, 0), src(1, 1)); + SRC[ER] = cv::Point2f(src(ER, 0), src(ER, 1)); + DST[0] = cv::Point2f(dst(0, 0), dst(0, 1)); + DST[1] = cv::Point2f(dst(1, 0), dst(1, 1)); + DST[ER] = cv::Point2f(dst(ER, 0), dst(ER, 1)); + if (1 == inv) { + trans = cv::getAffineTransform(DST, SRC); + } + else + { + trans = cv::getAffineTransform(SRC, DST); + } + return trans; +} + +nc::NdArray MxpiPNetPostprocess::get_dir(nc::NdArray src_point, float rot_rad) { + float sn = nc::sin(rot_rad); + float cs = nc::cos(rot_rad); + nc::NdArray src_result = {0, 0}; + src_result[0] = src_point[0] * cs - src_point[1] * sn; + src_result[1] = src_point[0] * sn + src_point[1] * cs; + return src_result; +} + +nc::NdArray MxpiPNetPostprocess::get_3rd_point(nc::NdArray a, nc::NdArray b) { + nc::NdArray direct = a - b; + nc::NdArray c = b + nc::NdArray { -direct[1], direct[0] }; + return c; +} + +APP_ERROR MxpiPNetPostprocess::Process(std::vector& mxpiBuffer) { + LogInfo << "MxpiPNetPostprocess::Process start"; + MxpiBuffer* buffer = mxpiBuffer[0]; + MxpiMetadataManager mxpiMetadataManager(*buffer); + MxpiErrorInfo mxpiErrorInfo; + ErrorInfo_.str(""); + auto errorInfoPtr = mxpiMetadataManager.GetErrorInfo(); + if (errorInfoPtr != nullptr) { + ErrorInfo_ << GetError(APP_ERR_COMM_FAILURE, pluginName_) << "MxpiPNetPostprocess process is not implemented"; + mxpiErrorInfo.ret = APP_ERR_COMM_FAILURE; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + LogError << "MxpiPNetPostprocess process is not implemented"; + return APP_ERR_COMM_FAILURE; + } + // Get the data (infer tensor) from buffer + shared_ptr metadata = mxpiMetadataManager.GetMetadata(parentName_); + if (metadata == nullptr) { + ErrorInfo_ << GetError(APP_ERR_METADATA_IS_NULL, pluginName_) << "Metadata of tensor is NULL, failed"; + mxpiErrorInfo.ret = APP_ERR_METADATA_IS_NULL; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return APP_ERR_METADATA_IS_NULL; // self define the error code + } + // Check the proto struct name + google::protobuf::Message* msg = (google::protobuf::Message*)metadata.get(); + const google::protobuf::Descriptor* desc = msg->GetDescriptor(); + if (desc->name() != SAMPLE_KEY) { + ErrorInfo_ << GetError(APP_ERR_PROTOBUF_NAME_MISMATCH, pluginName_) + << "Proto struct name is not MxpiTensorPackageList, failed"; + mxpiErrorInfo.ret = APP_ERR_PROTOBUF_NAME_MISMATCH; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return APP_ERR_PROTOBUF_NAME_MISMATCH; // self define the error code + } + LogInfo << "MxpiPNetPostprocess::Get Tensor Completed"; + + // Get the data (image information) from buffer + shared_ptr Info_metadata = mxpiMetadataManager.GetMetadata(infoPlugName_); + if (Info_metadata == nullptr) { + ErrorInfo_ << GetError(APP_ERR_METADATA_IS_NULL, infoPlugName_) << "Metadata of original image is NULL, failed"; + mxpiErrorInfo.ret = APP_ERR_METADATA_IS_NULL; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, infoPlugName_, mxpiErrorInfo); + return APP_ERR_METADATA_IS_NULL; // self define the error code + } + // Check the proto struct name + google::protobuf::Message* info_msg = (google::protobuf::Message*)Info_metadata.get(); + const google::protobuf::Descriptor* info_desc = info_msg->GetDescriptor(); + if (info_desc->name() != INFO_KEY) { + ErrorInfo_ << GetError(APP_ERR_PROTOBUF_NAME_MISMATCH, infoPlugName_) + << "Proto struct name is not MxpiTensorPackageList, failed"; + mxpiErrorInfo.ret = APP_ERR_PROTOBUF_NAME_MISMATCH; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, infoPlugName_, mxpiErrorInfo); + return APP_ERR_PROTOBUF_NAME_MISMATCH; // self define the error code + } + LogInfo << "MxpiPNetPostprocess::Get Image Info Completed"; + + // Generate WHENet output + shared_ptr srcMxpiTensorPackageListSptr = static_pointer_cast(metadata); + shared_ptr srcMxpiVisionListSptr = static_pointer_cast(Info_metadata); + shared_ptr dstMxpiObjectListSptr = make_shared(); + APP_ERROR ret = GenerateHeadPoseInfo(*srcMxpiTensorPackageListSptr, *srcMxpiVisionListSptr, *dstMxpiObjectListSptr); + if (ret != APP_ERR_OK) { + LogError << GetError(ret, pluginName_) << "MxpiPNetPostprocess gets inference information failed."; + mxpiErrorInfo.ret = ret; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return ret; + } + + ret = mxpiMetadataManager.AddProtoMetadata(pluginName_, static_pointer_cast(dstMxpiObjectListSptr)); + if (ret != APP_ERR_OK) { + ErrorInfo_ << GetError(ret, pluginName_) << "MxpiPNetPostprocess add metadata failed."; + mxpiErrorInfo.ret = ret; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return ret; + } + // Send the data to downstream plugin + SendData(0, *buffer); + LogInfo << "MxpiPNetPostprocess::Process end"; + return APP_ERR_OK; +} + +std::vector> MxpiPNetPostprocess::DefineProperties() { + // Define an A to store properties + std::vector> properties; + // Set the type and related information of the properties, and the key is the name + auto parentNameProSptr = std::make_shared> (ElementProperty{ + STRING, "dataSource", "name", "the name of previous plugin", "mxpi_tensorinfer2", "NULL", "NULL"}); + auto infoPlugProSptr = std::make_shared> (ElementProperty{ + STRING, "InfoSource", "name", "the name of needed decoder/crop plugin", "mxpi_imagedecoder0", "NULL", "NULL"}); + auto descriptionMessageProSptr = std::make_shared> (ElementProperty{ + STRING, "descriptionMessage", "message", "Description mesasge of plugin", "This is MxpiSamplePlugin", "NULL", "NULL"}); + properties.push_back(parentNameProSptr); + properties.push_back(infoPlugProSptr); + properties.push_back(descriptionMessageProSptr); + return properties; +} + +// Register the Sample plugin through macro +MX_PLUGIN_GENERATE(MxpiPNetPostprocess) \ No newline at end of file diff --git a/contrib/PoseEstNet/plugins/postprocess/MxpiPNetPostprocess.h b/contrib/PoseEstNet/plugins/postprocess/MxpiPNetPostprocess.h new file mode 100755 index 0000000000000000000000000000000000000000..fa82dede96828d07e24997980a3f5339a22d01d8 --- /dev/null +++ b/contrib/PoseEstNet/plugins/postprocess/MxpiPNetPostprocess.h @@ -0,0 +1,111 @@ +/* + * Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef SDKMEMORY_MXPIPNETPOSTPROCESS_H +#define SDKMEMORY_MXPIPNETPOSTPROCESS_H + +#include "MxTools/PluginToolkit/base/MxPluginGenerator.h" +#include "MxTools/PluginToolkit/base/MxPluginBase.h" +#include "MxTools/PluginToolkit/metadata/MxpiMetadataManager.h" +#include "MxTools/Proto/MxpiDataType.pb.h" +#include "MxBase/PostProcessBases/PostProcessDataType.h" +#include "MxBase/ErrorCode/ErrorCode.h" +#include "opencv2/opencv.hpp" +#include + +#if defined(__cpp_lib_math_special_functions) || !defined(DNUMCPP_NO_USE_BOOST) +#include "NumCpp/Core/Internal/StaticAsserts.hpp" +#include "NumCpp/Core/Internal/StlAlgorithms.hpp" +#include "NumCpp/NdArray.hpp" +#include "NumCpp/Functions.hpp" +#endif +#include + +#ifndef __cpp_lib_math_special_functions +#include +#else +#include "boost/math/special_functions/bessel.hpp" +#endif + +/** +* @api +* @brief Definition of MxpiHeadPosePlugin class. +*/ + +namespace MxPlugins { + class MxpiPNetPostprocess : public MxTools::MxPluginBase { + public: + /** + * @api + * @brief Initialize configure parameter. + * @param configParamMap + * @return APP_ERROR + */ + APP_ERROR Init(std::map>& configParamMap) override; + /** + * @api + * @brief DeInitialize configure parameter. + * @return APP_ERROR + */ + APP_ERROR DeInit() override; + /** + * @api + * @brief Process the data of MxpiBuffer. + * @param mxpiBuffer + * @return APP_ERROR + */ + APP_ERROR Process(std::vector& mxpiBuffer) override; + /** + * @api + * @brief Definition the parameter of configure properties. + * @return std::vector> + */ + static std::vector> DefineProperties(); + /** + * @api + * @brief Get the number of class id and confidence from model inference. + * @param key + * @param buffer + * @return APP_ERROR + */ + APP_ERROR GenerateHeadPoseInfo(const MxTools::MxpiTensorPackageList srcMxpiTensorPackage, + const MxTools::MxpiVisionList srcMxpiVisionPackage, + MxTools::MxpiObjectList& dstMxpiHeadPoseListSptr); + + nc::NdArray transform_preds(nc::NdArray coords, nc::NdArray center, + float scale, int output_size[2]); + + cv::Mat get_affine_transform(nc::NdArray center, float scale, float rot, int output_size[2], + nc::NdArray shift, int inv); + + nc::NdArray get_dir(nc::NdArray src_point, float rot_rad); + + nc::NdArray get_3rd_point(nc::NdArray a, nc::NdArray b); + + nc::NdArray affine_transform(nc::NdArray pt, nc::NdArray t); + + private: + APP_ERROR SetMxpiErrorInfo(MxTools::MxpiBuffer& buffer, const std::string pluginName, + const MxTools::MxpiErrorInfo mxpiErrorInfo); + std::string parentName_; + std::string infoPlugName_; + std::string descriptionMessage_; + std::ostringstream ErrorInfo_; + }; +} + + +#endif // SDKMEMORY_MXPIPNETPOSTPROCESS_H diff --git a/contrib/PoseEstNet/plugins/postprocess/build.sh b/contrib/PoseEstNet/plugins/postprocess/build.sh new file mode 100755 index 0000000000000000000000000000000000000000..162319a9dd81ae4abff73650c25a1cc3ce4a1a2b --- /dev/null +++ b/contrib/PoseEstNet/plugins/postprocess/build.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright 2022 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License.mitations under the License. + +set -e + +current_folder="$( cd "$(dirname "$0")" ;pwd -P )" + +function build_plugin() { + build_path=$current_folder/build + if [ -d "$build_path" ]; then + rm -rf "$build_path" + else + echo "file $build_path is not exist." + fi + mkdir -p "$build_path" + cd "$build_path" + cmake .. + make -j + cd .. + exit 0 +} + +build_plugin +exit 0 diff --git a/contrib/PoseEstNet/plugins/preprocess/CMakeLists.txt b/contrib/PoseEstNet/plugins/preprocess/CMakeLists.txt new file mode 100755 index 0000000000000000000000000000000000000000..56e2a811bd42c52ec9902f945477d9093b249e9a --- /dev/null +++ b/contrib/PoseEstNet/plugins/preprocess/CMakeLists.txt @@ -0,0 +1,35 @@ +cmake_minimum_required(VERSION 3.5.2) +project(MxpiPNetPreprocess) + +add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0) +add_definitions(-Dgoogle=mindxsdk_private) +add_definitions(-DNUMCPP_NO_USE_BOOST) + +set(PLUGIN_NAME "mxpi_pnetpreprocessplugin") +set(TARGET_LIBRARY ${PLUGIN_NAME}) +set(MX_SDK_HOME $ENV{MX_SDK_HOME}) +set(LIBRARY_OUTPUT_PATH ${MX_SDK_HOME}/lib/plugins) +set(CMAKE_CXX_STANDARD 14) + +include_directories(${MX_SDK_HOME}/include) +include_directories(${MX_SDK_HOME}/opensource/include) +include_directories(${MX_SDK_HOME}/opensource/include/gstreamer-1.0) +include_directories(${MX_SDK_HOME}/opensource/include/glib-2.0) +include_directories(${MX_SDK_HOME}/opensource/lib/glib-2.0/include) +include_directories(${MX_SDK_HOME}/opensource/include/opencv4) +include_directories(../include) + +link_directories(${MX_SDK_HOME}/lib) +link_directories(${MX_SDK_HOME}/opensource/lib) + +add_compile_options(-std=c++11 -fPIC -fstack-protector-all -pie -Wno-deprecated-declarations -g) +add_compile_options("-DPLUGIN_NAME=${PLUGIN_NAME}") + +add_definitions(-DENABLE_DVPP_INTERFACE) + +add_library(${TARGET_LIBRARY} SHARED MxpiPNetPreprocess.cpp) +target_link_libraries(${TARGET_LIBRARY} glib-2.0 gstreamer-1.0 gobject-2.0 gstbase-1.0 gmodule-2.0 glog) + + +target_link_libraries(${TARGET_LIBRARY} mxpidatatype plugintoolkit mxbase mindxsdk_protobuf) +target_link_libraries(${TARGET_LIBRARY} -Wl,-z,relro,-z,now,-z,noexecstack -s) \ No newline at end of file diff --git a/contrib/PoseEstNet/plugins/preprocess/MxpiPNetPreprocess.cpp b/contrib/PoseEstNet/plugins/preprocess/MxpiPNetPreprocess.cpp new file mode 100755 index 0000000000000000000000000000000000000000..f97d67ecc705bc91ef2047822bd1e6a7bed787fc --- /dev/null +++ b/contrib/PoseEstNet/plugins/preprocess/MxpiPNetPreprocess.cpp @@ -0,0 +1,457 @@ +/* + * Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "MxpiPNetPreprocess.h" +#include "MxBase/Log/Log.h" +#include "MxBase/Tensor/TensorBase/TensorBase.h" + + +using namespace MxBase; +using namespace MxPlugins; +using namespace MxTools; +using namespace std; +namespace { + const string VISION_KEY = "MxpiVisionList"; + const mindxsdk_private::protobuf::uint32 MXBASE_PIXEL_FORMAT_YUV_SEMIPLANAR_420 = 1; + const mindxsdk_private::protobuf::uint32 MXBASE_PIXEL_FORMAT_RGB_888 = 12; + const mindxsdk_private::protobuf::uint32 MXBASE_PIXEL_FORMAT_BGR_888 = 13; + const int YUV_U = 2; + const int YUV_V = 3; + const int ER = 2; + const int ER_WU_LIU = 256 + const float LING_DIAN_WU = 0.5 +} + + +APP_ERROR MxpiPNetPreprocess::Init(std::map>& configParamMap) { + LogInfo << "MxpiPoseEstNetPrepostPlugin::Init start."; + APP_ERROR ret = APP_ERR_OK; + // Get the property values by key + std::shared_ptr parentNamePropSptr = std::static_pointer_cast(configParamMap["dataSource"]); + parentName_ = *parentNamePropSptr.get(); + std::shared_ptr outputFormatPropSptr = std::static_pointer_cast(configParamMap["outputFormat"]); + outputDataFormat = *outputFormatPropSptr.get(); + std::shared_ptr descriptionMessageProSptr = std::static_pointer_cast(configParamMap["descriptionMessage"]); + descriptionMessage_ = *descriptionMessageProSptr.get(); + LogInfo << "MxpiPoseEstNetPrepostPlugin::Init complete."; + return APP_ERR_OK; +} + + +APP_ERROR MxpiPNetPreprocess::DeInit() { + LogInfo << "MxpiPNetPreprocess::DeInit start."; + LogInfo << "MxpiPNetPreprocess::DeInit complete."; + return APP_ERR_OK; +} + + +APP_ERROR MxpiPNetPreprocess::SetMxpiErrorInfo(MxpiBuffer& buffer, const std::string pluginName, + const MxpiErrorInfo mxpiErrorInfo) { + APP_ERROR ret = APP_ERR_OK; + // Define an object of MxpiMetadataManager + MxpiMetadataManager mxpiMetadataManager(buffer); + ret = mxpiMetadataManager.AddErrorInfo(pluginName, mxpiErrorInfo); + if (ret != APP_ERR_OK) { + LogError << "Failed to AddErrorInfo."; + return ret; + } + ret = SendData(0, buffer); + return ret; +} + + +APP_ERROR MxpiPNetPreprocess::GenerateVisionList(const MxpiVisionList srcMxpiVisionList, + MxpiVisionList& dstMxpiVisionList) { + LogInfo <<"input type:" <CopyFrom(dstVision); + } + if (dstMxpiVisionList.visionvec_size() == 0) { + LogError << "element("<< elementName_<<") dst vision vec size is 0!"; + return APP_ERR_COMM_FAILURE; + } + LogInfo << "Generate done"; + return APP_ERR_OK; +} + + +APP_ERROR MxpiPNetPreprocess::DoAffineTransform(size_t idx, + const MxTools::MxpiVision srcMxpiVision, + MxTools::MxpiVision& dstMxpiVision) { + LogInfo << "Till now is ok"; + auto &visionInfo = srcMxpiVision.visioninfo(); + auto &visionData = srcMxpiVision.visiondata(); + + MemoryData memorySrc = {}; + memorySrc.deviceId = visionData.deviceid(); + memorySrc.type = (MxBase::MemoryData::MemoryType)visionData.memtype(); + memorySrc.size = visionData.datasize(); + memorySrc.ptrData = (void *)visionData.dataptr(); + MemoryData memoryDst(visionData.datasize(), MemoryData::MEMORY_HOST_NEW); + APP_ERROR ret = MxBase::MemoryHelper::MxbsMallocAndCopy(memoryDst, memorySrc); + if (ret != APP_ERR_OK) { + LogError << "Fail to malloc and copy host memory."; + return ret; + } + + int orignal_width; + int orignal_height; + + cv::Mat src; + cv::Mat imgBGR; + cv::Mat imgRGB; + + // check the format of the input image, decode it accordingly. + outputPixelFormat_ = visionInfo.format(); + LogInfo << outputPixelFormat_; + if (outputPixelFormat_ == MXBASE_PIXEL_FORMAT_YUV_SEMIPLANAR_420) { + LogInfo << "the format of input image is YUV, using dvpp to decode."; + imgRGB = cv::Mat(visionInfo.height(), visionInfo.width(), CV_8UC3); + orignal_width = visionInfo.width(); + orignal_height = visionInfo.height(); + + cv::Mat transedImgRBG = cv::Mat(visionInfo.heightaligned(), visionInfo.widthaligned(), CV_8UC3); + src = cv::Mat(visionInfo.heightaligned()* YUV_V / YUV_U, visionInfo.widthaligned(), CV_8UC1, memoryDst.ptrData); + cv::cvtColor(src, transedImgRBG, cv::COLOR_YUV2RGB_NV12); + + vector cropRange; + cropRange.push_back(0); // x_0 + cropRange.push_back(0); // y_0 + cropRange.push_back(orignal_width); // x_1 + cropRange.push_back(orignal_height); // y_1 + crop_img(transedImgRBG, imgRGB, cropRange); + } else { + imgRGB = cv::Mat(visionInfo.heightaligned(), visionInfo.widthaligned(), CV_8UC3); + orignal_width = visionInfo.widthaligned(); + orignal_height = visionInfo.heightaligned(); + if (outputPixelFormat_ == MXBASE_PIXEL_FORMAT_RGB_888) { + LogInfo << "the format of input image is RGB, using opencv to decode."; + imgRGB = cv::Mat(visionInfo.heightaligned(), visionInfo.widthaligned(), CV_8UC3, memoryDst.ptrData); + } else if (outputPixelFormat_ == MXBASE_PIXEL_FORMAT_BGR_888) { + LogInfo << "the format of input image is BGR, using opencv to decode."; + src = cv::Mat(visionInfo.heightaligned(), visionInfo.widthaligned(), CV_8UC3, memoryDst.ptrData); + cv::cvtColor(src, imgRGB, cv::COLOR_BGR2RGB); + } + } + + cv::Mat dst; + MxBase::MemoryData memoryNewDst(dst.data, MxBase::MemoryData::MEMORY_HOST_NEW); + auto pixel_std = 200; + nc::NdArray center = {float(orignal_width / 2.0), float(orignal_height / 2.0)}; + float scale = MAX(orignal_width, orignal_height) * 1.25 / pixel_std; + int outputSize[2] = { ER_WU_LIU, ER_WU_LIU }; + cv::Size dstSize = { ER_WU_LIU, ER_WU_LIU }; + cv::Mat trans = get_affine_transform(center, scale, 0, outputSize, {0, 0}, 0); + cv::warpAffine(imgRGB, dst, trans, dstSize); + + cv::Mat imgYuv; + cv::Mat yuv_mat; + cv::Mat img_nv12; + if (outputDataFormat == "YUV") { + LogInfo << "output in yuv"; + imgYuv = cv::Mat(ER_WU_LIU, ER_WU_LIU, CV_8UC1); + cv::cvtColor(dst, imgYuv, cv::COLOR_RGB2YUV_I420); + yuv_mat = imgYuv.clone(); + Bgr2Yuv(yuv_mat, img_nv12); + ret = Mat2MxpiVisionDvpp(idx, img_nv12, dstMxpiVision); + } else { + if (outputDataFormat == "RGB") { + LogInfo <<"output in rgb"; + ret = Mat2MxpiVisionOpencv(idx, dst, dstMxpiVision); + } else { + LogInfo <<"output in bgr"; + imgBGR = cv::Mat(ER_WU_LIU, ER_WU_LIU, CV_8UC3); + cv::cvtColor(dst, imgBGR, cv::COLOR_RGB2BGR); + ret = Mat2MxpiVisionOpencv(idx, imgBGR, dstMxpiVision); + } + } + + if (ret != APP_ERR_OK) { + LogError << "convert mat to mxvision failed!"; + return ret; + } + LogInfo << "affine_transform done"; + return APP_ERR_OK; +} + + +APP_ERROR MxpiPNetPreprocess::Mat2MxpiVisionOpencv(size_t idx, const cv::Mat& mat, MxTools::MxpiVision& vision) { + LogInfo << "Mat2MxpiVision begin"; + auto header = vision.add_headervec(); + header->set_memberid(idx); + header->set_datasource(parentName_); + + auto visionInfo = vision.mutable_visioninfo(); + if (outputDataFormat == "YUV") { + visionInfo->set_format(MXBASE_PIXEL_FORMAT_YUV_SEMIPLANAR_420); + } else { + if (outputDataFormat == "RGB") { + visionInfo->set_format(MXBASE_PIXEL_FORMAT_RGB_888); + } else { + visionInfo->set_format(MXBASE_PIXEL_FORMAT_BGR_888); + } + } + + visionInfo->set_height(mat.rows); + visionInfo->set_heightaligned(mat.rows); + visionInfo->set_width(mat.cols); + visionInfo->set_widthaligned(mat.cols); + auto visionData = vision.mutable_visiondata(); + LogInfo << "elemSize = " << mat.elemSize(); + LogInfo << "col = " << mat.cols; + LogInfo << "rows = " << mat.rows; + LogInfo << "size = " << mat.size(); + visionData->set_datasize(mat.cols * mat.rows * mat.elemSize()); + MemoryData memoryDataDst(visionData->datasize(), MemoryData::MEMORY_HOST, deviceId_); + MemoryData memoryDataStr(mat.data, visionData->datasize(), MemoryData::MEMORY_HOST_MALLOC); + APP_ERROR ret = MemoryHelper::MxbsMallocAndCopy(memoryDataDst, memoryDataStr); + if (ret != APP_ERR_OK) { + LogError << "copy memory error."; + return ret; + } + visionData->set_dataptr((uint64)memoryDataDst.ptrData); + visionData->set_deviceid(deviceId_); + visionData->set_memtype(MxTools::MXPI_MEMORY_HOST); + visionData->set_datatype(MxTools::MxpiDataType::MXPI_DATA_TYPE_UINT8); + LogInfo << "Mat2MxpiVision done"; + return APP_ERR_OK; +} + + +// area is the upper left corner coordinate and width height of the cutting area +void MxpiPNetPreprocess::crop_img(cv::Mat &img, cv::Mat &crop_img, std::vector &area) { + int crop_x1 = std::max(0, area[0]); + int crop_y1 = std::max(0, area[1]); + int crop_x2 = std::min(img.cols -1, area[0] + area[2] - 1); + int crop_y2 = std::min(img.rows - 1, area[1] + area[3] - 1); + + crop_img = img(cv::Range(crop_y1, crop_y2+1), cv::Range(crop_x1, crop_x2 + 1)); +} + + +APP_ERROR MxpiPNetPreprocess::Bgr2Yuv(cv::Mat &yuv_mat, cv::Mat &img_nv12) { + uint8_t *yuv = yuv_mat.ptr(); + img_nv12 = cv::Mat(ER_WU_LIU * YUV_V / YUV_U, ER_WU_LIU, CV_8UC1); + uint8_t *ynv12 = img_nv12.ptr(); + int32_t uv_height = 256 / 2; + int32_t uv_width = 256 / 2; + int32_t y_size = 256 * 256; + memcpy(ynv12, yuv, y_size); + uint8_t *nv12 = ynv12 + y_size; + uint8_t *u_data = yuv + y_size; + uint8_t *v_data = u_data + uv_height * uv_width; + for (int32_t i = 0; i < uv_width * uv_height; i++) { + *nv12++ = *u_data++; + *nv12++ = *v_data++; + } + return APP_ERR_OK; +} + + +APP_ERROR MxpiPNetPreprocess::Mat2MxpiVisionDvpp(size_t idx, const cv::Mat& mat, MxTools::MxpiVision& vision) { + LogInfo << "Mat2MxpiVision begin"; + auto header = vision.add_headervec(); + header->set_memberid(idx); + header->set_datasource(parentName_); + + auto visionInfo = vision.mutable_visioninfo(); + visionInfo->set_format(outputPixelFormat_); + visionInfo->set_height(mat.rows*YUV_U/YUV_V); + visionInfo->set_heightaligned(mat.rows*YUV_U/YUV_V); + visionInfo->set_width(mat.cols); + visionInfo->set_widthaligned(mat.cols); + + auto visionData = vision.mutable_visiondata(); + LogInfo << "elemSize = " << mat.elemSize(); + LogInfo << "col = " << mat.cols; + LogInfo << "rows = " << mat.rows; + LogInfo << "size = " << mat.size(); + visionData->set_datasize(mat.cols * mat.rows * mat.elemSize()); + MemoryData memoryDataDst(visionData->datasize(), MemoryData::MEMORY_DVPP, deviceId_); + MemoryData memoryDataStr(mat.data, visionData->datasize(), MemoryData::MEMORY_HOST_MALLOC); + APP_ERROR ret = MemoryHelper::MxbsMallocAndCopy(memoryDataDst, memoryDataStr); + if (ret != APP_ERR_OK) { + LogError << "copy memory error."; + return ret; + } + visionData->set_dataptr((uint64)memoryDataDst.ptrData); + visionData->set_deviceid(deviceId_); + visionData->set_memtype(MxTools::MXPI_MEMORY_DVPP); + visionData->set_datatype(MxTools::MxpiDataType::MXPI_DATA_TYPE_UINT8); + LogInfo << "Mat2MxpiVision done"; + return APP_ERR_OK; +}; + + +cv::Mat MxpiPNetPreprocess::get_affine_transform(nc::NdArray center, float scale, float rot, + int output_size[2], nc::NdArray shift = {0, 0}, int inv = 0) { + nc::NdArray scales = { scale * 200, scale * 200 }; + nc::NdArray scale_tmp = scales; + float src_w = scale_tmp[0]; + int dst_w = output_size[0]; + int dst_h = output_size[1]; + float rot_rad = nc::constants::pi * rot / 180; + nc::NdArray src_dir = get_dir(nc::NdArray { 0, float(src_w * -0.5) }, rot_rad); + nc::NdArray dst_dir = { 0, float(dst_w * -0.5) }; + nc::NdArray src = nc::zeros(3, 2); + nc::NdArray dst = nc::zeros(3, 2); + nc::NdArray temp; + + temp = center + scale_tmp * shift; + src(0, 0) = temp(0, 0); + src(0, 1) = temp(0, 1); + temp = center + src_dir + scale_tmp * shift; + src(1, 0) = temp(0, 0); + src(1, 1) = temp(0, 1); + temp = dst_w * LING_DIAN_WU, dst_h * LING_DIAN_WU; + dst(0, 0) = temp(0, 0); + dst(0, 1) = temp(0, 1); + temp = nc::NdArray {float(dst_w * LING_DIAN_WU), float(dst_h * LING_DIAN_WU)} + dst_dir; + dst(1, 0) = temp(0, 0); + dst(1, 1) = temp(0, 1); + temp = get_3rd_point(src(0, src.cSlice()), src(1, src.cSlice())); + src(ER, 0) = temp(0, 0); + src(ER, 1) = temp(0, 1); + temp = get_3rd_point(dst(0, dst.cSlice()), dst(1, dst.cSlice())); + dst(ER, 0) = temp(0, 0); + dst(ER, 1) = temp(0, 1); + + cv::Mat trans; + cv::Point2f SRC[3]; + cv::Point2f DST[3]; + SRC[0] = cv::Point2f(src(0, 0), src(0, 1)); + SRC[1] = cv::Point2f(src(1, 0), src(1, 1)); + SRC[ER] = cv::Point2f(src(ER, 0), src(ER, 1)); + DST[0] = cv::Point2f(dst(0, 0), dst(0, 1)); + DST[1] = cv::Point2f(dst(1, 0), dst(1, 1)); + DST[ER] = cv::Point2f(dst(ER, 0), dst(ER, 1)); + if (1 == inv) { + trans = cv::getAffineTransform(DST, SRC); + } + else + { + trans = cv::getAffineTransform(SRC, DST); + } + return trans; +} + + +nc::NdArray MxpiPNetPreprocess::get_dir(nc::NdArray src_point, float rot_rad) { + float sn = nc::sin(rot_rad); + float cs = nc::cos(rot_rad); + nc::NdArray src_result = {0, 0}; + src_result[0] = src_point[0] * cs - src_point[1] * sn; + src_result[1] = src_point[0] * sn + src_point[1] * cs; + return src_result; +} + + +nc::NdArray MxpiPNetPreprocess::get_3rd_point(nc::NdArray a, nc::NdArray b) { + nc::NdArray direct = a - b; + nc::NdArray c = b + nc::NdArray { -direct[1], direct[0] }; + return c; +} + + +APP_ERROR MxpiPNetPreprocess::Process(std::vector& mxpiBuffer) { + LogInfo << "MxpiPNetPreprocess::Process start"; + MxpiBuffer* buffer = mxpiBuffer[0]; + MxpiMetadataManager mxpiMetadataManager(*buffer); + MxpiErrorInfo mxpiErrorInfo; + ErrorInfo_.str(""); + auto errorInfoPtr = mxpiMetadataManager.GetErrorInfo(); + if (errorInfoPtr != nullptr) { + ErrorInfo_ << GetError(APP_ERR_COMM_FAILURE, pluginName_) << "MxpiPNetPreprocess process is not implemented"; + mxpiErrorInfo.ret = APP_ERR_COMM_FAILURE; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + LogError << "MxpiPNetPreprocess process is not implemented"; + return APP_ERR_COMM_FAILURE; + } + // Get the (image) data from buffer + shared_ptr metadata = mxpiMetadataManager.GetMetadata(parentName_); + if (metadata == nullptr) { + ErrorInfo_ << GetError(APP_ERR_METADATA_IS_NULL, pluginName_) << "Metadata of tensor is NULL, failed"; + mxpiErrorInfo.ret = APP_ERR_METADATA_IS_NULL; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return APP_ERR_METADATA_IS_NULL; // self define the error code + } + // Check the proto struct name + google::protobuf::Message* msg = (google::protobuf::Message*)metadata.get(); + const google::protobuf::Descriptor* desc = msg->GetDescriptor(); + if (desc->name() != VISION_KEY) { + ErrorInfo_ << GetError(APP_ERR_PROTOBUF_NAME_MISMATCH, pluginName_) + << "Proto struct name is not MxpiVisionList, failed"; + mxpiErrorInfo.ret = APP_ERR_PROTOBUF_NAME_MISMATCH; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return APP_ERR_PROTOBUF_NAME_MISMATCH; // self define the error code + } + LogInfo << "MxpiPNetPreprocess::Get Original Image Completed"; + + // Generate WHENet output + shared_ptr srcMxpiVisionListSptr = static_pointer_cast(metadata); + shared_ptr dstMxpiVisionListSptr = make_shared(); + APP_ERROR ret = GenerateVisionList(*srcMxpiVisionListSptr, *dstMxpiVisionListSptr); + if (ret != APP_ERR_OK) { + LogError << GetError(ret, pluginName_) << "MxpiPNetPreprocess gets inference information failed."; + mxpiErrorInfo.ret = ret; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return ret; + } + + ret = mxpiMetadataManager.AddProtoMetadata(pluginName_, static_pointer_cast(dstMxpiVisionListSptr)); + if (ret != APP_ERR_OK) { + ErrorInfo_ << GetError(ret, pluginName_) << "MxpiPNetPreprocess add metadata failed."; + mxpiErrorInfo.ret = ret; + mxpiErrorInfo.errorInfo = ErrorInfo_.str(); + SetMxpiErrorInfo(*buffer, pluginName_, mxpiErrorInfo); + return ret; + } + // Send the data to downstream plugin + SendData(0, *buffer); + LogInfo << "MxpiPNetPreprocess::Process end"; + return APP_ERR_OK; +} + + +std::vector> MxpiPNetPreprocess::DefineProperties() { + // Define an A to store properties + std::vector> properties; + // Set the type and related information of the properties, and the key is the name + auto parentNameProSptr = std::make_shared>(ElementProperty { + STRING, "dataSource", "name", "the name of previous plugin", "mxpi_imagedecoder0", "NULL", "NULL"}); + auto outputPixelFormatProSptr = std::make_shared>(ElementProperty { + STRING, "outputFormat", "name", "the format of output image", "YUV", "NULL", "NULL"}); + auto descriptionMessageProSptr = std::make_shared>(ElementProperty { + STRING, "descriptionMessage", "message", "Description mesasge of plugin", "This is MxpiSamplePlugin", "NULL", "NULL"}); + properties.push_back(parentNameProSptr); + properties.push_back(outputPixelFormatProSptr); + properties.push_back(descriptionMessageProSptr); + + return properties; +} + +// Register the Sample plugin through macro +MX_PLUGIN_GENERATE(MxpiPNetPreprocess) \ No newline at end of file diff --git a/contrib/PoseEstNet/plugins/preprocess/MxpiPNetPreprocess.h b/contrib/PoseEstNet/plugins/preprocess/MxpiPNetPreprocess.h new file mode 100755 index 0000000000000000000000000000000000000000..af6f82e57af40b6ee4130fd09b114d1aa4c5dbbd --- /dev/null +++ b/contrib/PoseEstNet/plugins/preprocess/MxpiPNetPreprocess.h @@ -0,0 +1,118 @@ +/* + * Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef SDKMEMORY_MXPIPNETPREPROCESS_H +#define SDKMEMORY_MXPIPNETPREPROCESS_H +#include "MxTools/PluginToolkit/base/MxPluginGenerator.h" +#include "MxTools/PluginToolkit/base/MxPluginBase.h" +#include "MxTools/PluginToolkit/metadata/MxpiMetadataManager.h" +#include "MxTools/Proto/MxpiDataType.pb.h" +#include "MxBase/PostProcessBases/PostProcessDataType.h" +#include "MxBase/ErrorCode/ErrorCode.h" +#include "opencv2/opencv.hpp" +#include +#include + +#if defined(__cpp_lib_math_special_functions) || !defined(DNUMCPP_NO_USE_BOOST) +#include "NumCpp/Core/Internal/StaticAsserts.hpp" +#include "NumCpp/Core/Internal/StlAlgorithms.hpp" +#include "NumCpp/NdArray.hpp" +#include "NumCpp/Functions.hpp" +#endif +#include + +#ifndef __cpp_lib_math_special_functions +#include +#else +#include "boost/math/special_functions/bessel.hpp" +#endif + + +/** +* @api +* @brief Definition of MxpiPoseEstNetPrepostPlugin class. +*/ +namespace MxPlugins { + class MxpiPNetPreprocess : public MxTools::MxPluginBase { + public: + /** + * @api + * @brief Initialize configure parameter. + * @param configParamMap + * @return APP_ERROR + */ + APP_ERROR Init(std::map>& configParamMap) override; + /** + * @api + * @brief DeInitialize configure parameter. + * @return APP_ERROR + */ + APP_ERROR DeInit() override; + /** + * @api + * @brief Process the data of MxpiBuffer. + * @param mxpiBuffer + * @return APP_ERROR + */ + APP_ERROR Process(std::vector& mxpiBuffer) override; + /** + * @api + * @brief Definition the parameter of configure properties. + * @return std::vector> + */ + static std::vector> DefineProperties(); + /** + * @api + * @brief Get the number of class id and confidence from model inference. + * @param key + * @param buffer + * @return APP_ERROR + */ + + APP_ERROR GenerateVisionList(const MxTools::MxpiVisionList srcMxpiVisionList, + MxTools::MxpiVisionList& dstMxpiVisionList); + + APP_ERROR Mat2MxpiVisionOpencv(size_t idx, const cv::Mat& mat, MxTools::MxpiVision& vision); + + APP_ERROR Bgr2Yuv(cv::Mat &yuv_mat, cv::Mat &img_nv12); + + APP_ERROR Mat2MxpiVisionDvpp(size_t idx, const cv::Mat& mat, MxTools::MxpiVision& vision); + + void crop_img(cv::Mat &img, cv::Mat &crop_img, std::vector &area); + + APP_ERROR DoAffineTransform(size_t idx, const MxTools::MxpiVision srcMxpiVision, + MxTools::MxpiVision& dstMxpiVision); + + cv::Mat get_affine_transform(nc::NdArray center, float scale, float rot, int output_size[2], + nc::NdArray shift, int inv); + + nc::NdArray get_dir(nc::NdArray src_point, float rot_rad); + + nc::NdArray get_3rd_point(nc::NdArray a, nc::NdArray b); + + private: + APP_ERROR SetMxpiErrorInfo(MxTools::MxpiBuffer& buffer, const std::string pluginName, + const MxTools::MxpiErrorInfo mxpiErrorInfo); + std::string parentName_; + std::string descriptionMessage_; + std::string outputDataFormat; + std::ostringstream ErrorInfo_; + mindxsdk_private::protobuf::uint32 outputPixelFormat_; + }; +} + + +#endif // SDKMEMORY_MXPIPNETPREPROCESS_H diff --git a/contrib/PoseEstNet/plugins/preprocess/build.sh b/contrib/PoseEstNet/plugins/preprocess/build.sh new file mode 100755 index 0000000000000000000000000000000000000000..162319a9dd81ae4abff73650c25a1cc3ce4a1a2b --- /dev/null +++ b/contrib/PoseEstNet/plugins/preprocess/build.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright 2022 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License.mitations under the License. + +set -e + +current_folder="$( cd "$(dirname "$0")" ;pwd -P )" + +function build_plugin() { + build_path=$current_folder/build + if [ -d "$build_path" ]; then + rm -rf "$build_path" + else + echo "file $build_path is not exist." + fi + mkdir -p "$build_path" + cd "$build_path" + cmake .. + make -j + cd .. + exit 0 +} + +build_plugin +exit 0 diff --git a/contrib/PoseEstNet/pth2onnx.py b/contrib/PoseEstNet/pth2onnx.py new file mode 100644 index 0000000000000000000000000000000000000000..237cbc8e60914bb6814b9789010e66499f1e9df2 --- /dev/null +++ b/contrib/PoseEstNet/pth2onnx.py @@ -0,0 +1,134 @@ +# ------------------------------------------------------------------------------ +# pose.pytorch +# Copyright (c) 2018-present Microsoft +# Licensed under The Apache-2.0 License [see LICENSE for details] +# Written by Bin Xiao (Bin.Xiao@microsoft.com) +# ------------------------------------------------------------------------------ + +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import argparse +import os +import pprint + +import torch +import torch.nn.parallel +import torch.backends.cudnn as cudnn +import torch.optim +import torch.utils.data +import torch.utils.data.distributed +import torchvision.transforms as transforms + +import _init_paths +from config import cfg +from config import update_config +from core.loss import JointsMSELoss +from core.function import validate, output_preds +from utils.utils import create_logger + +import dataset +import models + + +def parse_args(): + parser = argparse.ArgumentParser(description='Train keypoints network') + # general + parser.add_argument('--cfg', + help='experiment configure file name', + required=True, + type=str) + + parser.add_argument('opts', + help="modify config options using the command-line", + default=None, + nargs=argparse.REMAINDER) + + parser.add_argument('--evaluate', + help="flag of evaluating with ground-truth labels", + default=False, + action='store_true') + parser.add_argument('--outputPreds', + help="flag of output predicted pose (keypoints), heatmaps and segments", + default=False, + action='store_true') + + parser.add_argument('--modelDir', + help='model directory', + type=str, + default='') + parser.add_argument('--logDir', + help='log directory', + type=str, + default='') + parser.add_argument('--dataDir', + help='data directory', + type=str, + default='') + parser.add_argument('--prevModelDir', + help='prev model directory', + type=str, + default='') + parser.add_argument('--outputDir', + help='output directory', + type=str, + default='') + + args = parser.parse_args() + return args + + +def main(): + args = parse_args() + update_config(cfg, args) + + logger, final_output_dir, tb_log_dir = create_logger( + cfg, args.cfg, 'valid') + + logger.info(pprint.pformat(args)) + logger.info(cfg) + + # cudnn related setting + cudnn.benchmark = cfg.CUDNN.BENCHMARK + torch.backends.cudnn.deterministic = cfg.CUDNN.DETERMINISTIC + torch.backends.cudnn.enabled = cfg.CUDNN.ENABLED + + model = eval('models.'+cfg.MODEL.NAME+'.get_pose_net')( + cfg, is_train=False + ) + + if cfg.TEST.MODEL_FILE: + logger.info('=> loading model from {}'.format(cfg.TEST.MODEL_FILE)) + model.load_state_dict(torch.load(cfg.TEST.MODEL_FILE), strict=False) + else: + model_state_file = os.path.join( + final_output_dir, 'final_state.pth' + ) + logger.info('=> loading model from {}'.format(model_state_file)) + model.load_state_dict(torch.load(model_state_file)) + + model.eval() + + device = torch.device("cpu") + dummy_input = torch.randn(1, 3, 256, 256, device=device) + + input_names = ["image"] + output_names = ["class"] + dynamic_axes = {'image': {0: '-1'}, 'class': {0: '-1'}} + export_onnx_file = "PoseEstNet.onnx" + + torch.onnx.export(model, dummy_input, export_onnx_file, input_names=input_names, dynamic_axes=dynamic_axes, + output_names=output_names, opset_version=11, verbose=True) + + output = model(dummy_input) + print(output) + print(output.size()) + print(output.shape[0]) + print(output.shape[1]) + print(output.shape[2]) + print(output.shape[3]) + + +if __name__ == '__main__': + main()