diff --git a/contrib/FCOS/README.md b/contrib/FCOS/README.md new file mode 100644 index 0000000000000000000000000000000000000000..0e0a539251a4404c211de87b79e557fbd031dcaf --- /dev/null +++ b/contrib/FCOS/README.md @@ -0,0 +1,252 @@ +# FCOS + +## 1 介绍 + +本开发项目演示FCOS模型实现目标检测。本系统基于mxVision SDK进行开发，以昇腾Atlas310卡为主要的硬件平台，主要实现目标检测。待检测的图片中物体不能被遮挡太严重，并且物体要完全出现在图片中。图片亮度不能过低。输入一张图片，最后会输出图片中能检测到的物体。项目主要流程： + +1.环境搭建； +2.模型转换； +3.生成后处理插件； +4.进行精度、性能对比。 + +### 1.1支持的产品 + +本产品以昇腾310（推理）卡为硬件平台。 + +### 1.2支持的版本 + +该项目支持的SDK版本为2.0.4，CANN版本为5.0.4。 + +### 1.3软件方案介绍 + +基于MindXSDK的FCOS目标检测的业务流程为： + +1. 将待检测的图片放到相应的文件夹下。 +2. 首先使用mmcv库对图片进行前处理（改变图片大小、归一化、补边操作）。‘ +3. 将图片转换成为二进制的numpy数组，传入pipeline中。 +4. 放缩后的图片输入模型推理插件mxpi_tensorinfer进行处理； +5. 将经过模型推理输出时候的张量数据流输入到mxpi_objectpostprocessor中，对FCOS目标检测模型推理输出的张量进行后处理； +6. 处理完毕之后将数据传入mxpi_dataserialize0插件中，将stream结果组装成json字符串输出。 + +下表为系统方案各个子系统功能的描述： + +| 序号 | 子系统 | 功能描述 | +| ---- | ---------- | ---------------------------------------------------------------------------------------------------------------------- | +| 1 | 图片输入 | 传入图片，修改图片的大小和格式为符合模型要求的格式 | +| 2 | 图像前处理 | 对图片进行改变大小、补边、归一化等操作。 | +| 3 | 模型推理 | 将已经处理好的图片传入到mxpi_tensorinfer中，使用目标检测模型FCOS进行推理，得到推理的张量数据流 | +| 4 | 模型后处理 | 将模型推理得到的张量数据流传入mxpi_objectpostprocessor中进行张量后处理。对模型输出的目标框进行去重，排序和筛选等工作。 | +| 5 | 组装字符串 | stream结果组装成json字符串输出。 | + +### 1.4代码目录结构与说明 + +本项目名为FCOS目标检测，项目的目录如下所示： + +``` +|- models +| |- fcos.onnx //onnx模型 +| |_ Fcos_tf_bs.cfg +|- pipeline +| |_ FCOSdetection.pipeline +|- plugin +| |_FCOSPostprocess +| |- CMakeLists.txt +| |- FCOSDetectionPostProcess.cpp +| |- FCOSDetectionPostProcess.h +| |_ build.sh +|- image +| |- image1.png +| |_ image2.png +|- build.sh +|- evaluate.py +|- colorlist.txt +|_ main.py +``` + +### 1.5技术实现流程图 + +本项目实现对输入的图片进行目标检测，整体流程如下： + +![avatar](./image/image1.png) + +### 1.6特性以及适用场景 +本项目是根据COCO数据集进行训练，仅仅适合COCO官方数据集中的80类物体进行识别。在此这八十类物体不一一列出。 +## 2环境依赖 + +推荐系统为ubuntu 18.04,环境软件和版本如下： + +| 软件名称 | 版本 | 说明 | 获取方式 | +| ------------------- | ------ | ----------------------------- | ----------------------------------------------------------------- | +| MindX SDK | 2.0.4 | mxVision软件包 | [点击打开链接](https://www.hiascend.com/software/Mindx-sdk) | +| ubuntu | 18.04 | 操作系统 | 请上ubuntu官网获取 | +| Ascend-CANN-toolkit | 5.0.4 | Ascend-cann-toolkit开发套件包 | [点击打开链接](https://www.hiascend.com/software/cann/commercial) | +| mmdetection | 2.25.0 | 用于评估准确度 | 请上mmdetection官网 | + +在项目开始运行前需要设置环境变量： + +``` +. /usr/local/Ascend/ascend-toolkit/set_env.sh +. ${SDK安装路径}/mxVision/set_env.sh +``` + +## 3 软件依赖 + +项目运行过程中涉及到的第三方软件依赖如下表所示： + +| 软件名称 | 版本 | 说明 | 使用教程 | +| ----------- | ------ | -------------------- | --------------------------------------------------------- | +| pycocotools | 2.0.0 | 用于实现代码测评 | [点击打开链接](https://cocodataset.org/) | +| mmdetection | 2.25.0 | 用于实现模型精度评估 | [点击打开链接](https://github.com/open-mmlab/mmdetection) | +| mmcv | 1.3.17 | 用于实现图片前处理 | [点击打开链接](https://github.com/open-mmlab/mmcv) | + +.安装python COCO测评工具,mmcv和mmdetection。执行命令： + +``` +pip3 install pycocotools +pip3 install mmcv-full +pip3 install mmdet +``` + +## 4 模型转换 + +本项目使用的模型是FCOS目标检测模型这个模型是一个无anchor检测器。FCOS直接把预测特征图上的每个位置$(x,y)$当作训练样本，若这个位置在某个ground truth box的内部，则视为正样本，该位置的类别标签$c$对应这个box的类别，反之则视为负样本。这个网络的输出为目标框的左上角坐标、右下角坐标、类别和置信度。本项目的onnx模型可以直接[下载](https://www.hiascend.com/zh/software/modelzoo/models/detail/1/6fcc4747a48245d29351c26cd052dd13)。下载后，里面自带的om模型是可以直接使用的，或者自行使用ATC工具将onnx模型转换成为om模型，模型转换工具的使用说明参考[链接](https://support.huawei.com/enterprise/zh/doc/EDOC1100234054/79fc5a60)。 + +模型转换步骤如下： + +1.从下载链接处下载onnx模型至FCOS/models文件夹下。 + +2.进入models文件夹目录下，设置环境变量如下： + + +设置完环境变量之后，就进行模型的转换： + +模型转换语句如下： + +``` +atc --model=fcos.onnx --framework=5 --soc_version=Ascend310 --input_format=NCHW --input_shape="input:1,3,800,1333" --output=fcos_bs1 --precision_mode=allow_fp32_to_fp16 +``` + +执行完该命令之后，会在models文件夹下生成.om模型，并且转换成功之后会在终端输出： + +``` +ATC start working now, please wait for a moment. +ATC run success, welcome to the next use. +``` + + + +## 5准备 + +### 步骤1 + +准备一张待检测图片，并上传到FCOS文件夹下。然后修改main.py文件里面的图片路径为待检测的图片路径。并且从https://github.com/pjreddie/darknet/blob/master/data/coco.names 里面下载coco.names文件，并且将这个文件存放到models文件夹下。并且修改main.py里IMAGENAME为图片的路径： +```python +IMAGENAME = '{image path}' +``` +### 步骤2 + +进入FCOS/plugin/FCOSPostprocess目录，在该目录下运行下列命令： + +``` +bash build.sh +``` +这个后处理插件即可以使用。 + +## 6编译与运行 + +在FCOS目录下执行命令： + +``` +python3 main.py +``` + +最后生成的结果会在FCOS文件夹目录下result.jpg图片中。 + +## 7精度测试 + +1.下载COCO VAL 2017[数据集](http://images.cocodataset.org/zips/val2017.zip +)和[标注文件](http://images.cocodataset.org/annotations/annotations_trainval2017.zip +)。在FCOS目录下创建dataset目录，将数据集压缩文件和标注数据压缩文件都解压至 `FCOS/dataset` 目录下。再次创建一个文件夹名为binresult，在FCOS目录下。确保解压后的FCOS目录结构为： + +``` +|- models +| |- fcos.onnx //onnx模型 +| |_ Fcos_tf_bs.cfg +|- pipeline +| |_ FCOSdetection.pipeline +|- plugin +| |_FCOSPostprocess +| |- CMakeLists.txt +| |- FCOSDetectionPostProcess.cpp +| |- FCOSDetectionPostProcess.h +| |_ build.sh +|- binresult +|- dataset +| |- annotations +| | |_ instances_val2017.json +| |_ val2017 +| |-000000581615.jpg +| |-000000581781.jpg +| |_other-images +|- image +| |- image1.png +| |_ image2.png +|- build.sh +|- colorlist.txt +|- evaluate.py +|_ main.py +``` + +2.再修改后处理插件cpp文件中的： +```cpp + //107行 + if (*(beginRes + CENTERPOINT) < THRESHOLD_) { + continue; + } + //123行 + MxBase::NmsSort(objectInfo, RATE); +``` +将上面的代码全部注释掉，再重新生成一次插件。 + +3.执行命令： + +``` +python3 evaluate.py +``` + +命令执行完毕之后，会在binresult文件夹，生成模型输出的目标框、置信度等信息。 + +4.最后在终端输出COCO格式的测评结果，输出结果如下： + +![avatar](./image/image2.png) + +上图中第一行可以看到，这个模型在COCO VAL 2017数据集上，IOU阈值为0.50:0.05:0.95时的精确度为0.347。原模型的推理精度也为0.347这个数据与原模型的推理精度误差范围在0.01内。 + +## 8常见问题 +### 8.1 模型路径配置问题： +问题描述： +检测过程中用到的模型以及模型后处理插件需配置路径属性。 + +后处理插件以及模型推理插件配置例子： + +```json +// 模型推理插件 + "mxpi_tensorinfer0": { + "props": { + "dataSource": "mxpi_imageresize0", + "modelPath": "./models/fcos_bs1.om" + }, + "factory": "mxpi_tensorinfer", + "next": "mxpi_objectpostprocessor0" + }, +// 模型后处理插件 + "mxpi_objectpostprocessor0":{ + "props": { + "dataSource" : "mxpi_tensorinfer0", + "postProcessConfigPath": "./models/Fcos_tf_bs.cfg", + "postProcessLibPath": "libFCOSDetectionPostProcess.so" + }, + "factory": "mxpi_objectpostprocessor", + "next": "mxpi_dataserialize0" + }, +``` diff --git a/contrib/FCOS/build.sh b/contrib/FCOS/build.sh new file mode 100644 index 0000000000000000000000000000000000000000..2f013a99a2680a4ee547ceb099a5aa8d33a0e598 --- /dev/null +++ b/contrib/FCOS/build.sh @@ -0,0 +1,33 @@ +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +set -e +current_folder="$( cd "$(dirname "$0")" ;pwd -P )" + +SAMPLE_FOLDER=( +/plugin/FCOSPostprocess/ +) + +err_flag=0 +for sample in "${SAMPLE_FOLDER[@]}";do + cd "${current_folder}/${sample}" + bash build.sh || { + echo -e "Failed to build ${sample}" + err_flag=1 + } +done + +if [ ${err_flag} -eq 1]; then + exit 1 +fi +exit 0 \ No newline at end of file diff --git a/contrib/FCOS/colorlist.txt b/contrib/FCOS/colorlist.txt new file mode 100644 index 0000000000000000000000000000000000000000..ab262c96e27545688a9b6702e2d1e349e791fc74 --- /dev/null +++ b/contrib/FCOS/colorlist.txt @@ -0,0 +1 @@ +LawnGreen,Chartreuse,Aqua,Beige,Azure,BlanchedAlmond,Bisque,Aquamarine,BlueViolet,BurlyWood,CadetBlue,AntiqueWhite,Chocolate,Coral,CornflowerBlue,Cornsilk,Crimson,Cyan,DarkCyan,DarkGoldenRod,DarkGrey,DarkKhaki,DarkOrange,DarkOrchid,DarkSalmon,DarkSeaGreen,DarkTurquoise,DarkViolet,DeepPink,DeepSkyBlue,DodgerBlue,FireBrick,FloralWhite,ForestGreen,Fuchsia,Gainsboro,GhostWhite,Gold,GoldenRod,Salmon,Tan,HoneyDew,HotPink,IndianRed,Ivory,Khaki,Lavender,LavenderBlush,AliceBlue,LemonChiffon,LightBlue,LightCoral,LightCyan,LightGoldenRodYellow,LightGray,LightGrey,LightGreen,LightPink,LightSalmon,LightSeaGreen,LightSkyBlue,LightSlateGray,LightSlateGrey,LightSteelBlue,LightYellow,Lime,LimeGreen,Linen,Magenta,MediumAquaMarine,MediumOrchid,MediumPurple,MediumSeaGreen,MediumSlateBlue,MediumSpringGreen,MediumTurquoise,MediumVioletRed,MintCream,MistyRose,Moccasin,NavajoWhite,OldLace,Olive,OliveDrab,Orange,OrangeRed,Orchid,PaleGoldenRod,PaleGreen,PaleTurquoise,PaleVioletRed,PapayaWhip,PeachPuff,Peru,Pink,Plum,PowderBlue,Purple,Red,RosyBrown,RoyalBlue,SaddleBrown,Green,SandyBrown,SeaGreen,SeaShell,Sienna,Silver,SkyBlue,SlateBlue,SlateGray,SlateGrey,Snow,SpringGreen,SteelBlue,GreenYellow,Teal,Thistle,Tomato,Turquoise,Violet,Wheat,White,WhiteSmoke,Yellow,YellowGreen \ No newline at end of file diff --git a/contrib/FCOS/evaluate.py b/contrib/FCOS/evaluate.py new file mode 100644 index 0000000000000000000000000000000000000000..7a5248bd0042372efe8c2683de0fb4160a366e70 --- /dev/null +++ b/contrib/FCOS/evaluate.py @@ -0,0 +1,35 @@ +import os +from mmdet.core import bbox2result +from mmdet.core import encode_mask_results +from mmdet.datasets import CocoDataset +import numpy as np + +if __name__ == "__main__": + coco_dataset = CocoDataset( + ann_file='./dataset/annotations/instances_val2017.json', pipeline=[]) + results = [] + + for ids in coco_dataset.img_ids: + print('image ids = {}'.format(ids)) + bbox_results = [] + # read bbox information + BBOX_RES_PATH = './binresult/bboxres' + str(ids) + '.bin' + REBLES_RES_PATH = './binresult/class' + str(ids) + '.bin' + bboxes = np.fromfile(BBOX_RES_PATH, dtype=np.float32) + bboxes = np.reshape(bboxes, [100, 5]) + bboxes.tolist() + # read label information + labels = np.fromfile(REBLES_RES_PATH, dtype=np.int64) + labels = np.reshape(labels, [100, 1]) + labels.tolist() + bbox_results = [bbox2result(bboxes, labels[:, 0], 80)] + + result = bbox_results + results.extend(result) + print('Evaluating...') + eval_results = coco_dataset.evaluate(results, + metric=[ + 'bbox', + ], + classwise=True) + print(eval_results) \ No newline at end of file diff --git a/contrib/FCOS/image/image1.png b/contrib/FCOS/image/image1.png new file mode 100644 index 0000000000000000000000000000000000000000..035bb1e72896dd8bbc812236ad242c0de86f4bd1 Binary files /dev/null and b/contrib/FCOS/image/image1.png differ diff --git a/contrib/FCOS/image/image2.png b/contrib/FCOS/image/image2.png new file mode 100644 index 0000000000000000000000000000000000000000..b5c62be596f510d624766da680bc5af2e41aaa87 Binary files /dev/null and b/contrib/FCOS/image/image2.png differ diff --git a/contrib/FCOS/main.py b/contrib/FCOS/main.py new file mode 100644 index 0000000000000000000000000000000000000000..8ee2fd40f919c113747840a46be68730c1cb1842 --- /dev/null +++ b/contrib/FCOS/main.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import json +import os +import time +import cv2 +import numpy as np +import mmcv +import webcolors +import MxpiDataType_pb2 as MxpiDataType +from StreamManagerApi import StreamManagerApi, MxProtobufIn, InProtobufVector, StringVector + + +def from_colorname_to_bgr(color): + rgb_color = webcolors.name_to_rgb(color) + result_change = (rgb_color.blue, rgb_color.green, rgb_color.red) + return result_change + + +def standard_to_bgr(): + list_color_name = [] + with open("colorlist.txt", "r") as ff: + list_color_name = ff.read() + list_color_name = list_color_name.split(',') + + standard = [] + for i in range(len(list_color_name) - + 36): # -36 used to match the len(obj_list) + standard.append(from_colorname_to_bgr(list_color_name[i])) + + return standard + + +def plot_one_box(origin_img, box, color=None, line_thickness=None): + tl = line_thickness or int(round( + 0.001 * max(origin_img.shape[0:2]))) # line thickness + if tl < 1: + tl = 1 + c1, c2 = (int(box['x0']), int(box['y0'])), (int(box['x1']), int(box['y1'])) + cv2.rectangle(origin_img, c1, c2, color=color, thickness=tl) + if box['text']: + tf = max(tl - 2, 1) # font thickness + s_size = cv2.getTextSize(str('{:.4%}'.format(box['confidence'])), + 0, + fontScale=float(tl) / 3, + thickness=tf)[0] + t_size = cv2.getTextSize(box['text'], + 0, + fontScale=float(tl) / 3, + thickness=tf)[0] + c2 = c1[0] + t_size[0] + s_size[0] + 15, c1[1] - t_size[1] - 3 + cv2.rectangle(origin_img, c1, c2, color, -1) # filled + cv2.putText(origin_img, + '{}: {:.4%}'.format(box['text'], box['confidence']), + (c1[0], c1[1] - 2), + 0, + float(tl) / 3, [0, 0, 0], + thickness=tf, + lineType=cv2.FONT_HERSHEY_SIMPLEX) + + +def resize_image(image, size): + old_h = image.shape[0] + old_w = image.shape[1] + scale_ratio = min(size[0] / old_w, size[1] / old_h) + new_w = int(np.floor(old_w * scale_ratio)) + new_h = int(np.floor(old_h * scale_ratio)) + image = mmcv.imresize(image, (new_w, new_h), backend='cv2') + return image, scale_ratio + + +def preprocess(imagepath): + image = mmcv.imread(imagepath, backend='cv2') + image, scale = resize_image(image, (1333, 800)) + h = image.shape[0] + w = image.shape[1] + mean = np.array([102.9801, 115.9465, 122.7717], dtype=np.float32) + std = np.array([1.0, 1.0, 1.0], dtype=np.float32) + norm_img_data = mmcv.imnormalize(image, mean, std, to_rgb=False) + padleft = (1333 - w) // 2 + padtop = (800 - h) // 2 + padright = 1333 - w - padleft + padbottom = 800 - h - padtop + image_for_infer = mmcv.impad(norm_img_data, + padding=(padleft, padtop, padright, + padbottom), + pad_val=0) + image_for_infer = image_for_infer.transpose(2, 0, 1) + return image_for_infer, [scale, padleft, padtop, padright, padbottom] + + +if __name__ == '__main__': + streamManagerApi = StreamManagerApi() + # create a StreamManager and init it + ret = streamManagerApi.InitManager() + if ret != 0: + print("Failed to init Stream manager, ret=%s" % str(ret)) + exit() + + # create pipeline + with open("./pipeline/FCOSdetectiontest.pipeline", 'rb') as f: + pipelineStr = f.read() + ret = streamManagerApi.CreateMultipleStreams(pipelineStr) + if ret != 0: + print("Failed to create Stream, ret=%s" % str(ret)) + exit() + IMAGENAME = '{image path}' + # create detection object + if os.path.exists(IMAGENAME) != 1: + print("The test image does not exist.") + exit() + + # get the size of picture. + tensor_data, return_image = preprocess(IMAGENAME) + tensor = tensor_data[None, :] + + visionList = MxpiDataType.MxpiVisionList() + visionVec = visionList.visionVec.add() + visionInfo = visionVec.visionInfo + # The standard input size of FCOS model is 1333*800 + visionInfo.width = 1333 + visionInfo.height = 800 + visionInfo.widthAligned = 1333 + visionInfo.heightAligned = 800 + visionData = visionVec.visionData + visionData.dataStr = tensor.tobytes() + visionData.deviceId = 0 + visionData.memType = 0 + visionData.dataSize = len(tensor) + + STREAMNAME = b'detection' + INPLUGINID = 0 + + # put the detection object to stream + KEY_VALUE = b"appsrc0" + protobufVec = InProtobufVector() + + protobuf = MxProtobufIn() + protobuf.key = KEY_VALUE + protobuf.type = b"MxTools.MxpiVisionList" + protobuf.protobuf = visionList.SerializeToString() + protobufVec.push_back(protobuf) + + uniqueId = streamManagerApi.SendProtobuf(STREAMNAME, INPLUGINID, + protobufVec) + + if uniqueId < 0: + print("Failed to send data to stream.") + exit() + + YUV_BYTES_NU = 3 + YUV_BYTES_DE = 2 + keys = [b"mxpi_objectpostprocessor0"] + keyVec = StringVector() + for key in keys: + keyVec.push_back(key) + + # get the output data from the stream plugin + infer_result = streamManagerApi.GetProtobuf(STREAMNAME, 0, keyVec) + if infer_result.size() == 0: + print("infer_result is null") + exit() + + if infer_result[0].errorCode != 0: + print("GetProtobuf error. errorCode=%d" % (infer_result[0].errorCode)) + exit() + + # get output information from mxpi_objectpostprocessor0 + objectList = MxpiDataType.MxpiObjectList() + objectList.ParseFromString(infer_result[0].messageBuf) + print(objectList) + + img = cv2.imread(IMAGENAME) + result = objectList.objectVec + myscale, pad_left, pad_top, pad_right, pad_bottom = return_image + + classify = [] + with open('./models/coco.names') as f: + for line in f.readlines(): + line = line.strip('\n') + classify.append(str(line)) + color_list = standard_to_bgr() + for x in result: + TEXTNAME = "{}{}".format(str(round(x.classVec[0].confidence, 4)), " ") + TEXTNAME += str(classify[int(x.classVec[0].classId)]) + new_x0 = max(int((x.x0 - pad_left) / myscale), 0) + new_x1 = max(int((x.x1 - pad_left) / myscale), 0) + new_y0 = max(int((x.y0 - pad_top) / myscale), 0) + new_y1 = max(int((x.y1 - pad_top) / myscale), 0) + objBox = { + 'x0': new_x0, + 'x1': new_x1, + 'y0': new_y0, + 'y1': new_y1, + 'text': str(classify[int(x.classVec[0].classId)]), + 'classId': int(x.classVec[0].classId), + 'confidence': round(x.classVec[0].confidence, 4) + } + plot_one_box(img, objBox, color=color_list[objBox.get('classId')]) + cv2.imwrite("./result.jpg", img) + # destroy streams + streamManagerApi.DestroyAllStreams() diff --git a/contrib/FCOS/models/Fcos_tf_bs.cfg b/contrib/FCOS/models/Fcos_tf_bs.cfg new file mode 100644 index 0000000000000000000000000000000000000000..f4da5cccb168051e0555868ab662d737a6188314 --- /dev/null +++ b/contrib/FCOS/models/Fcos_tf_bs.cfg @@ -0,0 +1,6 @@ +CLASS_NUM=80 +SCORE_THRESH=0.05 +OBJECTNESS_THRESH=0.2 +IOU_THRESH=0.5 +MODEL_TYPE=1 +FRAMEWORK=Pytorch \ No newline at end of file diff --git a/contrib/FCOS/pipeline/FCOSdetectiontest.pipeline b/contrib/FCOS/pipeline/FCOSdetectiontest.pipeline new file mode 100644 index 0000000000000000000000000000000000000000..541bf2974d22a832d2f7875b7b6521ac472a563c --- /dev/null +++ b/contrib/FCOS/pipeline/FCOSdetectiontest.pipeline @@ -0,0 +1,45 @@ +{ + "detection": { + "stream_config": { + "deviceId": "0" + }, + "appsrc0": { + "props": { + "blocksize": "409600" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0" + }, + "mxpi_tensorinfer0": { + "props": { + "dataSource": "appsrc0", + "modelPath": "./models/fcos_bs1.om" + }, + "factory": "mxpi_tensorinfer", + "next": "mxpi_objectpostprocessor0" + }, + "mxpi_objectpostprocessor0":{ + "props": { + "dataSource" : "mxpi_tensorinfer0", + "postProcessConfigPath": "./models/Fcos_tf_bs.cfg", + "postProcessLibPath": "libFCOSDetectionPostProcess.so", + "labelPath": "models/coco.names" + }, + "factory": "mxpi_objectpostprocessor", + "next": "mxpi_dataserialize0" + }, + "mxpi_dataserialize0": { + "props": { + "outputDataKeys": "mxpi_objectpostprocessor0" + }, + "factory": "mxpi_dataserialize", + "next": "appsink0" + }, + "appsink0": { + "props": { + "blocksize": "4096000" + }, + "factory": "appsink" + } + } +} \ No newline at end of file diff --git a/contrib/FCOS/plugin/FCOSPostprocess/CMakeLists.txt b/contrib/FCOS/plugin/FCOSPostprocess/CMakeLists.txt new file mode 100644 index 0000000000000000000000000000000000000000..14fdc0321d6bd7f47e78e9e313125cf4fd9f1099 --- /dev/null +++ b/contrib/FCOS/plugin/FCOSPostprocess/CMakeLists.txt @@ -0,0 +1,31 @@ +cmake_minimum_required(VERSION 3.5.2) +project(FCOSDetectionPostProcess) + +add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0) +add_definitions(-Dgoogle=mindxsdk_private) + +set(PLUGIN_NAME "FCOSDetectionPostProcess") +set(TARGET_LIBRARY ${PLUGIN_NAME}) +set(MX_SDK_HOME "$ENV{MX_SDK_HOME}") + +include_directories(${MX_SDK_HOME}/include) +include_directories(${MX_SDK_HOME}/opensource/include) +include_directories(${MX_SDK_HOME}/opensource/include/gstreamer-1.0) +include_directories(${MX_SDK_HOME}/opensource/include/glib-2.0) +include_directories(${MX_SDK_HOME}/opensource/lib/glib-2.0/include) +include_directories(${MX_SDK_HOME}/opensource/include/opencv4) + +link_directories(${MX_SDK_HOME}/opensource/lib/) +link_directories(${MX_SDK_HOME}/lib) + +add_compile_options(-std=c++11 -fPIC -fstack-protector-all -pie -Wno-deprecated-declarations) +add_compile_options("-DPLUGIN_NAME=${PLUGIN_NAME}") + +add_definitions(-DENABLE_DVPP_INTERFACE) +add_library(${TARGET_LIBRARY} SHARED FCOSDetectionPostProcess.cpp) + +target_link_libraries(${TARGET_LIBRARY} glib-2.0 gstreamer-1.0 gobject-2.0 gstbase-1.0 gmodule-2.0) +target_link_libraries(${TARGET_LIBRARY} plugintoolkit mxpidatatype mxbase opencv_world) +target_link_libraries(${TARGET_LIBRARY} -Wl,-z,relro,-z,now,-z,noexecstack -s) + +install(TARGETS ${TARGET_LIBRARY} LIBRARY DESTINATION ${MX_SDK_HOME}/lib/modelpostprocessors) diff --git a/contrib/FCOS/plugin/FCOSPostprocess/FCOSDetectionPostProcess.cpp b/contrib/FCOS/plugin/FCOSPostprocess/FCOSDetectionPostProcess.cpp new file mode 100644 index 0000000000000000000000000000000000000000..36571de2496b37915e4dd7e1ea7da5ca4db53308 --- /dev/null +++ b/contrib/FCOS/plugin/FCOSPostprocess/FCOSDetectionPostProcess.cpp @@ -0,0 +1,154 @@ +/* + * Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "FCOSDetectionPostProcess.h" + +#include "MxBase/Log/Log.h" + +namespace { +const float RATE = 0.3; +const uint32_t L = 0; +const uint32_t T = 1; +const uint32_t R = 2; +const uint32_t B = 3; +const int NETINPUTWIDTH = 1333; +const int NETINPUTHEIGHT = 800; +const uint32_t CENTERPOINT = 4; +const float THRESHOLD_ = 0.3; +} // namespace +namespace MxBase { +FCOSPostProcess &FCOSPostProcess::operator=(const FCOSPostProcess &other) { + if (this == &other) { + return *this; + } + ObjectPostProcessBase::operator=(other); + return *this; +} + +APP_ERROR FCOSPostProcess::Init( + const std::map> &postConfig) { + LogInfo << "Start to Init FCOSDetectionPostProcess"; + APP_ERROR ret = ObjectPostProcessBase::Init(postConfig); + if (ret != APP_ERR_OK) { + LogError << GetError(ret) + << "Fail to superInit in FCOSDetectionPostProcess."; + return ret; + } + LogInfo << "End to Init FCOSDetectionPostProcess."; + return APP_ERR_OK; +} + +APP_ERROR FCOSPostProcess::DeInit() { return APP_ERR_OK; } + +/* + input: + tensors:the output of mxpi_tensorinfer0 , the output of the model. + objectInfos:save result. + return: + return the postprocess result. +*/ +APP_ERROR FCOSPostProcess::Process( + const std::vector &tensors, + std::vector> &objectInfos, + const std::vector &resizedImageInfos, + const std::map> &configParamMap) { + LogInfo << "Start to Process FCOSDetectionPostProcess."; + APP_ERROR ret = APP_ERR_OK; + auto inputs = tensors; + ret = CheckAndMoveTensors(inputs); + if (ret != APP_ERR_OK) { + LogError << "CheckAndMoveTensors failed. ret=" << ret; + return ret; + } + + if (tensors.size() == 0) { + return APP_ERR_OK; + } + + LogInfo << "FCOSDetectionPostProcess start to write results."; + + for (auto num : {0, 1}) { + if ((num >= tensors.size()) || (num < 0)) { + LogError << GetError(APP_ERR_INVALID_PARAM) << "TENSOR(" << num + << ") must ben less than tensors'size(" << tensors.size() + << ") and larger than 0."; + } + } + auto shape = tensors[0].GetShape(); + if (shape.size() == 0) { + return APP_ERR_OK; + } + + LogInfo << "start to process."; + if (tensors[0].GetBuffer() == NULL || tensors[1].GetBuffer() == NULL) { + LogError << "tensors buffer is NULL.\n"; + return APP_ERR_OK; + } + std::vector objectInfo; + auto res0 = (float *)tensors[0].GetBuffer(); + auto classIdx = (__int64 *)tensors[1].GetBuffer(); + auto shape1 = tensors[1].GetShape(); + for (uint32_t i = 0; i < shape[1]; i++) { + float *beginRes = res0 + i * 5; + ObjectInfo objInfo; + if (*(beginRes + CENTERPOINT) < THRESHOLD_) { + continue; + } + objInfo.x0 = *(beginRes + L); + objInfo.y0 = *(beginRes + T); + objInfo.x1 = *(beginRes + R); + objInfo.y1 = *(beginRes + B); + objInfo.confidence = *(beginRes + CENTERPOINT); + objInfo.classId = (float)classIdx[i]; + LogInfo << "start postprocessbbox."; + PostprocessBBox(objInfo, resizedImageInfos[0].widthOriginal, + resizedImageInfos[0].heightOriginal, NETINPUTWIDTH, + NETINPUTHEIGHT); + LogInfo << "postprocessbbox successfully."; + objectInfo.push_back(objInfo); + } + MxBase::NmsSort(objectInfo, RATE); + objectInfos.push_back(objectInfo); + LogInfo << "FCOSDetectionPostProcess write results successed."; + LogInfo << "End to Process FCOSDetectionPostProcess."; + return APP_ERR_OK; +} + +void FCOSPostProcess::PostprocessBBox(ObjectInfo &objInfo, int imageWidth, + int imageHeight, int netInWidth, + int netInHeight) { + float scale = netInWidth * 1.0 / imageWidth * 1.0; + if (scale > (netInHeight * 1.0 / imageHeight * 1.0)) { + scale = (netInHeight * 1.0 / imageHeight * 1.0); + } + float padW = netInWidth * 1.0 - imageWidth * 1.0 * scale; + float padH = netInHeight * 1.0 - imageHeight * 1.0 * scale; + float padLeft = padW / 2; + float padTop = padH / 2; + objInfo.x0 = (objInfo.x0 - padLeft) / scale; + objInfo.y0 = (objInfo.y0 - padTop) / scale; + objInfo.x1 = (objInfo.x1 - padLeft) / scale; + objInfo.y1 = (objInfo.y1 - padTop) / scale; +} +extern "C" { +std::shared_ptr GetObjectInstance() { + LogInfo << "Begin to get FCOSPostProcess instance."; + auto instance = std::make_shared(); + LogInfo << "End to get FCOSPostProcess instance."; + return instance; +} +} +} // namespace MxBase \ No newline at end of file diff --git a/contrib/FCOS/plugin/FCOSPostprocess/FCOSDetectionPostProcess.h b/contrib/FCOS/plugin/FCOSPostprocess/FCOSDetectionPostProcess.h new file mode 100644 index 0000000000000000000000000000000000000000..1d61167b87c50e70b16af0f6a9e45245f0dd0363 --- /dev/null +++ b/contrib/FCOS/plugin/FCOSPostprocess/FCOSDetectionPostProcess.h @@ -0,0 +1,58 @@ +/* + * Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef FCOS_POST_PROCESS_H +#define FCOS_POST_PROCESS_H +#include "MxBase/CV/ObjectDetection/Nms/Nms.h" +#include "MxBase/PostProcessBases/ObjectPostProcessBase.h" +#include "opencv2/opencv.hpp" + +namespace MxBase { + class FCOSPostProcess : public ObjectPostProcessBase { + public: + FCOSPostProcess() = default; + + ~FCOSPostProcess() = default; + + FCOSPostProcess(const FCOSPostProcess &other); + + APP_ERROR Init( + const std::map> &postConfig) override; + + APP_ERROR DeInit() override; + + APP_ERROR Process(const std::vector &tensors, + std::vector> &objectInfos, + const std::vector &resizedImageInfos = {}, + const std::map> + &configParamMap = {}) override; + + FCOSPostProcess &operator=(const FCOSPostProcess &other); + + protected: + void PostprocessBBox(ObjectInfo &objInfo, int imageWidth, int imageHeight, int netInWidth, int netInHeight); + private: + uint32_t classNum_ = 0; + bool softmax_ = true; + uint32_t topK_ = 1; + float min_confidence = 0.5; + }; + + extern "C" { + std::shared_ptr GetObjectInstance(); + } +} // namespace MxBase +#endif \ No newline at end of file diff --git a/contrib/FCOS/plugin/FCOSPostprocess/build.sh b/contrib/FCOS/plugin/FCOSPostprocess/build.sh new file mode 100644 index 0000000000000000000000000000000000000000..a3e08287babfc52a9f4b3e96835e00585e3e867a --- /dev/null +++ b/contrib/FCOS/plugin/FCOSPostprocess/build.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e + +current_folder="$( cd "$(dirname "$0")" ;pwd -P )" + +function build_plugin() { + build_path=$current_folder/build + if [ -d "$build_path" ]; then + rm -rf "$build_path" + else + echo "file $build_path is not exist." + fi + mkdir -p "$build_path" + cd "$build_path" + cmake .. + make -j + make install + chmod 640 $MX_SDK_HOME/lib/modelpostprocessors/libFCOSDetectionPostProcess.so + cd .. + exit 0 +} + +build_plugin +exit 0 \ No newline at end of file diff --git a/contrib/build_all.sh b/contrib/build_all.sh index f14e926376aff90b4adccbb0a1dc727a8d110fc4..d5405a9d5e4d97e2aebc540270ffcda5b9320647 100644 --- a/contrib/build_all.sh +++ b/contrib/build_all.sh @@ -18,20 +18,20 @@ current_folder="$( cd "$(dirname "$0")" ;pwd -P )" SAMPLE_FOLDER=( - # ActionRecognition/ - # CrowdCounting/ + # ActionRecognition/ + # CrowdCounting/ # mxBase_wheatDetection/ - # EdgeDetectionPicture/ + # EdgeDetectionPicture/ HelmetIdentification/ Individual/ # human_segmentation/ - # OpenposeKeypointDetection/ - PersonCount/ - FatigueDrivingRecognition/ - # CartoonGANPicture/ - # HeadPoseEstimation/ - FaceBoxes/ - BertTextClassification/ + # OpenposeKeypointDetection/ + PersonCount/ + FatigueDrivingRecognition/ + # CartoonGANPicture/ + # HeadPoseEstimation/ + FaceBoxes/ + BertTextClassification/ # RTM3DTargetDetection/ EfficientDet/ SentimentAnalysis/ @@ -47,9 +47,11 @@ SAMPLE_FOLDER=( PassengerflowEstimation/ CenterFace/ YOLOX/ - PicoDet/ + PicoDet/ SOLOV2/ OpenCVPlugin/ + RefineDet/ + FCOS/ ) err_flag=0 @@ -57,12 +59,12 @@ for sample in ${SAMPLE_FOLDER[@]};do cd ${current_folder}/${sample} bash build.sh || { echo -e "Failed to build ${sample}" - err_flag=1 + err_flag=1 } done if [ ${err_flag} -eq 1 ]; then - exit 1 + exit 1 fi exit 0 diff --git a/mindxsdk-referenceapps b/mindxsdk-referenceapps new file mode 160000 index 0000000000000000000000000000000000000000..b23ecef88d2cf58c7421a2a9be1179dfc20f5fde --- /dev/null +++ b/mindxsdk-referenceapps @@ -0,0 +1 @@ +Subproject commit b23ecef88d2cf58c7421a2a9be1179dfc20f5fde