# DeepSparkInference **Repository Path**: wangzhpeng/deepsparkinference ## Basic Information - **Project Name**: DeepSparkInference - **Description**: DeepSparkInference推理模型示例库一期甄选了48个推理模型示例，涵盖计算机视觉，自然语言处理，语音识别等领域，后续将逐步拓展更多AI领域。 - **Primary Language**: Python - **License**: Apache-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 26 - **Created**: 2024-10-17 - **Last Updated**: 2024-10-17 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # DeepSparkInference DeepSparkInference推理模型库作为DeepSpark开源社区的核心项目，于2024年3月正式开源，一期甄选了48个推理模型示例，涵盖计算机视觉，自然语言处理，语音识别等领域，后续将逐步拓展更多AI领域。 DeepSparkInference中的模型提供了在国产推理引擎IGIE或IxRT下运行的推理示例和指导文档，部分模型提供了基于国产通用GPU[智铠100](https://www.iluvatar.com/productDetails?fullCode=cpjs-yj-tlxltt-zk100)的评测结果。 IGIE（Iluvatar GPU Inference Engine）是基于TVM框架研发的高性能、高通用、全流程的AI推理引擎。支持多框架模型导入、量化、图优化、多算子库支持、多后端支持、算子自动调优等特性，为推理场景提供易部署、高吞吐量、低延迟的完整方案。 IxRT（Iluvatar CoreX RunTime）是天数智芯自研的高性能推理引擎，专注于最大限度发挥天数智芯通用GPU 的性能，实现各领域模型的高性能推理。IxRT支持动态形状推理、插件和INT8/FP16推理等特性。 DeepSparkInference将按季度进行版本更新，后续会逐步丰富模型类别并拓展大模型推理。 ## Computer Vision ### Classification

Models	Precision	IGIE	IxRT
AlexNet	FP16	Supported	Supported
AlexNet	INT8	Supported	Supported
CLIP	FP16	Supported	-
CLIP	INT8	-	-
Conformer-B	FP16	Supported	-
Conformer-B	INT8	-	-
ConvNeXt-Small	FP16	Supported	-
ConvNeXt-Small	INT8	-	-
CSPDarkNet50	FP16	Supported	-
CSPDarkNet50	INT8	-	-
CSPResNet50	FP16	-	Supported
CSPResNet50	INT8	-	Supported
DeiT-tiny	FP16	Supported	-
DeiT-tiny	INT8	-	-
DenseNet121	FP16	Supported	Supported
DenseNet121	INT8	-	-
DenseNet161	FP16	Supported	-
DenseNet161	INT8	-	-
DenseNet169	FP16	Supported	-
DenseNet169	INT8	-	-
EfficientNet-B0	FP16	Supported	Supported
EfficientNet-B0	INT8	-	Supported
EfficientNet-B1	FP16	Supported	Supported
EfficientNet-B1	INT8	-	Supported
EfficientNet-B2	FP16	Supported	-
EfficientNet-B2	INT8	-	-
EfficientNetV2	FP16	Supported	Supported
EfficientNetV2	INT8	-	Supported
EfficientNetv2_rw_t	FP16	Supported	-
EfficientNetv2_rw_t	INT8	-	-
GoogLeNet	FP16	Supported	Supported
GoogLeNet	INT8	Supported	Supported
HRNet-W18	FP16	Supported	Supported
HRNet-W18	INT8	-	Supported
InceptionV3	FP16	Supported	Supported
InceptionV3	INT8	Supported	Supported
Inception_ResNet_V2	FP16	-	Supported
Inception_ResNet_V2	INT8	-	Supported
MobileNetV2	FP16	Supported	Supported
MobileNetV2	INT8	Supported	Supported
MobileNetV3_Large	FP16	Supported	-
MobileNetV3_Large	INT8	-	-
MobileNetV3_Small	FP16	Supported	Supported
MobileNetV3_Small	INT8	-	-
RegNet_x_1_6gf	FP16	Supported	-
RegNet_x_1_6gf	INT8	-	-
RepVGG	FP16	Supported	Supported
RepVGG	INT8	-	-
Res2Net50	FP16	Supported	Supported
Res2Net50	INT8	-	Supported
ResNeSt50	FP16	Supported	-
ResNeSt50	INT8	-	-
ResNet101	FP16	Supported	Supported
ResNet101	INT8	Supported	Supported
ResNet152	FP16	Supported	-
ResNet152	INT8	Supported	-
ResNet18	FP16	Supported	Supported
ResNet18	INT8	Supported	Supported
ResNet34	FP16	-	Supported
ResNet34	INT8	-	Supported
ResNet50	FP16	Supported	Supported
ResNet50	INT8	Supported	-
ResNet_V1_D50	FP16	-	Supported
ResNet_V1_D50	INT8	-	Supported
ResNeXt50_32x4d	FP16	Supported	-
ResNeXt50_32x4d	INT8	-	-
SEResNet50	FP16	Supported	-
SEResNet50	INT8	-	-
ShuffleNetV1	FP16	-	Supported
ShuffleNetV1	INT8	-	-
ShuffleNetV2_x0_5	FP16	Supported	-
ShuffleNetV2_x0_5	INT8	-	-
ShuffleNetV2_x1_0	FP16	Supported	-
ShuffleNetV2_x1_0	INT8	-	-
SqueezeNet 1.0	FP16	-	Supported
SqueezeNet 1.0	INT8	-	Supported
SqueezeNet 1.1	FP16	-	Supported
SqueezeNet 1.1	INT8	-	Supported
Swin Transformer	FP16	Supported	-
Swin Transformer	INT8	-	-
Swin Transformer Large	FP16	-	Supported
Swin Transformer Large	INT8	-	-
VGG16	FP16	Supported	Supported
VGG16	INT8	Supported	-
Wide ResNet50	FP16	Supported	Supported
Wide ResNet50	INT8	Supported	Supported

### Detection

Models	Precision	IGIE	IxRT
ATSS	FP16	Supported	-
ATSS	INT8	-	-
CenterNet	FP16	Supported	-
CenterNet	INT8	-	-
DETR	FP16	-	Supported
DETR	INT8	-	-
FCOS	FP16	Supported	Supported
FCOS	INT8	-	-
FoveaBox	FP16	Supported	-
FoveaBox	INT8	-	-
FSAF	FP16	Supported	-
FSAF	INT8	-	-
HRNet	FP16	Supported	-
HRNet	INT8	-	-
RetinaFace	FP16	Supported	-
RetinaFace	INT8	-	-
RetinaNet	FP16	Supported	-
RetinaNet	INT8	-	-
RTMDet	FP16	Supported	-
RTMDet	INT8	-	-
YOLOv3	FP16	Supported	Supported
YOLOv3	INT8	Supported	Supported
YOLOv4	FP16	Supported	Supported
YOLOv4	INT8	Supported	Supported
YOLOv5	FP16	Supported	Supported
YOLOv5	INT8	Supported	Supported
YOLOv5s	FP16	-	Supported
YOLOv5s	INT8	-	Supported
YOLOv6	FP16	Supported	Supported
YOLOv6	INT8	-	Supported
YOLOv7	FP16	Supported	Supported
YOLOv7	INT8	Supported	Supported
YOLOv8	FP16	Supported	Supported
YOLOv8	INT8	Supported	Supported
YOLOX	FP16	Supported	Supported
YOLOX	INT8	Supported	Supported

### Face Recognition

Models	Precision	IGIE	IxRT
FaceNet	FP16	-	Supported
FaceNet	INT8	-	Supported

### Pose Estimation

Models	Precision	IGIE	IxRT
RTMPose	FP16	Supported	-
RTMPose	INT8	-	-

### Segmentation

Models	Precision	IGIE	IxRT
Mask R-CNN	FP16	-	Supported
Mask R-CNN	INT8	-	-
SOLOv1	FP16	-	Supported
SOLOv1	INT8	-	-

### Trace

Models	Precision	IGIE	IxRT
FastReID	FP16	Supported	-
FastReID	INT8	-	-
DeepSort	FP16	Supported	-
DeepSort	INT8	Supported	-
RepNet-Vehicle-ReID	FP16	Supported	-
RepNet-Vehicle-ReID	INT8	-	-

## NLP ### Language Modelling

Models	Precision	IGIE	IxRT
ALBERT	FP16	-	Supported
ALBERT	INT8	-	-
BERT Base NER	FP16	-	-
BERT Base NER	INT8	Supported	-
BERT Base SQuAD	FP16	Supported	Supported
BERT Base SQuAD	INT8	-	Supported
BERT Large SQuAD	FP16	Supported	Supported
BERT Large SQuAD	INT8	Supported	Supported
DeBERTa	FP16	-	Supported
DeBERTa	INT8	-	-
RoBERTa	FP16	-	Supported
RoBERTa	INT8	-	-
RoFormer	FP16	-	Supported
RoFormer	INT8	-	-
VideoBERT	FP16	-	Supported
VideoBERT	INT8	-	-

### Large Language Model

Models	vLLM	TensorRT-LLM	TGI
Baichuan2-7B	Supported	-	-
ChatGLM-3-6B	Supported	-	-
Llama2-7B	-	Supported	-
Llama3-70B	Supported	-	-
Qwen-7B	-	-	Supported
Qwen1.5-7B	-	-	Supported
Qwen1.5-7B	Supported	-	-
Qwen1.5-14B	Supported	-	-
Qwen1.5-72B	Supported	-	-

## Speech ### Speech Recognition

Models	Precision	IGIE	IxRT
Conformer	FP16	Supported	Supported
Conformer	INT8	-	-
Transformer ASR	FP16	-	Supported
Transformer ASR	INT8	-	-

--- ## 社区 ### 治理请参见 DeepSpark Code of Conduct on [Gitee](https://gitee.com/deep-spark/deepspark/blob/master/CODE_OF_CONDUCT.md) or on [GitHub](https://github.com/Deep-Spark/deepspark/blob/main/CODE_OF_CONDUCT.md)。 ### 交流请联系。 ### 贡献请参见 [DeepSparkInference Contributing Guidelines](CONTRIBUTING.md)。 ## 许可证本项目许可证遵循[Apache-2.0](LICENSE)。