From 96d68f94d0c2493b891a3778a73119dbfff6f3d1 Mon Sep 17 00:00:00 2001 From: Will Li Date: Thu, 19 Mar 2026 10:59:47 +0800 Subject: [PATCH 1/2] =?UTF-8?q?=E6=96=B0=E5=A2=9ENPU=E5=BA=94=E7=94=A8?= =?UTF-8?q?=EF=BC=8C=E7=AE=97=E6=B3=95=E8=B6=85=E5=B8=82=E6=96=87=E6=A1=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../en/NPU-Applications/README.md | 9 + .../en/NPU-Applications/ai_market.md | 797 ++++++++++++++++++ docs/Quectel-Pi-M1/en/sidebar.yaml | 10 +- .../zh/NPU-Applications/README.md | 7 + .../zh/NPU-Applications/ai_market.md | 796 +++++++++++++++++ docs/Quectel-Pi-M1/zh/sidebar.yaml | 8 +- pages/index/zh/README.md | 4 +- 7 files changed, 1624 insertions(+), 7 deletions(-) create mode 100644 docs/Quectel-Pi-M1/en/NPU-Applications/README.md create mode 100644 docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md create mode 100644 docs/Quectel-Pi-M1/zh/NPU-Applications/README.md create mode 100644 docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md diff --git a/docs/Quectel-Pi-M1/en/NPU-Applications/README.md b/docs/Quectel-Pi-M1/en/NPU-Applications/README.md new file mode 100644 index 0000000..b9eb877 --- /dev/null +++ b/docs/Quectel-Pi-M1/en/NPU-Applications/README.md @@ -0,0 +1,9 @@ +# NPU applications + +
+This chapter introduces guides and examples for developing edge intelligence applications on the Quectel Pi M1 platform. It aims to provide developers with a complete solution from model deployment to practical implementation, enabling you to fully utilize the platform's edge computing capabilities. +
+ +> **Quick access** + +- [AIMarket](./ai_market.md) diff --git a/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md b/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md new file mode 100644 index 0000000..65b7dbb --- /dev/null +++ b/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md @@ -0,0 +1,797 @@ +# AIMarket + +## Introduction + +This document is the integration development guide for **AIMarket** Android SDK, designed to help developers quickly understand, deploy, and use various AI models and algorithm services provided by the algorithm supermarket. + +### What is AIMarket + +AIMarket is a one-stop AI algorithm integration platform developed by Quectel, providing high-performance edge AI inference capabilities for Android devices. Built on Qualcomm SNPE (Snapdragon Neural Processing Engine) framework, the platform fully utilizes NPU/DSP hardware acceleration to achieve low-latency, low-power local AI inference. + +### Core Features + +- **Local Inference**: All algorithms run on-device, no network required, privacy protected +- **Hardware Acceleration**: Based on SNPE framework, supports DSP/NPU acceleration +- **Unified Interface**: Provides standardized JNI interface, simplifying integration process +- **Multi-modal Support**: Supports both image and real-time camera stream input methods +- **High Reliability**: Industrial-grade stability, suitable for commercial deployment + +### Supported Algorithm Services + +| Algorithm Type | Function Description | Application Scenarios | +|---------|---------|---------| +| Face Recognition | Face detection, keypoint localization, feature extraction and comparison | Access control attendance, identity verification | +| Mask Detection | Detect mask wearing status (correct/incorrect/not worn) | Public place safety monitoring | +| Gesture Recognition | Recognize common gesture actions | Human-computer interaction, smart home | +| QR Code Recognition | Quickly scan and decode QR codes/barcodes | Payment, information retrieval | +| OCR Text Recognition | Text extraction from images and camera streams | ID card recognition, receipt processing | + +--- + +## Pre-deployment Preparation + +### Development Environment Requirements + +| Project | Requirements | +|-----|------| +| IDE | Android Studio Arctic Fox (2020.3.1) or higher | +| Android SDK | compileSdk 32 | +| Android NDK | 21.4.7075529 | +| CMake | 3.22.1 | +| Gradle | 7.0+ | +| JDK | 1.8 | +| Target Device | ARM64 architecture (arm64-v8a) | +| Minimum System Version | Android 5.0 (API 21) | +| Target System Version | Android 11 (API 30) | + +**Note:** The device needs to be connected to adb tools, execute `adb root`, then execute `adb shell setenforce 0` + +### Required Software Installation + +#### Android Studio Configuration + +1. **Install NDK and CMake** + - Open Android Studio → Settings → Appearance & Behavior → System Settings → Android SDK + - Switch to **SDK Tools** tab + - Check and install the following components: + - NDK (Side by side) → Select version 21.4.7075529 + - CMake → Select version 3.22.1 + - Android SDK Build-Tools + - Android SDK Platform-Tools + +2. **Configure Environment Variables** (Windows System) + ``` + ANDROID_NDK_HOME = C:\Users\[Username]\AppData\Local\Android\Sdk\ndk\21.4.7075529 + ``` + +#### Gradle Configuration + +Project root directory `build.gradle`: +```gradle +plugins { + alias(libs.plugins.android.application) apply false + alias(libs.plugins.kotlin.android) apply false +} +``` + +### Project Dependencies + +#### Java/Kotlin Dependencies + +#### Native Libraries (.so files) + +The project depends on the following pre-compiled Native libraries (contact relevant personnel to obtain), which need to be placed in the `app/src/main/jniLibs/arm64-v8a/` directory: + +| Library Name | Description | +|-------|------| +| libaimarket.so | Main algorithm library (project build output) | +| libSNPE.so | Qualcomm SNPE inference engine | +| libopencv_java4.so | OpenCV image processing library | +| libfacerecognition.so | Face recognition algorithm library | +| libmaskdet.so | Mask detection algorithm library | +| libpipeline.so | Gesture recognition pipeline library | +| libdeepocr.so | OCR text recognition library | +| libzbar.so | QR code recognition library | +| libzbar_ex.so | QR code extension library | +| libjsoncpp.so | JSON parsing library | +| libc++_shared.so | C++ runtime library | + +### Model File Deployment + +Model files need to be deployed to the specified directory on the device, project default path structure: + +``` +/data/user/0/com.quectel.aimarket/files/model/ +├── face_model/ +│ ├── facedet_q_enc.dlc # Face detection model +│ ├── facelandmark_q_enc.dlc # Keypoint detection model +│ └── facereg_q_enc.dlc # Feature extraction model +├── mask_model/ +│ └── mask_q_enc.dlc # Mask detection model +├── gesture_model/ +│ ├── model.json # Model configuration file +│ ├── detection/ +│ │ └── sim.engine # Gesture detection model +│ └── classification/ +│ └── sim.engine # Gesture classification model +├── ocr/ +│ ├── model.json # OCR configuration file +│ └── sim.engine # OCR model +└── facedb/ # Face feature database (registered face images) + ├── 00.jpg + ├── 01.jpg + └── ... +``` + +--- + +## Model Core Information + +### Model Algorithm List + +| No. | Algorithm Name | Model Type | Input Size | Inference Framework | Hardware Acceleration | Description | +|----|---------|---------|---------|---------|---------|------| +| 1 | Face Recognition | Detection Model | 256×256 | SNPE | DSP | Detect face positions in images | +| 2 | Mask Detection | YOLO Detection | 640×640 | SNPE | DSP | Three categories: correct/incorrect/not worn | +| 3 | Gesture Detection | Detection Model | 640×640 | SNPE | DSP | Detect gesture regions | +| 4 | OCR Text Recognition | Recognition Model | - | SNPE | DSP | Text content recognition | +| 5 | QR Code Recognition | Decoding Algorithm | - | ZBar | CPU | Supports QR Code and other code formats | + +### Detailed Algorithm Descriptions + +#### Face Recognition + +**Functional Modules:** +- **Face Detection**: Locate face positions in images/video streams, output bounding box coordinates +- **Keypoint Localization**: Extract 5 facial keypoints (left eye, right eye, nose tip, left mouth corner, right mouth corner) +- **Feature Extraction**: Generate 128-dimensional face feature vectors +- **Face Comparison**: Calculate face similarity through cosine similarity + +**Technical Parameters:** +| Parameter | Value | +|-----|------| +| Detection confidence threshold | 0.5 | +| NMS threshold | 0.5 | +| Minimum face area ratio | 1% (image area) | +| Feature dimension | 128-dimensional | +| Similarity threshold | 0.6 | + +#### Mask Detection + +**Detection Categories:** +| Category ID | Category Name | Description | +|-------|---------|------| +| 0 | Correctly_Worn | Correctly wearing mask | +| 1 | Not_Worn | Not wearing mask | +| 2 | Incorrectly_Worn | Incorrectly wearing mask (covering mouth only/nose only, etc.) | + +**Technical Parameters:** +| Parameter | Default Value | +|-----|-------| +| Confidence threshold | 0.25 | +| NMS threshold | 0.45 | +| Minimum detection box area | 1% (image area) | + +#### Gesture Recognition + +**Supported Gesture Types:** +| No. | Gesture Name | Description | +|-----|---------|------| +| 1 | like | Thumbs up gesture | +| 2 | dislike | Thumbs down gesture | +| 3 | call | Phone call gesture | +| 4 | fist | Fist gesture | +| 5 | four | Four fingers gesture | +| 6 | mute | Mute gesture | +| 7 | ok | OK gesture | +| 8 | one | Index finger gesture | +| 9 | plam | Open palm | +| 10 | other | Other gestures | +| 11 | three | Three fingers gesture | +| 12 | yeah | Victory/V gesture | + +**Pipeline Process:** Detection → Cropping → Classification + +#### OCR Text Recognition + +**Functional Features:** +- Supports multi-line text detection +- Automatically recognizes mixed Chinese-English text +- Returns text content and position information + +**Technical Parameters:** +| Parameter | Value | +|-----|------| +| Confidence threshold | 0.2 | +| Input size | 256×256 | +| Supported languages | Chinese, English | + +#### QR Code Recognition + +**Supported Code Formats:** +- QR Code +- EAN-13 (configurable to disable) +- Other ZBar-supported code formats + +**Configuration Parameters:** +```cpp +g_qrScanner->set_config(ZBAR_QRCODE, ZBAR_CFG_X_DENSITY, 4); +g_qrScanner->set_config(ZBAR_QRCODE, ZBAR_CFG_Y_DENSITY, 4); +``` + +--- + +## Model Migration Process (Face Recognition Example) + +This section takes face recognition as an example to introduce in detail how to integrate algorithm models from scratch in Android Studio. + +### Project Structure Creation + +``` +app/src/main/ +├── cpp/ +│ ├── CMakeLists.txt # CMake configuration file +│ ├── aimarket.cpp # JNI interface implementation +│ └── include/ # Header file directory +│ ├── faceRecogInterfaceEX.hpp +│ ├── SNPEClass.hpp +│ └── ... +├── java/com/quectel/aimarket/ +│ └── manager/ +│ └── JniManager.java # JNI management class +├── jniLibs/arm64-v8a/ # Native library directory +│ ├── libSNPE.so +│ ├── libopencv_java4.so +│ └── ... +└── assets/models/ # Model file directory + └── face_model/ + ├── facedet_q_enc.dlc + ├── facelandmark_q_enc.dlc + └── facereg_q_enc.dlc +``` + +### CMake Configuration + +Create `app/src/main/cpp/CMakeLists.txt`: + +```cmake +cmake_minimum_required(VERSION 3.22.1) +project("aimarket" LANGUAGES CXX) + +# ========== Path Configuration ========== +set(THIRD_PARTY_SO_ROOT "${CMAKE_SOURCE_DIR}/../jniLibs") +set(INCLUDE_ROOT "${CMAKE_SOURCE_DIR}/include") +set(LOCAL_SRC_DIR "${CMAKE_SOURCE_DIR}") +set(SUPPORTED_ABI "arm64-v8a") +set(THIRD_PARTY_SO_DIR "${THIRD_PARTY_SO_ROOT}/${SUPPORTED_ABI}") + +# ========== Import Pre-compiled Libraries ========== + +# OpenCV library +set(OPENCV_LIB_NAME "opencv_java4") +add_library(${OPENCV_LIB_NAME} SHARED IMPORTED) +set_target_properties(${OPENCV_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${OPENCV_LIB_NAME}.so" +) + +# SNPE inference engine +set(SNPE_LIB_NAME "SNPE") +add_library(${SNPE_LIB_NAME} SHARED IMPORTED) +set_target_properties(${SNPE_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${SNPE_LIB_NAME}.so" +) + +# C++ runtime +set(C_SHARED_LIB_NAME "c++_shared") +add_library(${C_SHARED_LIB_NAME} SHARED IMPORTED) +set_target_properties(${C_SHARED_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${C_SHARED_LIB_NAME}.so" +) + +# Face recognition library +set(FACE_LIB_NAME "facerecognition") +add_library(${FACE_LIB_NAME} SHARED IMPORTED) +set_target_properties(${FACE_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${FACE_LIB_NAME}.so" +) + +# ========== Header File Configuration ========== +function(recursive_include dir) + if(IS_DIRECTORY ${dir}) + include_directories(${dir}) + file(GLOB SUB_DIRS RELATIVE ${dir} "${dir}/*") + foreach(sub_dir ${SUB_DIRS}) + set(full_sub_dir "${dir}/${sub_dir}") + if(IS_DIRECTORY ${full_sub_dir}) + recursive_include(${full_sub_dir}) + endif() + endforeach() + endif() +endfunction() +recursive_include(${INCLUDE_ROOT}) + +# ========== Build aimarket.so ========== +file(GLOB LOCAL_SRC_FILES + "${LOCAL_SRC_DIR}/aimarket.cpp" +) + +find_library(LOG_LIB log REQUIRED) +find_library(JNI_GRAPHICS_LIB jnigraphics REQUIRED) + +add_library(aimarket SHARED ${LOCAL_SRC_FILES}) + +target_compile_features(aimarket PRIVATE cxx_std_17) +target_compile_options(aimarket PRIVATE + -frtti + -fexceptions + -Wno-error=format-security + -fPIC +) + +target_link_libraries(aimarket PRIVATE + ${C_SHARED_LIB_NAME} + ${FACE_LIB_NAME} + ${OPENCV_LIB_NAME} + ${SNPE_LIB_NAME} + ${JNI_GRAPHICS_LIB} + ${LOG_LIB} + c m dl atomic +) +``` + +### JNI Interface Layer Implementation + +#### Java Layer Interface Definition + +Create `JniManager.java`: + +```java +package com.quectel.aimarket.manager; + +public class JniManager { + private static final String TAG = "JniManager"; + private static final String SO_NAME = "aimarket"; + private static volatile JniManager INSTANCE; + private static boolean isSoLoaded = false; + + // Load Native library + static { + try { + System.loadLibrary(SO_NAME); + isSoLoaded = true; + } catch (UnsatisfiedLinkError e) { + isSoLoaded = false; + } + } + + // Singleton pattern + public static JniManager getInstance() { + if (INSTANCE == null) { + synchronized (JniManager.class) { + if (INSTANCE == null) { + INSTANCE = new JniManager(); + } + } + } + return INSTANCE; + } + + // ========== Face Recognition Interface ========== + + /** + * Set SNPE environment variable + * @param soPath Native library path + */ + public native void setAdspEnv(String soPath); + + /** + * Initialize face recognition model + * @param faceDetModelPath Face detection model path + * @param faceLandmarkModelPath Keypoint model path + * @param faceRegModelPath Feature extraction model path + * @param faceDbPath Face database path + * @return 0-success, negative value-error code + */ + public native int initFaceRecognition( + String faceDetModelPath, + String faceLandmarkModelPath, + String faceRegModelPath, + String faceDbPath + ); + + /** + * Face recognition from image + * @param imagePath Input image path + * @param saveDir Result image save directory + * @return Result image path + */ + public native String faceRecognitionFromImage(String imagePath, String saveDir); + + /** + * Real-time face recognition from camera stream + * @param mergedYuvData YUV data + * @param width Image width + * @param height Image height + * @param rotation Rotation angle + * @return Detection result array [face count, x, y, w, h, id, score, keypoint count, keypoint coordinates...] + */ + public native float[] faceRecognitionFromCameraX( + byte[] mergedYuvData, + int width, + int height, + int rotation + ); + + /** + * Release face recognition resources + */ + public native void releaseCameraFaceRecog(); +} +``` + +#### C++ Layer JNI Implementation + +Create `aimarket.cpp` (core code snippet): + +```cpp +#include +#include +#include +#include +#include "faceRecogInterfaceEX.hpp" +#include "opencv2/opencv.hpp" + +#define LOG_TAG "JNI_aimarket" +#define LOGD(...) __android_log_print(ANDROID_LOG_DEBUG, LOG_TAG, __VA_ARGS__) +#define LOGE(...) __android_log_print(ANDROID_LOG_ERROR, LOG_TAG, __VA_ARGS__) + +// Global face recognition instance +static std::unique_ptr g_frIfEx = nullptr; +static std::vector> g_faceFeatures; +static int g_validFeatureDim = 0; + +void SetAdspLibraryPath(const std::string &nativeLibPath) { + std::stringstream spath; + std::stringstream spath1; + spath << nativeLibPath << ";/vendor/lib/rfsa/adsp;/vendor/dsp;/vendor/dsp/cdsp"; + spath1 << nativeLibPath; + LOGD("nativeLibPath is %s", nativeLibPath.c_str()); + if (setenv("ADSP_LIBRARY_PATH", spath.str().c_str(), 1 /*override*/) == 0) { + LOGD("SNPE SUCCESS"); + } else { + LOGD("SNPE FAIL"); + } + if (setenv("LD_LIBRARY_PATH", spath1.str().c_str(), 1 /*override*/) == 0) { + LOGD("SNPE SUCCESS"); + } else { + LOGD("SNPE FAIL"); + } + LOGD("ADSP_LIBRARY_PATH is %s", getenv("ADSP_LIBRARY_PATH")); + LOGD("LD_LIBRARY_PATH is %s", getenv("LD_LIBRARY_PATH")); +} + +extern "C" JNIEXPORT void JNICALL +Java_com_quectel_aimarket_manager_JniManager_setAdspEnv( + JNIEnv *env, + jobject thiz, + jstring soLibraryPath +) { + const char *ldPath = env->GetStringUTFChars(soLibraryPath, nullptr); + + if (ldPath == nullptr) { + LOGE("Environment variable path is empty!"); + goto end; + } + + SetAdspLibraryPath(ldPath); + + end: + env->ReleaseStringUTFChars(soLibraryPath, ldPath); +} + + +/** + * Initialize face recognition model + */ +extern "C" +JNIEXPORT jint JNICALL +Java_com_quectel_aimarket_manager_JniManager_initFaceRecognition( + JNIEnv *env, jobject thiz, + jstring face_det_model_path, + jstring face_landmark_model_path, + jstring face_reg_model_path, + jstring face_db_path) { + + try { + if (g_frIfEx != nullptr) { + LOGD("Face recognition already initialized"); + return 0; + } + + // Get model paths + char *detModel = const_cast(env->GetStringUTFChars(face_det_model_path, nullptr)); + char *landmarkModel = const_cast(env->GetStringUTFChars(face_landmark_model_path, nullptr)); + char *regModel = const_cast(env->GetStringUTFChars(face_reg_model_path, nullptr)); + char *dbPath = const_cast(env->GetStringUTFChars(face_db_path, nullptr)); + + // Create face recognition instance + g_frIfEx = std::make_unique( + detModel, landmarkModel, regModel); + + int err = g_frIfEx->init(); + if (err != 0) { + g_frIfEx.reset(); + return err; + } + + // Load face database + std::vector filePaths; + glob(dbPath, filePaths); + g_faceFeatures.clear(); + + for (const auto &path: filePaths) { + cv::Mat img = cv::imread(path); + if (img.empty()) continue; + + std::vector faceResults; + err = g_frIfEx->detect_face(img, faceResults, 0.5, 0.5); + + if (err == 1 && !faceResults.empty() && !faceResults[0].feature.empty()) { + g_faceFeatures.emplace_back(faceResults[0].feature); + g_validFeatureDim = faceResults[0].feature.size(); + } + } + + // Release string resources + env->ReleaseStringUTFChars(face_det_model_path, detModel); + env->ReleaseStringUTFChars(face_landmark_model_path, landmarkModel); + env->ReleaseStringUTFChars(face_reg_model_path, regModel); + env->ReleaseStringUTFChars(face_db_path, dbPath); + + return g_faceFeatures.empty() ? -2 : 0; + } catch (...) { + return -5; + } +} + +/** + * Camera stream face recognition + */ +extern "C" +JNIEXPORT jfloatArray JNICALL +Java_com_quectel_aimarket_manager_JniManager_faceRecognitionFromCameraX( + JNIEnv *env, jobject thiz, + jbyteArray mergedYuvData_, + jint width, jint height, jint rotation) { + + if (g_frIfEx == nullptr || g_faceFeatures.empty()) { + jfloatArray emptyArray = env->NewFloatArray(1); + float emptyVal = 0.0f; + env->SetFloatArrayRegion(emptyArray, 0, 1, &emptyVal); + return emptyArray; + } + + // Get YUV data + jbyte *mergedYuvData = env->GetByteArrayElements(mergedYuvData_, nullptr); + + // YUV to BGR + cv::Mat yuvMat(height * 3 / 2, width, CV_8UC1, (unsigned char *) mergedYuvData); + cv::Mat bgrMat; + cv::cvtColor(yuvMat, bgrMat, cv::COLOR_YUV2BGR_I420); + + // Image rotation + cv::Mat rotatedMat; + switch (rotation) { + case 90: cv::rotate(bgrMat, rotatedMat, cv::ROTATE_90_CLOCKWISE); break; + case 180: cv::rotate(bgrMat, rotatedMat, cv::ROTATE_180); break; + case 270: cv::rotate(bgrMat, rotatedMat, cv::ROTATE_90_COUNTERCLOCKWISE); break; + default: rotatedMat = bgrMat.clone(); break; + } + + // Face detection + std::vector faceResults; + g_frIfEx->detect_face(rotatedMat, faceResults, 0.5f, 0.5f); + + // Feature matching + for (auto &face: faceResults) { + if (!face.feature.empty() && face.feature.size() == g_validFeatureDim) { + std::pair matchRes = g_frIfEx->cosine_similarity( + g_faceFeatures, face.feature, 0.6f); + face.code = matchRes.first; + } + } + + // Build return result + int resultSize = 1; + for (const auto &face: faceResults) { + resultSize += 7 + face.key_pts.size() * 2; + } + + jfloatArray resultArray = env->NewFloatArray(resultSize); + std::unique_ptr resultData(new float[resultSize]); + int idx = 0; + resultData[idx++] = static_cast(faceResults.size()); + + for (const auto &face: faceResults) { + resultData[idx++] = static_cast(face.x); + resultData[idx++] = static_cast(face.y); + resultData[idx++] = static_cast(face.width); + resultData[idx++] = static_cast(face.height); + resultData[idx++] = static_cast(face.code); + resultData[idx++] = 0.0f; // score + resultData[idx++] = static_cast(face.key_pts.size()); + for (const auto &pt: face.key_pts) { + resultData[idx++] = static_cast(pt.x); + resultData[idx++] = static_cast(pt.y); + } + } + + env->SetFloatArrayRegion(resultArray, 0, resultSize, resultData.get()); + env->ReleaseByteArrayElements(mergedYuvData_, mergedYuvData, JNI_ABORT); + return resultArray; +} + +/** + * Release resources + */ +extern "C" +JNIEXPORT void JNICALL +Java_com_quectel_aimarket_manager_JniManager_releaseCameraFaceRecog( + JNIEnv *env, jobject thiz) { + g_frIfEx.reset(); + g_faceFeatures.clear(); + g_validFeatureDim = 0; +} +``` + +### build.gradle Configuration + +Add the following in `app/build.gradle`: + +```gradle +android { + // ... other configurations ... + + defaultConfig { + // ... other configurations ... + + externalNativeBuild { + cmake { + cppFlags "" + abiFilters 'arm64-v8a' + } + } + ndk { + abiFilters 'arm64-v8a' + } + } + + externalNativeBuild { + cmake { + path "src/main/cpp/CMakeLists.txt" + version "3.22.1" + } + } + ndkVersion '21.4.7075529' +} +``` + +### Usage Examples + +#### Model Initialization + +```java +public class MainActivity extends AppCompatActivity { + private JniManager jniManager; + + @Override + protected void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + + // Get JNI manager instance + jniManager = JniManager.getInstance(); + + // Set SNPE environment + String nativeLibDir = getApplicationInfo().nativeLibraryDir; + jniManager.setAdspEnv(nativeLibDir); + + // Initialize face recognition + String modelPath = getFilesDir() + "/model/face_model"; + String faceDbPath = getFilesDir() + "/model/facedb"; + + int result = jniManager.initFaceRecognition( + modelPath + "/facedet_q_enc.dlc", + modelPath + "/facelandmark_q_enc.dlc", + modelPath + "/facereg_q_enc.dlc", + faceDbPath + ); + + if (result == 0) { + Log.d("AIMarket", "Face recognition initialized successfully"); + } else { + Log.e("AIMarket", "Face recognition initialization failed: " + result); + } + } +} +``` + +#### Camera Stream Face Recognition (Using CameraX as example, requires importing relevant dependencies, you can also choose other camera implementation methods) + +```java +// Use CameraX to get preview frames +previewView.getPreviewStreamState().observe(this, state -> { + // Camera ready +}); + +// Process frame data in ImageAnalysis +ImageAnalysis imageAnalysis = new ImageAnalysis.Builder() + .setTargetResolution(new Size(640, 480)) + .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) + .build(); + +imageAnalysis.setAnalyzer(executor, image -> { + // Get YUV data + ImageProxy imageProxy = image; + byte[] yuvData = yuv420ToNv21(imageProxy); + + // Call face recognition + float[] results = jniManager.faceRecognitionFromCameraX( + yuvData, + imageProxy.getWidth(), + imageProxy.getHeight(), + getRotationDegrees() + ); + + // Parse results + if (results != null && results.length > 1) { + int faceCount = (int) results[0]; + int idx = 1; + for (int i = 0; i < faceCount; i++) { + float x = results[idx++]; + float y = results[idx++]; + float w = results[idx++]; + float h = results[idx++]; + int id = (int) results[idx++]; + // Skip score and keypoints + idx += 2 + (int) results[idx - 1] * 2; + + Log.d("FaceResult", "Face " + i + ": pos=(" + x + "," + y + + "), id=" + id); + } + } + + image.close(); +}); +``` + +#### Resource Release + +```java +@Override +protected void onDestroy() { + super.onDestroy(); + if (jniManager != null) { + jniManager.releaseCameraFaceRecog(); + jniManager.releaseAiMarketResources(); + } +} +``` + +## Demo Demonstration +1. Select the desired function from the left tab. Local Picture is for selecting local images for recognition, Live Camera is for invoking the device camera for recognition, APM is for viewing device performance changes and runtime logs +2. Select Local Picture, click the gray area in the Before region to upload an image +3. Select Live Camera, click Start Camera to begin camera recognition + + +## Common Troubleshooting + +**Note:** This project requires the use of system libraries, ensure targetSdk ≤ 30. Before installing the application, you need to connect to the device via adb and execute the `adb root` command, then execute `adb shell setenforce 0` command. Also, the ADSP environment variable must be set before initializing the algorithm's .so library for normal operation. Refer to the code above. + +| Problem | Possible Cause | Solution | +|----------------------|------------------------------------------|--------------------------------| +| Algorithm validity expired | Algorithm has a one-hour validity period after initialization, will expire after timeout | Exit current application and clear background to restart | +| UnsatisfiedLinkError | .so library not loaded correctly | Check jniLibs directory structure and architecture match, and whether environment variables are set | +| Model initialization failed (-1) | Model file does not exist or path error | Check if model files are correctly copied to device | +| Face database loading failed (-2) | Face database image format incorrect | Ensure using standard JPEG format images | +| Detection result is empty | Image format or size incorrect, some cameras have fixed angles causing captured images to have rotation/mirroring issues | Check YUV data format and rotation angle, correctly rotate input images | +| DSP unavailable | Device does not support SNPE DSP | Replace device | diff --git a/docs/Quectel-Pi-M1/en/sidebar.yaml b/docs/Quectel-Pi-M1/en/sidebar.yaml index 2b9b0df..90e6475 100644 --- a/docs/Quectel-Pi-M1/en/sidebar.yaml +++ b/docs/Quectel-Pi-M1/en/sidebar.yaml @@ -198,7 +198,12 @@ items: #- label: Research Applications # file: Applications/Open-Source-Projects/research_applications.md - # - label: NPU Applications + - label: NPU Applications + file: NPU-Applications/README.md + items: + - label: AIMarket + file: NPU-Applications/ai_market.md +# - label: NPU Applications # file: NPU-Applications/README.md # items: # - label: NPU Development Guide @@ -214,8 +219,7 @@ items: # - label: Deepseek Deployment # file: NPU-Applications/deepseek_deployment.md #- label: ChatGPT Deployment - # file: AI-pplications/chatgpt_deployment.md - + # file: AI-pplications/chatgpt_deployment.md #- label: Q&A #file: FAQ/README.md #items: diff --git a/docs/Quectel-Pi-M1/zh/NPU-Applications/README.md b/docs/Quectel-Pi-M1/zh/NPU-Applications/README.md new file mode 100644 index 0000000..bc6d220 --- /dev/null +++ b/docs/Quectel-Pi-M1/zh/NPU-Applications/README.md @@ -0,0 +1,7 @@ +# NPU应用 + +本章节介绍在Quectel Pi M1平台上进行边缘智能应用开发的相关指南和示例，旨在为开发者提供从模型部署到实际应用的完整解决方案，以充分利用平台的边缘计算能力。 + +> **快速访问** + +- [算法超市](./ai_market.md) diff --git a/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md b/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md new file mode 100644 index 0000000..6f11b71 --- /dev/null +++ b/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md @@ -0,0 +1,796 @@ +# 算法超市 + +## 简介 + +本文档是**算法超市** Android SDK的集成开发指南，旨在帮助开发者快速了解、部署和使用算法超市提供的各类AI模型和算法服务。 + +### 什么是算法超市 + +算法超市是由移远通信（Quectel）开发的一站式AI算法集成平台，为Android设备提供高性能的边缘AI推理能力。该平台基于高通SNPE（Snapdragon Neural Processing Engine）框架构建，充分利用NPU/DSP硬件加速，实现低延迟、低功耗的本地AI推理。 + +### 核心特性 + +- **本地推理**：所有算法均在设备端运行，无需联网，保护隐私 +- **硬件加速**：基于SNPE框架，支持DSP/NPU加速 +- **统一接口**：提供标准化的JNI接口，简化集成流程 +- **多模态支持**：支持图片和实时相机流两种输入方式 +- **高可靠性**：工业级稳定性，适用于商业部署 + +### 支持的算法服务 + +| 算法类型 | 功能描述 | 应用场景 | +|---------|---------|---------| +| 人脸识别 | 人脸检测、关键点定位、特征提取与比对 | 门禁考勤、身份验证 | +| 口罩检测 | 检测口罩佩戴状态（正确/错误/未佩戴） | 公共场所安全监测 | +| 手势识别 | 识别常见手势动作 | 人机交互、智能家居 | +| 二维码识别 | 快速扫描和解码二维码/条形码 | 支付、信息获取 | +| OCR文字识别 | 图片和相机流中的文字提取 | 证件识别、票据处理 | + +--- + +## 部署前准备 + +### 开发环境要求 + +| 项目 | 要求 | +|-----|------| +| IDE | Android Studio Arctic Fox (2020.3.1) 或更高版本 | +| Android SDK | compileSdk 32 | +| Android NDK | 21.4.7075529 | +| CMake | 3.22.1 | +| Gradle | 7.0+ | +| JDK | 1.8 | +| 目标设备 | ARM64架构（arm64-v8a）| +| 最低系统版本 | Android 5.0 (API 21) | +| 目标系统版本 | Android 11 (API 30) | + +**注：** 设备需连接adb工具，执行`adb root`,再执行`adb shell setenforce 0` + +### 必需软件安装 + +#### Android Studio配置 + +1. **安装NDK和CMake** + - 打开 Android Studio → Settings → Appearance & Behavior → System Settings → Android SDK + - 切换到 **SDK Tools** 标签页 + - 勾选以下组件并安装： + - NDK (Side by side) → 选择版本 21.4.7075529 + - CMake → 选择版本 3.22.1 + - Android SDK Build-Tools + - Android SDK Platform-Tools + +2. **配置环境变量**（Windows系统） + ``` + ANDROID_NDK_HOME = C:\Users\[用户名]\AppData\Local\Android\Sdk\ndk\21.4.7075529 + ``` + +#### Gradle配置 + +项目根目录 `build.gradle`: +```gradle +plugins { + alias(libs.plugins.android.application) apply false + alias(libs.plugins.kotlin.android) apply false +} +``` + +### 项目依赖库 + +#### Java/Kotlin依赖 + +#### Native依赖库（.so文件） + +项目依赖以下预编译的Native库（可寻找相关人员提供），需放置在 `app/src/main/jniLibs/arm64-v8a/` 目录： + +| 库名称 | 说明 | +|-------|------| +| libaimarket.so | 主算法库（项目编译产物） | +| libSNPE.so | 高通SNPE推理引擎 | +| libopencv_java4.so | OpenCV图像处理库 | +| libfacerecognition.so | 人脸识别算法库 | +| libmaskdet.so | 口罩检测算法库 | +| libpipeline.so | 手势识别Pipeline库 | +| libdeepocr.so | OCR文字识别库 | +| libzbar.so | 二维码识别库 | +| libzbar_ex.so | 二维码扩展库 | +| libjsoncpp.so | JSON解析库 | +| libc++_shared.so | C++运行时库 | + +### 模型文件部署 + +模型文件需部署到设备的指定目录，项目默认路径结构： + +``` +/data/user/0/com.quectel.aimarket/files/model/ +├── face_model/ +│ ├── facedet_q_enc.dlc # 人脸检测模型 +│ ├── facelandmark_q_enc.dlc # 关键点检测模型 +│ └── facereg_q_enc.dlc # 特征提取模型 +├── mask_model/ +│ └── mask_q_enc.dlc # 口罩检测模型 +├── gesture_model/ +│ ├── model.json # 模型配置文件 +│ ├── detection/ +│ │ └── sim.engine # 手势检测模型 +│ └── classification/ +│ └── sim.engine # 手势分类模型 +├── ocr/ +│ ├── model.json # OCR配置文件 +│ └── sim.engine # OCR模型 +└── facedb/ # 人脸特征库（注册人脸图片） + ├── 00.jpg + ├── 01.jpg + └── ... +``` + +--- + +## 模型核心信息 + +### 模型算法列表 + +| 序号 | 算法名称 | 模型类型 | 输入尺寸 | 推理框架 | 硬件加速 | 说明 | +|----|---------|---------|---------|---------|---------|------| +| 1 | 人脸识别 | 检测模型 | 256×256 | SNPE | DSP | 检测图片中的人脸位置 | +| 2 | 口罩检测 | YOLO检测 | 640×640 | SNPE | DSP | 三分类：正确/错误/未佩戴 | +| 3 | 手势检测 | 检测模型 | 640×640 | SNPE | DSP | 检测手势区域 | +| 4 | OCR文字识别 | 识别模型 | - | SNPE | DSP | 文字内容识别 | +| 5 | 二维码识别 | 解码算法 | - | ZBar | CPU | 支持QR Code等多种码制 | + +### 各算法详细说明 + +#### 人脸识别 + +**功能模块：** +- **人脸检测**：定位图片/视频流中的人脸位置，输出边界框坐标 +- **关键点定位**：提取5个面部关键点（左眼、右眼、鼻尖、左嘴角、右嘴角） +- **特征提取**：生成128维人脸特征向量 +- **人脸比对**：通过余弦相似度计算人脸相似性 + +**技术参数：** +| 参数 | 数值 | +|-----|------| +| 检测置信度阈值 | 0.5 | +| NMS阈值 | 0.5 | +| 最小人脸面积比例 | 1%（图像面积） | +| 特征维度 | 128维 | +| 相似度阈值 | 0.6 | + +#### 口罩检测 + +**检测类别：** +| 类别ID | 类别名称 | 说明 | +|-------|---------|------| +| 0 | Correctly_Worn | 正确佩戴口罩 | +| 1 | Not_Worn | 未佩戴口罩 | +| 2 | Incorrectly_Worn | 错误佩戴口罩（仅遮口/仅遮鼻等） | + +**技术参数：** +| 参数 | 默认值 | +|-----|-------| +| 置信度阈值 | 0.25 | +| NMS阈值 | 0.45 | +| 最小检测框面积 | 1%（图像面积） | + +#### 手势识别 + +**支持的手势类型：** +| 序号 | 手势名称 | 说明 | +|-----|---------|------| +| 1 | like | 点赞手势 | +| 2 | dislike | 踩/倒赞手势 | +| 3 | call | 打电话手势 | +| 4 | fist | 握拳 | +| 5 | four | 四指手势 | +| 6 | mute | 静音手势 | +| 7 | ok | OK手势 | +| 8 | one | 食指手势 | +| 9 | plam | 手掌张开 | +| 10 | other | 其他手势 | +| 11 | three | 三指手势 | +| 12 | yeah | 胜利/V字手势 | + +**Pipeline流程：** 检测 → 裁剪 → 分类 + +#### OCR文字识别 + +**功能特性：** +- 支持多行文本检测 +- 自动识别中英文混合文本 +- 返回文本内容和位置信息 + +**技术参数：** +| 参数 | 数值 | +|-----|------| +| 置信度阈值 | 0.2 | +| 输入尺寸 | 256×256 | +| 支持语言 | 中文、英文 | + +#### 二维码识别 + +**支持的码制：** +- QR Code +- EAN-13（可配置关闭） +- 其他ZBar支持的码制 + +**配置参数：** +```cpp +g_qrScanner->set_config(ZBAR_QRCODE, ZBAR_CFG_X_DENSITY, 4); +g_qrScanner->set_config(ZBAR_QRCODE, ZBAR_CFG_Y_DENSITY, 4); +``` + +--- + +## 模型移植流程（以人脸识别为例） + +本节以人脸识别为例，详细介绍如何从零开始在Android Studio中集成算法模型。 + +### 项目结构创建 + +``` +app/src/main/ +├── cpp/ +│ ├── CMakeLists.txt # CMake配置文件 +│ ├── aimarket.cpp # JNI接口实现 +│ └── include/ # 头文件目录 +│ ├── faceRecogInterfaceEX.hpp +│ ├── SNPEClass.hpp +│ └── ... +├── java/com/quectel/aimarket/ +│ └── manager/ +│ └── JniManager.java # JNI管理类 +├── jniLibs/arm64-v8a/ # Native库目录 +│ ├── libSNPE.so +│ ├── libopencv_java4.so +│ └── ... +└── assets/models/ # 模型文件目录 + └── face_model/ + ├── facedet_q_enc.dlc + ├── facelandmark_q_enc.dlc + └── facereg_q_enc.dlc +``` + +### CMake配置 + +创建 `app/src/main/cpp/CMakeLists.txt`： + +```cmake +cmake_minimum_required(VERSION 3.22.1) +project("aimarket" LANGUAGES CXX) + +# ========== 路径配置 ========== +set(THIRD_PARTY_SO_ROOT "${CMAKE_SOURCE_DIR}/../jniLibs") +set(INCLUDE_ROOT "${CMAKE_SOURCE_DIR}/include") +set(LOCAL_SRC_DIR "${CMAKE_SOURCE_DIR}") +set(SUPPORTED_ABI "arm64-v8a") +set(THIRD_PARTY_SO_DIR "${THIRD_PARTY_SO_ROOT}/${SUPPORTED_ABI}") + +# ========== 导入预编译库 ========== + +# OpenCV库 +set(OPENCV_LIB_NAME "opencv_java4") +add_library(${OPENCV_LIB_NAME} SHARED IMPORTED) +set_target_properties(${OPENCV_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${OPENCV_LIB_NAME}.so" +) + +# SNPE推理引擎 +set(SNPE_LIB_NAME "SNPE") +add_library(${SNPE_LIB_NAME} SHARED IMPORTED) +set_target_properties(${SNPE_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${SNPE_LIB_NAME}.so" +) + +# C++运行时 +set(C_SHARED_LIB_NAME "c++_shared") +add_library(${C_SHARED_LIB_NAME} SHARED IMPORTED) +set_target_properties(${C_SHARED_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${C_SHARED_LIB_NAME}.so" +) + +# 人脸识别库 +set(FACE_LIB_NAME "facerecognition") +add_library(${FACE_LIB_NAME} SHARED IMPORTED) +set_target_properties(${FACE_LIB_NAME} PROPERTIES + IMPORTED_LOCATION "${THIRD_PARTY_SO_DIR}/lib${FACE_LIB_NAME}.so" +) + +# ========== 头文件配置 ========== +function(recursive_include dir) + if(IS_DIRECTORY ${dir}) + include_directories(${dir}) + file(GLOB SUB_DIRS RELATIVE ${dir} "${dir}/*") + foreach(sub_dir ${SUB_DIRS}) + set(full_sub_dir "${dir}/${sub_dir}") + if(IS_DIRECTORY ${full_sub_dir}) + recursive_include(${full_sub_dir}) + endif() + endforeach() + endif() +endfunction() +recursive_include(${INCLUDE_ROOT}) + +# ========== 编译aimarket.so ========== +file(GLOB LOCAL_SRC_FILES + "${LOCAL_SRC_DIR}/aimarket.cpp" +) + +find_library(LOG_LIB log REQUIRED) +find_library(JNI_GRAPHICS_LIB jnigraphics REQUIRED) + +add_library(aimarket SHARED ${LOCAL_SRC_FILES}) + +target_compile_features(aimarket PRIVATE cxx_std_17) +target_compile_options(aimarket PRIVATE + -frtti + -fexceptions + -Wno-error=format-security + -fPIC +) + +target_link_libraries(aimarket PRIVATE + ${C_SHARED_LIB_NAME} + ${FACE_LIB_NAME} + ${OPENCV_LIB_NAME} + ${SNPE_LIB_NAME} + ${JNI_GRAPHICS_LIB} + ${LOG_LIB} + c m dl atomic +) +``` + +### JNI接口层实现 + +#### Java层接口定义 + +创建 `JniManager.java`： + +```java +package com.quectel.aimarket.manager; + +public class JniManager { + private static final String TAG = "JniManager"; + private static final String SO_NAME = "aimarket"; + private static volatile JniManager INSTANCE; + private static boolean isSoLoaded = false; + + // 加载Native库 + static { + try { + System.loadLibrary(SO_NAME); + isSoLoaded = true; + } catch (UnsatisfiedLinkError e) { + isSoLoaded = false; + } + } + + // 单例模式 + public static JniManager getInstance() { + if (INSTANCE == null) { + synchronized (JniManager.class) { + if (INSTANCE == null) { + INSTANCE = new JniManager(); + } + } + } + return INSTANCE; + } + + // ========== 人脸识别接口 ========== + + /** + * 设置SNPE环境变量 + * @param soPath Native库路径 + */ + public native void setAdspEnv(String soPath); + + /** + * 初始化人脸识别模型 + * @param faceDetModelPath 人脸检测模型路径 + * @param faceLandmarkModelPath 关键点模型路径 + * @param faceRegModelPath 特征提取模型路径 + * @param faceDbPath 人脸库路径 + * @return 0-成功，负数-错误码 + */ + public native int initFaceRecognition( + String faceDetModelPath, + String faceLandmarkModelPath, + String faceRegModelPath, + String faceDbPath + ); + + /** + * 从图片进行人脸识别 + * @param imagePath 输入图片路径 + * @param saveDir 结果图片保存目录 + * @return 结果图片路径 + */ + public native String faceRecognitionFromImage(String imagePath, String saveDir); + + /** + * 从相机流进行实时人脸识别 + * @param mergedYuvData YUV数据 + * @param width 图像宽度 + * @param height 图像高度 + * @param rotation 旋转角度 + * @return 检测结果数组 [人脸数, x, y, w, h, id, score, 关键点数, 关键点坐标...] + */ + public native float[] faceRecognitionFromCameraX( + byte[] mergedYuvData, + int width, + int height, + int rotation + ); + + /** + * 释放人脸识别资源 + */ + public native void releaseCameraFaceRecog(); +} +``` + +#### C++层JNI实现 + +创建 `aimarket.cpp`（核心代码片段）： + +```cpp +#include +#include +#include +#include +#include "faceRecogInterfaceEX.hpp" +#include "opencv2/opencv.hpp" + +#define LOG_TAG "JNI_aimarket" +#define LOGD(...) __android_log_print(ANDROID_LOG_DEBUG, LOG_TAG, __VA_ARGS__) +#define LOGE(...) __android_log_print(ANDROID_LOG_ERROR, LOG_TAG, __VA_ARGS__) + +// 全局人脸识别实例 +static std::unique_ptr g_frIfEx = nullptr; +static std::vector> g_faceFeatures; +static int g_validFeatureDim = 0; + +void SetAdspLibraryPath(const std::string &nativeLibPath) { + std::stringstream spath; + std::stringstream spath1; + spath << nativeLibPath << ";/vendor/lib/rfsa/adsp;/vendor/dsp;/vendor/dsp/cdsp"; + spath1 << nativeLibPath; + LOGD("nativeLibPath is %s", nativeLibPath.c_str()); + if (setenv("ADSP_LIBRARY_PATH", spath.str().c_str(), 1 /*override*/) == 0) { + LOGD("SNPE环境配置成功"); + } else { + LOGD("SNPE环境配置失败"); + } + if (setenv("LD_LIBRARY_PATH", spath1.str().c_str(), 1 /*override*/) == 0) { + LOGD("SNPE环境配置成功"); + } else { + LOGD("SNPE环境配置失败"); + } + LOGD("ADSP_LIBRARY_PATH is %s", getenv("ADSP_LIBRARY_PATH")); + LOGD("LD_LIBRARY_PATH is %s", getenv("LD_LIBRARY_PATH")); +} + +extern "C" JNIEXPORT void JNICALL +Java_com_quectel_aimarket_manager_JniManager_setAdspEnv( + JNIEnv *env, + jobject thiz, + jstring soLibraryPath +) { + const char *ldPath = env->GetStringUTFChars(soLibraryPath, nullptr); + + if (ldPath == nullptr) { + LOGE("环境变量路径为空！"); + goto end; + } + + SetAdspLibraryPath(ldPath); + + end: + env->ReleaseStringUTFChars(soLibraryPath, ldPath); +} + +/** + * 初始化人脸识别模型 + */ +extern "C" +JNIEXPORT jint JNICALL +Java_com_quectel_aimarket_manager_JniManager_initFaceRecognition( + JNIEnv *env, jobject thiz, + jstring face_det_model_path, + jstring face_landmark_model_path, + jstring face_reg_model_path, + jstring face_db_path) { + + try { + if (g_frIfEx != nullptr) { + LOGD("人脸识别已初始化"); + return 0; + } + + // 获取模型路径 + char *detModel = const_cast(env->GetStringUTFChars(face_det_model_path, nullptr)); + char *landmarkModel = const_cast(env->GetStringUTFChars(face_landmark_model_path, nullptr)); + char *regModel = const_cast(env->GetStringUTFChars(face_reg_model_path, nullptr)); + char *dbPath = const_cast(env->GetStringUTFChars(face_db_path, nullptr)); + + // 创建人脸识别实例 + g_frIfEx = std::make_unique( + detModel, landmarkModel, regModel); + + int err = g_frIfEx->init(); + if (err != 0) { + g_frIfEx.reset(); + return err; + } + + // 加载人脸库 + std::vector filePaths; + glob(dbPath, filePaths); + g_faceFeatures.clear(); + + for (const auto &path: filePaths) { + cv::Mat img = cv::imread(path); + if (img.empty()) continue; + + std::vector faceResults; + err = g_frIfEx->detect_face(img, faceResults, 0.5, 0.5); + + if (err == 1 && !faceResults.empty() && !faceResults[0].feature.empty()) { + g_faceFeatures.emplace_back(faceResults[0].feature); + g_validFeatureDim = faceResults[0].feature.size(); + } + } + + // 释放字符串资源 + env->ReleaseStringUTFChars(face_det_model_path, detModel); + env->ReleaseStringUTFChars(face_landmark_model_path, landmarkModel); + env->ReleaseStringUTFChars(face_reg_model_path, regModel); + env->ReleaseStringUTFChars(face_db_path, dbPath); + + return g_faceFeatures.empty() ? -2 : 0; + } catch (...) { + return -5; + } +} + +/** + * 相机流人脸识别 + */ +extern "C" +JNIEXPORT jfloatArray JNICALL +Java_com_quectel_aimarket_manager_JniManager_faceRecognitionFromCameraX( + JNIEnv *env, jobject thiz, + jbyteArray mergedYuvData_, + jint width, jint height, jint rotation) { + + if (g_frIfEx == nullptr || g_faceFeatures.empty()) { + jfloatArray emptyArray = env->NewFloatArray(1); + float emptyVal = 0.0f; + env->SetFloatArrayRegion(emptyArray, 0, 1, &emptyVal); + return emptyArray; + } + + // 获取YUV数据 + jbyte *mergedYuvData = env->GetByteArrayElements(mergedYuvData_, nullptr); + + // YUV转BGR + cv::Mat yuvMat(height * 3 / 2, width, CV_8UC1, (unsigned char *) mergedYuvData); + cv::Mat bgrMat; + cv::cvtColor(yuvMat, bgrMat, cv::COLOR_YUV2BGR_I420); + + // 图像旋转 + cv::Mat rotatedMat; + switch (rotation) { + case 90: cv::rotate(bgrMat, rotatedMat, cv::ROTATE_90_CLOCKWISE); break; + case 180: cv::rotate(bgrMat, rotatedMat, cv::ROTATE_180); break; + case 270: cv::rotate(bgrMat, rotatedMat, cv::ROTATE_90_COUNTERCLOCKWISE); break; + default: rotatedMat = bgrMat.clone(); break; + } + + // 人脸检测 + std::vector faceResults; + g_frIfEx->detect_face(rotatedMat, faceResults, 0.5f, 0.5f); + + // 特征匹配 + for (auto &face: faceResults) { + if (!face.feature.empty() && face.feature.size() == g_validFeatureDim) { + std::pair matchRes = g_frIfEx->cosine_similarity( + g_faceFeatures, face.feature, 0.6f); + face.code = matchRes.first; + } + } + + // 构建返回结果 + int resultSize = 1; + for (const auto &face: faceResults) { + resultSize += 7 + face.key_pts.size() * 2; + } + + jfloatArray resultArray = env->NewFloatArray(resultSize); + std::unique_ptr resultData(new float[resultSize]); + int idx = 0; + resultData[idx++] = static_cast(faceResults.size()); + + for (const auto &face: faceResults) { + resultData[idx++] = static_cast(face.x); + resultData[idx++] = static_cast(face.y); + resultData[idx++] = static_cast(face.width); + resultData[idx++] = static_cast(face.height); + resultData[idx++] = static_cast(face.code); + resultData[idx++] = 0.0f; // score + resultData[idx++] = static_cast(face.key_pts.size()); + for (const auto &pt: face.key_pts) { + resultData[idx++] = static_cast(pt.x); + resultData[idx++] = static_cast(pt.y); + } + } + + env->SetFloatArrayRegion(resultArray, 0, resultSize, resultData.get()); + env->ReleaseByteArrayElements(mergedYuvData_, mergedYuvData, JNI_ABORT); + return resultArray; +} + +/** + * 释放资源 + */ +extern "C" +JNIEXPORT void JNICALL +Java_com_quectel_aimarket_manager_JniManager_releaseCameraFaceRecog( + JNIEnv *env, jobject thiz) { + g_frIfEx.reset(); + g_faceFeatures.clear(); + g_validFeatureDim = 0; +} +``` + +### build.gradle配置 + +在 `app/build.gradle` 中添加： + +```gradle +android { + // ... 其他配置 ... + + defaultConfig { + // ... 其他配置 ... + + externalNativeBuild { + cmake { + cppFlags "" + abiFilters 'arm64-v8a' + } + } + ndk { + abiFilters 'arm64-v8a' + } + } + + externalNativeBuild { + cmake { + path "src/main/cpp/CMakeLists.txt" + version "3.22.1" + } + } + ndkVersion '21.4.7075529' +} +``` + +### 使用示例 + +#### 初始化模型 + +```java +public class MainActivity extends AppCompatActivity { + private JniManager jniManager; + + @Override + protected void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + + // 获取JNI管理器实例 + jniManager = JniManager.getInstance(); + + // 设置SNPE环境 + String nativeLibDir = getApplicationInfo().nativeLibraryDir; + jniManager.setAdspEnv(nativeLibDir); + + // 初始化人脸识别 + String modelPath = getFilesDir() + "/model/face_model"; + String faceDbPath = getFilesDir() + "/model/facedb"; + + int result = jniManager.initFaceRecognition( + modelPath + "/facedet_q_enc.dlc", + modelPath + "/facelandmark_q_enc.dlc", + modelPath + "/facereg_q_enc.dlc", + faceDbPath + ); + + if (result == 0) { + Log.d("AIMarket", "人脸识别初始化成功"); + } else { + Log.e("AIMarket", "人脸识别初始化失败: " + result); + } + } +} +``` + +#### 相机流人脸识别（以CameraX为例，需要导入相关依赖，也可自行选择别的相机实现方式） + +```java +// 使用CameraX获取预览帧 +previewView.getPreviewStreamState().observe(this, state -> { + // 相机准备就绪 +}); + +// 在ImageAnalysis中处理帧数据 +ImageAnalysis imageAnalysis = new ImageAnalysis.Builder() + .setTargetResolution(new Size(640, 480)) + .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) + .build(); + +imageAnalysis.setAnalyzer(executor, image -> { + // 获取YUV数据 + ImageProxy imageProxy = image; + byte[] yuvData = yuv420ToNv21(imageProxy); + + // 调用人脸识别 + float[] results = jniManager.faceRecognitionFromCameraX( + yuvData, + imageProxy.getWidth(), + imageProxy.getHeight(), + getRotationDegrees() + ); + + // 解析结果 + if (results != null && results.length > 1) { + int faceCount = (int) results[0]; + int idx = 1; + for (int i = 0; i < faceCount; i++) { + float x = results[idx++]; + float y = results[idx++]; + float w = results[idx++]; + float h = results[idx++]; + int id = (int) results[idx++]; + // 跳过score和关键点 + idx += 2 + (int) results[idx - 1] * 2; + + Log.d("FaceResult", "Face " + i + ": pos=(" + x + "," + y + + "), id=" + id); + } + } + + image.close(); +}); +``` + +#### 资源释放 + +```java +@Override +protected void onDestroy() { + super.onDestroy(); + if (jniManager != null) { + jniManager.releaseCameraFaceRecog(); + jniManager.releaseAiMarketResources(); + } +} +``` + +## demo演示 +1. 左侧tab栏选择想要体验的功能，Local Picture是选择本地图片识别，Live Camera是调用设备摄像头识别,APM是查看设备性能变化以及运行日志 +2. 选择Local Picture，点击Before区域的灰白区域即可上传图片 +3. 选择Live Camera，点击开始拍照，即可开启摄像头识别 + + +## 常见问题排查 + +**注意：** 本项目需要使用到系统库，需确保targetSdk ≤ 30,在安装应用之前需要先adb连接设备执行 `adb root` 命令，然后执行 `adb shell setenforce 0` 命令,且需要在算法的so库初始化之前设置adsp环境变量，才能正常使用，参考上面的代码。 + +| 问题 | 可能原因 | 解决方案 | +|----------------------|------------------------------------------|--------------------------------| +| 算法时效性失效 | 算法初始化后会有一个小时的时效性，超时会失效 | 退出当前应用并清理后台重新启动 | +| UnsatisfiedLinkError | .so库未正确加载 | 检查jniLibs目录结构和架构是否匹配以及环境变量是否设置 | +| 模型初始化失败(-1) | 模型文件不存在或路径错误 | 检查模型文件是否正确复制到设备 | +| 人脸库加载失败(-2) | 人脸库图片格式不正确 | 确保使用标准JPEG格式图片 | +| 检测结果为空 | 图像格式或尺寸不正确，有些相机的角度不固定，导致拍摄出来的画面出现旋转镜像等问题 | 检查YUV数据格式和旋转角度，对传入的图片进行正确旋转 | +| DSP不可用 | 设备不支持SNPE DSP | 更换设备 | diff --git a/docs/Quectel-Pi-M1/zh/sidebar.yaml b/docs/Quectel-Pi-M1/zh/sidebar.yaml index fa3c92b..bb958e3 100644 --- a/docs/Quectel-Pi-M1/zh/sidebar.yaml +++ b/docs/Quectel-Pi-M1/zh/sidebar.yaml @@ -180,7 +180,12 @@ items: #- label: 科研应用 # file: Applications/Open-Source-Projects/research_applications.md - # - label: AI应用 + - label: NPU应用 + file: NPU-Applications/README.md + items: + - label: 算法超市 + file: NPU-Applications/ai_market.md + # - label: AI应用 # file: NPU-Applications/README.md # items: # - label: NPU开发指南 @@ -199,7 +204,6 @@ items: # file: NPU-Applications/deepseek_deployment.md #- label: Chatgpt部署 # file: NPU-Applications/chatgpt_deployment.md - #- label: Q&A #file: FAQ/README.md #items: diff --git a/pages/index/zh/README.md b/pages/index/zh/README.md index 46f5e32..09140f4 100644 --- a/pages/index/zh/README.md +++ b/pages/index/zh/README.md @@ -61,8 +61,8 @@ // 根据选中的型号显示或隐藏内容 if (model === 'Quectel Pi M1') { contentSections.forEach((section, index) => { - // 显示用户手册、操作系统、支持配件、应用开发 - if ([0, 1, 2, 3, 5, 6].includes(index)) { + // 显示用户手册、操作系统、支持配件、应用开发、NPU应用 + if ([0, 1, 2, 3, 4, 5, 6].includes(index)) { section.style.display = 'block'; } else { section.style.display = 'none'; -- Gitee From 35962d072d65f729509029b1c058dc465da36e9b Mon Sep 17 00:00:00 2001 From: Will Li Date: Fri, 20 Mar 2026 11:34:03 +0800 Subject: [PATCH 2/2] =?UTF-8?q?=E4=BF=AE=E6=94=B9=E7=AE=97=E6=B3=95?= =?UTF-8?q?=E8=B6=85=E5=B8=82=E8=AF=B4=E6=98=8E=E6=96=87=E6=A1=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md | 8 +++++++- docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md | 7 ++++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md b/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md index 65b7dbb..113d987 100644 --- a/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md +++ b/docs/Quectel-Pi-M1/en/NPU-Applications/ai_market.md @@ -8,6 +8,8 @@ This document is the integration development guide for **AIMarket** Android SDK, AIMarket is a one-stop AI algorithm integration platform developed by Quectel, providing high-performance edge AI inference capabilities for Android devices. Built on Qualcomm SNPE (Snapdragon Neural Processing Engine) framework, the platform fully utilizes NPU/DSP hardware acceleration to achieve low-latency, low-power local AI inference. +[Official Platform Experience](https://www.quectel.com.cn/ai-open-platform/algostore) + ### Core Features - **Local Inference**: All algorithms run on-device, no network required, privacy protected @@ -80,7 +82,7 @@ plugins { #### Native Libraries (.so files) -The project depends on the following pre-compiled Native libraries (contact relevant personnel to obtain), which need to be placed in the `app/src/main/jniLibs/arm64-v8a/` directory: +The project depends on the following pre-compiled Native libraries ([available on official website](https://www.quectel.com.cn/ai-open-platform/algostore)), which need to be placed in the `app/src/main/jniLibs/arm64-v8a/` directory: | Library Name | Description | |-------|------| @@ -221,6 +223,10 @@ g_qrScanner->set_config(ZBAR_QRCODE, ZBAR_CFG_Y_DENSITY, 4); --- +## Model Conversion and Optimization Platform + +[AI Development Toolchain Experience Platform](https://www.quectel.com.cn/ai-open-platform/aide) + ## Model Migration Process (Face Recognition Example) This section takes face recognition as an example to introduce in detail how to integrate algorithm models from scratch in Android Studio. diff --git a/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md b/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md index 6f11b71..961b6b1 100644 --- a/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md +++ b/docs/Quectel-Pi-M1/zh/NPU-Applications/ai_market.md @@ -8,6 +8,8 @@ 算法超市是由移远通信（Quectel）开发的一站式AI算法集成平台，为Android设备提供高性能的边缘AI推理能力。该平台基于高通SNPE（Snapdragon Neural Processing Engine）框架构建，充分利用NPU/DSP硬件加速，实现低延迟、低功耗的本地AI推理。 +[官网体验平台](https://www.quectel.com.cn/ai-open-platform/algostore) + ### 核心特性 - **本地推理**：所有算法均在设备端运行，无需联网，保护隐私 @@ -80,7 +82,7 @@ plugins { #### Native依赖库（.so文件） -项目依赖以下预编译的Native库（可寻找相关人员提供），需放置在 `app/src/main/jniLibs/arm64-v8a/` 目录： +项目依赖以下预编译的Native库（[官网获取](https://www.quectel.com.cn/ai-open-platform/algostore)），需放置在 `app/src/main/jniLibs/arm64-v8a/` 目录： | 库名称 | 说明 | |-------|------| @@ -221,6 +223,9 @@ g_qrScanner->set_config(ZBAR_QRCODE, ZBAR_CFG_Y_DENSITY, 4); --- +## 模型转换与优化平台 +[AI开发工具链体验平台](https://www.quectel.com.cn/ai-open-platform/aide) + ## 模型移植流程（以人脸识别为例） 本节以人脸识别为例，详细介绍如何从零开始在Android Studio中集成算法模型。 -- Gitee