# pure_c_llm_inference

**Repository Path**: xusun000/pure_c_llm_inference

## Basic Information

- **Project Name**: pure_c_llm_inference
- **Description**: 1234567890
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-07-07
- **Last Updated**: 2026-08-08

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Pure Inference

Portable C++20 inference runtime with a stable C ABI. The runtime is independent
of any single model family; Qwen is currently the first built-in backend.

Model backends and compute kernel backends are independent.

**Model backends**

- `qwen`: pure C++ Qwen family path (reference / portable)
- `openvino`: optional OpenVINO GenAI path for Intel **NPU / GPU / CPU** (see `docs/openvino-npu.md`)

**Kernel providers** (only used by the pure C++ `qwen` path)

- `portable`: dependency-free C++ fallback on every platform
- `accelerate`: automatically built on Apple platforms
- `metal`: Apple GPU backend using Metal Performance Shaders
- `mlx`: preferred Apple GPU backend when the `mlx-c` library is installed
- `cuda`: NVIDIA GPU backend using cuBLAS when the CUDA Toolkit is installed
- `cblas`: automatically built when CBLAS and BLAS are available

## Build

```sh
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure
```

Apple GPU build (Metal is enabled by default):

```sh
brew install mlx-c
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DPI_ENABLE_METAL=ON -DPI_ENABLE_MLX=ON
cmake --build build --parallel
```

If `mlx-c` is installed in a custom prefix, pass
`-DPI_MLX_C_ROOT=/path/to/prefix`. The build remains usable without MLX and
falls back to native Metal, Accelerate, or portable kernels.

NVIDIA GPU build (CUDA is detected automatically):

```sh
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DPI_ENABLE_CUDA=ON
cmake --build build --parallel
```

For a non-standard toolkit location, pass
`-DCUDAToolkit_ROOT=/path/to/cuda`. The provider registers only when a CUDA
device is available at runtime; otherwise kernel selection falls back normally.

Optional OpenVINO (NPU) backend:

```sh
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DPI_ENABLE_OPENVINO=ON
cmake --build build --parallel
```

Build outputs:

- `libpure_inference_core.a`: C++ runtime and built-in backends
- `libpure_inference`: shared C ABI library
- `pure-inference-cli`: interactive chat client
- `pure-inference-server`: OpenAI-compatible HTTP server

## Run

```sh
./build/pure-inference-cli /path/to/model
./build/pure-inference-server /path/to/model 8080
```

Use `--backend qwen` or `--backend openvino` to select a model backend.

Select a compute provider independently (pure `qwen` path only):

```sh
PI_KERNEL_BACKEND=portable ./run.sh /path/to/model
PI_KERNEL_BACKEND=accelerate ./run.sh /path/to/model
PI_KERNEL_BACKEND=metal ./run.sh /path/to/model
PI_KERNEL_BACKEND=mlx ./run.sh /path/to/model
PI_KERNEL_BACKEND=cblas ./run.sh /path/to/model
PI_KERNEL_BACKEND=cuda ./run.sh /path/to/model
```

OpenVINO device selection:

```sh
PI_OV_DEVICE=NPU ./build/pure-inference-cli /path/to/ov_ir --backend openvino
```

`auto` kernel selection picks `cuda` on NVIDIA systems, `mlx` or `metal` on
Apple systems, then the best available CPU provider, and always falls back to
`portable`. Immutable linear weights are cached by GPU providers so repeated
token decoding does not upload them for every matrix multiplication. Install
the CUDA Toolkit to enable the `cuda` provider, or an OpenBLAS development
package on Linux to enable the `cblas` provider.

The kernel interface remains vendor-neutral. Future AMD ROCm providers can be
added as separate translation units without changing Qwen
model code; see `docs/adding-a-kernel-backend.md`.

For Intel NPU setup, export, and trial scripts see `docs/openvino-npu.md`.
Also: `scripts/detect_npu.ps1`, `scripts/export_openvino.py`, `scripts/trial_genai.py`.

See `docs/adding-a-backend.md` for the model backend contract and
`docs/adding-a-kernel-backend.md` for the compute provider contract.