# QSync

**Repository Path**: ByteDance/QSync

## Basic Information

- **Project Name**: QSync
- **Description**: Official resporitory for "QSync: Adpative Mixed-Precision for Training Synchronization".
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-12-15
- **Last Updated**: 2026-01-31

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# QSync
Official resporitory for "IPDPS' 24 QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices".

### Description
QSync aims to explore the potential of removing unnecessary quantized operations to improve training accuracy. It achieves this through the following components:
- Quantization perturbation indicator/Replayer for analyzing the global data flow graph's memory and latency under mixed-precision (Predictor)
- Allocator for selecting the optimal quantized operations for training (Allocator / Syncer)
- Support for low-precision backends (CUTLASS, CUDNN) (LP-PyTorch)

In particular, QSync addresses a specific practical scenario: hybrid-cluster training, which involves inference GPUs with power capabilities (memory, compute) and training GPUs with higher capabilities.

The provided scripts support both convolution-based and transformer-based models.

*NOTE*: The project is a bit old. The performance of kernel implementation may not catch up with latest PyTorch.

### Set Environment
Clone the repo
`git clone --recursive https://github.com/bytedance/QSync.git`

#### Docker
- run `build.sh` under `dockerfile`
- run `run.sh`, specifiying the necessary path mounting inside.
- run `pip install -e .` right in the root folder of QSync, compilation of kernels will start.

#### Manual Installation
- Some libs may hard to install without proxy. Change `<abspath_to_root>` in `m_install.sh` to the absolute path to the root folder. Then
1. `bash m_install.sh`
2. `make`

### Usage
QSync is implemented under the `qsync` folder, composed of `syncer`, `predictor` and `LpTorch`.
- to use LpTorch and convert your model to mixed-biwdith model, use `model = QModule(model)`
- See detail for usage of predictor and syncer in the corresponding page.
- See sample under `benchmark_convs / benchmark_transformers`

*notice* the cross-node cost modeling is not as accurate as single-node is. Extra efforts required to align the communication start.