# ParaGen

**Repository Path**: ByteDance/ParaGen

## Basic Information

- **Project Name**: ParaGen
- **Description**: ParaGen is a PyTorch deep learning framework for parallel sequence generation.
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-08-26
- **Last Updated**: 2026-01-29

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# ParaGen 
  
ParaGen is a PyTorch deep learning framework for parallel sequence generation.
Apart from sequence generation, ParaGen also enhances various NLP tasks, including 
sequence-level classification, extraction and generation.
                     
## Requirements and Installation  

* Install third-party dependent package:
```bash
# Ubuntu
apt-get install libopenmpi-dev libssl-dev openssh-server
# CentOS
yum install openmpi openssl openssh-server
# Conda
conda install -c conda-forge mpi4py
```
* To install ParaGen from source:
```bash  
cd ParaGen
pip install -e .
``` 
* For distributed training on multiple GPUs, run `ParaGen` with `torch.distributed`
```bash
python -m torch.distributed.launch --nproc_per_node {GPU_NUM} paragen/entries/run.py --configs {config_file}
```
You can also use `horovod` for distributed training. Install `horovod` with
``` bash
# require CMake to install horovod. (https://cmake.org/install/)
HOROVOD_WITH_PYTORCH=1 HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_NCCL_HOME=${NCCL_ROOT_DIR} pip install horovod
```
Then run `ParaGen` with `horovod`:
```bash
horovodrun -np {GPU_NUM} -H localhost:{GPU_NUM} paragen-run --config {config_file}
```
* Install lightseq to faster train:
``` bash
pip install lightseq
```

## Getting Started

Before using `ParaGen`, it would be helpful to overview how `ParaGen` works.

`ParaGen` is designed as a `task-oriented` framework, where `task` is regarded as the core of all the codes.
A specific task selects all the components for support itself, such as model architectures, training strategies, dataset, and data processing.
Any component within `ParaGen` can be customized, while the existing modules and methods are used as a plug-in library.

As tasks are considered as the core of `ParaGen`, it works with various `modes`, such as `train`, `evaluate`, `preprocess` and `serve`.
Tasks act differently under different modes, by reorganizing the components without code modification.

Please refer to [examples](examples) for detailed instructions. 

# ParaGen Usage and Contribution

We welcome any experimental algorithms on ParaGen.

- Install ParaGen;
- Create your own paragen-plugin libraries under `third_party`;
- Experiment your own algorithms;
- Write a reproducible shell script;
- Create a merge request and assign reviewers to any of us.