# TrianFlow
**Repository Path**: james9/TrianFlow
## Basic Information
- **Project Name**: TrianFlow
- **Description**: Code release for "Towards Better Generalization: Joint Depth-Pose Learning without PoseNet".
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2021-10-27
- **Last Updated**: 2021-10-28
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
## Towards Better Generalization: Joint Depth-Pose Learning without PoseNet
## 走向更好的泛化：没有 PoseNet 的联合深度姿势学习
Created by Wang Zhao, Shaohui Liu, Yezhi Shu, Yong-Jin Liu.
## Introduction
This implementation is based on our CVPR'2020 paper "Towards Better Generalization: Joint Depth-Pose Learning without PoseNet". You can find the arXiv version of the paper here. In this repository we release code and pre-trained models for *TrianFlow* (our method) and a strong baseline *PoseNet-Flow* method.
此实现基于我们的 CVPR'2020 论文“Towards Better Generalization: Joint Depth-Pose Learning without PoseNet”。 您可以在此处找到该论文的 arXiv 版本。 在这个存储库中，我们发布了 TrianFlow（我们的方法）和强大的基线 PoseNet-Flow 方法的代码和预训练模型。
## Installation
The code is based on Python3.6. You could use either virtualenv or conda to setup a specified environment. And then run:
代码基于Python3.6。 您可以使用 virtualenv 或 conda 来设置指定的环境。 然后运行：
安装Python3.6：
https://www.cnblogs.com/netflix/p/15026768.html
https://blog.csdn.net/weixin_45385568/article/details/115116470
```bash
pip install -r requirements.txt
```
## Run a demo
To run a depth prediction demo, you may need to first download the pretrained model from here.
要运行深度预测演示，您可能需要先从...
```bash
python test.py --config_file ./config/default_1scale.yaml --gpu 0 --mode depth --task demo --image_path ./data/demo/kitti.png --pretrained_model ./models/pretrained/depth_pretrained.pth --result_dir ./data/demo
######
python test.py --config_file ./config/kitti.yaml --gpu 0 --mode flow_3stage --task kitti_flow --pretrained_model ./model/kitti_flow.pth --result_dir ./save/flow
###
python test.py --config_file ./config/kitti.yaml --gpu 0 --mode depth --task demo --image_path ./data/demo/kitti.png --pretrained_model ./pretrained_models/kitti_depth_pretrained.pth --result_dir ./data/demo
```
This will give you a predicted depth map for demo image.
## Run experiments 运行实验
### Prepare training data 准备训练数据 :
1. For KITTI depth and flow tasks, download KITTI raw dataset using the script provided on the official website. You also need to download KITTI 2015 dataset to evaluate the predicted optical flow. Run the following commands to generate groundtruth files for eigen test images.
对于 KITTI 深度和流任务，使用官网提供的script下载 KITTI 原始数据集。 您还需要下载 KITTI 2015 dataset来评估预测的光流。 运行以下命令为特征测试图像生成地面实况文件。
```
cd ./data/eigen
python export_gt_depth.py --data_path /path/to/your/kitti/root
```
2. For KITTI Odometry task, download KITTI Odometry dataset.
对于 KITTI 里程计任务，请下载 KITTI Odometry dataset。
3. For NYUv2 experiments, download NYUv2 raw sequences and labeled data mat, also the traing test split mat from here. Put the labeled data and splits file under the same directory. The data structure should be:
对于 NYUv2 实验，下载 NYUv2 原始序列和标记数据垫，以及来自 < a href="https://github.com/ankurhanda/nyuv2-meta-data">here。 将标记的数据和拆分文件放在同一目录下。 数据结构应该是：
```
nyuv2
| basements
| cafe
| ...
nyuv2_test
| nyu_depth_v2_labeled.mat
| splits.mat
```
### Training:
1. Modify the configuration file in the ./config directory to set up your path. The config file contains the important paths and default hyper-parameters used in the training process.
修改 ./config 目录中的配置文件以设置您的路径。 配置文件包含训练过程中使用的重要路径和默认超参数。
2. For KITTI depth, we have the three-stage training schedule.
对于 KITTI 深度，我们有三个阶段的训练计划。
```bash
1. python train.py --config_file ./config/kitti.yaml --gpu [gpu_id] --mode flow --prepared_save_dir [name_of_your_prepared_dataset] --model_dir [your/directory/to/save/training/models]
2. python train.py --config_file ./config/kitti.yaml --gpu [gpu_id] --mode depth --prepared_save_dir [name_of_your_prepared_dataset] --model_dir [your/directory/to/save/training/models] --flow_pretrained_model [path/to/your/stage1_flow_model]
3. python train.py --config_file ./config/kitti_3stage.yaml --gpu [gpu_id] --mode depth_pose --prepared_save_dir [name_of_your_prepared_dataset] --model_dir [your/directory/to/save/training/models] --depth_pretrained_model [path/to/your/stage2_depth_model]
```
If you are running experiments on the dataset for the first time, it would first process data and save in the [prepared_base_dir] path defined in your config file.
For other datasets like KITTI Odometry and NYUv2 dataset, you could run with the same commands using the appropriate config file.
如果您是第一次在数据集上运行实验，它会首先处理数据并保存在配置文件中定义的 [prepared_base_dir] 路径中。
对于其他数据集，如 KITTI Odometry 和 NYUv2 数据集，您可以使用适当的配置文件使用相同的命令运行。
We also implement and release codes for the strong baseline *PoseNet-Flow* method, you could run it by two-stage training:
我们还实现并发布了强基线 *PoseNet-Flow* 方法的代码，您可以通过两阶段训练来运行它：
```bash
1. python train.py --config_file [path/to/your/config/file] --gpu [gpu_id] --mode flow --prepared_save_dir [name_of_your_prepared_dataset] --model_dir [your/directory/to/save/training/models]
2. python train.py --config_file [path/to/your/config/file] --gpu [gpu_id] --mode flowposenet --prepared_save_dir [name_of_your_prepared_dataset] --model_dir [your/directory/to/save/training/models] --flow_pretrained_model [path/to/your/stage1_flow_model]
```
### Evaluation:
We provide pretrained models here for different tasks. The performance could be slightly different with the paper due to randomness.
我们在here为不同的任务提供了预训练模型。 由于随机性，性能可能与论文略有不同。
1. To evaluate the monocular depth estimation on kitti eigen test split, run:
要评估 kitti 特征测试拆分的单目深度估计，请运行：
```bash
python test.py --config_file ./config/kitti.yaml --gpu [gpu_id] --mode depth --task kitti_depth --pretrained_model [path/to/your/model] --result_dir [path/to/save/results]
```
2. To evaluate the monocular depth estimation on nyuv2 test split, run:
要评估 nyuv2 测试拆分上的单目深度估计，请运行：
```bash
python test.py --config_file ./config/nyu.yaml --gpu [gpu_id] --mode depth --task nyuv2 --pretrained_model [path/to/your/model] --result_dir [path/to/save/results]
```
3. To evaluate the optical flow estimation on KITTI 2015, run:
要评估 KITTI 2015 上的光流估计，请运行：
```bash
python test.py --config_file ./config/kitti.yaml --gpu [gpu_id] --mode flow_3stage --task kitti_flow --pretrained_model [path/to/your/model] --result_dir [path/to/save/results]
```
4. To evaluate the visual odometry task on KITTI Odometry dataset, first get predictions on a single sequence and then evaluate:
要评估 KITTI Odometry 数据集上的视觉里程计任务，首先获得对单个序列的预测，然后评估：
```bash
python infer_vo.py --config_file ./config/odo.yaml --gpu [gpu_id] --traj_save_dir_txt [where/to/save/the/prediction/file] --sequences_root_dir [the/root/dir/of/your/image/sequences] --sequence [the sequence id] ----pretrained_model [path/to/your/model]
python ./core/evaluation/eval_odom.py --gt_txt [path/to/your/groundtruth/poses/txt] --result_txt [path/to/your/prediction/txt] --seq [sequence id to evaluate]
```
You could evaluate on the sampled KITTI odometry dataset by simply sampling the raw image sequences and gt-pose txt. Then run *infer_vo.py* on the sampled image sequence and *eval_odom.py* with predicted txt and sampled gt txt to get results.
您可以通过简单地采样原始图像序列和 gt-pose txt 来评估采样的 KITTI 里程计数据集。 然后在采样的图像序列上运行 *infer_vo.py* 并使用预测的 txt 和采样的 gt txt 运行 *eval_odom.py* 以获得结果。
### Citation
If you find our work useful in your research, please consider citing:
如果您发现我们的工作对您的研究有用，请考虑引用：
```
@inproceedings{zhao2020towards,
title={Towards Better Generalization: Joint Depth-Pose Learning without PoseNet},
author={Zhao, Wang and Liu, Shaohui and Shu, Yezhi and Liu, Yong-Jin},
booktitle={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2020}
}
```
### Related Projects
Digging into Self-Supervised Monocular Depth Prediction.
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation.
Visual Odometry Revisited: What Should Be Learnt?