# research_semantic_localization

**Repository Path**: CSS365_admin/research_semantic_localization

## Basic Information

- **Project Name**: research_semantic_localization
- **Description**: 语义定位的研究资料
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 7
- **Created**: 2020-10-08
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# 语义定位 （图像相似性判断，重定位）

## 1. 研究目标

人类的感知没有精确计算自己所在场景的位置信息，仅仅通过场景的相似性来进行定位，既通过目前看到的场景和之前走过的那个场景相似从而完成定位。因此通过深度学习来对场景进行相似性判断，并和地图库中的场景进行比对，可以模拟人的认知过程，从而实现基于视觉（非几何计算）的定位。

主要的研究目标有：
1. 单张图像的相似性判断已经比较多，而序列化的图像的场景相似性判断还没有。研究如何将多张图像所拍摄的场景进行建模，可以构建图，然后在地图（图网络）上进行子图匹配，找到最优的匹配点。

2. 如何更好的生成场景中物体的特征，可以NetVLAD或参考《Image Matching Based on Deep Feature and Spatial Correlation Graph》中的FE-Net。

3. 如何构建图卷积神经网络(Graph Convolutional Networks)来计算相似性。

4. 如何将所经过的场景提取特征并保存到一个图中，从而完成语义地图的构建。



## 2. 主要思路

研究思路：
1. 先通读一下主要的参考文献，建立对所研究问题的基本认识，了解基本的方法等。
2. 找一些代码运行一下，建立直觉的认识，并熟悉数据集。
3. 可以从基本的深度学习提取特征开始，然后再深入到图卷积神经网络。



具体的研究方法（需要尝试）：

1. 使用EdgeBox或者其他方法找到感兴趣的区域，然后提取对象的深度学习特征
    - EdgeBox 仅仅提取感兴趣的区域 [论文](https://www.microsoft.com/en-us/research/wp-content/uploads/2014/09/ZitnickDollarECCV14edgeBoxes.pdf)，[中文解释](https://blog.csdn.net/wsj998689aa/article/details/39476551)
2. 研究Siamese网络，提高特征的区分能力 (可以参考《Image Matching Based on Deep Feature and Spatial Correlation Graph》中的FE-Net)
3. 研究Graph Convolutional Networks，如何提取网络节点的特征，并做图匹配等。



## 3. 参考代码

* Improved version of DBow2 (https://github.com/rmsalinas/DBow3)
* FBOW (Fast Bag of Words) is an extremmely optimized version of the DBow2/DBow3 libraries (https://github.com/rmsalinas/fbow)
* Robust Visual Robot Localization Across Seasons using Network Flows (https://github.com/MHassanNadeem/localization-network-flows)
* NetVLAD: CNN architecture for weakly supervised place recognition (https://www.di.ens.fr/willow/research/netvlad/)
* Graph Convolutional Networks in PyTorch (https://github.com/tkipf/pygcn)
* Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (https://github.com/mdeff/cnn_graph)
* Graph Convolutional Networks (GCNs) (https://github.com/sungyongs/graph-based-nn)



## 4. 参考文献

更多的参考文档可以参考[references](references)目录

### 4.0 综述
* [2015 Visual Place Recognition: A Survey](references/survey/2015 Visual Place Recognition: A Survey.pdf)
* [基于视觉地图的视觉定位](https://www.zhihu.com/column/c_1287353014030585856)


### 4.1 特征提取

* NetVLAD CNN architecture for weakly supervised place recognitiondf
* Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free
* 2020 Visual search over billions of aerial and satellite images

### 4.2 网络方法
* Image Matching Based on Deep Feature and Spatial Correlation Graph
* Location Graphs for Visual Place Recognition
* Learning Convolutional Neural Networks for Graphs
* Robust Visual Semi-Semantic Loop Closure Detection by a Covisibility Graph and CNN Features
* Siamese Network
    - https://github.com/delijati/pytorch-siamese

    
### 4.3 图神经网络
* Learning Convolutional Neural Networks for Graphs
* Image Matching Based on Deep Feature and Spatial Correlation Graph


### 4.4 多视角的检索
* Lending Orientation to Neural Networks for Cross-view Geo-localization  https://github.com/Liumouliu/OriCNN
* Optimal Feature Transport for Cross-View Image Geo-Localization https://github.com/shiyujiao/cross_view_localization_CVFT


### 4.5 索引方法
* Tree-based indexing for real-time ConvNet landmark-based visual place recognition
* 2020 Visual search over billions of aerial and satellite images


### 4.6 Codes

* Keras implementation of the Netvlad for visual place recognition https://github.com/crlz182/Netvlad-Keras

* LoST - Visual Place Recognition using Visual Semantics for Opposite Viewpoints across Day and Night https://github.com/oravus/lostX
* visual place recognition in changing enviroments https://github.com/PRBonn/vpr_relocalization
* PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018 https://github.com/mikacuy/pointnetvlad
* NetVLAD: CNN architecture for weakly supervised place recognition https://github.com/Relja/netvlad

* Visual place recognition from opposing viewpoints under extreme appearance variations https://github.com/oravus/seq2single
* Optimal Feature Transport for Cross-View Image Geo-Localization https://github.com/shiyujiao/cross_view_localization_CVFT


### 4.7 Dataset
* University1652-Baseline https://github.com/layumi/University1652-Baseline
* Places: A 10 million image database
 for scene recognition
* 24/7 Place Recognition by View Synthesis http://www.ok.ctrl.titech.ac.jp/~torii/project/247/