# pactran_metrics

**Repository Path**: mirrors_google-research/pactran_metrics

## Basic Information

- **Project Name**: pactran_metrics
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-07-19
- **Last Updated**: 2025-12-04

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# PACTran Metrics

This is the code repository for the paper: PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks (`https://arxiv.org/abs/2203.05126`).

This is not an officially supported Google product.

## Introduction

PACTran is a theoretically grounded family of metrics for pretrained model selection and transferability measurement. The family is derived from the optimal PAC-Bayesian bound under the transfer learning setting. It contains three metric instantiations: PACTran-Dirichlet, PACTran-Gamma, PACTran-Gaussian.
Our experiments showed that PACTran-Gaussian is a more consistent and effective transferability measure compared to other existing selection methods.

## How to use

In the following, we provide the instructions of using the metrics (while using the caltech101 task in the Visual Task Adaptation Benchmark (VTAB) [1] and the Sup-100% checkpoint as an example). Run the following scripts in the parent dir of `pactran_metrics`.

- Prerequisites:
  - Tensorflow
  - Tensorflow-Hub
  - Tensorflow-datasets
  - Numpy
  - Scipy
  - Scikit-learn

- For penultimate layer feature prediction, run
```
python -m pactran_metrics.adapt_and_eval \
--hub_module 'https://tfhub.dev/vtab/sup-100/1'  \
--hub_module_signature default \
--finetune_layer default \
--work_dir /tmp/all_features/sup-100/caltech101/ \
--dataset 'caltech101' \
--batch_size 512 \
--batch_size_eval 512 \
--initial_learning_rate 0.001 \
--decay_steps 1500,3000,4500 \
--max_steps 1 \
--run_adaptation=False \
--run_evaluation=False \
--run_prediction=True \
--linear_eval=False
```

- For whole network finetuning, run
```
python -m pactran_metrics.adapt_and_eval \
--hub_module 'https://tfhub.dev/vtab/sup-100/1'  \
--hub_module_signature default \
--finetune_layer default \
--work_dir /tmp/all_models/sup-100/caltech101/ \
--dataset 'caltech101' \
--batch_size 512 \
--batch_size_eval 512 \
--initial_learning_rate 0.001 \
--decay_steps 1500,3000,4500 \
--max_steps 5000 \
--run_adaptation \
--use_anil False
```

- For top-layer only finetuning, run
```
python -m pactran_metrics.anil_classifier \
--hub_module="https://tfhub.dev/vtab/sup-100/1" \
--dataset="caltech101" \
--work_dir=/tmp/all_results/caltech101 \
--feature_dir=/tmp/all_features
```

- For metrics estimation, run
```
python -m pactran_metrics.compute_metrics \
--hub_module="https://tfhub.dev/vtab/sup-100/1" \
--dataset="caltech101" \
--work_dir=/tmp/all_results/caltech101 \
--feature_dir=/tmp/all_features \
--num_examples=2
```
- All available datasets (in `./data` folder):
  - "caltech101"
  - "oxford_flowers102"
  - "patch_camelyon"
  - "sun397"
  - "dmlab"
  - "cifar"
  - "ddsm"
  - "oxford_iiit_pet"
  - "smallnorb"
  
- All available model URLs:
  - "https://tfhub.dev/vtab/sup-100/1" \
  - "https://tfhub.dev/vtab/sup-rotation-100/1" \
  - "https://tfhub.dev/vtab/sup-exemplar-100/1" \
  - "https://tfhub.dev/vtab/semi-exemplar-10/1" \
  - "https://tfhub.dev/vtab/semi-rotation-10/1" \
  - "https://tfhub.dev/vtab/rotation/1" \
  - "https://tfhub.dev/vtab/exemplar/1" \
  - "https://tfhub.dev/vtab/relative-patch-location/1" \
  - "https://tfhub.dev/vtab/jigsaw/1" \
  - "https://tfhub.dev/vtab/uncond-biggan/1" \
  - "https://tfhub.dev/vtab/cond-biggan/1" \
  - "https://tfhub.dev/vtab/wae-mmd/1" \
  - "https://tfhub.dev/vtab/vae/1" \
  - "https://tfhub.dev/vtab/wae-ukl/1" \
  - "https://tfhub.dev/vtab/wae-gan/1"

## Reference:

[1] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga,
J., Pinto, A.S., Neumann, M., Dosovitskiy, A., et al.: A large-scale study of
representation learning with the visual task adaptation benchmark. arXiv preprint
arXiv:1910.04867 (2019)