# PhyX

**Repository Path**: hf-datasets/PhyX

## Basic Information

- **Project Name**: PhyX
- **Description**: Mirror of https://huggingface.co/datasets/Cloudriver/PhyX
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-07-15
- **Last Updated**: 2025-07-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

---
license: mit
task_categories:
- question-answering
- visual-question-answering
- multiple-choice
language:
- en
tags:
- Physics
size_categories:
- 1K<n<10K
configs:
- config_name: default
  data_files:
  - split: test_mini
    path: dataset_card/test_mini-*
  - split: test
    path: dataset_card/test-*
- config_name: data_for_llms_eval
  data_files:
  - split: phyx_mc
    path: data_llms_eval/PhyX_MC.parquet
  - split: phyx_mini_mc
    path: data_llms_eval/PhyX_mini_MC.parquet
  - split: phyx_oe
    path: data_llms_eval/PhyX_OE.parquet
  - split: phyx_mini_oe
    path: data_llms_eval/PhyX_mini_OE.parquet
dataset_info:
- config_name: default
  features:
  - name: id
    dtype: string
  - name: question
    dtype: string
  - name: question_description
    dtype: string
  - name: question_description_simplified
    dtype: string
  - name: options
    sequence: string
  - name: answer
    dtype: string
  - name: image
    dtype: image
  - name: image_caption
    dtype: string
  - name: category
    dtype: string
  - name: subfield
    dtype: string
  - name: reasoning_type
    sequence: string
  splits:
  - name: test_mini
    num_bytes: 57952044.0
    num_examples: 1000
  - name: test
    num_bytes: 159959199.0
    num_examples: 3000
  download_size: 743182949
  dataset_size: 217911243.0
- config_name: data_for_llms_eval
  features:
  - name: index
    dtype: string
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: image
    dtype: string
  - name: category
    dtype: string
  - name: subfield
    dtype: string
  - name: reasoning_type
    sequence: string
  splits:
  - name: phyx_mc
    num_examples: 3000
  - name: phyx_mini_mc
    num_examples: 1000
  - name: phyx_oe
    num_examples: 3000
  - name: phyx_mini_oe
    num_examples: 1000
---
# PhyX: Does Your Model Have the "Wits" for Physical Reasoning?

Dataset for the paper "[PhyX: Does Your Model Have the &#34;Wits&#34; for Physical Reasoning?](https://arxiv.org/abs/2505.15929)".

For more details, please refer to the project page with **dataset exploration and visualization tools**: [https://phyx-bench.github.io/](https://phyx-bench.github.io/).

[[🌐 Project Page](https://phyx-bench.github.io/)] [[📖 Paper](https://arxiv.org/abs/2505.15929)] [[🔧 Evaluation Code](https://github.com/NastyMarcus/PhyX)]  [[🌐 Blog (中文)](https://mp.weixin.qq.com/s/okKn6WrWilPmo0_yOcDP3Q)]

## 🔔 News

- **[2025.05.27]** 🎉 PhyX is officially supported by [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) for easy evalution.
- **[2025.05.23]** 🚀 The [arXiv paper](https://arxiv.org/abs/2505.15929) is online!
- **[2025.05.21]** 🚀 We release the testmini set of PhyX at [Huggingface](https://huggingface.co/datasets/Cloudriver/PhyX) and the [evaluation code](https://github.com/NastyMarcus/PhyX)!

## 📝 About PhyX

PhyX is the first large-scale benchmark specifically designed to assess models' ability in physical reasoning through realistic, visually grounded scenarios.

PhyX includes 3,000 meticulously collected multimodal questions, covering 6 reasoning types across 25 sub-domains and 6 core domains: thermodynamics, electromagnetism, mechanics, modern physics, optics, and wave acoustics.

PhyX specializes in university-level challenging questions presented through realistic, high-fidelity visual scenarios. Unlike general-purpose benchmarks, our tasks require models to integrate visual cues with implicit physical laws, going beyond simple knowledge recall and demanding nuanced, context-driven inference. This design enables a rigorous evaluation of true multimodal reasoning about the physical world, exposing key limitations in current models’ capabilities when handling professional-level scientific problems.

PhyX consists of 3,000 visually grounded physics questions, carefully curated across six distinct physics domains:

- Mechanics (550)
- Electromagnetism (550)
- Thermodynamics (500)
- Wave/Acoustics (500)
- Optics (500)
- Modern Physics (400)

## 🔍 Dataset Versions

PhyX contains two subsets: testmini (1,000 questions) and test (3,000 questions). Each subset includes 12 versions tailored for different evaluation settings,
**📌 Notice: Our Hugging Face repository only includes the default-setting TSV files for both OE and MC formats (PhyX_mini_OE.tsv, PhyX_mini_MC.tsv, PhyX_OE.tsv, and PhyX_MC.tsv) to avoid confusion.
If you are looking for additional TSV files covering more evaluation settings as discussed in the paper, please refer to our full GitHub repository here:**
[PhyX](https://github.com/NastyMarcus/PhyX)

| File Name            | Type & Input Style                                                 | Description                            |
| -------------------- | ------------------------------------------------------------------ | -------------------------------------- |
| `PhyX_mini_OE.tsv` | OE / Text-DeRedundancy (Image + Simplified Description + Question) | OE version with simplified description |
| `PhyX_mini_MC.tsv` | MC / Text-DeRedundancy (Image + Simplified Description + Question) | MC version with simplified description |
| `PhyX_OE.tsv`      | OE / Text-DeRedundancy (Image + Simplified Description + Question) | OE version with simplified description |
| `PhyX_MC.tsv`      | MC / Text-DeRedundancy (Image + Simplified Description + Question) | MC version with simplified description |

- 🔍 `PhyX_mini_OE.tsv` and `PhyX_mini_MC.tsv` stands for the 1,000-questions testmini set;
- 🔍 `PhyX_OE.tsv` and `PhyX_MC.tsv` stands for the full version with 3,000-questions test set.

## 📘 Sample Format and Field Definitions

Each entry in PhyX is stored as a JSON object with the following fields:

| Field                    | Type   | Description                                                                  |
| ------------------------ | ------ | ---------------------------------------------------------------------------- |
| `index`                | int    | Index of the question                                                        |
| `question`             | string | Question                                                                     |
| `question_description` | string | Original full description of the problem                                     |
| `question_simply`      | string | Simplified version of the question description (used in `SIMPLY` versions) |
| `options`              | string | Answer options, format:`A:"...", B:"...", ...`                             |
| `answer`               | string | Ground truth answer                                                          |
| `image`                | string | Image filename (e.g.,`200.png`)                                            |
| `image_caption`        | string | Textual description of the image (only in `TL` versions)                   |
| `category`             | string | Physics category (e.g.,`"Optics"`)                                         |
| `subfield`             | string | Fine-grained physics subfield (e.g.,`"Geometrical Optics"`)                |
| `reasoning_type`       | string | Type(s) of Physical Reasoning                                                |

You can use this format to load and evaluate different question versions based on your model’s capability (e.g., multimodal, text-only).

## ✅ Cite

If you find **PhyX** useful for your your research and applications, please kindly cite using this BibTeX:

```bibtex
@misc{shen2025phyxdoesmodelwits,
      title={PhyX: Does Your Model Have the "Wits" for Physical Reasoning?}, 
      author={Hui Shen and Taiqiang Wu and Qi Han and Yunta Hsieh and Jizhou Wang and Yuyue Zhang and Yuxin Cheng and Zijian Hao and Yuansheng Ni and Xin Wang and Zhongwei Wan and Kai Zhang and Wendong Xu and Jing Xiong and Ping Luo and Wenhu Chen and Chaofan Tao and Zhuoqing Mao and Ngai Wong},
      year={2025},
      eprint={2505.15929},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2505.15929}, 
}
```

## ❤️ Contributors

> *Hui Shen `<sup>`1, 2 `</sup>`, Taiqiang Wu `<sup>`1 `</sup>`, Qi Han `<sup>`3 `</sup>`, Yunta Hsieh `<sup>`2 `</sup>`, Jizhou Wang `<sup>`4 `</sup>`, Yuyue Zhang `<sup>`3 `</sup>`, Yuxin Cheng `<sup>`1 `</sup>`, Zijian Hao `<sup>`3 `</sup>`, Yuansheng Ni `<sup>`5 `</sup>`, Xin Wang `<sup>`6 `</sup>`, Zhongwei Wan `<sup>`6 `</sup>`, Kai Zhang `<sup>`6 `</sup>`, Wendong Xu `<sup>`1 `</sup>`, Jing Xiong `<sup>`1 `</sup>`, Ping Luo `<sup>`1 `</sup>`, Wenhu Chen `<sup>`5 `</sup>`, Chaofan Tao `<sup>`1 `</sup>`, Z. Morley Mao `<sup>`2 `</sup>`, Ngai Wong `<sup>`1 `</sup>`.*

> *`<sup>`1 `</sup>`The University of Hong Kong, `<sup>`2 `</sup>`University of Michigan, `<sup>`3 `</sup>`Independent, `<sup>`4 `</sup>`University of Toronto, `<sup>`5 `</sup>`University of Waterloo, `<sup>`6 `</sup>`The Ohio State University.*