# LLMSafetyBenchmark

**Repository Path**: devdz/LLMSafetyBenchmark

## Basic Information

- **Project Name**: LLMSafetyBenchmark
- **Description**: A comprehensive framework for assessing the security capabilities of large language models (LLMs) through multi-dimensional testing.
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-03-09
- **Last Updated**: 2026-03-09

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Introduction to LLMSafetyBenchmark
This project aims to comprehensively evaluate the safety capabilities of large language models (LLMs) by using a multi-dimensional testing framework to assess model performance in security-related scenarios. The evaluation system includes various question types such as multiple-choice questions, subjective questions, and CTF practical questions, covering security issues at different difficulty levels.

![image](https://github.com/user-attachments/assets/f9462804-efdb-40c1-a20e-14021816e84f)
![image](https://github.com/user-attachments/assets/12c01b2b-7701-4003-ab66-76225349610c)
# Usage Instructions


## Test Script

```bash
# Parameter description:  
# --model       Specify the model name  
# --fewshot     Select the number of few-shot examples (optional: 2 or 5)  
# --apikey      API key for Silicomotion  
# --datasets    Specify the path to the test dataset  

python ./llms_eval_api.py \
  --model qwen25coder_7b \
  --fewshot 5 \
  --datasets /home/A_master/LLMsEval/codes/datas/all/obj_6352_aigen.json \
  --apikey sk-rcewjdxsgjvupcyedtqzttgjqxsjpvhtpeavunjccuvtdesn
```

## model download
```bash
python ./models/load_models --model [path of the target model in the modelscope] --cache-dir [local path of the downloaded model]
```

## Test Question Format Specification
```json
{
  "id": 1,
  "source": "owasp_mastg",
  "question": "An Android application developer is tasked with ensuring the security of sensitive user information. The developer has several options for data storage, each with different security implications. Which of the following storage options should the developer avoid to prevent exposing sensitive user data to other applications on the device?",
  "choices": [
    "A: Using SharedPreferences with MODE_PRIVATE to store user preferences and settings.",
    "B: Storing user credentials in an unencrypted SQLite database accessible to the app.",
    "C: Implementing SQLCipher to encrypt SQLite databases containing sensitive user information.",
    "D: Saving encrypted user data in the Android Keystore system."
  ],
  "answer": "B",
  "topics": [
    "ApplicationSecurity"
  ],
  "keyword": "SQLite",
  "tag": "ApplicationSecurity",
  "mission_class": "multi"
}
```
# Dataset Distribution
![image](https://github.com/user-attachments/assets/15154c35-7ba2-44c7-8b1b-e4e0a51f4b0d)
![image](https://github.com/user-attachments/assets/d67712be-c1ee-4faf-93b3-deb880ff47ec)

# Eval Results
![image](https://github.com/user-attachments/assets/000da0e7-5c64-4745-9499-9ba37add9938)
![image](https://github.com/user-attachments/assets/f96e4d5a-cb79-49c3-824c-d23402a1e1cd)
![image](https://github.com/user-attachments/assets/75891664-6e2e-4049-b8eb-2b8691a53f81)


# Web Display Solution

## Web program usage

### backend
```bash
cd ./backend

# Install dependencies (if not already installed)
pip install -r requirements.txt

# Start the backend server
python app.py
```
### frontend
```bash
# Navigate to frontend directory
cd ./frontend

# Install dependencies
npm install

# Start the development server
npm run serve
```