# preprocessed_commoncatalog-cc-by_DCAE

**Repository Path**: hf-datasets/preprocessed_commoncatalog-cc-by_DCAE

## Basic Information

- **Project Name**: preprocessed_commoncatalog-cc-by_DCAE
- **Description**: Mirror of https://huggingface.co/datasets/SwayStar123/preprocessed_commoncatalog-cc-by_DCAE
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-06-15
- **Last Updated**: 2025-06-15

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

---
license: cc
task_categories:
- text-to-image
language:
- en
pretty_name: 'Preprocessed Common catalogue (CC-BY) DCAE '
size_categories:
- 10M<n<100M
---

The images are resized and then encoded with the DC-AE f32 autoencoder. The resizing is done with a bucketmanager with base resolution 512x512, minimum side length 256, maximum side length 1024, all sides are divisible by 32 ofcourse as they needed to be encoded by the DCAEf32 encoder.
The captions are generated with moondream2, encoded with siglip and bert. (Bert embeddings variance is very high, so use a norm layer). The text embeddings are padded to 64 tokens, but i have provided the unpadded length aswell so you can prune it to the maximum in the batch and save compute.