diff --git a/tutorials/notebook/convert_dataset_to_mindspore_data_format/convert_dataset_to_mindspore_data_format.ipynb b/tutorials/notebook/convert_dataset_to_mindspore_data_format/convert_dataset_to_mindspore_data_format.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..60f0ffe0c6ad6562918a6295e1e0f7293abc0cfb
--- /dev/null
+++ b/tutorials/notebook/convert_dataset_to_mindspore_data_format/convert_dataset_to_mindspore_data_format.ipynb
@@ -0,0 +1,975 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 将数据集转换为MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 概述\n",
+ "\n",
+ "用户可以将非标准的数据集转换为MindSpore数据格式—MindRecord，从而方便地加载到MindSpore中进行训练。同时，MindSpore在部分场景做了性能优化，使用MindRecord数据格式可以获得更好的性能体验。\n",
+ "\n",
+ "MindSpore数据格式具备的特征如下：\n",
+ "- 实现多变的用户数据统一存储、访问，训练数据读取更加简便。\n",
+ "- 数据聚合存储，高效读取，且方便管理、移动。\n",
+ "- 高效的数据编解码操作，对用户透明、无感知。\n",
+ "- 可以灵活控制分区的大小，实现分布式训练。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 整体流程\n",
+ "\n",
+ "1. 准备环节。\n",
+ "2. 将MNIST数据集转换为MindSpore数据格式。\n",
+ "3. 将CSV数据集转换为MindSpore数据格式。\n",
+ "4. 将CIFAR-10数据集转换为MindSpore数据格式。\n",
+ "5. 将CIFAR-100数据集转换为MindSpore数据格式。\n",
+ "6. 将ImageNet数据集转换为MindSpore数据格式。\n",
+ "7. 用户自定义生成MindSpore数据格式。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 准备环节"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 导入模块\n",
+ "\n",
+ "该模块提供API以加载和处理数据集。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import mindspore.dataset as ds"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 创建目录"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- 在jupyter工作目录下创建dataset目录，本次体验将所有的原始数据集都放在该目录下。\n",
+ "- 在jupyter工作目录下创建transform目录，本次体验将所有的转换数据集都放在该目录下。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 将MNIST数据集转换为MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### MNIST数据集下载"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- 训练数据集：\n",
+ " > http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
+ " > http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
+ "- 测试数据集：\n",
+ " > http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
+ " > http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
+ "- 将下载好的数据集放在`jupyter工作目录/dataset/MnistData/`下，如图所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " dataset/MnistData/ \n",
+ " ├── t10k-images-idx3-ubyte.gz \n",
+ " ├── t10k-labels-idx1-ubyte.gz \n",
+ " ├── train-images-idx3-ubyte.gz \n",
+ " └── train-labels-idx1-ubyte.gz "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### MNIST数据集转换"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`MnistToMR`这个类用于将MNIST数据集转换为MindSpore数据格式，参数用法如下：\n",
+ "- `source` - 包含t10k-images-idx3-ubyte.gz，train-images-idx3-ubyte.gz，t10k-labels-idx1-ubyte.gz，train-labels-idx1-ubyte.gz的目录，本例中使用变量`mnist_path`传入该参数。\n",
+ "- `destination` - 要转换成MindSpore数据格式文件的目录，本例中使用变量`mnist_mindrecord_path`传入该参数。\n",
+ "- `partition_number` - 分区的大小，默认为1，本例中使用的默认参数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from mindspore.mindrecord import MnistToMR\n",
+ "\n",
+ "mnist_path = './dataset/MnistData'\n",
+ "mnist_mindrecord_path = './transform/mnist.record'\n",
+ "\n",
+ "mnist_transformer = MnistToMR(mnist_path,mnist_mindrecord_path)\n",
+ "# executes transformation from Mnist to MindRecord\n",
+ "mnist_transformer.transform()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "因为MNIST数据集包含训练数据集以及测试数据集，所以生成的MindSpore数据格式文件也分别是训练数据集和测试数据集的，其中.db结尾的文件保存的是描述MindSpore数据格式文件的元数据信息，切记一定不要删除它，生成的文件如下所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " transform/mnist.record_test.mindrecord \n",
+ " transform/mnist.record_test.mindrecord.db \n",
+ " transform/mnist.record_train.mindrecord \n",
+ " transform/mnist.record_train.mindrecord.db "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "以下内容首先加载MindSpore数据格式的数据集，本例只进行了训练数据集的加载，如果需要测试数据集的加载也是同样的操作，然后对数据创建了字典迭代器，并通过迭代器读取了一条数据记录。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'data': array([255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1,\n",
+ " 0, 0, 1, 0, 1, 0, 0, 255, 219, 0, 67, 0, 2,\n",
+ " 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 2, 2,\n",
+ " 2, 4, 3, 2, 2, 2, 2, 5, 4, 4, 3, 4, 6,\n",
+ " 5, 6, 6, 6, 5, 6, 6, 6, 7, 9, 8, 6, 7,\n",
+ " 9, 7, 6, 6, 8, 11, 8, 9, 10, 10, 10, 10, 10,\n",
+ " 6, 8, 11, 12, 11, 10, 12, 9, 10, 10, 10, 255, 192,\n",
+ " 0, 11, 8, 0, 28, 0, 28, 1, 1, 17, 0, 255, 196,\n",
+ " 0, 31, 0, 0, 1, 5, 1, 1, 1, 1, 1, 1, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6,\n",
+ " 7, 8, 9, 10, 11, 255, 196, 0, 181, 16, 0, 2, 1,\n",
+ " 3, 3, 2, 4, 3, 5, 5, 4, 4, 0, 0, 1, 125,\n",
+ " 1, 2, 3, 0, 4, 17, 5, 18, 33, 49, 65, 6, 19,\n",
+ " 81, 97, 7, 34, 113, 20, 50, 129, 145, 161, 8, 35, 66,\n",
+ " 177, 193, 21, 82, 209, 240, 36, 51, 98, 114, 130, 9, 10,\n",
+ " 22, 23, 24, 25, 26, 37, 38, 39, 40, 41, 42, 52, 53,\n",
+ " 54, 55, 56, 57, 58, 67, 68, 69, 70, 71, 72, 73, 74,\n",
+ " 83, 84, 85, 86, 87, 88, 89, 90, 99, 100, 101, 102, 103,\n",
+ " 104, 105, 106, 115, 116, 117, 118, 119, 120, 121, 122, 131, 132,\n",
+ " 133, 134, 135, 136, 137, 138, 146, 147, 148, 149, 150, 151, 152,\n",
+ " 153, 154, 162, 163, 164, 165, 166, 167, 168, 169, 170, 178, 179,\n",
+ " 180, 181, 182, 183, 184, 185, 186, 194, 195, 196, 197, 198, 199,\n",
+ " 200, 201, 202, 210, 211, 212, 213, 214, 215, 216, 217, 218, 225,\n",
+ " 226, 227, 228, 229, 230, 231, 232, 233, 234, 241, 242, 243, 244,\n",
+ " 245, 246, 247, 248, 249, 250, 255, 218, 0, 8, 1, 1, 0,\n",
+ " 0, 63, 0, 254, 127, 235, 238, 239, 128, 159, 240, 75, 111,\n",
+ " 248, 39, 79, 198, 159, 132, 30, 20, 241, 54, 183, 255, 0,\n",
+ " 5, 203, 248, 99, 224, 255, 0, 28, 107, 186, 76, 119, 122,\n",
+ " 207, 130, 252, 75, 224, 77, 66, 218, 219, 69, 153, 208, 55,\n",
+ " 217, 100, 212, 100, 149, 35, 119, 66, 66, 185, 216, 170, 8,\n",
+ " 109, 165, 192, 4, 248, 159, 252, 20, 87, 254, 9, 179, 241,\n",
+ " 239, 254, 9, 161, 241, 87, 70, 248, 109, 241, 175, 89, 240,\n",
+ " 198, 187, 103, 226, 159, 15, 69, 175, 120, 63, 197, 190, 12,\n",
+ " 213, 254, 221, 165, 235, 122, 108, 140, 202, 151, 16, 72, 85,\n",
+ " 27, 4, 175, 70, 81, 193, 86, 27, 149, 149, 143, 207, 180,\n",
+ " 87, 234, 55, 252, 18, 239, 193, 31, 240, 108, 247, 195, 77,\n",
+ " 47, 225, 247, 197, 95, 219, 195, 246, 144, 241, 191, 139, 188,\n",
+ " 119, 121, 28, 51, 107, 254, 8, 189, 240, 149, 252, 126, 28,\n",
+ " 208, 175, 119, 43, 129, 112, 208, 91, 239, 187, 141, 74, 237,\n",
+ " 56, 145, 226, 112, 199, 124, 101, 114, 107, 145, 255, 0, 131,\n",
+ " 151, 60, 25, 251, 95, 47, 237, 185, 165, 124, 106, 253, 160,\n",
+ " 188, 69, 225, 77, 123, 192, 30, 48, 240, 202, 127, 194, 137,\n",
+ " 215, 126, 31, 190, 116, 41, 60, 51, 11, 110, 130, 222, 217,\n",
+ " 119, 29, 143, 24, 157, 26, 65, 146, 25, 166, 14, 164, 163,\n",
+ " 46, 63, 57, 235, 91, 192, 39, 193, 195, 199, 122, 41, 248,\n",
+ " 136, 183, 71, 195, 227, 86, 182, 254, 221, 22, 67, 247, 198,\n",
+ " 207, 205, 95, 59, 203, 228, 124, 254, 94, 236, 123, 226, 191,\n",
+ " 94, 63, 111, 47, 248, 33, 148, 159, 240, 80, 31, 137, 94,\n",
+ " 31, 248, 207, 255, 0, 4, 45, 248, 47, 240, 179, 87, 248,\n",
+ " 73, 54, 131, 107, 109, 38, 165, 225, 47, 138, 169, 231, 73,\n",
+ " 46, 49, 230, 234, 54, 58, 139, 71, 37, 133, 210, 0, 86,\n",
+ " 72, 211, 121, 124, 6, 101, 18, 23, 21, 194, 127, 193, 124,\n",
+ " 62, 32, 124, 9, 248, 55, 251, 15, 254, 202, 191, 240, 75,\n",
+ " 143, 6, 252, 112, 209, 62, 35, 120, 247, 224, 174, 137, 168,\n",
+ " 77, 241, 3, 92, 240, 206, 167, 29, 245, 142, 155, 115, 114,\n",
+ " 80, 27, 5, 157, 73, 203, 164, 139, 42, 249, 103, 12, 145,\n",
+ " 195, 22, 229, 5, 128, 31, 149, 148, 81, 69, 21, 255, 217],\n",
+ " dtype=uint8), 'label': array(3, dtype=int64)}\n"
+ ]
+ }
+ ],
+ "source": [
+ "file_name = './transform/mnist.record_train.mindrecord'\n",
+ "# create MindDataset for reading data\n",
+ "mnist_data_set = ds.MindDataset(dataset_file=file_name)\n",
+ "# create a dictionary iterator and read a data record through the iterator\n",
+ "print(next(mnist_data_set.create_dict_iterator()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 将CSV数据集转换为MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### CSV数据集下载"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- 本例中需要的数据位置在https://gitee.com/mindspore/docs/tree/master/tutorials/notebook/convert_dataset_to_mindspore_data_format/csv_data/data.csv\n",
+ "中，使用过程中可以在此路径下找到文件并下载，并且保存在`jupyter工作目录/dataset/`下，如图所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " dataset/data.csv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### CSV数据集转换"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`CsvToMR`这个类用于将CSV数据集转换为MindSpore数据格式，参数用法如下：\n",
+ "- `source` - CSV的文件的路径，本例中使用变量`csv_path`传入该参数。\n",
+ "- `destination` - 要转换成MindSpore数据格式文件的目录，本例使用变量`csv_mindrecord_path`传入该参数。\n",
+ "- `columns_list` - 要读取的列的列表，默认为None，本例使用默认参数。\n",
+ "- `partition_number` - 分区的大小，默认为1，本例使用默认参数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from mindspore.mindrecord import CsvToMR\n",
+ "from mindspore.mindrecord import FileReader\n",
+ "csv_path = './dataset/data.csv'\n",
+ "csv_mindrecord_path = './transform/csv.mindrecord'\n",
+ "\n",
+ "csv_transformer = CsvToMR(csv_path,csv_mindrecord_path) \n",
+ "# executes transformation from Csv to MindRecord\n",
+ "csv_transformer.transform()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "生成的文件如下所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " transform/csv.mindrecord \n",
+ " transform/csv.mindrecord.db "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "以下内容首先加载MindSpore数据格式的数据集，然后对数据创建了字典迭代器，并通过迭代器读取了一条数据记录。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'b': array(b'q', dtype='|S1'), 'c': array(b'e', dtype='|S1'), 'a': array(b'w', dtype='|S1')}\n"
+ ]
+ }
+ ],
+ "source": [
+ "# create MindDataset for reading data\n",
+ "csv_data_set = ds.MindDataset(dataset_file=csv_mindrecord_path)\n",
+ "# create a dictionary iterator and read a data record through the iterator\n",
+ "print(next(csv_data_set.create_dict_iterator()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 将CIFAR-10数据集转换为MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### CIFAR-10数据集下载"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- CIFAR-10数据集：\n",
+ " > https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
+ "- 将下载好的CIFAR-10数据集放在`jupyter工作目录/dataset/Cifar10Data/`下，如图所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " dataset/Cifar10Data/ \n",
+ " ├── batches.meta \n",
+ " ├── data_batch_1 \n",
+ " ├── data_batch_2 \n",
+ " ├── data_batch_3 \n",
+ " ├── data_batch_4 \n",
+ " ├── data_batch_5 \n",
+ " ├── readme.html \n",
+ " └── test_batch "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### CIFAR-10数据集转换"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`Cifar10ToMR`这个类用于将CIFAR-10数据集转换为MindSpore数据格式，参数用法如下：\n",
+ "- `source` - 存放CIFAR-10数据集的目录，本例中使用变量`cifar10_path`传入该参数。\n",
+ "- `destination` - 要转换成MindSpore数据格式文件的目录，本例中使用变量`cifar10_mindrecord_path`传入该参数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from mindspore.mindrecord import Cifar10ToMR\n",
+ "\n",
+ "cifar10_path = './dataset/Cifar10Data/'\n",
+ "cifar10_mindrecord_path = './transform/cifar10.record'\n",
+ "\n",
+ "cifar10_transformer = Cifar10ToMR(cifar10_path,cifar10_mindrecord_path)\n",
+ "# executes transformation from Cifar10 to MindRecord\n",
+ "cifar10_transformer.transform(['label'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "因为CIFAR-10数据集包含训练数据集以及测试数据集，所以生成的MindSpore数据格式文件也分别是训练数据集和测试数据集的，生成的文件如下所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " transform/cifar10.record \n",
+ " transform/cifar10.record.db \n",
+ " transform/cifar10.record_test \n",
+ " transform/cifar10.record_test.db "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "以下内容首先加载MindSpore数据格式的数据集，本例只进行了训练数据集的加载，如果需要测试数据集的加载也是同样的操作，然后对数据创建了字典迭代器，并通过迭代器读取了一条数据记录。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'data': array([255, 216, 255, ..., 35, 255, 217], dtype=uint8), 'id': array(30707, dtype=int64), 'label': array(4, dtype=int64)}\n"
+ ]
+ }
+ ],
+ "source": [
+ "# create MindDataset for reading data\n",
+ "cifar10_data_set = ds.MindDataset(dataset_file=cifar10_mindrecord_path)\n",
+ "# create a dictionary iterator and read a data record through the iterator\n",
+ "print(next(cifar10_data_set.create_dict_iterator()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 将CIFAR-100数据集转换为MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### CIFAR-100数据集下载"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- CIFAR-100数据集：\n",
+ " > https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- 将下载好的CIFAR-100数据集放在`jupyter工作目录/dataset/Cifar100Data/`下，如图所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " dataset/Cifar100Data/ \n",
+ " ├── file.txt~ \n",
+ " ├── meta \n",
+ " ├── test \n",
+ " └── train "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### CIFAR-100数据集转换"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`Cifar100ToMR`这个类用于将CIFAR-100数据集转换为MindSpore数据格式，参数用法如下：\n",
+ "- `source` - 存放CIFAR-100数据集的目录，本例中使用变量`cifar100_path`传入该参数。\n",
+ "- `destination` - 要转换成MindSpore数据格式文件的目录，本例中使用变量`cifar100_mindrecord_path`传入该参数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from mindspore.mindrecord import Cifar100ToMR\n",
+ "\n",
+ "cifar100_path = './dataset/Cifar100Data/'\n",
+ "cifar100_mindrecord_path = './transform/cifar100.record'\n",
+ "\n",
+ "cifar100_transformer = Cifar100ToMR(cifar100_path,cifar100_mindrecord_path)\n",
+ "#executes transformation from Cifar100 to MindRecord\n",
+ "cifar100_transformer.transform(['fine_label','coarse_label'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "因为CIFAR-100数据集包含训练数据集以及测试数据集，所以生成的MindSpore数据格式文件也分别是训练数据集和测试数据集的，生成的文件如下所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " transform/cifar100.record \n",
+ " transform/cifar100.record.db \n",
+ " transform/cifar100.record_test \n",
+ " transform/cifar100.record_test.db "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "以下内容首先加载MindSpore数据格式的数据集，本例只进行了训练数据集的加载，如果需要测试数据集的加载也是同样的操作，然后对数据创建了字典迭代器，并通过迭代器读取了一条数据记录。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'data': array([255, 216, 255, ..., 127, 255, 217], dtype=uint8), 'fine_label': array(88, dtype=int64), 'coarse_label': array(8, dtype=int64), 'id': array(10349, dtype=int64)}\n"
+ ]
+ }
+ ],
+ "source": [
+ "# create MindDataset for reading data\n",
+ "cifar100_data_set = ds.MindDataset(dataset_file=cifar100_mindrecord_path)\n",
+ "# create a dictionary iterator and read a data record through the iterator\n",
+ "print(next(cifar100_data_set.create_dict_iterator()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 将ImageNet数据集转换为MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### ImageNet数据集下载"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "- ImageNet数据集：\n",
+ " > http://image-net.org/download\n",
+ "- 将下载好的数据集放在`jupyter工作目录/dataset/ImageNetData/`下，如图所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " dataset/ImageNetData/ \n",
+ " ├── bounding_boxes \n",
+ " ├── imagenet_map.txt \n",
+ " ├── train \n",
+ " └── validation "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### ImageNet数据集转换"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`ImageNetToMR`这个类用于将ImageNet数据集转换为MindSpore数据格式，参数用法如下：\n",
+ "- `map_file` - map file 应该显示标签，内容如下所示：\n",
+ "\n",
+ " n02119789 0 \n",
+ " n02100735 1 \n",
+ " n02110185 2 \n",
+ " n02096294 3 \n",
+ "本例中使用变量`imagenet_map_path`传入该参数。\n",
+ "\n",
+ "\n",
+ "- `image_dir` - image目录应该是包含n02119789、n02100735、n02110185、n02096294的目录，本例中使用变量`imagenet_image_dir`传入该参数。\n",
+ "- `destination` - 要转换成MindSpore数据格式文件的路径，本例中使用变量`imagenet_mindrecord_path`传入该参数。\n",
+ "- `partititon_number` - 分区的大小，本例中设置该值为4，表示会生成4个MindSpore格式文件，使用变量`partition_number`传入该参数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from mindspore.mindrecord import ImageNetToMR\n",
+ "\n",
+ "imagenet_map_path = './dataset/ImageNetData/imagenet_map.txt'\n",
+ "imagenet_image_dir = './dataset/ImageNetData/train'\n",
+ "imagenet_mindrecord_path = './transform/imagenet.record'\n",
+ "partition_number = 4\n",
+ "\n",
+ "imagenet_transformer = ImageNetToMR(imagenet_map_path,imagenet_image_dir,imagenet_mindrecord_path,partition_number)\n",
+ "#executes transformation from ImageNet to MindRecord\n",
+ "imagenet_transformer.transform()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "因为本例中指定了分区大小为4，所以会生成4个MindSpore数据格式文件，因为数据集比较大，转换时间有点长，请耐心等待，生成的文件如下所示："
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " transform/imagenet.record0 \n",
+ " transform/imagenet.record0.db \n",
+ " transform/imagenet.record1 \n",
+ " transform/imagenet.record1.db \n",
+ " transform/imagenet.record2 \n",
+ " transform/imagenet.record2.db \n",
+ " transform/imagenet.record3 \n",
+ " transform/imagenet.record3.db "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "以下内容首先加载MindSpore数据格式的数据集，虽然是自己指定了分区，但是文件之间还是相互有联系，所以不管对任何一个MindSpore数据格式文件加载，都会同时把其他三个文件一起加载，本例将imagenet.record0文件进行了加载，然后对数据创建了字典迭代器，并通过迭代器读取了一条数据记录。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'label': array(564, dtype=int64), 'file_name': array(b'./dataset/ImageNetData/train/n03388183/n03388183_8017.JPEG',\n",
+ " dtype='|S58'), 'data': array([255, 216, 255, ..., 39, 255, 217], dtype=uint8)}\n"
+ ]
+ }
+ ],
+ "source": [
+ "file_name = './transform/imagenet.record0'\n",
+ "# create MindDataset for reading data\n",
+ "imagenet_data_set = ds.MindDataset(dataset_file=file_name)\n",
+ "# create a dictionary iterator and read a data record through the iterator\n",
+ "print(next(imagenet_data_set.create_dict_iterator()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 用户自定义生成MindSpore数据格式"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "1. 导入`FileWriter`类，用于将用户定义的原始数据写入，参数用法如下：\n",
+ "\n",
+ "\n",
+ "- `file_name` - MindSpore数据格式文件的文件名，本例使用变量`data_record_path`传入该参数。\n",
+ "- `shard_num` - MindSpore数据格式文件的数量，默认为1，取值范围在[1,1000]，本例使用默认参数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mindspore.mindrecord import FileWriter\n",
+ "data_record_path = './transform/data.record'\n",
+ "writer = FileWriter(data_record_path,1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "2. 定义数据集Schema，Schema用于定义数据集包含哪些字段以及字段的类型，然后添加Schema，相关规范如下：\n",
+ "\n",
+ "\n",
+ "- 字段名：字母、数字、下划线。\n",
+ "- 字段属性type：int32、int64、float32、float64、string、bytes。\n",
+ "- 字段属性shape：如果是一维数组，用[-1]表示，如果是二维数组，用[m,n]表示，如果是三维数组，用[x,y,z]表示。\n",
+ "\n",
+ "本例中定义了`file_name`字段，用于标注准备写入数据的文件名字，定义了`label`字段，用于给数据打标签，定义了`data`字段，用于保存数据。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "data_schema = {\"file_name\":{\"type\":\"string\"},\"label\":{\"type\":\"int32\"},\"data\":{\"type\":\"bytes\"}}\n",
+ "writer.add_schema(data_schema,\"test_schema\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "3. 准备需要写入的数据，按照用户定义的Schema形式，准备需要写入的样本列表，本例中需要的数据位置在https://gitee.com/mindspore/docs/tree/master/tutorials/notebook/convert_dataset_to_mindspore_data_format/images/transform.jpg\n",
+ "中，使用过程中可以在此路径下找到图片并下载，并且保存在`jupyter工作目录/dataset/`下。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def image_to_bytes(file_name):\n",
+ " f = open(file_name,'rb')\n",
+ " image_bytes = f.read()\n",
+ " f.close()\n",
+ " return image_bytes\n",
+ "\n",
+ "data = [{\"file_name\":\"transform.jpg\",\"label\":1,\"data\":image_to_bytes('./dataset/transform.jpg')}]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "4. 添加索引字段，添加索引字段可以加速数据读取，改步骤为可选操作。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "indexes = [\"file_name\",\"label\"]\n",
+ "writer.add_index(indexes)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "5. 写入数据，最后生成MindSpore数据格式文件，`write_raw_data`接口可以被重复调用，方便用户将多个样本添加至MindSpore数据格式文件中。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "MSRStatus.SUCCESS"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "writer.write_raw_data(data)\n",
+ "writer.commit()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "6. 以下内容首先加载MindSpore数据格式的数据集，然后对数据创建了字典迭代器，并通过迭代器读取了一条数据记录。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'label': array(1, dtype=int32), 'file_name': array(b'transform.jpg', dtype='|S13'), 'data': array([255, 216, 255, ..., 159, 255, 217], dtype=uint8)}\n"
+ ]
+ }
+ ],
+ "source": [
+ "file_name = './transform/data.record'\n",
+ "# create MindDataset for reading data\n",
+ "define_data_set = ds.MindDataset(dataset_file=file_name)\n",
+ "# create a dictionary iterator and read a data record through the iterator\n",
+ "print(next(define_data_set.create_dict_iterator()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 总结\n",
+ "\n",
+ "以上便是本次体验的内容，我们通过此次体验全面了解了如何通过各个相应的子模块将其他格式的数据集转换为MindSpore数据格式。"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/tutorials/notebook/convert_dataset_to_mindspore_data_format/csv_data/data.csv b/tutorials/notebook/convert_dataset_to_mindspore_data_format/csv_data/data.csv
new file mode 100644
index 0000000000000000000000000000000000000000..de65cfe60cdbf3e2f9c487ceb8e9e767ae178329
--- /dev/null
+++ b/tutorials/notebook/convert_dataset_to_mindspore_data_format/csv_data/data.csv
@@ -0,0 +1,4 @@
+a,b,c
+w,q,e
+w,q,e
+w,q,e
diff --git a/tutorials/notebook/convert_dataset_to_mindspore_data_format/images/transform.jpg b/tutorials/notebook/convert_dataset_to_mindspore_data_format/images/transform.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..8507a75e4b6dba1444729af86c1de0b12d859b0c
Binary files /dev/null and b/tutorials/notebook/convert_dataset_to_mindspore_data_format/images/transform.jpg differ
diff --git a/tutorials/source_zh_cn/use/data_preparation/converting_datasets.md b/tutorials/source_zh_cn/use/data_preparation/converting_datasets.md
index ba744652da3054a25d64a8d55e95dfbf84da3071..550a58f434faa07e25ad609e167c406d869d4290 100644
--- a/tutorials/source_zh_cn/use/data_preparation/converting_datasets.md
+++ b/tutorials/source_zh_cn/use/data_preparation/converting_datasets.md
@@ -15,6 +15,8 @@
+
+
## 概述