[5.10] [Feature] virtio-mem: deferred hotplugged pages initialization · Pull Request !4622 · anolis/cloud-kernel

bugzilla: https://bugzilla.openanolis.cn/show_bug.cgi?id=18841

# Description of problem
    Memory hotplug is a serial process that adds memory to Linux in the
    granularity of memory blocks. We identified two memory initialization
    functions that consume significant time when onling memory blocks:
    - `__init_single_page`: initialize the struct page
    - `__free_pages_core`: add page to the buddy allocator
    
    We attempted to execute these two functions in parallel during the
    process of hotplugging a memory block. The experimental results showed
    that when the memory block size was 1GB, the hotplug speed was
    increased by approximately 200%. However, when the memory block size
    was 128MB, which is the more commonly used size, the hotplug speed
    was even worse than that of serial execution.
    
    Therefore, how to improve the hotplug speed when the memory block size
    is 128MB remains a challenge.
    
    Here is my idea:
    - Defer the execution of these two functions and their associated
      processs to the final phase of the entire hotplug process, so
      that the hotplug speed will no longer be limited by the memory
      block size.
    - Perform parallel execution in the final phase, as previous
      implementations have proven that this can accelerate the hotplug
      process.
    
    We introduce the new online function, `deferred_online_memory`, for
    deferring the actual online process of memory blocks.
    
    Additionally, we have added a command-line argument,
    parallel_hotplug_ratio, which sets the ratio of parallel workers to
    the number of CPUs on the node. When parallel_hotplug_ratio is 0,
    the memory online process will no longer be deferred.

# Version-Release number of selected component (if applicable):
5.10.134-18.al8.x86_64

# How reproducible
ecs.ebmg7.32xlarge

# Steps to Reproduce

# Actual results
    We applied parallel deferred memory online to virtio-mem and
    observed a significant speedup in the hot-plug process.
    
    We conducted an experiment to hotplug 400G of memory, and
    the results were as follows:
    - Before applying the patch:
      - Total Time = Origin Hotplug Time = 5537ms (72.24 GB/s)
    - After applying the patch (with `parallel_hotplug_ratio=80`):
      - Origin Hotplug Time = 178ms
      - Deferred Parallel Hotplug Time = 1200ms
      - Total Time = 1378ms (76% reduction, 290.28 GB/s)
    
    Lastly, there's an issue regarding the guest's plug
    request to the VMM. The VMM relies on the plug requests
    sent by the guest to determine the size of the hot-plugged
    memory. Therefore, we should defer the sending of the plug
    requests after the memory has been actually onlined.

# Expected results

# Additional info
测试数据： https://aliyuque.antfin.com/ouyi.yr/xg843l/grd84917c2kw8tmv#kx6qT
## 实验环境及配置
+ 实例规格：ecs.ebmg7.32xlarge，128 CPU，512 GB 内存
+ guest 配置：64 vCPU，256 MB boot mem
+ 命令行基础配置：

| base cmdline **parameters** |
| --- |
| memory_hotplug.skip_set_contiguous=1 |
| memory_hotplug.memmap_on_memory=1 |

## 稳定性测试
### ✅测试一：连续热插拔
测试连续热插、热拔，以及内存不对齐 memory block size （128MB）等情况下延迟并发扩容功能是否正常。

| extra cmdline **parameters** |
| --- |
| parallel_hotplug_ratio = 80 |

| request size（MB） | origin MemTotal（MB） | defer MemTotal（MB） |
| --- | --- | --- |
| 冷启动（初始状态） | 203 | 203 |
| 64 | 265 | 265 |
| 92 | 293 | 293 |
| 128 | 329 | 329 |
| 0 | 203 | 203 |
| 192 | 391 | 391 |
| 4096 | 4235 | 4235 |
| 2048 | 2219 | 2219 |
| 0 | 205 | 207 |
| 409600 | 403403 | 403403 |
| 3000 | 3155 | 3155 |
| 102400 | 101003 | 101003 |

### ✅测试二：多 NUMA node
测试在2个 numa node 情况下，每个 node 上有一个 virtio-mem 设备情况下延迟并发扩容功能是否正常。

| extra cmdline **parameters** |
| --- |
| parallel_hotplug_ratio = 80 |

| node 0 request size (MB) | node 1 request size (MB) | origin MemTotal（MB） | defer MemTotal（MB） |
| --- | --- | --- | --- |
| **0** | **0** | 203 | 203 |
| **64** | **0** | 265 | 265 |
| **64** | **64** | 327 | 327 |
| **128** | **256** | 581 | 581 |
| **0** | **256** | 455 | 455 |
| **3000** | **2000** | 5123 | 5123 |
| **1000** | **500** | 1679 | 1679 |
| **0** | **0** | 207 | 209 |
| **204800** | **1024** | 202811 | 202811 |
| **204800** | **204800** | 402403 | 402403 |

### ✅测试三：开启 auto-movable
测试在开启 auto-movable 情况下，热插内存是否正确按照比例插入到对应的 zone 上。

| extra cmdline **parameters** |
| --- |
| parallel_hotplug_ratio = 80 |
| memory_hotplug.online_policy=auto-movable |
| memory_hotplug.auto_movable_ratio=201 |

> 表格数据显示为 min/low/high/spanned/present/managed
>

| request size (MB) | origin normal zone | origin movable zone | defer normal zone | defer movable zone |
| --- | --- | --- | --- | --- |
| **0** | 0/0/0/0/0/0 | 0/0/0/0/0/0 | 0/0/0/0/0/0 | 0/0/0/0/0/0 |
| 1024 | 141/205/269/131072/65536/64512 | 424/617/810/229376/196608/193536 | 141/205/269/131072/65536/64512 | 424/617/810/229376/196608/193536 |
| 5120 | 381/768/1155/1114112/393216/387072 | 890/1793/2696/1310720/917504/903168 | 381/768/1155/1114112/393216/387072 | 890/1793/2696/1310720/917504/903168 |
| 8000 | 520/1165/1810/1900544/655360/645120 | 1104/2474/3844/2064384/1409024/1370624 | 520/1165/1810/1900544/655360/645120 | 1104/2474/3844/2064384/1409024/1370624 |
| 15000 | 722/1947/3172/3670016/1245184/1225728 | 1504/4057/6610/3866624/2621440/2553856 | 722/1947/3172/3670016/1245184/1225728 | 1504/4057/6610/3866624/2621440/2553856 |
| 10000 | 1063/2288/3513/3670016/1245184/1225728 | 1147/2469/3791/1966080/1343488/1293824 | 1063/2288/3513/3670016/1245184/1225728 | 1147/2469/3791/1966080/1343488/1293824 |
| 5000 | 2060/3285/4510/3670016/1245184/1225728 | 108/172/236/65536/65536/33792 | 2060/3285/4510/3670016/1245184/1225728 | 108/172/236/65536/65536/33792 |
| 20000 | 840/2485/4130/4947968/1671168/1645056 | 1734/5128/8522/5144576/3473408/3394560 | 840/2485/4130/4947968/1671168/1645056 | 1734/5128/8522/5144576/3473408/3394560 |
| 10000 | 1645/3290/4935/4947968/1671168/1645056 | 903/1806/2709/1310720/917504/874496 | 1645/3290/4935/4947968/1671168/1645056 | 903/1806/2709/1310720/917504/874496 |
| 0 | 220/275/330/32768/32768/2560 | 0/0/0/0/0/0 | 220/275/330/32768/32768/1536 | 0/0/0/0/0/0 |

## 性能测试
### ✅测试一：空载性能对比
空载情况下测试扩容 400G，额外命令行配置：

| extra cmdline **parameters** |
| --- |
| parallel_hotplug_ratio = 80 |

热插耗时降低**76%**，原流程热插速度**72G/s**，优化后**290G/s**

![](https://intranetproxy.alipay.com/skylark/lark/0/2025/png/147156307/1750160828992-96473196-f2e2-422b-b5c2-f8d9ceea87ac.png)

### ✅测试二：CPU负载测试
guest不同cpu负载下不同ratio的性能影响：

| **cpu load（%）** | **parallel_hotplug_ratio（%）** | **time（s）** | **speed（GB/s）** |
| --- | --- | --- | --- |
| 0 | 10 | 2.724 | 146.84 |
| 50 | 10 | 2.654 | 150.72 |
| 100 | 10 | 2.810 | 142.35 |
| 0 | 20 | 1.694 | 236.13 |
| 50 | 20 | 2.057 | 194.46 |
| 100 | 20 | 2.167 | 184.59 |
| 0 | 40 | 1.316 | 303.95 |
| 50 | 40 | 1.431 | 279.52 |
| 100 | 40 | 1.477 | 270.82 |
| 0 | 80 | 1.384 | 289.02 |
| 50 | 80 | 1.491 | 268.28 |
| 100 | 80 | 1.623 | 246.46 |
| 0 | 100 | 1.277 | 313.23 |
| 50 | 100 | 1.471 | 271.92 |
| 100 | 100 | 1.604 | 249.38 |

1. cpu负载越高，扩容性能越差
2. ratio越高扩容性能越好，但是存在边际效应

**生产环境中，推荐设置 ratio = 50%。100%容易影响正在运行的其他进程。**

类型	指派人员	状态
审查	xunlei zhongjiang baolin.wang	已完成 (0/0人)
测试	小龙已测试通过	已完成 (1/1人)

GVPanolis/cloud-kernel

搜索帮助

GVP anolis/cloud-kernel