[openEuler-20.03 SP2] ARM64 一个 DIE 上部署多台虚拟机测试 unixbench spawn 性能下降

原版的补丁有问题

> host依然能统计到大量dvm的原因找到了，是guest的patch没有打全，只改了一部分，这个函数中的is没有去掉。
> ![输入图片说明](https://images.gitee.com/uploads/images/2021/1018/111501_9f4c608a_422679.png "屏幕截图.png")

做如下修正

1.    首先 flush_tlb_mm_nosync 还是想使用广播的方式的，因此需要用 aside1is(IS) 的方式。
2.    ipi_flush_tlb_page -=> local_flush_tlb_page 是想发 IPI 的，因此不需要用 IS 的方式。

![输入图片说明](https://images.gitee.com/uploads/images/2021/1018/110547_cb5747a2_422679.png "屏幕截图.png")

```cpp
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 9c1912570823..5b2e81813dbc 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -269,8 +269,9 @@ static inline void flush_tlb_mm_nosync(struct mm_struct *mm)
        unsigned long asid = __TLBI_VADDR(0, ASID(mm));
 
        dsb(ishst);
-       __tlbi(aside1, asid);
-       __tlbi_user(aside1, asid);
+       __tlbi(aside1is, asid);
+       __tlbi_user(aside1is, asid);
+       dsb(ish);
 }
 
 static inline void flush_tlb_page_nosync(struct vm_area_struct *vma,
@@ -289,8 +290,8 @@ static inline void local_flush_tlb_page(struct vm_area_struct *vma,
        unsigned long addr = __TLBI_VADDR(uaddr, ASID(vma->vm_mm));
 
        dsb(nshst);
-       __tlbi(vale1is, addr);
-       __tlbi_user(vale1is, addr);
+       __tlbi(vale1, addr);
+       __tlbi_user(vale1, addr);
        dsb(nsh);
 }
```

补丁见附件中 V2

虚拟机固定绑核的时候，测试没有问题。
但是虚拟机设置成范围绑核的时候，测试出现了批量虚拟机无法启动的问题。

固定绑核如下所示：

![输入图片说明](https://images.gitee.com/uploads/images/2021/1021/095831_0a6e6e73_422679.png "屏幕截图.png")

范围绑核如下所示：
![输入图片说明](https://images.gitee.com/uploads/images/2021/1021/095851_e3492875_422679.png "屏幕截图.png")

![输入图片说明](https://images.gitee.com/uploads/images/2021/1021/095855_26a167ce_422679.png "屏幕截图.png")

进一步分析测试发现：
5220 上 HOST 没去掉 FB，虚拟机打了 patch，满虚拟机能正常起了，目前跑着也都正常。无异常启动的问题，
7260 HOST 不去掉FB用原始的内核，范围绑核也没出现虚拟机挂掉的情况。

HOST 不去掉 FB，就应该不会出现虚拟机挂掉的问题，但是这样性能数据上不去。

> 这个FB的特性为啥影响虚机稳定性?
>
> 实质上是vcpu迁移带来的tlbi不一致问题。 
> FB的作用是，将虚拟机发出来的 TLBI 强制转换为广播型 TLBI IS，若关掉，则可能发生：vcpu0本来在cpu0中，但中途被迁移到了cpu1，此时发出TLBI，host 只会在 cpu1 进行刷 tlb 操作，vcpu0 再返回 cpu0 时，tlb 依然是旧的，就会导致问题出现。

![输入图片说明](https://images.gitee.com/uploads/images/2021/1021/100149_c6563728_422679.png "屏幕截图.png")

![输入图片说明](https://images.gitee.com/uploads/images/2021/1021/100210_97052b32_422679.png "屏幕截图.png")

![输入图片说明](https://images.gitee.com/uploads/images/2021/1021/100222_bb754e8a_422679.png "屏幕截图.png")

之前 HISI 的汤年耀 KVM 里面的修改
https://www.spinics.net/lists/kvm-arm/msg42855.html

```cpp
diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 64ce29378467..f85ea9c649cb 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -75,7 +75,7 @@
  * PTW:		Take a stage2 fault if a stage1 walk steps in device memory
  */
 #define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
-			 HCR_BSU_IS | HCR_FB | HCR_TAC | \
+			 HCR_BSU_IS | HCR_TAC | \
 			 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
 			 HCR_FMO | HCR_IMO | HCR_PTW )
 #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index acf9a993dfb6..845be911f885 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -334,8 +334,10 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	/*
 	 * We might get preempted before the vCPU actually runs, but
 	 * over-invalidation doesn't affect correctness.
+	 * Dirty tlb might still exist when vcpu ran on other pcpu
+	 * and modified page mapping.
 	 */
-	if (*last_ran != vcpu->vcpu_id) {
+	if (*last_ran != vcpu->vcpu_id || vcpu->cpu != cpu) {
 		kvm_call_hyp(__kvm_tlb_flush_local_vmid, mmu);
 		*last_ran = vcpu->vcpu_id;
 	}
-- 
2.7.4
```

在 `vcpu` 切了 `CPU` 之后，通过 `__kvm_tlb_flush_local_vmid` 刷掉所有的 `TLB`.
但是 `HCR_FB` 本身的作用除了刷 `TLB` 以外, 还有 `flush icache`. 这个补丁当前遗留的问题就是没有刷 icache.

host 上增加刷 icache 的补丁
0001-KVM-arm64-don-t-force-broadcast-tlbi-and-ic.patch

```cpp
From fdc9ff90003d39c52e2ae53cc9f90ec86a1cc30e Mon Sep 17 00:00:00 2001
From: t00358190 <tangnianyao@huawei.com>
Date: Mon, 25 Oct 2021 11:47:27 +0800
Subject: [PATCH] KVM:arm64:don't force broadcast tlbi and ic

Signed-off-by: t00358190 <tangnianyao@huawei.com>
---
 arch/arm64/include/asm/kvm_arm.h | 2 +-
 arch/arm64/kvm/arm.c             | 6 +++++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 0cafa405c7f6..7a72c7de0884 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -80,7 +80,7 @@
  * PTW:		Take a stage2 fault if a stage1 walk steps in device memory
  */
 #define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
-			 HCR_BSU_IS | HCR_FB | HCR_TAC | \
+			 HCR_BSU_IS | HCR_TAC | \
 			 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
 			 HCR_FMO | HCR_IMO | HCR_PTW )
 #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index eb645005d80d..928c8f5532e2 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -435,9 +435,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	/*
 	 * We might get preempted before the vCPU actually runs, but
 	 * over-invalidation doesn't affect correctness.
+	 * For HCR_FB=0, we need to invalid icache and tlb in cases
+	 * that vCPU modified page mapping or instruction memory on
+	 * other pCPU and then comes back.
 	 */
-	if (*last_ran != vcpu->vcpu_id) {
+	if (*last_ran != vcpu->vcpu_id || vcpu->cpu != cpu) {
 		kvm_call_hyp(__kvm_tlb_flush_local_vmid, mmu);
+		asm volatile("ic iallu");
 		*last_ran = vcpu->vcpu_id;
 	}
 
-- 
2.30.0
```

GVP openEuler/kernel

内容风险标识

评论 (17)

GVPopenEuler/kernel

内容风险标识