[BugFix] Fix server shutdown stuck when some executors exited · Pull Request !715 · MindSpore/vllm-mindspore

1. Fix server shutdown stuck when some executors exited
---
<!--  Thanks for sending a pull request!  Here are some tips for you:

1) If this is your first time, please read our contributor guidelines: https://gitee.com/mindspore/mindspore/blob/master/CONTRIBUTING.md

2) If you want to contribute your code but don't know who will review and merge, please add label `mindspore-assistant` to the pull request, we will find and do it as soon as possible.
-->

**What type of PR is this?**

/kind bug

**What does this PR do / why do we need it**:
修复手动杀掉单个worker之后，api server shutdown卡住不退的问题

- 本地验证流程
1. 启动vllm
2. 待vllm成功启动后，通过`npu-smi info`命令，查看当前的worker进程，并通过kill命令杀掉一个进程
3. 客户端发送curl命令，执行在线推理

- 运行结果
server和client都可以退出，不会卡死
1. server 端日志
    ```text
    (VllmWorker rank=0 pid=18976) WARNING 07-30 11:09:53 vllm-mindspore[gpu_model_runner.py:449] ----------------------------------------------------
    (VllmWorker rank=0 pid=18976) WARNING 07-30 11:09:53 vllm-mindspore[gpu_model_runner.py:449] - C++ Call Stack: (For framework developers)
    (VllmWorker rank=0 pid=18976) WARNING 07-30 11:09:53 vllm-mindspore[gpu_model_runner.py:449] ----------------------------------------------------
    (VllmWorker rank=0 pid=18976) WARNING 07-30 11:09:53 vllm-mindspore[gpu_model_runner.py:449] mindspore/ccsrc/runtime/graph_scheduler/actor/kernel_actor.cc:1383 ExecuteLaunchKernelTask
    (VllmWorker rank=0 pid=18976) WARNING 07-30 11:09:53 vllm-mindspore[gpu_model_runner.py:449]
    (VllmWorker rank=0 pid=18976) WARNING 07-30 11:09:53 vllm-mindspore[gpu_model_runner.py:449]
    INFO 07-30 11:09:56 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
    ......
    INFO 07-30 11:12:26 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
    [INFO] PS(18976,fffe6a072120,python3):2025-07-30-11:12:34.644.204 [mindspore/ccsrc/ps/core/communicator/tcp_client.cc:318] Start] Event base dispatch success!
    [INFO] PS(18976,fffe84f9f120,python3):2025-07-30-11:12:34.644.398 [mindspore/ccsrc/ps/core/communicator/tcp_server.cc:220] Start] Event base dispatch success!
    INFO 07-30 11:12:34 vllm-mindspore[launcher.py:92] Shutting down FastAPI HTTP server.
    WARNING 07-30 11:12:34 [core.py:363] signum=15 17 huweidebug <frame at 0xffff39488680, file '/mnt/disk1/fanrui/vllm_space/vllm-v0.8.3/vllm/distributed/device_communicators/shm_broadcast.py', line 39, code sched_yield>
    [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
    ......
    [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
    INFO:     Shutting down
    INFO 07-30 11:12:39 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
    /root/miniconda3/envs/py310/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
      warnings.warn('resource_tracker: There appear to be %d '
    /root/miniconda3/envs/py310/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
      warnings.warn('resource_tracker: There appear to be %d '
    ERROR:    Exception in ASGI application
    ```
2. client 日志
    ```text
    $ curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "prompt": "你觉得mindspore怎么样", "temperature": 0, "top_p": 0.95}'
    Internal Server Error
    ```

**Which issue(s) this PR fixes**:

Fixes #ICPCA9

**Code review checklist [【代码检视checklist说明】](https://gitee.com/mindspore/community/blob/master/security/code_review_checklist_mechanism.md)**:

+ - [ ] 是否进行返回值校验 (禁止使用void屏蔽安全函数、自研函数返回值，C++标准库函数确认无问题可以屏蔽)
+ - [ ] 是否遵守 ***SOLID原则 / 迪米特法则***
+ - [ ] 是否具备UT测试用例看护 && 测试用例为有效用例 (若新特性无测试用例看护请说明原因)
+ - [ ] 是否为对外接口变更
+ - [ ] 是否涉及官网文档修改

类型	指派人员	状态
审查	TronZhang	已完成 (0/0人)
测试	TronZhang	已完成 (0/0人)

MindSpore/vllm-mindspore .gitee-modal { width: 500px !important; }

搜索帮助

MindSpore/vllm-mindspore