fix diverse mode router.

[hiworldwzj]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the router's request queue management by introducing a more robust mechanism for handling aborted requests. The changes centralize the logic for identifying, cleaning up, and removing requests that have been cancelled, leading to a more efficient and reliable request processing pipeline. This ensures that the system can gracefully manage and free resources associated with unneeded requests.

Highlights

• Aborted Request Handling: Introduced a dedicated method, _filter_aborted_reqs, to centralize and improve the management of aborted requests within the waiting queue. This method ensures that all related requests are marked as aborted before removal and proper resource cleanup.
• Queue Management: Refactored the generate_new_batch method to call _filter_aborted_reqs early in its execution, streamlining the process of removing aborted requests and preventing them from being considered for new batches.
• Dependency Update: Added the time module import, which is utilized by the new _filter_aborted_reqs method for a brief delay to ensure consistent state across related requests.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• lightllm/server/router/req_queue/chunked_prefill/beam_impl.py
  • Imported the time module.
  • Added a new private method _filter_aborted_reqs to process and remove aborted requests from the waiting queue, including a short delay to synchronize abortion states.
  • Integrated the call to _filter_aborted_reqs at the beginning of the generate_new_batch method.
  • Removed the redundant in-line logic for handling aborted requests from the generate_new_batch method, delegating this responsibility to the new _filter_aborted_reqs method.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

本次 PR 重构了对中止请求的处理，将逻辑移至新的 _filter_aborted_reqs 方法中。这改善了代码结构，并且似乎修复了 waiting_req_list 更新方式中一个潜在的 bug。然而，新方法引入了一个阻塞的 time.sleep(0.1)，这会带来严重的性能和可靠性问题。我的审查意见中包含一个提高新方法效率的建议，并指出了使用 time.sleep 进行同步的问题。

[gemini-code-assist]

这个方法有两点可以改进：

1. 性能: 当前实现为了过滤中止的请求，最多会遍历 self.waiting_req_list 三次。这可以优化，通过使用 any() 进行初步检查，然后在单次遍历中对列表进行分区。
2. 可靠性与性能: 使用阻塞的 time.sleep(0.1) 进行同步是脆弱的，并且存在性能问题。它会暂停线程，增加延迟，并且如果其他请求并发中止，等待时间可能不足以防止竞争条件。一个更健壮的同步机制，如条件变量或事件，将是更好的长期解决方案。

下面的建议代码重构了过滤逻辑以提高性能。time.sleep() 的问题更为根本，应通过更稳健的同步策略来解决。

Suggested change

	exist_aborted_req = len([req for req in self.waiting_req_list if req.is_aborted]) > 0
	if exist_aborted_req:
	time.sleep(0.1)
	aborted_reqs = [req for req in self.waiting_req_list if req.is_aborted]
	self.waiting_req_list = [req for req in self.waiting_req_list if not req.is_aborted]
	for req in aborted_reqs:
	req: Req = req
	logger.debug(f"router abort req id {req.request_id} shm_index: {req.index_in_shm_mem}")
	self.free_aborted_req_cpu_cache_pages(req)
	self.router.shm_req_manager.put_back_req_obj(req)
	if any(req.is_aborted for req in self.waiting_req_list):
	time.sleep(0.1)
	aborted_reqs = []
	new_waiting_req_list = []
	for req in self.waiting_req_list:
	if req.is_aborted:
	aborted_reqs.append(req)
	else:
	new_waiting_req_list.append(req)
	self.waiting_req_list = new_waiting_req_list
	for req in aborted_reqs:
	logger.debug(f"router abort req id {req.request_id} shm_index: {req.index_in_shm_mem}")
	self.free_aborted_req_cpu_cache_pages(req)
	self.router.shm_req_manager.put_back_req_obj(req)

[hiworldwzj]

等待队列一般规模不会超过1000，同时生成式写法python执行效率更高，在小数组情况下更适合。