bugfix: vlm models(language-only part) inference error, without --ena…

[shihaobai]

…ble_multimodal

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical bug affecting Vision-Language Models (VLMs) when they are configured to operate solely in language mode, without multimodal capabilities enabled. The fix ensures that the inference process can proceed smoothly by gracefully handling the absence of a multimodal embedding cache client, preventing runtime errors and improving the robustness of the system under varying operational configurations.

Highlights

• Bug Fix for VLM Models: Resolved an inference error that occurred in Vision-Language Models (VLMs) when they were run in a language-only configuration, specifically without the --enable_multimodal flag.
• Robust Multimodal Client Handling: Implemented a conditional check for the presence of cpu_embed_cache_client to prevent AttributeError when multimodal features are not enabled.
• Safe Tensor Initialization: Ensured that cpu_embed_cache_tensor is safely initialized as an empty tensor when the cpu_embed_cache_client is not available, maintaining compatibility and preventing crashes.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• lightllm/models/gemma3/layer_infer/pre_layer_infer.py
  • Modified the context_forward method to check for g_infer_context.cpu_embed_cache_client being None before attempting to access its cpu_embed_cache_tensor attribute. If None, an empty tensor is created.
• lightllm/models/qwen3_vl/layer_infer/pre_layer_infer.py
  • Updated the context_forward method to include a null check for g_infer_context.cpu_embed_cache_client, initializing cpu_embed_cache_tensor as an empty tensor if the client is not present.
• lightllm/models/qwen_vl/layer_infer/pre_layer_infer.py
  • Refactored the context_forward method to conditionally assign cpu_embed_cache_tensor based on whether g_infer_context.cpu_embed_cache_client exists, preventing errors in language-only inference.

Activity

• No specific activity (comments, reviews, progress updates) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly fixes a bug that occurs when running VLM models in language-only mode by handling cases where cpu_embed_cache_client is None. The fix prevents a potential AttributeError. However, the same fix has been applied to three different files, resulting in code duplication. My review includes a suggestion to refactor this duplicated logic to improve code maintainability.

[gemini-code-assist]

While this logic correctly handles the case where cpu_embed_cache_client is None, it is duplicated in qwen3_vl and qwen_vl pre-layer inference files. To improve maintainability and avoid repeating code, consider extracting this logic into a shared helper method in the base class LlamaMultimodalPreLayerInfer.

[gemini-code-assist]

This logic to safely retrieve cpu_embed_cache_tensor is also present in the gemma3 and qwen_vl model files. To adhere to the DRY (Don't Repeat Yourself) principle, this logic should be centralized. A helper method in the LlamaMultimodalPreLayerInfer base class would be an ideal place for it.

[gemini-code-assist]

This logic for safely initializing cpu_embed_cache_tensor is duplicated in the subclasses Gemma3PreLayerInfer and Qwen3VLMultimodalPreLayerInfer. Since this is the base class, you could define a protected helper method here (e.g., _get_cpu_embed_cache_tensor) to encapsulate this logic. The subclasses can then call this method, which would eliminate the code duplication and make future changes easier.