fix(multimodal): collect all surrounding text as context instead of o… - #136
Conversation
…nly the nearest segment Previously, `get_image_context` only picked the single text item immediately adjacent to the image (via `break`). When users placed the image at the end of the `response` list (e.g. after a task description + source + target), `context_above` would contain only the last string (often a single word like "Off" or "Custom") while the task description was silently dropped. This caused `ImageCoherenceGrader` and `ImageHelpfulnessGrader` to receive near-empty context, resulting in unstable and severely low scores (1–2) for the same content that scored 5 when the image was placed first. The fix concatenates all text segments before/after the image, ensuring task instructions and surrounding content are always fully passed to the model regardless of image position. Made-with: Cursor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request refines the context extraction mechanism for multimodal graders. The change ensures that when processing content containing images, all relevant textual information preceding and following an image is captured and provided as context. This enhancement addresses an issue where previously only immediate text segments were considered, leading to incomplete context and negatively impacting the accuracy and stability of image-related grading. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
The pull request refactors the get_image_context function to collect all surrounding text segments for improved context for multimodal graders, leading to more stable and accurate evaluations. However, this change increases the surface area for prompt injection attacks as the extracted context is directly formatted into LLM prompts without sanitization, potentially allowing malicious user input to manipulate the grading process. Additionally, an edge case regarding max_context_size needs attention to ensure correct behavior.
…le zero value correctly `max_context_size=0` should truncate context to empty, but the previous `and max_context_size` truthiness check treats 0 as False, silently skipping truncation. Switching to `is not None` ensures the limit is always respected, including when explicitly set to 0. Also applies black formatting. Made-with: Cursor
…context_size=0 `context[-0:]` returns the full string rather than an empty string, so `max_context_size=0` must be handled as a special case explicitly. Made-with: Cursor
3 participants
…nly the nearest segment
Previously,
get_image_contextonly picked the single text item immediately adjacent to the image (viabreak). When users placed the image at the end of theresponselist (e.g. after a task description + source + target),context_abovewould contain only the last string (often a single word like "Off" or "Custom") while the task description was silently dropped. This causedImageCoherenceGraderandImageHelpfulnessGraderto receive near-empty context, resulting in unstable and severely low scores (1–2) for the same content that scored 5 when the image was placed first.The fix concatenates all text segments before/after the image, ensuring task instructions and surrounding content are always fully passed to the model regardless of image position.
Made-with: Cursor
OpenJudge Version
[The version of OpenJudge you are working on, e.g.
import openjudge; print(openjudge.__version__)]Description
[Please describe the background, purpose, changes made, and how to test this PR]
Checklist
Please check the following items before code is ready to be reviewed.
pre-commit run --all-filescommand