Skip to content

Commit b90d90b

Browse files
XiaoBoAIcursoragent
XiaoBoAI
and
cursoragent
committed
fix(graders): remove angle brackets from JSON placeholders in prompt templates
Angle bracket placeholders like `<integer between 1 and 5>` in Output Schema sections could be misinterpreted as XML tags by LLMs, causing malformed output. Replaced with plain text descriptions to avoid ambiguity with surrounding XML tags. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent b0b96d0 commit b90d90b

10 files changed

Lines changed: 44 additions & 44 deletions

‎openjudge/graders/multi_turn/anaphora_resolution_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@
7373
<输出格式>
7474
请按以下JSON格式输出评估结果：
7575
{{
76-
"reason": "<详细说明评估理由，包括代词解析的正确性分析>",
77-
"score": <1-5的整数>
76+
"reason": "详细说明评估理由，包括代词解析的正确性分析",
77+
"score": 1-5的整数
7878
}}
7979
</输出格式>
8080
@@ -131,8 +131,8 @@
131131
<Output Schema>
132132
Output your evaluation in the following JSON format:
133133
{{
134-
"reason": "<detailed explanation including analysis of pronoun resolution correctness>",
135-
"score": <integer from 1-5>
134+
"reason": "detailed explanation including analysis of pronoun resolution correctness",
135+
"score": integer from 1-5
136136
}}
137137
</Output Schema>
138138

‎openjudge/graders/multi_turn/context_memory_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,8 @@
7272
<输出格式>
7373
请按以下JSON格式输出评估结果：
7474
{{
75-
"reason": "<详细说明评估理由，包括助手记住或遗忘了哪些关键信息>",
76-
"score": <1-5的整数>
75+
"reason": "详细说明评估理由，包括助手记住或遗忘了哪些关键信息",
76+
"score": 1-5的整数
7777
}}
7878
</输出格式>
7979
@@ -128,8 +128,8 @@
128128
<Output Schema>
129129
Output your evaluation in the following JSON format:
130130
{{
131-
"reason": "<detailed explanation including what key information the assistant remembered or forgot>",
132-
"score": <integer from 1-5>
131+
"reason": "detailed explanation including what key information the assistant remembered or forgot",
132+
"score": integer from 1-5
133133
}}
134134
</Output Schema>
135135

‎openjudge/graders/multi_turn/instruction_clarification_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@
7373
<输出格式>
7474
请按以下JSON格式输出评估结果：
7575
{{
76-
"reason": "<详细说明评估理由，包括模糊点识别和追问质量分析>",
77-
"score": <1-5的整数>
76+
"reason": "详细说明评估理由，包括模糊点识别和追问质量分析",
77+
"score": 1-5的整数
7878
}}
7979
</输出格式>
8080
@@ -130,8 +130,8 @@
130130
<Output Schema>
131131
Output your evaluation in the following JSON format:
132132
{{
133-
"reason": "<detailed explanation including ambiguity identification and question quality analysis>",
134-
"score": <integer from 1-5>
133+
"reason": "detailed explanation including ambiguity identification and question quality analysis",
134+
"score": integer from 1-5
135135
}}
136136
</Output Schema>
137137

‎openjudge/graders/multi_turn/proactive_interaction_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@
7373
<输出格式>
7474
请按以下JSON格式输出评估结果：
7575
{{
76-
"reason": "<详细说明评估理由，包括主动互动的质量和自然程度分析>",
77-
"score": <1-5的整数>
76+
"reason": "详细说明评估理由，包括主动互动的质量和自然程度分析",
77+
"score": 1-5的整数
7878
}}
7979
</输出格式>
8080
@@ -133,8 +133,8 @@
133133
<Output Schema>
134134
Output your evaluation in the following JSON format:
135135
{{
136-
"reason": "<detailed explanation including analysis of proactive interaction quality and naturalness>",
137-
"score": <integer from 1-5>
136+
"reason": "detailed explanation including analysis of proactive interaction quality and naturalness",
137+
"score": integer from 1-5
138138
}}
139139
</Output Schema>
140140

‎openjudge/graders/multi_turn/response_repetition_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,8 @@
7676
<输出格式>
7777
请按以下JSON格式输出评估结果：
7878
{{
79-
"reason": "<详细说明评估理由，包括发现的重复内容及其影响分析>",
80-
"score": <1-5的整数>
79+
"reason": "详细说明评估理由，包括发现的重复内容及其影响分析",
80+
"score": 1-5的整数
8181
}}
8282
</输出格式>
8383
@@ -136,8 +136,8 @@
136136
<Output Schema>
137137
Output your evaluation in the following JSON format:
138138
{{
139-
"reason": "<detailed explanation including identified repetitive content and impact analysis>",
140-
"score": <integer from 1-5>
139+
"reason": "detailed explanation including identified repetitive content and impact analysis",
140+
"score": integer from 1-5
141141
}}
142142
</Output Schema>
143143

‎openjudge/graders/multi_turn/self_correction_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@
7373
<输出格式>
7474
请按以下JSON格式输出评估结果：
7575
{{
76-
"reason": "<详细说明评估理由，包括错误识别、修正质量和态度分析>",
77-
"score": <1-5的整数>
76+
"reason": "详细说明评估理由，包括错误识别、修正质量和态度分析",
77+
"score": 1-5的整数
7878
}}
7979
</输出格式>
8080
@@ -133,8 +133,8 @@
133133
<Output Schema>
134134
Output your evaluation in the following JSON format:
135135
{{
136-
"reason": "<detailed explanation including error recognition, correction quality, and attitude analysis>",
137-
"score": <integer from 1-5>
136+
"reason": "detailed explanation including error recognition, correction quality, and attitude analysis",
137+
"score": integer from 1-5
138138
}}
139139
</Output Schema>
140140

‎openjudge/graders/multi_turn/topic_switch_grader.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,8 @@
7474
<输出格式>
7575
请按以下JSON格式输出评估结果：
7676
{{
77-
"reason": "<详细说明评估理由，包括话题切换的识别和处理分析>",
78-
"score": <1-5的整数>
77+
"reason": "详细说明评估理由，包括话题切换的识别和处理分析",
78+
"score": 1-5的整数
7979
}}
8080
</输出格式>
8181
@@ -131,8 +131,8 @@
131131
<Output Schema>
132132
Output your evaluation in the following JSON format:
133133
{{
134-
"reason": "<detailed explanation including analysis of topic switch recognition and handling>",
135-
"score": <integer from 1-5>
134+
"reason": "detailed explanation including analysis of topic switch recognition and handling",
135+
"score": integer from 1-5
136136
}}
137137
</Output Schema>
138138

‎openjudge/graders/multimodal/image_coherence.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,8 @@
6464
6565
<Output Schema>
6666
{{
67-
"reason": "<brief explanation for the assigned score>",
68-
"score": <integer between 1 and 5>
67+
"reason": "brief explanation for the assigned score",
68+
"score": integer between 1 and 5
6969
}}
7070
</Output Schema>
7171
JSON:
@@ -108,8 +108,8 @@
108108
109109
<输出格式>
110110
{{
111-
"reason": "<对所给分数的简要解释>",
112-
"score": <1到5之间的整数>
111+
"reason": "对所给分数的简要解释",
112+
"score": 1到5之间的整数
113113
}}
114114
</输出格式>
115115
JSON:

‎openjudge/graders/multimodal/image_helpfulness.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,8 @@
6565
6666
<Output Schema>
6767
{{
68-
"reason": "<brief explanation for the assigned score>",
69-
"score": <integer between 1 and 5>
68+
"reason": "brief explanation for the assigned score",
69+
"score": integer between 1 and 5
7070
}}
7171
</Output Schema>
7272
JSON:
@@ -109,8 +109,8 @@
109109
110110
<输出格式>
111111
{{
112-
"reason": "<对所给分数的简要解释>",
113-
"score": <1到5之间的整数>
112+
"reason": "对所给分数的简要解释",
113+
"score": 1到5之间的整数
114114
}}
115115
</输出格式>
116116
JSON:

‎openjudge/graders/multimodal/text_to_image.py‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@
6161
6262
<Output Schema>
6363
{{
64-
"score": [<integer between 1 and 5>],
65-
"reason": "<brief explanation>"
64+
"score": [integer between 1 and 5],
65+
"reason": "brief explanation"
6666
}}
6767
</Output Schema>
6868
JSON:
@@ -108,8 +108,8 @@
108108
109109
<Output Schema>
110110
{{
111-
"score": [<naturalness 1-5>, <artifacts 1-5>],
112-
"reason": "<brief explanation>"
111+
"score": [naturalness 1-5, artifacts 1-5],
112+
"reason": "brief explanation"
113113
}}
114114
</Output Schema>
115115
JSON:
@@ -150,8 +150,8 @@
150150
151151
<输出格式>
152152
{{
153-
"score": [<1到5之间的整数>],
154-
"reason": "<简要解释>"
153+
"score": [1到5之间的整数],
154+
"reason": "简要解释"
155155
}}
156156
</输出格式>
157157
JSON:
@@ -197,8 +197,8 @@
197197
198198
<输出格式>
199199
{{
200-
"score": [<自然度 1-5>, <伪影 1-5>],
201-
"reason": "<简要解释>"
200+
"score": [自然度 1-5, 伪影 1-5],
201+
"reason": "简要解释"
202202
}}
203203
</输出格式>
204204
JSON:

0 commit comments

Comments
 (0)