Loading eval run detail…
Loading eval run detail…
Run inspection
RAG QA Seed · gemini-3.1-flash-lite · v1
647d4df7-9029-48d6-b39f-875def2671a8
Pass rate
95.0%
Share of cases meeting the grader threshold
Average score
0.987
Mean final score after configured grader weighting
Cost
$0.0293967
Estimated total model cost
Avg latency
745.5 ms
P95 867 ms
Failed cases
1
Cases below the composite threshold
Status
Progress
Errors
Total cost
Average latency
P95 latency
Created
Started
Completed
Dataset ID
f67a8007-d9ce-4816-9bed-676547a5f78b
Prompt version ID
76f01278-2380-4481-82d0-51394f25f108
Model
gemini-3.1-flash-lite
Pass rate and score by workflow type.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| rag_qa | 95.0% | 0.987 | 19/201 failed | 745.5 ms | $0.029397 |
Pass rate and score by difficulty.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| easy | 100.0% | 1.000 | 8/8 | 733.3 ms | $0.016284 |
| hard | 80.0% | 0.950 | 4/51 failed | 789.2 ms |
Inspect model inputs, deterministic scores, judge feedback, and rubric details.
Showing 12 of 12
| Test case | Workflow | Difficulty | Final score | Judge score | Failure modes | Details |
|---|---|---|---|---|---|---|
1fcd0247-1220-4a06-b462-bbc543308fe6 | rag qa | easy | 1.000 | — | None | |
2c106ddc-d43e-4b74-8c18-b9025cf8f488 | rag qa | easy | 1.000 | — | None | |
2e3e13c9-2cdc-48b9-bd28-bdc80ddfdc1b | rag qa | easy | 1.000 | — | None | |
334fc972-81eb-409d-ae77-c66fcadb0e7c | rag qa | hard | 1.000 | — | None | |
3cbbf6ee-04c5-41d8-9552-09f9546957c3 | rag qa | medium | 1.000 | — | None | |
4d878405-abbc-41d5-9f56-97f69eac808e | rag qa | hard | 1.000 | — | None | |
5e9825dc-f444-496c-b4c7-6ba40904115b | rag qa | medium | 1.000 | — | None | |
80842481-39a0-4791-b2cf-f75b86c94913 | rag qa | easy | 1.000 | — | None | |
a3afb5e4-0820-4102-a879-7e7f7ed828e1 | rag qa | easy | 1.000 | — | None | |
ad45517e-4142-4336-bcae-4807f1d84ab7 | rag qa | hard | 0.749 | 0.750 | missing required phraselow text similarityunsupported claimmissing citation | |
cc3f0bd7-b8a3-4910-8140-8f7a984fa6e0 | rag qa | medium | 1.000 | — | None | |
e8b0f977-794b-4a5b-b8eb-953917641561 | rag qa | easy | 1.000 | — | None |
20 results with expandable grader details
| Test case | Final score | Judge score | Outcome | Failure modes | Latency | Input tokens | Output tokens | Cost | Error | Grader details | Model output |
|---|---|---|---|---|---|---|---|---|---|---|---|
1fcd0247-1220-4a06-b462-bbc543308fe6 | 1.000 | — | Passed | None | 861 ms | 127 | 48 | $0.0000319 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.662. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. |
| $0.002669 |
| medium | 100.0% | 1.000 | 7/7 | 728.3 ms | $0.010443 |
Pass rate and score by dataset tags.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| api | 100.0% | 1.000 | 2/2 | 841 ms | $0.000068 |
| auth | 100.0% | 1.000 | 2/2 | 651.5 ms | $0.002150 |
| billing | 66.7% | 0.916 | 2/31 failed | 818.3 ms | $0.013229 |
| communication | 100.0% | 1.000 | 1/1 | 694 ms | $0.002106 |
| compliance | 100.0% | 1.000 | 2/2 | 843 ms | $0.000074 |
| datasets | 100.0% | 1.000 | 1/1 | 729 ms | $0.000212 |
| distractors | 100.0% | 1.000 | 1/1 | 826 ms | $0.000217 |
| errors | 100.0% | 1.000 | 1/1 | 821 ms | $0.000036 |
| exports | 100.0% | 1.000 | 2/2 | 753 ms | $0.000239 |
| feature-flags | 100.0% | 1.000 | 2/2 | 766 ms | $0.000071 |
| incidents | 100.0% | 1.000 | 3/3 | 656.7 ms | $0.011001 |
| invoices | 100.0% | 1.000 | 1/1 | 740 ms | $0.007818 |
| limits | 100.0% | 1.000 | 1/1 | 861 ms | $0.000032 |
| logs | 100.0% | 1.000 | 1/1 | 837 ms | $0.000032 |
| multi-doc | 50.0% | 0.875 | 1/21 failed | 956.5 ms | $0.000296 |
| negation | 100.0% | 1.000 | 1/1 | 635 ms | $0.002118 |
| oncall | 100.0% | 1.000 | 2/2 | 668.5 ms | $0.000073 |
| paraphrase | 100.0% | 1.000 | 1/1 | 572 ms | $0.000039 |
| privacy | 100.0% | 1.000 | 3/3 | 786.3 ms | $0.000461 |
| refunds | 100.0% | 1.000 | 1/1 | 651 ms | $0.005157 |
| rollout | 100.0% | 1.000 | 1/1 | 665 ms | $0.000034 |
| safety | 100.0% | 1.000 | 1/1 | 867 ms | $0.000037 |
| security | 100.0% | 1.000 | 2/2 | 676 ms | $0.000235 |
| sla | 100.0% | 1.000 | 3/3 | 633 ms | $0.013074 |
| tricky | 66.7% | 0.916 | 2/31 failed | 757 ms | $0.002410 |
Deterministic graders
LLM judge: —
No judge reason available.
Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}
{ "answer": "The public API has a rate limit of 1,000 requests per minute for each API key.", "citations": [ "rate-limits" ] } |
1ff5795f-b8d0-4ccf-ba8d-dd70e457f8cb | 1.000 | 1.000 | Passed | None | 684 ms | 126 | 37 | $0.0002024 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model accurately answers the question using the provided document and includes the correct citation. Deterministic graders
LLM judge: 1.000 The model accurately answers the question using the provided document and includes the correct citation. | { "answer": "No, customer content is never used to train public models.", "citations": [ "privacy-policy" ] } |
2c106ddc-d43e-4b74-8c18-b9025cf8f488 | 1.000 | — | Passed | None | 777 ms | 124 | 38 | $0.0000276 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.673. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Dataset exports can be provided in either JSONL or CSV formats.", "citations": [ "export-guide" ] } |
2e3e13c9-2cdc-48b9-bd28-bdc80ddfdc1b | 1.000 | — | Passed | None | 668 ms | 129 | 48 | $0.0000321 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.726. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Users can reset their passwords by navigating to Settings > Security. The reset link remains valid for 60 minutes.", "citations": [ "password-reset" ] } |
334fc972-81eb-409d-ae77-c66fcadb0e7c | 1.000 | — | Passed | None | 572 ms | 176 | 53 | $0.0000388 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.554. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "If a region experiences an outage affecting more than 20% of requests, the platform on-call should be paged.", "citations": [ "oncall-runbook" ] } |
3cbbf6ee-04c5-41d8-9552-09f9546957c3 | 1.000 | — | Passed | None | 821 ms | 132 | 57 | $0.000036 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.592. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Once the public API exceeds the quota of 1,000 requests per minute per API key, it returns an HTTP 429 status code.", "citations": [ "rate-limits" ] } |
48f10ff4-1f2e-4485-8475-d660ace8291e | 1.000 | 1.000 | Passed | None | 694 ms | 134 | 41 | $0.0021065 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately identifies the 30-minute update requirement for severity-1 incidents and provides the correct citation. Deterministic graders
LLM judge: 1.000 The response accurately identifies the 30-minute update requirement for severity-1 incidents and provides the correct citation. | { "answer": "Customers should receive updates every 30 minutes during a severity-1 incident.", "citations": [ "sla-guide" ] } |
4ac9f9cc-24f0-4bfc-b1e8-e45bf7d03436 | 1.000 | 1.000 | Passed | None | 615 ms | 132 | 38 | $0.0109385 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model accurately identified the acknowledgment target for severity-1 incidents and included the correct citation. Deterministic graders
LLM judge: 1.000 The model accurately identified the acknowledgment target for severity-1 incidents and included the correct citation. | { "answer": "Severity-1 incidents must be acknowledged within 15 minutes.", "citations": [ "sla-guide" ] } |
4d878405-abbc-41d5-9f56-97f69eac808e | 1.000 | — | Passed | None | 849 ms | 165 | 64 | $0.0000421 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.705. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 2, cited 2, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Customer content is encrypted at rest using AES-256. Regarding retention, audit logs are kept for 365 days before being automatically purged.", "citations": [ "privacy-policy", "data-retention" ] } |
5e9825dc-f444-496c-b4c7-6ba40904115b | 1.000 | — | Passed | None | 765 ms | 132 | 52 | $0.000034 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.739. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "The platform on-call should be paged when there are regional outages that impact more than 20% of requests.", "citations": [ "oncall-runbook" ] } |
7dc549a8-6a4a-4813-be3a-2dc743d1cd6e | 1.000 | 1.000 | Passed | None | 740 ms | 121 | 48 | $0.0078182 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model accurately answers the question based on the provided billing policy document and includes the correct citation. Deterministic graders
LLM judge: 1.000 The model accurately answers the question based on the provided billing policy document and includes the correct citation. | { "answer": "Invoices are generated on the first calendar day of each month for the usage that occurred in the prior month.", "citations": [ "billing-cycles" ] } |
80842481-39a0-4791-b2cf-f75b86c94913 | 1.000 | — | Passed | None | 837 ms | 124 | 48 | $0.0000316 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.752. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Audit logs are retained for a period of 365 days, after which they are automatically purged.", "citations": [ "data-retention" ] } |
84e9370b-511f-4ab3-8f11-42ff4a0aca2d | 1.000 | 1.000 | Passed | None | 729 ms | 132 | 48 | $0.0002117 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The answer accurately provides the requested information regarding supported export formats and processing time, and correctly cites the source document. Deterministic graders
LLM judge: 1.000 The answer accurately provides the requested information regarding supported export formats and processing time, and correctly cites the source document. | { "answer": "Dataset exports are supported in JSONL or CSV formats and are processed asynchronously within a timeframe of 24 hours.", "citations": [ "export-guide" ] } |
9be3a757-073c-43c4-8498-fceb4c66ac1c | 1.000 | 1.000 | Passed | None | 826 ms | 159 | 46 | $0.0002168 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model correctly identified that the Privacy Policy governs the use of customer content and accurately cited the document. Deterministic graders
LLM judge: 1.000 The model correctly identified that the Privacy Policy governs the use of customer content and accurately cited the document. | { "answer": "The Privacy Policy governs the use of customer content, stating that it is never used to train public models.", "citations": [ "privacy-policy" ] } |
a3afb5e4-0820-4102-a879-7e7f7ed828e1 | 1.000 | — | Passed | None | 590 ms | 129 | 39 | $0.0000285 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.857. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "A severity-1 incident must be acknowledged within 15 minutes.", "citations": [ "sla-guide" ] } |
ab2c94b9-5432-4cf6-a685-b1c6c799575d | 1.000 | 1.000 | Passed | None | 635 ms | 122 | 43 | $0.0021175 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately answers the question and cites the correct source document. Deterministic graders
LLM judge: 1.000 The response accurately answers the question and cites the correct source document. | { "answer": "No, password reset links do not last indefinitely; they expire after 60 minutes.", "citations": [ "password-reset" ] } |
ad45517e-4142-4336-bcae-4807f1d84ab7 | 0.749 | 0.750 | Failed | missing required phraselow text similarityunsupported claimmissing citation | 1,064 ms | 159 | 71 | $0.0002541 | — | View grader detailsComposite score 0.749 with pass threshold 0.800. Deterministic score 0.749. LLM judge: The response provides accurate information but failed to properly link citations to specific claims as required by the rubric. Deterministic graders
LLM judge: 0.750 The response provides accurate information but failed to properly link citations to specific claims as required by the rubric. | { "answer": "Customers can request a full refund within 30 days of purchase, provided the subscription remains unused. Invoices are issued on the first day of every month, covering usage from the previous month.", "citations": [ "refund-policy", "billing-cycles" ] } |
cc3f0bd7-b8a3-4910-8140-8f7a984fa6e0 | 1.000 | — | Passed | None | 665 ms | 126 | 54 | $0.0000342 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.702. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Feature flags can be rolled out safely by targeting them by workspace ID and gradually increasing the rollout from 1% to 100%.", "citations": [ "feature-flags" ] } |
e8b0f977-794b-4a5b-b8eb-953917641561 | 1.000 | — | Passed | None | 867 ms | 131 | 60 | $0.0000371 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.405. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Yes, a workspace can gradually enable a feature by using feature flags, which allow for a rollout ranging from 1% to 100% targeted by workspace ID.", "citations": [ "feature-flags" ] } |
f54f13fd-1821-4ef6-9b22-a12edfcd4652 | 1.000 | 1.000 | Passed | None | 651 ms | 126 | 48 | $0.0051571 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model accurately identified the refund policy timeline and correctly cited the relevant document. Deterministic graders
LLM judge: 1.000 The model accurately identified the refund policy timeline and correctly cited the relevant document. | { "answer": "Customers have 30 days from the date of purchase to request a full refund, provided the subscription remains unused.", "citations": [ "refund-policy" ] } |