Loading eval run detail…
Loading eval run detail…
Run inspection
RAG QA Seed · gemini-3.1-flash-lite · v1-degraded
6fa61dcb-8a5a-4224-8523-8606fee31cc7
Pass rate
85.0%
Share of cases meeting the grader threshold
Average score
0.953
Mean final score after configured grader weighting
Cost
$0.0630011
Estimated total model cost
Avg latency
808.2 ms
P95 908 ms
Failed cases
3
Cases below the composite threshold
Status
Progress
Errors
Total cost
Average latency
P95 latency
Created
Started
Completed
Dataset ID
f67a8007-d9ce-4816-9bed-676547a5f78b
Prompt version ID
ca16be90-7da1-487a-adf8-35ef486bdb41
Model
gemini-3.1-flash-lite
Pass rate and score by workflow type.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| rag_qa | 85.0% | 0.953 | 17/203 failed | 808.2 ms | $0.063001 |
Pass rate and score by difficulty.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| easy | 87.5% | 0.948 | 7/81 failed | 768 ms | $0.020936 |
| hard | 80.0% | 0.952 | 4/51 failed | 822.8 ms |
Inspect model inputs, deterministic scores, judge feedback, and rubric details.
Showing 12 of 12
| Test case | Workflow | Difficulty | Final score | Judge score | Failure modes | Details |
|---|---|---|---|---|---|---|
1ff5795f-b8d0-4ccf-ba8d-dd70e457f8cb | rag qa | medium | 0.715 | 0.900 | missing required phraselow text similarityunsupported claim | |
2e3e13c9-2cdc-48b9-bd28-bdc80ddfdc1b | rag qa | easy | 1.000 | — | None | |
334fc972-81eb-409d-ae77-c66fcadb0e7c | rag qa | hard | 1.000 | — | None | |
48f10ff4-1f2e-4485-8475-d660ace8291e | rag qa | medium | 1.000 | — | None | |
4ac9f9cc-24f0-4bfc-b1e8-e45bf7d03436 | rag qa | easy | 1.000 | — | None | |
5e9825dc-f444-496c-b4c7-6ba40904115b | rag qa | medium | 1.000 | — | None | |
84e9370b-511f-4ab3-8f11-42ff4a0aca2d | rag qa | medium | 1.000 | — | None | |
a3afb5e4-0820-4102-a879-7e7f7ed828e1 | rag qa | easy | 1.000 | — | None | |
ab2c94b9-5432-4cf6-a685-b1c6c799575d | rag qa | hard | 1.000 | — | None | |
ad45517e-4142-4336-bcae-4807f1d84ab7 | rag qa | hard | 0.758 | — | missing required phraselow text similarityunsupported claim | |
e8b0f977-794b-4a5b-b8eb-953917641561 | rag qa | easy | 0.580 | 0.700 | missing required phraselow text similarityunsupported claim | |
f54f13fd-1821-4ef6-9b22-a12edfcd4652 | rag qa | easy | 1.000 | — | None |
20 results with expandable grader details
| Test case | Final score | Judge score | Outcome | Failure modes | Latency | Input tokens | Output tokens | Cost | Error | Grader details | Model output |
|---|---|---|---|---|---|---|---|---|---|---|---|
1fcd0247-1220-4a06-b462-bbc543308fe6 | 1.000 | 1.000 | Passed | None | 901 ms | 112 | 66 | $0.0045405 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately identifies the API rate limit and correctly cites the source document. Deterministic graders
|
| $0.012733 |
| medium | 85.7% | 0.959 | 6/71 failed | 843.6 ms | $0.029333 |
Pass rate and score by dataset tags.
| Key | Pass rate | Avg score | Cases | Avg latency | Cost |
|---|---|---|---|---|---|
| api | 100.0% | 1.000 | 2/2 | 880.5 ms | $0.009619 |
| auth | 100.0% | 1.000 | 2/2 | 728 ms | $0.000063 |
| billing | 66.7% | 0.919 | 2/31 failed | 860.7 ms | $0.006903 |
| communication | 100.0% | 1.000 | 1/1 | 804 ms | $0.000029 |
| compliance | 100.0% | 1.000 | 2/2 | 721 ms | $0.015213 |
| datasets | 100.0% | 1.000 | 1/1 | 673 ms | $0.000032 |
| distractors | 100.0% | 1.000 | 1/1 | 866 ms | $0.004962 |
| errors | 100.0% | 1.000 | 1/1 | 860 ms | $0.005079 |
| exports | 100.0% | 1.000 | 2/2 | 756 ms | $0.004408 |
| feature-flags | 50.0% | 0.790 | 1/21 failed | 892 ms | $0.014660 |
| incidents | 100.0% | 1.000 | 3/3 | 778.3 ms | $0.000089 |
| invoices | 100.0% | 1.000 | 1/1 | 888 ms | $0.006829 |
| limits | 100.0% | 1.000 | 1/1 | 901 ms | $0.004541 |
| logs | 100.0% | 1.000 | 1/1 | 612 ms | $0.007555 |
| multi-doc | 50.0% | 0.879 | 1/21 failed | 845 ms | $0.007700 |
| negation | 100.0% | 1.000 | 1/1 | 755 ms | $0.000032 |
| oncall | 100.0% | 1.000 | 2/2 | 878.5 ms | $0.000072 |
| paraphrase | 100.0% | 1.000 | 1/1 | 803 ms | $0.000039 |
| privacy | 66.7% | 0.905 | 2/31 failed | 838 ms | $0.019637 |
| refunds | 100.0% | 1.000 | 1/1 | 834 ms | $0.000031 |
| rollout | 100.0% | 1.000 | 1/1 | 908 ms | $0.010314 |
| safety | 0.0% | 0.580 | 0/11 failed | 876 ms | $0.004346 |
| security | 50.0% | 0.857 | 1/21 failed | 759.5 ms | $0.007049 |
| sla | 100.0% | 1.000 | 3/3 | 728.3 ms | $0.000084 |
| tricky | 66.7% | 0.919 | 2/31 failed | 806 ms | $0.000114 |
LLM judge: 1.000
The response accurately identifies the API rate limit and correctly cites the source document.
{ "answer": "The public API enforces a rate limit of 1,000 requests per minute for each API key. If this limit is exceeded, the server will return an HTTP 429 status code.", "citations": [ "rate-limits" ] } |
1ff5795f-b8d0-4ccf-ba8d-dd70e457f8cb | 0.715 | 0.900 | Failed | missing required phraselow text similarityunsupported claim | 818 ms | 111 | 54 | $0.0070176 | — | View grader detailsComposite score 0.715 with pass threshold 0.800. Deterministic score 0.592. LLM judge: The answer accurately addresses the question using the provided document, although it deviates slightly from the exact phrasing requested in the expected output. Deterministic graders
LLM judge: 0.900 The answer accurately addresses the question using the provided document, although it deviates slightly from the exact phrasing requested in the expected output. | { "answer": "No, customer content is not used to train public models. It is protected by AES-256 encryption at rest to ensure data privacy.", "citations": [ "privacy-policy" ] } |
2c106ddc-d43e-4b74-8c18-b9025cf8f488 | 1.000 | 1.000 | Passed | None | 839 ms | 109 | 38 | $0.0043759 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model correctly identified the export formats supported by the dataset as requested, citing the appropriate document. Deterministic graders
LLM judge: 1.000 The model correctly identified the export formats supported by the dataset as requested, citing the appropriate document. | { "answer": "Dataset exports can be generated in either JSONL or CSV formats.", "citations": [ "export-guide" ] } |
2e3e13c9-2cdc-48b9-bd28-bdc80ddfdc1b | 1.000 | — | Passed | None | 701 ms | 114 | 49 | $0.000031 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.687. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Users can reset their password by navigating to Settings > Security. The password reset link remains valid for 60 minutes.", "citations": [ "password-reset" ] } |
334fc972-81eb-409d-ae77-c66fcadb0e7c | 1.000 | — | Passed | None | 803 ms | 161 | 56 | $0.0000385 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.292. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "If a region experiences a loss of more than one fifth (20%) of its traffic, you should page the platform on-call team.", "citations": [ "oncall-runbook" ] } |
3cbbf6ee-04c5-41d8-9552-09f9546957c3 | 1.000 | 1.000 | Passed | None | 860 ms | 117 | 82 | $0.0050786 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The answer accurately describes the API rate limit behavior and correctly cites the provided document. Deterministic graders
LLM judge: 1.000 The answer accurately describes the API rate limit behavior and correctly cites the provided document. | { "answer": "Once the public API exceeds the limit of 1,000 requests per minute for a specific API key, the system will return an HTTP 429 'Too Many Requests' status code, indicating that the user has sent too many requests in a given amount of time.", "citations": [ "rate-limits" ] } |
48f10ff4-1f2e-4485-8475-d660ace8291e | 1.000 | — | Passed | None | 804 ms | 119 | 42 | $0.0000287 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.752. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "For severity-1 incidents, customers should receive status updates every 30 minutes.", "citations": [ "sla-guide" ] } |
4ac9f9cc-24f0-4bfc-b1e8-e45bf7d03436 | 1.000 | — | Passed | None | 644 ms | 117 | 39 | $0.0000273 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.520. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "The acknowledgment target for severity-1 incidents is 15 minutes.", "citations": [ "sla-guide" ] } |
4d878405-abbc-41d5-9f56-97f69eac808e | 1.000 | 1.000 | Passed | None | 830 ms | 150 | 85 | $0.0076571 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model accurately addressed both encryption and retention questions using the provided documents and included correct citations. Deterministic graders
LLM judge: 1.000 The model accurately addressed both encryption and retention questions using the provided documents and included correct citations. | { "answer": "Customer content is secured using AES-256 encryption at rest and is strictly excluded from use in training public models. Regarding data retention, audit logs are maintained for a period of 365 days, after which they are automatically purged from the system.", "citations": [ "privacy-policy", "data-retention" ] } |
5e9825dc-f444-496c-b4c7-6ba40904115b | 1.000 | — | Passed | None | 954 ms | 117 | 54 | $0.0000333 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.670. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "The platform on-call should be paged specifically when there are regional outages that impact more than 20% of total requests.", "citations": [ "oncall-runbook" ] } |
7dc549a8-6a4a-4813-be3a-2dc743d1cd6e | 1.000 | 1.000 | Passed | None | 888 ms | 106 | 46 | $0.0068286 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The answer accurately states when invoices are generated and provides the correct citation. Deterministic graders
LLM judge: 1.000 The answer accurately states when invoices are generated and provides the correct citation. | { "answer": "Invoices are generated on the first calendar day of every month, covering the usage from the preceding month.", "citations": [ "billing-cycles" ] } |
80842481-39a0-4791-b2cf-f75b86c94913 | 1.000 | 1.000 | Passed | None | 612 ms | 109 | 50 | $0.0075555 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The model correctly identifies the retention period and provides the appropriate citation. Deterministic graders
LLM judge: 1.000 The model correctly identifies the retention period and provides the appropriate citation. | { "answer": "Audit logs are retained for a period of 365 days, after which they are automatically purged from the system.", "citations": [ "data-retention" ] } |
84e9370b-511f-4ab3-8f11-42ff4a0aca2d | 1.000 | — | Passed | None | 673 ms | 117 | 50 | $0.0000317 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.587. Matched 2/2 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Dataset exports are supported in JSONL and CSV formats. These exports are processed asynchronously and typically complete within 24 hours.", "citations": [ "export-guide" ] } |
9be3a757-073c-43c4-8498-fceb4c66ac1c | 1.000 | 1.000 | Passed | None | 866 ms | 144 | 48 | $0.0049622 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The answer accurately identifies the correct document and provides the requested information with proper citation. Deterministic graders
LLM judge: 1.000 The answer accurately identifies the correct document and provides the requested information with proper citation. | { "answer": "The Privacy Policy governs the use of customer content, explicitly stating that it is never used to train public models.", "citations": [ "privacy-policy" ] } |
a3afb5e4-0820-4102-a879-7e7f7ed828e1 | 1.000 | — | Passed | None | 737 ms | 114 | 42 | $0.0000282 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.734. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "A severity-1 incident should be acknowledged within 15 minutes of being reported.", "citations": [ "sla-guide" ] } |
ab2c94b9-5432-4cf6-a685-b1c6c799575d | 1.000 | — | Passed | None | 755 ms | 107 | 53 | $0.0000319 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.354. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "No, password reset links do not last indefinitely. According to the security documentation, these links expire 60 minutes after they are generated.", "citations": [ "password-reset" ] } |
ad45517e-4142-4336-bcae-4807f1d84ab7 | 0.758 | — | Failed | missing required phraselow text similarityunsupported claim | 860 ms | 144 | 72 | $0.0000432 | — | View grader detailsComposite score 0.758 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=0.610 (Text similarity: 0.610. Matched 1/2 required phrases.) citation=0.750 (Citation score 0.750. Required 2, cited 2, missing 0, invalid 0, unsupported claims 1. Unsupported claims: first calendar day.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Refunds can be requested for unused subscriptions within 30 days of the initial purchase date. Invoices are issued on the first day of every month, covering all usage activity from the preceding month.", "citations": [ "refund-policy", "billing-cycles" ] } |
cc3f0bd7-b8a3-4910-8140-8f7a984fa6e0 | 1.000 | 1.000 | Passed | None | 908 ms | 111 | 94 | $0.0103141 | — | View grader detailsComposite score 1.000 with pass threshold 0.800. Deterministic score 1.000. LLM judge: The response accurately addresses the question using information from the provided documents and includes the required citation. Deterministic graders
LLM judge: 1.000 The response accurately addresses the question using information from the provided documents and includes the required citation. | { "answer": "To roll out feature flags safely, you should implement a gradual rollout strategy, starting with a small percentage of users (e.g., 1%) and incrementally increasing to 100%. Additionally, you can enhance safety by targeting specific segments, such as by workspace ID, to isolate the impact of the new feature during the testing phase.", "citations": [ "feature-flags" ] } |
e8b0f977-794b-4a5b-b8eb-953917641561 | 0.580 | 0.700 | Failed | missing required phraselow text similarityunsupported claim | 876 ms | 116 | 85 | $0.0043461 | — | View grader detailsComposite score 0.580 with pass threshold 0.800. Deterministic score 0.500. LLM judge: The answer provides correct information but fails to properly cite or incorporate the required 'workspace ID' phrasing as requested by the rubric, leading to an unsupported claim finding regarding the citation. Deterministic graders
LLM judge: 0.700 The answer provides correct information but fails to properly cite or incorporate the required 'workspace ID' phrasing as requested by the rubric, leading to an unsupported claim finding regarding the citation. | { "answer": "Yes, a workspace can gradually enable a risky feature by using feature flags. These flags allow for a controlled rollout, enabling the feature for a specific percentage of workspaces, starting from as low as 1% and scaling up to 100% as confidence in the feature's stability increases.", "citations": [ "feature-flags" ] } |
f54f13fd-1821-4ef6-9b22-a12edfcd4652 | 1.000 | — | Passed | None | 834 ms | 111 | 50 | $0.0000311 | — | View grader detailsComposite score 1.000 with pass threshold 0.850. json_schema=1.000 (Required fields: 2/2 valid. Field types: 2/2 present fields valid.) text_similarity=1.000 (Text similarity: 0.782. Matched 1/1 required phrases.) citation=1.000 (Citation score 1.000. Required 1, cited 1, missing 0, invalid 0, unsupported claims 0.) LLM judge error: Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}}. Deterministic graders
LLM judge: — No judge reason available. Gemini judge provider error: 504 DEADLINE_EXCEEDED. {'error': {'code': 504, 'message': 'Deadline expired before operation could complete.', 'status': 'DEADLINE_EXCEEDED'}} | { "answer": "Customers are eligible to request a full refund within 30 days of their purchase, provided that the subscription has remained unused.", "citations": [ "refund-policy" ] } |