Batch API Quickstart
Submit OpenRouter-compatible inline batches through the attested TrustedRouter gateway and poll encrypted results by batch ID.
Submit many requests. Poll one batch.
TrustedRouter implements the OpenRouter inline Batch API shape at /api/beta/batches. Existing integrations can switch the hostname and API key without uploading a JSONL file.
Batch items use the same attested authorization, routing, fallback, and integer billing as ordinary inference. When native execution is enabled for an eligible direct route, TrustedRouter uses that provider's Batch API and applies its published discount.
import os
import time
import requests
root = "https://api.trustedrouter.com"
headers = {
"Authorization": f"Bearer {os.environ['TRUSTEDROUTER_API_KEY']}",
"Content-Type": "application/json",
}
# Keep endpoint and model before requests.
created = requests.post(
f"{root}/api/beta/batches",
headers=headers,
json={
"endpoint": "/v1/chat/completions",
"model": "deepseek/deepseek-v4-flash",
"requests": [
{
"custom_id": "summary-001",
"body": {
"messages": [{
"role": "user",
"content": "Reply exactly PONG",
}],
"max_tokens": 64,
},
}
],
},
timeout=30,
)
created.raise_for_status()
batch = created.json()
while batch["status"] not in {
"completed", "failed", "expired", "cancelled"
}:
time.sleep(5)
response = requests.get(
f"{root}/api/beta/batches/{batch['id']}",
headers=headers,
timeout=30,
)
response.raise_for_status()
batch = response.json()
print(batch["results"])Drop-in contract
POST /api/beta/batches
Returns HTTP 202 with a batch_* ID, status, request counts, completion window, and aggregate usage fields.
GET /api/beta/batches/{id}
Poll with the same API key. A completed batch returns results inline in original request order.
Response or error per item
Every result preserves custom_id and has either a response with status, request ID, and body, or a stable error.
Use the provider's real Batch API.
When native execution is enabled, eligible direct OpenAI and Parasail chat or embedding jobs are uploaded by the attested gateway to that provider's native asynchronous Batch API. Both currently publish a 50% discount, which TrustedRouter applies with integer microdollar accounting after the provider completes each item.
If native submission is unsupported or a native item fails, TrustedRouter releases its native hold and runs that item through the ordinary attested fallback path at the selected fallback route's normal price.
No silent privacy downgrade
Native provider Batch APIs retain prompt and output content. TrustedRouter therefore keeps ZDR, E2E, confidential, EU, BYOK, custom, orchestration, and Broadcast-enabled jobs on the ordinary enclave-managed path.
One batch format across four APIs.
/v1/chat/completions/v1/responses/v1/messages/v1/embeddings
Streaming is not available inside a batch. Embedding batches accept text, text arrays, token arrays, and arrays of token arrays. Provider preferences, input_type, and multimodal embedding inputs are rejected before the job is created.
{
"endpoint": "/v1/responses",
"model": "your/model",
"requests": [
{
"custom_id": "task-001",
"body": {
"input": "Summarize this text.",
"max_output_tokens": 300
}
}
]
}Execution guarantees
Exactly-once settlement
Each item has a deterministic idempotency key. Worker restarts recover encrypted checkpoints instead of rerunning completed requests.
Regional lease safety
Workers claim jobs with generation-checked leases. Only one region executes an item while another region can recover an abandoned job.
Partial results
A provider or validation failure is attached to that item. Successful items remain successful and are billed once.
Encrypted job state. No plaintext storage.
Submitting to the Batch API opts into temporary encrypted storage so jobs survive restarts and results can be polled. Prompts and outputs are encrypted inside the attested gateway with a per-artifact key. The raw API key is validated and discarded during submission; it is never persisted, even inside an encrypted artifact.
Under the deployed GCP IAM policy, only approved production enclave image digests are granted access to the ciphertext and its Cloud KMS envelope key. The control plane, object storage, logs, billing records, Sentry, and analytics do not receive plaintext batch content. Batch artifacts are automatically deleted after 30 days.
When an eligible job uses a provider-native Batch API, that provider receives and temporarily retains plaintext input for up to 26 hours and provider results for up to 6 hours after availability. This is why native Batch execution is never used for privacy-constrained aliases or requests.
This encrypted-retention boundary depends on GCP KMS, Cloud Storage, and the project's IAM administration. It is intentionally stronger than ordinary application-layer storage, but it is not the same zero-retention property as synchronous and streaming inference.
Ordinary synchronous and streaming inference remains content-stateless. Strict EU data-residency batch execution is not included in this first beta.
Billing and limits
Native discount eligibility
Eligible OpenAI and Parasail native Batch items receive their published 50% discount when native execution is enabled. Managed fallback and providers without a verified native discount use normal route pricing. Read returned usage instead of assuming every item was discounted.
Current beta limits
- 24-hour completion window
- 32 MiB create request limit
- 50,000 requests per batch
- 64 MiB response limit per item
- Unique
custom_idvalues, up to 256 bytes - No list, cancel, or file-upload endpoints