Batches

The Batches API submits a JSONL-formatted set of requests to a provider’s async batch endpoint and tracks completion server-side. Batch submission is currently implemented for OpenAI (/v1/batches) and Anthropic (/v1/messages/batches); the background poller additionally understands the status APIs of Azure OpenAI, Mistral, Cohere, and Fireworks, but submission for those providers is not yet wired. Most providers offer a ~50% discount for batches with a 24h SLA.

Required capability: batch

Endpoints

Method	Path	Description
POST	/v1/batches	Submit a new batch.
GET	/v1/batches	List batches with cursor pagination.
GET	/v1/batches/:batchId	Fetch current status.
POST	/v1/batches/:batchId/cancel	Cancel an in-progress batch.

Submit

curl https://your-gateway.example.com/v1/batches \
  -H "Authorization: Bearer aigw_sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "openai",
    "endpoint": "/v1/chat/completions",
    "requests": [
      { "custom_id": "row-1", "method": "POST", "url": "/v1/chat/completions",
        "body": { "model": "gpt-4o-mini", "messages": [{"role":"user","content":"Hi"}] } },
      { "custom_id": "row-2", "method": "POST", "url": "/v1/chat/completions",
        "body": { "model": "gpt-4o-mini", "messages": [{"role":"user","content":"Hello"}] } }
    ],
    "completion_webhook_url": "https://yourapp.example.com/webhooks/batch"
  }'

Request fields

Field	Type	Required	Description
provider	string	Yes	Slug of a provider that supports batch submission (openai, anthropic).
endpoint	string	Yes	One of: /v1/chat/completions, /v1/messages, /v1/embeddings, /v1/completions.
requests	array	Yes	1–50,000 request lines. Each has custom_id, method: "POST", url, body.
completion_webhook_url	string	No	URL invoked once the batch reaches a terminal state.
metadata	object	No	Free-form tenant metadata persisted with the batch record.

File-id translation in batch input

If any request line’s body contains an input_file_id (e.g. {"model":"text-embedding-3-large","input_file_id":"file-abc"}), the gateway translates each gateway file ID to the provider’s file ID before submission. Make sure the file’s provider matches the batch’s provider.

Response

{
  "id": "batch-7c8b9d3e-...",
  "object": "batch",
  "provider": "openai",
  "provider_batch_id": "batch_OAI_xyz",
  "endpoint": "/v1/chat/completions",
  "status": "validating",
  "request_counts": { "total": 2, "completed": 0, "failed": 0 },
  "created_at": 1748284800,
  "metadata": null
}

Status lifecycle

validating → in_progress → (completed | failed | cancelled | expired)

A BullMQ background worker polls each non-terminal batch at a per-provider cadence (OpenAI 30s, Anthropic / Mistral / Cohere 60s, Fireworks 30s). When a batch transitions to a terminal state, the gateway:

1. Pulls the final status into the batch record.
2. Stores output / error file IDs.
3. Delivers an HMAC-signed webhook (if completion_webhook_url was provided).

Webhook delivery

Webhook payload is JSON-shaped:

{
  "type": "batch.completed",
  "batch_id": "batch-7c8b9d3e-...",
  "provider": "openai",
  "endpoint": "/v1/chat/completions",
  "status": "completed",
  "request_counts": { "total": 2, "completed": 2, "failed": 0 },
  "provider_batch_id": "batch_OAI_xyz",
  "output_file_url": "/v1/files/file-out-abc/content",
  "error_file_url": null,
  "completed_at": "2026-05-26T18:00:00.000Z",
  "metadata": null
}

output_file_url and error_file_url carry the gateway-relative path of the batch’s provider-side output/error file (the value embeds the upstream provider file ID, e.g. /v1/files/<provider-file-id>/content). Use it as a stable reference to the result file. Note that the gateway does not currently proxy raw file-content downloads — retrieve batch results using the provider file ID with your provider SDK, or via the provider’s own files endpoint.

Headers:

Header	Value
Content-Type	application/json
X-Gateway-Event	batch.completed
X-Gateway-Signature	t=<unix-seconds>,v1=<hex-sha256>
X-Gateway-Delivery	<batch_id>.<attempt-number>

Verify with HMAC-SHA256 of ${timestamp}.${body} using the per-batch secret the gateway generated at submit time. Tolerance: ±5 minutes. The gateway retries up to 5 times with exponential backoff.

Cancel

curl -X POST https://your-gateway.example.com/v1/batches/batch-abc/cancel \
  -H "Authorization: Bearer aigw_sk_your_api_key"

Idempotent — already-terminal batches return their current state.