LLM Console

Quick start

The API is OpenAI-compatible, so any OpenAI SDK works: point it at the base URL below and use your API key. Keys are issued by an admin in Settings → API keys.

Base URLBASE_URL

Chat & vision modelqwen3-vl-32b

Speech-to-text modelwhisper-large-v3

Embedding modelmultilingual-e5-large-instruct

curl BASE_URL/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-vl-32b", "messages": [{"role": "user", "content": "Write a one-line welcome for Grade 4 parents."}]}'

from openai import OpenAI  # pip install openai

client = OpenAI(base_url="BASE_URL", api_key="sk-llm-…")

reply = client.chat.completions.create(
    model="qwen3-vl-32b",
    messages=[{"role": "user", "content": "Write a one-line welcome for Grade 4 parents."}],
)
print(reply.choices[0].message.content)

import OpenAI from "openai"; // npm install openai

const client = new OpenAI({ baseURL: "BASE_URL", apiKey: process.env.LLM_API_KEY });

const reply = await client.chat.completions.create({
  model: "qwen3-vl-32b",
  messages: [{ role: "user", content: "Write a one-line welcome for Grade 4 parents." }],
});
console.log(reply.choices[0].message.content);

Authentication

Send the key as a bearer token on every request: Authorization: Bearer sk-llm-…. Each key belongs to one person or app, every request is logged against it, and a revoked key stops working immediately.

Keep keys on the server side (environment variables, not browser code or git). If a key leaks, ask an admin to revoke it and create a new one.

Your key may come with a console login (your email and a password from the admin). Signed in, you see your own requests, logs, usage and webhooks, plus how busy the GPU and CPU are. Other people's data stays hidden.

Wrong or revoked key

HTTP/1.1 401 Unauthorized
{
  "error": {
    "message": "Missing or invalid API key",
    "type": "invalid_request_error",
    "code": "invalid_api_key"
  }
}

AI coding agents (MCP)

Connect Claude Code, Cursor, VS Code (Copilot) or any other MCP client to MCP_URL. While it writes your integration, the agent can look up exactly how to call each model: endpoints, fields, limits, errors, retries and webhook signatures. It uses your normal API key and only serves this documentation. It never calls the models or sees your data.

# Run once in your project (LLM_API_KEY must be set in your shell)
claude mcp add --transport http orchids-llm MCP_URL \
  --header "Authorization: Bearer $LLM_API_KEY"

// ~/.cursor/mcp.json (or .cursor/mcp.json in the project); LLM_API_KEY must be set in your environment
{
  "mcpServers": {
    "orchids-llm": {
      "url": "MCP_URL",
      "headers": { "Authorization": "Bearer ${env:LLM_API_KEY}" }
    }
  }
}

// .vscode/mcp.json: VS Code asks for the key once and stores it securely
{
  "servers": {
    "orchids-llm": {
      "type": "http",
      "url": "MCP_URL",
      "headers": { "Authorization": "Bearer ${input:llm-api-key}" }
    }
  },
  "inputs": [{ "type": "promptString", "id": "llm-api-key", "description": "Orchids LLM API key", "password": true }]
}

Tools: list_models (what each model does) and get_integration_guide (the full brief for qwen, whisper or embeddings, the same text as the Copy for AI buttons).
Prompts: integrate_qwen, integrate_whisper and integrate_embeddings start an integration from a one-line description of your task.
Then just ask, for example: "Use orchids-llm to add Hindi and English semantic search over our FAQ table."

Chat & vision

POST /v1/chat/completions with model qwen3-vl-32b. Send text, or text plus images as URLs or base64 data URLs. Up to 16 images per request and 16,384 tokens of context (prompt plus reply). Video input is turned off. Tool calling (tools) is supported.

Building with an AI assistant?Copy a complete Qwen chat & vision brief for ChatGPT, Claude, Cursor or Copilot, then add what you are building.

Each base64 image can be at most 1.5 MB. A larger image is not sent to the model: you still get a normal reply in the format your prompt asks for, with nothing read from that image and The file size is very large. in its remarks (or as the whole reply if the format has no such field). Resize photos to about 1600 px on the long side.

Images

curl BASE_URL/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-vl-32b",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "What is on this worksheet?"},
      {"type": "image_url", "image_url": {"url": "https://example.com/page1.jpg"}}
    ]}]
  }'

import base64
from openai import OpenAI

client = OpenAI(base_url="BASE_URL", api_key="sk-llm-…")
image = base64.b64encode(open("page1.jpg", "rb").read()).decode()

reply = client.chat.completions.create(
    model="qwen3-vl-32b",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What is on this worksheet?"},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image}"}},
    ]}],
)
print(reply.choices[0].message.content)

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "BASE_URL", apiKey: process.env.LLM_API_KEY });
const image = fs.readFileSync("page1.jpg").toString("base64");

const reply = await client.chat.completions.create({
  model: "qwen3-vl-32b",
  messages: [{ role: "user", content: [
    { type: "text", text: "What is on this worksheet?" },
    { type: "image_url", image_url: { url: `data:image/jpeg;base64,${image}` } },
  ] }],
});
console.log(reply.choices[0].message.content);

Streaming

Set "stream": true to receive tokens as server-sent events. If an admin cancels a request, the stream ends with HTTP 499 request_cancelled.

curl -N BASE_URL/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-vl-32b", "stream": true, "messages": [{"role": "user", "content": "Explain photosynthesis simply."}]}'

stream = client.chat.completions.create(
    model="qwen3-vl-32b",
    messages=[{"role": "user", "content": "Explain photosynthesis simply."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

const stream = await client.chat.completions.create({
  model: "qwen3-vl-32b",
  messages: [{ role: "user", content: "Explain photosynthesis simply." }],
  stream: true,
});
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");

Embeddings

POST /v1/embeddings with model multilingual-e5-large-instruct turns text into a vector of 1024 numbers for semantic search, RAG, de-duplication or clustering. It understands English, Hindi and the other Indic languages (Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam…), Hinglish included, and matches across languages: a Hindi question finds an English answer. Vectors are normalized, so cosine similarity is a plain dot product.

Building with an AI assistant?Copy a complete embeddings brief for ChatGPT, Claude, Cursor or Copilot, then add what you are building.

Set input_type correctly, it matters for search quality: "document" (the default) for the texts you store, "query" for the user's search question. Mixing them up still works but ranks worse.

Field	Required	Description
`input`	yes	A string or an array of strings. At most 512 strings and about 65,000 tokens per request.
`model`	no	`multilingual-e5-large-instruct`
`input_type`	no	`document` (default) or `query`.
`instruction`	no	For queries only: one sentence describing the task, used instead of the default "Given a web search query, retrieve relevant passages that answer the query". Example: "Given a parent's question, retrieve the matching school FAQ".
`encoding_format`	no	`float` (default) or `base64` (little-endian float32; the OpenAI SDKs use and decode it automatically).

Each text is read in windows of 512 tokens (roughly 350 English words, fewer in Hindi). Longer texts are split automatically and you still get one vector per input, averaged over its parts. For long documents, search works better if you split them into paragraphs yourself and embed each one.

# Documents to store (input_type defaults to "document")
curl BASE_URL/embeddings \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "multilingual-e5-large-instruct",
       "input": ["School fees are paid online at the start of each quarter.",
                 "परीक्षा का परिणाम अगले सोमवार को वेबसाइट पर घोषित किया जाएगा।"]}'

# A user's search question
curl BASE_URL/embeddings \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "multilingual-e5-large-instruct", "input_type": "query",
       "input": "results kab aayenge?"}'

import numpy as np
from openai import OpenAI

client = OpenAI(base_url="BASE_URL", api_key="sk-llm-…")
MODEL = "multilingual-e5-large-instruct"

docs = ["School fees are paid online at the start of each quarter.",
        "परीक्षा का परिणाम अगले सोमवार को वेबसाइट पर घोषित किया जाएगा।"]
doc_vecs = np.array([d.embedding for d in client.embeddings.create(model=MODEL, input=docs).data])

q = client.embeddings.create(model=MODEL, input="results kab aayenge?",
                             extra_body={"input_type": "query"}).data[0].embedding
scores = doc_vecs @ np.array(q)          # cosine similarity (vectors are normalized)
print(docs[int(scores.argmax())])        # → the exam results sentence

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "BASE_URL", apiKey: process.env.LLM_API_KEY });
const model = "multilingual-e5-large-instruct";

const docs = ["School fees are paid online at the start of each quarter.",
              "परीक्षा का परिणाम अगले सोमवार को वेबसाइट पर घोषित किया जाएगा।"];
const { data } = await client.embeddings.create({ model, input: docs });

// Extra fields such as input_type are sent as-is.
const q = (await client.embeddings.create({ model, input: "results kab aayenge?", input_type: "query" })).data[0].embedding;
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
const best = data.map((d, i) => [dot(d.embedding, q), i]).sort((a, b) => b[0] - a[0])[0][1];
console.log(docs[best]); // → the exam results sentence

{
  "object": "list",
  "model": "multilingual-e5-large-instruct",
  "data": [
    {"object": "embedding", "index": 0, "embedding": [0.0213, -0.0087, 0.0441, …]},
    {"object": "embedding", "index": 1, "embedding": [-0.0129, 0.0352, 0.0016, …]}
  ],
  "usage": {"prompt_tokens": 41, "total_tokens": 41}
}

Speech to text

POST /v1/audio/transcriptions (multipart form) returns the transcript in the response. POST /v1/audio/translations takes the same fields and returns English.

Building with an AI assistant?Copy a complete speech-to-text brief for ChatGPT, Claude, Cursor or Copilot, then add what you are building.

Field	Required	Description
`file`	yes	Audio or video file (mp3, wav, m4a, ogg, mp4…). At most 100 MB and 60 minutes.
`model`	no	`whisper-large-v3` (`whisper-1` is accepted too).
`language`	no	ISO code such as `en` or `hi`. Defaults to `en`; set it for other languages.
`response_format`	no	`json` (default), `verbose_json` (with segments), `text`, `srt` or `vtt`.
`prompt`	no	Words to steer spelling, such as names or subject terms.
`temperature`	no	Number, default `0`.
`metadata`	no	Up to 8 KB of your own data (JSON or text), returned unchanged in the JSON response and in webhooks.
`async`	no	`true` to get HTTP 202 at once and the result by webhook. See Async transcriptions.

curl BASE_URL/audio/transcriptions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F model=whisper-large-v3 \
  -F language=en \
  -F file=@call.mp3

from openai import OpenAI

client = OpenAI(base_url="BASE_URL", api_key="sk-llm-…")

with open("call.mp3", "rb") as audio:
    result = client.audio.transcriptions.create(model="whisper-large-v3", file=audio, language="en")
print(result.text)

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "BASE_URL", apiKey: process.env.LLM_API_KEY });

const result = await client.audio.transcriptions.create({
  model: "whisper-large-v3",
  file: fs.createReadStream("call.mp3"),
  language: "en",
});
console.log(result.text);

Async transcriptions

For long calls or batch jobs, add async=true. The server replies 202 Accepted immediately, transcribes in the background and POSTs the result to your API key's webhook (Settings → Webhooks): whisper.completed, or whisper.failed with an error.

Your key needs an enabled whisper.completed webhook, otherwise the request gets HTTP 400 async_webhook_missing. Ask an admin to add one.
Pass your own IDs in metadata: they come back in the 202 reply and in the webhook.
Job status is available for 1 hour at GET /v1/audio/transcriptions/<job id> with the same key. It never contains the transcript.
At most 50 jobs (3 GB of audio) can wait at once; beyond that you get HTTP 429 async_queue_full.

curl BASE_URL/audio/transcriptions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -F model=whisper-large-v3 \
  -F async=true \
  -F 'metadata={"lead_id": 12345, "source": "call-centre"}' \
  -F file=@call.mp3

# Optional status check (same API key; kept 1 hour; never includes the transcript)
curl BASE_URL/audio/transcriptions/tr_3f9a1c2b7d4e \
  -H "Authorization: Bearer $LLM_API_KEY"

import json, requests  # pip install requests

with open("call.mp3", "rb") as audio:
    r = requests.post(
        "BASE_URL/audio/transcriptions",
        headers={"Authorization": "Bearer sk-llm-…"},
        data={"model": "whisper-large-v3", "async": "true",
              "metadata": json.dumps({"lead_id": 12345, "source": "call-centre"})},
        files={"file": audio},
    )
job = r.json()          # HTTP 202
print(job["id"], job["status"])  # tr_… queued

import fs from "node:fs";

const form = new FormData();
form.append("model", "whisper-large-v3");
form.append("async", "true");
form.append("metadata", JSON.stringify({ lead_id: 12345, source: "call-centre" }));
form.append("file", new Blob([fs.readFileSync("call.mp3")]), "call.mp3");

const res = await fetch("BASE_URL/audio/transcriptions", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.LLM_API_KEY}` },
  body: form,
});
const job = await res.json(); // HTTP 202
console.log(job.id, job.status); // tr_… queued

HTTP/1.1 202 Accepted
Location: /v1/audio/transcriptions/tr_3f9a1c2b7d4e

{
  "id": "tr_3f9a1c2b7d4e",
  "object": "audio.transcription.job",
  "status": "queued",
  "model": "whisper-large-v3",
  "endpoint": "/v1/audio/transcriptions",
  "created_at": "2026-09-17T09:12:03Z",
  "started_at": null,
  "finished_at": null,
  "error": null,
  "metadata": {"lead_id": 12345, "source": "call-centre"},
  "delivery": "The result is POSTed to the webhook(s) configured in the LLM Console …"
}

Webhooks

Each finished Whisper transcription (sync or async) is POSTed as JSON to the webhooks of the API key that made the request. You only ever receive your own key's transcripts. Delivery runs in the background, so API callers are never slowed down.

Event	Sent when
`whisper.completed`	A transcription finished. `response` holds the full result. Async failures are also sent here.
`whisper.failed`	An async transcription failed or was cancelled. The payload has `error` and `outcome` instead of `response`.

Reply with any 2xx within 15 seconds. Failed deliveries are retried after 10 s, 1 min and 5 min.
Redirects aren't followed. URLs pointing at this server or at loopback and link-local addresses are refused; other LAN hosts (such as an n8n box) are fine.
Use X-Webhook-Id to ignore duplicates if your endpoint was slow and a retry arrives.
An admin links each webhook to an API key. Signed in to the console, you can send a test event and rotate your signing secret.

Delivery

POST <your webhook URL>
Content-Type: application/json
User-Agent: LLM-Gateway-Webhooks/1.0
X-Webhook-Id: dlv_1ab903c9db9e3266
X-Webhook-Event: whisper.completed
X-Webhook-Timestamp: 1789636780
X-Webhook-Signature: sha256=<hex>

{
  "id": "dlv_1ab903c9db9e3266",
  "event": "whisper.completed",
  "created_at": "2026-09-17T09:12:44Z",
  "request_id": "3f9a1c2b7d4e",
  "job_id": "tr_3f9a1c2b7d4e",
  "async": true,
  "model": "whisper-large-v3",
  "endpoint": "/v1/audio/transcriptions",
  "status": 200,
  "response_format": "json",
  "received_at": "2026-09-17T09:12:03Z",
  "completed_at": "2026-09-17T09:12:44Z",
  "duration_s": 41.2,
  "client_ip": "192.168.0.44",
  "user_agent": "python-requests/2.32",
  "upload_bytes": 3840044,
  "api_key": {"id": "key_…", "name": "Call centre", "email": "ops@orchidsintl.edu.in"},
  "metadata": {"lead_id": 12345, "source": "call-centre"},
  "response": {"text": "Hello, I'm calling about admissions …"}
}

Verify signatures

X-Webhook-Signature is sha256= followed by the hex HMAC-SHA256 of "<X-Webhook-Timestamp>.<raw body>", keyed with the webhook's signing secret (shown once when the webhook is created or its secret is rotated). Verify against the raw bytes, before parsing the JSON.

import hashlib, hmac, time

def verify(secret: str, headers, raw_body: bytes) -> bool:
    ts = headers["X-Webhook-Timestamp"]
    if abs(time.time() - int(ts)) > 300:  # reject replays older than 5 minutes
        return False
    expected = "sha256=" + hmac.new(secret.encode(), ts.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, headers["X-Webhook-Signature"])

# Flask:  verify(SECRET, request.headers, request.get_data())

import crypto from "node:crypto";

function verify(secret, headers, rawBody /* Buffer */) {
  const ts = headers["x-webhook-timestamp"];
  if (Math.abs(Date.now() / 1000 - Number(ts)) > 300) return false; // reject replays
  const expected = "sha256=" + crypto.createHmac("sha256", secret).update(`${ts}.`).update(rawBody).digest("hex");
  const given = headers["x-webhook-signature"] || "";
  return given.length === expected.length && crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(given));
}

// Express: app.post("/hook", express.raw({ type: "application/json" }), (req, res) => { … verify(SECRET, req.headers, req.body) … })

Errors & limits

Errors use the OpenAI shape: {"error": {"message": …, "type": …, "code": …}}.

Status	Code	Meaning
400	`file`, `response_format`, `metadata_too_large`, `async_webhook_missing`	Bad upload, unsupported audio, audio over 60 minutes, metadata over 8 KB, async without a webhook, or an embeddings request with empty strings, over 512 inputs or over 65,000 tokens
401	`invalid_api_key`	Missing, wrong or revoked API key
404	`model`, `job_not_found`	Unknown model name, or an async job that doesn't exist, is older than 1 hour or belongs to another key
413	–	Upload larger than 100 MB
429	– , `async_queue_full`	Rate limit reached, or too many async jobs waiting. Retry after a short pause
499	`request_cancelled`	An administrator cancelled the request. Don't retry automatically
502	`upstream_error`	The model server is restarting or unavailable. Retry with backoff

LimitsChat, speech-to-text and embeddings each have their own budget, so a busy one never blocks the others: 120 requests per minute per client IP (bursts of 40) and 16 open connections per IP, per service. 100 MB per upload, 60 minutes of audio per file, 16,384 tokens of chat context, 16 images per chat request, 1.5 MB per base64 image, 512 texts and about 65,000 tokens per embeddings request.

Your usage

Server

Trends

GPU utilization

GPU memory by service

Requests running / waiting

Finished requests per minute

Qwen token throughput

Qwen latency

KV cache usage

GPU temperature & power

Trends

CPU loadpeak % of all server threads

Whisper CPU cores in usepeak and average · dashed: thread budget

Whisper speed× realtime while transcribing

Whisper memorydashed line: its limit

New analysis

Analyses

Summary by Qwen (built from these results only, the video is never sent)

Flagged moments

In flight

Finished requests

Usage by API key

API requests over time by status

Requests by endpointrequests

Response time by endpointp50 · p95, successful

Top clientsrequests

Rejected API keys by client401s

HTTP status codesrequests

Client softwarerequests

Requests by modelrequests

New API key

New webhook

Recent deliveries last 100

New console login