LABEL CHECK / DOSSIER

LiteLLM

https://github.com/BerriAI/litellm·verified at v1.90.5 (2026-07-17)·last verified 2026-07-17·volatile (re-verified per release (≤2 weeks))

13claims sampled

3as labeled

8partial

2not as labeled

0not implemented

11open deltas

4verified closed

0closed partial

closure trend: format-translation and release-shippable fixes close within weeks; concurrency, budget-enforcement, and fallback deltas are not closing — fix PRs sit unmerged or were closed while the backing issues were stale-bot closed unfixed

Verdict

The unified-API and provider-breadth claims hold as labeled, but the production-control claims — budgets, rate limits, fallback, and the README's 8ms-P95-at-1k-RPS figure — are contradicted by issue-traced behavior under exactly the concurrent multi-tenant conditions the proxy is marketed for, with the key fixes unmerged and the backing issues stale-bot closed unfixed.

Claim ledger — 13 rows

#statusclaimedobservedevidenceverified

1as labeledCall 100+ LLMs using the OpenAI Input/Output Format (docs.litellm.ai landing page, captured 2026-06-11)Full detail →Source-reviewed: the corpus confirms a unified OpenAI-compatible API surface across 100+ providers, and records LiteLLM as the dominant open-source abstraction layer in the gateway category. Full detail →[1]—

2as labeledDrop-in OpenAI compatibility — swap providers without rewriting your code (BerriAI/litellm README, captured 2026-06-11)Full detail →Source-reviewed: the input/output translation layer is the product's core and works as labeled for mainstream chat-completion use; the corpus's adoption evidence (Stripe, Google, Netflix recorded as adopters) rests on… Full detail →[1]—

3partialRouter with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) (docs.litellm.ai landing page and README, captured 2026-06-11)Full detail →Source-reviewed: fallback fires as documented for RateLimitError and AuthenticationError, but NotFoundError (404) unconditionally bypasses the entire fallback chain (router.py L5285-86; #21377 stale-bot closed… Full detail →[1][2][3][4][5][6]2026-07-18

4not as labeledTrack spend & set budgets per project (docs.litellm.ai landing page; 'Budgets, Rate Limits' docs page offers personal, team, team-member, and key-level budgets; captured 2026-06-11)Full detail →Source-reviewed: a $50 configured budget produced $764.78 actual spend via the AzureOpenAI client — a 15x overshoot with no alert (#12977, closed not-planned 2025-10-31, no fix); user budgets are skipped entirely for… Full detail →[1][2][3][4][5][6][7][8][9]2026-07-18

5partialSet tpm limits (tokens per minute), rpm limits (requests per minute), and max parallel requests per key, user, team, or model ('Budgets, Rate Limits' docs page, captured 2026-06-11)Full detail →Source-reviewed: 5 concurrent requests against a 100 TPM limit consumed 663 tokens — 6.6x the limit — because tokens are deducted after completion, not reserved upfront (#18730, open and reopened as of 2026-06-11; the… Full detail →[1][2][3][4][5]2026-07-18

6partialGuardrails out of the box; per-project customization (logging, guardrails, caching) (README 'Production-ready gateway' bullet and docs.litellm.ai comparison table, captured 2026-06-11)Full detail →Source-reviewed: guardrails exist and run in some configurations, but multiple configuration paths had documented non-enforcement windows: GUI/API-defined guardrails never invoked (#15584, vendor-closed completed… Full detail →[1][2][3][4][5][6]—

7partialPer-project caching (docs.litellm.ai comparison table, captured 2026-06-11)Full detail →Source-reviewed: caching works but its savings are unverifiable from the gateway's own telemetry: the cache_hit metric reports 0 even when the upstream provider returns cached_tokens (#6229, closed not-planned); Azure… Full detail →[1][2][3][4]—

8not as labeled8ms P95 latency at 1k RPS (BerriAI/litellm README, captured 2026-06-11)Full detail →Source-reviewed corpus evidence contradicts the number: a production post-mortem measured P99 latency of 18 seconds at 310 RPS and OOM at 350 RPS sustained over 2 hours — a ~300 RPS Python GIL ceiling that horizontal… Full detail →[1]—

9partialCentralized API gateway with authentication & authorization; virtual keys for secure access control (docs.litellm.ai comparison table, captured 2026-06-11)Full detail →Source-reviewed: the auth layer exists and functions, but its 2026 security record materially qualifies 'secure': 7 GitHub security advisories published 2026-04-03 through 2026-05-07, including a Critical OIDC… Full detail →[1][2][3][4][5][6]2026-07-18

10partialConsistent output - same response format regardless of which provider you use (docs.litellm.ai landing page, captured 2026-06-11)Full detail →Source-reviewed: holds for mainstream chat completions; documented divergences at the edges: non-compliant tool-call IDs are not normalized, breaking cross-provider multi-turn tool use (#22317); the thinking.summary… Full detail →[1][2][3]—

11partialException handling with OpenAI-compatible errors (docs.litellm.ai comparison table and README, captured 2026-06-11)Full detail →Source-reviewed: the exception mapping exists, but asyncio.CancelledError bypasses all three of LiteLLM's exception-handling layers — a DNS cancellation yields a bare 500 with no retry, no fallback, and no log entry… Full detail →[1][2][3][4]—

12partialMulti-tenant cost tracking and spend management per project/user (docs.litellm.ai comparison table, captured 2026-06-11)Full detail →Source-reviewed: cost tracking works in the common path; Azure cost tracking fails on model-name mismatch — Azure returns 'gpt-4' while the deployment is configured as 'azure/gpt-4-1106-preview' and cost calculation… Full detail →[1][2][3][4]—

13as labeledLoad balancing out of the box; application-level load balancing (README 'Production-ready gateway' bullet and docs.litellm.ai comparison table, captured 2026-06-11)Full detail →Source-reviewed: router load balancing across deployments is confirmed functional in the corpus; its one documented adverse interaction — scattering requests defeats provider-side cache warmup — was addressed by a… Full detail →[1]2026-07-18

Delta ledger — 11 open

LL-01opened 2026-03-01

In-memory budget/rate counters lose 75-93% of increments under concurrent load; atomic-increment fix unmerged

closes when:Atomic increment fix merged and a concurrent increment test (4+ threads, 50 increments each) yields the expected counter value on a current release, TD re-verified

evidence:[1][2]Issue stale-bot closed not-planned 2026-05-20 with PR #20979 still open unmerged — re-checked 2026-06-11Re-verified open 2026-07-17 against v1.90.5 — atomic-increment fix PR #20979 still open unmerged

LL-02opened 2026-03-01

Concurrent requests bypass TPM limits 6.6x — tokens deducted after completion, not reserved; reservation fix closed unmerged

closes when:Concurrent requests against a TPM limit cannot collectively exceed it (pre-call token reservation or equivalent), TD re-verified on a current release

evidence:[1][2]Issue open and reopened; PR #23775 closed without merge — re-checked 2026-06-11UPDATE 2026-06-28: pre-call token reservation merged in v1.85.0 (PR #27001); #18730 still open and reporter-unconfirmed; v1.90.0 added an opt-out (PR #30211) — falsification criterion partially met, awaiting TD execution[5]Re-verified open 2026-07-17 against v1.90.5 — no change since 2026-06-28; #18730 still open/reopened, opt-out (PR #30211) unchanged

LL-03opened 2026-03-01

Budget enforcement bypassed via AzureOpenAI client (15x overshoot) and team virtual keys; AWS Bedrock pass-through budget bypass fixed (other pass-through routes remain untracked)

closes when:A configured budget binds across the AzureOpenAI-client and team virtual-key vectors, and across non-Bedrock pass-through routes, in a TD re-test on a current release (AWS Bedrock pass-through narrowed out of scope 2026-07-17: budget tracking there confirmed fixed via merged PR #12847)

evidence:[1][2][3]SCOPE NARROWED 2026-07-17: AWS Bedrock pass-through budget bypass (#12789) fixed via merged PR #12847 (merged 2025-07-22) — the pass-through vector is no longer blanket-broken; general pass-through routes (#10750, e.g. Gemini) remain closed not-planned, unfixed[5][6]Team virtual-key budget bypass (#12905): fix merged 2026-07-14 via PR #32005 (breaking change — enforces the key owner's personal max_budget on team keys by default, with a skip_user_budget_on_team_key opt-out) — NOT yet in a released version as of 2026-07-17; source inspection of litellm/proxy/auth/auth_checks.py confirms the pre-fix bypass guard `(team_object is None or team_object.team_id is None)` is still present in both v1.90.5 (latest stable) and v1.93.0-rc.2 (latest prerelease)[8]AzureOpenAI-client budget bypass (#12977): no fix PR found — still closed not-planned, unfixedRe-verified open 2026-07-17 against v1.90.5 — falsification criterion not met overall (AzureOpenAI-client and team-key vectors remain open on any released version)

LL-04opened 2026-03-01

NotFoundError (404) unconditionally bypasses the fallback chain; fix PR open unmerged since Feb 2026

closes when:A 404/NotFoundError from one deployment triggers failover to an available alternative deployment, TD re-verified on a current release

evidence:[1][2]Issue stale-bot closed not-planned 2026-06-01; PR #21378 still open — re-checked 2026-06-11Fix attempt PR #21378 closed unmerged 2026-07-17 (open since Feb 2026); automated review (Greptile) found the fix incomplete — the new NotFoundError retry branch is unreachable because an earlier status-code guard (router.py L5280-5283) already raises the 404 before reaching itRe-verified open 2026-07-17 against v1.90.5 — no working fix exists; #21377 remains stale-closed not-planned

LL-05opened 2026-03-01

Mid-stream fallback injects a default English system prompt over the caller's persona; no disable option (closed not-planned)

closes when:Mid-stream fallback continuation preserves the caller's system prompt, or a configuration option disables the injected prompt; TD re-verified

evidence:[1]Re-verified open 2026-06-28 against v1.90.0Re-verified open 2026-07-17 against v1.90.5 — no new fix PR found for the disable-injected-prompt request; an unrelated prefill-block streaming bug (#27967, opened 2026-05-14) touches the same fallback code path but does not address this claim

LL-06opened 2026-03-01

Multimodal fallback OpenAI→Gemini drops image data via in-place message mutation (closed not-planned)

closes when:OpenAI-to-Gemini fallback delivers image content (non-zero image tokens in usage metrics), TD re-verified

evidence:[1]Re-verified open 2026-06-28 against v1.90.0Re-verified open 2026-07-17 against v1.90.5 — no new fix PR found; #15803 remains closed not-planned

LL-07opened 2026-03-01

README's 8ms P95 at 1k RPS contradicted by measured ~300 RPS GIL ceiling (P99 18s at 310 RPS; OOM at 350 RPS sustained)

closes when:An independent load test sustains 1k RPS with ~8ms P95 on a vendor-documented configuration, or re-measurement fails to reproduce the ~300 RPS ceiling; either result TD-verified

evidence:[1]Contradicting post-mortem article removed (HTTP 404 since 2026-04-22); measurement retained at secondary-research backingRe-verified open 2026-06-28 against v1.90.0Re-verified open 2026-07-17 — no new independent load-test data found in either direction; the contradicting post-mortem source remains unreachable (HTTP 404)

LL-08opened 2026-03-01

Gateway cache metrics unreliable — cache_hit reports 0 while upstream bills cached_tokens; savings unverifiable from gateway telemetry

closes when:Gateway cache_hit metrics match upstream provider cached-token billing in a TD re-test

evidence:[1]Re-verified open 2026-06-28 against v1.90.0NOTE 2026-07-17 (per #2535): mitigation shipped — PR #27660 merged 2026-06-01 added independent Prometheus provider-cache-token metrics (litellm_provider_cache_read_input_tokens_metric / litellm_provider_cache_creation_input_tokens_metric) as a workaround path; the underlying cache_hit-flag inaccuracy (#6229) was not itself re-confirmed or fixed this pass — #6229 remains closed not-planned with no code change[4]Re-verified open 2026-07-17 against v1.90.5

LL-09opened 2026-05-17

Upstream Retry-After header not forwarded under the standard header; clients retry immediately, amplifying provider rate-limit pressure

closes when:Upstream Retry-After is forwarded to clients under the standard header and respected by router backoff, TD re-verified

evidence:[1][2]Issue stale-bot closed not-planned 2026-05-29; fix PR #21648 closed unmerged — re-checked 2026-06-11UPDATE 2026-06-28: fix PR #21648 auto-closed without merge 2026-06-03; v1.90.0 PR #27687 enriches rate-limit error fields but does not forward Retry-After — no fix pathRe-verified open 2026-06-28 against v1.90.0Re-verified open 2026-07-17 against v1.90.5 — no activity on #21553 since the 2026-05-21 stale-bot mark; no new fix PR opened; fix PR #21648 remains closed unmerged (auto-closed 2026-06-03, confirmed dead)

LL-10opened 2026-06-18

Guardrail fail-open cluster — 8 implementations silently fail open (Lakera category_thresholds disables the blocked verdict, llm_as_a_judge defaults overall_score to 100, OpenAI/Azure moderation scans only the trailing turn, Presidio fails open on analyzer error)

closes when:Each guardrail blocks the policy-violating input/output it is configured for under analyzer-error and threshold edge cases, TD re-verified on a current release

evidence:[1][2]Two partial fix efforts in progress, both unmerged as of 2026-07-17: a contributor offered 2026-07-04 to fix the Lakera category_thresholds override (#30727) but no PR has landed yet; PR #31406 (opened 2026-06-26, unmerged) fixes only the tool-blocklist case/whitespace-normalization half of #30732, explicitly leaving the /v1/responses and /v1/messages route-coverage gap for a separate PR[4]Re-verified open 2026-07-17 against v1.90.5 — no fixes merged; cluster remains open

LL-11opened 2026-06-25

SSE error codes outside the 100-599 range silently discarded — router never fails over or cools down for ZhipuAI/DashScope upstream errors (#31284)

closes when:A non-standard upstream SSE error code triggers router fallback/cooldown rather than being silently dropped, TD re-verified

evidence:[1]Two competing fix PRs open unmerged as of 2026-07-17: #31312 (opened 2026-06-25) and #32109 (opened 2026-07-04, CI fully green — lint, code-quality, all unit-test groups, Semgrep, OSV, Codecov — pinged for maintainer review 2026-07-14 after 10 days with no pickup)[3][4]Re-verified open 2026-07-17 against v1.90.5

Closed deltas — 4 (falsification-gated)

LL-C1closed verifiedre-verified 2026-03-28

Deferred-HTTP providers (Vertex AI, Bedrock, Predibase, Codestral) bypassed streaming fallback entirely

closed by:PR #22375, merged 2026-02-28 — sync streaming fallback plus 429 handling for all streaming paths

LL-C2closed verifiedre-verified 2026-05-17

thinking.summary silently dropped routing Anthropic requests to the OpenAI Responses API

closed by:PRs #21441/#21491 (merged 2026-03-05), shipped in v1.82.3-stable (2026-03-17); TD re-verified present through v1.83.14-stable.patch.3

LL-C3closed verifiedre-verified 2026-05-17

Reported Bedrock guardrail bypass (original input returned instead of blocked output)

closed by:TD re-verification of #22949: root cause was reporter configuration (full guardrail ARN corrupting the URL path instead of the short ID) — not a LiteLLM code defect; the finding's row is corrected, not vendor-fixed

LL-C4closed verifiedre-verified 2026-04-10

API keys logged in plaintext in proxy log output (pre-v1.82.4)

closed by:v1.82.4 — plaintext key logging suppressed; not backported, mitigation on older versions is upgrade

Methodology

sampling: Theory Delta Method §2 material-claim sampling rule (theorydelta-policy strategy/method.md): a claim is sampled iff it appears on the vendor's own label and would plausibly influence an adoption decision. Label surfaces enumerated and captured 2026-06-11: the docs.litellm.ai landing page, the BerriAI/litellm README, and the 'Budgets, Rate Limits' docs page (docs.litellm.ai/docs/proxy/users). 13 material claims extracted; all sampled (full dossier, 10-claim floor met).
verification depth: source-reviewed — not executed by Theory Delta
verification statement: Evidence is drawn from LiteLLM's public issue tracker, pull requests, release notes, GitHub security advisories, the vendor's own security disclosure, and the Theory Delta corpus blocks backing them; the open/closed state of every cited issue and fix PR was re-checked against the GitHub API on 2026-06-11, again on 2026-06-28 (v1.90.0), and again on 2026-07-17 (v1.90.5). Theory Delta has not executed LiteLLM in its own environment for this dossier. Star counts are excluded from assessment as unreliable quantitative metrics.
backing confidence: medium
strongest case against: Many backing issues were stale-bot closed ('not planned') rather than triaged, so some behaviors may have been fixed incidentally in LiteLLM's roughly weekly releases since the finding's last full verification (2026-03-22) without an issue or PR recording it; the ~300 RPS ceiling contradicting the 8ms-P95 claim rests on a production post-mortem whose source article has been removed (HTTP 404 since 2026-04-22) and has not been re-measured; several guardrail issues were vendor-closed as completed and TD has not re-verified those fixes; and release velocity (v1.88.1 shipped 2026-06-09) can outpace any source-review window.