LLM gateway on LiteLLM: failures the health probe misses

In brief

• The gateway: LiteLLM is the single access point for AI agents, developer assistants and data pipelines to models from several cloud providers and to a local model on a GPU server. A caller names a model by its group name; the gateway router picks a deployment, retries on errors and keeps track of quotas and spend.
• Router failures: in one day 39% of requests were rejected, while the end-to-end probe stayed green because it called a different model. There turned out to be two causes, and both returned the same answer, "No deployments available for selected model": an hour-long cooldown of a single-deployment group after provider 500 errors (the LiteLLM team fixed this very case on 27 Oct 2026), and a daily token cap set in the gateway itself. An alert on the share of failed requests, created during the investigation, caught this outage at once: per the gateway log, 92% of requests over that half hour ended in rejection. After the router settings fix there was no hour-long cooldown again: over 30 days (5 Oct – 4 Nov 2026) provider 5xx errors and timeouts added up to 3.9% of the number of requests, while 0.34% of requests reached consumers as failures.
• A new model behind an old name: the provider gave the model name to the next version of the model. There were no errors, yet the share of items the scoring model passed on to a human dropped in one of the feeds from 10–11% to 5.9% on the day of the change and then to about 3%. Paired rescoring of 668 items showed that the order of scores stayed close to the former one (rank correlation 0.81–0.96) while the scale dropped by 0.8–1.6 points. Thresholds were recalculated, and the share of items passing the threshold is now watched.
• Spend: the gateway priced calls at rates the provider had already changed and recorded 61% of the cost of calls at the official price table. The price is now a blend weighted by the gateway's share of peak traffic: summed over 28 days it matches the cost at the official table, and a test keeps it within the official bounds.
• Method: alerts are set on request outcomes, not on resource headroom; a fix is verified on the next provider failures, not by a green response right after a restart; router settings, prices and the routes of the private/* groups are pinned by configuration tests.
• The same failures across the industry: on 27 Oct 2026 the LiteLLM team merged a fix for exactly this case (as of 4 Nov 2026 only in 1.109 pre-releases): when allowed_fails is set on the router, a single-deployment group with no fallback answers "No deployments available" for the whole cooldown. Anthropic wrote in its postmortem of 17 Sep 2025 that its own evaluations missed the degradation users were reporting. Yandex and DeepSeek record in their changelogs how a latest branch or an old model name ends up serving a new model. Links are in the sections "Risks and requirements" and "Checking against industry practice".

Risks and requirements

The case is addressed to CTOs and heads of platform and AI teams at organizations whose AI agents and pipelines reach several models through one LLM gateway, also called an AI gateway. Alfa-Bank's gateway is built this way: according to the speakers at Selectel's MLechny Put 2026 conference, the models deployed in the bank's cluster are connected to a single gateway on LiteLLM, and various teams and services go through it.

The gateway becomes a shared point of failure for every agent, so its behaviour on failure has to be decided in advance: for each check and each route, decide whether a request goes through when it fails (fail-open) or is rejected (fail-closed).

The hard part is that such a gateway almost never goes down as a whole. Ilya Zakharov, head of the monitoring department at Astra Group, put it this way in a TAdviser podcast about monitoring AI services in general: a 200 response used to mean the service works, and now a model can answer quickly and still be wrong. Researchers from Stanford and UC Berkeley showed back in 2023 that the behaviour of a model under the same name changes noticeably within three months: GPT-4 told prime numbers from composite ones correctly in 84% of cases in March 2023 and in 51% in June (Chen, Zaharia, Zou). In its postmortem of three recent issues of 17 Sep 2025, Anthropic wrote that its evaluations did not capture the degradation users were reporting, even though at the worst hour a routing error affected 16% of Sonnet 4 requests; the causes were infrastructure bugs, and the model name did not change. Some providers record such switches in their own changelogs: in the Yandex AI Studio release notes, different YandexGPT generations stood behind the latest branch at different times, YandexGPT Lite based on YandexGPT 3 from 24 Jun 2024 and the 4th-generation model from 2 Dec 2024; in the DeepSeek change log, the name deepseek-v4-flash was temporarily routed to the new model after the release of V4.1 Flash.

Since 1 March 2026, FSTEC Order No. 117 of 11 April 2025 has been in force in Russia; FSTEC of Russia is the Federal Service for Technical and Export Control. The order approves the Requirements for the protection of information contained in state information systems and in other information systems of state bodies, state unitary enterprises and state institutions. Clause 60 of the Requirements does not allow restricted-access information held in these systems to be passed to the developer of an artificial intelligence model. For commercial systems this is a reference point rather than a binding rule. In a gateway it is a question of routing: if a group has a fallback to a cloud model, the text goes to the provider when the local model fails. So the private/* groups are fail-closed: they have no fallbacks or aliases, their deployments are defined by private addresses or names in the organisation's domain, and a configuration test checks this; when the local model fails, the request is rejected rather than sent outside. In a gateway the perimeter is set by the group name, not by the model: calls by general names, even to the same local model, are outside the perimeter and can go to the cloud through a fallback on failure.

The task

An LLM gateway, also called an AI gateway, sits between applications and model providers: one API in front of many models, with routing, retries and fallbacks, keys and quotas, and spend tracking. LiteLLM is an open-source project: a Python library and a proxy server, LiteLLM Proxy, that accepts requests in the OpenAI API format and passes them to models from different providers. As an AI gateway it routes requests, issues API keys with limits and budgets, and tracks spend. Here AI agents and developer assistants, document-parsing pipelines and a pipeline that scores incoming material all work through such a gateway. Behind it are cloud models from several providers, some of them with daily quotas, and a local model on a GPU server. A caller names a model by its group name. The gateway router balances load across the deployments of a group, retries the request on errors and temporarily excludes a deployment that fails too often (cooldown); a limiter counts quotas, and the gateway records the cost of every call.

Such a gateway almost never goes down as a whole. Failure arrives differently: the router rejects requests while the provider is reachable; the provider gives a model name to the next version of the model; the gateway prices calls at a rate the provider has already changed. In all three cases the liveness check and an end-to-end probe through one model stay green. The task: find such failures and make sure the next one of the same kind shows up in the numbers, not by chance.

How the gateway works

Figure 1. The gateway, two outcome signals and the price check. A two-way arrow means a request and a response.

Agents and pipelines call the gateway through an OpenAI-compatible API and name a model by its group name. A group can hold one deployment or several, at different providers. The router picks a deployment, retries the request on errors and, if a deployment fails more often than allowed, removes it from selection for the cooldown period. Daily quotas are counted by a limiter in Redis, and the gateway writes the cost of every call to PostgreSQL using the prices in the gateway config. The private/* groups are defined only by private addresses and names in the organisation's domain and have no fallbacks.

Gateway failures show up in two signals that measure outcomes rather than liveness, and in a price check:

• Share of rejected requests. The gateway exports LiteLLM metrics to Prometheus, including counters of all requests and of failed responses (these include both router rejections and errors caused by the callers themselves); a Grafana rule raises an alert when more than a tenth of requests over an hour failed and this holds for 15 minutes. The share is computed for the gateway as a whole, so this rule will not catch a complete outage of a group with a small share of traffic; the model and error class labels show who exactly is hitting the wall.
• Share passing the scorer threshold. Every morning the scoring pipeline compares the share of items that passed the threshold with the 14-day median. If the threshold has not changed and the share has halved against the median, that is a reason for paired rescoring: both a shift in the model's scale and a change in the incoming flow move the share this way.
• Prices against the provider table. A configuration test checks the price of every model name against the provider's official table as of 23 Oct 2026: the price must lie between the off-peak and the peak price. The test will not notice the next tariff change by itself: the table in the test and the price blend are updated from the provider's change log.

- uid: llm-gateway-failure-share
  condition: C
  data:
  - refId: A
    model:
      # share of failed requests over an hour; while failures stay below
      # 0.003 per second (about one in 5 minutes), the numerator is empty
      expr: >
        (sum(rate(litellm_proxy_failed_requests_metric_total[1h])) > 0.003)
        / clamp_min(sum(rate(litellm_proxy_total_requests_metric_total[1h])), 0.001)
  - refId: C
    model:
      type: threshold
      expression: B   # B: the last value of A
      conditions:
      - evaluator: { type: gt, params: [0.1] }
  for: 15m
  noDataState: OK   # an empty numerator means "almost no failures"
  labels:
    severity: warning

Checking against industry practice

Each failure class was checked against what LiteLLM's developers, model providers and operations practitioners write about it publicly: the monitoring chapter of the Google SRE book, the LiteLLM documentation on health checks and on a model cooled down for everyone after one user's error, PR #48644, PR #41960, PR #40954 and issue #49089 in the LiteLLM repository, Andrey Krisanov's write-up on the cost of health checks, the DeepSeek pricing page, issue #47788 in the OpenCode repository, the FinOps Foundation paper on token spend, the paper by Chen, Zaharia, Zou, the Anthropic postmortem and the change logs of Yandex AI Studio and DeepSeek. The pages were opened on 4 Nov 2026, and quotes and numbers were checked against the pages themselves; where behaviour depends on version or settings, the table says so.

Method

Gateway failures leave no errors where people usually look for them, so each class was taken apart by measurement rather than by error messages or green checks. The same principle underlies the alerting audit: a rule that can never fire also looks healthy.

• Outcome, not headroom. First we counted how many gateway requests ended in a rejection: in one day of the gateway log, 39% of responses to generation requests were 429 rejections. The alert "pool 90% full" was at that time pointing at the daily pool of another model and said nothing about rejections: it measures resource headroom, not request outcomes.
• A fix is verified on the next failures. In this installation the cooldown list lives in process memory, and a container restart clears it; with Redis behind the router, according to the LiteLLM docs, a restart can instead reload the old entry. Either way, a successful response right after a restart proves nothing. The verification window is counted from the container start time, and confirmation comes from the hours when the provider again answers with errors while consumers get no failures. Provider errors show in the LiteLLM metric on deployment responses, consumer failures in the metric on failed responses, excluding errors caused by the callers themselves (400, 403).
• Paired rescoring of the scorer. 668 items scored by the former model between 26 Aug and 10 Sep were sampled evenly across the buckets of the former score and rescored by the new model with the same prompts and the same input assembly, without writing to the database. For each of five feeds we computed the shift in the mean, Spearman's rank correlation, and the precision and recall of the send decision at the new threshold against the former decision. The second way: pick the threshold at which the pass share matches the former one.
• Price check. The provider's official table, with off-peak and peak prices, and the share of peak calls in the cost of the gateway's traffic over 28 days give the blend that the gateway records. After the restart, the cost recorded on live calls was compared with the calculation at the new prices.
• A parameter is verified by a call. Whether the model's reasoning was switched off was checked by the usage.completion_tokens_details.reasoning_tokens field of a live response, not by the presence of the parameter in the code.

The 83 minutes after the fix of 13 Aug are one observation window; behaviour through later provider failures was measured over the 30 days Prometheus keeps (5 Oct – 4 Nov 2026), and in that period the cooldown is already 10 seconds. The scale shift was measured on one scorer across five feeds and is given as a range across feeds. The sample is uniform across the buckets of the former score, and on such a spread rank correlation is usually higher than on the natural flow of items. The price was checked against the provider's official table and recalculated from the gateway's traffic; the provider invoice and the gateway's spend records were not reconciled line by line. The blend is exact in total for the gateway; for a single key with a different share of peak traffic the recorded cost deviates, but the price stays between the off-peak and the peak price. The case of the reasoning parameter rests on one live call; the share of reasoning tokens after the fix comes from the gateway log as of 23 Oct 2026. The rejections over the day of 12–13 Aug are not split by cause: they include both the cooldown and the daily token cap.

Technical decisions

Cooldown does not protect quotas. By design LiteLLM does not cool down a single-deployment group: the router code says outright that cooldowns on such groups should be avoided by default. But when allowed_fails is set in the router settings, this exemption did not apply until the LiteLLM team's fix of 27 Oct 2026, and a single-deployment group with no fallback answered "No deployments available" for the whole cooldown. Here the pair allowed_fails: 1 and cooldown_time: 3600 was kept so that a model with a daily quota would not get a rejection every minute until midnight once the quota was spent. But quotas are guarded by the limiter, while cooldown reacts to errors, and that is a different mechanism: one-off provider 500 errors took the model out for an hour, with nothing in the group to replace it. At night the failure changed its cause for a while: the daily token cap set for this model in the gateway itself ran out, and the router stopped selecting the deployment without any cooldown at all. The error text in the gateway response was the same in both cases, "No deployments available for selected model", so the cause was determined each time from the cooldown list and the pool counter, not from the response. In the morning of 13 Aug the cap was removed, and the cooldown returned: under load the provider returned bursts of 500 errors while direct requests to it went through, and the gateway rejected 99.6% of requests over 15 minutes. The fix on 13 Aug in the router settings: allowed_fails 1 → 3, cooldown_time 3600 → 60 seconds. At the time the gateway config was assembled by a generator that rewrote the router settings as a whole, so until 28 Aug the hour-long cooldown came back with every rebuild; on 28 Aug the value was corrected in the generator itself. Today the cooldown is 10 seconds, error thresholds are set both overall and per error class, and the gateway retries a 429 after a one-second pause. These values are pinned by a configuration test: it fails if the cooldown is longer than 15 seconds, if fewer than three errors are allowed before a cooldown, or if a 429 is not retried.

A probe spends quota too. The end-to-end probe and the canary check called the aggregate groups, and the router sent these calls to deployments with a daily quota among others, so the probes themselves used up the quota. Deployments with a daily quota were taken out of the aggregates, and later the aggregate groups were removed altogether: each model is called by its name. Background health checks are switched off for metered deployments, and a configuration test checks this. A synthetic probe of a metered resource becomes a cost item of its own, and for LiteLLM this is a known story. In "Hidden Costs of LiteLLM's Health Checks", Andrey Krisanov points out that a LiteLLM model health check makes a real request to the model, and advises keeping background checks for local models while excluding paid external ones. In issue #49089, a LiteLLM user reported that after upgrading to 1.106.3 background checks started sending paid requests to image-generation models, which ultimately got the upstream billing account suspended.

The scorer's scale is watched through a share, not a control item. After the model change, a control item with a score of 9 got 8 against a threshold of 7 and stayed green: it catches a broken scorer, not a scale shift of one point. So the signal became the share of items passing an unchanged threshold, against the 14-day median.

Prices from the gateway's traffic. Since 16 Aug the provider charges twice as much during weekday peak hours as at other times. The DeepSeek prices in the gateway config had been set by hand under the former tariff, and a hand-set price overrides LiteLLM's built-in price table and remains one number for every hour of the day; so the recorded spend changed neither when the provider changed its tariff nor when LiteLLM fixed its built-in table. The built-in table has told peak from off-peak since PR #41960, but until 19 Sep it counted off-peak calls at the peak price itself: an explicit price in the config does not depend on when the table gets fixed, but it has to be checked against the provider's table, and the test does that. Now the config holds a blend weighted by the gateway's share of peak traffic, and a test keeps the price of every model name between the off-peak and the peak price and makes sure that different names of one model carry one price.

# Official prices per 1M tokens: (off-peak, peak)
DEEPSEEK_PRICES_PER_1M = {
    "deepseek/deepseek-flash": {
        "cache_read_input_token_cost": (0.003, 0.006),
        "input_cost_per_token": (0.15, 0.30),
        "output_cost_per_token": (0.60, 1.20),
    },
}
 
def test_deepseek_prices_match_official_table(config=None):
    cfg = config or load_config()
    seen = {}
    for m in cfg["model_list"]:
        lp = m.get("litellm_params", {})
        model = lp.get("model")
        if model not in DEEPSEEK_PRICES_PER_1M:
            continue
        for field, (low, high) in DEEPSEEK_PRICES_PER_1M[model].items():
            per_1m = float(lp[field]) * 1e6
            assert low <= per_1m <= high, f"{m['model_name']}: {field} = {per_1m:.4f}"
            seen.setdefault((model, field), set()).add(float(lp[field]))
    # every name of one model carries one price
    for (model, field), values in seen.items():
        assert len(values) == 1, f"{model}: different {field}: {sorted(values)}"

A parameter is verified by the model's response. DeepSeek V4 models reason by default. Backends behind an OpenAI-compatible API have different extensions: chat_template_kwargs passes chat template parameters to servers such as vLLM and SGLang, while DeepSeek has its own parameter for switching reasoning off. A request with chat_template_kwargs: {"thinking": false} went through the gateway without an error, but it did not switch reasoning off: one live call produced 146 reasoning tokens. Reasoning consumes the same max_tokens budget, so some responses came back empty or cut off. With DeepSeek's native parameter thinking: {"type": "disabled"} the same call had no reasoning, and the answer was five times shorter. As of 23 Oct 2026, consumers that switch reasoning off show 0–0.3% reasoning tokens in the gateway log. The same class is described for Z.AI models in LiteLLM PR #40954: the flag switching reasoning off never reached the model, and every request paid for reasoning tokens.

Results

Until 28 Aug the config generator kept bringing the hour-long cooldown back, and the same class was visible from the side of the end-to-end probe, which called an aggregate group of four deployments. Every long failure window in its measurements for 19–27 Aug, from 30 to 85 minutes, consisted of 429 responses caused by cooldown: all four deployments of the group were in it at once, and the window length was set by the cooldown, not by the duration of the provider's failure.

Since 28 Aug there has been no hour-long cooldown in the gateway log: the last rejection with "Try again in 3600 seconds" was logged on 25 Aug. How the fix holds up through later provider failures shows in LiteLLM metrics for 30 days (5 Oct – 4 Nov 2026). Provider 5xx errors and timeouts added up to 3.9% of the number of requests, while 0.34% of requests reached consumers as failures. In 163 of 717 hours provider errors amounted to at least a tenth of the number of requests, and in 158 of them (97%) not a single consumer request failed: the retry went through. The share of failed consumer requests rose above 10% within an hour in five hours. Two of them fell on 3 Nov, when the provider answered 503 in a row and there was nothing to retry against. The other three (15 Oct and 31 Oct) came from request limits set in the gateway itself on a deployment and on a key: such rejections are recognisable in the log by their text, and the alert on the share of failed requests shows them just like provider failures.

Conclusions

1. An LLM gateway fails not by going down but by answering: with a router rejection while the provider is reachable, a different model behind an old name, spend at an outdated price. The liveness check and a probe through one model stay green in these cases, so alerts are set on outcomes, on the share of rejected requests and the pass share, and prices in the config are checked against the provider table by a test.
2. A setting that looks like quota protection can itself cause failures: the hour-long cooldown on a single-deployment group, together with the daily token cap, produced 39% rejections in a day, and after the cap was removed, 99.6% of requests were rejected over one 15-minute window. After the router fix there was no hour-long cooldown again, and over 30 days, with provider errors at 3.9% of requests, 0.34% of requests reached consumers as failures.
3. A model change at the provider under an unchanged name is visible neither in the code nor in the gateway's responses and logs; it can be learned from the provider's change log, if the provider records it, and from the score distribution. Paired rescoring of 668 items showed an order close to the former one (rank correlation 0.81–0.96) and a shifted scale (0.8–1.6 points). So after a model change the thresholds need recalculating, and the score distribution, not a single control item, needs watching.
4. Prices and default model behaviour are set by the provider and change without any involvement of the gateway: the gateway recorded 61% of the cost of calls at the official table, and reasoning, which DeepSeek V4 models do by default, was not switched off by the chat template parameter. The price is held by a configuration test, and switching reasoning off was verified by a live call and by the share of reasoning tokens in the gateway log.
5. A fix is verified on the next provider failures, not by the first green response: a restart, if the cooldown list lives in memory, clears it and on its own looks like a repair. And it is pinned by a configuration test, otherwise the next rebuild of a generated config overwrites it.