The problem
What was actually wrong
A language model call is three lines in a prototype and a page of defensive code in production. Providers return 429s and 500s under load, the same question arrives a hundred times a day and is paid for a hundred times, an outage at one provider takes the feature down, and a loop with a bug can spend real money before anyone notices.
The data is the other half. Customer emails, phone numbers and card numbers reach the prompt unless something removes them first, tool results and user text can carry instructions aimed at the model, and the JSON that comes back is occasionally not JSON at all.
So every team writes the same file: a retry decorator, a cache dictionary, a regular expression for emails, a token cost formula, and a loop that asks the model again when parsing fails. It is written once per project, tested rarely, and owned by nobody.
Approach
What I built
Intercept instead of replace. Rather than asking people to call a new client, callm instruments the six SDK methods they already call and only acts while a decorated function is running. Outside that scope the SDK behaves exactly as before, and the caller gets the SDK's own response object back, even when the answer came from the cache or from a different provider.
Run everything as one stack. Injection scoring and masking happen first, so nothing sensitive reaches the cache key or the network. Then the cache, then validation, then fallback, then the cost guard, then retries. Each layer is independent and only switched on when it is configured.
Refuse to guess. The cache matches exactly unless similarity matching is requested, because two prompts that differ by one URL look identical to an embedding model and deserve different answers. A fallback across providers is skipped when the request carries tools, images or a response format, since translating those would quietly change the question.
Measure the claims. The benchmark script in the repository runs the real SDKs against a local server, so the overhead figure, the cache savings and the recovery rate can be reproduced by anyone in about ten seconds.
Where it got hard
3 decisions, each with tests behind it
Three problems that decided the design
None of these are visible from the outside, and each one would have shown up later as a report that quietly disagreed with reality. Two were settled before the first release; the third was found by a line by line review against reproductions.
- 01
One implementation for sync and async
Retries, caching and fallback all need to wait, call and branch, and writing them twice is how the two versions drift apart. Each layer is a generator that yields what it wants done, a sleep or a call, and receives the result. A synchronous driver runs those with time.sleep and direct calls, an asynchronous one with asyncio.sleep and await. The logic exists once, so a def and an async def cannot behave differently.
- 02
Translating a request without changing the question
Sending an OpenAI conversation to Claude means rebuilding it: the system prompt moves out of the message list, consecutive turns merge, and a required token limit appears from nowhere. Sampling parameters that current Claude models reject are dropped. Anything that cannot survive the trip, tools, images, response formats, marks the request as unportable, so it only ever falls back to models from the same provider.
- 03
The review that found ten defects
Before release I had the code reviewed line by line against reproductions rather than opinions. It found ten real defects, including masking silently switching off inside nested scopes and a crash when a schema class met the cache. The ones that matter were not in the interesting parts. They were in the seams between features, which is exactly where tests are usually thinnest, and each one now has a regression test.
Not established
What these numbers do not show
A library that sits between an application and its model provider is only worth using if it is honest about what it does not catch. Every limit here is also in the project README.
- Detection is pattern based. PII masking covers structured identifiers, emails, phone numbers, cards with a checksum, SSNs and IBANs, and person names only with an optional model. Injection detection is a heuristic scorer with an optional local classifier. Both reduce risk and neither removes it, so the documentation says so next to every claim.
- Cost limits are enforced against an estimate before the call and the real figure after it, so a single call can exceed a budget by its estimation error. Setting a token limit removes most of that gap.
- Streaming responses get masking, retries, budgets and telemetry, but they are not cached, validated or failed over.
- Six SDK methods are instrumented. Helpers that bypass them, such as the streaming and parsing wrappers, are not, and the documentation lists them rather than leaving people to discover it.