reclm

Record your llm calls and make your notebooks fast again.

When building AI based tooling and packaging we often call LLMs while prototyping and testing our code. A single LLM call can take 100’s of ms to run and the output isn’t deterministic. This can really slow down development especially if our notebook contains many LLM calls 😞.

While LLMs are new, working with external APIs in our code isn’t. Plenty of tooling already exists that make working with APIs much easier. For example, Python’s unittest mock object is commonly used to simulate or mock an API call so that it returns a hardcoded response. This works really well in the traditional Python development workflow and can make our tests fast and predictable.

However, it doesn’t work well in the nbdev workflow where oftentimes we’ll want to quickly run all cells in our notebook while we’re developing our code. While we can use mocks in our test cells we don’t want our exported code cells to be mocked. This leaves us with two choices:

• we temporarily mock our exported code cells but undo the mocking before we export these cells.
• we do nothing and just live with notebooks that take a long time to run.

Both options are pretty terrible as they pull us out of our flow state and slow down development 😞.

reclm builds on the underlying idea of mocks but adapts them to exploratory workflows. It initializes each sdk client (e.g. AsyncAnthropic) with a custom http_client. This http_client intercepts each LLM call, caches the response, and returns this cached response if the user makes the same LLM call again.

Usage

To use reclm

• install the package: pip install git+https://github.com/AnswerDotAI/reclm.git
• import the package from reclm.core import enable_reclm in each notebook
• add enable_reclm() to the top of each notebook

Note: enable_reclm should be added after you import the OpenAI and/or Anthropic SDK.

Every LLM call you make using OpenAI/Anthropic will now be cached in reclm.json.

Cache Location

If you’re using reclm in an nbdev project the cache (i.e. reclm.json) will be added to your project’s root dir. For all other projects the cache will be added to the current working directory. You can set a custom location for the cache by passing cache_dir to enable_reclm (e.g. enable_reclm(cache_dir='/path/to/cache')).

Your Tests

nbdev_test will automatically read from the cache. However, if your notebooks contain LLM calls that haven’t been cached, nbdev_test will call the OpenAI/Anthropic APIs and then cache the responses.

Cleaning the cache

It is recommended that you clean the cache before committing it. This will remove any stale LLM requests you’ve accumulated during development.

To clean the cache, run update_reclm_cache from your project’s root directory.

Note: Your LLM request/response data is stored in a file called reclm.json. Depending on your setup, it might be stored in your project’s root dir or your current working directory.

Note: All request headers are removed so it is safe to include this file in your version control system (e.g. git). In fact, it is expected that you’ll include this file in your vcs. There is one scenario where your api key could end up in the cache. This occurs if you make an LLM call that includes your api key in the response such as an authentication error.

Reclm Tests

The reclm test suite is found in test_reclm.py. Use the command python test_reclm.py to run the test suite.

Each test checks a specific SDK endpoint (e.g. anthropic streaming). If reclm is working correctly the SDK endpoint won’t be called and instead the response will be pulled from reclm.json.