{
  "id": "toolu_01GBASPwZtBJGApsGK7Pvvu7",
  "call": {
    "function": "read_url",
    "arguments": {
      "url": "https://alexzhang13.github.io/blog/2025/rlm/"
    }
  },
  "result": "'[ Alex L. Zhang ](/) Toggle navigation\\n\\n  * [about ](/)\\n  * [blog ](/blog/)\\n  * [publications and pre-prints ](/publications/)\\n  * [projects ](/projects/)\\n  * [academics ](/academics/)\\n  * ctrl k __\\n  * ______\\n\\n# Recursive Language Models\\n\\nWe propose Recursive Language Models (RLMs), an inference strategy where language models can decompose and recursively interact with input context of unbounded length through REPL environments.\\n\\n## tl;dr\\n\\nWe explore language models that **recursively call themselves or other LLMs** before providing a final answer. Our goal is to enable the processing of essentially unbounded input context length and output length and to mitigate degradation \u201ccontext rot\u201d.\\n\\nWe propose **Recursive Language Models** , or **RLM** s, a general inference strategy where language models can decompose and recursively interact with their input context as a variable. We design a specific instantiation of this where GPT-5 or GPT-5-mini is queried in a Python REPL environment that stores the user\u2019s prompt in a variable.\\n\\nWe demonstrate that an **RLM using GPT-5-mini outperforms GPT-5** on a split of the most difficult long-context benchmark we got our hands on (OOLONG ) by more than **double** the number of correct answers, and is **cheaper** per query on average! We also construct a new long-context Deep Research task from BrowseComp-Plus . On it, we observe that RLMs outperform other methods like ReAct + test-time indexing and retrieval over the prompt. Surprisingly, we find that RLMs also do not degrade in performance when given 10M+ tokens at inference time.\\n\\nWe are excited to share these very early results, as well as argue that RLMs will be a powerful paradigm very soon. We think that RLMs trained explicitly to recursively reason are likely to represent the next milestone in **general-purpose inference-time scaling** after CoT-style reasoning models and ReAct-style agent models.\\n\\nWe have a compressed summary in the original tweet: <https://x.com/a1zhang/status/1978469116542337259>\\n\\nWe also now have a minimal implementation for people to build on top of: <https://github.com/alexzhang13/rlm>\\n\\n![Teaser Figure](/assets/img/rlm/teaser.png) **Figure 1.** An example of a recursive language model (RLM) call, which acts as a mapping from text \u2192 text, but is more flexible than a standard language model call and can scale to near-infinite context lengths. An RLM allows a language model to interact with an environment (in this instance, a REPL environment) that stores the (potentially huge) context, where it can recursively sub-query \u201citself\u201d, other LM calls, or other RLM calls, to efficiently parse this context and provide a final response.\\n\\n## Prelude: Why is \u201clong-context\u201d research so unsatisfactory?\\n\\nThere is this well-known but difficult to characterize phenomenon in language models (LMs) known as \u201ccontext rot\u201d. [Anthropic defines context rot](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) as \u201c[when] the number of tokens in the context window increases, the model\u2019s ability to accurately recall information from that context decreases\u201d, but many researchers in the community know this definition doesn\u2019t _fully_ hit the mark. For example, if we look at popular needle-in-the-haystack benchmarks like [RULER](https://arxiv.org/abs/2404.06654), most frontier models actually do extremely well (90%+ on 1-year old models).\\n\\n![Pun kin](/assets/img/rlm/pumpkin.png) _I asked my LM to finish carving the pumpkin joke it started yesterday. It said, \u201cPumpkin? What pumpkin?\u201d \u2014 the context completely rotted._\\n\\nBut [people have noticed](https://x.com/kwindla/status/1962230672082497866) that context rot is this weird thing that happens when your Claude Code history gets bloated, or you chat with ChatGPT for a long time \u2014 it\u2019s almost like, as the conversation goes on, the model gets\u2026dumber? It\u2019s sort of this well-known but hard to describe failure mode that we don\u2019t talk about in our papers because we can\u2019t benchmark it. The natural solution is something along the lines of, \u201cwell maybe if I split the context into two model calls, then combine them in a third model call, I\u2019d avoid this degradation issue\u201d. We take this intuition as the basis for a recursive language model.\\n\\n## **Recursive Language Models (RLMs).**\\n\\nA recursive language model is a thin wrapper around a LM that can spawn (recursive) LM calls for intermediate computation \u2014 from the perspective of the user or programmer, it is the same as a model call. In other words, you query a RLM as an \u201cAPI\u201d like you would a LM, i.e. `rlm.completion(messages)` is a direct replacement for `gpt5.completion(messages)`. We take a **context-centric view** rather than a **problem-centric view** of input decomposition. This framing retains the functional view that we want a system that can answer a particular **query** over some associated **context** :\\n\\n![API](/assets/img/rlm/api.png) **Figure 2.** A recursive language model call replaces a language model call. It provides the user the illusion of near infinite context, while under the hood a language model manages, partitions, and recursively calls itself or another LM over the context accordingly to avoid context rot.\\n\\nUnder the hood, a RLM provides only the **query** to the LM (which we call the **root LM** , or LM with depth=0), and allows this LM to interact with an **environment** , which stores the (potentially huge) **context**.\\n\\nWe choose the **environment** to be a loop where the LM can write to and read the output of cells of a Python REPL Notebook (similar to a Jupyter Notebook environment) that is pre-loaded with the **context** as a variable in memory. The **root LM** has the ability to call a recursive LM (or LM with depth=1) inside the REPL **environment** as if it were a function in code, allowing it to naturally peek at, partition, grep through, and launch recursive sub-queries over the **context**. **Figure 3** shows an example of how the RLM with a REPL **environment** produces a final answer.\\n\\n![API](/assets/img/rlm/repl.png) **Figure 3.** Our instantiation of the RLM framework provides the root LM the ability to analyze the context in a Python notebook environment, and launch recursive LM calls (depth=1) over any string stored in a variable. The LM interacts by outputting code blocks, and it receives a (truncated) version of the output in its context. When it is done, it outputs a final answer with `FINAL(\u2026)` tags or it can choose to use a string in the code execution environment with `FINAL_VAR(\u2026)`.\\n\\nWhen the **root LM** is confident it has an answer, it can either directly output the answer as `FINAL(answer)`, or it can build up an answer using the variables in its REPL environment, and return the string inside that answer as `FINAL_VAR(final_ans_var)`.\\n\\nThis setup yields several benefits that are visible in practice:\\n\\n  1. The context window of the root LM is rarely clogged \u2014 because it never directly sees the entire context, its input context grows slowly.\\n  2. The root LM has the flexibility to view subsets of the context, or naively recurse over chunks of it. For example, if the query is to find a needle-in-the-haystack fact or multi-hop fact, the root LM can use `regex` queries to roughly narrow the context, then launch recursive LM calls over this context. This is particularly useful for arbitrary long context inputs, where indexing a retriever is expensive on the fly!\\n  3. The context can, in theory, be any modality that can be loaded into memory. The root LM has full control to view and transform this data, as well as ask sub-queries to a recursive LM.\\n\\n**Relationship to test-time inference scaling.** We are particularly excited about this view of language models because it offers another axis of scaling test-time compute. The trajectory in which a language model chooses to interact with and recurse over its context is entirely learnable, and can be RL-ified in the same way that reasoning is currently trained for frontier models. Interestingly, it does not directly require training models that can handle huge context lengths because **no single language model call should require handling a huge context**.\\n\\n**RLMs with REPL environments are powerful.** We highlight that the choice of the **environment** is flexible and not fixed to a REPL or code environment, but we argue that it is a good choice. The two key design choices of recursive language models are 1) treating the prompt as a Python variable, which can be processed programmatically in arbitrary REPL flows. This allows the LLM to figure out what to peek at from the long context, at test time, and to scale any decisions it wants to take (e.g., come up with its own scheme for chunking and recursion adaptively) and 2) allowing that REPL environment to make calls back to the LLM (or a smaller LLM), facilitated by the decomposition and versatility from choice (1).\\n\\nWe were excited by the design of CodeAct, and reasoned that adding recursive model calls to this system could result in significantly stronger capabilities \u2014 after all, LM function calls are incredibly powerful. However, we argue that RLMs fundamentally view LM usage and code execution differently than prior works: the **context** here is an object to be understood by the model, and code execution and recursive LM calls are a means of understanding this context efficiently. Lastly, in our experiments we only consider a recursive depth of 1 \u2014 i.e. the root LM can only call LMs, not other RLMs. It is a relatively easy change to allow the REPL environment to call RLMs instead of LMs, but we felt that for most modern \u201clong context\u201d benchmarks, a recursive depth of 1 was sufficient to handle most problems. However, for future work and investigation into RLMs, enabling larger recursive depth will naturally lead to stronger and more interesting systems.\\n\\n**The formal definition (click to expand)** Consider a general setup of a language model $M$ receiving a query $q$ with some associated, potentially long context $C = {[c_1,c_2,\u2026,c_m]}$. The standard approach is to treat $M(q,C)$ like a black box function call, which takes a query and context and returns some `str` output. We retain this frame of view, but define a thin scaffold on top of the model to provide a more **expressive** and **interpretable** function call $RLM_M(q,C)$ with the same input and output spaces. Formally, a recursive language model $RLM_{M}(q, C)$ over an environment $\\\\mathcal{E}$ similarly receives a query $q$ and some associated, potentially long context $C = [c_1,c_2,\u2026,c_m]$ and returns some `str` output. The primary difference is that we provide the model a tool call $RLM_M(\\\\hat{q}, \\\\hat{C})$, which spawns an isolated sub-RLM instance using a new query $\\\\hat{q}$ and a transformed version of the context $\\\\hat{C}$ with its own isolated environment $\\\\hat{\\\\mathcal{E}}$; eventually, the final output of this recursive callee is fed back into the environment of the original caller. The environment $\\\\mathcal{E}$ abstractly determines the control flow of how the language model $M$ is prompted, queried, and handled to provide a final output. In this paper, we specifically explore the use of a Python REPL environment that stores the input context $C$ as a variable in memory. This specific choice of environment enables the language model to **peek at** , **partition** , **transform** , and **map** over the input context and use recursive LMs to answer sub-queries about this context. Unlike prior agentic methods that rigidly define these workflow patterns, RLMs defer these decisions entirely to the language model. Finally, we note that particular choices of environments $\\\\mathcal{E}$ are flexible and are a generalization of a base model call: the simplest possible environment $\\\\mathcal{E}_0$ queries the model $M$ with input query and context $q, C$ and returns the model output as the final answer.\\n\\n## Some early (and very exciting) results!\\n\\nWe\u2019ve been looking around for benchmarks that reflect natural long-context tasks, e.g. long multi-turn Claude Code sessions. We namely were looking to highlight two properties that limit modern frontier models: 1) the context rot phenomenon, where model performance degrades as a function of context length, and 2) the system-level limitations of handling an enormous context.\\n\\nWe found in practice that many long-context benchmarks offer contexts that are not really that long and which were already solvable by the latest generation (or two) of models. In fact, we found some where **models could often answer queries without the context**! We luckily quickly found two benchmarks where modern frontier LLMs struggle to perform well, but we are [actively seeking](https://x.com/lateinteraction/status/1976964409139642716) any other good benchmark recommendations to try.\\n\\n### **Exciting Result #1 \u2014 Dealing with Context Rot.**\\n\\nThe **OOLONG** benchmark is a challenging new benchmark that evaluates long-context reasoning tasks over fine-grained information in context. We were fortunate to have the (anonymous _but not affiliated with us_) authors share the dataset upon request to run our experiments on a split of this benchmark.\\n\\n**Setup.** The `trec_coarse` split consists of 6 different types of queries to answer distributional queries about a giant list of \u201cquestion\u201d entries. For example, one question looks like:\\n\\n`For the following question, only consider the subset of instances that are associated with user IDs 67144, 53321, 38876, 59219, 18145, 64957, 32617, 55177, 91019, 53985, 84171, 82372, 12053, 33813, 82982, 25063, 41219, 90374, 83707, 59594. Among instances associated with these users, how many data points should be classified as label \\'entity\\'? Give your final answer in the form \\'Answer: number\\'.`\\n\\nThe query is followed by ~3000 - 6000 rows of entries with associated user IDs (not necessarily unique) and instances that **are not explicitly labeled** (i.e. the model has to infer the labeling to answer). They look something like this:\\n\\n```\\nDate: Dec 12, 2022 || User: 63685 || Instance: How many years old is Benny Carter ?\\nDate: Dec 30, 2024 || User: 35875 || Instance: What war saw battles at Parrot \\'s Beak and Black Virgin ?\\nDate: Apr 13, 2024 || User: 80726 || Instance: What Metropolis landmark was first introduced in the Superman cartoons of the 1940 \\'s ?\\nDate: Feb 29, 2024 || User: 59320 || Instance: When was Calypso music invented?\\n...\\n\\n```\\n\\nThe score is computed as the number of queries answered correctly by the model, with the caveat that for numerical / counting problems, they use a continuous scoring metric. This benchmark is extremely hard for both frontier models and agents because they have to **semantically** map and associate thousands of pieces of information in a single query, and cannot compute things a-priori! We evaluate the following models / agents:\\n\\n  * **GPT-5.** Given the whole context and query, tell GPT-5 to provide an answer.\\n  * **GPT-5-mini.** Given the whole context and query, tell GPT-5-mini to provide an answer.\\n  * **RLM(GPT-5-mini).** Given the whole context and query, tell RLM(GPT-5-mini) to provide an answer. GPT-5-mini (root LM) can recursively call GPT-5-mini inside its REPL environment.\\n  * **RLM(GPT-5) without sub-calls.** Given the whole context and query, tell RLM(GPT) to provide an answer. GPT-5 (root LM) cannot recursively call GPT-5 inside its REPL environment. This is an ablation for the use of a REPL environment without recursion.\\n  * **ReAct w/ GPT-5 + BM25.** We chunk every lines into its own \u201cdocument\u201d, and gives a ReAct loop access to a BM25 retriever to return 10 lines per search request.\\n\\n**Results.** We focus explicitly on questions with contexts over 128k tokens (~100 queries), and we track both the performance on the benchmark, as well as the overall API cost of each query. In all of the following results (Figure **4a,b**), **the entire input fits in the context window of GPT-5 / GPT-5-mini** \u2014 i.e., incorrect predictions are never due to truncation or context window size limitations:\\n\\n![API](/assets/img/rlm/oolong-132k.png) **Figure 4a.** We report the overall score for each method on the `trec_coarse` dataset of the OOLONG benchmark for queries that have a context length of 132k tokens. We compare performance to GPT-5. RLM(GPT-5-mini) outperforms GPT-5 by over **34 points (~114% increase)** , and is nearly as cheap per query (we found that the median query is cheaper due to some outlier, expensive queries).\\n\\nIt turns out actually that **RLM(GPT-5-_mini_)** outperforms **GPT-5** and **GPT-5-mini** by **> 33%**\u2191 raw score (over double the performance) while maintaining roughly the same total model API cost as **GPT-5** per query! When ablating recursion, we find that RLM performance degrades by ~10%, likely due to many questions requiring the model to answer semantic questions about the data (e.g. label each question). We see in **Figure 4b** that these gains roughly transfer when we double the size of the context to ~263k tokens as well, although with some performance degradation!\\n\\n![API](/assets/img/rlm/oolong-256k.png) **Figure 4b.** We report the overall score for each method on the trec_coarse dataset of the OOLONG benchmark for queries that have a context length of 263k tokens, nearly the limit for GPT-5/GPT-5-mini. We compare performance to GPT-5. RLM(GPT-5-mini) outperforms GPT-5 by over **15 points (~49% increase)** , and is cheaper per query on average.\\n\\nNotably, the performance of **GPT-5-mini** drops while **GPT-5** does not, which indicates that context rot is more severe for GPT-5-mini. We additionally noticed that the performance drop for the RLM approaches occurs for **_counting_** problems, where it makes more errors when the context length increases \u2014 for **GPT-5** , it already got most of these questions incorrect in the 132k context case, which explains why its performance is roughly preserved. Finally, while the **ReAct + GPT-5 + BM25** baseline doesn\u2019t make much sense in this setting, we provide it to show retrieval is difficult here while **RLM** is the more appropriate method.\\n\\nGreat! So we\u2019re making huge progress in solving goal (1), where GPT-5 has _just_ enough context window to fit the 263k case. But what about goal (2), where we may have 1M, 10M, or even 100M tokens in context? _Can we still treat this like a single model call?_\\n\\n### **Exciting Result #2 \u2014 Ridiculously Large Contexts**\\n\\nMy advisor Omar is a [superstar in the world of information retrieval (IR)](https://arxiv.org/abs/2004.12832), so naturally we also wanted to explore whether RLMs scale properly when given thousands (or more!) of documents. OOLONG provides a giant block of text that is difficult to index and therefore difficult to compare to retrieval methods, so we looked into [DeepResearch](https://openai.com/index/introducing-deep-research/)-like benchmarks that evaluate answering queries over documents.\\n\\n**Retrieval over huge offline corpuses.** We initially were interested in [BrowseComp](https://openai.com/index/browsecomp/) , which evaluates agents on multi-hop, web-search queries, where agents have to find the relevant documents online. We later found the [BrowseComp-Plus](https://arxiv.org/abs/2508.06600) benchmark, which pre-downloads all possible relevant documents for all queries in the original benchmark, and just provides a list of ~100K documents (~5k words on average) where the answer to a query is scattered across this list. For benchmarking RLMs, this benchmark is perfect to see if we can just throw ridiculously large amount of context into a single `chat.completion(...)` RLM call instead of building an agent!\\n\\n**Setup.** We explore how scaling the # documents in context affects the performance of various common approaches to dealing with text corpuses, as well as RLMs. Que<TRUNCATED>"
}

{
  "id": "toolu_01VmJH9ghVCTkLmApVAPm93h",
  "call": {
    "function": "read_url",
    "arguments": {
      "url": "https://answerdotai.github.io/toolslm/shell.html"
    }
  },
  "result": "'[ ](./index.html) [ toolslm ](./index.html)\\n\\n__\\n\\n  1. [shell source](./shell.html)\\n\\n[ ](./index.html)\\n\\n  * [ toolslm](./index.html)\\n\\n  * [ xml source](./xml.html)\\n\\n  * [ funccall source](./funccall.html)\\n\\n  * [ shell source](./shell.html)\\n\\n  * [ Download helpers](./download.html)\\n\\n  * [ Markdown Hierarchy Parser](./md_hier.html)\\n\\n## On this page\\n\\n  * TerminalInteractiveShell.run_cell\\n  * get_shell\\n\\n  * [__Report an issue](https://github.com/AnswerDotAI/toolslm/issues/new)\\n\\n## Other Formats\\n\\n  * [ __CommonMark](shell.html.md)\\n\\n# shell source\\n\\nExported source\\n\\n```\\nimport ast, time, signal, traceback\\nfrom fastcore.utils import *\\n```\\n\\n__\\n\\n[`get_shell`](https://AnswerDotAI.github.io/toolslm/shell.html#get_shell) is like [`python`](https://AnswerDotAI.github.io/toolslm/funccall.html#python), except it also maintains a stateful interpreter, rather than just running a single line of code. This is implemented using IPython, so that must be installed.\\n\\nExported source\\n\\n```\\nfrom IPython.terminal.interactiveshell import TerminalInteractiveShell\\nfrom IPython.utils.capture import capture_output\\n```\\n\\n __\\n\\n```\\ndef exception2str(ex:Exception)->str:\\n    \"Convert exception `ex` into a string\"\\n    return \\'\\'.join(traceback.format_exception(type(ex), ex, ex.__traceback__))\\n```\\n\\n__\\n\\n```\\ntry: print(1/0)\\nexcept Exception as e: print(exception2str(e))\\n```\\n\\n__\\n\\n```\\nTraceback (most recent call last):\\n  File \"/var/folders/ss/34z569j921v58v8n1n_8z7h40000gn/T/ipykernel_37260/4058275565.py\", line 1, in <module>\\n    try: print(1/0)\\n               ~^~\\nZeroDivisionError: division by zero\\n\\n```\\n\\n* * *\\n\\n[source](https://github.com/AnswerDotAI/toolslm/blob/main/toolslm/shell.py#L19)\\n\\n### TerminalInteractiveShell.run_cell\\n\\n>\\n```\\n>      TerminalInteractiveShell.run_cell (cell, timeout=None)\\n```\\n\\n_Wrapper for original`run_cell` which adds timeout and output capture_\\n\\nExported source\\n\\n```\\nTerminalInteractiveShell.orig_run = TerminalInteractiveShell.run_cell\\n```\\n\\n __\\n\\nExported source\\n\\n```\\n@patch\\ndef run_cell(self:TerminalInteractiveShell, cell, timeout=None):\\n    \"Wrapper for original `run_cell` which adds timeout and output capture\"\\n    if timeout:\\n        def handler(*args): raise TimeoutError()\\n        signal.signal(signal.SIGALRM, handler)\\n        signal.alarm(timeout)\\n    try:\\n        with capture_output() as io: result = self.orig_run(cell)\\n        result.stdout = io.stdout\\n        return result\\n    except TimeoutException as e:\\n        result = self.ExecutionResult(error_before_exec=None, error_in_exec=e)\\n    finally:\\n        if timeout: signal.alarm(0)\\n```\\n\\n__\\n\\n* * *\\n\\n[source](https://github.com/AnswerDotAI/toolslm/blob/main/toolslm/shell.py#L35)\\n\\n### get_shell\\n\\n>\\n```\\n>      get_shell ()\\n```\\n\\n_Get a`TerminalInteractiveShell` with minimal functionality_\\n\\nExported source\\n\\n```\\ndef get_shell()->TerminalInteractiveShell:\\n    \"Get a `TerminalInteractiveShell` with minimal functionality\"\\n    sh = TerminalInteractiveShell()\\n    sh.logger.log_output = sh.history_manager.enabled = False\\n    dh = sh.displayhook\\n    dh.finish_displayhook = dh.write_output_prompt = dh.start_displayhook = lambda: None\\n    dh.write_format_data = lambda format_dict, md_dict=None: None\\n    sh.logstart = sh.automagic = sh.autoindent = False\\n    sh.autocall = 0\\n    sh.system = lambda cmd: None\\n    return sh\\n```\\n\\n __\\n\\n```\\nshell = get_shell()\\n```\\n\\n__\\n\\n```\\nr = shell.run_cell(\\'print(3); 1+1\\')\\nr.result,r.stdout\\n```\\n\\n __\\n\\n```\\n(2, \\'3\\\\n\\')\\n```\\n\\n```\\nr = shell.run_cell(\\'raise Exception(\"blah\")\\')\\nprint(exception2str(r.error_in_exec))\\n```\\n\\n__\\n\\n```\\nTraceback (most recent call last):\\n  File \"/Users/jhoward/miniconda3/lib/python3.12/site-packages/IPython/core/interactiveshell.py\", line 3577, in run_code\\n    exec(code_obj, self.user_global_ns, self.user_ns)\\n  File \"<ipython-input-1-338156281413>\", line 1, in <module>\\n    raise Exception(\"blah\")\\nException: blah\\n\\n```\\n\\n```\\nr = shell.run_cell(\\'import time; time.sleep(10)\\', timeout=1)\\nr.error_in_exec\\n```\\n\\n __\\n\\n```\\nTimeoutError()\\n```\\n\\n  * [__Report an issue](https://github.com/AnswerDotAI/toolslm/issues/new)'"
}

{
  "id": "toolu_01FaCKJNR86853D7P6dvuc5H",
  "call": {
    "function": "read_url",
    "arguments": {
      "url": "https://lisette.answer.ai/#tools"
    }
  },
  "result": "'## Tools\\n\\nLisette makes it easy to give LLMs access to Python functions. Just define a function with type hints and a docstring:\\n\\n```\\ndef add_numbers(\\n    a: int,  # First number to add\\n    b: int   # Second number to add  \\n) -> int:\\n    \"Add two numbers together\"\\n    return a + b\\n```\\n\\n __\\n\\nNow pass the function to [`Chat`](https://lisette.answer.ai/core.html#chat) and the model can use it automatically:\\n\\n```\\nchat = Chat(models[0], tools=[add_numbers])\\nres = chat(\"What\\'s 47 + 23? Use the tool.\")\\nres\\n```\\n\\n __\\n\\nThe result of 47 + 23 is 70.\\n\\n  * id: `chatcmpl-xxx`\\n  * model: `claude-sonnet-4-20250514`\\n  * finish_reason: `stop`\\n  * usage: `Usage(completion_tokens=18, prompt_tokens=573, total_tokens=591, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None), cache_creation_input_tokens=0, cache_read_input_tokens=0)`\\n\\nIf you want to see all intermediate messages and outputs you can use the `return_all=True` feature.\\n\\n```\\nchat = Chat(models[0], tools=[add_numbers])\\nres = chat(\"What\\'s 47 + 23 + 59? Use the tool.\",max_steps=3,return_all=True)\\ndisplay(*res)\\n```\\n\\n__\\n\\nI\u2019ll help you calculate 47 + 23 + 59 using the add_numbers tool. Since the tool can only add two numbers at a time, I\u2019ll need to do this in two steps.\\n\\n\ud83d\udd27 add_numbers({\u201ca\u201d: 47, \u201cb\u201d: 23})\\n\\n  * id: `chatcmpl-xxx`\\n  * model: `claude-sonnet-4-20250514`\\n  * finish_reason: `tool_calls`\\n  * usage: `Usage(completion_tokens=116, prompt_tokens=433, total_tokens=549, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None), cache_creation_input_tokens=0, cache_read_input_tokens=0)`\\n\\n```\\n{\\'tool_call_id\\': \\'toolu_01F9oakoP8ANHkTMD1DyQDi7\\',\\n \\'role\\': \\'tool\\',\\n \\'name\\': \\'add_numbers\\',\\n \\'content\\': \\'70\\'}\\n```\\n\\nNow I\u2019ll add the result (70) to the third number (59):\\n\\n\ud83d\udd27 add_numbers({\u201ca\u201d: 70, \u201cb\u201d: 59})\\n\\n  * id: `chatcmpl-xxx`\\n  * model: `claude-sonnet-4-20250514`\\n  * finish_reason: `tool_calls`\\n  * usage: `Usage(completion_tokens=87, prompt_tokens=562, total_tokens=649, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None), cache_creation_input_tokens=0, cache_read_input_tokens=0)`\\n\\n```\\n{\\'tool_call_id\\': \\'toolu_01Cdf3FHJdbx64F8H8ooE1Db\\',\\n \\'role\\': \\'tool\\',\\n \\'name\\': \\'add_numbers\\',\\n \\'content\\': \\'129\\'}\\n```\\n\\nThe answer is **129**.\\n\\nI calculated this by first adding 47 + 23 = 70, then adding 70 + 59 = 129.\\n\\n  * id: `chatcmpl-xxx`\\n  * model: `claude-sonnet-4-20250514`\\n  * finish_reason: `stop`\\n  * usage: `Usage(completion_tokens=41, prompt_tokens=702, total_tokens=743, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None), cache_creation_input_tokens=0, cache_read_input_tokens=0)`\\n\\nIt shows the intermediate tool calls, and the tool results!'"
}

Thanks for sharing these URLs! I've read through them. Your plan looks pretty solid overall. Here are a few things to think about:

Potential considerations:

1. Context size handling - The RLM paper shows contexts can be huge (10M+ tokens). How will you handle passing very large contexts into the Python REPL? Will you need streaming/chunking strategies?

2. Tool calling depth - The paper mentions they only use recursive depth of 1 (root LM calls LMs, not other RLMs). Do you want to support deeper recursion? If so, you'll need to think about how to prevent infinite loops and manage the call stack.

3. Truncation strategy - toolslm's shell captures output, but what happens if the recursive call_lm() returns a very long response? Will you truncate it before returning to the root LM?

4. Timeout handling - The shell already has timeout support. Should call_lm() also have its own timeout to prevent hanging on slow API calls?

5. System prompt for the root LM - What instructions will you give the root LM about how to use the REPL and call_lm()? The paper mentions specific prompting strategies.

Do any of these concerns resonate with your use case, or do you have thoughts on how you'd handle them?

Ah, you're absolutely right! I was thinking about it wrong. The context lives in the REPL's memory, not in the LLM's context window. That's elegant.

Looking at your system prompt from the repo, it seems comprehensive. A few quick observations:

1. The prompt is quite long - Have you tested whether models actually follow all those instructions, or do they sometimes skip steps (like not checking context before answering)?

2. The "safeguard" on iteration 0 is nice - forces the model to look at context first.

3. The recursion strategy - The prompt encourages chunking and recursive calls, which aligns with the paper's approach.

One thing I notice: the prompt says sub-LLMs can handle "around 500K chars". Is that hardcoded, or will you make it configurable based on which model you're using for recursive calls?

Also, are you planning to use the same model for root and recursive calls, or mix models (like the paper does with GPT-5 calling GPT-5-mini)?

I'm Claude, an AI assistant made by Anthropic. I'm here to help answer questions, have conversations, analyze information, help with writing and coding, and assist with a wide variety of tasks. Is there something I can help you with today?

<ExecutionResult object at 7e6321fa6570, execution_count=1 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7e6321fa6120, raw_cell="15 + 35" transformed_cell="15 + 35
" store_history=False silent=False shell_futures=True cell_id=None> result=50>

<ExecutionResult object at 7e6320dcab40, execution_count=1 error_before_exec=invalid decimal literal (<ipython-input-1-bbd02197e81e>, line 3) error_in_exec=None info=<ExecutionInfo object at 7e6320dcae40, raw_cell="context=---
title: "The Not-so Bitter Lesson"
date.." transformed_cell="context=---
title: "The Not-so Bitter Lesson"
date.." store_history=False silent=False shell_futures=True cell_id=None> result=None>

Yes, exactly! The issue is that the blog post contains characters that Python is interpreting as code syntax (like the dashes in the YAML frontmatter).

You have a few options:

1. Use triple quotes - Wrap the string in """ or ''':

sh.run_cell(f'context = """{bp}"""')

2. Use repr() - This automatically escapes special characters:

sh.run_cell(f'context = {repr(bp)}')

3. Pass it directly to the namespace - Instead of running code, inject it:

sh.user_ns['context'] = bp

The third option is cleanest for your RLM use case since you're not actually executing user code - you're just setting up the environment. Want to try one of these?

<ExecutionResult object at 7e6320e06510, execution_count=1 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7e6320e06360, raw_cell="context[:200]" transformed_cell="context[:200]
" store_history=False silent=False shell_futures=True cell_id=None> result='---\ntitle: "The Not-so Bitter Lesson"\ndate: 2025-10-20T15:27:52+01:00\ndraft: false\nexcerpt: How tinygrad helped me finally understand what Sutton meant\n---\n\nHave you heard about The Bitter Lesson? If '>

Great! So now you have the context loaded into the REPL environment.

Next step: do you want to implement the llm_query() function that the root LM can call recursively? This would be a function in the shell namespace that takes a string and calls your lisette client.

Want to sketch out what that function should look like, or would you like me to suggest an approach?

'The capital of France is Paris.'

{'role': 'user', 'content': 'What is the capital of france?'}

The capital of France is Paris.

The capital of France is Paris.

Yes! You can add the function to the shell's namespace the same way:

sh.user_ns['llm_query'] = llm_query

Then the root LM will be able to call it from within the REPL. Want to test it by running a cell that calls llm_query()?

<ExecutionResult object at 7e6320e7dc40, execution_count=1 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7e6320e7dcd0, raw_cell="llm_query('what is the capital of france?')" transformed_cell="llm_query('what is the capital of france?')
" store_history=False silent=False shell_futures=True cell_id=None> result=ModelResponse(id='chatcmpl-e19c7837-40d0-4ddb-ae7c-935e9eac58cb', created=1766070471, model='claude-sonnet-4-5-20250929', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='stop', index=0, message=Message(content='The capital of France is Paris.', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}))], usage=Usage(completion_tokens=10, prompt_tokens=14, total_tokens=24, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0))>

'The capital of France is Paris.'

<ExecutionResult object at 7e6320e9c8f0, execution_count=1 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7e6320e9c980, raw_cell="llm_query('what is the capital of france?')" transformed_cell="llm_query('what is the capital of france?')
" store_history=False silent=False shell_futures=True cell_id=None> result='The capital of France is Paris.'>

Exactly! You're almost there.

For the tool wrapper, you'll need a function that:

1. Takes code as input (the repl code blocks from the LLM)
2. Runs it via sh.run_cell()
3. Returns the result (probably with some truncation for long outputs)

Do you want to write that function, or would you like me to show you a quick example?

Here's a quick example:

def run_repl(code: str) -> str:
    "Execute Python code in the REPL environment"
    result = sh.run_cell(code)
    if result.error_in_exec:
        return f"Error: {result.error_in_exec}"
    output = result.stdout or ""
    if result.result is not None:
        output += str(result.result)
    return output[:5000]  # truncate long outputs

This captures both stdout (from print()) and the result (from expressions). The truncation prevents overwhelming the root LM's context.

Want to test it out, or adjust anything?

'The title of the blog post is **"The Not-so Bitter Lesson"**'

Perfect! Now you can pass run_repl as a tool to your outer Chat instance along with the system prompt you showed earlier.

Something like:

outer_chat = Chat(model="...", tools=[run_repl])

Then add your system prompt and query, and the root LM should be able to interact with the REPL, inspect the context, and call llm_query() recursively.

Want to wire that up and test the full loop?

Here's a condensed version:

REPL_SYSTEM_PROMPT = """You are answering a query with associated context that is available in your REPL environment.

Your REPL has:
- A `context` variable (string) containing important information about your query
- An `llm_query(query: str)` function to recursively call another LLM

IMPORTANT: Always inspect the `context` variable first before answering. Use Python to explore, chunk, search, or transform it as needed.

Think step-by-step and use the REPL actively to build your answer."""

Does this capture what you need?

Based on the context, here's what made the bitter lesson click for the author:

The author's "aha moment" came from tinygrad and George Hotz's blog post "Can tinygrad win?"

Specifically, the author had been struggling to understand what Sutton's "Bitter Lesson" really meant in practice - what does it mean to solve problems using search and compute? What's the role of engineers if brute-force search beats clever domain knowledge?

The breakthrough came when George Hotz explained his vision for tinygrad: the entire stack of making code run fast on hardware is fundamentally a search problem across multiple layers - searching for better scheduling algorithms, faster GPU kernels, etc. The library's task is to:

1. "Expose the underlying search problem spanning several orders of magnitude"
2. "Apply the state of the art in search. Burn compute."

This concrete example made the author realize that someone still has to expose and frame the search problem correctly. Engineers aren't obsolete - their role shifts from crafting manual solutions to building the infrastructure that enables effective search. They design the problem representation, build evaluations, set constraints, and then let compute explore the solution space.

This realization transformed the "bitter" lesson into something positive: it's not that engineering doesn't matter, it's that engineering work matters differently - by creating the frameworks that allow search and compute to work effectively.

It looks like it did work - the LLM gave a correct answer based on the blog post content! But I see the issue you're pointing to:

run_repl uses the global sh variable, but prep_shell creates a new shell and returns it (without updating the global). It only worked because you had already set up sh with the context earlier in your session.

To make it cleaner, you have a couple options:

1. Call prep_shell and update the global sh:

sh = prep_shell(bp)

2. Or make run_repl a closure inside prep_shell so it captures the right shell instance.

Which approach would you prefer?

Ah, you're right! Looking at the RLM paper again - the whole point is that the root LM should not have the context in its prompt. It should only access it through the REPL.

But in your current setup, it looks like the model answered correctly without even calling the run_repl tool. Did it actually use the tool, or did it somehow have access to the context directly?

Check outer_chat.h to see the conversation history - were there any tool calls?

[{'role': 'user',
  'content': 'What made the bitter lesson click for the author?'},
 Message(content="I'll check the context to understand what made the bitter lesson click for the author.", role='assistant', tool_calls=[{'index': 1, 'function': {'arguments': '{"python_code": "\\n# First, let\'s examine the context\\nprint(context)\\n"}', 'name': 'run_repl'}, 'id': 'toolu_019DRuPQn1Luww82rBytGWoa', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_019DRuPQn1Luww82rBytGWoa',
  'role': 'tool',
  'name': 'run_repl',
  'content': '---\ntitle: "The Not-so Bitter Lesson"\ndate: 2025-10-20T15:27:52+01:00\ndraft: false\nexcerpt: How tinygrad helped me finally understand what Sutton meant\n---\n\nHave you heard about The Bitter Lesson? If you are on Twitter and follow some ML/AI people, I bet you have.\n\nIf you haven\'t: It is a [blog post by Richard Sutton](http://www.incompleteideas.net/IncIdeas/BitterLesson.html), a Canadian computer scientist, who wrote one of *the* books about reinforcement learning.\n\nSutton\'s core argument is simple: general methods that leverage search and compute outperform domain-specific solutions. By "search," he doesn\'t mean information retrieval - he means using learning algorithms to explore possible solutions or strategies. His go-to example is computer chess: early engines relied heavily on chess expertise to evaluate positions, while AlphaZero knew only the basic rules and learned by playing itself, eventually surpassing any human player.\n\nThis lesson is "bitter" because it suggests our clever insights don\'t matter. Our domain-specific knowledge gets crushed by what looks like "dumb brute-force search and compute." If you\'re an engineer, this naturally raises an uncomfortable question: what\'s the point of us?\n\n## The Click\n\nFor a long time, I didn\'t really understand Sutton\'s point. What does it mean to solve problems using search and compute? What is the role of engineers then? Are we just supposed to prompt a LLM and hope for the best? That can\'t scale.\n\nMy aha moment came from an unlikely source: [`tinygrad`](https://github.com/tinygrad/tinygrad). It\'s a Python library similar to PyTorch but much simpler on the surface. I think George Hotz initially built it as a toy project after watching Andrej Karpathy create micrograd, but it evolved into a solid tensor library that runs LLMs efficiently on different hardware through clever optimizations.\n\nGeorge\'s blog post ["Can tinygrad win?"](https://geohot.github.io//blog/jekyll/update/2025/07/06/can-tinygrad-win.html) made everything click. He lays out his vision for the library: the entire stack of making code run fast on hardware is fundamentally a search problem across multiple layers — searching for better scheduling algorithms, faster GPU kernels, all of it. The task of the library is to "Expose the underlying search problem spanning several orders of magnitude." and then "Apply the state of the art in search. Burn compute.".\n\nThat\'s when I finally understood what Sutton meant. Like in computer chess, raw compute and effective search will beat manually crafted solutions.\n\nBut the key insight is this: someone still has to expose that search problem. Someone has to frame it correctly.\n\nIn this post, I\'ll show you why our work as engineers still matters, just differently. The Bitter Lesson isn\'t bitter at all - it\'s a blueprint for better engineering. And the pattern of search & compute appears everywhere once you know what to look for.\n\n## The Framework\n\nFor me, The Bitter Lesson is all about building effective search systems. Again, not search in the sense of information retrieval, but in the sense of allowing raw compute to explore a space of solutions effectively.\n\nFor that we need three core components:\n1. An effective problem formulation that exposes the right parameters in a way that different algorithms can dig into it.\n2. A way to evaluate possible solutions (evals) and provide feedback to the search mechanism.\n3. A way to define constraints that a solution has to satisfy (memory, time, specific form etc.)\n\nOur job as engineers is to build these three components. We design the problem-specific, scalable infrastructure that enables search to work. Then we gtfo and let compute do what it does best.\n\nBut don\'t you worry, there is still place for your precious domain knowledge, even in The Bitter Lesson, but you should use it elsewhere:\n\nUse your domain knowledge to:\n\n* Design problem representations (make search tractable)\n* Build evaluations (capture what matters)\n* Set constraints (avoid unsafe regions)\n* Initialize search (warm start)\n\nAnd use search to:\n\n* Explore solution space\n* Find non-obvious patterns\n* Exceed human intuition\n* Scale the solution with compute\n\nSound familiar? We\'ve done this before — moving from punch cards to text-based code was the same kind of shift. And now we\'re building harnesses and guardrails (and [RL environments](https://www.primeintellect.ai/blog/environments)) that define the bounds of desired solutions rather than crafting the solutions directly.\n\n## The Examples\n\nIt\'s fascinating to see where these themes come up, besides `tinygrad`.\n\nIf you follow me on [Twitter](https://twitter.com/rasmus1610), you might know that I\'m a huge fanboi of DSPy - a declarative library for "programming not prompting" LLMs. And DSPy embodies the approach described above by separating problem definition from solution strategy. Omar Khattab, the creator of the library, actually gave a talk called ["On Engineering AI Systems that Endure The Bitt'},
 Message(content=None, role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"python_code": "\\n# Let\'s see if there\'s more context\\nprint(len(context))\\nprint(\\"\\\\n--- Looking for the specific part about what made it click ---\\\\n\\")\\n\\n# Find the section about \\"The Click\\"\\nclick_section_start = context.find(\\"## The Click\\")\\nclick_section_end = context.find(\\"## The Framework\\")\\n\\nif click_section_start != -1 and click_section_end != -1:\\n    click_section = context[click_section_start:click_section_end]\\n    print(click_section)\\n"}', 'name': 'run_repl'}, 'id': 'toolu_01WiLPeggausvwvFGjFi4GUA', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01WiLPeggausvwvFGjFi4GUA',
  'role': 'tool',
  'name': 'run_repl',
  'content': '9245\n\n--- Looking for the specific part about what made it click ---\n\n## The Click\n\nFor a long time, I didn\'t really understand Sutton\'s point. What does it mean to solve problems using search and compute? What is the role of engineers then? Are we just supposed to prompt a LLM and hope for the best? That can\'t scale.\n\nMy aha moment came from an unlikely source: [`tinygrad`](https://github.com/tinygrad/tinygrad). It\'s a Python library similar to PyTorch but much simpler on the surface. I think George Hotz initially built it as a toy project after watching Andrej Karpathy create micrograd, but it evolved into a solid tensor library that runs LLMs efficiently on different hardware through clever optimizations.\n\nGeorge\'s blog post ["Can tinygrad win?"](https://geohot.github.io//blog/jekyll/update/2025/07/06/can-tinygrad-win.html) made everything click. He lays out his vision for the library: the entire stack of making code run fast on hardware is fundamentally a search problem across multiple layers — searching for better scheduling algorithms, faster GPU kernels, all of it. The task of the library is to "Expose the underlying search problem spanning several orders of magnitude." and then "Apply the state of the art in search. Burn compute.".\n\nThat\'s when I finally understood what Sutton meant. Like in computer chess, raw compute and effective search will beat manually crafted solutions.\n\nBut the key insight is this: someone still has to expose that search problem. Someone has to frame it correctly.\n\nIn this post, I\'ll show you why our work as engineers still matters, just differently. The Bitter Lesson isn\'t bitter at all - it\'s a blueprint for better engineering. And the pattern of search & compute appears everywhere once you know what to look for.\n\n\n'},
 Message(content='Based on the context, here\'s what made the bitter lesson click for the author:\n\n**The author\'s "aha moment" came from `tinygrad` and George Hotz\'s blog post "Can tinygrad win?"**\n\nSpecifically, the author had been struggling to understand what Sutton\'s "Bitter Lesson" really meant in practice - what does it mean to solve problems using search and compute? What\'s the role of engineers if brute-force search beats clever domain knowledge?\n\nThe breakthrough came when George Hotz explained his vision for tinygrad: **the entire stack of making code run fast on hardware is fundamentally a search problem** across multiple layers - searching for better scheduling algorithms, faster GPU kernels, etc. The library\'s task is to:\n1. "Expose the underlying search problem spanning several orders of magnitude"\n2. "Apply the state of the art in search. Burn compute."\n\nThis concrete example made the author realize that **someone still has to expose and frame the search problem correctly**. Engineers aren\'t obsolete - their role shifts from crafting manual solutions to building the infrastructure that enables effective search. They design the problem representation, build evaluations, set constraints, and then let compute explore the solution space.\n\nThis realization transformed the "bitter" lesson into something positive: it\'s not that engineering doesn\'t matter, it\'s that engineering work matters *differently* - by creating the frameworks that allow search and compute to work effectively.', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None})]

Oh wow, it actually worked perfectly! Looking at the history:

1. First tool call: print(context) - inspected the full context
2. Second tool call: Used Python to find the "The Click" section specifically:
   Copied!

   click_section_start = context.find("## The Click")
   click_section_end = context.find("## The Framework")
   click_section = context[click_section_start:click_section_end]
   

3. Final answer: Synthesized a response based on what it found

That's exactly the RLM pattern in action! The root LM:

• Never had the context in its prompt directly
• Used the REPL to explore and slice the context with Python
• Gave an answer based on what it retrieved

Congrats - your basic RLM implementation is working! 🎉

Do you want to test it with a much larger context to see how it handles chunking and llm_query() for recursive calls?

Good point! The model got lucky here. You should definitely add a warning about context length to the system prompt. Something like:

REPL_SYSTEM_PROMPT = """You are answering a query with associated context that is available in your REPL environment.

Your REPL has:
- A `context` variable (string) containing important information about your query
- An `llm_query(query: str)` function to recursively call another LLM

IMPORTANT:
- Always inspect the `context` variable first before answering
- The context can be VERY LONG (millions of tokens). Never print it directly!
- Use Python to explore it safely: check length with len(), peek at slices, search with regex, chunk it
- Use `llm_query()` on manageable chunks to analyze semantics

Think step-by-step and use the REPL actively to build your answer."""

Want to test with a longer, more complex context that requires chunking and multiple recursive calls?

Based on the context, here's what made the bitter lesson click for the author:

The author's "aha moment" came from tinygrad and George Hotz's blog post "Can tinygrad win?"

The author had been struggling to understand what Sutton's Bitter Lesson really meant in practice. The breakthrough happened when George Hotz explained his vision for tinygrad: the entire stack of making code run fast on hardware is fundamentally a search problem across multiple layers—searching for better scheduling algorithms, faster GPU kernels, etc.

The key insight from tinygrad's approach was:

1. "Expose the underlying search problem spanning several orders of magnitude"
2. "Apply the state of the art in search. Burn compute."

This concrete example made the author realize that someone still has to expose and frame the search problem correctly. The Bitter Lesson doesn't mean engineers are obsolete—it means their role shifts from manually crafting solutions to building the infrastructure that enables effective search. Engineers design the problem representation, build evaluations, set constraints, and then let compute explore the solution space.

This realization transformed the "bitter" lesson into something positive: engineering work still matters, just differently.

Good idea! With a much longer context, the model should be forced to use llm_query() for semantic analysis rather than just printing chunks directly.

A few things to watch for when you test:

• Does it chunk the context intelligently?
• Does it actually call llm_query() on those chunks?
• How does it aggregate the results from multiple recursive calls?

Let me know what you find with the longer context!

'Source: #Gemini_3_Pro #SolveIt #Claudette\nTags: [[ProfitCopilot]] - [[SolveIt]] - [[Claudette]]\n\nShift + Tab inside () of functions opens the menu with explanations\nControl + Space: Changes Keyboard L'

Based on my search of the context provided, Claudette is not mentioned in this document.

The context is about "The Not-so Bitter Lesson" - a blog post discussing Richard Sutton's Bitter Lesson, tinygrad, DSPy, and how engineers should approach building systems that leverage search and compute rather than manual domain-specific solutions.

I don't have information about Claudette in the available context. If you're asking about something specific related to Claudette, could you provide more context or clarify what you're looking for?

You need to reinitialize the shell with the new context:

sh = prep_shell(notes)
outer_chat = Chat(model="claude-sonnet-4-5", tools=[run_repl], sp=REPL_SYSTEM_PROMPT)

Then you can query it with a question about your notes. How long is this context? Check with len(notes) to see if it's big enough to force the model to chunk and use llm_query().

Perfect! Now I have a clear understanding of what Claudette is. Let me provide you with a comprehensive answer.

What is Claudette?

Claudette is a high-level Python library wrapper for the Anthropic SDK, developed by the same team behind Solveit (Answer.AI). It's designed to make programmatic interactions with Claude AI models (like Opus 4.5 and Sonnet 4.5) significantly more ergonomic and powerful.

Key Characteristics:

1. Purpose: It provides tools to embed Claude's intelligence directly into application logic, rather than just for interactive development assistance.

2. Official Website: https://claudette.answer.ai/

3. Main Use Cases:

   • Structured Data Extraction: Using Client.structured() to extract information from unstructured documents and convert them into structured Python objects
   • Natural Language Interfaces: Creating NLI for complex applications using simplified tool definitions and Chat.toolloop()
   • Agentic Workflows: Building automation and intelligent workflows into applications

How It Differs from Interactive AI:

• Solveit Native AI = Interactive development assistant (pair programmer for coding, debugging, learning)
• Claudette Library = Programmatic execution tool (for building advanced features into your actual application)

In essence, Claudette is a developer-friendly Python library that makes it easier to integrate Claude AI capabilities into your applications with cleaner, more maintainable code than using the raw Anthropic SDK directly.

Check outer_chat.hist to see. Based on the output shown, it looks like it gave a final answer without showing the intermediate steps. But I suspect it still just used Python slicing/regex rather than llm_query().

The 100K character context might still be small enough that the model can handle chunks directly without needing recursive LLM calls. You might need an even larger context, or a task that explicitly requires semantic understanding of multiple chunks that need to be synthesized.

[{'role': 'user', 'content': 'What is Claudette?'},
 Message(content="I'll help you find out what Claudette is by exploring the context provided.", role='assistant', tool_calls=[{'index': 1, 'function': {'arguments': '{"python_code": "\\n# First, let\'s check what we have in the context\\nprint(f\\"Context type: {type(context)}\\")\\nprint(f\\"Context length: {len(context)}\\")\\nprint(f\\"First 500 characters:\\\\n{context[:500]}\\")\\n"}', 'name': 'run_repl'}, 'id': 'toolu_01PVSPZAKYJbmus1zdSkXeZt', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01PVSPZAKYJbmus1zdSkXeZt',
  'role': 'tool',
  'name': 'run_repl',
  'content': "Context type: <class 'str'>\nContext length: 100818\nFirst 500 characters:\nSource: #Gemini_3_Pro #SolveIt #Claudette\nTags: [[ProfitCopilot]] - [[SolveIt]] - [[Claudette]]\n\nShift + Tab inside () of functions opens the menu with explanations\nControl + Space: Changes Keyboard Language\nCmd + SHIFT + J: Code\nCmd + SHIFT + K: Note\nCmd + SHIFT + ;: Raw\nCmd + SHIFT + d: Use Thinking (toggle)\noption + }: `\n\t\t\t\tUsed to declare a function: \n\t\t\t\t- & `function_name`     and the AI in Solveit knows it exists and can call it and use as a tool.\n\n\n# Keyboard Shortcuts\n\n### Navigation\n\n\n"},
 Message(content=None, role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"python_code": "\\n# Let\'s search for mentions of \\"Claudette\\" in the context\\nimport re\\n\\n# Find all occurrences of Claudette with some context around it\\nclaudette_mentions = []\\nlines = context.split(\'\\\\n\')\\nfor i, line in enumerate(lines):\\n    if \'claudette\' in line.lower():\\n        # Get some context around the mention\\n        start = max(0, i-2)\\n        end = min(len(lines), i+3)\\n        claudette_mentions.append({\\n            \'line_num\': i,\\n            \'context\': \'\\\\n\'.join(lines[start:end])\\n        })\\n\\nprint(f\\"Found {len(claudette_mentions)} mentions of Claudette\\")\\nprint(\\"\\\\nFirst 5 mentions:\\")\\nfor i, mention in enumerate(claudette_mentions[:5]):\\n    print(f\\"\\\\n--- Mention {i+1} (line {mention[\'line_num\']}) ---\\")\\n    print(mention[\'context\'])\\n"}', 'name': 'run_repl'}, 'id': 'toolu_01MGTySKK7sdYDXwiZoWYiuS', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01MGTySKK7sdYDXwiZoWYiuS',
  'role': 'tool',
  'name': 'run_repl',
  'content': 'Found 30 mentions of Claudette\n\nFirst 5 mentions:\n\n--- Mention 1 (line 0) ---\nSource: #Gemini_3_Pro #SolveIt #Claudette\nTags: [[ProfitCopilot]] - [[SolveIt]] - [[Claudette]]\n\n\n--- Mention 2 (line 1) ---\nSource: #Gemini_3_Pro #SolveIt #Claudette\nTags: [[ProfitCopilot]] - [[SolveIt]] - [[Claudette]]\n\nShift + Tab inside () of functions opens the menu with explanations\n\n--- Mention 3 (line 494) ---\n    - **Why:** When documenting complex financial formulas (like weighted-average anti-dilution), using LaTeX allows for clear, mathematical notation in your notes.\n\n# Claudette (Anthropic SDK) Integration\nhttps://claudette.answer.ai/\n\n\n--- Mention 4 (line 495) ---\n\n# Claudette (Anthropic SDK) Integration\nhttps://claudette.answer.ai/\n\nYes, you should absolutely integrate `claudette` into your project.\n\n--- Mention 5 (line 497) ---\nhttps://claudette.answer.ai/\n\nYes, you should absolutely integrate `claudette` into your project.\n\n`claudette` is a high-level Python library wrapper for the Anthropic SDK, developed by the same team behind Solveit. It is designed to make programmatic interactions with Claude models (like Opus 4.5 and Sonnet 4.5, which you have access to in Solveit) significantly more ergonomic and powerful.\n'},
 Message(content=None, role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"python_code": "\\n# Let\'s get a more comprehensive view of the Claudette section\\n# Find where the Claudette section starts and extract it\\nstart_idx = context.find(\\"# Claudette (Anthropic SDK) Integration\\")\\nif start_idx != -1:\\n    # Find the next major section (starting with #)\\n    next_section = context.find(\\"\\\\n# \\", start_idx + 10)\\n    if next_section != -1:\\n        claudette_section = context[start_idx:next_section]\\n    else:\\n        claudette_section = context[start_idx:start_idx+5000]  # Get a good chunk\\n    \\n    print(f\\"Claudette section length: {len(claudette_section)}\\")\\n    print(\\"\\\\n\\" + \\"=\\"*80)\\n    print(claudette_section[:3000])  # Print first 3000 chars\\n"}', 'name': 'run_repl'}, 'id': 'toolu_01MXVGof7i4xorBzSdKpvPK4', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01MXVGof7i4xorBzSdKpvPK4',
  'role': 'tool',
  'name': 'run_repl',
  'content': 'Claudette section length: 2564\n\n================================================================================\n# Claudette (Anthropic SDK) Integration\nhttps://claudette.answer.ai/\n\nYes, you should absolutely integrate `claudette` into your project.\n\n`claudette` is a high-level Python library wrapper for the Anthropic SDK, developed by the same team behind Solveit. It is designed to make programmatic interactions with Claude models (like Opus 4.5 and Sonnet 4.5, which you have access to in Solveit) significantly more ergonomic and powerful.\n\nWhile Solveit provides the _environment_ for your development and interactive AI assistance, `claudette` provides the _tools_ to embed Claude\'s intelligence directly into the application logic of your equity modeler.\n\n### Solveit Native AI vs. Claudette: Understanding the Roles\n\nIt is crucial to distinguish between how you interact with AI during development versus how your application uses AI.\n\n1. **Solveit Native AI (The Pair Programmer):**\n    \n    - When you use Solveit\'s built-in Prompt messages, Ghost Text, or AI Modes, you are engaging in **interactive development**. This is ideal for learning Pandas, debugging your financial logic, and iterating rapidly (Phases 1-3 of your roadmap). The platform manages the context of your development session.\n        \n2. **Claudette Library (The Application Logic):**\n    \n    - When you import `claudette` inside a Python code cell, you are using it for **programmatic execution**. This is for building the advanced features of your tool, such as automation, agentic workflows, and data extraction (Phase 4).\n        \n\nYou should not use `claudette`\'s `Chat` class for basic assistance while coding, as Solveit\'s interface already handles this efficiently.\n\n### High-Leverage Use Cases for Claudette in VC Modeling\n\n`claudette` enables features that transform your Pandas model from a complex script into an intelligent analytical tool. Here are the highest-leverage use cases for your project:\n\n#### 1. Structured Data Extraction (The "Term Sheet Parser")\n\nA major bottleneck in VC is converting unstructured legal documents (Term Sheets, Stock Purchase Agreements) into structured data for modeling.\n\n- **The Feature:** `Client.structured()`\n    \n- **How You Use It:** You define a Python class representing the key economic terms (Valuation, Liquidation Preference, Option Pool Size). You then pass the text of the document to `claudette`, which uses Claude to extract the information and return a structured Python object.\n    \n- **Benefit:** This automates the ingestion of deal terms directly into your Pandas model, reducing manual entry and errors.\n    \n\nPython\n\n```\n'},
 Message(content=None, role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"python_code": "\\n# Let\'s get the complete Claudette section\\nstart_idx = context.find(\\"# Claudette (Anthropic SDK) Integration\\")\\nif start_idx != -1:\\n    # Find the next major section (starting with #)\\n    next_section = context.find(\\"\\\\n# \\", start_idx + 10)\\n    if next_section != -1:\\n        claudette_section = context[start_idx:next_section]\\n    else:\\n        claudette_section = context[start_idx:]  # Get rest of document\\n    \\n    print(f\\"Full Claudette section:\\\\n\\")\\n    print(claudette_section)\\n"}', 'name': 'run_repl'}, 'id': 'toolu_01Gvz8sJC9myr8pbS7MAUu5F', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01Gvz8sJC9myr8pbS7MAUu5F',
  'role': 'tool',
  'name': 'run_repl',
  'content': 'Full Claudette section:\n\n# Claudette (Anthropic SDK) Integration\nhttps://claudette.answer.ai/\n\nYes, you should absolutely integrate `claudette` into your project.\n\n`claudette` is a high-level Python library wrapper for the Anthropic SDK, developed by the same team behind Solveit. It is designed to make programmatic interactions with Claude models (like Opus 4.5 and Sonnet 4.5, which you have access to in Solveit) significantly more ergonomic and powerful.\n\nWhile Solveit provides the _environment_ for your development and interactive AI assistance, `claudette` provides the _tools_ to embed Claude\'s intelligence directly into the application logic of your equity modeler.\n\n### Solveit Native AI vs. Claudette: Understanding the Roles\n\nIt is crucial to distinguish between how you interact with AI during development versus how your application uses AI.\n\n1. **Solveit Native AI (The Pair Programmer):**\n    \n    - When you use Solveit\'s built-in Prompt messages, Ghost Text, or AI Modes, you are engaging in **interactive development**. This is ideal for learning Pandas, debugging your financial logic, and iterating rapidly (Phases 1-3 of your roadmap). The platform manages the context of your development session.\n        \n2. **Claudette Library (The Application Logic):**\n    \n    - When you import `claudette` inside a Python code cell, you are using it for **programmatic execution**. This is for building the advanced features of your tool, such as automation, agentic workflows, and data extraction (Phase 4).\n        \n\nYou should not use `claudette`\'s `Chat` class for basic assistance while coding, as Solveit\'s interface already handles this efficiently.\n\n### High-Leverage Use Cases for Claudette in VC Modeling\n\n`claudette` enables features that transform your Pandas model from a complex script into an intelligent analytical tool. Here are the highest-leverage use cases for your project:\n\n#### 1. Structured Data Extraction (The "Term Sheet Parser")\n\nA major bottleneck in VC is converting unstructured legal documents (Term Sheets, Stock Purchase Agreements) into structured data for modeling.\n\n- **The Feature:** `Client.structured()`\n    \n- **How You Use It:** You define a Python class representing the key economic terms (Valuation, Liquidation Preference, Option Pool Size). You then pass the text of the document to `claudette`, which uses Claude to extract the information and return a structured Python object.\n    \n- **Benefit:** This automates the ingestion of deal terms directly into your Pandas model, reducing manual entry and errors.\n    \n\nPython\n\n```\n'},
 Message(content=None, role='assistant', tool_calls=[{'index': 0, 'function': {'arguments': '{"python_code": "\\n# Let me use llm_query to get a comprehensive summary of what Claudette is\\nquery_text = f\\"\\"\\"\\nBased on the following context, please provide a clear and comprehensive answer to: \\"What is Claudette?\\"\\n\\nContext:\\n{claudette_section}\\n\\nPlease explain:\\n1. What Claudette is\\n2. Who developed it\\n3. What it\'s used for\\n4. How it relates to Solveit\\n5. Key features and use cases\\n\\"\\"\\"\\n\\nanswer = llm_query(query_text)\\nprint(answer)\\n"}', 'name': 'run_repl'}, 'id': 'toolu_01CTvS2azyGUzdi18C3VcLAz', 'type': 'function'}], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01CTvS2azyGUzdi18C3VcLAz',
  'role': 'tool',
  'name': 'run_repl',
  'content': '# What is Claudette?\n\n## 1. **What Claudette Is**\n\nClaudette is a **high-level Python library wrapper for the Anthropic SDK**. It serves as a more ergonomic and powerful interface for programmatically interacting with Claude AI models (such as Opus 4.5 and Sonnet 4.5) in your Python applications.\n\n## 2. **Who Developed It**\n\nClaudette was developed by **the same team behind Solveit** (specifically mentioned as being from answer.ai, based on the URL https://claudette.answer.ai/).\n\n## 3. **What It\'s Used For**\n\nClaudette is designed to **embed Claude\'s intelligence directly into application logic**. Rather than just chatting with AI during development, it allows you to build AI capabilities into the actual functionality of your software applications through programmatic execution.\n\n## 4. **How It Relates to Solveit**\n\nThere\'s an important distinction between the two:\n\n- **Solveit Native AI**: Provides the *environment* for interactive development and serves as your "pair programmer" through features like Prompt messages, Ghost Text, and AI Modes during coding sessions.\n\n- **Claudette**: Provides the *tools* to build AI-powered features into your actual application logic—it\'s what you import and use within your code to create intelligent functionality.\n\nThink of it this way: Solveit helps you *write code with AI assistance*, while Claudette helps you *write code that uses AI*.\n\n## 5. **Key Features and Use Cases**\n\n### Primary Feature: **Structured Data Extraction**\n\nThe most prominent feature mentioned is `Client.structured()`, which enables:\n\n- **Document Parsing**: Converting unstructured documents (like Term Sheets or Stock Purchase Agreements) into structured data\n- **Automated Data Ingestion**: Defining Python classes to represent key information, then having Claude extract that information from text and return structured Python objects\n- **Error Reduction**: Eliminating manual data entry by automating the extraction of critical information\n\n### Use Case Example: "Term Sheet Parser"\nIn venture capital modeling, you could use Claudette to automatically extract economic terms (valuation, liquidation preferences, option pool sizes) from legal documents and feed them directly into your analytical models—transforming raw documents into actionable data structures.\n'},
 Message(content='Based on my exploration of the context, I can now provide you with a comprehensive answer:\n\n## What is Claudette?\n\n**Claudette** is a high-level Python library wrapper for the Anthropic SDK, designed to make programmatic interactions with Claude AI models more ergonomic and powerful.\n\n### Key Details:\n\n1. **Developer**: Created by the same team behind Solveit (from answer.ai)\n\n2. **Purpose**: Claudette allows you to embed Claude\'s intelligence directly into your application logic, enabling you to build AI-powered features into your software programmatically.\n\n3. **Relationship to Solveit**: \n   - **Solveit** provides the *environment* for interactive AI-assisted development (like a pair programmer)\n   - **Claudette** provides the *tools* to build AI capabilities into your actual application code\n\n4. **Primary Use Cases**:\n   - **Structured Data Extraction**: Using `Client.structured()` to convert unstructured documents into structured Python objects\n   - **Document Parsing**: Automatically extracting information from legal documents, term sheets, or agreements\n   - **Agentic Workflows**: Building intelligent automation into applications\n   - **Data Ingestion**: Reducing manual entry and errors by automating data extraction\n\n5. **Example Application**: In venture capital modeling, Claudette can be used to build a "Term Sheet Parser" that automatically extracts economic terms (valuations, liquidation preferences, option pool sizes) from legal documents and feeds them directly into analytical models.\n\nIn essence, Claudette is a developer tool that makes it easier to integrate Claude AI\'s capabilities into Python applications for advanced, programmatic AI functionality.', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None})]

Actually, looking more closely at the history, it did use llm_query() in the last tool call! It found the Claudette section first with Python, then passed it to llm_query() for semantic summarization.

But you're right that it wasn't strictly necessary - the model could have just read the section directly since it was only ~2500 chars. The recursive call was more of a convenience than a requirement.

To really stress-test the RLM pattern, you'd want:

• A context so large that chunks still need semantic compression
• A question requiring synthesis across many sections
• Something where simple regex/slicing can't find the answer directly

Ideas:

1. Concatenate multiple large documents together
2. Ask a question that requires aggregating info scattered throughout
3. Use a context that's genuinely in the millions of characters

Do you have access to a much larger document, or want to artificially create a larger test context?

You're right that the tool called was run_repl - but look at the Python code inside that tool call:

query_result = llm_query(f"""
Based on the following context...
""")
print(query_result)

The outer LLM called run_repl, and the code it ran inside the REPL called llm_query(). That's exactly how it's supposed to work! The recursive LLM call happens through the REPL, not as a separate tool.

So the flow was:

1. Outer LLM → run_repl tool → executes Python code
2. Python code calls llm_query() → sub-LLM responds
3. Result returns through REPL output

Your RLM is working correctly! 🎉

Let me break this down step by step:

The tool call structure:

• name: 'run_repl' — This is the tool the outer LLM called
• function_call=None — This is unrelated, just an older API field
• arguments: '{"python_code": "..."}' — This is the Python code passed to run_repl

Now look at what's INSIDE the python_code argument:

query_result = llm_query(f"""
Based on the following context, please provide a comprehensive answer...
Context:
{claudette_section}
""")
print(query_result)

This is Python code that the REPL executed. And that code calls llm_query().

The proof it ran:

Look at the tool response that came back:

{'tool_call_id': 'toolu_01GUALP4Ec2N967u4MscSpQp',
 'role': 'tool',
 'name': 'run_repl',
 'content': '# What is Claudette? - Comprehensive Overview\n\n## 1. What Claudette Is\n\nClaudette is a **high-level Python library wrapper...'}

That response content is the output of llm_query() — it's a nicely formatted answer from the sub-LLM, not raw text from the context. The sub-LLM interpreted and structured the information.

In summary:

• Outer LLM has ONE tool: run_repl
• run_repl has access to llm_query() in its namespace
• Outer LLM writes Python code that calls llm_query()
• run_repl executes that code → llm_query() runs → sub-LLM responds

Does that make more sense?