Problem w/ context window exceeds

[ndebard]

Getting context window exceeds on main branch from jan26th. Not repeatedly, but it's happened twice so far since yesterday in ~20 runs perhaps.

Error: Error code: 400 - {'error': {'message': 'Your input exceeds the context window of this model. Please adjust your input and try again.', 'type': 'invalid_request_error', 'param': 'input', 'code': 'context_length_exceeded'}}
Traceback (most recent call last):
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/examples/two_agent_examples/plan_execute/plan_execute_from_yaml.py", line 1678, in main
    prev_summary = run_substeps(
                   ^^^^^^^^^^^^^
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/examples/two_agent_examples/plan_execute/plan_execute_from_yaml.py", line 1188, in run_substeps
    sub_result = executor.invoke(
                 ^^^^^^^^^^^^^^^^
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/src/ursa/agents/base.py", line 389, in invoke
    return self._invoke_engine(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/src/ursa/agents/base.py", line 326, in _invoke_engine
    return invoke_method(normalized, config=config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/src/ursa/agents/base.py", line 549, in _invoke
    return self.compiled_graph.invoke(input, config=config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langgraph/pregel/main.py", line 3071, in invoke
    for chunk in self.stream(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langgraph/pregel/main.py", line 2646, in stream
    for _ in runner.tick(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langgraph/pregel/_runner.py", line 167, in tick
    run_with_retry(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langgraph/pregel/_retry.py", line 42, in run_with_retry
    return task.proc.invoke(task.input, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langgraph/_internal/_runnable.py", line 656, in invoke
    input = context.run(step.invoke, input, config, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 5557, in invoke
    return self.bound.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4880, in invoke
    return self._call_with_config(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2058, in _call_with_config
    context.run(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 435, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4737, in _invoke
    output = call_func_with_variable_args(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 435, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/src/ursa/agents/execution_agent.py", line 549, in safety_check
    self.llm.invoke(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 5557, in invoke
    return self.bound.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 402, in invoke
    self.generate_prompt(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 1121, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 931, in generate
    self._generate_with_cache(
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 1233, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_openai/chat_models/base.py", line 1386, in _generate
    raise e
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/langchain_openai/chat_models/base.py", line 1368, in _generate
    raw_response = self.root_client.responses.with_raw_response.create(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/openai/_legacy_response.py", line 364, in wrapped
    return cast(LegacyAPIResponse[R], func(*args, **kwargs))
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/openai/resources/responses/responses.py", line 866, in create
    return self._post(
           ^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/openai/_base_client.py", line 1259, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndebard/miniconda3/envs/ursa-jan26/lib/python3.11/site-packages/openai/_base_client.py", line 1047, in request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'Your input exceeds the context window of this model. Please adjust your input and try again.', 'type': 'invalid_request_error', 'param': 'input', 'code': 'context_length_exceeded'}}
During task with name 'safety_check' and id '5b38ab2e-ca88-b458-5d9f-cfde6bd8b828'
Traceback (most recent call last):
  File "/Users/ndebard/laptop_local_workspace/ursa-where-i-make-changes/ursa-jan26/ursa/examples/two_agent_examples/plan_execute/plan_execute_from_yaml.py", line 1889, in <module>
    final_output, workspace = main(
    ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)
[ERROR] Command failed for task 4522001f with return code 1

[ndebard]

@mikegros and @awadell1 please think about this and how we should address this. Let's talk.

[mikegros]

Do you know what it was doing when you hit that? I wonder if it tried to read a document that by itself exceed the context window? Or something that generated stdout/stderr that exceeded the context like a long compiler message.

Just curious for figuring out the right fix.

[ndebard]

@mikegros - just happened twice more in 2 terminals at more or less the same time (I happened to be watching them). You can see in screenshot. In these cases, appeared to be right after writing out a piece of Python code? Very late into agentic run.

[ndebard]

3rd fail I was able to catch - seems about the same thing but it's definitely hard to tell. I'll see if I can add some debugging.