Simulation Environment #1880
Simulation Environment #1880
Conversation
csansoon
force-pushed
the
simulation-environment
branch
5 times, most recently
from
c0fd1dc to
0f0f47f
Compare
...b/src/components/ChatWrapper/Message/Content/ToolCall/_components/ToolCard/SimulationTag.tsx
Outdated
Show resolved
Hide resolved
| }) | ||
| } else if (streamManager.source === LogSources.Evaluation) { | ||
| definition.execute = instrumentToolHandler(mockClientToolResult, { | ||
| } else if (streamManager.source === LogSources.Playground) { |
There was a problem hiding this comment.
I'm confused, why are we changing here from logSources evaluation to playground when adding now simulations?
There was a problem hiding this comment.
It's not a change, it's GitHub's diff.
I REMOVED the condition for LogSources.Evaluation. The fact that the tool gets simulated or not does not depend where it is being called from anymore. Instead, it depends on the "simulationSettings".
| parameters, | ||
| source, | ||
| simulationSettings: { | ||
| simulateToolResponses: true, |
There was a problem hiding this comment.
This should not be true by default no?
There was a problem hiding this comment.
This job previously called runDocumentAtCommitWithAutoToolResponses, which is the old way to run the prompt with simulations enabled.
Not sure what this job is being used for and what should it do, but this setting ensures it still behaves as before.
csansoon
force-pushed
the
simulation-environment
branch
6 times, most recently
from
2a9003a to
f47fe57
Compare
| 'An optional list of all the required dependencies to run the script. Adding dependencies will severely increase the execution time, so do not include them unless required.', | ||
| ), | ||
| }), | ||
| outputSchema: z.string(), |
There was a problem hiding this comment.
Nah this is for the internal use of the simulation only
csansoon
force-pushed
the
simulation-environment
branch
from
f47fe57 to
cc16e67
Compare
| @@ -0,0 +1,206 @@ | |||
| --- | |||
csansoon
force-pushed
the
simulation-environment
branch
from
cc16e67 to
9cc4fb8
Compare
Improve Simulation Environment
In this PR, the simulation (tool mocking) pipeline has been reworked and fully integrated into the core execution flow.
The previously independent
runDocumentAtCommitWithAutoToolResponsesservice has been removed.Instead, the simulation environment is now seamlessly integrated into the existing
runDocumentAtCommitservice.A new
simulationSettingsattribute has been introduced torunDocumentAtCommit.This attribute, which is undefined by default, configures the simulation behavior.
When
simulateToolResponsesis enabled, all tools are automatically routed through the simulation layer instead of their real implementations.This simulation applies to the following tool categories:
This integration ensures all logic is shared with the regular execution service, simplifying the overall architecture and eliminating redundant code.
Additionally, the toolSource data for generated tools now includes a new
simulatedattribute, indicating whether the tool response was produced within a simulation.This allows the Tool Call UI to display a “Simulated” badge whenever a tool’s output originates from a simulated execution, regardless of its source.
With these changes, we can potentially: