This gives us the ability to say, "This is how you can use DImOS to discover all rooms in {standardized environment} in less than {some time frame}"
Example eval:
- agent initializes with no knowledge outside its current frame
- Is asked, "Go to the room next to you and tell me its extents"
- […multiple agent steps]
And we are testing for:
- correctness
- token usage
- amount of agent steps it took
- time to completion
Out of scope for this specific task, but forward thinking infrastructure:
- for every agent step, we are saving the entire request/response payload to LLM provider (most eval viewers assume the request sent to LLM providers across steps is exactly the same as the previous step + an additional item in the messages array, but that hasn't been true for most harnesses for over a year)
- some way to track this progress over the months/years
Synced from DIM-1521
This gives us the ability to say, "This is how you can use DImOS to discover all rooms in {standardized environment} in less than {some time frame}"
Example eval:
And we are testing for:
Out of scope for this specific task, but forward thinking infrastructure:
Synced from DIM-1521