An agent that will accept a description of an analysis and use experiment knowledge bases to look up datasets for your analysis. It will work by looking up the nearest neighbors to your analsysis and retrieving from the research notes the datasets used.
To begin with, we are collecting some benchmarks based on existing analyses and recording the actual datasets used. We will also record the Analysis ID so we can exclude those datasets from the benchmarking.
Each benchmark will be a subdirectory of the benchmarks/<experiment_name> directory. Name the subdirectory after the analysis ID and include two text files: prompt.txt: A prompt that describes the analysis. datasets.txt: A list of datasets used in the analysis.
We can have multiple prompts for each analysis to explore different ways of describing the analysis.
We will evaluate the agent's performance by comparing the datasets it retrieves with the actual datasets used in the analysis.
We will use MLFLow to track these questions along with the context used to answer them.