Skip to content

Commit eea799f

Browse files
kohankhakikohankhaki
authored
Merge pull request #37 from VectorInstitute/agentic_task_gen
Agentic task generation feature (including solver) added.
2 parents 5735f3f + a379ecc commit eea799f

20 files changed

+2533
-94
lines changed

‎README.md‎

Lines changed: 98 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -73,21 +73,112 @@ Utilize the capability and the corresponding subject LLM score to select or gene
7373
```bash
7474
python -m src.run_lbo
7575
```
76-
7776
### Agentic Generation Scripts
7877

79-
Generate areas, capabilities, and tasks using multi-agent debate systems. Configure parameters in `src/cfg/agentic_config.yaml`.
78+
These scripts implement the multi-agent debate workflow for automated generation of areas, capabilities, tasks, and solutions.
79+
All configurable parameters are defined in `src/cfg/agentic_config.yaml`.
80+
81+
#### Understanding Pipeline Tags
82+
83+
The pipeline uses **auto-generated tags** to organize outputs from each step. Understanding how tags work is essential for running the pipeline:
84+
85+
- **Tag Format**: Tags are automatically generated timestamps in the format `_YYYYMMDD_HHMMSS` (e.g., `_20251104_143022`)
86+
- **Auto-Generation**: When you run a step (e.g., Generate Areas), the script automatically creates a tag and includes it in the output path
87+
- **Finding Tags**: After running a step, check the console output or the output directory to see the generated tag. The tag appears in the file path where outputs are saved
88+
- **Using Tags**: To run the next step in the pipeline, you need to specify the tag from the previous step's output:
89+
- Step 2 (Generate Capabilities) needs `areas_tag` from Step 1
90+
- Step 3 (Generate Tasks) needs `capabilities_tag` from Step 2
91+
- Step 4 (Generate Solutions) needs `tasks_tag` from Step 3
8092

93+
**Example Workflow**:
94+
1. Run `python -m src.agentic_area_generator` → outputs to `.../areas/_20251104_143022/areas.json`
95+
2. Use the tag `_20251104_143022` in the next step:
96+
```bash
97+
python -m src.agentic_capability_generator pipeline_tags.areas_tag=_20251104_143022
98+
```
99+
3. The capability generator outputs to `.../capabilities/_20251104_150315/...`
100+
4. Use this new tag for the next step, and so on.
101+
102+
---
103+
104+
#### 1. Generate Areas
105+
Generate domain areas using the scientist–moderator debate system:
81106
```bash
82-
# Generate capability areas
83107
python -m src.agentic_area_generator
108+
```
109+
110+
This step auto-generates a tag (e.g., `_20251104_143022`) and outputs the results to:
111+
112+
**Output location:**
113+
```
114+
~/<output_dir>/<domain>/<exp_id>/areas/<areas_tag>/areas.json
115+
```
116+
Where:
117+
- `<output_dir>` comes from `global_cfg.output_dir`
118+
- `<domain>` comes from `global_cfg.domain` (spaces replaced with underscores)
119+
- `<exp_id>` comes from `exp_cfg.exp_id`
120+
- `<areas_tag>` is the auto-generated tag for this run (use this tag in Step 2)
121+
122+
#### 2. Generate Capabilities
123+
Generate capabilities for each area:
124+
```bash
125+
# Use the areas_tag from Step 1 (Generate Areas) output
126+
python -m src.agentic_capability_generator pipeline_tags.areas_tag=_YYYYMMDD_HHMMSS pipeline_tags.resume_capabilities_tag=_YYYYMMDD_HHMMSS
127+
```
128+
129+
**Options:**
130+
- `pipeline_tags.areas_tag` specifies which set of areas to use when generating capabilities. This should be the `<areas_tag>` from the output of Step 1 (Generate Areas).
131+
- `pipeline_tags.resume_capabilities_tag` (optional) resumes a previous capability generation run.
132+
133+
This step auto-generates a new tag for the capabilities output.
134+
135+
**Output location:**
136+
```
137+
~/<output_dir>/<domain>/<exp_id>/capabilities/<capabilities_tag>/<area>/capabilities.json
138+
```
139+
Where:
140+
- `<capabilities_tag>` is the auto-generated tag for this run (use this tag in Step 3)
141+
142+
143+
#### 3. Generate Tasks
144+
Generate evaluation tasks for a specific capabilities tag:
145+
```bash
146+
# Use the capabilities_tag from Step 2 (Generate Capabilities) output
147+
python -m src.agentic_task_generator pipeline_tags.capabilities_tag=_YYYYMMDD_HHMMSS pipeline_tags.resume_tasks_tag=_YYYYMMDD_HHMMSS
148+
```
149+
150+
**Options:**
151+
- `pipeline_tags.capabilities_tag` specifies which set of capabilities to use when generating tasks. This should be the `<capabilities_tag>` from the output of Step 2 (Generate Capabilities).
152+
- `pipeline_tags.resume_tasks_tag` (optional) resumes a previous task generation run.
153+
154+
This step auto-generates a new tag for the tasks output.
84155

85-
# Generate capabilities for each area
86-
python -m src.agentic_capability_generator
156+
**Output location:**
157+
```
158+
~/<output_dir>/<domain>/<exp_id>/tasks/<tasks_tag>/[<area>]-[<capability>]/tasks.json
159+
```
160+
Where:
161+
- `<tasks_tag>` is the auto-generated tag for this run (use this tag in Step 4)
162+
163+
#### 4. Generate Solutions
164+
Solve generated tasks using the multi-agent debate system:
165+
```bash
166+
# Use the tasks_tag from Step 3 (Generate Tasks) output
167+
python -m src.agentic_task_solver pipeline_tags.tasks_tag=_YYYYMMDD_HHMMSS pipeline_tags.resume_solutions_tag=_YYYYMMDD_HHMMSS
168+
```
169+
170+
**Options:**
171+
- `pipeline_tags.tasks_tag` specifies which set of tasks to solve. This should be the `<tasks_tag>` from the output of Step 3 (Generate Tasks).
172+
- `pipeline_tags.resume_solutions_tag` (optional) resumes a previous solution generation run.
87173

88-
# Generate tasks for each capability
89-
python -m src.agentic_task_generator
174+
This step auto-generates a new tag for the solutions output.
175+
176+
**Output location:**
177+
```
178+
~/<output_dir>/<domain>/<exp_id>/task_solutions/<solutions_tag>/[<area>]-[<capability>]/<task_id>_solution.json
90179
```
180+
Where:
181+
- `<solutions_tag>` is the auto-generated tag for this run
91182

92183
### Wikipedia-Based Analysis Tools
93184

‎src/agentic_capability_generator.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ def main(cfg: DictConfig) -> None:
6363
error_msg = "No areas_tag provided. Please provide pipeline_tags.areas_tag=<tag> to specify which areas to use."
6464
log.warning(error_msg)
6565
span.update(
66-
level="WARNING",
66+
level="ERROR",
6767
status_message="Missing areas_tag",
6868
metadata={"areas_tag_missing": error_msg},
6969
)

‎src/agentic_task_generator.py‎

Lines changed: 98 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,39 +2,118 @@
22

33
import asyncio
44
import logging
5+
import os
56
import traceback
67

78
import hydra
9+
import openlit
10+
from langfuse import Langfuse
811
from omegaconf import DictConfig, OmegaConf
912

10-
from .task_generation import generate_tasks
13+
from src.task_generation import generate_tasks
1114

1215

16+
# Suppress OpenTelemetry console output
17+
os.environ["OTEL_LOG_LEVEL"] = "ERROR"
18+
os.environ["OTEL_METRICS_EXPORTER"] = "none"
19+
os.environ["OTEL_PYTHON_LOG_CORRELATION"] = "false"
20+
os.environ["OTEL_PYTHON_LOG_LEVEL"] = "ERROR"
21+
1322
log = logging.getLogger("agentic_task_gen")
1423

24+
lf = Langfuse()
25+
openlit.init(tracer=lf._otel_tracer, disable_batch=True, disable_metrics=True)
26+
1527

1628
@hydra.main(version_base=None, config_path="cfg", config_name="agentic_config")
1729
def main(cfg: DictConfig) -> None:
1830
"""Run the multi-agent task generation system."""
19-
log.info("Starting multi-agent task generation")
20-
log.info("Configuration:\n%s", OmegaConf.to_yaml(cfg, resolve=True))
21-
22-
# Check for capabilities_tag parameter
2331
capabilities_tag = cfg.pipeline_tags.capabilities_tag
24-
if capabilities_tag:
25-
log.info(f"Using capabilities from tag: {capabilities_tag}")
26-
else:
27-
log.warning(
28-
"No capabilities_tag provided. Please provide --pipeline_tags.capabilities_tag=<tag> to specify which capabilities to use."
29-
)
30-
return
31-
32-
try:
33-
asyncio.run(generate_tasks(cfg, capabilities_tag))
34-
except Exception as e:
35-
log.error(f"Task generation failed: {e}")
36-
log.error(f"Full traceback: {traceback.format_exc()}")
37-
raise
32+
resume_tag = getattr(cfg.pipeline_tags, "resume_tasks_tag", None)
33+
domain_name = cfg.global_cfg.domain
34+
exp_id = cfg.exp_cfg.exp_id
35+
36+
with lf.start_as_current_span(
37+
name=f"ace_agentic_task_generation:{domain_name}:{exp_id}"
38+
) as span:
39+
try:
40+
msg = "Starting multi-agent task generation"
41+
log.info(msg)
42+
span.update(metadata={"system_started": msg})
43+
44+
config_yaml = OmegaConf.to_yaml(cfg, resolve=True)
45+
msg = "Configuration loaded"
46+
log.info("Configuration:\n%s", config_yaml)
47+
span.update(
48+
metadata={
49+
"configuration_loaded": msg,
50+
"config": config_yaml,
51+
"domain": domain_name,
52+
"exp_id": exp_id,
53+
}
54+
)
55+
56+
if capabilities_tag:
57+
msg = f"Using capabilities from tag: {capabilities_tag}"
58+
log.info(msg)
59+
span.update(
60+
metadata={
61+
"capabilities_tag_found": msg,
62+
"capabilities_tag": capabilities_tag,
63+
}
64+
)
65+
else:
66+
error_msg = "No capabilities_tag provided. Please provide pipeline_tags.capabilities_tag=<tag> to specify which capabilities to use."
67+
log.warning(error_msg)
68+
span.update(
69+
level="ERROR",
70+
status_message="Missing capabilities_tag",
71+
metadata={"capabilities_tag_missing": error_msg},
72+
)
73+
return
74+
75+
if resume_tag:
76+
msg = f"Resuming task generation from tag: {resume_tag}"
77+
log.info(msg)
78+
span.update(
79+
metadata={"resume_tag_found": msg, "resume_tag": resume_tag}
80+
)
81+
82+
span.update_trace(
83+
metadata={
84+
"domain": domain_name,
85+
"exp_id": exp_id,
86+
"capabilities_tag": capabilities_tag,
87+
"resume_tag": resume_tag,
88+
"config": config_yaml,
89+
},
90+
tags=["agentic_task_generation", exp_id],
91+
)
92+
93+
asyncio.run(generate_tasks(cfg, capabilities_tag, lf, resume_tag))
94+
95+
msg = "Multi-agent task generation completed successfully"
96+
log.info(msg)
97+
span.update(metadata={"system_completed": msg})
98+
99+
except Exception as e:
100+
error_msg = f"Task generation failed: {e}"
101+
traceback_msg = f"Full traceback: {traceback.format_exc()}"
102+
103+
log.error(error_msg)
104+
log.error(traceback_msg)
105+
106+
span.update(
107+
level="ERROR",
108+
status_message=str(e),
109+
metadata={
110+
"system_error": error_msg,
111+
"error": str(e),
112+
"traceback": traceback_msg,
113+
},
114+
)
115+
116+
raise
38117

39118

40119
if __name__ == "__main__":

‎src/agentic_task_solver.py‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
"""Multi-agent debate system for solving generated tasks."""
2+
3+
import asyncio
4+
import logging
5+
import os
6+
import traceback
7+
8+
import hydra
9+
import openlit
10+
from langfuse import Langfuse
11+
from omegaconf import DictConfig, OmegaConf
12+
13+
from src.task_solver import solve_tasks
14+
15+
16+
# Suppress OpenTelemetry console output
17+
os.environ["OTEL_LOG_LEVEL"] = "ERROR"
18+
os.environ["OTEL_METRICS_EXPORTER"] = "none"
19+
os.environ["OTEL_PYTHON_LOG_CORRELATION"] = "false"
20+
os.environ["OTEL_PYTHON_LOG_LEVEL"] = "ERROR"
21+
22+
log = logging.getLogger("agentic_task_solver")
23+
24+
langfuse_client = Langfuse()
25+
openlit.init(
26+
tracer=langfuse_client._otel_tracer, disable_batch=True, disable_metrics=True
27+
)
28+
29+
30+
@hydra.main(version_base=None, config_path="cfg", config_name="agentic_config")
31+
def main(cfg: DictConfig) -> None:
32+
"""Run the multi-agent debate-based task solving system."""
33+
tasks_tag = cfg.pipeline_tags.get("tasks_tag")
34+
resume_tag = getattr(cfg.pipeline_tags, "resume_solutions_tag", None)
35+
domain_name = cfg.global_cfg.domain
36+
exp_id = cfg.exp_cfg.exp_id
37+
38+
with langfuse_client.start_as_current_span(
39+
name=f"ace_agentic_task_solver:{domain_name}:{exp_id}"
40+
) as span:
41+
try:
42+
msg = "Starting multi-agent debate-based task solver"
43+
log.info(msg)
44+
span.update(metadata={"system_started": msg})
45+
46+
config_yaml = OmegaConf.to_yaml(cfg, resolve=True)
47+
msg = "Configuration loaded"
48+
log.info("Configuration:\n%s", config_yaml)
49+
span.update(
50+
metadata={
51+
"configuration_loaded": msg,
52+
"config": config_yaml,
53+
"domain": domain_name,
54+
"exp_id": exp_id,
55+
}
56+
)
57+
58+
if tasks_tag:
59+
msg = f"Using tasks from tag: {tasks_tag}"
60+
log.info(msg)
61+
span.update(
62+
metadata={
63+
"tasks_tag_found": msg,
64+
"tasks_tag": tasks_tag,
65+
}
66+
)
67+
else:
68+
error_msg = "No tasks_tag provided. Please provide pipeline_tags.tasks_tag=<tag> to specify which tasks to solve."
69+
log.warning(error_msg)
70+
span.update(
71+
level="ERROR",
72+
status_message="Missing tasks_tag",
73+
metadata={"tasks_tag_missing": error_msg},
74+
)
75+
return
76+
77+
if resume_tag:
78+
msg = f"Resuming task solving from tag: {resume_tag}"
79+
log.info(msg)
80+
span.update(
81+
metadata={"resume_tag_found": msg, "resume_tag": resume_tag}
82+
)
83+
84+
span.update_trace(
85+
metadata={
86+
"domain": domain_name,
87+
"exp_id": exp_id,
88+
"tasks_tag": tasks_tag,
89+
"resume_tag": resume_tag,
90+
"config": config_yaml,
91+
},
92+
tags=["agentic_task_solver", exp_id],
93+
)
94+
95+
asyncio.run(solve_tasks(cfg, tasks_tag, langfuse_client, resume_tag))
96+
97+
msg = "Multi-agent debate-based task solving completed successfully"
98+
log.info(msg)
99+
span.update(metadata={"system_completed": msg})
100+
101+
except Exception as e:
102+
error_msg = f"Task solving failed: {e}"
103+
traceback_msg = f"Full traceback: {traceback.format_exc()}"
104+
105+
log.error(error_msg)
106+
log.error(traceback_msg)
107+
108+
span.update(
109+
level="ERROR",
110+
status_message=str(e),
111+
metadata={
112+
"system_error": error_msg,
113+
"error": str(e),
114+
"traceback": traceback_msg,
115+
},
116+
)
117+
118+
raise
119+
120+
finally:
121+
langfuse_client.flush()
122+
123+
124+
if __name__ == "__main__":
125+
main()

0 commit comments

Comments
 (0)