Skip to content

bug: run generate errors when the original training job has holdout = 0 #276

@seayang-nv

Description

@seayang-nv

Priority Level

Medium (Annoying but has workaround)

Describe the bug

When a user sets holdout=0, the pipeline still generates a test.csv under dataset. The generation will continue without issues.

However, when a user wants to use the saved run-path to run a generation job, it will hit the following error

2026-03-20T16:27:17.128 | Nemo Safe Synthesizer |  user    |  error |  run.py: run_generate: 361 
Error in SafeSynthesizer: No columns to parse from file : {'error_type': 'EmptyDataError'} 
Traceback (most recent call last):
  File "/root/Safe-Synthesizer/.venv/bin/safe-synthesizer", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/src/nemo_safe_synthesizer/cli/run.py", line 371, in run_generate
    ss.load_from_save_path()
  File "/root/Safe-Synthesizer/src/nemo_safe_synthesizer/observability.py", line 820, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/src/nemo_safe_synthesizer/sdk/library_builder.py", line 219, in load_from_save_path
    self._test_df = pd.read_csv(test_path)
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1898, in _make_engine
    return mapping[engine](f, **self.options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Safe-Synthesizer/.venv/lib/python3.11/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 93, in __init__
    self._reader = parsers.TextReader(src, **kwds)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/parsers.pyx", line 581, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file

Steps/Code to reproduce bug

run a generation only job from a saved run path

Expected behavior

we should avoid pd.read_csv(test_path) when holdout was set 0

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions