Skip to content

Error handling with stats and files with different dimensions. #19

@faeglas

Description

@faeglas

Currently when using the stats command and a file_id with multiple files, the error thrown is this:

Traceback (most recent call last):
  File "./probtest.py", line 57, in <module>
    cli()
  File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/engine/stats.py", line 91, in stats
    file_specification,
  File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/engine/stats.py", line 15, in create_stats_dataframe
    df = df_from_file_ids(file_id, input_dir, file_specification)
  File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/util/dataframe_ops.py", line 150, in df_from_file_ids
    fid_dfs = unify_time_index(fid_dfs)
  File "/scratch/mch/fgessler/ICON/GLORI_test_small/CPU/externals/probtest/util/dataframe_ops.py", line 174, in unify_time_index
    df = df.reindex(columns=unique_times, level=time_multiindex_index)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/util/_decorators.py", line 309, in wrapper
    return func(*args, **kwargs)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/frame.py", line 4036, in reindex
    return super().reindex(**kwargs)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/generic.py", line 4464, in reindex
    axes, level, limit, tolerance, method, fill_value, copy
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/frame.py", line 3877, in _reindex_axes
    columns, method, copy, level, fill_value, limit, tolerance
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/frame.py", line 3925, in _reindex_columns
    allow_dups=False,
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/generic.py", line 4532, in _reindex_with_indexers
    copy=copy,
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/internals/managers.py", line 1285, in reindex_indexer
    self.axes[axis]._can_reindex(indexer)
  File "/scratch/d1000/apps/buildbot/bb-venv/lib64/python3.6/site-packages/pandas/core/indexes/base.py", line 3292, in _can_reindex
    raise ValueError("cannot reindex from a duplicate axis")
ValueError: cannot reindex from a duplicate axis

We could make a simple check to make sure all files a user is trying to use stats on have the same dimensions and give a more meaningful and lightweight error message. E.g: "Error: File a and File b have different dimensions. They must have the same dimensions."

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions