[SL-3389] [Feature] MetricFlow doesn't support DBT_PROJECT_DIR environment variable

[mercedes-wu-wandb]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

commands from dbt project level:
mf health-checks

output:

AttributeError: 'Namespace' object has no attribute 'PROFILES_DIR'

Expected Behavior

✔ Health checks completed.
• ✅ SqlEngine.BIGQUERY - SELECT 1: Success!

Steps To Reproduce

1. Set up a dbt project with a valid profiles.yml.
2. Set the DBT_PROJECT_DIR environment variable to point to the project directory:
   export DBT_PROJECT_DIR=/path/to/dbt/project
3. Set the DBT_PROFILES_DIR environment variable to point to the profiles
   export DBT_PROFILES_DIR=/path/to/profiles/directory
4. Run the following command from the dbt directory:
   mf health-checks

Relevant log output

No response

Environment

- OS:Darwin Kernel Version 23.6.0
- Python: 3.12.3
- dbt: 1.8.9

Which database adapter are you using with dbt?

bigquery

Additional Context

I am trying to run metricflow on dbt-core

_SL-3389

[github-actions]

Thank you for your bug report! Our team is will be out of the office for Christmas and our Global Week of Rest, from December 25, 2024, through January 3, 2025.

We will review your issue as soon as possible after returning.
Thank you for your understanding, and happy holidays! 🎄🎉

If you are a customer of dbt Cloud, please contact our Customer Support team via the dbt Cloud web interface or email [email protected].

[dbeatty10]

Thanks for reaching out @mercedes-wu-wandb!

I'm going to transfer this issue to the metricflow repository for follow-up by another member of our team.

[dbeatty10]

Transferred from https://github.com/dbt-labs/dbt-core/issues/11179 to #1591.

[plypaul]

@mercedes-wu-wandb To clarify, mf works for you when run it from the dbt project directory?

[mercedes-wu-wandb]

Yes mf works in the dbt project directory but even with my environment variable set to DBT_PROJECT_DIR it does not seem to recognize it

analytics-py3.12➜  analytics git:(semantic-layer) export DBT_PROJECT_DIR=dbt/
analytics-py3.12➜  analytics git:(semantic-layer) mf health-checks           
❌ Unable to locate 'dbt_project.yml' in the current directory
In order to run the MetricFlow CLI, you must be running in the root directory of a working dbt project.
Please check out `https://docs.getdbt.com/reference/commands/init` if you want to get started on building a dbt project.
analytics-py3.12➜  analytics git:(semantic-layer) cd dbt
analytics-py3.12➜  dbt git:(semantic-layer) mf health-checks
ERROR:root:Logging exception
Traceback (most recent call last):
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/utils.py", line 116, in wrapper
    func(*args, **kwargs)
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/metricflow/telemetry/reporter.py", line 150, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/utils.py", line 170, in new_func
    return ctx.invoke(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/main.py", line 384, in health_checks
    cfg.setup()
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 54, in setup
    raise e
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 40, in setup
    self._dbt_project_metadata = dbtProjectMetadata.load_from_project_path(dbt_project_path)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/dbt_connectors/dbt_config_accessor.py", line 46, in load_from_project_path
    profile = load_profile(str(project_path), {})
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt/config/runtime.py", line 79, in load_profile
    profile = Profile.render(
              ^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt/config/profile.py", line 400, in render
    raw_profiles = read_profile(flags.PROFILES_DIR)
                                ^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'PROFILES_DIR'
Traceback (most recent call last):
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/utils.py", line 116, in wrapper
    func(*args, **kwargs)
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/metricflow/telemetry/reporter.py", line 150, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/utils.py", line 170, in new_func
    return ctx.invoke(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/main.py", line 384, in health_checks
    cfg.setup()
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 54, in setup
    raise e
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 40, in setup
    self._dbt_project_metadata = dbtProjectMetadata.load_from_project_path(dbt_project_path)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/dbt_connectors/dbt_config_accessor.py", line 46, in load_from_project_path
    profile = load_profile(str(project_path), {})
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt/config/runtime.py", line 79, in load_profile
    profile = Profile.render(
              ^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt/config/profile.py", line 400, in render
    raw_profiles = read_profile(flags.PROFILES_DIR)
                                ^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'PROFILES_DIR'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mercedeswu/wb-dev/analytics/.venv/bin/mf", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/utils.py", line 123, in wrapper
    click.echo(f"\nERROR: {str(e)}\nLog file: {cli_context.log_file_path}")
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 112, in log_file_path
    return pathlib.Path(self.dbt_project_metadata.project.log_path, "metricflow.log")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 97, in dbt_project_metadata
    return self._get_dbt_project_metadata()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mercedeswu/wb-dev/analytics/.venv/lib/python3.12/site-packages/dbt_metricflow/cli/cli_configuration.py", line 58, in _get_dbt_project_metadata
    raise RuntimeError(
RuntimeError: CLIConfiguration.setup() should have been called before accessing the configuration.

[plypaul]

It should be straightforward to add support for DBT_PROJECT_DIR, but did you see documentation that allows for that environment variable for the mf CLI? Looking to double check the classification of this as a bug vs. feature.

[mercedes-wu-wandb]

Hey @plypaul, so far i have not so I'm happy to change this to issue to a feature if that's helpful!

[plypaul]

Yes, that would be great. Thanks!

[mercedes-wu-wandb]

Changed to "[SL-3389] [Feature] MetricFlow doesn't support DBT_PROJECT_DIR environment variable"! Do you know when this would get worked on by chance?

[plypaul]

There was some relevant prior work that has already been done, and I'm aiming to get a PR out this week.

[mercedes-wu-wandb]

awesome, thanks a lot for the updates and looking forward to the PR!