Skip to content

Commit a4bf301

Browse files
suryaiyer95claude
andauthored
feat: add --debug flag and DATAPILOT_DEBUG env var for verbose logging (#111)
API failures were reported as a one-line summary such as `Error in uploading the manifest.` with no way to see why. `APIClient` already recorded the HTTP status and error body via `logger.debug`, but `dbt/cli/cli.py` pinned the root logger at `INFO` at import time, so every `DEBUG` record was discarded and there was no flag or env var to change it. Verbose output is now opt-in two ways, both resolving to the same click parameter: the `--debug` flag, and the `DATAPILOT_DEBUG` environment variable for CI/CD pipelines where the command line is generated and hard to edit. * Add `datapilot/utils/logging_utils.py` with `configure_logging()`, `is_debug_enabled()` and `redact_url()`. * Add a `debug_option` decorator and wire it into `dbt project-health` and `dbt onboard`. * Replace the import-time `logging.basicConfig(level=logging.INFO)` in `dbt/cli/cli.py` and `mcp.py` with `configure_logging()`, so `DATAPILOT_DEBUG` is honoured even on paths that never reach a command callback. * Debug is sticky, so an import-time call at `INFO` cannot undo `--debug`. * `configure_logging()` only installs a handler when the root logger has none, leaving handlers owned by embedding applications intact. Redact credentials, since debug output is meant to be shared with support: * Presigned upload URLs carry an AWS key and signature in the query string. `redact_url()` strips it in both the `put()` request log and the `Received signed URL` log, keeping the object path. * Hold `urllib3` at `INFO` in debug mode; it logs each request line verbatim, presigned query string included. * Log the `GET` request params instead, which identify the integration id, environment and file type without exposing secrets. Verified against `api.myaltimate.com`: the API token never appears in debug output, and a successful upload logs no `AWSAccessKeyId` or `Signature`. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent acc982f commit a4bf301

11 files changed

Lines changed: 376 additions & 7 deletions

File tree

README.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,35 @@ The [--catalog-path] is an optional argument. If you don't specify a catalog pat
4242

4343
The [--config-path] is an optional argument. You can provide a yaml file with overrides for the default behavior of the insights.
4444

45+
#### Verbose / Debug Logging
46+
47+
By default, API failures are reported as a short summary such as `Error in uploading the manifest.`
48+
To see the underlying HTTP status codes and the error bodies returned by the API, enable debug
49+
logging with either the `--debug` flag or the `DATAPILOT_DEBUG` environment variable:
50+
51+
```shell
52+
datapilot dbt onboard --debug ...
53+
54+
# Or, for CI/CD pipelines where the command line is generated:
55+
export DATAPILOT_DEBUG=1
56+
datapilot dbt onboard ...
57+
```
58+
59+
This turns the generic message into an actionable one:
60+
61+
```
62+
DEBUG:APIClient:Sending GET request for tenant acme at url: https://api.myaltimate.com/dbt/v1/signed_url
63+
DEBUG:APIClient:Request params: {'dbt_core_integration_id': '2', 'dbt_core_integration_environment_type': 'DEV', 'file_type': 'manifest'}
64+
DEBUG:APIClient:HTTP Error: {'detail': 'dbt_core_integration with id:2 and env:DEV not found'} - Status code: 400
65+
Error in uploading the manifest.
66+
```
67+
68+
Note that `--dbt_core_integration_environment` is matched exactly, including case, against the
69+
environments configured for your integration.
70+
71+
Debug output is safe to share: the API token is never logged, and the credentials in presigned
72+
upload URLs are redacted to `?<redacted>`.
73+
4574
#### Generating Manifest and Catalog Files for dbt Projects
4675

4776
1. Generate Manifest File (manifest.json). Open your dbt project's root directory in a terminal or command prompt. Run `dbt compile`. This command generates manifest.json in the target folder under your dbt project directory structure.

src/datapilot/cli/decorators.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,27 @@
99
import click
1010
from dotenv import load_dotenv
1111

12+
from datapilot.utils.logging_utils import DEBUG_ENV_VAR
13+
from datapilot.utils.logging_utils import configure_logging
14+
15+
16+
def debug_option(f):
17+
"""Decorator adding a --debug flag, also settable via DATAPILOT_DEBUG."""
18+
19+
@click.option(
20+
"--debug",
21+
is_flag=True,
22+
default=False,
23+
envvar=DEBUG_ENV_VAR,
24+
help=f"Enable verbose debug logging, including API status codes and error responses. Can also be set with {DEBUG_ENV_VAR}=1.",
25+
)
26+
@wraps(f)
27+
def wrapper(*args, **kwargs):
28+
configure_logging(debug=kwargs.pop("debug", False))
29+
return f(*args, **kwargs)
30+
31+
return wrapper
32+
1233

1334
def load_config_from_file() -> Optional[Dict]:
1435
"""Load configuration from ~/.altimate/altimate.json if it exists."""

src/datapilot/clients/altimate/client.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
from requests.exceptions import RequestException
77
from requests.exceptions import Timeout
88

9+
from datapilot.utils.logging_utils import redact_url
10+
911

1012
class APIClient:
1113
def __init__(self, api_token="", base_url="", tenant=""):
@@ -36,6 +38,8 @@ def get(self, endpoint, params=None, timeout=None):
3638

3739
try:
3840
self.logger.debug(f"Sending GET request for tenant {self.tenant} at url: {url}")
41+
if params:
42+
self.logger.debug(f"Request params: {params}")
3943
response = requests.get(url, headers=headers, params=params, timeout=timeout)
4044

4145
# Check if the response was successful
@@ -67,7 +71,8 @@ def post(self, endpoint, data=None, timeout=None):
6771
def put(self, endpoint, data, timeout=None):
6872
url = f"{self.base_url}{endpoint}"
6973

70-
self.logger.debug(f"Sending PUT request for tenant {self.tenant} at url: {url}")
74+
# Presigned upload URLs carry AWS credentials in the query string.
75+
self.logger.debug(f"Sending PUT request for tenant {self.tenant} at url: {redact_url(url)}")
7176
response = requests.put(url, data=data, timeout=timeout)
7277
self.logger.debug(f"Received PUT response with status: {response.status_code}")
7378
return response

src/datapilot/clients/altimate/utils.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
from datapilot.clients.altimate.client import APIClient
1010
from datapilot.clients.altimate.constants import SUPPORTED_ARTIFACT_TYPES
11+
from datapilot.utils.logging_utils import redact_url
1112

1213

1314
def check_token_and_instance(
@@ -89,7 +90,7 @@ def onboard_file(api_token, tenant, dbt_core_integration_id, dbt_core_integratio
8990
if signed_url_data:
9091
signed_url = signed_url_data.get("url")
9192
file_id = signed_url_data.get("dbt_core_integration_file_id")
92-
api_client.log(f"Received signed URL: {signed_url}")
93+
api_client.log(f"Received signed URL: {redact_url(signed_url)}")
9394
api_client.log(f"Received File ID: {file_id}")
9495

9596
upload_response = upload_content_to_signed_url(file_path, signed_url)

src/datapilot/core/mcp_utils/mcp.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
import asyncio
22
import json
3-
import logging
43
import shutil
54
from dataclasses import dataclass
65

@@ -10,7 +9,9 @@
109
from mcp import StdioServerParameters
1110
from mcp.client.stdio import stdio_client
1211

13-
logging.basicConfig(level=logging.INFO)
12+
from datapilot.utils.logging_utils import configure_logging
13+
14+
configure_logging()
1415

1516

1617
@dataclass

src/datapilot/core/platforms/dbt/cli/cli.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
1-
import logging
2-
31
import click
42

53
from datapilot.cli.decorators import auth_options
4+
from datapilot.cli.decorators import debug_option
65
from datapilot.clients.altimate.utils import check_token_and_instance
76
from datapilot.clients.altimate.utils import get_all_dbt_configs
87
from datapilot.clients.altimate.utils import onboard_file
@@ -21,9 +20,11 @@
2120
from datapilot.core.platforms.dbt.utils import load_run_results
2221
from datapilot.core.platforms.dbt.utils import load_sources
2322
from datapilot.utils.formatting.utils import tabulate_data
23+
from datapilot.utils.logging_utils import configure_logging
2424
from datapilot.utils.utils import map_url_to_instance
2525

26-
logging.basicConfig(level=logging.INFO)
26+
# Honour DATAPILOT_DEBUG even for code paths that never reach a command callback.
27+
configure_logging()
2728

2829

2930
# New dbt group
@@ -33,6 +34,7 @@ def dbt():
3334

3435

3536
@dbt.command("project-health")
37+
@debug_option
3638
@auth_options
3739
@click.option(
3840
"--manifest-path",
@@ -134,6 +136,7 @@ def project_health(
134136

135137

136138
@dbt.command("onboard")
139+
@debug_option
137140
@auth_options
138141
@click.option(
139142
"--dbt_core_integration_id",
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
"""Logging configuration for the datapilot CLI.
2+
3+
Verbose output is opt-in and can be turned on two ways:
4+
5+
- the ``DATAPILOT_DEBUG`` environment variable, which is the option to reach for
6+
in CI/CD pipelines where the command line is generated and hard to edit
7+
- the ``--debug`` flag on individual commands
8+
9+
Both raise the root logger to ``DEBUG``, which surfaces the HTTP status codes and
10+
API error bodies that ``APIClient`` already records but normally discards.
11+
"""
12+
13+
import logging
14+
import os
15+
from typing import Optional
16+
from urllib.parse import urlsplit
17+
from urllib.parse import urlunsplit
18+
19+
DEBUG_ENV_VAR = "DATAPILOT_DEBUG"
20+
21+
# Values that count as "on" for DATAPILOT_DEBUG. Anything else (including "0",
22+
# "false" and the empty string) leaves debug logging off.
23+
_TRUTHY_VALUES = frozenset({"1", "true", "yes", "on"})
24+
25+
# Debug mode is sticky: a command-level `--debug` must not be undone by a later
26+
# call that happens to default to False.
27+
_debug_enabled = False
28+
29+
30+
def debug_enabled_via_env() -> bool:
31+
"""Return True when DATAPILOT_DEBUG is set to a truthy value."""
32+
return os.environ.get(DEBUG_ENV_VAR, "").strip().lower() in _TRUTHY_VALUES
33+
34+
35+
def configure_logging(debug: bool = False) -> bool:
36+
"""Configure root logging for the CLI and return whether debug mode is on.
37+
38+
Safe to call more than once; the group callback and the command callback both
39+
call it, and the more verbose of the two wins.
40+
"""
41+
global _debug_enabled
42+
_debug_enabled = _debug_enabled or debug or debug_enabled_via_env()
43+
level = logging.DEBUG if _debug_enabled else logging.INFO
44+
45+
root = logging.getLogger()
46+
# Only install a handler when nothing else has, so embedding applications
47+
# (and pytest's caplog) keep theirs. Setting the level is what actually
48+
# decides whether the DEBUG records get through.
49+
if not root.handlers:
50+
logging.basicConfig(level=level)
51+
root.setLevel(level)
52+
53+
# urllib3 logs each request line in full, which for a presigned S3 upload means
54+
# the AWS key and signature. Our own client logs the status codes and request
55+
# params, so nothing diagnostic is lost by holding urllib3 at INFO.
56+
logging.getLogger("urllib3").setLevel(logging.INFO)
57+
58+
return _debug_enabled
59+
60+
61+
def is_debug_enabled() -> bool:
62+
"""Return whether debug logging is currently enabled."""
63+
return _debug_enabled
64+
65+
66+
def redact_url(url: Optional[str]) -> str:
67+
"""Strip a URL's query string so it is safe to log.
68+
69+
Presigned upload URLs carry AWS credentials and a signature in the query
70+
string, and debug output routinely gets pasted into support tickets.
71+
"""
72+
if not url:
73+
return ""
74+
75+
parts = urlsplit(url)
76+
if not parts.query:
77+
return url
78+
79+
return urlunsplit((parts.scheme, parts.netloc, parts.path, "<redacted>", ""))

tests/cli/__init__.py

Whitespace-only changes.

tests/cli/test_debug_option.py

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
import logging
2+
3+
import pytest
4+
from click.testing import CliRunner
5+
6+
from datapilot.cli.main import datapilot
7+
from datapilot.utils import logging_utils
8+
from datapilot.utils.logging_utils import DEBUG_ENV_VAR
9+
10+
11+
@pytest.fixture(autouse=True)
12+
def _reset_logging_state(monkeypatch):
13+
monkeypatch.setattr(logging_utils, "_debug_enabled", False)
14+
monkeypatch.delenv(DEBUG_ENV_VAR, raising=False)
15+
original_level = logging.getLogger().level
16+
yield
17+
logging.getLogger().setLevel(original_level)
18+
19+
20+
def invoke_project_health(env=None, extra_args=()):
21+
"""project-health runs fully offline, so it exercises the flag without network access."""
22+
return CliRunner().invoke(
23+
datapilot,
24+
["dbt", "project-health", "--manifest-path", "tests/data/manifest_v11.json", *extra_args],
25+
env=env or {},
26+
)
27+
28+
29+
class TestDebugFlag:
30+
def test_flag_is_advertised_in_help(self):
31+
result = CliRunner().invoke(datapilot, ["dbt", "onboard", "--help"])
32+
33+
assert result.exit_code == 0
34+
assert "--debug" in result.output
35+
assert DEBUG_ENV_VAR in result.output
36+
37+
def test_flag_enables_debug_logging(self):
38+
result = invoke_project_health(extra_args=["--debug"])
39+
40+
assert result.exit_code == 0
41+
assert logging_utils.is_debug_enabled() is True
42+
assert logging.getLogger().level == logging.DEBUG
43+
44+
def test_without_flag_stays_at_info(self):
45+
result = invoke_project_health()
46+
47+
assert result.exit_code == 0
48+
assert logging_utils.is_debug_enabled() is False
49+
assert logging.getLogger().level == logging.INFO
50+
51+
52+
class TestDebugEnvVar:
53+
def test_env_var_enables_debug_logging(self):
54+
result = invoke_project_health(env={DEBUG_ENV_VAR: "1"})
55+
56+
assert result.exit_code == 0
57+
assert logging_utils.is_debug_enabled() is True
58+
assert logging.getLogger().level == logging.DEBUG
59+
60+
def test_env_var_off_stays_at_info(self):
61+
result = invoke_project_health(env={DEBUG_ENV_VAR: "0"})
62+
63+
assert result.exit_code == 0
64+
assert logging_utils.is_debug_enabled() is False
65+
assert logging.getLogger().level == logging.INFO
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
import logging
2+
from unittest.mock import patch
3+
4+
import pytest
5+
from requests import Response
6+
7+
from datapilot.clients.altimate.client import APIClient
8+
from datapilot.utils import logging_utils
9+
from datapilot.utils.logging_utils import DEBUG_ENV_VAR
10+
from datapilot.utils.logging_utils import configure_logging
11+
12+
PRESIGNED_URL = (
13+
"https://altimate-datapilot-freemium-prod.s3.amazonaws.com/prd/tenant%3Delastic/manifest.json"
14+
"?AWSAccessKeyId=AKIAVXVSOK5H3JQFSSU4&Signature=2LV9nU5JHPVOFz&Expires=1785271367"
15+
)
16+
17+
18+
@pytest.fixture(autouse=True)
19+
def _reset_logging_state(monkeypatch):
20+
monkeypatch.setattr(logging_utils, "_debug_enabled", False)
21+
monkeypatch.delenv(DEBUG_ENV_VAR, raising=False)
22+
original_level = logging.getLogger().level
23+
yield
24+
logging.getLogger().setLevel(original_level)
25+
26+
27+
def make_response(status_code=200):
28+
response = Response()
29+
response.status_code = status_code
30+
response._content = b""
31+
return response
32+
33+
34+
class TestPresignedUrlRedaction:
35+
def test_put_does_not_log_aws_credentials(self, caplog):
36+
client = APIClient()
37+
38+
with caplog.at_level(logging.DEBUG), patch("requests.put", return_value=make_response()):
39+
client.put(PRESIGNED_URL, data=b"{}")
40+
41+
assert "AKIAVXVSOK5H3JQFSSU4" not in caplog.text
42+
assert "2LV9nU5JHPVOFz" not in caplog.text
43+
# The object path is still logged, which is the part that aids debugging.
44+
assert "manifest.json" in caplog.text
45+
assert "<redacted>" in caplog.text
46+
47+
def test_get_logs_request_params(self, caplog):
48+
"""Params identify the integration id/env, and contain no secrets."""
49+
client = APIClient(api_token="secret-token", base_url="https://api.myaltimate.com", tenant="elastic") # noqa: S106
50+
params = {"dbt_core_integration_id": "2", "dbt_core_integration_environment_type": "PROD"}
51+
52+
with caplog.at_level(logging.DEBUG), patch("requests.get", return_value=make_response()):
53+
client.get("/dbt/v1/signed_url", params=params)
54+
55+
assert "dbt_core_integration_environment_type" in caplog.text
56+
assert "PROD" in caplog.text
57+
assert "secret-token" not in caplog.text
58+
59+
60+
class TestUrllib3Silencing:
61+
def test_urllib3_is_held_at_info_in_debug_mode(self):
62+
"""urllib3 logs whole request lines, presigned query string included."""
63+
configure_logging(debug=True)
64+
65+
assert logging.getLogger().level == logging.DEBUG
66+
assert logging.getLogger("urllib3").level == logging.INFO

0 commit comments

Comments
 (0)