Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nfiann-manifest #6933

Open
wants to merge 7 commits into
base: current
Choose a base branch
from
Open

Nfiann-manifest #6933

wants to merge 7 commits into from
+18 −0

Conversation

nataliefiann
Copy link
Contributor

@nataliefiann nataliefiann commented Feb 20, 2025 •
edited by github-actions bot
Loading

What are you changing in this pull request and why?

I have created this PR raised by Jerco for this Git issue: #3482 to advise dbt overwrites the manifest.json file during parsing. If a user references --state information from the target/ directory, they may encounter a warning indicating that the saved manifest was not found, since the manifest.json is overwritten during parsing.

Closes: #3482

Checklist

  • I have reviewed the Content style guide so my content adheres to these guidelines.
  • The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the version a whole page and/or version a block of content guidelines.
  • I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."
  • The content in this PR requires a dbt release note, so I added one to the release notes page.

🚀 Deployment available! Here are the direct links to the updated files:

@nataliefiann nataliefiann requested a review from a team as a code owner February 20, 2025 15:17
@vercel Vercel
Copy link

vercel bot commented Feb 20, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
docs-getdbt-com ✅ Ready (Inspect) Visit Preview Feb 21, 2025 4:33pm

@github-actions github-actions bot added content Improvements or additions to content size: small This change will take 1 to 2 days to address labels Feb 20, 2025
mirnawong1
mirnawong1 reviewed Feb 20, 2025
View reviewed changes
website/docs/reference/node-selection/state-comparison-caveats.md Outdated

<Lightbox src="/img/docs/reference/saved-manifest-not-found.png" title="Saved manifest not found error" />

During the next run, the `target/manifest.json` is overwritten. Following this, it's read again to detect changes, but no changes are found.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what run?

mirnawong1
mirnawong1 reviewed Feb 20, 2025
View reviewed changes
website/docs/reference/node-selection/state-comparison-caveats.md Outdated

#### Workaround

- Move the manifest from the `target/` folder to a dedicated folder such as `state/` between step 2 of the run, and in step 4 of the run, use the command `mkdir state && mv target/manifest.json state/manifest.json`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont understand the steps referenced here? what's step 2, step 4?

mirnawong1
mirnawong1 reviewed Feb 20, 2025
View reviewed changes
website/docs/reference/node-selection/state-comparison-caveats.md Outdated

Additionally, it's not recommended to set `--state` and `--target-path` to the same path while using state-dependent features like `--defer` and `state:modified` as it can lead to non-idempotent behavior, which may not work as expected.

#### Workaround
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a workaround or are we saying 'to avoid having it overwritten'?

mirnawong1
mirnawong1 reviewed Feb 20, 2025
View reviewed changes
website/docs/reference/node-selection/state-comparison-caveats.md Outdated

<Lightbox src="/img/docs/reference/saved-manifest-not-found.png" title="Saved manifest not found error" />

During the next run, the `target/manifest.json` is overwritten. Following this, it's read again to detect changes, but no changes are found.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is read again?

mirnawong1
mirnawong1 requested changes Feb 20, 2025
View reviewed changes
Copy link
Contributor

@mirnawong1 mirnawong1 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey Natalie, thanks for this! have a dded some questions/suggestions:

  • clarification on what the steps in your referenced text
  • shortening the title
  • removing the word 'now'
  • add this to the 1.5 migration guide. @jtcohen6 mentions we should add to the v1.5 migration guide, which is missing this pr.
  • also should this content be added to the node selector methods doc as mentioned in the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mirnawong1 mirnawong1 mirnawong1 requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
content Improvements or additions to content size: small This change will take 1 to 2 days to address
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[v1.5] Manifest overwritten in subsequent parses, which can affect "stateful" worfklows
2 participants
@nataliefiann @mirnawong1