Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Looker parsing improvements for liquid templating and view/model aliasing #17912

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

sam-mccarty-mavenclinic
Copy link
Contributor

@sam-mccarty-mavenclinic sam-mccarty-mavenclinic commented Sep 18, 2024

I would like to propose two improvements to resolve issues with Looker ingestion, one with liquid template variables in sql_table_name and another with aliased views and explores

Describe your changes:

Fixes #17911

  • I added python-liquid as a requirement
  • I added a call to try to render the sql table name as a liquid template before passing it to build the lineage request so that it may render the template as the base condition
  • I added a check for a non-empty from_ attribute for views to replace the name while building the lineage to correctly map to the actual view being joined
  • I replaced the existing conditions with a check for a non-empty view_name on the model so that the underlying view it is based on is included as part of the lineage, regardless of whether there are other views joined

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Copy link
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@harshach
Copy link
Collaborator

@sam-mccarty-mavenclinic thanks for the contribution

@sam-mccarty-mavenclinic
Copy link
Contributor Author

@harshach no problem, i was making some tweaks to see if i could get the lineage fixed for our specific looker implementation, but i am open to more generic approaches to these issues if this initial solution is a bit too opinionated

@sushi30
Copy link
Contributor

sushi30 commented Sep 18, 2024

Hi @sam-mccarty-mavenclinic, thank you for the PR.

  1. You didn't submit the change to setup.py therefore the errors in the CI:
 E   ModuleNotFoundError: No module named 'liquid'
  1. Can you give an example of this scenario? Maybe an example liquid template that we can test?

@sam-mccarty-mavenclinic
Copy link
Contributor Author

One sec, I will push a fix to the setup file. Happy to add a test with sample file with the templating included, can you point me to the right location to add my test?

@sushi30
Copy link
Contributor

sushi30 commented Sep 18, 2024

Take a look here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Looker does not support liquid templates or aliased views/models
3 participants