Mydbtcourse

.github/workflows/run-dbt-prod.yml

@@ -0,0 +1,38 @@
+# .github/workflows/run-dbt-prod.yml
+name: run_dbt_prod
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  # schedule:
+  #   - cron: '0 8 * * *'
+
+env:
+  DBT_PROFILES_DIR: ./nyc_parking_violations
+  DBT_PROJECT_DIR: ./nyc_parking_violations
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Run dbt Prod
+      run: |
+        dbt debug
+        dbt compile --target prod
+        dbt run --target prod
+    - name: Test dbt Prod
+      run: |
+        dbt test --target prod

.vscode/settings.json

@@ -17,7 +17,6 @@
   "files.autoSave": "afterDelay",
   "screencastMode.onlyKeyboardShortcuts": true,
   "terminal.integrated.fontSize": 18,
-  "workbench.activityBar.visible": true,
   "workbench.colorTheme": "Visual Studio Dark",
   "workbench.fontAliasing": "antialiased",
   "workbench.statusBar.visible": true

NOTICE

@@ -1,15 +1,56 @@
-Copyright 2022 LinkedIn Corporation
+Copyright 2024 LinkedIn Corporation
 All Rights Reserved.
 
 Licensed under the LinkedIn Learning Exercise File License (the "License").
 See LICENSE in the project root for license information.
 
 ATTRIBUTIONS:
-[PLEASE PROVIDE ATTRIBUTIONS OR DELETE THIS AND THE ABOVE LINE “ATTRIBUTIONS”]
+
+Pandas
+https://github.com/pandas-dev/pandas
+Copyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team
+Copyright (c) 2011-2021, Open source contributors.
+License: BSD 3-Clause
+https://opensource.org/licenses/BSD-3-Clause
 
 Please note, this project may automatically load third party code from external 
 repositories (for example, NPM modules, Composer packages, or other dependencies). 
 If so, such third party code may be subject to other license terms than as set 
 forth above. In addition, such third party code may also depend on and load 
 multiple tiers of dependencies. Please review the applicable licenses of the 
 additional dependencies.
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+
+PANDAS
+BSD 3-Clause License
+
+Copyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team
+All rights reserved.
+
+Copyright (c) 2011-2023, Open source contributors.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ""AS IS""
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1,8 +1,10 @@
-# COURSENAME
-This is the repository for the LinkedIn Learning course `course-name`. The full course is available from [LinkedIn Learning][lil-course-url].
+# Data Engineering with Data Build Tool (DBT)
+This is the repository for the LinkedIn Learning course `Data Engineering with Data Build Tool (DBT)`. The full course is available from [LinkedIn Learning][lil-course-url].
 
 ![course-name-alt-text][lil-thumbnail-url] 
 
+Data Build Tool (dbt) has quickly become an essential tool in many data stacks ranging from startups to big tech for managing data transformations. In this course, data engineer Mark Freeman helps you get started with setting up, running, and managing a dbt project via the open-source offering dbt Core. Learn how to install dbt Core, configure an environment for dbt, create and manage a dbt project, and deploy a dbt project in production. If you’re a data professional tasked with implementing dbt within your organization, recently joined a team utilizing dbt and need to upskill, or just want to learn about dbt to increase your competitiveness within the data job market, check out this course.
+
 _See the readme file in the main branch for updated instructions and information._
 ## Instructions
 This repository has branches for each of the videos in the course. You can use the branch pop up menu in github to switch to a specific branch and take a look at the course at that stage, or you can add `/tree/BRANCH_NAME` to the URL to go to the branch you want to access.
@@ -22,15 +24,9 @@ To resolve this issue:
     Add changes to git using this command: git add .
 	Commit changes using this command: git commit -m "some message"
 
-## Installing
-1. To use these exercise files, you must have the following installed:
-	- [list of requirements for course]
-2. Clone this repository into your local machine using the terminal (Mac), CMD (Windows), or a GUI tool like SourceTree.
-3. [Course-specific instructions]
-
 
 [0]: # (Replace these placeholder URLs with actual course URLs)
 
-[lil-course-url]: https://www.linkedin.com/learning/
-[lil-thumbnail-url]: http://
+[lil-course-url]: https://www.linkedin.com/learning/data-engineering-with-dbt
+[lil-thumbnail-url]: https://media.licdn.com/dms/image/D560DAQHnrbGtKKAabg/learning-public-crop_675_1200/0/1703199806902?e=2147483647&v=beta&t=Q6jKFTslSvhl3bYTkd0PuIEVnIx26lCa-XWVE5hCmm0
 

logs/dbt.log

@@ -0,0 +1,67 @@
+19:55:38.151755 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x75cb4c2b5720>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x75cb4a0fb190>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x75cb4a0fb1c0>]}
+
+
+============================== 19:55:38.190124 | 4ae9eb80-7a61-493f-b7d8-e2223201dba5 ==============================
+19:55:38.190124 [info ] [MainThread]: Running with dbt=1.6.1
+19:55:38.190813 [debug] [MainThread]: running dbt with arguments {'quiet': 'False', 'introspect': 'True', 'partial_parse': 'True', 'use_colors': 'True', 'write_json': 'True', 'warn_error': 'None', 'log_path': 'logs', 'send_anonymous_usage_stats': 'True', 'printer_width': '80', 'target_path': 'None', 'debug': 'False', 'invocation_command': 'dbt init', 'static_parser': 'True', 'version_check': 'True', 'cache_selected_only': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'use_experimental_parser': 'False', 'log_cache_events': 'False', 'fail_fast': 'False', 'profiles_dir': '/home/vscode/.dbt', 'log_format': 'default', 'no_print': 'None', 'indirect_selection': 'eager'}
+19:55:38.191400 [warn ] [MainThread]: [ConfigFolderDirectory]: Unable to parse dict {'dir': PosixPath('/home/vscode/.dbt')}
+19:55:38.191862 [info ] [MainThread]: Creating dbt configuration folder at 
+19:56:18.260747 [debug] [MainThread]: Starter project path: /home/vscode/.local/lib/python3.10/site-packages/dbt/include/starter_project
+19:56:18.270153 [info ] [MainThread]: 
+Your new dbt project "nyc_parking_violations" was created!
+
+For more information on how to configure the profiles.yml file,
+please consult the dbt documentation here:
+
+  https://docs.getdbt.com/docs/configure-your-profile
+
+One more thing:
+
+Need help? Don't hesitate to reach out to us via GitHub issues or on Slack:
+
+  https://community.getdbt.com/
+
+Happy modeling!
+
+19:56:18.270776 [info ] [MainThread]: Setting up your profile.
+19:56:57.348418 [info ] [MainThread]: No sample profile found for duckdb.
+19:56:57.349268 [debug] [MainThread]: Command `dbt init` succeeded at 19:56:57.349088 after 79.20 seconds
+19:56:57.349734 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x75cb4c2b5720>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x75cb49f946d0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x75cb49f94760>]}
+19:56:57.350208 [debug] [MainThread]: Flushing usage events
+14:30:50.361209 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7db8bf429720>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7db8bd263220>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7db8bd263190>]}
+
+
+============================== 14:30:50.366278 | f00b8a90-b999-4239-9ebb-d05c745067ce ==============================
+14:30:50.366278 [info ] [MainThread]: Running with dbt=1.6.1
+14:30:50.366949 [debug] [MainThread]: running dbt with arguments {'log_format': 'default', 'no_print': 'None', 'write_json': 'True', 'send_anonymous_usage_stats': 'True', 'log_cache_events': 'False', 'use_colors': 'True', 'warn_error': 'None', 'fail_fast': 'False', 'use_experimental_parser': 'False', 'introspect': 'True', 'quiet': 'False', 'debug': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'target_path': 'None', 'cache_selected_only': 'False', 'indirect_selection': 'eager', 'log_path': 'logs', 'static_parser': 'True', 'profiles_dir': '/home/vscode/.dbt', 'invocation_command': 'dbt debug', 'printer_width': '80', 'version_check': 'True', 'partial_parse': 'True'}
+14:30:50.367574 [info ] [MainThread]: dbt version: 1.6.1
+14:30:50.368018 [info ] [MainThread]: python version: 3.10.12
+14:30:50.368513 [info ] [MainThread]: python path: /usr/local/bin/python
+14:30:50.368967 [info ] [MainThread]: os info: Linux-6.8.0-1027-azure-x86_64-with-glibc2.31
+14:30:50.369470 [info ] [MainThread]: Using profiles dir at /home/vscode/.dbt
+14:30:50.369922 [info ] [MainThread]: Using profiles.yml file at /home/vscode/.dbt/profiles.yml
+14:30:50.370396 [info ] [MainThread]: Using dbt_project.yml file at /workspaces/data-engineering-with-data-build-tool-dbt-4458303/dbt_project.yml
+14:30:50.370867 [info ] [MainThread]: Configuration:
+14:30:50.371364 [info ] [MainThread]:   profiles.yml file [ERROR not found]
+14:30:50.371890 [info ] [MainThread]:   dbt_project.yml file [ERROR not found]
+14:30:50.372384 [info ] [MainThread]: Required dependencies:
+14:30:50.372956 [debug] [MainThread]: Executing "git --help"
+14:30:50.376109 [debug] [MainThread]: STDOUT: "b"usage: git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]\n           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]\n           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]\n           [--config-env=<name>=<envvar>] <command> [<args>]\n\nThese are common Git commands used in various situations:\n\nstart a working area (see also: git help tutorial)\n   clone     Clone a repository into a new directory\n   init      Create an empty Git repository or reinitialize an existing one\n\nwork on the current change (see also: git help everyday)\n   add       Add file contents to the index\n   mv        Move or rename a file, a directory, or a symlink\n   restore   Restore working tree files\n   rm        Remove files from the working tree and from the index\n\nexamine the history and state (see also: git help revisions)\n   bisect    Use binary search to find the commit that introduced a bug\n   diff      Show changes between commits, commit and working tree, etc\n   grep      Print lines matching a pattern\n   log       Show commit logs\n   show      Show various types of objects\n   status    Show the working tree status\n\ngrow, mark and tweak your common history\n   branch    List, create, or delete branches\n   commit    Record changes to the repository\n   merge     Join two or more development histories together\n   rebase    Reapply commits on top of another base tip\n   reset     Reset current HEAD to the specified state\n   switch    Switch branches\n   tag       Create, list, delete or verify a tag object signed with GPG\n\ncollaborate (see also: git help workflows)\n   fetch     Download objects and refs from another repository\n   pull      Fetch from and integrate with another repository or a local branch\n   push      Update remote refs along with associated objects\n\n'git help -a' and 'git help -g' list available subcommands and some\nconcept guides. See 'git help <command>' or 'git help <concept>'\nto read about a specific subcommand or concept.\nSee 'git help git' for an overview of the system.\n""
+14:30:50.376742 [debug] [MainThread]: STDERR: "b''"
+14:30:50.377235 [info ] [MainThread]:  - git [OK found]
+
+14:30:50.377734 [info ] [MainThread]: Connection test skipped since no profile was found
+14:30:50.378387 [info ] [MainThread]: 2 checks failed:
+14:30:50.378976 [info ] [MainThread]: dbt looked for a profiles.yml file in /home/vscode/.dbt/profiles.yml, but did
+not find one. For more information on configuring your profile, consult the
+documentation:
+
+https://docs.getdbt.com/docs/configure-your-profile
+
+
+14:30:50.379621 [info ] [MainThread]: Project loading failed for the following reason:
+ project path </workspaces/data-engineering-with-data-build-tool-dbt-4458303/dbt_project.yml> not found
+
+14:30:50.380444 [debug] [MainThread]: Command `dbt debug` failed at 14:30:50.380303 after 0.02 seconds
+14:30:50.381042 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7db8bf429720>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7db8bd150670>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7db8bd150640>]}
+14:30:50.381726 [debug] [MainThread]: Flushing usage events

nyc_parking_violations/.gitignore

@@ -0,0 +1,4 @@
+
+target/
+dbt_packages/
+logs/

nyc_parking_violations/.user.yml

@@ -0,0 +1 @@
+id: 70761850-c03a-4f73-809a-02729b3f9420

nyc_parking_violations/README.md

@@ -0,0 +1,15 @@
+Welcome to your new dbt project!
+
+### Using the starter project
+
+Try running the following commands:
+- dbt run
+- dbt test
+
+
+### Resources:
+- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
+- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
+- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support
+- Find [dbt events](https://events.getdbt.com) near you
+- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices

nyc_parking_violations/dbt_project.yml

@@ -0,0 +1,52 @@
+
+# Name your project! Project names should contain only lowercase characters
+# and underscores. A good package name should reflect your organization's
+# name or the intended use of these models
+name: 'nyc_parking_violations'
+version: '1.0.0'
+config-version: 2
+
+# This setting configures which "profile" dbt uses for this project.
+profile: 'nyc_parking_violations'
+
+# These configurations specify where dbt should look for different types of files.
+# The `model-paths` config, for example, states that models in this project can be
+# found in the "models/" directory. You probably won't need to change these!
+model-paths: ["models"]
+analysis-paths: ["analyses"]
+test-paths: ["tests"]
+seed-paths: ["seeds"]
+macro-paths: ["macros"]
+snapshot-paths: ["snapshots"]
+
+clean-targets:         # directories to be removed by `dbt clean`
+  - "target"
+  - "dbt_packages"
+
+
+# Configuring models
+# Full documentation: https://docs.getdbt.com/docs/configuring-models
+
+# In this example config, we tell dbt to build all models in the example/
+# directory as views. These settings can be overridden in the individual model
+# files using the `{{ config(...) }}` macro.
+models:
+  nyc_parking_violations:
+    # Config indicated by + and applies to all files under models/example/
+    example:
+      +materialized: ephemeral
+    bronze:
+      +materialized: view
+    silver:
+      silver_parking_violation_codes:
+        +materialized: ephemeral
+      silver_parking_violations:
+        +materialized: ephemeral
+      silver_violation_tickets:
+        +materialized: view
+      silver_violation_vehicles:
+        +materialized: view
+    gold:
+      +materialized: table
+tests:
+  +store_failures: true

nyc_parking_violations/models/bronze/bronze_parking_violation_codes.sql

@@ -0,0 +1,7 @@
+SELECT
+    code AS violation_code,
+    definition,
+    manhattan_96th_st_below,
+    all_other_areas
+FROM
+    parking_violation_codes

nyc_parking_violations/models/bronze/bronze_parking_violations.sql

@@ -0,0 +1,23 @@
+SELECT
+    summons_number,
+    registration_state,
+    plate_type,
+    issue_date,
+    violation_code,
+    vehicle_body_type,
+    vehicle_make,
+    issuing_agency,
+    vehicle_expiration_date,
+    violation_location,
+    violation_precinct,
+    issuer_precinct,
+    issuer_code,
+    issuer_command,
+    issuer_squad,
+    violation_time,
+    violation_county,
+    violation_legal_code,
+    vehicle_color,
+    vehicle_year,
+FROM
+    parking_violations_2023