Chapter_11_deployment

.github/workflows/run-dbt-prod.yml

@@ -0,0 +1,38 @@
+# .github/workflows/run-dbt-prod.yml
+name: run_dbt_prod
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  # schedule:
+  #   - cron: '0 8 * * *'
+
+env:
+  DBT_PROFILES_DIR: ./nyc_parking_violations
+  DBT_PROJECT_DIR: ./nyc_parking_violations
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Run dbt Prod
+      run: |
+        dbt debug
+        dbt compile --target prod
+        dbt run --target prod
+    - name: Test dbt Prod
+      run: |
+        dbt test --target prod

NOTICE

@@ -1,15 +1,56 @@
-Copyright 2022 LinkedIn Corporation
+Copyright 2024 LinkedIn Corporation
 All Rights Reserved.
 
 Licensed under the LinkedIn Learning Exercise File License (the "License").
 See LICENSE in the project root for license information.
 
 ATTRIBUTIONS:
-[PLEASE PROVIDE ATTRIBUTIONS OR DELETE THIS AND THE ABOVE LINE “ATTRIBUTIONS”]
+
+Pandas
+https://github.com/pandas-dev/pandas
+Copyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team
+Copyright (c) 2011-2021, Open source contributors.
+License: BSD 3-Clause
+https://opensource.org/licenses/BSD-3-Clause
 
 Please note, this project may automatically load third party code from external 
 repositories (for example, NPM modules, Composer packages, or other dependencies). 
 If so, such third party code may be subject to other license terms than as set 
 forth above. In addition, such third party code may also depend on and load 
 multiple tiers of dependencies. Please review the applicable licenses of the 
 additional dependencies.
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+
+PANDAS
+BSD 3-Clause License
+
+Copyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team
+All rights reserved.
+
+Copyright (c) 2011-2023, Open source contributors.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ""AS IS""
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1,8 +1,10 @@
-# COURSENAME
-This is the repository for the LinkedIn Learning course `course-name`. The full course is available from [LinkedIn Learning][lil-course-url].
+# Data Engineering with Data Build Tool (DBT)
+This is the repository for the LinkedIn Learning course `Data Engineering with Data Build Tool (DBT)`. The full course is available from [LinkedIn Learning][lil-course-url].
 
 ![course-name-alt-text][lil-thumbnail-url] 
 
+Data Build Tool (dbt) has quickly become an essential tool in many data stacks ranging from startups to big tech for managing data transformations. In this course, data engineer Mark Freeman helps you get started with setting up, running, and managing a dbt project via the open-source offering dbt Core. Learn how to install dbt Core, configure an environment for dbt, create and manage a dbt project, and deploy a dbt project in production. If you’re a data professional tasked with implementing dbt within your organization, recently joined a team utilizing dbt and need to upskill, or just want to learn about dbt to increase your competitiveness within the data job market, check out this course.
+
 _See the readme file in the main branch for updated instructions and information._
 ## Instructions
 This repository has branches for each of the videos in the course. You can use the branch pop up menu in github to switch to a specific branch and take a look at the course at that stage, or you can add `/tree/BRANCH_NAME` to the URL to go to the branch you want to access.
@@ -22,15 +24,9 @@ To resolve this issue:
     Add changes to git using this command: git add .
 	Commit changes using this command: git commit -m "some message"
 
-## Installing
-1. To use these exercise files, you must have the following installed:
-	- [list of requirements for course]
-2. Clone this repository into your local machine using the terminal (Mac), CMD (Windows), or a GUI tool like SourceTree.
-3. [Course-specific instructions]
-
 
 [0]: # (Replace these placeholder URLs with actual course URLs)
 
-[lil-course-url]: https://www.linkedin.com/learning/
-[lil-thumbnail-url]: http://
+[lil-course-url]: https://www.linkedin.com/learning/data-engineering-with-dbt
+[lil-thumbnail-url]: https://media.licdn.com/dms/image/D560DAQHnrbGtKKAabg/learning-public-crop_675_1200/0/1703199806902?e=2147483647&v=beta&t=Q6jKFTslSvhl3bYTkd0PuIEVnIx26lCa-XWVE5hCmm0
 

create_dbs.ipynb

@@ -0,0 +1,66 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "69aa603b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import duckdb\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "f4456138",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sql_query_import_1 = '''\n",
+    "CREATE OR REPLACE TABLE parking_violation_codes AS\n",
+    "SELECT *\n",
+    "FROM read_csv_auto(\n",
+    "    'data/dof_parking_violation_codes.csv',\n",
+    "    normalize_names=True\n",
+    "    )\n",
+    "'''\n",
+    "\n",
+    "sql_query_import_2 = '''\n",
+    "CREATE OR REPLACE TABLE parking_violations_2023 AS\n",
+    "SELECT *\n",
+    "FROM read_csv_auto(\n",
+    "    'data/parking_violations_issued_fiscal_year_2023_sample.csv',\n",
+    "    normalize_names=True\n",
+    "    )\n",
+    "'''\n",
+    "\n",
+    "with duckdb.connect('data/nyc_parking_violations.db') as con:\n",
+    "    con.sql(sql_query_import_1)\n",
+    "    con.sql(sql_query_import_2)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "TRAIN-EPAM-DBT",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

nyc_parking_violations/dbt_project.yml

@@ -34,4 +34,20 @@ models:
   nyc_parking_violations:
     # Config indicated by + and applies to all files under models/example/
     example:
+      +materialized: ephemeral
+    bronze: 
       +materialized: view
+    silver:
+      silver_parking_violation_codes:
+        +materialized: ephemeral
+      silver_parking_violations:
+        +materialized: ephemeral
+      silver_violation_tickets:
+        +materialized: view
+      silver_violation_vehicles:
+        +materialized: view
+    gold:
+      +materialized: table
+tests:
+  +store_failures: true
+

nyc_parking_violations/models/bronze/bronze_parking_violation_codes.sql

@@ -4,4 +4,4 @@ SELECT
     manhattan_96th_st_below,
     all_other_areas
 FROM
-    parking_violation_codes
+    parking_violation_codes

nyc_parking_violations/models/bronze/bronze_parking_violations.sql

@@ -18,6 +18,6 @@ SELECT
     violation_county,
     violation_legal_code,
     vehicle_color,
-    vehicle_year,
+    vehicle_year
 FROM
-    parking_violations_2023
+    parking_violations_2023    

nyc_parking_violations/models/docs/docs_block.md

@@ -0,0 +1,107 @@
+{% docs violation_code %}
+Code representing the specific parking violation.
+{% enddocs %}
+
+{% docs definition %}
+Description of the violation for a respective code.
+{% enddocs %}
+
+{% docs manhattan_96th_st_below %}
+The fee in $USD for a violation on or below Manhattan 96th Street.
+{% enddocs %}
+
+{% docs all_other_areas %}
+The fee in $USD for a violation not on or below Manhattan 96th Street.
+{% enddocs %}
+
+{% docs summons_number %}
+Unique identifier for each summons issued for a parking violation.
+{% enddocs %}
+
+{% docs registration_state %}
+The state where the vehicle is registered.
+{% enddocs %}
+
+{% docs plate_type %}
+The type of license plate.
+{% enddocs %}
+
+{% docs issue_date %}
+The date when the summons was issued.
+{% enddocs %}
+
+{% docs vehicle_body_type %}
+The body type of the vehicle involved in the violation.
+{% enddocs %}
+
+{% docs vehicle_make %}
+The make or brand of the vehicle.
+{% enddocs %}
+
+{% docs issuing_agency %}
+The agency that issued the summons.
+{% enddocs %}
+
+{% docs vehicle_expiration_date %}
+The date when the vehicle's registration expires.
+{% enddocs %}
+
+{% docs violation_location %}
+General location where the violation occurred.
+{% enddocs %}
+
+{% docs violation_precinct %}
+Precinct where the violation was identified.
+{% enddocs %}
+
+{% docs issuer_precinct %}
+Precinct of the officer or official who issued the summons.
+{% enddocs %}
+
+{% docs issuer_code %}
+Unique code identifying the issuer.
+{% enddocs %}
+
+{% docs issuer_command %}
+Command or unit of the issuer.
+{% enddocs %}
+
+{% docs issuer_squad %}
+Squad detail for the issuer.
+{% enddocs %}
+
+{% docs violation_time %}
+Time when the violation occurred.
+{% enddocs %}
+
+{% docs violation_county %}
+County where the violation took place.
+{% enddocs %}
+
+{% docs violation_legal_code %}
+Legal code associated with the violation.
+{% enddocs %}
+
+{% docs vehicle_color %}
+Color of the vehicle involved in the violation.
+{% enddocs %}
+
+{% docs vehicle_year %}
+Manufacturing year of the vehicle.
+{% enddocs %}
+
+{% docs fee_usd %}
+The fee charged for a parking violation, specified in USD. This fee varies depending on the location of the violation.
+{% enddocs %}
+
+{% docs is_manhattan_96th_st_below %}
+A boolean value indicating whether the violation occurred in Manhattan on or below 96th Street.
+{% enddocs %}
+
+{% docs ticket_count %}
+The total number of tickets issued for a specific violation code.
+{% enddocs %}
+
+{% docs total_revenue_usd %}
+The total revenue accumulated from tickets, based on the violation code. This sum is represented in USD.
+{% enddocs %}