Add pre-commit config and a new workflow to run the pre-commit hooks on any PR (#503)

* properly configure lint scripts

* add flake8 as a dev requirements

* add pre-commit config

* add pre-commit workflow

* properly configure path of misconfigured lint-fix in cli/

* lint fix all files after adding pre-commit

* bump up pre-commit

* run node 18 dependencies before 16, so node 16 can do linting for both cli and webserver

* lint fix we rebase

* docs update for pre-commit

* docs update and add autoflake

* add eslint in the contributing guidelines

* grammatical correction and remove eslint-next for cli

Author: thesujai

.flake8

@@ -0,0 +1,2 @@
+[flake8]
+max-line-length = 110

.github/ISSUE_TEMPLATE/bug.md

@@ -27,7 +27,7 @@ labels: bug
 <!-- Feel Free to Remove this section if you are a user and don't have the exact server details -->
 <!-- Additional Information which may be useful. Add whatever information is available, skip the field which are not available-->
 
-- Operating System: 
+- Operating System:
 - Deployment Method: <!-- snap/docker/tar/etc -->
 
 

.github/ISSUE_TEMPLATE/config.yml

@@ -5,4 +5,4 @@ contact_links:
     about: Hang out with the community of Engineering Managers, Tech Leads, and CTOs.
   - name: Get on a Call
     url: https://calendly.com/middleware-dhruv/30min
-    about: Get a tailored demo for your use cases
+    about: Get a tailored demo for your use cases

.github/ISSUE_TEMPLATE/feature.md

@@ -17,7 +17,7 @@ labels: enhancement
 - [ ] TODO 2
 - [ ] TODO 3
 
-## Further Comments / References  
+## Further Comments / References
 
 <!-- Any additional comments for this . -->
-<!-- Add References to similar features provided by any other platform(if possible)  -->
+<!-- Add References to similar features provided by any other platform(if possible)  -->

.github/pull_request_template.md

@@ -1,4 +1,4 @@
-## Linked Issue(s) 
+## Linked Issue(s)
 
 <!-- Link the issues being closed by or related to this PR. For example, you can use Closes #594 if this PR closes issue number 594 -->
 

.github/workflows/black.yml

@@ -30,4 +30,4 @@ jobs:
         run: python -m pip install black==24.3.0
 
       - name: Run Black Check
-        run: black . --check
+        run: black . --check

.github/workflows/build.yml

@@ -5,7 +5,7 @@ on:
     types: [published]
   push:
     branches: ["main"]
-    paths:  
+    paths:
         - 'Dockerfile'
         - 'backend/**'
         - 'database-docker/**'
@@ -125,7 +125,7 @@ jobs:
         run: |
           mkdir -p /tmp/digests
           digest="${{ steps.build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"  
+          touch "/tmp/digests/${digest#sha256:}"
 
 
       - name: Upload digest
@@ -186,11 +186,11 @@ jobs:
   stop-runner-arm:
     name: Stop self-hosted EC2 arm runner
     needs:
-      - start-runner-arm 
+      - start-runner-arm
       - build
       - merge
     runs-on: ubuntu-latest
-    if: ${{ always() }} 
+    if: ${{ always() }}
     steps:
       - name: Configure AWS credentials
         uses: aws-actions/configure-aws-credentials@v1
@@ -205,4 +205,3 @@ jobs:
           github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
           label: ${{ needs.start-runner-arm.outputs.label }}
           ec2-instance-id: ${{ needs.start-runner-arm.outputs.ec2-instance-id }}
-

.github/workflows/pre-commit.yml

@@ -0,0 +1,51 @@
+name: Linting-Precommit
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  pre_job:
+    name: Path match check
+    runs-on: ubuntu-latest
+    outputs:
+      should_skip: ${{ steps.skip_check.outputs.should_skip }}
+    steps:
+      - id: skip_check
+        uses: fkirc/skip-duplicate-actions@master
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          paths_ignore: '["**.po", "**.json"]'
+
+  linting:
+    name: All file linting
+    needs: pre_job
+    if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Node.js 18
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18
+
+      - name: Install dependencies for cli
+        run: yarn add eslint@^8.40.0 eslint-plugin-import@^2.29.0 eslint-plugin-prettier@^5.0.1 eslint-plugin-react@^7.29.4 eslint-plugin-unused-imports@^3.0.0 --dev
+        working-directory: cli
+
+      - name: Setup Node.js 16
+        uses: actions/setup-node@v3
+        with:
+          node-version: 16
+
+      - name: Install dependencies for web-server
+        run: yarn add eslint@^8.40.0 [email protected] eslint-plugin-import@^2.29.0 eslint-plugin-prettier@^5.0.1 eslint-plugin-react@^7.29.4 eslint-plugin-unused-imports@^3.0.0 --dev
+        working-directory: web-server
+
+      - name: Run pre-commit hooks
+        uses: pre-commit/[email protected]

.github/workflows/pytest.yml

@@ -33,7 +33,7 @@ jobs:
         pip install -r requirements.txt
 
     - name: Test with pytest
-      env:  
+      env:
         DB_HOST: "localhost"
         DB_NAME: "mhq-oss"
         DB_PASS: "postgres"

.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+exclude: (\.git/|\.tox/|\.venv/|build/|static/|dist/|node_modules/|__init__.py|app.py|sync_app.py)
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.0.1
+    hooks:
+      - id: trailing-whitespace
+      - id: check-yaml
+      - id: end-of-file-fixer
+  - repo: https://github.com/PyCQA/flake8
+    rev: 7.1.1
+    hooks:
+      - id: flake8
+        args:
+          - --ignore=E501,W503,W605
+
+  # I dont find any dedicated hook for this, so using a custom one
+  - repo: local
+    hooks:
+    - id: linting webserver
+      name: Linting-webserver
+      entry: bash -c 'cd web-server && yarn lint-fix'
+      language: system
+  - repo: local
+    hooks:
+    - id: linting cli
+      name: Linting-cli
+      entry: bash -c 'cd cli && yarn lint-fix'
+      language: system
+  - repo: https://github.com/PyCQA/autoflake
+    rev: v2.3.1
+    hooks:
+    - id: autoflake
+      args: [--remove-all-unused-imports, --in-place]
+  # black fixes code, so let it be at the last
+  - repo: https://github.com/psf/black
+    rev: 24.8.0
+    hooks:
+      - id: black

CONTRIBUTING.md

@@ -14,13 +14,53 @@ We use pull requests for code contributions. To submit a pull request:
 
 1. Fork the repository.
 2. Create a new branch for your feature or bug fix.
-3. Make your changes and commit them with clear and concise messages.
+3. Make your changes and commit them with clear and concise messages. Refer [Making Commits](#making-commits)
 4. Push your changes to your fork.
 5. Submit a pull request to the main repository, detailing the changes you've made and referencing any relevant issues.
 
 ## Code Style
 
-Please follow the existing code style and conventions used in the project. If you're unsure, feel free to ask for clarification in the pull request or issue comments.
+Please follow the existing code style and conventions used in the project. If you're unsure, feel free to ask for clarification in the pull request or issue comments. For linting and all refer [Making Commits](#making-commits)
+
+## Making Commits
+
+1. Pre-commit should be installed as a dev dependency already. If not then Create a virual environment [venv](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments) or [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation) and run the following command from project root dir:
+
+    ```
+    pip install -r backend/dev-requirements.txt --upgrade
+    ```
+    Also install the eslint for both `cli/` and `webserver/`. Please use node 16 for that
+    *NOTE*: If NVM is not installed, install using [nvm](https://github.com/nvm-sh/nvm?tab=readme-ov-file#installing-and-updating)
+    ```
+    nvm install 16
+    nvm use 16
+    ```
+    ```
+    cd cli
+    yarn add eslint@^8.40.0 eslint-plugin-import@^2.29.0 eslint-plugin-prettier@^5.0.1 eslint-plugin-react@^7.29.4 eslint-plugin-unused-imports@^3.0.0 --dev
+    ```
+    ```
+    cd web-server
+    yarn add eslint@^8.40.0 [email protected] eslint-plugin-import@^2.29.0 eslint-plugin-prettier@^5.0.1 eslint-plugin-react@^7.29.4 eslint-plugin-unused-imports@^3.0.0 --dev
+    ```
+    Then install it:
+    ```
+    pre-commit install
+    ```
+2. Then after adding the changes to staging, commit it normally. You will observe pre-commit hooks running.
+
+    *NOTE*: The pre-commit hooks modifies the files and you have to again add the modified changes to staging and then commit
+
+3. You can run the pre-commit without committing anything as
+    ```
+    pre-commit run --all-files
+    ```
+    or
+    ```
+    pre-commit run --files [path/to/file]
+    ```
+
+*NOTE*: Make sure the terminal is using node 16 for the linting before committing(Until we upgrade node for webserver)
 
 ## Labels
 
@@ -33,7 +73,7 @@ We use the following labels to categorize and prioritize issues:
 - `good first issue`: Indicates that the issue is suitable for newcomers to the project.
 - `priority`: high/medium/low: Indicates the priority level of the issue.
 - `discussion`: Indicates that the issue is open for discussion and feedback.
-  
+
 If you're creating a new issue, please apply the appropriate labels to help us better organize and address it.
 
 ## Feedback

README.md

@@ -114,9 +114,9 @@ They are:
 
 ### ☁️ Using GitPod
 
-Gitpod enables development on remote machines and helps you get started with Middleware if your machine does not support running the project locally. 
+Gitpod enables development on remote machines and helps you get started with Middleware if your machine does not support running the project locally.
 
-If you want to run the project locally you can [setup using docker](#-using-docker) or [setup everything manually](#-manual-setup). 
+If you want to run the project locally you can [setup using docker](#-using-docker) or [setup everything manually](#-manual-setup).
 
 1. Click the button below to open this project in Gitpod.
 
@@ -162,8 +162,8 @@ Make sure docker is running.
     - The postgres database can be accessed at host: `localhost`, port: `5434`, username: `postgres`, password: `postgres`, db name: `mhq-oss`.
     - The redis server can be accessed at host: `localhost`, port: `6385`.
 
-5. **View the logs**: Although the CLI tracks all logs, the logs of services running inside the container can be viewed in different terminals using the following commands: 
-     
+5. **View the logs**: Although the CLI tracks all logs, the logs of services running inside the container can be viewed in different terminals using the following commands:
+
    **Frontend logs**
    ```bash
    docker exec -it middleware-dev tail --lines 500 -f /var/log/web-server/web-server.log
@@ -205,13 +205,13 @@ To set up middleware locally, follow these steps:
 3. **Run Redis and Postgres Containers**:
 
     If you don't have docker installed, please install docker [over here](https://docs.docker.com/get-docker/)
-  
+
     Run the following commands to run Postgres and Redis using docker.
 
      ```bash
      cd database-docker && docker-compose up -d
      ```
-  
+
     If you don't prefer Docker, you can choose to install [Postgres](https://www.postgresql.org/download/) and [Redis](https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/) manually.
 
     Once you are done with using or developing Middleware, you can choose to close these running container. (NOTE: Don't do this if you are following this document and trying to run Middleware.)
@@ -222,7 +222,7 @@ To set up middleware locally, follow these steps:
     ```
 
 4. **Generate Encryption keys**:
-    
+
     Generate encryption keys for the project by running the following command in the project root directory:
 
     ```bash
@@ -232,14 +232,14 @@ To set up middleware locally, follow these steps:
 5. **Backend Server Setup**
 
     - Install python version `3.11.6`
-    
+
       - For this you can install python from [over here](https://www.python.org/downloads/) if you don't have it on your machine.
       - Install pyenev
-  
+
         ```bash
         git clone https://github.com/pyenv/pyenv.git ~/.pyenv
         ```
-      
+
       - Add pyenv to your shell's configuration file (.bashrc, .bash_profile, .zshrc, etc.):
 
         ```bash
@@ -251,7 +251,7 @@ To set up middleware locally, follow these steps:
         ```
         source ~/.bashrc
         ```
-    - Move backend directory to create a virtual environment  
+    - Move backend directory to create a virtual environment
 
       ```bash
       cd backend
@@ -263,7 +263,7 @@ To set up middleware locally, follow these steps:
       ```bash
       . venv/bin/activate
         ```
-  
+
     - Install Dependencies
 
       ```bash
@@ -286,22 +286,22 @@ To set up middleware locally, follow these steps:
       ```
 
     - Start the backend servers
-    
+
       - Change Directory to analytics_server
         ```bash
         cd analytics_server
         ```
-     
+
       - For backend analytics server:
         ```bash
         flask --app app --debug run --port 9696
         ```
-     
+
       - For backend sync server:
         ```bash
         flask --app sync_app --debug run --port 9697
         ```
-        NOTE: Open this sync sever in a new terminal window after activating the virtual environment only after starting analytics server. 
+        NOTE: Open this sync sever in a new terminal window after activating the virtual environment only after starting analytics server.
 
 6. **Web Server Setup**
 
@@ -316,7 +316,7 @@ To set up middleware locally, follow these steps:
      cd web-server
      yarn
      ```
-   
+
    - Start the web-server
      ```bash
      yarn dev
@@ -413,7 +413,7 @@ We look forward to your part in keeping Middleware secure!
 
 
 ## License
- 
+
 This project is licensed under the [Apache 2.0](https://github.com/middlewarehq/middleware/blob/main/LICENSE) License - see the LICENSE.md file for details.
 
 

SECURITY.md

@@ -7,7 +7,7 @@ This document outlines the security practices and considerations for the middlew
 We appreciate your help in keeping middleware secure. If you discover a security vulnerability, please report it responsibly by following these steps:
 
 * **Email:** Send an email to [[email protected]](mailto:[email protected]).
-* **Include:** 
+* **Include:**
     * A detailed description of the vulnerability, including the affected components and potential impact.
     * Steps to reproduce the vulnerability (if possible), including any specific code snippets or configurations.
     * Any additional information that may help us understand and address the vulnerability.

backend/analytics_server/mhq/api/deployment_analytics.py

@@ -21,10 +21,10 @@
 )
 from mhq.service.deployments.analytics import get_deployment_analytics_service
 from mhq.service.query_validator import get_query_validator
-from mhq.store.models import SettingType, EntityType, Team
+from mhq.store.models import SettingType, EntityType
 from mhq.store.models.code.filter import PRFilter
 from mhq.store.models.code.pull_requests import PullRequest
-from mhq.store.models.code.repository import OrgRepo, TeamRepos
+from mhq.store.models.code.repository import OrgRepo
 from mhq.store.models.code.workflows.filter import WorkflowFilter
 from mhq.service.deployments.models.models import (
     Deployment,