Tech Assessment Property Data Scraping

Technical Specs for QA

• Python 3.10* or later
• postgresql 12.6.* or later

Backend Setup

Setting up a virtual environment with Python and pip

• clone the repo
• install a virtual env and activate it: python -m venv env; env/Scripts/activate[Windows]
• install a virtual env and activate it: virtualenv --no-site-packages env; source env/bin/activate[Linux/iOS]
• install requirements: pip install -r requirements.txt
• copy the configuration file: cp main/config/example.development.cfg > main/config/development.cfg.

Setting up the Database with PostgreSQL

Setup the PostgreSQL database (minimum version 12.*)

psql -U postgres
=# CREATE USER postgres WITH PASSWORD 'database-name';
=# CREATE DATABASE database-name;
=# GRANT ALL PRIVILEGES ON DATABASE database-name TO postgres;
=# \q

Or you can use the Graphic interface on windows

Deploying database changes

• App uses Flask-Migrate (which uses Alembic) to handle database migrations.
• Intialize db flask db init
• This will autogenerate a change. Double check that it make sense. To apply it on your machine, run flask db migrate
• To upgrade all versions, this ultimately delete all tables flask db upgrade

Pytest

• Powershell Run: $ENV:PYTHONPATH = "<name-of-project>"
• Linux/Mac to set the environment path export PYTOHNPATH=<name-of-project>

Then run pytest for simple test summary or pytest -vv for detailed test summary

Redis Setup

Redis is required for caching and background task management.

Install Redis:

1. On Mac OS X: brew install redis
2. On Windows: Use the Redis MSI installer
3. On Ubuntu: sudo apt-get update && sudo apt-get install redis-server
4. Update your development.cfg to include Redis configuration: REDIS_URL = "redis://<redis-host>"

Deployment Strategy

Deployment

• You can use Heroku, which supports a wide range of programming languages and has an easy deployment that links to your github repository and waits on push commits before deploying your work. There is another option that uses the Heroku-CLI if you don't want to use github.
• You can also setup a PostgreSQL database and Redis instance on Heroku and update your SQLALCHEMY_DATABASE_URI variable in the development.cfg
• You can setup Heroku Moonitoring to notify you of any issues.

Scheduling Tasks

• You can use the Astronomer To get started:

1. Initialize an Astronomer project by running astro dev init
2. Start Airflow locally by running astro dev start
3. Navigate to localhost:8080 in your browser and you should see the tutorial DAGs there
4. To stop the containers run astro dev stop alternatively to destroy the container and images run astro dev kill

Framework

• For the API I used flask-Restful to intall pip install flask-restful which runs on Flask. Flask is a micro web framework written in Python. to install pip install flask

To access the api-docs http://localhost:5000/swagger