Data Engineering | Kafka | ElasticSearch | ELK | Esper | Nifi

[raunak-r]

Nifi

• https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
• Github Repos - https://github.com/jfrazee/awesome-nifi

Integrating Nifi + Kafka

minifi, and scenarios of using multiple consumers vs partitions. - https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka

[raunak-r]

Esper

esper is the software to handle complex event processing, and EPL is the language (similar to SQL), to achieve this.

• getting started with esper, java code, EPL statements, handling events to cep engine - https://coffeeonesugar.wordpress.com/2009/07/21/getting-started-with-esper-in-5-minutes/#:~:text=Esper%20is%20available%20in%20Java,follows%20is%20written%20in%20Java.&text=0%2Fesper%2Flib%20folder.,pipe%20to%20our%20CEP%20engine.
• basics of esper, java code, usage on event handling - https://medium.com/@bruno.felix/complex-event-processing-with-esper-core-concepts-f97394b39c07

[raunak-r]

Redis and Celery with Django

Using Docker-Compose - https://testdriven.io/blog/django-and-celery/
Naked setup in Ubuntu - https://stackabuse.com/asynchronous-tasks-in-django-with-redis-and-celery

Running inside a simple docker container.

Protected Mode is required when connecting from a different server to a different server. Since the container is a different server from localhost, that's why required.

FROM        ubuntu:20.04
RUN         apt-get update && apt-get install -y redis-server
EXPOSE      6379
ENTRYPOINT  ["/usr/bin/redis-server", "--protected-mode no"]

Running Redis

• sudo docker run -p 6379:6379 --log-opt max-size=10m redis _# Run container
• sudo docker exec -it CID /bin/bash # access container bash

Deploying Celery | Dockerfile

Dockerfile - https://stackoverflow.com/questions/59651428/runtimewarningyoure-running-the-worker-with-superuser-privilegesthis-is-absol

# --uid and --gid options are required
# else this warning will show up RuntimeWarning:You're running the worker with superuser privileges:this is absolutely not recommended
- celery --app=app worker --loglevel=info --uid=nobody --gid=nogroup &

Testing

• redis-cli -h 127.0.0.1 -p 6379
• ping # will output PONG

Steps

1. Run Redis on a terminal as a docker container
2. Run Celery Worker on a different terminal
3. Call API from Postman.

[raunak-r]

ELK elastic - logstash - kibana

• When to use elasticsearch - https://qr.ae/pG1xMH

[raunak-r]

ElasticSearch

ES Setup

> echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
> sudo apt update
> sudo apt install elasticsearch

Optional - Port Configuration

> sudo nano /etc/elasticsearch/elasticsearch.yml

Run ES

> sudo systemctl start elasticsearch

Create Indexes

> curl -X PUT "localhost:9200/INDEX_NAME?pretty"

Verify Elastic Search

> List of all indexes
    curl -X GET "localhost:9200/_cat/indices?pretty"
> Cluster Health
    curl -X GET "localhost:9200/_cluster/health?pretty"

[raunak-r]

Data at Scale

• Nasdaq Case Study - Nasdaq Uses AWS to Pioneer Stock Exchange Data Storage in the Cloud - https://aws.amazon.com/solutions/case-studies/nasdaq-case-study/?pg=ln&sec=c