mlau_scraper

Current build status (via CircleCI):

A tool to extract insights from my Machine Learning Algorithms Unpacked publication, with the aim of exploring what content resonates with the audience, what doesn't, and recommendations on future topics that I should write about based on popular publications in the ML space.

The dashboard is live on Streamlit Community Cloud, linked here.

Key insights discovered

• The most popular ML-related newsletters discuss current-world trends in the AI space, including big tech and how to break into the industry.
• There does not appear to be a universal standard Substack upload cadence. That is to say, a weekly schedule does not necessarily beat out a fortnightly one- audience retention is far more dependent on post quality.

Limitations

• As my Substack is free to access, it only makes sense to compare with the non-paywalled posts of other newsletters. However, this means that statistics from other newsletters, such as upload frequencies, may not compare in a fully representative manner.
• The ML-related newsletters are filtered from Substack's "Technology" category, rather than being filtered from the full set of newsletters on the platform. This simplification allows for more efficient data retrieval, at the expense of potentially missing out on ML-related newsletters from other categories such as "Science".
• At the time of project development, the (unofficial) Substack API only allows for post filtering via simple strings like "machine learning", without the incorporation of more robust tools such as regex. Again, this may lead to representation bias in results.

Technical Components

Substack API call results saved to Supabase

The information on newsletters and their posts are accessed via the (unofficial) Substack API. These results are then saved to Supabase tables, which the live dashboard reads from.

Text processing + clustering

The project uses SpaCy for the the removal of stop words and emojis, as well as lemmatisation.

The clustering comes in two flavours:

• Simple DBSCAN, whose hyperparameters the user can play around with. This is purely for exploration, as I continue to build out my suite of written posts.
• HDBSCAN applied across my newsletter's posts, as well as the most popular ML-related newsletters on the platform. The goal is to investigate what other newsletters/posts my content naturally aligns with, as well as ML-related topics present in other clusters that are not captured in my writing to date, to help provide suggestions for future topics.

Investigation of the natural groupings of my posts, as well as outliers uncovered by DBSCAN and whether this correlates with top posts. The dashboard also allows users to train a DBSCAN model with their own choice of hyperparameters, with the results displayed.

Intra-cluster EDA

Following from the above point, the dashboard provides options for EDA, including word count, hour of upload and top word frequency by cluster.

CI/CD

• CI: linting checks are orchestrated w/ CircleCI, and are conducted upon any code pushes to GitHub.
• CD: The API data is retrieved via a GitHub Action CRON job set to run every Monday at 9:00am UTC. The job is executed by an Oracle VM, due to Client Forbidden errors arising from the use of standard Github-hosted runners.

How to run the webapp

The dashboard is already live at this link.

This project manages Python packages and their dependencies with uv.

To run the webapp locally, assuming uv is installed:

• Clone the repository
• Navigate to the root of the local project repository in the terminal and run

  uv venv
  

  and then

  uv sync
  

  to create the virtual environment called .venv and install all the project dependencies in it respectively.
• To run the webapp locally, run

  uv run streamlit run src/dashboard.py
  

  or, with Make installed, you can instead use

  make dashboard
  

To Develop

• Add HDBSCAN script to CRON job
• Experiment further with dashboard features to glean further insights
• Complete Pytest suite
• MLOps for HDBSCAN