Dev.to Mirror—The Set-and-Forget AI Crawler

🔗 Live Site: anchildress1.github.io/devto-mirror

This Copilot generated utility helps make your Dev.to blogs more discoverable by search engines by automatically generating and hosting a mirror site with generous robots.txt rules. Avoiding Dante’s DevOps and the maintenance headache. This is a simple html, no frills approach with a sitemap and robots.tx—that's it (although I'm slowly working through enhancements). If you're like me and treat some comments as mini-posts, you can selectively pull in the ones that deserve their own page.

Note

I'm slowly accepting that one or two brave souls might actually read my strong (and usually correct) opinions. 😅 I'm also always looking for ways to improve AI results across the board, because... well, somebody has to. 🧠

The internet already changed—blink and you missed it. We don't Google anymore; we ask ChatGPT (the wise ones even ask for sources). 🤖

• When I searched: my Dev.to showed up just as expected
• When I asked Gemini and ChatGPT the same thing: crickets. 🦗

So yeah, obvious disconnect... Also, I'm not hosting a blog on my domain (I'm a backend dev; hosting a pretty blog + analytics sounds like a relaxing afternoon with Dante's DevOps. Hard pass. 🔥🫠), but I still want control of robots.txt.

Enter the five-minute ChatGPT fix: a tiny static mirror with canonicals back to Dev.to—no domain, no analytics—just (practically) instantly crawlable 😉🐜.

P.S. "Five minutes" usually means two hours. Acceptable losses. 😅 And seriously, writing this blurb took longer than the code. 🤨 Alright.... 3 hours (it took me an hour to get the picture just right, enough anyway) and lots of follow up work. Still worth it! 😅

—Ashley 🦄

Repo Stuff

What is this?

Auto-generates a static mirror of your Dev.to blog with generous robots.txt for AI crawlers. Simple HTML, sitemap, canonical links—zero maintenance.

Quick Setup ⚡

1. Fork this repo
2. Set variables (Settings → Actions → Variables):

• DEVTO_USERNAME – your Dev.to username
• GH_USERNAME – your GitHub username

3. (Optional) Set API key (Settings → Actions → Secrets):

• DEVTO_KEY – for private/draft posts

4. Delete gh-pages branch if it exists
5. Update comments.txt file (or delete it completely)
6. Run workflow Actions → Generate and Publish Dev.to Mirror Site → Run workflow
7. Enable Pages → Settings → Pages → Deploy from branch → gh-pages

This will automatically pull new content from Dev every Wednesday at 9:40 AM EST.

Important

Deploying with a gh-pages branch is somewhat deprecated, but it was the most straightforward way to keep a running history. This eliminates unnecessary calls to the Dev API every week. If you want to force a complete refresh, you can manually trigger the publish.yaml workflow with the force_full_regen option.

How it works

Fetches posts via Dev.to API (incremental updates via last_run.txt). Generates plain HTML files with canonical links back to Dev.to, AI-specific optimizations, plus sitemap and robots.txt. Optional: include comments as standalone pages via comments.txt or delete it entirely.

Force full regeneration: Actions → Generate and Publish Dev.to Mirror Site → Run workflow with force_full_regen: true.

Warning

I've tinkered some with moving robots.txt and llms.txt to the base-level repo, but haven't been able to make it work yet. Research says it's possible, but I'm either doing it all wrong or AI lied to me. 🤷‍♀️ So, the Google Search Console have a difficult time locating these files currently. Otherwise, there doesn't seem to be any problems with keeping those files here.

Local Development

git clone https://github.com/anchildress1/devto-mirror.git
cd devto-mirror

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or update to latest version
uv self --no-python-downloads update

# Install dependencies and lefthook hooks
make install

# Configure environment
cp .env.example .env
# Edit .env with your DEVTO_USERNAME and GH_USERNAME

# Run validation
make ai-checks

Documentation 📚

Additional documentation is available in the docs/ directory:

• Development Guide - Local development setup and commands
• CI/CD Guide - GitHub Actions workflows and deployment
• Security Analysis - Security recommendations and workflows
• Migration Plan - AI optimization refactoring progress

License 📄

Every project has to have a stack of fine print somewhere. Keep going, keep going, keep going... Here's mine, as painless as possible:

You know where the license is, but I'll sum it up: this is not open source (even though you can still do just about anything you want with it). As long as you're not turning it into the next big SaaS or selling subscriptions in the cloud, then have fun! Else, you've gotta ask me first.

Basically? This project's got boundaries. Be cool, don't try to sneak it into a product launch, and we'll get along just fine. 😘