🎬 Como TV Scraper

A powerful web scraper for Como TV content with Telegram bot control

Features • Quick Start • Documentation • Telegram Bot

---

✨ Features

🔍 Web Scraping 📦 Scrape 1-1000 categories automatically 📄 Extract detailed content information 🔗 Fetch related content (up to 10 items) 🌐 Multi-language support (IT/EN) 🔄 Automatic retry mechanism ⏱️ Smart rate limiting

🤖 Telegram Bot 📱 Remote control via Telegram 📊 Real-time progress tracking 📥 Direct file downloads ⚙️ Live configuration updates 🧹 Integrated data cleaning 🌍 Proxy support

🛡️ Additional Features

• ✅ Multi-threading (4 parallel workers for 4x speed)
• ✅ Proxy support (HTTP/HTTPS) with authentication
• ✅ 3x retry mechanism for all requests (SSLError, Timeout, etc.)
• ✅ Duplicate removal & data cleaning
• ✅ Fancy progress bar with live stats & ETA
• ✅ CSV export with custom fields
• ✅ Environment-based configuration (.env)
• ✅ Error handling & logging

---

🚀 Quick Start

1️⃣ Installation

# Clone repository
git clone https://github.com/Veloruze/como-tv-scraper.git
cd como-tv-scraper

# Install dependencies
pip install -r requirements.txt

2️⃣ Configuration

Create .env file from template:

cp .env.example .env

Edit .env with your settings:

# Telegram Bot (optional - only if using bot)
TELEGRAM_BOT_TOKEN=your_bot_token_here

# Proxy Configuration
PROXY_URL=http://username:password@proxy.com:port
PROXY_ENABLED=true

# Scraper Settings
LANGUAGE_CODE=it-IT
CATEGORY_START=1
CATEGORY_END=1000
MAX_WORKERS=4

3️⃣ Run Scraper

# Scrape categories
python como_tv_scraper.py

# Clean data (remove duplicates)
python clean_data.py

# Scrape details
python detail_scraper.py

---

🤖 Telegram Bot

Control everything from Telegram with an interactive bot interface!

Setup Telegram Bot

1. Create Bot

   • Open Telegram → Search @BotFather
   • Send /newbot and follow instructions
   • Copy the bot token

2. Configure

   TELEGRAM_BOT_TOKEN=1234567890:ABCdefGHIjklMNOpqrsTUVwxyz

3. Run Bot

   python telegram_bot.py

Bot Commands

Command	Description
/start	Show main menu with interactive buttons
/setlang <code>	Set language (it-IT or en_US)
/setproxy <url>	Configure proxy URL
/setrange <start> <end>	Set category range
/toggleproxy	Enable/disable proxy

Interactive Menu

📁 Scrape Categories  →  Scrape all categories
📄 Scrape Details    →  Get detailed content info
🧹 Clean Data        →  Remove duplicates
📊 Status            →  Check scraping progress
⚙️ Settings          →  View configuration
📥 Download          →  Get CSV files

Workflow

graph LR
    A[📁 Scrape Categories] --> B[🧹 Clean Data]
    B --> C[📄 Scrape Details]
    C --> D[📥 Download Results]

Loading

---

📖 Documentation

Category Scraper

Scrapes content from categories with pagination support.

from como_tv_scraper import ComoTVScraper, ScraperConfig

# Auto-load from .env
config = ScraperConfig()
scraper = ComoTVScraper(config)

# Scan categories
scraper.scan_all_categories(
    start_id=1,
    end_id=1000,
    output_file="output/all_categories.csv"
)

Output: categoryId, categoryName, title, contentId

Detail Scraper

Fetches detailed information for each content item.

from detail_scraper import ComoTVDetailScraper, load_content_ids_from_csv

# Load config from .env
scraper = ComoTVDetailScraper()

# Load content IDs from categories
content_ids = load_content_ids_from_csv("output/all_categories_cleaned.csv")

# Scrape details
scraper.scrape_multiple_details(
    content_ids=content_ids,
    output_file="output/content_details.csv"
)

Output: contentId, name, type, templateModel, area, contentGrade, description, duration, categoryName, watermarkId, RelatedCount, Related1-10

Data Cleaning

Remove duplicates and empty rows automatically.

from clean_data import clean_categories_csv, clean_details_csv

# Clean category data
clean_categories_csv()

# Clean details data
clean_details_csv()

---

⚙️ Configuration

Environment Variables

Variable	Default	Description
TELEGRAM_BOT_TOKEN	-	Telegram bot token from @BotFather
PROXY_URL	-	Proxy URL with credentials
PROXY_ENABLED	true	Enable/disable proxy
LANGUAGE_CODE	it-IT	API language (see Language Options below)
CATEGORY_START	1	Start category ID
CATEGORY_END	1000	End category ID
MAX_WORKERS	4	Number of parallel threads (1-10 recommended)

Proxy Format

http://username:password@host:port

Example:

http://user123-country-it:pass456@proxy.com:6060

Language Options

Code	Language	Status	Notes
it-IT	Italian (Italy)	✅ Recommended	Excellent data quality, best results
en_US	English (US)	⚠️ Limited	Poor/no data availability

Note: Como TV is Italy-based, so Italian language provides the most comprehensive content.

---

📁 Project Structure

como-tv-scraper/
├── 📄 como_tv_scraper.py      # Category scraper
├── 📄 detail_scraper.py       # Detail scraper
├── 📄 clean_data.py           # Data cleaning utility
├── 🤖 telegram_bot.py         # Telegram bot interface
├── 📋 requirements.txt        # Python dependencies
├── 🔧 .env.example            # Configuration template
├── 📖 README.md               # This file
└── 📂 output/                 # Output directory
    ├── all_categories.csv           # Raw category data
    ├── all_categories_cleaned.csv   # Cleaned categories
    ├── content_details.csv          # Raw details
    └── content_details_cleaned.csv  # Cleaned details

---

🔧 Troubleshooting

Connection Errors / Timeouts

• ✅ Script auto-retries 3x with 5s delay
• ✅ Increase timeout in scraper config
• ✅ Check proxy connection
• ✅ Verify internet stability

Proxy Not Working

• ✅ Verify proxy URL format
• ✅ Test proxy: curl -x http://user:pass@proxy:port https://google.com
• ✅ Check proxy credentials
• ✅ Try disabling proxy: PROXY_ENABLED=false

Telegram Bot Not Responding

• ✅ Verify token in .env file
• ✅ Check bot is running: python telegram_bot.py
• ✅ Ensure .env file exists and formatted correctly
• ✅ Check internet connection

No Data Found

• ✅ Verify language code (it-IT works best)
• ✅ Check proxy is working (timeout = no data)
• ✅ Try different category range
• ✅ Increase timeout to 60s

---

📊 Sample Output

Category Scraping

Starting scan of categories 1 to 1000...
Proxy enabled: True
Language: it-IT

[206/1000] Scanning category 206...
  Scraping page 0...
  Scraping page 1...
  Found 45 items

==================================================
SCAN COMPLETE!
==================================================
Total categories scanned: 1000
Successful: 127
Failed/Empty: 873
Total items collected: 4,325
Output file: output/all_categories.csv
==================================================

Detail Scraping

Scraping content details with 4 workers...
100%|████████████████████| 4325/4325 [1:23:45<00:00, 51.67it/s]

Results:
  ✓ Successful: 4,312
  ✗ Failed: 13
  📊 Total: 4,325

Output saved to: output/content_details.csv

---

🔐 Security

⚠️ Important Security Notes

• 🔒 NEVER commit .env file to git (already in .gitignore)
• 🔒 NEVER share your Telegram bot token publicly
• 🔒 Keep proxy credentials secure
• 🔒 Use .env.example as template only (no real credentials)

---

📦 Requirements

• Python 3.8 or higher
• requests >= 2.31.0
• python-telegram-bot >= 20.0 (for bot only)
• python-dotenv >= 1.0.0
• tqdm >= 4.66.0 (for progress bars)

---

📝 License

MIT License - see LICENSE file for details

---

🤝 Contributing

Contributions are welcome! Feel free to:

• 🐛 Report bugs
• 💡 Suggest features
• 🔧 Submit pull requests

---

Made with ❤️ using Python

⬆ Back to Top