Skip to content

README.md

Latest commit

 

History

History
154 lines (113 loc) · 5.26 KB

README.md

File metadata and controls

154 lines (113 loc) · 5.26 KB
Raw

Arabic Corpus Refinery

A specialized tool for cleaning and normalizing Arabic text corpora using Google's Gemini. This tool is designed to prepare high-quality text data for training Large Language Models (LLMs) while preserving the unique characteristics of Arabic dialects and expressions.

Features

  • Automated Text Cleaning: Uses Google Gemini AI to intelligently clean and normalize Arabic text
  • Dialect Preservation: Maintains unique dialect expressions and vocabulary while standardizing orthography
  • Memory Efficient: Processes large corpora in chunks to work on low-memory systems (1GB RAM)
  • Resume Capability: Can resume processing from where it left off if interrupted
  • Progress Tracking: Real-time dashboard showing processing progress and statistics
  • Robust Error Handling: Includes retry logic for API calls and comprehensive logging

Prerequisites

  • Python 3.7 or higher
  • Google Gemini API key
  • Text corpus files in .txt format

Installation

  1. Clone the repository:

    git clone https://github.com/o96a/LLMCorpusKit.git
cd LLMCorpusKit

  2. Install dependencies:

    pip install -r requirements.txt

  3. Set up environment variables:

    • Copy .env.example to .env
    • Get your Google Gemini API key from Google AI Studio
    • Add your API key to the .env file:
    cp .env.example .env
# Edit .env file and add your API key

Usage

  1. Prepare your corpus:

    • Create a corpus folder in the project directory
    • Place your .txt files containing Arabic text in this folder

  2. Run the cleaning process:

    python main.py

  3. Monitor progress:

    • The script displays a real-time dashboard with progress information
    • Cleaned files will be saved in the cleaned_corpus folder
    • Processing state is automatically saved and can be resumed if interrupted

Configuration

You can modify the following settings in main.py:

  • CHUNK_SIZE: Size of text chunks processed at once (default: 15000 characters)
  • MODEL_NAME: Gemini model to use (default: 'gemini-2.0-flash')
  • CORPUS_PATH: Path to input corpus files (default: 'corpus')
  • CLEANED_PATH: Path for cleaned output files (default: 'cleaned_corpus')

Text Processing Rules

The tool applies the following cleaning and normalization rules:

Orthographic Normalization

  • Standardizes "ى" (alif maqsura) to "ي" (ya) where appropriate
  • Removes diacritics and tatweel marks
  • Standardizes laughter expressions (e.g., "ههههه" → "ههه")
  • Removes excessive punctuation

Spelling Correction

  • Fixes common typos and spelling mistakes
  • Standardizes word variations while preserving dialect

Content Refinement

  • Removes conversational filler words with no semantic value
  • Improves sentence structure and punctuation
  • Handles transliterated foreign words consistently

Preservation Constraints

  • Does NOT translate to Modern Standard Arabic (MSA)
  • Does NOT remove unique dialect expressions
  • Maintains the authentic flavor of the dialect

File Structure

LLMCorpusKit/
├── main.py                 # Main processing script
├── requirements.txt        # Python dependencies
├── .env.example           # Environment variables template
├── .gitignore             # Git ignore rules
├── README.md              # This file
├── corpus/                # Input text files (create this folder)
├── cleaned_corpus/        # Output cleaned files (auto-created)
├── processing_state.json  # Processing state (auto-created)
└── processing.log         # Processing logs (auto-created)

Troubleshooting

Common Issues

  1. API Key Error:

    • Ensure your .env file contains a valid GOOGLE_API_KEY
    • Verify your API key has access to Gemini models

  2. Memory Issues:

    • Reduce CHUNK_SIZE if running on very low memory systems
    • Ensure you have enough disk space for output files

  3. Processing Interruption:

    • The script automatically saves progress and can resume from the last checkpoint
    • Simply run python main.py again to continue

Logs and Debugging

  • Check processing.log for detailed error messages
  • The script provides verbose console output during processing
  • Processing state is saved in processing_state.json

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/new-feature)
  3. Commit your changes (git commit -am 'Add new feature')
  4. Push to the branch (git push origin feature/new-feature)
  5. Create a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

  • Google Gemini AI for powerful text processing capabilities
  • The Arabic language community for preserving rich dialects

Support

If you encounter any issues or have questions, please:

  1. Check the troubleshooting section above
  2. Search existing GitHub Issues
  3. Create a new issue with detailed information about your problem

Note: This tool is designed for Arabic text processing. It can work with various Arabic dialects and maintains their unique characteristics while improving text quality.