An interactive AI-powered mobile app to learn and practice sign language using speech, text, and gesture recognition
Demo • Features • Installation • Usage • Contributing
Sign Language Learner App is a cutting-edge educational mobile application designed to make sign language learning accessible, engaging, and effective for everyone. Using advanced AI and machine learning technologies, the app provides personalized learning experiences through gesture recognition, interactive lessons, and gamified challenges.
- 🎮 Gamified Learning: Interactive quizzes, challenges, and progress tracking
- 🤖 AI-Powered Feedback: Real-time gesture analysis and personalized suggestions
- 📱 Mobile-First Design: Optimized for smartphones and tablets
- 🌐 Offline Capability: Learn anytime, anywhere without internet
- 👥 Community-Driven: Built with input from the deaf and hard-of-hearing community
- 🎓 Curriculum-Based: Structured learning path from beginner to advanced
- Real-time Detection: Uses MediaPipe and TensorFlow Lite for accurate hand tracking
- Gesture Evaluation: AI-powered feedback on gesture accuracy and form
- Multiple Angles: Recognizes gestures from different camera positions
- Adaptive Learning: Improves recognition based on user's signing style
- Bidirectional Learning: Practice both sign-to-speech and speech-to-sign
- Natural Language Processing: Understands context and sentence structure
- Voice Synthesis: High-quality text-to-speech with multiple voice options
- Multilingual Support: Learn sign language in your preferred spoken language
- Progress Tracking: Visual progress bars and achievement system
- Daily Challenges: Keep users engaged with daily learning goals
- Streak System: Reward consistent practice with streak counters
- Leaderboards: Compete with friends and global learners
- Badges & Rewards: Unlock achievements for milestones and skills
- Interactive Chatbot: Practice conversations in real-world scenarios
- Personalized Feedback: Tailored suggestions based on learning patterns
- Adaptive Difficulty: Automatically adjusts lesson complexity
- Learning Analytics: Track strengths and areas for improvement
- Structured Lessons: Beginner to advanced learning paths
- Topic Categories: Everyday conversations, professional terms, emotional expressions
- Cultural Context: Learn about deaf culture and community
- Regional Variations: Support for different sign language dialects
- Download Lessons: Access content without internet connection
- Local Model Processing: On-device AI for privacy and speed
- Sync Progress: Automatically sync when connected to internet
- Reduced Data Usage: Optimized for mobile data plans
| Component | Technology | Purpose |
|---|---|---|
| Mobile Framework | Flutter (Dart) | Cross-platform mobile development |
| Backend Services | Firebase | Authentication, database, cloud functions |
| Machine Learning | TensorFlow Lite | On-device gesture recognition |
| Computer Vision | MediaPipe | Hand tracking and pose detection |
| Local Database | SQLite/Hive | Offline data storage |
| State Management | Riverpod/Provider | App state and data flow |
| Animation | Rive/Lottie | Interactive animations and feedback |
| Speech APIs | Google Speech/Azure | STT and TTS capabilities |
┌─────────────────────────────────────────────────────────────┐
│ Presentation Layer │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Screens │ │ Widgets │ │ Animations │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────┬───────────────────────────────────────────┘
│
┌─────────────────▼───────────────────────────────────────────┐
│ Business Logic Layer │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Controllers │ │ Services │ │ Repositories│ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────┬───────────────────────────────────────────┘
│
┌─────────────────▼───────────────────────────────────────────┐
│ Data Layer │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Firebase │ │Local Database│ │ ML Models │ │
│ │ (Cloud) │ │ (SQLite) │ │(TF Lite) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
SignLanguageLearner/
├── 📱 lib/
│ ├── 📁 core/
│ │ ├── 📄 constants.dart # App-wide constants
│ │ ├── 📄 theme.dart # App theming and styles
│ │ └── 📄 utils.dart # Utility functions
│ ├── 📁 data/
│ │ ├── 📁 models/
│ │ │ ├── 📄 user.dart # User data model
│ │ │ ├── 📄 lesson.dart # Lesson structure
│ │ │ └── 📄 progress.dart # Progress tracking
│ │ ├── 📁 repositories/
│ │ │ ├── 📄 user_repository.dart
│ │ │ ├── 📄 lesson_repository.dart
│ │ │ └── 📄 progress_repository.dart
│ │ └── 📁 datasources/
│ │ ├── 📄 firebase_datasource.dart
│ │ └── 📄 local_datasource.dart
│ ├── 📁 domain/
│ │ ├── 📁 entities/
│ │ ├── 📁 repositories/
│ │ └── 📁 usecases/
│ ├── 📁 presentation/
│ │ ├── 📁 screens/
│ │ │ ├── 📁 auth/
│ │ │ │ ├── 📄 login_screen.dart
│ │ │ │ └── 📄 signup_screen.dart
│ │ │ ├── 📁 home/
│ │ │ │ ├── 📄 home_screen.dart
│ │ │ │ └── 📄 dashboard_screen.dart
│ │ │ ├── 📁 learning/
│ │ │ │ ├── 📄 lesson_screen.dart
│ │ │ │ ├── 📄 practice_screen.dart
│ │ │ │ └── 📄 quiz_screen.dart
│ │ │ ├── 📁 profile/
│ │ │ │ ├── 📄 profile_screen.dart
│ │ │ │ └── 📄 progress_screen.dart
│ │ │ └── 📁 camera/
│ │ │ └── 📄 gesture_camera_screen.dart
│ │ ├── 📁 widgets/
│ │ │ ├── 📁 common/
│ │ │ │ ├── 📄 custom_button.dart
│ │ │ │ ├── 📄 progress_bar.dart
│ │ │ │ └── 📄 loading_indicator.dart
│ │ │ ├── 📁 lesson/
│ │ │ │ ├── 📄 lesson_card.dart
│ │ │ │ ├── 📄 gesture_display.dart
│ │ │ │ └── 📄 feedback_widget.dart
│ │ │ └── 📁 camera/
│ │ │ ├── 📄 camera_preview.dart
│ │ │ └── 📄 gesture_overlay.dart
│ │ └── 📁 controllers/
│ │ ├── 📄 auth_controller.dart
│ │ ├── 📄 lesson_controller.dart
│ │ └── 📄 camera_controller.dart
│ ├── 📁 services/
│ │ ├── 📄 firebase_service.dart # Firebase integration
│ │ ├── 📄 ml_service.dart # Machine learning service
│ │ ├── 📄 camera_service.dart # Camera handling
│ │ ├── 📄 speech_service.dart # STT/TTS integration
│ │ └── 📄 notification_service.dart
│ └── 📄 main.dart # App entry point
├── 📁 assets/
│ ├── 📁 images/
│ │ ├── 📁 gestures/ # Sign language reference images
│ │ ├── 📁 avatars/ # User avatars and characters
│ │ └── 📁 icons/ # Custom app icons
│ ├── 📁 animations/
│ │ ├── 📁 rive/ # Rive animation files
│ │ └── 📁 lottie/ # Lottie animation files
│ ├── 📁 audio/
│ │ ├── 📁 sounds/ # UI sound effects
│ │ └── 📁 pronunciations/ # Word pronunciations
│ └── 📁 fonts/ # Custom fonts
├── 📁 ml_models/
│ ├── 📄 gesture_classifier.tflite # Trained gesture model
│ ├── 📄 hand_landmark.tflite # MediaPipe hand model
│ └── 📄 model_metadata.json # Model information
├── 📁 test/
│ ├── 📁 unit/ # Unit tests
│ ├── 📁 widget/ # Widget tests
│ └── 📁 integration/ # Integration tests
├── 📁 docs/
│ ├── 📄 API.md # API documentation
│ ├── 📄 ARCHITECTURE.md # Architecture guide
│ └── 📄 USER_GUIDE.md # User manual
├── 📄 pubspec.yaml # Flutter dependencies
├── 📄 analysis_options.yaml # Code analysis rules
├── 📄 firebase.json # Firebase configuration
└── 📄 README.md # This file
- Flutter SDK 3.0 or higher
- Dart SDK 2.17 or higher
- Android Studio / VS Code with Flutter extensions
- Firebase Account for backend services
- Physical Device or Emulator with camera support
git clone https://github.com/yourusername/SignLanguageLearner.git
cd SignLanguageLearner# Get all packages
flutter pub get
# Verify Flutter installation
flutter doctor# Install Firebase CLI
npm install -g firebase-tools
# Login to Firebase
firebase login
# Install FlutterFire CLI
dart pub global activate flutterfire_cli
# Configure Firebase for your project
flutterfire configure# Copy environment template
cp lib/core/env_template.dart lib/core/env.dart
# Edit env.dart with your API keys and configuration# Download pre-trained models
flutter pub run build_runner build
# Or manually download models to ml_models/ directory# Debug mode
flutter run
# Release mode
flutter run --release
# Specific device
flutter run -d <device_id># android/app/build.gradle
android {
compileSdkVersion 33
minSdkVersion 21
targetSdkVersion 33
}
# Add permissions in android/app/src/main/AndroidManifest.xml
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.INTERNET" /># ios/Runner/Info.plist
<key>NSCameraUsageDescription</key>
<string>This app uses camera for gesture recognition</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app uses microphone for speech recognition</string>┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Splash Screen │ => │ Onboarding │ => │ Auth Screen │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
┌─────────────────┐ ┌─────────────────┐ ┌─────────▼───────┐
│ Profile Setup │ <= │ Home/Dashboard │ <= │ Main Navigation│
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
┌───────────────────────┼───────────────────────┐
│ │ │
┌───────▼───────┐ ┌─────────▼─────────┐ ┌───────▼───────┐
│ Lessons │ │ Practice Mode │ │ Progress │
│ Browser │ │ (Camera) │ │ Tracking │
└───────────────┘ └───────────────────┘ └───────────────┘
| Screen | Purpose | Key Features |
|---|---|---|
| Onboarding | First-time user introduction | Tutorial, permissions, setup |
| Dashboard | Main hub for learning activities | Progress overview, daily goals |
| Lesson Browser | Browse available lessons | Categories, difficulty levels |
| Practice Mode | Interactive learning session | Gesture recognition, feedback |
| Quiz Mode | Test knowledge and skills | Timed challenges, scoring |
| Progress | Track learning journey | Statistics, achievements |
| Profile | User settings and preferences | Account settings, app config |
- Welcome & Permissions: Grant camera and microphone access
- User Profile: Set up your learning preferences and goals
- Skill Assessment: Take a quick test to determine your starting level
- Tutorial: Learn how to use gesture recognition and app features
- Dashboard Check: Review daily goals and progress
- Lesson Selection: Choose from recommended or browse all lessons
- Interactive Learning: Follow guided instructions with AI feedback
- Practice Session: Use camera to practice gestures
- Quiz Challenge: Test your knowledge with interactive quizzes
- Progress Review: Check achievements and plan next session
// Example gesture practice flow
1. Position camera at arm's length
2. Ensure good lighting and clear background
3. Follow on-screen gesture guide
4. Receive real-time feedback
5. Repeat until accuracy target is met// Speech-to-sign learning
1. Listen to spoken word/phrase
2. Perform corresponding sign language gesture
3. App recognizes your gesture
4. Get feedback on accuracy and timing// Daily challenge example
1. Complete 5 new gestures
2. Practice for 15 minutes
3. Achieve 85% accuracy rate
4. Earn points and badges
5. Maintain learning streak| Metric | Description | Visualization |
|---|---|---|
| Accuracy Score | Gesture recognition accuracy | Progress bars, charts |
| Learning Streak | Consecutive days of practice | Streak counter, calendar |
| Skill Level | Current proficiency level | Level badges, progress rings |
| Time Practiced | Total learning time | Time tracking, goals |
| Gestures Learned | Number of mastered signs | Achievement counters |
- Difficulty Adjustment: Automatically increases challenge based on performance
- Personalized Recommendations: Suggests lessons based on learning patterns
- Weakness Identification: Highlights areas needing improvement
- Optimal Timing: Reminds users when to review previously learned content
# Run all tests
flutter test
# Run with coverage
flutter test --coverage
# Generate coverage report
genhtml coverage/lcov.info -o coverage/html
# Integration tests
flutter drive --target=test_driver/app.dart- Model validation
- Service logic testing
- Utility function verification
- UI component testing
- User interaction simulation
- Layout validation
- Complete user flow testing
- API integration verification
- Camera and ML model testing
# Debug APK
flutter build apk --debug
# Release APK
flutter build apk --release
# App Bundle (for Play Store)
flutter build appbundle --release# Debug build
flutter build ios --debug
# Release build
flutter build ios --release
# Create archive for App Store
flutter build ipa# Upload using fastlane
fastlane android deploy
# Manual upload
# Upload generated app-release.aab to Google Play Console# Upload using fastlane
fastlane ios deploy
# Manual upload via Xcode
# Archive and upload through Xcode Organizer- Basic Alphabet: A-Z fingerspelling
- Numbers: 0-20 counting
- Common Greetings: Hello, goodbye, thank you
- Family Members: Mom, dad, brother, sister
- Daily Activities: Eat, sleep, work, study
- Emotions & Feelings: Happy, sad, angry, excited
- Colors & Shapes: Basic color vocabulary
- Time & Dates: Days, months, telling time
- Food & Drinks: Common meal vocabulary
- Transportation: Car, bus, train, airplane
- Professional Terms: Job titles, workplace vocabulary
- Complex Conversations: Multi-sentence dialogues
- Abstract Concepts: Ideas, opinions, emotions
- Cultural Context: Deaf culture, community values
- Regional Variations: Different sign language dialects
- Medical Signs: Healthcare vocabulary
- Educational Signs: School and learning terms
- Emergency Signs: Safety and urgent communication
- Religious Signs: Spiritual and ceremonial vocabulary
- High Contrast Mode: Better visibility for users with visual impairments
- Text Size Options: Adjustable font sizes for readability
- Voice Commands: Navigate app using voice controls
- Haptic Feedback: Vibration feedback for interactions
- Screen Reader Support: Compatible with TalkBack and VoiceOver
- Community Input: Developed with deaf community feedback
- Cultural Context: Includes deaf culture education
- Regional Support: Multiple sign language variants
- Inclusive Design: Welcomes learners from all backgrounds
- Basic gesture recognition (ASL alphabet & numbers)
- Interactive lessons and quizzes
- Progress tracking and gamification
- Offline mode support
- Speech-to-text integration
- Advanced gesture recognition (words and phrases)
- AI chatbot for conversation practice
- Social features (friend challenges, leaderboards)
- Multiple sign languages (BSL, JSL)
- Improved accuracy with better ML models
- AR overlays for gesture guidance
- Live video chat with interpreters
- Advanced analytics and learning insights
- Teacher/parent dashboard
- Certification and achievement system
- VR immersive learning experiences
- Integration with video calling platforms
- Global sign language community platform
- AI-generated personalized curriculum
- Wearable device integration
We welcome contributions from developers, educators, and the deaf community!
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes following our coding standards
- Test your changes thoroughly
- Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- 🐛 Bug Fixes: Help identify and fix issues
- ✨ New Features: Implement new learning modules
- 🎨 UI/UX Improvements: Enhance user experience
- 📚 Content Creation: Add new lessons and gestures
- 🌍 Localization: Translate app into new languages
- 📖 Documentation: Improve guides and tutorials
- Follow Flutter/Dart best practices
- Write comprehensive tests for new features
- Use meaningful commit messages
- Update documentation for significant changes
- Respect accessibility guidelines
This project is licensed under the MIT License - see the LICENSE file for details.
MIT License
Copyright (c) 2024 Sign Language Learner Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software")...
- National Association of the Deaf (NAD) - Guidance and feedback
- Gallaudet University - Sign language curriculum consultation
- Google MediaPipe Team - Hand tracking technology
- Flutter Community - Open source packages and support
- Beta Testers - Early feedback from deaf community members
- MediaPipe - Hand landmark detection
- TensorFlow - Machine learning framework
- Firebase - Backend services
- Flutter - Mobile app framework
- 📖 Documentation: docs/USER_GUIDE.md
- 🐛 Bug Reports: GitHub Issues
- 💬 Discussions: GitHub Discussions
- 📧 Email Support: [email protected]
- 🎮 Discord: Join our learning community
- 🐦 Twitter: @SignLearnerApp
- 📘 Facebook: Sign Language Learner Community
- 📱 Reddit: r/SignLanguageLearning
Made with ❤️ for inclusive education and accessibility
Help us make sign language learning accessible to everyone!
⭐ Star this repository if you found it helpful! ⭐