"Sometimes you gotta run before you can walk." — Tony Stark
A voice-activated AI desktop assistant powered by Groq AI and Llama models, featuring a live 3D neural orb dashboard, real-time state synchronization, screen awareness, smart web and person lookup, natural-language command processing, and desktop automation.
| Feature | Description |
|---|---|
| Always-On Wake Word | Say "Hey Jarvis" to activate the assistant without pressing a button. |
| Groq AI Brain | Uses Groq-hosted Llama models for natural-language interaction and AI responses. |
| Real-Time Dashboard | Interactive Three.js neural orb synchronized with assistant states. |
| Chat Interface | Displays user and JARVIS conversations directly in the dashboard. |
| SSE Live Sync | Server-Sent Events keep voice, responses, and UI state synchronized in real time. |
| Smart Commands | Natural-language commands can trigger web searches, application launches, and other actions. |
| Screen Awareness | Captures and analyzes the current screen using Groq vision capabilities. |
| Barge-In / Interrupt | Saying "Hey Jarvis" while JARVIS is speaking can interrupt the current response. |
| Person Lookup | Searches for people and provides information through web resources and AI responses. |
| Smart Image Search | Opens image search results for requested people or subjects. |
| Instagram Finder | Searches for Instagram profiles through natural-language commands. |
| Sleep / Wake Mode | Put JARVIS into sleep mode and wake it again using voice commands. |
| AI File Save | AI-generated responses can be saved to files. |
| System Monitoring | Displays CPU and RAM usage on the dashboard. |
| Hot Restart | Restart JARVIS through a voice command without manually restarting the application. |
JARVIS follows an event-driven architecture where voice or text input is interpreted, routed to the appropriate execution path, and synchronized with the browser dashboard through Flask and Server-Sent Events.
USER
|
v
+--------------------+
| Voice / Text Input |
+---------+----------+
|
v
+--------------------+
| Speech Recognition |
| Command Processing |
+---------+----------+
|
v
+--------------------+
| Intent / Command |
| Handling |
+---------+----------+
|
+------------+------------+
| | |
v v v
+-----------+ +-----------+ +-------------+
| System | | Web / | | Groq AI |
| Actions | | Search | | Llama/Vision|
+-----------+ +-----------+ +-------------+
| | |
+------------+------------+
|
v
+--------------------+
| Action / Response |
| State Updates |
+---------+----------+
|
| SSE
v
+--------------------+
| Flask Server |
| API + SSE Stream |
+---------+----------+
|
| HTTP / SSE
v
+--------------------+
| Browser Dashboard |
| |
| Three.js Neural Orb|
| Chat Interface |
| Status Display |
| System Statistics |
+--------------------+
- Voice Layer — listens for the wake word and captures spoken commands.
- Command Processing — interprets natural-language requests and determines the appropriate action.
- System Integration — handles desktop applications, browser actions, system information, and other local operations.
- Groq AI Layer — handles conversational AI requests and screen-analysis tasks using supported Llama models.
- Flask Backend — provides the local application server and API endpoints.
- SSE Communication — streams assistant states, messages, and events to the browser in real time.
- Three.js Dashboard — provides the visual interface, neural orb, chat display, status information, and system statistics.
JARVIS converts a voice command into an action or AI response through the following flow:
User speaks
|
v
Wake-word / Speech Recognition
|
v
Command Processing
|
+---------------------+
| |
v v
System / Web Action AI Request
| |
| v
| Groq AI / Vision
| |
+----------+----------+
|
v
Action / Response
|
v
SSE Stream
|
+------+------+
| |
v v
Dashboard Voice
- Voice Input — JARVIS listens for the wake word
"Hey Jarvis". - Speech Recognition — the spoken command is converted into text.
- Command Processing — JARVIS determines what type of request was made.
- Execution — the appropriate system, web, or application action is performed.
- AI Processing — AI requests are handled using Groq-hosted Llama models, including vision capabilities where applicable.
- Real-Time Updates — Server-Sent Events synchronize the browser dashboard with assistant states and responses.
- Response — JARVIS provides feedback through the dashboard and voice output.
- Python 3.11+
- Flask
- Groq API
- Llama Models
- pyttsx3
- Speech Recognition
- HTML5
- CSS3
- JavaScript
- Three.js
- WebGL
- HTTP
- Server-Sent Events (SSE)
- Windows
- Python 3.11+
- Windows
- Working microphone
- Groq API key
- Internet connection for AI and web-based features
git clone https://github.com/rishabhbhardwaj-dev/Jarvis-voice-assistant.git
cd Jarvis-voice-assistantpip install -r requirements.txtIf PyAudio installation fails, try:
pip install pipwin
pipwin install pyaudioConfigure your Groq API key using the configuration expected by config.py.
Do not commit your real API key to GitHub.
python main.pyOr launch it using:
run_jarvis.bat
The dashboard runs locally at:
http://localhost:5000
| Say | Action |
|---|---|
"Hey Jarvis" |
Wake up JARVIS |
"Jarvis sleep" |
Enter sleep mode |
"Wake up Jarvis" |
Resume from sleep mode |
"Jarvis restart" |
Restart JARVIS |
"Jarvis quit" |
Shut down JARVIS |
| Say | Action |
|---|---|
"Open YouTube" |
Opens YouTube |
"Search YouTube for lofi music" |
Searches YouTube |
"Search Google for AI news" |
Performs a Google search |
"Find React hooks on GitHub" |
Searches GitHub |
"Look up quantum physics on Wikipedia" |
Searches Wikipedia |
| Say | Action |
|---|---|
"Who is Rohit Sharma?" |
Opens relevant information and provides an AI response |
"Tell me about Elon Musk" |
Searches for information and provides a response |
"Show me a picture of Virat Kohli" |
Opens image search results |
"Photo of Cristiano Ronaldo" |
Opens image search results |
"Rohit Sharma Instagram" |
Searches for the relevant Instagram profile |
| Say | Action |
|---|---|
"What time is it" |
Provides the current time |
"What's the date" |
Provides the current date |
"System status" |
Shows CPU and RAM usage |
"Open Notepad" |
Launches Notepad |
"Open Calculator" |
Launches Calculator |
"Open VS Code" |
Launches VS Code |
"Play music" |
Plays local music files |
| Say | Action |
|---|---|
"What's on my screen?" |
Analyzes the current screen using Groq vision capabilities |
"Analyze my screen" |
Performs screen analysis |
"Use AI to write a cover letter" |
Generates an AI response that can be saved to a file |
"Reset chat" |
Clears conversation context |
| Any other supported question | Interacts with Groq AI |
JARVIS AI/
│
├── demo/
│ ├── Jarvis_SleepMode.mp4
│ └── ...
│
├── frontend/
│ ├── index.html
│ ├── script.js
│ └── style.css
│
├── screenshots/
│ ├── Jarvis_Interface(Dashboard).png
│ └── Voice Command Execution.png
│
├── .env
├── .gitignore
├── config.py
├── main.py
├── requirements.txt
├── run_jarvis.bat
└── README.md
.envcontains local configuration and should not contain credentials that are committed to the repository.
| Problem | Possible Fix |
|---|---|
| Microphone not working | Set the correct microphone as the default Windows input device. |
| PyAudio installation error | Try pip install pipwin followed by pipwin install pyaudio. |
| Groq API key error | Verify the API key configuration used by config.py. |
| Screen awareness fails | Verify the required screen-capture dependencies are installed. |
| Port 5000 is already in use | Stop the process using port 5000 or change the application port. |
| JARVIS keeps speaking | Use the wake word to trigger the interrupt/barging-in behavior. |
A typical demonstration can include:
- Start JARVIS with
python main.py. - Say "Hey Jarvis" to activate the assistant.
- Ask "What time is it" and observe the voice and dashboard response.
- Ask JARVIS to search YouTube for something.
- Ask a person-related question and observe the web/AI response.
- Ask "What's on my screen?" to demonstrate screen awareness.
- Interrupt an active response using the wake word.
- Say "Jarvis sleep" and observe the dashboard state.
- Say "Wake up Jarvis" to restore the assistant.
- Use the dashboard chat input for a text-based interaction.
- Say "Jarvis quit" to shut down the application.
.png)
Rishabh Bhardwaj
- Portfolio: https://rishabh-portfolio-lac.vercel.app/
- GitHub: https://github.com/rishabhbhardwaj-dev
- LinkedIn: https://www.linkedin.com/in/rishabhbhardwaj-tech/
No open-source license has been specified for this project.
Built with Python, Flask, Groq AI, Llama models, and Three.js.