Talk to your Windows computer like you talk to a human.
Racchha is an open-source Windows computer-use agent that understands natural language, observes the desktop, plans validated actions, controls applications, verifies results, and performs bounded recovery when actions fail.
It is Windows-focused, experimental, and not IoT. Natural language stays flexible. The internal action vocabulary is typed, allowlisted, and validated. This is an evolving research/engineering project — not a finished Jarvis and not universal Windows control.
“Racchha, Notepad kholo aur mera ye text likh do.”
⭐ Star · 🐛 Issues · 💬 Discussions · 🤝 Contributing · 🗺️ Roadmap
Don't just watch tutorials.
Pick an issue. Read the architecture. Understand the existing implementation. Build the solution. Write tests. Open a PR.
Racchha is designed as a real-world open-source project where you can practice:
Python FastAPI React Electron TypeScript Windows automation UI Automation AI agents LLM integration software architecture testing Git/GitHub CI/CD open-source collaboration
The goal isn't to copy code. The goal is to understand an existing system and make it better.
Many desktop “assistants” are one of these:
- hardcoded command collections and aliases
- chatbots that cannot reliably touch the real UI
- LLM wrappers that jump straight to unsafe shell / PowerShell / CMD
Racchha is trying a different loop:
Understand → Observe → Plan → Act → Verify → Recover
The work is specifically about making computer interaction:
- observable — a bounded desktop snapshot, not a guess
- structured — typed
ComputerActionJSON, not free-form scripts - testable — pytest + Vitest, with Windows CI configured
- verifiable — post-action observation before calling success
- safety-constrained — allowlists, forbidden parameters, no validator bypass
- recoverable — bounded replan, not an infinite agent loop
- contributor-friendly — issues, docs, and a real architecture to extend
If you care about computer-use agents that you can inspect, test, and keep from becoming a shell trampoline, this repo is for you.
Students and developers often learn by building todo lists, calculators, weather apps, basic CRUD, or a thin chatbot wrapper. Those are useful first steps.
Racchha is a longer-horizon project. One codebase forces you to work across:
| Discipline | Where it shows up |
|---|---|
| Python / FastAPI | backend/app/ — routes, services, DI |
| React / TypeScript / Electron | frontend/ — desktop shell, listening, IPC |
| REST + frontend/backend contracts | POST /command, speech routes |
| LLM structured output | Gemini intents, goal_actions, replan JSON |
| Prompt engineering | compact computer/browser/goal context, not the whole UI tree |
| Windows UI Automation | observation + semantic click/select |
| Browser automation | Chrome DevTools Protocol infrastructure |
| Planning / verification / recovery | GoalExecutor observe → act → verify → replan |
| Context | in-memory ComputerContext / GoalContext |
| Testing + CI | 316 backend tests, 21 frontend tests, Windows workflow |
| GitHub | issues, PR template, MIT, contributing guide |
Don't just clone Racchha. Pick an issue and build a piece of it.
If you are serious about becoming a stronger developer, don’t only watch tutorials. Spend real time reading, running, breaking, debugging, testing, and extending a system like this. That is a serious hands-on path — it does not guarantee anyone becomes “the best developer.”
Start here if you want a real repo without owning the whole agent loop:
- documentation and setup troubleshooting
- reproducing bugs with clear logs (no secrets)
- extra unit tests and fixtures
- clearer validation / API error messages
- small React/Electron UI copy and error display
See CONTRIBUTING.md and docs/github-issues.md. Look for good first issue on GitHub.
Good fit if you already write Python or TypeScript and want production-shaped modules:
- FastAPI services and dependency injection
- React / Electron / TypeScript listening and HTTP client
- observation collectors and summarization
- context engine (compact state, no invented paths)
- UI Automation discovery ranking
- integration-style tests (opt-in Windows, not Linux UIA)
The hard problems are still open:
- bounded replanning quality (fixtures, prompts, policy)
- UIA trees that lie or are empty
- safer browser + desktop goal handoff (without rewriting CDP)
- Phase 3 system controls as typed actions, not shell
- Phase 4 vision/OCR as research first — not implemented today
| Phase | Status |
|---|---|
| 0 Typed action foundation | Complete |
| 1 Computer Control Layer | Complete |
| 2A Perception / observation | Complete |
| 2B Semantic UI discovery | Complete |
| 2C Observe → Act → Observe | Complete |
| 2D Computer + browser context | Complete |
| 2E Bounded replanning | Complete |
| 3 Windows system controls | Not implemented |
| 4 Vision / OCR | Not implemented |
Local automated coverage: 316 backend tests, 21 frontend tests, frontend production build passing.
GitHub Actions runs the suite on windows-latest (see .github/workflows/tests.yml). Check the Actions tab for the latest run — this README does not claim a badge color.
Not every Windows application is guaranteed. Third-party UIs with weak automation trees fail more often. Treat Racchha as an evolving architecture, not a finished general-purpose desktop AI.
Currently supports (experimental, Windows 10/11):
- Natural-language intent resolution (Gemini + fallback)
- Goal-based execution and the legacy plugin execute path
- Launching allowlisted apps (Notepad, Calculator, Explorer, Chrome, Settings, Task Manager, and aliases in the safety policy)
- Window management (focus, min/max/close/restore)
- Keyboard keys, hotkeys, and text typing
- Semantic UI discovery and UI Automation clicks/selects (not coordinate clicking)
- Explorer navigation when a path can be determined
- Rich desktop observation and compact computer/browser context
- Observe → act → observe, verification, bounded replanning
- Chrome DevTools Protocol infrastructure for browser workflows
- Speech input and TTS
Not implemented / not a feature:
- Universal computer vision, OCR, or screenshot fallback
- Unlimited autonomous loops
- Arbitrary shell / PowerShell / CMD
- Coordinate-based unrestricted automation
- Complete Windows Settings, brightness, volume, or media control (Phase 3)
- Every application, or perfect UI understanding
- A permanently free Gemini tier — you bring your own API access; Google’s limits change
If context is insufficient, Racchha should clarify rather than guess a folder, button, or app.
These are illustrative. Success depends on the live UI, observation quality, and Gemini planning.
| You say | What Racchha is building toward |
|---|---|
Notepad kholo |
Launch / focus Notepad |
Notepad kholo aur ye likho Hello Gurubhai |
Launch, observe, type, verify |
Notepad minimize karo |
Window management on the Notepad referent |
File Manager kholo |
Launch Explorer |
D drive kholo |
navigate_path when allowed |
D drive me My Resume folder kholo |
Navigate, then open/select the folder when discovery can see it |
Save button pe click karo |
Semantic selector, not x/y |
Chrome kholo aur Google search karo |
Chrome/CDP + intent routing |
Is window ko maximize karo |
Window action on the referent |
Us folder me jo resume hai woh kholo |
Uses computer context, not a hardcoded “usme → Explorer” rule |
User: “Notepad kholo aur likho Hello Gurubhai”
- Intent resolver returns
mode=goalwith validated actions. - Goal executor launches Notepad and waits.
- Observation captures the foreground window.
type_textruns through the Computer Control Layer.- Post-observation + verifier check the step.
- TTS speaks a short completion.
User: “File Manager kholo, D drive me jao aur My Resume folder kholo”
Observe after each navigation. Computer context stores a compact Explorer path when known. A later “ab usme …” can use that path in the Gemini prompt. If the path is unknown, the model is instructed to clarify.
User: “Save button pe click karo”
Racchha does not click screen coordinates. ElementDiscoveryService matches a semantic UiSelector (name, control type, match mode, enabled/visible) against the observed tree.
flowchart TD
U[User voice or text]
STT[Speech recognition]
API["FastAPI POST /command"]
INT[Intent resolver Gemini plus fallback]
CS[CommandService]
LEGACY[Legacy execute path]
TP[TaskPlanner]
TE[TaskExecutor]
CE[CommandEngine]
PL[Plugins]
GOAL[Goal path]
PLAN[GoalPlanner]
GE[GoalExecutor]
OBS[Observation layer]
CCL[Computer Control Layer]
VERIFY[Verifier]
REPLAN[Bounded replanner]
CTX[Session computer context]
TTS[Text to speech]
UIA[Windows UI Automation]
CHROME[Chrome CDP]
U --> STT
STT --> API
API --> INT
INT --> CS
CS --> LEGACY
CS --> GOAL
LEGACY --> TP --> TE --> CE --> PL --> TTS
GOAL --> PLAN --> GE
GE --> OBS --> CCL
CCL --> UIA
CCL --> CHROME
CCL --> VERIFY
VERIFY -->|success| CTX
VERIFY -->|failure and budget left| REPLAN
REPLAN --> PLAN
VERIFY -->|done or exhausted| TTS
CTX --> INT
Details: docs/architecture.md.
Gemini decides WHAT needs to be accomplished.
The Computer Control Layer decides HOW the computer may safely do it.
Gemini must not output Python, PowerShell, CMD, shell, arbitrary executable paths, or click coordinates. It outputs structured intents / goal_actions. Those objects are parsed by parse_goal_actions and rejected if they violate safety_policy.py. Replanned actions go through the same validator.
Natural language ≠ hardcoded command list. Phrases like “usme”, “isme”, “thoda aur”, or “phir” are resolved from context, not if maps.
Racchha can control the local Windows session. The safety model is part of the product.
- Typed actions only. Gemini proposes
ComputerActionJSON. The Computer Control Layer executes allowlisted types. - Validation on every plan and every replan. Unknown types and forbidden keys (
shell,cmd,powershell,python, coordinates, arbitrary executable paths, …) are rejected. - Application allowlist. Launch targets are a small set (see
safety_policy.py). - No unrestricted shell. PowerShell/CMD/arbitrary Python are not a supported feature path.
- Semantic UI, not coordinate spraying. Clicks use observed UIA names/control types.
- Bounded work. Defaults include max 20 actions per goal, max 2 replan attempts, max 5 actions per replan, 30s action timeout, 180s goal timeout.
- Audit records. Compact pre/post summaries and replan metadata — not screenshots or full UI trees in session memory.
PRs that bypass the validator will be rejected.
- Frontend captures audio (push-to-talk / toggle listening) or you POST text to
/command. - Backend speech routes transcribe when needed.
IntentResolverclassifies: conversation, execute, goal, clarify, unsupported.CommandServicekeeps the legacy plugin path and the goal agent path.- Goal execution is observe → act → observe → verify; Phase 2E may replan inside limits.
- Compact computer/browser/goal context is stored in memory for the next turn.
- TTS speaks the result.
Typed actions in backend/app/core/computer_control/: launch, windows, keys, type, semantic UI, Explorer navigation, wait, inspect UI. Allowlists live in safety_policy.py.
backend/app/core/observation/ captures foreground process/title, a bounded UIA tree, and Explorer path when detectable. Session memory stores summaries, not screenshots or full trees.
element_discovery.py ranks observed nodes against a selector — how Racchha clicks “Save” without becoming a coordinate bot.
GoalPlanner turns validated goal_actions into a GoalPlan. Replanning is bounded (attempts, action counts, goal timeout). There is no unlimited self-directed loop. Invalid replan JSON or forbidden actions abort recovery.
Chrome CDP lives under backend/app/core/automation/ and the Chrome plugin. Browser URL/title can be mirrored into ComputerContext. Website coverage is incomplete.
| Layer | Stack |
|---|---|
| Desktop | Electron 34, React 19, TypeScript, Vite 6, Vitest |
| Backend | Python 3.11+, FastAPI, Uvicorn, Pydantic Settings |
| AI | Gemini API (google-genai), structured JSON intents and replans |
| Windows | pywin32, uiautomation, WinRT speech synthesis |
| Browser | Chrome DevTools Protocol |
| Speech | SpeechRecognition, pydub, pyttsx3 / OneCore TTS |
agent-racchha/
├── backend/
│ ├── app/
│ │ ├── api/ # HTTP routes: /, /command, /speech, plugins
│ │ ├── core/
│ │ │ ├── intent/ # Gemini + fallback intent resolution
│ │ │ ├── goal/ # Planner, executor, verifier, replan
│ │ │ ├── computer_control/# Typed actions, UIA, discovery, safety
│ │ │ ├── observation/ # Collectors, snapshot, summarizer
│ │ │ ├── context/ # Session, computer, browser, goal context
│ │ │ ├── automation/ # Chrome / CDP
│ │ │ ├── plugins/ # Legacy command plugins
│ │ │ ├── task/ # Legacy task planner / executor
│ │ │ ├── speech/ # STT / TTS
│ │ │ └── command_engine/ # Parser, registry, engine
│ │ ├── domain/ # API schemas
│ │ ├── services/ # CommandService, TTS, speech
│ │ ├── dependencies.py
│ │ └── main.py
│ ├── scripts/ # Manual Windows verification
│ └── requirements.txt
├── frontend/
│ ├── electron/ # Main + preload
│ └── src/ # React UI, audio, listening hooks
├── tests/backend/ # Pytest suite
├── docs/
├── .github/ # Issue/PR templates, CI
└── package.json # npm run dev / install:all
This tree matches the repository.
Requirements
- Windows 10 or 11 (computer control and several Python packages are Windows-native)
- Python 3.11+
- Node.js 20+ and npm
- Git
- Chrome, if you use browser automation
- Optional ffmpeg for WebM/Opus transcription (
FFMPEG_PATHorPATH)
git clone https://github.com/pradipNP/agent-racchha.git
cd agent-racchha
npm run install:all
pip install -r tests/requirements.txtEquivalent to: npm install, npm install --prefix frontend, pip install -r backend/requirements.txt.
- Create Gemini API access in Google’s AI / Gemini developer console. Model names, quotas, and pricing change; nothing here is permanently free.
- Copy the example env file:
copy backend\.env.example backend\.env
- Set placeholders only in your local file:
GEMINI_API_KEY=your_gemini_api_key_here GEMINI_MODEL=your_gemini_model_here - Never commit
.env. Restart the backend after changes.
Root .env.example documents the same keys. FastAPI Settings loads .env relative to the backend working directory when you use npm run dev:backend.
The project is intended to stay free and community-driven as open source. That does not mean every API or dependency will always be free of charge.
npm run dev- FastAPI:
http://127.0.0.1:8000 - Vite:
http://localhost:5173 - Electron window starts with the Vite plugin
Backend only: npm run dev:backend
Frontend / Electron only: npm run dev:frontend
GET / returns app name, version, and status: running. POST /command is the main command entry.
See CONTRIBUTING.md and docs/development.md.
Short version: fork → branch → code → tests → PR. Do not bypass the action validator. Do not add IoT or unrestricted shell.
python -m pytest tests/backend
npm test --prefix frontend -- --run
npm run build --prefix frontendCI (.github/workflows/tests.yml) runs these on windows-latest. It does not run Linux desktop automation. Manual scripts such as backend/scripts/verify_semantic_ui.py are local Windows checks and can drive Notepad/Explorer.
- Experimental: apps with weak UIA support will fail more often.
- Observation can miss Explorer paths; context then stays
None. - Replanning is bounded; exhausted budgets fail the goal.
- Speech quality depends on the OS recognizer, microphone, and optional ffmpeg.
- Browser automation is infrastructure plus selected workflows, not “any site.”
- Phase 3 system controls (brightness, volume, media) are not implemented. Do not hardcode “thoda aur” to brightness.
Completed: Phase 0, 1, 2A–2E.
Upcoming (not done): Phase 3 system controls; Phase 4 vision/OCR; Phase 5 unified desktop+browser goals; Phase 6 stronger planning and more allowlisted apps.
Full list: docs/roadmap.md.
- Fork and clone pradipNP/agent-racchha.
- Install (
npm run install:all). - Configure
backend/.envlocally. - Pick an issue: github.com/pradipNP/agent-racchha/issues — drafts also live in docs/github-issues.md.
- Add tests. Run the suites above.
- Open a PR with the template.
Good first themes: docs, selector tests, validation messages, observation summaries, frontend error mapping.
- Computer-control actions are safety-sensitive. Never bypass the validator.
- Do not commit API keys. See SECURITY.md.
- Session context is in-process memory, not a cloud desktop dump. You are still automating your PC.
- Screenshots/OCR are not part of the current phase; do not add silent screen upload.
MIT — Copyright (c) 2026 Racchha AI.
Racchha stands on FastAPI, Electron, React, Windows UI Automation, Chrome DevTools Protocol, and the Gemini API. Contributors who add tests and honest docs are doing the work this kind of agent actually needs.
If Racchha is useful or interesting:
- Star the repository
- Open or discuss an issue
- Start a discussion
- Fork, implement a slice, and open a pull request
Every real contribution helps move Racchha closer to a general-purpose Windows computer-use agent — still experimental, still honest about what is not done yet.