How We Built Khandu AI: A Desktop Assistant in Python

# The Idea

It started as a weekend challenge: build something like Jarvis. Not a chatbot wrapper. An actual desktop assistant that could open apps, search the web, and answer questions — right on your Windows PC.

What started as a Python script with subprocess.run and the speech_recognition library turned into something significantly more ambitious.

# Why We Went Offline-First

The first version of Khandu AI relied entirely on an external AI API. It worked great, until it didn't — the moment internet dropped, the assistant was useless.

We made a hard call early: core features must work without internet. This pushed us toward local models (we experimented with Whisper for transcription and smaller quantized LLMs for response generation).

The trade-off was response time — local models are slower — but users told us reliability mattered more than speed.

# The Tech Stack

Here's what's powering Khandu AI under the hood:

# Core dependencies
# text to speech
# app control
# browser automation
# UI automation
# optional cloud AI
# Local LLM

• Speech-to-text: tiny model, 39MB, runs locally
• Text-to-speech: pyttsx3 (offline, no API calls)
• Intent parsing: Custom rule engine + optional GPT-4o for complex queries
• App automation: gui for mouse/keyboard control

# The Biggest Technical Challenges

1. Wake Word Detection**

We tried using pvporcupine for wake word detection. It works, but the free tier felt too limiting. We ended up shipping a hotkey-based trigger (Ctrl+Shift+K) as the default, with an option to enable always-listening mode.

2. Noise Cancellation

Office environments are noisy. The default speech_recognition threshold kept triggering on background noise. We implemented an adaptive noise threshold that calibrates on startup:

with sr.Microphone() as source:
    recognizer.adjust_for_ambient_noise(source, duration=2)

It's not perfect, but it dramatically reduced false triggers.

3. App Context Awareness

"Close the tab" means different things depending on what's open. We built a basic context stack — a queue of recently interacted apps — so the assistant knows what "it" refers to:

"Open Notepad and type Hello World" → opens Notepad, focuses it, types the text.

This required hooking into Windows event APIs to track focus changes, which was tricky but ultimately very satisfying to get right.

What's Next

Version 2.0 (coming soon) ships with:

• Plugin system so you can extend Khandu with custom commands
• Better app integration (VSCode, Chrome, Excel)
• Faster local model (we're evaluating Phi-3 mini)
• A proper installer (no more running from terminal)

If any of this sounds interesting, you can get notified when it launches from our Khandu AI page.

---

Built with ❤️ by AISkyTools. All our tools are free forever — because we build them for ourselves first.