I Built an AI That Replies to My WhatsApp, and Taught It to Ask Permission First

The problem, and what I built

You open WhatsApp after a long day and there's a wall of green badges. A friend asking if dinner is still on. A cousin's voice note. A group where someone has asked you a direct question.

Most replies are tiny: "yes, 8 works", "haha", "send me the link". They pile up, and some get forgotten.

I wanted help with the small ones, without handing my social life to a bot that might say something stupid in my name. So I set one rule before writing any code: the AI has to ask permission first.

The result is a small app on my own machine. It links to my WhatsApp as a "linked device", like WhatsApp Web. It drafts replies that sound like me and shows them on a phone-friendly dashboard, where I send, edit, ignore or regenerate them.

Under the hood, briefly

It's one Node.js app with a SQLite store. Here's the path a message takes.

• The connector uses Baileys, an open-source library that speaks the WhatsApp Web protocol. It's unofficial, so it sits behind a small interface I can swap if it breaks.
• Media becomes text. Voice notes are transcribed, images described, PDFs read. If that fails, the message becomes a clear placeholder, never silently dropped.
• The router saves every message, skips duplicates and checks the chat's mode. In groups it only drafts when I'm @mentioned, by default.
• The context builder sends recent messages, a rolling summary of older ones, a short bio and real examples of how I write in that chat.
• The model can be a hosted one or a fully local one in its own container.

Ordinary group chatter never reaches the AI, which saves money as well as embarrassment.

The trust ladder: OFF, DRAFT, AUTO

Each chat has one setting, its mode:

• OFF: messages are stored, nothing is drafted. Every new chat starts here.
• DRAFT: the AI writes a reply and waits for me.
• AUTO: the AI may send on its own, but only if the reply passes the risk gate. Otherwise it becomes a normal draft.
• PAUSED: a per-chat kill switch: still observing, never drafting.

Above all of this is a global pause that stops every auto-send at once. On a fresh install it starts on, so nothing auto-sends until I choose.

The dashboard tracks how often I send a draft unchanged, edit it or throw it away. After at least 50 reviewed drafts with 85% or more sent unchanged, it shows a "ready for AUTO" hint. Only a hint: nothing flips a chat to AUTO by itself.

A risk gate made of boring rules

Before any auto-send, a short list of checks runs. Any single "yes" holds the draft for me.

The gate asks:

• Is the global pause on?
• Is the incoming message sensitive?
• Is the AI's reply sensitive?
• Is WhatsApp disconnected?

Then it waits a few random seconds and checks the pause and connection again.

"Sensitive" means money, passwords and OTPs, legal matters, medical emergencies and job matters. I can add my own words in a setting. Checking the reply too means the AI can't wander into a sensitive topic on its own.

Why rules, not AI? Rules are free, instant, predictable and easy to explain. They do flag harmless things (a "wifi password" message got held). That's the right way round: a false alarm costs one tap, a miss could cost far more.

If the model fails, the app records a failed draft. It never sends a made-up fallback. Every send, hold and failure is logged with its reason.

Honest about the bot, human about the style

If my friend thinks they're chatting with me and it's actually a model, that's a small deception, even when the reply is harmless.

So every real auto-send starts with a short bot tag. The rule: if a human didn't press send, the other person gets told. Anything I send myself goes out clean, because I chose to send it.

The tag is added only to the delivered message, not the stored copy. Otherwise it would leak into future prompts as if it were how I talk.

Style is learned per chat, because I don't text my mum the way I text school friends. Each chat's prompt sees only that chat's examples, on top of a general baseline.

Just as important is which words count. Messages I type, and drafts I edit, count. Drafts sent unchanged and auto-sends don't: those are the AI's words. Feeding them back would slowly turn my style into the model's, like a photocopy of a photocopy.

A simple style guard also rejects assistant-speak like "Sure!" or "Hope this helps", and essays in reply to one-liners.

Sometimes the best reply is a meme (or a film line)

Some messages don't want words. A friend gets the job, or their flight is delayed again, and the perfect answer is a reaction picture or a well-known Hindi film line. So I added an optional "playful mode".

• Cheap gates first. Playful mode must be on for the chat (it's off by default), the chat must be in DRAFT, nothing may look sensitive, and a cooldown and dice roll keep it rare. A meme is tried on 40% of eligible drafts, at most once per chat every 6 hours; film lines use 25% and 4 hours.
• The AI picks, or says none. It gets a short menu of matching memes or film lines and must answer with one choice or "none". Plans, times, money and anything serious get nothing.
• Code has the last word. Plain code checks the pick: Roman script only, nothing offensive or sensitive, no assistant-speak.
• Always my tap. Extras never appear in AUTO. A meme is shown exactly as it will go out; a film line joins the reply only if I add it.

In groups, the model is told everyone must enjoy it and it must never target one person. When in doubt, nothing. That means fewer memes, which is the right side to err on for jokes in my name.

Two word lists block abuse, violence, sexual content, religion, caste, politics and drugs when the library is built. The tagging model's own safety verdict is a second net.

Memes are drawn by memegen, an open-source renderer I run myself. Its open-source build stamps a website name on every image, so a six-line patch removes that watermark at build time, keeping the licence notice. A meme sent to a friend shouldn't carry an ad.

Privacy, and trying it yourself

It runs in one Docker container. You need a WhatsApp account you're comfortable linking through an unofficial client, which carries some risk, so keep AUTO conservative.

cp .env.example .env
docker compose up -d --build

Scan the QR code from the logs, add an AI provider key, and open the dashboard. A fully local model setup is optional.

What was hard (and what I learned)

• The wrong script. I chat in English, Hindi and Bengali. The usual speech model often wrote Bengali audio in the wrong script, so voice notes from mostly-Indic chats can go to a separate Indic speech model.
• Blank and badly placed memes. The renderer silently dropped Hindi and Bengali letters, so captions are now Roman script only. A two-panel template got text drawn across a face, so outside templates are limited to classic top-and-bottom ones.
• "Successful" empty replies. One reasoning model spent its whole budget on hidden thinking and returned nothing, while reporting success. Empty replies now count as failures.
• Testing ethically. I tested AUTO only in my own "message yourself" chat. Switching on a real group would have sent other people's messages to an AI without asking them, so automated tests covered group features.

Almost every real bug showed up only with real usage, never in a clean demo. Test against real life, but in a safe corner of it.

Wrapping up

The most useful thing I built here isn't the AI. It's the permission system around it: OFF by default, trust earned per chat, a boring rule-based gate, and honesty whenever the bot speaks.

I think that pattern fits any agent that acts on your behalf.

If you want to self-host something like this, here is how I run my own services at home: my palm-sized homelab.

Would you let an AI reply to your messages? Where would you draw the line? Tell me in the comments, and subscribe if you'd like the next build in your inbox.