Security

Satya-Vani: Catching a Cloned Voice While the Call Is Still On

Seven pluggable signals, one explainable 0–100 risk score that updates every half second: how I built live voice-spoof detection.

Subroto Sanyal

— 7 min read
Share
Satya-Vani logo beside a phone showing an incoming call from an unknown number at 23:40, a waveform turning red, and a high-risk meter
💡
TL;DR
• I built Satya-Vani ("true voice"). It listens to a call and estimates, while the call is still going, how likely it is that the voice is a clone or the caller is running a scam.
• Seven independent signals score every 2-second slice of audio. One small YAML file combines them into a single 0–100 risk score that updates every half second.
• Nothing is a black box. Every score shows what each signal said, how much it counted, and one plain-English sentence explaining why.

When a familiar voice isn't who you think

A late-night call from an unknown number. The voice sounds exactly like someone you love, and they're in trouble. They need money, right now, and you mustn't tell anyone.

Sometimes a second voice joins in, claiming to be from the police or the "cyber crime cell", warning of legal action if you don't transfer the money immediately.

Your brain does what any brain would do: I know that voice, so this is real. That instinct is exactly what the attack relies on.

Modern voice cloning can copy a voice from a few seconds of recorded audio: a voice note, a video, a talk. Caller ID is easy to fake. Calling back only helps if you think of it while someone is shouting at you to hurry.

The same trick works on a bank employee who gets a call that sounds just like their CFO.

🔑
Key idea: don't ask "who is speaking?", which is exactly what a clone fools. Ask "how was this sound made?" A neural voice generator rebuilds a waveform rather than recording a human throat, and the rebuild leaves fingerprints.

What I built

Satya-Vani is a self-hosted service with a browser dashboard. You give it a call, either by speaking into your microphone or by uploading a recording.

It cuts the audio into overlapping 2-second slices and sends each slice past seven independent "judges". It then combines their opinions into one risk score from 0 to 100, which moves as the call goes on.

The dashboard is a plain HTML and JavaScript page with four tabs: Upload a file, Microphone (live), Voiceprints (enroll someone's voice once) and History (every past session, replayable). Enrolled voiceprints are stored as numbers only, never raw audio.

Four steps: a phone call with call details, Satya-Vani slicing audio and scoring it with seven signals, one 0-100 risk score with an explanation, and three action bands: low, elevated, high
Listen, score, explain, act.

The seven signals

  • Acoustic: a neural network trained to tell real speech from synthetic speech, with its final layer retrained on Hindi data.
  • Prosodic: tiny wobbles in pitch, loudness and breathiness. Human throats have them; synthesis only approximates them.
  • Voiceprint / context: compares the voice with an enrolled voiceprint of the person the caller claims to be, or falls back to common-sense scam rules.
  • Intent: a classifier that reads the transcript and rates how well it fits fraud-like labels.
  • Semantic risk: a small local LLM (no cloud call) that looks for urgency, money requests, authority claims and "don't tell anyone".
  • Watermark: checks for the invisible watermark some cloning tools embed in every clip.
  • Phase incoherence: a cheap check on the waveform. Voice generators tend to rebuild "too tidy" phase.

Context clues: the scam script, as rules

Back to that late-night call. Even without any voice analysis, it has red flags a careful friend would notice. When nobody is enrolled, the third signal turns those flags into points:

  • Unknown number: the number isn't on file.
  • Unusual hour: between 22:00 and 07:00.
  • Financial request: words like "transfer", "OTP" or "account", including Hindi and Marathi equivalents.
  • Urgency language: "urgent", "right now", "don't tell anyone".
  • Authority claim: "cyber crime", "police", "this is your bank".
  • Authority and money together: the classic scam script, worth extra points.

A classic scam call trips every rule. Each match points to an actual word in the actual transcript, so you can see exactly what triggered it.

The live transcript only covers about the last 8 seconds, so I made these flags sticky: once a call has heard "cyber crime cell", it remembers that for the rest of the call.

Follow one slice of a live call

Here's what happens every half second while the microphone tab is on:

Sequence diagram with five lanes: browser, ws_router, live transcription, engine, fusion and history. Nine numbered steps from sending a 2-second window to receiving the FusedScore back
One 2-second window's round trip. Transcription runs on its own clock so it never slows the score.
  • The browser cuts a 2-second window of mic audio and sends it over a WebSocket with the call details (known number, hour, claimed identity).
  • Every 4 seconds of new audio, a background task transcribes the last 8 seconds. Separate tasks run the intent classifier and the local LLM, so a slow one never blocks the others.
  • The latest finished transcript and flags are merged into this window's context.
  • The engine applies the sticky flags and runs all seven detectors. Each one scores or abstains.
  • The results are combined into one score, saved to history and logged.
  • The score and its explanation go back over the same WebSocket, and the meter moves.

Uploads use the same engine, but transcription, intent and the LLM run once for the whole file.

One explainable score

Think of the seven signals as a panel of judges. Some are more experienced, so their vote counts for more. The weights live in one config file.

Concept diagram of the risk formula: bar chart of the seven configured weights (acoustic 0.60, prosodic 0.20, third signal 0.20, intent 0.15, semantic 0.15, Perth watermark 0.15, phase incoherence 0.10), steps for abstain-aware renormalisation and per-signal smoothing with alpha 0.35, the weighted-sum formula, and the three bands
The real weights, smoothing and bands from the config file.

Three rules turn the votes into one score:

  1. Silent judges don't vote zero. If there's no transcript yet, or nobody is enrolled, that signal abstains and the other weights are rescaled.
  2. Opinions are smoothed. One odd half-second can't swing the score wildly.
  3. The number is banded. 0–34 is Low (monitor), 35–69 is Elevated (quiet advisory) and 70–100 is High (block the sensitive action and call back on a known number).

Newer, less-proven signals carry small weights, so none of them can push a genuine call into High on its own.

A score of 72 with no explanation is useless, especially to a bank clerk who has to justify blocking a transfer. So every result lists each signal's score (or why it abstained), its weight, its exact contribution, and one plain sentence, such as "the call was placed at an unusual hour (23:00)".

The neural acoustic model is the honest exception. It can't point at a specific feature, so its sentence says what it actually knows, the probability it assigned to "synthetic", rather than inventing a reason.

Scores are also reproducible: the same audio, context and config always give the same score.

Dashboard result: risk score 74 HIGH with per-signal bars and explanations
A synthetic scam call scores 74, HIGH, with every signal explained.

Design decisions worth stealing

  • A new signal is one class plus one YAML entry. No engine or API changes. That's how four signals were added beyond the original three. The trade-off: a typo in the YAML shows up at startup, not in your editor.
  • Abstain, don't zero. In a fraud detector, a zero quietly means "looks safe", the most dangerous possible default. Because weights shift when a signal abstains, each result reports both its configured and its effective weight.
  • Run expensive models once, read many times. Transcription and the LLM run in the background, not for every 2-second slice. The trade-off: transcript signals lag speech by a few seconds.

What was hard (and what I learned)

  • A fast model that lies quietly. The Indian-language speech engine was more accurate and much faster on Hindi and Marathi. Given English, though, it returned Malayalam-script gibberish with no error. Hence a quick language check in front of it.
  • Dinner plans are not fraud. The intent model rated "are we still on for dinner tonight?" as 38.9% "creating urgency". That's why it carries a low weight, and why the local LLM runs alongside it. In hand checks, the LLM scored that sentence as zero urgency.
  • Real time is unforgiving. Routing live-mic audio through the Indian-language engine built up a backlog of 9–13 seconds. The fix: decide the language once per call, and keep one persistent connection per call.
  • My hand-tuned rule was a coin toss. Measured on real data, my prosody rule scored exactly 50% balanced accuracy. A small logistic regression on the same three features reached 82.5% on held-out data.

Try it yourself

The quickest way is the live dashboard at play.sanyal.net/satya-vani. Upload a sample recording, add some call details, and watch every signal explain its part of the score.

Try Satya-Vani live

To run your own copy, you need Docker:

docker compose up --build
open http://localhost:8020/

What's built, and what's next

Built: the Live Call Path (slicing, seven signals, scoring and explanations), plus the dashboard with upload, live mic, voiceprints and history.

Next: a mock banking screen that actually blocks a transfer on High, calibrated thresholds, and more test data to make the Hindi numbers solid.

Closing

The instinct "I know that voice" is exactly what voice cloning exploits. Satya-Vani's answer is to stop trusting that one instinct and gather many small, explainable clues instead, while the call is still on. You can try the live dashboard and see it for yourself.

If practical security tooling is your thing, I also wrote about SentinelScan, a self-hosted security workbench for homelabs and small IT teams.

If you've worked on voice fraud, deepfake detection or Indian-language speech, I'd love to hear what you'd add as an eighth signal. Leave a comment, and subscribe if you'd like the next build log in your inbox.

Read more