How I Built VoiceMax: Reading Emotion From a Voice Recording With Three Small AI Flows

By Tanbir Hossain Ramim. Project page: VoiceMax. Source: github.com/TanbirRamim/VoiceMax.

VoiceMax started at Hackaburg 2025, where I built it with my team (the footer still says "Team 2.1"). The idea was simple: your voice carries a lot of emotional information you do not consciously notice, so let people record a few seconds of speech and get a short, honest reading of how they sound, followed by something supportive.

It is a Next.js app with TypeScript, shadcn/ui and Tailwind on the front, and Genkit with a Gemini model doing the analysis. This post walks through how it is put together and the decisions I would keep.

One model, three narrow jobs

The whole AI layer is configured in seven lines:

import {genkit} from 'genkit';
import {googleAI} from '@genkit-ai/googleai';

export const ai = genkit({
  plugins: [googleAI()],
  model: 'googleai/gemini-2.0-flash',
});

The temptation at a hackathon is to write one giant prompt that returns everything at once. I split it into three flows instead:

1. analyzeAudioEmotion listens to the recording and describes it.
2. suggestAdditionalEmotions takes that description and proposes up to three secondary emotions.
3. providePersonalizedFeedback takes only the primary emotion and writes feedback, calling a tool when the emotion is negative.

Each flow has one job, one input schema and one output schema. That made prompts much easier to iterate on, because when the feedback was off I knew exactly which prompt to change, and I could run each flow on its own in the Genkit developer UI (npm run genkit:dev).

Getting structured answers out of audio

The first flow is the only one that sees audio. The recording goes in as a base64 data URI, and the output is a Zod schema with five fields:

const AnalyzeAudioEmotionOutputSchema = z.object({
  primaryEmotion: z.string().describe('The primary emotion expressed in the audio.'),
  perceivedStressLevel: z.string().describe('A qualitative description of the perceived stress level in the voice (e.g., calm, moderate stress, high tension).'),
  speechCharacteristics: z.string().describe('Observations about speech patterns like pace, pauses, or fluency (e.g., fluid and confident, some hesitation, frequent pauses, rapid pace).'),
  perceivedConfidence: z.string().describe("Description of the speaker's perceived confidence (e.g., confident and assertive, somewhat hesitant, appears unsure)."),
  vocalEnergy: z.string().describe("Qualitative assessment of the vocal energy or enthusiasm conveyed (e.g., high energy, moderate, low energy/flat)."),
});

The .describe() strings matter more than they look. Genkit passes the schema to the model, so the descriptions double as instructions, and giving example values ("calm, moderate stress, high tension") keeps the answers short and comparable instead of turning into paragraphs.

All five fields are qualitative strings on purpose. A model listening to ten seconds of audio has no business producing "stress: 73%". A phrase like "some hesitation noted" is closer to what it can actually tell, and it reads better to the person on the other side.

The prompt itself passes the audio through Handlebars media syntax:

Audio: {{media url=audioDataUri}}

Chaining the flows without a second audio upload

Only the first flow gets the audio. The second one gets a text summary built from the first flow's output:

const suggestionsInput: SuggestAdditionalEmotionsInput = {
  primaryEmotion,
  audioAnalysisContext: `The primary emotion detected is "${primaryEmotion}". The voice also showed signs of "${perceivedStressLevel}" stress, speech characteristics were noted as "${speechCharacteristics}", perceived confidence as "${perceivedConfidence}", and vocal energy as "${vocalEnergy}". Consider nuances.`,
};

This keeps the audio payload to a single request, and it means the secondary emotions are reasoned from the same observations the user sees on screen, so the two parts of the result cannot contradict each other in obvious ways.

Letting a tool, not the model, write the exercise

The feedback flow was the one I cared most about getting right. If someone sounds anxious, I did not want the model improvising breathing instructions. So the exercise text comes from a Genkit tool with fixed answers:

async (input) => {
    const emotion = input.emotion.toLowerCase();

    if (['anxious', 'anxiety', 'stressed', 'stress', 'worried', 'nervous', 'fear'].some(e => emotion.includes(e))) {
      return 'Try Box Breathing: Inhale for 4 seconds, hold your breath for 4 seconds, exhale for 4 seconds, and then hold your breath again for 4 seconds. Repeat this cycle several times to calm your nervous system.';
    }
    // ... sad and angry branches, then a mindful-breathing default
}

The prompt then tells the model that for negative emotions it must call breathingExerciseSuggestion and put the tool's output in the suggestion field verbatim, with no "Here is the exercise:" wrapper. For positive emotions it writes a short tip itself and must not call the tool.

The split is deliberate. The model is good at the empathetic sentence in feedback. The deterministic part, the thing someone might actually follow, is plain code I can read and test. The matching uses includes, so "slightly anxious" or "stressed out" still land in the right branch.

Recording audio in the browser

The recorder is the most defensive code in the project, because microphone handling is where things break across browsers. Before creating a MediaRecorder it negotiates a format:

let options = { mimeType: 'audio/webm' };
if (MediaRecorder.isTypeSupported && !MediaRecorder.isTypeSupported(options.mimeType)) {
  options.mimeType = 'audio/ogg';
  if (!MediaRecorder.isTypeSupported(options.mimeType)) {
    (options as any).mimeType = '';
  }
}

WebM first, Ogg second, and otherwise an empty string so the browser picks its own default. When recording stops, the chunks become a Blob, the file name takes its extension from the real MIME type, and a FileReader turns it into the data URI the first flow expects.

Permission errors get their own messages: NotAllowedError explains how to re-enable the microphone, NotFoundError says no microphone was found. Every path that acquires a stream also stops its tracks on reset, so the browser's recording indicator actually goes away when the user starts over. That cleanup is spread over four steps in a useEffect, and every one of them was added after the indicator stayed on in testing.

Errors a person can act on

The three flows run one after another inside a single try. When something fails, the raw error is mapped to a sentence a user can do something with:

if (err.message.includes('429 Too Many Requests') || err.message.includes('QuotaFailure') || err.message.includes('rate limit')) {
  userFriendlyError = 'Analysis failed due to API rate limits. ...';
} else if (err.message.includes('400 Bad Request') || err.message.toLowerCase().includes('invalid argument')) {
  userFriendlyError = 'Analysis failed: The recorded audio might be too short, silent, corrupted, or in a format the AI could not process. ...';
}

A 400 from the model almost always meant the recording was too short or silent, so the message says that instead of "Bad Request". Anything else is trimmed and capped at 200 characters so a stack trace never ends up on screen.

What I would change

Reading the code again, there is one thing I would fix first. After each flow the page calls setAnalysisResult with a partial result, with a comment saying it is to "show partial results sooner". But the results section only renders when !isLoading, and loading stays true until all three flows finish. So the progressive rendering is written but never visible. The fix is small: render each card as soon as its field exists and keep the spinner only for the parts still pending.

The other change is running flows two and three in parallel. The feedback flow only needs primaryEmotion, so it does not have to wait for the secondary emotions.

The pattern I would reuse everywhere, though, is the shape of the AI layer: small flows with typed inputs and outputs, and plain code for anything that must be exactly right.

If you want to read the whole thing, the source is on GitHub, and there is a short summary on the VoiceMax project page.