Testing Whisper‑Tiny

Sometimes, you don’t need fancy gear to have big fun. In this post, I’ll share my experience running the locally installed OpenAI Whisper‑Tiny speech recognition model with the ultimate test setup: a $5‑special “chipiest” microphone. Will it understand me? Will it confuse “coffee with milk” with “coughing in silk”? Let’s find out.

Why Whisper‑Tiny?

• Lightweight: Tiny model → runs on CPU, even on modest laptops.
• Fast: Trade‑off between accuracy and speed, but perfect for experimenting.
• Self‑contained: Works offline—your audio data never leaves your machine.

The Test Dataset

I prepared a set of 50 short phrases covering everyday commands, food, numbers, locations, tongue twisters, and “ummmm” sounds. The idea is to stress‑test the model with a mix of easy wins and likely failure points.

Here are 50 short phrases, clean and ready for testing:

Everyday Commands & Queries

1. Open the door
2. Turn off the lights
3. Call my mom
4. Send a text message
5. What’s the weather?
6. Play the next song
7. Stop the music
8. Start the timer
9. Set an alarm for seven
10. Cancel reminder

Food & Drink

11. Coffee with milk
12. Pizza delivery tonight
13. Two bottles of water
14. Hot chocolate please
15. Fresh apple juice

Numbers & Dates

16. One two three four five
17. Eleven twelve thirteen
18. Twenty twenty-four
19. March third, nineteen ninety-nine
20. Ten o’clock sharp

Locations & Travel

21. Take me home
22. Nearest gas station
23. Central train station
24. Go to the airport
25. Map of New York

Random Words for Clarity

26. Red blue green yellow
27. Cat dog bird fish
28. Alpha beta gamma delta
29. Yes no maybe later
30. Up down left right

Conversational Snippets

31. How are you today?
32. I’m feeling great
33. That was funny
34. I don’t know
35. See you tomorrow

Tech Stuff

36. Open the settings
37. Restart computer now
38. Wi‑Fi disconnected
39. Bluetooth headphones paired
40. Battery is low

Tricky / Fun Utterances

41. She sells seashells
42. Peter picked a pepper
43. Unique New York
44. Toy boat toy boat toy boat
45. The quick brown fox jumps

Edge Cases

46. Zero point zero one percent
47. Nine nine nine nine
48. Zzzzz sound
49. Hmmm, let me think
50. Okay okay okay

Setup

1. Hardware

• Microphone: the cheapest USB mic I could find online (cost less than a sandwich).
• Computer: GPU NVIDIA GeForce GTX 1660 SUPER.

2. Software

• Whisper‑Tiny model downloaded locally – https://github.com/AIHelpers/docker-whisper-tiny
• Desktop audio recording tool (or Python package like sounddevice) to capture clips

Recording the Audio

I recited each of the 50 test phrases into the bargain‑bin mic, making sure to include:

• Clear speaking (to test best case)
• Mumbling and background noise (to test worst case)

Transcription Process

1. Record and transcribe audio.
2. Collect the outputs in a simple table with columns:
   Phrase ID | Original Phrase | Whisper Output | Error/Match

Clear speaking (to test best case)

Phrase ID	Original Phrase	Whisper Output	Error/Match
1	Open the door	Open the door	match
2	Turn off the lights	Turn off the lights	match
3	Call my mom	Go, my mom.	error
4	Send a text message	Send a text message	match
5	What’s the weather?	Wchodz ze wezor	error
6	Play the next song	Play the next song	match
7	Stop the music	Stop the music	match
8	Start the timer	Start the timer	match
9	Set an alarm for seven	Set an alarm for 7	match
10	Cancel reminder	Council Reminder	error
11	Coffee with milk	coffee, vis milk	error
12	Pizza delivery tonight	Pizza delivery tonight.	match
13	Two bottles of water	Two bottles of water	match
14	Hot chocolate please	hot chocolate please	match
15	Fresh apple juice	Fresh Apple Juice	match
16	One two three four five	1, 2, 3, 4, 5	match
17	Eleven twelve thirteen	11 12 13	match
18	Twenty twenty-four	20 24	match
19	March third, nineteen ninety-nine	March, sod, 1999	error
20	Ten o’clock sharp	Then, Ocklock Sharp	error
21	Take me home	Take me home	match
22	Nearest gas station	nearest gas station	match
23	Central train station	Central train station	match
24	Go to the airport	Go to the airport	match
25	Map of New York	map of New York	match
26	Red blue green yellow	Red, blue, green, yellow	match
27	Cat dog bird fish	cat, dog, bird, fish	match
28	Alpha beta gamma delta	Alpha, beta, gamma, delta	match
29	Yes no maybe later	Yes, no, maybe later	match
30	Up down left right	Up, down, left, right	match
31	How are you today?	How are you today?	match
32	I’m feeling great	I’m feeling great	match
33	That was funny	That was funny	match
34	I don’t know	I don’t know	match
35	See you tomorrow	See you tomorrow	match
36	Open the settings	Open the settings	match
37	Restart computer now	Restart Computer Now	match
38	Wi-Fi disconnected	Wi-Fi disconnected	match
39	Bluetooth headphones paired	Bluetooth headphones paired	match
40	Battery is low	Battery is low	match
41	She sells seashells	She sells seashells	match
42	Peter picked a pepper	Peter, pickid, and pepper	error
43	Unique New York	unique New York	match
44	Toy boat toy boat toy boat	to a Buddha, to a Buddh	error
45	The quick brown fox jumps	The quick brown fox jumps	match
46	Zero point zero one percent	0.01%	match
47	Nine nine nine nine	9 9 9 9	match
48	Zzzzz sound	This sound	error
49	Hmmm, let me think	Hmm. Let me sink	error
50	Okay okay okay	Okay okay okay	match

Mumbling and background noise (to test worst case)

Phrase ID	Original Phrase	Whisper Output	Error/Match
1	Open the door	Open the door	match
2	Turn off the lights	It removes the lads	error
3	Call my mom	Goal my mom	error
4	Send a text message	Send it to the message	error
5	What’s the weather?	what is the VZR?	error
6	Play the next song	Blaze the next song	error
7	Stop the music	Stubbs in music	error
8	Start the timer	Starts with time on	error
9	Set an alarm for seven	Set an alarm for seven	match
10	Cancel reminder	Consular Mander	error
11	Coffee with milk	Coffee is milk	error
12	Pizza delivery tonight	Pizza delivery tonet	error
13	Two bottles of water	Two bottles of water	match
14	Hot chocolate please	Hot chocolate please	match
15	Fresh apple juice	Fresh apple juice	match
16	One two three four five	1, 2, 3, 4, 5	match
17	Eleven twelve thirteen	11 12 13	match
18	Twenty twenty-four	20, 24	match
19	March third, nineteen ninety-nine	Mart, soat, 1999	error
20	Ten o’clock sharp	Then a clog sharp	error
21	Take me home	Take me home	match
22	Nearest gas station	nearest gas station	match
23	Central train station	Central Drain Station	error
24	Go to the airport	go to the airport	match
25	Map of New York	Map of the New York	error
26	Red blue green yellow	Red, blue, green, yellow	match
27	Cat dog bird fish	Cat dog bird fish	match
28	Alpha beta gamma delta	alpha, beta, gamma, delta	match
29	Yes no maybe later	Yes, no, maybe later	match
30	Up down left right	up down left right	match
31	How are you today?	How are you today?	match
32	I’m feeling great	I’m feeling great	match
33	That was funny	That was funny	match
34	I don’t know	I don’t know	match
35	See you tomorrow	Siv till morgon	error
36	Open the settings	Open the settings	match
37	Restart computer now	We’re start computer now	error
38	Wi-Fi disconnected	Why do I disconnect?	error
39	Bluetooth headphones paired	Bluetooth et de prendre sprueste	error
40	Battery is low	battery law	error
41	She sells seashells	She sells, she sells	error
42	Peter picked a pepper	Bit of a big, a bit better	error
43	Unique New York	You need milk	error
44	Toy boat toy boat toy boat	Toi bort, toi bort, toi bort	error
45	The quick brown fox jumps	and the quick brown folks jumps	error
46	Zero point zero one percent	0.01%	match
47	Nine nine nine nine	No, no, no, no	error
48	Zzzzz sound	Sound	error
49	Hmmm, let me think	Hmm, let me sink	error
50	Hmmm, let me think	Okay, okay, okay	match

Summary results

Test type	Match	Error	Match %	Error %
Clear speaking	40	10	80%	20%
Mumbling and background noise	23	27	46%	54%

Raw Performance Stats

• Clear speaking:
  • Matches: 40/50
  • Accuracy: 80%
• Mumbling + noise:
  • Matches: 23/50
  • Accuracy: 46%

That’s a huge gap: roughly a 34 percentage-point drop when conditions get difficult.

---

What This Means

• Tiny model strengths:
  • Relatively solid under ideal conditions—80% isn’t bad for a very small model.
  • Fast and resource-efficient, works on lower-powered devices.
• Tiny model weaknesses:
  • Struggles significantly with noisy, “imperfect” speech.
  • This is expected: whisper-tiny has fewer parameters, so its “ear” for dealing with accents, mumbling, and background sounds is limited.

---

How to Interpret

Think of whisper-tiny as the bicycle of ASR models: lightweight, efficient, easy to deploy—but not the champion for carrying heavy loads (like messy audio).
Whereas larger models like whisper-base or whisper-small/medium are like scooters or cars: heavier, need more resources, but handle more complicated journeys.

---

Next Steps You Might Explore

• Comparison with bigger models: Run the same 50-question test with whisper-base or small. You’ll instantly see whether accuracy in noisy cases jumps (it usually does).
• Preprocessing tricks:
  • Noise reduction (e.g. with pyannote.audio or even simple filters).
  • Volume normalization.
• Data augmentation for robustness: If deploying on a custom task, fine-tuning with examples that include your tricky noise situations can dramatically help.
• Error analysis: What types of words were consistently misrecognized—short function words, numbers, names? This often reveals model limits.

---

A Tiny Bit of Humor

In short: whisper-tiny can handle a calm podcast session, but if you invite it to a crowded party, it sort of nods and smiles while guessing what you said.

Conclusion

The evaluation of the Whisper-tiny model demonstrates that it performs reasonably well under clear speaking conditions, achieving 80% accuracy across the test set. However, its performance drops sharply in the presence of mumbling and background noise, where accuracy falls to 46%. This contrast highlights both the efficiency and limitations of the model: while Whisper-tiny is lightweight and suitable for resource-constrained environments, it is not robust enough to handle real-world scenarios where speech may be unclear or disturbed by noise. For applications requiring higher reliability in noisy conditions, using a larger Whisper model or applying audio preprocessing techniques would be advisable.

To obtain more reliable and generalizable results, the evaluation should be expanded with a larger set of questions and a wider range of speakers featuring different accents, languages, and environmental conditions. This broader testing will provide a clearer picture of the model’s strengths and weaknesses across realistic usage scenarios.