How to Build an AI Podcast Show-Notes Workflow with Whisper and Notion

Why Build It in Small, Public Steps

Shipping a workflow in small steps and showing it to the people it serves exposes wrong assumptions early, before you polish something nobody needs. For an AI-generated show-notes pipeline, the friction points to watch are transcript accuracy, formatting consistency, and how easily listeners can find past episodes. Tackle them one at a time.

Core Components of the Pipeline

Component	Role	Typical Tool	Free / Low‑Cost Alternative
Audio capture	Record episode	Any recorder (phone, USB mic)	Smartphone voice memos
Speech‑to‑text	Generate raw transcript	OpenAI Whisper (local)	Whisper.cpp, Vosk
Speaker diarization (optional)	Label who said what	PyAnnote, WhisperX	Manual tagging in transcript
Clean‑up & summarization	Remove filler, create bullet points	GPT‑4 via API, Claude	Local LLMs (Llama 2 7B) via Ollama
Knowledge‑base entry	Store notes, links, timestamps	Notion database	Airtable, Coda
Automation	Move files between steps	Zapier, Make (Integromat)	n8n (self‑hosted)
Publishing	Export to blog or newsletter	Notion → Web‑share, RSS	Ghost, Substack

The free alternatives are functional but may require more setup time or accept slightly lower accuracy. Choose based on your tolerance for maintenance versus upfront cost.

Step‑by‑Step Workflow

1. Record and Upload

   • Save the episode as an MP3/WAV file in a dedicated folder (e.g., podcast/raw/).
   • Use a naming convention: YYYY-MM-DD_episode-title.ext. Consistency makes later automation reliable.

2. Transcribe with Whisper

   • Run locally: whisper audio.mp3 --model medium --language en.
   • Output: audio.txt (plain text) and audio.vtt (timed subtitles).
   • Tip: Keep the model on a GPU if possible; otherwise, the tiny model runs on CPU in ~2‑3x real‑time and is sufficient for clean speech.

3. Optional Diarization

   • If you need speaker labels, feed the VTT into WhisperX or PyAnnote.
   • For most indie podcasts, a single‑speaker assumption works; you can manually add [Host] or [Guest] tags later.

4. Clean‑up with LLM

   • Prompt example:

     You are a podcast editor. Convert the following transcript into concise show notes:
     - Remove filler words (um, uh, like).
     - Group related sentences into bullet points.
     - Add timestamps (MM:SS) for each major topic.
     - Include a 2‑sentence summary at the top.
     

   • Send the raw text to the LLM API (temperature 0.2 for deterministic output).
   • Store the LLM response as shownotes.md.

5. Import to Notion

   • Create a Notion database with properties: Episode Title, Date, Audio File (file), Transcript (text), Show Notes (text), Tags (multi‑select), Published? (checkbox).
   • Use Zapier/Make: trigger on new file in podcast/processed/ → create Notion page → map fields.
   • Free alternative: n8n workflow watching a Google Drive folder and using the Notion API node.

6. Link and Tag

   • Add relevant tags (e.g., #AI, #Startup, #Tooling).
   • Insert links to mentioned tools, articles, or GitHub repos directly in the Notion page.
   • Use Notion’s relation feature to connect episodes to a “Resources” database for cross‑episode discoverability.

7. Publish / Share

   • Enable public sharing on the Notion page or export as HTML via Notion’s export.
   • If you maintain a blog, set up a simple RSS feed from the Notion database using a service like Super.so or Fruition.

What to Watch For

1. Bigger Whisper models give diminishing returns

Each step up in Whisper model size improves accuracy but costs more CPU time, and the gains shrink as you go up. Test small and medium on two or three of your own episodes and stop at the smallest model whose transcript you would not have to fix by hand.

2. Speaker labels may not be worth automating

Automatic diarization adds processing time and still mislabels speakers in crosstalk. For a show with one host and one guest, adding [Host]/[Guest] tags by hand is often faster and more reliable. Keep diarization optional for interview-heavy episodes.

3. Fix the prompt, not the output

If you keep editing the generated notes by hand, put that editing into the prompt instead: remove filler, add timestamps, write a summary, match your house style. A prompt you refine once saves you cleanup on every later episode.

4. Plan for Notion at scale

Large Notion databases get slower to load and to query through the API. Archive old episodes to a separate database, use filtered views instead of loading every row, and consider Airtable or Coda if the show grows large.

5. Make the automation idempotent

A re-uploaded file or a retried run can create duplicate pages. Add a "processed" flag, or check Notion for an existing page with the same title before creating one. This matters most when the pipeline runs on a schedule.

6. Ask readers what they want from the notes

Share a live notes page with your audience and ask what they want. Additions worth offering: a TL;DR block at the top (an LLM summary does this), links to timestamped audio, and a downloadable version (Notion can export to PDF).

Checklist for Replicating the Workflow

• Choose transcription model (Whisper medium recommended).
• Set up a folder hierarchy: raw/, processed/, archive/.
• Write LLM prompt for show‑notes generation; test on 2‑3 episodes.
• Create Notion database with needed properties.
• Build automation (Zapier/Make/n8n) to move file → transcript → notes → Notion.
• Add idempotency check (title lookup).
• Enable public sharing or export for distribution.
• Schedule a weekly review to prune old files and update prompts.