Forrest Note: AI Voice Notes on ESP32-S3 with Whisper & GPT

Oct 13, 2026 · 6 min read

What You Will Build

Forrest Note is firmware that turns a Waveshare ESP32-S3 e-ink development board into a pocket AI notepad. The workflow is deliberately minimal: hold the record button, say whatever is on your mind, and release. The device transcribes your audio with OpenAI Whisper, sends the raw transcript to gpt-4o-mini for cleanup, and pushes a finished Markdown file — complete with a YAML frontmatter title, a one-sentence summary callout, topic backlinks for Obsidian, and the verbatim transcript tucked in a foldable quote block — directly to a GitHub repository. Open that repository as an Obsidian vault and your notes organise themselves.

This is a fork of the original Pala Note project, which contributed the hardware bring-up, e-ink and audio drivers, recording engine, and 3D-printable enclosure. Forrest Note adds the AI cleanup layer, GitHub sync, runtime Wi-Fi and key provisioning, real TLS certificate validation, OTA updates, two-way delete, and calendar event extraction from note content.

What You Need

Hardware

• Waveshare ESP32-S3 1.54-inch e-Paper AIoT Development Board — specifically the B/W (non-"G") variant. The board integrates the MCU, display, audio codec with microphone and speaker, microSD slot, RTC, SHTC3 sensor, and LiPo charging in one unit.
• A microSD card (for on-device note storage).
• A LiPo battery (supported by the onboard charger).
• A USB-C cable for flashing.
• The 3D-printable case from the original Pala Note project at https://ko-fi.com/s/674a1a82e0 (not redistributed in this repo).

Accounts and keys

• An OpenAI API key with billing enabled (for Whisper transcription and GPT-4o-mini cleanup).
• A GitHub repository (can be private) to store your notes, plus a fine-grained Personal Access Token with Contents: Read and write permission on that repo.
• Your 2.4 GHz Wi-Fi credentials (the ESP32-S3 radio is 2.4 GHz only).

None of these go into source code. You enter them through a captive portal after flashing.

Software (build machine)

• arduino-cli (or Arduino IDE as an alternative)
• ESP32 Arduino board core version 3.2.0
• Arduino libraries: Adafruit GFX Library and ArduinoJson

Typical use cases

How the AI Pipeline Works

Understanding the pipeline helps when things go wrong.

1. Record: holding the record button captures audio through the ES7210 microphone codec.
2. Transcribe: on sync, the raw audio file is sent to OpenAI's whisper-1 model, which returns a verbatim transcript.
3. Enrich: the firmware sends that transcript to gpt-4o-mini, which returns a structured response containing a one-word topic title, a one-sentence summary, a rewritten body (filler words, false starts, and repetition removed, but every fact and number preserved), up to six topic tags, and — when the note describes a dated plan — a calendar event object with resolved start and end times.
4. Write: the firmware assembles a Markdown file named after the topic (e.g. Soho.md), embedding all of the above in YAML frontmatter and Obsidian-compatible callout blocks. The verbatim transcript is kept in a foldable > [!quote]- Original transcript callout.
5. Push: the file is uploaded to your GitHub repository via the GitHub Contents API over HTTPS validated against the Mozilla CA bundle.

A chunked-HTTP decoder is included to handle OpenAI's Transfer-Encoding: chunked responses correctly — without it the enrichment step silently fails.

Build and Flash

Step 1 — Install arduino-cli

# macOS (Homebrew)
brew install arduino-cli
# or the official installer
curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh

Step 2 — Add the ESP32 core and libraries

arduino-cli config init
arduino-cli config add board_manager.additional_urls \
  https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
arduino-cli core update-index
arduino-cli core install esp32:[email protected]
arduino-cli lib install "Adafruit GFX Library" "ArduinoJson"

Step 3 — Compile

Run this from the root of the cloned repository:

arduino-cli compile \
  -b "esp32:esp32:esp32s3:PSRAM=opi,PartitionScheme=custom,CDCOnBoot=cdc,FlashSize=8M" \
  ./forrest_note

A successful build reports something like Sketch uses ~1.43 MB (8%) of program storage space.

Step 4 — Flash (read the BOOT quirk first)

This board cuts power to its USB port about one second after plug-in unless it is parked in the ROM bootloader. If you plug in and immediately try to flash you will get a port-not-found error. The correct sequence:

1. Press and hold the record button (BOOT / GPIO0).
2. While still holding, plug in the USB-C cable.
3. Keep holding through the entire write.

# Find your port first
arduino-cli board list

# Then compile and upload in one step
arduino-cli compile --upload -p /dev/cu.usbmodemXXXX \
  -b "esp32:esp32:esp32s3:PSRAM=opi,PartitionScheme=custom,CDCOnBoot=cdc,FlashSize=8M" \
  ./forrest_note

Replace /dev/cu.usbmodemXXXX with your actual port (/dev/ttyACM0 on Linux, COMx on Windows). When the terminal prints Hash of data verified. and Hard resetting..., release the button.

Arduino IDE alternative: open forrest_note/forrest_note.ino, select ESP32S3 Dev Module, and set PSRAM → OPI PSRAM, Partition Scheme → custom, USB CDC On Boot → Enabled, Flash Size → 8 MB. Hold BOOT, plug in, click Upload.

First-Time Setup via the Hotspot

The firmware ships with no credentials stored. On first boot the device broadcasts an open Wi-Fi network named ForrestNote-Setup.

1. Power on the device.
2. On your phone or laptop, connect to ForrestNote-Setup.
3. Open a browser to http://192.168.4.1. Most phones pop up the captive portal automatically.
4. Fill in the form:

5. Tap Save. The device stores everything in on-device NVS (non-volatile storage) and reboots onto your Wi-Fi.

You can revisit this page any time at http://<device-ip>/provision. Leaving a field blank keeps its current value. OTA firmware updates are also available at /ota in the same portal.

Extending the Build and Known Limitations

Ideas to take it further

• Calendar sync bridge: the firmware already extracts event frontmatter (event_title, event_start, event_end, event_allday) when a note describes a dated plan. A small script on your computer or a GitHub Action could read those fields and create calendar entries in your calendar app of choice.
• Obsidian auto-pull: clone your notes repo as an Obsidian vault and install the Obsidian Git community plugin configured to pull every minute. New notes will appear automatically without any manual action.
• Private vault: the GitHub repo can be private — the firmware uses the GitHub Contents API with your PAT, so visibility does not affect sync.
• Tag index pages: the firmware auto-builds a Map of Content (MOC) page per tag in your vault, so browsing by topic works out of the box in Obsidian's graph view.

Limitations to keep in mind

• The firmware targets the Waveshare ESP32-S3 1.54-inch B/W board specifically. The four-colour "1.54G" variant is not supported.
• The 3D-printable enclosure is not in this repository. You must download it separately from the original Pala Note project.
• Transcription and AI cleanup require an active internet connection and a funded OpenAI account — there is no on-device inference.
• Wi-Fi is 2.4 GHz only; 5 GHz networks will not appear.
• The project was created in June 2026 and has no released binary yet, so you must build from source.

Facts in this article come from the project's public README and GitHub metadata at the time of writing. Images belong to their respective owners and link back to the original source.