JQ on glasses — Cloud SDK → Bluetooth SDK migration

Decided 2026-07-25 (Bluetooth SDK / own-the-stack). Forcing function: MentraOS 3.0 (Aug 3, 2026) moves miniapps off the cloud; the Cloud SDK is no longer developed and its miniapps stop working in the official Mentra app on Aug 3 (legacy builds available at MentraGlass.com/Legacy until Oct 2026). The Mentra Bluetooth SDK is unaffected and under active development.

What breaks

apps/glasses/ is a MentraOS AppServer built on @mentra/sdk — the deprecated Cloud SDK. It runs on Railway and the glasses reach it via MentraOS cloud. On Aug 3 this stops working in the official app.

Capabilities it uses today (session.*): audio.playAudio (play an HTTPS MP3), audio.speak (TTS), camera.requestPhoto, events.onButtonPress / onDisconnected / onTranscription.

Why NOT fork MentraOS

Forking = maintaining a glasses OS (huge, zero differentiation for us) and chasing a target Mentra just re-architected. The thing a fork would buy — freedom from their cloud + store — is exactly what the Bluetooth SDK already gives: our own iOS/ Android app drives Mentra Live directly over BLE. Ownership without owning an OS.

Target architecture

The phone becomes the JQ brain; the glasses are a BLE peripheral. This converges with [[MOBILE_JQ_PRESENCE]], [[AR_GLASSES_LIVING_VOICE]], and the owned-hardware horizon — and Sophia's "decide locally" seam fits on-phone even better.

JQ mobile app  ──BLE──▶  Mentra Live (mic / camera / speaker / display)
      │
      └── HTTPS ──▶ HiveJournal backend  (UNCHANGED — see below)

The backend is transport-agnostic — we keep all of it

Everything apps/glasses calls today is a plain authed HTTPS endpoint the mobile app calls verbatim; only the glasses transport changes (MentraOS session → BLE):

POST /api/jq/voice/say { text } → { audio_url } (DreamPro speak path)
GET /api/jq/voice/nudge?occasion= (own-voice nudge)
GET /api/downstream/soundtracks/for-phase, GET /api/levers/interventions
AisleAsk: GET /:storeId/next, /route, GET /stores, POST /:storeId/scan-section, POST /:storeId/scan-deal
The whole intent layer (apps/glasses/src/intent.ts, unit-tested) is pure and moves to the phone unchanged — parse utterance → action.

So the migration is a transport swap + a mobile shell, not a rewrite of the JQ logic.

Capability mapping (Cloud SDK → Bluetooth SDK)

Today (MentraOS session)	BLE-SDK target	Confidence
`camera.requestPhoto`	BLE image send (email: "send images locally, without WiFi") → phone → scan endpoints	High — explicitly shipped
`audio.playAudio(mp3)`	play JQ's MP3 to the glasses speaker over BLE, or fall back to the phone speaker	⚠️ Verify BLE audio-out
`audio.speak(text)`	phone TTS (or our cloned-voice MP3 via `/voice/say`) → same audio path	Medium
`events.onTranscription`	mic audio over BLE → phone STT (on-device or cloud)	⚠️ Verify BLE mic streaming
`events.onButtonPress`	BLE button events (but buttons are hardware-reserved on Live)	Low — likely voice-first anyway
`events.onDisconnected`	BLE connection state	High

Phases

Phase 0 — Bridge (do first, ~9 days to Aug 3): keep JQ-on-glasses alive past Aug 3 → runway to Oct 2026. MentraOS Legacy is a separate older build of the phone app (v2.11) that still runs Cloud miniapps — it lives beside the 3.0 official app, not on our server. Our AppServer needs ZERO change (the Legacy page specifies no developer changes; @mentra/sdk ^2.1.29's caret already blocks a jump to the incompatible 3.x line; legacy app 2.11 = same Cloud-SDK generation).
- iOS: install via TestFlight — https://testflight.apple.com/join/D32Vvssy (coexists with the App Store build; use Legacy for JQ).
- Android: sideload the APK — https://github.com/Mentra-Community/MentraOS/releases/download/v2.11/Mentra_2p11_Beta_27.apk (enable unknown sources; may share the package name with the store app → you may have to pick one).
- Our side: keep the Railway apps/glasses service deployed + the app enabled at console.mentraglass.com; do NOT bump @mentra/sdk to 3.x while on the bridge.
Phase 1 — Mobile shell + BLE bring-up: JQ app (reuse [[MOBILE_JQ_PRESENCE]] foundation) pairs Mentra Live via the Bluetooth SDK. Connect, battery, disconnect, button events. Port intent.ts to the phone.
Phase 2 — Speak path (DreamPro-first): utterance → intent → POST /voice/say → play MP3 (glasses over BLE if supported, else phone). This is the build-first path (own-voice encouragement).
Phase 3 — Capture path (AisleAsk / Odessa): "scan this aisle/deal" → BLE image → POST /:storeId/scan-section|scan-deal. Reuses the tuned scan prompts.
Phase 4 — Voice-in: mic over BLE → STT on phone → intent. Verify SDK mic access first; if limited, fall back to phone mic (glasses still show/speak).
Phase 5 — Retire the AppServer: once the mobile path is at parity, decommission the Railway apps/glasses AppServer (keep only as the legacy bridge until Oct 2026).

Phase 1 — detailed scope (researched 2026-07-25)

The SDK research collapsed the biggest unknowns. The Mentra Bluetooth SDK is @mentra/bluetooth-sdk (npm, currently 0.1.20) with first-class React Native / Expo support — and it explicitly does audio playback to the glasses speaker and microphone input (both open questions above → answered YES). So this is a normal RN SDK integration, not a native-bridge project.

Where our app is

apps/mobile = Expo 54 / RN 0.81, managed workflow, expo-av audio, no BLE, no native modules. eas.json exists. The main lift is moving to a custom dev client (the SDK can't run in Expo Go).

Phase 1 goal (one sentence)

On a real device, our Expo dev-client app pairs Mentra Live over BLE and plays a JQ voice line to the glasses speaker — proving the transport + the DreamPro speak path end-to-end. Nothing else.

Workstreams

Dev-build foundation (most of the effort). Config plumbing DONE — app.config.js gates the @mentra/bluetooth-sdk + expo-build-properties plugins + BLE/mic perms + iOS background modes behind MENTRA_BLE=1 (default + Workout Window builds untouched). Remaining is on-device: install + MENTRA_BLE=1 expo prebuild + EAS dev build — runbook at apps/mobile/MENTRA_BLE_SETUP.md. Perms: Android BLUETOOTH_SCAN/CONNECT, ACCESS_FINE_LOCATION, RECORD_AUDIO, POST_NOTIFICATIONS; iOS NSBluetoothAlwaysUsageDescription, NSMicrophoneUsageDescription, background modes bluetooth-central+audio, setAudioModeAsync(...). Risk: prebuild churn on an existing managed app + first native build setup (Apple cert for iOS).
Pairing bring-up. A "Connect glasses" screen using useMentraBluetooth({ defaultModel: DeviceModels.MentraLive }) + useBluetoothEvent('battery'| 'button_press'): scan → connect → requestVersionInfo. DoD: shows "Connected · Mentra Live · 82%" and survives a disconnect/reconnect.
First capability round-trip = audio-out (the wedge). Fetch a JQ MP3 from the existing POST /api/jq/voice/say { text } and play it to the glasses speaker via the SDK audio route; wire a minimal "say a nudge" button (→ /api/jq/voice/ nudge). DoD: JQ speaks in the owner's cloned voice through the glasses, from our own app. (Small spike: confirm the exact audio-play method name in api-reference.md, and that it targets the glasses speaker vs the phone.)
(Parallel, low-risk) Port intent.ts. Copy the pure, unit-tested apps/glasses/src/intent.ts into the app (or a shared pkg) so utterance→action is ready for Phase 2 — no BLE dependency.

Phase 1 Definition of Done

Real device: our dev-client app connects to Mentra Live, shows battery/connection, and plays a JQ voice line through the glasses. Transport + speak path proven.

Effort / gates

~1–2 weeks, most of it Workstream 1 (dev-build plumbing) + confirming the audio API. Needs: the user's Mentra Live + a phone (BLE needs real hardware, no simulator), an EAS account (have it), and — for iOS — a Mac + Apple dev cert. Note: the SDK's Expo plugin bundles onnxruntime (.so pick-first rules) → on-device ML is in the box, worth watching for the Sophia local-NLU angle ([[SOPHIA_LICENSING]]).

NOT in Phase 1

Voice-in/STT, Odessa capture, retiring the Railway AppServer — those are Phases 2–5.

Open questions (verify against the Bluetooth SDK docs when in hand)

~~Audio-out over BLE~~ — ANSWERED YES (2026-07-25): the SDK routes phone audio playback to the glasses speaker. Confirm the exact method name + MP3-vs-stream in api-reference.md during Phase 1 WS3.
~~Mic streaming over BLE~~ — ANSWERED YES: the SDK exposes microphone input ('microphone' events + RECORD_AUDIO). Wire our STT in Phase 4.
iOS + Android parity — the SDK claims both; confirm the audio/mic bits on iOS.
On-phone Miniapp SDK — track as a fallback distribution path if BLE audio/mic prove limited (keeps store distribution, less ownership).
Sophia on-phone — the local-NLU pre-filter fits the phone model; still gated on the Aquila Labs enterprise license before shipping in any path ([[SOPHIA_LICENSING]]).

Landmines carried over

@mentra/sdk Node-version quirk (run via tsx) is a Cloud-SDK-only problem — gone once we're on BLE.
Buttons are hardware-reserved on Mentra Live → stay voice-first.
Don't over-invest in the legacy AppServer; it's a bridge, not the destination.