Move media from Drive to a Cloud Storage bucket with stored thumbnails

docs/data.md

@@ -1,6 +1,6 @@
 # Data model
 
-The entities the directory serves, and how they are assembled. Structured data lives in one Google Sheet; blobs (photos, audio) are files beside it in the community shared drive, discovered by naming convention. Access rides on drive membership: the service account is a content manager of the drive, never granted through project IAM. The organized model is held in memory — nothing computed is ever written back to the sheet.
+The entities the directory serves, and how they are assembled. Structured data lives in one Google Sheet in the community shared drive, reached through drive membership rather than project IAM; blobs (photos, audio) are objects in the media bucket, reached through project IAM. Each has exactly one home. The organized model is held in memory — nothing computed is ever written back to the sheet.
 
 ## Spreadsheet layout
 
@@ -18,7 +18,7 @@ School structure lives nowhere in the sheet: membership and the classroom and cr
 1. Read the raw import.
 2. Transform each import row into canonical person and household records (explosion, below). The Name to Email mapping applies during this step, since it fixes the key everything else uses.
 3. Apply the Overrides tab onto the canonical records by email: unflagged rows patch existing records, flagged rows create new ones.
-4. List the media drive and attach photo and pronunciation blobs to records by filename convention.
+4. List the media bucket and attach photo and pronunciation blobs to records by object-name convention.
 5. Hold the organized result in memory; the server refuses to start if the load fails, and the model reloads periodically.
 
 Because Overrides rows are authored post-transform, their values skip import normalization — so canonical-value validation runs on every layer, not just the import.
@@ -53,7 +53,7 @@ The explosion turns each import row into one student record plus up to four adul
 Canonical model columns, keyed by lowercased email, one row per person. Three kinds of content share the tab, distinguished only by authorship and one flag:
 
 - **Corrections** (hand): fix anything the import gets wrong — swapped name fields, bad phone numbers — and carry person flags with no import source, like room-parent assignments and the new-to-Helios marker.
-- **Self-service text** (app): facts, pronouns, preferred name, phone, address — the latter two hideable via the `-` clear — and the Opted Out flag. Every self-service edit warns that it doesn't affect the values shown in Veracross. The app writes these cells directly; moderating a contribution is the same act as any other correction. Photo and pronunciation uploads go straight to the media drive and never touch the sheet.
+- **Self-service text** (app): facts, pronouns, preferred name, phone, address — the latter two hideable via the `-` clear — and the Opted Out flag. Every self-service edit warns that it doesn't affect the values shown in Veracross. The app writes these cells directly; moderating a contribution is the same act as any other correction. Photo and pronunciation uploads go straight to the media bucket and never touch the sheet.
 - **Additions** (hand, flagged): people with no import row at all. The flag inverts the source expectation.
 
 Cell semantics are sparse: an empty cell contributes nothing, `-` clears the underlying value. An addition is just an override applied to an empty base record, so the merge logic is uniform; the flag selects the validation instead:
@@ -69,11 +69,13 @@ Family-level fields (address, family photo caption, family phone) ride on a pare
 
 **Opted Out** removes the person entirely at load: their record, their membership in families and parent-contact lists, and any room-parent assignment all vanish from the model. Because viewing the directory requires being in it, opting out also locks the person out — they get a permissions error until the school clears the flag. People set it from their own page, parents set it for their kids (each with a confirmation spelling out the consequences), or an admin sets the cell by hand.
 
-Every change to Overrides appends a Change Log row: timestamp, actor, the row's email, then the previous value of each column that changed — `-` marking a previously empty cell, untouched columns left blank. Media uploads are not logged here; the drive archive is their history.
+Every change to Overrides appends a Change Log row: timestamp, actor, the row's email, then the previous value of each column that changed — `-` marking a previously empty cell, untouched columns left blank. Media uploads are not logged here; bucket object versions are their history.
 
 ## Media blobs
 
-Photos and pronunciation recordings are files in the media shared drive, named by convention: `<email local part>-photo` and `<email local part>-pronunciation` for people, `<family key hash>-photo` and `<family key hash>-pronunciation` for families. Presence means existence — no sheet cell records a filename — and freshness comes from the file's modified time. Uploads replace the file and archive the previous version; superseded versions stay in an archive folder.
+Photos and pronunciation recordings are objects in the media bucket `gs://heliosian-media`, under a `people/` or `families/` prefix and named by convention: `<email local part>-photo` and `<email local part>-pronunciation` for people, `<family key hash>-photo` and `<family key hash>-pronunciation` for families, each keeping its source extension. Presence means existence — no sheet cell records a filename — and freshness comes from the object generation.
+
+Every image is stored with a `-thumb.jpg` sibling written at the same time, so startup loads thumbnails instead of computing them; an image object with no stored thumbnail is fatal. Uploads overwrite the object and its thumbnail, and bucket versioning retains the superseded generations. An upload arriving in a different format also deletes the previous extension's object, so a key never resolves to two files.
 
 ## Person
 
@@ -84,8 +86,8 @@ One record per person, all roles in one shape:
 - **Roles**: student, parent, and staff booleans, derived from sourcing; combinations are valid. Display strings derive from the flags.
 - **New to Helios**: override-carried flag marking people who just joined the community; drives the matching filter toggle.
 - **Pronouns**: optional; a curated list plus a freeform escape hatch.
-- **Pronunciation**: optional audio recording of the person's name, from the media drive.
-- **Photo**: official portrait or personal upload, from the media drive; people may opt for an illustrated avatar instead.
+- **Pronunciation**: optional audio recording of the person's name, from the media bucket.
+- **Photo**: official portrait or personal upload, from the media bucket; people may opt for an illustrated avatar instead.
 - **Facts**: optional about-me text — first-person blurbs for students, professional bios for staff.
 - **Student fields**: grade, classroom, and crew from the homeroom split; parent contact emails derive from the student's household adults.
 - **Staff fields**: job title, department, grade band, and classroom/crew assignment for teaching staff.
@@ -98,8 +100,8 @@ A household groups adults and kids; a student belongs to one household normally,
 
 - **Key**: a hash of the sorted emails of every member — students and adults alike — so identity is order-insensitive and derives from nothing but membership. Any membership change (new student, student leaves, parent change) produces a new key, deliberately: the family's URL and photo association reset along with its composition.
 - **Members**: the adults in the household and the students whose rows name it.
-- **Photo and caption**: the family photo from the media drive plus a who's-who description naming everyone in it.
-- **Pronunciation**: optional audio recording of the family name, from the media drive.
+- **Photo and caption**: the family photo from the media bucket plus a who's-who description naming everyone in it.
+- **Pronunciation**: optional audio recording of the family name, from the media bucket.
 - **Address**: as much as the family chooses to share — full postal address or just city and state, seeded from the import and updatable via self-service.
 - **Phone**: optional household phone.
 

docs/deploy.md

@@ -32,7 +32,7 @@ The service was created once with:
 Each flag is load-bearing:
 
 - `--service-account` — the runtime identity. Application-default credentials inside the container resolve to `directory@` through the metadata server; there is no key file anywhere in the system.
-- `--min-instances 1` — startup preloads every media file from Drive before listening (about two and a half minutes all told); far too slow for scale-to-zero, and close enough to Cloud Run's four-minute startup probe window to watch as the media set grows.
+- `--min-instances 1` — startup preloads every media object before listening (well under a minute); still far too slow for scale-to-zero.
 - `--max-instances 1` — the directory model and blob store live in per-instance memory with no cross-instance coherency; a self-service edit refreshes only the instance that handled it, so a second instance would serve stale data.
 - `--memory 4Gi` — the blob store holds all media and thumbnails in RAM, and the preload peaks well above the steady-state footprint (a roughly 600 MB blob store OOMed a 2 GiB instance during preload). The startup log line `blob store: … MB in memory` reports the steady state; resize when boots start failing or the footprint approaches half the limit.
 - `--no-cpu-throttling` — the directory model and blob store refresh on five-minute tickers between requests; default throttling would starve them.
@@ -53,11 +53,18 @@ Secret Manager secrets, delivered as environment variables per `--set-secrets` a
 - `heliosian-geocoding-key` — the Geocoding API server key, mirrored locally as `creds/geocoding.key`
 - `heliosian-maps-browser-key` — the Maps JavaScript browser key, mirrored locally as `creds/maps.key`
 
+## Media storage
+
+Photos and pronunciation recordings live in `gs://heliosian-media` (us-west1, uniform access, public access prevented, object versioning on), the single source of truth for media — see `docs/data.md` for the naming convention and stored thumbnails. It sits in the same region as the service, so the whole set preloads into memory in about half a minute with no per-request throttle to work around, and versioning carries the upload history that used to need an archive folder.
+
+Because the bucket is private and every read goes through the app's own sign-in gate, no object is ever publicly readable; the service reads and writes it as `directory@`.
+
 ## IAM
 
 `directory@heliosian.iam.gserviceaccount.com` is both the data identity and the runtime identity:
 
-- Data access: content manager on the community shared drive — which covers editing the `Directory` sheet inside it (self-service edits write cells and append to the Change Log tab) and managing media (uploads create files and archive old versions). Shared in Drive directly, never through project IAM.
+- Data access: content manager on the community shared drive, which covers editing the `Directory` sheet inside it (self-service edits write cells and append to the Change Log tab). Shared in Drive directly, never through project IAM.
+- Media: `roles/storage.objectAdmin` on `gs://heliosian-media`, granted on the bucket.
 - Runtime: the Cloud Run service runs as it, and it holds Secret Manager Secret Accessor on each secret individually.
 - Humans: `roles/iam.serviceAccountTokenCreator` on this account enables the local impersonation that real-data development uses (`docs/dev.md`).
 

docs/dev.md

@@ -16,14 +16,14 @@ In sample mode `tools/screenshot` captures pages directly, no session needed (se
 
     DIRECTORY_SHEET=<spreadsheet id> go run .
 
-serves from the production spreadsheet and media drive (see `docs/data.md`) and turns on the full stack. The model loads at startup — the server refuses to start if the load fails — and reloads every five minutes. Real data never leaves the process: nothing is written to disk. Requirements:
+serves from the production spreadsheet and media bucket (see `docs/data.md`) and turns on the full stack. The model loads at startup — the server refuses to start if the load fails — and reloads every five minutes. Real data never leaves the process: nothing is written to disk. Requirements:
 
 - **Sign-in** — everything sits behind Google sign-in restricted to the school's Workspace domain (API paths get a 401 instead of the login page). The OAuth web client is read from `creds/oauth-client.json` or `GOOGLE_CLIENT_ID`; its authorized JavaScript origins must include `http://localhost:8080`. The server issues its own HMAC-signed session cookie; set `SESSION_KEY` to keep sessions valid across restarts.
 - **Data access** — Google credentials come from application-default credentials impersonating the data service account, set up once per machine:
 
       gcloud auth application-default login --impersonate-service-account=directory@heliosian.iam.gserviceaccount.com
 
-  This requires `roles/iam.serviceAccountTokenCreator` on that account. The community shared drive — which holds the `Directory` sheet and all media — is shared with `directory@` as content manager; local processes then act as exactly the identity production runs as, with no key file anywhere (the org forbids creating one).
+  This requires `roles/iam.serviceAccountTokenCreator` on that account. `directory@` reaches the `Directory` sheet as a content manager of the community shared drive, and the media bucket through project IAM; local processes then act as exactly the identity production runs as, with no key file anywhere (the org forbids creating one).
 - **Maps** — a server key for the Geocoding API (`creds/geocoding.key` or `GOOGLE_MAPS_SERVER_KEY`; never rendered into pages, restrict by server IP or leave unrestricted for dev) and a browser key for the Maps JavaScript API (`creds/maps.key` or `GOOGLE_MAPS_BROWSER_KEY`; rendered into pages, restrict by HTTP referer). Geocoding results are cached in memory per address.
 
 To capture authenticated real-data pages, launch the capture browser (`go run ./tools/capturebrowser`), sign in to the local server there once, and use `tools/browse` or `tools/screenshot -remote` — the session cookie lives in the capture profile.
@@ -42,7 +42,7 @@ No Node, no Docker, and no cloud credentials are needed for local development. R
 
 ## Tools
 
-Each runs as `go run ./tools/<name>`. The sheet and drive tools authenticate with the same impersonated application-default credentials as the server (see Real data).
+Each runs as `go run ./tools/<name>`. The sheet, drive, and bucket tools authenticate with the same impersonated application-default credentials as the server (see Real data).
 
 - `screenshot`, `capturebrowser`, `browse` — page capture and browser driving; see `docs/screenshots.md`
 - `startserver` — launch the app detached, wait for it to listen, print the pid, log path, and a minted session cookie (needs `SESSION_KEY` and `DIRECTORY_SHEET`)
@@ -55,4 +55,5 @@ Each runs as `go run ./tools/<name>`. The sheet and drive tools authenticate wit
 - `createtabs` — create the directory sheet's local-layer tabs with their header rows
 - `setcell` — set one cell in a tab by key column, appending the row if missing
 - `probeblob` — time the download of a few drive media files
+- `migratemedia` — copy the drive media folders into the media bucket, generating thumbnails; retired once the bucket is the only store
 - `splash` — regenerate the iOS splash battery from the captured original page; see `docs/pwa.md`

docs/directory.md

@@ -62,4 +62,4 @@ A copyable contact table for party planning and outreach: full name, email, role
 - Favorites/bookmarks mark people and feed the email list's bookmark tab.
 - Photos lazy-load; full-size view on click where the photo is the subject (family pages).
 - All data is community-only, behind sign-in; opt-out removes a person on request.
-- Self-service: viewing your own record, your kids', or your family page shows inline edit affordances — photo upload, pronunciation recording or upload, About Me text, preferred name, phone, address — plus opt-out for yourself or your kids. Media uploads replace the Drive file and archive the previous version; sheet-backed edits write the Overrides tab and append a Change Log row (see `docs/data.md`).
+- Self-service: viewing your own record, your kids', or your family page shows inline edit affordances — photo upload, pronunciation recording or upload, About Me text, preferred name, phone, address — plus opt-out for yourself or your kids. Media uploads overwrite the bucket object, which retains the previous version; sheet-backed edits write the Overrides tab and append a Change Log row (see `docs/data.md`).

docs/goals.md

@@ -23,5 +23,5 @@ More apps follow over time, all served by the same binary.
 ## Data
 
 - Structured data comes from Google Sheets. The system of record is Veracross, and a direct integration replaces Sheets when an API becomes available.
-- Blobs (photos and other uploads) live in Google Drive.
+- Blobs (photos and other uploads) live in Cloud Storage.
 - Existing data is imported from the current platforms.