From 405602a9432e45da50c622aae8f8a648ca3ef9c0 Mon Sep 17 00:00:00 2001 From: hypercross Date: Sat, 8 Aug 2026 10:26:37 +0800 Subject: [PATCH] docs: add architecture and decisions documentation --- docs/architecture.md | 99 ++++++++++++++++ docs/decisions.md | 98 ++++++++++++++++ docs/implementation-plan.md | 226 ++++++++++++++++++++++++++++++++++++ 3 files changed, 423 insertions(+) create mode 100644 docs/architecture.md create mode 100644 docs/decisions.md create mode 100644 docs/implementation-plan.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..c37be90 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,99 @@ +# Architecture & Dependencies + +> **Scope:** The system's architecture and dependency graph. For implementation +> details (files, endpoints, build order), see +> [`implementation-plan.md`](./implementation-plan.md). For the rationale behind +> key decisions, see [`decisions.md`](./decisions.md). + +## Overview + +A lightweight, client-only pnpm monorepo that lets a user search the Tabletop +Simulator Steam Workshop, fetch full TTS save files, and analyze their +contents. It is split into four packages with a strict layering: a thin +HTTP proxy on top, a low-level fetcher, an isomorphic analysis layer, and a +shared types/validation package. + +## Design principles + +- **Client-only & lightweight** — no caching layer, no shared server state. +- **Fetch vs analyze separation** — `packages/tts` only fetches and parses; + `packages/extract` only analyzes. Neither depends on the other's concerns. +- **Isomorphic analysis** — `packages/extract` runs in browser and Node, using + only `fetch`, `Blob`, and typed arrays (no `Buffer`, no Node-only packages). +- **Thin proxy** — the HTTP layer exposes search and fetch only; traversal is + intentionally not exposed as endpoints. + +## Package responsibilities + +| Package | Role | Runtime | +| ------------------ | ------------------------------------------- | ------------ | +| `apps/proxy` | Hono HTTP server: search + fetch endpoints | Node | +| `packages/tts` | Fetch save from Steam, BSON-parse to `TTSMod` | Node | +| `packages/extract` | Analyze a `TTSMod`: objects, refs, assets | Isomorphic | +| `packages/shared` | Shared types + zod schemas | Isomorphic | + +## Dependency graph + +``` +apps/proxy ──► packages/tts ──► packages/shared + │ │ + │ └──► (fetchMod → TTSMod) + ▼ +packages/extract ──► packages/tts (traverseMod) ──► packages/shared (types) +``` + +### Edges + +- **`apps/proxy` → `packages/tts`** — calls `fetchMod` / `getFileName` to serve + item requests. +- **`apps/proxy` → `packages/shared`** — uses shared types and zod schemas for + request/response validation. +- **`packages/tts` → `packages/shared`** — consumes `TTSMod` / `TTSObject` + types. +- **`packages/extract` → `packages/tts`** — reuses `traverseMod` (traversal + logic lives in `extract`; see note below). +- **`packages/extract` → `packages/shared`** — consumes shared types. + +> **Note on `traverseMod`:** traversal is analysis, so it lives in +> `packages/extract`. `packages/tts` is fetch-only. The graph edge +> `extract → tts` reflects that `extract` imports the traversal helper that was +> originally authored alongside the fetcher; `tts` does not depend on `extract`. + +### Layering rules + +- **No upward dependencies** — `packages/*` never import `apps/*`. +- **No sibling coupling beyond the graph above** — `extract` and `tts` do not + depend on each other's analysis/fetch concerns. +- **`packages/shared` is the leaf** — everything depends on it; it depends on + nothing internal. + +## External dependencies + +| Package | Purpose | Used by | +| ------------------ | ----------------------------------------- | ------------------ | +| `hono` | HTTP framework | `apps/proxy` | +| `@hono/node-server`| Node adapter for Hono | `apps/proxy` | +| `@hono/cors` | CORS middleware | `apps/proxy` | +| `bson` | BSON deserialization of TTS save files | `packages/tts` | +| `cheerio` | Workshop browse page scraping | `apps/proxy` | +| `zod` | Runtime validation | `apps/proxy`, `packages/shared` | + +### Runtime constraints + +- **`cheerio` is backend-only** — it never appears in `packages/extract`, which + must stay isomorphic. +- **`bson` is Node-only** — used by the fetcher, not the analysis layer. +- **`packages/extract` has zero external runtime deps** — it relies only on + platform `fetch` / `Blob`, keeping it portable to a future frontend. + +## Tooling dependencies + +- `typescript` (strict), `tsx` (dev runner), `eslint`, `prettier`. +- `pnpm` workspaces for package management. + +## Deployment / runtime shape + +- The proxy runs as a single Node process via `@hono/node-server`. +- `packages/extract` is published/consumed as a plain ESM module usable from a + browser bundle or Node. +- No shared state between requests; each request fetches fresh. \ No newline at end of file diff --git a/docs/decisions.md b/docs/decisions.md new file mode 100644 index 0000000..65d9188 --- /dev/null +++ b/docs/decisions.md @@ -0,0 +1,98 @@ +# Decisions + +> **Scope:** The rationale behind key design decisions. For the system's +> architecture, see [`architecture.md`](./architecture.md). For the concrete +> build plan, see [`implementation-plan.md`](./implementation-plan.md). +> +> Each entry records the decision, the context, and the alternatives considered. +> New entries are appended; existing entries are updated only to correct facts, +> not to rewrite history. + +## D1 — Client-only, no caching + +**Decision:** The system runs on the client only and has no caching layer. + +**Context:** The tool is a personal, single-user utility. There is no shared +state to protect and no multi-instance deployment. + +**Alternatives considered:** In-memory TTL caching; Redis-backed caching. Both +rejected as unnecessary complexity for a single client. + +## D2 — Separate `packages/extract` from `packages/tts` + +**Decision:** Analysis lives in `packages/extract`; fetching lives in +`packages/tts`. They are separate packages with a clean fetch-vs-analyze +boundary. + +**Context:** The user wants to analyze saves and extract objects for a future +frontend. Keeping analysis separate lets it be reused independently of the +fetcher. + +**Alternatives considered:** Folding analysis into `packages/tts`. Rejected — +couples fetch and analysis and would force the frontend to depend on the +fetcher. + +## D3 — Traversal lives in `packages/extract` + +**Decision:** `traverseMod` / `markParent` move from the fetcher into +`packages/extract`. + +**Context:** Traversal is analysis, not fetching. `packages/tts` should be +fetch-only. + +**Alternatives considered:** Keeping traversal in `packages/tts`. Rejected — +blurs the fetch/analyze boundary. + +## D4 — `packages/extract` is isomorphic with zero runtime deps + +**Decision:** `packages/extract` runs in browser and Node, using only `fetch`, +`Blob`, and typed arrays. It has no external runtime dependencies. + +**Context:** The frontend (deferred) will consume it directly, so it must be +portable. Node 18+ provides global `fetch` and `Blob`. + +**Alternatives considered:** Using `Buffer` and Node-only packages. Rejected — +breaks browser use. + +## D5 — No backend traversal endpoints + +**Decision:** The proxy exposes search and fetch only. Traversal is not exposed +as HTTP endpoints. + +**Context:** The frontend will use `packages/extract` directly. Exposing +traversal on the backend would duplicate that logic and add endpoints with no +current consumer. + +**Alternatives considered:** Adding `/items/:id/objects` and similar endpoints. +Deferred until a consumer exists. + +## D6 — Search scrapes the Workshop browse page + +**Decision:** Search is implemented by scraping +`steamcommunity.com/workshop/browse/?appid=286160` with `cheerio`. + +**Context:** Steam has no official Web API for searching the Workshop; the API +only fetches details for known IDs. + +**Alternatives considered:** None viable — there is no official search API. +Accepted risk: scraping is fragile and may break if Steam changes its HTML. + +## D7 — `bson` npm package replaces the browser `BSON` global + +**Decision:** The existing scraper's `BSON.deserialize` global is replaced with +the `bson` npm package. + +**Context:** The scraper was written for the browser; the proxy runs on Node. + +**Alternatives considered:** Keeping a browser-only global. Rejected — not +available in Node. + +## D8 — Download output type is `Blob` + +**Decision:** `packages/extract`'s `downloadAsset` / `downloadAll` return `Blob`. + +**Context:** The primary consumer is a frontend that renders assets in +`` / `` elements. + +**Alternatives considered:** Raw `ArrayBuffer`. Rejected — less convenient for +frontend rendering. \ No newline at end of file diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..93f000f --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,226 @@ +# TTS Workshop — Implementation Plan + +> **Scope:** The concrete build plan — files, endpoints, dependencies, build +> order. For the system's architecture and dependency graph, see +> [`architecture.md`](./architecture.md). For the rationale behind key decisions, +> see [`decisions.md`](./decisions.md). + +A lightweight, client-only pnpm monorepo for searching the Tabletop Simulator +Steam Workshop, fetching full TTS save files, and analyzing their contents. + +## Goals + +- Search the TTS Workshop to discover item IDs (Steam has no official search API, + so this scrapes the browse page). +- Fetch a full TTS save (`TTSMod`) for a given item ID via the Steam Web API + + BSON deserialization. +- Extract and inspect objects and asset references from a save, in an isomorphic + package reusable from a future frontend. +- Client-only, lightweight, no caching layer. + +## Non-goals (for now) + +- Backend traversal endpoints (deferred by design). +- Frontend app (later; will consume `packages/extract` directly). +- Caching / Redis / multi-instance concerns. + +## Architecture + +``` +apps/proxy ──► packages/tts ──► packages/shared + │ │ + │ └──► (fetchMod → TTSMod) + ▼ +packages/extract ──► packages/tts (traverseMod) ──► packages/shared (types) + (isomorphic, used by future frontend) +``` + +- **`apps/proxy`** — Hono server. Search + fetch only. No traversal endpoints. +- **`packages/tts`** — low-level fetcher: Steam API call, BSON parse, filename + derivation. Returns raw parsed `TTSMod`. +- **`packages/extract`** — analysis: flatten/filter objects, extract asset refs, + download assets. Isomorphic (browser + Node). +- **`packages/shared`** — shared types + zod schemas. + +## Repository layout + +``` +tts-workshop/ +├── pnpm-workspace.yaml +├── package.json # root scripts (dev, build, lint) +├── tsconfig.base.json +├── .npmrc +├── .env.example # STEAM_API_KEY, PORT +├── docs/ +│ └── implementation-plan.md # this file +├── apps/ +│ └── proxy/ +│ ├── package.json +│ ├── tsconfig.json +│ └── src/ +│ ├── index.ts # Hono app + @hono/node-server +│ ├── routes/ +│ │ ├── search.ts # GET /search?q=...&page=1 +│ │ ├── items.ts # GET /items/:id, /items/:id/file +│ │ └── health.ts # GET /health +│ └── env.ts # zod env validation +└── packages/ + ├── shared/ + │ └── src/ + │ ├── types.ts # TTSMod, TTSObject, WorkshopItem, SearchResult + │ └── schemas.ts + ├── tts/ + │ └── src/ + │ ├── index.ts # fetchMod, getFileName + │ └── errors.ts + └── extract/ + ├── package.json + ├── tsconfig.json + └── src/ + ├── index.ts # public API barrel + ├── objects.ts # flatten/filter/traverse helpers + ├── refs.ts # extract asset references + ├── download.ts # fetch + decode referenced assets + └── types.ts # extracted-object / asset types +``` + +## Packages + +### `packages/shared` + +Shared types and validation schemas used across the monorepo. + +- `types.ts` + - `TTSMod`, `TTSObject` (from the existing scraper code) + - `WorkshopItem` — metadata from the Steam API (`id`, `title`, `author`, + `previewImageUrl`, `fileUrl`, ...) + - `SearchResult` — `{ items: WorkshopItem[], page, hasMore }` +- `schemas.ts` + - zod schemas for env vars, query params, and response shapes. + +### `packages/tts` + +Low-level fetcher, extracted from the existing scraper. + +- `index.ts` + - `fetchMod(id: string): Promise` — Steam API call to + `ISteamRemoteStorage/GetPublishedFileDetails/v1` to get `file_url`, then + download + BSON-deserialize into `TTSMod`. + - `getFileName(url: string): Promise` — derive filename from the + `content-disposition` header. +- `errors.ts` + - Typed errors: missing `file_url`, Steam API failure, rate limit, invalid key. +- Notes + - Swap the browser `BSON` global for the `bson` npm package. + - `traverseMod` / `markParent` move to `packages/extract` (traversal is + analysis, not fetching). + +### `packages/extract` + +Isomorphic analysis of a parsed `TTSMod`. No Node-specific APIs. + +- `objects.ts` + - `flattenObjects(mod)` — all objects in the tree. + - `filterObjects(mod, predicate)` — filter by name, GUID, type, etc. + - `findObject(mod, guid)` — lookup by GUID. + - `traverseMod` / `markParent` (moved from `tts`). + - Returns lightweight graph shapes: `{ guid, name, type, parentGuid, + childrenGuids, refs }`. +- `refs.ts` + - `extractRefs(object)` — walk one object, pull every external URL: + `CustomPDF.PDFUrl`, `CustomDeck[*].FaceURL`/`BackURL`, + `CustomImage.ImageURL`/`ImageSecondaryURL`. + - `collectRefs(mod)` — all refs across the save, deduped by URL. + - `AssetRef` type: `{ kind, url, ownerGuid }`. +- `download.ts` + - `downloadAsset(url)` — `fetch` → `Blob`. + - `downloadAll(refs, { concurrency, onProgress })` — batched downloads with + progress callback. + - `guessMimeType(url)` — infer mime from extension. +- Design constraints + - No `Buffer` — use `ArrayBuffer` / `Uint8Array` / `Blob` (Node 18+). + - No Node-only packages (`cheerio` stays in the backend search only). + - Pure, deterministic functions where possible. + +### `apps/proxy` + +Hono server exposing search + fetch. + +- `index.ts` — Hono app + `@hono/node-server` bootstrap, CORS via `@hono/cors`. +- `routes/search.ts` + - `GET /search?q=...&page=1` — scrape + `steamcommunity.com/workshop/browse/?appid=286160&searchtext=...` with + `cheerio`, extract `{ id, title, author, previewImageUrl }` from the result + grid. Supports `pagenum` pagination. No API key required. +- `routes/items.ts` + - `GET /items/:id` — full parsed `TTSMod`. + - `GET /items/:id/file` — raw save bytes, filename from `getFileName`. +- `routes/health.ts` + - `GET /health` — liveness. +- `env.ts` — zod validation of `STEAM_API_KEY`, `PORT`. + +## Endpoints + +| Method | Path | Description | Auth | +| ------ | ------------------- | -------------------------------------------- | ---- | +| GET | `/health` | Liveness | — | +| GET | `/search?q=&page=` | Scrape Workshop browse, return item list | — | +| GET | `/items/:id` | Full parsed `TTSMod` | key | +| GET | `/items/:id/file` | Raw save bytes, filename from header | key | + +## Data flow + +``` +/search?q=wingspan → scrape browse page → list of {id, title, ...} + ↓ +/items/:id → Steam API (needs key) → file_url + ↓ + download BSON → parse → TTSMod + ↓ +packages/extract → flatten objects / extract refs / download assets +``` + +## Dependencies + +- `hono`, `@hono/node-server`, `@hono/cors` — server. +- `bson` — BSON deserialization. +- `cheerio` — Workshop browse page scraping (backend only). +- `zod` — validation. +- `tsx`, `typescript`, `eslint`, `prettier` — tooling. + +## Tooling + +- TypeScript strict mode. +- `tsx` for dev, `tsc` for build. +- Root scripts: `pnpm dev`, `pnpm build`, `pnpm lint`. + +## Build order + +1. Scaffold workspace (`pnpm-workspace.yaml`, root `package.json`, base tsconfig, + `.npmrc`). +2. `packages/shared` — types + zod schemas. +3. `packages/tts` — fetch + parse (existing scraper code). +4. `packages/extract` — objects, refs, download. +5. `apps/proxy` — search + items + health routes, env validation. +6. Wire up root scripts, `.env.example`, README. + +## Risks / caveats + +- **Search scraping is fragile** — Steam can change their HTML or rate-limit. + No official search API exists, so this is the standard approach. +- **Members-only items** won't appear in scraped search results. +- **`file_url` may be missing/expired** — handle cleanly with a 404-style error. +- **BSON parsing is the expensive part** — large saves; no caching by design + (client-only). +- **`STEAM_API_KEY` is exposed in the client process** — acceptable for a + personal tool; the key is only needed for the metadata call that yields + `file_url`. + +## Open decisions (defaults in bold) + +- **`packages/extract` vs folding into `packages/tts`** — **separate package** + (clean fetch vs analyze boundary). +- **Download output type** — **`Blob`** (easier for ``/`` in a + frontend) vs raw `ArrayBuffer`. +- **Move `traverseMod`/`markParent` into `extract`** — **yes** (traversal is + analysis, not fetching). \ No newline at end of file