docs: add architecture and decisions documentation
This commit is contained in:
@@ -0,0 +1,226 @@
|
||||
# TTS Workshop — Implementation Plan
|
||||
|
||||
> **Scope:** The concrete build plan — files, endpoints, dependencies, build
|
||||
> order. For the system's architecture and dependency graph, see
|
||||
> [`architecture.md`](./architecture.md). For the rationale behind key decisions,
|
||||
> see [`decisions.md`](./decisions.md).
|
||||
|
||||
A lightweight, client-only pnpm monorepo for searching the Tabletop Simulator
|
||||
Steam Workshop, fetching full TTS save files, and analyzing their contents.
|
||||
|
||||
## Goals
|
||||
|
||||
- Search the TTS Workshop to discover item IDs (Steam has no official search API,
|
||||
so this scrapes the browse page).
|
||||
- Fetch a full TTS save (`TTSMod`) for a given item ID via the Steam Web API +
|
||||
BSON deserialization.
|
||||
- Extract and inspect objects and asset references from a save, in an isomorphic
|
||||
package reusable from a future frontend.
|
||||
- Client-only, lightweight, no caching layer.
|
||||
|
||||
## Non-goals (for now)
|
||||
|
||||
- Backend traversal endpoints (deferred by design).
|
||||
- Frontend app (later; will consume `packages/extract` directly).
|
||||
- Caching / Redis / multi-instance concerns.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
apps/proxy ──► packages/tts ──► packages/shared
|
||||
│ │
|
||||
│ └──► (fetchMod → TTSMod)
|
||||
▼
|
||||
packages/extract ──► packages/tts (traverseMod) ──► packages/shared (types)
|
||||
(isomorphic, used by future frontend)
|
||||
```
|
||||
|
||||
- **`apps/proxy`** — Hono server. Search + fetch only. No traversal endpoints.
|
||||
- **`packages/tts`** — low-level fetcher: Steam API call, BSON parse, filename
|
||||
derivation. Returns raw parsed `TTSMod`.
|
||||
- **`packages/extract`** — analysis: flatten/filter objects, extract asset refs,
|
||||
download assets. Isomorphic (browser + Node).
|
||||
- **`packages/shared`** — shared types + zod schemas.
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
tts-workshop/
|
||||
├── pnpm-workspace.yaml
|
||||
├── package.json # root scripts (dev, build, lint)
|
||||
├── tsconfig.base.json
|
||||
├── .npmrc
|
||||
├── .env.example # STEAM_API_KEY, PORT
|
||||
├── docs/
|
||||
│ └── implementation-plan.md # this file
|
||||
├── apps/
|
||||
│ └── proxy/
|
||||
│ ├── package.json
|
||||
│ ├── tsconfig.json
|
||||
│ └── src/
|
||||
│ ├── index.ts # Hono app + @hono/node-server
|
||||
│ ├── routes/
|
||||
│ │ ├── search.ts # GET /search?q=...&page=1
|
||||
│ │ ├── items.ts # GET /items/:id, /items/:id/file
|
||||
│ │ └── health.ts # GET /health
|
||||
│ └── env.ts # zod env validation
|
||||
└── packages/
|
||||
├── shared/
|
||||
│ └── src/
|
||||
│ ├── types.ts # TTSMod, TTSObject, WorkshopItem, SearchResult
|
||||
│ └── schemas.ts
|
||||
├── tts/
|
||||
│ └── src/
|
||||
│ ├── index.ts # fetchMod, getFileName
|
||||
│ └── errors.ts
|
||||
└── extract/
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── src/
|
||||
├── index.ts # public API barrel
|
||||
├── objects.ts # flatten/filter/traverse helpers
|
||||
├── refs.ts # extract asset references
|
||||
├── download.ts # fetch + decode referenced assets
|
||||
└── types.ts # extracted-object / asset types
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
### `packages/shared`
|
||||
|
||||
Shared types and validation schemas used across the monorepo.
|
||||
|
||||
- `types.ts`
|
||||
- `TTSMod`, `TTSObject` (from the existing scraper code)
|
||||
- `WorkshopItem` — metadata from the Steam API (`id`, `title`, `author`,
|
||||
`previewImageUrl`, `fileUrl`, ...)
|
||||
- `SearchResult` — `{ items: WorkshopItem[], page, hasMore }`
|
||||
- `schemas.ts`
|
||||
- zod schemas for env vars, query params, and response shapes.
|
||||
|
||||
### `packages/tts`
|
||||
|
||||
Low-level fetcher, extracted from the existing scraper.
|
||||
|
||||
- `index.ts`
|
||||
- `fetchMod(id: string): Promise<TTSMod>` — Steam API call to
|
||||
`ISteamRemoteStorage/GetPublishedFileDetails/v1` to get `file_url`, then
|
||||
download + BSON-deserialize into `TTSMod`.
|
||||
- `getFileName(url: string): Promise<string>` — derive filename from the
|
||||
`content-disposition` header.
|
||||
- `errors.ts`
|
||||
- Typed errors: missing `file_url`, Steam API failure, rate limit, invalid key.
|
||||
- Notes
|
||||
- Swap the browser `BSON` global for the `bson` npm package.
|
||||
- `traverseMod` / `markParent` move to `packages/extract` (traversal is
|
||||
analysis, not fetching).
|
||||
|
||||
### `packages/extract`
|
||||
|
||||
Isomorphic analysis of a parsed `TTSMod`. No Node-specific APIs.
|
||||
|
||||
- `objects.ts`
|
||||
- `flattenObjects(mod)` — all objects in the tree.
|
||||
- `filterObjects(mod, predicate)` — filter by name, GUID, type, etc.
|
||||
- `findObject(mod, guid)` — lookup by GUID.
|
||||
- `traverseMod` / `markParent` (moved from `tts`).
|
||||
- Returns lightweight graph shapes: `{ guid, name, type, parentGuid,
|
||||
childrenGuids, refs }`.
|
||||
- `refs.ts`
|
||||
- `extractRefs(object)` — walk one object, pull every external URL:
|
||||
`CustomPDF.PDFUrl`, `CustomDeck[*].FaceURL`/`BackURL`,
|
||||
`CustomImage.ImageURL`/`ImageSecondaryURL`.
|
||||
- `collectRefs(mod)` — all refs across the save, deduped by URL.
|
||||
- `AssetRef` type: `{ kind, url, ownerGuid }`.
|
||||
- `download.ts`
|
||||
- `downloadAsset(url)` — `fetch` → `Blob`.
|
||||
- `downloadAll(refs, { concurrency, onProgress })` — batched downloads with
|
||||
progress callback.
|
||||
- `guessMimeType(url)` — infer mime from extension.
|
||||
- Design constraints
|
||||
- No `Buffer` — use `ArrayBuffer` / `Uint8Array` / `Blob` (Node 18+).
|
||||
- No Node-only packages (`cheerio` stays in the backend search only).
|
||||
- Pure, deterministic functions where possible.
|
||||
|
||||
### `apps/proxy`
|
||||
|
||||
Hono server exposing search + fetch.
|
||||
|
||||
- `index.ts` — Hono app + `@hono/node-server` bootstrap, CORS via `@hono/cors`.
|
||||
- `routes/search.ts`
|
||||
- `GET /search?q=...&page=1` — scrape
|
||||
`steamcommunity.com/workshop/browse/?appid=286160&searchtext=...` with
|
||||
`cheerio`, extract `{ id, title, author, previewImageUrl }` from the result
|
||||
grid. Supports `pagenum` pagination. No API key required.
|
||||
- `routes/items.ts`
|
||||
- `GET /items/:id` — full parsed `TTSMod`.
|
||||
- `GET /items/:id/file` — raw save bytes, filename from `getFileName`.
|
||||
- `routes/health.ts`
|
||||
- `GET /health` — liveness.
|
||||
- `env.ts` — zod validation of `STEAM_API_KEY`, `PORT`.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Path | Description | Auth |
|
||||
| ------ | ------------------- | -------------------------------------------- | ---- |
|
||||
| GET | `/health` | Liveness | — |
|
||||
| GET | `/search?q=&page=` | Scrape Workshop browse, return item list | — |
|
||||
| GET | `/items/:id` | Full parsed `TTSMod` | key |
|
||||
| GET | `/items/:id/file` | Raw save bytes, filename from header | key |
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
/search?q=wingspan → scrape browse page → list of {id, title, ...}
|
||||
↓
|
||||
/items/:id → Steam API (needs key) → file_url
|
||||
↓
|
||||
download BSON → parse → TTSMod
|
||||
↓
|
||||
packages/extract → flatten objects / extract refs / download assets
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `hono`, `@hono/node-server`, `@hono/cors` — server.
|
||||
- `bson` — BSON deserialization.
|
||||
- `cheerio` — Workshop browse page scraping (backend only).
|
||||
- `zod` — validation.
|
||||
- `tsx`, `typescript`, `eslint`, `prettier` — tooling.
|
||||
|
||||
## Tooling
|
||||
|
||||
- TypeScript strict mode.
|
||||
- `tsx` for dev, `tsc` for build.
|
||||
- Root scripts: `pnpm dev`, `pnpm build`, `pnpm lint`.
|
||||
|
||||
## Build order
|
||||
|
||||
1. Scaffold workspace (`pnpm-workspace.yaml`, root `package.json`, base tsconfig,
|
||||
`.npmrc`).
|
||||
2. `packages/shared` — types + zod schemas.
|
||||
3. `packages/tts` — fetch + parse (existing scraper code).
|
||||
4. `packages/extract` — objects, refs, download.
|
||||
5. `apps/proxy` — search + items + health routes, env validation.
|
||||
6. Wire up root scripts, `.env.example`, README.
|
||||
|
||||
## Risks / caveats
|
||||
|
||||
- **Search scraping is fragile** — Steam can change their HTML or rate-limit.
|
||||
No official search API exists, so this is the standard approach.
|
||||
- **Members-only items** won't appear in scraped search results.
|
||||
- **`file_url` may be missing/expired** — handle cleanly with a 404-style error.
|
||||
- **BSON parsing is the expensive part** — large saves; no caching by design
|
||||
(client-only).
|
||||
- **`STEAM_API_KEY` is exposed in the client process** — acceptable for a
|
||||
personal tool; the key is only needed for the metadata call that yields
|
||||
`file_url`.
|
||||
|
||||
## Open decisions (defaults in bold)
|
||||
|
||||
- **`packages/extract` vs folding into `packages/tts`** — **separate package**
|
||||
(clean fetch vs analyze boundary).
|
||||
- **Download output type** — **`Blob`** (easier for `<img>`/`<object>` in a
|
||||
frontend) vs raw `ArrayBuffer`.
|
||||
- **Move `traverseMod`/`markParent` into `extract`** — **yes** (traversal is
|
||||
analysis, not fetching).
|
||||
Reference in New Issue
Block a user