hypercross/tts-workshop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add architecture and decisions documentation

Browse Source
This commit is contained in:
hypercross
2026-08-08 10:26:37 +08:00
commit 405602a943
3 changed files with 423 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+99
View File
@@ -0,0 +1,99 @@
# Architecture & Dependencies
> **Scope:** The system's architecture and dependency graph. For implementation
> details (files, endpoints, build order), see
> [`implementation-plan.md`](./implementation-plan.md). For the rationale behind
> key decisions, see [`decisions.md`](./decisions.md).
## Overview
A lightweight, client-only pnpm monorepo that lets a user search the Tabletop
Simulator Steam Workshop, fetch full TTS save files, and analyze their
contents. It is split into four packages with a strict layering: a thin
HTTP proxy on top, a low-level fetcher, an isomorphic analysis layer, and a
shared types/validation package.
## Design principles
- **Client-only & lightweight** — no caching layer, no shared server state.
- **Fetch vs analyze separation** — `packages/tts` only fetches and parses;
`packages/extract` only analyzes. Neither depends on the other's concerns.
- **Isomorphic analysis** — `packages/extract` runs in browser and Node, using
only `fetch`, `Blob`, and typed arrays (no `Buffer`, no Node-only packages).
- **Thin proxy** — the HTTP layer exposes search and fetch only; traversal is
intentionally not exposed as endpoints.
## Package responsibilities
| Package | Role | Runtime |
| ------------------ | ------------------------------------------- | ------------ |
| `apps/proxy` | Hono HTTP server: search + fetch endpoints | Node |
| `packages/tts` | Fetch save from Steam, BSON-parse to `TTSMod` | Node |
| `packages/extract` | Analyze a `TTSMod`: objects, refs, assets | Isomorphic |
| `packages/shared` | Shared types + zod schemas | Isomorphic |
## Dependency graph
```
apps/proxy ──► packages/tts ──► packages/shared
│ │
│ └──► (fetchMod → TTSMod)
▼
packages/extract ──► packages/tts (traverseMod) ──► packages/shared (types)
```
### Edges
- **`apps/proxy` → `packages/tts`** — calls `fetchMod` / `getFileName` to serve
item requests.
- **`apps/proxy` → `packages/shared`** — uses shared types and zod schemas for
request/response validation.
- **`packages/tts` → `packages/shared`** — consumes `TTSMod` / `TTSObject`
types.
- **`packages/extract` → `packages/tts`** — reuses `traverseMod` (traversal
logic lives in `extract`; see note below).
- **`packages/extract` → `packages/shared`** — consumes shared types.
> **Note on `traverseMod`:** traversal is analysis, so it lives in
> `packages/extract`. `packages/tts` is fetch-only. The graph edge
> `extract → tts` reflects that `extract` imports the traversal helper that was
> originally authored alongside the fetcher; `tts` does not depend on `extract`.
### Layering rules
- **No upward dependencies** — `packages/*` never import `apps/*`.
- **No sibling coupling beyond the graph above** — `extract` and `tts` do not
depend on each other's analysis/fetch concerns.
- **`packages/shared` is the leaf** — everything depends on it; it depends on
nothing internal.
## External dependencies
| Package | Purpose | Used by |
| ------------------ | ----------------------------------------- | ------------------ |
| `hono` | HTTP framework | `apps/proxy` |
| `@hono/node-server`| Node adapter for Hono | `apps/proxy` |
| `@hono/cors` | CORS middleware | `apps/proxy` |
| `bson` | BSON deserialization of TTS save files | `packages/tts` |
| `cheerio` | Workshop browse page scraping | `apps/proxy` |
| `zod` | Runtime validation | `apps/proxy`, `packages/shared` |
### Runtime constraints
- **`cheerio` is backend-only** — it never appears in `packages/extract`, which
must stay isomorphic.
- **`bson` is Node-only** — used by the fetcher, not the analysis layer.
- **`packages/extract` has zero external runtime deps** — it relies only on
platform `fetch` / `Blob`, keeping it portable to a future frontend.
## Tooling dependencies
- `typescript` (strict), `tsx` (dev runner), `eslint`, `prettier`.
- `pnpm` workspaces for package management.
## Deployment / runtime shape
- The proxy runs as a single Node process via `@hono/node-server`.
- `packages/extract` is published/consumed as a plain ESM module usable from a
browser bundle or Node.
- No shared state between requests; each request fetches fresh.
docs/decisions.md
+98
View File
@@ -0,0 +1,98 @@
# Decisions
> **Scope:** The rationale behind key design decisions. For the system's
> architecture, see [`architecture.md`](./architecture.md). For the concrete
> build plan, see [`implementation-plan.md`](./implementation-plan.md).
>
> Each entry records the decision, the context, and the alternatives considered.
> New entries are appended; existing entries are updated only to correct facts,
> not to rewrite history.
## D1 — Client-only, no caching
**Decision:** The system runs on the client only and has no caching layer.
**Context:** The tool is a personal, single-user utility. There is no shared
state to protect and no multi-instance deployment.
**Alternatives considered:** In-memory TTL caching; Redis-backed caching. Both
rejected as unnecessary complexity for a single client.
## D2 — Separate `packages/extract` from `packages/tts`
**Decision:** Analysis lives in `packages/extract`; fetching lives in
`packages/tts`. They are separate packages with a clean fetch-vs-analyze
boundary.
**Context:** The user wants to analyze saves and extract objects for a future
frontend. Keeping analysis separate lets it be reused independently of the
fetcher.
**Alternatives considered:** Folding analysis into `packages/tts`. Rejected —
couples fetch and analysis and would force the frontend to depend on the
fetcher.
## D3 — Traversal lives in `packages/extract`
**Decision:** `traverseMod` / `markParent` move from the fetcher into
`packages/extract`.
**Context:** Traversal is analysis, not fetching. `packages/tts` should be
fetch-only.
**Alternatives considered:** Keeping traversal in `packages/tts`. Rejected —
blurs the fetch/analyze boundary.
## D4 — `packages/extract` is isomorphic with zero runtime deps
**Decision:** `packages/extract` runs in browser and Node, using only `fetch`,
`Blob`, and typed arrays. It has no external runtime dependencies.
**Context:** The frontend (deferred) will consume it directly, so it must be
portable. Node 18+ provides global `fetch` and `Blob`.
**Alternatives considered:** Using `Buffer` and Node-only packages. Rejected —
breaks browser use.
## D5 — No backend traversal endpoints
**Decision:** The proxy exposes search and fetch only. Traversal is not exposed
as HTTP endpoints.
**Context:** The frontend will use `packages/extract` directly. Exposing
traversal on the backend would duplicate that logic and add endpoints with no
current consumer.
**Alternatives considered:** Adding `/items/:id/objects` and similar endpoints.
Deferred until a consumer exists.
## D6 — Search scrapes the Workshop browse page
**Decision:** Search is implemented by scraping
`steamcommunity.com/workshop/browse/?appid=286160` with `cheerio`.
**Context:** Steam has no official Web API for searching the Workshop; the API
only fetches details for known IDs.
**Alternatives considered:** None viable — there is no official search API.
Accepted risk: scraping is fragile and may break if Steam changes its HTML.
## D7 — `bson` npm package replaces the browser `BSON` global
**Decision:** The existing scraper's `BSON.deserialize` global is replaced with
the `bson` npm package.
**Context:** The scraper was written for the browser; the proxy runs on Node.
**Alternatives considered:** Keeping a browser-only global. Rejected — not
available in Node.
## D8 — Download output type is `Blob`
**Decision:** `packages/extract`'s `downloadAsset` / `downloadAll` return `Blob`.
**Context:** The primary consumer is a frontend that renders assets in
`<img>` / `<object>` elements.
**Alternatives considered:** Raw `ArrayBuffer`. Rejected — less convenient for
frontend rendering.
docs/implementation-plan.md
+226
View File
@@ -0,0 +1,226 @@
# TTS Workshop — Implementation Plan
> **Scope:** The concrete build plan — files, endpoints, dependencies, build
> order. For the system's architecture and dependency graph, see
> [`architecture.md`](./architecture.md). For the rationale behind key decisions,
> see [`decisions.md`](./decisions.md).
A lightweight, client-only pnpm monorepo for searching the Tabletop Simulator
Steam Workshop, fetching full TTS save files, and analyzing their contents.
## Goals
- Search the TTS Workshop to discover item IDs (Steam has no official search API,
so this scrapes the browse page).
- Fetch a full TTS save (`TTSMod`) for a given item ID via the Steam Web API +
BSON deserialization.
- Extract and inspect objects and asset references from a save, in an isomorphic
package reusable from a future frontend.
- Client-only, lightweight, no caching layer.
## Non-goals (for now)
- Backend traversal endpoints (deferred by design).
- Frontend app (later; will consume `packages/extract` directly).
- Caching / Redis / multi-instance concerns.
## Architecture
```
apps/proxy ──► packages/tts ──► packages/shared
│ │
│ └──► (fetchMod → TTSMod)
▼
packages/extract ──► packages/tts (traverseMod) ──► packages/shared (types)
(isomorphic, used by future frontend)
```
- **`apps/proxy`** — Hono server. Search + fetch only. No traversal endpoints.
- **`packages/tts`** — low-level fetcher: Steam API call, BSON parse, filename
derivation. Returns raw parsed `TTSMod`.
- **`packages/extract`** — analysis: flatten/filter objects, extract asset refs,
download assets. Isomorphic (browser + Node).
- **`packages/shared`** — shared types + zod schemas.
## Repository layout
```
tts-workshop/
├── pnpm-workspace.yaml
├── package.json # root scripts (dev, build, lint)
├── tsconfig.base.json
├── .npmrc
├── .env.example # STEAM_API_KEY, PORT
├── docs/
│ └── implementation-plan.md # this file
├── apps/
│ └── proxy/
│ ├── package.json
│ ├── tsconfig.json
│ └── src/
│ ├── index.ts # Hono app + @hono/node-server
│ ├── routes/
│ │ ├── search.ts # GET /search?q=...&page=1
│ │ ├── items.ts # GET /items/:id, /items/:id/file
│ │ └── health.ts # GET /health
│ └── env.ts # zod env validation
└── packages/
├── shared/
│ └── src/
│ ├── types.ts # TTSMod, TTSObject, WorkshopItem, SearchResult
│ └── schemas.ts
├── tts/
│ └── src/
│ ├── index.ts # fetchMod, getFileName
│ └── errors.ts
└── extract/
├── package.json
├── tsconfig.json
└── src/
├── index.ts # public API barrel
├── objects.ts # flatten/filter/traverse helpers
├── refs.ts # extract asset references
├── download.ts # fetch + decode referenced assets
└── types.ts # extracted-object / asset types
```
## Packages
### `packages/shared`
Shared types and validation schemas used across the monorepo.
- `types.ts`
- `TTSMod`, `TTSObject` (from the existing scraper code)
- `WorkshopItem` — metadata from the Steam API (`id`, `title`, `author`,
`previewImageUrl`, `fileUrl`, ...)
- `SearchResult` — `{ items: WorkshopItem[], page, hasMore }`
- `schemas.ts`
- zod schemas for env vars, query params, and response shapes.
### `packages/tts`
Low-level fetcher, extracted from the existing scraper.
- `index.ts`
- `fetchMod(id: string): Promise<TTSMod>` — Steam API call to
`ISteamRemoteStorage/GetPublishedFileDetails/v1` to get `file_url`, then
download + BSON-deserialize into `TTSMod`.
- `getFileName(url: string): Promise<string>` — derive filename from the
`content-disposition` header.
- `errors.ts`
- Typed errors: missing `file_url`, Steam API failure, rate limit, invalid key.
- Notes
- Swap the browser `BSON` global for the `bson` npm package.
- `traverseMod` / `markParent` move to `packages/extract` (traversal is
analysis, not fetching).
### `packages/extract`
Isomorphic analysis of a parsed `TTSMod`. No Node-specific APIs.
- `objects.ts`
- `flattenObjects(mod)` — all objects in the tree.
- `filterObjects(mod, predicate)` — filter by name, GUID, type, etc.
- `findObject(mod, guid)` — lookup by GUID.
- `traverseMod` / `markParent` (moved from `tts`).
- Returns lightweight graph shapes: `{ guid, name, type, parentGuid,
childrenGuids, refs }`.
- `refs.ts`
- `extractRefs(object)` — walk one object, pull every external URL:
`CustomPDF.PDFUrl`, `CustomDeck[*].FaceURL`/`BackURL`,
`CustomImage.ImageURL`/`ImageSecondaryURL`.
- `collectRefs(mod)` — all refs across the save, deduped by URL.
- `AssetRef` type: `{ kind, url, ownerGuid }`.
- `download.ts`
- `downloadAsset(url)` — `fetch` → `Blob`.
- `downloadAll(refs, { concurrency, onProgress })` — batched downloads with
progress callback.
- `guessMimeType(url)` — infer mime from extension.
- Design constraints
- No `Buffer` — use `ArrayBuffer` / `Uint8Array` / `Blob` (Node 18+).
- No Node-only packages (`cheerio` stays in the backend search only).
- Pure, deterministic functions where possible.
### `apps/proxy`
Hono server exposing search + fetch.
- `index.ts` — Hono app + `@hono/node-server` bootstrap, CORS via `@hono/cors`.
- `routes/search.ts`
- `GET /search?q=...&page=1` — scrape
`steamcommunity.com/workshop/browse/?appid=286160&searchtext=...` with
`cheerio`, extract `{ id, title, author, previewImageUrl }` from the result
grid. Supports `pagenum` pagination. No API key required.
- `routes/items.ts`
- `GET /items/:id` — full parsed `TTSMod`.
- `GET /items/:id/file` — raw save bytes, filename from `getFileName`.
- `routes/health.ts`
- `GET /health` — liveness.
- `env.ts` — zod validation of `STEAM_API_KEY`, `PORT`.
## Endpoints
| Method | Path | Description | Auth |
| ------ | ------------------- | -------------------------------------------- | ---- |
| GET | `/health` | Liveness | — |
| GET | `/search?q=&page=` | Scrape Workshop browse, return item list | — |
| GET | `/items/:id` | Full parsed `TTSMod` | key |
| GET | `/items/:id/file` | Raw save bytes, filename from header | key |
## Data flow
```
/search?q=wingspan → scrape browse page → list of {id, title, ...}
↓
/items/:id → Steam API (needs key) → file_url
↓
download BSON → parse → TTSMod
↓
packages/extract → flatten objects / extract refs / download assets
```
## Dependencies
- `hono`, `@hono/node-server`, `@hono/cors` — server.
- `bson` — BSON deserialization.
- `cheerio` — Workshop browse page scraping (backend only).
- `zod` — validation.
- `tsx`, `typescript`, `eslint`, `prettier` — tooling.
## Tooling
- TypeScript strict mode.
- `tsx` for dev, `tsc` for build.
- Root scripts: `pnpm dev`, `pnpm build`, `pnpm lint`.
## Build order
1. Scaffold workspace (`pnpm-workspace.yaml`, root `package.json`, base tsconfig,
`.npmrc`).
2. `packages/shared` — types + zod schemas.
3. `packages/tts` — fetch + parse (existing scraper code).
4. `packages/extract` — objects, refs, download.
5. `apps/proxy` — search + items + health routes, env validation.
6. Wire up root scripts, `.env.example`, README.
## Risks / caveats
- **Search scraping is fragile** — Steam can change their HTML or rate-limit.
No official search API exists, so this is the standard approach.
- **Members-only items** won't appear in scraped search results.
- **`file_url` may be missing/expired** — handle cleanly with a 404-style error.
- **BSON parsing is the expensive part** — large saves; no caching by design
(client-only).
- **`STEAM_API_KEY` is exposed in the client process** — acceptable for a
personal tool; the key is only needed for the metadata call that yields
`file_url`.
## Open decisions (defaults in bold)
- **`packages/extract` vs folding into `packages/tts`** — **separate package**
(clean fetch vs analyze boundary).
- **Download output type** — **`Blob`** (easier for `<img>`/`<object>` in a
frontend) vs raw `ArrayBuffer`.
- **Move `traverseMod`/`markParent` into `extract`** — **yes** (traversal is
analysis, not fetching).