tts-workshop/docs/implementation-plan.md

TTS Workshop — Implementation Plan

Scope: The concrete build plan — files, endpoints, dependencies, build order. For the system's architecture and dependency graph, see architecture.md. For the rationale behind key decisions, see decisions.md.

A lightweight, client-only pnpm monorepo for searching the Tabletop Simulator Steam Workshop, fetching full TTS save files, and analyzing their contents.

Goals

• Search the TTS Workshop to discover item IDs (Steam has no official search API, so this scrapes the browse page).
• Fetch a full TTS save (TTSMod) for a given item ID via the Steam Web API + BSON deserialization.
• Extract and inspect objects and asset references from a save, in an isomorphic package reusable from a future frontend.
• Client-only, lightweight, no caching layer.

Non-goals (for now)

• Backend traversal endpoints (deferred by design).
• Frontend app (later; will consume packages/extract directly).
• Caching / Redis / multi-instance concerns.

Architecture

apps/proxy ──► packages/tts ──► packages/shared
      │            │
      │            └──► (fetchMod → TTSMod)
      ▼
packages/extract ──► packages/tts (traverseMod) ──► packages/shared (types)
   (isomorphic, used by future frontend)

• apps/proxy — Hono server. Search + fetch only. No traversal endpoints.
• packages/tts — low-level fetcher: Steam API call, BSON parse, filename derivation. Returns raw parsed TTSMod.
• packages/extract — analysis: flatten/filter objects, extract asset refs, download assets. Isomorphic (browser + Node).
• packages/shared — shared types + zod schemas.

Repository layout

tts-workshop/
├── pnpm-workspace.yaml
├── package.json                # root scripts (dev, build, lint)
├── tsconfig.base.json
├── .npmrc
├── .env.example                # STEAM_API_KEY, PORT
├── docs/
│   └── implementation-plan.md  # this file
├── apps/
│   └── proxy/
│       ├── package.json
│       ├── tsconfig.json
│       └── src/
│           ├── index.ts        # Hono app + @hono/node-server
│           ├── routes/
│           │   ├── search.ts   # GET /search?q=...&page=1
│           │   ├── items.ts    # GET /items/:id, /items/:id/file
│           │   └── health.ts   # GET /health
│           └── env.ts          # zod env validation
└── packages/
    ├── shared/
    │   └── src/
    │       ├── types.ts        # TTSMod, TTSObject, WorkshopItem, SearchResult
    │       └── schemas.ts
    ├── tts/
    │   └── src/
    │       ├── index.ts        # fetchMod, getFileName
    │       └── errors.ts
    └── extract/
        ├── package.json
        ├── tsconfig.json
        └── src/
            ├── index.ts        # public API barrel
            ├── objects.ts      # flatten/filter/traverse helpers
            ├── refs.ts         # extract asset references
            ├── download.ts     # fetch + decode referenced assets
            └── types.ts        # extracted-object / asset types

Packages

packages/shared

Shared types and validation schemas used across the monorepo.

• types.ts
  • TTSMod, TTSObject (from the existing scraper code)
  • WorkshopItem — metadata from the Steam API (id, title, author, previewImageUrl, fileUrl, ...)
  • SearchResult — { items: WorkshopItem[], page, hasMore }
• schemas.ts
  • zod schemas for env vars, query params, and response shapes.

packages/tts

Low-level fetcher, extracted from the existing scraper.

• index.ts
  • fetchMod(id: string): Promise<TTSMod> — Steam API call to ISteamRemoteStorage/GetPublishedFileDetails/v1 to get file_url, then download + BSON-deserialize into TTSMod.
  • getFileName(url: string): Promise<string> — derive filename from the content-disposition header.
• errors.ts
  • Typed errors: missing file_url, Steam API failure, rate limit, invalid key.
• Notes
  • Swap the browser BSON global for the bson npm package.
  • traverseMod / markParent move to packages/extract (traversal is analysis, not fetching).

packages/extract

Isomorphic analysis of a parsed TTSMod. No Node-specific APIs.

• objects.ts
  • flattenObjects(mod) — all objects in the tree.
  • filterObjects(mod, predicate) — filter by name, GUID, type, etc.
  • findObject(mod, guid) — lookup by GUID.
  • traverseMod / markParent (moved from tts).
  • Returns lightweight graph shapes: { guid, name, type, parentGuid, childrenGuids, refs }.
• refs.ts
  • extractRefs(object) — walk one object, pull every external URL: CustomPDF.PDFUrl, CustomDeck[*].FaceURL/BackURL, CustomImage.ImageURL/ImageSecondaryURL.
  • collectRefs(mod) — all refs across the save, deduped by URL.
  • AssetRef type: { kind, url, ownerGuid }.
• download.ts
  • downloadAsset(url) — fetch → Blob.
  • downloadAll(refs, { concurrency, onProgress }) — batched downloads with progress callback.
  • guessMimeType(url) — infer mime from extension.
• Design constraints
  • No Buffer — use ArrayBuffer / Uint8Array / Blob (Node 18+).
  • No Node-only packages (cheerio stays in the backend search only).
  • Pure, deterministic functions where possible.

apps/proxy

Hono server exposing search + fetch.

• index.ts — Hono app + @hono/node-server bootstrap, CORS via @hono/cors.
• routes/search.ts
  • GET /search?q=...&page=1 — scrape steamcommunity.com/workshop/browse/?appid=286160&searchtext=... with cheerio, extract { id, title, author, previewImageUrl } from the result grid. Supports pagenum pagination. No API key required.
• routes/items.ts
  • GET /items/:id — full parsed TTSMod.
  • GET /items/:id/file — raw save bytes, filename from getFileName.
• routes/health.ts
  • GET /health — liveness.
• env.ts — zod validation of STEAM_API_KEY, PORT.

Endpoints

Method	Path	Description	Auth
GET	/health	Liveness	—
GET	/search?q=&page=	Scrape Workshop browse, return item list	—
GET	/items/:id	Full parsed TTSMod	key
GET	/items/:id/file	Raw save bytes, filename from header	key

Data flow

/search?q=wingspan  →  scrape browse page  →  list of {id, title, ...}
        ↓
/items/:id          →  Steam API (needs key)  →  file_url
        ↓
                      download BSON → parse → TTSMod
        ↓
packages/extract    →  flatten objects / extract refs / download assets

Dependencies

• hono, @hono/node-server, @hono/cors — server.
• bson — BSON deserialization.
• cheerio — Workshop browse page scraping (backend only).
• zod — validation.
• tsx, typescript, eslint, prettier — tooling.

Tooling

• TypeScript strict mode.
• tsx for dev, tsc for build.
• vitest for unit tests, colocated as *.test.ts next to sources.
• Root scripts: pnpm dev, pnpm build, pnpm test, pnpm lint.

Build order

1. Scaffold workspace (pnpm-workspace.yaml, root package.json, base tsconfig, .npmrc).
2. packages/shared — types + zod schemas.
3. packages/tts — fetch + parse (existing scraper code).
4. packages/extract — objects, refs, download.
5. apps/proxy — search + items + health routes, env validation.
6. Wire up root scripts, .env.example, README.

Risks / caveats

• Search scraping is fragile — Steam can change their HTML or rate-limit. No official search API exists, so this is the standard approach.
• Members-only items won't appear in scraped search results.
• file_url may be missing/expired — handle cleanly with a 404-style error.
• BSON parsing is the expensive part — large saves; no caching by design (client-only).
• STEAM_API_KEY is exposed in the client process — acceptable for a personal tool; the key is only needed for the metadata call that yields file_url.

Open decisions (defaults in bold)

• packages/extract vs folding into packages/tts — separate package (clean fetch vs analyze boundary).
• Download output type — Blob (easier for <img>/<object> in a frontend) vs raw ArrayBuffer.
• Move traverseMod/markParent into extract — yes (traversal is analysis, not fetching).