diff --git a/docs/syntax-rework-plan.md b/docs/syntax-rework-plan.md new file mode 100644 index 0000000..6c3b255 --- /dev/null +++ b/docs/syntax-rework-plan.md @@ -0,0 +1,109 @@ +# Implementation Plan: Syntax Clarity & Parser Robustness + +Status: **Draft — not yet applied** + +## Goals + +1. Eliminate the backtracking heuristics in the value parser by making the value grammar context-free. +2. Remove redundant/ambiguous syntax forms. +3. Make union resolution deterministic and structural (not error-message-driven). +4. Address the `csv-parse` quote conflict. +5. Keep the `;` separator (forced by CSV constraints — not the source of ambiguity). + +All changes are **breaking** to the DSL → bump to `2.0.0`, update README + `csv-loader.md`, add a migration note. + +--- + +## Phase 1 — Mandatory brackets for composite values + +**Files:** `src/value-parser.ts`, `src/index.test.ts`, `src/csv-loader/reference-resolver.ts` + +- Remove the `allowOmitBrackets` parameter from `parseValue`, `parseTupleValue`, `parseArrayValue`. +- Delete the `elementIsTupleOrArray` disambiguation block and all `savedPos` restore logic in `parseArrayValue`. +- In `parseValue` (top-level), drop the `allowOmitBrackets = schema.type === "tuple" || "array"` special case — brackets are always required. +- Values now must be fully bracketed: `[a; 1]; [b; 2]` (no more `[a; 1]; [b; 2]` without outer brackets). + +**Tests to update** (currently assert bracket-optional behavior): +- `index.test.ts`: "should parse tuple without brackets" [L177], "should parse array without brackets" [L215], "should parse array of tuples without outer brackets" [L249]. +- `reference-resolver.ts` / `index.test.ts` reference tests: "should parse array reference IDs without brackets" [L685]. + +**Decision point:** Full mandatory brackets, or a *top-level-only* carve-out (brackets optional only when the value is the whole cell and the element is non-composite)? Recommend **full mandatory** for a clean grammar; note the carve-out as a fallback if ergonomics matter more. + +--- + +## Phase 2 — Single array form `Type[]` + +**Files:** `src/parser.ts`, `src/index.test.ts` + +- Remove the `[Type][]` array syntax from `parseSchemaInternal`. Keep only `Type[]`. +- This makes `[string]` a **1-tuple** (currently it collapses to an array). Update the tuple branch: `[t]` → `{ type: "tuple", elements: [t] }`; the special case `elements.length === 1 && !elements[0].name` becomes a real tuple. +- Update `schemaToTypeString` in `src/type-utils.ts` — the `array` case currently special-cases tuple elements; with `[Type][]` gone, arrays are always `elementType[]` (keep the `(union)[]` paren wrapping). + +**Tests to update:** +- `index.test.ts`: "should parse array of tuples" [L239] uses `[string; number][]` → becomes `[string; number][]` (unchanged) but "array of tuples without outer brackets" [L249] changes per Phase 1. +- Any test using `[number][]`, `[string][]` schema strings (e.g. README examples, `csv-loader.md`). + +--- + +## Phase 3 — Deterministic union resolution + +**Files:** `src/csv-loader/reference-resolver.ts`, `src/validator.ts` (via `type-utils.ts`), `src/csv-loader/module-gen.ts` + +- **Rule:** In a union, always try **non-reference members before reference members**, regardless of author order. This makes fallback structural instead of error-message-driven. +- In `parseValueWithReferences` and `resolveNestedReferences`, replace the `/not found|Circular reference|Failed to load/` error-message inspection with a fixed ordering: partition members into `nonRef` / `ref`, try `nonRef` first, then `ref`. +- In `module-gen.ts` `generateSchemaResolutionCode` union case, apply the same ordering so generated code matches runtime behavior. +- **Document the rule** in README (union member ordering is currently "first match wins" per AGENTS.md). + +**Note:** This changes behavior for `@users[] | string` — `string` would now win for plain strings. Flag as a deliberate semantic change. + +--- + +## Phase 4 — Quote conflict resolution + +**Files:** `src/parser.ts`, `src/csv-loader/loader.ts`, README, `csv-loader.md` + +**Options (pick one):** +- **(a) Standardize on single-quoted literals only** — minimal change, but leaves a footgun. +- **(b) Quote the entire schema cell** so `csv-parse` treats it as one field — requires loader-side handling to strip the outer quotes before parsing. +- **(c) Drop quotes entirely** — use bare-token literals (e.g. `on` / `off` as identifiers). Most robust, largest change. + +**Recommendation:** (b) — it's the only option that fixes the root cause (schema cells containing `"` break `csv-parse`) without redesigning the literal syntax. Requires: in `loader.ts`, detect schema-row cells that are fully quoted and unwrap before `parseSchema`. + +--- + +## Phase 5 — Cleanup & consistency + +**Files:** `src/csv-loader/reference-resolver.ts`, `src/csv-loader/loader.ts`, `src/csv-loader/module-gen.ts` + +- **Reverse-reference performance:** `resolveReverseReference` does `refTable.filter(...)` per row. Build a `Map` lookup once per referenced table (mirroring what `module-gen.ts` already generates) and reuse it. Cache the lookup alongside the parsed table in `referenceTableCache`. +- **Remove the `,` stop-character in `parseReferenceValue`** (value parser has a `,` stop char the schema parser doesn't — drift). Make reference ID parsing consistent with the rest of the value grammar. +- **`int`/`float`/`number` → `number`:** document the collapse in README (type-level collapse is intentional; parse-time distinction remains). + +--- + +## Phase 6 — Docs & migration + +**Files:** `README.md`, `csv-loader.md`, `AGENTS.md` + +- Update all syntax tables and examples for mandatory brackets + single array form. +- Add a **Migration section** listing the breaking changes: + 1. Composite values must be fully bracketed. + 2. `[Type][]` array form removed — use `Type[]`. + 3. `[single]` is now a 1-tuple, not an array. + 4. Union resolution now prefers non-reference members. + 5. (If Phase 4b) schema cells may be fully quoted. +- Update `AGENTS.md` gotchas: union ordering rule, quote handling. + +--- + +## Validation + +- `npm run typecheck` +- `npm run test` (update `src/index.test.ts`, `src/csv-loader/*.test.ts` first) +- Manually verify the integration fixture (`user_rev.csv` / `order_rev.csv`) still resolves. + +--- + +## Suggested execution order + +Phases 1 → 2 are tightly coupled (both touch bracket parsing) — do them together. Phase 3 is independent. Phase 4 is independent. Phases 5–6 are cleanup/docs and can go last. \ No newline at end of file