Implementation Plan: Syntax Clarity & Parser Robustness

Status: Draft — not yet applied

Goals

Eliminate the backtracking heuristics in the value parser by making the value grammar context-free.
Remove redundant/ambiguous syntax forms.
Make union resolution deterministic and structural (not error-message-driven).
Address the csv-parse quote conflict.
Keep the ; separator (forced by CSV constraints — not the source of ambiguity).

All changes are breaking to the DSL → bump to 2.0.0, update README + csv-loader.md, add a migration note.

Phase 1 — Mandatory brackets for composite values

Files: src/value-parser.ts, src/index.test.ts, src/csv-loader/reference-resolver.ts

Remove the allowOmitBrackets parameter from parseValue, parseTupleValue, parseArrayValue.
Delete the elementIsTupleOrArray disambiguation block and all savedPos restore logic in parseArrayValue.
In parseValue (top-level), drop the allowOmitBrackets = schema.type === "tuple" || "array" special case — brackets are always required.
Values now must be fully bracketed: [a; 1]; [b; 2] (no more [a; 1]; [b; 2] without outer brackets).

Tests to update (currently assert bracket-optional behavior):

index.test.ts: "should parse tuple without brackets" [L177], "should parse array without brackets" [L215], "should parse array of tuples without outer brackets" [L249].
reference-resolver.ts / index.test.ts reference tests: "should parse array reference IDs without brackets" [L685].

Decision point: Full mandatory brackets, or a top-level-only carve-out (brackets optional only when the value is the whole cell and the element is non-composite)? Recommend full mandatory for a clean grammar; note the carve-out as a fallback if ergonomics matter more.

Phase 2 — Single array form `Type[]`

Files: src/parser.ts, src/index.test.ts

Remove the [Type][] array syntax from parseSchemaInternal. Keep only Type[].
This makes [string] a 1-tuple (currently it collapses to an array). Update the tuple branch: [t] → { type: "tuple", elements: [t] }; the special case elements.length === 1 && !elements[0].name becomes a real tuple.
Update schemaToTypeString in src/type-utils.ts — the array case currently special-cases tuple elements; with [Type][] gone, arrays are always elementType[] (keep the (union)[] paren wrapping).

Tests to update:

index.test.ts: "should parse array of tuples" [L239] uses [string; number][] → becomes [string; number][] (unchanged) but "array of tuples without outer brackets" [L249] changes per Phase 1.
Any test using [number][], [string][] schema strings (e.g. README examples, csv-loader.md).

Phase 3 — Deterministic union resolution

Files: src/csv-loader/reference-resolver.ts, src/validator.ts (via type-utils.ts), src/csv-loader/module-gen.ts

Rule: In a union, always try non-reference members before reference members, regardless of author order. This makes fallback structural instead of error-message-driven.
In parseValueWithReferences and resolveNestedReferences, replace the /not found|Circular reference|Failed to load/ error-message inspection with a fixed ordering: partition members into nonRef / ref, try nonRef first, then ref.
In module-gen.ts generateSchemaResolutionCode union case, apply the same ordering so generated code matches runtime behavior.
Document the rule in README (union member ordering is currently "first match wins" per AGENTS.md).

Note: This changes behavior for @users[] | string — string would now win for plain strings. Flag as a deliberate semantic change.

Phase 4 — Quote conflict resolution

Files: src/parser.ts, src/csv-loader/loader.ts, README, csv-loader.md

Options (pick one):

(a) Standardize on single-quoted literals only — minimal change, but leaves a footgun.
(b) Quote the entire schema cell so csv-parse treats it as one field — requires loader-side handling to strip the outer quotes before parsing.
(c) Drop quotes entirely — use bare-token literals (e.g. on / off as identifiers). Most robust, largest change.

Recommendation: (b) — it's the only option that fixes the root cause (schema cells containing " break csv-parse) without redesigning the literal syntax. Requires: in loader.ts, detect schema-row cells that are fully quoted and unwrap before parseSchema.

Phase 5 — Cleanup & consistency

Files: src/csv-loader/reference-resolver.ts, src/csv-loader/loader.ts, src/csv-loader/module-gen.ts

Reverse-reference performance: resolveReverseReference does refTable.filter(...) per row. Build a Map<fk, rows[]> lookup once per referenced table (mirroring what module-gen.ts already generates) and reuse it. Cache the lookup alongside the parsed table in referenceTableCache.
Remove the , stop-character in parseReferenceValue (value parser has a , stop char the schema parser doesn't — drift). Make reference ID parsing consistent with the rest of the value grammar.
int/float/number → number: document the collapse in README (type-level collapse is intentional; parse-time distinction remains).

Phase 6 — Docs & migration

Files: README.md, csv-loader.md, AGENTS.md

Update all syntax tables and examples for mandatory brackets + single array form.
Add a Migration section listing the breaking changes:
1. Composite values must be fully bracketed.
2. [Type][] array form removed — use Type[].
3. [single] is now a 1-tuple, not an array.
4. Union resolution now prefers non-reference members.
5. (If Phase 4b) schema cells may be fully quoted.
Update AGENTS.md gotchas: union ordering rule, quote handling.

Validation

npm run typecheck
npm run test (update src/index.test.ts, src/csv-loader/*.test.ts first)
Manually verify the integration fixture (user_rev.csv / order_rev.csv) still resolves.

Suggested execution order

Phases 1 → 2 are tightly coupled (both touch bracket parsing) — do them together. Phase 3 is independent. Phase 4 is independent. Phases 5–6 are cleanup/docs and can go last.

6.2 KiB Raw Blame History Unescape Escape