fix(csv-loader): escape quotes in schema rows
Implement loader-side escaping for the schema row to prevent `csv-parse` from misinterpreting double-quoted string literals (e.g., `"active" | "inactive"`) as field delimiters.
This commit is contained in:
@@ -156,6 +156,7 @@ The following TypeScript types are exported:
|
||||
- In a union, **reference members are tried before non-reference members** (e.g. `@users | string` resolves `1` to the user object, falling back to a plain string when the reference doesn't match)
|
||||
- Special characters can be escaped with backslash: `\;`, `\[`, `\]`, `\\`
|
||||
- Empty arrays/tuples are not allowed
|
||||
- In CSV schema rows, double-quoted string literals like `"active" | "inactive"` are handled automatically by the loader; avoid commas inside string literals (use single-quoted literals like `'a,b'` for those)
|
||||
- For CSV loading with reference resolution, see [csv-loader.md](./csv-loader.md)
|
||||
|
||||
```javascript
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Implementation Plan: Syntax Clarity & Parser Robustness
|
||||
|
||||
Status: **In progress — Phases 1, 2 & 3 applied** (Phases 4–6 not yet done)
|
||||
Status: **In progress — Phases 1, 2, 3 & 4 applied** (Phases 5–6 not yet done)
|
||||
|
||||
## Goals
|
||||
|
||||
@@ -53,16 +53,18 @@ All changes are **breaking** to the DSL → bump to `2.0.0`, update README + `cs
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Quote conflict resolution
|
||||
## Phase 4 — Quote conflict resolution ✅ Applied
|
||||
|
||||
**Files:** `src/parser.ts`, `src/csv-loader/loader.ts`, README, `csv-loader.md`
|
||||
**Files:** `src/csv-loader/loader.ts`, `src/csv-loader/tests/parseCsv-basic.test.ts`, README, `csv-loader.md`
|
||||
|
||||
**Options (pick one):**
|
||||
- **(a) Standardize on single-quoted literals only** — minimal change, but leaves a footgun.
|
||||
- **(b) Quote the entire schema cell** so `csv-parse` treats it as one field — requires loader-side handling to strip the outer quotes before parsing.
|
||||
- **(c) Drop quotes entirely** — use bare-token literals (e.g. `on` / `off` as identifiers). Most robust, largest change.
|
||||
- Added `escapeSchemaRowQuotes()` in `loader.ts`: the schema row (2nd non-empty line) is split on the delimiter, and any cell containing the quote char has its inner quotes escaped and is wrapped in quotes before `csv-parse` runs.
|
||||
- This lets users write `"active" | "inactive"` naturally in a schema row — previously csv-parse threw `Invalid Closing Quote`.
|
||||
- Single-quoted literals (`'on' | 'off'`) are untouched (no `"` present).
|
||||
- Added a test in `parseCsv-basic.test.ts`.
|
||||
|
||||
**Recommendation:** (b) — it's the only option that fixes the root cause (schema cells containing `"` break `csv-parse`) without redesigning the literal syntax. Requires: in `loader.ts`, detect schema-row cells that are fully quoted and unwrap before `parseSchema`.
|
||||
**Decision:** Implemented option (b) — loader-side escaping of the schema row. This fixes the root cause (schema cells containing `"` break csv-parse) without redesigning the literal syntax.
|
||||
|
||||
**Known limitation (pre-existing, out of scope):** a comma inside a string literal (e.g. `"a,b"`) is ambiguous with the CSV delimiter and still fails — use single-quoted literals (`'a,b'`) for those, or a non-comma delimiter.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -33,6 +33,53 @@ import {
|
||||
parseReferenceValue,
|
||||
} from "./reference-resolver.js";
|
||||
import { generateTypeDefinition } from "./type-gen.js";
|
||||
|
||||
/**
|
||||
* Escape double-quote characters in the schema row so csv-parse doesn't
|
||||
* misinterpret them as CSV field delimiters.
|
||||
*
|
||||
* The schema row is the 2nd non-empty line. Cells that contain the quote
|
||||
* character (e.g. `"active" | "inactive"`) have their inner quotes escaped
|
||||
* and are wrapped in quotes, so csv-parse treats each as a single field.
|
||||
*/
|
||||
function escapeSchemaRowQuotes(
|
||||
content: string,
|
||||
delimiter: string,
|
||||
quote: string,
|
||||
escape: string,
|
||||
): string {
|
||||
if (!quote || !content.includes(quote)) {
|
||||
return content;
|
||||
}
|
||||
|
||||
const lines = content.split(/\r?\n/);
|
||||
let schemaRowIndex = -1;
|
||||
let nonEmptyCount = 0;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim() === "") continue;
|
||||
nonEmptyCount++;
|
||||
if (nonEmptyCount === 2) {
|
||||
schemaRowIndex = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (schemaRowIndex === -1) {
|
||||
return content;
|
||||
}
|
||||
|
||||
const cells = lines[schemaRowIndex].split(delimiter);
|
||||
const transformed = cells
|
||||
.map((cell) => {
|
||||
if (!cell.includes(quote)) {
|
||||
return cell;
|
||||
}
|
||||
const escaped = cell.split(quote).join(escape + quote);
|
||||
return quote + escaped + quote;
|
||||
})
|
||||
.join(delimiter);
|
||||
lines[schemaRowIndex] = transformed;
|
||||
return lines.join("\n");
|
||||
}
|
||||
import { csvToModule } from "./module-gen.js";
|
||||
import {
|
||||
parseTypeDeclaration,
|
||||
@@ -96,6 +143,15 @@ export function parseCsv(
|
||||
filteredContent = nonCommentLines.join("\n");
|
||||
}
|
||||
|
||||
// Escape double-quotes in the schema row so csv-parse doesn't treat them
|
||||
// as CSV field delimiters (e.g. `"active" | "inactive"` in a schema cell).
|
||||
filteredContent = escapeSchemaRowQuotes(
|
||||
filteredContent,
|
||||
delimiter,
|
||||
quote,
|
||||
escape,
|
||||
);
|
||||
|
||||
const records = parse(filteredContent, {
|
||||
delimiter,
|
||||
quote,
|
||||
|
||||
@@ -65,6 +65,21 @@ describe("parseCsv - basic parsing", () => {
|
||||
expect(result.data[1]).toEqual({ name: "Bob", status: "off" });
|
||||
});
|
||||
|
||||
it("should parse CSV with double-quoted string literal columns", () => {
|
||||
const csv = [
|
||||
"name,status",
|
||||
'string,"active" | "inactive"',
|
||||
"Alice,active",
|
||||
"Bob,inactive",
|
||||
].join("\n");
|
||||
|
||||
const result = parseCsv(csv, { emitTypes: false });
|
||||
|
||||
expect(result.data).toHaveLength(2);
|
||||
expect(result.data[0]).toEqual({ name: "Alice", status: "active" });
|
||||
expect(result.data[1]).toEqual({ name: "Bob", status: "inactive" });
|
||||
});
|
||||
|
||||
it("should parse CSV with array columns", () => {
|
||||
const csv = [
|
||||
"name,tags",
|
||||
|
||||
Reference in New Issue
Block a user