fix(csv-loader): escape quotes in schema rows
Implement loader-side escaping for the schema row to prevent `csv-parse` from misinterpreting double-quoted string literals (e.g., `"active" | "inactive"`) as field delimiters.
This commit is contained in:
@@ -156,6 +156,7 @@ The following TypeScript types are exported:
|
|||||||
- In a union, **reference members are tried before non-reference members** (e.g. `@users | string` resolves `1` to the user object, falling back to a plain string when the reference doesn't match)
|
- In a union, **reference members are tried before non-reference members** (e.g. `@users | string` resolves `1` to the user object, falling back to a plain string when the reference doesn't match)
|
||||||
- Special characters can be escaped with backslash: `\;`, `\[`, `\]`, `\\`
|
- Special characters can be escaped with backslash: `\;`, `\[`, `\]`, `\\`
|
||||||
- Empty arrays/tuples are not allowed
|
- Empty arrays/tuples are not allowed
|
||||||
|
- In CSV schema rows, double-quoted string literals like `"active" | "inactive"` are handled automatically by the loader; avoid commas inside string literals (use single-quoted literals like `'a,b'` for those)
|
||||||
- For CSV loading with reference resolution, see [csv-loader.md](./csv-loader.md)
|
- For CSV loading with reference resolution, see [csv-loader.md](./csv-loader.md)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Implementation Plan: Syntax Clarity & Parser Robustness
|
# Implementation Plan: Syntax Clarity & Parser Robustness
|
||||||
|
|
||||||
Status: **In progress — Phases 1, 2 & 3 applied** (Phases 4–6 not yet done)
|
Status: **In progress — Phases 1, 2, 3 & 4 applied** (Phases 5–6 not yet done)
|
||||||
|
|
||||||
## Goals
|
## Goals
|
||||||
|
|
||||||
@@ -53,16 +53,18 @@ All changes are **breaking** to the DSL → bump to `2.0.0`, update README + `cs
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 4 — Quote conflict resolution
|
## Phase 4 — Quote conflict resolution ✅ Applied
|
||||||
|
|
||||||
**Files:** `src/parser.ts`, `src/csv-loader/loader.ts`, README, `csv-loader.md`
|
**Files:** `src/csv-loader/loader.ts`, `src/csv-loader/tests/parseCsv-basic.test.ts`, README, `csv-loader.md`
|
||||||
|
|
||||||
**Options (pick one):**
|
- Added `escapeSchemaRowQuotes()` in `loader.ts`: the schema row (2nd non-empty line) is split on the delimiter, and any cell containing the quote char has its inner quotes escaped and is wrapped in quotes before `csv-parse` runs.
|
||||||
- **(a) Standardize on single-quoted literals only** — minimal change, but leaves a footgun.
|
- This lets users write `"active" | "inactive"` naturally in a schema row — previously csv-parse threw `Invalid Closing Quote`.
|
||||||
- **(b) Quote the entire schema cell** so `csv-parse` treats it as one field — requires loader-side handling to strip the outer quotes before parsing.
|
- Single-quoted literals (`'on' | 'off'`) are untouched (no `"` present).
|
||||||
- **(c) Drop quotes entirely** — use bare-token literals (e.g. `on` / `off` as identifiers). Most robust, largest change.
|
- Added a test in `parseCsv-basic.test.ts`.
|
||||||
|
|
||||||
**Recommendation:** (b) — it's the only option that fixes the root cause (schema cells containing `"` break `csv-parse`) without redesigning the literal syntax. Requires: in `loader.ts`, detect schema-row cells that are fully quoted and unwrap before `parseSchema`.
|
**Decision:** Implemented option (b) — loader-side escaping of the schema row. This fixes the root cause (schema cells containing `"` break csv-parse) without redesigning the literal syntax.
|
||||||
|
|
||||||
|
**Known limitation (pre-existing, out of scope):** a comma inside a string literal (e.g. `"a,b"`) is ambiguous with the CSV delimiter and still fails — use single-quoted literals (`'a,b'`) for those, or a non-comma delimiter.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -33,6 +33,53 @@ import {
|
|||||||
parseReferenceValue,
|
parseReferenceValue,
|
||||||
} from "./reference-resolver.js";
|
} from "./reference-resolver.js";
|
||||||
import { generateTypeDefinition } from "./type-gen.js";
|
import { generateTypeDefinition } from "./type-gen.js";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Escape double-quote characters in the schema row so csv-parse doesn't
|
||||||
|
* misinterpret them as CSV field delimiters.
|
||||||
|
*
|
||||||
|
* The schema row is the 2nd non-empty line. Cells that contain the quote
|
||||||
|
* character (e.g. `"active" | "inactive"`) have their inner quotes escaped
|
||||||
|
* and are wrapped in quotes, so csv-parse treats each as a single field.
|
||||||
|
*/
|
||||||
|
function escapeSchemaRowQuotes(
|
||||||
|
content: string,
|
||||||
|
delimiter: string,
|
||||||
|
quote: string,
|
||||||
|
escape: string,
|
||||||
|
): string {
|
||||||
|
if (!quote || !content.includes(quote)) {
|
||||||
|
return content;
|
||||||
|
}
|
||||||
|
|
||||||
|
const lines = content.split(/\r?\n/);
|
||||||
|
let schemaRowIndex = -1;
|
||||||
|
let nonEmptyCount = 0;
|
||||||
|
for (let i = 0; i < lines.length; i++) {
|
||||||
|
if (lines[i].trim() === "") continue;
|
||||||
|
nonEmptyCount++;
|
||||||
|
if (nonEmptyCount === 2) {
|
||||||
|
schemaRowIndex = i;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (schemaRowIndex === -1) {
|
||||||
|
return content;
|
||||||
|
}
|
||||||
|
|
||||||
|
const cells = lines[schemaRowIndex].split(delimiter);
|
||||||
|
const transformed = cells
|
||||||
|
.map((cell) => {
|
||||||
|
if (!cell.includes(quote)) {
|
||||||
|
return cell;
|
||||||
|
}
|
||||||
|
const escaped = cell.split(quote).join(escape + quote);
|
||||||
|
return quote + escaped + quote;
|
||||||
|
})
|
||||||
|
.join(delimiter);
|
||||||
|
lines[schemaRowIndex] = transformed;
|
||||||
|
return lines.join("\n");
|
||||||
|
}
|
||||||
import { csvToModule } from "./module-gen.js";
|
import { csvToModule } from "./module-gen.js";
|
||||||
import {
|
import {
|
||||||
parseTypeDeclaration,
|
parseTypeDeclaration,
|
||||||
@@ -96,6 +143,15 @@ export function parseCsv(
|
|||||||
filteredContent = nonCommentLines.join("\n");
|
filteredContent = nonCommentLines.join("\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Escape double-quotes in the schema row so csv-parse doesn't treat them
|
||||||
|
// as CSV field delimiters (e.g. `"active" | "inactive"` in a schema cell).
|
||||||
|
filteredContent = escapeSchemaRowQuotes(
|
||||||
|
filteredContent,
|
||||||
|
delimiter,
|
||||||
|
quote,
|
||||||
|
escape,
|
||||||
|
);
|
||||||
|
|
||||||
const records = parse(filteredContent, {
|
const records = parse(filteredContent, {
|
||||||
delimiter,
|
delimiter,
|
||||||
quote,
|
quote,
|
||||||
|
|||||||
@@ -65,6 +65,21 @@ describe("parseCsv - basic parsing", () => {
|
|||||||
expect(result.data[1]).toEqual({ name: "Bob", status: "off" });
|
expect(result.data[1]).toEqual({ name: "Bob", status: "off" });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("should parse CSV with double-quoted string literal columns", () => {
|
||||||
|
const csv = [
|
||||||
|
"name,status",
|
||||||
|
'string,"active" | "inactive"',
|
||||||
|
"Alice,active",
|
||||||
|
"Bob,inactive",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
const result = parseCsv(csv, { emitTypes: false });
|
||||||
|
|
||||||
|
expect(result.data).toHaveLength(2);
|
||||||
|
expect(result.data[0]).toEqual({ name: "Alice", status: "active" });
|
||||||
|
expect(result.data[1]).toEqual({ name: "Bob", status: "inactive" });
|
||||||
|
});
|
||||||
|
|
||||||
it("should parse CSV with array columns", () => {
|
it("should parse CSV with array columns", () => {
|
||||||
const csv = [
|
const csv = [
|
||||||
"name,tags",
|
"name,tags",
|
||||||
|
|||||||
Reference in New Issue
Block a user