perf(csv-loader): optimize reverse reference resolution

Implement a reverse lookup cache to avoid re-filtering the entire
referenced table for every row during reverse reference resolution.
This improves performance from O(N*M) to O(N+M) where N is the number
of rows in the current table and M is the number of rows in the
referenced table.

Also update documentation to reflect the new union resolution
behavior and migration notes for version 2.0.0.
This commit is contained in:
2026-08-06 10:27:18 +08:00
parent 37e3514c0c
commit 641af7341a
5 changed files with 115 additions and 12 deletions
+12
View File
@@ -139,6 +139,8 @@ The following TypeScript types are exported:
| Int | `int` | `42` | | Int | `int` | `42` |
| Float | `float` | `3.14` | | Float | `float` | `3.14` |
| Number | `number` | `42` or `3.14` | | Number | `number` | `42` or `3.14` |
> **Note:** `int`, `float`, and `number` all collapse to the `number` type in generated TypeScript declarations (`schemaToTypeString`). The distinction is preserved at parse time — `int` rejects non-integer values while `float`/`number` accept them.
| Boolean | `boolean` | `true` or `false` | | Boolean | `boolean` | `true` or `false` |
| Tuple | `[Type1; Type2; ...]` | `[hello; 42; true]` | | Tuple | `[Type1; Type2; ...]` | `[hello; 42; true]` |
| Array | `Type[]` | `[1; 2; 3]` | | Array | `Type[]` | `[1; 2; 3]` |
@@ -159,6 +161,16 @@ The following TypeScript types are exported:
- In CSV schema rows, double-quoted string literals like `"active" | "inactive"` are handled automatically by the loader; avoid commas inside string literals (use single-quoted literals like `'a,b'` for those) - In CSV schema rows, double-quoted string literals like `"active" | "inactive"` are handled automatically by the loader; avoid commas inside string literals (use single-quoted literals like `'a,b'` for those)
- For CSV loading with reference resolution, see [csv-loader.md](./csv-loader.md) - For CSV loading with reference resolution, see [csv-loader.md](./csv-loader.md)
## Migration from 1.x
Version 2.0.0 introduces breaking changes to the schema DSL:
1. **Composite values must be fully bracketed.** Tuple and array values now always require `[]``[a; 1]; [b; 2]` is no longer accepted; write `[[a; 1]; [b; 2]]`.
2. **The `[Type][]` array form is removed.** Use `Type[]` (e.g. `[string; number][]` instead of `[[string; number]][]`).
3. **`[single]` is now a 1-tuple, not an array.** Use `Type[]` for single-element arrays.
4. **Union resolution prefers reference members.** In a union containing references, reference members are tried before non-reference members regardless of author order (see the notes above).
5. **Schema cells may be fully quoted.** Double-quoted string literals like `"active" | "inactive"` in a CSV schema row are handled automatically by the loader.
```javascript ```javascript
// rspack.config.js // rspack.config.js
module.exports = { module.exports = {
+10
View File
@@ -149,6 +149,16 @@ Uses [typed-csv](https://github.com/your-repo/typed-csv) syntax:
| Array | `string[]` | `[a; b; c]` | | Array | `string[]` | `[a; b; c]` |
| Tuple | `[string; number]` | `[hello; 42]` | | Tuple | `[string; number]` | `[hello; 42]` |
> **Note:** `int` and `float` are accepted in schemas and validated at parse time, but both collapse to `number` in generated type declarations.
## Migration from 1.x
- Composite values must be fully bracketed: `[[a; 1]; [b; 2]]`, not `[a; 1]; [b; 2]`.
- The `[Type][]` array form is removed — use `Type[]`.
- `[single]` is a 1-tuple, not an array.
- In unions containing references, reference members are tried first.
- Double-quoted string literals (`"active" | "inactive"`) in a schema row are handled automatically.
## License ## License
ISC ISC
+1 -1
View File
@@ -87,7 +87,7 @@ All changes are **breaking** to the DSL → bump to `2.0.0`, update README + `cs
1. Composite values must be fully bracketed. 1. Composite values must be fully bracketed.
2. `[Type][]` array form removed — use `Type[]`. 2. `[Type][]` array form removed — use `Type[]`.
3. `[single]` is now a 1-tuple, not an array. 3. `[single]` is now a 1-tuple, not an array.
4. Union resolution now prefers non-reference members. 4. Union resolution now prefers reference members (tried before non-reference members).
5. (If Phase 4b) schema cells may be fully quoted. 5. (If Phase 4b) schema cells may be fully quoted.
- Update `AGENTS.md` gotchas: union ordering rule, quote handling. - Update `AGENTS.md` gotchas: union ordering rule, quote handling.
+58 -11
View File
@@ -12,6 +12,16 @@ import { parseCsv } from "./loader.js";
/** Cache for loaded referenced tables */ /** Cache for loaded referenced tables */
const referenceTableCache = new Map<string, Record<string, unknown>[]>(); const referenceTableCache = new Map<string, Record<string, unknown>[]>();
/**
* Cache for reverse-reference lookups: filePath -> (foreignKey -> (fkValue -> rows[])).
* Built once per referenced table + foreign key, mirroring what module-gen emits,
* so resolveReverseReference doesn't re-filter the whole table per row.
*/
const reverseLookupCache = new Map<
string,
Map<string, Map<string, Record<string, unknown>[]>>
>();
/** Set of file paths currently being loaded (to detect circular references) */ /** Set of file paths currently being loaded (to detect circular references) */
const loadingFiles = new Set<string>(); const loadingFiles = new Set<string>();
@@ -39,6 +49,7 @@ export function loadReferenceTable(
): { ): {
lookup: Map<string, Record<string, unknown>>; lookup: Map<string, Record<string, unknown>>;
refTable: Record<string, unknown>[]; refTable: Record<string, unknown>[];
refFilePath: string;
} { } {
const baseDir = const baseDir =
refBaseDir || refBaseDir ||
@@ -83,7 +94,51 @@ export function loadReferenceTable(
} }
}); });
return { lookup, refTable }; return { lookup, refTable, refFilePath };
}
/**
* Build (and cache) a reverse lookup for a referenced table + foreign key:
* a map from the foreign-key value to the rows that reference it.
*/
function getReverseLookup(
schema: ReverseReferenceSchema,
refBaseDir: string | undefined,
defaultPrimaryKey: string,
currentFilePath: string | undefined,
): Map<string, Record<string, unknown>[]> {
const { refTable, refFilePath } = loadReferenceTable(
schema,
refBaseDir,
defaultPrimaryKey,
currentFilePath,
);
let byForeignKey = reverseLookupCache.get(refFilePath);
if (!byForeignKey) {
byForeignKey = new Map();
reverseLookupCache.set(refFilePath, byForeignKey);
}
const cached = byForeignKey.get(schema.foreignKey);
if (cached) return cached;
const lookup = new Map<string, Record<string, unknown>[]>();
for (const row of refTable) {
const fkValue = row[schema.foreignKey];
const fkStr =
fkValue !== null && fkValue !== undefined && typeof fkValue === "object"
? String((fkValue as Record<string, unknown>)[defaultPrimaryKey])
: String(fkValue);
const bucket = lookup.get(fkStr);
if (bucket) {
bucket.push(row);
} else {
lookup.set(fkStr, [row]);
}
}
byForeignKey.set(schema.foreignKey, lookup);
return lookup;
} }
export function resolveReferenceId( export function resolveReferenceId(
@@ -333,21 +388,13 @@ export function resolveReverseReference(
defaultPrimaryKey: string, defaultPrimaryKey: string,
currentFilePath: string | undefined, currentFilePath: string | undefined,
): Record<string, unknown>[] { ): Record<string, unknown>[] {
const { refTable } = loadReferenceTable( const lookup = getReverseLookup(
schema, schema,
refBaseDir, refBaseDir,
defaultPrimaryKey, defaultPrimaryKey,
currentFilePath, currentFilePath,
); );
const pkStr = String(pkValue); return lookup.get(String(pkValue)) ?? [];
return refTable.filter((row) => {
const fkValue = row[schema.foreignKey];
const fkStr =
fkValue !== null && fkValue !== undefined && typeof fkValue === "object"
? String((fkValue as Record<string, unknown>)[defaultPrimaryKey])
: String(fkValue);
return fkStr === pkStr;
});
} }
export function resolveNestedReferences( export function resolveNestedReferences(
@@ -43,6 +43,40 @@ describe("parseCsv - reverse reference resolution", () => {
} }
}); });
it("should resolve reverse reference when the foreign key is an object reference", () => {
// The referenced table's foreign key column is itself a reference, so the
// stored value is an object and the reverse lookup must key on its primary key.
const ordersCsvPath = path.join(fixturesDir, "rev_orders.csv");
const ordersContent = [
"id,customer,total",
"string,@users,number",
"1,1,100",
"2,1,50",
].join("\n");
fs.writeFileSync(ordersCsvPath, ordersContent);
try {
const csv = [
"id,name",
"string,string",
"# inject orders = ~rev_orders(customer)",
"1,Alice",
].join("\n");
const result = parseCsv(csv, {
emitTypes: false,
currentFilePath: path.join(fixturesDir, "test.csv"),
});
const orders = result.data[0].orders as Record<string, unknown>[];
expect(orders).toHaveLength(2);
expect(orders[0].id).toBe("1");
expect(orders[1].id).toBe("2");
} finally {
fs.unlinkSync(ordersCsvPath);
}
});
it("should return empty array for reverse reference with no matches", () => { it("should return empty array for reverse reference with no matches", () => {
const ordersCsvPath = path.join(fixturesDir, "rev_orders.csv"); const ordersCsvPath = path.join(fixturesDir, "rev_orders.csv");
const ordersContent = [ const ordersContent = [