dataforge — structured data, reshaped deterministically

Endpoints & prices

{
  "from": "csv",
  "to": "json",
  "data": "name,age\r\nAda,36\r\n\"Grace, R\",45\n"
}

{
  "format": "json",
  "from": "csv",
  "data": "[\n  {\n    \"name\": \"Ada\",\n    \"age\": 36\n  },\n  {\n    \"name\": \"Grace, R\",\n    \"age\": 45\n  }\n]\n",
  "rows": 2,
  "warnings": []
}

curl -sS https://dataforge.x.c00l.site/convert \
  -H 'content-type: application/json' \
  -d '{ "from": "csv", "to": "json", "data": "name,age\r\nAda,36\r\n\"Grace, R\",45\n" }'

{
  "from": "json",
  "data": "{\"store\":{\"book\":[{\"author\":\"Rees\",\"price\":8.95},{\"author\":\"Waugh\",\"price\":12.99}]}}",
  "path": "$.store.book[?(@.price < 10)].author"
}

{
  "count": 1,
  "matches": ["Rees"],
  "paths": ["$.store.book[0].author"],
  "pointers": ["/store/book/0/author"]
}

curl -sS https://dataforge.x.c00l.site/query \
  -H 'content-type: application/json' \
  -d '{ "from": "json", "data": "{\"store\":{\"book\":[{\"author\":\"Rees\",\"price\":8.95},{\"author\":\"Waugh\",\"price\":12.99}]}}", "path": "$.store.book[?(@.price < 10)].author" }'

{
  "from": "json",
  "data": "[{\"id\":1,\"email\":\"a@b.com\",\"tier\":\"pro\"},{\"id\":2,\"email\":\"c@d.io\",\"tier\":\"free\"}]"
}

{
  "mode": "infer",
  "schema": {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "id": { "type": "integer" },
        "email": { "type": "string", "format": "email" },
        "tier": { "enum": ["pro", "free"] }
      },
      "required": ["id", "email", "tier"]
    }
  }
}

curl -sS https://dataforge.x.c00l.site/schema \
  -H 'content-type: application/json' \
  -d '{ "from": "json", "data": "[{\"id\":1,\"email\":\"a@b.com\",\"tier\":\"pro\"},{\"id\":2,\"email\":\"c@d.io\",\"tier\":\"free\"}]" }'

{
  "from": "csv",
  "to": "csv",
  "data": "dept,name,salary\neng,Ada,120\neng,Alan,110\nops,Grace,100\n",
  "pipeline": [
    { "op": "groupBy", "by": ["dept"],
      "aggregations": { "headcount": "count", "total": "sum:salary" } },
    { "op": "sort", "by": ["-total"] }
  ]
}

{
  "format": "csv",
  "rows": 2,
  "columns": ["dept", "headcount", "total"],
  "data": "dept,headcount,total\neng,2,230\nops,1,100\n"
}

curl -sS https://dataforge.x.c00l.site/transform \
  -H 'content-type: application/json' \
  -d '{ "from": "csv", "to": "csv", "data": "dept,name,salary\neng,Ada,120\neng,Alan,110\nops,Grace,100\n", "pipeline": [ { "op": "groupBy", "by": ["dept"], "aggregations": { "headcount": "count", "total": "sum:salary" } }, { "op": "sort", "by": ["-total"] } ] }'

{
  "from": "yaml",
  "a": "name: app\nreplicas: 2\n",
  "b": "name: app\nreplicas: 3\nimage: nginx\n"
}

{
  "equal": false,
  "patch": [
    { "op": "replace", "path": "/replicas", "value": 3 },
    { "op": "add", "path": "/image", "value": "nginx" }
  ],
  "summary": ["changed /replicas: 2 -> 3", "added /image = \"nginx\""],
  "counts": { "replace": 1, "add": 1, "total": 2 }
}

curl -sS https://dataforge.x.c00l.site/diff \
  -H 'content-type: application/json' \
  -d '{ "from": "yaml", "a": "name: app\nreplicas: 2\n", "b": "name: app\nreplicas: 3\nimage: nginx\n" }'

{
  "from": "yaml",
  "to": "yaml",
  "data": "name: app\nreplicas: 2\n",
  "patch": [{ "op": "replace", "path": "/replicas", "value": 3 }]
}

{
  "format": "yaml",
  "applied": 1,
  "data": "name: app\nreplicas: 3\n"
}

curl -sS https://dataforge.x.c00l.site/patch \
  -H 'content-type: application/json' \
  -d '{ "from": "yaml", "to": "yaml", "data": "name: app\nreplicas: 2\n", "patch": [{ "op": "replace", "path": "/replicas", "value": 3 }] }'

Transform operations

Steps run in order. Every operation is a fixed primitive with validated arguments; expressions are parsed and interpreted, never eval'd.

• select
• drop
• rename
• filter
• compute
• cast
• sort
• groupBy
• flatten
• unflatten
• pivot
• unpivot
• dedupe
• limit
• offset

filter takes either a predicate object ({"field":"price","op":"lt","value":10}, composable with and/or/not) or an expression string ("price < 10 && tier == 'pro'"). groupBy aggregations: count, countDistinct, sum, avg, min, max, first, last, join, list.

Honest limits

• YAML is a practical subset. Block and flow collections, all three scalar styles with multi-line folding, block scalars |/> with chomping and explicit indentation, anchors, aliases, << merge keys, multiple documents and the YAML 1.2 core schema. Not supported, and rejected rather than mis-parsed: explicit keys (? foo), non-scalar keys, %TAG directives and custom tags, !!set/!!omap/ !!binary. Tabs used for indentation are an error.
• Numbers are precision-safe by default. 007 stays "007"; a 19-digit id stays a string rather than rounding. Quoted CSV fields are never coerced. Set options.coerce to "aggressive" for classic loose parsing, or "none" for strings only.
• TOML date-times become strings. JSON has no date type, so RFC 3339 text round-trips verbatim. TOML has no null: null keys are dropped on output (configurable).
• XML DTDs are refused. Internal <!ENTITY> declarations return 400 — that is the billion-laughs defence — and external SYSTEM/PUBLIC identifiers are never fetched. Only the five predefined entities and numeric character references are expanded, with a total expansion cap.
• JMESPath is not offered. JSONPath is implemented properly instead; asking for syntax:"jmespath" gets a clear 400.
• Caps — data 1,048,576 chars, body 2 MiB, depth 64, 200,000 rows, 20,000 query matches. Over the cap is a 400/413, not a crash.

Errors

{ "error": { "code": "csv_parse_error",
             "message": "CSV parse error at line 3, column 12: unterminated quoted field",
             "details": { "line": 3, "column": 12 } } }

A 4xx or 5xx is never settled, so a failed call is free.