# Files, CSV and JSON Read text with `•nget`, parse it with `•csv` or `•json`, and calculate. `•csv⁻¹` and `•json⁻¹` produce text, which `•nput` writes to a file. ``` text sales←•csv •nget "sales.csv" totals←+/¨sales "totals.json" •nput •json⁻¹ totals ``` Options go on the left as a keyed vector, and the data goes on the right. Option names are case-insensitive. ## Files ``` bpl ]help •nget ``` # `•nget` — Read a file `•nget path` reads a UTF-8 text file. `•nget "-"` reads the rest of standard input. A path that starts with `./` or `../` is relative to the file holding the code, as for [`•load`](system/load.qmd). `X •nget path` takes options on the left: `binary` (`1` reads a vector of byte values) and `encoding` (`"UTF-8"`). [Files](data.ipynb#files) covers bytes and options. Errors: `IO` for missing files, invalid UTF-8, other file errors, and standard input that fails or that the frontend doesn’t provide; `DOMAIN` for invalid options. ``` bpl ]help •nput ``` # `•nput` — Write a file `path •nput data` writes `data` to a new file. It returns the number of bytes written, which isn’t displayed, as with an assignment. Text is written as UTF-8. An array of numbers is written as bytes, each from 0 to 255, in ravel order. A path that starts with `./` or `../` is relative to the file holding the code, as for [`•load`](system/load.qmd). `"-" •nput data` writes `data` to standard output, with no line ending added. Text written this way just before a read of `⎕` is that read’s prompt. See [`⎕`](glyphs/quad.qmd#input). `X •nput data` takes options on the left: `path`, `overwrite` (`1` replaces an existing file) and `encoding` (`"UTF-8"`, for text). With the option `unique`, `•nput` writes a new file with a unique name inside the directory `path`. It returns the new file’s path, which is displayed. The option `prefix` sets the start of the name, as for [`•mkdir`](system/mkdir.qmd). Plain text on the left is the path. [Files](data.ipynb#files) covers bytes and options. Errors: `IO` for an existing file without `overwrite`, and for other file errors; `DOMAIN` for invalid options or byte values, and for bytes written to standard output that aren’t UTF-8. ``` bpl text←•nget "sales.csv" "copy.csv" •nput text ["path":"copy.csv" "overwrite":1] •nput text ``` Directories must already exist. Binary reads return exact integers in `0…255`. `•nput` writes an array of numbers as bytes, each an integer in that range. It checks the data before opening the file. ``` bpl bytes←["binary":1] •nget "image.bin" "copy.bin" •nput bytes "UTF-8" •ucs bytes ⍝ decode UTF-8 when appropriate bytes-256ₓ×bytes≥128ₓ ⍝ signed-byte interpretation ``` ## Numeric input ``` bpl ]help •vfi ``` # `•vfi` — Numeric input `•vfi text` returns two vectors, `[valid numbers]`, for the whitespace-separated fields of `text`. An invalid field has the flag `0ₓ` and the value `0`. Fields are parsed as numbers, never executed. `separators •vfi text` splits on each character in `separators` instead. It trims whitespace around fields. An empty field is a valid `0`. [Numeric input](data.ipynb#numeric-input) covers it. Errors: `DOMAIN` when either argument is not text. Invalid fields have flag `0ₓ` and value `0`. Valid fields have flag `1ₓ`: ``` bpl [valid nums]←•vfi "12 nope -3 1.5" valid nums valid#nums ``` [1 0 1 1]ₓ 12 0 ¯3 1.5 12 ¯3 1.5 A number written plainly, such as `2` or `1.5`, is approximate. A number written with an `x` suffix, such as `3x`, or as a rational, such as `1r4`, is exact. `•vfi` also accepts complex numbers written with `j` or `J`, `∞`, `inf`, `infinity`, ASCII signs and signed exponents. `nan` gives `$n`. An invalid field gives the number `0` with the flag `0ₓ`. The numbers all become approximate when one of them is approximate and a float can represent every one exactly. Otherwise each number keeps its kind. The `0` from an invalid field doesn’t count as approximate. ``` bpl 1⊃•vfi "2 3x 1r4 1j-2 -1e-3 ∞" ``` 2 3ₓ 1ᵣ4 1ⱼ¯2 ¯0.001 ∞ `separators •vfi text` splits on any supplied character and trims surrounding whitespace. Empty fields are valid zero. Internal whitespace remains part of the field. ``` bpl ',' •vfi "3.9,2.4,,76," '⋄' •vfi "1 ⋄ 2 3 ⋄ 4" ``` (1ₓ 1ₓ 1ₓ 1ₓ 1ₓ⋄3.9 2.4 0 76 0) (1ₓ 0ₓ 1ₓ⋄1 0 4) Empty input returns two empty vectors. With no separator characters (`""` on the left), nonempty input is one field. DOMAIN: either argument is not character text. ## JSON ``` bpl ]help •json ``` # `•json` — JSON `•json text` reads JSON5, which includes all JSON. JSON5 adds comments, trailing commas, single-quoted strings, unquoted keys, hexadecimal numbers, `Infinity` and `NaN`. Objects become keyed vectors. Arrays become vectors. Strings become character vectors. Integers of up to 128 bits stay exact. `true` and `false` become `$t` and `$f`. `•json⁻¹ Y` writes JSON text. Keyed axes become objects. Unkeyed axes become arrays. Character vectors become strings. Keyed entries that hold functions are left out. NaN writes as `null`. `["fill":v] •json text` reads `null` as `v`. The default is NaN. An integer array with a `null` stays exact. `["fill":v] •json⁻¹ Y` writes `v` as `null`. [JSON](data.ipynb#json) covers the conversions. Errors: `DOMAIN` for malformed JSON, with its line and column, and for values that JSON can’t hold: an infinity that isn’t `fill`, out-of-range floats, nonintegral rationals, complex numbers and other functions. Nested structure is kept: ``` bpl person←•json "{""name"":""Ann"",""scores"":[10,20]}" person.name person.scores ``` Ann [10 20]ₓ `•json` also reads JSON5, with comments, trailing commas, single-quoted strings and unquoted keys. ``` bpl •json "{name: 'Ann', scores: [10, 20,]} // JSON5" ``` ["name":"Ann" "scores":[10 20]ₓ] `"f.json" •nput •json⁻¹ Y` writes JSON to a file. Scalar arrays export their contents. Axis names and empty-array prototypes are left out: ``` bpl •json⁻¹ ["name":"Ann" "scores":[10 20]ₓ] •json⁻¹ [1 2⋄3 4]ₓ •json⁻¹ •json "{}" ``` {"name":"Ann","scores":[10,20]} [[1,2],[3,4]] {} JSON integers are exact. Numbers with a decimal point or an exponent are approximate. An array’s numbers combine by the rule in [Numeric input](#numeric-input). `true` and `false` read as `$t` and `$f`. `•json⁻¹` writes Booleans back as `true` and `false`: ``` bpl •json "[9223372036854775809,1.5,true,false]" •json⁻¹ •json "[true,false]" ``` 9223372036854775809ₓ 1.5 $t $f [true,false] `fill` replaces `null` on import. The default is NaN. On export, NaN and the `fill` value become `null`: ``` bpl •json "[1,null,3]" ["fill":¯1ₓ] •json "[1,null,3]" ["fill":∞] •json⁻¹ 1ₓ ∞ 3ₓ ``` [1 $n 3]ₓ [1 ¯1 3]ₓ [1,null,3] Duplicate object members keep the last value. ## CSV ``` bpl ]help •csv ``` # `•csv` — CSV `•csv text` parses CSV into a vector of columns. Headers become keys. Numeric columns become numbers. Missing numeric cells become NaN. An integer column with missing cells stays exact. Missing text cells become `""`. `•csv⁻¹ T` writes CSV text for a vector of columns. Keys supply the header. Column lengths must agree. NaN writes as an empty cell, and so does the exact value given as `fill`. Both directions take options on the left: `header`, `separator`, `quotechar`, `doublequote`, `escapechar`, `decimal`, `thousands`, `trim` and `fill`. Reading also takes `text_columns`, `numeric_columns` and `missing`, and writing takes `forcequotes` and `lineending`. `"forcequotes":2` quotes every field. [CSV](data.ipynb#csv) describes the options. Errors: `DOMAIN` for invalid options, duplicate headers, and cells that CSV can’t hold: nonintegral rationals, complex numbers, functions or nested cells; `LENGTH` for unequal record widths or columns. Each numeric column uses compact integer or float storage where possible: ``` bpl text←"price,qty 10.5,2 20.0,4" T←•csv text T T.qty +/¨T ``` ["price":[10.5 20] "qty":[2 4]ₓ] [2 4]ₓ ["price":30.5 "qty":6ₓ] `"f.csv" •nput •csv⁻¹ T` writes CSV to a file. Unkeyed input writes the data without a header. Parsing the text restores the columns: ``` bpl T←["price":10.5 20;"qty":2ₓ 4ₓ] •csv •csv⁻¹ T ``` ["price":[10.5 20] "qty":[2 4]ₓ] ### CSV options ``` bpl text←"price;qty 10,5;2 20,0;4" opts←["separator":';' "decimal":','] T←opts •csv text T csv←opts •csv⁻¹ T opts •csv csv ``` ["price":[10.5 20] "qty":[2 4]ₓ] ["price":[10.5 20] "qty":[2 4]ₓ]
Option Default Meaning
header Import: 1; export: has keys Read/write column names
separator ',' Field separator; use •ucs 9 for TSV
quotechar '"' Quote character; "" disables quoting
doublequote 1 Represent a quote inside a quoted field by doubling it
escapechar "" Escape character inside quoted fields
decimal '.' Decimal mark: '.' or ','
thousands "" Group separator; groups after the first contain three digits
trim 0 Trim surrounding whitespace from fields
fill Import: NaN; export: none Numeric missing-cell replacement. On export, NaN and this exact value write as an empty cell
text_columns ⍬ Import these columns as text
numeric_columns ⍬ Import these columns as numbers; invalid nonmissing text errors
missing ⍬ Additional missing-cell strings; empty cells are always missing
forcequotes 0 Export: 0 as needed, 2 all fields
lineending •ucs 10 Export: LF or CRLF (•ucs 13 10)
Column selectors are names or positions, counting from 0. A character vector names one column. Use a vector for several selectors. ``` bpl text←"id,qty 00123,5" T←["text_columns":"id"] •csv text T.id ``` ["00123"] Separator, quote and escape must be distinct ASCII characters other than CR, LF or NUL. Fields support Unicode, doubled quotes and embedded newlines. Import accepts LF, CRLF and CR record endings. Export with `escapechar` quotes every field and doubles literal escape characters. With quoting disabled, export errors if a field requires quoting. ### Numbers and missing cells Inference examines a whole column, ignoring missing cells. Integer fields produce exact numbers. Decimal/scientific notation produces floats. A column containing other text stays text. Quoted numbers participate in inference too. ``` bpl T←•csv "count,label 2,001 3,yes" T.count T.label ``` [2 3]ₓ "001" "yes" A column of integers uses integer storage when every value fits in 64 bits. Larger integers stay exact. A value with a decimal point or an exponent is approximate. A column’s numbers combine by the rule in [Numeric input](#numeric-input). Missing numeric cells become NaN, written `$n`. An integer column with a gap becomes floats. A column that also holds an integer too large for a float to hold exactly keeps each item’s kind, in mixed storage. Missing text cells become `""`. Entirely missing columns are text unless forced numeric. ``` bpl T←•csv "price,qty 10.5,2 ,4" T.price T.qty ``` 10.5 $n [2 4]ₓ Configure extra missing markers and numeric fill explicitly. ``` bpl text←"qty NA 4" ["missing":"NA" "fill":¯1ₓ] •csv text ``` ["qty":[¯1 4]ₓ] Export uses `-` for negative numbers, decimal/scientific float spelling and plain exact integers. Infinity writes as `inf` or `-inf`. Set `fill` to export a chosen numeric sentinel as an empty field. Nonintegral rationals, complex numbers, functions and nested nonstring cells give DOMAIN. Empty input returns an empty unkeyed vector. Header-only input returns keyed empty columns. Empty header names are allowed. Duplicate header names give DOMAIN. Unequal record widths or column lengths give LENGTH. Invalid forced numeric fields report their row and column.