iversonnb j

Running J in-process through libj, and j magics for Jupyter and IPython

J’s console, jconsole, is a thin wrapper around libj, the J engine shared library. Rather than scraping the console over a pipe, iversonnb loads libj directly: JDo runs one line of J and returns only when it’s done, output arrives through a callback we register, and JInterrupt stops a long computation from another thread. The engine, its C API (jsource/jsrc/jlib.h), and profile.ijs (the standard library bootstrap) all ship with every J install, including pip install jlanguage. This notebook builds the binding up one step at a time.

Finding J

find_j

def find_j():

Locate the J binary directory: the one containing libj and profile.ijs

A J install is a directory with bin/ holding the engine library and profile.ijs side by side. which finds a real jconsole and resolves symlinks to its directory (the pip install jlanguage console script is a Python shim with no libj next to it, so the existence check skips it and the jlang import supplies the path instead). The fallbacks scan the standard per-OS install locations.

find_j()

Path('/Users/jhoward/aai-ws/.venv/lib/python3.13/site-packages/jlang/bin')

The engine

load_j

def load_j(
    jbin:NoneType=None
):

Load libj from jbin (or find_j()), declare its C signatures, and return (jbin,lib)

The whole API is in jsource/jsrc/jlib.h: JInit2 returns an engine instance told where its own directory is (bare JInit exists too, but then the engine can’t find libgmp next to itself, so extended precision breaks – and mixing the two in one process can crash), JDo runs one line of J against it, and JSM registers a 5-slot callback array {output, wd, input, unused, options} (jconsole passes exactly {Joutput, 0, Jinput, 0, SMCON}). The output callback receives everything J writes, tagged with a type: 1 is a formatted result, 2 an error display, 5 an exit request from 2!:55 (the text pointer carries the exit code, so it’s declared c_void_p and decoded per type – it can be NULL). The input callback supplies continuation lines when a multi-line definition is open. Here’s the smallest working setup, callbacks that just collect into a list:

jbin,lib = load_j()
jt = lib.JInit2(str(jbin).encode())
out = []
@_OUTCB
def outcb(j,typ,p): out.append((typ, ctypes.string_at(p).decode('utf-8','replace') if p else p))
cbs = (c_void_p*5)(cast(outcb,c_void_p), None, None, None, _SMCON)
lib.JSM(jt, cbs)
lib.JDo(jt, b'+/ % # 1 2 3 4'), out

(0, [(1, '0.25\n')])

JDo returns 0 for success (the J error number otherwise), and by the time it returns, the result has already arrived through the callback: completion detection is free, no prompt-scraping or sentinel needed. But this is a bare engine: no standard library. jconsole boots one by running a first sentence that sets BINPATH and ARGV and then runs profile.ijs (see jefirst in jsource/jsrc/jeload.c); we do the same. J executes right to left, so BINPATH is set first:

out.clear()
boot = f"(3 : '0!:0 y')<BINPATH,'/profile.ijs'[ARGV_z_=:<'jconsole'[BINPATH_z_=:'{jbin}'"
lib.JDo(jt, boot.encode()), out

(0, [])

The stdlib is now loaded:

out.clear()
lib.JDo(jt, b"toupper 'j is here'"), out

(0, [(1, 'J IS HERE\n')])

Errors

Errors need no scraping either: JDo returns the J error number, and the error display arrives as type-2 output. JError carries the session’s display as its message, like AplError does:

out.clear()
lib.JDo(jt, b"1 + 'a'"), out

(3, [(2, "|domain error, executing dyad +\n|y is character\n|   1    +'a'\n")])

JError

def JError(
    *args, **kwargs
):

A J error, carrying the session’s output (including the error display) as its message

A session object

J packages all of the above: load, init, register callbacks, boot the profile. The output callback collects (type,text) pairs, records exit requests, and guards against NULL. The input callback pops lines from a queue that run fills: J calls it whenever a multi-line definition is open, so a whole cell can be queued and each line lands where it belongs – top-level lines via JDo, body lines via the callback. The returned buffer is kept alive on self because the engine reads it after the callback returns (returning a Python bytes directly would leak: ctypes can’t manage the lifetime, and warns so).

J

def J(
    jbin:NoneType=None
):

A J session: the libj engine loaded in-process, with the stdlib profile booted

run queues the cell’s lines and feeds each remaining top-level line to JDo (a definition’s body lines are consumed from the queue by the input callback in between). Any nonzero return raises JError with everything the session printed, error display included; otherwise that output is returned:

J.run

def run(
    code
):

Run code (one or more lines of J) in the session, returning its output; raises JError on J errors

j = J()
print(j.run('m =: 2 3 $ 10 * 1 + i. 6\nm'))

10 20 30
40 50 60

State persists across calls, multi-line definitions work (the body lines travel through the input callback), and errors raise JError with the session’s error display:

test_eq(j.run('+/ , m'), '210\n')
test_eq(j.run('mean =: 3 : 0\n(+/ y) % # y\n)\nmean 1 2 3 4'), '2.5\n')
test_fail(lambda: j.run("m + 'x'"), contains='domain error')

Wrapping every use in print(j.run(...)) is clunky. __call__ makes the session itself a function, and JOut’s verbatim __repr__ means bare expressions display exactly as they do in a J session, while still comparing and slicing as ordinary strings. A call with no output returns None, so nothing displays:

J.call

def __call__(
    code
):

Run code, returning session output (or None if there is none)

JOut

def JOut(
    *args, **kwargs
):

Output text from a J call; displays verbatim

j('m ,. |. m')

10 20 30 40 50 60
40 50 60 10 20 30

test_eq(j('+/ , m'), '210\n')
test_is(j('m2 =: 10 * m'), None)

Extended precision works because JInit2 lets the engine find its bundled libgmp:

j('*/ 1 + i. 25x')

15511210043330985984000000

Getting values into Python

run returns display text: what you want to look at, not what you want to compute with. JGetM hands over a noun’s actual data: type, rank, and pointers to shape and ravel, straight out of engine memory. The common types map directly (booleans and integers, floats, characters); anything else (boxed, extended, rational) raises. _nest folds the flat ravel back into the array’s shape:

J.getm

def getm(
    name
):

Read noun name into Python: str for characters; int/float scalars and (nested) lists otherwise

pyval evaluates an arbitrary expression by assigning it to a scratch noun, reading that, and erasing it. Square brackets read better still:

J.getitem

def __getitem__(
    expr
):

Call self as a function.

J.pyval

def pyval(
    expr
):

Evaluate expr and return the result as a Python value

test_eq(j['m'], [[10,20,30],[40,50,60]])
test_eq(j['m > 25'], [[0,0,1],[1,1,1]])
test_eq(j['+/ % # 1 2 3 4'], 0.25)
test_eq(j["'py' , 'val'"], 'pyval')

JSetM is the same interface in reverse, so Python values can go into the workspace too: _jdat builds the typed buffers (strings as characters; numbers as integers, or floats if any element is one), and __setitem__ ships them. fn rounds the ergonomics out, turning any J verb into a Python callable – one argument applies it monadically, two dyadically, in J’s left-argument-first order:

J.setitem

def __setitem__(
    nm, v
):

Assign Python value v (scalar, string, or nested list) to noun nm

j['q'] = [[1,2],[3,4.5]]
test_eq(j['q'], [[1,2],[3,4.5]])
j['s'] = "it's"
test_eq(j['s'], "it's")
test_eq(j['+/ , q'], 10.5)

J.fn

def fn(
    code
):

A Python callable applying J verb code monadically or dyadically (left argument first)

sq = j.fn('*:')
test_eq(sq([1,2,3]), [1,4,9])
test_eq(j.fn('+/')([1,2,3]), 6)
test_eq(j.fn('{.')(2, [5,6,7]), [5,6])

Interrupting

JDo blocks its calling thread at the C level, so a Python signal handler can’t run while J computes – whoever hosts a J session must call interrupt from another thread (that’s exactly what the jkernel worker does with SIGINT). It’s safe cross-thread: it just sets the engine’s break flag, which J polls between sentences, so the running line stops with an attention interrupt error and the session survives:

J.interrupt

def interrupt():

Stop the currently running sentence with an attention interrupt; safe to call from another thread

import threading,time

j('spin =: 3 : 0\nn =. 0\nwhile. n < 1e8 do. n =. n + 1 end.\n)')
t0 = time.time()
threading.Timer(0.5, j.interrupt).start()
test_fail(lambda: j('spin 0'), contains='attention interrupt')
assert time.time()-t0 < 5
test_eq(j('2+2'), '4\n')

Exit requests and shutdown

When J code asks to leave (exit 0, or the 2!:55 foreign it wraps), the request arrives as a type-5 output whose text pointer carries the exit code. The engine itself keeps going – it’s the front end’s decision – so J records the code in .exited, run returns quietly instead of raising, and a host like the jkernel can honor it. Unlike a Dyalog session there’s no child process to orphan: the engine lives and dies with the host process, so no atexit handling is needed and close (or the context manager) exists just to free the instance’s memory explicitly. A closed session is unusable.

J.exit

def __exit__(
    *args
):

Call self as a function.

J.enter

def __enter__():

Call self as a function.

J.close

def close():

Free the engine instance; the session is unusable afterwards

with J() as j2:
    test_is(j2.exited, None)
    j2('exit 7')
    test_eq(j2.exited, 7)

The `j` magics

%%j runs a cell and displays the session output verbatim; %j expr returns the expression’s value as a Python object (it’s just j[expr]), so it works on the right of an assignment: z = %j z. A trailing ; on a cell suppresses its output, and the engine only starts on first use, not when the magic is registered. Loading the iversonnb.j extension registers it.

JMagic

def JMagic(
    jbin:NoneType=None
):

IPython %j/%%j magics, driving a lazily-started J session

load_ipython_extension

def load_ipython_extension(
    ipython
):

Required function for creating magic

create_j_magic

def create_j_magic(
    shell:NoneType=None
):

Create a JMagic and register its j line/cell magic with shell, returning it

# Only required if you don't load the extension
magic = create_j_magic()

%%j
m3 =: 3 3 $ i. 9
m3 +/ . * m3

15 18  21
42 54  66
69 90 111

The line magic brings values back into Python, and a trailing ; suppresses cell output entirely:

z = %j m3
test_eq(z, [[0,1,2],[3,4,5],[6,7,8]])

%%j
big =: 1000 1000 $ i. 5
big + big;

Cleanup

Free the engines this notebook started: the magic’s session, the J object’s, and the raw walkthrough one.

if magic.o: magic.o.close()
j.close()
lib.JFree(jt)