iversonnb core

Driving Dyalog APL over the RIDE protocol, and apl magics for Jupyter and IPython

iversonnb talks to Dyalog using the RIDE protocol, the same TCP protocol used by Dyalog’s own IDE and by the official Jupyter kernel. Any Dyalog interpreter can serve it, nothing extra needs to be loaded into the workspace, and the message framing tells us exactly when output ends, whether it was an error, and when the interpreter is ready for more. This notebook builds up the client one step at a time, showing what actually goes over the wire.

Finding and starting Dyalog

---

source

find_dyalog

def find_dyalog():

Locate the Dyalog interpreter binary

On macOS the installer symlinks the interpreter into /usr/local/bin, so which normally finds it. The fallbacks scan the standard install locations on macOS and Linux.

find_dyalog()

'/usr/local/bin/dyalog'

RIDE connections can go in either direction. Dyalog’s Jupyter kernel uses RIDE_INIT=SERVE:*:port, where the interpreter listens and the client polls until the port opens, which means guessing a free port and racing to connect. CONNECT mode reverses the roles: we listen on a port the OS just gave us, tell the interpreter to dial back, and accept() blocks until it does. No race, no polling. RIDE_SPAWNED=1 tells the interpreter it belongs to this connection, so it exits when the connection drops.

---

source

start_dyalog

def start_dyalog(
    dyalog:NoneType=None, # Path to the interpreter binary; `find_dyalog()` result if None
    timeout:int=10, # Socket timeout (secs), so a client bug can never hang the caller
):

Spawn a Dyalog interpreter that connects back to us over RIDE; return (socket,Popen)

sock,proc = start_dyalog()

Message framing

The interpreter speaks first. Here are the raw bytes:

raw = sock.recv(64)
raw

b'\x00\x00\x00\x1cRIDESupportedProtocols=2'

Every message is framed as a 4-byte big-endian total length, the literal RIDE, then a UTF-8 payload. Here 0x1c is 28: 4 length bytes, 4 for RIDE, and the 20 characters of SupportedProtocols=2. The first two messages in each direction are plain strings negotiating the protocol version; every payload after that is JSON, a 2-element array of command name and arguments. One quirk: the interpreter sometimes sends raw control characters inside JSON strings, which strict parsers reject, so ride_recv escapes them before decoding (the official Jupyter kernel does the same).

---

source

ride_recv

def ride_recv(
    sock
):

Receive one RIDE message, JSON-decoded unless it’s a handshake string

---

source

ride_send

def ride_send(
    sock, msg
):

Send one RIDE message: a handshake str, or a [cmd,args] list sent as JSON

The handshake is symmetric: each side sends SupportedProtocols=2 and UsingProtocol=2, then we identify ourselves and the interpreter replies with its details:

ride_send(sock, 'SupportedProtocols=2')
ride_send(sock, 'UsingProtocol=2')
ride_recv(sock)

'UsingProtocol=2'

ride_send(sock, ['Identify',{'apiVersion':1,'identity':1}])
info = ride_recv(sock)[1]
{k:info[k] for k in ('Vendor','Language','version','arch','platform')}

{'Vendor': 'Dyalog Limited',
 'Language': 'APL',
 'version': '20.0.53963',
 'arch': 'Unicode/64',
 'platform': 'Mac-64'}

The interpreter then sends a series of status messages on its own. SetPromptType with type 1 means the session is ready for input, so read until that arrives:

msgs = []
while True:
    m = ride_recv(sock)
    msgs.append(m)
    if m[0]=='SetPromptType' and m[1]['type']==1: break
msgs

[['UpdateDisplayName', {'displayName': 'CLEAR WS'}],
 ['SetPromptType', {'type': 1}]]

Executing code

Execute takes a line of session input (the trailing newline is required). The interpreter echoes the input, streams back session output as AppendSessionOutput messages, and finally sends SetPromptType 1 to say it’s ready for the next line. That final message is what makes RIDE reliable to drive: there’s never a question of whether output has finished.

ride_send(sock, ['Execute',{'text':'3×⍳4\n','trace':0}])
msgs = []
while True:
    m = ride_recv(sock)
    msgs.append(m)
    if m[0]=='SetPromptType' and m[1]['type']==1: break
msgs

[['UpdateSessionCaption', {'text': 'CLEAR WS - Dyalog APL'}],
 ['AppendSessionOutput', {'result': '3×⍳4\n', 'type': 14, 'group': 0}],
 ['SetPromptType', {'type': 0}],
 ['AppendSessionOutput', {'result': '3 6 9 12\n', 'type': 2, 'group': 0}],
 ['SetPromptType', {'type': 1}]]

The type field distinguishes kinds of output: 14 is the echo of our own input (clients normally drop it), and the rest are genuine session output. Errors are flagged out-of-band with a HadError message, so there’s no scraping of output text to detect failure:

ride_send(sock, ['Execute',{'text':'1÷0\n','trace':0}])
msgs = []
while True:
    m = ride_recv(sock)
    msgs.append(m)
    if m[0]=='SetPromptType' and m[1]['type']==1: break
msgs

[['AppendSessionOutput', {'result': '1÷0\n', 'type': 14, 'group': 0}],
 ['SetPromptType', {'type': 0}],
 ['HadError', {'error': 11, 'dmx': 1}],
 ['AppendSessionOutput',
  {'result': 'DOMAIN ERROR: Divide by zero\n', 'type': 5, 'group': 0}],
 ['AppendSessionOutput', {'result': '      1÷0\n', 'type': 5, 'group': 0}],
 ['AppendSessionOutput', {'result': '       ∧\n', 'type': 5, 'group': 0}],
 ['SetPromptType', {'type': 1}]]

APL errors and unusual prompt states get their own exceptions: AplError for ordinary APL errors (carrying the session’s error display, plus a reset flag for when the interpreter had to be replaced), and AplPrompt, used internally when the session stops at a prompt other than ready.

---

source

AplPrompt

def AplPrompt(
    ptype
):

The session stopped at a non-ready prompt: 2=⎕ input, 3=incomplete input, 4=⍞ input

---

source

AplError

def AplError(
    msg, reset:bool=False
):

An APL error, carrying the session’s error display as its message; reset means the interpreter was replaced and workspace state lost

ride_run wraps the whole exchange: normalize the code to non-blank lines, send them all as one Execute, drop the echoes and prompt strings, collect everything else, and return the output plus the error number (0 if none) once the session is ready again. The echo counting is the subtle part: each input line is echoed back as a type 14 message, and a prompt change only counts as final once every line we sent has been echoed.

---

source

ride_run

def ride_run(
    sock, code
):

Run one or more lines of APL in the session; return (output,errno) once the session is ready again

Multi-line input

Since 20.0 the session has multiline input on by default, so a complete block construct can go in a single Execute. The interpreter echoes each line as it consumes it, switches the prompt to type 3 while a block is open, then runs the block once it’s complete. This is why ride_run counts echoes: a type 3 prompt seen before all our input has been echoed just means “still reading the block”, while one seen after means the input really was incomplete. (One thing to avoid: sending a fresh Execute message while the prompt is type 3 crashes the interpreter, so a batch of lines must go in one message.)

ride_run(sock, '''
:If 1
    ⎕←6×7
:EndIf''')

('42\n', 0)

Plain multi-statement input works the same way, and session state persists across calls:

test_eq(ride_run(sock, 'x←⍳3\nz←x×x\n⎕←z'), ('1 4 9\n', 0))

Errors come back with the HadError number and the session error display:

out,err = ride_run(sock, '1÷0')
print(out)
test_eq(err, 11)

DOMAIN ERROR: Divide by zero
      1÷0
       ∧

A session object

Apl packages all of the above: spawn the interpreter, shake hands, identify, widen the print width so long results aren’t wrapped (the same 32767 the official kernel uses), and wait for the first ready prompt.

---

source

Apl

def Apl(
    dyalog:NoneType=None, timeout:int=10
):

A Dyalog APL session over the RIDE protocol

close asks the interpreter to exit with an Exit message, then falls back to killing the process: a wedged interpreter ignores both Exit and SIGTERM. The interpreter doesn’t exit just because the connection drops, so _connect registers close to run at process exit; without that, every notebook kernel restart would leak a dyalog process.

---

source

Apl.close

def close():

Shut down the interpreter and close the connection

apl = Apl()
apl.info['version']

'20.0.53963'

run sends the code and interprets what comes back. APL errors raise AplError with the session’s error display. A ⎕ or ⍞ input request is cancelled (the session survives) and raises. Incomplete input, such as an unclosed :If, is the nasty case: it wedges the interpreter beyond recovery (Dyalog/ride#1401), so the only honest move is to replace the interpreter with a fresh one and raise with reset set, so callers know workspace state was lost.

---

source

Apl.run

def run(
    code
):

Run code in the session, returning its output; raises AplError on APL errors

print(apl.run('m←2 3⍴⍳6\n⎕←m'))

1 2 3
4 5 6

Output comes back exactly as the session formats it, so matrices look like APL. State persists for the life of the session, and errors raise AplError carrying the session’s error display:

test_eq(apl.run('+/,m'), '21\n')
test_fail(lambda: apl.run('m+\'x\''), contains='DOMAIN ERROR')

Wrapping every use in print(apl.run(...)) is clunky. Defining __call__ lets us treat the session itself as a function, and giving the output a verbatim __repr__ means bare expressions in a notebook display exactly as they do in a Dyalog session. AplOut subclasses str, so results still compare, slice, and print like ordinary strings. A cell with no output returns None, so nothing displays at all:

---

source

Apl.__call__

def __call__(
    code
):

Run code, returning session output (or None if there is none)

---

source

AplOut

def AplOut(
    *args, **kwargs
):

Output text from an Apl call; displays verbatim, in the SAX2 APL font where HTML is available

apl('m ∘.× ⍳4')

1  2  3  4
2  4  6  8
3  6  9 12
          
4  8 12 16
5 10 15 20
6 12 18 24

test_eq(apl('+/,m'), '21\n')
test_is(apl('m2←m×10'), None)

Getting values into Python

run returns display text, which is what you want to look at, but not what you want to compute with. For that, ask the interpreter to serialize the value with ⎕JSON and parse it on our side. The HighRank variant option makes rank≥2 arrays serialize as nested lists (by default ⎕JSON refuses them). The 1 left argument forces export: monadic ⎕JSON on a character vector would parse it as JSON instead of serializing it.

---

source

Apl.pyval

def pyval(
    expr
):

Evaluate expr and return the result as a Python value

test_eq(apl.pyval('m'), [[1,2,3],[4,5,6]])
test_eq(apl.pyval('3×⍳4'), [3,6,9,12])
test_eq(apl.pyval('⎕A'), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ')

pyval is wordy for the most common case, grabbing a variable. Square brackets read better, and ⎕JSON works in both directions, so we can assign Python values into the workspace too:

---

source

Apl.__setitem__

def __setitem__(
    nm, v
):

Call self as a function.

---

source

Apl.__getitem__

def __getitem__(
    expr
):

Call self as a function.

apl['q'] = [[1,2],[3,4.5]]
test_eq(apl['q'], [[1,2],[3,4.5]])
test_eq(apl["'nested: ',⍕⎕NC'q'"], 'nested: 2')
apl['s'] = "it's"
test_eq(apl['s'], "it's")

Note that ⎕JSON imports nested lists as vectors of vectors, not as rank-2 arrays. When you want a proper matrix, mix with ↑:

apl('↑q')

1 2  
3 4.5

fn goes one step further: it turns any APL function into a Python callable. One argument applies it monadically; two applies it dyadically, in APL’s left-argument-first order. Arguments travel in as ⎕JSON and the result comes back through pyval:

---

source

Apl.fn

def fn(
    code
):

A Python callable applying APL function code monadically or dyadically

sq = apl.fn('{⍵*2}')
test_eq(sq([1,2,3]), [1,4,9])
test_eq(apl.fn('+/')([1,2,3]), 6)
test_eq(apl.fn('↑')(2, [5,6,7]), [5,6])

Sessions close themselves at process exit, but a context manager is tidier for short-lived ones:

---

source

Apl.__exit__

def __exit__(
    *args
):

Call self as a function.

---

source

Apl.__enter__

def __enter__():

Call self as a function.

with Apl() as a2: test_eq(a2('2+2'), '4\n')

Input requests and incomplete input

A cell that asks for keyboard input can’t be satisfied in a notebook, so run cancels the request and raises, leaving the session usable:

test_fail(lambda: apl('x←⎕'), contains='not supported')
test_eq(apl('2+2'), '4\n')

test_fail(lambda: apl('x←⍞'), contains='not supported')
test_eq(apl('2+2'), '4\n')

Incomplete input is unrecoverable, so run replaces the wedged interpreter with a fresh workspace and raises with reset set:

try: apl(':If 1')
except AplError as ex: err = ex
test_eq(err.reset, True)
assert 'wedged' in str(err)
test_eq(apl('2+2'), '4\n')

Long computations must not be cut off: the socket timeout guards startup only, and is lifted once the session is ready, so run waits as long as the code takes.

with Apl(timeout=2) as a2: assert float(a2.run('⎕←⎕DL 3')) >= 3

The apl magics

%%apl runs a cell and displays the session output, rendered in Adám Brudzewsky’s SAX2 APL font (loaded from a CDN, falling back to plain monospace offline), so results look exactly as they do in a Dyalog session. %apl expr returns the expression’s value as a Python object (it’s just apl[expr]), so it works on the right of an assignment: z = %apl z. A trailing ; on a cell suppresses its output, and the interpreter only starts on first use, not when the magic is registered. The APL language bar is injected into the page the first time a magic runs.

---

source

APLMagic

def APLMagic(
    dyalog:NoneType=None
):

IPython %apl/%%apl magics, driving a lazily-started Apl session

---

source

create_magic

def create_magic(
    shell:NoneType=None
):

Create an APLMagic and register its apl line/cell magic with shell, returning it

# Only required if you don't load the extension
magic = create_magic()

%%apl
m2←3 3⍴⍳9
⎕←m2

1 2 3
4 5 6
7 8 9

The line magic brings values back into Python:

z = %apl m2  ⍝ comments are fine here too
test_eq(z, [[1,2,3],[4,5,6],[7,8,9]])

] user commands work, and a trailing ; suppresses cell output entirely:

%%apl
]display 2 2⍴'ab' 'cd' 1 2

┌→──────────┐
↓ ┌→─┐ ┌→─┐ │
│ │ab│ │cd│ │
│ └──┘ └──┘ │
│           │
│ 1    2    │
│           │
└∊──────────┘

%%apl
big←1000 1000⍴⍳12;

---

source

load_ipython_extension

def load_ipython_extension(
    ipython
):

Required function for creating magic

---

source

create_ipython_config

def create_ipython_config():

Called by iversonnb_install to install magic

Cleanup

Shut down the sessions this notebook started: the magic’s, the Apl object’s, and the raw-socket walkthrough one.

magic.o.close()
apl.close()
ride_send(sock, ['Exit',{'code':0}])
sock.close()