aplnb core

Driving Dyalog APL over the RIDE protocol, and apl magics for Jupyter and IPython

aplnb runs Dyalog APL from Python and provides apl magics for Jupyter and IPython. It uses the RIDE protocol, as do Dyalog’s IDE and official Jupyter kernel. It needs no extra code loaded into the APL workspace.

RIDE messages distinguish output, errors, and readiness for input. The examples below show the protocol before building the Apl session object.

Finding and starting Dyalog

source

find_dyalog

def find_dyalog():

Locate the Dyalog interpreter binary

find_dyalog first looks for mapl or dyalog on PATH. It then checks standard installation directories on macOS and Linux.

find_dyalog()

'/usr/local/bin/dyalog'

RIDE supports two connection directions. With RIDE_INIT=SERVE:*:port, the interpreter listens for a client. Dyalog’s Jupyter kernel uses this mode and polls for the listening port.

aplnb uses CONNECT mode. It listens on a port allocated by the OS, then starts Dyalog with that address. accept() waits for the interpreter to connect. This avoids selecting an unused port before binding it. The environment also sets RIDE_SPAWNED=1; explicit shutdown is covered below.

source

start_dyalog

def start_dyalog(
    dyalog:NoneType=None, # Path to the interpreter binary; `find_dyalog()` result if None
    timeout:int=10, # Socket timeout (secs), so a client bug can never hang the caller
):

Spawn a Dyalog interpreter that connects back to us over RIDE; return (socket,Popen)

sock,proc = start_dyalog()

Message framing

The interpreter speaks first. Here are the raw bytes:

raw = sock.recv(64)
raw

b'\x00\x00\x00\x1cRIDESupportedProtocols=2'

A RIDE message contains a 4-byte big-endian total length, the literal RIDE, and a UTF-8 payload. Here 0x1c is 28: 4 length bytes, 4 bytes for RIDE, and 20 for SupportedProtocols=2.

The first two messages in each direction negotiate the protocol version as plain strings. Later payloads are JSON arrays containing a command name and its arguments.

Dyalog sometimes sends raw control characters inside JSON strings. ride_recv escapes these before decoding, as does the official Jupyter kernel.

source

ride_recv

def ride_recv(
    sock
):

Receive one RIDE message, JSON-decoded unless it’s a handshake string

source

ride_send

def ride_send(
    sock, msg
):

Send one RIDE message: a handshake str, or a [cmd,args] list sent as JSON

Each side sends SupportedProtocols=2 and UsingProtocol=2. The client then sends Identify and reads the interpreter’s details:

ride_send(sock, 'SupportedProtocols=2')
ride_send(sock, 'UsingProtocol=2')
ride_recv(sock)

'UsingProtocol=2'

ride_send(sock, ['Identify',{'apiVersion':1,'identity':1}])
info = ride_recv(sock)[1]
{k:info[k] for k in ('Vendor','Language','version','arch','platform')}

{'Vendor': 'Dyalog Limited',
 'Language': 'APL',
 'version': '20.0.53963',
 'arch': 'Unicode/64',
 'platform': 'Mac-64'}

Wait for SetPromptType with type 1, which signals readiness for input:

msgs = []
while True:
    m = ride_recv(sock)
    msgs.append(m)
    if m[0]=='SetPromptType' and m[1]['type']==1: break
msgs

[['UpdateDisplayName', {'displayName': 'CLEAR WS'}],
 ['SetPromptType', {'type': 1}]]

Executing code

Execute sends session input with a required trailing newline. Dyalog echoes the input and sends output in AppendSessionOutput messages. SetPromptType 1 signals that execution has finished and the interpreter is ready for more input.

ride_send(sock, ['Execute',{'text':'3×⍳4\n','trace':0}])
msgs = []
while True:
    m = ride_recv(sock)
    msgs.append(m)
    if m[0]=='SetPromptType' and m[1]['type']==1: break
msgs

[['UpdateSessionCaption', {'text': 'CLEAR WS - Dyalog APL'}],
 ['AppendSessionOutput', {'result': '3×⍳4\n', 'type': 14, 'group': 0}],
 ['SetPromptType', {'type': 0}],
 ['AppendSessionOutput', {'result': '3 6 9 12\n', 'type': 2, 'group': 0}],
 ['SetPromptType', {'type': 1}]]

Output type 14 is an input echo. HadError reports an error independently of its displayed text:

ride_send(sock, ['Execute',{'text':'1÷0\n','trace':0}])
msgs = []
while True:
    m = ride_recv(sock)
    msgs.append(m)
    if m[0]=='SetPromptType' and m[1]['type']==1: break
msgs

[['AppendSessionOutput', {'result': '1÷0\n', 'type': 14, 'group': 0}],
 ['SetPromptType', {'type': 0}],
 ['HadError', {'error': 11, 'dmx': 1}],
 ['AppendSessionOutput',
  {'result': 'DOMAIN ERROR: Divide by zero\n', 'type': 5, 'group': 0}],
 ['AppendSessionOutput', {'result': '      1÷0\n', 'type': 5, 'group': 0}],
 ['AppendSessionOutput', {'result': '       ∧\n', 'type': 5, 'group': 0}],
 ['SetPromptType', {'type': 1}]]

AplError includes Dyalog’s error display. Its reset flag indicates that the interpreter was replaced and workspace state was lost. AplPrompt is an internal exception for prompts other than the ready prompt.

source

AplPrompt

def AplPrompt(
    ptype
):

The session stopped at a non-ready prompt: 2=⎕ input, 3=incomplete input, 4=⍞ input

source

AplError

def AplError(
    msg, reset:bool=False
):

An APL error, carrying the session’s error display as its message; reset means the interpreter was replaced and workspace state lost

ride_run sends all non-blank input lines in one Execute message. It collects output, excluding input echoes and prompt strings. The result is (output, errno), with error number 0 for success.

A final prompt counts only after Dyalog has echoed every input line. This matters for multiline blocks, as the next example shows.

source

ride_run

def ride_run(
    sock, code
):

Run one or more lines of APL in the session; return (output,errno) once the session is ready again

Multi-line input

Dyalog 20.0 enables multiline session input by default. Send a complete block in one Execute message. The interpreter echoes each line and uses prompt type 3 while the block is open.

A type 3 prompt before all line echoes means Dyalog is still reading the block. After all echoes, it means the submitted block is incomplete. Do not send another Execute at that prompt: the interpreter can crash. Apl.run handles incomplete input by replacing the interpreter.

ride_run(sock, '''
:If 1
    ⎕←6×7
:EndIf''')

('42\n', 0)

Session state persists across calls:

test_eq(ride_run(sock, 'x←⍳3\nz←x×x\n⎕←z'), ('1 4 9\n', 0))

Errors come back with the HadError number and the session error display:

out,err = ride_run(sock, '1÷0')
print(out)
test_eq(err, 11)

DOMAIN ERROR: Divide by zero
      1÷0
       ∧

A session object

Apl starts Dyalog and completes the handshake before accepting input. It sets the print width to 32767, matching the official kernel, to avoid wrapping long output.

source

Apl

def Apl(
    dyalog:NoneType=None, timeout:int=10
):

A Dyalog APL session over the RIDE protocol

close sends Exit and waits up to three seconds. It kills the process if it has not exited. _connect registers this cleanup with atexit to avoid leaving Dyalog processes behind. Closing the socket alone does not reliably shut down a stuck interpreter.

source

Apl.close

def close():

Shut down the interpreter and close the connection

apl = Apl()
apl.info['version']

'20.0.53963'

run returns output or raises AplError:

Ordinary APL errors include the session’s error display.
Requests for ⎕ or ⍞ input are cancelled. The session remains usable.
Incomplete input, such as an unclosed :If, requires a new interpreter because of Dyalog/ride#1401. The exception has reset=True, indicating that workspace state was lost.

source

Apl.run

def run(
    code
):

Run code in the session, returning its output; raises AplError on APL errors

print(apl.run('m←2 3⍴⍳6\n⎕←m'))

1 2 3
4 5 6

The matrix remains available for later calls. Ordinary APL errors do not reset the session:

test_eq(apl.run('+/,m'), '21\n')
test_fail(lambda: apl.run('m+\'x\''), contains='DOMAIN ERROR')

Calling apl(code) displays output without an extra print. It returns AplOut, a str subclass that preserves Dyalog’s formatting in notebook displays. Results still support ordinary string operations. Calls with no output return None.

source

Apl.call

def __call__(
    code
):

Run code, returning session output (or None if there is none)

source

AplOut

def AplOut(
    *args, **kwargs
):

Output text from an Apl call; displays verbatim, in the SAX2 APL font where HTML is available

apl('m ∘.× ⍳4')

test_eq(apl('+/,m'), '21\n')
test_is(apl('m2←m×10'), None)

Getting values into Python

pyval returns a Python value instead of display text. It asks Dyalog to serialize the expression with ⎕JSON, then parses the JSON.

The HighRank option serializes arrays of rank 2 or higher as nested lists. Without it, ⎕JSON rejects these arrays. The left argument 1 forces serialization. Monadic ⎕JSON would try to parse a character vector as JSON.

source

Apl.pyval

def pyval(
    expr
):

Evaluate expr and return the result as a Python value

test_eq(apl.pyval('m'), [[1,2,3],[4,5,6]])
test_eq(apl.pyval('3×⍳4'), [3,6,9,12])
test_eq(apl.pyval('⎕A'), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ')

Use square brackets to read an APL expression as a Python value or assign a Python value to an APL variable:

source

Apl.setitem

def __setitem__(
    nm, v
):

source

Apl.getitem

def __getitem__(
    expr
):

apl['q'] = [[1,2],[3,4.5]]
test_eq(apl['q'], [[1,2],[3,4.5]])
test_eq(apl["'nested: ',⍕⎕NC'q'"], 'nested: 2')
apl['s'] = "it's"
test_eq(apl['s'], "it's")

⎕JSON imports nested lists as vectors of vectors. Use ↑ to make a rank-2 matrix:

apl('↑q')

1 2  
3 4.5

fn returns a Python callable for an APL function. One argument calls it monadically. Two arguments call it dyadically, with the left argument first. Arguments and results use the same JSON conversions as pyval and assignment.

source

Apl.fn

def fn(
    code
):

A Python callable applying APL function code monadically or dyadically

sq = apl.fn('{⍵*2}')
test_eq(sq([1,2,3]), [1,4,9])
test_eq(apl.fn('+/')([1,2,3]), 6)
test_eq(apl.fn('↑')(2, [5,6,7]), [5,6])

Use a context manager to close a session before process exit:

source

Apl.exit

def __exit__(
    *args
):

source

Apl.enter

def __enter__():

with Apl() as a2: test_eq(a2('2+2'), '4\n')

Input requests and incomplete input

Check that cancelled input requests leave the session usable:

test_fail(lambda: apl('x←⎕'), contains='not supported')
test_eq(apl('2+2'), '4\n')

test_fail(lambda: apl('x←⍞'), contains='not supported')
test_eq(apl('2+2'), '4\n')

Check that incomplete input resets the workspace:

try: apl(':If 1')
except AplError as ex: err = ex
test_eq(err.reset, True)
assert 'wedged' in str(err)
test_eq(apl('2+2'), '4\n')

The timeout parameter limits startup. Once connected, Apl.run waits as long as the computation takes:

with Apl(timeout=2) as a2: assert float(a2.run('⎕←⎕DL 3')) >= 3

The `apl` magics

%%apl runs a cell and displays its session output. A trailing ; suppresses the display. %apl expr returns a Python value, as with apl[expr], and can appear in an assignment: z = %apl z.

The first magic call starts the interpreter and adds the APL language bar to the page. Registration alone does not start Dyalog.

Output uses Adám Brudzewsky’s SAX2 APL font, loaded locally or from a CDN. Monospace is the fallback when SAX2 is unavailable.

source

APLMagic

def APLMagic(
    dyalog:NoneType=None
):

IPython %apl/%%apl magics, driving a lazily-started Apl session

source

create_magic

def create_magic(
    shell:NoneType=None
):

Create an APLMagic and register its apl line/cell magic with shell, returning it

# Only required if you don't load the extension
magic = create_magic()

%%apl
m2←3 3⍴⍳9
⎕←m2

1 2 3
4 5 6
7 8 9

The line magic brings values back into Python:

z = %apl m2  ⍝ comments are fine here too
test_eq(z, [[1,2,3],[4,5,6],[7,8,9]])

The first example runs ]display, a Dyalog user command. The second suppresses cell output with a trailing ;.

%%apl
]display 2 2⍴'ab' 'cd' 1 2

┌→──────────┐
↓ ┌→─┐ ┌→─┐ │
│ │ab│ │cd│ │
│ └──┘ └──┘ │
│           │
│ 1    2    │
│           │
└∊──────────┘

%%apl
big←1000 1000⍴⍳12;

source

load_ipython_extension

def load_ipython_extension(
    ipython
):

Required function for creating magic

source

create_ipython_config

def create_ipython_config():

Called by aplnb_install to install magic

Cleanup

Shut down the sessions this notebook started: the magic’s, the Apl object’s, and the raw-socket walkthrough one.

magic.o.close()
apl.close()
ride_send(sock, ['Exit',{'code':0}])
sock.close()

Finding and starting Dyalog

find_dyalog

start_dyalog

Message framing

ride_recv

ride_send

Executing code

AplPrompt

AplError

ride_run

Multi-line input

A session object

Apl

Apl.close

Apl.run

Apl.__call__

AplOut

Getting values into Python

Apl.pyval

Apl.__setitem__

Apl.__getitem__

Apl.fn

Apl.__exit__

Apl.__enter__

Input requests and incomplete input

The apl magics

APLMagic

create_magic

load_ipython_extension

create_ipython_config

Cleanup

Apl.call

Apl.setitem

Apl.getitem

Apl.exit

Apl.enter

The `apl` magics