test

Run dialogs’ code messages through a live solveit, nbdev-test-style

Testing a notebook with nbdev-test runs its cells under execnb, outside solveit: no dialog context, no current-message tracking, no dialog kernel. For notebooks written as solveit dialogs (like dialoghelper’s own), that tests a different thing than what runs in production. solveit-test instead asks a running solveit instance to execute a notebook exactly as the app does – the production runloop, a fresh dialog kernel, real current-message state – and reports the code messages whose outputs contain errors.

from fastcore.test import *

Running one dialog

test_dlg is the whole protocol: restart the dialog’s kernel (starting the dialog if needed, so every run begins from clean state), queue all its code messages in order – solveit’s own run-all semantics, including messages hidden from the AI – then poll each queued message’s run flag until the queue drains, and return the messages that ended up with error outputs. The run happens server-side, so each message executes with the dialog itself as context, exactly as when its cells were authored.

---

source

test_dlg

async def test_dlg(
    dname:str, # Name/path of the dialog (relative to current dialog's folder, or absolute if starts with '/')
    ids:list=None, # Message ids to run (default: all code messages, including those hidden from the AI)
    timeout:int=600, # Max seconds to wait for the run to finish
    poll:float=0.5, # Seconds between completion checks
)->list: # Run messages left with error outputs (empty means the dialog passed)

Restart dname’s kernel, run its code messages through the solveit runloop, and return those that errored

To see it work we need a dialog on the live server, so we create a throwaway one with three code messages, the middle one broken. Note that the run continues past the failure – solveit’s run queue records the error and keeps going, so one bad cell doesn’t hide later ones:

tnm = f'/tmp_soltest_{os.urandom(4).hex()}'
await create_or_run_dialog(tnm, template=False)
for src in ('a=1', '1/0', 'print(a)'): await add_msg(src, msg_type='code', dname=tnm, placement='at_end')
errs = await test_dlg(tnm)
[m.id for m in errs]

['aa871758']

Exactly one message failed, and its output carries the traceback:

test_eq(len(errs), 1)
assert 'ZeroDivisionError' in str(errs[0].output)
errs[0].output

'---------------------------------------------------------------------------\nZeroDivisionError                         Traceback (most recent call last)\nCell In[11], line 1\n----> 1 1/0\n\nZeroDivisionError: division by zero'

A dialog whose run leaves no error outputs passes, and the scratch dialog cleans up fully – kernel stopped, file removed:

await del_msgs(errs[0].id, dname=tnm)
test_eq(await test_dlg(tnm), [])
await stop_dialog(tnm)
await rm_dialog(tnm)

{'success': 'deleted "/Users/jhoward/tmp_soltest_9140b8c8"'}

The command line

The solveit-test entrypoint maps each notebook path to its dialog name through the server’s own base path (so the notebooks must live under the tree the instance serves), runs each in turn, and reports nbdev-test-style. It respects nbdev’s own skip machinery, computed with nbdev’s API (a dev dependency, imported lazily so the rest of this module works without it): skip_exec: true frontmatter skips a whole notebook, #|eval: false (comment or meta form) skips a cell, and nbdev_export cells never run. A dialog that’s open in solveit gets its kernel restarted by its test run, so state you had in that session is lost – that’s the price of every run starting clean. Runs execute the notebooks’ code for real, mutating messages and outputs on disk, so run it on a clean checkout and review the diff: an example that doesn’t clean up after itself shows up there, which is itself worth knowing.

---

source

test_nbs

async def test_nbs(
    path:str='.', # An .ipynb file, or a directory of them, to test
    timeout:int=600, # Max seconds to wait per dialog
    keep:bool=False, # Leave dialog kernels running after their test?
    n_workers:int=None, # Max dialogs tested concurrently (default: min(num_cpus(), 8))
)->dict: # Failures per notebook name: erroring message ids, or a repr'd exception

Test each notebook under path as a dialog on the local solveit instance, printing progress nbdev-test-style

---

source

solveit_test

async def solveit_test(
    path:str='.', # An .ipynb file, or a directory of them, to test
    timeout:int=600, # Max seconds to wait per dialog
    keep:bool=False, # Leave dialog kernels running after their test?
    n_workers:int=None, # Max dialogs tested concurrently (default: min(num_cpus(), 8))
):

Run notebooks as dialogs on the local solveit instance, nbdev-test-style; they must live under its data path