# test
Testing a notebook with `nbdev-test` runs its cells under execnb,
outside solveit: no dialog context, no current-message tracking, no
dialog kernel. For notebooks written as solveit dialogs (like
dialoghelper’s own), that tests a different thing than what runs in
production. `solveit-test` instead asks a running solveit instance to
execute a notebook exactly as the app does – the production runloop, a
fresh dialog kernel, real current-message state – and reports the code
messages whose outputs contain errors.
``` python
from fastcore.test import *
```
## Running one dialog
[`test_dlg`](https://AnswerDotAI.github.io/dialoghelper/test.html#test_dlg)
is the whole protocol: restart the dialog’s kernel (starting the dialog
if needed, so every run begins from clean state), queue all its code
messages in order – solveit’s own run-all semantics, including messages
hidden from the AI – then poll each queued message’s `run` flag until
the queue drains, and return the messages that ended up with error
outputs. The run happens server-side, so each message executes with the
dialog itself as context, exactly as when its cells were authored.
------------------------------------------------------------------------
source
### test_dlg
``` python
async def test_dlg(
dname:str, # Name/path of the dialog (relative to current dialog's folder, or absolute if starts with '/')
ids:list=None, # Message ids to run (default: all code messages, including those hidden from the AI)
timeout:int=600, # Max seconds to wait for the run to finish
poll:float=0.5, # Seconds between completion checks
)->list: # Run messages left with error outputs (empty means the dialog passed)
```
*Restart `dname`’s kernel, run its code messages through the solveit
runloop, and return those that errored*
To see it work we need a dialog on the live server, so we create a
throwaway one with three code messages, the middle one broken. Note that
the run continues past the failure – solveit’s run queue records the
error and keeps going, so one bad cell doesn’t hide later ones:
``` python
tnm = f'/tmp_soltest_{os.urandom(4).hex()}'
await create_or_run_dialog(tnm, template=False)
for src in ('a=1', '1/0', 'print(a)'): await add_msg(src, msg_type='code', dname=tnm, placement='at_end')
errs = await test_dlg(tnm)
[m.id for m in errs]
```
['aa871758']
Exactly one message failed, and its output carries the traceback:
``` python
test_eq(len(errs), 1)
assert 'ZeroDivisionError' in str(errs[0].output)
errs[0].output
```
'---------------------------------------------------------------------------\nZeroDivisionError Traceback (most recent call last)\nCell In[11], line 1\n----> 1 1/0\n\nZeroDivisionError: division by zero'
A dialog whose run leaves no error outputs passes, and the scratch
dialog cleans up fully – kernel stopped, file removed:
``` python
await del_msgs(errs[0].id, dname=tnm)
test_eq(await test_dlg(tnm), [])
await stop_dialog(tnm)
await rm_dialog(tnm)
```
{'success': 'deleted "/Users/jhoward/tmp_soltest_9140b8c8"'}
## The command line
The `solveit-test` entrypoint maps each notebook path to its dialog name
through the server’s own base path (so the notebooks must live under the
tree the instance serves), runs each in turn, and reports
`nbdev-test`-style. It respects nbdev’s own skip machinery, computed
with nbdev’s API (a dev dependency, imported lazily so the rest of this
module works without it): `skip_exec: true` frontmatter skips a whole
notebook, `#|eval: false` (comment or meta form) skips a cell, and
`nbdev_export` cells never run. A dialog that’s open in solveit gets its
kernel restarted by its test run, so state you had in that session is
lost – that’s the price of every run starting clean. Runs execute the
notebooks’ code for real, mutating messages and outputs on disk, so run
it on a clean checkout and review the diff: an example that doesn’t
clean up after itself shows up there, which is itself worth knowing.
------------------------------------------------------------------------
source
### test_nbs
``` python
async def test_nbs(
path:str='.', # An .ipynb file, or a directory of them, to test
timeout:int=600, # Max seconds to wait per dialog
keep:bool=False, # Leave dialog kernels running after their test?
n_workers:int=None, # Max dialogs tested concurrently (default: min(num_cpus(), 8))
)->dict: # Failures per notebook name: erroring message ids, or a repr'd exception
```
*Test each notebook under `path` as a dialog on the local solveit
instance, printing progress `nbdev-test`-style*
------------------------------------------------------------------------
source
### solveit_test
``` python
async def solveit_test(
path:str='.', # An .ipynb file, or a directory of them, to test
timeout:int=600, # Max seconds to wait per dialog
keep:bool=False, # Leave dialog kernels running after their test?
n_workers:int=None, # Max dialogs tested concurrently (default: min(num_cpus(), 8))
):
```
*Run notebooks as dialogs on the local solveit instance,
`nbdev-test`-style; they must live under its data path*