Collections

Notes, cards, decks and search, straight to the sqlite file

With fastanki.schema handling the file format, this module does the actual work: a Collection class that adds, finds, updates and removes notes, generating cards the way Anki would. Every change is bookkept for sync (usn=-1 marks pending rows, deletions go to graves, col.mod tracks modification time), so a collection managed here syncs cleanly with AnkiWeb and any other Anki client.

import tempfile
from fastcore.test import *
from anki.collection import Collection as AnkiCollection
import anki.lang
from anki.utils import field_checksum as anki_csum
from unittest.mock import patch as mock_patch

Text utilities

Anki fingerprints the first field of every note with a checksum used for duplicate detection, computed over the text with HTML stripped but media filenames kept (so <img src=cat.jpg> and a note that mentions cat.jpg collide, as they should). The guid identifies a note across collections, as 10 or so characters of base91.

---

source

guid64

def guid64():

A new random note guid, in Anki’s base91 encoding

---

source

field_csum

def field_csum(
    text:str
):

Anki’s dupe-detection checksum: first 4 bytes of the sha1 of the stripped first field

---

source

strip_html_media

def strip_html_media(
    s:str
):

Strip HTML from s, preserving media filenames, matching Anki’s checksum preprocessing

Checked directly against Anki’s own implementation, on plain text, HTML, entities, and media references:

anki.lang.set_lang('en')  # pylib's checksum helper strips HTML through a backend
for s in ['hello', 'dos', '<b>test</b>', 'a &amp; b', '<img src=cat.jpg>', "<img src='dog gif.gif'>x",
          '[sound:beep.mp3]', 'caf\xe9&nbsp;au lait', '<style>p {}</style>text']:
    test_eq(field_csum(s), anki_csum(s))

test_eq(field_csum('hello'), 2868168221)
g = guid64()
assert 1 <= len(g) <= 11 and set(g) <= set(map(chr, _B91))
g

'_>o$MB$jHA'

Opening a collection

fastanki keeps its own collection, separate from any desktop Anki profile: on machines where fastanki runs there’s usually no desktop Anki at all, and where there is one, staying out of its data folder means never fighting it for the sqlite lock. The desktop sees our changes the same way your phone does, through AnkiWeb sync.

---

source

Collection

def Collection(
    path
):

Initialize self. See help(type(self)) for accurate signature.

---

source

data_dir

def data_dir():

fastanki’s data folder (override with FASTANKI_DIR)

q and q1 keep the sql one-liners readable throughout the rest of the module. _dirty bumps col.mod, which is how sync notices there’s something to send.

td = Path(tempfile.mkdtemp())
col = Collection.open(td/'collection.anki2')
test_eq(col.q1('select ver from col'), 18)

Notetypes

A notetype’s structure spans three tables plus a protobuf config. NT gathers what note operations need: field names, sort field, kind, and the per-template card generation requirements.

---

source

Collection.notetypes

def notetypes():

Call self as a function.

---

source

Collection.nt

def nt(
    name
):

Notetype structure by name (case-insensitive)

---

source

NT

def NT(
    con, ntid, name
):

A notetype’s structure: fields, templates, and card generation rules

col.nt('Basic')

NT('Basic', flds=['Front', 'Back'], 1 templates)

The name=? lookup goes through sqlite’s unicase collation on the name column, so col.nt('basic') finds Basic:

test_eq(col.nt('basic').id, col.nt('Basic').id)
test_eq(col.notetypes(), ['Basic','Cloze'])

Decks

Deck names nest with :: in Anki’s UI, but the table stores the separator as \x1f, one row per level. Adding Spanish::Vocab creates Spanish too, as Anki does.

---

source

Collection.decks

def decks():

Call self as a function.

---

source

Collection.deck_id

def deck_id(
    name, create:bool=False
):

Id of deck name (::-separated), optionally creating it (and any missing parents)

did = col.deck_id('Spanish::Vocab', create=True)
test_eq(col.decks(), ['Default','Spanish','Spanish::Vocab'])
test_eq(col.deck_id('Spanish::Vocab'), did)
test_is(col.deck_id('Nope'), None)

Adding notes

Note is a plain holder for one row of the notes table, with dict-style field access by name. The interesting work is in add_note: joining fields with \x1f, computing sfld and csum, and deciding which cards to generate. For normal notetypes each template carries a requirement (“any of these fields non-empty” or “all of them”); for cloze notetypes there’s one card per distinct {c1::...} number across the fields.

Card generation follows Anki’s rule rather than the stored req cache: a normal template generates a card when its front would render non-empty for the note’s non-empty fields. renders_with_fields parses the front template ({Field}, {#Field}…{{/Field}}, {^Field}…{{/Field}}, comments) and reports whether any non-empty field reaches the output; static text alone never counts. field_is_empty matches Anki: whitespace and empty <br>/<div> tags only.

---

source

renders_with_fields

def renders_with_fields(
    qfmt, nonempty
):

Would front template qfmt render non-empty given the set of nonempty field names?

---

source

field_is_empty

def field_is_empty(
    s
):

Call self as a function.

---

source

note_cards

def note_cards(
    nt, fields, tags:tuple=()
):

Template ordinals that generate cards for a note’s fields (values in field order) and tags, using Anki’s render rule

---

source

Note

def Note(
    id, guid, mid, mod, usn, tags, flds, nt
):

Initialize self. See help(type(self)) for accurate signature.

Cloze ordinals are 1-based in note text ({c1::...}) but 0-based as card ords, so note_cards returns them shifted. KIND_NONE (0) requirements mean the template never generates; any is 1 and all is 2 in the protobuf enum.

bnt = col.nt('Basic')
test_eq(note_cards(bnt, ['hi','']), [0])
test_eq(note_cards(bnt, ['','back only']), [])
cnt = col.nt('Cloze')
test_eq(note_cards(cnt, ['{{c1::a}} and {{c3::b}}','']), [0,2])
test_eq(note_cards(cnt, ['no clozes here','']), [])

And the render rule agrees with Anki itself across the reversed and optional-reversed (conditional {#Add Reverse}) notetypes: Anki generates the cards, and note_cards predicts the same ordinals from the note’s fields.

gp = Path(tempfile.mkdtemp())/'gen.anki2'
aoc = AnkiCollection(str(gp))
want = []
for mname, flds in [('Basic (and reversed card)', dict(Front='q', Back='a')),
                    ('Basic (and reversed card)', dict(Front='q', Back='')),
                    ('Basic (optional reversed card)', dict(Front='q', Back='a')),
                    ('Basic (optional reversed card)', dict(Front='q', Back='a', **{'Add Reverse':'y'}))]:
    n = aoc.new_note(aoc.models.by_name(mname))
    for k,v in flds.items(): n[k] = v
    aoc.add_note(n, 1)
    want.append((mname, flds, sorted(c.ord for c in n.cards())))
aoc.close()
fc = Collection(gp)
for mname, flds, ords in want: test_eq(note_cards(fc.nt(mname), [flds.get(f,'') for f in fc.nt(mname).flds]), ords)
fc.close()

Now add itself. Anki assigns each new card a position (due for a new card is its queue position, not a date), drawn from the nextPos counter:

---

source

Collection.add

def add(
    model:str='Basic', deck:str='Default', tags:NoneType=None, **fields
):

Add a note (and its cards), returning the Note

---

source

Collection.get_note

def get_note(
    nid
):

Retrieve a Note by id

n = col.add(Front='hola', Back='hello', deck='Spanish::Vocab', tags='spanish')
n

Front: hola | Back: hello | 🏷 spanish

test_eq(n['Front'], 'hola')
test_eq(n.tags, ['spanish'])
test_eq(col.q1('select count(*) from cards where nid=?', n.id), 1)
test_eq(col.q1('select due from cards where nid=?', n.id), 1)
cz = col.add(model='Cloze', Text='{{c1::uno}} y {{c2::dos}}')
test_eq(col.q('select ord, due from cards where nid=? order by ord', cz.id), [(0,2),(1,3)])

Updating and removing

Updates rewrite the note row with fresh mod, usn=-1, sfld and csum, and generate any cards a new cloze number now requires (never deleting existing ones, matching Anki). Removals record graves, the tombstones sync uses to propagate deletions: type 0 is a card, 1 a note, 2 a deck.

---

source

Collection.update_note

def update_note(
    note, tags:NoneType=None, add_tags:NoneType=None, **fields
):

Update fields and/or tags of note (a Note or note id), returning the updated Note

n2 = col.update_note(n, Back='hello!', add_tags='greeting')
test_eq(n2['Back'], 'hello!')
test_eq(n2.tags, ['spanish','greeting'])
cz2 = col.update_note(cz.id, Text='{{c1::uno}} {{c2::dos}} {{c3::tres}}')
test_eq(col.q1('select count(*) from cards where nid=?', cz.id), 3)

---

source

Collection.remove_deck

def remove_deck(
    name
):

Remove deck name and its subdecks, with their cards and notes

---

source

Collection.remove_notes

def remove_notes(
    nids
):

Remove notes (and their cards) by id, recording graves for sync

Anki’s “remove deck” takes the notes down with it when their cards all lived there; we do the simpler, stricter thing and remove every note that had a card in the deck. The functional API only exposes deck removal through Collection, so surprises stay contained.

col.add(Front='bye', deck='Doomed')
test_eq(col.remove_deck('Doomed'), 1)
test_is(col.deck_id('Doomed'), None)
test_eq(col.q1("select count(*) from notes where flds like 'bye%'"), 0)
test_eq(col.q1("select count(*) from graves where type=2"), 1)

Finding

Anki has a search language; we have Python. Keyword arguments compose to SQL: deck, tag and added_days map to indexed columns, any other keyword is treated as a field name and matched case-insensitively as a substring, and where/args drop through to raw SQL against the notes table (aliased n) joined with cards (aliased c) for anything the keywords don’t cover.

---

source

Collection.find_note_ids

def find_note_ids(
    **kw
):

Call self as a function.

---

source

Collection.find_notes

def find_notes(
    deck:NoneType=None, tag:NoneType=None, added_days:NoneType=None, where:NoneType=None, args:tuple=(), **fields
):

Notes matching all given criteria; field kwargs are case-insensitive substring matches

test_eq(col.find_note_ids(deck='Spanish'), [n.id])
test_eq(col.find_note_ids(tag='greeting'), [n.id])
test_eq([x.id for x in col.find_notes(Front='HOLA')], [n.id])
test_eq(col.find_notes(Front='nope'), [])
test_eq(len(col.find_notes()), 2)
test_eq(col.find_note_ids(where='n.id=?', args=[cz.id]), [cz.id])

Cards get the same treatment. Card carries the scheduling columns; is_due selects learning cards whose time has come and review cards due today or earlier.

---

source

Collection.find_card_ids

def find_card_ids(
    **kw
):

Call self as a function.

---

source

Collection.find_cards

def find_cards(
    deck:NoneType=None, tag:NoneType=None, added_days:NoneType=None, is_due:NoneType=None, where:NoneType=None,
    args:tuple=(), **fields
):

Cards matching all given criteria (same keywords as find_notes, plus is_due)

---

source

Collection.today

def today():

Days since collection creation, Anki’s day counter

---

source

Card

def Card(
    id, nid, did, ord, mod, usn, type, queue, due, ivl
):

Initialize self. See help(type(self)) for accurate signature.

cards = col.find_cards(deck='Spanish')
test_eq(len(cards), 1)
cards[0]

Card 1784759309663 (nid: 1784759309663, due: 1, ivl: 0d, queue: 0)

New cards aren’t “due” in Anki’s sense, and nothing has been studied in this collection, so:

test_eq(col.find_cards(is_due=True), [])
test_eq(len(col.find_card_ids()), 4)

Due counts

---

source

Collection.due_counts

def due_counts(
    deck:NoneType=None
):

(new, learning, review) counts for deck and its subdecks (or the whole collection)

test_eq(col.due_counts(), (4,0,0))
test_eq(col.due_counts('Spanish'), (1,0,0))

The oracle test

Everything above wrote rows we believe are Anki-compatible. Time to ask Anki: open the collection we’ve been mutating, run its database check, use its own search to find our notes, add a note of its own, and make sure we can read that back.

col.close()
ac = AnkiCollection(str(td/'collection.anki2'))
report, ok = ac.fix_integrity()
test_eq(ok, True)

test_eq(ac.find_notes('deck:Spanish::Vocab'), [n.id])
test_eq(ac.find_notes('tag:greeting'), [n.id])
test_eq(sorted(ac.find_notes('')), sorted([n.id, cz.id]))
anote = ac.get_note(cz.id)
test_eq(anote['Text'], '{{c1::uno}} {{c2::dos}} {{c3::tres}}')

theirs = ac.new_note(ac.models.by_name('Basic'))
theirs['Front'] = 'from anki'
ac.add_note(theirs, 1)
ac.close()
col = Collection.open(td/'collection.anki2')
test_eq(col.find_note_ids(Front='from anki'), [theirs.id])
col.close()