Syncing

A pure-python client for the AnkiWeb sync protocol

fastanki syncs collections with AnkiWeb or a self-hosted Anki server. A delta sync exchanges changes since the previous sync. A full sync replaces one side’s entire SQLite database.

import tempfile
from fastcore.test import *
from google.protobuf.json_format import MessageToDict

The wire

SyncServer sends POST requests to {endpoint}sync/. It compresses request bodies with zstd. Bodies contain JSON, except for full uploads, which contain the database file. The client also decompresses responses with an anki-original-size header.

The anki-sync header contains:

• v: protocol version, currently 11.
• k: authentication key, empty during login.
• c: client version string.
• s: random session ID.

AnkiWeb uses HTTP 308 redirects to assign users to servers. The client follows the redirect and saves the new endpoint with the authentication key.

---

source

SyncServer

def SyncServer(
    endpoint:NoneType=None, hkey:str=''
):

One AnkiWeb-protocol server (AnkiWeb itself, or self-hosted)

Schema 11 conversions

Delta sync exchanges notetypes, decks and deck configurations in Anki’s 2012 JSON format, called schema 11. This supports older clients even when both databases use a newer schema.

The conversion functions preserve unknown keys in an other JSON blob. Some fields change representation between formats. For example, autoplay becomes disable_autoplay, with the opposite Boolean value. An initial ease of 2500 becomes 2.5.

---

source

nt_to_s11

def nt_to_s11(
    con, ntid
):

Convert notetype ntid’s rows to a schema-11 dict

---

source

nt_from_s11

def nt_from_s11(
    m
):

Convert a schema-11 notetype dict to (notetype_row, field_rows, template_rows)

---

source

deck_to_s11

def deck_to_s11(
    con, did
):

Convert deck did’s row to a schema-11 dict

---

source

deck_from_s11

def deck_from_s11(
    d
):

Convert a schema-11 deck dict to a decks table row (filtered decks unsupported: kept as-is if present locally)

---

source

dconf_to_s11

def dconf_to_s11(
    con, dcid
):

Convert deck config dcid’s row to a schema-11 dict

---

source

dconf_from_s11

def dconf_from_s11(
    d
):

Convert a schema-11 deck config dict to a deck_config table row

Check that conversions preserve values from Anki’s Python API.

from anki.collection import Collection as AnkiCollection
import anki.lang

def sub_eq(a, b, path=''):
    "Assert every key/value in `a` appears in `b` (recursively, with numeric tolerance)"
    for k,v in a.items():
        w = b.get(k)
        if isinstance(v,dict): sub_eq(v, w or {}, f'{path}{k}.')
        elif isinstance(v,float) or isinstance(w,float): assert abs(v-(w or 0))<1e-6, f'{path}{k}: {v} vs {w}'
        elif isinstance(v,bool) or isinstance(w,bool): assert bool(v)==bool(w), f'{path}{k}: {v} vs {w}'
        else: assert v==w, f'{path}{k}: {v!r} vs {w!r}'

td = Path(tempfile.mkdtemp())
ac = AnkiCollection(str(td/'oracle.anki2'))
mcon = connect(td/'scratch.anki2')
mcon.execute(SCHEMA)
m11 = ac.models.by_name('Cloze')
row,flds,tmpls = nt_from_s11(m11)
mcon.execute('insert into notetypes values (?,?,?,?,?)', row)
mcon.executemany('insert into fields values (?,?,?,?)', flds)
mcon.executemany('insert into templates values (?,?,?,?,?,?)', tmpls)
back = nt_to_s11(mcon, row[0])
sub_eq(dict(m11), back)

d11 = ac.decks.get(1)
mcon.execute('insert into decks values (?,?,?,?,?,?)', deck_from_s11(d11))
sub_eq({k:v for k,v in d11.items() if k!='mid'}, deck_to_s11(mcon, 1))
c11 = ac.decks.get_config(1)
mcon.execute('insert into deck_config values (?,?,?,?,?)', dconf_from_s11(c11))
sub_eq(dict(c11), dconf_to_s11(mcon, 1))

Gathering and applying changes

A delta sync exchanges data in this order:

• Graves record deletions of cards, notes and decks.
• Unchunked changes contain notetypes, decks, deck configurations and tags.
• Chunks contain notes, cards and review log entries as positional JSON arrays.

Pending local rows have usn=-1. The client assigns the server’s current usn to these rows when sending them.

The sync state machine

sync_collection calls meta to compare collection state. It returns noChanges when the modification timestamps match. If changes require a full sync, it raises FullSyncRequired.

A delta sync calls the remaining endpoints in order:

1. start
2. applyGraves
3. applyChanges
4. chunk, repeated until all server chunks have been downloaded
5. applyChunk, repeated until all local chunks have been uploaded
6. sanityCheck2
7. finish

sanityCheck2 compares object counts. On a mismatch, the client rolls back local changes and raises SanityCheckFailed. It changes the schema timestamp to require a full sync on the next attempt.

---

source

Collection.sync_collection

def sync_collection(
    srv
):

Delta-sync with srv, returning ‘noChanges’, ‘success’, or raising FullSyncRequired

---

source

Collection.sync_meta

def sync_meta():

---

source

SanityCheckFailed

def SanityCheckFailed(
    *args, **kwargs
):

Common base class for all non-exit exceptions.

---

source

FullSyncRequired

def FullSyncRequired(
    *args, **kwargs
):

Common base class for all non-exit exceptions.

---

source

SyncRequired

def SyncRequired(
    *args, **kwargs
):

Common base class for all non-exit exceptions.

Full sync and login

full_download replaces the local database with the server’s copy. full_upload replaces the server’s database with the local copy. The server checks the uploaded database’s integrity before returning OK.

Login uses hostKey to obtain an authentication key. sync saves the key and endpoint in auth.json beside the collection.

---

source

Collection.sync

def sync(
    user:NoneType=None, passw:NoneType=None, endpoint:NoneType=None, upload:bool=False
):

Sync with AnkiWeb (or endpoint): logs in if needed, handles delta and full syncs

---

source

Collection.load_auth

def load_auth(
    endpoint:NoneType=None
):

A SyncServer from saved auth, or None

---

source

Collection.save_auth

def save_auth(
    srv
):

---

source

Collection.full_upload

def full_upload(
    srv
):

Replace the server’s collection with ours

---

source

Collection.full_download

def full_download(
    srv
):

Replace the local collection with the server’s copy

Against a real server

Anki’s Rust implementation is the protocol reference. The examples below use fastanki and Anki as clients of a local Anki sync server. They check changes in both directions without contacting AnkiWeb.

These cells use eval: false. tests/test_sync.py runs the sequence with a fixture that starts and stops the server in a temporary data directory.

start_sync_server requires the anki package, a development dependency. It runs python -m anki.syncserver on a free local port and waits for a response. Call terminate() on the returned process to stop it. It also stops when Python exits.

---

source

start_sync_server

def start_sync_server(
    base, # Folder for the server's data
    user:str='tester', # Username the server will accept
    passw:str='s3kret', # Password for `user`
):

Start the anki wheel’s sync server on a free port, returning (process, endpoint); it stops at exit, or sooner via .terminate()

server, EP = start_sync_server(td/'server')

Upload a new collection. The local and server schema timestamps differ, requiring a full sync.

col = Collection.open(td/'ours'/'collection.anki2')
n1 = col.add(Front='syncme', Back='please', tags='wire', deck='Sync::Deep')
ncz = col.add(model='Cloze', Text='{{c1::first}} {{c2::second}}')
test_fail(lambda: col.sync_collection(SyncServer(EP, hkey='bad')))
res = col.sync(user='tester', passw='s3kret', endpoint=EP, upload=True)
test_eq(res, 'full sync')

test_eq(col.sync(), 'noChanges')

Use Anki to download and check the collection.

(td/'second').mkdir()
oc2 = AnkiCollection(str(td/'second'/'collection.anki2'))
auth = oc2.sync_login('tester', 's3kret', endpoint=EP)
st = oc2.sync_collection(auth, sync_media=False)
oc2.close_for_full_sync()
oc2.full_upload_or_download(auth=auth, server_usn=st.server_media_usn, upload=False)
oc2.reopen(after_full_sync=True)
test_eq(sorted(oc2.find_notes('')), sorted([n1.id, ncz.id]))
test_eq(oc2.find_notes('tag:wire'), [n1.id])
test_eq(oc2.find_notes('deck:Sync::Deep'), [n1.id])
onote = oc2.get_note(ncz.id)
test_eq(onote['Text'], '{{c1::first}} {{c2::second}}')

Add a note and deck in Anki, edit another note, and delete the cloze note. Sync these changes to the server.

theirs = oc2.new_note(oc2.models.by_name('Basic'))
theirs['Front'] = 'from the other side'
oc2.add_note(theirs, oc2.decks.add_normal_deck_with_name('Oracle').id)
on = oc2.get_note(n1.id)
on['Back'] = 'edited remotely'
oc2.update_note(on)
oc2.remove_notes([ncz.id])
st = oc2.sync_collection(auth, sync_media=False)
test_eq(st.required, st.NO_CHANGES)

Check that fastanki downloads Anki’s changes.

test_eq(col.sync(), 'success')
test_eq(col.find_note_ids(Front='from the other side'), [theirs.id])
test_eq(col.get_note(n1.id)['Back'], 'edited remotely')
test_fail(lambda: col.get_note(ncz.id))
test_eq([d for d in col.decks() if d=='Oracle'], ['Oracle'])

Check that Anki receives fastanki’s deletion and new tagged note.

n2 = col.add(Front='round trip', tags=['back','forth'])
col.remove_notes(n1.id)
test_eq(col.sync(), 'success')
st = oc2.sync_collection(auth, sync_media=False)
oc2.close()
oc3 = AnkiCollection(str(td/'second'/'collection.anki2'))
test_eq(sorted(oc3.find_notes('')), sorted([theirs.id, n2.id]))
test_eq(oc3.find_notes('tag:forth'), [n2.id])
oc3.close()
col.close()
server.terminate()