import tempfile
from fastcore.test import *
from anki.collection import Collection as AnkiCollection
from anki.utils import checksumMedia
Media files and the AnkiWeb media sync protocol
Media sync is a protocol of its own, independent of collection sync: collection sync moves note and card rows, media sync moves files, and neither orders the other. It has its own usn sequence and no full-sync path, so every media sync is incremental. Local state is a media folder next to the collection plus a small sqlite db with one row per file; the folder is the truth, and a scan before each sync brings the db up to date. Like the rest of fastanki this was built against Anki’s Rust code (rslib/src/sync/media/) as the reference, and is tested against a real Anki server below with Anki itself as the second client.
Filenames
A media file’s name is its identity across devices, so Anki normalizes names aggressively before they may sync: NFC form (APFS stores and serves NFD), characters that break some platform stripped, Windows device names defused, no trailing dot or space, 120 bytes at most. A file whose name can’t survive this round trip is skipped by sync rather than mangled.
norm_fname
def norm_fname(
fname
):Normalize a media filename as Anki does: NFC, problem characters stripped, device names defused, 120 bytes max
The awkward cases are pinned by Anki’s own test suite (files.rs); we also push names through the oracle’s write_data, which runs the genuine Rust normalization, and check we picked the same name:
test_eq(norm_fname('foo.jpg'), 'foo.jpg')
test_eq(norm_fname('con.jpg[]><:"/?*^\\|\0\r\n'), 'con_.jpg')
test_eq(norm_fname('test.'), 'test._')
test_eq(norm_fname('test '), 'test _')
test_eq(norm_fname('x'*130+'.jpg'), 'x'*115+'.jpg')
td = Path(tempfile.mkdtemp())
ac = AnkiCollection(str(td/'oracle.anki2'))
names = ['ok.jpg', 'con.', 'foo bar.png', 'te\u0301st.gif', 'x'*130+'.webp'] # NFD input comes back NFC
for name in names: test_eq(ac.media.write_data(name, b'data'), norm_fname(name))
[(name, norm_fname(name)) for name in names[:4]][('ok.jpg', 'ok.jpg'),
('con.', 'con_._'),
('foo bar.png', 'foo bar.png'),
('tést.gif', 'tést.gif')]Adding files
Adding a file whose name is taken must never silently clobber different content, because after a sync every device would see the change. So an add is content-aware: same name and same bytes is a no-op, while same name and different bytes stores the data under name-{sha1}.ext instead. Case-insensitive filesystems make Foo.jpg and foo.jpg the same file, so when the name is uppercase and can’t be used as-is, Anki retries with the lowercased name before falling back to the hash rename - which means a fresh uppercase name is stored lowercased.
add_media_file
def add_media_file(
folder, data, fname
):Write data into folder under (normalized) fname without clobbering different content; returns the name used
The oracle shows every branch, and we behave identically (note the platform wrinkle in the middle: on a case-insensitive filesystem, an existing foo.jpg satisfies the same-content check for Foo.jpg, so the name comes back with its case preserved):
mf = td/'add'
cases = [('Foo.jpg', b'aaa'), ('Foo.jpg', b'aaa'), ('Foo.jpg', b'bbb'), ('dup.png', b'x'), ('dup.png', b'y')]
theirs = [ac.media.write_data(n, d) for n,d in cases]
ours = [add_media_file(mf, d, n) for n,d in cases]
test_eq(ours, theirs) # we make exactly the oracle's choices, branch by branch
test_eq(_sha1(b'aaa'), checksum(b'aaa')) # and our checksums are Anki's: hex sha1
test_eq(theirs[0], 'foo.jpg') # fresh uppercase is stored lowercased
test_eq(theirs[2], f'foo-{_sha1(b"bbb")}.jpg') # same name, different content: hash-renamed
test_eq(theirs[3:], ['dup.png', f'dup-{_sha1(b"y")}.png'])
ours['foo.jpg',
'Foo.jpg',
'foo-5cb138284d431abd6a053a56625ec088bfb88912.jpg',
'dup.png',
'dup-95cb0bfd2977c761298d9624e4b4d4c72a39974a.png']The media database
Sync state lives in a sqlite db of two tiny tables: one row per file (csum NULL means “deleted, deletion not yet uploaded”; dirty means “changed since last sync”), and a meta row holding the media folder’s last-seen mtime plus the last server usn we’ve caught up to. The layout matches Anki’s own: for collection.anki2, the folder is collection.media and the db collection.mdb, the same names the desktop app would pick.
Media
def Media(
folder, db_path
):A collection’s media folder plus the db tracking its sync state
m = Media(td/'m'/'media', td/'m'/'collection.mdb')
test_eq(m.count(), 0)
test_is(m.entry('test.mp3'), None)
m.set_entry('test.mp3', None, 0, False)
test_eq(m.entry('test.mp3'), (None, 0, 0))
m.set_entry('test.mp3', _sha1(b'hello'), 123, True)
test_eq(m.pending(25), [('test.mp3', _sha1(b'hello'), 123)])
test_eq(m.count(), 1)
m.set_meta(123, 321)
test_eq(m.meta(), (123, 321))
m.force_resync()
test_eq((m.count(), m.meta()), (0, (0, 0)))Change tracking
The media folder is the truth; the db just caches what sync needs to know about it. Before every sync, a scan reconciles the two: a file that’s new or has a changed mtime is re-hashed and marked dirty if its content really changed, and a db row whose file is gone becomes a deletion tombstone. Two mtime tricks keep this cheap: the folder’s own mtime (stored in dirMod, milliseconds) skips the whole scan when no directory entry changed, and per-file mtimes (seconds, matching Anki’s db format) decide what to re-hash. Some files can never sync, so the scan ignores them: names that aren’t valid NFC (macOS lists names in NFD; they’re treated as their NFC form when that form is valid), thumbs.db/.ds_store, empty files, and files over 100MiB.
The scenario below is Anki’s own change-tracking test: add, modify, delete, with a “touched but unchanged” case in the middle proving a bare mtime bump doesn’t cause a pointless upload. Since scans compare against the stored folder mtime, the test backdates mtimes rather than sleeping through the clock’s resolution:
def _backdate(p):
t = p.stat().st_mtime - 3
os.utime(p, (t, t))
trk = Media(td/'trk'/'media', td/'trk'/'collection.mdb')
f1 = trk.folder/'file.jpg'
f1.write_bytes(b'hello')
trk._register_changes()
test_eq(trk.count(), 1)
test_eq(trk.entry('file.jpg'), (_sha1(b'hello'), _mtime_s(f1), 1))
trk.set_entry('file.jpg', _sha1(b'hello'), _mtime_s(f1), False) # as if it synced
os.utime(f1); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.entry('file.jpg')[2], 0) # touched, content unchanged: still clean
f1.write_bytes(b'hello1'); _backdate(f1); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.entry('file.jpg'), (_sha1(b'hello1'), _mtime_s(f1), 1))
trk.set_entry('file.jpg', _sha1(b'hello1'), _mtime_s(f1), False)
(trk.folder/'Thumbs.db').write_bytes(b'x') # never syncs
(trk.folder/'empty.gif').touch() # nor do empty files
f1.unlink(); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.count(), 0)
test_eq(trk.entry('file.jpg'), (None, 0, 1)) # a deletion tombstone, awaiting upload
trk.pending(25)[('file.jpg', None, 0)]Zips on the wire
Files travel in zip batches, stored uncompressed. Each zip holds at most 25 files and stops accepting more once ~2.5MB has accumulated. Members are named 0, 1, … with a _meta JSON member mapping them to real filenames - but the two directions differ: an upload’s _meta is a list of [filename, zip_name] pairs where a null zip name means “this file was deleted”, while a download’s is a plain {zip_name: filename} dict (deletions never travel downward as zip entries; the change list already said what to delete).
up = _zip_up([('a.jpg', b'aaa'), ('gone.mp3', None), ('b.png', b'bb')])
with zipfile.ZipFile(io.BytesIO(up)) as z:
test_eq(json.loads(z.read('_meta')), [['a.jpg', '0'], ['gone.mp3', None], ['b.png', '2']])
test_eq(z.read('0'), b'aaa')
down = io.BytesIO()
with zipfile.ZipFile(down, 'w', zipfile.ZIP_STORED) as z: # what a server download looks like
z.writestr('0', b'data0'); z.writestr('_meta', json.dumps({'0':'pic.jpg'}))
got = _unzip_down(down.getvalue())
test_eq(got, [('pic.jpg', b'data0')])
got[('pic.jpg', b'data0')]_gather_zip reads the pending entries’ current bytes, honoring the size cutoff, uploading a since-vanished file as a deletion, and pruning entries that can never upload (the batch is then rebuilt from the db):
gz = Media(td/'gz'/'media', td/'gz'/'collection.mdb')
for i in range(3): (gz.folder/f'big{i}.bin').write_bytes(bytes(1500_000))
(gz.folder/'small.txt').write_bytes(b'hi')
gz._register_changes()
got = gz._gather_zip([(f'big{i}.bin', _sha1(bytes(1500_000)), 0) for i in range(3)])
test_eq([f for f,_ in got], ['big0.bin', 'big1.bin']) # 2.5MB cutoff: third file waits for the next batch
got = gz._gather_zip([('small.txt', _sha1(b'hi'), 0), ('vanished.png', 'deadbeef', 0), ('del.png', None, 0)])
test_eq(got, [('small.txt', b'hi'), ('vanished.png', None), ('del.png', None)]) # missing file -> uploaded as deletion
gz.set_entry('empty.bin', _sha1(b''), 1, True) # a zero-byte file can never upload
(gz.folder/'empty.bin').touch()
test_is(gz._gather_zip([('empty.bin', _sha1(b''), 1)]), None) # pruned; batch must be rebuilt...
test_eq([f for f,_,_ in gz.pending(25)], ['big0.bin', 'big1.bin', 'big2.bin', 'small.txt']) # ...without itTalking to the server
Media endpoints live under msync/ beside the collection sync’s sync/, and use the very same envelope (zstd body, anki-sync header), so SyncServer handles them via its prefix argument. Two quirks are media-specific: JSON replies arrive wrapped in a legacy {"data": ..., "err": ""} envelope, and some payloads are JSON arrays rather than objects (Rust’s serde_tuple): each change-list row is [fname, usn, sha1], and an upload reply is [processed, current_usn]. The five methods are begin (returns the server’s current media usn), mediaChanges, downloadFiles, uploadChanges, and mediaSanity.
MediaSanityFailed
def MediaSanityFailed(
*args, **kwargs
):Common base class for all non-exit exceptions.
Deciding what to do with a server change
Each change-list row names a file and the sha1 it now has on the server (empty means deleted there). Against our db entry - its checksum, and whether it’s pending upload - there are exactly nine cases, and Anki’s changes.rs resolves them with three rules: a genuine content conflict favors the server (the loser’s bytes are already gone from the server’s history either way); a local pending addition survives a remote deletion (it’ll upload next); and matching sides just tidy the dirty flag. We keep the decision as a pure function of (local sha1, remote sha1, state), with Anki’s own test table pinning all nine cases below.
required_change
def required_change(
local, remote, state
):Resolve one server row: sha1s (’’ means deleted), state in none/clean/dirty -> None, ‘download’, ‘delete’, or ‘clean’
for args,exp in [(('','','none'),None), (('','','clean'),'delete'), (('','1','dirty'),'download'),
(('1','','dirty'),None), (('1','','clean'),'delete'), (('1','1','clean'),None),
(('1','1','dirty'),'clean'),(('a','b','dirty'),'download'),(('a','b','clean'),'download')]:
test_eq(required_change(*args), exp)Files in and out
Applying a remote deletion moves the file to a media.trash folder beside the media folder rather than unlinking it, exactly as Anki does - a sync bug then costs a manual restore, not data. Incoming files normally write straight to the folder, but AnkiWeb still holds pre-normalization filenames from old clients; such a file is stored under a corrected name, and two dirty entries record the fixup for the next upload: a tombstone for the server’s bad name, and the renamed file itself.
tf = Media(td/'tf'/'media', td/'tf'/'collection.mdb')
test_eq(tf._add_from_server('ok.png', b'fine'), [('ok.png', _sha1(b'fine'), _mtime_s(tf.folder/'ok.png'), False)])
bad = 'we|ird.png' # a name today's Anki would never create
test_eq(tf._add_from_server(bad, b'x'), [(bad, None, 0, True), ('weird.png', _sha1(b'x'), _mtime_s(tf.folder/'weird.png'), True)])
tf._trash_files(['ok.png', 'never-existed.png'])
test_eq((tf.folder/'ok.png').exists(), False)
test_eq((tf.folder.with_name('media.trash')/'ok.png').read_bytes(), b'fine')The sync procedure
A media sync is: scan the folder, learn the server’s usn (begin, unless the caller already has it from the collection sync’s meta), pull whatever the server has that we haven’t seen, push whatever’s dirty, and - only if anything actually moved - ask the server to compare file counts. Pulling loops over change batches, applying each in one db transaction that also advances lastUsn, so an interrupted sync resumes where it stopped. Pushing has one subtlety: after an upload the server reports its new usn, and we may adopt it as “caught up” only when it’s exactly our old usn plus the files we sent - anything else means another client uploaded concurrently, and we must leave lastUsn alone so the next pull fetches their changes. A failed count check clears the local db and errors, making the next sync re-list everything against the server - Anki’s own recovery path.
Media.sync
def sync(
srv, server_usn:NoneType=None
):Sync the media folder with srv: pull server changes, push local ones, then verify counts
Media on a Collection
Collection gets three conveniences: media() opens the collection’s own media folder and db (a context manager, like Collection itself), add_media copies data in and returns the filename to cite in a note field - <img src="name"> for pictures, [sound:name] for audio - and sync_media runs a media sync with the saved auth. With this, the one safety rule from the days before media support (“never reference a file we can’t upload”) is discharged: referencing what add_media returns is always syncable.
Collection.sync_media
def sync_media(
srv:NoneType=None
):Media-sync with srv (default: the saved auth from a previous sync)
Collection.add_media
def add_media(
file, fname:NoneType=None
):Copy file (a path, or bytes with fname) into the media folder; returns the name to reference in note fields
Collection.media
def media():This collection’s Media; close it after use (or use as a context manager)
col = Collection.open(td/'ex'/'collection.anki2')
name = col.add_media(b'not really a png', 'diagram.png')
n = col.add(Front=f'What does this show? <img src="{name}">', Back='the architecture')
test_eq(name, 'diagram.png')
test_eq((td/'ex'/'collection.media'/name).read_bytes(), b'not really a png')
with col.media() as cm:
cm._register_changes()
test_eq(cm.pending(25), [(name, _sha1(b'not really a png'), _mtime_s(td/'ex'/'collection.media'/name))])
col.close()
nFront: What does this show? 
| Back: the architecture | 🏷 -
Against a real server
The same arrangement as the syncer notebook: the anki wheel’s genuine sync server on localhost, with Anki itself playing second client - its media sync runs in a backend thread, so we poll media_sync_status (which also rethrows any error). These cells are eval: false because they need the server process; tests/test_sync.py runs this sequence, plus batching and count-check-failure cases, under pytest.
import timeserver, EP = start_sync_server(td/'server')mcol = Collection.open(td/'sync'/'collection.anki2')
img = mcol.add_media(b'PNG our image', 'pic.png')
mcol.add(Front=f'What is this? <img src="{img}">', Back='our pic')
mcol.sync(user='tester', passw='s3kret', endpoint=EP, upload=True)
mcol.sync_media()
mcol.sync_media() # caught up: a no-opAnki pulls the collection and then the media, and both sides of the story check out: the bytes match, and its media check finds every reference resolved:
(td/'sync2').mkdir()
moc = AnkiCollection(str(td/'sync2'/'collection.anki2'))
auth = moc.sync_login('tester', 's3kret', endpoint=EP)
st = moc.sync_collection(auth, sync_media=False)
moc.close_for_full_sync()
moc.full_upload_or_download(auth=auth, server_usn=st.server_media_usn, upload=False)
moc.reopen(after_full_sync=True)
moc.sync_media(auth)
while moc.media_sync_status().active: time.sleep(0.05)
test_eq(Path(moc.media.dir(), 'pic.png').read_bytes(), b'PNG our image')
chk = moc.media.check()
test_eq((list(chk.missing), list(chk.unused)), ([], []))Then the reverse direction: Anki adds a file and deletes ours, and one sync_media brings us both changes - their audio arrives, and our deleted picture moves to media.trash, leaving a clean sync state:
(td/'oracle.mp3').write_bytes(b'ORACLE AUDIO')
oname = moc.media.add_file(str(td/'oracle.mp3'))
moc.media.trash_files(['pic.png'])
moc.sync_media(auth)
while moc.media_sync_status().active: time.sleep(0.05)
mcol.sync_media()
with mcol.media() as m:
test_eq((m.folder/oname).read_bytes(), b'ORACLE AUDIO')
test_eq((m.folder/'pic.png').exists(), False)
test_eq((m.folder.with_name('media.trash')/'pic.png').exists(), True)
test_eq((m.pending(25), m.count()), ([], 1))
moc.close()
mcol.close()
server.terminate()