Media

Media files and the AnkiWeb media sync protocol

Media sync exchanges files with AnkiWeb independently of the notes and cards in collection sync. Neither protocol requires the other to run first. Media sync has its own update sequence numbers (USNs) and is always incremental.

fastanki keeps the files beside the collection and tracks their sync state in a sqlite database. Before syncing, it checks the folder for changes. The implementation follows Anki’s Rust code in rslib/src/sync/media/. The examples below compare it with Anki as a second client.

import tempfile
from fastcore.test import *
from anki.collection import Collection as AnkiCollection
from anki.utils import checksum

Filenames

Devices identify a media file by its filename. norm_fname follows Anki’s rules for names that work across platforms:

• Convert to Unicode NFC.
• Strip disallowed characters.
• Add underscores to Windows device names and names ending in a dot or space.
• Limit the name to 120 UTF-8 bytes.

Sync skips local files whose names don’t meet these rules. macOS needs special handling because it lists names in NFD form.

---

source

norm_fname

def norm_fname(
    fname
):

Normalize a media filename as Anki does: NFC, problem characters stripped, device names defused, 120 bytes max

Compare filename normalization with Anki:

test_eq(norm_fname('foo.jpg'), 'foo.jpg')
test_eq(norm_fname('con.jpg[]><:"/?*^\\|\0\r\n'), 'con_.jpg')
test_eq(norm_fname('test.'), 'test._')
test_eq(norm_fname('test '), 'test _')
test_eq(norm_fname('x'*130+'.jpg'), 'x'*115+'.jpg')

td = Path(tempfile.mkdtemp())
ac = AnkiCollection(str(td/'oracle.anki2'))
names = ['ok.jpg', 'con.', 'foo bar.png', 'te\u0301st.gif', 'x'*130+'.webp']   # NFD input comes back NFC
for name in names: test_eq(ac.media.write_data(name, b'data'), norm_fname(name))
[(name, norm_fname(name)) for name in names[:4]]

[('ok.jpg', 'ok.jpg'),
 ('con.', 'con_._'),
 ('foo bar.png', 'foo bar.png'),
 ('tést.gif', 'tést.gif')]

Adding files

add_media_file preserves existing files. If the name and content match, it returns the name without writing. If the content differs, it writes to name-{sha1}.ext.

Before adding the checksum, it retries names containing uppercase letters in lowercase. This also applies to new files: adding Foo.jpg creates foo.jpg.

---

source

add_media_file

def add_media_file(
    folder, data, fname
):

Write data into folder under (normalized) fname without clobbering different content; returns the name used

On a case-insensitive filesystem, Foo.jpg matches an existing foo.jpg. If the content matches too, add_media_file returns Foo.jpg without changing its case. Compare these choices with Anki:

mf = td/'add'
cases = [('Foo.jpg', b'aaa'), ('Foo.jpg', b'aaa'), ('Foo.jpg', b'bbb'), ('dup.png', b'x'), ('dup.png', b'y')]
theirs = [ac.media.write_data(n, d) for n,d in cases]
ours = [add_media_file(mf, d, n) for n,d in cases]
test_eq(ours, theirs)                              # we make exactly the oracle's choices, branch by branch
test_eq(_sha1(b'aaa'), checksum(b'aaa'))           # and our checksums are Anki's: hex sha1
test_eq(theirs[0], 'foo.jpg')                      # fresh uppercase is stored lowercased
test_eq(theirs[2], f'foo-{_sha1(b"bbb")}.jpg')     # same name, different content: hash-renamed
test_eq(theirs[3:], ['dup.png', f'dup-{_sha1(b"y")}.png'])
ours

['foo.jpg',
 'Foo.jpg',
 'foo-5cb138284d431abd6a053a56625ec088bfb88912.jpg',
 'dup.png',
 'dup-95cb0bfd2977c761298d9624e4b4d4c72a39974a.png']

The media database

For collection.anki2, fastanki uses the folder collection.media and database collection.mdb, matching Anki’s paths.

The media table tracks each file’s checksum, mtime and upload status. A null csum records a deletion. dirty marks entries awaiting upload. The meta table stores the folder’s last observed mtime and the last server USN processed.

---

source

Media

def Media(
    folder, db_path
):

A collection’s media folder plus the db tracking its sync state

m = Media(td/'m'/'media', td/'m'/'collection.mdb')
test_eq(m.count(), 0)
test_is(m.entry('test.mp3'), None)
m.set_entry('test.mp3', None, 0, False)
test_eq(m.entry('test.mp3'), (None, 0, 0))
m.set_entry('test.mp3', _sha1(b'hello'), 123, True)
test_eq(m.pending(25), [('test.mp3', _sha1(b'hello'), 123)])
test_eq(m.count(), 1)
m.set_meta(123, 321)
test_eq(m.meta(), (123, 321))
m.force_resync()
test_eq((m.count(), m.meta()), (0, (0, 0)))

Change tracking

The scan records additions, changes and deletions for upload. It hashes new files and files whose mtimes have changed. A changed mtime alone doesn’t require an upload if the checksum still matches. Missing files get null checksums and await deletion uploads.

The scan skips the folder if its mtime matches dirMod, stored in milliseconds. It compares individual file mtimes in seconds, matching Anki’s database format.

The scan excludes empty files, files over 100MiB, thumbs.db, .ds_store and invalid filenames. On macOS it accepts NFD filenames whose NFC equivalents are valid.

Check additions, modifications and deletions. Backdating mtimes lets the test trigger scans without waiting for the clock:

def _backdate(p):
    t = p.stat().st_mtime - 3
    os.utime(p, (t, t))

trk = Media(td/'trk'/'media', td/'trk'/'collection.mdb')
f1 = trk.folder/'file.jpg'
f1.write_bytes(b'hello')
trk._register_changes()
test_eq(trk.count(), 1)
test_eq(trk.entry('file.jpg'), (_sha1(b'hello'), _mtime_s(f1), 1))

trk.set_entry('file.jpg', _sha1(b'hello'), _mtime_s(f1), False)     # as if it synced
os.utime(f1); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.entry('file.jpg')[2], 0)                                # touched, content unchanged: still clean

f1.write_bytes(b'hello1'); _backdate(f1); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.entry('file.jpg'), (_sha1(b'hello1'), _mtime_s(f1), 1))

trk.set_entry('file.jpg', _sha1(b'hello1'), _mtime_s(f1), False)
(trk.folder/'Thumbs.db').write_bytes(b'x')                          # never syncs
(trk.folder/'empty.gif').touch()                                    # nor do empty files
f1.unlink(); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.count(), 0)
test_eq(trk.entry('file.jpg'), (None, 0, 1))                        # a deletion tombstone, awaiting upload
trk.pending(25)

[('file.jpg', None, 0)]

Zips on the wire

Media sync transfers uncompressed zip archives. Upload batches contain at most 25 entries. A batch stops adding files once it exceeds 2.5MiB.

Zip entries use numeric names. _meta maps those names to filenames:

• Uploads use [filename, zip_name] pairs. A null zip_name records a deletion.
• Downloads use a {zip_name: filename} mapping. The server reports deletions in the change list, not in the zip.

up = _zip_up([('a.jpg', b'aaa'), ('gone.mp3', None), ('b.png', b'bb')])
with zipfile.ZipFile(io.BytesIO(up)) as z:
    test_eq(json.loads(z.read('_meta')), [['a.jpg', '0'], ['gone.mp3', None], ['b.png', '2']])
    test_eq(z.read('0'), b'aaa')

down = io.BytesIO()
with zipfile.ZipFile(down, 'w', zipfile.ZIP_STORED) as z:      # what a server download looks like
    z.writestr('0', b'data0'); z.writestr('_meta', json.dumps({'0':'pic.jpg'}))
got = _unzip_down(down.getvalue())
test_eq(got, [('pic.jpg', b'data0')])
got

[('pic.jpg', b'data0')]

Files can change after the scan. _gather_zip reads their current contents and treats missing files as deletions. It removes entries that no longer meet the upload rules. The caller then rebuilds the batch:

gz = Media(td/'gz'/'media', td/'gz'/'collection.mdb')
for i in range(3): (gz.folder/f'big{i}.bin').write_bytes(bytes(1500_000))
(gz.folder/'small.txt').write_bytes(b'hi')
gz._register_changes()

got = gz._gather_zip([(f'big{i}.bin', _sha1(bytes(1500_000)), 0) for i in range(3)])
test_eq([f for f,_ in got], ['big0.bin', 'big1.bin'])               # 2.5MB cutoff: third file waits for the next batch

got = gz._gather_zip([('small.txt', _sha1(b'hi'), 0), ('vanished.png', 'deadbeef', 0), ('del.png', None, 0)])
test_eq(got, [('small.txt', b'hi'), ('vanished.png', None), ('del.png', None)])   # missing file -> uploaded as deletion

gz.set_entry('empty.bin', _sha1(b''), 1, True)                      # a zero-byte file can never upload
(gz.folder/'empty.bin').touch()
test_is(gz._gather_zip([('empty.bin', _sha1(b''), 1)]), None)       # pruned; batch must be rebuilt...
test_eq([f for f,_,_ in gz.pending(25)], ['big0.bin', 'big1.bin', 'big2.bin', 'small.txt'])   # ...without it

Talking to the server

SyncServer uses prefix='msync' for media requests instead of collection sync’s sync/. Both use a zstd body and the anki-sync header.

Media JSON replies have the form {"data": ..., "err": ""}. Change-list rows are [fname, usn, sha1] arrays. Upload replies are [processed, current_usn] arrays.

The media methods are begin, mediaChanges, downloadFiles, uploadChanges and mediaSanity. begin returns the server’s current media USN.

---

source

MediaSanityFailed

def MediaSanityFailed(
    *args, **kwargs
):

Common base class for all non-exit exceptions.

Deciding what to do with a server change

Each server change gives a filename and its current SHA-1 checksum. An empty checksum means the server deleted the file. required_change compares this with the local checksum and upload status:

• Download the server’s file if it’s missing locally or has different content.
• Keep a local file awaiting upload when the server reports a deletion.
• Apply other server deletions locally.
• Clear the upload flag when both files have the same content.

The tests cover Anki’s nine cases from changes.rs.

---

source

required_change

def required_change(
    local, remote, state
):

Resolve one server row: sha1s (’’ means deleted), state in none/clean/dirty -> None, ‘download’, ‘delete’, or ‘clean’

for args,exp in [(('','','none'),None),      (('','','clean'),'delete'),   (('','1','dirty'),'download'),
                 (('1','','dirty'),None),    (('1','','clean'),'delete'),  (('1','1','clean'),None),
                 (('1','1','dirty'),'clean'),(('a','b','dirty'),'download'),(('a','b','clean'),'download')]:
    test_eq(required_change(*args), exp)

Files in and out

Remote deletions move files to a sibling media.trash folder. You can restore them manually.

Some files on AnkiWeb have names from older clients that no longer meet the filename rules. _add_from_server saves them under corrected names. It records the old name as a pending deletion and the corrected file as a pending upload.

tf = Media(td/'tf'/'media', td/'tf'/'collection.mdb')
test_eq(tf._add_from_server('ok.png', b'fine'), [('ok.png', _sha1(b'fine'), _mtime_s(tf.folder/'ok.png'), False)])
bad = 'we|ird.png'                                                   # a name today's Anki would never create
test_eq(tf._add_from_server(bad, b'x'), [(bad, None, 0, True), ('weird.png', _sha1(b'x'), _mtime_s(tf.folder/'weird.png'), True)])
tf._trash_files(['ok.png', 'never-existed.png'])
test_eq((tf.folder/'ok.png').exists(), False)
test_eq((tf.folder.with_name('media.trash')/'ok.png').read_bytes(), b'fine')

The sync procedure

sync scans local files, downloads server changes, then uploads pending local changes. It gets the server’s media USN from begin unless the caller provides it from collection sync’s meta. After exchanging changes, it asks the server to compare file counts. A sync with no changes skips this check.

Each downloaded batch updates the database and lastUsn in one transaction. An interrupted sync resumes from the last recorded batch.

An upload advances lastUsn only when the server’s new USN equals the old lastUsn plus the number of entries it accepted. Otherwise, another client has uploaded concurrently. Keeping lastUsn unchanged lets the next sync fetch that client’s changes.

A failed count check clears local sync state and raises MediaSanityFailed. It leaves the files intact. The next sync compares all files with the server again.

---

source

Media.sync

def sync(
    srv, server_usn:NoneType=None
):

Sync the media folder with srv: pull server changes, push local ones, then verify counts

Media on a Collection

Use Collection.add_media to copy a file into the media folder. It returns the filename to use in a note field: <img src="name"> for pictures or [sound:name] for audio.

Collection.media() opens the media folder and database as a context manager. Collection.sync_media() syncs using the saved authentication.

---

source

Collection.sync_media

def sync_media(
    srv:NoneType=None
):

Media-sync with srv (default: the saved auth from a previous sync)

---

source

Collection.add_media

def add_media(
    file, fname:NoneType=None
):

Copy file (a path, or bytes with fname) into the media folder; returns the name to reference in note fields

---

source

Collection.media

def media():

This collection’s Media; close it after use (or use as a context manager)

col = Collection.open(td/'ex'/'collection.anki2')
name = col.add_media(b'not really a png', 'diagram.png')
n = col.add(Front=f'What does this show? <img src="{name}">', Back='the architecture')
test_eq(name, 'diagram.png')
test_eq((td/'ex'/'collection.media'/name).read_bytes(), b'not really a png')
with col.media() as cm:
    cm._register_changes()
    test_eq(cm.pending(25), [(name, _sha1(b'not really a png'), _mtime_s(td/'ex'/'collection.media'/name))])
col.close()
n

Front: What does this show? | Back: the architecture | 🏷 -

Against a real server

These examples use the anki wheel’s sync server on localhost, as in the syncer notebook. Anki is the second client. Its media sync runs in a backend thread. Poll media_sync_status until its active flag is false. The call raises an exception if sync fails.

The examples require a server process and have eval: false. tests/test_sync.py runs this sequence under pytest. It also covers batching and count-check failures.

import time

server, EP = start_sync_server(td/'server')

mcol = Collection.open(td/'sync'/'collection.anki2')
img = mcol.add_media(b'PNG our image', 'pic.png')
mcol.add(Front=f'What is this? <img src="{img}">', Back='our pic')
mcol.sync(user='tester', passw='s3kret', endpoint=EP, upload=True)
mcol.sync_media()
mcol.sync_media()      # caught up: a no-op

Check that Anki downloads the image and resolves its reference:

(td/'sync2').mkdir()
moc = AnkiCollection(str(td/'sync2'/'collection.anki2'))
auth = moc.sync_login('tester', 's3kret', endpoint=EP)
st = moc.sync_collection(auth, sync_media=False)
moc.close_for_full_sync()
moc.full_upload_or_download(auth=auth, server_usn=st.server_media_usn, upload=False)
moc.reopen(after_full_sync=True)
moc.sync_media(auth)
while moc.media_sync_status().active: time.sleep(0.05)
test_eq(Path(moc.media.dir(), 'pic.png').read_bytes(), b'PNG our image')
chk = moc.media.check()
test_eq((list(chk.missing), list(chk.unused)), ([], []))

Add an audio file and delete the picture in Anki. Sync Anki, then fastanki. Check that fastanki downloads the audio, moves the picture to media.trash, and has no pending uploads:

(td/'oracle.mp3').write_bytes(b'ORACLE AUDIO')
oname = moc.media.add_file(str(td/'oracle.mp3'))
moc.media.trash_files(['pic.png'])
moc.sync_media(auth)
while moc.media_sync_status().active: time.sleep(0.05)

mcol.sync_media()
with mcol.media() as m:
    test_eq((m.folder/oname).read_bytes(), b'ORACLE AUDIO')
    test_eq((m.folder/'pic.png').exists(), False)
    test_eq((m.folder.with_name('media.trash')/'pic.png').exists(), True)
    test_eq((m.pending(25), m.count()), ([], 1))
moc.close()
mcol.close()
server.terminate()