# Media


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

Media sync is a protocol of its own, independent of collection sync:
collection sync moves note and card rows, media sync moves files, and
neither orders the other. It has its own usn sequence and no full-sync
path, so every media sync is incremental. Local state is a media folder
next to the collection plus a small sqlite db with one row per file; the
folder is the truth, and a scan before each sync brings the db up to
date. Like the rest of fastanki this was built against Anki’s Rust code
(`rslib/src/sync/media/`) as the reference, and is tested against a real
Anki server below with Anki itself as the second client.

``` python
import tempfile
from fastcore.test import *
from anki.collection import Collection as AnkiCollection
from anki.utils import checksum
```

## Filenames

A media file’s name is its identity across devices, so Anki normalizes
names aggressively before they may sync: NFC form (APFS stores and
serves NFD), characters that break some platform stripped, Windows
device names defused, no trailing dot or space, 120 bytes at most. A
file whose name can’t survive this round trip is skipped by sync rather
than mangled.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L37"
target="_blank" style="float:right; font-size:smaller">source</a>

### norm_fname

``` python
def norm_fname(
    fname
):
```

*Normalize a media filename as Anki does: NFC, problem characters
stripped, device names defused, 120 bytes max*

The awkward cases are pinned by Anki’s own test suite (`files.rs`); we
also push names through the oracle’s `write_data`, which runs the
genuine Rust normalization, and check we picked the same name:

``` python
test_eq(norm_fname('foo.jpg'), 'foo.jpg')
test_eq(norm_fname('con.jpg[]><:"/?*^\\|\0\r\n'), 'con_.jpg')
test_eq(norm_fname('test.'), 'test._')
test_eq(norm_fname('test '), 'test _')
test_eq(norm_fname('x'*130+'.jpg'), 'x'*115+'.jpg')

td = Path(tempfile.mkdtemp())
ac = AnkiCollection(str(td/'oracle.anki2'))
names = ['ok.jpg', 'con.', 'foo bar.png', 'te\u0301st.gif', 'x'*130+'.webp']   # NFD input comes back NFC
for name in names: test_eq(ac.media.write_data(name, b'data'), norm_fname(name))
[(name, norm_fname(name)) for name in names[:4]]
```

    [('ok.jpg', 'ok.jpg'),
     ('con.', 'con_._'),
     ('foo bar.png', 'foo bar.png'),
     ('tést.gif', 'tést.gif')]

## Adding files

Adding a file whose name is taken must never silently clobber different
content, because after a sync every device would see the change. So an
add is content-aware: same name and same bytes is a no-op, while same
name and different bytes stores the data under `name-{sha1}.ext`
instead. Case-insensitive filesystems make `Foo.jpg` and `foo.jpg` the
same file, so when the name is uppercase and can’t be used as-is, Anki
retries with the lowercased name before falling back to the hash
rename - which means a fresh uppercase name is stored lowercased.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L56"
target="_blank" style="float:right; font-size:smaller">source</a>

### add_media_file

``` python
def add_media_file(
    folder, data, fname
):
```

*Write `data` into `folder` under (normalized) `fname` without
clobbering different content; returns the name used*

The oracle shows every branch, and we behave identically (note the
platform wrinkle in the middle: on a case-insensitive filesystem, an
existing `foo.jpg` satisfies the same-content check for `Foo.jpg`, so
the name comes back with its case preserved):

``` python
mf = td/'add'
cases = [('Foo.jpg', b'aaa'), ('Foo.jpg', b'aaa'), ('Foo.jpg', b'bbb'), ('dup.png', b'x'), ('dup.png', b'y')]
theirs = [ac.media.write_data(n, d) for n,d in cases]
ours = [add_media_file(mf, d, n) for n,d in cases]
test_eq(ours, theirs)                              # we make exactly the oracle's choices, branch by branch
test_eq(_sha1(b'aaa'), checksum(b'aaa'))           # and our checksums are Anki's: hex sha1
test_eq(theirs[0], 'foo.jpg')                      # fresh uppercase is stored lowercased
test_eq(theirs[2], f'foo-{_sha1(b"bbb")}.jpg')     # same name, different content: hash-renamed
test_eq(theirs[3:], ['dup.png', f'dup-{_sha1(b"y")}.png'])
ours
```

    ['foo.jpg',
     'Foo.jpg',
     'foo-5cb138284d431abd6a053a56625ec088bfb88912.jpg',
     'dup.png',
     'dup-95cb0bfd2977c761298d9624e4b4d4c72a39974a.png']

## The media database

Sync state lives in a sqlite db of two tiny tables: one row per file
(`csum` NULL means “deleted, deletion not yet uploaded”; `dirty` means
“changed since last sync”), and a `meta` row holding the media folder’s
last-seen mtime plus the last server usn we’ve caught up to. The layout
matches Anki’s own: for `collection.anki2`, the folder is
`collection.media` and the db `collection.mdb`, the same names the
desktop app would pick.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L82"
target="_blank" style="float:right; font-size:smaller">source</a>

### Media

``` python
def Media(
    folder, db_path
):
```

*A collection’s media folder plus the db tracking its sync state*

``` python
m = Media(td/'m'/'media', td/'m'/'collection.mdb')
test_eq(m.count(), 0)
test_is(m.entry('test.mp3'), None)
m.set_entry('test.mp3', None, 0, False)
test_eq(m.entry('test.mp3'), (None, 0, 0))
m.set_entry('test.mp3', _sha1(b'hello'), 123, True)
test_eq(m.pending(25), [('test.mp3', _sha1(b'hello'), 123)])
test_eq(m.count(), 1)
m.set_meta(123, 321)
test_eq(m.meta(), (123, 321))
m.force_resync()
test_eq((m.count(), m.meta()), (0, (0, 0)))
```

## Change tracking

The media folder is the truth; the db just caches what sync needs to
know about it. Before every sync, a scan reconciles the two: a file
that’s new or has a changed mtime is re-hashed and marked dirty if its
content really changed, and a db row whose file is gone becomes a
deletion tombstone. Two mtime tricks keep this cheap: the folder’s own
mtime (stored in `dirMod`, milliseconds) skips the whole scan when no
directory entry changed, and per-file mtimes (seconds, matching Anki’s
db format) decide what to re-hash. Some files can never sync, so the
scan ignores them: names that aren’t valid NFC (macOS lists names in
NFD; they’re treated as their NFC form when that form is valid),
`thumbs.db`/`.ds_store`, empty files, and files over 100MiB.

The scenario below is Anki’s own change-tracking test: add, modify,
delete, with a “touched but unchanged” case in the middle proving a bare
mtime bump doesn’t cause a pointless upload. Since scans compare against
the *stored* folder mtime, the test backdates mtimes rather than
sleeping through the clock’s resolution:

``` python
def _backdate(p):
    t = p.stat().st_mtime - 3
    os.utime(p, (t, t))

trk = Media(td/'trk'/'media', td/'trk'/'collection.mdb')
f1 = trk.folder/'file.jpg'
f1.write_bytes(b'hello')
trk._register_changes()
test_eq(trk.count(), 1)
test_eq(trk.entry('file.jpg'), (_sha1(b'hello'), _mtime_s(f1), 1))

trk.set_entry('file.jpg', _sha1(b'hello'), _mtime_s(f1), False)     # as if it synced
os.utime(f1); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.entry('file.jpg')[2], 0)                                # touched, content unchanged: still clean

f1.write_bytes(b'hello1'); _backdate(f1); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.entry('file.jpg'), (_sha1(b'hello1'), _mtime_s(f1), 1))

trk.set_entry('file.jpg', _sha1(b'hello1'), _mtime_s(f1), False)
(trk.folder/'Thumbs.db').write_bytes(b'x')                          # never syncs
(trk.folder/'empty.gif').touch()                                    # nor do empty files
f1.unlink(); _backdate(trk.folder)
trk._register_changes()
test_eq(trk.count(), 0)
test_eq(trk.entry('file.jpg'), (None, 0, 1))                        # a deletion tombstone, awaiting upload
trk.pending(25)
```

    [('file.jpg', None, 0)]

## Zips on the wire

Files travel in zip batches, stored uncompressed. Each zip holds at most
25 files and stops accepting more once ~2.5MB has accumulated. Members
are named `0`, `1`, … with a `_meta` JSON member mapping them to real
filenames - but the two directions differ: an upload’s `_meta` is a list
of `[filename, zip_name]` pairs where a null zip name means “this file
was deleted”, while a download’s is a plain `{zip_name: filename}` dict
(deletions never travel downward as zip entries; the change list already
said what to delete).

``` python
up = _zip_up([('a.jpg', b'aaa'), ('gone.mp3', None), ('b.png', b'bb')])
with zipfile.ZipFile(io.BytesIO(up)) as z:
    test_eq(json.loads(z.read('_meta')), [['a.jpg', '0'], ['gone.mp3', None], ['b.png', '2']])
    test_eq(z.read('0'), b'aaa')

down = io.BytesIO()
with zipfile.ZipFile(down, 'w', zipfile.ZIP_STORED) as z:      # what a server download looks like
    z.writestr('0', b'data0'); z.writestr('_meta', json.dumps({'0':'pic.jpg'}))
got = _unzip_down(down.getvalue())
test_eq(got, [('pic.jpg', b'data0')])
got
```

    [('pic.jpg', b'data0')]

`_gather_zip` reads the pending entries’ current bytes, honoring the
size cutoff, uploading a since-vanished file as a deletion, and pruning
entries that can never upload (the batch is then rebuilt from the db):

``` python
gz = Media(td/'gz'/'media', td/'gz'/'collection.mdb')
for i in range(3): (gz.folder/f'big{i}.bin').write_bytes(bytes(1500_000))
(gz.folder/'small.txt').write_bytes(b'hi')
gz._register_changes()

got = gz._gather_zip([(f'big{i}.bin', _sha1(bytes(1500_000)), 0) for i in range(3)])
test_eq([f for f,_ in got], ['big0.bin', 'big1.bin'])               # 2.5MB cutoff: third file waits for the next batch

got = gz._gather_zip([('small.txt', _sha1(b'hi'), 0), ('vanished.png', 'deadbeef', 0), ('del.png', None, 0)])
test_eq(got, [('small.txt', b'hi'), ('vanished.png', None), ('del.png', None)])   # missing file -> uploaded as deletion

gz.set_entry('empty.bin', _sha1(b''), 1, True)                      # a zero-byte file can never upload
(gz.folder/'empty.bin').touch()
test_is(gz._gather_zip([('empty.bin', _sha1(b''), 1)]), None)       # pruned; batch must be rebuilt...
test_eq([f for f,_,_ in gz.pending(25)], ['big0.bin', 'big1.bin', 'big2.bin', 'small.txt'])   # ...without it
```

## Talking to the server

Media endpoints live under `msync/` beside the collection sync’s
`sync/`, and use the very same envelope (zstd body, `anki-sync` header),
so
[`SyncServer`](https://AnswerDotAI.github.io/fastanki/syncer.html#syncserver)
handles them via its `prefix` argument. Two quirks are media-specific:
JSON replies arrive wrapped in a legacy `{"data": ..., "err": ""}`
envelope, and some payloads are JSON *arrays* rather than objects
(Rust’s `serde_tuple`): each change-list row is `[fname, usn, sha1]`,
and an upload reply is `[processed, current_usn]`. The five methods are
`begin` (returns the server’s current media usn), `mediaChanges`,
`downloadFiles`, `uploadChanges`, and `mediaSanity`.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L192"
target="_blank" style="float:right; font-size:smaller">source</a>

### MediaSanityFailed

``` python
def MediaSanityFailed(
    *args, **kwargs
):
```

*Common base class for all non-exit exceptions.*

## Deciding what to do with a server change

Each change-list row names a file and the sha1 it now has on the server
(empty means deleted there). Against our db entry - its checksum, and
whether it’s pending upload - there are exactly nine cases, and Anki’s
`changes.rs` resolves them with three rules: a genuine content conflict
favors the server (the loser’s bytes are already gone from the server’s
history either way); a local pending *addition* survives a remote
deletion (it’ll upload next); and matching sides just tidy the dirty
flag. We keep the decision as a pure function of
`(local sha1, remote sha1, state)`, with Anki’s own test table pinning
all nine cases below.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L202"
target="_blank" style="float:right; font-size:smaller">source</a>

### required_change

``` python
def required_change(
    local, remote, state
):
```

*Resolve one server row: sha1s (’’ means deleted), `state` in
none/clean/dirty -\> None, ‘download’, ‘delete’, or ‘clean’*

``` python
for args,exp in [(('','','none'),None),      (('','','clean'),'delete'),   (('','1','dirty'),'download'),
                 (('1','','dirty'),None),    (('1','','clean'),'delete'),  (('1','1','clean'),None),
                 (('1','1','dirty'),'clean'),(('a','b','dirty'),'download'),(('a','b','clean'),'download')]:
    test_eq(required_change(*args), exp)
```

## Files in and out

Applying a remote deletion moves the file to a `media.trash` folder
beside the media folder rather than unlinking it, exactly as Anki does -
a sync bug then costs a manual restore, not data. Incoming files
normally write straight to the folder, but AnkiWeb still holds
pre-normalization filenames from old clients; such a file is stored
under a corrected name, and two dirty entries record the fixup for the
next upload: a tombstone for the server’s bad name, and the renamed file
itself.

``` python
tf = Media(td/'tf'/'media', td/'tf'/'collection.mdb')
test_eq(tf._add_from_server('ok.png', b'fine'), [('ok.png', _sha1(b'fine'), _mtime_s(tf.folder/'ok.png'), False)])
bad = 'we|ird.png'                                                   # a name today's Anki would never create
test_eq(tf._add_from_server(bad, b'x'), [(bad, None, 0, True), ('weird.png', _sha1(b'x'), _mtime_s(tf.folder/'weird.png'), True)])
tf._trash_files(['ok.png', 'never-existed.png'])
test_eq((tf.folder/'ok.png').exists(), False)
test_eq((tf.folder.with_name('media.trash')/'ok.png').read_bytes(), b'fine')
```

## The sync procedure

A media sync is: scan the folder, learn the server’s usn (`begin`,
unless the caller already has it from the collection sync’s `meta`),
pull whatever the server has that we haven’t seen, push whatever’s
dirty, and - only if anything actually moved - ask the server to compare
file counts. Pulling loops over change batches, applying each in one db
transaction that also advances `lastUsn`, so an interrupted sync resumes
where it stopped. Pushing has one subtlety: after an upload the server
reports its new usn, and we may adopt it as “caught up” only when it’s
exactly our old usn plus the files we sent - anything else means another
client uploaded concurrently, and we must leave `lastUsn` alone so the
next pull fetches their changes. A failed count check clears the local
db and errors, making the next sync re-list everything against the
server - Anki’s own recovery path.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L246"
target="_blank" style="float:right; font-size:smaller">source</a>

### Media.sync

``` python
def sync(
    srv, server_usn:NoneType=None
):
```

*Sync the media folder with `srv`: pull server changes, push local ones,
then verify counts*

## Media on a Collection

[`Collection`](https://AnswerDotAI.github.io/fastanki/collection.html#collection)
gets three conveniences: `media()` opens the collection’s own media
folder and db (a context manager, like
[`Collection`](https://AnswerDotAI.github.io/fastanki/collection.html#collection)
itself),
[`add_media`](https://AnswerDotAI.github.io/fastanki/core.html#add_media)
copies data in and returns the filename to cite in a note field -
`<img src="name">` for pictures, `[sound:name]` for audio - and
`sync_media` runs a media sync with the saved auth. With this, the one
safety rule from the days before media support (“never reference a file
we can’t upload”) is discharged: referencing what
[`add_media`](https://AnswerDotAI.github.io/fastanki/core.html#add_media)
returns is always syncable.

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L317"
target="_blank" style="float:right; font-size:smaller">source</a>

### Collection.sync_media

``` python
def sync_media(
    srv:NoneType=None
):
```

*Media-sync with `srv` (default: the saved auth from a previous
[`sync`](https://AnswerDotAI.github.io/fastanki/core.html#sync))*

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L310"
target="_blank" style="float:right; font-size:smaller">source</a>

### Collection.add_media

``` python
def add_media(
    file, fname:NoneType=None
):
```

*Copy `file` (a path, or bytes with `fname`) into the media folder;
returns the name to reference in note fields*

------------------------------------------------------------------------

<a
href="https://github.com/AnswerDotAI/fastanki/blob/main/fastanki/media.py#L305"
target="_blank" style="float:right; font-size:smaller">source</a>

### Collection.media

``` python
def media():
```

*This collection’s
[`Media`](https://AnswerDotAI.github.io/fastanki/media.html#media);
close it after use (or use as a context manager)*

``` python
col = Collection.open(td/'ex'/'collection.anki2')
name = col.add_media(b'not really a png', 'diagram.png')
n = col.add(Front=f'What does this show? <img src="{name}">', Back='the architecture')
test_eq(name, 'diagram.png')
test_eq((td/'ex'/'collection.media'/name).read_bytes(), b'not really a png')
with col.media() as cm:
    cm._register_changes()
    test_eq(cm.pending(25), [(name, _sha1(b'not really a png'), _mtime_s(td/'ex'/'collection.media'/name))])
col.close()
n
```

<div class="prose" data-markdown="1">

**Front**: What does this show? <img src="diagram.png"> | **Back**: the
architecture | 🏷 -

</div>

## Against a real server

The same arrangement as the syncer notebook: the `anki` wheel’s genuine
sync server on localhost, with Anki itself playing second client - its
media sync runs in a backend thread, so we poll `media_sync_status`
(which also rethrows any error). These cells are `eval: false` because
they need the server process; `tests/test_sync.py` runs this sequence,
plus batching and count-check-failure cases, under pytest.

``` python
import time
```

``` python
server, EP = start_sync_server(td/'server')
```

``` python
mcol = Collection.open(td/'sync'/'collection.anki2')
img = mcol.add_media(b'PNG our image', 'pic.png')
mcol.add(Front=f'What is this? <img src="{img}">', Back='our pic')
mcol.sync(user='tester', passw='s3kret', endpoint=EP, upload=True)
mcol.sync_media()
mcol.sync_media()      # caught up: a no-op
```

Anki pulls the collection and then the media, and both sides of the
story check out: the bytes match, and its media check finds every
reference resolved:

``` python
(td/'sync2').mkdir()
moc = AnkiCollection(str(td/'sync2'/'collection.anki2'))
auth = moc.sync_login('tester', 's3kret', endpoint=EP)
st = moc.sync_collection(auth, sync_media=False)
moc.close_for_full_sync()
moc.full_upload_or_download(auth=auth, server_usn=st.server_media_usn, upload=False)
moc.reopen(after_full_sync=True)
moc.sync_media(auth)
while moc.media_sync_status().active: time.sleep(0.05)
test_eq(Path(moc.media.dir(), 'pic.png').read_bytes(), b'PNG our image')
chk = moc.media.check()
test_eq((list(chk.missing), list(chk.unused)), ([], []))
```

Then the reverse direction: Anki adds a file and deletes ours, and one
`sync_media` brings us both changes - their audio arrives, and our
deleted picture moves to `media.trash`, leaving a clean sync state:

``` python
(td/'oracle.mp3').write_bytes(b'ORACLE AUDIO')
oname = moc.media.add_file(str(td/'oracle.mp3'))
moc.media.trash_files(['pic.png'])
moc.sync_media(auth)
while moc.media_sync_status().active: time.sleep(0.05)

mcol.sync_media()
with mcol.media() as m:
    test_eq((m.folder/oname).read_bytes(), b'ORACLE AUDIO')
    test_eq((m.folder/'pic.png').exists(), False)
    test_eq((m.folder.with_name('media.trash')/'pic.png').exists(), True)
    test_eq((m.pending(25), m.count()), ([], 1))
moc.close()
mcol.close()
server.terminate()
```