Project field notes
Migrating reading progress to Kavita
Two Python scripts that import per-issue reading progress into Kavita's SQLite database — one from a ComicRack XML library, one from a Codex server — without touching any other table.
tl;dr
https://github.com/DieselTech/Kavita-Database-Migration
Three scripts:
| Coming from | Script | Source file |
|---|---|---|
| ComicRack | ComicRack_Migration.py | ComicDb.xml |
| Codex | Codex_Migration.py | codex.sqlite |
| Kavita | Kavita_Migrate_Progress.py | kavita.db |
ComicRack and Codex scripts can be ran with --dry-run --report report.txt to get an idea what it will do. If you like
what you see, then remove --dry-run and let it rip. The Kavita to Kavita script just has --dry-run to test.
Why??
People have told me they wanted to move over to Kavita but didn’t want to lose their years if not decades of reading progress stored in ComicRack. Then the same thing came up for Codex, which is a much newer server but has the same “problem”.
ComicRack and Codex scripts write to exactly one Kavita table, AppUserProgresses, for exactly one named user.
Nothing else in the database is touched. The Kavita to Kavita script was made to let you start with a fresh copy of your database
without bringing forward years of heavy changes.
Supported reading servers
| ComicRack | Codex | |
|---|---|---|
| Storage | one XML file, ComicDb.xml | Django SQLite database, codex.sqlite |
| Users | single user | multi-user, plus anonymous sessions |
| File paths | not usable for matching | full path per comic, and Kavita has them too |
| Match strategy | fuzzy: series name + run identity + issue number | exact: relative folder path, then issue number |
That difference drives everything below. ComicRack forces a scoring contest since their “metadata” is weak. Codex migrations should be a lot better because it has proper metadata and even file paths we can look at.
Part 1 — ComicRack
ComicRack stores its library and reading state in a single XML file, ComicDb.xml.
The script reads ComicDb.xml, matches each book that has reading progress against a series and
chapter in Kavita, and writes rows into Kavita’s AppUserProgresses table for one named user.
Since ComicRack is a “single user” application, you just pick the Kavita user to land on.
The two data models
The relevant fields on each side:
ComicRack <Book> | Kavita |
|---|---|
Series — series name, no year | Series.Name — name with a (year) or (volume) suffix |
Volume — run ordinal (8), sometimes a year | Volume.Name — the run’s start year as a string |
Year / Month — the issue’s cover date | Series.NormalizedName — name + year, lowercased, alphanumeric |
Number — issue number, may be non-numeric | Chapter.MinNumber / MaxNumber (REAL), Chapter.Range (TEXT) |
CurrentPage / LastPageRead | AppUserProgresses.PagesRead |
Opened — .NET timestamp with offset | Created / LastModified (+ ...Utc), naive local time |
Year means different things on each side
Kavita’s NormalizedName embeds the year the run started: Avengers (2023) normalizes to
avengers2023. ComicRack’s Year is the cover date of the individual issue.
So Avengers #7, published in 2024 as part of the run that began in 2023, has Year=2024 in
ComicRack and lives under avengers2023 in Kavita.
Hahah you thought because it is called Issue Number that they would be int or float
Marvel loves to screw with people - 1.MU, 16.HU, 50.LR, 78.BEY. Because of this, we need to treat
“issue numbers” as strings. Kavita stores these in Chapter.Range as text and sets MinNumber to 0.0.
Any matcher that casts the issue number to a float drops them, and any matcher that trusts MinNumber
collides them with a genuine issue #0.
Matching
Name normalization
Series names are reduced to lowercase alphanumerics with the trailing discriminator split off, so
Avengers (2023) yields base avengers and discriminator 2023. A discriminator ≥ 1900 is treated
as a year; below that it is a run ordinal, which is how Kavita records series like
Black Panther (7).
A second lookup key is generated with a leading article stripped, so ComicRack’s
Punisher: War Zone reaches Kavita’s The Punisher: War Zone (1992).
Candidate gating
A base name like avengers resolves to a dozen Kavita runs. Every one of them is a candidate, and
each candidate is checked for whether it actually contains the issue number before it is scored.
A run that does not hold the issue cannot win on name similarity alone.
This gate is what stops the single worst failure mode. A Kavita library will often contain a stub
series — a bare Black Panther with one chapter in it, alongside the real
Black Panther (1998), (2005), (2016) and (2018). Name-only matching sends every Black
Panther book to the stub. Coverage gating spreads them across the correct runs.
Scoring
Surviving candidates are scored on how well the run identity lines up:
| Evidence | Score |
|---|---|
ComicRack Volume is a 4-digit year equal to the run’s year | 120 |
ComicRack Volume is an ordinal equal to the run’s ordinal | 110 |
| Issue cover year equals the run’s start year | 100 |
| Run started at or before the issue’s cover year | 60 − (gap, capped at 50) |
| Run started after the issue’s cover year | negative |
| Candidate has no discriminator at all (likely a stub) | −20 |
The “started at or before” rule is the one that recovers the bulk of the library: it accepts that an issue dated 2025 belongs to the run that began in 2023, and prefers the most recent run that could plausibly contain it.
Issue lookup
Text numbers are tried first, against Chapter.Range and Chapter.Number, because Kavita stores
them with MinNumber = 0.0 and a numeric-first lookup would mis-hit issue #0. Numeric lookup then
tries an exact match on the parsed Range, and falls back to any chapter whose
MinNumber ≤ n ≤ MaxNumber to catch collected or merged chapters.
Conflict arbitration
Matching is per book, so two different ComicRack runs can independently land on the same Kavita chapter. That is not hypothetical. In the test dataset:
- ComicRack
Punisher, volume 54, cover year 2009 →Punisher (2009)#1. Correct, score 100. - ComicRack
The Punisher, volume 6, cover year 2011 →Punisher (2009)#1. Wrong.
Writes are therefore deferred. All books are matched, claims are grouped by chapter, and any chapter claimed by two different ComicRack runs is resolved before anything is written:
- Same run claiming a chapter twice is a duplicate file. It merges — highest page count wins.
- Different runs claiming one chapter: the higher-scoring claim is written, the other is dropped and reported. A tie between distinct runs is unresolvable evidence, so both are dropped.
- Only the overlapping issues are dropped. The losing run keeps every issue it alone claims.
Skipping a book costs nothing. Writing progress to the wrong issue is silent bad data, and this is the only class of error the script cannot detect after the fact.
Usage
# Preview. Reads the database, writes nothing.
python ComicRack_Migration.py \
--comicrack-xml ComicDb.xml \
--kavita-db kavita.db \
--username $Name \
--dry-run
# Preview, saving the full list of what would be skipped.
python ComicRack_Migration.py --comicrack-xml ComicDb.xml --kavita-db kavita.db --username $Name --dry-run --report skipped.txt
# Apply.
python ComicRack_Migration.py --comicrack-xml ComicDb.xml --kavita-db kavita.db --username $Name
--username is the Kavita username, matched against AspNetUsers.UserName and case-sensitive. An
unknown name aborts and lists the valid ones. --verbose prints a line per book;
--report FILE writes the complete skip breakdown, grouped by reason, with every affected series.
Output
ComicRack -> Kavita | user: you | DRY RUN (no changes written)
source: ComicDb.xml
target: kavita.db
Read 3911 books with reading progress (4228 total in ComicRack)
Indexed 29103 series / 177718 chapters from Kavita
Would match 3578 of 3911 books (91.5%)
3573 new, 1 would raise existing progress, 4 already at or beyond this point
Skipped 333 books
120 series not in Kavita 35 series: PunisherMAX, Deadpool: Merc With a Mouth, ...
98 ambiguous run, not guessed 11 series: The Punisher, Punisher: War Zone, Daredevil, ...
101 no series/number in ComicRack
14 issue not in Kavita 3 series: Fray, Blood Hunt: Red Band, ...
Run with --report FILE for the full list of skipped series.
DRY RUN - nothing was written. Re-run without --dry-run to apply.
Part 2 — Codex
Codex is a Django app, so its database is a plain SQLite file,
codex.sqlite. Progress lives in codex_bookmark: one row per (user, comic) holding a 0-indexed
page and a finished flag.
The important difference is that Codex stores the real file path of every comic in codex_comic.path,
and Kavita stores the same thing in MangaFile.FilePath. If both servers point at the same library
tree, the path is the match. There is no name normalization, no run scoring and no arbitration
phase in this script, because there is nothing to guess at.
Lining up the two library roots
The two servers rarely mount the library at the same place. Codex might call it /comics, Kavita
/data/media/comics. Both databases declare their own roots — codex_library.path and Kavita’s
FolderPath.Path — so the script strips the declared root from each side and matches on what is left:
Codex /comics/Marvel/Avengers (2023)/Avengers 007.cbz
Kavita /data/media/comics/Marvel/Avengers (2023)/Avengers 007.cbz
→ both become Marvel/Avengers (2023)
Kavita’s chapters are indexed by that relative directory. A chapter can own more than one file (a
.cbr and a .cbz of the same issue), so the index is keyed on chapter and remembers the first
file’s page count for tie-breaking.
Two escape hatches for when the trees don’t line up that cleanly:
- Suffix fallback. If the relative directory doesn’t hit, the longest trailing fragment that identifies exactly one Kavita directory is used instead. A fragment matching two directories is treated as no match, not as a coin flip.
--path-map OLD=NEW. For libraries that were genuinely reorganised below the root, not just remounted. Repeatable, applied longest-prefix-first before root stripping.
Picking the issue inside the folder
Once the folder is anchored, the issue is resolved on Codex’s issue_number against Kavita’s parsed
Chapter.MinNumber:
- Exactly one chapter carries that number → done.
- Several do → narrow by page count, because a duplicate scan of the same issue is the usual cause.
If they all resolve to the same
ChapterIdit was never a real ambiguity, just two files. Genuinely different chapters with the same number and no page-count separation are skipped. - No issue number at all, or Kavita parsed the run’s numbering differently → a unique page count inside that directory is still solid evidence. Anything less is skipped.
Translating a bookmark into PagesRead
Codex counts from page 0 and tracks a finished flag; Kavita counts pages completed. The mapping is
not a straight copy:
- Finished → written as fully read using Kavita’s page count, not Codex’s. The two servers may hold different scans of the same issue.
- Unfinished → clamped to one page short of the end, so a partial read can never be mistaken for a completed one.
- Chapter Kavita hasn’t analyzed yet (0 pages known) → Codex’s count is used. Any positive value already reads as finished, and the row becomes exactly right after the next library scan.
- Opened but never read past page 1 → skipped entirely.
Which user?
Codex is multi-user and also keeps anonymous, session-scoped bookmarks. Guessing between two real users would silently import a stranger’s history, so:
- One user with progress -> used automatically.
- More than one and the script aborts and lists them with bookmark counts. Pass
--codex-user. - Only anonymous session bookmarks = aborts, because there is nothing attributable to migrate.
Usage
# Preview. Reads both databases, writes nothing.
python Codex_Migration.py \
--codex-db codex.sqlite \
--kavita-db kavita.db \
--username $Name \
--dry-run --report skipped.txt
# Pick one of several Codex users.
python Codex_Migration.py --codex-db codex.sqlite --kavita-db kavita.db --username $Name --codex-user "them@example.com"
# The tree below the root differs too, not just the mount point.
python Codex_Migration.py --codex-db codex.sqlite --kavita-db kavita.db --username $Name --path-map "/comics/Marvel=/comics/Publishers/Marvel"
# Apply.
python Codex_Migration.py --codex-db codex.sqlite --kavita-db kavita.db --username $Name
Output
Codex -> Kavita | you@example.com -> you | DRY RUN (no changes written)
source: codex.sqlite
target: kavita.db
Read 1415 bookmarks for Codex user you@example.com (1415 total)
Indexed 178960 chapters across 18250 folders from Kavita
Would match 1139 of 1415 bookmarks (80.5%)
992 new, 109 would raise existing progress, 38 already at or beyond this point
Note 508 matched comics have a different page count in Kavita than in Codex
(different scans of the same issue; Kavita's count was used)
Note 2 matched comics have not been analyzed by Kavita yet (0 pages known)
Codex's page count was used; run a Kavita library scan to correct them
Skipped 276 bookmarks
208 not in this Kavita library 50 series: X-Force, X-Men, New Mutants, ...
22 issue not in Kavita 12 series: Astonishing X-Men Infinity Comic, Avengers, ...
46 no progress recorded in Codex
Run with --report FILE for the full list of skipped series.
DRY RUN - nothing was written. Re-run without --dry-run to apply.
Note the shape of the skips: 208 of 276 are series the target Kavita library simply does not hold. Because the path anchor is exact, a Codex skip is almost always a statement about the library, not about the matcher.
What both scripts share
Stop Kavita before running either one. SQLite tolerates concurrent readers, but Kavita caches progress in memory and can write it back over the imported rows.
Existing progress is never lowered. A row is updated only when the imported page count exceeds the stored one, which makes re-running a no-op.
Dry runs simulate their own pending writes. Duplicate source entries resolve against each other exactly as they would in a real run, so the preview summary and the applied summary are byte-identical. The preview is a prediction, not an approximation.
One user per invocation. Progress is per-user in Kavita. Run once per username.
AppUserProgresses is denormalized — it carries VolumeId, SeriesId and LibraryId alongside
ChapterId. Both scripts derive those from the chapter’s own Series row in the same insert rather
than carrying them through Python, so the columns cannot drift from the chapter they describe.
Timestamps come from the source: ComicRack’s Opened field, Codex’s created_at / updated_at.
.NET writes seven fractional digits and a timezone offset; strptime accepts six and Kavita stores
naive local time, so the offset is stripped and the fraction truncated. Books with no readable date
fall back to the current time — visible in the data as a cluster of rows all stamped with the run
time.
Known limitations
ComicRack
Cover year colliding with a neighbouring run’s start year. An issue whose own cover date happens
to equal a different run’s debut year scores 100 against the wrong run. ComicRack Daredevil
volume 7 holds the 2023-dated issues of the run that began in 2022; every one of them scores a
perfect match against Daredevil (2023). Resolving this needs an order-preserving map from
ComicRack volume ordinals to Kavita run years, which the source data cannot support. In practice
these usually collide with the correct run’s own books and get caught by arbitration.
Legacy numbering. Marvel restarts issue numbering and periodically reverts to a cumulative count. Avengers #673 exists in ComicRack; the Kavita run that shipped it files it under its relaunch number. The coverage gate correctly refuses the run, then picks an older run that happens to contain a #673.
Codex
The libraries have to be the same tree. Path matching is the whole design. If Kavita’s library
was rebuilt with a different folder layout — renamed series folders, a flattened publisher level —
the suffix fallback will carry some of it and --path-map the rest, but a wholesale reorganisation
is out of scope.
Page counts drift between scans. Where Kavita and Codex hold different rips of the same issue, a partial read lands on a proportionally different page. Finished reads are always exact; unfinished ones are approximate by however much the two files differ.
Anonymous bookmarks are not migrated. Codex records progress for logged-out sessions. There is no user to attribute them to, so they are skipped.