Release notes

What changed in each kage release.

The authoritative, commit-level history lives in CHANGELOG.md and on the releases page. This page summarises each version.

v0.3.12

The largest community release kage has had. Most of what is below arrived as pull requests from other people, and the rest came from bug reports that were specific enough to act on.

--resume actually resumes. state.json only ever persisted the visited set, so a resumed run found its seed already visited, queued nothing, printed pages 0 and exited successfully with most of the site still missing. The unfinished frontier is now saved next to the visited set, with each page's depth so --max-depth keeps its meaning across a restart, and a run reports what it is leaving behind. A page that failed is carried into the next run instead of being lost, which is the memory of what failed asked for in #36. --max-pages no longer throws away the pages it held back, so inspecting a site with -p 20 and then finishing it later works as a workflow.
Saved pages keep their doctype. kage serialises a rendered page as the outerHTML of <html>, and a doctype is a sibling of <html> rather than a child, so every page kage had ever written came out without one. A document with no doctype is quirks mode in every browser, which changes the box model and strips the <meta charset> declaration of its authority, and that is the encoding problem behind #16. The original doctype is now reproduced exactly, because the string itself selects the rendering mode, and a page that genuinely had none still gets none.
Saved pages declare their real encoding. Non-UTF-8 <meta charset> and Content-Type charset declarations are rewritten to UTF-8, matching the bytes kage writes to disk (#16). Rewriting covers the whole document, but whether the page declares an encoding is answered from <head> alone, since a charset meta Chrome left in <body> sits past the 1024 bytes a reader pre-scans and is no use to anyone.
--scroll scrolls the element that actually scrolls. It called window.scrollBy, which moves nothing on a site whose body is pinned to the viewport with the document inside an inner container, and that is how Feishu, Notion, Linear and most dashboards are built (#61). kage now picks the largest genuinely scrollable element, reads the position back after each step instead of assuming the step landed, and keeps going until both the position and the height stop changing. The whole scroll is bounded by half the render timeout, so a page that appends content forever cannot hold a worker.
Redirects resolve correctly, and no longer end the crawl. Relative links use the post-redirect URL and the document's first <base href>, and every consumed base href is removed so it cannot re-root the saved page, while a base target is preserved. A redirect that leaves the crawl scope used to take the page's links with it: a seed redirecting apex to www resolved every link onto a host the scope rejects, so nothing was enqueued and the run saved one page and stopped, while assets kept downloading and made it look complete. The resolution base now falls back to the document <base href> and then to the enqueued URL whenever the redirect target is out of scope.
Multi-page ZIM packs get a usable landing page. A mirror with no root index.html gets a bounded, title-sorted list of its pages instead of opening on an arbitrary one, and a single-page archive still opens directly on its article and keeps that article's title metadata (#62).
Crawl path controls use path boundaries. --exclude and --scope-prefix match a path and its descendants rather than any substring, so pass the full path prefix if you relied on the old behaviour. --max-pages is now documented accurately as a cap on queued page URLs rather than on renders, so a page that fails, that robots.txt disallows, or that is not HTML still spends its slot.
A headful crawl stops stealing focus. Render pages are created in the background, so driving kage through --control-url against a logged-in browser no longer pulls the Chrome window in front of whatever you are working in (#70).
--traversal is deprecated. The flag was never read and crawls have always been breadth-first. It is still accepted so existing scripts keep working, and it is planned for removal in the next minor release.
New documentation. CONTRIBUTING.md covers the development setup, the test commands, and what a focused pull request looks like, and a new robots.txt reference explains the kage agent token, Crawl-delay, and the advisory --no-robots override (#8).

Thanks

Special thanks to @SihanTeng, who wrote seven of the pull requests in this release: the redirect base resolution (#74), the synthetic ZIM landing page (#75), the crawl control path semantics (#76), the charset rewriting (#81), the --traversal deprecation (#82), the contributor guide (#83), and the robots.txt reference (#84). That work started as two large pull requests, and after a review asking for one concern each it came back as seven focused ones, which is the single most useful thing a contributor can do for a reviewer.

Thanks to @wangpi26 for the background render fix (#70), which is the kind of bug you only find by living with the tool.

Thanks to the people whose reports drove the rest of this release: @pepa65 for the UTF-8 and quirks mode report (#16), @j2l for the resume and long object chain report (#36), @edenzhong for the app-shell scrolling report (#61), @klerpi for the homepage-less ZIM report (#62), and @Xe for asking how site owners are meant to block this tool (#8).

Thanks also to everyone who has contributed to kage before this release: @Xirui, @zkd-11, @GautamKumarOffical, @Valid-Systems, and @AddictArts.

v0.3.11

go install ...@latest works again. The v0.3.9 antivirus fix replaced Rod's leakless dependency with a local stub. That kept the flagged helper out of kage.exe, but Go refuses versioned installation of a module containing a dependency-changing replace directive (#72). Windows now launches Chrome through a small platform-specific launcher that never imports leakless. Other platforms keep Rod's launcher, the Windows binary remains free of the flagged helper, and the module no longer needs replace.

v0.3.10

Go 1.26.5. The release updates kage's Go toolchain requirement to the latest patch release.

v0.3.9

A fix for the antivirus warning some Windows users hit when installing kage.

The Windows build no longer ships the leakless helper antivirus flags. kage renders pages with go-rod, whose launcher pulls in leakless, a small watchdog that force-kills Chrome if kage exits. leakless carries a prebuilt helper binary for every platform and links the Windows one straight into kage.exe. Windows Defender recognises that helper as Trojan:Win32/Kepavll!rfn and quarantines it, so a fresh scoop install failed with a virus warning on leakless.exe (#68). kage already launches Chrome with leakless switched off, so the helper never ran anyway. v0.3.9 replaced it with a stub that carries no embedded binary, which dropped about 1.28 MB from the Windows build and cleared the warning. Thanks to John Pywtorak for the report. The local replacement also broke versioned go install; the next release corrects that regression without restoring the helper (#72).

v0.3.4

Two community fixes: a clean stop for kage serve, and pages with heavy JavaScript that used to be dropped.

kage serve stops on Ctrl-C. The preview server was started with a blocking call that never watched for an interrupt, so stopping it meant killing the process. kage now shuts the server down gracefully on an interrupt or a SIGTERM, with a short timeout before forcing the listener closed. Thanks to Xirui Wang (#35) and Kaidi Zhao (#38).
Pages with deeply nested JavaScript still clone. Chrome's DevTools Protocol returns "Object reference chain is too long" while loading a page whose script builds a deeply nested object graph, but the page's HTML has already loaded and the error is only about Chrome's internal object tracking. kage now recognises that error and finishes rendering instead of dropping the page (#36). Thanks to Gautam Kumar (#39).

v0.3.3

A fix for Chrome saving a file to your Downloads folder mid-crawl.

A crawl never writes to your Downloads folder. A link with no file extension is queued as a page, so the page worker opened it in Chrome. When such a link served a binary, a zip or a CSV, Chrome saved the file to ~/Downloads, a surprise side effect of running a clone (#32). kage now denies Chrome-initiated downloads outright, since every asset is fetched through kage's own downloader and the browser never needs to write a file. As a second layer, kage detects a navigation whose response is not HTML and reroutes that URL to the asset downloader, where the size and media policy decides whether to localise it or leave it on the live web, instead of saving a broken page.

v0.3.2

A fix for garbled text on pages that did not carry a charset of their own.

Saved pages declare UTF-8. kage writes every page as UTF-8, but a site that set its charset only in the HTTP Content-Type header, with no <meta charset> in the markup, lost that signal once the page became a standalone file. A reader serving the bytes without a charset fell back to its locale encoding and turned every curly quote, dash, and non-breaking space into mojibake. kage now inserts a <meta charset="utf-8"> at the top of <head> when the page does not already declare one, so the page renders correctly in any reader.

v0.3.1

A fix for broken styling when a packed mirror's home page is a nested page.

/ redirects to the main page instead of serving it in place. A page's saved asset links are mirror-relative (../_kage/...), computed for that page's own location. The viewer was answering / with the main page's bytes directly, so the browser resolved those links against / and 404ed the page's CSS and images. A developer.apple.com/documentation mirror opened at / came up completely unstyled. kage now redirects / to the main page's canonical path, the way the archive's W/mainPage redirect does, so relative assets resolve correctly. Kiwix already followed that redirect, so it was never affected.

v0.3.0

Leaner mirrors, and a way to publish one as a dataset. A clone now keeps the assets that make a site readable offline and leaves the bulk downloads on the live web, and a packed archive converts to a columnar table that drops straight into dataset tooling.

Bulk downloads stay remote by default. Video and audio, installers and disk images (.dmg, .pkg, .exe, .msi, ...), archives, and PDFs are left pointing at their live URL rather than downloaded, because they are rarely needed to read a site offline yet routinely make up most of its bytes. On a developer.apple.com crawl that class was 18 of 19 GB. Page-rendering assets (images, fonts, CSS) are untouched. --keep-media restores the old behaviour, and --skip-ext .foo leaves more extensions remote.
Assets come only from the site's own domain by default. Localising is scoped to the seed's registrable domain, so developer.apple.com still pulls from www.apple.com and images.apple.com but not a separate brand domain or an off-topic third party (an embedded tracker, an unrelated CDN). --all-asset-hosts downloads from any host as before.
The size cap skips instead of truncating. An asset over --max-asset-mb was being saved as exactly the first N MB of itself, a corrupt fragment that would never play or run. kage now checks the response size and leaves an over-cap asset out of the mirror entirely, pointing at its live URL. On the apple crawl this was about a gigabyte of half-downloaded WWDC videos and .dmg installers.
kage parquet export and import. A packed ZIM converts to a flat Parquet table, one row per entry with clear columns (doc_id, url, host, crawl_date, mime, text, content), the shape a dataset host like Hugging Face expects and that DuckDB or pandas reads as is. The column names follow the open-index/open-markdown dataset, with doc_id a deterministic UUID v5 of the page URL, so a kage export sits alongside other web-crawl datasets. The conversion is lossless: a ZIM round-tripped through Parquet reproduces every entry, its metadata, and the main page byte for byte.
kage pack --incremental. Packing keeps a small cache sidecar next to the output and reuses the compression of any cluster whose bytes have not changed since the last pack. Compressing clusters with zstd is the dominant cost of packing a large mirror, so re-packing after a small change (a --refresh, a handful of edited pages) only compresses what actually changed. A reused cluster is byte-for-byte what a fresh compression produces, so the archive stays deterministic.
Identical pages are stored once. When a rendered page's bytes match a page already written, kage stores it as a hard link to the first copy instead of a second full file, collapsing the duplicate content a faceted site spawns when many ?q=…/?page=… URLs render the same page. The summary reports how many were deduped.
Cleaner progress counting. The live counter shows distinct URL paths as "pages" and the query-string permutations one path can spawn separately as "variants", so the number tracks the site's real size instead of being inflated by ?q=… URLs.

v0.2.1

Packed ZIM archives now carry the metadata Kiwix expects, so a mirror shows up in a ZIM reader's library with a title, a description, and an icon instead of as a blank entry.

Mandatory metadata is always written. Every archive now gets a Name and a Description (a line derived from the host when --description is not given), the two fields zimcheck flags as missing otherwise.
The favicon becomes the book icon. When the mirror has a usable icon (an apple-touch-icon.png, favicon.png, or a PNG-based favicon.ico), kage rescales it to a 48x48 PNG and stores it as Illustrator_48x48@1, which is the icon Kiwix shows for the archive in its library. A site with no usable icon is packed without one rather than with a broken image.

v0.2.0

Double-click apps, so a packed mirror opens like a real desktop app instead of a terminal program.

kage pack --app wraps the viewer in a double-click app with the site's favicon as its icon. The flag builds on the binary format, so it composes with --base (including a webview base) and --icon. On macOS that is a .app bundle; on Linux, with a Linux --base, an AppImage-style .AppDir that becomes a single .AppImage when appimagetool is installed. The icon is pulled from the mirror automatically, or set with --icon.
A GUI-subsystem Windows base ships in the release as kage_<version>_windows-gui_<arch>.zip. Pack a viewer onto it with --format binary --base and the resulting .exe opens with no console window behind it.
Smarter cross-platform packing. kage reads the base binary's executable header to detect its target OS, so a Windows viewer always gets a .exe name and the right run hint, regardless of how the base file is named.

v0.1.2

A security fix for how kage launches Chrome, clearer crawl errors, and a container image that actually runs.

Chrome keeps its sandbox on by default. Earlier versions launched Chrome with --no-sandbox on every run, which switched off the browser's main security boundary even on an ordinary desktop where the sandbox works fine (#10). The sandbox now stays on, and is dropped only where it genuinely cannot start: inside a container (detected from IN_DOCKER or /.dockerenv) or when running as root. Whenever it is dropped, kage says so on stderr, so the choice is never silent.
Transient asset failures retry. A download that hits a 403/429, a 5xx, or a network blip is retried with a short backoff, which recovers files that bot-protection rejects on the first request of a burst. Permanent failures like a 404 are not retried.
Clearer crawl errors. Each failure now logs a classified reason (HTTP 403 Forbidden, timed out, ...), the URL, and the page that referenced it, and the end-of-run summary lists what went wrong instead of printing only a count.
The container image runs. Chrome aborted in the image with chrome_crashpad_handler: --database is required, so the crash reporter is now disabled inside a container, and the kage user has a writable home (the mounted /out volume) so output, resume state, and Chrome's profile no longer fail with a permission error (#7).

v0.1.1

Packing, so a clone can travel as one file instead of a folder.

kage pack <mirror-dir> collapses a mirror into a single distributable file. --format zim (the default) writes an open ZIM archive, the same format Kiwix uses, so the file opens in any ZIM reader and not just kage. --format binary appends that archive to a copy of kage to make a self-contained executable that serves the site offline when run. Packing is deterministic, so the same mirror produces a byte-identical file.
kage open <file.zim> serves a packed ZIM back over a local HTTP server, the read side of kage pack --format zim.
An optional native-window viewer. Built with -tags webview, kage open and a packed binary show the site in a real window backed by the operating system's WebView instead of a browser tab. The default build stays pure Go and opens the browser, so the release pipeline is unchanged.
A pure-Go zim package that reads and writes the ZIM format: a fixed header, MIME and pointer lists, zstd or stored clusters, redirects, and a trailing MD5.

v0.1.0

The first release. kage clones a live website into a self-contained folder you can browse offline, with every script stripped out.

kage clone <url> renders each page in headless Chrome, strips all JavaScript, and localises CSS, images, and fonts to relative paths.
kage serve [dir] previews a cloned folder over a local file server.
Idempotent and resumable. Each page is keyed by the file it writes, so a page reached over http and https, or as /index.html versus /, is fetched once. Re-running resumes; --refresh re-renders in place; --force starts clean.
Polite by default. Honours robots.txt, seeds from sitemap.xml, scopes to the seed host, and runs three parallel worker tiers.
Packaged everywhere. Archives, .deb/.rpm/.apk, a multi-arch GHCR image with Chromium bundled, checksums, SBOMs, and a cosign signature.