docs: fix chunks index / memory usage internals

[ThomasWaldmann]

The internals docs still described the borg 1.x C HashIndex and a chunks index value of (refcount, size). Both are gone since the switch to borghash and the pack-based repository layout, so the "chunks index" / "memory usage" / "HashIndex" sections were misleading and the memory formulas were off by a lot.

chunks index

Document the real ChunkIndexEntry (flags, size, pack_id, obj_offset, obj_size), 32 + 48 == 80 bytes per entry, which is also the exact on-disk size per entry.

Also document that _store_chunkindex_fragment zeroes flags and size before writing a fragment, so a freshly built chunks index has size == 0 for all entries (both from the fragments and from the slow pack-header rebuild). The plaintext size is only filled in for the chunks the current borg run actually processes.

files cache

Document that entries are "compressed" (chunk ids reduced to 32bit chunks index indexes via k_to_idx) and msgpacked, instead of claiming they are python objects with a lot of overhead. Also note that the on-disk format stores the uncompressed chunks list, since the indexes are only valid for one specific in-memory chunks index.

memory usage

	old	new
chunks_index_usage	chunk_count * 40	chunk_count * 100
files_cache_usage	total_file_count * 240 + chunk_count * 165	total_file_count * 230 + chunk_count * 6

Measured rather than derived: filling a real ChunkIndex / files cache dict with 1Mi entries and reading RSS, on CPython 3.11 / 64bit.

• chunks index: 99.3 B/entry measured, matching the arithmetic (80 B payload, keys/values arrays over-allocated by up to 30%, plus 4 B per bucket at load factor 0.25 .. 0.5, so 88 .. 120 B).
• files cache: 228 B/entry at 1 chunk per file, rising to 309 B at 16 chunks per file; 230 + 6 * chunks fits that within ~5%. The per-chunk number dropped from 165 to 6 because the files cache now stores 32bit indexes instead of 256bit chunk ids.

Also dropped the mem_allocation = mem_usage / load_factor / mem_allocation_peak formulas: they predate borghash and now double-count, as the hash table overhead is already included in the per-entry numbers. Replaced by a short note on the resize peaks. Both worked examples were recomputed (1.9GiB / 0.28GiB, was 2.8GiB / 0.31GiB).

HashIndex

Rewritten for borghash: the three-array layout (uint32 bucket table + separate keys/values arrays), the stable keys-array index the files cache relies on, the real thresholds (grow 2x above load factor 0.5 including tombstones, shrink to 40% below 0.10, floor of 1000 buckets, kv arrays grown by 1.3x), and the BORGHASH + JSON metadata serialization format.

Removed the description of the removed C implementation: BORG_IDX / struct HashHeader, the 75%/25% thresholds, the 93% effective load factor, and the MAX_VALUE in-band signalling (values are no longer stored in the buckets, so there is no in-band signalling in the value anymore).

The internals_hashindex label is kept - repo_create_cmd.py references it for the none encryption DoS note, which still applies (linear probing on attacker-chosen key prefixes).

---

make html builds clean (the single warning, misc/asciinema/README.rst not in a toctree, is pre-existing and unrelated).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.49%. Comparing base (5878040) to head (e5ed8f6).
⚠️ Report is 43 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #9941   +/-   ##
=======================================
  Coverage   85.49%   85.49%           
=======================================
  Files          93       93           
  Lines       16122    16122           
  Branches     2466     2466           
=======================================
  Hits        13783    13783           
  Misses       1632     1632           
  Partials      707      707           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.