[jeffwidman]

WIP - please ignore while this is in Draft.

Thanks!
-Jeff

What was the end-user or developer problem that led to this PR?

Bundler's compact-index fetcher recursively expands dependencies from every
version in a fetched gem's info file. That includes versions already excluded
by the Gemfile's direct requirements.

For example, if the Gemfile requires target < 2.0, dependencies introduced
only by target 2.0 cannot participate in a successful resolution. Fetching
their info files, then recursively fetching their dependencies, cannot help
satisfy the request. Long version histories can therefore pull unrelated
dependency branches into metadata preparation.

The important distinction is between fetching a gem's version information and
expanding every version's dependencies: Bundler still needs the former for
resolution, checksums, and available-version diagnostics. This PR skips only
the latter when a mandatory direct requirement rules out the version.

What is your fix for the problem, implemented in this PR?

Pass applicable direct requirements through the default RubyGems source and
fetcher to the compact-index backend. Keep every fetched version row, but
do not enqueue dependency names from a version that fails its direct requirement.

The implementation is 22 added lines and one removed line across five library
files. It is deliberately conservative:

• Only source-less direct declarations applying to every locked platform
  contribute requirements, with one eligible declaration per name.
• Any override disables pruning.
• Explicit source indexes are not pruned, preserving their existing source
  affinity and ambiguous-source handling.
• Locked versions and alternative parents' transitive requirements are not
  treated as hard constraints.
• A dependency branch is still fetched if another eligible version or another
  root needs it.
• The dependency-API and full-index fallbacks remain unchanged.

This works for exact pins and ranges such as < 2.0. It does not change version
selection, add a metadata cache, persist a new index, or change fetching
concurrency. Its added costs are a small map referencing existing requirement
objects, per-row requirement checks, and a temporary selection array. Those
costs are why CPU improvement depends on the workload rather than being assumed.

This branch is independent of #9952 and does not include its
exact-pin resolution shortcut.

Benchmarks

Actual isolated patch on current master

A synthetic HTTP registry on loopback serves 20 direct gems, each with versions
0.9 and 1.0 through 8.0. Versions 0.9 and 1.0 depend on one retained child;
each excluded version 2.0 through 8.0 adds a separate two-gem branch. The registry
has 320 gem names and 480 version rows. The input locks 40 names, with the roots
at 0.9; the Gemfile pins all roots to 1.0 and explicitly updates them.

Each sample uses a fresh Ruby process, a fresh application and compact-index
cache, and the same input lockfile. Seven alternating baseline/patched pairs
follow one discarded warmup per library. Timing covers Definition.build,
resolve_remotely!, and to_lock, excluding process startup and fixture
generation. CPU is the Ruby client process's CPU, not the registry process's CPU.
There is no simulated network delay.

Metric	Baseline	This patch
Median Ruby process CPU	0.644 s	0.274 s
Median elapsed time	0.531 s	0.260 s
HTTP requests per sample	321	41
/info/ requests per sample	320	40

Every sample produced the same lockfile, and the driver checked the exact
requested paths, not just the counts. All remaining requests returned HTTP 200.
The 280 skipped requests are the two-gem branches introduced only by the seven
excluded versions of each of the 20 roots.

As a no-pruning control, changing all direct requirements to >= 0.9 kept all
versions eligible. Seven more alternating pairs made exactly 321 HTTP requests
in both arms, again with identical lockfiles. Median CPU was 0.683 -> 0.694 s
and elapsed time 0.567 -> 0.577 s. That small increase is reported rather than
claiming the additional requirement checks are free; these short shared-host
samples do not establish a stable overhead percentage.

This deliberately branch-heavy fixture demonstrates the avoided work, not an
expected percentage for real repositories. The loopback server and host affect
elapsed time; it is not an Internet-latency benchmark.

Standalone reproduction for both synthetic comparisons

Save the following as metadata-pruning-benchmark.py. With unmodified and
patched checkouts at the same base revision and the desired Ruby on PATH:

python3 metadata-pruning-benchmark.py /path/to/baseline/lib /path/to/patched/lib --pairs 7
python3 metadata-pruning-benchmark.py /path/to/baseline/lib /path/to/patched/lib --pairs 7 --unrestricted

Both commands passed locally with the results above. The driver uses the
Python standard library, serves only synthetic metadata on loopback, and
removes its temporary application/cache directories when finished.

import argparse
import base64
from collections import Counter
import hashlib
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import os
from pathlib import Path
import statistics
import subprocess
import tempfile
import time


RUBY_DRIVER = r'''
require "bundler"
require "json"
require "digest"
Bundler.ui = Bundler::UI::Silent.new
cpu = Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID)
wall = Process.clock_gettime(Process::CLOCK_MONOTONIC)
definition = Bundler::Definition.build("Gemfile", "Gemfile.lock",
  gems: JSON.parse(ENV.fetch("BENCHMARK_TARGETS")))
definition.resolve_remotely!
lock = definition.to_lock
puts JSON.generate(
  cpu: Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID) - cpu,
  wall: Process.clock_gettime(Process::CLOCK_MONOTONIC) - wall,
  lock_sha256: Digest::SHA256.hexdigest(lock),
  bundler: Bundler::VERSION, rubygems: Gem::VERSION, ruby: RUBY_VERSION
)
'''


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("baseline_lib", type=Path)
    parser.add_argument("patched_lib", type=Path)
    parser.add_argument("--pairs", type=int, default=7)
    parser.add_argument("--unrestricted", action="store_true")
    args = parser.parse_args()
    if args.pairs < 1:
        parser.error("--pairs must be positive")
    libraries = {
        "baseline": args.baseline_lib.resolve(),
        "patched": args.patched_lib.resolve(),
    }
    roots = ["fixture{:02d}".format(i) for i in range(20)]
    specs = {}
    for root in roots:
        specs[root] = [
            ("0.9", {root + "_used": "= 1.0"}),
            ("1.0", {root + "_used": "= 1.0"}),
        ]
        specs[root + "_used"] = [("1.0", {})]
        for version in range(2, 9):
            old = root + "_old{}".format(version)
            specs[root].append(("{}.0".format(version), {old: "= 1.0"}))
            specs[old] = [("1.0", {old + "_child": "= 1.0"})]
            specs[old + "_child"] = [("1.0", {})]

    responses = {}
    for name, versions in specs.items():
        lines = []
        for version, dependencies in versions:
            checksum = hashlib.sha256((name + version).encode()).hexdigest()
            deps = ",".join(child + ":" + req for child, req in dependencies.items())
            lines.append(version + " " + deps + "|checksum:" + checksum)
        responses["/info/" + name] = ("---\n" + "\n".join(lines) + "\n").encode()
    responses["/versions"] = (
        "---\n" + "".join(
            "{} {} {}\n".format(
                name, ",".join(version for version, _ in versions),
                hashlib.md5(responses["/info/" + name]).hexdigest()
            ) for name, versions in specs.items()
        )
    ).encode()
    requests = []

    class Registry(BaseHTTPRequestHandler):
        def do_GET(self):
            requests.append(self.path)
            if self.path not in responses:
                self.send_error(404, "Unexpected benchmark endpoint")
                return
            data = responses[self.path]
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
            self.send_header("Content-Length", str(len(data)))
            self.send_header("ETag", '"' + hashlib.md5(data).hexdigest() + '"')
            digest = base64.b64encode(hashlib.sha256(data).digest()).decode()
            self.send_header("Repr-Digest", "sha-256=:" + digest + ":")
            self.end_headers()
            self.wfile.write(data)

        def log_message(self, *_args):
            pass

    class RegistryServer(HTTPServer):
        request_queue_size = 1024

    runs = []
    with RegistryServer(("127.0.0.1", 0), Registry) as server, tempfile.TemporaryDirectory() as temp:
        server.timeout = 0.05
        url = "http://127.0.0.1:{}".format(server.server_port)
        directory = Path(temp)
        for iteration in range(args.pairs + 1):
            order = list(libraries.items())
            if iteration % 2:
                order.reverse()
            for label, library in order:
                app = directory / "{}-{}".format(iteration, label)
                app.mkdir()
                (app / "Gemfile").write_text(
                    'source "{}"\n'.format(url) +
                    "".join('gem "{}", "{}"\n'.format(name, ">= 0.9" if args.unrestricted else "= 1.0") for name in roots)
                )
                locked = []
                for name in sorted(roots + [r + "_used" for r in roots]):
                    locked.append("    {} ({})\n".format(name, "0.9" if name in roots else "1.0"))
                    if name in roots:
                        locked.append("      {}_used (= 1.0)\n".format(name))
                (app / "Gemfile.lock").write_text(
                    "GEM\n  remote: {}/\n  specs:\n".format(url) + "".join(locked) +
                    "\nPLATFORMS\n  ruby\n\nDEPENDENCIES\n" +
                    "".join("  {} (= 0.9)\n".format(name) for name in roots)
                )
                env = {k: os.environ[k] for k in ("PATH", "GEM_HOME", "GEM_PATH") if k in os.environ}
                env.update({
                    "HOME": str(app), "BUNDLE_USER_HOME": str(app / "user"),
                    "BUNDLE_IGNORE_CONFIG": "true", "BUNDLE_RETRY": "0",
                    "BENCHMARK_TARGETS": json.dumps(roots),
                })
                requests.clear()
                with (app / "stdout").open("w+") as stdout, (app / "stderr").open("w+") as stderr:
                    process = subprocess.Popen(
                        ["ruby", "-I", str(library), "-e", RUBY_DRIVER],
                        cwd=app, env=env, stdout=stdout, stderr=stderr
                    )
                    try:
                        deadline = time.monotonic() + 120
                        while process.poll() is None:
                            if time.monotonic() > deadline:
                                raise TimeoutError("Ruby benchmark exceeded 120 seconds")
                            server.handle_request()
                        stdout.seek(0)
                        stderr.seek(0)
                        output, errors = stdout.read(), stderr.read()
                        if process.returncode:
                            raise RuntimeError("{} failed:\n{}\n{}".format(label, errors, output))
                        unknown = set(requests) - set(responses)
                        if unknown:
                            raise RuntimeError("Unexpected requests: {}".format(unknown))
                        result = json.loads(output)
                    finally:
                        if process.poll() is None:
                            process.terminate()
                            process.wait(timeout=10)
                result.update(
                    library=label, iteration=iteration, http_requests=len(requests),
                    info_requests=sum(path.startswith("/info/") for path in requests),
                )
                expected_names = set(specs) if label == "baseline" or args.unrestricted else set(roots + [r + "_used" for r in roots])
                expected_requests = Counter(["/versions"] + ["/info/" + name for name in expected_names])
                if Counter(requests) != expected_requests:
                    raise AssertionError("Unexpected request set for {}: {}".format(label, requests))
                if iteration:
                    runs.append(result)
                print(json.dumps(result), flush=True)
    if len({r["lock_sha256"] for r in runs}) != 1:
        raise AssertionError("Baseline and patched lockfiles differ")
    summary = {}
    for label in libraries:
        samples = [r for r in runs if r["library"] == label]
        summary[label] = {
            "cpu_median": statistics.median(r["cpu"] for r in samples),
            "wall_median": statistics.median(r["wall"] for r in samples),
            "http_requests": sorted({r["http_requests"] for r in samples}),
            "info_requests": sorted({r["info_requests"] for r in samples}),
        }
    print(json.dumps({
        "pairs": args.pairs, "unrestricted": args.unrestricted,
        "direct_dependencies": len(roots), "locked_names": 2 * len(roots),
        "registry_names": len(specs), "registry_rows": sum(len(v) for v in specs.values()),
        "identical_lockfiles": True, "summary": summary,
    }, indent=2))


if __name__ == "__main__":
    main()

Earlier Dependabot measurements

An earlier runtime prototype was also measured against an internal GitHub
repository with 463 direct dependencies and 663 unique locked gem names.
The same successful three-update sample was compared with and without
metadata pruning. Both arms already included the exact-pin shortcut and the
upstream gzip-verification-removal optimization; only metadata pruning differed.
These are incremental measurements, not this branch's current-master timings.

In the sequential pair:

Metric	Before metadata pruning	After metadata pruning
Actual RubyGems /info/ HTTP GETs	771	576
All proxy-visible HTTP requests	1,008	813
Sum of native-helper process CPU	114.91 s	108.19 s
CLI elapsed time	185.97 s	165.42 s

That is 195 fewer HTTP requests: 25.3% fewer RubyGems info requests and 19.3%
fewer requests overall. Other request counts were unchanged. These are counted
requests, not per-helper metadata lookups multiplied by the number of helpers.

CPU results were less consistent. A second pair for that repository showed
128.24 -> 127.16 seconds of native CPU, a 0.85% reduction rather than 5.85%.
A complete root update job for public dependabot/dependabot-core showed
220.38 -> 220.78 seconds: effectively unchanged, not a demonstrated CPU win.
The successful proposed dependency records and all proposed text/binary file
contents matched within the comparisons.

Those jobs used Ruby 4.0.5, Bundler 4.0.22, and RubyGems 4.0.14 on a shared
development host with live registry access. They were individual paired jobs,
not controlled statistical wall-time trials. Native-helper CPU is not total
updater/container CPU, and the large repository sample was not a complete
update job. The claim is avoided metadata work and HTTP requests where excluded
versions introduce otherwise-unneeded branches, not a universal speedup.

Validation

Base: a099e068a86c0a912d558b2c107c6c263cb5b4ff. Local validation uses Linux MRI
Ruby 4.0.5 and checked-out Bundler/RubyGems 4.1.0.beta2.

ruby bin/rspec spec/bundler/definition_spec.rb spec/bundler/fetcher_spec.rb spec/bundler/fetcher/compact_index_spec.rb spec/bundler/source/rubygems_spec.rb spec/install/gems/compact_index_spec.rb --format progress --seed 45123

• Unmodified baseline: 113 examples, 0 failures.
• With the patch: 125 examples, 0 failures.
• The 12 added examples cover exact/range pruning on fresh locks and updates,
  shared dependency branches, loose bounds, inactive and additional platforms,
  overrides, changed manifests, missing-version diagnostics, explicit sources,
  propagation, and preserving fetched rows and checksum metadata.

ruby bin/rubocop lib/bundler/definition.rb lib/bundler/source/rubygems.rb lib/bundler/fetcher.rb lib/bundler/fetcher/base.rb lib/bundler/fetcher/compact_index.rb spec/bundler/fetcher_spec.rb spec/bundler/fetcher/compact_index_spec.rb spec/install/gems/compact_index_spec.rb

Passed: eight files inspected, no offenses.

ruby bin/parallel_rspec -n 4 spec --seed 45123

Passed: 4,166 examples, 0 failures, 29 pending.

The network-dependent realworld suite, RubyGems test-unit suite, and the
Windows/JRuby/TruffleRuby matrix have not been run locally.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags