From fbfeacac5b3f27ba210aa70fcbfff4b6246fa990 Mon Sep 17 00:00:00 2001 From: Blake Owens Date: Tue, 25 Aug 2026 16:55:26 -0500 Subject: [PATCH 01/24] feat(dedupe): let a caller supply the candidate scope and the winner order Deduplication derives its own candidate scope: a finding matches within its own product, narrowed to a single engagement by the engagement checkbox, and that is the only scope an installation can get. This makes both the scope and the preference order injectable, so a plugin can express a different one, while leaving every existing call site on exactly the behaviour it had. build_candidate_scope_queryset gains candidate_qs. When supplied it replaces the scope derivation; the loading strategy (defer, select_related, prefetch_related) is still applied here, so a caller decides which findings are candidates while the engine keeps deciding how to load them. Candidate confirmation walks locations, vulnerability ids and CWEs per candidate, so a scope handed in without those prefetches would silently turn one query into thousands. The four match generators gain ordering_key: a plain sort key over candidates that only changes which of several valid candidates is preferred. _is_candidate_older still runs afterwards, so an ordering key cannot make a newer finding win, and the global antisymmetry concurrent batches depend on is unaffected. uid_or_hash applies it to the merged candidate set, since merging two buckets loses their query order. Both kwargs thread through find_candidates_for_deduplication_*, match_batch_*, _dedupe_batch_* and their dispatchers. dedupe_batch_of_findings forwards them to a custom deduplication method only when set, so a plugin that predates them sees exactly the arguments it saw before. false_positive_history gains the same seam as scope_filter, since it builds its own queryset per algorithm from filter kwargs rather than filtering a supplied one. Every kwarg defaults to None and every default path is the previous code, so an open-source install is unaffected. The pairwise engagement guard is deliberately untouched: it stays correct for installs that rely on it, and a caller supplying a scope owns expressing its own isolation. --- dojo/finding/deduplication.py | 174 ++++++++++------ unittests/test_dedupe_injectable_scope.py | 229 ++++++++++++++++++++++ 2 files changed, 342 insertions(+), 61 deletions(-) create mode 100644 unittests/test_dedupe_injectable_scope.py diff --git a/dojo/finding/deduplication.py b/dojo/finding/deduplication.py index 1e8c7c0a615..3d8f48bea54 100644 --- a/dojo/finding/deduplication.py +++ b/dojo/finding/deduplication.py @@ -321,7 +321,7 @@ def are_locations_duplicates(new_finding, to_duplicate_finding): return False -def build_candidate_scope_queryset(test, mode="deduplication", service=None): +def build_candidate_scope_queryset(test, mode="deduplication", service=None, candidate_qs=None): """ Build a queryset for candidate finding. @@ -329,9 +329,24 @@ def build_candidate_scope_queryset(test, mode="deduplication", service=None): test: The test to scope from mode: "deduplication" (can match across tests) or "reimport" (same test only) service: Optional service filter (for deduplication mode, not used for reimport since service is in hash) + candidate_qs: Optional Finding queryset supplying the candidate scope. When given it + replaces the scope derivation below. + + A supplied ``candidate_qs`` must already express scope and engagement isolation: this + function adds neither to it. It does still apply the loading strategy (defer, + select_related, prefetch_related), so a caller decides *which* findings are candidates + while the engine keeps deciding how to load them. That split matters because the match + step walks locations, vulnerability ids and CWEs per candidate, so a scope handed in + without those prefetches would turn one query into thousands. + + The pairwise ``is_deduplication_on_engagement_mismatch`` guard still runs at match time + and is not disabled by passing a scope. A supplied scope that leaves isolated + engagements in it therefore gets those candidates rejected at match time instead. """ - if mode == "reimport": + if candidate_qs is not None: + queryset = candidate_qs + elif mode == "reimport": # For reimport, only filter by test. Service filtering is not needed because # service is included in hash_code calculation (HASH_CODE_FIELDS_ALWAYS = ["service"]), # so matching by hash_code automatically ensures correct service match. @@ -378,7 +393,7 @@ def build_candidate_scope_queryset(test, mode="deduplication", service=None): ) -def find_candidates_for_deduplication_hash(test, findings, mode="deduplication", service=None): +def find_candidates_for_deduplication_hash(test, findings, mode="deduplication", service=None, *, candidate_qs=None): """ Find candidates by hash_code. Works for both deduplication and reimport. @@ -389,7 +404,7 @@ def find_candidates_for_deduplication_hash(test, findings, mode="deduplication", service: Optional service filter (for deduplication mode, not used for reimport since service is in hash) """ - base_queryset = build_candidate_scope_queryset(test, mode=mode, service=service) + base_queryset = build_candidate_scope_queryset(test, mode=mode, service=service, candidate_qs=candidate_qs) hash_codes = {f.hash_code for f in findings if getattr(f, "hash_code", None) is not None} if not hash_codes: return {} @@ -408,7 +423,7 @@ def find_candidates_for_deduplication_hash(test, findings, mode="deduplication", return existing_by_hash -def find_candidates_for_deduplication_unique_id(test, findings, mode="deduplication", service=None): +def find_candidates_for_deduplication_unique_id(test, findings, mode="deduplication", service=None, *, candidate_qs=None): """ Find candidates by unique_id_from_tool. Works for both deduplication and reimport. @@ -419,7 +434,7 @@ def find_candidates_for_deduplication_unique_id(test, findings, mode="deduplicat service: Optional service filter (for deduplication mode, not used for reimport since service is in hash) """ - base_queryset = build_candidate_scope_queryset(test, mode=mode, service=service) + base_queryset = build_candidate_scope_queryset(test, mode=mode, service=service, candidate_qs=candidate_qs) unique_ids = {f.unique_id_from_tool for f in findings if getattr(f, "unique_id_from_tool", None) is not None} if not unique_ids: return {} @@ -439,7 +454,7 @@ def find_candidates_for_deduplication_unique_id(test, findings, mode="deduplicat return existing_by_uid -def find_candidates_for_deduplication_uid_or_hash(test, findings, mode="deduplication", service=None): +def find_candidates_for_deduplication_uid_or_hash(test, findings, mode="deduplication", service=None, *, candidate_qs=None): """ Find candidates by unique_id_from_tool or hash_code. Works for both deduplication and reimport. @@ -450,7 +465,7 @@ def find_candidates_for_deduplication_uid_or_hash(test, findings, mode="deduplic service: Optional service filter (for deduplication mode, not used for reimport since service is in hash) """ - base_queryset = build_candidate_scope_queryset(test, mode=mode, service=service) + base_queryset = build_candidate_scope_queryset(test, mode=mode, service=service, candidate_qs=candidate_qs) hash_codes = {f.hash_code for f in findings if getattr(f, "hash_code", None) is not None} unique_ids = {f.unique_id_from_tool for f in findings if getattr(f, "unique_id_from_tool", None) is not None} if not hash_codes and not unique_ids: @@ -484,8 +499,8 @@ def find_candidates_for_deduplication_uid_or_hash(test, findings, mode="deduplic return existing_by_uid, existing_by_hash -def find_candidates_for_deduplication_legacy(test, findings): - base_queryset = build_candidate_scope_queryset(test, mode="deduplication") +def find_candidates_for_deduplication_legacy(test, findings, *, candidate_qs=None): + base_queryset = build_candidate_scope_queryset(test, mode="deduplication", candidate_qs=candidate_qs) titles = {f.title for f in findings if getattr(f, "title", None)} cwes = {f.cwe for f in findings if getattr(f, "cwe", 0)} cwes.discard(0) @@ -508,7 +523,7 @@ def find_candidates_for_deduplication_legacy(test, findings): # TODO: should we align this with deduplication? -def find_candidates_for_reimport_legacy(test, findings, service=None): +def find_candidates_for_reimport_legacy(test, findings, service=None, *, candidate_qs=None): """ Find all existing findings in the test that match any of the given findings by title and severity. Used for batch reimport to avoid 1+N query problem. @@ -517,7 +532,7 @@ def find_candidates_for_reimport_legacy(test, findings, service=None): than legacy deduplication (title+severity vs title+CWE). Note: service parameter is kept for backward compatibility but not used since service is in hash_code. """ - base_queryset = build_candidate_scope_queryset(test, mode="reimport", service=None) + base_queryset = build_candidate_scope_queryset(test, mode="reimport", service=None, candidate_qs=candidate_qs) # Collect all unique title/severity combinations title_severity_pairs = set() @@ -600,10 +615,29 @@ def _is_candidate_older(new_finding, candidate): return is_older -def get_matches_from_hash_candidates(new_finding, candidates_by_hash) -> Iterator[Finding]: +def _preferred_candidate_order(candidates, ordering_key): + """ + Order the candidates a finding could deduplicate against, best first. + + Default (``ordering_key`` is None) keeps the order the caller built: every candidate + query is ``order_by("id")``, so the oldest finding wins, which is what deduplication has + always done. A supplied ``ordering_key`` is a plain sort key over candidates and only + changes which of several valid candidates is preferred. + + It cannot make a newer finding win: ``_is_candidate_older`` is still evaluated per + candidate afterwards, so the ordering chooses among candidates that are already legal + originals. Keeping those two separate is what stops a placement preference from breaking + the global antisymmetry that concurrent dedupe batches depend on. + """ + if ordering_key is None: + return candidates + return sorted(candidates, key=ordering_key) + + +def get_matches_from_hash_candidates(new_finding, candidates_by_hash, *, ordering_key=None) -> Iterator[Finding]: if new_finding.hash_code is None: return - possible_matches = candidates_by_hash.get(new_finding.hash_code, []) + possible_matches = _preferred_candidate_order(candidates_by_hash.get(new_finding.hash_code, []), ordering_key) deduplicationLogger.debug(f"Finding {new_finding.id}: Found {len(possible_matches)} findings with same hash_code, ids={[(c.id, c.hash_code) for c in possible_matches]}") for candidate in possible_matches: @@ -616,11 +650,11 @@ def get_matches_from_hash_candidates(new_finding, candidates_by_hash) -> Iterato yield candidate -def get_matches_from_unique_id_candidates(new_finding, candidates_by_uid) -> Iterator[Finding]: +def get_matches_from_unique_id_candidates(new_finding, candidates_by_uid, *, ordering_key=None) -> Iterator[Finding]: if new_finding.unique_id_from_tool is None: return - possible_matches = candidates_by_uid.get(new_finding.unique_id_from_tool, []) + possible_matches = _preferred_candidate_order(candidates_by_uid.get(new_finding.unique_id_from_tool, []), ordering_key) deduplicationLogger.debug(f"Finding {new_finding.id}: Found {len(possible_matches)} findings with same unique_id_from_tool, ids={[(c.id, c.unique_id_from_tool) for c in possible_matches]}") for candidate in possible_matches: if not _is_candidate_older(new_finding, candidate): @@ -632,7 +666,7 @@ def get_matches_from_unique_id_candidates(new_finding, candidates_by_uid) -> Ite yield candidate -def get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, candidates_by_hash) -> Iterator[Finding]: +def get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, candidates_by_hash, *, ordering_key=None) -> Iterator[Finding]: # Combine UID and hash candidates and walk oldest-first uid_list = candidates_by_uid.get(new_finding.unique_id_from_tool, []) if new_finding.unique_id_from_tool is not None else [] hash_list = candidates_by_hash.get(new_finding.hash_code, []) if new_finding.hash_code is not None else [] @@ -641,8 +675,10 @@ def get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, cand for c in hash_list: combined_by_id.setdefault(c.id, c) deduplicationLogger.debug("Finding %s: UID_OR_HASH: combined candidate ids (sorted)=%s", new_finding.id, sorted(combined_by_id.keys())) - for candidate_id in sorted(combined_by_id.keys()): - candidate = combined_by_id[candidate_id] + # Merging two buckets loses their query order, so this walk re-establishes it by id -- + # the same oldest-first rule the other algorithms get from order_by("id"). + combined = [combined_by_id[candidate_id] for candidate_id in sorted(combined_by_id.keys())] + for candidate in _preferred_candidate_order(combined, ordering_key): if not _is_candidate_older(new_finding, candidate): continue if is_deduplication_on_engagement_mismatch(new_finding, candidate): @@ -655,7 +691,7 @@ def get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, cand deduplicationLogger.debug("UID_OR_HASH: locations mismatch, skipping candidate %s", candidate.id) -def get_matches_from_legacy_candidates(new_finding, candidates_by_title, candidates_by_cwe) -> Iterator[Finding]: +def get_matches_from_legacy_candidates(new_finding, candidates_by_title, candidates_by_cwe, *, ordering_key=None) -> Iterator[Finding]: # --------------------------------------------------------- # 1) Collects all the findings that have the same: # (title and static_finding and dynamic_finding) @@ -669,7 +705,7 @@ def get_matches_from_legacy_candidates(new_finding, candidates_by_title, candida if getattr(new_finding, "cwe", 0): candidates.extend(candidates_by_cwe.get(new_finding.cwe, [])) - for candidate in candidates: + for candidate in _preferred_candidate_order(candidates, ordering_key): if not _is_candidate_older(new_finding, candidate): continue if is_deduplication_on_engagement_mismatch(new_finding, candidate): @@ -826,73 +862,73 @@ def _drop_links_to_deleted_originals(modified_new_findings): # --------------------------------------------------------------------------- -def match_batch_hash_code(findings): +def match_batch_hash_code(findings, *, candidate_qs=None, ordering_key=None): """Find dedup matches by hash_code without persisting. Returns [(finding, candidate), ...].""" if not findings: return [] test = findings[0].test - candidates_by_hash = find_candidates_for_deduplication_hash(test, findings) + candidates_by_hash = find_candidates_for_deduplication_hash(test, findings, candidate_qs=candidate_qs) if not candidates_by_hash: return [] matches = [] for new_finding in findings: - for match in get_matches_from_hash_candidates(new_finding, candidates_by_hash): + for match in get_matches_from_hash_candidates(new_finding, candidates_by_hash, ordering_key=ordering_key): matches.append((new_finding, match)) break return matches -def match_batch_unique_id(findings): +def match_batch_unique_id(findings, *, candidate_qs=None, ordering_key=None): """Find dedup matches by unique_id_from_tool without persisting. Returns [(finding, candidate), ...].""" if not findings: return [] test = findings[0].test - candidates_by_uid = find_candidates_for_deduplication_unique_id(test, findings) + candidates_by_uid = find_candidates_for_deduplication_unique_id(test, findings, candidate_qs=candidate_qs) if not candidates_by_uid: return [] matches = [] for new_finding in findings: - for match in get_matches_from_unique_id_candidates(new_finding, candidates_by_uid): + for match in get_matches_from_unique_id_candidates(new_finding, candidates_by_uid, ordering_key=ordering_key): matches.append((new_finding, match)) break return matches -def match_batch_uid_or_hash(findings): +def match_batch_uid_or_hash(findings, *, candidate_qs=None, ordering_key=None): """Find dedup matches by uid or hash_code without persisting. Returns [(finding, candidate), ...].""" if not findings: return [] test = findings[0].test - candidates_by_uid, existing_by_hash = find_candidates_for_deduplication_uid_or_hash(test, findings) + candidates_by_uid, existing_by_hash = find_candidates_for_deduplication_uid_or_hash(test, findings, candidate_qs=candidate_qs) if not (candidates_by_uid or existing_by_hash): return [] matches = [] for new_finding in findings: if new_finding.duplicate: continue - for match in get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, existing_by_hash): + for match in get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, existing_by_hash, ordering_key=ordering_key): matches.append((new_finding, match)) break return matches -def match_batch_legacy(findings): +def match_batch_legacy(findings, *, candidate_qs=None, ordering_key=None): """Find dedup matches by legacy algorithm without persisting. Returns [(finding, candidate), ...].""" if not findings: return [] test = findings[0].test - candidates_by_title, candidates_by_cwe = find_candidates_for_deduplication_legacy(test, findings) + candidates_by_title, candidates_by_cwe = find_candidates_for_deduplication_legacy(test, findings, candidate_qs=candidate_qs) if not (candidates_by_title or candidates_by_cwe): return [] matches = [] for new_finding in findings: - for match in get_matches_from_legacy_candidates(new_finding, candidates_by_title, candidates_by_cwe): + for match in get_matches_from_legacy_candidates(new_finding, candidates_by_title, candidates_by_cwe, ordering_key=ordering_key): matches.append((new_finding, match)) break return matches -def match_batch_of_findings(findings): +def match_batch_of_findings(findings, *, candidate_qs=None, ordering_key=None): """ Batch match findings against existing candidates without persisting. @@ -910,12 +946,12 @@ def match_batch_of_findings(findings): test = findings[0].test dedup_alg = test.deduplication_algorithm if dedup_alg == settings.DEDUPE_ALGO_HASH_CODE: - return match_batch_hash_code(findings) + return match_batch_hash_code(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) if dedup_alg == settings.DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL: - return match_batch_unique_id(findings) + return match_batch_unique_id(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) if dedup_alg == settings.DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE: - return match_batch_uid_or_hash(findings) - return match_batch_legacy(findings) + return match_batch_uid_or_hash(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) + return match_batch_legacy(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) # --------------------------------------------------------------------------- @@ -924,7 +960,7 @@ def match_batch_of_findings(findings): # --------------------------------------------------------------------------- -def _dedupe_batch_hash_code(findings): +def _dedupe_batch_hash_code(findings, *, candidate_qs=None, ordering_key=None): # NOTE: These functions intentionally interleave matching and set_duplicate() # rather than calling the match_batch_*() functions above. This is because # set_duplicate() modifies finding.duplicate in-memory, which affects the @@ -932,13 +968,13 @@ def _dedupe_batch_hash_code(findings): if not findings: return [] test = findings[0].test - candidates_by_hash = find_candidates_for_deduplication_hash(test, findings) + candidates_by_hash = find_candidates_for_deduplication_hash(test, findings, candidate_qs=candidate_qs) if not candidates_by_hash: return [] modified_new_findings = [] for new_finding in findings: deduplicationLogger.debug(f"deduplication start for finding {new_finding.id} with DEDUPE_ALGO_HASH_CODE") - for match in get_matches_from_hash_candidates(new_finding, candidates_by_hash): + for match in get_matches_from_hash_candidates(new_finding, candidates_by_hash, ordering_key=ordering_key): try: modified_new_findings.extend(set_duplicate(new_finding, match, save=False)) break @@ -947,17 +983,17 @@ def _dedupe_batch_hash_code(findings): return _flush_duplicate_changes(modified_new_findings) -def _dedupe_batch_unique_id(findings): +def _dedupe_batch_unique_id(findings, *, candidate_qs=None, ordering_key=None): if not findings: return [] test = findings[0].test - candidates_by_uid = find_candidates_for_deduplication_unique_id(test, findings) + candidates_by_uid = find_candidates_for_deduplication_unique_id(test, findings, candidate_qs=candidate_qs) if not candidates_by_uid: return [] modified_new_findings = [] for new_finding in findings: deduplicationLogger.debug(f"deduplication start for finding {new_finding.id} with DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL") - for match in get_matches_from_unique_id_candidates(new_finding, candidates_by_uid): + for match in get_matches_from_unique_id_candidates(new_finding, candidates_by_uid, ordering_key=ordering_key): deduplicationLogger.debug(f"Trying to deduplicate finding {new_finding.id} against candidate {match.id}") try: modified_new_findings.extend(set_duplicate(new_finding, match, save=False)) @@ -968,12 +1004,12 @@ def _dedupe_batch_unique_id(findings): return _flush_duplicate_changes(modified_new_findings) -def _dedupe_batch_uid_or_hash(findings): +def _dedupe_batch_uid_or_hash(findings, *, candidate_qs=None, ordering_key=None): if not findings: return [] test = findings[0].test - candidates_by_uid, existing_by_hash = find_candidates_for_deduplication_uid_or_hash(test, findings) + candidates_by_uid, existing_by_hash = find_candidates_for_deduplication_uid_or_hash(test, findings, candidate_qs=candidate_qs) if not (candidates_by_uid or existing_by_hash): return [] modified_new_findings = [] @@ -982,7 +1018,7 @@ def _dedupe_batch_uid_or_hash(findings): if new_finding.duplicate: continue - for match in get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, existing_by_hash): + for match in get_matches_from_uid_or_hash_candidates(new_finding, candidates_by_uid, existing_by_hash, ordering_key=ordering_key): try: modified_new_findings.extend(set_duplicate(new_finding, match, save=False)) break @@ -991,17 +1027,17 @@ def _dedupe_batch_uid_or_hash(findings): return _flush_duplicate_changes(modified_new_findings) -def _dedupe_batch_legacy(findings): +def _dedupe_batch_legacy(findings, *, candidate_qs=None, ordering_key=None): if not findings: return [] test = findings[0].test - candidates_by_title, candidates_by_cwe = find_candidates_for_deduplication_legacy(test, findings) + candidates_by_title, candidates_by_cwe = find_candidates_for_deduplication_legacy(test, findings, candidate_qs=candidate_qs) if not (candidates_by_title or candidates_by_cwe): return [] modified_new_findings = [] for new_finding in findings: deduplicationLogger.debug(f"deduplication start for finding {new_finding.id} with DEDUPE_ALGO_LEGACY") - for match in get_matches_from_legacy_candidates(new_finding, candidates_by_title, candidates_by_cwe): + for match in get_matches_from_legacy_candidates(new_finding, candidates_by_title, candidates_by_cwe, ordering_key=ordering_key): try: modified_new_findings.extend(set_duplicate(new_finding, match, save=False)) break @@ -1010,13 +1046,23 @@ def _dedupe_batch_legacy(findings): return _flush_duplicate_changes(modified_new_findings) -def dedupe_batch_of_findings(findings, *args, **kwargs): - """Batch deduplicate a list of findings. The findings are assumed to be in the same test.""" +def dedupe_batch_of_findings(findings, *args, candidate_qs=None, ordering_key=None, **kwargs): + """ + Batch deduplicate a list of findings. The findings are assumed to be in the same test. + + ``candidate_qs`` and ``ordering_key`` are forwarded only when set, so a custom + deduplication method that predates them keeps seeing exactly the arguments it saw before. + """ # Pro has customer implementation which will call the Pro dedupe methods, but also the normal OS dedupe methods. from dojo.utils import get_custom_method # noqa: PLC0415 -- circular import + scope_kwargs = {} + if candidate_qs is not None: + scope_kwargs["candidate_qs"] = candidate_qs + if ordering_key is not None: + scope_kwargs["ordering_key"] = ordering_key if batch_dedupe_method := get_custom_method("FINDING_DEDUPE_BATCH_METHOD"): deduplicationLogger.debug(f"Using custom deduplication method: {batch_dedupe_method.__name__}") - return batch_dedupe_method(findings, *args, **kwargs) + return batch_dedupe_method(findings, *args, **scope_kwargs, **kwargs) if not findings: logger.debug("dedupe_batch_of_findings called with no findings") @@ -1033,15 +1079,15 @@ def dedupe_batch_of_findings(findings, *args, **kwargs): if dedup_alg == settings.DEDUPE_ALGO_HASH_CODE: logger.debug(f"deduplicating finding batch with DEDUPE_ALGO_HASH_CODE - {len(findings)} findings") - return _dedupe_batch_hash_code(findings) + return _dedupe_batch_hash_code(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) if dedup_alg == settings.DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL: logger.debug(f"deduplicating finding batch with DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL - {len(findings)} findings") - return _dedupe_batch_unique_id(findings) + return _dedupe_batch_unique_id(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) if dedup_alg == settings.DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE: logger.debug(f"deduplicating finding batch with DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE - {len(findings)} findings") - return _dedupe_batch_uid_or_hash(findings) + return _dedupe_batch_uid_or_hash(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) logger.debug(f"deduplicating finding batch with LEGACY - {len(findings)} findings") - return _dedupe_batch_legacy(findings) + return _dedupe_batch_legacy(findings, candidate_qs=candidate_qs, ordering_key=ordering_key) deduplicationLogger.debug("dedupe: skipping dedupe because it's disabled in system settings get()") return [] @@ -1117,15 +1163,21 @@ def _fp_candidates_qs(scope_filter, dedup_alg, findings, exclude_ids=None): return Finding.objects.none() -def _fetch_fp_candidates_for_batch(findings, product, dedup_alg): +def _fetch_fp_candidates_for_batch(findings, product, dedup_alg, *, scope_filter=None): """ Fetch all existing findings in the product that could be FP matches for a batch, returning a dict keyed by match identifier for in-memory lookup. For unique_id_from_tool_or_hash_code the return value is a tuple (by_uid, by_hash). For all other algorithms it is a plain dict. + + ``scope_filter`` overrides which findings are searched, as a dict of filter keyword + arguments. Defaults to the finding's own product, which is what false-positive history + has always searched. A caller supplying one owns the scope entirely, including any + engagement isolation it needs to express. """ - scope_filter = {"test__engagement__product": product} + if scope_filter is None: + scope_filter = {"test__engagement__product": product} exclude_ids = {f.id for f in findings if f.id} qs = _fp_candidates_qs(scope_filter, dedup_alg, findings, exclude_ids).only( # Keep this list in sync with every field read from candidate objects in this function. @@ -1165,7 +1217,7 @@ def _fetch_fp_candidates_for_batch(findings, product, dedup_alg): return {} -def do_false_positive_history_batch(findings): +def do_false_positive_history_batch(findings, *, scope_filter=None): """ Batch version of do_false_positive_history. @@ -1187,7 +1239,7 @@ def do_false_positive_history_batch(findings): dedup_alg = findings[0].test.deduplication_algorithm # Fetch all candidate existing findings with one DB query - candidates = _fetch_fp_candidates_for_batch(findings, product, dedup_alg) + candidates = _fetch_fp_candidates_for_batch(findings, product, dedup_alg, scope_filter=scope_filter) # Optional plugin hook: refine the per-finding candidate list after it is resolved by # deduplication_algorithm. Lets a plugin (e.g. Pro) narrow candidates by fields that are diff --git a/unittests/test_dedupe_injectable_scope.py b/unittests/test_dedupe_injectable_scope.py new file mode 100644 index 00000000000..d0e7d5d50ea --- /dev/null +++ b/unittests/test_dedupe_injectable_scope.py @@ -0,0 +1,229 @@ +""" +Tests for the injectable candidate scope in dojo.finding.deduplication. + +The engine derives its own candidate scope from the incoming test: a finding +deduplicates against its own product, narrowed to one engagement by the +``deduplication_on_engagement`` flag. That derivation is the only scope an +installation can get, which is why ``build_candidate_scope_queryset`` now accepts a +``candidate_qs``, and why the winner rule accepts an ``ordering_key``. + +Both are opt-in and default to the behaviour that was there before, so these tests +assert two things at once: that supplying them works, and that not supplying them +changes nothing. +""" + +import logging + +from django.utils import timezone + +from dojo.finding.deduplication import ( + _dedupe_batch_hash_code, # noqa: PLC2701 + build_candidate_scope_queryset, + match_batch_hash_code, +) +from dojo.models import ( + Engagement, + Finding, + Product, + Product_Type, + Test, + Test_Type, + User, + UserContactInfo, +) + +from .dojo_test_case import DojoTestCase + +logger = logging.getLogger(__name__) + +SHARED_HASH = "a" * 64 + + +class TestInjectableCandidateScope(DojoTestCase): + + """A caller may supply the candidate scope instead of letting the engine derive it.""" + + def setUp(self): + super().setUp() + self.testuser = User.objects.create( + username="dedupe_scope_user", + is_staff=True, + is_superuser=True, + ) + UserContactInfo.objects.create(user=self.testuser, block_execution=True) + self.system_settings(enable_deduplication=False) + self.system_settings(enable_product_grade=False) + + self.product_type = Product_Type.objects.create(name="Dedupe Scope PT") + self.test_type = Test_Type.objects.get_or_create(name="Manual Test")[0] + # Two products, so the default scope cannot see across them. + self.test_a = self._create_test("Dedupe Scope Product A", "Scope Engagement A") + self.test_b = self._create_test("Dedupe Scope Product B", "Scope Engagement B") + + def _create_test(self, product_name, engagement_name): + product = Product.objects.create( + name=product_name, + description="Test", + prod_type=self.product_type, + ) + engagement = Engagement.objects.create( + name=engagement_name, + product=product, + target_start=timezone.now(), + target_end=timezone.now(), + ) + return Test.objects.create( + engagement=engagement, + test_type=self.test_type, + target_start=timezone.now(), + target_end=timezone.now(), + ) + + def _create_finding(self, test, title, hash_code=SHARED_HASH): + finding = Finding.objects.create( + test=test, + title=title, + severity="High", + description="Test", + mitigation="Test", + impact="Test", + reporter=self.testuser, + active=True, + verified=True, + ) + # Assigning after create keeps Finding.save() from recomputing it. + Finding.objects.filter(pk=finding.pk).update(hash_code=hash_code) + finding.refresh_from_db() + return finding + + # --- the default: unchanged ------------------------------------------ + + def test_default_scope_is_the_product_and_excludes_other_products(self): + """With no candidate_qs the scope is derived exactly as before.""" + mine = self._create_finding(self.test_a, "Scope default mine") + theirs = self._create_finding(self.test_b, "Scope default theirs") + + scope = build_candidate_scope_queryset(self.test_a) + scoped_ids = set(scope.values_list("id", flat=True)) + + self.assertIn(mine.id, scoped_ids, "a finding in the test's own product is a candidate") + self.assertNotIn( + theirs.id, scoped_ids, + "a finding in another product must not be a candidate under the derived scope", + ) + + def test_default_batch_dedupe_does_not_match_across_products(self): + """The pre-existing behaviour this change must not disturb.""" + original = self._create_finding(self.test_b, "Cross-product original") + newer = self._create_finding(self.test_a, "Cross-product newer") + + _dedupe_batch_hash_code([newer]) + + newer.refresh_from_db() + self.assertFalse( + newer.duplicate, + "identical hashes in two products must not deduplicate without an explicit scope", + ) + self.assertNotEqual(newer.duplicate_finding_id, original.id) + + # --- the seam --------------------------------------------------------- + + def test_supplied_scope_replaces_the_derivation(self): + """A supplied queryset is used as the candidate scope verbatim.""" + mine = self._create_finding(self.test_a, "Scope supplied mine") + theirs = self._create_finding(self.test_b, "Scope supplied theirs") + + both_products = Finding.objects.filter( + test__engagement__product__in=[ + self.test_a.engagement.product, + self.test_b.engagement.product, + ], + ) + scope = build_candidate_scope_queryset(self.test_a, candidate_qs=both_products) + scoped_ids = set(scope.values_list("id", flat=True)) + + self.assertIn(mine.id, scoped_ids) + self.assertIn( + theirs.id, scoped_ids, + "the supplied scope decides which findings are candidates, not the test's product", + ) + + def test_supplied_scope_matches_across_products(self): + """Matching against a cross-product scope finds the other product's finding.""" + original = self._create_finding(self.test_b, "Cross-product scoped original") + newer = self._create_finding(self.test_a, "Cross-product scoped newer") + + matches = match_batch_hash_code([newer], candidate_qs=Finding.objects.all()) + + self.assertEqual(len(matches), 1, "the cross-product candidate should have matched") + matched_new, matched_candidate = matches[0] + self.assertEqual(matched_new.id, newer.id) + self.assertEqual(matched_candidate.id, original.id) + + def test_supplied_scope_persists_the_cross_product_link(self): + """The persisting path honours the supplied scope too, not just the match-only one.""" + original = self._create_finding(self.test_b, "Cross-product persisted original") + newer = self._create_finding(self.test_a, "Cross-product persisted newer") + + _dedupe_batch_hash_code([newer], candidate_qs=Finding.objects.all()) + + newer.refresh_from_db() + self.assertTrue(newer.duplicate, "the finding should have been marked a duplicate") + self.assertEqual(newer.duplicate_finding_id, original.id) + + def test_supplied_scope_still_prefers_the_older_candidate(self): + """Widening the scope does not weaken the never-link-to-a-newer-finding rule.""" + newer_candidate = self._create_finding(self.test_b, "Ordering newer candidate") + target = self._create_finding(self.test_a, "Ordering target") + + matches = match_batch_hash_code([target], candidate_qs=Finding.objects.all()) + + self.assertEqual(len(matches), 0, msg=( + "the only candidate has a higher id than the target, so it is not a legal " + f"original (candidate id={newer_candidate.id}, target id={target.id})" + )) + + # --- ordering_key ----------------------------------------------------- + + def test_ordering_key_selects_among_several_valid_candidates(self): + """The preference order picks which older candidate becomes the original.""" + first = self._create_finding(self.test_b, "Ordering first original") + second = self._create_finding(self.test_b, "Ordering second original") + target = self._create_finding(self.test_a, "Ordering key target") + + # Default: oldest wins. + default_matches = match_batch_hash_code([target], candidate_qs=Finding.objects.all()) + self.assertEqual(default_matches[0][1].id, first.id, "the default winner is the lowest id") + + # Prefer the second finding, then fall back to id order. + preferred_matches = match_batch_hash_code( + [target], + candidate_qs=Finding.objects.all(), + ordering_key=lambda candidate: (candidate.id != second.id, candidate.id), + ) + self.assertEqual( + preferred_matches[0][1].id, second.id, + "the supplied ordering key should decide which of the valid candidates wins", + ) + + def test_ordering_key_cannot_promote_a_newer_candidate(self): + """A preference order chooses among legal originals; it cannot create one.""" + older = self._create_finding(self.test_b, "Ordering older legal candidate") + target = self._create_finding(self.test_a, "Ordering antisymmetry target") + newer = self._create_finding(self.test_b, "Ordering newer illegal candidate") + + matches = match_batch_hash_code( + [target], + candidate_qs=Finding.objects.all(), + # Ask for the newer finding first; the age guard must still reject it. + ordering_key=lambda candidate: (candidate.id != newer.id, candidate.id), + ) + + self.assertEqual(len(matches), 1) + self.assertEqual( + matches[0][1].id, older.id, + msg=( + "an ordering key must not be able to make a newer finding the original " + f"(newer id={newer.id}, target id={target.id})" + ), + ) From 98fc2f9f5f87c3c7805f683d8ccfc6fb8e70b7aa Mon Sep 17 00:00:00 2001 From: Blake Owens Date: Tue, 25 Aug 2026 18:47:46 -0500 Subject: [PATCH 02/24] test(dedupe): the target must outlive the only candidate for the age guard to bite The candidate was created first, so it held the lower id and was a legal original after all. Creating the target first makes the candidate genuinely newer, which is the case the assertion is about. --- unittests/test_dedupe_injectable_scope.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/unittests/test_dedupe_injectable_scope.py b/unittests/test_dedupe_injectable_scope.py index d0e7d5d50ea..184961645a3 100644 --- a/unittests/test_dedupe_injectable_scope.py +++ b/unittests/test_dedupe_injectable_scope.py @@ -173,8 +173,10 @@ def test_supplied_scope_persists_the_cross_product_link(self): def test_supplied_scope_still_prefers_the_older_candidate(self): """Widening the scope does not weaken the never-link-to-a-newer-finding rule.""" - newer_candidate = self._create_finding(self.test_b, "Ordering newer candidate") + # Creation order is the assertion here: the target must exist BEFORE the only + # candidate, so the candidate is the newer of the two and cannot be its original. target = self._create_finding(self.test_a, "Ordering target") + newer_candidate = self._create_finding(self.test_b, "Ordering newer candidate") matches = match_batch_hash_code([target], candidate_qs=Finding.objects.all()) From f8b3138df945854c1bfc61215b22051caeef1650 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Wed, 26 Aug 2026 16:22:34 -0500 Subject: [PATCH 03/24] docs: dedupe pools Deduplication has always been scoped to one Asset, narrowable to an Engagement. Pools are the other direction: a named group of Assets whose Findings deduplicate against each other, per matching kind. The new page covers what a pool is and is not (it changes which Findings are eligible to be compared, never how two are compared), the per-kind membership rule, why reimport appears alongside the two kinds that scope and yet cannot widen scope, the preview-then-acknowledge contract on both retroactive actions, where originals collect and why there is no newest-wins, and the parent-edges- only subtree toggle. Three statements elsewhere became incomplete rather than wrong, so they are updated in the same batch: the scope paragraph and the Pro algorithm summary in About Deduplication, its troubleshooting table (which offered only instance-wide answers to a per-Asset scope problem), and the Enabling Deduplication intro. --- .../PRO__dedupe_pools.md | 114 ++++++++++++++++++ .../PRO_enabling_product_deduplication.md | 2 +- .../about_deduplication.md | 5 +- 3 files changed, 119 insertions(+), 2 deletions(-) create mode 100644 docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md new file mode 100644 index 00000000000..17d45add932 --- /dev/null +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -0,0 +1,114 @@ +--- +title: "Dedupe Pools" +description: "Group Assets so their Findings deduplicate against each other, per matching kind" +weight: 3 +audience: pro +--- + +By default a Finding only deduplicates against other Findings in **its own Asset**, and an Engagement can narrow that further. A **Dedupe Pool** is the other direction: a named group of Assets whose Findings deduplicate against each other. + +Pools are for the case where the same thing is genuinely deployed in several places you model as separate Assets. Three services that all ship the same base image, or a monorepo split into an Asset per component, will each report the same vulnerability separately, and no per-Asset setting can make those Findings meet. + +A pool may span Organizations. You only ever see the members you have access to, and a member you cannot read is shown as a placeholder rather than hidden, so a pool never looks smaller than it is. + +Find pools at **Settings \> Deduplication Settings \> Dedupe Pools**. + +## Pools vs. the global algorithms + +Pools and the global algorithms solve the same problem at different scales, and they are not alternatives so much as different blast radii. + +| | Scope | Matches on | Choose it when | +| --- | --- | --- | --- | +| **Dedupe Pool** | The Assets you put in it | Whatever the tool's normal algorithm already uses | Some Assets should share matching and the rest should not | +| **Global Component** | Every Asset in the instance | Component name and version | Every SCA Finding for a dependency is the same Finding wherever it appears | +| **Global Locations** | Every Asset in the instance | Package URL, or URL for DAST Findings | As above, keyed on the full location under the Locations data model | + +A pool does not change **how** two Findings are compared. It changes **which** Findings are eligible to be compared at all. Tuning stays where it was, on the [Deduplication Tuning](/triage_findings/finding_deduplication/pro__deduplication_tuning/) pages. + +## Membership is per matching kind + +An Asset joins a pool for one **matching kind** at a time, and can be in at most one pool per kind. The same Asset can therefore share same-tool matching with one group and cross-tool matching with another. + +* **Same tool.** Findings from the same scanner deduplicate across the pool's Assets. +* **Cross tool.** Findings from different scanners deduplicate across the pool's Assets. +* **Reimport.** Selects the matching formula a reimport uses. It does **not** widen what is compared: reimport always matches inside its own Test. + +That last one is worth reading twice. Reimport appears alongside the other two because it is a matching decision a pool can carry, but pooling Assets for reimport does not make a reimport look outside its own Test. Only same tool and cross tool change scope. + +If you try to add an Asset that already matches within another pool for that kind, DefectDojo refuses the change and names the pool holding it. Take it out of that pool first if the move is deliberate. + +## Creating a pool changes nothing + +A new pool has no members, so nothing about deduplication changes until you add some. This is deliberate: creating a pool to look at it is safe. + +1. Open **Settings \> Deduplication Settings \> Dedupe Pools**. +2. Enter a name under **New pool** and click **Create Pool**. +3. Select the pool, then pick the **Matching kind** you want to configure. + +## Adding Assets, and previewing first + +Under **Add Assets**, choose the Assets and click **Preview Impact** before **Add to Pool**. + +The preview reports how many of the Findings **you can see** would become comparable with the rest of the pool, split by hash and by vendor ID. It is an **upper bound**, not a prediction. It answers an exact question (which Findings share an identity with the rest of the pool) rather than re-running the deduplication engine, because a preview that re-simulated every algorithm, set-matching rule and location predicate would produce confident numbers that were quietly wrong. The real run can only mark fewer. + +Adding members applies to **future imports**. Findings already in DefectDojo are untouched until you ask for them to be reprocessed. + +## Applying a pool to Findings that already exist + +**Apply to existing findings** re-runs deduplication over the Findings already in the pool's Assets. This can mark a large number of Findings as duplicates at once, so it is gated: + +1. Click **Preview Re-run**. This reports how many Findings you can see share an identity with a Finding in another Asset in the pool. +2. **Apply Now** stays disabled until that preview has run, and uses the acknowledgement the preview returned. + +The acknowledgement is derived from the specific change it describes, so the preview you ran for adding Assets does not authorize a re-run, and a re-run preview goes stale if the pool changes underneath it. Preview the thing you are about to do. + +Reimport offers no Apply Now, for the reason above: it cannot widen scope, so there is nothing retroactive to apply. + +## Where originals collect + +**Where originals collect** decides which Finding a pool's duplicates point at. + +* **Oldest finding wins.** The default, and what deduplication has always done. +* **Designated Asset, then oldest.** Duplicates point at the chosen Asset wherever it has a matching Finding, and at the oldest Finding otherwise. + +Use the second when one Asset is the place your team actually works, and you want the originals to land there rather than wherever the earliest scan happened to run. + +There is deliberately no newest-wins option. It would let an established original change hands, which breaks the guarantee that a Finding your team has already mitigated is not reopened as a duplicate of something newer. + +Changing the placement affects **new** matches. Existing duplicates keep their current original until they are re-pointed. + +## Removing an Asset from a pool + +Removing a member also applies to future imports. Findings already linked **keep their links**, including links to an original in an Asset the removed Asset no longer shares a pool with. + +That is the safe default, but it leaves duplicates pointing outside their own Asset. When you want those cleaned up, **Reset external links** clears exactly those links. It never deletes anything: a Finding whose link is cleared goes back to being an ordinary active Finding. + +## Pooling from the Asset page + +The **Dedupe Pool** panel on an Asset page shows which pool that Asset matches within, per kind, and lets you change it in place. Add it from the page layout editor if it is not already on your Asset pages. + +The panel also offers **Pool this Asset and everything under it**, which pools the Asset and its descendants for that kind in one action. Two things about it are worth knowing: + +* It follows **parent relationships only**. A reference between two Assets is not containment, so an Asset that merely uses another is not pulled in. +* It **skips rather than steals**. A descendant already pooled elsewhere for that kind is reported back as left alone, not moved. + +A membership created this way is marked **from parent**. **Untoggle subtree** removes only the memberships the toggle created; a membership someone added by hand survives it. + +## Pooling automatically with a Rule + +The Rules Engine action **Assign to a Dedupe Pool** puts an Asset into a pool, or takes it out of one. Run it on Asset creation and new Assets get pooled the way their siblings are, without anyone remembering to do it. + +Like the subtree toggle, it counts an Asset already pooled elsewhere for that kind as skipped rather than moving it. An Asset's pool is a deliberate decision, and a rule that silently relocated it would change which Findings deduplicate against each other with nothing in the run saying so. + +## Permissions + +Pools are governed by four global permissions, granted through global roles: + +| Permission | Allows | +| --- | --- | +| **View Dedupe Pool** | See pools and their members | +| **Add Dedupe Pool** | Create a pool | +| **Edit Dedupe Pool** | Change membership, placement, and run Apply Now | +| **Delete Dedupe Pool** | Delete a pool | + +Membership lists and every preview are filtered to the Assets you can read, so the numbers a preview reports are the numbers for **your** visibility, not the instance's. diff --git a/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md index b874bf59994..19a1b197cb0 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md @@ -6,7 +6,7 @@ audience: pro aliases: - /en/working_with_findings/finding_deduplication/enabling_product_deduplication --- -Deduplication can be applied at an Asset\-wide level, or scoped more narrowly to a single Engagement. +Deduplication can be applied at an Asset\-wide level, or scoped more narrowly to a single Engagement. To scope it the other way, across a chosen group of Assets, see [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/). ## Deduplication for Assets diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index 97f21b56288..3cf9943923f 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -28,6 +28,8 @@ Within a *single* report, the order the scanner happens to list its findings in By default, these Tests would need to be nested under the same Asset for Deduplication to be applied. If you wish, you can further limit the Deduplication scope to a single Engagement. +DefectDojo Pro can also widen it. A [Dedupe Pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) is a named group of Assets whose Findings deduplicate against each other, so the same vulnerability found in several Assets you deliberately model separately can still resolve to one original. + ![Deduplication on Asset and engagement level](images/deduplication.png) Duplicate Findings are set as Inactive by default. This does not mean the Duplicate Finding itself is Inactive. Rather, this is so that your team only has a single active Finding to work on and remediate, with the implication being that once the original Finding is Mitigated, the Duplicates will also be Mitigated. @@ -77,7 +79,7 @@ DefectDojo Open Source supports four deduplication algorithms that can be select - **Unique ID From Tool or Hash Code**: Prefer the tool’s unique ID; fall back to hash when no matching unique ID is found. - **Legacy**: Historical algorithm with multiple conditions; only available in the Open Source version. -**DefectDojo Pro adds more.** Two additional algorithms match across **all Assets** in the instance rather than within a single Asset or Engagement — **Global Component** (by component name and version) and **Global Vulnerability ID** (by CVE, GHSA, …). Both are off by default and enabled by DefectDojo Support. Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. +**DefectDojo Pro adds more.** [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) widen the scope of the existing algorithms to a chosen group of Assets, per matching kind, without changing how two Findings are compared. Two additional algorithms instead match across **all Assets** in the instance rather than within a single Asset or Engagement — **Global Component** (by component name and version) and **Global Vulnerability ID** (by CVE, GHSA, …). Both are off by default and enabled by DefectDojo Support. Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. ### An alternative to Deduplication: False Positive History @@ -171,6 +173,7 @@ Sometimes, Deduplication does not work as expected. Here are some examples of w | Duplicates are created across different tools | Cross-tool matching is disabled or too strict | Cross Tool Deduplication (Pro only) (hash-based matching) | | The same SCA dependency imported into multiple Assets creates separate Findings instead of duplicates | Deduplication is scoped per Asset by default | Global Component Deduplication (Pro only) ([enable for your SCA tools](/triage_findings/finding_deduplication/pro__global_component_deduplication/)), or, under the Locations data model, Global Locations Deduplication (Pro only) ([match on shared location](/triage_findings/finding_deduplication/pro__global_locations_deduplication/)) | | The same URL / web Finding imported into multiple Assets creates separate Findings instead of duplicates | Deduplication is scoped per Asset by default, and Global Component matches only components | Global Locations Deduplication (Pro only) ([match DAST/URL Findings across Assets](/triage_findings/finding_deduplication/pro__global_locations_deduplication/)) | +| The same vulnerability in a handful of related Assets creates separate Findings, but you do not want instance-wide matching | Deduplication is scoped per Asset by default, and the global algorithms are all-or-nothing | Dedupe Pools (Pro only) ([group just those Assets](/triage_findings/finding_deduplication/pro__dedupe_pools/)) | | Excess duplicates of the same Finding are being created, across Tests | Asset Hierarchy is not set up correctly | [Consider Reimport for continual testing](/triage_findings/finding_deduplication/avoid_excess_duplicates/) | When automatic deduplication misses Findings that you believe belong together, you can link them by hand from the View Finding page. See Similar Findings for how to discover related Findings and mark them as duplicates manually ([Open Source](/triage_findings/finding_deduplication/os__similar_findings/) | [Pro](/triage_findings/finding_deduplication/pro__similar_findings/)). From 061836ec2aa6ccdc69df8c382618b0efe99ff4d4 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Fri, 28 Aug 2026 07:27:05 -0500 Subject: [PATCH 04/24] docs: reimports do deduplicate across a dedupe pool The pools page said a reimport "does not widen what is compared" and left it there. True of a reimport's own matching, and it reads as "reimports never deduplicate across a pool", which is false. That reading is exactly how it was caught. Both halves are now stated. A reimport's own matching stays inside its Test, because that matching decides whether a Finding is updated, created or closed and is scoped to what the scan is authoritative over. The Findings it creates are then deduplicated under same tool and cross tool, which are pool-scoped. About Deduplication already said the second half in general terms ("Findings that remain after Reimport Deduplication are still subject to Same-Tool Deduplication"); it now names pools as a case of it, so the two pages agree. The apply-now paragraph carried the same framing and is reworded: pooling for reimport picks a formula rather than a scope, so there is no widened scope to re-run, and Findings a reimport created are covered by the other two kinds. --- .../PRO__dedupe_pools.md | 22 ++++++++++++++++--- .../about_deduplication.md | 2 ++ 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index 17d45add932..c12c8325d56 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -31,9 +31,22 @@ An Asset joins a pool for one **matching kind** at a time, and can be in at most * **Same tool.** Findings from the same scanner deduplicate across the pool's Assets. * **Cross tool.** Findings from different scanners deduplicate across the pool's Assets. -* **Reimport.** Selects the matching formula a reimport uses. It does **not** widen what is compared: reimport always matches inside its own Test. +* **Reimport.** Selects the matching formula a reimport uses. It does **not** widen what a reimport itself compares: a reimport matches inside its own Test. Findings it creates are still deduplicated across the pool under same tool and cross tool. -That last one is worth reading twice. Reimport appears alongside the other two because it is a matching decision a pool can carry, but pooling Assets for reimport does not make a reimport look outside its own Test. Only same tool and cross tool change scope. +That last one is worth reading twice, because the obvious reading is wrong. Two things are true +at once: + +* A reimport's **own** matching stays inside its Test. That matching decides whether an incoming + Finding updates an existing one, is created fresh, or whether a Finding missing from the scan + gets closed, so it is scoped to the Test the scan is authoritative over. Pooling Assets for + reimport does not change that. +* Findings a reimport **creates** are then deduplicated like any other new Finding, under same + tool and cross tool. If the Asset is pooled for those kinds, that deduplication reaches across + the pool. + +So pooling does affect reimports; it just affects what happens to the Findings a reimport +produces, rather than what the reimport itself compares against. Only same tool and cross tool +change scope. If you try to add an Asset that already matches within another pool for that kind, DefectDojo refuses the change and names the pool holding it. Take it out of that pool first if the move is deliberate. @@ -62,7 +75,10 @@ Adding members applies to **future imports**. Findings already in DefectDojo are The acknowledgement is derived from the specific change it describes, so the preview you ran for adding Assets does not authorize a re-run, and a re-run preview goes stale if the pool changes underneath it. Preview the thing you are about to do. -Reimport offers no Apply Now, for the reason above: it cannot widen scope, so there is nothing retroactive to apply. +Reimport offers no Apply Now, for the reason above: pooling for reimport changes which formula a +reimport uses, not which Findings it compares against, so there is no widened scope to re-run +over existing Findings. Re-running deduplication across the pool is what the same tool and +cross tool kinds do, and Findings a reimport created are included in that like any other. ## Where originals collect diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index 3cf9943923f..b9edacb2e94 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -42,6 +42,8 @@ Deduplication and Reimport are similar processes, but they use different algorit However, any Findings that remain after Reimport Deduplication are still subject to Same-Tool Deduplication. So if you use narrower a scope for Same-Tool Deduplication, you can end up with Duplicates within a Reimport pipeline. +This is also how a [Dedupe Pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) reaches a reimport. The reimport itself only ever matches inside its own Test, but the Findings it creates go through Same-Tool and Cross-Tool Deduplication afterwards, and those are pool-scoped. Pooling an Asset therefore does affect its reimports, by way of what happens to the Findings they produce. + ### Example Here's a tool with a Reimport Deduplication algorithm which is different from the Same-Tool Deduplication algorithm. From a2160f257efee552d1df8eb0aa00cec6c0e4fa63 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 31 Aug 2026 00:51:46 -0500 Subject: [PATCH 05/24] docs: the deduplication tuning page now describes Matching Configuration The three tuner deduplication pages were replaced by a single Matching Configuration page, so this page was describing a UI that no longer exists: a menu path that is gone, a tool dropdown on one of three pages, and four screenshots of retired screens. Two claims were not merely stale but wrong in a way that matters. It promised that changing a tool's settings "automatically triggers a background re-hash of all existing Findings". Changing an algorithm does not re-hash anything, and cannot: the algorithm selects which already-stored value is compared, so there is nothing to recompute. A reader following the old text would wait for a backlog re-hash that is never coming. That section is replaced with what actually happens, and points at a pool's Apply Now for the case it was reaching for. It also described selecting hash fields, which is not possible in this release. Changing hash fields changes how every stored hash was computed, so it needs a new generation written behind it before matching moves across, and that is not shipped. The page says so and points at support rather than describing a control that is not there. The reference material that is still accurate is kept as is: the algorithms, Content Fingerprint, the set-based vulnerability-id and CWE matchers, and location drift tracking. --- .../PRO__dedupe_pools.md | 2 +- .../PRO__deduplication_tuning.md | 83 +++++++++---------- 2 files changed, 40 insertions(+), 45 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index c12c8325d56..b37302fdc63 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -23,7 +23,7 @@ Pools and the global algorithms solve the same problem at different scales, and | **Global Component** | Every Asset in the instance | Component name and version | Every SCA Finding for a dependency is the same Finding wherever it appears | | **Global Locations** | Every Asset in the instance | Package URL, or URL for DAST Findings | As above, keyed on the full location under the Locations data model | -A pool does not change **how** two Findings are compared. It changes **which** Findings are eligible to be compared at all. Tuning stays where it was, on the [Deduplication Tuning](/triage_findings/finding_deduplication/pro__deduplication_tuning/) pages. +A pool does not change **how** two Findings are compared. It changes **which** Findings are eligible to be compared at all. How they are compared is set per tool on [Matching Configuration](/triage_findings/finding_deduplication/pro__deduplication_tuning/), which a pool can override for its own members. ## Membership is per matching kind diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index f0c0ae11455..36215fae381 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -5,39 +5,47 @@ weight: 4 audience: pro aliases: - /en/working_with_findings/finding_deduplication/tune_deduplication + - /en/triage_findings/finding_deduplication/matching_configuration --- Deduplication Tuning is a DefectDojo Pro feature that gives you fine-grained control over how findings are deduplicated, allowing you to optimize duplicate detection for your specific security testing workflow. -## Deduplication Settings +## Matching Configuration -In DefectDojo Pro, you can access Deduplication Tuning through: -**Settings > Finding Workflow** (**Settings > Pro Settings > Deduplication Settings** on instances still using the previous menu layout) +In DefectDojo Pro, matching is configured at **Settings > Matching Configuration**. -![image](images/deduplication_tuning.png) +This page replaced three separate pages (Same Tool Deduplication, Cross Tool Deduplication and Reimport Deduplication). Bookmarks to those pages redirect here. Instead of picking a tool from a dropdown on one of three pages, every tool is listed once with a column for each of the three matching kinds: -The Deduplication Settings page offers three key configuration areas: -- Same Tool Deduplication -- Cross Tool Deduplication -- Reimport Deduplication +- **Same tool**: how repeated scans from one tool are recognised as the same finding. +- **Cross tool**: how findings from different tools are matched against each other. +- **Reimport**: which formula a reimport uses inside its own test. -## Same Tool Deduplication +A tool's row shows the algorithm in force for each kind, how many hash fields it uses, whether anyone has changed it from the shipped default, and whether a [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) overrides it. -Same Tool Deduplication is enabled by default for all security tool parsers. This ensures findings from consecutive scans using the same tool are properly deduplicated. +### Changing an algorithm + +Select the algorithm in a tool's row to change it. Because a matching change decides which findings are treated as the same finding, it is not saved directly: + +1. Choose the new algorithm. The page explains what each one matches on. +2. Select **Review impact**. DefectDojo reports how many findings of that tool are in scope, and warns you when the new algorithm would leave findings with no identity to match on at all: a tool whose findings carry no unique ID matches nothing once the algorithm requires one, and nothing errors when that happens. +3. Confirm you understand the change, then select **Apply**. -To adjust Same Tool Deduplication: +The change decides what the next import compares. Existing duplicate links are left exactly as they are, so applying it does not re-link or unlink anything that is already recorded. + +> **Hash fields are not editable in this release.** The algorithm is. Changing which fields make up a tool's hash changes how every hash already stored was computed, so it needs a new generation of hashes written behind it before matching can move across; that work is not released yet. To change a tool's hash fields, contact [DefectDojo Support](mailto:support@defectdojo.com). + +## Same Tool Deduplication -1. Select a specific **Security Tool** from the dropdown -2. Choose a **Deduplication Algorithm** from the available options +Same Tool Deduplication is enabled by default for all security tool parsers. This ensures findings from consecutive scans using the same tool are properly deduplicated. -![image](images/same_tool_deduplication.png) +To adjust Same Tool Deduplication, select the algorithm in the tool's **Same tool** column on **Settings > Matching Configuration** and follow the review-and-confirm steps above. ### Available Deduplication Algorithms DefectDojo Pro offers the following deduplication methods for same-tool deduplication: #### Hash Code -Uses a combination of selected fields to generate a unique hash. When selected, a third dropdown will appear showing the fields being used to calculate the hash. +Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Matching Configuration** shows how many fields make up its hash; the fields themselves are not editable in this release (see the note above). ##### Content Fingerprint @@ -94,13 +102,9 @@ The `_partial` and `_subset` fields are compared per finding pair rather than fo Cross Tool Deduplication is disabled by default, as deduplication between different security tools requires careful configuration due to variations in how tools report the same vulnerabilities. -![image](images/cross_tool_deduplication.png) - -To enable Cross Tool Deduplication: +To enable Cross Tool Deduplication, select the algorithm in the tool's **Cross tool** column on **Settings > Matching Configuration** and change it to Hash Code. -1. Select a **Security Tool** from the dropdown -2. Change the **Deduplication Algorithm** from "Disabled" to "Hash Code" -3. Select which fields should be used for generating the hash in the **Hash Code Fields** dropdown +The hash fields it uses are the ones already configured for that tool; they cannot be changed in this release (see the note above). Cross Tool Deduplication supports the Hash Code algorithm, which is suitable for most workflows, as different tools rarely share compatible unique identifiers. For SCA tools reporting the same dependencies, [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) is also available as a cross-tool option (off by default). @@ -114,12 +118,7 @@ Reimport Deduplication Settings can be used to set an algorithm for Universal Pa Reimport Deduplication cannot be adjusted for other tools by default. Users who want to adjust the Reimport Deduplication algorithm for other tools in their instance should reach out to [DefectDojo Support](mailto:support@defectdojo.com) for assistance. -![image](images/reimport_deduplication.png) - -When configuring Reimport Deduplication: - -1. Select the **Security Tool** (Universal or Generic Parser) -2. Choose the appropriate **Deduplication Algorithm** +To configure Reimport Deduplication, select the algorithm in the tool's **Reimport** column on **Settings > Matching Configuration**. The following algorithm options are available for Reimport Deduplication: - Hash Code @@ -130,35 +129,29 @@ Reimport can completely discard Findings before they are recorded, so Reimport D ### Track Findings as Locations Change -When a tool's Reimport Deduplication algorithm is **Hash Code**, an additional toggle appears: **Track findings as locations change**. With it enabled, a finding whose location moved between reimports — a line shift or file rename, a URL move, or a dependency version bump — is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created. +A tool whose Reimport algorithm is **Hash Code** can also track findings as their locations change. With that enabled, a finding whose location moved between reimports — a line shift or file rename, a URL move, or a dependency version bump — is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created. -The toggle is off by default and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Enabling it automatically re-hashes the tool's existing findings in the background so historical data participates immediately. +It is off by default, is set by DefectDojo Support in this release, and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Enabling it automatically re-hashes the tool's existing findings in the background so historical data participates immediately. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) for how the matching works, what is preserved, and guidance for enabling it on large instances. -## Running Deduplication Retroactively on Existing Data +## What changing an algorithm does to existing findings -A common situation when first turning on Deduplication Tuning is having a large backlog of Findings that were imported *before* the dedup configuration changed. In DefectDojo Pro, you do not need to run a separate command to dedupe this historical data — **changing the Deduplication Settings for a tool automatically triggers a background re-hash of all existing Findings associated with that test type**. +Nothing, by design. Changing a tool's algorithm decides what the **next** import compares. Findings already in the instance keep the duplicate links they have, and no hashes are recomputed. -What this means in practice: +That is a deliberate difference from how a hash-field change behaves. Hash fields determine the value stored on each finding, so changing them requires recomputing that value across the tool's whole backlog; an algorithm change only selects which already-stored value is compared, so there is nothing to recompute. -- When you change the **Deduplication Algorithm** or the **Hash Code Fields** for a tool, DefectDojo queues a background job to recompute hashes for every Finding from that tool already in the instance. -- The job runs asynchronously. On large instances (millions of Findings), this can take some time to complete and you will not see immediate changes in the Findings table. -- Newly-computed hashes apply to subsequent dedup decisions across the whole backlog. +If you need existing findings re-evaluated against a new configuration, use a [dedupe pool's](/triage_findings/finding_deduplication/pro__dedupe_pools/) **Apply Now**, which re-runs deduplication over the findings already in scope and reports what it would link before it does anything. -If you make several configuration changes in quick succession, each one queues its own re-hash job. Allow the previous job to finish before evaluating results, especially when comparing Findings counts before and after the change. - -> **Note for self-hosted Pro:** The background job runs in the Celery worker pool. If you have starved or backlogged workers, the re-hash can take longer than expected — check worker health if results don't appear within the timeframe you would expect for your instance size. - -> **Feature flags do not gate an existing configuration.** A tool's saved Deduplication Settings stay in effect for as long as they are configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change or stop a tool's deduplication behavior, update its Deduplication Settings directly (which also queues the background re-hash described above). +> **Feature flags do not gate an existing configuration.** A tool's saved matching configuration stays in effect for as long as it is configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change a tool's behavior, change its algorithm on **Settings > Matching Configuration**. ## Deduplication Best Practices For optimal results with Deduplication Tuning: - **Start with defaults**: The preconfigured deduplication settings work well for most scenarios -- **Test changes carefully**: After adjusting deduplication settings, monitor a few imports to ensure proper behavior. -- **Plan retroactive re-hashes**: Changing dedup settings re-hashes every existing Finding from that tool in the background. See [Running Deduplication Retroactively on Existing Data](#running-deduplication-retroactively-on-existing-data) above. +- **Read the impact review before applying**: it tells you how many findings are in scope and, more importantly, whether the new algorithm leaves any of them with nothing to match on. +- **Test changes carefully**: After adjusting matching configuration, monitor a few imports to ensure proper behavior. - **Use Hash Code for cross-tool deduplication**: When enabling cross-tool deduplication, select fields that reliably identify the same finding across different tools (such as vulnerability name, location, and severity). **IMPORTANT** Each tool enabled for cross-tool deduplication **MUST** have the same fields selected. - **Keep cross-tool sources in the same Asset**: Cross-Tool Deduplication is Asset-scoped. Findings split across separate Assets will not dedupe even with matching hash fields. See [Cross Tool Deduplication](#cross-tool-deduplication) above. - **Avoid overly broad deduplication**: Cross-tool deduplication with too few hash fields may result in false duplicates @@ -167,6 +160,8 @@ For optimal results with Deduplication Tuning: By tuning deduplication settings to your specific tools, you can significantly reduce duplicate noise. -## Locked Findings +## Where a tool's matching came from + +A tool's row on **Settings > Matching Configuration** marks configuration that has been changed from the shipped default, and names any dedupe pool that overrides it. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool responsible when it differs from the instance default. -Whenever Deduplication Settings are changed for a given tool, Deduplication hashes are re-calculated for that tool across the entire DefectDojo instance. \ No newline at end of file +That pairing is what answers "why did these two findings deduplicate differently" without a support ticket: two tests on the same tool showing different algorithms is a pool override, not a fault. \ No newline at end of file From fc614be2c504af774c6d8587e6dd74a1cbe1d42b Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 31 Aug 2026 06:32:21 -0500 Subject: [PATCH 06/24] docs: hash fields are editable again, per axis Follows the Pro change restoring hash-field editing to Matching Configuration. The previous revision said they were not editable and pointed at support, which was true for one commit and is not now. Splits the retroactive-re-hash guidance by axis rather than making one claim about both, since they behave oppositely and conflating them is what made the original page wrong: changing hash fields recomputes the tool's whole backlog in the background, and changing the algorithm recomputes nothing at all. Also records that hash fields are set on the instance default rather than per pool, and why: a finding stores one hash and every other view of that finding reads it. --- .../PRO__deduplication_tuning.md | 48 ++++++++++++------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 36215fae381..714b2804d1f 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -22,30 +22,35 @@ This page replaced three separate pages (Same Tool Deduplication, Cross Tool Ded A tool's row shows the algorithm in force for each kind, how many hash fields it uses, whether anyone has changed it from the shipped default, and whether a [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) overrides it. -### Changing an algorithm +### Changing a tool's matching -Select the algorithm in a tool's row to change it. Because a matching change decides which findings are treated as the same finding, it is not saved directly: +Select a tool's row to change its algorithm, its hash fields, or both. Because a matching change decides which findings are treated as the same finding, it is not saved directly: -1. Choose the new algorithm. The page explains what each one matches on. -2. Select **Review impact**. DefectDojo reports how many findings of that tool are in scope, and warns you when the new algorithm would leave findings with no identity to match on at all: a tool whose findings carry no unique ID matches nothing once the algorithm requires one, and nothing errors when that happens. +1. Choose the new algorithm, the new hash fields, or both. The page explains what each algorithm matches on. +2. Select **Review impact**. DefectDojo reports how many findings of that tool are in scope, and warns you about the two things that are easy to miss (see below). 3. Confirm you understand the change, then select **Apply**. -The change decides what the next import compares. Existing duplicate links are left exactly as they are, so applying it does not re-link or unlink anything that is already recorded. +**The two axes behave very differently, and the impact review says which one you are moving.** -> **Hash fields are not editable in this release.** The algorithm is. Changing which fields make up a tool's hash changes how every hash already stored was computed, so it needs a new generation of hashes written behind it before matching can move across; that work is not released yet. To change a tool's hash fields, contact [DefectDojo Support](mailto:support@defectdojo.com). +- **Changing the algorithm** selects which stored value candidates are looked up by. Nothing is recomputed, and the change decides what the next import compares; existing duplicate links are left exactly as they are. The review warns you when the new algorithm would leave findings with no identity to match on at all — a tool whose findings carry no unique ID matches nothing once the algorithm requires one, and nothing errors when that happens. +- **Changing the hash fields** changes the value stored on every finding of that tool, so every hash already stored for it becomes stale. Applying queues a background recompute of the tool's whole backlog. Until that finishes, the tool's findings are hashed under two different definitions and may not match each other. The review tells you how many findings will be recomputed before you commit to it. + +Two rules are enforced when you save a field selection, for the reasons in [Set-based Hash Code Fields](#set-based-hash-code-fields-vulnerability-ids-and-cwes) below: a vulnerability IDs field may stand on its own, and CWE fields may not be the only criteria. + +> **Hash fields are set on the instance default, not per pool.** A [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) can give its members a different **algorithm**, but not a different field list. A finding stores one hash, and the classic UI, the v2 API and CSV exports all read that same value, so a pool-specific field list would change what every other view of that finding shows. ## Same Tool Deduplication Same Tool Deduplication is enabled by default for all security tool parsers. This ensures findings from consecutive scans using the same tool are properly deduplicated. -To adjust Same Tool Deduplication, select the algorithm in the tool's **Same tool** column on **Settings > Matching Configuration** and follow the review-and-confirm steps above. +To adjust Same Tool Deduplication, select the tool's **Same tool** column on **Settings > Matching Configuration** and follow the review-and-confirm steps above. ### Available Deduplication Algorithms DefectDojo Pro offers the following deduplication methods for same-tool deduplication: #### Hash Code -Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Matching Configuration** shows how many fields make up its hash; the fields themselves are not editable in this release (see the note above). +Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Matching Configuration** shows how many fields make up its hash, and selecting the row lets you change them. ##### Content Fingerprint @@ -102,9 +107,7 @@ The `_partial` and `_subset` fields are compared per finding pair rather than fo Cross Tool Deduplication is disabled by default, as deduplication between different security tools requires careful configuration due to variations in how tools report the same vulnerabilities. -To enable Cross Tool Deduplication, select the algorithm in the tool's **Cross tool** column on **Settings > Matching Configuration** and change it to Hash Code. - -The hash fields it uses are the ones already configured for that tool; they cannot be changed in this release (see the note above). +To enable Cross Tool Deduplication, select the tool's **Cross tool** column on **Settings > Matching Configuration**, change the algorithm to Hash Code, and select the fields the hash should be built from. Cross Tool Deduplication supports the Hash Code algorithm, which is suitable for most workflows, as different tools rarely share compatible unique identifiers. For SCA tools reporting the same dependencies, [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) is also available as a cross-tool option (off by default). @@ -118,7 +121,7 @@ Reimport Deduplication Settings can be used to set an algorithm for Universal Pa Reimport Deduplication cannot be adjusted for other tools by default. Users who want to adjust the Reimport Deduplication algorithm for other tools in their instance should reach out to [DefectDojo Support](mailto:support@defectdojo.com) for assistance. -To configure Reimport Deduplication, select the algorithm in the tool's **Reimport** column on **Settings > Matching Configuration**. +To configure Reimport Deduplication, select the tool's **Reimport** column on **Settings > Matching Configuration**. The following algorithm options are available for Reimport Deduplication: - Hash Code @@ -135,13 +138,23 @@ It is off by default, is set by DefectDojo Support in this release, and applies See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) for how the matching works, what is preserved, and guidance for enabling it on large instances. -## What changing an algorithm does to existing findings +## Running Deduplication Retroactively on Existing Data + +A common situation when first tuning matching is having a large backlog of Findings that were imported *before* the configuration changed. What happens to them depends on which axis you changed. + +**Changing the hash fields re-hashes the backlog.** DefectDojo queues a background job to recompute the stored hash for every Finding from that tool, because the fields determine that value. The impact review tells you how many Findings that is before you apply. + +- The job runs asynchronously. On large instances (millions of Findings), this takes time and you will not see immediate changes in the Findings table. +- Until it finishes, that tool's Findings are hashed under two different definitions and may not match each other. +- Existing duplicate links are not revisited. Re-hashing changes what future comparisons produce, not what was already decided. + +If you make several changes in quick succession, each queues its own job. Allow the previous one to finish before evaluating results, especially when comparing Finding counts before and after. -Nothing, by design. Changing a tool's algorithm decides what the **next** import compares. Findings already in the instance keep the duplicate links they have, and no hashes are recomputed. +> **Note for self-hosted Pro:** the job runs in the Celery worker pool. If workers are starved or backlogged, the re-hash takes longer than expected — check worker health if results do not appear within the timeframe you would expect for your instance size. -That is a deliberate difference from how a hash-field change behaves. Hash fields determine the value stored on each finding, so changing them requires recomputing that value across the tool's whole backlog; an algorithm change only selects which already-stored value is compared, so there is nothing to recompute. +**Changing the algorithm re-hashes nothing**, by design: it selects which already-stored value is compared, so there is nothing to recompute. It decides what the next import compares, and existing duplicate links are left as they are. -If you need existing findings re-evaluated against a new configuration, use a [dedupe pool's](/triage_findings/finding_deduplication/pro__dedupe_pools/) **Apply Now**, which re-runs deduplication over the findings already in scope and reports what it would link before it does anything. +If you need existing Findings re-evaluated against a new configuration, use a [dedupe pool's](/triage_findings/finding_deduplication/pro__dedupe_pools/) **Apply Now**, which re-runs deduplication over the Findings already in scope and reports what it would link before it does anything. > **Feature flags do not gate an existing configuration.** A tool's saved matching configuration stays in effect for as long as it is configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change a tool's behavior, change its algorithm on **Settings > Matching Configuration**. @@ -150,7 +163,8 @@ If you need existing findings re-evaluated against a new configuration, use a [d For optimal results with Deduplication Tuning: - **Start with defaults**: The preconfigured deduplication settings work well for most scenarios -- **Read the impact review before applying**: it tells you how many findings are in scope and, more importantly, whether the new algorithm leaves any of them with nothing to match on. +- **Read the impact review before applying**: it tells you how many findings are in scope, whether a new algorithm leaves any of them with nothing to match on, and how many findings a field change will re-hash. +- **Plan retroactive re-hashes**: changing hash fields recomputes every existing Finding from that tool in the background. See [Running Deduplication Retroactively on Existing Data](#running-deduplication-retroactively-on-existing-data). - **Test changes carefully**: After adjusting matching configuration, monitor a few imports to ensure proper behavior. - **Use Hash Code for cross-tool deduplication**: When enabling cross-tool deduplication, select fields that reliably identify the same finding across different tools (such as vulnerability name, location, and severity). **IMPORTANT** Each tool enabled for cross-tool deduplication **MUST** have the same fields selected. - **Keep cross-tool sources in the same Asset**: Cross-Tool Deduplication is Asset-scoped. Findings split across separate Assets will not dedupe even with matching hash fields. See [Cross Tool Deduplication](#cross-tool-deduplication) above. From c01f4b1f4d75d7caea1540d5cef788e88f8feffc Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Fri, 4 Sep 2026 07:40:24 -0500 Subject: [PATCH 07/24] docs: fix the stale drift page, the global-algorithm blast radius, and the nav paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four review findings on the dedupe docs batch. **PRO__location_drift_matching.md still walked users to a deleted page.** It said Settings > Finding Workflow > Reimport Deduplication and "Enable Track findings as locations change" — a page this batch removes — while the tuning page claimed the toggle was Support-set. One of those had to be wrong, and the drift page is the one a user follows step by step. The toggle is self-serve again on Matching Configuration (the Pro side of this batch adds it), so both pages now describe that, including the impact review, because turning it on or off changes which fields the reimport hash is built from and recomputes the tool's backlog. **Pooling narrows a global algorithm rather than leaving it alone.** The "Pools vs. the global algorithms" table read as two independent choices at different blast radii, but pro/dedupe/scope.py bounds Global Component and Global Locations to the pool once an Asset joins one for that kind. A Global Component user who creates a pool would silently narrow matching they believed was instance-wide. Called out under the table and on both PRO__global_* pages, which are the ones that promise "across all Assets". **The pages disagreed about where the feature lives.** Dedupe Pools said Settings > Deduplication Settings > Dedupe Pools; the tuning page said Settings > Matching Configuration. Both entries are in the same nav group, so every page now names the full path and says the two sit beside each other. Translations are left for the usual separate pass; only the English pages are updated here. --- .../finding_deduplication/PRO__dedupe_pools.md | 11 ++++++++++- .../PRO__deduplication_tuning.md | 16 ++++++++-------- .../PRO__global_component_deduplication.md | 10 +++++++++- .../PRO__global_locations_deduplication.md | 10 +++++++++- .../PRO__location_drift_matching.md | 12 ++++++------ 5 files changed, 42 insertions(+), 17 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index b37302fdc63..a4f4d3e11ce 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -11,7 +11,7 @@ Pools are for the case where the same thing is genuinely deployed in several pla A pool may span Organizations. You only ever see the members you have access to, and a member you cannot read is shown as a placeholder rather than hidden, so a pool never looks smaller than it is. -Find pools at **Settings \> Deduplication Settings \> Dedupe Pools**. +Find pools at **Settings \> Deduplication Settings \> Dedupe Pools**. Matching Configuration sits beside it in the same group. ## Pools vs. the global algorithms @@ -23,6 +23,15 @@ Pools and the global algorithms solve the same problem at different scales, and | **Global Component** | Every Asset in the instance | Component name and version | Every SCA Finding for a dependency is the same Finding wherever it appears | | **Global Locations** | Every Asset in the instance | Package URL, or URL for DAST Findings | As above, keyed on the full location under the Locations data model | +> **Pooling an Asset narrows a global algorithm rather than leaving it alone.** The two are not +> independent settings at different blast radii. While an Asset is unpooled, Global Component and +> Global Locations reach the whole instance as described above. Once that Asset joins a pool for +> the matching kind in question, those algorithms are **bounded to the pool**: its Findings match +> only against the pool's other members, not instance-wide. So creating a pool that happens to +> contain an Asset running Global Component silently narrows matching that was previously +> instance-wide. If you want a tool to keep matching across every Asset, leave its Assets out of +> a pool for that kind. + A pool does not change **how** two Findings are compared. It changes **which** Findings are eligible to be compared at all. How they are compared is set per tool on [Matching Configuration](/triage_findings/finding_deduplication/pro__deduplication_tuning/), which a pool can override for its own members. ## Membership is per matching kind diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 714b2804d1f..211f355c731 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -12,7 +12,7 @@ Deduplication Tuning is a DefectDojo Pro feature that gives you fine-grained con ## Matching Configuration -In DefectDojo Pro, matching is configured at **Settings > Matching Configuration**. +In DefectDojo Pro, matching is configured at **Settings > Deduplication Settings > Matching Configuration**, beside [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) in the same group. This page replaced three separate pages (Same Tool Deduplication, Cross Tool Deduplication and Reimport Deduplication). Bookmarks to those pages redirect here. Instead of picking a tool from a dropdown on one of three pages, every tool is listed once with a column for each of the three matching kinds: @@ -43,14 +43,14 @@ Two rules are enforced when you save a field selection, for the reasons in [Set- Same Tool Deduplication is enabled by default for all security tool parsers. This ensures findings from consecutive scans using the same tool are properly deduplicated. -To adjust Same Tool Deduplication, select the tool's **Same tool** column on **Settings > Matching Configuration** and follow the review-and-confirm steps above. +To adjust Same Tool Deduplication, select the tool's **Same tool** column on **Settings > Deduplication Settings > Matching Configuration** and follow the review-and-confirm steps above. ### Available Deduplication Algorithms DefectDojo Pro offers the following deduplication methods for same-tool deduplication: #### Hash Code -Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Matching Configuration** shows how many fields make up its hash, and selecting the row lets you change them. +Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Deduplication Settings > Matching Configuration** shows how many fields make up its hash, and selecting the row lets you change them. ##### Content Fingerprint @@ -107,7 +107,7 @@ The `_partial` and `_subset` fields are compared per finding pair rather than fo Cross Tool Deduplication is disabled by default, as deduplication between different security tools requires careful configuration due to variations in how tools report the same vulnerabilities. -To enable Cross Tool Deduplication, select the tool's **Cross tool** column on **Settings > Matching Configuration**, change the algorithm to Hash Code, and select the fields the hash should be built from. +To enable Cross Tool Deduplication, select the tool's **Cross tool** column on **Settings > Deduplication Settings > Matching Configuration**, change the algorithm to Hash Code, and select the fields the hash should be built from. Cross Tool Deduplication supports the Hash Code algorithm, which is suitable for most workflows, as different tools rarely share compatible unique identifiers. For SCA tools reporting the same dependencies, [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) is also available as a cross-tool option (off by default). @@ -121,7 +121,7 @@ Reimport Deduplication Settings can be used to set an algorithm for Universal Pa Reimport Deduplication cannot be adjusted for other tools by default. Users who want to adjust the Reimport Deduplication algorithm for other tools in their instance should reach out to [DefectDojo Support](mailto:support@defectdojo.com) for assistance. -To configure Reimport Deduplication, select the tool's **Reimport** column on **Settings > Matching Configuration**. +To configure Reimport Deduplication, select the tool's **Reimport** column on **Settings > Deduplication Settings > Matching Configuration**. The following algorithm options are available for Reimport Deduplication: - Hash Code @@ -134,7 +134,7 @@ Reimport can completely discard Findings before they are recorded, so Reimport D A tool whose Reimport algorithm is **Hash Code** can also track findings as their locations change. With that enabled, a finding whose location moved between reimports — a line shift or file rename, a URL move, or a dependency version bump — is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created. -It is off by default, is set by DefectDojo Support in this release, and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Enabling it automatically re-hashes the tool's existing findings in the background so historical data participates immediately. +It is off by default and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Tick it in the tool's **Reimport** column on **Settings > Deduplication Settings > Matching Configuration**. Because it changes which fields the reimport hash is built from, applying it re-hashes the tool's existing findings in the background; the impact review tells you how many before you commit. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) for how the matching works, what is preserved, and guidance for enabling it on large instances. @@ -156,7 +156,7 @@ If you make several changes in quick succession, each queues its own job. Allow If you need existing Findings re-evaluated against a new configuration, use a [dedupe pool's](/triage_findings/finding_deduplication/pro__dedupe_pools/) **Apply Now**, which re-runs deduplication over the Findings already in scope and reports what it would link before it does anything. -> **Feature flags do not gate an existing configuration.** A tool's saved matching configuration stays in effect for as long as it is configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change a tool's behavior, change its algorithm on **Settings > Matching Configuration**. +> **Feature flags do not gate an existing configuration.** A tool's saved matching configuration stays in effect for as long as it is configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change a tool's behavior, change its algorithm on **Settings > Deduplication Settings > Matching Configuration**. ## Deduplication Best Practices @@ -176,6 +176,6 @@ By tuning deduplication settings to your specific tools, you can significantly r ## Where a tool's matching came from -A tool's row on **Settings > Matching Configuration** marks configuration that has been changed from the shipped default, and names any dedupe pool that overrides it. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool responsible when it differs from the instance default. +A tool's row on **Settings > Deduplication Settings > Matching Configuration** marks configuration that has been changed from the shipped default, and names any dedupe pool that overrides it. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool responsible when it differs from the instance default. That pairing is what answers "why did these two findings deduplicate differently" without a support ticket: two tests on the same tool showing different algorithms is a pool override, not a fault. \ No newline at end of file diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md index 0bb85c958e6..5224b3447fb 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md @@ -19,7 +19,15 @@ Once the feature is enabled, **Global Component** will become available as an op ## Configuring Global Component Deduplication -Global Component can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Finding Workflow** (**Settings > Pro Settings > Deduplication Settings** on instances still using the previous menu layout; see [The Sidebar Menu](/navigation/pro__sidebar/)). +Global Component can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Deduplication Settings > Matching Configuration** (see [The Sidebar Menu](/navigation/pro__sidebar/)). + +> **A pooled Asset is bounded to its pool.** "Across all Assets" holds while an Asset is not in a +> [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) for the matching kind +> in question. Once it joins one, this algorithm matches its Findings only against that pool's +> other members, not instance-wide. Pooling therefore narrows this algorithm rather than leaving +> it untouched, which is worth knowing before creating a pool that happens to contain Assets +> relying on it. + ### Same-Tool diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md index 033cec7228b..becf3c43fe4 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md @@ -23,7 +23,15 @@ Once the feature is enabled, **Global Locations** becomes available as an option ## Configuring Global Locations Deduplication -Global Locations can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Finding Workflow** (**Settings > Pro Settings > Deduplication Settings** on instances still using the previous menu layout; see [The Sidebar Menu](/navigation/pro__sidebar/)). +Global Locations can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Deduplication Settings > Matching Configuration** (see [The Sidebar Menu](/navigation/pro__sidebar/)). + +> **A pooled Asset is bounded to its pool.** "Across all Assets" holds while an Asset is not in a +> [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) for the matching kind +> in question. Once it joins one, this algorithm matches its Findings only against that pool's +> other members, not instance-wide. Pooling therefore narrows this algorithm rather than leaving +> it untouched, which is worth knowing before creating a pool that happens to contain Assets +> relying on it. + When you select **Global Locations**, the Hash Code Fields selector is hidden (it does not apply) and a **Location Types** selector appears instead. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md b/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md index c6a018098a1..08cb50e74fe 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md @@ -18,14 +18,14 @@ Each of these previously produced a closed finding plus a "new" finding — losi ## Enabling Location Tracking -Location tracking is configured per tool under: -**Settings > Finding Workflow > Reimport Deduplication** (**Settings > Pro Settings > Deduplication Settings > Reimport Deduplication** on instances still using the previous menu layout) +Location tracking is configured per tool on **Settings > Deduplication Settings > Matching Configuration**. -1. Select the **Security Tool**. -2. Set the **Deduplication Algorithm** to **Hash Code**. Location tracking applies to the Hash Code algorithm only — tools with a reliable **Unique ID From Tool** already track movement through their stable IDs and do not need it. -3. Enable **Track findings as locations change**. +1. Find the tool's row and select its **Reimport** column. +2. Set the **Algorithm** to **Hash code**. Location tracking applies to that algorithm only: tools with a reliable **Unique ID From Tool** already track movement through their stable IDs and do not need it. +3. Tick **Track findings as locations change**. +4. Select **Review impact**, then **Apply**. -Saving the setting automatically triggers a background re-hash of the tool's existing findings (see [Enabling on Existing Data](#enabling-on-existing-data-upgrades) below), so findings imported before the toggle participate immediately. +The review step matters here. Turning tracking on or off changes which fields the reimport hash is built from, so every hash already stored for that tool becomes stale and the whole backlog is recomputed in the background. The review tells you how many findings that is before you commit, and until the recompute finishes the tool's findings are hashed under two different definitions and may not match each other. See [Enabling on Existing Data](#enabling-on-existing-data-upgrades) below. ## How Matching Works From 752daea4d855640c0e18a4a37e90a2c2bb02cee3 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Fri, 4 Sep 2026 09:37:27 -0500 Subject: [PATCH 08/24] docs(dedupe): the Apply Now ceiling, and drop em dashes from the Pro dedupe pages Apply Now runs inside the request and refuses above 50,000 findings, and reports a reason rather than a silent zero when deduplication is off instance-wide. Neither was written down, so a user with a large pool met the refusal with no way to anticipate it. Also replaces the em dashes these pages introduced with colons, parentheses and periods, per the house copy style. --- .../finding_deduplication/PRO__dedupe_pools.md | 7 +++++++ .../finding_deduplication/PRO__deduplication_tuning.md | 6 +++--- .../finding_deduplication/about_deduplication.md | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index a4f4d3e11ce..3d0f5624a82 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -84,6 +84,13 @@ Adding members applies to **future imports**. Findings already in DefectDojo are The acknowledgement is derived from the specific change it describes, so the preview you ran for adding Assets does not authorize a re-run, and a re-run preview goes stale if the pool changes underneath it. Preview the thing you are about to do. +Apply Now runs while you wait, so it is capped at 50,000 Findings across the pool. Above that +it refuses and tells you the count rather than running past the request. Narrow the pool, or +contact DefectDojo Support to have the re-run queued in the background. + +If deduplication is turned off for the instance, Apply Now reports that nothing was matched and +points you at System Settings rather than reporting a silent zero. + Reimport offers no Apply Now, for the reason above: pooling for reimport changes which formula a reimport uses, not which Findings it compares against, so there is no widened scope to re-run over existing Findings. Re-running deduplication across the pool is what the same tool and diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 211f355c731..99f0dd3c4fd 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -32,7 +32,7 @@ Select a tool's row to change its algorithm, its hash fields, or both. Because a **The two axes behave very differently, and the impact review says which one you are moving.** -- **Changing the algorithm** selects which stored value candidates are looked up by. Nothing is recomputed, and the change decides what the next import compares; existing duplicate links are left exactly as they are. The review warns you when the new algorithm would leave findings with no identity to match on at all — a tool whose findings carry no unique ID matches nothing once the algorithm requires one, and nothing errors when that happens. +- **Changing the algorithm** selects which stored value candidates are looked up by. Nothing is recomputed, and the change decides what the next import compares; existing duplicate links are left exactly as they are. The review warns you when the new algorithm would leave findings with no identity to match on at all: a tool whose findings carry no unique ID matches nothing once the algorithm requires one, and nothing errors when that happens. - **Changing the hash fields** changes the value stored on every finding of that tool, so every hash already stored for it becomes stale. Applying queues a background recompute of the tool's whole backlog. Until that finishes, the tool's findings are hashed under two different definitions and may not match each other. The review tells you how many findings will be recomputed before you commit to it. Two rules are enforced when you save a field selection, for the reasons in [Set-based Hash Code Fields](#set-based-hash-code-fields-vulnerability-ids-and-cwes) below: a vulnerability IDs field may stand on its own, and CWE fields may not be the only criteria. @@ -132,7 +132,7 @@ Reimport can completely discard Findings before they are recorded, so Reimport D ### Track Findings as Locations Change -A tool whose Reimport algorithm is **Hash Code** can also track findings as their locations change. With that enabled, a finding whose location moved between reimports — a line shift or file rename, a URL move, or a dependency version bump — is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created. +A tool whose Reimport algorithm is **Hash Code** can also track findings as their locations change. With that enabled, a finding whose location moved between reimports (a line shift or file rename, a URL move, or a dependency version bump) is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created. It is off by default and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Tick it in the tool's **Reimport** column on **Settings > Deduplication Settings > Matching Configuration**. Because it changes which fields the reimport hash is built from, applying it re-hashes the tool's existing findings in the background; the impact review tells you how many before you commit. @@ -150,7 +150,7 @@ A common situation when first tuning matching is having a large backlog of Findi If you make several changes in quick succession, each queues its own job. Allow the previous one to finish before evaluating results, especially when comparing Finding counts before and after. -> **Note for self-hosted Pro:** the job runs in the Celery worker pool. If workers are starved or backlogged, the re-hash takes longer than expected — check worker health if results do not appear within the timeframe you would expect for your instance size. +> **Note for self-hosted Pro:** the job runs in the Celery worker pool. If workers are starved or backlogged, the re-hash takes longer than expected. Check worker health if results do not appear within the timeframe you would expect for your instance size. **Changing the algorithm re-hashes nothing**, by design: it selects which already-stored value is compared, so there is nothing to recompute. It decides what the next import compares, and existing duplicate links are left as they are. diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index b9edacb2e94..8008848c985 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -81,7 +81,7 @@ DefectDojo Open Source supports four deduplication algorithms that can be select - **Unique ID From Tool or Hash Code**: Prefer the tool’s unique ID; fall back to hash when no matching unique ID is found. - **Legacy**: Historical algorithm with multiple conditions; only available in the Open Source version. -**DefectDojo Pro adds more.** [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) widen the scope of the existing algorithms to a chosen group of Assets, per matching kind, without changing how two Findings are compared. Two additional algorithms instead match across **all Assets** in the instance rather than within a single Asset or Engagement — **Global Component** (by component name and version) and **Global Vulnerability ID** (by CVE, GHSA, …). Both are off by default and enabled by DefectDojo Support. Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. +**DefectDojo Pro adds more.** [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) widen the scope of the existing algorithms to a chosen group of Assets, per matching kind, without changing how two Findings are compared. Two additional algorithms instead match across **all Assets** in the instance rather than within a single Asset or Engagement: **Global Component** (by component name and version) and **Global Vulnerability ID** (by CVE, GHSA, and similar). Both are off by default and enabled by DefectDojo Support. Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. ### An alternative to Deduplication: False Positive History From 6d218893521d5b2a28d538c1b62b3fbcd193c944 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Fri, 4 Sep 2026 10:16:47 -0500 Subject: [PATCH 09/24] refactor(dedupe): make candidate_qs keyword-only, and pin its forwarding on every finder build_candidate_scope_queryset took candidate_qs positionally while all five finders made it keyword-only. Every caller already passes it by keyword, so closing the asymmetry costs nothing and stops a positional service argument from ever landing in the wrong slot. The tests reached candidate_qs only through the hash path. Each of the five finders builds its own base queryset, so any of the others could have dropped the argument and stayed green. All five now assert the same property directly and in both directions: the derived scope does not reach the other product, and a supplied one does. The finders returning two maps (uid-or-hash, legacy) are checked on both, since forwarding could reach one and not the other. --- dojo/finding/deduplication.py | 2 +- unittests/test_dedupe_injectable_scope.py | 128 +++++++++++++++++++++- 2 files changed, 127 insertions(+), 3 deletions(-) diff --git a/dojo/finding/deduplication.py b/dojo/finding/deduplication.py index 3d8f48bea54..6caa86eda90 100644 --- a/dojo/finding/deduplication.py +++ b/dojo/finding/deduplication.py @@ -321,7 +321,7 @@ def are_locations_duplicates(new_finding, to_duplicate_finding): return False -def build_candidate_scope_queryset(test, mode="deduplication", service=None, candidate_qs=None): +def build_candidate_scope_queryset(test, mode="deduplication", service=None, *, candidate_qs=None): """ Build a queryset for candidate finding. diff --git a/unittests/test_dedupe_injectable_scope.py b/unittests/test_dedupe_injectable_scope.py index 184961645a3..1104738805c 100644 --- a/unittests/test_dedupe_injectable_scope.py +++ b/unittests/test_dedupe_injectable_scope.py @@ -19,6 +19,11 @@ from dojo.finding.deduplication import ( _dedupe_batch_hash_code, # noqa: PLC2701 build_candidate_scope_queryset, + find_candidates_for_deduplication_hash, + find_candidates_for_deduplication_legacy, + find_candidates_for_deduplication_uid_or_hash, + find_candidates_for_deduplication_unique_id, + find_candidates_for_reimport_legacy, match_batch_hash_code, ) from dojo.models import ( @@ -79,7 +84,7 @@ def _create_test(self, product_name, engagement_name): target_end=timezone.now(), ) - def _create_finding(self, test, title, hash_code=SHARED_HASH): + def _create_finding(self, test, title, hash_code=SHARED_HASH, *, unique_id=None, cwe=0): finding = Finding.objects.create( test=test, title=title, @@ -90,9 +95,10 @@ def _create_finding(self, test, title, hash_code=SHARED_HASH): reporter=self.testuser, active=True, verified=True, + cwe=cwe, ) # Assigning after create keeps Finding.save() from recomputing it. - Finding.objects.filter(pk=finding.pk).update(hash_code=hash_code) + Finding.objects.filter(pk=finding.pk).update(hash_code=hash_code, unique_id_from_tool=unique_id) finding.refresh_from_db() return finding @@ -229,3 +235,121 @@ def test_ordering_key_cannot_promote_a_newer_candidate(self): f"(newer id={newer.id}, target id={target.id})" ), ) + + # --- every finder forwards it ---------------------------------------- + # + # Five finders each build their own base queryset, so candidate_qs has to be threaded + # through every one of them and any of them could drop it and stay green. The hash finder + # is also reached through match_batch_hash_code above; it is called directly here so all + # five state the property the same way. + + def test_supplied_scope_reaches_the_hash_finder(self): + theirs = self._create_finding(self.test_b, "Hash scoped original") + mine = self._create_finding(self.test_a, "Hash scoped newer") + + default = find_candidates_for_deduplication_hash(self.test_a, [mine]) + self.assertNotIn(theirs.id, [c.id for c in default.get(SHARED_HASH, [])]) + + supplied = find_candidates_for_deduplication_hash( + self.test_a, [mine], candidate_qs=Finding.objects.all(), + ) + self.assertIn( + theirs.id, [c.id for c in supplied.get(SHARED_HASH, [])], + "the supplied scope did not reach the hash finder", + ) + + def test_supplied_scope_reaches_the_unique_id_finder(self): + """ + Forwarding, not just the hash path. + + Each finder builds its own base queryset, so ``candidate_qs`` has to be threaded + through every one of them. Covering only the hash finder would leave the unique-id and + legacy paths free to drop the argument and stay green. + """ + shared_uid = "SCOPE-UID-1" + theirs = self._create_finding(self.test_b, "Uid scoped original", unique_id=shared_uid) + mine = self._create_finding(self.test_a, "Uid scoped newer", unique_id=shared_uid) + + default = find_candidates_for_deduplication_unique_id(self.test_a, [mine]) + self.assertNotIn( + theirs.id, [candidate.id for group in default.values() for candidate in group], + "the derived scope must not reach the other product", + ) + + supplied = find_candidates_for_deduplication_unique_id( + self.test_a, [mine], candidate_qs=Finding.objects.all(), + ) + self.assertIn( + theirs.id, [candidate.id for candidate in supplied.get(shared_uid, [])], + "the supplied scope did not reach the unique-id finder", + ) + + def test_supplied_scope_reaches_the_legacy_finder(self): + """Same forwarding property for the title/CWE finder, which returns two maps.""" + theirs = self._create_finding(self.test_b, "Legacy scoped finding", cwe=79) + mine = self._create_finding(self.test_a, "Legacy scoped finding", cwe=79) + # The map is keyed by the PERSISTED title, and Finding.save() titlecases it, so the + # string passed to _create_finding is not the key. Read it back off the saved row. + stored_title = mine.title + + default_by_title, default_by_cwe = find_candidates_for_deduplication_legacy(self.test_a, [mine]) + self.assertNotIn( + theirs.id, [candidate.id for candidate in default_by_title.get(stored_title, [])], + "the derived scope must not reach the other product", + ) + self.assertNotIn(theirs.id, [candidate.id for candidate in default_by_cwe.get(79, [])]) + + by_title, by_cwe = find_candidates_for_deduplication_legacy( + self.test_a, [mine], candidate_qs=Finding.objects.all(), + ) + self.assertIn( + theirs.id, [candidate.id for candidate in by_title.get(stored_title, [])], + "the supplied scope did not reach the legacy finder's title map", + ) + self.assertIn( + theirs.id, [candidate.id for candidate in by_cwe.get(79, [])], + "the supplied scope did not reach the legacy finder's CWE map", + ) + + def test_supplied_scope_reaches_the_uid_or_hash_finder(self): + """The combined finder builds its own base queryset too, and returns two maps.""" + shared_uid = "SCOPE-UID-2" + theirs = self._create_finding(self.test_b, "Uid-or-hash scoped original", unique_id=shared_uid) + mine = self._create_finding(self.test_a, "Uid-or-hash scoped newer", unique_id=shared_uid) + + default_by_uid, default_by_hash = find_candidates_for_deduplication_uid_or_hash(self.test_a, [mine]) + self.assertNotIn(theirs.id, [c.id for c in default_by_uid.get(shared_uid, [])]) + self.assertNotIn(theirs.id, [c.id for c in default_by_hash.get(SHARED_HASH, [])]) + + by_uid, by_hash = find_candidates_for_deduplication_uid_or_hash( + self.test_a, [mine], candidate_qs=Finding.objects.all(), + ) + self.assertIn( + theirs.id, [c.id for c in by_uid.get(shared_uid, [])], + "the supplied scope did not reach the combined finder's unique-id map", + ) + self.assertIn( + theirs.id, [c.id for c in by_hash.get(SHARED_HASH, [])], + "the supplied scope did not reach the combined finder's hash map", + ) + + def test_supplied_scope_reaches_the_legacy_reimport_finder(self): + """Reimport normally scopes to the incoming test alone, which a supplied scope replaces.""" + shared_title = "Legacy reimport scoped finding" + theirs = self._create_finding(self.test_b, shared_title) + mine = self._create_finding(self.test_a, shared_title) + key = (shared_title.lower(), "High") + + default = find_candidates_for_reimport_legacy(self.test_a, [mine]) + self.assertNotIn( + theirs.id, [c.id for c in default.get(key, [])], + "reimport's derived scope is the incoming test, so the other product must not appear", + ) + + supplied = find_candidates_for_reimport_legacy( + self.test_a, [mine], candidate_qs=Finding.objects.all(), + ) + self.assertIn( + theirs.id, [c.id for c in supplied.get(key, [])], + "the supplied scope did not reach the legacy reimport finder", + ) From b519c0bc24be3cc323e47fcd93180945b34c1de1 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Sat, 5 Sep 2026 20:31:14 -0500 Subject: [PATCH 10/24] docs(dedupe): scope of false-positive history, pool overrides and reimport membership as they ship, upgrade and custom-role notes False-positive history now uses the same scope as deduplication, which narrows replication for instances with engagement-scoped deduplication: an isolated engagement is excluded from every other engagement's history and its own imports read only its own. About Deduplication says so; before, the search always covered the whole asset. The pools page and Deduplication Tuning both claimed a pool can give its members a different algorithm. No shipping path creates a pool-level configuration row, so both now say per-pool overrides are not yet available and every member uses the instance default. The pools page offered a reimport membership kind and described it as selecting the formula a reimport uses. The resolver never consults a pool for that kind, so the membership did nothing; the kind is withdrawn from the page and the text explains why, and that findings a reimport creates still deduplicate across the pool. Deduplication Tuning gains an Upgrading section: the cutover from the tuner pages is one-way and a database backup should precede the upgrade. The pools page gains a custom-roles note: the upgrade carries Edit Tuner to all four pool permissions and View Tuner to View Dedupe Pool, and only roles created afterwards need the grants made by hand. --- .../PRO__dedupe_pools.md | 36 ++++++++----------- .../PRO__deduplication_tuning.md | 23 +++++++++--- .../about_deduplication.md | 10 +++++- 3 files changed, 43 insertions(+), 26 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index 3d0f5624a82..ebfa8ad121b 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -32,7 +32,7 @@ Pools and the global algorithms solve the same problem at different scales, and > instance-wide. If you want a tool to keep matching across every Asset, leave its Assets out of > a pool for that kind. -A pool does not change **how** two Findings are compared. It changes **which** Findings are eligible to be compared at all. How they are compared is set per tool on [Matching Configuration](/triage_findings/finding_deduplication/pro__deduplication_tuning/), which a pool can override for its own members. +A pool does not change **how** two Findings are compared. It changes **which** Findings are eligible to be compared at all. How they are compared is set per tool on [Matching Configuration](/triage_findings/finding_deduplication/pro__deduplication_tuning/). Per-pool overrides of that configuration are not yet available; today every member uses the instance default. ## Membership is per matching kind @@ -40,22 +40,15 @@ An Asset joins a pool for one **matching kind** at a time, and can be in at most * **Same tool.** Findings from the same scanner deduplicate across the pool's Assets. * **Cross tool.** Findings from different scanners deduplicate across the pool's Assets. -* **Reimport.** Selects the matching formula a reimport uses. It does **not** widen what a reimport itself compares: a reimport matches inside its own Test. Findings it creates are still deduplicated across the pool under same tool and cross tool. +Membership is offered for these two kinds only. There is no reimport kind to pool for, and that +is deliberate rather than an omission: a reimport matches inside its own Test, so pooling could +never widen what it compares, and the one thing a reimport membership could do (select a +per-pool reimport formula) depends on per-pool overrides, which are not yet available. Offering +the kind would accept a membership that changes nothing. -That last one is worth reading twice, because the obvious reading is wrong. Two things are true -at once: - -* A reimport's **own** matching stays inside its Test. That matching decides whether an incoming - Finding updates an existing one, is created fresh, or whether a Finding missing from the scan - gets closed, so it is scoped to the Test the scan is authoritative over. Pooling Assets for - reimport does not change that. -* Findings a reimport **creates** are then deduplicated like any other new Finding, under same - tool and cross tool. If the Asset is pooled for those kinds, that deduplication reaches across - the pool. - -So pooling does affect reimports; it just affects what happens to the Findings a reimport -produces, rather than what the reimport itself compares against. Only same tool and cross tool -change scope. +Pooling still affects reimports, in the way that matters: the Findings a reimport **creates** +are deduplicated like any other new Finding, under same tool and cross tool, and if the Asset is +pooled for those kinds that deduplication reaches across the pool. If you try to add an Asset that already matches within another pool for that kind, DefectDojo refuses the change and names the pool holding it. Take it out of that pool first if the move is deliberate. @@ -91,11 +84,6 @@ contact DefectDojo Support to have the re-run queued in the background. If deduplication is turned off for the instance, Apply Now reports that nothing was matched and points you at System Settings rather than reporting a silent zero. -Reimport offers no Apply Now, for the reason above: pooling for reimport changes which formula a -reimport uses, not which Findings it compares against, so there is no widened scope to re-run -over existing Findings. Re-running deduplication across the pool is what the same tool and -cross tool kinds do, and Findings a reimport created are included in that like any other. - ## Where originals collect **Where originals collect** decides which Finding a pool's duplicates point at. @@ -144,3 +132,9 @@ Pools are governed by four global permissions, granted through global roles: | **Delete Dedupe Pool** | Delete a pool | Membership lists and every preview are filtered to the Assets you can read, so the numbers a preview reports are the numbers for **your** visibility, not the instance's. + +### Custom roles on upgrade + +Matching Configuration used to be read through the tuner's **View Tuner** permission and edited through **Edit Tuner**. Neither gates the new pages: reading pools and Matching Configuration needs **View Dedupe Pool**, and editing a matching formula needs **Edit Dedupe Pool**. + +The upgrade carries existing grants over. A custom role holding **Edit Tuner** receives all four pool permissions; a role holding only **View Tuner** receives **View Dedupe Pool**, so a view-only tuner role keeps its read access without gaining any write. Built-in roles are re-seeded from the shipped definitions. Only a custom role created **after** the upgrade needs the pool permissions granted explicitly by an administrator. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 99f0dd3c4fd..f286a588176 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -20,7 +20,7 @@ This page replaced three separate pages (Same Tool Deduplication, Cross Tool Ded - **Cross tool**: how findings from different tools are matched against each other. - **Reimport**: which formula a reimport uses inside its own test. -A tool's row shows the algorithm in force for each kind, how many hash fields it uses, whether anyone has changed it from the shipped default, and whether a [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) overrides it. +A tool's row shows the algorithm in force for each kind, how many hash fields it uses, and whether anyone has changed it from the shipped default. ### Changing a tool's matching @@ -37,7 +37,7 @@ Select a tool's row to change its algorithm, its hash fields, or both. Because a Two rules are enforced when you save a field selection, for the reasons in [Set-based Hash Code Fields](#set-based-hash-code-fields-vulnerability-ids-and-cwes) below: a vulnerability IDs field may stand on its own, and CWE fields may not be the only criteria. -> **Hash fields are set on the instance default, not per pool.** A [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) can give its members a different **algorithm**, but not a different field list. A finding stores one hash, and the classic UI, the v2 API and CSV exports all read that same value, so a pool-specific field list would change what every other view of that finding shows. +> **One configuration per tool.** Matching Configuration is instance-wide: every Asset uses the same algorithm and field list for a given tool, including Assets in a [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/). Per-pool algorithm overrides are not yet available. Hash fields will stay instance-wide even when they are, because a finding stores one hash and the classic UI, the v2 API and CSV exports all read that same value, so a pool-specific field list would change what every other view of that finding shows. ## Same Tool Deduplication @@ -138,6 +138,21 @@ It is off by default and applies only to the Hash Code reimport algorithm (tools See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) for how the matching works, what is preserved, and guidance for enabling it on large instances. +## Upgrading from the tuner pages + +The three tuner deduplication pages were replaced by Matching Configuration, and the upgrade +moves their configuration into it. The migration copies every per-tool entry from the tuner's +three stored settings into Matching Configuration rows, marks the ones you had changed from the +shipped defaults as edited, and then removes the tuner's stored settings. + +**That last step is one-way. Take a database backup before upgrading.** The migration cannot be +reversed: rolling back to the previous release means restoring the backup, not running the +migration in reverse. Nothing about your matching behaviour changes at upgrade time; every tool +keeps the algorithm and fields it had. The backup is for the case where you need the previous +release back for some other reason. + +Permission changes for custom roles are described under [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/#custom-roles-on-upgrade). + ## Running Deduplication Retroactively on Existing Data A common situation when first tuning matching is having a large backlog of Findings that were imported *before* the configuration changed. What happens to them depends on which axis you changed. @@ -176,6 +191,6 @@ By tuning deduplication settings to your specific tools, you can significantly r ## Where a tool's matching came from -A tool's row on **Settings > Deduplication Settings > Matching Configuration** marks configuration that has been changed from the shipped default, and names any dedupe pool that overrides it. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool responsible when it differs from the instance default. +A tool's row on **Settings > Deduplication Settings > Matching Configuration** marks configuration that has been changed from the shipped default. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool its Asset matches within. -That pairing is what answers "why did these two findings deduplicate differently" without a support ticket: two tests on the same tool showing different algorithms is a pool override, not a fault. \ No newline at end of file +That pairing is what answers "why did these two findings deduplicate differently" without a support ticket: the panel names the scope each test matched within, so two tests on the same tool with different results point at different pools rather than at a fault. \ No newline at end of file diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index 8008848c985..db1b2b19e65 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -85,7 +85,15 @@ DefectDojo Open Source supports four deduplication algorithms that can be select ### An alternative to Deduplication: False Positive History -Instances that deliberately do **not** deduplicate can instead use [False Positive History](/triage_findings/finding_deduplication/false_positive_history/), which automatically marks an incoming Finding as a false positive when a matching Finding in the same Asset was already triaged that way. It is **mutually exclusive with Deduplication** — DefectDojo does not allow both to be enabled — and it is still marked experimental. +Instances that deliberately do **not** deduplicate can instead use [False Positive History](/triage_findings/finding_deduplication/false_positive_history/), which automatically marks an incoming Finding as a false positive when a matching Finding in the same Asset was already triaged that way. It is **mutually exclusive with Deduplication** (DefectDojo does not allow both to be enabled) and it is still marked experimental. + +**In DefectDojo Pro, False Positive History uses the same scope as deduplication.** Three consequences follow, and the first two narrow replication for instances that use engagement-scoped deduplication: + +* An Engagement that has deduplication scoped to itself is **excluded** from every other Engagement's false positive history in the same Asset. +* An import into such an Engagement reads **only that Engagement's** history. +* An Asset in a [Dedupe Pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) replicates a false positive across the pool's Assets, not only within itself. + +Previously the search always covered the whole Asset regardless of engagement scoping. ## How endpoints are assessed per algorithm From 743ce14767f7fa81210cf03e63c1aea780f84d89 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Sun, 6 Sep 2026 13:14:23 -0500 Subject: [PATCH 11/24] docs(dedupe): 3.3.0 changelog entries, the Apply Now ceiling, disabling cross tool Adds the 3.3.0 section to the Pro changelog with the two behaviour changes the matching-configuration work introduces (false-positive history follows deduplication scope; Global Component and Global Locations matching bounded to the pool for a pooled Asset) and the two feature lines. The dedupe-pools page states the lowered Apply Now ceiling (10,000 Findings), and the tuning page says what setting a tool's cross-tool algorithm back to Disabled does. --- docs/content/releases/pro/changelog.md | 12 ++++++++++++ .../finding_deduplication/PRO__dedupe_pools.md | 2 +- .../PRO__deduplication_tuning.md | 2 ++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index e06e6944894..9bc1472493c 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -16,6 +16,18 @@ You can subscribe to these release notes with the [RSS feed](/releases/pro/chang For Open Source release notes, please see the [Releases page on GitHub](https://github.com/DefectDojo/django-DefectDojo/releases), or alternatively consult the Open Source [upgrade notes](/releases/os_upgrading/upgrading_guide/). +## September 2026: v3.3 + +### September 8, 2026: v3.3.0 + +New features: +* **(Deduplication)** Added Dedupe Pools: group the Assets that should deduplicate against each other, choose where their originals collect, preview what a membership change would link, and re-run deduplication over the Findings already in scope with Apply Now. +* **(Deduplication)** The three deduplication tuning pages are now one Matching Configuration page: every tool listed once, with its same-tool, cross-tool and reimport matching side by side, and every change previewed before it is saved. + +Behavior changes: +* **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. Instances using false-positive history across such Engagements see narrower replication than before. +* **(Deduplication)** For an Asset in a Dedupe Pool, Global Component and Global Locations matching is bounded to the pool rather than the whole instance. + ## August 2026: v3.2 ### August 31, 2026: v3.2.400 diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index ebfa8ad121b..085521448c2 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -77,7 +77,7 @@ Adding members applies to **future imports**. Findings already in DefectDojo are The acknowledgement is derived from the specific change it describes, so the preview you ran for adding Assets does not authorize a re-run, and a re-run preview goes stale if the pool changes underneath it. Preview the thing you are about to do. -Apply Now runs while you wait, so it is capped at 50,000 Findings across the pool. Above that +Apply Now runs while you wait, so it is capped at 10,000 Findings across the pool. Above that it refuses and tells you the count rather than running past the request. Narrow the pool, or contact DefectDojo Support to have the re-run queued in the background. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index f286a588176..d9655c51470 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -111,6 +111,8 @@ To enable Cross Tool Deduplication, select the tool's **Cross tool** column on * Cross Tool Deduplication supports the Hash Code algorithm, which is suitable for most workflows, as different tools rarely share compatible unique identifiers. For SCA tools reporting the same dependencies, [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) is also available as a cross-tool option (off by default). +To turn Cross Tool Deduplication off again for a tool, set its algorithm back to **Disabled**. That also clears the tool's cross-tool hash fields, and the cross-tool hashes already stored for its Findings are recomputed to empty in the background. Until that finishes, other tools' imports can still match against those Findings. + Note that Cross Tool Deduplication is also scoped to individual Assets only. ## Reimport Deduplication From f08a3a07f5e9a5f284dcd19a2bd63865d2dd1836 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 7 Sep 2026 04:16:48 -0500 Subject: [PATCH 12/24] docs(dedupe): navigation paths, permissions, scope statements and procedures for the new pages Every deduplication page now gives the Finding Workflow path first, with the previous menu's path in parentheses, the way the enabling page does. The sidebar page no longer describes the three retired tuner pages. The global algorithm pages describe the hub procedure instead of the tuner form, say that a pooled Asset bounds them to its pool rather than that matching is always global, and no longer claim an algorithm change recomputes hashes. The tuning page drops the Support-only and Assets-only statements that pools and the hub made false, and explains that every tool's cross-tool cell starts as Disabled. The pools page names the Dedupe Pool permissions the way the roles editor does, notes that the designated Asset must be a member and that the engine's age check still applies, and says the subtree toggle pools only what the caller can read. The changelog and the overview count three global algorithms and scope false-positive replication to same-tool matching. --- docs/content/navigation/PRO__sidebar.md | 4 +-- docs/content/releases/pro/changelog.md | 4 +-- .../PRO__dedupe_pools.md | 29 +++++++++--------- .../PRO__deduplication_tuning.md | 30 +++++++++---------- .../PRO__global_component_deduplication.md | 15 +++++----- .../PRO__global_locations_deduplication.md | 15 +++++----- .../PRO__location_drift_matching.md | 2 +- .../about_deduplication.md | 4 +-- 8 files changed, 50 insertions(+), 53 deletions(-) diff --git a/docs/content/navigation/PRO__sidebar.md b/docs/content/navigation/PRO__sidebar.md index 19ef9c5f409..0dcd5c9c830 100644 --- a/docs/content/navigation/PRO__sidebar.md +++ b/docs/content/navigation/PRO__sidebar.md @@ -128,7 +128,7 @@ Settings is divided into eight groups, named for what you are trying to do rathe | **System** | System Settings, Appearance, Announcement Banner, Login Banner, Email | | **UI Defaults** | Form Configuration, Layout Defaults | | **Users & Permissions** | Users, Groups, Roles | -| **Finding Workflow** | The three Deduplication pages, Finding Enrichment, Service Level Agreements, Prioritization Engines, Mitigation Policies | +| **Finding Workflow** | Dedupe Pools, Matching Configuration, Finding Enrichment, Service Level Agreements, Prioritization Engines, Mitigation Policies | | **Configuration** | Environments, Regulations, Note Types, Test Types, CI/CD Infrastructure, Tool Types, Tool Configurations | | **Notifications** | Notification Events, Notification Webhooks | | **Operations** | Audit Logs, Usage Logs, Schedules, Celery Status, and on DefectDojo Cloud, Message Portal, Firewall Rules, Maintenance Windows | @@ -180,7 +180,7 @@ If you are used to the previous layout: | Settings → Users → All Users / New User | Settings → Users & Permissions → Users | | Settings → Users → All Groups / New Group | Settings → Users & Permissions → Groups | | Settings → Users → Roles | Settings → Users & Permissions → Roles | -| Settings → Pro Settings → Deduplication Settings → *(three pages)* | Settings → Finding Workflow → Same Tool / Cross Tool / Reimport Deduplication | +| Settings → Pro Settings → Deduplication Settings → *(three pages)* | Settings → Finding Workflow → Matching Configuration (one page covering same-tool, cross-tool and reimport matching), beside Dedupe Pools | | Settings → Pro Settings → Finding Enrichment Settings | Settings → Finding Workflow → Finding Enrichment | | Settings → Configuration → Service Level Agreements | Settings → Finding Workflow → Service Level Agreements | | Settings → Configuration → Prioritization Engines | Settings → Finding Workflow → Prioritization Engines | diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 9bc1472493c..8fdb9ea6e49 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -25,8 +25,8 @@ New features: * **(Deduplication)** The three deduplication tuning pages are now one Matching Configuration page: every tool listed once, with its same-tool, cross-tool and reimport matching side by side, and every change previewed before it is saved. Behavior changes: -* **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. Instances using false-positive history across such Engagements see narrower replication than before. -* **(Deduplication)** For an Asset in a Dedupe Pool, Global Component and Global Locations matching is bounded to the pool rather than the whole instance. +* **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. An Asset in a Dedupe Pool replicates its false positives across the pool for same-tool matching. Instances using false-positive history across such Engagements see narrower replication than before. +* **(Deduplication)** For an Asset in a Dedupe Pool, Global Component, Global Vulnerability ID and Global Locations matching is bounded to the pool rather than the whole instance. ## August 2026: v3.2 diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index 085521448c2..56a985933a7 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -11,7 +11,7 @@ Pools are for the case where the same thing is genuinely deployed in several pla A pool may span Organizations. You only ever see the members you have access to, and a member you cannot read is shown as a placeholder rather than hidden, so a pool never looks smaller than it is. -Find pools at **Settings \> Deduplication Settings \> Dedupe Pools**. Matching Configuration sits beside it in the same group. +Find pools at **Settings \> Finding Workflow \> Dedupe Pools** (**Settings \> Pro Settings \> Deduplication Settings \> Dedupe Pools** on instances still using the previous menu layout). Matching Configuration sits beside it in the same group. ## Pools vs. the global algorithms @@ -24,8 +24,8 @@ Pools and the global algorithms solve the same problem at different scales, and | **Global Locations** | Every Asset in the instance | Package URL, or URL for DAST Findings | As above, keyed on the full location under the Locations data model | > **Pooling an Asset narrows a global algorithm rather than leaving it alone.** The two are not -> independent settings at different blast radii. While an Asset is unpooled, Global Component and -> Global Locations reach the whole instance as described above. Once that Asset joins a pool for +> independent settings at different blast radii. While an Asset is unpooled, Global Component, +> Global Vulnerability ID and Global Locations reach the whole instance as described above. Once that Asset joins a pool for > the matching kind in question, those algorithms are **bounded to the pool**: its Findings match > only against the pool's other members, not instance-wide. So creating a pool that happens to > contain an Asset running Global Component silently narrows matching that was previously @@ -56,7 +56,7 @@ If you try to add an Asset that already matches within another pool for that kin A new pool has no members, so nothing about deduplication changes until you add some. This is deliberate: creating a pool to look at it is safe. -1. Open **Settings \> Deduplication Settings \> Dedupe Pools**. +1. Open **Settings \> Finding Workflow \> Dedupe Pools**. 2. Enter a name under **New pool** and click **Create Pool**. 3. Select the pool, then pick the **Matching kind** you want to configure. @@ -89,7 +89,7 @@ points you at System Settings rather than reporting a silent zero. **Where originals collect** decides which Finding a pool's duplicates point at. * **Oldest finding wins.** The default, and what deduplication has always done. -* **Designated Asset, then oldest.** Duplicates point at the chosen Asset wherever it has a matching Finding, and at the oldest Finding otherwise. +* **Designated asset, then oldest.** Duplicates point at the chosen Asset where it has a matching Finding old enough to be the original (the engine's age check still applies), and at the oldest Finding otherwise. The designated Asset has to be a member of the pool for same-tool or cross-tool matching: add it first, then designate it. Use the second when one Asset is the place your team actually works, and you want the originals to land there rather than wherever the earliest scan happened to run. @@ -101,7 +101,7 @@ Changing the placement affects **new** matches. Existing duplicates keep their c Removing a member also applies to future imports. Findings already linked **keep their links**, including links to an original in an Asset the removed Asset no longer shares a pool with. -That is the safe default, but it leaves duplicates pointing outside their own Asset. When you want those cleaned up, **Reset external links** clears exactly those links. It never deletes anything: a Finding whose link is cleared goes back to being an ordinary active Finding. +That is the safe default, but it leaves duplicates pointing outside their own Asset. When you want those cleaned up, **Reset External Links** clears exactly those links. Like Apply Now it is preview-gated: **Preview Cleanup** counts the links first and hands back the acknowledgement the reset requires. It never deletes anything: a Finding whose link is cleared goes back to being an ordinary active Finding. ## Pooling from the Asset page @@ -111,6 +111,7 @@ The panel also offers **Pool this Asset and everything under it**, which pools t * It follows **parent relationships only**. A reference between two Assets is not containment, so an Asset that merely uses another is not pulled in. * It **skips rather than steals**. A descendant already pooled elsewhere for that kind is reported back as left alone, not moved. +* It pools only what you can read. A descendant you do not have access to is neither pooled nor named; the panel reports how many were left alone for that reason. A membership created this way is marked **from parent**. **Untoggle subtree** removes only the memberships the toggle created; a membership someone added by hand survives it. @@ -122,19 +123,19 @@ Like the subtree toggle, it counts an Asset already pooled elsewhere for that ki ## Permissions -Pools are governed by four global permissions, granted through global roles: +Pools are governed by the **Dedupe Pool** row of the roles editor, four global permissions that take effect only through a global role: -| Permission | Allows | +| Dedupe Pool column | Allows | | --- | --- | -| **View Dedupe Pool** | See pools and their members | -| **Add Dedupe Pool** | Create a pool | -| **Edit Dedupe Pool** | Change membership, placement, and run Apply Now | -| **Delete Dedupe Pool** | Delete a pool | +| **View** | See pools, their members, and Matching Configuration | +| **Add** | Create a pool | +| **Edit** | Change membership, placement, matching formulas, and run Apply Now | +| **Delete** | Delete a pool | Membership lists and every preview are filtered to the Assets you can read, so the numbers a preview reports are the numbers for **your** visibility, not the instance's. ### Custom roles on upgrade -Matching Configuration used to be read through the tuner's **View Tuner** permission and edited through **Edit Tuner**. Neither gates the new pages: reading pools and Matching Configuration needs **View Dedupe Pool**, and editing a matching formula needs **Edit Dedupe Pool**. +Matching Configuration used to be read through the tuner's **View Tuner** permission and edited through **Edit Tuner**. Neither gates the new pages: reading pools and Matching Configuration needs **Dedupe Pool: View**, and editing a matching formula needs **Dedupe Pool: Edit**. Both are global permissions, so a role scoped to an Asset or Organization grants nothing here. -The upgrade carries existing grants over. A custom role holding **Edit Tuner** receives all four pool permissions; a role holding only **View Tuner** receives **View Dedupe Pool**, so a view-only tuner role keeps its read access without gaining any write. Built-in roles are re-seeded from the shipped definitions. Only a custom role created **after** the upgrade needs the pool permissions granted explicitly by an administrator. +The upgrade carries existing grants over. A custom role holding **Edit Tuner** receives all four Dedupe Pool permissions; a role holding only **View Tuner** receives **Dedupe Pool: View**, so a view-only tuner role keeps its read access without gaining any write. Built-in roles are re-seeded from the shipped definitions. A custom role that held neither tuner permission receives nothing, and so does a custom role created **after** the upgrade: both need the Dedupe Pool permissions granted explicitly by an administrator. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index d9655c51470..4899ac564f1 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -12,7 +12,7 @@ Deduplication Tuning is a DefectDojo Pro feature that gives you fine-grained con ## Matching Configuration -In DefectDojo Pro, matching is configured at **Settings > Deduplication Settings > Matching Configuration**, beside [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) in the same group. +In DefectDojo Pro, matching is configured at **Settings > Finding Workflow > Matching Configuration** (**Settings > Pro Settings > Deduplication Settings > Matching Configuration** on instances still using the previous menu layout), beside [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) in the same group. This page replaced three separate pages (Same Tool Deduplication, Cross Tool Deduplication and Reimport Deduplication). Bookmarks to those pages redirect here. Instead of picking a tool from a dropdown on one of three pages, every tool is listed once with a column for each of the three matching kinds: @@ -24,7 +24,7 @@ A tool's row shows the algorithm in force for each kind, how many hash fields it ### Changing a tool's matching -Select a tool's row to change its algorithm, its hash fields, or both. Because a matching change decides which findings are treated as the same finding, it is not saved directly: +In a tool's row, select the algorithm shown under the matching kind you want to change to edit that kind's algorithm, its hash fields, or both. Because a matching change decides which findings are treated as the same finding, it is not saved directly: 1. Choose the new algorithm, the new hash fields, or both. The page explains what each algorithm matches on. 2. Select **Review impact**. DefectDojo reports how many findings of that tool are in scope, and warns you about the two things that are easy to miss (see below). @@ -43,14 +43,14 @@ Two rules are enforced when you save a field selection, for the reasons in [Set- Same Tool Deduplication is enabled by default for all security tool parsers. This ensures findings from consecutive scans using the same tool are properly deduplicated. -To adjust Same Tool Deduplication, select the tool's **Same tool** column on **Settings > Deduplication Settings > Matching Configuration** and follow the review-and-confirm steps above. +To adjust Same Tool Deduplication, select the tool's **Same tool** column on **Settings > Finding Workflow > Matching Configuration** and follow the review-and-confirm steps above. ### Available Deduplication Algorithms DefectDojo Pro offers the following deduplication methods for same-tool deduplication: #### Hash Code -Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Deduplication Settings > Matching Configuration** shows how many fields make up its hash, and selecting the row lets you change them. +Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Finding Workflow > Matching Configuration** shows how many fields make up its hash, and selecting the row lets you change them. ##### Content Fingerprint @@ -71,10 +71,10 @@ This algorithm can be useful when working with SAST scanners, or situations wher Attempts to use the tool's unique ID first, then falls back to the hash code if no unique ID is available. This provides the most flexible deduplication option. #### Global Component -Matches findings by component name and version across **all Assets** in the instance, rather than within a single Asset or Engagement. Intended for SCA tools where the same vulnerable dependency appears in many Assets. This algorithm is off by default and must be enabled by DefectDojo Support. See [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) for details. +Matches findings by component name and version across **all Assets** in the instance, rather than within a single Asset or Engagement. Intended for SCA tools where the same vulnerable dependency appears in many Assets. Gated behind a feature flag and off by default; a superuser can turn it on from **Settings > Feature Flags**. An Asset in a [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) for the matching kind is bounded to that pool instead of the whole instance. See [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) for details. #### Global Vulnerability ID -Matches findings by their **vulnerability IDs** (CVE, GHSA, …) across **all Assets** in the instance, rather than within a single Asset or Engagement. Intended for tools that report the same CVE across many Assets. Off by default and enabled by DefectDojo Support. +Matches findings by their **vulnerability IDs** (CVE, GHSA, …) across **all Assets** in the instance, rather than within a single Asset or Engagement. Intended for tools that report the same CVE across many Assets. Gated behind a feature flag and off by default; a superuser can turn it on from **Settings > Feature Flags**. Like the other instance-wide algorithms it is bounded to the Asset's dedupe pool when the Asset is in one for that matching kind. > **Two tools on the same instance-wide algorithm become mutual deduplication candidates.** When two *different* tools are both configured with an instance-wide algorithm (Global Component, or Global Vulnerability ID), their findings share a constant grouping hash, so a finding from either tool is considered for deduplication against the other on that shared dimension (component, or vulnerability ID). This is the intended cross-tool behavior — enable it only when you want those tools to dedupe together. @@ -107,23 +107,21 @@ The `_partial` and `_subset` fields are compared per finding pair rather than fo Cross Tool Deduplication is disabled by default, as deduplication between different security tools requires careful configuration due to variations in how tools report the same vulnerabilities. -To enable Cross Tool Deduplication, select the tool's **Cross tool** column on **Settings > Deduplication Settings > Matching Configuration**, change the algorithm to Hash Code, and select the fields the hash should be built from. +Every tool's **Cross tool** cell reads **Disabled** until you enable it. To enable Cross Tool Deduplication for a tool, select that cell on **Settings > Finding Workflow > Matching Configuration**, change the algorithm to Hash Code, and select the fields the hash should be built from. The editor will not save an algorithm with no fields, because cross-tool matching builds its hash from those fields and nothing else. Cross Tool Deduplication supports the Hash Code algorithm, which is suitable for most workflows, as different tools rarely share compatible unique identifiers. For SCA tools reporting the same dependencies, [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/) is also available as a cross-tool option (off by default). To turn Cross Tool Deduplication off again for a tool, set its algorithm back to **Disabled**. That also clears the tool's cross-tool hash fields, and the cross-tool hashes already stored for its Findings are recomputed to empty in the background. Until that finishes, other tools' imports can still match against those Findings. -Note that Cross Tool Deduplication is also scoped to individual Assets only. +Cross Tool Deduplication is scoped to the Asset, or to the Asset's [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) when it is in one for cross-tool matching. ## Reimport Deduplication **⚠️ Reimport processes can completely discard Findings before they are recorded. This can lead to data loss if set incorrectly, so Reimport Deduplication settings should be adjusted with caution.** -Reimport Deduplication Settings can be used to set an algorithm for Universal Parsers, or for a Generic Findings Import Parser. +Reimport Deduplication can be adjusted for any tool listed on Matching Configuration. Universal Parsers and the Generic Findings Import parser are where it is most often changed, because what they emit varies per installation; for a shipped parser the default reimport formula is usually right. -Reimport Deduplication cannot be adjusted for other tools by default. Users who want to adjust the Reimport Deduplication algorithm for other tools in their instance should reach out to [DefectDojo Support](mailto:support@defectdojo.com) for assistance. - -To configure Reimport Deduplication, select the tool's **Reimport** column on **Settings > Deduplication Settings > Matching Configuration**. +To configure Reimport Deduplication, select the tool's **Reimport** column on **Settings > Finding Workflow > Matching Configuration**. The following algorithm options are available for Reimport Deduplication: - Hash Code @@ -136,7 +134,7 @@ Reimport can completely discard Findings before they are recorded, so Reimport D A tool whose Reimport algorithm is **Hash Code** can also track findings as their locations change. With that enabled, a finding whose location moved between reimports (a line shift or file rename, a URL move, or a dependency version bump) is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created. -It is off by default and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Tick it in the tool's **Reimport** column on **Settings > Deduplication Settings > Matching Configuration**. Because it changes which fields the reimport hash is built from, applying it re-hashes the tool's existing findings in the background; the impact review tells you how many before you commit. +It is off by default and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Tick it in the tool's **Reimport** column on **Settings > Finding Workflow > Matching Configuration**. Because it changes which fields the reimport hash is built from, applying it re-hashes the tool's existing findings in the background; the impact review tells you how many before you commit. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) for how the matching works, what is preserved, and guidance for enabling it on large instances. @@ -145,7 +143,7 @@ See [Location Drift Matching](/triage_findings/finding_deduplication/pro__locati The three tuner deduplication pages were replaced by Matching Configuration, and the upgrade moves their configuration into it. The migration copies every per-tool entry from the tuner's three stored settings into Matching Configuration rows, marks the ones you had changed from the -shipped defaults as edited, and then removes the tuner's stored settings. +shipped defaults as changed, and then removes the tuner's stored settings. **That last step is one-way. Take a database backup before upgrading.** The migration cannot be reversed: rolling back to the previous release means restoring the backup, not running the @@ -173,7 +171,7 @@ If you make several changes in quick succession, each queues its own job. Allow If you need existing Findings re-evaluated against a new configuration, use a [dedupe pool's](/triage_findings/finding_deduplication/pro__dedupe_pools/) **Apply Now**, which re-runs deduplication over the Findings already in scope and reports what it would link before it does anything. -> **Feature flags do not gate an existing configuration.** A tool's saved matching configuration stays in effect for as long as it is configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change a tool's behavior, change its algorithm on **Settings > Deduplication Settings > Matching Configuration**. +> **Feature flags do not gate an existing configuration.** A tool's saved matching configuration stays in effect for as long as it is configured; turning off a related feature flag does **not** retroactively revert that tool to default deduplication. To change a tool's behavior, change its algorithm on **Settings > Finding Workflow > Matching Configuration**. ## Deduplication Best Practices @@ -193,6 +191,6 @@ By tuning deduplication settings to your specific tools, you can significantly r ## Where a tool's matching came from -A tool's row on **Settings > Deduplication Settings > Matching Configuration** marks configuration that has been changed from the shipped default. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool its Asset matches within. +A tool's row on **Settings > Finding Workflow > Matching Configuration** marks configuration that has been changed from the shipped default. A test's **Matching Policy** panel shows the same thing from the other direction: the algorithm actually in force for that test, and the pool its Asset matches within. That pairing is what answers "why did these two findings deduplicate differently" without a support ticket: the panel names the scope each test matched within, so two tests on the same tool with different results point at different pools rather than at a fault. \ No newline at end of file diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md index 5224b3447fb..e31967bf24a 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md @@ -19,7 +19,7 @@ Once the feature is enabled, **Global Component** will become available as an op ## Configuring Global Component Deduplication -Global Component can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Deduplication Settings > Matching Configuration** (see [The Sidebar Menu](/navigation/pro__sidebar/)). +Global Component can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Finding Workflow > Matching Configuration** (**Settings > Pro Settings > Deduplication Settings > Matching Configuration** on instances still using the previous menu layout; see [The Sidebar Menu](/navigation/pro__sidebar/)). > **A pooled Asset is bounded to its pool.** "Across all Assets" holds while an Asset is not in a > [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) for the matching kind @@ -33,10 +33,9 @@ Global Component can be applied to Same-Tool Deduplication, Cross-Tool Deduplica Use Same-Tool Deduplication with the Global Component algorithm when you want to deduplicate findings from a single SCA tool across multiple Assets. -1. Open the **Same Tool Deduplication** tab. -2. Select the SCA tool from the **Security Tool** dropdown (for example, `Dependency Track Finding Packaging Format (FPF) Export`). -3. Set the **Deduplication Algorithm** to **Global Component**. -4. Submit the form. +1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Same tool** cell. +3. Set the **Algorithm** to **Global Component**. +4. Review the impact and confirm. Hash Code Fields are not used by this algorithm and are hidden when it is selected. @@ -46,7 +45,7 @@ Use Cross-Tool Deduplication with the Global Component algorithm when you want t Cross-tool matching requires Global Component to be configured on **each** tool that should participate. -1. Open the **Cross Tool Deduplication** tab. +1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Cross tool** cell. 2. For each tool to include: select it from the **Security Tool** dropdown, set the algorithm to **Global Component**, and submit. ## How Matching Works @@ -58,7 +57,7 @@ A new Finding is marked as a duplicate of an existing Finding when: Component version matching is exact. A Finding for `timespan@2.3.0` will **not** deduplicate against one for `timespan@2.3.1`. -The Engagement-scoped deduplication setting is ignored for this algorithm; matching is always global. +The Engagement-scoped deduplication setting is ignored for this algorithm. Matching is instance-wide unless the Asset is in a dedupe pool for that matching kind, in which case it is bounded to the pool (see the callout above). ## Example @@ -94,4 +93,4 @@ For **Cross Tool** Deduplication: - Hash Code - Disabled -Changing the algorithm triggers a background recalculation of deduplication hashes for the tool's existing Findings. +Changing the algorithm changes what the next import compares and recomputes nothing; existing duplicate links are left as they are. Changing a tool's hash fields (or, for Global Locations, its location types) is what triggers the background recalculation of that tool's stored hashes. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md index becf3c43fe4..feb9a00744c 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md @@ -23,7 +23,7 @@ Once the feature is enabled, **Global Locations** becomes available as an option ## Configuring Global Locations Deduplication -Global Locations can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Deduplication Settings > Matching Configuration** (see [The Sidebar Menu](/navigation/pro__sidebar/)). +Global Locations can be applied to Same-Tool Deduplication, Cross-Tool Deduplication, or both, and is configured per security tool from **Settings > Finding Workflow > Matching Configuration** (**Settings > Pro Settings > Deduplication Settings > Matching Configuration** on instances still using the previous menu layout; see [The Sidebar Menu](/navigation/pro__sidebar/)). > **A pooled Asset is bounded to its pool.** "Across all Assets" holds while an Asset is not in a > [dedupe pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) for the matching kind @@ -48,11 +48,10 @@ At least one type must be selected; both are selected by default. A tool configu Use Same-Tool Deduplication with the Global Locations algorithm when you want to deduplicate Findings from a single tool across multiple Assets by shared location. -1. Open the **Same Tool Deduplication** tab. -2. Select the tool from the **Security Tool** dropdown. -3. Set the **Deduplication Algorithm** to **Global Locations**. +1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Same tool** cell. +3. Set the **Algorithm** to **Global Locations**. 4. Choose the **Location Types** to match on. -5. Submit the form. +5. Review the impact and confirm. ### Cross-Tool @@ -60,7 +59,7 @@ Use Cross-Tool Deduplication with the Global Locations algorithm when you want t Cross-tool matching reads the importing tool's location-type selection, so configure Global Locations on **each** tool that should participate, with matching Location Types. -1. Open the **Cross Tool Deduplication** tab. +1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Cross tool** cell. 2. For each tool to include: select it from the **Security Tool** dropdown, set the algorithm to **Global Locations**, choose the Location Types, and submit. ## How Matching Works @@ -72,7 +71,7 @@ A new Finding is marked as a duplicate of an existing Finding anywhere in the in The match is **strict and non-vacuous**: two Findings that have no locations of a selected type are **never** deduplicated (unlike scoped location matching, "both empty" is not a match). If endpoint-field comparison is disabled (`DEDUPE_ALGO_ENDPOINT_FIELDS = []`), URLs cannot establish a match at all — only a shared dependency can. -Same-Tool matching stays within a single tool (test type). Cross-Tool matching crosses tools intentionally. The Engagement-scoped deduplication setting is ignored for this algorithm; matching is always global, and the `service` field still partitions deduplication as it does for the other global algorithms. +Same-Tool matching stays within a single tool (test type). Cross-Tool matching crosses tools intentionally. The Engagement-scoped deduplication setting is ignored for this algorithm. Matching is instance-wide unless the Asset is in a dedupe pool for that matching kind, in which case it is bounded to the pool (see the callout above), and the `service` field still partitions deduplication as it does for the other global algorithms. ## Example @@ -124,4 +123,4 @@ For **Cross Tool** Deduplication: - Hash Code - Disabled -Changing the algorithm triggers a background recalculation of deduplication hashes for the tool's existing Findings. +Changing the algorithm changes what the next import compares and recomputes nothing; existing duplicate links are left as they are. Changing a tool's hash fields (or, for Global Locations, its location types) is what triggers the background recalculation of that tool's stored hashes. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md b/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md index 08cb50e74fe..21457bd7210 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md @@ -18,7 +18,7 @@ Each of these previously produced a closed finding plus a "new" finding — losi ## Enabling Location Tracking -Location tracking is configured per tool on **Settings > Deduplication Settings > Matching Configuration**. +Location tracking is configured per tool on **Settings > Finding Workflow > Matching Configuration** (**Settings > Pro Settings > Deduplication Settings > Matching Configuration** on instances still using the previous menu layout). 1. Find the tool's row and select its **Reimport** column. 2. Set the **Algorithm** to **Hash code**. Location tracking applies to that algorithm only: tools with a reliable **Unique ID From Tool** already track movement through their stable IDs and do not need it. diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index db1b2b19e65..29d3f447ce0 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -81,7 +81,7 @@ DefectDojo Open Source supports four deduplication algorithms that can be select - **Unique ID From Tool or Hash Code**: Prefer the tool’s unique ID; fall back to hash when no matching unique ID is found. - **Legacy**: Historical algorithm with multiple conditions; only available in the Open Source version. -**DefectDojo Pro adds more.** [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) widen the scope of the existing algorithms to a chosen group of Assets, per matching kind, without changing how two Findings are compared. Two additional algorithms instead match across **all Assets** in the instance rather than within a single Asset or Engagement: **Global Component** (by component name and version) and **Global Vulnerability ID** (by CVE, GHSA, and similar). Both are off by default and enabled by DefectDojo Support. Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. +**DefectDojo Pro adds more.** [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) widen the scope of the existing algorithms to a chosen group of Assets, per matching kind, without changing how two Findings are compared. Three additional algorithms instead match across **all Assets** in the instance rather than within a single Asset or Engagement, or across the Asset's pool when it is in one for that matching kind: **Global Component** (by component name and version), **Global Vulnerability ID** (by CVE, GHSA, and similar) and **Global Locations** (by shared URLs or dependencies). All three are off by default and gated behind feature flags (**Settings > Feature Flags**). Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. ### An alternative to Deduplication: False Positive History @@ -91,7 +91,7 @@ Instances that deliberately do **not** deduplicate can instead use [False Positi * An Engagement that has deduplication scoped to itself is **excluded** from every other Engagement's false positive history in the same Asset. * An import into such an Engagement reads **only that Engagement's** history. -* An Asset in a [Dedupe Pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) replicates a false positive across the pool's Assets, not only within itself. +* An Asset in a [Dedupe Pool](/triage_findings/finding_deduplication/pro__dedupe_pools/) replicates a false positive across the pool's Assets for same-tool matching, not only within itself. Previously the search always covered the whole Asset regardless of engagement scoping. From a588d982f702b7fc57efb4a7a147b46ddda16941 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 7 Sep 2026 05:36:26 -0500 Subject: [PATCH 13/24] feat(dedupe): let a plugin supply the false-positive history scope do_false_positive_history_batch accepts a scope_filter, but the post-import task and the bulk edit call it without one, so a plugin that widens deduplication to a group of products (FINDING_DEDUPE_BATCH_METHOD plus the candidate scope hook) could not widen the history search the same way: a false positive marked in a sibling product never reached a new import. When the caller supplied no scope, the batch now asks the optional FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD for one. The default stays the product, a provider returning None keeps the default, and an explicit scope_filter always wins. Four tests pin those rules. Co-Authored-By: Claude Opus 5 --- dojo/finding/deduplication.py | 15 ++++- unittests/test_dedupe_injectable_scope.py | 79 ++++++++++++++++++++++- 2 files changed, 91 insertions(+), 3 deletions(-) diff --git a/dojo/finding/deduplication.py b/dojo/finding/deduplication.py index 6caa86eda90..8e24aed0563 100644 --- a/dojo/finding/deduplication.py +++ b/dojo/finding/deduplication.py @@ -1228,6 +1228,10 @@ def do_false_positive_history_batch(findings, *, scope_filter=None): Args: findings: list of :model:`dojo.Finding` instances + scope_filter: which findings the history is searched over, as filter keyword + arguments for ``Finding.objects.filter``. ``None`` asks the + ``FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD`` plugin hook, and falls back to + the findings' own product when no hook is configured or it returns ``None``. """ if not findings: @@ -1238,6 +1242,16 @@ def do_false_positive_history_batch(findings, *, scope_filter=None): product = findings[0].test.engagement.product dedup_alg = findings[0].test.deduplication_algorithm + from dojo.utils import get_custom_method # noqa: PLC0415 -- circular import + + # Optional plugin hook: the scope the history is searched over, when the caller did not say. + # The engine's own scope is the product; a plugin (e.g. Pro) can widen it to a group of + # products or narrow it to one engagement by returning filter keyword arguments, and keeps + # the default by returning None. Every caller that passes no scope (the post-import task + # included) gets the same answer, so a plugin's scope cannot depend on which door was used. + if scope_filter is None and (scope_provider := get_custom_method("FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD")): + scope_filter = scope_provider(findings) + # Fetch all candidate existing findings with one DB query candidates = _fetch_fp_candidates_for_batch(findings, product, dedup_alg, scope_filter=scope_filter) @@ -1245,7 +1259,6 @@ def do_false_positive_history_batch(findings, *, scope_filter=None): # deduplication_algorithm. Lets a plugin (e.g. Pro) narrow candidates by fields that are # excluded from the hash string but compared per pair (set-match tokens on # vulnerability_ids / CWEs). Resolved once; a no-op when unset. See get_custom_method. - from dojo.utils import get_custom_method # noqa: PLC0415 -- circular import fp_candidate_filter = get_custom_method("FINDING_FALSE_POSITIVE_HISTORY_CANDIDATE_FILTER_METHOD") to_mark_as_fp_ids: set = set() diff --git a/unittests/test_dedupe_injectable_scope.py b/unittests/test_dedupe_injectable_scope.py index 1104738805c..71876f5559e 100644 --- a/unittests/test_dedupe_injectable_scope.py +++ b/unittests/test_dedupe_injectable_scope.py @@ -14,11 +14,13 @@ import logging +from django.test import override_settings from django.utils import timezone from dojo.finding.deduplication import ( _dedupe_batch_hash_code, # noqa: PLC2701 build_candidate_scope_queryset, + do_false_positive_history_batch, find_candidates_for_deduplication_hash, find_candidates_for_deduplication_legacy, find_candidates_for_deduplication_uid_or_hash, @@ -44,9 +46,9 @@ SHARED_HASH = "a" * 64 -class TestInjectableCandidateScope(DojoTestCase): +class _TwoProductFixture(DojoTestCase): - """A caller may supply the candidate scope instead of letting the engine derive it.""" + """Two products under one type, with helpers for tests and findings that share an identity.""" def setUp(self): super().setUp() @@ -102,6 +104,11 @@ def _create_finding(self, test, title, hash_code=SHARED_HASH, *, unique_id=None, finding.refresh_from_db() return finding + +class TestInjectableCandidateScope(_TwoProductFixture): + + """A caller may supply the candidate scope instead of letting the engine derive it.""" + # --- the default: unchanged ------------------------------------------ def test_default_scope_is_the_product_and_excludes_other_products(self): @@ -353,3 +360,71 @@ def test_supplied_scope_reaches_the_legacy_reimport_finder(self): theirs.id, [c.id for c in supplied.get(key, [])], "the supplied scope did not reach the legacy reimport finder", ) + + +#: What the module-level scope provider below answers with. A test sets it, the provider +#: returns a copy, so the hook can be exercised through the same settings path production uses. +_FP_SCOPE: dict = {} +_FP_SCOPE_CALLS: list = [] + + +def _module_fp_scope(findings): + _FP_SCOPE_CALLS.append([finding.pk for finding in findings]) + return dict(_FP_SCOPE) or None + + +class TestInjectableFalsePositiveHistoryScope(_TwoProductFixture): + + """ + False-positive history asks a plugin for its scope when the caller supplied none. + + The post-import task calls ``do_false_positive_history_batch(findings)`` with no scope, so + without this hook a plugin that widens deduplication to a group of products could not widen + the history search the same way, and a false positive marked in one product never reached a + sibling on import. The default stays the product. + """ + + def setUp(self): + super().setUp() + _FP_SCOPE.clear() + _FP_SCOPE_CALLS.clear() + # Same title and hash on both sides: whichever algorithm the test type resolves to, + # the two findings share an identity and only the scope decides whether they meet. + self.marked = self._create_finding(self.test_a, "Same identity in two products") + Finding.objects.filter(pk=self.marked.pk).update(false_p=True, active=False) + self.sibling = self._create_finding(self.test_b, "Same identity in two products") + + def test_the_default_scope_is_still_the_product(self): + do_false_positive_history_batch([self.sibling]) + + self.sibling.refresh_from_db() + self.assertFalse(self.sibling.false_p) + self.assertEqual(_FP_SCOPE_CALLS, [], "no hook is configured, so none may be consulted") + + @override_settings(FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD="unittests.test_dedupe_injectable_scope._module_fp_scope") + def test_a_configured_scope_provider_decides_the_search_when_the_caller_did_not(self): + _FP_SCOPE.update({"test__engagement__product__in": [self.test_a.engagement.product_id, self.test_b.engagement.product_id]}) + + do_false_positive_history_batch([self.sibling]) + + self.sibling.refresh_from_db() + self.assertTrue(self.sibling.false_p, "the provider widened the search to both products") + self.assertEqual(_FP_SCOPE_CALLS, [[self.sibling.pk]]) + + @override_settings(FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD="unittests.test_dedupe_injectable_scope._module_fp_scope") + def test_a_provider_returning_none_keeps_the_default(self): + do_false_positive_history_batch([self.sibling]) + + self.sibling.refresh_from_db() + self.assertFalse(self.sibling.false_p) + self.assertEqual(len(_FP_SCOPE_CALLS), 1) + + @override_settings(FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD="unittests.test_dedupe_injectable_scope._module_fp_scope") + def test_an_explicit_scope_wins_over_the_provider(self): + _FP_SCOPE.update({"test__engagement__product__in": [self.test_a.engagement.product_id, self.test_b.engagement.product_id]}) + + do_false_positive_history_batch([self.sibling], scope_filter={"test__engagement__product": self.test_b.engagement.product}) + + self.sibling.refresh_from_db() + self.assertFalse(self.sibling.false_p) + self.assertEqual(_FP_SCOPE_CALLS, [], "a caller that said where to search is not second-guessed") From 8fdf6e614290200d327c9c98ac5d5208cf3e88c1 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 7 Sep 2026 15:39:43 -0500 Subject: [PATCH 14/24] docs(dedupe): hub procedures for the global pages, two deduplication pages, table row, wording The Global Component and Global Locations pages describe the per-cell Matching Configuration procedure for cross tool and for reverting, and their step lists are numbered without gaps. The component page's matching rule says the instance-wide reach stops at a pool. The pools page's comparison table gains the Global Vulnerability ID row the callout beneath it already named, and quotes the subtree action the way the panel renders it. The sidebar page counts the two deduplication pages. The tuning page says selecting the cell edits the fields. The em dashes on these pages are replaced. Co-Authored-By: Claude Opus 5 --- docs/content/navigation/PRO__sidebar.md | 4 ++-- .../finding_deduplication/PRO__dedupe_pools.md | 3 ++- .../PRO__deduplication_tuning.md | 2 +- .../PRO__global_component_deduplication.md | 16 ++++++++-------- .../PRO__global_locations_deduplication.md | 16 ++++++++-------- 5 files changed, 21 insertions(+), 20 deletions(-) diff --git a/docs/content/navigation/PRO__sidebar.md b/docs/content/navigation/PRO__sidebar.md index 0dcd5c9c830..132b9021328 100644 --- a/docs/content/navigation/PRO__sidebar.md +++ b/docs/content/navigation/PRO__sidebar.md @@ -138,7 +138,7 @@ Settings is divided into eight groups, named for what you are trying to do rathe ### All Settings -The first entry in the section, **All Settings**, opens a directory of every settings page your account can reach, arranged in the same groups as the menu and searchable by name or by what the page does. Searching `deduplication` finds the three deduplication pages *and* System Settings, because System Settings holds deduplication options too. +The first entry in the section, **All Settings**, opens a directory of every settings page your account can reach, arranged in the same groups as the menu and searchable by name or by what the page does. Searching `deduplication` finds the two deduplication pages (Dedupe Pools and Matching Configuration) *and* System Settings, because System Settings holds deduplication options too. The last category, **Elsewhere in the app**, lists pages that configure DefectDojo but live in other sidebar sections: the authorization providers, Login and MFA settings, Jira instances, the Upstream and Downstream connectors, and the Universal Parser. Each tile is chipped with the section it belongs to. @@ -171,7 +171,7 @@ If you are used to the previous layout: | Manage → Rules Engine and Rules Engine 2.0 | Act → Triage Engine | | Manage → *(any)* → New *(record)* | The **New** button on the matching list page | | Dashboards → Home | Overview → Dashboards *(when Dashboards 2.0 is on)* | -| Settings → *(top level)* → Feature Flags | Unchanged — still at the top level, below All Settings | +| Settings → *(top level)* → Feature Flags | Unchanged: still at the top level, below All Settings | | Settings → Pro Settings → System Settings | Settings → System → System Settings | | Settings → Pro Settings → Appearance | Settings → System → Appearance | | Settings → Pro Settings → Banner Settings → Announcement Banner Settings | Settings → System → Announcement Banner | diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index 56a985933a7..ec5d32e875f 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -21,6 +21,7 @@ Pools and the global algorithms solve the same problem at different scales, and | --- | --- | --- | --- | | **Dedupe Pool** | The Assets you put in it | Whatever the tool's normal algorithm already uses | Some Assets should share matching and the rest should not | | **Global Component** | Every Asset in the instance | Component name and version | Every SCA Finding for a dependency is the same Finding wherever it appears | +| **Global Vulnerability ID** | Every Asset in the instance | A shared vulnerability identifier (CVE, GHSA, and so on) | Every Finding for a given vulnerability is the same Finding wherever it appears | | **Global Locations** | Every Asset in the instance | Package URL, or URL for DAST Findings | As above, keyed on the full location under the Locations data model | > **Pooling an Asset narrows a global algorithm rather than leaving it alone.** The two are not @@ -107,7 +108,7 @@ That is the safe default, but it leaves duplicates pointing outside their own As The **Dedupe Pool** panel on an Asset page shows which pool that Asset matches within, per kind, and lets you change it in place. Add it from the page layout editor if it is not already on your Asset pages. -The panel also offers **Pool this Asset and everything under it**, which pools the Asset and its descendants for that kind in one action. Two things about it are worth knowing: +The panel also offers **Pool this asset and everything under it**, which pools the Asset and its descendants for that kind in one action. Two things about it are worth knowing: * It follows **parent relationships only**. A reference between two Assets is not containment, so an Asset that merely uses another is not pulled in. * It **skips rather than steals**. A descendant already pooled elsewhere for that kind is reported back as left alone, not moved. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 4899ac564f1..dd1fe78faf8 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -50,7 +50,7 @@ To adjust Same Tool Deduplication, select the tool's **Same tool** column on **S DefectDojo Pro offers the following deduplication methods for same-tool deduplication: #### Hash Code -Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Finding Workflow > Matching Configuration** shows how many fields make up its hash, and selecting the row lets you change them. +Uses a combination of selected fields to generate a unique hash. A tool's row on **Settings > Finding Workflow > Matching Configuration** shows how many fields make up its hash, and selecting the cell lets you change them. ##### Content Fingerprint diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md index e31967bf24a..f17718e1ad3 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md @@ -9,7 +9,7 @@ Global Component Deduplication is a DefectDojo Pro algorithm that identifies dup Unlike the other deduplication algorithms, Global Component matching is **not scoped to a single Asset or Engagement**. A Finding imported into Asset B can be marked as a duplicate of an older Finding in Asset A, even if the two Assets are unrelated. -> **Global Component vs. Global Locations:** Global Component matches only on component name and version. If your instance uses the Locations data model, [Global Locations Deduplication](/triage_findings/finding_deduplication/pro__global_locations_deduplication/) is the more precise successor — it keys dependencies on the full Package URL and additionally deduplicates URL/DAST Findings across Assets. See that page's comparison table for which to choose. +> **Global Component vs. Global Locations:** Global Component matches only on component name and version. If your instance uses the Locations data model, [Global Locations Deduplication](/triage_findings/finding_deduplication/pro__global_locations_deduplication/) is the more precise successor: it keys dependencies on the full Package URL and additionally deduplicates URL/DAST Findings across Assets. See that page's comparison table for which to choose. ## Enabling the Global Component Algorithm @@ -34,8 +34,8 @@ Global Component can be applied to Same-Tool Deduplication, Cross-Tool Deduplica Use Same-Tool Deduplication with the Global Component algorithm when you want to deduplicate findings from a single SCA tool across multiple Assets. 1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Same tool** cell. -3. Set the **Algorithm** to **Global Component**. -4. Review the impact and confirm. +2. Set the **Algorithm** to **Global Component**. +3. Review the impact and confirm. Hash Code Fields are not used by this algorithm and are hidden when it is selected. @@ -45,15 +45,15 @@ Use Cross-Tool Deduplication with the Global Component algorithm when you want t Cross-tool matching requires Global Component to be configured on **each** tool that should participate. -1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Cross tool** cell. -2. For each tool to include: select it from the **Security Tool** dropdown, set the algorithm to **Global Component**, and submit. +1. Open **Settings > Finding Workflow > Matching Configuration**. +2. For each tool to include: select its **Cross tool** cell, set the **Algorithm** to **Global Component**, review the impact and confirm. ## How Matching Works A new Finding is marked as a duplicate of an existing Finding when: - The component name and component version match exactly, **and** -- An older Finding with the same component name and version exists anywhere in the DefectDojo instance — in any Asset or Engagement. +- An older Finding with the same component name and version exists anywhere in the DefectDojo instance, in any Asset or Engagement, unless the Asset is in a pool (see below). Component version matching is exact. A Finding for `timespan@2.3.0` will **not** deduplicate against one for `timespan@2.3.1`. @@ -68,7 +68,7 @@ Assume Global Component is enabled on `Dependency Track Finding Packaging Format | 1 | Dependency Track scan for `timespan@2.3.0` | Application 0 | 1 active Finding created | | 2 | Same Dependency Track scan | Application 1 | 1 Finding created, marked as duplicate of the Application 0 Finding | | 3 | Generic Findings Import for `timespan@2.3.0` | Application 2 | 1 Finding created, marked as duplicate of the Application 0 Finding (cross-tool match) | -| 4 | Dependency Track scan for `timespan@2.3.1` | Application 3 | 1 active Finding created — different version, no match | +| 4 | Dependency Track scan for `timespan@2.3.1` | Application 3 | 1 active Finding created (different version, no match) | Each duplicate Finding shows its original at the bottom of the Finding page in the duplicate chain. @@ -80,7 +80,7 @@ In that case, the Finding is visible and labelled as a duplicate, but the user w ## Reverting -To stop using Global Component for a given tool, open its Deduplication Settings and switch the algorithm back to one of the scoped options. +To stop using Global Component for a given tool, open **Settings > Finding Workflow > Matching Configuration**, select the tool's cell for the matching kind in question, and switch the algorithm back to one of the scoped options. For **Same Tool** Deduplication: diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md index feb9a00744c..a5dcd14a937 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md @@ -5,9 +5,9 @@ weight: 6 audience: pro --- -Global Locations Deduplication is a DefectDojo Pro algorithm that identifies duplicate Findings across **all Assets** based purely on a **shared location**: a URL, or a dependency (identified by its Package URL). Two Findings that share a location of a selected type are treated as duplicates regardless of their title, severity, CWE, or vulnerability IDs — the location alone is the identity. +Global Locations Deduplication is a DefectDojo Pro algorithm that identifies duplicate Findings across **all Assets** based purely on a **shared location**: a URL, or a dependency (identified by its Package URL). Two Findings that share a location of a selected type are treated as duplicates regardless of their title, severity, CWE, or vulnerability IDs: the location alone is the identity. -It is the location-aware counterpart to [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/), applied to the DefectDojo Locations data model. Where Global Component matches only on a component name and version, Global Locations matches on the same dependency **by full Package URL** *and* on shared **URLs** — so it can deduplicate DAST/web Findings across Assets, which Global Component cannot. +It is the location-aware counterpart to [Global Component Deduplication](/triage_findings/finding_deduplication/pro__global_component_deduplication/), applied to the DefectDojo Locations data model. Where Global Component matches only on a component name and version, Global Locations matches on the same dependency **by full Package URL** *and* on shared **URLs**, so it can deduplicate DAST/web Findings across Assets, which Global Component cannot. Unlike the scoped algorithms, Global Locations matching is **not scoped to a single Asset or Engagement**. A Finding imported into Asset B can be marked as a duplicate of an older Finding in Asset A, even if the two Assets are unrelated. @@ -49,9 +49,9 @@ At least one type must be selected; both are selected by default. A tool configu Use Same-Tool Deduplication with the Global Locations algorithm when you want to deduplicate Findings from a single tool across multiple Assets by shared location. 1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Same tool** cell. -3. Set the **Algorithm** to **Global Locations**. -4. Choose the **Location Types** to match on. -5. Review the impact and confirm. +2. Set the **Algorithm** to **Global Locations**. +3. Choose the **Location Types** to match on. +4. Review the impact and confirm. ### Cross-Tool @@ -59,8 +59,8 @@ Use Cross-Tool Deduplication with the Global Locations algorithm when you want t Cross-tool matching reads the importing tool's location-type selection, so configure Global Locations on **each** tool that should participate, with matching Location Types. -1. Open **Settings > Finding Workflow > Matching Configuration** and select the tool's **Cross tool** cell. -2. For each tool to include: select it from the **Security Tool** dropdown, set the algorithm to **Global Locations**, choose the Location Types, and submit. +1. Open **Settings > Finding Workflow > Matching Configuration**. +2. For each tool to include: select its **Cross tool** cell, set the **Algorithm** to **Global Locations**, choose the Location Types, review the impact and confirm. ## How Matching Works @@ -110,7 +110,7 @@ In that case, the Finding is visible and labelled as a duplicate, but the user w ## Reverting -To stop using Global Locations for a given tool, open its Deduplication Settings and switch the algorithm back to one of the scoped options. +To stop using Global Locations for a given tool, open **Settings > Finding Workflow > Matching Configuration**, select the tool's cell for the matching kind in question, and switch the algorithm back to one of the scoped options. For **Same Tool** Deduplication: From f2c3fb525297604420942984be0ddd3cb6683489 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 7 Sep 2026 17:06:42 -0500 Subject: [PATCH 15/24] docs(dedupe): the global pages name the hub, and the last em dashes go The Global Component and Global Locations pages said the algorithm becomes available "in the Tuner"; they now name Settings > Finding Workflow > Matching Configuration like the rest of the page. The remaining em dashes on the tuning and Global Locations pages are replaced with colons, commas and parentheses. Co-Authored-By: Claude Opus 5 --- .../PRO__deduplication_tuning.md | 16 ++++++++-------- .../PRO__global_component_deduplication.md | 2 +- .../PRO__global_locations_deduplication.md | 14 +++++++------- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index dd1fe78faf8..5677b67bed5 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -54,11 +54,11 @@ Uses a combination of selected fields to generate a unique hash. A tool's row on ##### Content Fingerprint -**Content Fingerprint** is a selectable hash field (available in all three configuration areas) that provides a *location-invariant* identity for static-analysis findings. It is derived from the vulnerable code snippet a tool includes in the finding — normalized so that indentation, line-number annotations, and formatting differences do not change it. Two findings about the same vulnerable code hash identically even when the code moved to a different line or file. +**Content Fingerprint** is a selectable hash field (available in all three configuration areas) that provides a *location-invariant* identity for static-analysis findings. It is derived from the vulnerable code snippet a tool includes in the finding, normalized so that indentation, line-number annotations, and formatting differences do not change it. Two findings about the same vulnerable code hash identically even when the code moved to a different line or file. -Content Fingerprint is computed for tools that include a code snippet in the finding description — including **Bandit**, **Gosec**, **Brakeman**, **Checkmarx One**, and any tool whose description carries a fenced code block or SARIF snippet. +Content Fingerprint is computed for tools that include a code snippet in the finding description, including **Bandit**, **Gosec**, **Brakeman**, **Checkmarx One**, and any tool whose description carries a fenced code block or SARIF snippet. -> **Before selecting Content Fingerprint as a hash field**, populate fingerprints for existing findings by running `./manage.py backfill_fingerprints`. Findings imported after the feature is present get fingerprints automatically, but pre-existing findings have none — selecting the field without backfilling makes existing and incoming findings hash differently, splitting every match until the backfill runs. +> **Before selecting Content Fingerprint as a hash field**, populate fingerprints for existing findings by running `./manage.py backfill_fingerprints`. Findings imported after the feature is present get fingerprints automatically, but pre-existing findings have none. Selecting the field without backfilling makes existing and incoming findings hash differently, splitting every match until the backfill runs. Content Fingerprint pairs well with **CWE** for tools that embed file paths or line numbers inside their titles, where other identity fields change every time the code moves. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/#choosing-hash-fields-for-tracked-tools). @@ -76,7 +76,7 @@ Matches findings by component name and version across **all Assets** in the inst #### Global Vulnerability ID Matches findings by their **vulnerability IDs** (CVE, GHSA, …) across **all Assets** in the instance, rather than within a single Asset or Engagement. Intended for tools that report the same CVE across many Assets. Gated behind a feature flag and off by default; a superuser can turn it on from **Settings > Feature Flags**. Like the other instance-wide algorithms it is bounded to the Asset's dedupe pool when the Asset is in one for that matching kind. -> **Two tools on the same instance-wide algorithm become mutual deduplication candidates.** When two *different* tools are both configured with an instance-wide algorithm (Global Component, or Global Vulnerability ID), their findings share a constant grouping hash, so a finding from either tool is considered for deduplication against the other on that shared dimension (component, or vulnerability ID). This is the intended cross-tool behavior — enable it only when you want those tools to dedupe together. +> **Two tools on the same instance-wide algorithm become mutual deduplication candidates.** When two *different* tools are both configured with an instance-wide algorithm (Global Component, or Global Vulnerability ID), their findings share a constant grouping hash, so a finding from either tool is considered for deduplication against the other on that shared dimension (component, or vulnerability ID). This is the intended cross-tool behavior: enable it only when you want those tools to dedupe together. ### Set-based Hash Code Fields (Vulnerability IDs and CWEs) @@ -91,16 +91,16 @@ Two finding attributes hold a *set* of values rather than a single value: vulner | `cwes_partial` | they share **at least one** CWE | | `cwes_subset` | one finding's CWEs are a **subset** of the other's | -The `_partial` and `_subset` fields are compared per finding pair rather than folded into the hash: the remaining Hash Code Fields group the candidate findings, and the set comparison then narrows that group. (Exact matching — `vulnerability_ids` and `cwes` — is folded into the hash directly.) +The `_partial` and `_subset` fields are compared per finding pair rather than folded into the hash: the remaining Hash Code Fields group the candidate findings, and the set comparison then narrows that group. (Exact matching, `vulnerability_ids` and `cwes`, is folded into the hash directly.) **Empty values.** If a finding has no vulnerability IDs (or no CWEs) for the configured matcher: -- If Hash Code Fields also include an ordinary field (for example `title`), that field carries the identity — the set matcher is skipped for the pair and the findings can still match on the rest of the hash. +- If Hash Code Fields also include an ordinary field (for example `title`), that field carries the identity: the set matcher is skipped for the pair and the findings can still match on the rest of the hash. - If a set matcher is the **only** field, a finding with no values does not match anything: with nothing else to identify it, an empty set is not treated as matching every other finding. **Configuration rules** (enforced when you save settings): -- A vulnerability IDs field (`vulnerability_ids`, `vulnerability_ids_partial`, or `vulnerability_ids_subset`) may be used on its own — a CVE or GHSA identifies a specific vulnerability instance. +- A vulnerability IDs field (`vulnerability_ids`, `vulnerability_ids_partial`, or `vulnerability_ids_subset`) may be used on its own: a CVE or GHSA identifies a specific vulnerability instance. - CWE fields (`cwes`, `cwes_partial`, `cwes_subset`) may **not** be the only criteria. A CWE is a weakness *class*, not a specific instance, so matching on CWE alone would merge unrelated findings. Pair a CWE matcher with an identifying field such as `title` or `file_path`. ## Cross Tool Deduplication @@ -184,7 +184,7 @@ For optimal results with Deduplication Tuning: - **Use Hash Code for cross-tool deduplication**: When enabling cross-tool deduplication, select fields that reliably identify the same finding across different tools (such as vulnerability name, location, and severity). **IMPORTANT** Each tool enabled for cross-tool deduplication **MUST** have the same fields selected. - **Keep cross-tool sources in the same Asset**: Cross-Tool Deduplication is Asset-scoped. Findings split across separate Assets will not dedupe even with matching hash fields. See [Cross Tool Deduplication](#cross-tool-deduplication) above. - **Avoid overly broad deduplication**: Cross-tool deduplication with too few hash fields may result in false duplicates -- **Backfill before selecting Content Fingerprint**: run `./manage.py backfill_fingerprints` first, then select the field — the triggered re-hash then has fingerprints to work with. See [Content Fingerprint](#content-fingerprint) above. +- **Backfill before selecting Content Fingerprint**: run `./manage.py backfill_fingerprints` first, then select the field: the triggered re-hash then has fingerprints to work with. See [Content Fingerprint](#content-fingerprint) above. - **Enable location tracking between scan runs**: the toggle's automatic re-hash covers the tool's whole backlog; on large instances let it finish before the next scheduled reimport. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/#enabling-on-existing-data-upgrades). By tuning deduplication settings to your specific tools, you can significantly reduce duplicate noise. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md index f17718e1ad3..7874077efcd 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_component_deduplication.md @@ -15,7 +15,7 @@ Unlike the other deduplication algorithms, Global Component matching is **not sc Global Component Deduplication is gated behind a feature flag and is **off by default**. A superuser can turn it on from **Settings > Feature Flags** on both Cloud and On-Premise instances. See [Feature Flags](/admin/feature_flags/pro__feature_flags/). -Once the feature is enabled, **Global Component** will become available as an option in the **Deduplication Algorithm** dropdown for both Same Tool and Cross Tool Deduplication settings in the Tuner. +Once the feature is enabled, **Global Component** becomes available as an **Algorithm** for both Same tool and Cross tool on **Settings > Finding Workflow > Matching Configuration**. ## Configuring Global Component Deduplication diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md index a5dcd14a937..f7b53158b0e 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md @@ -19,7 +19,7 @@ Global Locations is defined over the DefectDojo **Locations** data model and is Global Locations Deduplication is gated behind a feature flag and is **off by default**. Once Locations is enabled, a superuser can turn it on from **Settings > Feature Flags** on both Cloud and On-Premise instances. See [Feature Flags](/admin/feature_flags/pro__feature_flags/). -Once the feature is enabled, **Global Locations** becomes available as an option in the **Deduplication Algorithm** dropdown for both Same Tool and Cross Tool Deduplication settings in the Tuner. +Once the feature is enabled, **Global Locations** becomes available as an **Algorithm** for both Same tool and Cross tool on **Settings > Finding Workflow > Matching Configuration**. ## Configuring Global Locations Deduplication @@ -39,8 +39,8 @@ When you select **Global Locations**, the Hash Code Fields selector is hidden (i Choose which location types participate in matching: -- **URLs** — two Findings match when they share a URL (compared on the configured endpoint fields, `DEDUPE_ALGO_ENDPOINT_FIELDS`). -- **Dependencies** — two Findings match when they reference the same dependency, by full Package URL identity. +- **URLs**: two Findings match when they share a URL (compared on the configured endpoint fields, `DEDUPE_ALGO_ENDPOINT_FIELDS`). +- **Dependencies**: two Findings match when they reference the same dependency, by full Package URL identity. At least one type must be selected; both are selected by default. A tool configured for **URLs** only ignores shared dependencies, and a tool configured for **Dependencies** only ignores shared URLs. @@ -69,7 +69,7 @@ A new Finding is marked as a duplicate of an existing Finding anywhere in the in - **A URL** whose configured endpoint fields (`DEDUPE_ALGO_ENDPOINT_FIELDS`) all match, **or** - **A dependency** with the same Package URL (an exact purl match, so `pkg:npm/timespan@2.3.0` does **not** match `pkg:npm/timespan@2.3.1`). -The match is **strict and non-vacuous**: two Findings that have no locations of a selected type are **never** deduplicated (unlike scoped location matching, "both empty" is not a match). If endpoint-field comparison is disabled (`DEDUPE_ALGO_ENDPOINT_FIELDS = []`), URLs cannot establish a match at all — only a shared dependency can. +The match is **strict and non-vacuous**: two Findings that have no locations of a selected type are **never** deduplicated (unlike scoped location matching, "both empty" is not a match). If endpoint-field comparison is disabled (`DEDUPE_ALGO_ENDPOINT_FIELDS = []`), URLs cannot establish a match at all: only a shared dependency can. Same-Tool matching stays within a single tool (test type). Cross-Tool matching crosses tools intentionally. The Engagement-scoped deduplication setting is ignored for this algorithm. Matching is instance-wide unless the Asset is in a dedupe pool for that matching kind, in which case it is bounded to the pool (see the callout above), and the `service` field still partitions deduplication as it does for the other global algorithms. @@ -82,8 +82,8 @@ Assume Global Locations (both location types) is enabled on a DAST tool (Same To | 1 | DAST Finding at `https://shared.example.com/login` | Application 0 | 1 active Finding created | | 2 | Same URL, **different** vulnerability (title + severity) | Application 1 | 1 Finding created, marked as duplicate of the Application 0 Finding (location alone matches) | | 3 | Second DAST tool, same URL | Application 2 | 1 Finding created, marked as duplicate of the Application 0 Finding (cross-tool match) | -| 4 | DAST Finding at `https://other.example.com/admin` | Application 3 | 1 active Finding created — different URL, no shared location | -| 5 | Finding with no URL and no dependency | Application 4 | 1 active Finding created — no location to share | +| 4 | DAST Finding at `https://other.example.com/admin` | Application 3 | 1 active Finding created (different URL, no shared location) | +| 5 | Finding with no URL and no dependency | Application 4 | 1 active Finding created (no location to share) | Each duplicate Finding shows its original at the bottom of the Finding page in the duplicate chain. @@ -96,7 +96,7 @@ Both are global (cross-Asset) algorithms that ignore the Engagement scope and ma | Matches on | Component **name + version** | A shared **location**: a URL and/or a dependency | | Dependency identity | Name and version | Full **Package URL** (type, namespace, name, version, qualifiers) | | URL / DAST Findings | Not matched | Matched (on the configured endpoint fields) | -| Configurable | No | Yes — choose URLs, Dependencies, or both per tool | +| Configurable | No | Yes: choose URLs, Dependencies, or both per tool | | Data model | Works with or without Locations | Requires **Locations** (Pro) | | Best for | SCA tools where a package name+version is the identity | Web/DAST tools and SCA under the Locations model, where the URL or exact dependency is the identity | From 6e28604c44aa721beb22b18f187777fcf0251f86 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 7 Sep 2026 20:15:37 -0500 Subject: [PATCH 16/24] docs(dedupe): the Global Locations intro names the hub; last em dashes on two pages The Global Locations page said the algorithm "does not appear in the Tuner" when Locations is off; it now names Settings > Finding Workflow > Matching Configuration. The em dashes on the Location Drift Matching page and the one on About Deduplication are replaced with colons, commas and parentheses. Co-Authored-By: Claude Opus 5 --- .../PRO__global_locations_deduplication.md | 2 +- .../PRO__location_drift_matching.md | 42 +++++++++---------- .../about_deduplication.md | 2 +- 3 files changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md index f7b53158b0e..85c551a85d5 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__global_locations_deduplication.md @@ -13,7 +13,7 @@ Unlike the scoped algorithms, Global Locations matching is **not scoped to a sin ## Requirements -Global Locations is defined over the DefectDojo **Locations** data model and is only offered when the **Locations** feature is enabled. On instances where Locations is turned off, the Global Locations feature flag is shown as locked ("Requires Locations to be enabled") and the algorithm does not appear in the Tuner. +Global Locations is defined over the DefectDojo **Locations** data model and is only offered when the **Locations** feature is enabled. On instances where Locations is turned off, the Global Locations feature flag is shown as locked ("Requires Locations to be enabled") and the algorithm is not offered on **Settings > Finding Workflow > Matching Configuration**. ## Enabling the Global Locations Algorithm diff --git a/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md b/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md index 21457bd7210..0db858b051e 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__location_drift_matching.md @@ -1,18 +1,18 @@ --- title: "Location Drift Matching" -description: "Track findings as their locations change across reimports — line shifts, file renames, URL moves, and dependency version bumps no longer close and recreate findings" +description: "Track findings as their locations change across reimports: line shifts, file renames, URL moves, and dependency version bumps no longer close and recreate findings" weight: 6 audience: pro --- -**Location Drift Matching** lets reimport recognize a finding whose *location* moved as the **same finding**. Without it, reimport matches findings by an exact identity hash that includes location fields — so every location movement closes the old finding and creates an identical new one: +**Location Drift Matching** lets reimport recognize a finding whose *location* moved as the **same finding**. Without it, reimport matches findings by an exact identity hash that includes location fields, so every location movement closes the old finding and creates an identical new one: - A commit shifts code and the finding's **line number** changes. - A refactor **renames or moves the file**. - A web application's **URL, port, or host** changes between DAST scans. - A dependency **version bump** changes the vulnerable package version an SCA tool reports. -Each of these previously produced a closed finding plus a "new" finding — losing the status, notes, SLA clock, risk acceptance, and JIRA linkage on the original, and generating false "new critical finding" noise. With Location Drift Matching enabled, one finding is maintained in place: its location is updated from the latest scan and its history is preserved. +Each of these previously produced a closed finding plus a "new" finding, losing the status, notes, SLA clock, risk acceptance, and JIRA linkage on the original, and generating false "new critical finding" noise. With Location Drift Matching enabled, one finding is maintained in place: its location is updated from the latest scan and its history is preserved. > Location Drift Matching is a DefectDojo Pro feature. It is **off by default** and enabled per security tool. @@ -31,7 +31,7 @@ The review step matters here. Turning tracking on or off changes which fields th With tracking enabled, reimport matching happens in two stages: -1. **Stable identity.** The reimport hash is computed *without* the volatile location fields (line, file path, description, component name/version, endpoints) — so a finding's identity captures *what* the finding is, not *where* it currently lives. Findings that did not move still match exactly, first, and are never disturbed. +1. **Stable identity.** The reimport hash is computed *without* the volatile location fields (line, file path, description, component name/version, endpoints), so a finding's identity captures *what* the finding is, not *where* it currently lives. Findings that did not move still match exactly, first, and are never disturbed. 2. **Evidence pairing.** Within each group of findings that share a stable identity, a location matcher pairs incoming findings with existing ones using location evidence, in deterministic passes from strongest to weakest. A finding is routed to exactly one matcher based on the location data it carries. ### Code findings (SAST) @@ -41,7 +41,7 @@ With tracking enabled, reimport matching happens in two stages: | Exact | Same file and line | Always wins; a moved neighbor can never "steal" an unmoved finding's match | | Dataflow | Same source/sink objects (`sast_source_object` / `sast_sink_object`) | For tools that report dataflow; immune to line renumbering | | Nearest line | Same file, closest line number | Greedy, closest-first; same-file only | -| File rename | Different file | Only when exactly **one** incoming and **one** existing finding remain — ambiguity fails closed | +| File rename | Different file | Only when exactly **one** incoming and **one** existing finding remain (ambiguity fails closed) | ### URL findings (DAST) @@ -51,7 +51,7 @@ With tracking enabled, reimport matching happens in two stages: | Endpoint set drift | Overlapping endpoint sets (endpoints added/removed) | | Port move | Same host and path, different port | | Path drift | Same host, similar path (mutual-best segment similarity) | -| Host move | Different host — only as an unambiguous 1×1 pairing, with a wildcard-DNS guard | +| Host move | Different host, only as an unambiguous 1×1 pairing, with a wildcard-DNS guard | ### Dependency findings (SCA) @@ -61,11 +61,11 @@ With tracking enabled, reimport matching happens in two stages: | Version bump | Same package, different version | | Manifest move | Same package, different lockfile/manifest path | -When the same vulnerable package appears in **several manifests**, each manifest's finding is tracked independently — a version bump in one lockfile never swallows the finding from another. +When the same vulnerable package appears in **several manifests**, each manifest's finding is tracked independently: a version bump in one lockfile never swallows the finding from another. ### Severity re-scores -Security tools re-score severities as their rule engines evolve. With tracking enabled, a tool-reported severity change does **not** split a finding's identity: the finding matches, and its severity is updated from the scan — unless a person has re-triaged the severity by hand, in which case the human's value always wins (see below). +Security tools re-score severities as their rule engines evolve. With tracking enabled, a tool-reported severity change does **not** split a finding's identity: the finding matches, and its severity is updated from the scan, unless a person has re-triaged the severity by hand, in which case the human's value always wins (see below). ## What Is Preserved, What Refreshes @@ -73,47 +73,47 @@ A drift-matched finding keeps everything that matters about its lifecycle: statu Its **location fields** (file path, line, dataflow fields, endpoints, component version) refresh from the incoming scan. -Its **descriptive fields** (title, description, severity, component version) refresh from the scan *only when the scan still owns them*: DefectDojo records a digest of each field as last written by import/reimport. If the current value still matches that digest, the tool wrote it and the scan may update it; if a person edited the field since, the human's value is preserved permanently. Findings created before this feature have no digests and are treated as human-owned — reimport will never overwrite their descriptive fields. The single exception is **component version**, which is scan telemetry that people essentially never hand-edit: it refreshes even without a digest, so migrated SCA findings still receive version updates. +Its **descriptive fields** (title, description, severity, component version) refresh from the scan *only when the scan still owns them*: DefectDojo records a digest of each field as last written by import/reimport. If the current value still matches that digest, the tool wrote it and the scan may update it; if a person edited the field since, the human's value is preserved permanently. Findings created before this feature have no digests and are treated as human-owned: reimport will never overwrite their descriptive fields. The single exception is **component version**, which is scan telemetry that people essentially never hand-edit: it refreshes even without a digest, so migrated SCA findings still receive version updates. ### Identity always tracks the tool's report -When a matched finding is refreshed, its stored identity hashes are **adopted from the incoming scan's values** — never recomputed from the finding's current fields. This distinction matters: the finding's fields after a refresh are a *merge* of scan values and human edits, and a hash computed from that merge would contain values no scan will ever report again, silently breaking every future reimport for that finding. Adoption guarantees that a person renaming a finding, re-triaging its severity, or editing its description can never break its ability to match the next scan. +When a matched finding is refreshed, its stored identity hashes are **adopted from the incoming scan's values**, never recomputed from the finding's current fields. This distinction matters: the finding's fields after a refresh are a *merge* of scan values and human edits, and a hash computed from that merge would contain values no scan will ever report again, silently breaking every future reimport for that finding. Adoption guarantees that a person renaming a finding, re-triaging its severity, or editing its description can never break its ability to match the next scan. ## Location History -Under **Locations** (Beta), every drift match records where the finding used to live: the superseded source-code location, URL, or dependency version is kept as a reference on the finding, stamped with where it moved and why. The finding's location timeline — "this finding lived at `auth.py:42`, then `auth.py:57`, then `session.py:31`" — is visible on the finding page. See [Source Code Locations](/asset_modelling/locations/pro__source_code_locations/). +Under **Locations** (Beta), every drift match records where the finding used to live: the superseded source-code location, URL, or dependency version is kept as a reference on the finding, stamped with where it moved and why. The finding's location timeline ("this finding lived at `auth.py:42`, then `auth.py:57`, then `session.py:31`") is visible on the finding page. See [Source Code Locations](/asset_modelling/locations/pro__source_code_locations/). -Location Drift Matching itself works **with or without** the Locations feature: matching pairs on the finding's own fields and endpoints, so findings survive movement either way. Locations adds the recorded, visible history on top. History starts recording from the moment Locations is enabled — earlier moves were applied but not recorded. +Location Drift Matching itself works **with or without** the Locations feature: matching pairs on the finding's own fields and endpoints, so findings survive movement either way. Locations adds the recorded, visible history on top. History starts recording from the moment Locations is enabled: earlier moves were applied but not recorded. ## Enabling on Existing Data (Upgrades) The feature is designed to be self-migrating: - **Nothing changes until you opt in.** With the toggle off, reimport hashes compute exactly as before. -- **Saving the toggle re-hashes existing findings.** The background job recomputes the tool's stored reimport hashes with the new (location-free) identity, and creates any missing Pro finding records for data migrated from open-source. Once it completes, old and new findings speak the same identity language — a finding imported months ago is tracked exactly like one imported yesterday. +- **Saving the toggle re-hashes existing findings.** The background job recomputes the tool's stored reimport hashes with the new (location-free) identity, and creates any missing Pro finding records for data migrated from open-source. Once it completes, old and new findings speak the same identity language: a finding imported months ago is tracked exactly like one imported yesterday. - **Enable between scan runs on large instances.** The re-hash is a background job over the tool's whole finding population. A reimport that lands while it is mid-flight can see a mix of old and new hashes and churn the unprocessed slice once. Flip the toggle at a quiet time, and let the job finish before the next scheduled reimport. -- **Hand-edited titles.** The opt-in re-hash computes from current database values. Every commonly-edited field is excluded from the tracked identity — severity edits are actually *healed* by the re-hash — but if a person renamed a finding's **title** (and title is a hash field for that tool), that one finding will churn once on its next reimport before stabilizing. +- **Hand-edited titles.** The opt-in re-hash computes from current database values. Every commonly-edited field is excluded from the tracked identity (severity edits are actually *healed* by the re-hash), but if a person renamed a finding's **title** (and title is a hash field for that tool), that one finding will churn once on its next reimport before stabilizing. ## Choosing Hash Fields for Tracked Tools -Location tracking removes the volatile location fields from the reimport hash automatically — you do not need to remove `line` or `file_path` from a tool's hash configuration yourself. Two configurations deserve attention: +Location tracking removes the volatile location fields from the reimport hash automatically: you do not need to remove `line` or `file_path` from a tool's hash configuration yourself. Two configurations deserve attention: -- **All-volatile configurations.** If a tool's hash fields are *entirely* location fields (for example just `file_path` + `line`), stripping them leaves nothing, and the hash falls back to the legacy title+CWE identity. Matching still works — the evidence passes carry the discrimination — but identity is much coarser. Prefer configurations that keep at least one stable content field. -- **Location embedded in stable fields.** Field exclusions cannot help when location data hides *inside* a field that must stay in the hash. A tool that titles findings "SQL Injection in queries.py:42" changes its title on every line move — the identity splits and tracking cannot see the pair. For such tools, choose hash fields that avoid the leaking field; **CWE + Content Fingerprint** is the strong combination (see [Content Fingerprint](/triage_findings/finding_deduplication/pro__deduplication_tuning/#content-fingerprint)). +- **All-volatile configurations.** If a tool's hash fields are *entirely* location fields (for example just `file_path` + `line`), stripping them leaves nothing, and the hash falls back to the legacy title+CWE identity. Matching still works (the evidence passes carry the discrimination), but identity is much coarser. Prefer configurations that keep at least one stable content field. +- **Location embedded in stable fields.** Field exclusions cannot help when location data hides *inside* a field that must stay in the hash. A tool that titles findings "SQL Injection in queries.py:42" changes its title on every line move: the identity splits and tracking cannot see the pair. For such tools, choose hash fields that avoid the leaking field; **CWE + Content Fingerprint** is the strong combination (see [Content Fingerprint](/triage_findings/finding_deduplication/pro__deduplication_tuning/#content-fingerprint)). ## Interaction with Deduplication -Location tracking is a **reimport** feature: Same Tool and Cross Tool Deduplication are unchanged — their hashes compute exactly as before, and the exclusions never apply to them. Two deliberate integrations: +Location tracking is a **reimport** feature: Same Tool and Cross Tool Deduplication are unchanged: their hashes compute exactly as before, and the exclusions never apply to them. Two deliberate integrations: -- **Version bumps no longer block dependency deduplication.** The deduplication location gate normally requires two SCA findings to reference the *identical* package version. For tracking-enabled tools, a shared package identity (ecosystem + package name, with the namespace compared whenever both sides carry one) is enough — consistent with reimport treating a version bump as the same finding. This applies to Same Tool deduplication under Locations only. -- **Clean identity inputs.** Because matched findings adopt scan-reported hashes, the values deduplication consumes always reflect what the tool last reported — human edits can no longer contaminate them. +- **Version bumps no longer block dependency deduplication.** The deduplication location gate normally requires two SCA findings to reference the *identical* package version. For tracking-enabled tools, a shared package identity (ecosystem + package name, with the namespace compared whenever both sides carry one) is enough, consistent with reimport treating a version bump as the same finding. This applies to Same Tool deduplication under Locations only. +- **Clean identity inputs.** Because matched findings adopt scan-reported hashes, the values deduplication consumes always reflect what the tool last reported: human edits can no longer contaminate them. ## Consolidating Historical Churn Instances that ran for years without tracking accumulate close-and-recreate chains: the same finding closed and reopened as a new record every time it moved. A management command finds those chains (linked hop-by-hop by the same matchers, with a lifetime-overlap guard so findings that genuinely coexisted never merge) and consolidates each chain onto its most recent finding, marking the older copies as duplicates of the survivor: ```bash -# Dry run — reports what would be consolidated, changes nothing +# Dry run: reports what would be consolidated, changes nothing ./manage.py consolidate_location_churn --product # Apply, with a confirmation prompt diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index 29d3f447ce0..39927aba386 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -22,7 +22,7 @@ By creating and marking Duplicates in this way, DefectDojo ensures that all the ### Which Finding becomes the original -Deduplication always treats the **earliest-created** Finding in a duplicate chain as the canonical original, so a Finding from an earlier import is never demoted to a duplicate of a newer one — an original that is already established does not change hands. +Deduplication always treats the **earliest-created** Finding in a duplicate chain as the canonical original, so a Finding from an earlier import is never demoted to a duplicate of a newer one: an original that is already established does not change hands. Within a *single* report, the order the scanner happens to list its findings in does not decide the winner. Findings from one import are created in a stable, content-derived order, so a report that contains several findings colliding on the same deduplication key produces the **same original every time it is imported**. Re-scanning and re-importing the same results will not shuffle which Finding your team has been working on. From ab414f544582c8b62b82d4c64d3a2dd3f6031308 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 7 Sep 2026 20:16:24 -0500 Subject: [PATCH 17/24] docs(dedupe): the last three em dashes on pages this PR touches Two older changelog entries and the enabling-deduplication page kept an em dash each; with these replaced, every documentation page this PR touches is free of them. Co-Authored-By: Claude Opus 5 --- docs/content/releases/pro/changelog.md | 4 ++-- .../PRO_enabling_product_deduplication.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 8fdb9ea6e49..85656319337 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -333,7 +333,7 @@ Additional features: ### June 15, 2026: v3.0.0 * **(Locations)** Locations are now enabled by default, superseding the legacy Endpoint model. The legacy Endpoint API stays read-compatible and your data is preserved. See [Locations enabled by default](/releases/os_upgrading/3.0/#locations-enabled-by-default). -* **(Assets & Organizations)** "Product Type" → "Organization" and "Product" → "Asset" relabeling (UI labels + URL routing) is now on by default. The change is cosmetic — API endpoints and field names are unchanged. See [Asset / Organization labels enabled by default](/releases/os_upgrading/3.0/#asset--organization-labels-enabled-by-default). +* **(Assets & Organizations)** "Product Type" → "Organization" and "Product" → "Asset" relabeling (UI labels + URL routing) is now on by default. The change is cosmetic: API endpoints and field names are unchanged. See [Asset / Organization labels enabled by default](/releases/os_upgrading/3.0/#asset--organization-labels-enabled-by-default). * **(Authorization)** Open Source restores the **Authorized Users** panel on Product/Product Type detail under the legacy authorization model; Pro deployments retain full RBAC and are not impacted. See [Authorized Users panel replaces Members/Groups under legacy authorization](/releases/os_upgrading/3.0/#authorized-users-panel-replaces-membersgroups-under-legacy-authorization). * **(SSO)** SSO providers (SAML, OIDC, Google, Okta, Azure AD, GitLab, Auth0, Keycloak, GitHub Enterprise, remote-user header auth) are now DefectDojo Pro-only. See [SSO providers are available in DefectDojo Pro only](/releases/os_upgrading/3.0/#sso-providers-are-available-in-defectdojo-pro-only). * **(API)** Removed the Questionnaire API endpoints. See [Removal: Questionnaire API Endpoints](/releases/os_upgrading/3.0/#removal-questionnaire-api-endpoints). @@ -367,7 +367,7 @@ Additional features: * **(Pro UI)** You can now activate or deactivate Test Types and Users directly from their list menus, so retiring or restoring entries no longer requires opening the edit form. * **(Pro UI)** Anchor links now open in a new tab as expected, so following a reference no longer pulls you away from the page you were working on. * **(Pro UI)** Adding Findings to an existing Risk Acceptance works reliably again. A recent performance improvement caused the form to fail for some users; you can now resume managing accepted Findings without errors. -* **(Pro UI)** Your customized table column order is now preserved across page refreshes. Previously only column visibility carried over, so any rearranging you did would silently revert to the default — forcing you to reorder columns every session. +* **(Pro UI)** Your customized table column order is now preserved across page refreshes. Previously only column visibility carried over, so any rearranging you did would silently revert to the default, forcing you to reorder columns every session. * **(API)** Fixed a 500 error when fetching vulnerable endpoints (`GET /api/vue/endpoints/{id}/vulnerable/`), restoring reliable access to vulnerability data for an endpoint. ### May 4, 2026: v2.58.0 diff --git a/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md b/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md index 19a1b197cb0..04f66d1f1db 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/PRO_enabling_product_deduplication.md @@ -20,7 +20,7 @@ Deduplication can be applied at an Asset\-wide level, or scoped more narrowly to ### Enable Finding Deduplication -**Enable Finding Deduplication** turns on the Deduplication Algorithm for all Findings. Once enabled, Deduplication runs on every subsequent import — DefectDojo compares imported Findings against existing Findings in the destination Asset and marks duplicates according to your configuration. +**Enable Finding Deduplication** turns on the Deduplication Algorithm for all Findings. Once enabled, Deduplication runs on every subsequent import: DefectDojo compares imported Findings against existing Findings in the destination Asset and marks duplicates according to your configuration. ### Delete Duplicate Findings From 22d1e4e2d507d9d15f097f82f80920ab6f8a2644 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Tue, 8 Sep 2026 02:59:39 -0500 Subject: [PATCH 18/24] fix(dedupe): ask the history scope provider once per engagement; eighth-review docs A batch that mixes engagements (the classic bulk edit groups by product and algorithm only) has no single right scope when a plugin isolates an engagement, so do_false_positive_history_batch partitions by engagement before asking the provider. Docs: Legacy is offered by Matching Configuration, cross tool is pool-scoped, Apply Now is per pool and kind, the rules action leaves a pooled asset unchanged, and the 3.3.0 changelog carries the one-way migration and backup, the permission cutover, the rules action, the asset panel and the audit-log tracking. Co-Authored-By: Claude Opus 5 --- docs/content/releases/pro/changelog.md | 5 ++++ .../PRO__dedupe_pools.md | 6 ++-- .../PRO__deduplication_tuning.md | 2 +- .../about_deduplication.md | 2 +- dojo/finding/deduplication.py | 29 +++++++++++++++---- unittests/test_dedupe_injectable_scope.py | 29 +++++++++++++++++++ 6 files changed, 63 insertions(+), 10 deletions(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 85656319337..8ffe94b6f64 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -27,6 +27,11 @@ New features: Behavior changes: * **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. An Asset in a Dedupe Pool replicates its false positives across the pool for same-tool matching. Instances using false-positive history across such Engagements see narrower replication than before. * **(Deduplication)** For an Asset in a Dedupe Pool, Global Component, Global Vulnerability ID and Global Locations matching is bounded to the pool rather than the whole instance. +* **(Deduplication)** The upgrade copies the deduplication tuning into per-tool matching rows and then drops the tuning columns. That second step is one-way: take a database backup before upgrading, because the migration refuses to reverse and the way back is the backup. +* **(Deduplication)** The tuner's permissions are retired in favour of four Dedupe Pool permissions (view, add, edit, delete). Roles that held the tuner permissions are carried over: tuner edit maps to all four, tuner view to view only. +* **(Rules)** A new asset rule action, Assign to Dedupe Pool, pools an Asset or removes the rows a rule created; it never moves an Asset another pool holds, and the rule owner needs the Dedupe Pool edit permission. +* **(Assets)** The Asset page gains a Dedupe Pool panel showing which pool the Asset matches within, per kind, with the pool change, subtree pooling and untoggle available in place. +* **(Audit Log)** Dedupe pools, their memberships and the per-tool matching rows are tracked in the audit log. ## August 2026: v3.2 diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index ec5d32e875f..b63cb4193cb 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -1,7 +1,7 @@ --- title: "Dedupe Pools" description: "Group Assets so their Findings deduplicate against each other, per matching kind" -weight: 3 +weight: 8 audience: pro --- @@ -76,6 +76,8 @@ Adding members applies to **future imports**. Findings already in DefectDojo are 1. Click **Preview Re-run**. This reports how many Findings you can see share an identity with a Finding in another Asset in the pool. 2. **Apply Now** stays disabled until that preview has run, and uses the acknowledgement the preview returned. +Apply Now is scoped to one pool and one matching kind: it re-runs deduplication over the pool's members for the kind selected on the page, over the Findings that are not already duplicates. Existing duplicate links are left as they are, so it is not a general re-run of deduplication. + The acknowledgement is derived from the specific change it describes, so the preview you ran for adding Assets does not authorize a re-run, and a re-run preview goes stale if the pool changes underneath it. Preview the thing you are about to do. Apply Now runs while you wait, so it is capped at 10,000 Findings across the pool. Above that @@ -120,7 +122,7 @@ A membership created this way is marked **from parent**. **Untoggle subtree** re The Rules Engine action **Assign to a Dedupe Pool** puts an Asset into a pool, or takes it out of one. Run it on Asset creation and new Assets get pooled the way their siblings are, without anyone remembering to do it. -Like the subtree toggle, it counts an Asset already pooled elsewhere for that kind as skipped rather than moving it. An Asset's pool is a deliberate decision, and a rule that silently relocated it would change which Findings deduplicate against each other with nothing in the run saying so. +Like the subtree toggle, it leaves an Asset already pooled elsewhere for that kind unchanged rather than moving it. An Asset's pool is a deliberate decision, and a rule that silently relocated it would change which Findings deduplicate against each other with nothing in the run saying so. ## Permissions diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 5677b67bed5..59cba130f04 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -182,7 +182,7 @@ For optimal results with Deduplication Tuning: - **Plan retroactive re-hashes**: changing hash fields recomputes every existing Finding from that tool in the background. See [Running Deduplication Retroactively on Existing Data](#running-deduplication-retroactively-on-existing-data). - **Test changes carefully**: After adjusting matching configuration, monitor a few imports to ensure proper behavior. - **Use Hash Code for cross-tool deduplication**: When enabling cross-tool deduplication, select fields that reliably identify the same finding across different tools (such as vulnerability name, location, and severity). **IMPORTANT** Each tool enabled for cross-tool deduplication **MUST** have the same fields selected. -- **Keep cross-tool sources in the same Asset**: Cross-Tool Deduplication is Asset-scoped. Findings split across separate Assets will not dedupe even with matching hash fields. See [Cross Tool Deduplication](#cross-tool-deduplication) above. +- **Keep cross-tool sources in the same Asset, or in the same Dedupe Pool**: Cross Tool Deduplication is scoped to the Asset, or to the Asset's pool when it is in one for cross-tool matching. Findings split across Assets that share neither will not dedupe even with matching hash fields. See [Cross Tool Deduplication](#cross-tool-deduplication) above. - **Avoid overly broad deduplication**: Cross-tool deduplication with too few hash fields may result in false duplicates - **Backfill before selecting Content Fingerprint**: run `./manage.py backfill_fingerprints` first, then select the field: the triggered re-hash then has fingerprints to work with. See [Content Fingerprint](#content-fingerprint) above. - **Enable location tracking between scan runs**: the toggle's automatic re-hash covers the tool's whole backlog; on large instances let it finish before the next scheduled reimport. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/#enabling-on-existing-data-upgrades). diff --git a/docs/content/triage_findings/finding_deduplication/about_deduplication.md b/docs/content/triage_findings/finding_deduplication/about_deduplication.md index 39927aba386..827ea9476a4 100644 --- a/docs/content/triage_findings/finding_deduplication/about_deduplication.md +++ b/docs/content/triage_findings/finding_deduplication/about_deduplication.md @@ -79,7 +79,7 @@ DefectDojo Open Source supports four deduplication algorithms that can be select - **Unique ID From Tool**: Uses the scanner-provided unique identifier. - **Hash Code**: Uses a configured set of fields to compute a hash. - **Unique ID From Tool or Hash Code**: Prefer the tool’s unique ID; fall back to hash when no matching unique ID is found. -- **Legacy**: Historical algorithm with multiple conditions; only available in the Open Source version. +- **Legacy**: Historical algorithm with multiple conditions. Matching Configuration offers it for same-tool and reimport matching; it is the fallback when a tool has no other configuration. **DefectDojo Pro adds more.** [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/) widen the scope of the existing algorithms to a chosen group of Assets, per matching kind, without changing how two Findings are compared. Three additional algorithms instead match across **all Assets** in the instance rather than within a single Asset or Engagement, or across the Asset's pool when it is in one for that matching kind: **Global Component** (by component name and version), **Global Vulnerability ID** (by CVE, GHSA, and similar) and **Global Locations** (by shared URLs or dependencies). All three are off by default and gated behind feature flags (**Settings > Feature Flags**). Pro also lets the Hash Code algorithm treat a Finding's vulnerability IDs and CWEs as **sets**, matching on the exact set, on any shared value (`_partial`), or on one being a subset of the other (`_subset`). See [Deduplication Tuning (Pro)](/triage_findings/finding_deduplication/pro__deduplication_tuning/) for the full list, the set-matching fields, and the rules governing them. diff --git a/dojo/finding/deduplication.py b/dojo/finding/deduplication.py index e2a1ac3089a..6fdbcf0720c 100644 --- a/dojo/finding/deduplication.py +++ b/dojo/finding/deduplication.py @@ -1299,11 +1299,6 @@ def do_false_positive_history_batch(findings, *, scope_filter=None): if not findings: return - system_settings = System_Settings.objects.get() - - product = findings[0].test.engagement.product - dedup_alg = findings[0].test.deduplication_algorithm - from dojo.utils import get_custom_method # noqa: PLC0415 -- circular import # Optional plugin hook: the scope the history is searched over, when the caller did not say. @@ -1312,7 +1307,29 @@ def do_false_positive_history_batch(findings, *, scope_filter=None): # the default by returning None. Every caller that passes no scope (the post-import task # included) gets the same answer, so a plugin's scope cannot depend on which door was used. if scope_filter is None and (scope_provider := get_custom_method("FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD")): - scope_filter = scope_provider(findings) + # The provider answers for one engagement: a plugin may isolate an engagement from the + # rest of its product, and a batch that mixes an isolated engagement with a normal one + # (the classic bulk edit groups by product and algorithm only) has no single right scope. + # Ask once per engagement and process each group with its own answer; a None answer + # keeps the default for that group and is not asked again. + by_engagement: dict = {} + for finding in findings: + by_engagement.setdefault(finding.test.engagement_id, []).append(finding) + for group in by_engagement.values(): + _do_false_positive_history_batch_in_scope(group, scope_provider(group)) + return + + _do_false_positive_history_batch_in_scope(findings, scope_filter) + + +def _do_false_positive_history_batch_in_scope(findings, scope_filter): + """The batch itself, once the scope is settled: ``None`` searches the findings' own product.""" + system_settings = System_Settings.objects.get() + + product = findings[0].test.engagement.product + dedup_alg = findings[0].test.deduplication_algorithm + + from dojo.utils import get_custom_method # noqa: PLC0415 -- circular import # Fetch all candidate existing findings with one DB query candidates = _fetch_fp_candidates_for_batch(findings, product, dedup_alg, scope_filter=scope_filter) diff --git a/unittests/test_dedupe_injectable_scope.py b/unittests/test_dedupe_injectable_scope.py index 71876f5559e..3befe5ae4ca 100644 --- a/unittests/test_dedupe_injectable_scope.py +++ b/unittests/test_dedupe_injectable_scope.py @@ -419,6 +419,35 @@ def test_a_provider_returning_none_keeps_the_default(self): self.assertFalse(self.sibling.false_p) self.assertEqual(len(_FP_SCOPE_CALLS), 1) + @override_settings(FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD="unittests.test_dedupe_injectable_scope._module_fp_scope") + def test_a_batch_spanning_engagements_asks_the_provider_once_per_engagement(self): + """ + The classic bulk edit groups by product and algorithm only, so its batches can mix + engagements whose scopes differ; the provider is asked per engagement, with that + engagement's findings, rather than once for the first finding's engagement. + """ + other_engagement = Engagement.objects.create( + name="Scope Engagement B2", + product=self.test_b.engagement.product, + target_start=timezone.now(), + target_end=timezone.now(), + ) + other_test = Test.objects.create( + engagement=other_engagement, + test_type=self.test_type, + target_start=timezone.now(), + target_end=timezone.now(), + ) + elsewhere = self._create_finding(other_test, "Same identity in two products") + + do_false_positive_history_batch([self.sibling, elsewhere]) + + self.assertEqual( + sorted(sorted(group) for group in _FP_SCOPE_CALLS), + sorted([[self.sibling.pk], [elsewhere.pk]]), + "one call per engagement, each with only that engagement's findings", + ) + @override_settings(FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD="unittests.test_dedupe_injectable_scope._module_fp_scope") def test_an_explicit_scope_wins_over_the_provider(self): _FP_SCOPE.update({"test__engagement__product__in": [self.test_a.engagement.product_id, self.test_b.engagement.product_id]}) From 237d6296b9ca341fbe4724addf515f2ac7a47815 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Tue, 8 Sep 2026 13:40:28 -0500 Subject: [PATCH 19/24] docs(changelog): the dedupe pools entries ship in 3.3.100, the patch the PR is milestoned for Co-Authored-By: Claude Opus 5 --- docs/content/releases/pro/changelog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 8ffe94b6f64..decc7195a7b 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -18,7 +18,7 @@ For Open Source release notes, please see the [Releases page on GitHub](https:// ## September 2026: v3.3 -### September 8, 2026: v3.3.0 +### September 14, 2026: v3.3.100 New features: * **(Deduplication)** Added Dedupe Pools: group the Assets that should deduplicate against each other, choose where their originals collect, preview what a membership change would link, and re-run deduplication over the Findings already in scope with Apply Now. From cc7a14ea5aa5f5987012b1f6f2355df094d4f7cf Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Sat, 12 Sep 2026 14:27:29 -0500 Subject: [PATCH 20/24] docs(dedupe): correct the tuner permission scope and add the upgrade ordering Two corrections to the 3.3.100 entry. The permissions line claimed the tuner's permissions are retired. They are not: Tuner_View and Tuner_Edit still gate the other 14 tuner sections (SSO, LDAP, SCIM, email, MFA and the rest). Only the three deduplication pages moved onto the Dedupe Pool permissions, and only for those pages does the carry-over apply. The single-transaction fold is also an operator instruction nobody was given. The copy into matching rows and the drop of the tuning columns commit together, so the changeover is instant: a pod still on the old image reads columns that no longer exist, and a new-image pod that started before the commit has no rows to read. Say to migrate first, roll pods second, and hold imports across the window. The one-way warning moves into that block, where an operator reads it in time. Co-Authored-By: Claude Opus 5 --- docs/content/releases/pro/changelog.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 039d508a8e7..59c1edc5f1e 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -27,12 +27,15 @@ New features: Behavior changes: * **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. An Asset in a Dedupe Pool replicates its false positives across the pool for same-tool matching. Instances using false-positive history across such Engagements see narrower replication than before. * **(Deduplication)** For an Asset in a Dedupe Pool, Global Component, Global Vulnerability ID and Global Locations matching is bounded to the pool rather than the whole instance. -* **(Deduplication)** The upgrade copies the deduplication tuning into per-tool matching rows and then drops the tuning columns. That second step is one-way: take a database backup before upgrading, because the migration refuses to reverse and the way back is the backup. -* **(Deduplication)** The tuner's permissions are retired in favour of four Dedupe Pool permissions (view, add, edit, delete). Roles that held the tuner permissions are carried over: tuner edit maps to all four, tuner view to view only. +* **(Deduplication)** The three deduplication pages move off the Tuner permissions onto four Dedupe Pool permissions (view, add, edit, delete). Roles that held the Tuner permissions are carried over for those pages: Tuner edit maps to all four, Tuner view to view only. The Tuner permissions themselves are unchanged and still gate the other 14 Tuner sections (SSO, LDAP, SCIM, email, MFA and the rest). * **(Rules)** A new asset rule action, Assign to Dedupe Pool, pools an Asset or removes the rows a rule created; it never moves an Asset another pool holds, and the rule owner needs the Dedupe Pool edit permission. * **(Assets)** The Asset page gains a Dedupe Pool panel showing which pool the Asset matches within, per kind, with the pool change, subtree pooling and untoggle available in place. * **(Audit Log)** Dedupe pools, their memberships and the per-tool matching rows are tracked in the audit log. +Upgrade notes: +* **(Deduplication)** Run the migration before rolling pods, and hold imports until the roll finishes. The upgrade copies the deduplication tuning into per-tool matching rows and drops the tuning columns in a single transaction, so the changeover is instant rather than gradual. A pod still on the old image reads the tuning columns, so it fails on every deduplication path the moment the migration job commits. A pod already on the new image that starts before the job commits has no matching rows to read: DefectDojo falls back to the tuning columns while they exist, and the Go matching service cannot answer at all until the table is there. Migrate first, then roll the web, worker and matching pods, then resume imports, so no import straddles the changeover. +* **(Deduplication)** Take a database backup before upgrading. Dropping the tuning columns is one-way: the migration refuses to reverse, so the backup is the way back. + ### September 9, 2026: v3.3.0 New features: From 3a6665f99ac23c645c276d7a7d6176137a77dec0 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Sat, 12 Sep 2026 14:41:08 -0500 Subject: [PATCH 21/24] docs(dedupe): give the upgrade page the deploy ordering, not just the changelog The changelog scrolls away; the upgrade section is where an operator reads before upgrading. Same instruction: the copy and the removal commit together, so migrate first, roll pods second, resume imports last. Says plainly that a single-node deployment already does this and needs nothing extra, so the note does not read as a new requirement for everyone. Co-Authored-By: Claude Opus 5 --- .../finding_deduplication/PRO__deduplication_tuning.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 59cba130f04..ae300876d55 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -151,6 +151,14 @@ migration in reverse. Nothing about your matching behaviour changes at upgrade t keeps the algorithm and fields it had. The backup is for the case where you need the previous release back for some other reason. +**Run the migration before you roll pods, and hold imports until the roll finishes.** The copy +and the removal commit together, so the changeover is instant rather than gradual. A pod still +running the previous release reads settings that no longer exist the moment the migration +commits, and a pod on the new release that started before it has no Matching Configuration rows +to read yet. Migrating first, rolling second, and resuming imports last means no import straddles +the changeover. On a single-node deployment this is the ordinary upgrade order and needs nothing +extra; it matters where web, worker and matching pods restart independently of the migration job. + Permission changes for custom roles are described under [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/#custom-roles-on-upgrade). ## Running Deduplication Retroactively on Existing Data From f4dfc88f73d4b31c6c40f6569fbeebbb8472bd57 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Sun, 13 Sep 2026 21:12:30 -0500 Subject: [PATCH 22/24] docs(dedupe): say what old and new pods actually do across the upgrade, and what crosses an Organization The upgrade notes described an old-image pod as failing on every deduplication path. It does not fail; it degrades, which is worse to diagnose. That release's settings reader swallows the ProgrammingError from the dropped columns and returns None, so every Tuner-backed setting read at request time comes back empty: SCIM, finding enrichment, the OSV source, notifications, the health check, and the import drift settings, which raise on the None. Sign-in and MFA survive because they were applied at process start. The notes now say that. The new-pod sentence was also wrong in the reassuring direction. Matching does resolve from the old columns while they exist, but pool membership has no such fallback, so an import on a new pod before the migration commits fails on the membership table. That is the actual reason imports are held; the notes now give it. Both places carry the same correction: the 3.3.100 Upgrade notes and the upgrade section of the deduplication tuning page. Two consequences nobody had written down. The migration holds an exclusive lock on the settings table from dropping the columns to commit, and requests reading settings wait through it. And the first nightly identity check after upgrading may notify about cross-tool tools that had fields but no algorithm: the seed records the Hash code the previous release only implied, so the definition moved while the hashes did not, and the suggested rehash is safe. The pools page and the changelog said a pool may span Organizations without saying what that means. Now they do: duplicate marks and false-positive replication follow the pool across that boundary. Co-Authored-By: Claude Fable 5.1 --- docs/content/releases/pro/changelog.md | 6 ++-- .../PRO__dedupe_pools.md | 2 +- .../PRO__deduplication_tuning.md | 30 ++++++++++++++----- 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 59c1edc5f1e..8b3a9b62d96 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -25,7 +25,7 @@ New features: * **(Deduplication)** The three deduplication tuning pages are now one Matching Configuration page: every tool listed once, with its same-tool, cross-tool and reimport matching side by side, and every change previewed before it is saved. Behavior changes: -* **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. An Asset in a Dedupe Pool replicates its false positives across the pool for same-tool matching. Instances using false-positive history across such Engagements see narrower replication than before. +* **(Deduplication)** False-positive history now follows deduplication scope. A Finding is compared against the Assets it deduplicates with, so an Engagement that deduplicates within itself only replicates false positives inside that Engagement. An Asset in a Dedupe Pool replicates its false positives across the pool for same-tool matching. Instances using false-positive history across such Engagements see narrower replication than before. A pool may span Organizations, and both effects follow the pool: a duplicate mark or a replicated false positive originating in one Organization can change a Finding in another Organization that shares the pool. * **(Deduplication)** For an Asset in a Dedupe Pool, Global Component, Global Vulnerability ID and Global Locations matching is bounded to the pool rather than the whole instance. * **(Deduplication)** The three deduplication pages move off the Tuner permissions onto four Dedupe Pool permissions (view, add, edit, delete). Roles that held the Tuner permissions are carried over for those pages: Tuner edit maps to all four, Tuner view to view only. The Tuner permissions themselves are unchanged and still gate the other 14 Tuner sections (SSO, LDAP, SCIM, email, MFA and the rest). * **(Rules)** A new asset rule action, Assign to Dedupe Pool, pools an Asset or removes the rows a rule created; it never moves an Asset another pool holds, and the rule owner needs the Dedupe Pool edit permission. @@ -33,7 +33,9 @@ Behavior changes: * **(Audit Log)** Dedupe pools, their memberships and the per-tool matching rows are tracked in the audit log. Upgrade notes: -* **(Deduplication)** Run the migration before rolling pods, and hold imports until the roll finishes. The upgrade copies the deduplication tuning into per-tool matching rows and drops the tuning columns in a single transaction, so the changeover is instant rather than gradual. A pod still on the old image reads the tuning columns, so it fails on every deduplication path the moment the migration job commits. A pod already on the new image that starts before the job commits has no matching rows to read: DefectDojo falls back to the tuning columns while they exist, and the Go matching service cannot answer at all until the table is there. Migrate first, then roll the web, worker and matching pods, then resume imports, so no import straddles the changeover. +* **(Deduplication)** Run the migration before rolling pods, hold imports until the roll finishes, and roll the Go matching service to the same release. The upgrade copies the deduplication tuning into per-tool matching rows and drops the tuning columns in a single transaction, so the changeover is instant rather than gradual. A pod still on the old image keeps serving, but from the moment the migration commits every Tuner-backed setting it reads at request time degrades or fails: its settings query names columns that no longer exist, the error is swallowed, and the read comes back empty. Sign-in and MFA settings applied at process start survive; SCIM, finding enrichment, the OSV source, notifications, the health check and the import drift settings do not. A pod already on the new image that started before the job commits cannot import: matching resolves from the old tuning columns while they exist, but pool membership has no such fallback, so an import on that pod fails until the table it needs is there. Migrate first, then roll the web, worker and matching pods, then resume imports, so no import straddles the changeover. +* **(Deduplication)** The migration holds an exclusive lock on the settings table for its final steps, from dropping the tuning columns to commit. Requests reading settings during those statements wait rather than fail. The copy that precedes it, which is the long step on a large instance, does not hold that lock. +* **(Deduplication)** The first nightly identity check after the upgrade may send a system notification saying the cross-tool identity changed for some tools. Those tools had cross-tool hash fields configured but no algorithm; the previous release treated that as Hash code, and the upgrade records Hash code explicitly, so the identity definition moved while the stored hashes did not. The rehash the notification suggests (`manage.py identity_drift --kind cross_tool --rehash`) is safe, recomputes the same values, and records the new baseline so the notice does not repeat. * **(Deduplication)** Take a database backup before upgrading. Dropping the tuning columns is one-way: the migration refuses to reverse, so the backup is the way back. ### September 9, 2026: v3.3.0 diff --git a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md index b63cb4193cb..b544134daa4 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__dedupe_pools.md @@ -9,7 +9,7 @@ By default a Finding only deduplicates against other Findings in **its own Asset Pools are for the case where the same thing is genuinely deployed in several places you model as separate Assets. Three services that all ship the same base image, or a monorepo split into an Asset per component, will each report the same vulnerability separately, and no per-Asset setting can make those Findings meet. -A pool may span Organizations. You only ever see the members you have access to, and a member you cannot read is shown as a placeholder rather than hidden, so a pool never looks smaller than it is. +A pool may span Organizations, and everything a pool does crosses that boundary with it: a Finding in one Organization can be marked a duplicate of a Finding in another, and a false positive recorded in one Organization is replicated to matching Findings in the others that share the pool. You only ever see the members you have access to, and a member you cannot read is shown as a placeholder rather than hidden, so a pool never looks smaller than it is. Find pools at **Settings \> Finding Workflow \> Dedupe Pools** (**Settings \> Pro Settings \> Deduplication Settings \> Dedupe Pools** on instances still using the previous menu layout). Matching Configuration sits beside it in the same group. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index ae300876d55..9e314da67ef 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -151,13 +151,29 @@ migration in reverse. Nothing about your matching behaviour changes at upgrade t keeps the algorithm and fields it had. The backup is for the case where you need the previous release back for some other reason. -**Run the migration before you roll pods, and hold imports until the roll finishes.** The copy -and the removal commit together, so the changeover is instant rather than gradual. A pod still -running the previous release reads settings that no longer exist the moment the migration -commits, and a pod on the new release that started before it has no Matching Configuration rows -to read yet. Migrating first, rolling second, and resuming imports last means no import straddles -the changeover. On a single-node deployment this is the ordinary upgrade order and needs nothing -extra; it matters where web, worker and matching pods restart independently of the migration job. +**Run the migration before you roll pods, hold imports until the roll finishes, and roll the Go +matching service to the same release.** The copy and the removal commit together, so the +changeover is instant rather than gradual. A pod still running the previous release keeps +serving, but every Tuner-backed setting it reads at request time degrades or fails from the moment +the migration commits: its settings query names columns that no longer exist, the error is +swallowed, and the read comes back empty. Sign-in and MFA settings applied when the process +started survive; SCIM, finding enrichment, the OSV source, notifications, the health check and the +import drift settings do not. A pod on the new release that started before the migration committed +cannot import: matching resolves from the old settings while they exist, but pool membership has +no such fallback, so an import on that pod fails until its table is there. Migrating first, +rolling second, and resuming imports last means no import straddles the changeover. On a +single-node deployment this is the ordinary upgrade order and needs nothing extra; it matters +where web, worker and matching pods restart independently of the migration job. + +Two things to expect from the upgrade itself. The migration holds an exclusive lock on the +settings table for its final steps, from dropping the old columns to commit; requests reading +settings during those statements wait rather than fail, and the copy step before it, which is the +long one on a large instance, does not hold that lock. And the first nightly identity check after +the upgrade may send a system notification that the cross-tool identity changed for some tools. +Those tools had cross-tool hash fields configured but no algorithm; the previous release treated +that as Hash code and the upgrade records Hash code explicitly, so the definition moved while the +stored hashes did not. The rehash it suggests is safe, recomputes the same values, and records the +new baseline so the notice does not repeat. Permission changes for custom roles are described under [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/#custom-roles-on-upgrade). From 327a01f69482d133a09ad7b0074cb62615e22e0f Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Sun, 13 Sep 2026 21:15:30 -0500 Subject: [PATCH 23/24] docs(dedupe): the seeded matching rows have no audit trail, and say so pro.0211 writes the matching rows before it installs their pghistory triggers, so the configuration the upgrade carries over from the Tuner is not an audit event. An operator looking for who set a tool's fields to what they are on upgrade day finds nothing, and should know that is expected rather than a gap in the log. History starts with the first change made after the upgrade. Co-Authored-By: Claude Fable 5.1 --- docs/content/releases/pro/changelog.md | 1 + .../finding_deduplication/PRO__deduplication_tuning.md | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index 8b3a9b62d96..b6eb52661b3 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -35,6 +35,7 @@ Behavior changes: Upgrade notes: * **(Deduplication)** Run the migration before rolling pods, hold imports until the roll finishes, and roll the Go matching service to the same release. The upgrade copies the deduplication tuning into per-tool matching rows and drops the tuning columns in a single transaction, so the changeover is instant rather than gradual. A pod still on the old image keeps serving, but from the moment the migration commits every Tuner-backed setting it reads at request time degrades or fails: its settings query names columns that no longer exist, the error is swallowed, and the read comes back empty. Sign-in and MFA settings applied at process start survive; SCIM, finding enrichment, the OSV source, notifications, the health check and the import drift settings do not. A pod already on the new image that started before the job commits cannot import: matching resolves from the old tuning columns while they exist, but pool membership has no such fallback, so an import on that pod fails until the table it needs is there. Migrate first, then roll the web, worker and matching pods, then resume imports, so no import straddles the changeover. * **(Deduplication)** The migration holds an exclusive lock on the settings table for its final steps, from dropping the tuning columns to commit. Requests reading settings during those statements wait rather than fail. The copy that precedes it, which is the long step on a large instance, does not hold that lock. +* **(Deduplication)** The matching rows the upgrade seeds carry no audit log entry: the migration writes them before it installs their audit triggers. Audit history for Matching Configuration starts with the first change made after the upgrade; the seeded state itself is what the Tuner held, and is not recorded as an event. * **(Deduplication)** The first nightly identity check after the upgrade may send a system notification saying the cross-tool identity changed for some tools. Those tools had cross-tool hash fields configured but no algorithm; the previous release treated that as Hash code, and the upgrade records Hash code explicitly, so the identity definition moved while the stored hashes did not. The rehash the notification suggests (`manage.py identity_drift --kind cross_tool --rehash`) is safe, recomputes the same values, and records the new baseline so the notice does not repeat. * **(Deduplication)** Take a database backup before upgrading. Dropping the tuning columns is one-way: the migration refuses to reverse, so the backup is the way back. diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 9e314da67ef..134f7bb8d3d 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -168,7 +168,10 @@ where web, worker and matching pods restart independently of the migration job. Two things to expect from the upgrade itself. The migration holds an exclusive lock on the settings table for its final steps, from dropping the old columns to commit; requests reading settings during those statements wait rather than fail, and the copy step before it, which is the -long one on a large instance, does not hold that lock. And the first nightly identity check after +long one on a large instance, does not hold that lock. The rows the migration seeds carry no audit +log entry, because they are written before their audit triggers are installed; audit history for +Matching Configuration starts with the first change made after the upgrade. And the first nightly +identity check after the upgrade may send a system notification that the cross-tool identity changed for some tools. Those tools had cross-tool hash fields configured but no algorithm; the previous release treated that as Hash code and the upgrade records Hash code explicitly, so the definition moved while the From d77fe2b636a8c9496f190ca6626cd23eafac37e1 Mon Sep 17 00:00:00 2001 From: blakeaowens Date: Mon, 14 Sep 2026 00:07:42 -0500 Subject: [PATCH 24/24] docs(dedupe): the upgrade no longer has a required order, and the migration reverses The pod-order requirement was a deal breaker, and it existed for one reason: the migration dropped the tuner's three columns in the same release as the code that stopped reading them. It no longer drops them. The fields leave the model, the columns stay for this release, and a later release removes them once no pod on the previous image can exist. So a pod still on the previous image keeps working through a roll, a pod on the new image reaches a pre-migration database and matches without pools, and the order stops mattering. The notes now say that, in both the changelog and the upgrade section of the tuning page. With the drop gone the migration is reversible, so "one-way, the backup is the way back" is replaced with what is now true: rolling back to the previous node works, and a backup is ordinary upgrade hygiene. The exclusive-lock note goes with the drop; the audit-trail and identity-notification notes stay. Co-Authored-By: Claude Fable 5.1 --- docs/content/releases/pro/changelog.md | 5 +- .../PRO__deduplication_tuning.md | 51 ++++++++----------- 2 files changed, 22 insertions(+), 34 deletions(-) diff --git a/docs/content/releases/pro/changelog.md b/docs/content/releases/pro/changelog.md index b6eb52661b3..1e50d825c1d 100644 --- a/docs/content/releases/pro/changelog.md +++ b/docs/content/releases/pro/changelog.md @@ -33,11 +33,10 @@ Behavior changes: * **(Audit Log)** Dedupe pools, their memberships and the per-tool matching rows are tracked in the audit log. Upgrade notes: -* **(Deduplication)** Run the migration before rolling pods, hold imports until the roll finishes, and roll the Go matching service to the same release. The upgrade copies the deduplication tuning into per-tool matching rows and drops the tuning columns in a single transaction, so the changeover is instant rather than gradual. A pod still on the old image keeps serving, but from the moment the migration commits every Tuner-backed setting it reads at request time degrades or fails: its settings query names columns that no longer exist, the error is swallowed, and the read comes back empty. Sign-in and MFA settings applied at process start survive; SCIM, finding enrichment, the OSV source, notifications, the health check and the import drift settings do not. A pod already on the new image that started before the job commits cannot import: matching resolves from the old tuning columns while they exist, but pool membership has no such fallback, so an import on that pod fails until the table it needs is there. Migrate first, then roll the web, worker and matching pods, then resume imports, so no import straddles the changeover. -* **(Deduplication)** The migration holds an exclusive lock on the settings table for its final steps, from dropping the tuning columns to commit. Requests reading settings during those statements wait rather than fail. The copy that precedes it, which is the long step on a large instance, does not hold that lock. +* **(Deduplication)** Pods may roll in either order relative to the migration. The upgrade copies the deduplication tuning into per-tool matching rows and retires the old tuning fields from the application, but leaves their columns in the database for this release. A pod still on the previous image keeps reading and writing those columns and behaves exactly as before until it is rolled; a pod on the new image reaches a database that has not migrated yet and matches without pools, reading the old tuning where it needs to, until the migration lands. The columns are removed by a later release, once no pod on the previous image can exist. Hold imports across the roll if you want no import to straddle the changeover; nothing fails if you do not. +* **(Deduplication)** The migration is reversible. Rolling back to the previous node drops the new pool tables and restores the previous release's view of the settings; the tuning columns never left. Take a database backup before upgrading anyway, as ordinary upgrade hygiene. * **(Deduplication)** The matching rows the upgrade seeds carry no audit log entry: the migration writes them before it installs their audit triggers. Audit history for Matching Configuration starts with the first change made after the upgrade; the seeded state itself is what the Tuner held, and is not recorded as an event. * **(Deduplication)** The first nightly identity check after the upgrade may send a system notification saying the cross-tool identity changed for some tools. Those tools had cross-tool hash fields configured but no algorithm; the previous release treated that as Hash code, and the upgrade records Hash code explicitly, so the identity definition moved while the stored hashes did not. The rehash the notification suggests (`manage.py identity_drift --kind cross_tool --rehash`) is safe, recomputes the same values, and records the new baseline so the notice does not repeat. -* **(Deduplication)** Take a database backup before upgrading. Dropping the tuning columns is one-way: the migration refuses to reverse, so the backup is the way back. ### September 9, 2026: v3.3.0 diff --git a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md index 134f7bb8d3d..26e26fb042e 100644 --- a/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md +++ b/docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md @@ -145,38 +145,27 @@ moves their configuration into it. The migration copies every per-tool entry fro three stored settings into Matching Configuration rows, marks the ones you had changed from the shipped defaults as changed, and then removes the tuner's stored settings. -**That last step is one-way. Take a database backup before upgrading.** The migration cannot be -reversed: rolling back to the previous release means restoring the backup, not running the -migration in reverse. Nothing about your matching behaviour changes at upgrade time; every tool -keeps the algorithm and fields it had. The backup is for the case where you need the previous -release back for some other reason. - -**Run the migration before you roll pods, hold imports until the roll finishes, and roll the Go -matching service to the same release.** The copy and the removal commit together, so the -changeover is instant rather than gradual. A pod still running the previous release keeps -serving, but every Tuner-backed setting it reads at request time degrades or fails from the moment -the migration commits: its settings query names columns that no longer exist, the error is -swallowed, and the read comes back empty. Sign-in and MFA settings applied when the process -started survive; SCIM, finding enrichment, the OSV source, notifications, the health check and the -import drift settings do not. A pod on the new release that started before the migration committed -cannot import: matching resolves from the old settings while they exist, but pool membership has -no such fallback, so an import on that pod fails until its table is there. Migrating first, -rolling second, and resuming imports last means no import straddles the changeover. On a -single-node deployment this is the ordinary upgrade order and needs nothing extra; it matters -where web, worker and matching pods restart independently of the migration job. - -Two things to expect from the upgrade itself. The migration holds an exclusive lock on the -settings table for its final steps, from dropping the old columns to commit; requests reading -settings during those statements wait rather than fail, and the copy step before it, which is the -long one on a large instance, does not hold that lock. The rows the migration seeds carry no audit -log entry, because they are written before their audit triggers are installed; audit history for +**Pods may roll in either order relative to the migration.** The migration copies the tuning into +Matching Configuration rows and retires the old tuning fields from the application, but leaves +their columns in the database for this release. A pod still running the previous release keeps +reading and writing those columns and behaves exactly as before until it is rolled. A pod on the +new release that reaches a database which has not migrated yet matches without pools, reading the +old tuning where it needs to, until the migration lands. The columns are removed by a later +release, once no pod on the previous release can exist. If you would rather no import straddle +the changeover, hold imports across the roll; nothing fails if you do not. + +**The migration is reversible, and you should still take a backup.** Rolling back to the previous +migration drops the new pool tables and restores the previous release's view of the settings; the +tuning columns never left. The backup is ordinary upgrade hygiene, not the only way back. + +Two things to expect from the upgrade itself. The rows the migration seeds carry no audit log +entry, because they are written before their audit triggers are installed; audit history for Matching Configuration starts with the first change made after the upgrade. And the first nightly -identity check after -the upgrade may send a system notification that the cross-tool identity changed for some tools. -Those tools had cross-tool hash fields configured but no algorithm; the previous release treated -that as Hash code and the upgrade records Hash code explicitly, so the definition moved while the -stored hashes did not. The rehash it suggests is safe, recomputes the same values, and records the -new baseline so the notice does not repeat. +identity check after the upgrade may send a system notification that the cross-tool identity +changed for some tools. Those tools had cross-tool hash fields configured but no algorithm; the +previous release treated that as Hash code and the upgrade records Hash code explicitly, so the +definition moved while the stored hashes did not. The rehash it suggests is safe, recomputes the +same values, and records the new baseline so the notice does not repeat. Permission changes for custom roles are described under [Dedupe Pools](/triage_findings/finding_deduplication/pro__dedupe_pools/#custom-roles-on-upgrade).