feat: simplify contradicting and redundant predicates on a column - #25207
Open
wudidapaopao wants to merge 4 commits into
Open
feat: simplify contradicting and redundant predicates on a column#25207wudidapaopao wants to merge 4 commits into
wudidapaopao wants to merge 4 commits into
Conversation
…ication
`simplify_predicates` grouped every `column <op> literal` comparison by
column, including ones whose literal is NULL, and then reduced each group
with `ScalarValue::try_cmp`. That comparison follows sort order, where NULL
is an ordinary value below every other one, rather than SQL three-valued
logic. A predicate such as `a > NULL` was therefore treated as a real but
weaker lower bound and dropped as redundant:
a > NULL AND a > 5 => a > 5
`a > NULL` never evaluates to true, so the conjunction matches no row while
the simplified `a > 5` does. Skip comparisons against a NULL literal when
grouping so they are carried through untouched.
Queries do not reach this today because `SimplifyExpressions` folds
comparisons with NULL literals before `PushDownFilter` runs, but
`simplify_predicates` is public and callers can hit it directly.
`simplify_predicates` reduced the `>`/`>=` and `<`/`<=` comparisons on a column to their most restrictive bound, but never compared the two groups with each other, and only looked for contradictions between equalities. Conjunctions that no row can satisfy were therefore left in the plan, and comparisons already implied by an equality were still evaluated per row. Reason across the groups instead, taking the same approach DuckDB's `FilterCombiner::AddFilter` does: - Contradicting bounds reduce the conjunction to `false`, so that `EliminateFilter` and `PropagateEmptyRelation` can prune the plan they filter. `x > 6 AND x < 5` is unsatisfiable, and so is `x > 1 AND x < 1` because a strict comparison excludes the value the bounds share. Note that DuckDB stops short of the latter. - An equality pins the column to a single value, so it subsumes every other predicate that value satisfies, and contradicts the rest: `x = 5 AND x > 3` becomes `x = 5`, while `x = 5 AND x > 5` becomes `false`. - `!=` predicates now take part in the analysis. One is dropped once a bound already excludes its value, as in `x > 10 AND x != 5`, and one that contradicts an equality reduces the conjunction to `false`. A column whose predicates contradict each other now short circuits the whole list, since predicates on other columns cannot make the conjunction true again. `false` stands for a conjunction that never evaluates to true, which under three-valued logic includes evaluating to NULL. That is only equivalent for the predicates of a `Filter`, which keeps a row solely when they evaluate to true, and is where this runs.
github-actions
Bot
added
optimizer
Add unit tests for the reasoning that spans the comparison groups of a column, and sqllogictest cases that check the plans it produces: - an equality subsuming the predicates its value satisfies, and being rejected by each of the six comparison operators; - bounds that leave no value, for every combination of strict and inclusive comparisons, next to the inclusive pair that admits one; - one column's contradiction discarding the predicates on other columns; - `!=` being dropped once a bound excludes its value, and kept otherwise; - comparisons against a NULL literal staying untouched, which only a unit test can reach since `SimplifyExpressions` folds them beforehand.
wudidapaopao
force-pushed
the
simplify-predicates-contradictions
branch
from
September 11, 2026 20:31
e4043ff to
9fb1c19
Compare
Author
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25207 +/- ##
==========================================
+ Coverage 81.80% 81.94% +0.13%
==========================================
Files 1130 1133 +3
Lines 417754 423654 +5900
Branches 417754 423654 +5900
==========================================
+ Hits 341754 347143 +5389
- Misses 55875 55910 +35
- Partials 20125 20601 +476 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Grouping `!=` predicates by column moves them behind the ones that stay ungrouped, so `p_brand != 'Brand#45'` now follows `p_size IN (...)` in the conjunction. The predicates themselves are unchanged.
Author
|
Updated the TPC-H q16 plan. The filter changed from to
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
simplify_predicatesreduces the>/>=and the</<=comparisons on a column to their most restrictive bound, but never compares the two groups with each other, and only looks for contradictions between equalities. A filter no row can satisfy therefore still scans the table.WHEREclausea > 3 AND a < 1a > 3 AND a < 1EmptyRelationa > 1 AND a < 1a > 1 AND a < 1EmptyRelationa >= 1 AND a < 1a >= 1 AND a < 1EmptyRelationa = 7 AND a < 2a = 7 AND a < 2EmptyRelationa = 7 AND a != 7a = 7 AND a != 7EmptyRelationa = 7 AND a > 5a = 7 AND a > 5a = 7a > 10 AND a != 5a > 10 AND a != 5a > 10a >= 1 AND a <= 1still simplifies toa = 1: bounds meeting at a value both admit stay satisfiable.What changes are included in this PR?
fix: skip comparisons against a NULL literal when grouping.ScalarValue::try_cmpfollows sort order, where NULL sits below every other value, soa > NULL AND a > 5was reduced toa > 5althougha > NULLis never true. Reachable only through the publicsimplify_predicates, asSimplifyExpressionsfolds these first.feat: compare the groups with each other. Bounds leaving no value, and an equality that contradicts another predicate, reduce the conjunction tofalse; an equality drops what it subsumes;!=joins the analysis and is dropped once a bound excludes its value.test: unit tests andsqllogictestcases.Reducing to
falseis valid here because aFilterkeeps a row only when its predicate is true, making NULL andfalseinterchangeable.What is the testing strategy for this PR?
Unit tests in
simplify_predicates.rscover each comparison operator against an equality, every combination of strict and inclusive bounds,!=dropped and kept, and comparisons against NULL.simplify_predicates.sltchecks the plans and drops two# TODOmarkers this PR implements.Are there any user-facing changes?
Filters no row can satisfy no longer scan their input. Result sets and public APIs are unchanged.