HDDS-16273. OM bootstrap never completes when a single SST file is larger than ozone.om.ratis.snapshot.max.total.sst.size - #11214
Conversation
…rger than ozone.om.ratis.snapshot.max.total.sst.size Leader: record an oversized file when nothing has been collected for the request yet, so every part carries at least one file; treat a file that exactly fills the budget as fitting. Follower: abort with a diagnosable IOException after two consecutive parts that bring no new file instead of looping forever.
There was a problem hiding this comment.
🟢 Approval recommended
The functional fix is well-scoped, matches the PR description, and is covered by new targeted tests; remaining feedback is non-blocking.
Pull request overview
Fixes an OM HA bootstrap livelock where incremental checkpoint download never completes if a single SST exceeds ozone.om.ratis.snapshot.max.total.sst.size, by ensuring leader transfers always make progress and follower transfers abort instead of looping forever.
Changes:
- Leader: allow a single oversized SST to be transferred alone (and treat exact-fit as in-budget).
- Follower: detect consecutive no-progress parts during snapshot download and fail fast with a diagnosable
IOException. - Tests/docs: add targeted regression tests and clarify the config description in
ozone-default.xml.
File summaries
| File | Description |
|---|---|
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java | Ensures oversized SSTs are still sent (one per part) and fixes exact-fit budget semantics. |
| hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java | Adds coverage for oversized-file transfer progress and updates batching assertion for exact-fit behavior. |
| hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/RDBSnapshotProvider.java | Adds “no progress” part detection to prevent infinite looping during snapshot download. |
| hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRDBSnapshotProvider.java | Adds regression test validating abort after consecutive no-progress parts. |
| hadoop-hdds/common/src/main/resources/ozone-default.xml | Documents that an oversized SST is still transferred to ensure progress. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
chihsuan
left a comment
chihsuan
left a comment
There was a problem hiding this comment.
Thanks for the patch! @Eason09053360 Overall, the approach looks good. I left a few minor inline comments.
It looks like the new tests check file selection and stalled downloads, but don’t verify that the follower completes the transfer when an SST exceeds the limit. Could we add coverage for that, since it’s the main issue this PR fixes?
| + " %d parts brought no new file into %s and the transfer is still incomplete. A single file on the" | ||
| + " leader is likely larger than its per-request SST size limit" | ||
| + " (OM: ozone.om.ratis.snapshot.max.total.sst.size).", | ||
| leaderNodeID, numParts, numNoProgressParts, candidateDir)); |
There was a problem hiding this comment.
nit: Missing exclusions could also cause the same issue. Would it make sense to say an oversized SST is one possible cause rather than the likely cause?
What changes were proposed in this pull request?
OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir()checks the per-request budget(
ozone.om.ratis.snapshot.max.total.sst.size) before recording a file. When a single SST file is larger thanthe whole budget it is the first non-excluded file of every request, so the leader answers with an empty
tarball each time. The follower's exclude list never changes, and
RDBSnapshotProvider.downloadDBSnapshotFromLeader()loops forever while holding the
OzoneManagermonitor: no error, no bound, no progress. The comparison was also<= 0, so a file exactly equal to the remaining budget was treated as over budget.Leader side: when a file does not fit and nothing has been collected for the request yet, record it anyway and
end the part, so every request transfers at least one file. The budget check is now
< 0, so an exact fit isnot over budget. The v1
OMDBCheckpointServletrecords before checking and was never affected.Follower side:
RDBSnapshotProvidercounts consecutive parts that bring no new file into the candidate dir andaborts with a diagnosable
IOExceptionafter two instead of looping forever. One no-progress part is toleratedfor transient causes such as a truncated exclude list. The
part : Nlog is now 1-based.The
ozone-default.xmldescription now states that an oversized file is still sent.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16273
How was this patch tested?
TestOMDbCheckpointServletInodeBasedXfer#testCollectFilesFromDirTransfersFileLargerThanMaxSize: twofiles larger than the budget are sent one per request, the limit is enforced again once the tarball holds a
file, and a file exactly filling the budget is accepted.
testTarballBatchingassertion changed to<=to match the exact-fit semantics; both parameterizations pass.TestRDBSnapshotProvider#testDownloadAbortsWhenPartsBringNoNewFile: the first empty part is tolerated,the second aborts with the new message.
checkstyle.sh— 0 violations.Generated-by: Claude Code (Claude Fable 5.1)