fix: honor the S3 profile name and file and Hadoop's addressing mode for custom endpoints - #5872
Open
dwsmith1983 wants to merge 1 commit into
Open
fix: honor the S3 profile name and file and Hadoop's addressing mode for custom endpoints#5872dwsmith1983 wants to merge 1 commit into
dwsmith1983 wants to merge 1 commit into
Conversation
…for custom endpoints The profile credentials provider ignored fs.s3a.auth.profile.name and fs.s3a.auth.profile.file, and fs.s3a.path.style.access was applied inverted, so every custom endpoint was addressed path-style whatever the flag said and virtual-hosted addressing was never produced. Carry the profile name and file into the SDK builder, derive the virtual-hosted flag from the path-style setting the way Hadoop does, rebuild the endpoint as bucket.host for virtual-hosted addressing while forcing path-style for IP-literal hosts as the AWS SDK does, and return the effective mode with the endpoint so the two cannot disagree. Closes apache#4245 Closes apache#2802
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4245, closes #2802.
Rationale for this change
Two gaps in how the native S3 store is configured from
fs.s3a.*settings.The profile credentials provider ignored
fs.s3a.auth.profile.nameandfs.s3a.auth.profile.file, so a job that selects a named profile or a non-default credentials file on the Hadoop side got the SDK defaults on the native side.fs.s3a.path.style.accesswas applied inverted:trueset object_store'svirtual_hosted_style_requestto true and then appended/bucketto the endpoint, which object_store, treating a virtual-hosted endpoint as already containing the bucket, sent as a path-style URL anyway. The net effect was that every custom endpoint was addressed path-style whatever the flag said, and virtual-hosted addressing (bucket.host) was never produced.What changes are included in this PR?
CredentialProviderMetadata::Profilecarriesnameandfile, read through the existing per-bucket config lookup with blanks treated as unset, and passed to the SDK builder. The file is loaded in credentials-file format, which is what Hadoop'sProfileAWSCredentialsProviderdoes; with no file the SDK default applies as before.aws-runtimebecomes a direct dependency because the file-kind types re-exported byaws_configare deprecated aliases; it was already in the tree.path.style.accessis parsed the way Hadoop'sConfiguration.getBooleanparses it (default false, non-boolean text falls back to the default),virtual_hosted_style_requestis its negation and is always passed, andnormalize_endpointreturns the endpoint together with the effective mode so the two cannot disagree: virtual-hosted rebuildsscheme://bucket.host[:port][/path], path-style leaves the endpoint alone for object_store to append the bucket, and an IP-literal host forces path-style the way the AWS SDK's endpoint rules do, sohttp://127.0.0.1:9000keeps working without the flag.localhostis not special-cased, matching Hadoop. Thes3.amazonaws.comskip is unchanged.Behavior change: a custom
fs.s3a.endpointwithfs.s3a.path.style.accessunset is now addressed virtual-hosted, as Hadoop S3A addresses it. Deployments on MinIO, Ceph RGW or similar services behind a hostname that relied on the previous always-path-style behavior needfs.s3a.path.style.access=true, which Hadoop already requires for those services; IP-address endpoints keep working either way. Vendor alias schemes are unaffected because the JVM side already synthesizes the flag for them.How are these changes tested?
52 unit tests in the S3 module, 11 of them written first and failing on the previous code (the profile metadata carried no name or file;
path.style.accessunset produced no flag; a hostname endpoint was never rewritten; an IP endpoint was rewritten tobucket.127.0.0.1). Coverage: the flag unset,true,false, mixed case with whitespace, and an invalid value; per-bucket override of the flag and of the endpoint, each against a global value set the other way; thes3.amazonaws.comskip in both modes; scheme-less,http://, port, trailing slash, path suffix, an AWS regional host and a dotted bucket name; IPv4 and IPv6 hosts with and without a port;create_storecalled with a custom endpoint in each mode and with an IP endpoint; profile name only, file only, both, neither, blank, trimmed and per-bucket, plus the provider chain building with the profile provider among others. Two existing tests that had encoded the inverted flag were replaced; one that asserted an empty config now asserts the endpoint key is absent, since the flag is always present.The
create_storecalls show object_store accepts each flag and endpoint pair but do not issue a request, since object_store parses the endpoint on first use. The four Scala tests that set an endpoint either set path-style access or are pure config-translation tests, so none needed changing. Full core crate suite passes, clippy and fmt clean.