Skip to content

fix: honor the S3 profile name and file and Hadoop's addressing mode for custom endpoints - #5872

Open
dwsmith1983 wants to merge 1 commit into
apache:mainfrom
dwsmith1983:fix/s3-profile-and-virtual-hosted
Open

fix: honor the S3 profile name and file and Hadoop's addressing mode for custom endpoints#5872
dwsmith1983 wants to merge 1 commit into
apache:mainfrom
dwsmith1983:fix/s3-profile-and-virtual-hosted

Conversation

@dwsmith1983

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #4245, closes #2802.

Rationale for this change

Two gaps in how the native S3 store is configured from fs.s3a.* settings.

The profile credentials provider ignored fs.s3a.auth.profile.name and fs.s3a.auth.profile.file, so a job that selects a named profile or a non-default credentials file on the Hadoop side got the SDK defaults on the native side.

fs.s3a.path.style.access was applied inverted: true set object_store's virtual_hosted_style_request to true and then appended /bucket to the endpoint, which object_store, treating a virtual-hosted endpoint as already containing the bucket, sent as a path-style URL anyway. The net effect was that every custom endpoint was addressed path-style whatever the flag said, and virtual-hosted addressing (bucket.host) was never produced.

What changes are included in this PR?

  • CredentialProviderMetadata::Profile carries name and file, read through the existing per-bucket config lookup with blanks treated as unset, and passed to the SDK builder. The file is loaded in credentials-file format, which is what Hadoop's ProfileAWSCredentialsProvider does; with no file the SDK default applies as before. aws-runtime becomes a direct dependency because the file-kind types re-exported by aws_config are deprecated aliases; it was already in the tree.
  • path.style.access is parsed the way Hadoop's Configuration.getBoolean parses it (default false, non-boolean text falls back to the default), virtual_hosted_style_request is its negation and is always passed, and normalize_endpoint returns the endpoint together with the effective mode so the two cannot disagree: virtual-hosted rebuilds scheme://bucket.host[:port][/path], path-style leaves the endpoint alone for object_store to append the bucket, and an IP-literal host forces path-style the way the AWS SDK's endpoint rules do, so http://127.0.0.1:9000 keeps working without the flag. localhost is not special-cased, matching Hadoop. The s3.amazonaws.com skip is unchanged.
  • The data sources page lists the two profile keys and explains the addressing rule.

Behavior change: a custom fs.s3a.endpoint with fs.s3a.path.style.access unset is now addressed virtual-hosted, as Hadoop S3A addresses it. Deployments on MinIO, Ceph RGW or similar services behind a hostname that relied on the previous always-path-style behavior need fs.s3a.path.style.access=true, which Hadoop already requires for those services; IP-address endpoints keep working either way. Vendor alias schemes are unaffected because the JVM side already synthesizes the flag for them.

How are these changes tested?

52 unit tests in the S3 module, 11 of them written first and failing on the previous code (the profile metadata carried no name or file; path.style.access unset produced no flag; a hostname endpoint was never rewritten; an IP endpoint was rewritten to bucket.127.0.0.1). Coverage: the flag unset, true, false, mixed case with whitespace, and an invalid value; per-bucket override of the flag and of the endpoint, each against a global value set the other way; the s3.amazonaws.com skip in both modes; scheme-less, http://, port, trailing slash, path suffix, an AWS regional host and a dotted bucket name; IPv4 and IPv6 hosts with and without a port; create_store called with a custom endpoint in each mode and with an IP endpoint; profile name only, file only, both, neither, blank, trimmed and per-bucket, plus the provider chain building with the profile provider among others. Two existing tests that had encoded the inverted flag were replaced; one that asserted an empty config now asserts the endpoint key is absent, since the flag is always present.

The create_store calls show object_store accepts each flag and endpoint pair but do not issue a request, since object_store parses the endpoint on first use. The four Scala tests that set an endpoint either set path-style access or are pure config-translation tests, so none needed changing. Full core crate suite passes, clippy and fmt clean.

…for custom endpoints

The profile credentials provider ignored fs.s3a.auth.profile.name and
fs.s3a.auth.profile.file, and fs.s3a.path.style.access was applied
inverted, so every custom endpoint was addressed path-style whatever the
flag said and virtual-hosted addressing was never produced.

Carry the profile name and file into the SDK builder, derive the
virtual-hosted flag from the path-style setting the way Hadoop does,
rebuild the endpoint as bucket.host for virtual-hosted addressing while
forcing path-style for IP-literal hosts as the AWS SDK does, and return
the effective mode with the endpoint so the two cannot disagree.

Closes apache#4245
Closes apache#2802
@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading labels Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scan Parquet scan / data reading bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support fs.s3a.auth.profile.name and fs.s3a.auth.profile.file for ProfileCredentialsProvider virtual_hosted_style_request bad calculation

1 participant