Conversation
…hemaTransforms and Beam YAML - Expose `with_gcp_adc` (`withGcpAdc`) in KafkaIO read/write SchemaTransform, Python xlang IO and YAML - Add `num_partitions` in KafkaIO read SchemaTransform, Python xlang IO and YAML. This avoids the need to Kafka instance connection at pipeline submission time for Dataflow runner v1 (Streaming runner) if using SchemaTransform
|
Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| public abstract Boolean getRedistributeByRecordKey(); | ||
|
|
||
| @SchemaFieldDescription( | ||
| "Whether to use Google Cloud Platform Application Default Credentials (ADC) for" |
There was a problem hiding this comment.
Nit: Google auth library docs typically use the name "Application Default Credentials" or in rare cases "Google Application Default Credentials" since the provider can be used for Google APIs in general.
| + " construction time.") | ||
| @SchemaFieldNumber("18") | ||
| @Nullable | ||
| public abstract Integer getNumPartitions(); |
There was a problem hiding this comment.
In most cases the partition set for a topic will match the range of integers from 0 to N, but users may arbitrarily retire individual partitions.
Examples:
- Create a partition per entity with a non-sequential integer id (e.g., a partition per player in a game).
- Retire a partition after the last usable offset is used to produce a record.
Replacing the number of partitions property with a topic partition list property would spare adding it later with more boilerplate to validate the different combinations of configuration properties.
| + " authenticating with a Google Managed Kafka cluster.") | ||
| @SchemaFieldNumber("17") | ||
| @Nullable | ||
| public abstract Boolean getWithGcpAdc(); |
There was a problem hiding this comment.
Nit: This doesn't match the name used to configure the transform. Application default credentials is annoyingly long to spell so I'm not opposed to ADC, but I'd consider keeping it the same for consistency and clarity to users who might be unfamiliar with the acronym.
2 participants
Expose
with_gcp_adc(withGcpAdc) in KafkaIO read/write SchemaTransform, Python xlang IO and YAML. This option is already present in Java KafkaIO but not exposed to xlang. It make Python and YAML user easier to use Google managed Kafka (GMK).Add
num_partitionsin KafkaIO read SchemaTransform, Python xlang IO and YAML. This avoids the need to Kafka instance connection at pipeline submission time for Dataflow runner v1 (Streaming runner) if using SchemaTransformPlease add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.