Inconsistency in S3 Transfer Operators #37213
Assignees
Labels
area:providers
good first issue
kind:feature
Feature Requests
provider:amazon-aws
AWS/Amazon - related issues
Comments
|
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
|
Hi, I would like to bé assigned on this as my first issue, if possible. @jroachgolf84 would it be ok for you? |
|
@bricefotzo, were you able to complete this issue? If not, I'm looking forward to working on it. |
|
@jroachgolf84 I have added those changes 3da0dcf |
|
@bricefotzo do you plan to raise a PR for this issue? |
|
Yes, I'll create a PR this week |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Standardize similar AWS transfer operators to have similar functionality, as well as similar naming conventions.
This would start with updating transfer operators using S3 as a source or destination to have similar behavior and naming.
Use case/motivation
There seems to be substantial discrepancies between transfer operators, especially those using S3 as a source or destination. For example, in the
S3ToSFTPOperator, only a single file can be transferred from the SFTP to the S3 bucket. However, in theS3ToFTPOperatoran arbitrary number of files can be transferred from S3 to an FTP site. To add to this, theFTPToS3Operatoronly supports a single file being transferred.These sort of discrepancies exist for a number of transfer operators. In addition to that, parameter names are quite different from operator to operator, which makes using them quite difficult.
This will allow for a more seamless experience for Airflow users, and ensure common functionality across similar tools.
Related issues
No related issues.
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: