terraform-provider-aws: S3 settings on aws_dms_endpoint conflict with "extra_connection_attributes"

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

v0.11.13

Affected Resource(s)

  • aws_dms_endpoint.s3_settings

Terraform Configuration Files

resource "aws_dms_endpoint" "s3_raw" {
  endpoint_id = "s3-raw"
  engine_name = "s3"
  endpoint_type = "target"
  extra_connection_attributes = "dataFormat=parquet;"
  s3_settings {
    service_access_role_arn = "${aws_iam_role.role.arn}"
    bucket_name = "${var.s3_bucket}"
    bucket_folder = "${var.raw_data_path}/dms"
    compression_type = "GZIP"
  }
}

Expected Behavior

The “extra connection status” in the DMS endpoint should be:

bucketFolder=data/raw/dms;bucketName=MY_BUCKET_NAME;compressionType=GZIP;csvDelimiter=,;csvRowDelimiter=\n;dataFormat=parquet;

Actual Behavior

The “extra connection status” in the DMS endpoint is:

bucketFolder=data/raw/dms;bucketName=MY_BUCKET_NAME;compressionType=GZIP;csvDelimiter=,;csvRowDelimiter=\n;

Notice the lack of dataFormat=parquet

Steps to Reproduce

  1. terraform apply

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 89
  • Comments: 21 (5 by maintainers)

Commits related to this issue

Most upvoted comments

@aeschright is this being actively worked on?

Hi all! đź‘‹ Just wanted to direct you to our public roadmap for this quarter (Nov-Jan) in which this item has been mentioned.

Due to the significant community interest in resolving this issue, we will be looking at merging existing contributions soon.

We appreciate all the contributions and feedback thus far.

anyone working on this? seem the fix has not passed QA for a while?

@rory-lamendola @mchudoba Please upvote the pull request if you want it prioritised. It is not a full solution, but it will serve most of the use-cases out there by simply extending the supported s3_settings attributes to cover attributes that would otherwise need to be defined in extra_connection_attributes.