Are requests to Firehose compressed?

[pugnascotia]

I've been raking through the SDK code, trying to understand if compression works with Firehose requests. I understand from some AWS docs that compression is enabled by default, but I'm trying to find evidence of this. I've been trying to force compression using the awsconfig.WithRequestMinCompressSizeBytes option, and tracing requests with wsconfig.WithClientLogMode(aws.LogRequest|aws.LogRetries|aws.LogResponse), but without success.

Can anyone point me to how the Firehose client configures compression, or suggest another way of proving that requests to Firehose are being compressed?

[RanVaknin]

Hi @pugnascotia,

Firehose does the compression on the service side, not on the SDK side. With the SDK you can set the desired compression type like GZIP, then send the data as raw bytes and then Firehose does all of the work.

or suggest another way of proving that requests to Firehose are being compressed?

You can set S3 as your stream's destination, and Firehose will write the stream data to your desired bucket. You can then download the .gz file and see if its compressed or not.

Thanks,
Ran~

[pugnascotia]

Hi @RanVaknin, thanks for the reply. I'm not talking about what Firehose does with the data it receives, but how the AWS SDK applies compression to HTTP requests. I'm aware of the AWS_DISABLE_REQUEST_COMPRESSION and AWS_REQUEST_MIN_COMPRESSION_SIZE_BYTES environment variables and their corresponding configuration options, but I'm struggling to prove that the AWS SDK is applying compression to my HTTP requests to Firehose. This is important due to the network charges for moving data around e.g. between regions.

[lucix-aws]

The settings you're talking about manipulate operation-specific request compression - services can model that specific operations in their service support this compression, if so we generate support for it in their service client.

Today the only operation that supports this across aws is cloudwatch's PutMetricData. There's no concrete indicator of this in the SDK. Broadly speaking, if a service client's Options has the DisableRequestCompression and related fields, one or more operations in that service support the feature.

[lucix-aws]

#2762

[pugnascotia]

So to be 100% clear, client side HTTP request compression isn't actually supported in the SDK? If so, the official AWS docs are entirely misleading. I see you've raised an issue about this, but I do feel that a better outcome for AWS customers would be to extend compression support in the SDK.

[pugnascotia]

@lucix-aws is it possible to compress requests to Firehose e.g. using some middleware, or a custom http.Client? I've been experimenting with these approaches but so far without success. I managed to compress the request with a custom http.Transport, but the request failed because the request signature no longer matched.