When working with R2 as an "s3-compatible" endpoint – avoid multi-part upload…

When working with R2 as an "s3-compatible" endpoint – avoid multi-part uploads unless you have direct control over the splitting process.

Cloudflare's R2 is my favorite cloud storage. No-egress fees is music to my ears.

But its compatibility with S3 is anything but flawless.

I first mentioned this in relation to using Transmit.

While setting up a parquet-based data pipeline on airbyte between stripe and r2, I ran into the following failure on large tables:

"All non-trailing parts must have the same length"

This seems to be a specific implementation detail of the airbyte + R2 connector. Airbyte splits large files asymmetrically but with R2 multi-part uploads must all be the same length – except the last piece.

The fastest work-around (and this was a fast client project) – was to set parquet block size to 8mb, which greatly reduced the chance that an individual parquet file would be split into multi-part.