nicornk

I hope AWS doesn’t dilute the DuckDB brand by calling their managed DuckDB offering SageMaker<something>…

This would be my bet and hope as well.

In that sense all popular ERP vendors seems to be the same.

I agree. Pixi solves all of that issues and is fully open source including the packages from conda-forge.

Too bad there is nowadays the confusion with anaconda (the distribution that requires a license) and the FOSS pieces of conda-forge. Explain that to your legacy IT or Procurement -.-

Fivetran tried to upstream write support but it was not accepted https://github.com/duckdb/duckdb-iceberg/pull/95

If you just want to enable ssh to ec2 instances (through SSM) using ssh i-… you can add the following lines to your ssh config

https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0...

host i-* IdentityFile ~/.ssh/id_rsa TCPKeepAlive yes ServerAliveInterval 120 User ec2-user ProxyCommand sh -c "aws ec2 start-instances --instance-ids %h ; aws ec2 wait instance-running --instance-ids %h ; aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user %r --ssh-public-key 'file://~/.ssh/id_rsa.pub' --availability-zone $(aws ec2 describe-instances --instance-ids %h --query 'Reservations[0].Instances[0].Placement.AvailabilityZone') ; aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

This will also allow VSCode remote development.

The blog post echos my experience that duckDB just works (due to superior disk spilling capabilities) and polars OOMs a lot.

Snowflake usually unloads data to an internal stage bucket in the same region as your snowflake account. If you use an s3 gateway endpoint getting that data is free of egress charges.

and this is unfortunately the case for Velox.

Yeah, finally a method to figure out the org id of all those snowflake internal stage buckets that snowflake does not want to share for our VPC Endpoint Policies…

So how did you „further optimized [buildkit] to build Docker images up to 40x faster“

Pixi is great, highly recommended!

There are Aldis in Germany with self checkout.

Thanks for sharing your experience. I always think of starting in that space but your post shows me it’s probably better to wait for a few more months.

With S3 you can do something similar my misusing the multiparty upload functionality, e.g.: https://github.com/fsspec/s3fs/blob/fa1c76a3b75c6d0330ed03c4...

Isn’t it a little bit unrealistic that your proxy fleet can keep up with the massive traffic that some s3 buckets experience? I think every solution that Proxies traffic to s3 will at a certain point break.

SAP

Uploading data into AWS is free.

Really interesting going through your code, thanks for sharing. Why did you opt for running the parallelization code in a separate python process (s3op)?

You might want to update section ‚Caution: Overwriting data in S3‘ in the docs since S3 offers strong read after write consistency since dec 2020.

https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

That‘s basically MS Teams.