dm03514

What’s the furthest a recipient will be forced to travel? How many offices per population area? What’s the current rate of calls and direct deposit modifications? At that rate what load will it put on the offices? How long is the average interaction? What will the throughput be if the current office staffing?

I've mentioend this whenever iceberg comes up. It's wild how immature the ecosystem is still. Duckdb itself lacks the ability to write iceberg....

https://duckdb.org/docs/stable/extensions/iceberg/overview.h...

Apache iceberg go ? Nope

https://github.com/apache/iceberg-go?tab=readme-ov-file#read...

Basically java iceberg is the only mature way to do this, it's not a very accessible ecosystem.

For a side project I'm using pyiceberg to sink streaming data to iceberg (using DuckDB as the stream processor):

https://sql-flow.com/docs/tutorials/iceberg-sink

It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it!

https://github.com/turbolytics/sql-flow

Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion:

https://sql-flow.com/docs/tutorials/clickhouse-sink

This makes the glue code trivial to sink into these different systems as long as arrow is used.

pfsh who needs to write data??? ;p

If you have streaming data as a source, I built a side project to write streaming data to s3 in iceberg format:

https://sql-flow.com/docs/tutorials/iceberg-sink

https://github.com/turbolytics/sql-flow

I realize it's not quite what you asked for but wanted to mention it. I'm surprised at lack of native iceberg write support in these tools.

Pyiceberg though was quite easy to use, arrow-based API was very helpful as well.

Interesting would love to learn more about their methodology.

Like are they straight searching “software developer” on indeed?

https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE

—-

It’s rough out there, I see former coworkers laid off last year and the year before still struggling to find work

First of all I think this article was put together quite well, it was succinct had a great flow and was very easy to follow. The data was laid out to support the case; overall it was great to read.

Understanding a problem requires a holistic view of the larger system. An egg producer may say prices are higher Because supply is lower due to culling. Part of that statement is true. The author then zooms out to better understand the larger system.

The truth of a matter deviates significantly from one party’s assertions.

it’s scary to me that people lack the skills, the desire or incentives to understand and objectively seek understanding of what’s happening around us especially in highly charged US political landscape. This isn’t like pure scientific pursuit of knowledge but a more practical day to day ability to question, understand and gather new information with the purpose of developing an objective understanding of the world.

Fun, I did a similar project some years ago , I always like it when multiple people arrive at the same approach without ever communicating :)

https://github.com/dm03514/test-engine/blob/master/tutorials...

https://github.com/dm03514/func-y-task-engine

Very Cool! Which use cases are you targeting?

I'm surprised at the relatively sparse state of streaming. I feel like there are:

- Flink - Spark - ~Benthos - Arroyo - Then mostly custom / bespoke frameworks

Maybe there just isn't that much money in it? But I think there's still lots of opportunity to improve the Dev/UX over the JVM and the enterprise solutions.

-----

I would say recovery is achieved through kafka consumer groups right now, which results in at least once processing semantics.

We also support websocket input for the bluesky firehose but that is completely ephemeral. I have a story in the backlog to write ahead to disk which should allow for tolerating process crash/failure.

The tumbling windows stores state using duckdb. The end user can configure a disk-based duckdb databases which achieves durability as well.

The troubleshooting was actually pretty funky, i didn't realize how tricky it could be with stateful stream processing. I certainly don't have a good story for it, but what I did was added a sql debug HTTP handler. When this is enabled it exposes the duckdb execution context over HTTP. This was how I debugged the tumbling window logic during development. I would start SQLFlow, query to make sure the windows were empty, send some known messages, query the windows to make sure they were aggregated correctly, wait for the windows to close and flush, query the tumbling window state, etc.

For troubleshooting it also supports prometheus metrics oriented towards stream processing, # of messages, processing duration, success/failures, etc.

SQLFlow also ships with a dev framework, this allows you to execute your sqlflow pipeline against a json input file to make sure the sql processing logic is correct. I wanted to give the ability to decouple testing the logic, from actually having to stream data.

I would love to know about more what you're hacking on!

I'm building a stream processing framework using DuckDB.

https://github.com/turbolytics/sql-flow

The goal is to create a stream processing framework that supports SQL jobs. Apache Flink supports this but is very heavy-weight overall. I work with cost constrained companies that just can't run Flink but still want access to high performance streaming primitives.

We are taking a slightly different approach from other competitors by building cloud-native tools for engineers. SQLFlow is built on DuckDB, Native Kafka Library, and Arrow. This allows SQLFlow to handle ~70k+ events / second with low memory overhead (~250MiB).

Would love your questions, thoughts and feedback or feature Ideas! Thank you

We just added Clickhouse sink support to SQLFlow:

- https://github.com/turbolytics/sql-flow/issues/100

- https://github.com/turbolytics/sql-flow/pull/116

The Clickhouse python API made this trivial thanks to supporting Arrow!

We have a tutorial explaining how to stream data from Kafka to Clickhouse using SQLFlow:

https://sql-flow.com/docs/tutorials/clickhouse-sink

Local benchmarks show 70,000 rows / second insert using a batch size of 50k!

I think iceberg solves a lot of big data problems, for handling huge amounts of data on blob storage, including partitioning, compaction and ACID semantics.

I really like the way the catalog standard can decouple underlying storage as well.

My biggest concern is how inaccessible the implementations are, Java / spark has the only mature implementation right now,

Even DuckDB doesn’t support writing yet.

I built out a tool to stream data to iceberg which uses the python iceberg client:

https://www.linkedin.com/pulse/streaming-iceberg-using-sqlfl...

Hello! Just released a version of SQLFlow that adds support for streaming data to iceberg tables.

https://github.com/turbolytics/sql-flow?tab=readme-ov-file#s...

pyiceberg does the heavy lifting. It is a great project with a lot of active development.

It's interesting to see how immature iceberg ecosystem is. For example duckdb iceberg does not support writing to tables, neither does the go iceberg project. Iceberg writing was just recently added to the pyiceberg project!

Would love to learn more about peoples actual use of iceberg in the wild.

Are you using iceberg? How are you writing data to it? Which iceberg features are you using? Do you leverage snapshots at all?

Thank you

Created a tutorial on using SQLFlow to interact with the Bluesky firehose:

https://www.linkedin.com/pulse/learning-sqlflow-using-bluesk...

I’m thinking of this as a lightweight (single node) alternative in the same way duckdb is focused on data that can be processed by a single node.

I think / hope sqlflow will be a viable/lightweight/cost effective/easy to operate alternative to flink when working with small-medium sized data (on the orders of <10,000 messages / second)

Hello, I’ve been working on a project that embeds duckdb for stream processing.

I just added support for websocket sources which enables sql over the Bluesky firehouse.

https://github.com/turbolytics/sql-flow?tab=readme-ov-file#c...

Duckdb does all the sql execution, and python is responsible for sourcing the data.

The project is still quite young and I’m very much still experimenting, but I’d love any feedback. Thank you.

This looks really interesting, I instantly found a drawing of my kids to upload because I thought he would love it, but I was unable to even try it once without paying.

I saw one paragraph on transaction isolation levels hidden in the word salad

I agree with the post :)

Writing a book was very personally fulfilling.

I wrote about my self published book earnings a couple years ago.

https://on-systems.tech/blog/130-2022-self-published-book-ea...

This quote got me pretty excited, and I since yesterday I started sketching out some ideas for other books ;p

"First, simply: you believe strongly in your view of the world, and you’re pursuing it with intensity. Right now nobody else is really able to download your brain. Your book becomes how others can download it."

Hello Leo!

Are you open for remote for the EM role, or is it on-site?

Thank you!

Hello everyone! I'm researching how to generate guaranteed globally unique IDS, and have been reading about Twitter's snowflake ID:

https://en.wikipedia.org/wiki/Snowflake_ID

To understand it better I created a fun weekend flake id generator using go!

https://github.com/turbolytics/flake

Have you used flake ids before at a company? Have you ever had to globally identify billions of entities? How did you do it?

I often work at small scale, so self incrementing primary keys in DB or UUID as primary key serve me fine. Many years ago I was working at a company that had to globally identify clickstream events and that is where I was introduced to flake ids. We used a 64bit id for during that time.

I chose 128bit here to provide for 64bit milliseconds epoch, 48bit worker identifier (allowing for mac address) and 16 bits (65k ids) per millisecond for high throughput scenarios.

I'm not using this project and don't have any intention to, but I thought it was fun to do and to share :p

This is a n00b question, but how do regulations apply to private human crewed missions in the US?

Are there clear guidelines already written? I’m assuming it inherits a lot of rules of general aviation? But since private firms launching into space is new, do the rules and regulations keep up? Or are they still to be written?

Hello Everyone, having lots of fun figuring out ways to use DuckDB in common data operations. Already seeing huge savings in traditional data engineering applications replacing data warehouse queries with duckdb over blob storage (parquet).

SQLFlow uses duckdb to process kafka streaming data.

The goal is to create an environment where stream logic can be expressed in pure SQL with a simple and performant deployment, leveraging kafka consumer groups for state management.

The new in-memory processor easily handles 45,000 messages / second for simple aggregation.

https://github.com/turbolytics/sql-flow?tab=readme-ov-file#b...

------

I'd really appreciate feedback on features that would make this more useful. It's basically just a POC at this point.

Thank you

Test containers is such a game changer for integration testing, they have language specific docker apis that make it trivial to bring up containers and verify that they are fully initialized and ready to accept connections.

Pretty much every project I create now has testcontainers for integration testing :)

I setup CI so it lints, builds, unit tests then integration tests (using testcontainers)

https://github.com/turbolytics/latte/blob/main/.github/workf...

Their language bindings provide nice helper functions for common database operations (like generating a connection uri from a container user)

https://github.com/turbolytics/latte/blob/main/internal/sour...

I use them in $day job use them in side projects use them everywhere :)

Auth0 truly did an amazing job on hiring for culture, being transparent and living their values.

The pre-acquisition Glassdoor reviews were astounding and back this up.

For myself, working at auth0 was truly life altering. It was the first time I was in a long standing multi-national environment.

It was the first time I was deeply exposed to other cultures and ways of living.

As a side effect of working at Auth0, This exposure completely reshaped my worldview and thoughts on luck and privilege. Watching Argentina inflation fluctuate as people I spent 8 hours a day with struggle to pay their bills and provide for their families as I live stably with no worry I really internalized how much pure raw luck plays in setting us up for life: Who we’re born to and where we’re born.

Sorry nothing positive to say here.

I’ve been using dbt and MDS for nearly 3.5 years and I believe the entire approach is profoundly broken. There’s really nothing “modern” about it, especially compared to software engineering.

https://on-systems.tech/blog/135-draining-the-data-swamp/

Building on Extracted operational data is hard at best and a business altering security liability at worse.

I believe The MDS trails at least a decade behind modern software engineering practices, lacking industry guidance and generic tooling to support: CI/CD, versioned deployment artifacts, zero downtime deployments, unit testing, observability, monitoring and alerting.

MDS Data engineering is a meme in the industry, the expectation of “100%” combined with the lack of modern tooling makes success really hard to achieve.

I’m currently running 25mpw, and have maxed out my free time available per week (3 children). The only way I can do more distance And experience more woods is by running faster :)

If I weren’t time constrained I’d def be happy continuing to jog, I feel like slower speeds are less injury prone for myself

I noticed this too, after losing my sense of smell/taste in june 2022 I've been able to easily tolerate spice that I wasn't able to tolerate prior.

My sense of taste and smell is ~30% since Covid in June of 2022. Luckily there’s still mild taste so eating isn’t completely joyless. another symptom that started at this time was extreme nasal snoring so bad that I wake myself up and that has never happened before and it just started happening in June 2022.

I finally got to an ENT, literally yesterday, where they did a nasal endoscopy. There was mild inflammation, but no major deviation or polyps or anything worrying the diagnosis from them was Covid related since nothing structural was observed.

I just got a taste and smell training kit that arrived from Amazon today and I think that’s basically my only course of action at this point. Here’s hoping

I was going to respond same thing :sweat:. this seems like _normal_ startup hiring based on my experiences. People pass, people make stuff up to pass. Sticker shock could have factored in for CEO. I def empathize with the author. They thought they were a shoe in, thought everything was smooth and then wasn't responded to promptly, that sucks. I'm sure it was a rollercoaster during the time, excited about the new role, opportunity and $ and then denied for no known reason.

Not sure if it's reasonable to jump to illegal or unethical information?

Lucky stiff

Hello, I write about software, tech lead and other random things!

https://on-systems.tech/

I just published a post 2 days ago about 2022 Self Published Book earnings! I was kind of hoping hn would find it interesting but no luck so far!

https://on-systems.tech/blog/130-2022-self-published-book-ea...