HNHacker News
TopNewBestAskShowJobs

georgewfraser

2,609 karma · joined October 28, 2012

fivetran.com

twitter.com/frasergeorgew

george at fivetran dot com

submissionscomments
georgewfraser··on Cryo–electron microscopy breaks the atomic resolution barrier
When I was in grad school I was a TA for a biochemistry class and the prof asked me to deliver the lecture on crystallography while he was on vaca. Part of the lecture was about how electron microscopy was easier but pretty inaccurate compared to x-ray crystallography. But the slides were 10 years out of date, the accuracy of EM was still not as good but the gap was much smaller. So I updated them with more recent EM results from the same labs.

When the prof came back he spent the first 5m of the next lecture "correcting" me, explaining that EM sucked, x-ray was the only good technique. I realized that he had basically made up his mind decades earlier and no matter what changed in the technology he wasn't going to change his opinion. Science has its own politics, similar to actual politics in that conclusions drive observations, but instead of being about minimum wage or housing supply it's about arcane scientific techniques.

georgewfraser··on Applying Textbook Data Structures for Real Life Wins
It seems to me that under this scheme, if I make a single erroneous identify call, I will irreversibly merge two users. This is a surprising approach. Given that identify calls may occasionally be wrong, I would expect that

  identify(anonymous_id, new_canonical_id)
would map anonymous_id => new_canonical_id, but would leave the rest of the set find(anonymous_id) alone.
georgewfraser··on Applying Textbook Data Structures for Real Life Wins
Do I understand correctly that when someone calls

  identify(anonymous_id, new_canonical_id)
you merge the entire set of anonymous ids associated with

  find(anonymous_id)
with the set of anonymous ids associated with new_canonical_id?
georgewfraser··on Why is Snowflake so Valuable?
In order to grow into this valuation, snowflake will need to become the biggest data warehouse company ever. They have a great product and that will probably happen! But you need to be highly confident of the best case scenario happening to buy at the current price.
georgewfraser··on Apache Arrow and MinIO
For sure, but you're not going to fix that by making your own data lake using, for example, Parquet-on-S3. You're still going to pay the cost of compute when you analyze that data, and a well-optimized commercial database system is extremely hard to beat. Even if you look at Presto, and you exclude the people costs of managing it yourself, it still can't beat the commercial systems: https://fivetran.com/blog/warehouse-benchmark
georgewfraser··on Apache Arrow and MinIO
I agree that no matter what you're going to store unstructured binary data, like audio files, in object storage. But that is perfectly compatible with storing structured data in a relational database.

> they are great if the use case is to get answers to questions like "top 3 operators per team in volume of calls..."

You are straw-manning Snowflake/BQ. Just because they are SQL database systems doesn't mean you have to do 100% of your analysis in SQL. You can use other systems, like Spark, PyTorch, Tensorflow, to work with data that you manage inside a RDBMS. There are some issues with bandwidth getting data between systems, but these issues are getting solved (by Arrow!) and in the meantime unload-to/load-from S3 is a good workaround.

I've heard a lot of people make these same arguments and I've tentatively concluded it's mostly motivated reasoning. Engineers like to engineer things. They start by trying to make the obvious, boring system work, but when they run into an obstacle they immediately jump to "I need to build a new system using $TECHNOLOGY."

georgewfraser··on Apache Arrow and MinIO
> Snowflake and BigQuery are quite expensive for big datasets.

This is just flatly untrue---they are nearly the same cost/TB as object storage, and they store everything in compressed columnarized format, so they're about as efficient as you can get.

I have heard many people make the same claim. I can't figure it out. Is there something wrong with my calculator???

georgewfraser··on Apache Arrow and MinIO
I am truly baffled by the desire to build new data lakes. I understand why Hadoop data lakes came to be: Teradata charged a lot for storage. But now there are database systems that use object storage under-the-hood: Snowflake, BigQuery, Redshift RA3, Databricks Delta Lake. They give you the same low cost of object storage, but with a nice high-level relational interface. I am mystified by why someone would want to build their own data lake rather than just use one of these systems.
georgewfraser··on Please stop calling databases CP or AP (2015)
OK, my comment was over the top, apologies to Kleppmann if you’re still reading. He is a thought leader, and he’s the first introduction to distributed systems for a lot of people, and sometimes people stop with his work and don’t learn how much depth this field has. But that’s not really a criticism of him. And nobody should be compared to Nassim Taleb, I was clearly letting the heat wave get to me ;)
georgewfraser··on Please stop calling databases CP or AP (2015)
It’s a trivial observation that tells you nothing about the trade offs of building distributed systems.
georgewfraser··on Please stop calling databases CP or AP (2015)
Not that I know of. His lectures are amazing though.
georgewfraser··on Please stop calling databases CP or AP (2015)
YES!
georgewfraser··on Please stop calling databases CP or AP (2015)
Klepmann is a wannabe thought-leader, who repackages old ideas into pseudo-profound “insights”, designed to capture the attention of amateurs rather than move the state of the art forward. He’s the Nassim Taleb of distributed systems. If you’re looking for enlightenment, read Daniel Abadi or Andy Pavlo.
georgewfraser··on Please stop calling databases CP or AP (2015)
The CAP theorem is so dumb and yet so influential. If you want to gain a more useful understanding of the trade offs of distributed systems, I highly highly recommend the “PACELC theorem” by Daniel Abadi:

https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf

georgewfraser··on We Replaced an SSD with Storage Class Memory
Andy Pavlo talks about this in his class at CMU. You shouldn’t expect to get better performance by running a disk-optimized storage engine on memory, because you’re still paying all the overhead of locks and pages to work around the latency of disk, even though that latency no longer exists. Instead, you have to build a new, simpler storage engine that skips all the bookkeeping of a disk-oriented storage engine.

https://youtu.be/a70jRWLjQFk

georgewfraser··on The Splitgraph Data Delivery Network – query over 40k public datasets
Postgres foreign data wrappers is a weird choice of engine. Most queries to this service will be scans, in which case a column-oriented, vectorized, massively parallel engine like Presto will be 1000 times faster or so. Postgres’ underlying engine is optimized for scenarios where you read a small number of rows using an index.
georgewfraser··on Arm co-founder: Sale to Nvidia would be a disaster
I find it baffling that SoftBank isn’t simply taking ARM public. If ever there was a company that should be independent, it’s ARM. Is there a good explanation out there for why this isn’t the obvious choice?
georgewfraser··on I want to have an AWS region where everything breaks with high frequency
I think people overestimate the importance of failures of the underlying cloud platform. One of the most surprising lessons of the last 5 years at my company has been how rarely single points of failure actually fail. A simple load-balanced group of EC2 instances, pointed at a single RDS Postgres database, is astonishingly reliable. If you get fancy and build a multi-master system, you can easily end up creating more downtime than you prevent when your own failover/recovery system runs amok.
georgewfraser··on ZipPy: fast and lightweight Python 3 implementation using the Truffle framework
Yes
georgewfraser··on DuckDB: SQLite for Analytics
It’s much faster for analytical queries, because it’s a column store rather than a row store.
georgewfraser··on What every software engineer should know about Apache Kafka
ksql does not solve any of the hard consistency or contention problems you will face if you attempt to use Kafka as a datastore. Consider the simplest possible example: you write an "update event" to a topic and then read a ksql view of that topic. The view may or may not yet reflect the update. This is called read-after-write consistency, and you will need to create it in your application code.
georgewfraser··on What every software engineer should know about Apache Kafka
This notion of “stream-table duality” might be the most misleading, damaging idea floating around in software engineering today. Yes, you can turn a stream of events into a table of the present state. However, during that process you will eventually confront every single hard problem that relational database management systems have faced for decades. You will more or less have to write a full-fledged DBMS in your application code. And you will probably not do a great job, and will end up with dirty reads, phantoms, and all the other symptoms of a buggy database.

Kafka is a message broker. It’s not a database and it’s not close to being a database. This idea of stream-table duality is not nearly as profound or important as it seems at first.

georgewfraser··on IQ is largely a pseudoscientific swindle (2019)
I think we all know what happened here: Nassim Taleb took an IQ test, it didn’t show him to be the great genius he knows himself to be, and so the entire concept of IQ must be invalid.
georgewfraser··on 73% Of Inmates At An Ohio Prison Test Positive For Coronavirus. 0 deaths so far
10 prisoners in Ohio have now died of COVID, 1 from the prison mentioned in this article and 9 from another prison where most prisoners are infected: https://drc.ohio.gov/Portals/0/DRC%20COVID-19%20Information%...
georgewfraser··on Rust Survey 2019 Results
I have used both, and I’ve found rust analyzer to be better but not amazing. It seems it often “gives up” and autocomplete stops working. It is improving over time though.
georgewfraser··on Covid-19 Antibody Seroprevalence in Santa Clara County, California
There are now multiple sources of evidence pointing to a very low true infection fatality rate, as low as 0.1% in some regions. Iceland has been testing a semi-random sample using PCR since early in the epidemic, and there have been several small antibody surveys in “hot spots” showing infection rates over 15%. You can also use CDC surveillance of flu-like illnesses to estimate the total number of infected nationwide; this morning I posted an analysis of this data estimating only 1 in 40 cases were detected in March, consistent with the Stanford results: https://fivetran.com/blog/covid-19-count
georgewfraser··on Using SIMD to aggregate billions of values per second
Why your own dialect instead of Calcite SQL or ZetaSQL?
georgewfraser··on Using SIMD to aggregate billions of values per second
What dialect of SQL are you using?
georgewfraser··on SpaceX bans Zoom over privacy concerns
You can have 200, and you can also live-stream large meetings. It is a bit weird that you have to split the stream, but it's worked great for us doing ~300 person all-hands meetings.
georgewfraser··on SpaceX bans Zoom over privacy concerns
I've found Hangouts to be equal in call quality to Zoom, and it runs entirely in the web browser. The only down side is it consumes a LOT of power, so you need to stay plugged in while you use it.
← PreviousPage 4 of 11Next →