aduffy

So in the case from the README, where you're trying to curate a sample of your data, the only thing that you're reading is the metadata, UNTIL you run `export_files` and that actually copies the binary data to your local machine?

Are you suggesting that a newspaper making an endorsement is "malicious election interference", or that there was "malicious election interference" involved in them deciding not to publish the endorsement?

Ostensibly, the LA Times owner's daughter claims that the decision not to endorse is tied to the Biden-Harris administrations stance on Gaza. Not sure whether to take that at face value or just a cop-out.

https://www.nytimes.com/2024/10/26/us/los-angeles-times-endo...

I'm of two minds about this. On the one hand, I think there's a valid argument that traditional media is struggling to hold the attention of modern readers, and trust in news organizations is at all-time lows.

On the other hand, there has always been a solid separation between the editorial page and the newsroom of every major paper, just look at WSJ. WSJ has not endorsed a presidential candidate since 1928, but it's no secret that their editorial board is robustly conservative. The Post is clearly a left of center publication and their editorial board has opinions that they should be free to share. In a world where Musk has turned Twitter into a pay-to-play MAGA playpen, I don't think there is anything wrong with a newspaper *opinion section* providing a clear, well-articulated endorsement of a candidate based on their merits.

This post brought to you by smart money

So the maintainers that got the boot were working for sanctioned Russian organizations (or at least, could not prove that their particular employers were not under the broad OFAC sanctions regime).

Totally reasonable for the Kernel maintainers who largely live in US/EU to do this, it's hard enough working on open source without worrying about ending up in jail.

This has existed for a long time, it's called "RPA" or Robotic Process Automation. The biggest incumbent in this space is UiPath, but there are a host of startups and large companies alike that are tackling it.

Most of the things that RPA is used for can be easily scripted, e.g. download a form from one website, open up Adobe. There are a lot of startups that are trying to build agentic versions of RPA, I'm glad to see Anthropic is investing in it now too.

Geolocation does not use GPS, it uses IP addresses.

The lawsuit was bound to happen, the alternative being to accept the narrative without fighting back.

The Chinese National Security Laws allow the CCP to compel Chinese citizens and corporations anywhere they may be to perform work on behalf of their intelligence services. Even if you don’t think DJI is actively a threat, a government agency would be foolish to use anything of theirs.

Hopefully PagerGate made it clear to officials what the consequences can be of having an enemy actor infiltrate your physical supply chains.

- Brand new burner account

- upset about “AI slop” (the image is clearly not AI)

- mentions tech buzzwords that annoy you

- claiming the article is not as rigorous as an academic paper

Perhaps I’m just old school. But I miss the HN where the best way to get upvotes was to be insightful and not to send low-effort snarky replies

I’m much happier with the TSA having those scans than airlines and services like CLEAR. At least there are rules the government is held to about how the data is stored and privacy/deletion requirements.

I love Starlink, and it is a great way for many people who live outside of where it is profitable to run broadband to have a high-speed Internet connection.

What is concerning is how much direct control Elon wields over the service. I wouldn't be surprised if he disabled access to BlueSky, or other forms of censorship.

If that sounds ridiculous to you, keep in mind this is the same man who disrupted a live Ukrainian op that made use of Starlink: https://www.nbcnews.com/news/world/musk-stopped-ukraine-atta...

Those new service sector jobs: get paid to respond to HN comments

This is the most Tyler Cowen-coded response I could imagine, and I mean this in the best way possible.

Dropbox comes to mind.

Buried under the memes/vibes there is an actual reason this is important for data tools.

The previous generation of analytics/"Big Data" projects (think Hadoop, Spark, Kafka, Elastic) were all built in the JVM. They were monolithic distributed systems clusters hosted on VMs or on-premise. They were servers with clients implemented in Java. It is effectively impossible to embed a Java library into anything non-Java, the best you can do is fork a JVM with a carefully maintained classpath and hit it over the network (c.f. PySpark). Kafka has externally maintained bindings that lag the official JVM client.

Parquet was built during this era, so naturally its reference implementation was written in Java. For many years, the only implementation of Parquet was in Java. Even when parquet-cpp and subsequent implementations began to pop up, the Parquet Java implementation was still the best maintained. Over time as the spec got updated and new features made their way into Parquet, different implementations had different support. Files written by parquet-cpp or parquet-rs could not be opened via Spark or Presto.

The newer generation of data analytics tooling is meant to be easily embedded, so that generally means a native language that can export shared objects with a C ABI that can be consumed by the FFI layer of different languages. That leaves you a few options, and of those Rust is arguably the best for reasons of tooling and ecosystem, though different projects make different choices. DuckDB for example is an extremely popular library with bindings in several languages and it was built in C++ long after Rust became in-vogue.

While Vortex doesn't (yet) have a C API, we do have Python bindings that we expect to be the main way people use it.

Will and I actually work on Vortex :wave:

Perhaps we should clean up the wording in the intro, but yes there is in fact a file format!

We actually built the toolkit first, before building the file format. The interesting thing here is that we have a consistent in-memory and on-disk representation of compressed, typed arrays.

This is nice for a couple of reasons:

(a) It makes it really easy to test out new compression algorithms and compute functions. We just implement a new codec and it's automatically available for the file format.

(b) We spend a lot of energy on efficient push down. Many compute functions such as slicing and cloning are zero-cost, and all compute operations can execute directly over compressed data.

Highly encourage you to checkout the vortex-serde crate in the repo for file format things, and the vortex-datafusion crate for some examples of integrating the format into a query engine!

Yea so the thing you do for this is called "compaction", where you effectively merge the original + edits/deletes into a new immutable file. You then change your table metadata pointer to point at the new compacted file, and delete the old files from S3.

Due to the way S3 and the ilk are structured as globally replicated KV stores, you're not likely to get in-place edits anytime soon, and until the cost structure incentivizes otherwise you're going to continue to see data systems that preference immutable cloud storage.

React is arguably quite stable?

You're unlikely to find this with any analytic file format (including Vortex). The main reason is that OLAP systems generally assume an immutable distributed object/block layer (S3, HDFS, ABFS, etc.).

It's then generally up to a higher-level component called a table format to handle the idea of edits. See for example how Apache Iceberg handles deletes https://iceberg.apache.org/spec/#row-level-deletes

We have a TableProvider for use with Datafusion, checkout this crate and its examples: https://github.com/spiraldb/vortex/tree/develop/vortex-dataf...

open access link: http://www.vldb.org/pvldb/vol13/p1162-ferragina.pdf

These patent trolls are greedy, extractive, and contribute nothing to society while wasting vast public and private legal resources.

Fuck. Them. Excellent work to the entire litigation team at Cloudflare.

Thank you for the close reading! That’s definitely a mistake on my part, I’ll fix it shortly.

The dictionary quality was definitely highly sensitive to some of the tricks that the original authors implemented in their C++ code, many were documented in the paper but a few were not:

1. Always promoting single-bytes by boosting their scores by a factor of 8 in candidate search

2. Boosting the calculated gains of single-byte candidates by a factor of 8 to prevent them from falling off in later generations

3. Having an adaptive threshold for which symbols are included as the rounds go on

I didn't document these in the blog post to keep the content accessible, but it's definitely something you find once you start digging into compression ratios! Perhaps they will end up in a part 2 at some point.

[1]: https://github.com/spiraldb/fsst/blob/develop/src/builder.rs...

German strings are cool, and we're also using them in Vortex! They're also commonly referred to as "variable-length view arrays", which is what Arrow calls them [1]. They were first published by folks at TUM as part of the Umbra database (checkout Figure 4) [2].

German-style strings/views are not a compression algorithm, they're just a way for storing string data and making it quick to compare them in-memory. You can in fact store views, while storing the corresponding full-length strings in compressed format with FSST. We don't currently do that but we're working on it.

[1] https://arrow.apache.org/docs/format/Columnar.html#variable-...

[2] https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf

Probably the right choice. Go open the YT comments for just about any video and count how many high-quality discussions are being had. I'm usually surprised to see even one well-thought out question or comment.

Article version: https://www.computerenhance.com/p/zen-cuda-and-tensor-cores-...

This, 100%

As someone who got their first drivers license in California, I can say with certainty that there are in fact speed limits.