kleebeesh

I'm a random dude on the Internet, but my partner completed her PhD at MIT. While there I knew and knew of a few PhD grads who worked at MIT in some non-tenure-track role (postdoc, staff researcher, etc). Typically for a couple years and then they get a better-paying or more permanent job. But several remained "affiliated" in some way. They kept their MIT website/email, some in academia continued to collaborate to some extent. Things like that. But AFAIK they weren't getting a paycheck from MIT. And it's somewhere between neat and genuinely professionally valuable to be affiliated w/ a prestigious university, so I don't blame them for claiming affiliation. My best guess is he's "affiliated" in a similar way.

Neat!

> Right now, accessing my apps requires typing in the IP address of my machine (or Tailscale address) together with the app’s port number.

You might try running Nginx as an application, and configure it as a reverse proxy to the other apps. In your router config you can setup foo.home and bar.home to point to the Nginx IP address. And then the Nginx config tells it to redirect foo.home to IP:8080 and bar.home to IP:9090. That's not a thorough explanation but I'm sure you can plug this into an LLM and it'll spell it out for you.

Maybe a more accurate take: Half-assed soft deletion definitely isn't worth it.

If you're just going to throw in some deleted bool or deleted_at timestamp without thorough testing, you might as well just skip it. It's virtually certain to go wrong.

Maybe out of touch. I've used them all (Mint, ynab, pocketsmith, buxfer, some google-sheets-based app, probably tried a dozen others). All of the auto-sync features had obvious bugs. It works for a few months and then you end up with junk data. But if it works for you, carry on!

I guess it comes down to the tradeoffs you're willing to accept for privacy. I personally find it quite sketchy that Plaid takes your username and password to your literal money and then does some opaque screen-scraping just to grab your transactions. I also spent a lot of time working with their API and it did not inspire confidence.

I've been using firefly over two years now. Data entry has not been been a problem:

* Run the data-importer as a separate container. Takes maybe 20 minutes to configure correctly if you already know docker and docker-compose.

* Download the transactions as CSVs from each of my 5 or so credit card and bank accounts that get regular use.

* Upload them through the data-importer. It lets you configure and save settings for each CSV format. I took five minutes to set that up the first time I imported CSVs and just keeping using the same settings.

I upload all of my transactions once a month and it takes about an hour to download them, import them, and categorize all of them (I also have a bunch of rules to auto-categorize, but there are inevitably a dozen or so bespoke transactions).

I've found that any of the solutions built on top of syncing backends like Plaid inevitably have issues: duplicates, missing txs, debit/credit mixed up. I even built my own custom Plaid-to-Firefly syncer at one point and found the data quality was very mixed, even when all my accounts are at major US banks. The data-importer takes some more up-front work, but it's more secure, way more predictable, and generally a solved problem.

Sadly he's not some kind of local hero. The area much prefers sports to academics. I studied CS there for undergrad and knew of him but rarely heard his name mentioned or celebrated.

Ah, totally misread it. Thanks.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

I've personally had some painful experiences with refreshing materialized views in Postgres. In particular, highly variable performance on read replicas that were receiving a refreshed matview every few minutes. Maybe we were just doing it wrong, but I tend to avoid it if I can. Plus the eventual consistency can introduce confusion.

In any case, there's an interesting feature called Incremental View Maintenance that is being worked on by some Postgres developers: https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc...

This would let us define a materialized view that gets automatically updated as the source tables change. When I last checked (late 2021), they were saying it might land in PG15.

Thanks for the feedback -- sorry I missed this a couple days ago! Had no clue it got posted to HN.

FWIW, I've found GIN is a bit faster if you're just looking to filter. IIRC it was maybe 10-15% faster for the particular use-case I was looking at. So, worth a try, but don't expect a 10x improvement.

Yeah it's a shame the concat_ws function doesn't quite work here.

I've had success using Slick, an ORM-ish Scala library, to abstract away this tedious concatenation in app code.

Thanks for the feedback -- sorry I missed this a couple days ago! Had no clue it got posted to HN.

I've deployed a solution that uses roughly this same method with multiple tables. I experimented with a materialized view that would centralize all the text columns, but ultimately found that it was much simpler and fast enough to have a single expression index in each of the tables. Then the query either joins the tables and checks each column, or you can run a separate query for each of the tables and stitch together the results in app code.

> I think maybe one reason people shy away from things like Presto (and the above) is the uneven performance guarantees; waiting for an unoptimized Hadoop or Orcfile query by accident because you joined on something or another is fine for one-offs, but might become costly in prod workflows.

Right, so my question is: how is that solved with Hydra? Seems like you'd arrive at the same issue?

Looks neat, but wasn't this the promise of Presto? Presto didn't seem to really work out. From what I've seen it converged to a mostly analytical engine. It's still very useful, but I've never seen it used (successfully) in an OLTP workload. Maybe there's some difference in the intended product trajectory that I'm overlooking here?

Does anyone know if there is a way to monitor sudden changes in planning behavior for a given query? For example, I'd like to monitor/alert on this kind of situation: I'm executing the same query for many weeks/months, some table involved in the query is slowly growing over time, at some point the query planner responds to the growth by changing the plan.

I wonder how many of those are "!g" queries? I've tried very hard to adopt duckduckgo but it just doesn't cut it, especially for technical topics.

Fantastic. Any write-ups or descriptions for how they made it happen?

Hi HN! Elastiknn is an Elasticsearch plugin that lets users store numerical vectors within documents and run exact and approximate nearest neighbor queries on them. It supports dense and sparse vectors with five similarity functions (L1, L2, Angular, Jaccard, Hamming).

It's a hobby project of mine, based on an application where I needed to to continuously update an index of vectors for approximate searches. Very few of the existing approximate search libraries satisfy this use-case; most require re-building the index from scratch.

Elastiknn lets you add/update/delete vectors without rebuilding the index and combine vector queries with existing Elasticsearch queries, all through the standard JSON/HTTP API, all in your existing cluster. You just specify a field type of `elastiknn_sparse_bool_vector` or `elastiknn_dense_float_vector`, specify some optional settings for approximate indexing, index vectors as JSON lists, and then use the `elastiknn_nearest_neighbors` query in your search requests.

Nothing is without tradeoffs. In terms of raw throughput, Elastiknn is slower than many existing solutions. If you need a high performance solution for offline batch jobs, it's likely not for you. There are some benchmark results at https://elastiknn.com/performance/.

My current goals for the project are: integrating with the ann-benchmarks project, releasing builds for multiple Elasticsearch versions, and creating some example applications.

All feedback is appreciated. At this point there are several teams evaluating the plugin. There's a pretty thorough test suite, but nothing beats real-world usage and feedback. Cheers!

Yep, I'm aware.

The Lucene implementation seems early and slow-moving. Seems they are trying to create new storage formats and use graph-based search methods. OpenDistro wrapped a C++ binary that also uses a graph-based method. It works quite well, but only for L2 similarity and comes with the operational burden of running a rather large sidecar process completely disjoint from the JVM.

The approach I've taken is to support five similarity functions (L1, L2, Angular, Jaccard, Hamming), support sparse and dense vectors, implement everything inside the JVM with no sidecar processes and no changes to Lucene, and to use hashing-based search methods (i.e. LSH). IMO the last point has a clear advantage over using graph-based methods, because the hashes are treated just like regular text tokens which is clearly the optimal access pattern in ES/Lucene. Of course it will likely lose to a C++ implementation in terms of raw speed because it's the JVM, but IMO that matters less than making the plugin trivial to run and scale.

I don't think there's a definitively better approach yet. It's an interesting problem and it'll be interesting to see what ends up working well.

I'm working on an ANN plugin for Elasticsearch. All data is stored on disk, you automatically get horizontal/distributed scaling handled by ES, and you can combine ANN queries with Elasticsearch queries. http://elastiknn.klibisz.com/

It's currently not as fast as the in-memory alternatives. Though it's not a perfect apples/apples comparison. Data is stored on disk, it's a JVM implementation rather than C/C++, and it's optimized for single queries rather than bulk.

I've found EFS enticing in theory but painfully slow and riddled with issues in practice. In the past I've tried it thinking "it's basically an EBS volume I can mount on > 1 EC2 instance," only to find terrible read performance and misc. low-level NFS errors.

Some storage options are: - Store many vectors in a single HDF5 or LMDB file. - Store single vectors in many small binary files (e.g. using numpy save() function).

To look up neighbors you might: - Compute neighbors exhaustively (e.g. using Scipy distance functions). - Use an Approximate Nearest Neighbors approach like one of the ones benchmarked here (https://github.com/erikbern/ann-benchmarks). These can be much faster than exhaustively computing neighbors, at the cost of some accuracy and having to periodically re-build an index.

That's not really a problem if the user is frequently ingesting and vectorizing new data. They need a place to store it and efficiently query it. They can cache the queries for an old vector, but still need to compute new queries every time they have a new piece of content. Also old vectors might have relationships to the new vectors that have been inserted since caching, so you don't want to serve stale results.

Nice write-up! I also found this one useful in terms of high-level implementation: http://adventuresinmachinelearning.com/word2vec-keras-tutori...

Initially it was not obvious to me that the dot-product was even part of the model. In hindsight it's intuitive: a pair of high similarity vectors will have a high dot-product, which yields a high sigmoid activation. This also motivates the use of cosine similarity, which is just a normalized dot-product. Likely obvious to some but this eluded me the first few days I studied this model.

EU citizens can travel freely regardless of citizenship, but getting citizenship in another country is a separate process.

How does it compare to maidsafe? https://maidsafe.net/

Maidsafe made some noise in early 2014 but I haven't heard much from them since.

This might be helpful:http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-...

How do you compute the gradient of a non differentiable function? I'm not an expert, but that contradicts anything I've learned about gradient descent.

+1 Happens all the time. Can't tell specifically what the cause is. Most common seems to be the display hangs after unplugging HDMI while the laptop was in sleep mode.