acidbaseextract

What's your feeling on the status of elasticlunr as a project? I've been eying the the functionality for a project, but it hasn't received updates in quite a few years.

In my more paranoid moments I suspect that some of these disputes are false flags meant to discredit the social justice movement. In particular, some of the issues raised seem purpose built to piss people off without hope of achieving real inclusiveness or equality goals.

That I can't tell if that's the case or not is equally alarming.

Go with off the shelf. You'll get something far better that you can build yourself, and if you need something custom, you'll have a much clearer idea what your analysis is missing.

Writing to Google Analytics, Amplitude, Mixpanel (all of which have free tiers) or equivalent all should handle your case well.

Modern GPUs are extremely programmable, but this flexibility isn't that heavily used by neural networks. NN inference is pretty much just a huge amount of matrix multiplication.

Especially for mobile applications (most Arm customers), you pay extra energy for all that pipeline flexibility that isn't being used. A dedicated chip will save a bunch of power.

I agree that these BI tools are not a trivial investment and are not trivial to use, but in defense of building on a SQL data warehouse:

> Of course there's ANOTHER company to help me generate the scripts to popular 3rd parties :P

> I guess what I want is some powerful individual-use product that has some easy analytics automation on spreadsheets / APIs.

The point is that everything speaks SQL, so vendor lock-in becomes much harder and there's no data source that "is not currently in our plans to support" — if you can get the data into the warehouse, it's supported. Definitely a higher barrier to entry for an individual, but I wouldn't do business BI without it.

We've done some insane things, joining across data from 4+ different systems (CRM, email open tracking, analytics, primary app database, email to a blessed account) in ways that would be impossible for an "off the shelf" solution.

Typically these platforms will query on a data warehouse that you provide: redshift, bigquery, a big postgres instance, or the like.

In terms of getting data into that warehouse, you can write ETL (extract, transform, load) scripts yourself. There are also a number of ETL-as-a-service providers like Stitch or Fivetran that cover common things you might want to sync into your data warehouse and query like salesforce, google analytics, quickbooks, etc: https://www.stitchdata.com/integrations/sources/

Sounds like I did way overestimate. Thank you for the structure!!

Any blog posts or talks that inspired this? How do you handle retention? Any libraries or tools that helped with it?

I can see the use of such logging clearly, but I work at a young startup and implementing that ourselves sounds like infrastructure overkill. Maybe I'm overestimating the work?

Thanks for any thoughts!

I lived on the Balboa peninsula for a few years. Although it's rare, if the waves are very calm for a few days the water clears up a great deal.

Not arguing that they're aren't huge problems with the health of the ocean, but I have seen the water clear up that much a few times a year.

> types that may make complete sense in your program's world, but they drift out of sync as the larger system evolves.

At the risk of being inflammatory, I would say that you've been looking at poorly designed systems.

A running joke about Google engineers is that all they do is shuffle data from one ProtoBuf to another. It's largely true. The upshot of that is that every single service/system has common, typed definitions of the data structures they use and require. Data never drifts out of sync.

To be sure, a lot of the problems I dealt with there were around data structure migration on large systems — but there was never any uncertainty about the structure of the data itself.

I like JSON APIs, but I think JSON especially contributes to people getting hand-wavy and casual about the structure of data across systems in stupid ways (see NoSQL vs SQL).

> That's why I prefer dynamically typed languages for web systems; they start with the rigth assumption about the ever-evolving nature of the project.

I agree with your assessment that we need to design for evolution.

I disagree with your assessment that dynamic languages provide that. Dynamic languages certainly do hit a sweet spot for fast iteration on small projects. As soon as your project gets large or long running, the assurances that static types provide get really nice. Witness how much effort has gone into building type systems for dynamic languages: Typescript (MS) is exploding and had vigorous competition from Flow (Facebook). Python has mypy plus a bunch of corporate backing: MonkeyType (Instagram) and PyAnnotate (Dropbox).

I feel much more comfortable aggressively refactoring in static languages.

I'm sure the plastic is fine, I'd just prefer a metal or glass version.

On every discussion of the Aeropress, I've seen the ask for a non-plastic version pop up. Every time I've then seen essentially your reply above. I'd just prefer a metal or glass version.

If anyone from Zapier is reading, how do you folks do it? What does your support staff look like? Do you have allocated support engineers?

I'm on a team of fiveish serving a whole company of fiftyish. Sometimes we resolve issues quickly, but often it pains me how long resolutions take. I'd love to do better but it's not clear how, short of staffing an engineer towards those reactive things specifically.

The issue is that Mozilo's behavior resulted in personal gains against extreme public losses. It's in our best interest as a society to deter such behavior, while enabling whatever legitimate business Countrywide conducted. How do we best do that? The public at large wins — Mozilo should win, the public at large loses — Mozilo should lose.

I think GP's point is that a $60M fine (against $520M in earnings) is not a an effective deterrent against such antisocial behavior.

Ed Catmull, Pixar cofounder and inventor of the Z-buffer, has a great take on mistakes and failure in his book Creativity, Inc. Here's a pretty decent summary:

https://www.brainpickings.org/2014/05/02/creativity-inc-ed-c...

Essentially, we're going to fail. It happens. Might as well get it out of the way.

Secondly, failure averse cultures don't actually prevent failures, and they have a tendency to squash innovation.

> This is why I always have my visualizations overlay both failure rates and usage.

Do you have an example? I love having more viz tools in my toolbox!

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about?

Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want?

Every time I start looking at centralized logging stuff it seems like a rabbit hole of problems we're too small to be worrying about, stuff that's not shipping features on my app.

Dead simple practices you can start implementing tomorrow: https://www.amazon.com/Effective-Manager-Mark-Horstman/dp/11...

Their podcast, Manager Tools, is also excellent.

I highly recommend the full book. Robert Greene is killer at providing examples, proving his points then boiling down those examples into action you can take. The persuasive force of hearing the patterns of mastery in the lives of people we all admire, and seeing the variations on the theme that can apply to our own lives, is lost in a summary.

Also the audiobook is excellently read, if you're into that sort of thing.

In the same way that a chainsaw really is still a beautiful tool.

That's the nature of the beast. Pareto/power law distributions are very common in economic transactions. 80% of a lot of sales orgs revenue comes from 20% of deals, 80% of many freelancer's income of many freelancer's income comes from 20% of their clients, etc... Many people do sales or consulting anyway.

Going to echo this, really hard to read, both because of the color palette and the bizarre white spacing between the black lines of code.

I highly highly recommend both of Phil Rozenzweig's books, The Halo Effect and Left Brain, Right Stuff.

The Halo Effect in particular is a clinical deconstruction of the fallacies the business book genera. It tears Good to Great and company apart.

An great article on exactly this issue: https://www.scotthyoung.com/blog/2010/12/14/persist-without-...

A whole post about hash tables and he doesn't mention the real reason trees are better than hash tables: lower variance!

> Hash tables become full, and bad things happen

The bad thing is not a crazy long probe or even rehashing to enlarge the table—it's that some random individual insert is stuck eating the entire cost of rebuilding the table. For many applications, I'm happy with a slightly higher cost per operation in return for predictable performance.

> Hash functions are slow

Great, another unmentioned footgun - hash functions are hard. For example, the hash function he provides doesn't use a prime number of buckets. Oops, non-uniform input data in combination with unlucky bucket counts can generate high numbers of collisions.

Writing a comparison function to put your data in a tree is pretty stupidproof!

Yeah I totally agree. I tried to read some of the guy's other articles as the ideas are really interesting, but man are they buried in jargon.

I think his conclusion is more subtle than that. It's not that getting out of the way of the economy means it will all work out—far from it. Rather, continuing to buy into the magical thinking of the current priest-kings (i.e. state-directed action) opens the doors to much nastier alt-priests (e.g. totalitarianism).

The specific process? Further buying into our current priest-kings puts pressure on them to make their magic work. Magical thinking eventually fails, inevitably. Then the historical process of succession of priests kicks in because "we need the eggs" (to use the article's phrasing), and the more we need the eggs, the more nasty the alt-priests we'll turn to in search of eggs.

Because then the reviews for the connected PRs ends up being hideous to track.

Either you can:

• Submit the PRs one at a time... which is awful. Why not work on them in parallel?

• Submit your series of PRs all at once, then you have to set the PR base of each one to the following one to make the github diff view actually reasonable for reviews. THEN, when you have a code change based on feedback on the first PR, you have to merge it into all the later dependent PRs. THEN, you have to have to jiggle the PR bases very carefully when you're landing the PRs, otherwise it will merge them all into each other, and you end up with one ginormous commit landing—defeating the point of the small commits. AND unless you're using a squash based workflow, you'll end up with that giant merge mess in the middle of your commit history.

The contempt for people outside security, and their perspectives, is a huge problem. Another great article, "Infosec's Jerk Problem" (2013):

http://adversari.es/blog/2013/06/19/cant-we-all-just-get-alo...

Dang, that is sad. Data viz research sounds fun and useful. Thanks for the reply!