teej

You’ve kinda just described AWS Athena.

That’s a nice sentiment but we don’t always get to choose.

I feel like Amplitude's schema management tools are best-in-class right now but still missing a lot. I've been frustrated at the absence of any integration with my eng teams build/CI systems or really any automated way to catch problems at the PR or code review stage. Is this something on your radar?

> no one that really wanted to maintain this

I feel strongly that the data organization needs to own this outright, and the only thing holding them back was the right tooling. This seems like the perfect tool to enable it. Very excited to give it a spin.

I'll say this much as a new Tesla owner - they have at least 3 systems. Their internal system which drives anything you see on tesla.com or the app, Salesforce for tracking customer interactions and the source of truth for some order state, and third party freight platforms used for shipping cars around. These systems do not automatically sync and require human intervention to keep up to date.

I originally test drove in Boston and placed my order there. I informed them I would take delivery on the west coast and they set that up for me. I later found out about the three systems when my car was weeks late for delivery. No one seemed to know where it was or where it was headed. Tesla's internal system said it was en route to Hayward, CA. Salesforce said it was headed to a different location that turned out doesn't do deliveries. The shipping company said the car was headed to New York. The car's GPS pinged it to a railroad track in Georgia.

My experience is an outlier, but it does indicate that these APIs might not always be correct.

My understanding of timing attacks is that they can be used when you can trigger the code path thousands of times quickly. This gives you execution time averages and distributions. Also, they're typically used against cryptographic hash functions which are more expensive than just a string compare.

https://en.wikipedia.org/wiki/Timing_attack

> random

https://en.wikipedia.org/wiki/Ulam_spiral

I appreciate the clarification, but this reads less as “DO is chill” and more as “wait for your surprise bill in a year”

I mean, this has been exactly the situation since recaptcha was invented.

For those trying to up their SQL game from basic selects and grouping, I suggest learning:

- Common Table Expressions (CTEs). They will equip you to write clean, organized, and expressive SQL.

- CREATE TABLE AS SELECT. This command powers SQL-driven data transformation.

- Window functions, particularly SUM, LEAD, LAG, and ROW_NUMBER. These enable complex calculations without messy self-joins.

Learning those three will change your relationship with SQL and blow the door open of problems you can solve with the language.

I don’t think it’s crazy to think you are more capable of predicting the unique demand curves of your business better than a heuristic designed to be good enough for the median Amazon customer.

This article seems like an edited version of this talk, or some variant of it. Worth a watch - https://www.youtube.com/watch?v=Ol42iSDddOs

Citus improves the performance of OLAP query loads but it's not an analytics solution first. They say so themselves -

https://www.citusdata.com/blog/2018/06/07/what-is-citus-good...

---

When we first started building Citus, we began on the OLAP side. As time has gone on Citus has evolved to have full transactional support, first when targeting a single shard, and now fully distributed across your Citus database cluster.

Today, we find most who use the Citus database do so for either:

(OLTP) Fully transactional database powering their system of record or system of engagement (often multi-tenant)

(HTAP) For providing real-time insights directly to internal or external users across large amounts of data.

AWS Snowmobile cost is $0.005/GB per month which comes out to $1.5M/month for 300PB for however long the data is on the truck.

How are you getting data from DynamoDB to S3 so Athena can read it?

Saying a portfolio allocation of 100% stocks 0% bonds outperformed 60/40 during the longest bull run in history is shortsighted. Here's a more thorough analysis - https://finpage.blog/2017/01/04/three-fund-portfolio-2016-up...

There’s also an AWS offering called Spectrum and a data enrichment platform called Spectrum. Pretty overloaded name.

$100-250k range given this includes logo, brand identity, motion design, comprehensive visual design system.

Plus Washington DC, the US territories, and US military bases abroad.

There’s validated data types and unvalidated data types.

“Our logs are stored as newline-delimited JSON objects.” No, it’s improperly escaped so it’s actually a JSON-like string of unknown character encoding.

“This is a postal code field.” Nope, it’s a user-generated string that occasionally converges on something approximating a postal code.

“This is the user’s IP address.” Well I clearly see localhost in here so unless folks are browsing the site from our servers, this is “sometimes” the user’s IP address.

10% less than planned for Q1, not 10% less than Q4.

$25M is chump change at the scale of Sears, with 89,000 employees. It's only $280 per employee which is not nearly enough money to keep everyone working.

> Which pages were featured and on which social media platforms?

This was the “killer feature” of the Google Analytics replacement I built for myself a few years back. I would grab referrers for social and scrape them to give a more useful report of the source of traffic.

Unfortunately this is challenging or impossible for many social platforms due to HTTPS everywhere, the prevalence of outbound link scrubbing, and app-driven embedded web views.

I still think it would be a killer feature to know who tweeted you out or which subreddit you are trending on but it’s just not feasible to do based on a website pixel alone.

Franchises don’t sell a way to get rich quick. They sell a job.

If you’re keeping a balanced portfolio of assets and the price of one goes up, you sell to rebalance, not buy.

“A Programmer’s Introduction to Mathematics” by Jeremy Kun. It’s funny, approachable, and manages to get deep into math topics important to programmers. It goes into the history and culture of math, helping you understand why math is the way it is. Couldn’t be happier with it!

Most people seem to ascribe it to the login change (mid 2012 I think?) but at the same time, Quora was -exploding- in India. This came with it a big shift in the types and relevance of questions. By the start of 2013, India had more users on the site than any other country. As a heavy site user at the time, it felt like it happened overnight. For me this seems more impactful than the login change.

Some people don’t want to put a GA tag on their site because of concerns around how Google uses the data. Also you can’t arbitrarily query GA data so this gives you that capability.

S3 access logs alone are not sufficient to replicate this pipeline. This pixel is stateful (for the anonymous user ID) and S3 access logs don’t include arbitrary headers, in this case the cookie with the user id. Fastly would let you eliminate API gateway, Lambda, and both Kinesis steps.

API Gateway by itself is $3.50/million requests, which is 2-4x more expensive than Fastly at $0.75 - $1.60/million

This sort of thing works until you have one person run a security scan on your site, corrupting your user agents and event types.

I’ve seen folks put their pixel endpoint behind Fastly and process the access log delivered in S3. A Fastly VCL can handle the same transform that this Lambda is doing.