Why?
2,578 karma · joined October 28, 2009
previously cofounder of periscope data
Why?
Bram's conservatism makes perfect sense for someone who has to maintain this huge, consequential, intimidating codebase. Why hasn't he gotten some help?
Yes yes yes yes yes.
"Front-end engineering" is a craft that happens to be relevant right now. It's an artifact of the particular client-server computing model we currently have. It is not likely to be relevant 5-10-20 years from now.
"Computer Science" is (among other things) a discipline that will give you the ability to learn the next craft, and the next one after that. It will keep you relevant your whole career.
Definitely go to college if you have the opportunity to do so.
Realistically you are going to be tracking down your first 100 users yourself manually. Figure out who among your friends is a good fit and bug them. Then ask all your other friends who they know who'd be a good fit. If they don't click, follow up and ask why. If they click but then don't come back, follow up and ask why. If they become an active user, you are about to become best friends, always talking about what they like and why.
You're going to feel like a mooch for a while -- like you're always asking your contacts for things and not giving back. This is normal.
Also, this thread is on the front page, so post a link. Quick -- an opportunity!
Here are the same queries tested on MySQL, MSSQL and Oracle as well: https://periscope.io/blog/count-distinct-in-mysql-postgres-s...
tl;dr: Oracle and MSSQL do very well, and even better on the more specific queries. Postgres's naive query is much worse than all the others.
First query: 996s
First query with your variation: 593s
Second query: 67s
Third query: 21sFirst query:
QUERY PLAN
-----------------------------------------------------------------------------------------------------
Sort (cost=3475858.60..3475859.56 rows=385 width=22)
Sort Key: (count(DISTINCT time_on_site_logs.user_id))
-> GroupAggregate (cost=3370756.94..3475842.06 rows=385 width=22)
-> Sort (cost=3370756.94..3405784.03 rows=14010837 width=22)
Sort Key: dashboards.name
-> Hash Join (cost=44.90..558337.80 rows=14010837 width=22)
Hash Cond: (time_on_site_logs.dashboard_id = dashboards.id)
-> Seq Scan on time_on_site_logs (cost=0.00..365621.11 rows=14016911 width=8)
-> Hash (cost=29.40..29.40 rows=1240 width=22)
-> Seq Scan on dashboards (cost=0.00..29.40 rows=1240 width=22)
(10 rows)Second Query:
QUERY PLAN
-----------------------------------------------------------------------------------------------------
Sort (cost=2709630.36..2709631.09 rows=291 width=26)
Sort Key: (count(DISTINCT time_on_site_logs.user_id))
-> Merge Join (cost=2604392.06..2709618.45 rows=291 width=26)
Merge Cond: (time_on_site_logs.dashboard_id = dashboards.id)
-> GroupAggregate (cost=2604392.06..2709521.80 rows=291 width=8)
-> Sort (cost=2604392.06..2639434.34 rows=14016911 width=8)
Sort Key: time_on_site_logs.dashboard_id
-> Seq Scan on time_on_site_logs (cost=0.00..365621.11 rows=14016911 width=8)
-> Index Scan using dashboards_pkey on dashboards (cost=0.00..87.00 rows=1240 width=22)
Third Query: QUERY PLAN
-----------------------------------------------------------------------------------------------------------------
Sort (cost=436381.86..436382.36 rows=200 width=26)
Sort Key: log_counts.ct
-> Hash Join (cost=436308.72..436374.22 rows=200 width=26)
Hash Cond: (dashboards.id = log_counts.dashboard_id)
-> Seq Scan on dashboards (cost=0.00..29.40 rows=1240 width=22)
-> Hash (cost=436306.22..436306.22 rows=200 width=12)
-> Subquery Scan on log_counts (cost=436302.22..436306.22 rows=200 width=12)
-> HashAggregate (cost=436302.22..436304.22 rows=200 width=4)
-> HashAggregate (cost=435705.66..435944.28 rows=23862 width=8)
-> Seq Scan on time_on_site_logs (cost=0.00..365621.11 rows=14016911 width=8)Comparing across different RDBMS's is a great idea. Maybe we can squeeze it into a future blog post. :)
Here's a reprint. It's very interesting: http://www.freerepublic.com/focus/f-news/3041469/posts
Does the Retina MacBook Pro just keep getting thinner until the Air is obsolete? Is there some fundamental technical reason they can't keep iterating the Air thinner and put a Retina display on it? Or a business reason why that's not a priority?
Country Requests Population Requests Per Capita
Malta 89 415,654 0.000214120398216
United States (Min) 11000 313,900,000 0.000035043007327
United Kingdom 1975 62,740,000 0.000031479120179
Italy 1705 60,720,000 0.000028079710145
Australia 546 22,320,000 0.000024462365591
New Zealand 106 4,405,000 0.000024063564132
France 1547 65,430,000 0.000023643588568
Germany 1886 81,800,000 0.000023056234719
Singapore 107 5,184,000 0.000020640432099
Portugal 177 10,560,000 0.000016761363636
Here's the full spreadsheet: https://docs.google.com/spreadsheet/ccc?key=0AiKnxhgG6s6VdE1...[1] I used the minimum of the range for the US. Its ranking doesn't change if you use the max.
Population data is from Google searches, which lists the World Bank as a source.
Can anyone confirm that these include National Security Letters?
CSV export would be nice. I immediately wanted to compute requests per capita by country.
How so? What's different? What's better?
A government website showcases a design that is leaps and bounds ahead of any other government site we can think of, and HN's top comment casually dismisses it.
To rewrite something you said: "This is the perfect example of everything wrong with [HN comments]."
I, for one, am floored that someone in city government was able to rise above the bureaucracy and ship something truly useful and delightful for citizens. I would love nothing more than to meet the people involved and buy them a pint. Well done, guys.
That's not to say I wouldn't send it to a lawyer. But I'd do it with some hustle. Something like: "This is going out in 24 hours. Please comment ASAP!"
Periscope makes data analysis tools that use probabilistic algorithms to achieve 100-1,000x real-world-measured speedups over existing tools.
We're well-funded by awesome investors, and the team is currently 2x ex-Google and 2x ex-Microsoft engineers. Come be our fifth full-stack hacker!
Email harry@periscope.io.
(Not doubting, honestly curious.)
OUTRAGEOUS HACKER -- FULLTIME -- SAN FRANCISCO, CA
Periscope uses probabilistic algorithms to dramatically speed up data analysis. Our current algorithms achieve 13-225x real-world-measured speedups against up to 100M data rows. Come be our fourth Outrageous Hacker, working like crazy to make both those numbers go up while shipping the first version of the product!
We're well-funded by top VCs and our office in the heart of Soma is walking-distance from Caltrain and Bart. We're experience-agnostic as long as you're smart, get shit done, and excited to work at all levels of the stack.
Email harry@periscope.io
H1B Welcome