hamilton

Iodide does actually support that workflow, both in JS and our wasm python stack (Pyodide). We have some data scientists internally who have made Iodide reports using only pandas & matplotlib. It's still early days for this tech, but it's quite doable.

(disclaimer: I work on Iodide)

+1 - this is the right attitude to have! If you have a productive data science team, you should continue to use the tools you have - we'd be the first to tell you that. Re: the JS thing, we've found through user testing and our own experience as data scientists that folks don't really want to use JS for anything, and we want to meet some segment of data scientists where they are. As explained in the post, JS as a language and ecosystem isn't quite there in the way that it needs to be. In a few years, it may be, if you follow along w/ the various TC39 proposals out there.

To clarify, we do have support for Python / NumPy / Pandas / Matplotlib / others with Pyodide, which is natively supported in Iodide with the `%% py` delimiter. It doesn't address all the other workflow issues you have, of course, and there are some differences b/t a server kernel and what we have.

(iodide dev here) Yeah, that's a good point. We called it JSMD before we even thought of starting Pyodide, and in hindsight seems a bit constricting.

This New Yorker piece on Andy Byford provides a lot more context for why the MTA is the way that it is:

https://www.newyorker.com/magazine/2018/07/09/can-andy-byfor...

A handful of years ago, Paul Ohm wrote about a concept he called the "Database of Ruin" [0]. I think about it every time one of these pieces of news breaks.

"Once we have created this database, it is unlikely we will ever be able to tear it apart."

[0] https://hbr.org/2012/08/dont-build-a-database-of-ruin

I'll take any chance I can get to up this Lydia Davis short story, called "new year's resolution"[0]:

I ask my friend Bob what his New Year’s Resolutions are and he says, with a shrug (indicating that this is obvious or not surprising ): to drink less, to lose weight… He asks me the same, but I am not ready to answer him yet. I have been studying my Zen again, in a mild way, out of desperation over the holidays, though mild desperation. A medal or a rotten tomato, it’s all the same, says the book I have been reading. After a few days of consideration, I think the most truthful answer to my friend Bob would be: My New Year’s Resolution is to learn to see myself as nothing. Is this com¬petitive? He wants to lose some weight, I want to learn to see myself as nothing. Of course, to be competitive is not in keeping with any Buddhist philosophy. A true nothing is not competitive. But I don’t think I’m being competitive when I say it. I am feeling truly humble, at that moment. Or I think I am—in fact, can anyone be truly humble at the moment they say they want to learn to be nothing? But there is another problem, which I have been wanting to describe to Bob for a few weeks now: at last, halfway through your life, you are smart enough to see that it all amounts to nothing, even success amounts to nothing. But how does a person learn to see herself as nothing when she has already had so much trouble learning to see herself as, something in the first place? It’s so confusing. You spend the first half of your life learning that you are something after all, now you have to spend the second half learning to see yourself as nothing. You have been a negative nothing, now you want to be a positive nothing. I have begun trying, in these first days of the New Year, bur so far it’s pretty difficult. I’m pretty close to nothing all morning, but by late afternoon what is in me that is something starts throwing its weight around. This happens many days. By evening, I’m full of something and it’s often something nasty and pushy. So what I think at this point is that I’m aiming too high, that maybe nothing is too much, to begin with. Maybe for now I should just try, each day, to be a little less than I usually am.

[0] https://poetrydispatch.wordpress.com/2009/01/16/lydia-davis-...

West-coaster here. Twitter unreacheable, Github unreacheable, Box unreacheable, Soundcloud unreacheable (I'm a musician, saw a friend's band's big song premiere go stale). Tons more. Just because you didn't feel it doesn't mean others didn't.

This, right here, must be why they're betting big on self-driving vehicles.

Titles would clarify everything (which is easy to add in the data_graphic calls)

[disclaimer: I am one of the authors of the graphing library used. So neat to see it in action!]

Hi - yep, certainly on the horizon.

Hi - author here. Yes, we use Voronoi + clipPaths to make these kinds of rollovers possible.

Howdy - one of the authors here. I believe to my knowledge the only thing that requires bootstrap.js is the popover stuff, which we plan to decouple anyway.

Make an issue on github and we'll get to it soon: https://github.com/mozilla/metrics-graphics/issues

Hi - one of the authors here (Dygraphs is fantastic, btw!)

We've talked about this quite a bit internally (nothing formalized in a github issue), and are still figuring out how we want to handle it, if we want to.

Hi - author here. Completely agree - if you want to make much more complex data graphics, learning the nuts and bolts of d3 is vital. You certainly can't escape that.

But I would add that not everyone wants to make more complex ones. For a lot of needs, 'using a library like this will only get you so far' is often exactly what folks need. Many of us just want to make charts that work really well, and are really simple to use. We grew tired of having to re-implement the same ideas over and over again for new dashboards and reports. Hence this library - a way of simplifying what we've already done, and making it easier to add new features in a principled way.

Hi - one of the authors here. We've had the pie chart request before. We will probably never implement pie charts. I agree that there are cognitive issues with them, and given that at least for our immediate needs there are better alternatives.

Hi - we've actually talked about a number of implementation ideas for this, and probably should add a github issue for this.

Agreed. It's a shame to write off Dweck's research because of poor phrasing on the part of the writer of the article. Her work is massively important. If you're not willing to understand the psychological underpinnings of motivation, then you're leaving out a big part of the puzzle.

I worked at a movie startup, very similar to this idea, way back in 2007-8. It was my first startup experience, and I knew back then that it was not really an interesting product. We did a lot of cool text mining stuff on movie reviews we scraped (I am a statistician by trade) and at the time provided a way of searching based on emotions and moods. It was kind of neat, but even still - we were making a glorified movie list, which is not a problem anyone actually has.

At the time, our biggest 'competitor' (in our eyes) was jinni.com, which still seems to be doing something similar to what they set out to do way back then.

It's 2014 now, and I still see about 1-3 of these types of sites pop up every year. I just met a guy who has a similar movie list app out in Berkeley last week - no joke.

So, does your product suck? Maybe not. Is it a product solving a real problem that people will pay for? I don't think so. I personally don't see any value in using this site over Netflix, and I'm sure a lot of others would agree with me.

But it is okay if no one likes it - you can get a lot out of making something without it being successful or ultimately useful. I learned a lot working at that movie startup, and I am glad I did it, even though the damn thing fell apart before we could generate revenue. Revel in it, pal! You have created something!

Interesting seasonal trend. Must be more than just students, though.

I think these explorations are very worthwhile. People tend to underestimate how much work it takes to implement all the distribution functions.

Looking forward to seeing where this goes.

I'm hoping to see how the lib authors:

1.) deal with model workflow (for example, how do I go from model building to prediction?); 2.) explain what data structures they hope to introduce (if all the authors did was make a strong matrix library for server-side javascript, that would be enough to make everyone's day); 3.) outsource the computational grunt work to C (or Fortran, if they're really adventurous)

Having seen early versions of Rust (I work at Mozilla) I can tell you that the syntax is the last thing they're even considering (though that may have changed over the past few months). Seems like Graydon wants to get the guts right first.

This is a great point, and one we should have addressed in the post.

That said, there is not generally a spike in usage at lunch for anything else. For many other metrics of interest we see a familiar sinusoidal wave of usage, with the low point at around 3 or 4am, and a high point at approximately 6-7pm.

Thus the spike in usage at lunch turns out to be quite unexpected.

The game music is a Shostakovich piece. Awesome.

The author misses a large point, and it sort of confounds his other arguments. While form and representation are absolutely important, so is context. People cared about his facebook visualization because the subject matter was of immediate interest to us, given the current news cycle. A visualization has to be part of a compelling story.

Not all visualizations are great, and not all well-made visualizations express a compelling narrative. And not all visualizations, regardless of how well-made and compelling, are right for a target audience. Relevance is different than form; let's not confuse them.

Thanks to liebke for posting this, and Bradford for writing it.

I have two points.

First, your point about programming is incredibly important. I've worked with people who had amazing insights about statistical problems, but went cross-eyed upon being asked about SVN and Git. This makes a CS homework assignment unpleasant, and a real world research project impossible.

Second, this really begs another post. It should be called "Learning how to read a textbook on your own." Successful self-learners don't just __read__ a textbook. They toil with it, try proving things on paper themselves, work through exercises, attempt to apply it to some real-world situation, and hunt down someone who's smarter than they are to explain something that seems unclear.

Not all textbooks - certainly not every one on your list - can be read with a great application in mind. A reader must interrogate mercilessly the book on analysis or the rigorous probability book mentioned in the post.

This seems intuitive to someone who can successfully learn on their own, but most people are not taught to do that. The difficulties of relicating a portion of the classroom learning experience is a major barrier to entry. This is why online intro lectures for programming, math, and certain CS topics like algorithms can steer a learner in the right direction. Stanford Engineering Everywhere and of course MIT's OCW, links to which have been posted on HN at least once a month, are great starts.

Having lived across the street from the Delancey St. Foundation for a while, I've had the privilege of enjoying their many services. Deeply impressive organization. If you ever find yourself in the SoMa area, go eat at Crossroads Cafe or the Delancey St. Restaurant. Great food, and by far some of the best service in SF.

Great writeup!

igraph is also a must-have for network analysis in R. Great for data analysts familiar with the environment.

I think you're on to something regarding "significance." Over in my dept. we like to say that significance is a measure of sample size. The question, then, hinges on whether or not something has practical significance. Because we've built the whole research reputation incentive structure on the .05 significance level, studies can be designed to get that.

A great article on the problem with statistical significance (by a Frequentist) is here, called Why Most Published Research Findings Are False: http://www.plosmedicine.org/article/info:doi/10.1371/journal...

The punchline seems to be, well, that there's also a large human element that contributes to the problem. I think it's one thing to rail on the Frequentist way-of-thinking; it's entirely another to state that the institution of scientific research built on it creates unwanted incentives.

I beg to differ (not sure if you actually read the Efron talk I just posted). I get the thrust of the Less Wrong article, though I think that frankly his language and vitriol is direly misplaced. He's railing against some class of Frequentists who attack Bayesians. This is sort of a Fox News tactic; pick someone, in this case, a commenter on a blog, and generalize their comment to the entire population. It's fairly unstatistical, if you ask me. I don't get why anyone would do this. A good Frequentist is not a lost idiot, nor does she rail against Bayesians. In fact, a good Frequentist understands the beauty of Bayesian methods, too. I'd even wager that a good Bayesian understands why Frequentist methods have dominated the 20th century.

I'm attempting to realize that rare event in these vitriolic philosophical academic debates - add a bit of middle-groundness. In this case, the middle ground of Empirical Bayes has proven to be very, very useful in large-scale simultaneous inference problems. It's proof that there IS such thing as convergence in this debate. Taking the shrill tone the author takes as a signal of how little regard the two sides have for each other, it's obvious that we need some convergence.

This debate has been raging for 250 years. Brad Efron, one of the greatest living statisticians today and one of the few who can transcend the debate, has VERY interesting things to say about it. He believes that Empirical Bayes, or using some of your data to model your prior, will win the day.

http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode...