benfrederickson

While there are some implementation differences (py-spy is written in rust, profiling.sampling is a mix of python and C etc), the end result seems pretty similar to me.

One thing to note is that there are some differences in blocking behaviour of the target process. Py-spy blocks by default and profiling.sampling doesn't. I wrote a bit about why py-spy blocks by default here https://www.benfrederickson.com/why-python-needs-paused-duri... - the first version of py-spy also didn't block and since we got incredibly misleading results at times this was one of the first changes I made to py-spy

Sorry about leaving your program suspended - I believe that was because of a bug that should be fixed now https://github.com/benfred/py-spy/issues/390 =(

> you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample.

Py-spy defaults to blocking because the results can be pretty wrong otherwise: https://github.com/benfred/py-spy/issues/56 . You can see this problem profiling a program like https://github.com/benfred/py-spy/blob/master/tests/scripts/... with or without the nonblocking flag in py-spy - the nonblocking version produces garbage output.

Somewhat interestingly, this problem doesn't seem to occur with Ruby - and rbspy can get away without pausing the target program with only minor errors seen when profiling a similar function. I suspect this is because of differences between how the Ruby and Python interpreters store call stack information, but haven't had a chance to dig into the specifics.

Interesting article. While I definitely think you should be profiling your code to figure out the hot spots, cProfile has some limitations for profiling: cProfile doesn't give you line numbers, doesn’t work with threads, and significantly slows your program down.

I wrote a tool py-spy (https://github.com/benfred/py-spy) that is worth checking out if you’re interesting in profiling python programs. Not only does it solve those problems with cProfile - py-spy also lets you generate a flamegraph, profile running programs in production, works with multiprocess python applications, can profile native python extensions etc.

Author here. It's worth noting that since I wrote this post, py-spy has gained the ability to profile multiprocess python applications - and can also now show local variables in the dump command.

I wrote something that will get you the python interpreter stack from any running cpython process : https://github.com/benfred/py-spy/ , and rbspy can do the same for ruby https://github.com/rbspy/rbspy

the name pyspy was taken on pypi already : https://github.com/tdfischer/pyspy =)

Thanks! Both of your suggestions totally make sense. I've created an issue to track the poll() issue here https://github.com/benfred/py-spy/issues/13 - I think that should be an easy fix.

Not yet - but I'm hoping to have a version that supports this next week. Will update this issue when it's done: https://github.com/benfred/py-spy/issues/3

Check out rbspy https://github.com/rbspy/rbspy (rbspy was the inspiration for this project =)

I analyzed the top 1 million robots.txt files looking for sites that allow google and block everyone else here: https://www.benfrederickson.com/robots-txt-analysis/ - it's a relatively common pattern for major websites

There is a pretty good breakdown of a bunch of different options for visualizing sets here: http://www.cvast.tuwien.ac.at/SetViz

Venn/Euler diagrams don't work all that well past 3 sets, not all areas will be shown if using circles - so unless some of the sets are disjoint it will be a misleading diagram (like in the music example). However, I think it works well for 3 set diagams, I have an interactive example on last.fm data here https://www.benfrederickson.com/distance-metrics/ in the context of explaining some simple distance metrics.

Neat demonstration!

A while back I wrote a small package in Javascript for computing area proportional Venn and Euler diagrams: https://github.com/benfred/venn.js . The 2 circle case here is relatively easy, but the problem gets tricky when you have 3+ sets. I wrote up my approach here https://www.benfrederickson.com/venn-diagrams-with-d3.js/ and https://www.benfrederickson.com/better-venn-diagrams/

Alias tables are pretty cool, I wrote an interactive visualization of how they get built as part of this post a while ago: http://engineering.flipboard.com/2017/02/storyclustering . We used alias tables with MCMC and the hastings metropolis test to build a super fast LDA.

Also worth reading up on are sum-heaps. Alias tables are O(1) to sample from but O(n) to build/modify. Sum-heaps let you modify in O(log(n)) at the cost of sampling in O(log(n)) as well. A good writeup is here: https://timvieira.github.io/blog/post/2016/11/21/heaps-for-i...

Should have been clearer here: I've only been interviewed once, but I've given hundreds of interviews over the same time frame.

I just saw your question on proggit, and I lazily cut-and-paste the answer for here =):

For your first question - yes this means few people use more than one language in a month. There is also a power law distribution happening with user activity each month, so most users only have a handful of events each month (which happen to be mostly in a single language). I'm trying to measure how broad support it so this was mostly done on purpose. I was finding counting total events was getting biased by things that I most have been automatic activity (I was seeing single accounts with 10K commits a day for instance).

Percent of MAU in the charts is the total percentage of unique users who were active that month. I haven't tried out with yearly active users =(

Thanks! It was fun to put together.

Author here - happy to answer any questions anyone has.

I also wrote up an analysis of the top 1M robots.txt files: http://www.benfrederickson.com/robots-txt-analysis/

I ended up analyzing very different things from this article though, so this article was still pretty interesting to me.

Your visualization of using momentum with gradient descent in that post is really great - nice work!

For the venn code, I wrote up the approach here : http://www.benfrederickson.com/better-venn-diagrams/

Basically though, I'm using the non-linear CG method - so it doesn't require a positive definite matrix. The loss function is a little funky with handling the disjoint set/ subset relationships in the euler diagrams appropriately (defines the loss/gradient to be 0 if these constraints are satisfied), but this approach still works pretty well.

That venn diagram post has a couple interactive demos of how this works, and also a randomized test showing overall performance.

I actually believe its the best known algorithm for laying out area proportional venn diagrams. I benchmarked against the code from the venneuler paper here: http://benfred.github.io/venn.js/tests/venneuler_comparison/

Author here! This post is really just me messing around with Javascript, and trying to make some pretty graphs with D3. All the code is up on my github account, http://github.com/benfred/fmin if you want to check it out.

While these algorithms aren't maybe the most useful thing to have in JS, I am using both the nelder-mead and CG code to calculate area proportional venn diagrams in my venn.js project (which is why I originally wrote all this code in the first place).

Author here - happy to answer any questions

Not only can ants be used as pesticide - in some cases they also act as a herbicide and kill weeds and other plants that compete with their host tree: http://www.nature.com/nature/journal/v437/n7058/full/437495a... . Probably not useful for farming, but still cool.

Flipboard, Vancouver or Palo Alto, ONSITE

We're looking for a talented Data Engineer or Data Scientist to the join the data products team at Flipboard in Vancouver. Send me an email at ben@flipboard.com or check out the job description for more info: http://hire.jobvite.com/CompanyJobs/Careers.aspx?nl=1&k=Job&...

good point! I actually used this technique recently, to speed up a numerical optimization for laying out venn diagrams: http://www.benfrederickson.com/better-venn-diagrams/

I didn't think of it when I was writing this post though =(

Once I figured out the solution, I was kicking myself for not figuring it our earlier =)

I think I went wrong in 2 ways. My first attempt was to solve using calculus, and though I came up with a solution for the 2 circle case - I totally failed at finding the integral for 3+ circles.

My bigger mistake was to google for papers after I failed initially. I looked for other peoples venn diagram solutions, and found that they were using approximation techniques to calculate. Since the people writing these papers have a much better math pedigree than I do, it made me think I should use an approximation too.

While I did waste a bit of extra time on this, most of the time was spent in writing javascript code. Since the whole point of the venn diagram library in the first place was to learn javascript, I can’t really say it was a total waste. Also I used the quad tree estimate to verify the exact solution.

Author here. I wrote this post a while back, but never submitted to HN. Its one of those posts that I wasn't sure if it would really resonate with anyone (like my most recent one on venn diagram layout algorithms).

Anyways, If you have any questions ask away =).

Its amazing and insane =). I was trying to highlight the complexities behind something that many people people naively think is simple - I probably should have made the clearer based on the reaction elsewhere though

Thanks!

The examples were done with d3.js, the code for all the graphs is here https://github.com/benfred/bens-blog-code/tree/master/distan...

I'm using the python code to generate a series of json files, one for each artist. I stored them all in S3, and then load them via ajax calls. Its not an elegant solution, but it lets me keep my website statically generated.