gjreda

Nodding my head in agreement, only to realize I'm agreeing with an IRL friend.

His Think Bayes is another good one: http://greenteapress.com/thinkbayes/

Great post! I've been thinking about writing something similar with that same BeerAdvocate data. Good job beating me to it :)

Instead, I ended up writing a satirical beer snob bot [1] which tweets nonsensical beer reviews using Markov Chains. Some are bad, but some are pure gold. You can read about it here [2]. The code's also on GitHub [3].

[1] https://twitter.com/BeerSnobSays

[2] http://www.gregreda.com/2015/03/30/beer-review-markov-chains...

[3] https://github.com/gjreda/beer-snob-says

Are there ever opportunities for freelance/contract work?

You can find the rest of the materials here: https://github.com/gjreda/pydata2014nyc

Another fan of Pelican here. I've also written a theme for it [1]. One let's you use IPython Notebook for your Markdown + code.

I use S3 to host my blog. It's super cheap and it's really easy to push out changes with S3cmd [2]. Just a simple:

  s3cmd put mydir/* s3://mybucket/

1. https://github.com/gjreda/gregreda.com/tree/master/theme

2. http://s3tools.org/s3cmd

I spent a Sunday at the top of Hacker News about a year ago and I don't recall ever paying more than $5 a month for my S3 hosting.

For context, last month I had ~20,000 pageviews and my S3 + Data Transfer charges were $2.43.

It seems like there is a good amount of posts hitting the front page lately that are simply collections of tutorials, libraries, or other references without much substance.

I'd love to read about some in-depth experiences with said tutorials and libraries, why some options might be better than others, or the tradeoffs made amongst them.

SEEKING WORK - Remote (based in Chicago) (part-time)

Who I can help:

People/companies/startups looking to gain some value out of the data they're collecting, specifically around user behavior.

Experience:

First data hire at GrubHub and currently leading the data analytics team. I take pride in being a data generalist (can do the data science or data engineering thing, while keeping business value in mind - just because some 'discovery' is interesting, doesn't mean it's valuable for the business).

Prior to GrubHub, I spent three years as an analytics consultant at one of the Big Four.

Technologies:

My day-to-day stack is some combination of: Python + pandas + MySQL + Amazon Elastic MapReduce + Hive + Amazon Redshift.

Others: HTML, CSS, d3js, R, Flask (python), nearly all RDBMS platforms (SQL Server, Oracle, Postgres, SQLite), Google Analytics

Projects:

- Open Source contributions to pandas (https://github.com/pydata/pandas) and tweepy (https://github.com/tweepy/tweepy)

- My blog (http://www.gregreda.com), which has been at the top of HN several times (tutorials on web scraping, pandas, unix commands)

Links:

Website - http://www.gregreda.com

LinkedIn - https://www.linkedin.com/in/gjreda

GitHub - https://github.com/gjreda

Stack Overflow - http://stackoverflow.com/users/1419514/gjreda

Email - gjreda@gmail.com

I did this recently when trying to get tickets to a sold out Cloud Nothings show. I'd scrape Craiglist for postings every 10 minutes, and then send myself a text if any of the posts were new. I ended up getting tickets the day before the show.

Since the show was at a very small venue (capacity of maybe 500), I didn't have to worry about a constant stream of false positives. I would have needed to handle these if I were searching for tickets to a sold out <popular band> show, since ticket brokers just spam Craigslist constantly with popular terms.

Commit history banner I appreciated: https://twitter.com/gjreda/status/432661964209143808/photo/1

I forgot to mention reproducibility. Show your work (share the code).

For those interested, I wrote a "web scraping 101" tutorial in the past that uses BeautifulSoup: http://www.gregreda.com/2013/03/03/web-scraping-101-with-pyt...

Author here. This was my first foray into PyMC. I'd love feedback (good or bad).

I'm super interested in the chapter on creating reusable command line tools.

I've found the command line to be ideal for performing a lot of simple, memory-intensive tasks (filtering/munging/sorting/etc. a massive text file).

However, after data collection (and munging), data science is typically A LOT of _exploratory_ analysis. I think it's extremely important that all practitioners approach analysis with the mindset of making it easily reproducible (and if possible, flexible - don't hard code date ranges, file paths, etc.).

I tend to stick with IPython Notebook (and heavily recommend it). I fear that heavy analysis at the command line would consist of too many one-liners and thus be difficult to read and maintain.

Do you guys have any concerns about websites' terms of use restrictions?

I've done a bit of crawling and have always been curious how selling this sort of service would handle those potential use restrictions / legal issues.

If anyone is interested in reading it as one long Notebook, you can use NBViewer: http://nbviewer.ipython.org/urls/raw.github.com/gjreda/gregr...

Alternatively, I think you could also do:

    myframe.myfield.dropna().astype(float)

I'd need to test that out though - type conversion in pandas/NumPy has always been a little confusing for me.

Thanks! My concern was that it actually _was_ too long winded, so I'm glad to hear otherwise.

Thanks for pointing this out too - had no idea it'd handle a URL.

There's also Vincent[1], which has some mapping capabilities and is built on top of Vega (a "visualization grammar" for d3js).

[1]: https://github.com/wrobstory/vincent

My workflow is very similar. All data analysis, all the time.

Tools: Lots of SQL (MySQL). AWS Elastic MapReduce + Hive for log analysis. Python (pandas). R (ggplot2 + stat modeling - haven't fully grok'd Python's statsmodels package yet). A little bit of D3, and a lot of command line. Maybe some Flask every once in a while to prototype out a reporting web app.

I try to avoid Excel entirely because it always crashes on my MBP, but still find myself in it every so often to simply look at my dataset.

Pretty impressive for a first post with d3. Really nicely done.

I like that this only covers left outer joins as opposed to left and right outer joins.

The difference between the two is sometimes confusing for people, so I've always suggested that people learn one and stick with it, since you can accomplish the exact same thing with either.

I think the lack of a ggplot2 equivalent is one of the main things holding people back from fully converting from R to Python - I know it is for me. Once you really understand the ggplot2 syntax and style, making your graphics seems to come naturally. It's fantastic.

I'm extremely excited to see Bokeh develop more.

I mostly enjoyed Python for Data Analysis, but do agree that there was too much focus on pandas (though that was to be expected with Wes McKinney writing it).

I would have loved if the book had included sections on the statsmodels and scikit libraries, especially since they are integrated with pandas.

I've been using BeautifulSoup for a couple years, so it's what I'm most comfortable with. I'd heard of scrapy before, but had never given it a seriously look. That'll change based on all the positive things I've read about it in this thread.

I've heard this from countless people who have read the post. It's definitely made me want to look into Scrapy.

Brian Boyer did the same when he was leading the news apps team at the Chicago Tribune (I believe he left for NPR this past summer). The team's motto was "Show your work."

https://github.com/newsapps

SciPy Superpack easily installs iPython, SciPy, NumPy, Matplotlib, and others on OSX.