zeratul

Let me know if I can help anyway: scripts, source code, bandwidth.

OP: Please note that you can fit a line that predicts price based on the mileage (linear regression). For any car that the actual price is smaller than the one predicted is a good deal - that's what you want to have most visible. Obviously, you can use more variables for predicting the price, but the point is that cars that deviate from the fitted curve are either a bad deal or a good deal and that's where machine learning gives you monetary value on your web service. You could make even one step further give API for other websites to use your "bad deal"/"good deal" prediction engine and set your revenue this way.

[EDIT] The more variables you add the more accurate your model can be. You could even use NLP to extract the narrative about the car. You would disrupt KBB, by providing more user friendly interface. It does not have to be the visuals but just the fact that you can boil down the number of all cars that sell down to a hand full number of cars that deviate from the fitted curve. Even if you are not able to tell whether the deviation makes it a good or a bad deal, you are selecting a smaller number of cars worth looking at than any other website. At least you build a time sever which is worth money, too.

[EDIT2] I think I had an epiphany. The distance from the fitted curve is a buyer's risk. Someone comes to your website and you ask what car and what risk. Some people don't want a good deal, they want a fair deal; they want a car that is priced just right. Others might venture to get cars that are outliers, maybe those car have an engine modification or someone is in a rush to sell. In any case you would have a criteria for searching cars that no one has. KBB gives you the sound and fair deal prices but yours index could be calculated in near-real-time and thus more accurate. All depends how you implement regression model.

Note to self: excellent data mining application with interesting visualization

There is a value in providing a "social news paper for a city" but that's not what OP is doing now. "Swarm" intelligence is superior to an individual. Filtering news articles by number of re-tweets sets higher threshold for signal-to-noise ratio, however, 140 characters hardly constitute a "news article". Below, my suggestion for a data mining web application that is focused on delivering a news paper customized for a city with high signal-to-noise ratio news articles.

Input: X = country, Y = city, tweets from twitter

For each X scan last N million tweets that have a link to a web-site, output M most linked domains (Facebook.com, techcrunch.com, cnn.com, wsj.com, etc) - this gives you a base for news feeds customized for every country

For each Y in X, scan last n thousand tweets that have a link to a web-page, output m most linked web-pages that are hosted on M domains for Y country - this gives you the "social news paper for Y city in X country"

Parameters M and N prevent spamming and shape long term trends. Parameters n and m give you short term trends, which in essence are the "swarm" intelligence about which articles are worth reading for a given city in a given country. The value here is a time saver of what is worth reading to keep up with your community.

Note to self: could be a data mining web app

First, a distinction whether we are dealing with sparse data (nearly-binary data) or dense data (nearly-continuous) should be made. Second, a distinction whether we are calculating similarity for an unsupervised or a supervised problem should be made. Then, present most common similarity measures and variable selection algorithms for each of four type of problems.

This is a vast area of research. Diving head on might result in serious injury. For example, R statistical package simba (http://cran.r-project.org/web/packages/simba/index.html) lists 56 different similarity/dissimilarity measures just for binary data.

I thought this is actually interesting but I would like to know if >4k misspellings is a lot or not. Here is one way to do it:

    LANGUAGE #LENGHT #LENGTH = #LENGHT/#LENGTH  
    JavaScript 4252 2907459 = 0.0015
    C 18981 2902857 = 0.0065
    Java 7706 2348900 = 0.0033
    Ruby 10789 1690604 = 0.0064
    C++ 9458 1315552 = 0.0072
    PHP 3116 1167924 = 0.0027
    C# 1352 937647 = 0.0014
    Python 3662 737292 = 0.0050
    Ruby 1232 380484 = 0.0032
    Perl 1239 258892 = 0.0048
    Objective-C 679 238051 = 0.0029

P.S. There is something wrong with Github's language breakdown algorithm, sometimes it shows same language twice with a different number of hits.

Why don't you try 10-fold CV (http://en.wikipedia.org/wiki/Cross-validation_%28statistics%...) - the graph might drastically change. Here is example how to do it: https://onlinecourses.science.psu.edu/stat857/book/export/ht...

If precision & recall monotonically go down when increasing NN then it means you don't have enough training data.

DanielRapp: in file twss.js/lib/classifier/knn.js, number of NN should be odd to prevent ties [EDIT: also, NN should be large enough to prevent over-fitting; small NN would mean that the difference (decision boundary) between twss and not-twss is highly non-linear; you need to implement cross-validation to find best NN]

Note to self: machine learning using node.js; what's the speed of calculations, what's the memory management in node.js, can I find pure JS implementation of SVM?

> I love clever touches like this in TV and films.

Me too, "Contagion" movie hired Dr. W. Ian Lipkin who is a specialist in molecular methods for pathogen discovery: http://www.newscientist.com/blogs/culturelab/2011/09/contagi...

This is great! I surely would like to see your codebase. I wander what is your angle on this matter. Why are you interested in failed startups? I want to see patterns and draw conclusions.

Here is a snippet from Friendly Gaming Simplifier Message Board:

   On the day of 6th January 2012 02:05 GMT+1, I have  
   received a legal note from Perkins Coie attorney company, 
   which is representing Facebook Inc., based in Menlo Park, 
   California.

   According to this note I must follow all of the below 
   demands „In order to avoid further escalation of this 
   matter”:
   - Stop offering the FGS Browser Software, or any other 
   software that interferes with Facebook’s intended 
   limitations or impairs the proper working of Facebook,
   - I must cease and not access the Facebook site(s) and/or 
   services for any reason whatsoever.

I cannot find explanation why he got this message.

I was thinking about a database where (co-)founders would enter the data - a drop-down menus, where possible, for data mining. Something like:

    1. Here is what I was trying to build
    2. Here is the source code (link to GitHub?)
    3. Here is a screenshot
    4. Here is how I wanted to make money
    5. Here are my competitors
    6. Here is how much I raised money
    7. Here are top 5 reasons why I failed
    8. ???

Yes, CrunchBase has some of the above and, yes, some founders will put it on their blog. But wouldn't be it so much nicer to have a centralized database that could spit out some analitycs/trends and give guidance to newcomers? The volume of startups will grow. What if some failures are seasonal or can be pick-up and fixed? I think we have a chance for a data driven approach but only if founders are willing to contribute.

EDIT: I suspected that CrunchBase will have some errors and missing info since ANYONE can edit those entries. Maybe there was NO way to have founders enter such data ...

Very useful. Thanks.

There are so many books on data visualization it's hard to judge just from the reviews which are the canonical textbooks. So far I was just recommended this book: http://book.flowingdata.com/

I've updated the graph according to your suggestion.

Before every variable was normalized according to its own min/max, now there is a global min/max. Thank you all for the suggestions.

This graph is about daily domain changes. Maybe if you looked here, it would help (the four values in the four tab headings): http://www.dailychanges.com/domaincontrol.com/

I'm experimenting and leaning new ways to visualize the data. I think there are better ways of doing it than a bar graphs or line plots. Well, it seems it did not work that well. GitHub uses similar visualization technique, though. Maybe next time I get it right. Here is my inspiration:

http://raphaeljs.com/github/impact.html

EDIT: Initially I wanted to do Sankey diagram but that came out way too complicated and there wasn't a good way to show new and deleted domains.

Sure, data is there; tools are there; interest is there; but I'm always wary if skills are there.

We teach computer science, math, statistics, and machine learning but there is no good data mining class. Data mining is about being cunning. Sometimes quantile bootstrapping will work, other times you need cross-validate L1 radial support vector machines with non-linear feature selection. Good dialog with domain expert, broker, game designer, or medical doctor, will save you weeks of work. This is what is so hard to teach ... and we are left with practice.

SaintSal: It's a good start but apart of semantics (icons with meaning) you also need a grammar to call it a language. In case of visual language it means that there is a way of merging simple glyphs into more complicated icon. Here is an article that describe how it can be done in medicine:

http://www.sd.polyu.edu.hk/iasdr/proceeding/papers/Collabora...

I'm looking for an open source project that I could use in my applications. There is http://thenounproject.com/ but it does not have grammar. The merging of icons is done for you (e.g., the iron icon with one, two, or three dots)

I'm hoping that Raphael will get higher level API. I think g.Raphael went in that direction but more needs to be done.

jnbiche: maybe I can interest you in https://gist.github.com/1515418 please let me know if there is anything I should add. thanks

This mirror does not auto-delete (as posted on r-devel):

http://stat.ethz.ch/CRAN/web/packages/

There should be a ton of R packages to download here:

http://cran.r-project.org/web/packages

This change is propagating to the mirror websites. For some it might mean downtime with their statistical analysis.

EDIT: Actually, you can still get the packages source code here:

http://cran.r-project.org/src/contrib/

Binaries for mac os x are here:

http://cran.r-project.org/bin/macosx/leopard/contrib/

But I don't know where to get the PDF documentation for the individual packages.

EDIT2: Google keeps cached PDFs, you can view them using Google docs pdf reader.

P.S. I've contacted Uwe, Kurt, and Stefan but probably they are already gone for the day.

I suspected that there will be a sentiment for this game here ... one of many reasons for me to become a hacker. Through games, their small imperfections, I've started programming computers and get passionate about it. Nowadays with game consoles, many kids will miss the joy of hex-editing binary files.

Is there a more strict definition? Would this work:

  SAAS - HTML = BAAS

Too broad or too narrow?

I found this Chrome app that adds IMDb scores:

https://chrome.google.com/webstore/detail/pfjnpblfpdeldjpfok...

I've learned by observing and reading. You can start here: http://dribbble.com/ . After that, use http://www.hnsearch.com/ to find good articles about UI and UX design.

One important lesson for me was that web design (UI and UX) and web programming are two different tasks. Best outcome is achieved when they are separated. Paper and pencil will be your best friends.

At the first glance: This could be complimentary tool to WEKA. It has some features that WEKA does not have. To name one: non-linear feature selection (e.g., Manifold Sculpting). It has extensive visualization libraries (2D and 3D). It can store sparse representation of data which is a huge memory saver for text mining (NLP). The biggest complaint would be that I don't see how this tool could do feature selection INSIDE cross-validation loop. It seems that authors are unaware that feature selection on the whole data set is prone to overfitting.

Note to self: It has mean margin trees but no SVM? Thread safe? Portable to R? C++ codebase. Why SHA?

nicoslepicos, ig1, OrionJ - Thanks a lot! Is there a source code? I would like to contribute by adding IMDB scores. What do you think?

Instapaper is for bookmarking and reading. It strips unnecessary clutter. You provide links to interesting stories - one might want to read them in a more readable format or send it to Instapaper and read it later on a tablet.

Here is my rumble: I'm thinking about an algorithm that would work like singnal-to-noise ratio detector. This requires a lot download bandwidth. It means counting how fast people like, +1, or any other on the internet (techcrunch, etc). Simply, replicating similar logic that HN uses for moving articles from newest page to the front page just for the top 10-20 most popular tech websites (techcrunch, gizmodo, etc). Then you add nice readability features (e.g., Instapaper) and you have a business. I might have a prototype next year but it seems that you have infrastructure that sets you ahead.

Note to self: this could be a data mining web app if encumbered with proper algorithms

earlyriser: would you mind sharing what is the logic when deciding which article is "the most" popular? is it a simple time snapshot or something more intricate? have you considered using Instapaper API?

I'm looking for a way to have machine learning algorithms on google app engine (GAE). Ideally, it would be Python 2.7 + Numpy with C or C++. Is there a hack to do it? Do I need to rewrite C code in Numpy code? GAE allows .pyc files - can I use inline C inside Python/Numpy and compile it to .pyc, then deploy the whole thing on GAE? What are my options (beside switching to Java, that is)?

This should be like ~4h project. Many people on HN are interested in data mining and web apps. I'm guessing it would be appreciated contribution here.

There are also papers that talk about the methods in depth: http://arxiv.org/PS_cache/arxiv/pdf/1107/1107.5728v2.pdf

Actually, it's more than that (e.g., http://www.nejm.org/doi/full/10.1056/NEJMp1109649) but there are two serious bottlenecks: you need medical experts and the data might be too noisy. This means that any medical data mining project will cost a lot of money and half-way in the project you will find out that your data has too much noise. Most people don't want to take that risk, despite that benefits might be enormous.

TL;DR: yes it's BS but nothing that high-risk/high-ROI startups couldn't deal with.