davidhollander

Yes I was, I apologize. It must have been MongoDB that required total data be no larger than amount of RAM.

The test was run using protocol buffers and Python client on the same computer testing reads vs. a naive map reduce. The naive map reduce was storing 4,000 files in a folder, treating the filename as the key and parsing the text file contents from JSON into a python dictionary to see if an attribute matches. Basically I figure doing a loop of thousands of blocking disk accesses from a laptop harddrive on a standard filesystem buffering nothing in memory should always be much slower than any database.

This was last year so maybe Riak's performance has increased since then. I'd be interested if TokyoCabinet was added as a backend.

>keeps values on disk in log-structured files

Of course it does. Log files are the easiest way to make an in memory datastructure persistant. Logs are generally used for periodically dropping fast writes as a backup in case you want to reload the memory datastructure in the future. Read requests are structured and served from the in memory datastructure and do not require a disk access.

The fact that you have something using all your RAM that should require 0 disk accesses for reads NOT substantially beating something performing thousands of disk accesses is a FUBAR situation.

Now you can simply increase complexity and create multiple Riak clusters for different data schemes and treat it as a replication tool. However, their in memory bitcask map-reduce was actually slower than a hard-drive map-reduce on text JSON files in a folder. Where each file was individually opened, read from disk, encoded, and closed in a loop (with nothing held in memory). Which was rather scary!

Their replication scheme appears to be a pretty genuine copy of Amazon Dynamo though (consistent hashing ring, quorum, merkle trees) which is nice.

Reality Distortion Field Enabled.

http://www.blogcdn.com/www.engadget.com/media/2011/03/201103...

>so although your "working dataset" can be far larger than RAM your actual "active working dataset" can fit in there

To me that's the exact same pattern as cacheing, merely stated in reverse. If you want full persistence, everything's going to have to hit the disk anyway no matter how you phrase it and how and when you store it. In regards to your user logs problem stated below, that sounds easy to parallelize. I would hash+modulo the username to a data server number, on the selected data server traverse a B+ tree of dates\entries in order, and buffer or stream until 20 entries matching the usernames have been retrieved in response. High performance and fully persistent.

Waiting for a new memory+disk database pattern that does everything more intelligently and faster than every other disk+memory pattern seems like procrastination of parallelization, which is the unavoidable long term answer to these problems.

Why use an in memory database for persistence instead of for cacheing? Considering that RAM is the scarcest and thus most valuable resource on nearly every server, in memory DBs like Redis are better used for storing already rendered data ready for output rather than every bit of raw data that always needs to be duplicated or logged on disk at the end of the day anyway. Persistence==disk!=memory.

>Redis Diskstore is something I'm eagerly waiting for

If your data can be represented using hashmaps (unordered) and b+trees (ordered), check out TokyoCabinet for on-disk persistance. It has the fastest on-disk hashmaps and b+trees (of non-fixed size) and has been around for a while.

Come up with your own curriculum then. Invent homework assignments like the following:

1. Install a linux distribution: http://www.ubuntu.com/desktop/get-ubuntu/download

2. Install the languages you mentioned: sudo apt-get install sbcl ruby

3. Type 'irb', 'python','sbcl' to open interpreters for Ruby, Python, or Lisp

4. List the names of all the data structures and algorithms you've heard of and their operations. Be sure to include hashtables and b+ trees.

5. List the worst\best\average case runtimes for each operation.

6. Describe a practical use case for each.

7. Implement each by typing directly into one of the above interpreters.

You'll learn much faster, always get A's, and not have to spend any money.

Limit the number of links submitted per account per day to 1.

Why

Prevents spammers and karmafarmers from submitting the entire TechCrunch\Wired back-catalog at a rate of 25+ a day.

Further Analysis

Increasing the scarcity of a resource (link submission ability) will increase the value of items it is traded for (links). HNers value independent news related to code or unique analysis. HN already gets the independent submissions people want. They just die an early death on the new page due to overcrowding by links from webzines\newspapers with a profit incentive for maximum linkbaitery. This feature reduces the rate of dropoff for independent news.

Limit the number of links submitted per account per day to 1.

Why

Prevents spammers and karmafarmers from submitting the entire TechCrunch\Wired back-catalog at a rate of 25+ a day.

Further Analysis

Increasing the scarcity of a resource (link submission ability) will increase the value of items it is traded for (links).

HN already gets the independent code submissions people want. They just die an early death on the new page due to overcrowding by webzines\newspapers with builtin linkbait titles. This reduces the rate of dropoff for independent news.

Would be interesting to see how its performance compares to Ry's HTTP parser from the Node.js library: https://github.com/ry/http-parser

That's how these things work Chris.

"The Net interprets censorship as damage and routes around it." -John Gilmore 1993

418 I'm a teapot

...seemed worth investigating. http://en.wikipedia.org/wiki/Http_status_codes

I make everything in SVG and use CSS to continually change the size without loosing detail.

I only convert to pixels\PNG when ready to deploy. Don't design any fixed pixel art during the development process and you will save tons of time.

Correct. 1 refers to the rate per individual inbox that was entirely essential for their initial growth and traction. Total volume and market segmentation > 1 ... is a no brainer for any business. While they can now afford to alienate and loose businessmen and others with low inbox noise tolerance, limiting the bandwidth they consume in each individual's inbox is still their weakest link and one of the only ways for them to actually loose subscribers.

Here's why Groupon works: data compression. A groupon is a signal compressing all of the economic data in an entire city into a single very best offer, at a rate of one per day. Consumers are overloaded with information and seek simplification, the value derived is a function of the ratio of compression.

Here's how Groupon can fail: they start offering more than one coupon per day. Anything that increases the perceived noise/signal ratio in crowded inboxes.

facebook.com appears to be a distraction black hole, sucking all other useless advertising sites and hypercasual games into it.

This is largely a good thing, as you'll now only have to remember one domain name not to visit.

>New York City and Microsoft Partner to Modernize City's IT Infrastructure - Mayor Bloomberg and Microsoft CEO Steve Ballmer announce first of its kind partnership to keep New York City at the cutting edge of technological innovation while saving taxpayer dollars.

Long term licensing of proprietary software somehow does not seem like the best way to save tax payer dollars. I wonder how many thousands of IT departments and millions of computers exist across federal, state, and local levels? It seems likely that either the Department of Energy, NSA, or DoD already maintains their own Linux or BSD distribution that could be used to run open office on every government computer in the country for free. Even if you are a Windows fan, it seems hard to justify spending tax dollars on that many licenses for government use.

The obvious solution is to limit the number of articles you can submit a day to, say, 1. Increase the scarcity of the resource to increase value of what it's used for and limit the power of karma farmers and spammers.

I think the key game mechanic increasing the longevity of Hacker News is the minimum karma threshold necessary to downvote comments. In effect, it requires new users to undergo a period of socialization to community norms. During this period they train in recognizing what the schema for comments containing negative informational value looks like. First out of fear, then out of mandate as an acquired power.

If you ever want to keep a community alive, implement similar withheld powers of negative punishment and adjust the level of input work required to earn them.

>mongrel2, lua and Tir

Also, respect to Zed and MVC fans. However I've found it simpler to write a pure Lua coroutine\socket based stack on top of HTTP and TokyoCabinet than to use preexisting servers, DBMS, or frameworks. The advantage of LuaJIT is it allows you to develop highly refactorable multiprocess stack in one language and have it probably be faster than I/O that you won't need to use non-refactorable C black boxes or frameworks.

Welcome to the field of Applied Psychology and Cognitive Science. Once you have reverse engineered the mind, the only practical application is manipulating it.

>just as likely that modern marketers are just doing what people want

For a TRILLION dollars we are not talking about "marketers" in the Mad Men, I-made-a-website-and-wrote-a-book, or "social media expert" sense. We are at minimum talking about behavioral psychologist PhDs and global corporate parasites like Coca Cola, Philip Moris Tobacco, etc.

In reality, those are still on the billion dollar level and not on the top of the food chain. The only trillion dollar companies I'm aware of though are financial institutions, so that would be my criticism of the article. If you really want to make a trillion dollars, you'll need to be much closer to the monetary spigot. In other words, a bank.

Function call: halt further processing of current call stack, execute procedures, resume call stack after function call when control returns.

Yield: halt further processing of current call stack, execute procedures, resume call stack after yield when control returns.

Return: destroy current call stack.

The variation present on the code continuation axis is far more significant than the variation on the coughing-up-values axis.

The proposed solution seems to be increasing syntactic complexity rather than reducing it by adding another keyword, "from" into the mix which is hardly optimal imo.

This to me indicates a systematic logical flaw of thinking of hitting a "yield" in the call stack as analagous to "return". When in fact it is the inverse and analagous to waiting for a function to return and thus more similar to a print() function call. The whole point of coroutine yielding is that it inverts the point of view of a call stack and allows the called routine to be the calling routine as well.

I still use Python for some things and have too many fond memories to ever hate it, but even if generators get cleaned up I'll probably stay w/ Lua for pipeline type projects as I've grown too used to runtime within an order of magnitude of C, better coroutines, and its more Scheme-like nature.

LuaSocket implementation of the above: http://w3.impa.br/~diego/software/luasocket/ltn12.html

They're great at first, but then let's say you need to subdivide pieces of the generator into helper functions for better organization. Or yield recursively. Turns out you can't, because yielding is A) syntactical and B) can only be done from the main function body.

It took us until Python 3 to replace "print x" with a more sensical, functional "print(x)". Yet for yield we are stuck with even worse arbitrary syntactical rules for "yield x", instead of being able to do "yield(x)" where it yields to the wrapping generator\coroutine from wherever it is in a call stack. The result is nonintuitive, less-refactorable code.

http://lua-users.org/wiki/LuaCoroutinesVersusPythonGenerator...

It's not high value from a monetary perspective either. It's low value, high volume. And anything low-value and untargetted enough IS spam.

I would quantify value of an informational chunk by viewing it as a signal and measuring the degree of data compression present. For example, a Groupon listing has extremely high value as it compresses all of the economic data in an entire city to a single very best coupon at a rate of one per day.

The only difference between spam, ads, and content is the probability that the degree of compression or informational value they offer is greater than the attentional resources required to absorb it. The threshold values between categories vary person to person of course.