cmurphycode

I think that infringement protection can be handled using the following scheme:

User uploads file. Mega computes the convergent encryption E(F) using the hash of the file H(F). The hash of E(F) is H(E(F)) and determines the file already exists in Mega, and thus is deduplicated.

Mega does not tell the user this, and their used storage size increases (thus, RIAA cannot upload The Hobbit and determine it's already there). The user enters their password P, the convergent hash H(E(F)) is encrypted with the user's password - P(H(E(F))) - and is only stored correlated with the user as such. The hash of the original H(F) (used to convergently encrypt the file) is also encrypted with the user's password, as P(H(F)).

On retrieval, the user enters their password P, the hash P(H(E(F))) and the hash of the original P(H(F)) is decrypted. Now Mega knows where to find the convergently encrypted file, using H(E(F)) to locate E(F). Mega decrypts using the hash of the original, and returns the file F to the user.

If the password P is only stored hashed (as it should!), then there is no way to correlate a given infringing file with any other user's ownership. The user's account only contains P(H(E(F))) and P(H(F)), both of which are unique to that user.

Anyone see a problem (other than the implementation details and possible lack of motivation for Mega to do so)?

Actually, there are encryption schemes that allow deduplication. They leak information (that the file you have already exists), but the encrypted bits themselves are secure.

The keyword is "convergent" encryption. We used something like this at Iron Mountain Digital many years ago (they still do, AFAIK), and it is used in BitCasa today.

You should read the papers, but essentially the concept can be boiled down to encrypting the plaintext with a hash of the plaintext.

Since there is no way to derive the hash of a plaintext from an encrypted block, there is no way to hack the key other than regular old brute force. But if the same data is uploaded twice, the same hash is computed, and thus the same encryption is used, and thus the encrypted cipher text is identical.

The encryption keys can be stored separately from the cipher text. In particular, the user who uploaded the data would store the hashes (this would already happen in most backup applications anyway). Then, for retrieval, they give the hash and the block location to the server, who is now able to decrypt it. By stealing the server, you gain zero access to plaintext data.

Very cool stuff :)

Looking forward to seeing the additional measurements.

Can anyone explain why "I can’t leave it partially open or the radiator will fill with water and start spitting it out of the auto-bleed valve."? Why is partially open different from completely closed?

Colin explained it, but I wanted to apologize for the dissonance. Essentially, I'm phrasing the abstract "pick an element, then pick again" problem in terms of SHA collisions. Just for fun.

Yeah, that was a phrasing mistake in my original post. I meant if there are ANY hashes that hash to itself (any fixed point). Thanks for noticing.

At my day job, SHA-1 is the basis for our deduplication. As such, we love talking about frames of reference for the probability of a SHA-1 collision. One of our presentations includes such facts as "the probability of dying in a vending machine accident", etc.

If you enjoy this stuff, you'll get a pleasant tickle out of this one:

What is the probability that there exists a SHA-1 hash which hashes to itself? In other words, are there any fixed points?

(This post was edited to fix the phrasing mistake nicely pointed out by a commenter. Originally, I asked "probability that a SHA-1" when I meant "probability that any SHA-1")

Software Engineers in Boston area (Waltham, MA)

Actifio is looking for software engineers. Actifio is radically simple copy data management. We've invented some really cool technology to do things no other system can, and we're dead easy to use. This is not your mother's enterprise software!

Actifio has been invested in by top VC firms (one of the very few East Coast companies in Andreessen Horowitz's portfolio) and is growing super fast.

We're always looking for smart engineers in every team, but since I'm on the deduplication team, I'll have to pitch you on that :)

Dedup seems like an easy problem to solve. Split the data up into blocks and make sure you only store one of each. But at scale, it becomes a whole different animal.

When you've got 128TB of 4KB blocks, you have 32 billion unique blocks of data to manage. Making our dedup ingest, restore, and garbage collect fast is a great challenge, especially with "only" 128 GB of RAM.

If you'd like to chat about a job opportunity, I'm at chris.murphy@actifio.com. Since I'm an engineer, I'm not going to type a list of silly requirements. Just send me a resume and whatever else you think I'd like to know.

The generally accepted wisdom in the running community (backed up by a couple of studies, but what isn't these days) is that HIIT is great for making big gains in the short term, but that the gains will peak before a program with proper high volume base training would.

In fact, the way most people are encouraged to train is to build up to some decent mileage (at least 20 miles per week) and then mix in speed work, which varies from simply faster runs, to Fartlek (speed play), to intervals.

Knowing this, it becomes obvious why runners fall in love with speed work: they might do a "traditional" training plan and run slow, high volume for 3 months, and see minimal but steady gains. Then they do 3 weeks of sprints and make huge gains! Awesome! But those gains were a reflection of the base work. Think of it like sharpening an old, abused, rusty knife: you don't go right for the fine grit. First, you grind it down, then successively apply finer grits. The knife won't cut a tomato without the fine grits, but if you skipped straight to them, you'd be at it forever.

The other reason one should be wary of HIIT is the increased risk of injury. Especially in running, if you go from couch to sprinting, you're highly likely to be injured. It is important to run slow and often, so that your tendons, ligaments, and joints (not just your muscles and heart!) are ready for the strenuous demands of running.

Similar concepts are often spoken of in weightlifting, where you are encouraged to start with just the bar and work up.

That's just the zsh regex quirk; try quoting *.java.

I do believe the GP's message was that the tutor told him his solution took one "step", which is not equivalent to O(1), because sorted() takes O(nlgn).

In other words, he was pointing out the limitations of the tutor program's analysis.

I think one huge benefit is the "instant"-esque nature of a find as you type fuzzy search in a GUI.

In Bash, you can do incremental (find as you type) history search with Ctrl+R. Something analogous for grep would be fantastic. Displaying more than one result at a time would also be great.

Another thing that grep/find lacks is a index that could make large fuzzy searches faster, but clearly that can be added too.

I understand. I worded my question poorly- what I was looking for was what they found that invalidated it as a cryptographic hash, so I could figure out whether it would fit my (non-crypto) requirements.

True. But in the case I'm evaluating, that isn't my worry. All I care about is the expected number of collisions over some extremely large number of hashes.

The page outright says that it is not a cryptographic hash, but the Hash Quality section doesn't expound on why (except for a minor flaw on short strings for one of the functions). Does anyone know what the expected collision rate for this hash is?

In other words, I don't care if the hash doesn't avalanche perfectly (half changed for one bit of input change; a requirement for cryptographic hashing so changes are "obvious") as long as the collision rate is ~SHA256. Or, if I should care, explanations would be wonderful.

I must be reading it wrong, because I see 8.4% unadjusted for high school graduates in Nov 2011. How are you calculating aggregate regardless of higher level education? I don't understand how you can just add the high school, some college, and bachelors to get 20%.

Can anyone find a source for the unemployment rate of high school graduates? 22.9% is astoundingly high, if true.

I couldn't find a source (beyond tons of articles echoing the study), and found several contrarian sources, including http://www.wiredacademic.com/2012/01/unemployment-data-most-...

Nice and lightweight- I like it!

Seems like a really nice addition to this would be to demonstrate a quick integration with git bisect (http://book.git-scm.com/5_finding_issues_-_git_bisect.html). That could help you quickly identify troublesome commits where the speed dropped significantly.

Wow.

I understood the argument that the speed of the Earth's orbit dominated the Moon's, but I was having a really tough time visualizing it. The wiki article (http://en.wikipedia.org/wiki/Orbit_of_the_Moon#Path_of_Earth...) has a diagram that explains it (http://en.wikipedia.org/wiki/File:Moon_trajectory1.svg)

Yes-- sample size, and effect size. He's saying that the results were strong enough that given the sample size, they were statistically significant.

Simple example: if I have a study of 20,000 people and I find that 51% of people prefer foos to bars, that may be similarly significant to your study of 20 people, all of which prefer foos to bars.

In general, I agree, but that is the use case for the mute button.

Yup, without question. The only thing I'd warn you is that when you're doing heavy work (video encoding or lots of heavy compilation/running) or heavy play (lots of flash), the battery life will probably be just under 5. Not at all like the 7 they claim, which is really only for casual surfing where the processor is bored out of its mind.

(I have the same i5 13", it's been great)

This is actually quite good. Around slide 100, he started bringing in some very interesting numbers. Among them:

  - using facial recognition, they determined the average number of unique people in your Facebook photos is 6.6, against an average friend count of 130.
  - 80% of Skype calls are made to 2 friends

He makes a very strong point about caring about these "strong tie" relationships, both from the point of view of the user, and of Google (the company making use of the relationships you cultivate).

This has implications for how you analyze Plus's success. If Plus ends up with a couple hundred million users, but an average friend count of 30 or so, I suspect Google will be just as happy, or happier, than if they achieved Facebook's level of friend count.

Seconded. And if you're looking for the real in-depth (although slightly older) reference, check out Harold McGee's Science of Food and Cooking.

I've been wishing for this for a long time. Actually, it was Google Reader's behavior that made me think of it- when you subscribe to a blog via RSS, Reader will show you all of the archived entries back as far as anyone else on Google has subscribed to it. Very nice.

I can only imagine that it's a content usage restriction, but you'd think they'd be able to lawyer their way out of it. The cable companies wouldn't even have to mimic Google's behavior- start by just allowing access to things you've personally recorded.

The future, of course, is to skip all of this nonsense and just let you stream what you want, whenever you want, and pay exactly for that. Ah, well.

I was surprised to see that our FIOS box power button does absolutely nothing. After you turn it off, it's still recording two channels, so when you turn it back on, you can rewind and watch the last hour of the current channel and the last channel you were on. It also constantly shows a "Press the menu button to turn on FIOS", so that consumers don't get confused when they power on the TV but not the STB.

Basically, the power button is completely pointless. I might as well leave the system fully on all of the time.

If you unplug the box, it takes about 90 seconds to bring the system back.

It doesn't seem impossible to program a state where the box is sleeping, but ready to wake to record scheduled shows, etc. Of course, that would still take phantom power, but at least it wouldn't be running full-blast all the time.

If you're having issues with ssh picking up your keys, you can always call it out explicitly: ssh -i path/to/key.pem ec2-user@blah.amazonaws.com

It's probably not as simple as just the fiber in fruits protecting against fructose/glucose, but it does help. (EDIT: what I mean is, don't replace fruits with a glass of metamucil and sugar :))

Soluble fiber is just that: soluble in water. In your stomach, it absorbs water and turns into a sort of carb-trapping gel. While the carbs are still eventually digested, the gel slows down the absorption. This is good, because it helps to regulate your blood sugar (not as much of a big spike and drop).

Since blood sugar spikes == insulin response == hungry signals, this is quite useful.

Insoluble fiber is also useful, but mostly just for waste processing.

"checkout and reset do completely different things when given files or when not given files. reset on files should really have been called unadd. reset on refspecs should really have been jumpto, moveto or something else indicative that the current branch ptr is moved to a new refspec. --soft and friends could have been --no-update-index or --no-update-files."

I can understand your confusion, given the seemingly separate use cases for reset, but in fact, it makes perfect sense. Reset always does what it says it does. Let's break it down:

git reset --mixed <commit> will make your current HEAD point to <commit>, reset the index to <commit>, and leave your working tree alone. This is useful for "uncommitting" the last commit, e.g. so you can split it up into smaller commits. Example:

  git commit -am "lots of changes"
  # realize you should really do better
  git reset --mixed HEAD~1
  git add myfile.py
  git commit -m "implemented feature x"
  git add yourfile.py
  git commit -m "bugfix #3182"

Handy. Now let's look at the "unadd' scenario:

  git add dontstage.py
  git reset HEAD dontstage.py == git reset --mixed HEAD dontstage.py, since --mixed is the implicit default

git doesn't touch your commits, since you are already on HEAD. Git does reset the index to HEAD, which is before you added dontstage.py. If you had other changes that you added, it won't reset those, since you provided the limiter of dontstage.py. Git does not touch your working tree, so dontstage.py stays modified. The end result? Your working tree, index, and commits look exactly like before you ran git add dontstage.py.

Now, if someone (e.g. easy git: http://people.gnome.org/~newren/eg/) wants to make git reset HEAD to unadd, that's fine by me. I'm speculating here, but I imagine that the Linus/git dev point of view is, why call it anything other than exactly what it is? It's just nice and elegant that it happens to suffice multiple use cases.

The more you get into git, the more you start to realize why some of the commands that seemed arcane in the beginning are simple and elegantly named.

That doesn't work for me. git commit -a --amend prompts for commit message in $editor git commit -a --amend -C complains about lack of input

Maybe this is a git version specific thing.

Exactly. And if you know that you just want the SVN/HG commit-everything, just add a -a. Pretty intuitive.

Once you get used to having the ability to create arbitrary commits from a huge changeset, you'll find it aggravating when everyone else commits "3 bug fixes to core, swapping out a style sheet, and cleanup in database code" :)