Hacker News Reader

If you're familiar with the Willy Report (Google it), the simplest working theory is that someone figured out a way to award themselves USD/etc balances inside of the Mt. Gox system, purchased large amounts of Bitcoins with them, then transferred them out in the ordinary course of business. This caused positive improvements in the set of metrics that Gox and the community actively monitored, and the metrics in which the discrepancy would have been obvious were apparently not actively monitored. (e.g. Comparing the sum of all USD/etc liabilities as reflected in the database with the sum of balances at Mt. Gox's single-digit number of bank accounts.)

You could call it the most total failure of accounting controls in history, if one had any reason to suspect that Mt. Gox possessed meaningful accounting controls. There was no reason to suspect this, but many people were willfully blind to that, because it appeared that Bitcoiners were making money hand-over-fist.

Edit to add: One reason Gox may not have done the math on the assets vs. liabilities is because they knew they were insolvent from some point in early to mid 2013, as a result of some combination of external events, including the since-reversed freezing of their US bank accounts. The clearest publicly available evidence of insolvency was persistent unwillingness to allow people to withdraw USD/etc and ludicrous excuses for why this was the case. (At one point, the CEO alleged that their daily wire volume had overwhelmed the second largest bank in Japan. This kind of claim is very useful because it is trivially falsifiable.)

> You could call it the most total failure of accounting controls in history

Enron?

Nah, Enron's prominence is purely political, it was useful for various narratives. There are far worse examples, and Mt. Gox sounds like it's in the running.

E.g. there's cheating on your general ledger (which at 50,000 feet might be the sort of thing Enron did), then there's not keeping a general ledger at all, which is being posited in this discussion.

Even I, who's never run a company, but did watch my parents run some while I was growing up, know how essential it is to keep a general ledger. Otherwise you have no real idea what your financial state is.

not keeping a general ledger at all

To be fair, you don't need a general ledger if you are setting up a website to trade playing cards. Say an Online eXchange for Magic the Gathering.

Every time an Mt Gox story comes up, the most incredulous thing to me is, why was anyone even expecting these guys to act like a bank? Why did anyone even consider trusting them with actual money?

Because bitcoiners were willing to believe anything which fit their narrative, logic and diligence be damned.

LOL Bitcoin fanboys have shown up on this thread I see judging by our down votes...

You'd probably still want a ledger there, if you're trading physical goods. That way you can correlate confirmed receipts on both ends of whatever cards were traded.

No probably. Presumably you're doing this for a cut of the action, so that's income on one side, then there's your out of pocket costs to run the system etc. Knowing how these two compare can tell you all sorts of things, including how much you can safely take out for yourself if things are going well enough, vs. knowing you'll have to pay next month's co-lo fees out of your pocket.

Now, of course, at that level something as formal as a general ledger isn't quite required, but you still need to do the same things in e.g. watching the organization's bank balance.

I am admittedly a bit of an accounting fanboy, and may jump to more rigorous tracking of things I control or are accountable for than is needed. For instance, upon discovering a large portion of the intelligence funding in the United States is "off-budget", I was concerned about their financial rigor (your budget should include all the things you spend money on -- it's odd to think the US Government is quite a few steps and security clearances away from being able to answer "How much do you spend on pens a year?")

It wasn't an online exchange for MtG cards, it was an exchange for MtG:Online cards (digital collectibles), and, IIRC, other than buying the name, the bitcoin exchange didn't have much in common with it.

Mt. Gox was popular. In an environment without some sort of central accrediting authority or standards body for Bitcoin businesses (the very idea would be abhorrent), popularity confers legitimacy, and convenience confers popularity.

I also suspect there were also more than a few people who trusted it implicitly because it wasn't a bank, they just assumed it worked because Bitcoin.

When you begin from the premise that the models followed by the financial world are at best obsolete and at worst a criminal facade, of course you might not bother referring to them if you start a business, or care to find out if an exchange follows the best practices of an industry you consider corrupt and useless.

I'm not saying this describes all bitcoin holders or Mt. Gox customers by any means, but the politics around Bitcoin do seem to lend themselves to an unwillingness to hold third parties up to any sort of standard.

Bigger than Enron? Not even close: http://usatoday30.usatoday.com/money/energy/2002-01-22-enron...

And that's just the primary, don't forget about the fall-out (Arthur Andersen implosion, for one (miraculously, they survived)).

Only in a technical sense; Arthur Anderson is in a line of business where a criminal prosecution is pretty much a death sentence, a conviction absolutely so, even if later reserved. Per Wikipedia:

In 2002, the firm voluntarily surrendered its licenses to practice as Certified Public Accountants in the United States after being found guilty of criminal charges relating to the firm's handling of the auditing of Enron, an energy corporation based in Texas, which had filed for bankruptcy in 2001 and later failed. The other national accounting and consulting firms bought most of the practices of Arthur Andersen. The verdict was subsequently overturned by the Supreme Court of the United States. The damage to its reputation, however, has prevented it from returning as a viable business, though it still nominally exists.

How difficult is it to keep the right amount of bitcoin and USD in deposit? ... P.S. I'm just making all this up. I've never worked in payments or accounting.

Mt. Gox had a financial control problem less complex than that faced by the typical supermarket. A supermarket has cash going in and out, multiple cash registers, multiple cashiers on different shifts, credit cards, checks, benefit cards, and automated checkout lanes. That's just the front side - there's a comparable operation at the back, where merchandise comes in and is accounted for. A big supermarket has a higher transaction rate than Mt. Gox did.

If a $10 bill goes missing in a supermarket, the management will know at the end of the day, and will probably be able to figure out where it went. This is routine cash control. It's a solved problem.

Mt. Gox had no financial controls; we know that from press reports. None. They don't even seem to have had a general ledger. This is rare in real businesses, but it's common in fraud schemes. Fraudsters don't want accounting controls, with everything they're stealing on the record.

Anyway, on Jan 3, we'll hear more. That this story came from the Tokyo Metropolitan Police is significant. The police haven't said anything until now, except that they were investigating. That silence has ended. Arrests may be announced. There's a good chance that Mark Karpeles is about to experience the standard 21-day interrogation used in Japan.

> If a $10 bill goes missing in a supermarket, the management will know at the end of the day, and will probably be able to figure out where it went. This is routine cash control. It's a solved problem.

Not necessarily, differences are a daily part of working as a supermarket cashier. Most supermarkets have agreements with their employees (sometimes mandated by law), varying from full employee liability to full coverage by employer. It's actually quite common to have differences in your register, e.g. because customers let you keep small-ish returns or give "real" tips or you as cashier in a high-frequency environment make a mistake in sorting in a bill into the right compartment - or fraudsters trick you (http://www.weser-kurier.de/region/delmenhorster-kurier_artik...).

The point is that cash differences in a supermarket get noticed within hours. Some level of error can be accepted, but if somebody comes up $40 short several days each week, they're going to be looking for a new job. Mt. Gox claims they didn't know they were being drained of assets for months.

$40 short weeks in a row, or in this case, ~650,000 $500 bills short in a couple years :)

Exactly. I have a friend who while high school in the '60s worked in one of those highway restaurants, if you travel in the right parts of the US you'd remember their name.

Of all their cashiers she had the highest "volatility", you might say, from day to day. That conceptually annoyed her superiors, but they loved her because she never made a big mistake, and her daily errors averaged out very nicely.

Not to mention the 90+% conviction rate in Japan ...

Fun fact: in no year since 1957 has the Japanese court system failed to convict at least 99.9% of indicted suspects.

Even more fun fact: this is not the strongest statement one could make. The last number I remember citing for it was something on the order of two dozen not guilty verdicts, nationwide, out of about 125,000 prosecutions in a year, but don't quote me on that.

And you're not afraid? Do arrests seem legit when you hear about one?

I'm generally not afraid of the police when they're not actively questioning me on suspicion of me having committed a crime. That has happened eight times. Six of the eight times, it was on suspicion of stealing my own bicycle, the relevant evidence being that I was a) not Japanese and b) riding a bicycle. One time, after the cops had called my bicycle's anti-theft registration number into HQ and received the radio report "That bike is registered to one Patrick McKenzie", I was accused of stealing Patrick McKenzie's bicycle and identity. [+]

Does this answer your questions?

[+] I realize this sounds like the kind of story an Irish storyteller would make up to prove a point and feel that I must add "I swear by all the saints, by my hope of heaven and my fear of hell, this did literally happen."

It's the perfect crime. Steal someone's bicycle, along with their identity, and ride the bicycle with impunity.

How did you convince them you were him and that you hadn't stolen your own identity?

What next? They make you prove you've never been cloned?

I showed him a particular immigration document issued to me that he is not, strictly speaking, allowed to ask for prior to starting a custodial interrogation. He then started second guessing Immigration's decision to offer me a visa.

That was good news, since it convinced a colleague he was overstepping his authority. I was released 5 minutes later, with an admonition to lock up my bike properly to avoid the scourge of foreign bike thieves plaguing Ogaki.

And the other two times?

That seems incredibly scary to me. When something is that insane, what stops it from taking the next step of insanity? I'm sorry that it happened to you (let alone six times), and am glad you made it out ok.

One part of it is that the Japanese police won't arrest someone unless they think they can nail them. A lot of murders may be ruled suicide because there's no convenient scapegoat, he-said-she-said rape cases not pursued, etc.

OK, but that's approximately true of criminal justice in the US. For instance: wiretaps are administratively expensive to pursue, and so police don't request them until they're sure they've got enough evidence to win the warrant. But civilians aren't persuaded by this logic; instead, they see it as evidence that the whole system is tilted in favor of the police.

It should be as disquieting (or not disquieting) in Japan as it is in the US.

If you look at the criminal justice system as having some amount of diagnostic sensitivity at each stage in the process (having both false positives and false negatives as all systems do), then our comparatively lower conviction rate means that (assuming end-of-the-day-justice is equal), we should be even more willing to waste people's time developing cases that have no merit (because more of the diagnostic sensitivity is in the trial/conviction stage). So it's not unreasonable to assume it's easier to get a wiretap in the US than it is in Japan -- another explanation is that Japan is substantially more likely to have false negatives (as implied elsewhere -- but the false negative rate for many crimes in the US is quite high, as roughly half of all murders go unsolved). Could also be that the diagnostic sensitivity is extremely bumpy in America or Japan (read: basically one or two steps / people get an extremely high share of the criminal justice discretion).

Should be easy enough to prove or disprove by comparing the number of X-type arrests made to the number of X-type crimes committed.

If it's absurdly high, or significantly higher than some comparable country, then you've got a problem - with a 99.9% conviction rate.

99.9808%, given those figures. Some numbers I can find online pin it at 99.97%, which in either case is comparable but greater to that of Communist China (99.93%).

Just curious, searched and couldn't find anything meaningful, what is the "standard 21-day interrogation"?

http://bbc.co.uk/news/magazine-20810572

http://news.bbc.co.uk/2/hi/8290767.stm

Police can hold suspects for 23 days; those interrogations are private; Japan only recently re-introduced trial-by-jury (2009!); etc

Here's a nice 28 minite radio programme talking about the problems. http://www.bbc.co.uk/programmes/b01phktd

To make a long story short: the general standard of Japanese policing at present is quite similar to the prevailing standard of US policing in the 1950s. The primary investigative tactic is coercing confessions, and I do not use the word "coercing" lightly. Japanese cops can jail someone on suspicion of wrongdoing for 20-something days prior to having to bring a magistrate into the picture, during which time they're routinely physically and emotionally abused and made to think that signing whatever the cops put in front of them will be the quickest way to end the ordeal.

Doesn't this get abused heavily?

It is systematic abuse; I'm not sure it makes sense to talk about it being abused...

Indeed. Japan is a police state. A remarkably polite one, but things at this level are done for the convenience of the police, prosecutors and judges (at least prior to adding juries to the system, but I'm quite skeptical how that'll turn out for quite a while). Closing cases is more important than finding the real criminal, and atrocities like their handling of the Aum Shinrikyo cult suggest they're not so good when put to serious test.

Are they juries? A lay judge is different, and given the implementation of it, it seems barely better than what they had before, since a lay judge is typically going to just defer to the professional judge most of the time, anyway.

I figured the system would end up being "juries" only in form and didn't investigate it, but it's of course it turns out to be much worse as you point out. And then there's this gem from Wikipedia which echoes the early issues with common law juries, which were not much respected by the ruling class if the case was important and they rendered the "wrong" verdict:

"Others have written with concern regarding the harsh secrecy provision in the statute which includes the risk of criminal penalties for those lay judges who would publicly share confidential deliberation room discussion even after trial proceedings are complete.:

I'm sure there's a lot more where that came from, plus of course as you note extreme deference would likely be paid to the professional judges. The above is from the end of https://en.wikipedia.org/wiki/Lay_judges_in_Japan

"Atrocities like their handling of the Aum Shinrikyo cult"? Are you referring to the guy falsely blamed for the Matsumoto attack? The police are indeed likely to blame for leaking his name to the media, but he was never even formally charged, much less convicted:

https://en.wikipedia.org/wiki/Yoshiyuki_Kouno

The investigation of the Tokyo sarin attacks, on the other hand, seems to have been carried out pretty much by the book and I'm not aware of anybody outside the tinfoil hat brigade who contests that they got the right people.

After the sarin attacks, they did a good job. I'm referring to the multiple times they dropped the ball prior to that point, which among other things allowed the sarin attacks to happen. Their screwup on the Matsumoto attack is an obvious example; they they harmed an innocent in an incorrect focus on him is in a way much less important than letting the culprits later do much worse.

I don't know if this is actually "standard" or if there are just outliers that receive press attention, but there have been reports of innocent people confessing after being interrogated for 15 hours a day [1].

[1] http://www.nytimes.com/2007/05/11/world/asia/11japan.html?pa...

> I'd set up a write-only logging system to be used for auditing company systems

I bet Marc Karpeles had 5 people calling him at once that the trading engine was down again so THAT's where he put is efforts... He didn't get any calls from people saying "By the way, I hope you have robust auditing procedures."

Like some people live paycheck to paycheck, I'm betting the MtGox guys were living from critical bug report to critical bug report, without time for "luxuries" like "atomicity" or "database auditing"

(Let me say up front I am in no way intending for my comments to be a defense of the behavior of these criminals.)

It has been credibly alleged [+] that Mt. Gox's general level of engineering acumen can be summed up as "No tests, no staging environment, live coding by SSHing into production."

[+] Unnamed employee of Tibanne, the parent company which supplied substantially all of 0-employee Mt. Gox's staffing: http://www.pcworld.com/article/2105920/lost-in-translation-t...

On the topic of SSH: It appears that they used their own version of SSH reimplemented in PHP: https://web.archive.org/web/20140102104411/http://blog.magic...

True, but a system like I'm describing ought not to be terribly complicated to set up. Perhaps it's simply a remote syslog daemon under the control of a trusted exec. Preferably offsite or in a separate cage. Or if SAAS, under a distinct account.

Maybe we don't block transactions on writing to the log, but whenever discrepancies are detected of more than a short time (transactions more than 60 seconds old not appearing), we page all founders and senior leaders. The trading engine stays up, but we draw much attention to the logging outage.

(I'd also want controls like offline wallets and accounts that would prevent an attacker from obliterating our assets in a single withdrawal)

I'd like to note that Gonzague Gay-Bouchery is an oft-ignored part of this saga. He has gone to a lot of effort to silently vanish (deleting most of his online footprint just before the announcement last February) and has been mostly ignored, but reportedly had a large role in the business side of Mtgox, while Karpeles handled the code aspects.

This is just wild speculation now...

...but my best guess is their trading engine was so unstable that they ended up using one server to run a stripped-down clearance house algorithm for executing the trades as quickly as feasible, and a separate system that was the "source of truth" for how many physical assets in bitcoin/yen/USD were actually in possession by the company.

I'm guessing that there was absolutely nothing "atomic" about the system Marc Karpeles had built, their storage system was just an amalgam of disconnected, duct-taped databases living in different places, that were "kinda, sorta" in sync with each other, with lots of unpredictable, undefined behavior.

MtGox was pretty buggy and many of those bugs lost money, which could easily make them insolvent. Also, interfacing with Bitcoin at scale (e.g. managing thousands of addresses) required a lot of custom programming.