HNHacker News
TopNewBestAskShowJobs

gabbo

1,438 karma · joined September 8, 2010

submissionscomments
gabbo··on Canada forgot to plan for its future by leaning on oil and the loonie
It's crazy and upsetting to watch Canada squander the talent its great engineering schools produce, especially given the low cost.

I left Toronto 8 years ago for the US but have close ties to home and keep regular tabs on the labour market. Even at 1:1 CAD:USD, before factoring in increased cost of living in Toronto (vs. where I am, which is not SF), I'm making close to double what I'd hope to fetch back home. If I really bust my ass and kill it where I am, the disparity is only going to grow.

Once you account for other factors like the increased cost of everything and the general rarity of high-paying tech jobs, returning to Toronto feels like too much of a risk: even if I were to strike relative "gold" and make $120k-140k+ CAD as a senior engineer (a far cry from what I get how), what happens when I move on to something else? A close, highly-talented friend of mine has one of those jobs but feels trapped and doesn't even know where else he could go.

I love that the Toronto startup scene is growing and maturing and I have friends who are really working to, but I fear what's going to happen when funding in the US begins to contract. SF, NYC, and Seattle all have profitable "anchor" employers which will continue to bid for talent even when startup funding won't sustain high tech salaries. Toronto has small branch offices of American companies and some banks (I'm skeptical about the latter). Is there much else?

As far as I can tell there just isn't as much good work. I want to go home some day, but as someone who was fortunate enough to land a good tech job in the US: returning is a massive step down in pay for: fewer choices of work, a more fleeting labour market, a less ambitious environment, an expensive city with overpriced real estate, and a public transit/commute crisis which might not get materially better before I retire. I really do love the place though.

tl;dr I'm an exceptionally fortunate Canadian spoiled by great career prospects in a major American tech hub and find it hard to justify returning home :(

gabbo··on State of U.S. Salaries Report [pdf]
Oh I meant 2 hours each way (or 4 hours per day) is an extra 2-3 hours on top of what most would view as a typical North American commute (ballparking at 30-60 minutes each way, or 1-2 hours total per day).

The absolute level of craziness of 4 hours commuting daily is pretty high, but the incremental craziness on top of your typical commute is really what I meant here. :)

gabbo··on State of U.S. Salaries Report [pdf]
2 hours each way sounds absolutely, life-alteringly, macro-level awful to me. It boggles my mind that someone would consider that "a little on the high end for a commute" - it might be a worthy tradeoff all things considered, and "totally do-able" in that one can do it...but personally (and I'd expect for many people), I wouldn't be happy if I was in a position where I had no choice but to be commuting for 4 hours.

Yeah you can fill that time up with podcasts or audiobooks or that sort of thing, or work for a Bay Area tech company with a work shuttle and maybe-kinda-sorta-work along the way, but you're still giving up 2-3 hours of free/unconstrained time (one might make the argument "but I would be doing the same things at home anyway" but in my mind there's a huge difference between those two situations).

gabbo··on American big business faces the G20 in a fight for $2.1T in unpaid tax
It is hard, though: not least because determining where income is earned depends on defining "income". Revenue is maybe easy enough, but what about expenses? Depending on the industry, there's plenty of opportunity for chicanery which makes it far from "not that hard".

Example: you're a large tech company who can afford good accountants and complicated legal structures. Your high-tax jurisdiction European subsidiaries can "license" a whole bunch of "intellectual property" from their counterparts in Ireland/Luxembourg which significantly beef up their costs to the point that net income in the high-tax jurisdiction is minimal and it all gets booked somewhere which will charge you less.

Kind of like how Apple manages its assets via a Nevada subsidiary to avoid paying tax in California, and how Microsoft does something similar with software licensing. Except at least in the US case there's still tax revenue going to the federal government.

gabbo··on Money Flooding Out of Canada at Fastest Pace in Developed World
Depends on the market. I don't live in the Bay Area or NY, but knowing how much I make where I am (plus that of my peers) and similar pay levels in the Toronto tech labour market (one of the better ones in Canada), I would probably end up with ~50% of my US compensation in Canada even when assuming 1 USD = 1 CAD. Once you factor in the exchange, it's probably in the low 40s.

And that doesn't even account for take-home pay after taxes and the large Canadian urban markets' cost of living, since the tax brackets top out quite a bit lower in Canada (not that I think the higher taxes are a bad thing; I don't feel my taxes in the US are high enough).

gabbo··on Lessons learned writing highly available code
Easier said than done, if you don't have any state you're probably not doing something all that interesting.

The trick is to keep your state in the right place, the answer usually being to punt state management to a database/message queue/etc. - you definitely want to avoid, say, stateful frontends.

gabbo··on Lessons learned writing highly available code
Fixed-interval + jitter can work, exponential + jitter can work too. Be super careful with retries that don't involve jitter, depending on your system architecture you may end up with highly correlated bursts of retry load. You need to be careful of intermediate levels in your system when retrying. And be able to reason about what's going to happen under common failure modes (you won't able to anticipate the rare ones or freak occurrences, but you can defend in depth as best as you can).

If a low-level dependency starts to flake out (bad code/config/data push, unreliable host or network, load spike causing your storage to melt), retrying at intermediate stages can amplify the load and turn a small/medium-sized problem into a cascading failure. You can partially mitigate by retrying 0 or 1 times and then just give up, especially if you control all your callers or can afford that flakiness in your error budget.

Otherwise, depending on your needs you could maintain a retry budget (don't unless you really know you need it): instead of a flat K retries before giving up, enforce a process-wide per-operation cap on your retry rate. When the system is healthy, overall retry rate will be low so you have "room" to issue all K retries if needed. But if your load spikes, or your downstream dependencies start flaking out for some reason (your "fault" or otherwise), you'll avoid a positive feedback loop of death by quickly hitting your retry rate cap and shedding load which only would have made things worse and was destined to fail anyway.

EDIT: A few other remarks:

* OP's general point about putting limits on everything is really important, and closely related to the general wisdom in distributed systems development: everything can and will fail, so make the failure modes obvious and explicit. You get general robustness by avoiding runaway queries, but you also force discussions at design/development time like "what happens when this queue fills up" or "what happens if you're never able to take that lock" (though you should try to avoid needing to take locks ;)

* Instrument the hell out of the paths through your system and add lots of metrics. Logging is great, do it wherever it makes sense and you can, but a lot of the time you'll be able to get more insight (or the same amount of insight with less time spent doing ad-hoc log analytics) with something simple like counters whose value you can track over time and plot on a graph somewhere. Examples: classify downstream dependency latency into buckets, count of times you hit a code path (maybe some users require a more complex DB query than others or you're coding defensively, never expect a case to happen, but want to know if it does), cache hit rate, top sources of traffic in your system. Eventually you'll want to optimize bottlenecks and without data to identify them and prove they're fixed, you're flying blind.

gabbo··on Backblaze B2 Cloud Storage
As jbeda mentions, hardware errors are one big reason: with the scale S3/Azure/GCS/Backblaze operate at it's a matter of when and not if you're going to run into problems. Also: TLS may guarantee the bits your client sends are the ones their server receives, but that's just one cause of errors.

There's the write path from B2 receives your bits to when they're stored on disk, for one. You could have unforeseen bugs in the code sitting on the other end of their upload URL (it's probably not all theirs, and even if it was it was written by human developers).

Or B2's internal network path (if they have any) between that and the disk. Ideally that would provide integrity too, but maybe not. They offer a low price point and call out other compromises they make to achieve it (e.g. limited load balancing) - so while I really doubt it, it's remotely plausible they deem the internal overhead of SSL too high.

But then there's the potential for mismatch between "what the customer thinks they uploaded" and "what the customer actually uploaded" too! Less of an issue for now because their API only appears to support uploading files all at once, but eventually I'm sure they'll support a multipart upload scheme like the other platforms do. At which point uploads become more complicated since clients need to retain state and potentially resume. What if a client screws it up and there's some off-by-one error (or whatever)? If you can provide instant feedback, at upload time, that your clients provided bogus data, that's a good thing.

You can argue it's a painful requirement to force on users since it means they have to track/compute it themselves (might be nontrivial for streaming applications), which is fair. But there are enough points of failure, and the numbers so large, that errors happening is a fact and you really need to insure against it. Especially here, your entire reason for existing is to reliably store bits so it's kinda important to get it provably right.

It seems completely sensible to err on the side of caution, especially as a new and relatively unproven platform (as an object storage platform provider I mean, obviously they have tons of experience storing things).

gabbo··on Fury after Saudi Arabia 'chosen to head key UN human rights panel'
This is crazy. It would be like choosing America to head up the UN Social Welfare Commission.

(edit: this was meant to be a mildly sarcastic zinger, if it wasn't clear)

gabbo··on Downsides of Caching

  There's no real need to be running your cache in-process with your GC'd implementation language.
Fundamentally there's no need, but in-memory caching may still be the right choice. As always, there are tradeoffs. Standing up a separate cache component incurs non-trivial costs. Your service now has a new "unit of management" - a new thing you need to deploy, monitor, and scale. It's a separate thing which might go down unless it's provisioned for sufficient load, and you need to be careful about unwittingly introducing a new bottleneck or failure mode in your system. These are all solvable problems, but solving them comes at a cost.

You can totally argue that engineers should be forced to think about and address these issues up front with more rigor, and in a perfect world I think I'd agree. :)

gabbo··on Downsides of Caching
Good article which touches on real issues that a lot of developers won't really appreciate themselves until it happens to them (unless they have a strong background in distributed systems theory, and maybe even not then). A little strange that it doesn't use the word "consistency" even once though. :)

By dropping a cache into an existing system, you're weakening consistency in the name of performance. At best, your strongly-consistent system has started taking on eventually-consistent properties (but maybe not even eventual depending on how you invalidate/expire what's in your cache). Eventual consistency can help you scale, but reasoning about it is really hard.

In some sense caching as described by OP is a tool to implement CAP theorem tradeoffs, and Eric Brewer described the reality of trading off the C (consistency) for A/P (availability/partition-tolerance) better than I ever could:

  Another aspect of CAP confusion is the hidden cost of
  forfeiting consistency, which is the need to know the
  system’s invariants. The subtle beauty of a consistent
  system is that the invariants tend to hold even when the
  designer does not know what they are. Consequently, a
  wide range of reasonable invariants will work just fine.
  Conversely, when designers choose A, which requires
  restoring invariants after a partition, they must be
  explicit about all the invariants, which is both
  challenging and prone to error. At the core, this is the
  same concurrent updates problem that makes multithreading
  harder than sequential programming.
gabbo··on China Stock Market Drops 8.5%
I almost entirely agree with you, just wanted to add a bit more colour to your comments on Canada and Saudi Arabia.

Canada is clearly in recession but I don't think we can call it a "serious" recession yet. So far more of a mild contraction (but pretty terrible if you focus in on energy). I do think the rest of the economy is lagging behind energy and bad times are ahead overall, especially with so much of Canadian growth/prosperity tied to the housing boom. I'm probably preaching to the choir here, but seriously: when the prime minister promises a home renovation tax credit during his reelection campaign - and explicitly states he's doing to help boost the value of Canadians' homes - you've gotta know people are captured by the real estate boom narrative.

On Saudi, they can afford low oil prices for a while and can just borrow if they need to. Super low debt around 3% of GDP, low cost basis as you say, and they've done this before and come back from it (100% debt-to-GDP in the 90s o the back of low oil prices). I've seen interesting speculation that Saudi is willing to hurt a bit from cheap oil because it'll hurt an increasingly-economically-integrated Iran more.

gabbo··on China Stock Market Drops 8.5%
Saudi can just borrow money to help ride things out, their debt-to-GDP is incredibly low (like 3%). That got up to 100% in the early 90s, IIRC, when government revenues had been hurt by a protracted period of lower oil prices.
gabbo··on The Software Engineer’s Guide to Negotiating a Raise
It really depends. The numbers aren't huge life-changing at the beginning, but they add up over time and if you're really strong^H^H^H^H^Hlucky^H^H^H^H^Hshrewd and get promoted quickly, the numbers can get pretty large. The numbers below are hypotheticals but not too far off from people I know doing the BigTechCo circuit. Some do better, some do worse.

As a new grad you might see $5-10k in extra annual compensation coming from vesting RSUs (i.e. in addition to your salary and cash bonus). But RSU grants start to stack. Say your grants start vesting after 1 year at 25% per year. Year by year you might see:

Year 1: $40k/4 = $10k vest, new $10k grant (also vesting over 4 years)

Year 2: $40k/4 + $10k/4 = $12.5k vest, new $15k grant

Year 3: $40k/4 + $10k/4 + $15k/4 = $16.25k vest, new $25k grant

Year 4: $40k/4 + $10k/4 + $15k/4 + $25k/4 = $22.5k vest, new $35k grant

Year 5: $10k/4 + $15k/4 + $25k/4 + $35k/4 = $21k vest, new $40k grant

At year 5 you've seen $82.25k worth of stock vest (assuming a flat share price). Which leaves you with $82.75k left to vest.

Now you're tired of working at BigTechCo #1 and get an offer from their competitor, BigTechCo #2. You know how this works, so you tell them you've got $83k in RSUs. To make it worth your while, they offer you $160k of their own RSUs vesting on a similar schedule. You're an experienced, senior hire at this point so your RSU grants are larger, but maybe because you're a new hire who doesn't know the ropes your first couple of years at BigTechCo #2 show respectable but not explosive growth:

Year 6: $160k/4 = $40k vest, new $30k grant

Year 7: $160k/4 + $30k/4 = $47.5k vest, new $50k grant

Then you interview again and move to BigTechCo #3. Or back to BigTechCo #1. Or you tell BigTechCo #2 you're out and they give you a retention offer. Or you get a big promotion. Either way, the RSU grants add up. All the while you've been pulling in a respectable base salary, raises, and cash bonuses. And maybe, if you're lucky, your company's stock price has gone up too.

Or you realize your gross income of $200-300k in the Bay Area earns you the ability to buy property, which means you pick 2-3 of the following outcomes: [1] small and/or shitty house [2] 2-3 hour daily commute [3] overextended and house poor. So you move to the midwest and buy a palace.

gabbo··on Amazon Finally Discloses Cloud Services Sales, Showing 49% Jump
Amazon makes extensive use of capital leases support their expansion. In 2014 they were roughly on par with their other capex (in the $4B-5B range, don't remember exact number), except that their capital leases don't deduct from Amazon's free cash flow. If you subtract those out, the FCF situation ends up looking quite a bit worse.

http://seekingalpha.com/article/2743335-understanding-how-am... and http://www.fool.com/investing/general/2015/02/04/amazon-just... discuss the mechanics.

They're sort of taking on a bunch of debt (or things that look like debt) and downplaying it with clever accounting. In the current market environment I don't know that it's really a problem (they just borrowed $6B via a bond issue, for example). Just an interesting reality for a company competing with two cash-rich rivals in a hugely capital-intensive space.

gabbo··on Eric Schmidt's book is wrong about how Google works
No, monopolies aren't necessarily illegal in the USA. But they can be depending on the behavior of a monopolist.

From http://www.girardgibbs.com/antitrust-monopoly/:

> Monopolies are illegal if they are established or maintained through improper conduct, such as exclusionary or predatory acts. This is known as anticompetitive monopolization.

gabbo··on The Tech Worker Shortage Doesn't Really Exist
Your point may be valid, but the math is a little off. If you're an H1B holder going through the employment-based green card process (which could take anywhere from months to 10+ years), you can extend your H1B beyond the 6 year cutoff in 1 or 3 year increments depending on where you are in the process.

A former boss of mine was on H1B from India (big tech company and he wasn't even remotely underpaid) for 12 years, most of that time waiting for a green card.

gabbo··on The Tech Worker Shortage Doesn't Really Exist
Microsoft is hiring plenty. Go to https://careers.microsoft.com/search.aspx and search for software engineering jobs, the result is "only the first 500 results are shown."

Broad layoffs like what Microsoft did often serve as cover for divesting from product areas or culling dead wood (despite many US states being at-will, it can be surprisingly difficult to fire someone who isn't performing at a large company).

gabbo··on The Tech Worker Shortage Doesn't Really Exist
You can get TNs good for 1 year or 3, but you can technically renew your TN status as many times as you want. The downside is that TN status is strictly for people with "non-immigrant intent" and if a border agent believes you have immigrant intent when requesting a TN or entering America with one, they can revoke it/deny you entry at their discretion. So a 3-year TN is fine, maybe even a second one, but if you've been in the US on a TN after 8 or 9 years you're probably going to get some trouble.
gabbo··on The Tech Worker Shortage Doesn't Really Exist
Nit: Dunning-Kruger is actually when you're bad but don't realize it. I think you mean "Impostor syndrome," where you're good but feel like an unskilled fraud. :)
gabbo··on The Tech Worker Shortage Doesn't Really Exist
In all but the most special/uncommon cases it's likely to take more than months (unless you also consider 9-12 months to be a matter of months). I know many, many engineers who have gone through the US immigration process. Some had good experiences, some had bad ones, but I don't know anyone for whom the entire process took a matter of months - and their employers are the sort who would spend quite a bit of money to move things faster if they could.

For employment-based green cards you generally have to get PERM labor certification (where the sponsoring employer literally advertises your job and has to show that there aren't American citizens/green card holders who can do it). That takes a while on its own (weeks, generally months).

Then depending on your education/experience, you get a date which corresponds to your position in an immigrant visa queue. The more education/experience you have the higher the queue you can sit in, and it's done by birth country (not citizenship). There are separate queues for China, India, Mexico, Philippines, and "everywhere else." If you have a graduate degree, significant job experience directly related to the US position, or some skill the lawyers can pass off as highly rare/sought after, you might end up in the EB-1/EB-2 queues which are quite short (if you entered the queue today you'd already be at the front). In which case the processing time might only be a few weeks/months.

Or you could end up in the EB-3 queue (bachelor's or equivalent experience a.k.a. many software engineers). Right now people who entered the queue 2 years ago are at the front if they're not from China/India/Mexico/Philippines (http://travel.state.gov/content/visas/english/law-and-policy...), 4 years ago for China, and 11 years ago for India.

Once you get to the front of that queue, you need to "adjust status" which means a physical exam, fingerprinting, and other busy work none of which appear to be based on country of origin. You could end up here for a few months or over a year, it's hard to know until you find out what USCIS service center is processing your case and can look up historical data. Until you get to the end of this you're technically still bound to your employer though there are some loopholes. Adjustment of status can take sufficiently long that the government passed a law in 2000 allowing green card petitions in this state to be fully portable across employers after 180 days.

gabbo··on How to Make the Best Pourover Coffee
Brewing coffee is all about extracting good tasting stuff from ground coffee beans - some factors which affect extraction include:

  Water temperature (hotter water => more extraction)
  Grind size (finer grind => more extraction)
  Brew time (longer time => more extraction)
Overextracted coffee tends to taste bitter (or, as many people say when they're used to bad coffee, "like coffee" :)) and underextracted tends to taste sour. A bad grinder won't give you a uniform result (some will be more coarse, some will be more fine). Brewing should match up an appropriate water temperature + brew time with your grind size, but if your grind isn't uniform the coarser stuff will underextract and the finer stuff will overextract. So your coffee can end up having both sour/bitter flavor if the grind isn't even.
gabbo··on How to Make the Best Pourover Coffee
I have the same grinder recommended by devindotcom (http://www.amazon.com/Porlex-JP-30-Stainless-Coffee-Grinder/...) for work/travel and it's awesome.

Another recommendation on the brewing is the AeroPress (http://www.amazon.com/Aeropress-Coffee-and-Espresso-Maker/dp...). Pour overs are great and I love my Chemex, but if you're looking to brew a single serving - or want something which is easy to pack for travel - the AeroPress is incredibly hard to beat. It's cheap, fast, small, easy to clean, and makes it trivial to consistently brew a great cup of coffee. The Porlex grinder mentioned above happens to fit exactly inside the AeroPress too.

If you're going to do a pour over, a gooseneck kettle really helps. If you don't, it's nice to have but not a necessity. Either way this kettle might be one of the most useful appliances I have: http://www.amazon.com/Bonavita-Variable-Temperature-Electric.... It's a gooseneck kettle with digital temperature control, built-in timer, and hold function. You give it a temperature and it'll keep your water there for an hour. It's ridiculously convenient.

FWIW, I started off with a French press, which uses a coarser metal mesh to separate the grounds from the coffee. Because of this you often end up with a small amount of "coffee ground sludge" at the bottom of your French press. The remnants continue to "brew" in your cup after you remove the rest of the coffee grounds, so you might end up with more bitter, overextracted coffee if you aren't careful.

gabbo··on Show HN: Free, anonymous coding interview practice
You can do better by taking advantage of the fact that the lists are already sorted. O(n1 + n2) time, where n1 and n2 are the sizes of the original lists and O(n1 + n2) space for the output.

The "trick" is to iteratively merge your two lists: at each step you select the smallest of the two items you're at in both lists then move forward on the list whose item you picked.

An extension to this problem (I've asked, and been asked, many variations of this) is to merge K sorted lists of N items each. The naive solution is to merge them all together and sort but you can do better if you extend the algorithm above from 2 lists to K and choose a helpful intermediate data structure to optimize the "select the smallest of the K items" operation. AFAIK, all-up the best you can do then is O(NKlog[K]).

EDIT: Re-read the question, realized I read "unsorted" as "sorted" - not sure what you can do to merge two unsorted lists in better than O(Nlog[N]) time beyond non-comparison-based sorts where you make assumptions about the range or distribution of your input (radix/counting/bucket sorts).

gabbo··on Microsoft Supercharges Bing Search With Programmable Chips

  FPGAs are something you tell what to be, not what
  to do (and high level languages tell CPUs what to
  do, not what to be).
That's a really great way of putting it. I never thought of it that way before, but you're spot on.
gabbo··on Canada's Pitch to Tech Entrepreneur: We'll Pay 80% of Your Salaries
Hiring is really hard, but from anecdotal experience and hearsay, I still think the number of Canadian tech workers leaving for the US is a depressingly major factor. A large number of my ambitious Canadian peers ended up moving to the US, and Canadians are really well-represented among US tech companies. Many end up moving back (or plan on moving back), but it's almost always for personal reasons at significant cost.
gabbo··on Canada's Pitch to Tech Entrepreneur: We'll Pay 80% of Your Salaries
As someone who keeps tabs on the poutine and labour markets back home, I can vouch for everything you said.

I don't even live in CA/NY (so my gross pay multiple isn't quite as high, but income net tax/cost of living is pretty great) and there's just no comparison. I probably make 60-80% more total comp in the USA than I would expect in a major Canadian city, and that's not even accounting for higher taxes (depending on state/province), frothy Canadian real estate valuations, and higher consumer prices almost across the board.

But Canada sure is a great place to live and I miss it dearly. I still hope to move back one day.

gabbo··on T-Mobile to offer free unlimited international data, texts
Doesn't seem like it:

> New and existing T-Mobile customers who are on a post-paid plan are eligible for the new international plan, which kicks off on October 31

gabbo··on Windows Azure Storage certificate expired?
If this outage is solely caused by an expired SSL certificate then this is really bad. It makes Azure look like amateur hour (even though it's definitely not). This may be acceptable for a smaller operation, but not Microsoft's cloud platform. There should be processes (manual and automated) to deal with this.

On the bright side, after this you know they're not going to make the same mistake twice.

gabbo··on Windows 8 Developer Preview is Live
I installed this on a Hyper-V VM - everything went pretty quickly except after install, the first time I tried to log in it sat there a while. I shut it down (cleanly) then it booted up fine the next time.

Everything pretty much works now, I've been playing around with it mostly via Remote Desktop. You'll probably want to disable some of the animated window switching transitions if you do because they don't seem to remote very well (yet?).

← PreviousPage 2 of 3Next →