HNHacker News
TopNewBestAskShowJobs

salmonlogs

439 karma · joined October 12, 2016

submissionscomments
salmonlogs··on Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
The issue for me is that 99.9% uptime isn't a useful or meaningful metric. End users only care about the experience and if the application isn't performant, reliable and durable it doesn't matter if the lights are flashing - they will tell you that it's not working as intended. And when you rely on SLAs from third party providers the liability is not equally shared; they might credit you some % of your bill if its offline, but your reputational impact and opportunity cost is likely orders of magnitude greater. You also can't control how that 99.9% will happen, and more often than not its going to happen at the worst possible time (payroll dates, reports due, board need statistics etc.)

Mitigating these failures will always lead you down the path of replication, load balancing, high availability or at the very least frequent backups and restore strategies. And all of that is going to need to be done across multiple physical locations because I am never going to stake my reputation on a single physical site not losing power, connectivity or cooling. Now you are in the realms of worrying about network reliability, bandwidth availability for those replication and backup services in a way that doesn't impact user applications. And monitoring all of that, managing failures etc. etc.

As someone who helps organisations with their IT strategy and overall budget allocation process the focus is always on delivering reliable applications to customers and business users. Using a cloud provider helps us to ignore all the complexity behind the scenes that require significant investment in people and resources to manage once you hit a non-trivial scale. Paying a premium to do that is absolutely worthwhile compared to the downside of it going wrong, and the opportunity cost of wasting time on minute details that do not add value.

[Edit] And for context I DID used to buy servers one eBay for testing and development, and then migrated to bare metal colo, and all the while thought I was winning and it was cheaper. But over the years I've experienced enough issues and worked with enough companies to understand this was a false economy and now see the errors of my decisions and seek to help others avoid them.

salmonlogs··on Automate any tasks that require multiple entries of data in a browser
I'm not the OP and have absolutely nothing to do with the company. I replied to agree with your thoughts the pricing model was wrong and shared some more thoughts of my own; apologies if this was confusing, perhaps I should have made a new reply of my own.

Hope you are doing okay and find a way to vent your frustrations better than all caps on HN comments.

salmonlogs··on Automate any tasks that require multiple entries of data in a browser
I agree the pricing model could be improved and personally think simplification would be best change.

Imagine you have a single user subscribe who works inside a big company. They implement whatever process they needed and show a colleague who wants to do that too - either run the same process to build a new one. Your goal in this situation is to rapidly expand adoption inside that company and complex licensing will really slow you down. The second they need to place another order to get more access or sign a bigger cheque you will get dragged into sign offs and approvals and procurement and all kinds of wasted time.

"Land and Expand" is the right strategy for this type of solution and you should do everything you can to not slow down adoption. Your current pricing model is definitely limiting this and the initial plan of 350/mo is quite a jump up - that WILL require levels of sign off and approvals and slow you down.

Try this exercise - create personas for users of your service (power user who wants to automate EVERYTHING, administrators who will use it because its their job, end users, managers) and work what is important to them and what variables will increase or decrease the speed and volume of adoption.

I think pricing based on number of users alone would be easier to understand and not restrict growth. If one person gets really into your product and wants to automate 100 processes DO NOT GET IN THEIR WAY! Let that champion sing your praises and tell everyone that will listen how incredible your product is, because in a few months time when they roll those processes out they will come back for more user licenses. And if you select the right features into the higher plans they may well opt for a more expensive plan to meet the needs of security, billing, management, reporting etc

salmonlogs··on Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
I'm a huge fan of open source and open standards, in fact I always push for and expect portability and avoid proprietary systems where possible. Abstractions like Kubernetes are a fantastic middle ground to provide portability across platforms whilst taking advantage of cloud provider services where they exist. The same for apps and frameworks built on open standards like Kubeflow and Apache Beam.

The supporting services and systems come when you run services that require strong guarantees for reliability and resiliency, and meeting the needs of different lines of business.

If I think of a mid-size company that wants to run these kind of workloads and demands minimal downtime, resilience against local disaster and minimal data loss: - Customer facing applications in a reasonable scalable manner to meet peaks and troughs of demand without needing to size of peak demand

- CRM/ERP systems to manage customer data, payments, sales processes and inventory that CANNOT have corrupted or lost data

- Data platforms for running reasonable scale analytics and analysis on reasonable size volumes of data (say a few Petabytes online accessible, analyse 10s of Terabytes per quer)

- Capability for mid-level machine learning and access to modern acceleration hardware, up to date GPUs and maybe some NVIDIA Ampere type equipment

- Tools and platforms for operations and security that can capture, store and analyse all the logs produced by all those systems, plus some half decent cyber services - network level netflow analysis, maybe IDS if you are feeling fancy, endpoint scanning and analysis, threat intelligence capabilities to correlate against all of that data

- Tooling and platforms for developers - source control, artifact repositories, container registries, CI platforms like Jenkins ideally with automated security scanning integrated, CD for deployments like Spinnaker to canary and deploy your releases safely

- Networking for all of that equipment, ideally private backbones, leased Ethernet or MPLS - and that all needs to be resilience, redundant and duplicated

- Storage for all of the above that meets performance and cost needs, replicated, and backed up offline

Yes, you can do all of that yourself! But let's be clear, buying a server on eBay is not even a fraction of 1% of the reality of running real infrastructure for real systems for real businesses. There ARE reasons why you might do this but that is increasingly the exception due to either extreme scale, regulatory and privacy requirements typically from data sovereignty or very unique hardware requirements.

salmonlogs··on Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
Sounds like you found an amazing support rep and made a great case for it - good job!
salmonlogs··on Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
^^ this. Hard spending limits seem great until your app/service gets super popular and you have to explain to the CEO why you were down during the exact window you needed to be serving the demand.
salmonlogs··on Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
And this comment makes you look incredibly naive and narrow minded.

Running some code on a CPU != running a startup. Great you can buy a Dell server on eBay, or you can build a powerful desktop, or rent a VM or get a droplet or scrape on lowendbox. These are not a secret and there is a great reason no one does this other than hobbyists and neckbeards.

You do your testing and it works, then what? You have to deliver scalable reliable systems in production that require identity management, security, backups, resiliency, reliability, various networking services and a million other supporting services and all the systems that come with it. Never-mind actually scaling the application, monitoring it and all the tools, systems and processes needed to run reliable systems in production.

The eBay servers provide you exactly zero of that and you've just wasted time setting up an environment that is a snowflake and doesn't represent reality. Testing on the cloud on exactly the same platform you would use for production has a lot of benefits when you look at value as limited developer time delivering value to customers and the business.

Whilst the $1200 server on eBay might be cheap today, you are entirely missing the hidden cost of lost time when your team of developers costing $M/year are wasting on testing in an environment that doesn't help them find and solve production issues. You don't need many hours of wasted time or downtime to lose all of your so called cost gains.

Optimising for absolute minimum cost is a fools errand that only slows down actually delivering production systems that deliver value to your customers.

Please spend some time thinking bigger about the opportunity cost and value delivery of technology beyond the immediate dollars and cents - it might surprise you.

salmonlogs··on Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life.

Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of tickets and attended many, many meetings to make this happen.

salmonlogs··on The UK's “Chief Digital Officer” job has been open for over a year
As a former London based FAANG employee my experience is that base salary might be that for a relatively junior engineer, but a mid-level with 5-8 years experience would make around £150-200k total comp (base, bonus, stock, benefits). 10-15 years experience is £250-400k++ total comp range
salmonlogs··on The curious tale of Bhutan's playable record postage stamps (2015)
> web2.0 was a mistake.

Can you elaborate on this statement? I don't understand what this refers to

salmonlogs··on Using SIMD to aggregate billions of values per second
"its not possible to be accurate to that resolution with current hardware"

Are you referring to the clock precision of consumer grade hardware here?

In my experience the vast majority of financial time series data is reported in nanoseconds. The data providers, vendors, exchanges and data brokers absolutely have hardware capable of measuring timestamps in nanoseconds.

The accuracy doesn't have to be to 1ns of resolution to warrant measuring in nanos - even to the nearest 100ns is a useful and meaningful improvement beyond micros.

salmonlogs··on Ask HN: Is there a service like Heroku for hosting .NET core applications / dbs?
App Engine Flexible for .NET Core is fairly low effort to go from code to deployed

https://cloud.google.com/appengine/docs/flexible/dotnet/an-o...

salmonlogs··on YouTube bans content “showing users how to bypass secure computer systems”
Google's own bug bounty program (Vulnerability Reward Program) allows you to submit YouTube videos and even has a tutorial on how best to make one:

https://sites.google.com/site/bughunteruniversity/improve/ho...

salmonlogs··on Best Practices for Building a Microservice Architecture
Great article, very comprehensive
salmonlogs··on ZetaVM, my new compiler project
+1 for the amazing domain name
salmonlogs··on Google will now tell you whether a bar or shop is busy in real-time
Shout out to http://www.avoidhumans.com/ which tells you which shops/restaurants are NOT busy. If you're ever on the road and need somewhere quiet for a phone call, this is a lifesaver.

Hopefully they can integrate the Google data too to improve accuracy

salmonlogs··on Ask HN: Programmers of the 80s how do I learn to build the things you built?
Counterargument: Projects built in the 80s/90s that are still alive today are the 0.001% of projects that have survived for a myriad of reasons, not necessarily code quality or programmer ability

Given the time period there has been ample time for a large number of highly skilled people to contribute high quality code, increasing the overall project quality and stability.

I would hazard a guess the early beta versions of Apache, MySQL etc. were far from the quality and stability we see today

salmonlogs··on LinkedIn accesses Gmail contacts via ‘auto-authorization’
Agreed - this is the assumption I have been working on and experienced too
salmonlogs··on Most drivers who own cars with built-in GPS systems use phones for directions
Maybe it's a regional thing?

I'm in the UK and my maps get updated frequently (it often has a % downloading bar for the update) and when I compare with my phone the route and traffic is very similar to Google Maps.

salmonlogs··on Most drivers who own cars with built-in GPS systems use phones for directions
I totally get your point, and agree.

CarPlay/Android Auto certainly improve quality on average for most cars, but I'm not convinced they will solve the problem of outdated technology.

I previous had a Mercedes with "Apple Certified Connectivity" which was great, until they switched to lightning cables and the connectivity broke.

Whilst I'm sure CarPlay/Android Auto will last longer, I don't believe there won't be any breaking changes in future that will break compatibility. Given that we replace phones a lot faster than cars I can see a future where there are cars which only support CarPlay version X, and the standard is now version Y.

At that point you then end up with no navigation or media, because you aren't going to use an older phone just to have it work with your older car.

salmonlogs··on Most drivers who own cars with built-in GPS systems use phones for directions
Reading the comments I appear to be the exception. I lease a 2015 BMW 4-series which I added "Professional Media" to and the nav works really well. I've driven plenty of other cars where the nav is useless, but not all are bad.

Reasons to use it over my phone:

* 11" screen shows me a high quality map overlayed with current traffic, with 1/3 split for lane guidance when needed

* Directions also shown directly on the dash - no need to look far away

* Spoken instructions dim the music volume

* Pretty accurate traffic information with automatic OTA updates

Downsides:

* Voice control is shitty; saying "set destination to leamington spa" will change to a random radio frequency

* Slightly awkward to send routes from PC to car, has to be via Remote Control App

← PreviousPage 3 of 3