Choosing Go at American Express (2019) | Hacker News Reader

Choosing Go at American Express (2019)

americanexpress.io

62 pointsveqq69 comments

This seems a lot more like someone deciding on Go up front and then pushing a process through to pick it - it wasn't the fastest in their tests and it was the only language without existing usage at the company, and basically all the qualitative advantages are either not very unique or not quantifiable or both. No other new languages besides Go were even considered, despite some that easily meet the criteria for consideration.

Oracle's lawsuit against Google's use of Java API's started in 2019[0]. The writing was on the wall; many Java shops started looking at other options.

[0] https://en.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_....

The lawsuit started in 2009

https://en.m.wikipedia.org/wiki/Google_LLC_v._Oracle_America....

So if you're a Java shop you pick Scala because it seems amazing until you read about real world usage/find out for yourself, and then you pick Kotlin.

If you forced me to conduct best language olympics, the process I would use might look something like this:

1. Each team develops the same vertical slice of the business in their ecosystem of choice.

2. Test officials will make an edit to each codebase (somehow invisible to git history) that introduces a subtle bug. Alternatively, they may propose a small feature enhancement.

3. Time how long it takes each team to get a corrected/updated build deployed & re-tested successfully.

4. Go to 2 until statisticians are satisfied with the results.

Wouldn't that test the team as much as it tests the language?

That might be the point. The best tool is often the one that the team knows best. So you might really be testing for the case where the team knows a tool best and thus can respond the fastest to the work request.

I actually worked with Ben for a couple of my formative years as a software engineer - he's an extremely smart guy.

I'm no longer with Amex (still up for debate in my mind if that was a good idea to leave or not), but last I heard the department was very successful in onboarding a lot of their Java engineers to Go and transitioning to a Golang microservice architecture.

I find as an end user/consumer that Amex has one of the best web/app experiences of all the traditional financial companies I've interacted with. How did it feel working on it? I get the sense a lot of banking software is makeup on an ancient mainframe but not with Amex. Was it a fairly good environment for developers?

Are you sure that isn't the case for Amex as well? I recall being forced to use a very particular set of legal characters when defining my password for use with Amex online account services. The exact same constraints I have with several other financial institutions.

I suspect that somewhere in the heart of the beast there still lurks an IBM-branded system of record.

Just because a machine from the 90s is running the database doesn't mean you can't put a webGL frontend on top of it.

I used to write for a service that submitted transactions to AmexDirect. It's still done in EBCDIC, an encoding which famously led to a lawsuit against a European bank by a customer who couldn't create an account with his real name.

Amex is probably the one company of my three previous jobs I'd gladly work for again.

This is from 2019. I'd be curious how all of this worked out for AMEX. We've got over 4 years behind this decision now.

Edit: I found a followup from 2021 - https://youtu.be/e7PtBOsTpXE

Whether they had chosen Go or Java, or anything else, what I see here is an internal culture that really takes ownership of their language choice. That is its weight worth in gold. That is what matters in the end ("can we ship?") - not potential issues with the language itself.

Go is a decent industrial programming language alternative to Java, only things I'm not super enthused about is the logging experience (for the love of Christ, please create a standardised logging facade along the lines of slf4j, and allow the person running the application to set logging levels for individual loggers and choose whether stack-traces are emitted or not! Bonus points if I can change logging conf for a running app without restarting it) and libs that use C bindings, I have enough problems with that in Python already.

E.g., my manager is really interested in Dapr as a way to make basic Kafka usage easier for teams across our org.

Dapr's written in Go, and have made the reasonable decision to not use any libs that have C bindings for the same reason I don't like them.

However, this means that they can't use the Confluent Go Kafka client, as it's just a wrapper around librdkafka.

So they use Sarama, as it's written in pure Go, and is widely used, but is one of the most problematic Kafka clients I've had the misfortune to support. Although now that IBM has taken ownership of the project from Shopify, some of the more egregious bugs and oddities are finally being fixed.

But yeah, while you _can_ bind to C libs in Java, I've never encountered a library that did, so cross-platform lib compatibility wasn't something I ever had to worry about when working with Java.

You may rejoice on the logging front: https://go.dev/blog/slog

This is nice but it’s not a simple library to use and I think that’s going to hurt it.

It's as easy to use as other 3rd party logging libs.

I frequently make use and it's as easy as slog.Info("text", "field", value, "field", value)

It’s very simple to use, but dilly-dallying about a standard structured logging package means it’s now not widespread across third party libraries.

Many non-trivial Go apps have 4-5 logging frameworks pulled in, it’s a mess.

franz-go is the least painful kafka library out there

I personally don’t trust anything other than the confluent-kafka/v2 package.

franz-go is like 4x-20x faster than standard confluent client

I’ll try it

Having looked at the benchmarking code in franz-go, I'm very dubious about those claims.

1) What settings did they use for those published numbers? 2) They don't change the default `linger.ms` for confluent-kafka-go and indeed, don't provide a argument to do so, unlike for the code to test their client.

Automatically that renders the benchmark useless - the code running franz-go defaults to linger.ms of 0, librdkafka defaults to 5ms, so already you're not comparing apples with apples.

Slightly longer linger.ms should squeeze more throughput (at the expense of higher latency) not the other way around.

I ran some tests and it does appear to be about 3x faster at consuming, for my use-case.

It’s not that hard to get librdkafka linked to your Go code.

I did it in a day, behind a corporate firewall, without root access, and an outdated GCC compiler.

Most of the day was spent compiling a new version of GCC.

It's not that hard, but it is an extra step.

>Go is a decent industrial programming language

Industrial?

Yeah, as in writing code in the general software industry, CRUD apps for businesses etc. Didn't mean for running machinery / sensors etc. although it could be good there? I have no experience at that level.

I say this as a personal rust fan, but practically if a language like a combination of Kotlin+Go existed, that'd be like an awesome standard business language. Kotlin is good but just very tied to the JVM, Go has a hostile syntax/developer experience. Another way to frame it is like Rust with a garbage collector. Most biz APIs don't really need zero cost abstractions of Rust.

If you're interested, take a look at Crystal (https://crystal-lang.org/)!

This is Swift. It‘s just not being used a lot outside of iOS / Macos dev.

I see OCaml described as "Rust with a GC" pretty often, but maybe it's too heavily functional for what you're seeking.

Is being tied to the JVM that much of a problem these days? Developer workstations and servers have much more compute and RAM these days, and native compilation with GraalVM seems to eliminate high JVM startup times and memory overhead. Quarkus for example is known for a snappy developer experience.

I need Go to take one lesson from rust and that’s specifying if a function will mutate values passed through it or not.

This can technically be done as a breaking change spread over several versions of go and I think the cost would be worth what we’d get out of it. Utilities can also be made to edit code to support this automatically without having to have some user go fix it all.

Basically, I want an introduction of either a function level or value level ‘nut’ keyword like Rust where the function can not mutate a field unless it has explicitly stated mut.

This would fundamentally solve a whole class of bugs and errors that happen in go where some library ends up modifying a struct (pointer or a struct with a pointer field) without the caller intending that.

I'm not sure I fully understand what you're proposing? Is it about function purity or immutable values?

I think the way to do the latter in Go is to not export the field and only have it modifiable by a dedicated method. (or even enforce copy-on-write).

And passing values instead of pointers where needed.

I don't think that breaking compatibility to add a mut keyword would fit Go. It makes more sense for rust.

(especially wrt interior mutability... That would make it a very pervasive change, probably too much of a change)

Another thing is that wrapping values in an empty interface essentially make those immutable. It doesn't necessarily have mechanical sympathy but as long as you don't wrap pointers, it would be difficult to modify (no aliasing).

Then again, it's more defensive and pragmatic coding than something that proves functional purity. Seems that it is enough.

How did they ensure their service benchmark wasn’t I/O bound?

Lately I have been more interested in rust. Similar trade offs but “fearless concurrency”, access to low level APIs, and no garbage collector sold it for me.

I love how stubbornly plain and lacking in magic Go is. There is almost no hidden control flow, there's no aspect oriented programming, or operator overloading, etc., etc. - pretty much everything the code actually does is in front of you.

I think this means you have a bigger upfront cost in writing code the first time, with the benefit of easier debugging and easy maintenance later (which is a tradeoff I'm very happy to take).

TBH go does have a lot of weird magic stuff.

For example:

- types vs type aliases

- the init function

- how you define a struct can have significant impact on performance. There are lints that align struct better.

- changing a returned value in a defer function. Very useful but can easily hide complexity.

- interfaces are.. weird. They can also easily hide weirdness.

Build directives are magic comments. Embed is a magic comment but you also need to import it. Weird magic syntax to declare enums. Etc.

I was not very explicit on what magic means to me (so fair do's). By magic I mean logic of a program which is not plain function calls, conditionals, etc. or unexpected/hidden logic.

In that light, I agree init calls and defers are a bit magic. I'll even add panics in there. But I don't use these much personally, so I'm hesitant to say it is a lot of magic - especially compared to something like Java for example.

Discussed at the time:

Choosing Go at American Express - https://news.ycombinator.com/item?id=21637916 - Nov 2019 (15 comments)

not to be the “but rust” person, but I’m curious why Rust wasn’t in the test group. it seems to meet all their criteria, though the community was a bit smaller back then.

Go is a perfectly reasonable choice though, and easier to learn on average.

https://news.ycombinator.com/item?id=21638455

Python is not in the list also, despite them using nodejs.

But when I read the following:

   Based on these criteria, we narrowed our testing languages to C++17, Java, Node.js, and Go. The first three languages were already in use at American Express, whereas Go was not.

What I understand is that they did not really wanted to do a real benchmark but they decided to use Go first, and then they had to setup a kind of "bullshit neutral study" to pretend to leadership that Go was the best choice for all the languages that could fit the job.

Maybe it is the case in the end for them, but it is another subject.

My understanding is that JS running on Node is insanely fast compared to most other interpreted languages. This is because of how much work has been put into the V8 engine, especially the JIT compiler.

Based on this, it’s fair to put Node into a league above languages like Python and Ruby when considering performance.

This might have played a role in their decision to include Node.

Rust crypto libraries were definitely not ready in 2019. I think the story is a bit better today.

After using both java and go, I prefer go. The tooling is simpler and more intuitive, the feature set is less bloated (fewer useless things reinventing the wheel like streams api), it also isn't affiliated with Oracle

In general java feels more enshittified in comparison

As someone who's got a lot of Java experience but almost no Go experience, I worry that I've been coddled with a humongous standard library for pretty much everything, from collections of every sort to handling of time, units, strings, I/O buffers, and pretty much every data structure I might need. Do you find that the Go ecosystem is rich enough that you can find whatever you require?

Yes I had the exact same question. Philosophically I like Go better but I worry in practice that Java has me spoiled

The Go ecosystem has everything one needs. The cryptography stuff is nicer than any Java library (sorry BouncyCastle), first class tls everywhere, very decent http client/server support and tons community routers, third-party collections are easy to come by, ssh in standard library, any package from the internet can be directly pulled as a dependency, packages can be renamed without the need of shadowing, kafka clients, any cloud sdk, everything what one can imagine.

I was a heavy java snd scala user before 2018 and 5 years later I’m not missing anything. For what it’s worth, life got easier when I moved to Go.

Go introduced generics as of recently therefore you won’t find many of the more sophisticated collection types like trees and sorted lists etc. However there is an effort to include the usual suspects in the standard lib.

> Do you find that the Go ecosystem is rich enough that you can find whatever you require?

For what I require (which is atm concurrent web services, restful and grpc APIs, occasional CLIs, k8s tools), it's on par with java

For collections, I don't think go has the same variety as java built in but there are 3rd party implementations for more niche data structures like rb trees. Bare essentials like arraylists and hashmaps come built in

In general, yeah ecosystem is more than good enough for me. Only big thing that I can think of that's missing is gui stuff like swing in java

In my experience, in most cases you realize you’re imminent to reinvent a wheel, you start sifting through the packages available and check GitHub, see what one or another package is up to, then I check whether the functionality that I need is well tested, in many cases it is not, sometimes you get perplexed skimming through the issues, there are few uniformly good packages in many cases, you always see something popping up that’s been smoothed out in a Java pendent long ago. It’s tricky.

With JVM bound projects I’ve rarely felt compelled to actually examine the rather foundational packages. With Go, however, I’m often surprised, even when looking at the more popular packages or framework, kind of a different attitude, you know. Same goes to the Go runtime system behavior. The deeper you look the more you can see, I guess. There’s always a caveat somewhere around the corner.

That said, I do have a few Go systems in production. I love the lower memory footprint though it often comes at a cost of unnecessary CPU thrashing, which for the IO bound services I consider acceptable. I see some companies move to Go, the sentiment I’ve perceived is mixed.

As someone who has been through Java->Go, get ready to write a LOT of for loops. You will need to use your IDE abbreviations a lot. Especially for "if err != nil". No exceptions nor result types. Go is a LOT more verbose than Java when compared at the whole file level thanks to these missing features.

Go has no comfortable streams or standardized functional interfaces and generics is still in the "baby" phase. Though to be fair, Java generics is also merely toddler level, compared to C++/Rust.

To be honest, I like the latest Java releases better than Go. Go had an advantage over Java thanks to in-built virtual concurrency, but that advantage has been lost with Java 21 virtual threads.

Go still retains the advantages of a stream-lined yet full-featured and cohesive standard library, fast compilation and crisp, native executables. (However, for the last bit, GraalVM has been rapidly improving for Java)

The dependency management story (and noted by the author) is one where I wish people would just steal Java's approach.

1) Every package is namespaced 2) Namespaces are DNS names that you can prove you own

It eliminates typosquatting, and the drama of "someone created a bunch of helloworld crates just to grab the good package names"

I think Go's approach is similar - if you have a public repo, then whatever the DNS name is gives you a unique module name. For example, both of these modules can exist, and I can import both into a given Go project:

github.com/rs/zerolog

mysite.com/myforks/zerolog

“foo.int” is a valid domain but is not possible to use as a Java namespace. What do you do when your company gets bought or changes names? (com.sun.whatever vs. com.oracle.whatever)

I think that requirement applies to the maven central GroupID, which may or may not be related to the Java package name.

Yeah, not a valid Java package name, but can still be used as a group id for your artifact although it'd feel a tad odd.

Sorry, I realise I've been overloading package to mean "dependency you download" (I'll stick to artifact now) when package has an existing meaning in Java, and I'm muddying the waters a tad there.

There's no requirement for Java package names in an artifact to match the group id. E.g.,

> my-domain.com -> com.my-domain (NOTE: The groupId should reverse the domain name exactly, even if the domain name contains hyphens or other characters that would result in an invalid Java package name. Hyphens are perfectly acceptable in groupIds, and you would not need to change your Java package name to match it)

https://central.sonatype.org/publish/requirements/coordinate...

> com.sun.whatever

Oracle owns sun.com and runs artifacts that use that group id prefix, so yeah, when you get bought or change names, you can just keep the old name up and running. If you can't, or don't want to, I have seen some projects change group id, and there's tooling for artifact publishers to do so in a way that doesn't break end users, it's a bit clunky, but doesn't tend to occur that often. https://maven.apache.org/guides/mini/guide-relocation.html

Having artifact group id tied to DNS names isn't super-duper flexible, but it offers some assurance that the dependency you're introducing is published by the people you think, so attackers can't easily steal crypto wallet info by publishing pandass pndas pandsa to catch people who typoed pandas etc. etc.

I was a Java dev for many years and I vastly prefer Go. A lot of it for me has to do with the inversion of control libraries providing "fake" simplicity, but becoming a nightmare when you have to try and figure out what thing in the background is know causing a bean conflict.

The extremely straight control flow of Go makes it much easier to maintain in the long term IMO.

Part of maintainability is readability and ease of debugging, and I agree that Go shines here. But another part is the ease of refactoring and rewriting. My experience is that Go codebases which aren't fundamentally doing very much nevertheless become ossified through sheer volume. This is especially the case when you have layered packages (MVC, etc) and consequently a lot of mocks. Most conceivable changes implicate dozens to hundreds of mock expectation lines, and thus become more trouble than they're worth. This is largely because fully explicit control flow needs to be fully explicitly unit tested every single time. Those tests weigh a lot.

> Most conceivable changes implicate dozens to hundreds of mock expectation lines, and thus become more trouble than they're worth.

I've experienced the same thing in java, and similarly with testing different conditionals/exception cases in java

Usually most of the mock annoyingness I've personally dealt with is from repetitive, unrefactored test coverage

Does go have something similar to spring, or is there a framework baked into the language

You can write service structs which depend on service structs similar to spring, and there are libraries which can generate the code that does the wiring for you.

I've always wired my stuff manually in Go though, which is useful if I want to optionally load different subsystems because of some config, etc.

Curious what other languages were evaluated.

From [0] "Based on these criteria, we narrowed our testing languages to C++17, Java, Node.js, and Go. The first three languages were already in use at American Express, whereas Go was not."

[0]: https://americanexpress.io/choosing-go/

Reply on news.ycombinator.com