adhocpotato

Thinking about this some more: I tend to be a 'scratch your own itch' type person, and I haven't had a place where I needed this elsewhere. I think it would work pretty well think anything that was wrapping a series of commands with a specific tool. The first thing that came to mind was docker. I could put together an example for how to do something simple in docker, maybe following the steps in https://docs.docker.com/get-started/02_our_app/. If that sounds interesting, let me know.

The only place I've used it so far is for Ziti stuff. I did add a (really tiny) example to the project itself to show how to structure it: https://github.com/openziti/runzmd/tree/main/example

That was one reason I left the ziti actions in the project, as I thought they'd be useful as real life examples for someone writing their own actions.

Second in a series about Go patterns used in the open source project (https://github.com/openziti/ziti) I work on for new devs joining the project. The first one covered channels, this one talks about things learned integrating generics. It's focused on things specific to the Go implementation of generics. Since it's a bit early days still for Go generics I'm curious if people have hit other issues or have found better workarounds for the issues we hit.

I'll add some more detail. There are a lot of different ways your application can break as it scales up. From something which is handles data flow, the three I tend to think about the most are: the model, throughput and number of connected clients.

We've tested the model with 100k identities with the operations that clients use: auth, listing services, creating sessions, etc, to make sure that the model scales reasonable well. We had to add additional denormalization and make some other tweaks, but now the controller scales relatively well for those operations. I'm sure there are still edge cases where it may break down, but we'll have to fix them if someone hits them.

We've done thoughput testing to make sure we can handle high throughput use cases. This also resulted in lots of changes, including reworking the end-to-end flow control. This is an area where we're happy with the progress we've made, and performance is in a reasonably good place, but where we have lots of ideas of how to continue to improve and will be continuing to test and iterate.

Having tons of connected clients (even without much traffic flowing) is it's own scaling challenge. We've done some amount of testing here. As part of the testing mentioned above we had to make some changes to make sure that slow clients didn't hold up fast clients. More generally,this is where things start getting complicated and very specific. You can have very different types of traffic flow, so it's hard to model anything generic. We have not done as much in this area because we've not seen any cases where we're memory constrained, which is the usual sign of a concurrent connections scaling bottleneck.

Hope that's helpful.

That's an interesting application. I tend to use an IDE if I'm debugging in realtime and otherwise I'm getting stackdumps from the support org.

For CLI goroutine analysis there's another tool: https://github.com/linuxerwang/goroutine-inspect which is pretty good.

I think if I do a rewrite it might make more sense to go the web route, like the go tool pprof UIs.

I did poke around for a bit, but I really needed the tool, and I knew it would be days of work if I learned a new tool and hours if I stuck with what I knew.

My coworker Andrew did start a similar tool, which is here: https://github.com/andrewpmartinez/grid and uses https://github.com/AllenDang/giu for the UI.

Next time I need a feature I probably take a look at that and see what the model looks like and and how straightforward it is to work with.

I wrote a tool to help dig into goroutine stack dumps from customer deployments. One of the things the open source project (https://github.com/openziti/ziti) I work on does is mesh networking. This has been very helpful for looking at goroutines from multiple processes, debugging distributed systems issues.

It can:

  * Load multiple stackdumps
  * Group similar goroutines automatically or manually
  * Group goroutines by regular expression

It is written in Java, because I needed a tool and I've done Java UIs in the past. Since I do Go backend development I figured I'd spend my time getting the tool written.

We also use a Go agent library (https://github.com/openziti/agent) which helps us gather stack dumps (and pprof dumps, etc) at runtime.

Some doc I'm putting together about go patterns in the open source project I work on to help new devs get up to speed. Thought it might be interesting for others so I've blogified it. First one one is focused on go channels.