cfors

More engineering effort going into reinventing things that already exist to upsell people on Grafana cloud.

What about focusing on the core value that Grafana provides, dashboards?

Grafana 8 alerting is still in my opinion at a beta level. Dashboards as code has made no meaningful progress outside of community attempts in the past 3 years. The documentation for Grafana 8 alerts is still subpar.

All of these things as a paid offering are more interesting than migrating my logging system or metrics system. Developers don't want to migrate their observability.

This will no doubt be overshadowed by Generics, but Workspaces[0] are going to be incredibly useful for companies using a monorepo with shared packages.

Once again, great job to the Go team!

[0] https://go.dev/doc/tutorial/workspaces

Agree. I really can only imagine this being useful for an organization that publishes dozens of different libraries that a "product team" needs to use such as logging, database access, secret managers, etc. At that point, speaking the common language of a DI framework might be easier.

For example, rather than have to read the GoDoc for every single constructor i'm supposed to use, I can just see how to use my DI framework to set up this client library and move on with my life.

That being said, DI always strikes me as a layer of abstraction that I don't need in my day to day. But I don't work somewhere at Google/Uber scale so YMMV.

Performance is relative, so I'm not really sure the point being made here. Sure if your program has regex as the constrained resource this matters, but again it's all relative.

For one, I'm not convinced that the Grafana 8 Alerting API Swagger docs are up to date or ready for the public [0].

I've literally copied an alert's json format, and then tried to post it back and never got it to work.

Here's an example from my bash history:

> curl -X POST -H "Authorization: Bearer $GRAFANA_API_KEY" -H "accept: application/json" -d @rule.json some_endpoint/api/ruler/grafana/api/v1/rules/test1

I spent a solid day trying to play around with this to get it to work. Because of this the alerts are impossible to code review or store in a git source. Which stinks because Grafana's datasource API's would be amazing to use for alerting. But they're either unusable because anybody can change them or the administrator could bork them at any given point (which has happened before), or just undocumented to the point where they are useless.

That's not even to begin on dealing with the "big blob of json" problem [1] that was clearly important enough to be given an entire spot at GrafanaCon, but even Grafonnet is not supported with Grafana 8. There is apparently some CUE way of doing this, but I can't seem to find any official documentation on that.

Anyways, I've moved back to alertmanager for the time being.

edit: is all of grafana labs downvoting the GP? this is very honest and candid feedback here.

[0]: https://editor.swagger.io/?url=https://raw.githubusercontent...

[1]: https://grafana.com/go/grafanaconline/2021/dashboards-as-cod...

I wish Grafana would stop trying to make offerings that already exist and focus on making their dashboards and alerts as code usable.

I would even pay money for an actual offering that worked.

In today's world of excellent CLI tools I don't think grep is a good choice, especially for checking irregular languages like XML. [0]

I use tools like `jq` [1] or `yq` [2] all the time for CI checks. One useful check, is we have a configuration file stored as several hundred lines of YAML. Its a nice thing to maintain a sorted order for that, so we have a git pre-commit hook that runs the following:

> yq eval --inplace '.my_key|= sort' my_file.yaml

Of course, a pre-commit hook or CI both work. There's pros and cons of both. For our team, the pre-commit hook is a low enough level of effort, and doesn't require a CI check for something that executes in milliseconds.

[0] https://stackoverflow.com/a/1732454

[1] https://github.com/stedolan/jq

[2] https://github.com/mikefarah/yq

Surely, this is the very definition of too many options?

*Disclaimer: Mostly happy user of many of these projects

IBM already purchased red hat for $34b 3 years ago, so no need to wait any longer :)

Take vault for example. Rather than have to toggle a flag in your service to get a secret, you could have the vault sidecar inject the secret automatically into your container, as opposed to having to pass a configuration flag `USE_VAULT` to your application, which will conditionally have a baked in vault client that fetches your secret for you.

Your service doesn't really care where the secret comes from, as long it can use that secret to connect to some database, API or whatever. So IMO it makes your application code a bit cleaner knowing that it doesn't have to worry about where to fetch a secret from.

You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that.

Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of the potential SDK's for teams.

Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that.

Also, there is a decent amount of work being done on gRPC to speak XDS which also removes the need for the sidecar [0].

[0] https://istio.io/latest/blog/2021/proxyless-grpc/

A benefit of Kafka is that it is relatively trivial to spin up multiple consumers from a single CDC stream.

If you have some user attribute table that you set up with Debezium, there are a couple downstream consumers (teams!) that also want this data. So you hook up the table with debezium to Kafka and write all changes to a Kafka log.

A team of data scientists can stream these straight into their model, or a team of data engineers can dump these changes real time into an analytics store for business intelligence, or another team that needs access to this data can also create their own denormalized table of user attributes as well.

For the data producing team, once they get their data into Kafka basically any team can consume from this data which is a lot easier to maintain for the producing team and does not stress the database so its a very nice pattern for bigger orgs.

For the most part, keeping the data in a single postgres instance as long as possible would be my recommended solution. But there is a point where getting the data to consumers of that data becomes tedious enough that this additional infrastructure is worth it.

This is essentially the idea behind CDC (Change Data Capture) [0].

Martin Kleppmann has some great blogs about this as well [1].

[0] https://en.wikipedia.org/wiki/Change_data_capture

[1] https://www.confluent.io/blog/turning-the-database-inside-ou...

Yes containers don't solve for dealing with the mess of third party saas that every company is built around.

But that's why anytime you integrate with one of these tools you should be aware that there is a cost for maintaining that integration.

Its right here from the "Other Testimonies" section:

https://elmtown.simplecast.com/episodes/it-just-goes-on-and-...

Obviously Microsoft is a huge company so they use all of the languages, but I find it surprising given the amount of investment they've made into Typescript.

I don't need to know anything about user123 outside of the fact that they are located in Perth, Australia.

Nothing else matters to me for this, I don't have access to any PII data like email or device (look, I know user ID's can technically be considered PII depending on which infosec person you're talking too).

Is this still a problem?

I understand your metaphors but without knowing that user123, who created a ticket in our system, is in Perth Australia (which for some reason that locality in my own metaphorical example is having issues processing payments) how we're supposed to resolve this.

Maybe I'm just hopelessly optimistic that people aren't as awful as we want them to be, or naive.

Let me be clear here, I'm not advocating for ZERO access control or audit logs here!

Let's take that bank argument, I'm definitely not advocating for not locking vaults or safe deposit boxes. But somebody has access to those, and when they need access they have a process for getting to it. Frankly, it definitely can be abused and banks wouldn't know for the better until after the abuse when the employee would be terminated and taken to court.

That's because they have audit logs in place. And the reason why its part of your contract as an engineer not to abuse your access to customer data.

I think the larger point I'm trying to make here is that its really, really hard to build a system that prevents any type of abuse of data. Now I'm not saying that we shouldn't strive for systems that make it hard to abuse customers data, but bad actors have ways of beating these systems and I have some empathy for a policy that places trust in employees (who need access, by definition of their job!) to not abuse it.

Anyways, these are all good comments made in response to mine. I agree with them!

[EDIT] Okay - I see the incoming point about them not needing access to that data for their job, that's a fair point. But I think most of us have been at a point in our careers where knowing the piece of information about a user that might have gotten them into a certain state is occasionally a valuable debugging tool.

These comment's are all relatively ignorant of the fact that implementing these sorts of privacy controls generally makes your product worse and your engineers miserable.

> Facebook employees were granted user data access in order to “cut away the red tape that slowed down engineers,” the book says.

If we can take a step back, this is a totally reasonable policy. Unfortunately Facebook is facing the reality of the law of large numbers in that once you have 1000+ people the chances of having a bad actor in your system is much higher than 10 people.

Maybe this is a hot take, but I for one prefer that my company trusts me to do the right thing rather than make it hard to do my job. I'm not saying that there isn't a solution for this, but behind the "facebook corporation" there is generally just a bunch of engineers that want to do a good job at work.

Not a perfect solution but I add a prompt signaling both my current namespace and cluster, along with some safeguards for any changes on our production environment. In practice I haven't deployed something wrongfully in production ever.

I use a custom written script but I've used this one in the past - its pretty nice.

https://github.com/jonmosco/kube-ps1/blob/master/kube-ps1.sh

I upvoted, but then I felt like that wasn't enough. There is something about this book that seems more real than life. The author not only was a neurosurgeon, but also had a Master of Arts in Literature from Stanford. He is an absolutely incredible and deep writer.

Couldn't recommend it enough.

What is `rdep`? Couldn't seem to find it with a cursory google search.

> Developers want the durability, stability, and scalability of a SQL database but do not want to be constrained by managing a schema. It has been our goal to give both, not compromising on the power of your datastore but making changes feel as easy as deploying code.

So is this a wrapper around managing Schemas powered by Vitesse? (Btw, had to go to your github to figure that out)

If you want to be the database for developers you should know that developers do care about how you do this scaling.

Hinge | Software Engineer(s!) (Backend) | New York (NYC), Remote until September 2021 | Full-Time

Hinge is the dating app for people who want to get off dating apps. In today’s digital world, singles are so busy matching that they’re not actually connecting, in person, where it counts. Hinge is on a mission to change that. So we built an app that’s designed to be deleted. On Hinge, there are no rules, timers, or games. Instead, you’ll meet your most compatible matches and you’ll have unique conversations over what you’ve shared on your detailed profile. It’s a natural way to find a great first date. Currently, 3 out of 4 first dates lead to second dates, we’re the #1 mobile-first dating app mentioned in the New York Times wedding section, and we’re the fastest growing dating app in the US, UK, Canada, and Australia.

Hinge is looking for software engineers to join the Backend team. As the newest member of the backend team, we are looking for someone to help us design and build the next generation of high-performance APIs and backend services in Go. You will be on a cross-disciplinary team including other backend engineers, mobile engineers, product managers, and designers where you will lead development of new features and improve the quality and reliability of our services.

https://jobs.lever.co/matchgroup/d5d5d665-4715-4e16-9c24-b2e...

Makes me wonder what their partitioning strategy was. I can't imagine a cache that is over 4GB+ that couldn't be partitioned at least a bit further. But good call out, thank you for the correction.

Sounds like Discord was running gigantic in memory Go caches.

Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense.

Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite.

[0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

Using fzf and ctrl+r for fuzzy finding commands is a godsend.

Basically all snippets of code that I ever run are straight from this.

Forget how to exactly reset a committed local file that you want to place into another commit? Not something I do everyday, so may require some Googling if I can't remember exactly.

What I do know, is that I use something with `git reset`.

I ctrl+r and type `git reset` and here is what comes up:

> 9488 git reset HEAD^

> 10535* git reset --hard

> 10555* git reset HEAD file

> 10999* git reset --soft HEAD

Oh right, I need to reset the commits and then remove that file from the commit and I can recommit.

It is a complete gamechanger.

Awesome product, btw. Definitely see where it fits some niches.

Actual concerns about running a real application with real uptime requirements.

1. Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you

2. No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory.

3. Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have.

While I was being sarcastic, I was not being intellectually dishonest. There are entire hosts of problems that you call out you are trying to solve without providing any real solution.

> If you exceed the capacity of a single node, sharding your data can allow you to scale horizontally to multiple nodes.

It's so simple, just don't have a lot of data that doesn't need to interact with other data and your problem is solved!

/s

The README is a treasure trove of information on go internals, thank you for writing all of that out. Fascinating to read what looks like your your thought process as you built this library, I always sort of thought README's should sometimes be a "diary of design decisions"

That being said, I'm not really at a point where json decoding is a bottleneck in any of the code I write, but will keep this in mind. Great job!

Tech companies do this all the time when the cheap servers they run their businesses on can't handle the traffic.

I find it hard to believe that anybody would find it hard to empathize with the fact that your company failed to predict a wild meme craze leading to an overload of your systems.

Now that its said, it would have made more sense for Robinhood to just nuke its servers and return 500s to their users rather than deal with this whole mess.