209 karma · joined February 10, 2015
@TomGranot pretty much everywhere else except here.
I'm a technical product marketer, this is delightful and saves me so much time.
Wrong audience for the tool, since they focus on things like accessibility and browser support and what you promise to bring to the table in comparison to what you actually bring.
None of these matter - this is a TERRIFIC piece of tooling and I will use it starting literally next week for my work.
We've built what we call an Unstructured Data Transformation Layer. Think about it like an assembly line from raw text to tables in your data warehouse.
We don't use Llamaindex, we have our own (proprietary) piece of tech that does this. We can and have been outputting gold-standard tables on top of a lot of different types of context (legal docs, call transcripts, customer reviews, chat logs, emails, blog posts, social media posts, etc...) and looking to expand to more interesting domains soon.
If anyone wants to hear more hit me up at tom [at] flexor [dot] ai (does this still work or are scrapers smart enough nowadays to just grep for this too lol)
Hit me up at writing@granot.dev :)
Just use the latest & greatest open-source library, no?
Where do I see what permissions I just granted from my (naturally, dummy) gmail account? What happens next time I want to login?
Quick and dirty summaries of interesting stuff I don’t really dive into. Guy’s clearly a talented dev and has a knack for drawing diagrams that make sense
This is not monkey-patching or hot-swapping, but a different approach - we call it dynamic instrumentation. This amounts to having an agent (an SDK/library, in essence) perform the addition of Lightrun Logs, Snapshots and Metrics, then (potentially) pipe the information where it needs to go. We've got a nice diagram here [0].
I think that the mechanism itself - which changes by runtime, naturally - is well explained in the link above. However, the core security mechanisms we have are enabled by another component of the agent we dub the Sandbox.
In essence, we've got a (patented) way to verify everything that we do at runtime. That means we ensure that each evaluated expression does not have side effects (like changing the value of a member of an array, or editing a variable value), every metric we could think of is throttled and rate-limited (that includes our usage of CPU, RAM, the rate of I/O and a bunch of other things).
Given this sandbox mechanism, and the way the networking requirements look like (again, look at [0] - no need to open a debug port / inbound ports, and a pretty agnostic deployment model) I think we've got a pretty robust defense layer against a variety of failures. Also see my comment a few comments above regarding data security.
Regarding profiling - we're not actively a profiler, but we do offer a set of code-level metrics you can use to do performance analysis and detect bottlenecks [1].
Regarding adoption - we're doing OK:) I think this is a new approach, one that is quite different than what developers are used to. But it's also one of those things that - as you mentioned - make so much more sense than the alternatives, and with costs rising it's really even more evident than ever. If you're interested in the cost of logging specifically we've got it broken down in a study [2] we recently released.
[0] https://docs.lightrun.com/data-security/
[1] https://docs.lightrun.com/metrics/
[2] https://lightrun.com/resources/lightruns-economic-impact-on-...
If you've got a bunch of logs labelled as INFO, WARN and ERROR in your system, and you tweak the debug level - you get more detailed debug logs as need be. This will stream EXISTING logs to stdout and then to your favorite APM.
However, if the exact bit of information you wanted isn't there (that variable isn't logged, that specific piece of code isn't instrumented so you can't know it was reached, etc...) you're stuck.
In addition, sometimes (often) it's hard to correlate the exact path the code took, since it's not clear which condition or class or package or server were actually involved in the process of execution. This tool enables you to conditionally log just what you need in real time - basically "paint a path" through the code at runtime.
Hope that's clearer, can elaborate more if need be.
We've also got a zero-config (in-browser!) version of the whole thing [2], using code-server [3].
[1] https://github.com/lightrun-platform/lightrun/tree/main/exam...
Logging is a surprisingly pricey bit of observability at scale.
If you're doing anything highly-transactional you're pretty much guaranteed nowadays to get major observability bills (for things like ingestion, transmission, storage and analysis/querying).
It also creeps up on you - you're used to getting billed incessantly for cloud stuff, monitoring doesn't sound too pricey. Until, well, it is.
Can't help but point out that this is static logging, meaning logging added during development. This stems from an the approach colloquially referred to as "log everything, analyze later", rather than more disciplined, case-specific logging.
We're used to add "just-in-case-logs" to ensure that we've got ourselves covered when the sh*t hits the fan, but we rarely look at them.
An alternative approach would be using a tool, like the one Lightrun[0] builds (see disclaimer above) to add Logs in real-time to applications.
This means that instead of logging IN ADVANCE (i.e. during development) you can add logs when and where you need them, in real-time. The tool works using a variety of dynamic instrumentation techniques (depends on the runtime), and currently supports Java, Node.js & Python (.NET soon).
It can also pipe these logs into Loki, Elastic or SigNoz since they can be dumped like normal logs, right into the stdout.
In any case, we're seeing mass reduction in static logging (up to 60% in volume, 40% in cost) by going dynamic instead of mostly static. You'll always have to log some stuff for forensics and traceability, but dynamic instrumentation removes a lot of the dead weight.
And then I read your first sentence - why is it always the assumption that people are going the hyperbole route? This is a well-upvoted link, with enough comments to merit a good discussion, and what appears to be a nicely-thought out post after the link.
Also, I've been following Snir for quite a while - dude wrote a massive Data Engineering roadmap which is heavily used by people learning how to ETL and has a very interesting investing blog. He's an engineer turned investor, and looks at the market from this perspective.
The derogatory quotes are unnecessary.
Dude's got a screenshot. I doubt he faked it for the attention - he probably actually got that bill.
He feels it's expensive and he should pay less for the value he's getting considering his alternatives (Azure VMs in a 1/3 of the price).
Perhaps refer to the actual price of a billed hour there? That feels like the core theme here.
I tried finding you on twitter but no go since DMs are closed.
Would be happy to pick your brain about the topic - tom@granot.dev is where I’m at if you have the time!
We're more after ongoing situations, where the issue is either hard to reproduce locally or requires very specific state - APIs returning wrong data, vague API 500 errors, application transactions issues, misbehaving caches, 3rd party library errors - that kind of stuff.
If you're looking at the app and your approach would normally be to add another hotfix with logging because some specific piece of information is missing, this approach works beautifully.
There is another way to tackle the problem for most normal, back-end applications: Dynamic Logging[0].
Instead of adding a large of amount of logs during development (and then having to deal with compressing and transforming them later) one can instead choose to only add the logs required at runtime.
This is a workflow shift, and as such should be handled with care. But for the majority of logs used for troubleshooting, it's actually a saner approach: Don't make a priori assumptions about what you might need in production, then try and "massage" the right parts out of it when the problem rears its head.
Instead, when facing an issue, add logs where and when you need them to almost "surgically" only get the bits you want. This way, logging cost reduction happens naturally - because you're never writing many of the logs to begin with.
Note: we're not talking about removing logs needed for compliance, forensics or other regulatory reasons here, of course. We're talking about those logs that are used by developers to better understand what's going on inside the application: the "print this variable" or "show this user's state" or "show me which path the execution took" type logs, the ones you look at once and then forget about (while their costs piles on and on).
We call this workflow "Dynamic Logging", and have a fully-featured version of the product available for use at the website with up to 3 live instances.
On a personal - albeit obviously biased - note, I was an SRE before I joined the company, and saw an early demo of the product. I remember uttering a very verbal f-word during the demonstration, and thinking that I want me one of these nice little IDE thingies this company makes. It's a different way to think about logging - I'll give you that - but it makes a world of sense to me.
Funnily enough, this post coincides with the release of a newsletter issue[0] by a friend of mine - unzip.dev - about lattice-based cryptography.
A bit of a shameless plug, but it really is a great bit of intro for noobs in the area like myself.
100% would recommend.
First, a disclaimer: I have no idea how much Rockstar employees are paid, nor how their workdays look like. I don't know what their team sizes are, where they worked before, or who manages who and in what fashion. I actually don't know anything about the Rockstar engineering organization at all.
I am also not a GTA player (or, more accurately, haven't been since San Andreas came out many moons ago). This is my perspective as someone who has worked for various organizations in his life ( SWE-centered ones but also other, more "traditional" ones too).
We're all familiar with technical debt - it's a well established concept by now. Reducing it is part of the normal workload for well-functioning organizations, and (good) tech leads think about it often.
What isn't talked about as often is the "organizational" debt. Some things are "verboten" - you don't touch them, you don't talk about them, and you run like hell if you're ever assigned to deal with them.
Every large enough company has (at least) one of those. It might be a critical service written in a language nobody in the team knows anymore. Maybe it's a central piece of infra that somebody wrote and then left, and it's seen patch after patch ever since. These are things that end your career at the company if the BLAME points to you after you attempted to fix them.
I have a gut feeling - not based on anything concrete, as I mentioned - that the loading part for GTA Online might be one of those things. If someone breaks the process that loads the game - that's a big deal. No one would be able to play. Money would be lost, and business-folk will come knocking.
So sure, there might be some mitigations in place - if some part fails to load they allow the game to load anyways, and then attempt to fix it mid-fly. It's not black and white. But it feels like one of those things, and so people might have just been running like hell from it for years and years. Teams change. Projects change hands. People leave, more people join. It's life in the industry.
I would be REALLY interested in learning how software orgs deal with these types of behemoths in their projects. I have yet to find someone who knows how to - methodically and repetitively - break these apart when they appear.