Good to know how much you're "looking" closely after everything your customers are doing. The name of your company is in your profile, right?
For instance, one customer has 15(!) cronjobs. And they all used LLM calls to do stuff. So they ran out of tokens super quick and asked us for help because they didn' tknow what to do. Its not an engineering company, the owner has a sales background. So we went in and wrote scripts for them so that the cron calls don't need to use the LLM anymore.
How do you then ensure it’s fixed for customers in the future? Do you add instructions to agents to write scripts where possible?
Another question I have, is from a product perspective, how do you decide how much of the internals you show to users (like token usage).
I always thought that seems like (from a non technical users POV), like a implementation detail; and quite hard to map to actual usage. For example, one request might use lots of tokens, and another will barely use any; but it can be difficult for a non technical user to undertand the difference.
I always assumed most SeriousBusiness companies over a certain size, not run by 20 year olds, operate similarly.
Also, this idea that it's bad or weird to anonymously discuss customers is strange. I've known cops, doctors, dentists, and all sorts of other people who routinely share stories about weird things that happen with people and customers at work, anonymously. This is the norm in the world and always has been.
It's also the norm for the companies who own your data to look at it, when asked to. What do you think customer support even is?
It's only really a small corner of the tech world that treats these things as strange and sacrilege.
Sounds like your software might require HIPAA compliance in which case everything that you just said makes sense and is likely required in that context. But for the average SaaS that doesn't need HIPAA compliance it doesn't work like that.
And it's not particularly unusual to discuss vague details of your customers anonymously. Even in the medical field, with very strict privacy rules, doctors will often publicly discuss encounters they have had with patients, and they only cross the line when they identify the patient.
I mean I'm all for privacy and security but this statement right here just comes across as pretty paranoid
We are giving an unprecedented amount of personal data to these tools at the same time that the technology is being leveraged against people at an unprecedented scope and scale. No US legal framework exists to protect you against this. You should take this risk seriously.
But they didn't ask US for help. We didn't need to know a customer of yours is running 15 cronjobs. Even if this factoid itself may be harmless, it's a good thing to practice on how you talk about these things. I'm not saying you're doing anything wrong, I don't know that, but I totally get why how you talk about it rubs some people (including me) the wrong way.
> Its not an engineering company, the owner has a sales background.
Again, it's good if you have a close relationship with your customers, but we don't, and they don't have one with us.