777 karma · joined December 2, 2024
I was curious about this so I skimmed the paper [0]:
> SalesRLAgent achieved 96.7% accuracy, outperforming the best commercial alternative by 23.7 percentage points and the best LLM approach by 34.7 percentage points.
For a fuzzy natural language task like this, this magnitude of improvement should already set off alarm bells (Though i admit I'm not even sure what accuracy is even measured here, and the paper doesn't help either). Also, "best LLM" here refers to GPT-4 (at the time of upload, the public already had access to GPT-o3 and). I would have loved to contextualize the performance by looking at model size, but the paper is frustratingly devoid of detail in that regard:
> The core of SalesRLAgent is a reinforcement learning architecture consisting of: • A state encoder network that processes Azure OpenAI embeddings and features • A policy network that estimates conversion probability based on the current state • A value network that estimates the expected cumulative reward • A meta-learning module that assesses prediction confi dence
Also:
> Beyond technical metrics, we evaluated SalesRLAgent in real-world sales environments through A/B testing. [...] After 90 days across 217 representatives and 12,433 con versations, we observed: • 43.2% increase in conversion rate for the test group
This would be a pretty huge result but the fact that this is just shoved into a single paragrpah with no further discussion on methodology, baselines and setup makes me very suspicious.
Also I feel like the obvious way to read the very first sentence is that GLM is a language model
> As we develop GLM, the model sometimes exhibits capabilities that surprise us
> The [k-server] problem’s definition is simple: There are k servers located at points of a metric space. At each time step, a request arrives at a point of the metric space. An online algorithm must serve the request immediately by moving a server to the requested location, without knowledge of future requests. The goal is to minimize the total distance traveled by servers.
> The k-server conjecture states that a deterministic online algorithm can achieve competitive ratio k on every metric space.
I only had to look up what "competitive" means in this context, and wikipedia [0] had this to say about it:
> An algorithm is competitive if its competitive ratio—the ratio between its performance and the offline algorithm's performance—is bounded.
The ratio by which this performance is bounded for a k-competitive algorithm is k (plus some constant) [1]. We can consider the analogy of k support technicians ("servers) located in different (physical) locations ("in metric space"): The conjecture/theorem states that in any metric space (Not necessarily two- or three-dimensional), there exists an online algorithm that results in travelled distances of no more than roughly k times that of the optimal distance if all requests were known in advance.
[0] https://en.wikipedia.org/wiki/Competitive_analysis_(online_a...
[1] https://www14.in.tum.de/personen/albers/papers/brics.pdf Section 1.1
which links to: https://archive.org/details/s9notesqueries03londuoft/page/12...
which is in reference to the original proquiritations here: https://archive.org/details/worksofsirthomas00mait/page/416/...
i had also never heard of this before today and wonder if people had even seriously tried to decipher this at all?
For false negatives, that depends on the type of article at hand. If an error leads to me spending 2 minutes reading an article that ended up being not worth reading I'd be fine with an order of magnitude more than if false positives happen on articles that take 30 minutes to go through. Maybe on the order of 1-3% overall?
Not quite but close enough :D https://www.youtube.com/watch?v=DWzWwx8T3nE
Are there any regular typing keyboards with velocity sensitive keys? I genuinely don't know, it's just the first thing I'd compare this to
> The first issue I have with it is that it uses a logarithmic scale on the cost axis. Using a log scale is the only way to make you spot the difference between a model that costs $0.015 per task and one that costs $0.032, while the same plot contains a model that costs $3.69 — almost 250 times as expensive. However, the net result is that the viewers can no longer appreciate the immensity of the price difference between the cheap models and the heavy ones; nor can they realize how inconsequential the price differences are between the cheap models.
This is an asinine complaint, and nobody can seriously tell me that the last plot on their page [0] is more readable than the AA one [1]. If I'm using a model at the lower range of the cost scale for whatever list of tasks, and i switch to another model at the lower end of the cost scale, my spending might double anyways! This should be reflected in the plot, and linear scale doesn't do it justice.
It's also much easier to see the mentioned pareto frontier in the log plot than in the linear one.
I can see why they disagree with the pricing determination for open/local models, but I don't think there is one clear right way to do it. So how do they do it instead?
>Hardware is priced at zero, on the basis that both an RTX 3090 PC and a 64GB Strix Halo are desirable gaming/work machines anyways.
...oh
Would have been nice to mention explicitly how the pareto frontier changes with those new calculations.
[0] https://openteams.com/wp-content/uploads/2026/09/all_models-... [1] https://artificialanalysis.ai/#intelligence-comparison-tabs
Can't wait to have an ion trap in my living room!
I would love to appreciate the work that is going into this curriculum, but when the lesson overview is overwhelming amounts of AI copy slop I just can't bring myself to be interested.
Edit: i'm talking about the text, not the pictures. It's not overwhelming in the sense that it is a lot of text, but in the sense that the amount of Claude-isms in relation to both word count and actual content is way too high for me to care about. Take lesson 9 for example:
9. Where the quantum lives
every page is a classical drawing
the animation is impossible
each step has a classical model; they refuse to glue
one story per step, no story overall
where the quantum actually lives
and the gap has a price, in coherent memory
I have no idea what is going on here or what the topic of this lesson is, and the same is true for like half of the lesson overviews here.
Like i said, i'd love to be able to judge the content, but there is none here.I think you might be victimizing yourself a bit much there
This is wrong and you are misunderstanding what collinearity means. You could have a set of points where all but one are on the same line, and the set of a whole will be not collinear, while we can obviously draw a line that passes through more than two of them.
A different way of stating the theorem is that any finite set of points has either a line passing through all points (i.e. the set is collinear) or there exists a line that passes through exactly two points. This dichotomy (why two and not three? Why can't we construct a set where any line passes through at least three points?) is not immediately obvious.
The theorem doesn't presume that no three points are collinear, it presumes that the set as a whole isn't collinear, which is a much weaker statement.
I think you meant premise?