4,157 karma · joined May 28, 2011
Email: <myusername>@gmail.com
Now I could imagine such an API allowing to request a model from huggingface for example, and caching it long term that way, yes just like LM Studio does. But doing this based on some external resource requesting it, vs you doing it purposefully, has major security implications, not to mention not really getting around the lead time problem you mention whenever a new model is requested.
> The technical facts: - np.column_stack([x, y]): 20.63 µs - np.vstack([x, y]).T: 13.18 µs - 36% faster
Does anyone know if this is even true? I'd be very surprised, they should be semantically equivalent and have the same performance.
In any case, "column_stack" is a clearer way to express the intention of what is happening. I would agree with the maintainer that unless this is a very hot loop (I didn't look into it) the sacrifice of semantic clarity for shaving off 7 microseconds is absolutely not worth it.
That the AI refuses to understand this is really poor, shows a total lack of understanding of what programming is about.
Having to close spurious, automatically-generated PRs that make minor inconsequential changes is just really annoying. It's annoying enough when humans do it, let alone automated agents that have nothing to gain. Having the AI pretend to then be offended is just awful behaviour.
I find it a bit rare that this is the case though. Usually I have to carefully review what it's doing and guide it. Either by specific suggestions, or by specific tests, etc. I treat it as a "code writer" that doesn't necessarily understand the big picture. So I expect it to fuck up, and correcting it feels far less frustrating if you consider it a tool you are driving rather than letting it drive you. It's great when it gets things right but even then it's you that is confirming this.
Maybe a bit different but I think it's worth pointing out how this parallels the state of the job market right now.
It is so hard to get hired, with so many moving and diverse frameworks, libraries, and technologies you are expected to know, that it's almost impossible to keep up and stand out.
The only way to do it is to develop "projects" that demonstrate your abilities in each target domain, and in these days of vibe coding these need to be more than sketches but like full fledged applications that can draw real attention to you, if your lucky get on the front page somewhere.
And with vibe coding it can be done relatively quickly.
So we're in this state of new projects, very impressive looking projects, getting posted every day, all the time, and about 1% of them will see any kind of longevity because the vast majority will be dumped as soon as the author gets a job.
This makes it increasingly difficult to select dependencies for downstream work.
I've often wondered how much it happens in practice. What does the per-token distribution of expert selection actually look like during inference? For example does it act like uniform random variable, or does it stick with the same 2 or 3 experts for 10 tokens in a row? I haven't been able to find much info on this.
Obviously it depends on what model you are talking about, so some kind of survey would be interesting. I'm sure this must but something that the big inference labs are knowledgeable about.
Although, I guess if you are batching things, then even if a subset of experts is selected for a single query, maybe over the batch it appears completely random, that would destroy any efficiency gains. Perhaps it's possible to intelligently batch queries that are "similar" somehow? It's quite an interesting research problem when you think about it.
Come to think of it, how does it work then for the "prompt ingestion" stage, where it likely runs all experts in parallel to generate the KV cache? I guess that would destroy any efficiency gains due to MoE too, so the prompt ingestion and AR generation stages will have quite different execution profiles.
I get that, but I guess what I'm asking is, why does it matter what you did?
The result is working, documented source code, which seems to me to be the important part. What value does keeping the prompt have?
I'm not trying to needle, I just don't see it.
Another thought, just occurred when thinking about readmes and coding LLMs: obviously this model wouldn't have any coding knowledge, but I wonder if it could be possible to combine this somehow with a modern LLM in such a way that it does have coding knowledge, but it renders out all the text in the style / knowledge level of the 1800's model.
Offhand I can't think of a non-fine-tuning trick that would achieve this. I'm thinking back to how the old style transfer models used to work, where they would swap layers between models to get different stylistic effects applied. I don't know if that's doable with an LLM.
Basically in some ways it was a superior idea: benefit from the optimizations we are already doing for JS, but define a subset that is a good compilation target and for which we know the JS VM already performs pretty optimally. So apart from defining the subset there is no extra work to do. On the other hand I'm sure there are JS limitations that you inherit. And probably your "binaries" are a bit larger than WASM. (But, I would guess, highly compressible.)
I guess the good news is that you can still use this approach. Just that no one does, because WASM stole the thunder. Again, not sure if this is a good or bad thing, but interesting to think about... for instance, whether we could have gotten to the current state much faster by just fully adopting asm.js instead of diverting resources into a new runtime.
https://arxiv.org/abs/2407.01392
of course it doesn't redraw the image on every step, so not exactly what you're suggesting (interesting idea btw) but i think it's relevant.
ok of course the human reviewers could still use AI here but then so could the authors, ad infinitum..
In my case I was until recently working on TTS and this was a huge barrier for us. We used all the common signal quality and MOS-simulation models that judged so called "naturalness" and "expressiveness" etc. But we found that none of these really helped us much in deciding when one model was better than another, or when a model was "good enough" for release. Our internal evaluations correlated poorly with them, and we even disagreed quite a bit within the team on the quality of output. This made hyperparameter tuning as well as commercial planning extremely difficult and we suffered greatly for it. (Notice my use of past tense here..)
Having good metrics is just really key and I'm now at the point where I'd go as far as to say that if good metrics don't exist it's almost not even worth working on something. (Almost.)
Isn't that basically what the sigmoid operator does? Or more in the direction of averaging many logical computations, we have random forests.
I know that non-fiction would be "good for me." Particularly reading more in topics I'm less knowledgable about, like finance and business and politics. Personal growth. However, I do find that fiction helps expand my perspective and even, somehow, knowledge, but it's different from non-fiction, less direct. I don't read for that, explicitly, although I do like the effect. But I read because.. I guess, because it's nice for my brain to be somewhere else. I don't know. But non-fiction has never done it for me.. my mind just gets.. bored, I think, trying to absorb what someone else wants me to know. Even when I find the topic interesting.
I guess there are people who like non-fiction and people who like fiction and they often cross-over but I think most people lean one way or the other. I can see there being positives and negatives to either side. People who equally read both must be rare? Or maybe it's just my impression.
Is it mainly post-training that causes this behaviour? They seem to do it for everything, like they are really biased towards super verbose output these days. Maybe something to do with reasoning models being trained for longer output?
Remote: Yes (Remote or Hybrid, EU Timezone)
Willing to relocate: No
Technologies: [Deep Learning] PyTorch, TensorFlow, MLFlow; [Languages] Python, C, C++; [Infrastructure] AWS, Docker, PostgreSQL, DynamoDB.
I'm a Senior Machine Learning Engineer with 10+ years of R&D. My core expertise is Deep Learning Model Development and Pipeline Engineering, taking specialized models from concept to reliable output.
My recent work spans Computer Vision (traffic scenario analysis, SLAM, skin deformation analysis) and Generative Audio (speech synthesis focused on naturalness, novel voice generation, and controllability/editability).
I understand the full ML lifecycle, from novel research to scalable, cloud-ready API deployments. Seeking hands-on Senior-level roles and Lead positions to drive innovative model development. As Head of ML R&D (3 years), work included model development, AWS deployment, and rapid prototyping of LLM/GenAI applications for demos, all very hands-on along with a team of 10.
Background: PhD & Master's in Music Technology and Audio-Haptic Robotics (McGill).
CV and more details: https://sinclairs.gitlab.io/cv/sinclair_cv2025.pdf
Email: stephen.sinclair [..at ..] nonnegativ.com