I was also quite surprised to see that. Considering the effort Aleph Alpha put into to their new model, it seems like the Qwen team needs access to vast amounts of german data o.0
I'm really glad for these efforts for open models from within Europe.
302 karma · joined February 28, 2019
From my understanding distillation pretty much copies behaviour. If someone intends to distill from a Model, it can't extract unsafe behaviour, but would learn the same safety mechanisms, no?
These parts here somehow trigger me:
- Enter TorchTPU. As an engineering team, our mandate was to build a stack that leads with usability, portability, and excellent performance.
- Engineering the TorchTPU Stack: The Technical Reality
- Eager First: Flexibility Without Compromise
- The breakthrough, however, is our fused eager mode.
- The Road Ahead: 2026 and Beyond
I have mixed feelings about this. On one hand, we all seem to be using the same tools and converging to the same style. On the other hand, if we all use the same models with the same system prompts, we might lose a lot of creativity and diversity in online content.
Do not get me wrong. I use AI all day to speed things up. But I believe that there is only a small group, maybe 5 percent or less, that actually knows how to use AI properly (I'd count myself not yet in that 5%), which I see as potentially dangerous. The other issue I see is inexperienced software engineers writing software. Although I see this as a great value add and productivity boost for prototyping, I am afraid of the “I do not know much about coding but can also make PRs to our codebase” mentality.
For those of you that run things on autopilot, how do you keep code quality under control? And how do you handle refactoring? I am really curious, because one option now is also to just YOLO your LLMs to write code based on the maturity of the product. You can refactor an app or parts of it pretty fast again with LLMs. While tech debt accumulates faster, we also have the opportunity to rebuild faster.
FYI: the non thinking variant has the same price.
What strikes me is that this difference can make EVs more expensive per kilometer if you only compare energy cost with fuel cost.
Here is the math with numbers. Tesla chargers in Switzerland and Germany are usually at most CHF 0.50 or EUR 0.60 per kilowatt hour at the more expensive locations, along highways for example. They offer fast charging of 150 kW or more. Alternative providers often start at around CHF 0.75 for 50 kW or CHF 1.00 for more than 250 kW fast charging. If your electric car consumes 20 kWh (Model 3 is at around 15 I think) per 100 km you end up with costs of CHF 10.00, CHF 15.00, or CHF 20.00 per 100 km at CHF 0.50, CHF 0.75, or CHF 1.00 per kilowatt hour. If you drive a petrol car that uses 8 l per 100 km and the cost per liter is CHF 1.70 you pay CHF 13.60 per 100 km.
Disclaimer: I’m Swiss and studied at ETH. We’ve got the brainpower, but not much large-scale training experience yet. And IMHO, a lot of the “magic” in LLMs is infrastructure-driven.
Our intention with LightlyTrain (AGPL/Commercial license option) is to offer a streamlined, production-ready pretraining engine. This contrasts with our other library, LightlySSL (github.com/lightly-ai/lightly), which is MIT-licensed and geared towards researchers needing flexible building blocks.
We found many companies wanted a simpler "it just works" solution for pretraining, which is why LightlyTrain exists with its specific licensing options tailored for commercial teams alongside the AGPL.
Thanks again for the clarification!
We designed LightlyTrain specifically for production teams who need a robust, easy-to-use pretraining solution without getting lost in research papers. It builds on learnings from our MIT-licensed research framework, LightlySSL (github.com/lightly-ai/lightly), but is tailored for scalability and ease of integration.
For commercial use where the AGPL terms might not fit your needs, we offer straightforward commercial licenses for LightlyTrain. Happy to chat more if that's relevant for you!
As liopeer mentioned, we have results for medical (DeepLesion) and agriculture (DeepWeeds) in the blog post. We haven't published specific benchmarks on satellite or industrial inspection data yet, but those are definitely the kinds of niche domains where pretraining on specific unlabeled data should yield significant benefits. We're keen to explore more areas like these.
Our goal is exactly what you pointed out - bridging the gap between SSL research and practical application where labels are scarce. Appreciate the encouragement!
We just released LightlyTrain, a new open-source Python package (AGPL-3.0, free for research and educational purpose) for self-supervised pretraining of computer vision models: https://github.com/lightly-ai/lightly-train
Standard vision models pretrained on generic datasets like ImageNet or COCO often underperform on specific domains (e.g., medical, agriculture, autonomous driving). Fine-tuning helps, but performance is limited, and getting enough labeled data is expensive and slow.
LightlyTrain uses self-supervised learning (SSL) to pretrain models directly on your own unlabeled images or videos. This adapts the model to your specific visual domain before fine-tuning, leading to significantly better performance with less labeled data.
Key Features:
- No Labels Needed: Pretrain using your existing unlabeled image data.
- Better Performance: Consistently outperforms training from scratch and ImageNet-pretrained weights, especially in low-data regimes and domain-specific tasks (benchmarks in README/blog). We see gains across detection, classification, and segmentation.
- Domain Adaptation: Tailor models to your specific industry (manufacturing, healthcare, retail, etc.).
- Supports Popular Models: Works out-of-the-box with YOLO (v5-v12), RT-DETR, ResNet, ViTs, etc., integrating with frameworks like Ultralytics, TIMM, Torchvision.
- Easy to Use & Scalable: Simple pip install, minimal code to start, scales to millions of images, runs fully on-premise (single/multi-GPU). We built this because while SSL research is mature, making it easily accessible and effective for industry computer vision teams was hard. LightlyTrain aims to bridge that gap.
We’ve benchmarked it on COCO, BDD100K (driving), DeepLesion (medical), and DeepWeeds (agriculture), showing strong improvements over baselines (details in the repo/blog post linked below). For example, on COCO with only 10% labels, LightlyTrain pretraining boosted YOLOv8-s mAP by +14% over ImageNet weights and +34% over no pretraining.
- GitHub Repo: https://github.com/lightly-ai/lightly-train
- Docs: https://docs.lightly.ai/train
- Detailed Blog Post/Benchmarks: https://www.lightly.ai/blog/introducing-lightly-train
- Quick Demo Video: https://youtu.be/5Lmry1k_cA8
We’re here to answer any questions! Happy to discuss the tech, benchmarks, or use cases. Commercial licenses are also available for businesses needing different terms.
I’m on an M-series MacBook now but still want to target x86 as well, and without portable SIMD that would’ve been a headache.
If anyone’s curious, the project is here: https://github.com/IgorSusmelj/rustynum. It's just a learning exercise for learning Rust, but I’m having a lot of fun with it.
I understand that keeping the same data and curating it might be beneficial. But it sounds odd to roll back in time with the knowledge cutoff. AFAIK, the only event that happened around that time was the start of the Gaza conflict.
Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. To focus on speed you'd need to get rid of the memory bottleneck. Then you end up designing your own ASIC with it's own memory. The NPUs we see in most devices are part of the SoC around the CPU to offload AI computations. It would be interesting to run this benchmark in a infinite loop for the three devices (CPU, NPU, GPU) and measure power consumption. I'd expect the NPU to be lowest and also best in terms of "ops/watt"
So gpt-4 level 8B models running on phones and notebooks seems feasible within the next 5 years. I imaging having (voice) assistans running locally. Crazy how fast we progress.