It's called self attention because you only look at the input itself to determine which parts of the input to focus on. Transformers where first proposed for machine translation, and the neural net used in that publication consisted of two parts: an "encoder" part that takes an input language sentence and uses this self-attention to better understand it (and "encode" it into some neural representation), and a decoder which takes both the encoded source-language input as well as already translated parts of the target level sentence as input. These encoders use cross attention, ie they use the input sentence to determine on which part of the output to focus on next.
That sounds fairly childish. Aside from the very obvious fact that you'll miss out on a lot of cool stuff, if you don't agree with the value judgements of your peers, the problem most likely resides with your choice of peers.
Bayesian optimization and bayesian data analysis are two different things. Your suggestion seems off topic to me, and it feels like you didn't even check the linked content before posting.
I can't really any good ways for them to raise money, outside of becoming a subscription service or selling user data. What ways did you imagine? Because the more I think about it, the harder this problem appears to be...
That makes sense, thanks for elaborating! I could imagine a different solution where instead you just don't report the overall star-average, to make it a harder metric to game, but still get to have a stronger feedback signal for your recommendations.
If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.
I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.
In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always the case back then). I benched this extensively for all my R usecases and for several systems that i cared about back then. So there is usually no need to use MKL in the first place.
> in this case it takes 4-8 threads or so for KySync to match Zsync's performance.
Assuming zsynch is single threaded, in an apples/apples comparison (1 thread) zsynch wins by over 10x. I can't help but wonder how much faster kysynch would be if they focused more on single threaded performance. The way things stand, kysynch's implementation sounds rather wasteful I'm terms of CPU resources, in spite of what this announcement claims. Still, kudos on this impressive project!
It's always nice to see big labs working more towards building an understanding of things instead of just chasing SOTA. On the other hand, I'm not sure there is a lot of actionable findings in here. I guess that's the trade off with these things though....
You're being cryptic. The people your criticizing (this might include me, i have no way of knowing) will not be able to learn anything from your reply. It would be more constructive I'd you shared your thoughts so we'd have a chance to improve. What is your point, exactly? Spell it out for us, please.
As far as I know, hogwild doesn't really scale to very large systems and doesn't get used in practice, but maybe that's just because it's most useful on CPUs, which are inefficient for other reasons
> whatever will eventually become the standard, will be lossless, as it became with audio
At least ony bubble, the standard for audio is still lossless: I'm fairly sure 99.9% of all audio i consume is using something lossy:streamed over Spotify, in movies like Netflix, clips like YouTube, video Chats... Is that just my impression and technology moved to FLAC without me noticing?
You make it sound like this is Netflix's fault, whereas tlit seems to me that 10 years ago it was mich easier to acquire licenses for streaming. now some of the big studios have their own streaming services, so licensing isich harder/impossible
Master/Slave nomenclature is considered "problematic language" (because it harks back to slavery), and thus often replaced with more PC language these days. My organization had a whole "fix this nomenclature throughout the whole codebase" week (things like "black-/whitelist, master/slave, ....)
It sounds like the people that are being made redundant were mainly tasked with labeling data (which also matches them being temps instead of full time employees, and them training replacements in Costa Rica), so I don't think they're software engineers that can easily switch to other teams.
> Were all dancing around the elephant in the room that clearly everyone panicked due to china's lies at the start of 2020
Given what I've witnessed since the start of the pandemic in my own country, it seems more plausible to me that china's actions are a product of people and states not knowing how to deal with an unknown situation and changing it's course as situations unfold. What are the lies China has told that I'm not aware of?