voxgen

> Her channel has strayed far beyond the topics she has credibility in.

I appreciate that she makes her videos so easily verifiable, by prominently showing her research, that it was easy to see the point when this started happening and tune out. A lot of opinion-faucets on the internet try to be irrefutable by hiding their sources.

I don't trust Sabine intrinsically, but I trust that I can notice when she under-researches a topic or makes a leap of logic. She conveys enough good information that I find it worth my time to watch.

The difference is sources. Sabine shows her sources prominently on screen, with searchable citations to find the original. She makes it clear in her phrasing whether she's paraphrasing a source, or passing her own judgement.

It's easy to know whether to internalize what she says when you view it critically. Ask "does the presented research seem legit, complete, and impartial?" and "is her conclusion logical?". She gives you the receipts to check. This is not the same as deciding whether to put blind faith into a comedian's off-the-cuff anecdotes and opinions.

I often disagree with her conclusions, but at least she makes it very easy to validate her chain of though, find where our views diverge, and only absorb the information I trust.

I've seen similar reactions and I can't help but think she's intentionally communicating provocatively to make people engage their brains.

You shouldn't just "take her seriously", you should take what she says *critically*. Hear the information and opinions, then decide for yourself whether to accept them.

The patients know this. Asking for consent would still yield the vast majority of the data.

It would also mean more people analyzing it - I know at least one big pharma company that won't touch patient data that was taken without consent because they don't want to be associated with such unethical practices.

I've had 2 sets of AirPods die because of the batteries aging and losing their ability to hold a charge. Though to be fair, their microphones failed much earlier then their batteries, and noise cancelling gets some really annoying failure modes when the microphone has issues. I should have replaced them sooner. I also should find a brand that doesn't have consistent manufacturing quality issues...

The *GLU-based activations functions like GEGLU and SwiGLU use 2 input values to produce 1 output value, which makes these numbers weird. In each value pair, one goes through the GELU/SiLU activation function and is then multiplied by the other "gate" value.

In the report, "hidden dim" matches the number of GEGLU inputs. In the config, "intermediate_size" matches the number of GEGLU outputs. Most *GLU models so far have used intermediate_size=8/3*d_model as this makes have the same number of matmul FLOPS & parameters as a 4x-expanded non-GLU model, and PaLM vaguely showed that 4x is better than a smaller expansion factor.

If one considers Llama-2-7B's FFN expansion factor to be ~5.33x, Gemma's expansion factor is 16x.

Thank you very much for releasing these models! It's great to see Google enter the battle with a strong hand.

I'm wondering if you're able to provide any insight into the below hyperparameter decisions in Gemma's architecture, as they differ significantly from what we've seen with other recent models?

* On the 7B model, the `d_model` (3072) is smaller than `num_heads * d_head` (16*256=4096). I don't know of any other model where these numbers don't match.

* The FFN expansion factor of 16x is MUCH higher than the Llama-2-7B's 5.4x, which itself was chosen to be equi-FLOPS with PaLM's 4x.

* The vocab is much larger - 256k, where most small models use 32k-64k.

* GQA is only used on the 2B model, where we've seen other models prefer to save it for larger models.

These observations are in no way meant to be criticism - I understand that Llama's hyperparameters are also somewhat arbitrarily inherited from its predecessors like PaLM and GPT-2, and that it's non-trivial to run hyperopt on such large models. I'm just really curious about what findings motivated these choices.

If it was cheaper, or part of a bundle, or included unlimited API access, I'd consider it. GPT4's high price is always in the back of my mind when I use it.

But does Bard really qualify as the second best? Gemini Pro is pretty bad. I'd choose Claude over it any day.

Gemini Ultra's claimed superiority required a mountain of prompt optimization and repeated generations to get good scores. The "CoT@32" benchmark results costs at least 32x as many tokens, probably closer to 64x with the hidden chain of thought. I don't have high hopes this mode will be affordable or fast enough for chat.

One can also make a model to learn the necessary context length for each layer and head to save a huge amount of FLOPs: https://arxiv.org/abs/1905.07799

Actually, blood pressure is the determiner of whether high intake of salt is bad for you.

High sodium intake is only associated with increased risk in people with high blood pressure. However, having too little sodium is associated with increased risk regardless of blood pressure. Source: https://www.thelancet.com/journals/lancet/article/PIIS0140-6...

This was an expected step of the evolution of the Memory Hierarchy. RAM is just bigger-but-slower CPU cache. Swapfile is just bigger-but-slower RAM. There's always a trade-off between size and speed.

Incorporating RAM into the CPU package gives a big performance boost at the expense of maximum capacity. To compensate for the reduced performance in tasks that require more memory, you can either add another layer into the Memory Hierarchy (i.e. CPU Cache -> Fast RAM -> Slow RAM -> Swapfile), or you can improve Swapfile speed. It appears Apple have done the latter - they claim their SSD is up to 2x faster.

Video game consoles have done something similar with the latest generation. Only a 2x increase in RAM from the last generation (compared to an 8x/16x increase from the generation before), but a focus on storage performance.

The Murdoch news empire has fluctuated between 5 and 6 on that scale for as long as I can remember.

Not trying to excuse Facebook and Twitter's behavior here, but let's not let social media become a scapegoat while News Corp continues to run rampant.

> Sorry for the long post.

Don't be. This is the first post I've seen that actually answers more questions than it provokes. Thanks for helping me understand what happened.

That's a symptom of bad healthcare systems.

In the UK, I didn't have to pay anything for refills of concerta. In NZ, I only had to pay the GP who gave me a referral to a free psychiatrist for methylphenidate and dexamphetamine sulphate.

In both places, it has been a relatively long journey to get medication. They were keen to try to resolve my issues without drugs - it was about half a year between when I started seeing a psychiatrist and when they suggested medication.

Why is Dart able to run so much faster than JS?

The Dart team came from the V8 team, and presumably they're still involved in V8. I'd expect any optimizations that were developed for Dart and were applicable to JS would be backported into the V8 engine, so there must be something different about the Dart language that enables more optimizations.

From what I've seen of Dart, if you ignore the type annotations (like the VM does), and you don't consider syntax, the only performance-relevant difference between JS and Dart is the OOP system, which wouldn't explain a 50% performance difference...