xcodevn

There is a simple web page where you can explore the neurons and discover interesting features yourself! With more than 400k neurons in the Llama3-8B model, there are plenty of fascinating neurons for you to uncover. Check it out and explore all the neurons at https://neuralblog.github.io/llama3-neurons/neuron_viewer.ht...

tldr: mamba is not as good as transformer.

please help yourself and do a quick Google search about "zero shot" and "few shot" learning.

Of course, it will require exponential data for zero shot. The keyword here is zero shot. If you think about it for a second, this applies to humans too. We also need exponential training data to do things without examples.

> Probably not.

Lmao.

I must say, understanding how transformers work is arguably the most important research problem in history, assuming that AGI can be achieved by just scaling up current LLM models on text, video, audio, etc.

I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?

Even if OpenAI falls apart, this is still a good move.

> 5. It has deep implications for the trajectory of a technology that many see as heralding a revolution at least as significant as — if not more than — agriculture or industry, with truly existential implications for humanity.

Yes, this is likely to be one of the most important events in human history. We are living through a special period of evolution on Earth.

So this is what it looks like inside a car's dreams.

Can anyone conduct a proper measurement of resistivity versus temperature and determine whether it's a superconductor or not?

"Analysing the agent’s internal representations, we can say that by taking this approach to reinforcement learning in a vast task space, our agents are aware of the basics of their bodies and the passage of time and that they understand the high-level structure of the games they encounter."

Wow, really amazing if true.

P.S.: After looking into their paper, it's not that impressive. They use agent's internal states (LSTM cells, attention outputs, etc.) to predict whether it is early in the episode, or whether the agent is holding an object.

It is a single matrix multiplication to predict probabilities of all possible outputs. For example, with a hidden state of 1024 dimensions, and 8 bits output, it is 1024x256 parameters. 10 bits will need 1024x1024 params.

This is a well known problem. The noise is due to mu-law compression. The 16 bit audio samples are compressed to 8, 9, or 10 bits before feeding to the neutral net. The reason is because predicting a categorical distribution of 2^16 values requires too many parameters. The noise was also in samples from the famous Wavenet from Deepmind (they used 8 bit mu law).

There are two ways to avoid this: 1. predict 8 high (coarse) bits, 8 low (fine) bits separately as in the original waveRNN paper. 2. use a mixture of logistic distributions as the predictive output as in the recent Lyra vocoder from Google.

The most amazing thing in this video is NOT how we can read signals from the monkey brain, but how a monkey brain can learn to generate signals to control the cursor. I think it is remarkable that a brain can learn to have such fine control of its neural activities.

My immediate thought: train a Transformer (or Tacotron2) that transforms text to the encoded Lyra codes... And, we will finally have a good real-time open-source text-to-speech system running on mobile devices.

It is called Quantum Field Theory.

I would pick variational auto-encoders (VAEs) as the closest analogy to how the brain functions. Brain observes data and encodes it to a latent vector. When we imagine something or dreaming, brain decodes latent vector to video.

Moreover, usually our brain cannot imagine something as sharp and real as GAN's output. It's more like a blurry image from VAE's output.

Google RAPPOR technique adds noise to the report at client (user) side. The noisy reports are then collected to a central server for analysis.

Doing this guarantees differential privacy, a strong privacy protection.

However, the major disadvantage of RAPPOR is that we get a collection of very noisy data points. Therefore, we need a very big collection to learn anything useful at all!

If the number of clients/users/records is small, RAPPOR is practically useless.

Prio, on the other hand, adds NO noise to client's reports. It uses crypto magic to compute some functions (mean/sum/...) on the collection of reports by a network of servers. Only outputs of these functions are known. No reports are leaked until at least one server is honest!

However, Prio has no guarantee that the computed output doesn't leak information about client's reports (in fact, it surely leaks some information). While, RAPPOR guarantees this protection statistically!

A current research direction is to add noise to the output of Prio as such it guarantees differential privacy.

You don't need to trust all third-parties. Crypto magic guarantees that what you need is at least ONE server being honest.

If you trust Mozilla, then you have nothing to worry about, except the proof or program may be wrong ;)

Third-parties are for someone who doesn't trust Mozilla. Or that Mozilla could be compromised by an unknown adversary.

> I can bring a more accurate wind speed sensor to calibrate the windmill against the wind [...] But what should the prior be? [...] I have to have a crisp number — a full distribution actually, accurate out to the tails.

What would you do when the sensor returns negative wind speeds due to noise or errors?

The wind speed cannot be negative, or greater than speed of light. An expert in windmill can narrow down the prior distribution much more.

> But priors can be much harder.

Choosing a prior is hard because it requires thinking explicitly about the problem and its assumptions. It merely exposes our lack of expertise on the problem.

When you're lazy, you can pick a uniform prior Uniform(0, c) and call it a day.

> We can sharpen the problem. Say my number and yours are different. How do we tell who is right?

Forget about prior, say, we have 2 sensors which output two slightly different wind speeds. Which wind speeds is right? The lower one or the average speed.

This is a deep philosophical problem. However, it's a problem for any model.

> The prior will mostly wash out in any well-posed problem.

I don't think so. Any well-posed problem should include the prior, or else how can we tell: 2 data points is not enough?

> HN tends to invoke the Bayesian framework as a complete solution to inference [...]

Bayesian framework is indeed a complete solution to inference in a formal/logical sense. However, I agree that there are many problems in applying Bayesian framework to real world problems that requires serious thinking about our assumptions on the problem.

By definition of conditional probability: P(A|B) = P(A,B) / P(B) and P(B|A) = P(A,B) / P(A). Therefore, P(B|A) = P(A|B) * P(B) / P(A).

> It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right.

Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the formal world, to the real world for validation.

Prior predictive simulation method, which generates random data points from the prior, is a good heuristic to check if the prior is NOT plausible.

See the discovery paper of gravitational waves [1] which uses informative priors from physical evidences/constraints.

[1]: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.11...

I strongly disagree. Bayesian inference is the only known self-consistent formal system for doing science, i.e. updating our belief system about the world based on the current evidence.

There are several libraries. PyMC4 [1], the next version of PyMC3, will introduce TF as a backend. Tensorflow probability [2] from Google. PYRO [3], from Uber, uses Pytorch backend.

[1]: https://github.com/pymc-devs/pymc4

[2]: https://www.tensorflow.org/probability/

[3]: http://pyro.ai/

I truly believe that Bayesian inference is the statistics of the 21st century. Recent advances in MCMC (e.g., NUTS, Stan [1]) and variational inference (e.g., ADVI [2], VAE [3], etc.) + more computing power than ever promise a near future in which Bayesian inference is the default inference engine.

Prior distribution is a beautiful and logical mechanism for adding regularization, domain-specific knowledge to our model.

[1] Stan, a platform for statistical modeling http://mc-stan.org/

[2] Automatic Differentiation Variational Inference https://arxiv.org/abs/1603.00788

[3] Auto-Encoding Variational Bayes https://arxiv.org/abs/1312.6114

Now let's think about the feeling of people in other countries when be attacked by US missiles.

A quote from Microsoft's video: "[A] computer that computes in a billion parallel universes at the same time".

This is a bold statement with no physical evidence to backup and no idea how quantum computer works.

If you read the paper https://arxiv.org/abs/1705.09558, in section 2.1, it defines two conditional posteriors for the parameters of the generator and discriminator. Then, classical GAN is just a maximum likelihood estimate (or, a MAP estimate with uniform prior) of the parameters.

Next, the paper said: how about sampling the whole posterior distribution instead of finding only one point of maximum likelihood as classical GANs do. This is the point where the famous Markov chain Monte Carlo (MCMC) algorithms become useful. They use something called stochastic gradient Hamiltonian Monte Carlo, basically, it is a random walk algorithm, at each step, you follow a noisy gradient, as a result, you converge to the posterior distribution instead of a local minima as gradient descent does.

The paper claims that sampling the whole posterior helps to resolve problems with classical GANs. IMHO, this isn't a surprise claim, this is exactly what is good about Bayesian statistics.