renonce

Yeah but it’s not representing a known number - I would rather say it is a proof that the busy beaver function for BLC at 49 bits is higher than Graham’s number. The fact that it represents known numbers concisely is more interesting to me.

One invention that drew me to this topic was Binary Lambda Calculus, invented by John Tromp (“Tromp” as in Tromp-Taylor Rules). It’s just a direct binary encoding of a lambda calculus term, but it gives you a concrete and concise representation which is useful for evaluating the complexity of an expression. I encountered it when viewing https://codegolf.stackexchange.com/questions/6430/shortest-t... and found that this language was the one that could write the most precise representation of a very large number with the fewest bits possible. I later learned that Graham’s number could be encoded in 120 bits (maybe 3~4 less), much more concise than equivalent mathematical language, and I’ve since been drawn into the field of googology. It was fascinating.

Essentially the algorithm is about pruning parameters on the fly and making the weight matrix sparse by setting least significant weights to zero, where "least significant" is defined in terms of the rank of the absolute value of the pruned weight within its group?

A search for model pruning turns out many results, including https://arxiv.org/abs/2305.11627 which discusses "magnitude pruning" as a baseline, and refers to https://arxiv.org/pdf/2301.00774.pdf, which asserts in the introduction:

> First, as shown in Figure 1, SparseGPT can induce uniform layerwise sparsity of up to 60% in e.g. the 175-billion-parameter variant of the OPT family (Zhang et al., 2022), with minor accuracy loss. By contrast, the only known one-shot baseline which easily extends to this scale, Magnitude Pruning (Hagiwara, 1994; Han et al., 2015), preserves accuracy only until 10% sparsity, and completely collapses beyond 30% sparsity.

I don't like how these papers boast their own methods by poorly implementing a baseline and describing their own methods using lots of mathematical jargon - the OP's blog post is a breeze in making the methods accessible to people with very little background.

Quantum teleportation does not allow you to transfer an amount of information faster than speed of light. Say you have two labs with two entangled particles 300000km away such that when one particle is “up” the other will be “down”. When lab A observes the particle it has 1/2 probability of being “up” and 1/2 probability of being “down”. Same for lab B. But once both observations are known by someone C (could also be A or B), C will find that one of the particles is “up” and the other is “down”. Bell’s theorem further tells us that there are cases (probability distributions of A and B) where the distribution could not be explained by assuming that the probability distribution of A and B is determined before observation.

There are never ending debates on what happened while observing A, but I dislike calling it “quantum teleportation” as it only further confuses people as if any information was teleported during observation. My theory is that A and B are totally independent until both observations reach a middleman C, and it’s at this point that the entanglement starts to matter.

Based on your argument, the reason for legalization of weed is that the current police system has totally failed and decided the only way forward is to give up policing it? That would make this move seem more of a failure than a victory…

I don’t think zero knowledge systems are practical at the moment. It will take over around 8 orders of magnitude more compute and memory to produce a ZKP proof of generic computation like compilation. Even 2 orders of magnitude is barely acceptable.

While I don't think it's practical to expect every paper author to write in the most consise and elegant style possible, one thing I do wonder is the lack of a "comment section" for papers: why isn't there a centralized place for academics to just leave comments or their understanding of the paper, like most of the web forums? I don't expect the author to respond or even be active in every comment section but there should certainly be such a place. Every other form of Internet-based discussion, including Reddit, any random Internet forum, mail lists, Discord, etc. are all interactive and everyone is expected to participate in a centralized and public discussion.

This way, even if the paper itself is hard to read, for the good ones I do expect someone else to leave a comment with a sketch of the most interesting parts of the paper that would get upvoted to the top, and the author could click a "promote" button or something to make it official. This way the author doesn't have to write very well as long as the paper is valuable enough for someone else to be interested.

While I don’t expect the company to do such PR themselves, from my experience in China it is indeed the case that many major public figures or products began their explosive growth from negative comments like this

I understand the definitions of such crypto algorithms but have no idea about differential cryptanalysis. Can someone explain how attacks like this are constructed, and why it took 8 years to advance cryptanalysis by 3 rounds? What insight was needed that took 8 years to discover and formulate as a practical attack?

I'm not sure what this is for - can't I use the API of the underlying models? Using Supervision means I need to learn a new API for using existing CV tools and I don't see how it reduces enough complexity to justify that extra layer of abstraction. I see value in unifying different models but these connectors are just thin wrappers that rename properties and it's not like I have to compare 7 different classes of models every time to find which one works?

What I do prefer is something like a GUI with features like previewing detections and specifying data for training, maybe in the form of built-in components, which would have a smoother learning curve but still allow me to dig deeper whenever needed. Any such tools available?

Competitors are catching up and one day the chat API providers will race the prices to bottom

Depends on whether Apple’s own browser engine requires JIT. I think the point is to ensure fair competition.

Highlight from this paper:

> Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI’s gpt-3.5-turbo to be about 4,096.

Discussion of the other paper is at https://news.ycombinator.com/item?id=39675735 but DeepMind seems to care more about helping their peer keep secrets and did not release GPT-3.5’s embedding size. Also the other paper is mentioned in Section 9 “Simultaneous discovery” of this paper in case anyone wonders.

> the researchers examined the relations between participants’ self-reported engagement in both types of BSing and their ratings of how profound, truthful, or accurate they found pseudo-profound and pseudo-scientific statements and fake news headlines

If I understand correctly, they simply asked the participants themselves if they are BS’ing and drew the conclusions solely from that? How do you expect people BS’ing others would admit doing that?

Sounds like an interesting direction of research. There is a tool called mergekit and the models produced by merging different models is called frankenmerge. Search for “frankenmerge” and you can find a lot of interesting results and models and discussions on what works and what doesn’t. Might be a good idea to check previous experiments with these keywords.

> The obvious alternative to gradient descent here would be Bayes Formula

If you know a little about the math behind gradient descent you can see that an embedding layer followed by a softmax layer gives you exactly the best Bayes estimate. If you want a bit of structure, like every word depends on previous n words, you get a convolutional RNN which is also well studied. These ideas are natural and elegant but maybe a better idea is to comprehend the research already done to avoid diving into dead ends too much.

At least in theory we can achieve incremental learning by training from scratch every time we got some new training data. There are drawbacks with this approach such as inconsistent performance for different training runs and significantly higher training cost but it's achievable. Now the problem is if there exist methods more efficient than gradient descent? I think it's very clear now that there are no other algorithm in sight that could achieve the level of intelligence without gradient descent at its core, and the problem is just how gradient descent is used.

It’s stealing the last layer (softmax head), not an arbitrary part, also it targets “production models whose APIs expose full logprobs, or a logit bias”. Not all language model APIs have these features and this characterizes what APIs can be targeted and what can’t. These important pieces of information should have been written in the title or abstract rather than “typical API access”.

You dont need to wait for that one day. Right now I’m using cloud servers from hetzner and store data in Backblaze S3 and train models on Lambda or even vast.ai.

I learned the easiest way to do d-dimension sampling in Foundations of Data Science: see https://news.ycombinator.com/item?id=34575637 or https://www.cs.cornell.edu/jeh/book.pdf?file=book.pdf

I don't think it's a good idea to introduce over 20 different methods before talking about the correct one that works for any number of dimensions, say n, and the reason behind its correctness is very obvious:

* Generate a random vector by sampling n standard normal distributions: `vector = np.random.randn(n)`

* Key step: show that the vector has a uniform direction.

The proof is as follows: you look at the probability density function of a normal distribution, which is `p(x)=1/sqrt(2pi)*exp(-x^2/2)`, and the probability density function of the vector is the product of all these densities of its individual dimensions. Now the product `p(x1)p(x2)...p(xn)=1/sqrt(2pi)^n*exp(-(x1^2+x2^2+...+xn^2)/2)` since `exp(a)exp(b)=exp(a+b)` due to power functions.

Now it's easy to see that probability density is invariant to vector length, which means the vector has uniform probability for any specific value of x1^2+x2^2+...+xn^2. Whatever rotation you apply after sampling this vector, since rotation preserves x1^2+x2^2+...+xn^2 by definition, you get the exact same probability density function and therefore the same distribution of vectors.

* Now that the direction of the vector is uniformly sampled, decide the radius separately: for n-sphere the radius is just 1, and for n-ball the volume of the radius is proportional to the nth power of the radius, so you sample a uniform number from [0,1] as the volume and take the nth root as the radius: `radius = np.random.uniform(0, 1)*(1/n)`

* Normalize the vector to the radius: `vector = vector / np.sqrt((vector*2).sum()) * radius`

I think Section 2 of this book provides a much better perspective on the problem of generating uniform random points, since it also provides intuition behind the geometry of high dimensions and properties of the unit ball, etc.

Two places where the project is very impressive:

1. Emulating a desktop with windows so smoothly on both desktop and mobile. This webapp is much snappier than most webapps these days.

2. A storage and file explorer with a friendly API for third party apps. Now any app can use a cloud storage synced across devices where the storage costs are paid by the user.

Can't wait until more apps are available! I see some limitations like VS Code unable to open git repositories which seems a limitation of the storage API (append-only, cannot seek or download partially)? Hope it gets closer to native experience in the future

> Founders and employees would be paid only if the acquisition exceeded $559M

I think the problem with the article is that it emphasized the absolute acquisition value and omitted the fact that the company raised $400M. Considering the amount they raised, $559M seems much more reasonable and not quite bad terms?

Here is my shiny new super business plan for a startup that will profit thousands from a supposedly non-paying customer:

1. Offer “free static website” with lots of templates and guides to help you build one

2. The first 100GB is free and beyond that it’s $0.01/MB. But no worries! Very few customers actually use up that free bandwidth and in case you need more you can purchase packages for $100/TB. Also we offer a free service that will help you get your site more visible by advertising it, it’s included by default.

3. After a month or so, randomly help a customer bump the website and make it popular by putting it in some list that is frequently crawled. Secretly hire someone else to crawl these websites and make lots of download requests

4. Once the customer suddenly gets 10TB of traffic, bill them for 9900GB which is $99000

5. As long as 1 out of 100 customers pay, you are profiting $990 per customer! For the rest of customers, offer a 5% discount so they only have to pay $1980. Threat taking them to collections if they refuse.

Become the next millionare by just selling free static websites to 1000 customers! Anyone join us?

So is this GPT for images? They take a generative model and apply finetuning via LoRA on some downstream task such as surface normal and conclude that these models intrinsically learn these representations, and find that they do better than supervised approaches?

I think this is awesome but maybe it’s not really that surprising given how this “generate and then finetune” approach has already worked so well?

That prompt looks great. I tried it on OpenAI's incident response and got: (only last paragraph taken for brievity)

> ... > In conclusion, while the passage does provide a basic account of the incident, its lack of detail, reliance on jargon, and vague explanations leave room for skepticism regarding the severity of the issue, the transparency of the response, and the effectiveness of preventative measures. A more thorough and less technical explanation could have mitigated some of these concerns.

Cloudflare's postmortem: (https://blog.cloudflare.com/cloudflare-outage-on-june-21-202...)

> In summary, the passage does not exhibit overt deception, manipulation, or dishonesty. Cloudflare's statement seems to be a forthright attempt to explain what happened, accept responsibility, and outline steps to prevent future incidents. The inclusion of technical details and the acknowledgment of their error reflect a commitment to transparency, even if the promotional content at the end might seem slightly incongruent with the preceding apology and explanation.

Now the thing is to make a fast and reliable plugin that automatically puts the warning on the webpage and shame these corporates

Presigned URLs are long and ugly and b2 requires an extra API call to generate that, so I tend to prefer not to use them. The other reason for obscure names is so that you can't download or update files that you don't know the name even if you have the upload key. I know it goes against common sense security practice but compromising convenience can mean compromising security in many cases. The thing is to make sure your scheme is rigourous.

That sounds like a good use case for GPT. A GPT that automatically highlights such corporate speak and hints “WARNING: bullshit ahead”. I’m 100% sure it’s technically very easy to engineer such a model.

Do you think OpenAI’s superalignment will ever allow you to make such a model?

So now you say on your webpage your pricing is $1/TB or something. Great. But there is a caveat: the amount you pay depends on some complex factors such as the size of the table or the duration of your code. If the factors are so simple that no more than grade-level arithmetic is required to calculate my costs then that would be fine. But what if it gets a little bit more complex than that, such as “table size is 1PB and cost per 1TB is $1”? Did you know 1PB=1000TB rather than 0.001TB? What about “you need another $10 query to figure out the size of the table”? Or “the cost depends on number of function calls that your code makes and if you accidentally recursed yourself too many times you can’t limit it”? Or “The server is $5/mon but IP is $1/h and outbound traffic is $10/GB and if someone download something 1TB on your server you will pay $10000 within 2 hours”? At some point the factors related to billing is going to become non-trivial and every sentence in a long 10 page document could have 100x’d your costs, what makes this service different from a scam? You could have allowed me to set a billing cap so I won’t have to pay anything beyond $10, so that “$10” is everything I have to care about, could you?

But it’s costly. It would be even more useful if it were 1000 times cheaper than it is now. And we don’t yet know if OpenAI earns more than it spent on renting cards or not

Now museums are green but I'll have to pay $500 for a new phone every two or three years plus cellular data (Seems to average $70/mo for unlimited data in US?) and these costs pay for cellular stations and electricity and electronics that become garbage after just a few years. Is that really greener?