mkaic

Damn, I had no idea. That's really unfortunate.

This reminded me of Bishop's Castle in Colorado, USA — an incredible project built almost entirely by one man (who sadly died last year) working on it nonstop for 40 years:

https://en.wikipedia.org/wiki/Bishop_Castle

As a queer person who hasn't used Grindr but has heard about it from others, my understanding is that it has a reputation for being more for hookups than actual dating. Heard a story about a gay male friend-of-a-friend setting up an account on it and getting 4 unsolicited dick pics within a very short period of time.

Obviously this is just secondhand anecdotal evidence—I'm sure someone who has personally used the app could chime in and correct me if I am wrong.

Hell yeah. They do strike me as opening up some exciting possibilities—the fact that we can now do math on semantic information (something that has famously evaded prior attempts to do math on it) is really intriguing.

This reminds me of the antics of streamer DougDoug, who often uses LLM APIs to live-summarize, analyze, or interact with his (often multi-thousand-strong) Twitch chat. Most recently I saw him do a GeoGuessr stream where he had ChatGPT assume the role of a detective who must comb through the thousands of chat messages for clues about where the chat thinks the location is, then synthesizes the clamor into a final guess. Aside from constantly being trolled by people spamming nothing but "Kyoto, Japan" in chat, it occasionaly demonstrated a pretty effective incarnation of "the wisdom of the crowd" and was strikingly accurate at times.

I seem to be going in the opposite order lol—I've been getting my facial hair removed for almost a year now, but am only barely starting to become comfortable with the idea of actually transitioning. And if I did transition, I think I'd likely only come out socially to a very small group of people until I'd also transitioned hormonally! Perhaps my priorities are all backwards...

I read your linked post and enjoyed it—I've heard vaguely similar sentiments expressed before by some of my trans friends. It's like that RuPaul quote: "We're all born naked, and the rest is drag."

I hope you have a nice day, and thank you for the well-wishes :)

As someone pondering the exact same sentiment of "I like women so much that I kinda want to be one" but who hasn't fully committed to it yet, I really appreciated your vulnerability and lucid writing. I hope folks are kind here in the comments.

> It took me a while to remove my facial hair, I still haven’t trained my voice.

The facial hair removal really does take forever, it's so annoying :sob:. And I've found voice training (particularly around other people) to be really intimidating. I wish you the best of luck if you decide to pursue it!

Take care, OP.

Starlink satellites are placed in extremely low orbits specifically to avoid their becoming dangerous space-junk — their orbits are intended to decay after around 5 years, at which point they burn up in the atmosphere and leave no debris behind in LEO. Future iterations of the satellites may have even shorter lifetimes as launch costs get cheaper.

Monk and Robot is such a wonderful series. I have often wondered about the "now what" part of living in a hypothetical utopia, and Chambers explores that existential question really thoughtfully.

A friendly reminder: if you have tech-illiterate people in your life (parents, grandparents, friends, etc), please reach out to them and inform them about advances in AI text, image, audio, and (as of very recently) video generation. Many folks are not aware of what modern algorithms are capable of, and this puts them at risk. GenAI makes it easier and cheaper than ever for bad actors to create targeted, believable scams. Let your loved ones know that it is possible to create believable images, audio, and videos which may depict anything from “Politician Says OUTRAGEOUS Thing!” to “a member of your own family is begging you for money.” The best defense you can give them is to make them aware of what they’re up against. These tools are currently the worst they will ever be, and their capabilities will only grow in the coming months and years. They are already widely used by scammers.

Anecdotally, I feel like I've heard "powerhouse" used somewhat regularly when describing impressive people, i.e.: "Such-and-such is an absolute powerhouse on the field", or "That person is so productive, they're a powerhouse".

So maybe the original usage has been subsumed by "power plant", but I think the word has alternative meanings which persist.

Counterpoint: the hidden state at the beginning of ([the][cat][is][black]) x 3 is (probably) initialized to all zeros, but after seeing those first 4 tokens, it will not be all zeros. Thus, going into the second repetition of the sentence, the model has a different initial hidden state, and should exhibit different behavior. I think this makes it possible for the model to learn to recognize repeated sequences and avoid your proposed pitfall.

> there would be a massive amount of redundant information stored in hidden activations

Is there a way to prove this? One potential caveat that comes to mind for me is that perhaps the action of lerping between the old state and the new could be used by the model to perform semantically meaningful transformations on the old state. I guess in my mind it just doesn't seem obvious that the hidden state is necessarily a collection of "redundant information" — perhaps the information is culled/distilled the further along in the sequence you go? There will always be some redundancy, sure, but I don't think that such redundancy necessarily means we have to use superlinear methods like attention.

The counterargument here is that you can just scale the size of the hidden state sufficiently such that it can hold compressed representations of whatever-length sequence you like. Ultimately, what I care about is whether RNNs could compete with transformers if FLOPs are held constant—something TFA doesn't really investigate.

I strongly enjoy the simplicity of their "minGRU" architecture. It's basically just:

  class MinGRU(nn.Module):
    def __init__(self, token_size, hidden_state_size):
      self.token_to_proposal = nn.Linear(token_size, hidden_size)
      self.token_to_mix_factors = nn.Linear(token_size, hidden_size)

    def forward(self, previous_hidden_state, current_token):
      proposed_hidden_state = self.token_to_proposal(current_token)
      mix_factors = torch.sigmoid(self.token_to_mix_factors(current_token))
      return torch.lerp(proposed_hidden_state, previous_hidden_state, mix_factors)

And since the proposed hidden states and mix factors for each layer are both only dependent on the current token, you can compute all of them in parallel if you know the whole sequence ahead of time (like during training), and then combine them in linear time using parallel scan.

The fact that this is competitive with transformers and state-space models in their small-scale experiments is gratifying to the "best PRs are the ones that delete code" side of me. That said, we won't know for sure if this is a capital-B Breakthrough until someone tries scaling it up to parameter and data counts comparable to SOTA models.

One detail I found really interesting is that they seem to do all their calculations in log-space, according to the Appendix. They say it's for numerical stability, which is curious to me—I'm not sure I have a good intuition for why running everything in log-space makes the model more stable. Is it because they removed the tanh from the output, making it possible for values to explode if calculations are done in linear space?

EDIT: Another thought—it's kind of fascinating that this sort of sequence modeling works at all. It's like if I gave you all the pages of a book individually torn out and in a random order, and asked you to try to make a vector representation for each page as well as instructions for how to mix that vector with the vector representing all previous pages — except you have zero knowledge of those previous pages. Then, I take all your page vectors, sequentially mix them together in-order, and grade you based on how good of a whole-book summary the final vector represents. Wild stuff.

FURTHER EDIT: Yet another thought—right now, they're just using two dense linear layers to transform the token into the proposed hidden state and the lerp mix factors. I'm curious what would happen if you made those transforms MLPs instead of singular linear layers.

Haven't written anything up about it, but the code is here: https://github.com/mkaic/dagnabbit. I haven't gotten it to work particularly well yet, so I don't have any demos posted.

If I ever get it to a point where I feel proud of the result, I'll write a blog post about it and submit it to HN.

This is fascinating—I've recently been experimenting with something spookily similar after coming across the same paper as OP, only I'm trying to make random DAGs of logic gates reconstruct images rather than audio signals. Great minds like a think, I guess :P

Liberty costs 1,009 gallons of water per day :P

I have used Comic Mono as my coding font for the past 2 years and unironically love it. I installed it as a joke so I could take some screenshots and get funny reactions out of my friends, but found myself genuinely enjoying the readability. These days I frequently forget it's even installed except when someone new joins the team and sees my IDE setup for the first time:

"What font is that??"

"Oh, haha, yeah... It's Comic Sans, but monospaced!"

"Uh huh. Okie dokie then."

Regarding "What makes people fat?", I feel like there are few charts as damning as the one [0] halfway through TFA showing a spike in obesity rates that coincides nearly perfectly with the introduction of the modern USDA Food Pyramid.

The quote below it sums things up nicely, imo:

> Dr. Michael Eades: “A few years ago, I went back and pulled some labels off some feed sacks in a farmer’s coop and ran them through my little nutrition computer and there’s virtually no difference in the macronutrient composition that farmers use to feed animals to fatten them up and that the USDA uses to tell us to supposedly slim down.”

It is truly insane how we continue to act as though anything other than drastic carbohydrate intake reduction will fix the American obesity epidemic. We've been eating way too much flour and sugar for decades now.

[0]https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_...

My intention was less to imply "I'll just spin it up" and more to imply "Some competitor of Mistral's will spin it up". I agree that from my perspective as a casual user, Mistral's API is quite convenient. What I don't understand is why they aren't driven to zero-margin instantaneously by an onslaught of clones of their business model.

I guess I was just under the impression that cloud inference is such a competitive market that it'd be nigh-on-impossible to compete with the major players.

As much as I love their work, I can't be the only one who really struggles to see a path to profitability for Mistral, right? How do you make money selling API access to a model which anyone else can spin up an API for (license is Apache 2.0) on AWS or GCP or similar? Do they have some sort of magic inference optimization that allows them to be cheaper per-token than other hosting providers? Why would I use their API instead of anybody else's?

Asking these questions as a genuine fan of this company—I really want to believe they can succeed and not go the way of StabilityAI.

> "...they're not capable of thinking or coming up with novel ideas."

This claim is unfalsifiable given common definitions of "thinking" and "coming up with novel ideas".

The article addresses this and notes that most of the cost reductions we've seen in the past decade are only really applicable to utility-scale installations, as it's only then that the one-time costs can be amortized sufficiently. Small consumer installations are dominated by costs other than the panels themselves to a much greater degree than utility-scale installations.

Unfortunately, many parents do not have the time or energy to spend this much time teaching kids to read. There are parents who have to work all the time so they can feed their kids, there are single parents who work even more to feed their kids, and there are parents who are raising multiple kids and can't give much time to each individual child as a result. There are also parents who struggle with reading themselves, and therefore aren't capable of teaching their kids!

I agree that in a perfect world, all kids would receive personalized reading instruction from their parents. In reality, though, the poor/disadvantaged kids who need such instruction the most are also least likely to have parents capable of giving it to them. Well-funded public education is one of the greatest socioeconomic equalizers we have and it's crucial that we continue to support and expand it.

You might find this paper interesting: [JPEG-LM: LLMs as Image Generators with Canonical Codec Representations](https://arxiv.org/abs/2408.08459)

I can totally see where you are coming from — and I think I agree more in the context of purposefully artistic photography. however, for me personally most photography is not artistic, but documentary. totally support using all sorts of crazy techniques and manipulations for artistic expression tho!

I want photos to be a record of my perception. I want them to match what I see in front of me as closely as possible. I do not care if data is mangled to make this happen — in fact, the reason I love my iPhone's camera so much is because it mangles the raw sensor data so much. Apple's computational photography pipeline is state-of-the-art, and it is designed to make photos look as close to my perception as possible. All my favorite sunset photos have all been taken on my iPhone.

I do not like these AI photo-editing features. Or rather, I do not like the use of them, and will not be using them myself. The technology itself is stunning and exciting! But it entirely defeats the purpose of taking a photo for me. If someone blinks in a group picture, I want the blink to stay. If there's some obnoxious piece of garbage in the background, I want it to stay, because the whole point is for a photo to be a representation of reality consistent with my eyesight at the time it was taken.

I truly love candid photos. I am open to computational photography and even AI processing, but it should exclusively serve the goal of making the photo more candid. I feel that the AI photo editing features being pushed out these days will only make photos less candid.

I wish YouTube Premium (and honestly, Spotify too!) had a feature where I could voluntarily commit X additional dollars per month to be directly distributed to the creators I watch according to their share of my total watchtime, with some kind of manual opt-out button for individual videos/creators that I explicitly do not want to support. I am already a member of several Patreons but wish I could cast a bit of a wider support net for the people I watch enough-to-want-to-support-them-but-not-enough-to-join-their-Patreon, yknow?