tmabraham

Sophont Inc. | Founding Research Engineer / Scientist | Remote | Full-time

Sophont is a public-benefit corporation building open multimodal foundation models for medicine. Joining Sophont means you are paid to lead and contribute to high-impact medical AI research ultimately aimed to transform healthcare and life sciences.

We are looking for exceptional, high-agency ML research engineers who are passionate about using AI to build the future of healthcare. No previous background in medical AI is needed!

If you’re interested in working on highly impactful AI projects that could have the potential to transform and save many lives, Sophont is the company to work at.

## Roles

Founding Research Engineer, LLM Team - https://sophont.med/job_postings/llm Founding Research Scientist, Vision Team - https://sophont.med/job_postings/vision Founding Research Scientist, Miscellaneous - https://sophont.med/job_postings/misc

How to apply:

Email resume to hiring@sophontai.com

I'm the co-founder of Sophont, happy to answer any questions!

it was already planned for open-sourcing, the leak did not affect the plans in any way

Nicolas Bourbaki is the name of a character in twenty one pilots' concept album

"He'll always try to stop me, that Nicolas Bourbaki He's got no friends close, but those who know him most know He goes by Nico"

It is indeed inspired by the actual pseudonym, not just a pun.

The paper has preliminary results for video as well

Because you can take advantage of pretrained CNNs and perform transfer learning, which is significantly more data-efficient than training from scratch, which is what you'd likely have to do with raw digital signals. This paper is not unique in this approach and many papers have obtained SOTA results by processing digital signals as images.

Yeah using data from a 7T MRI giving higher spatial resolution definitely helps!

The fMRI dataset includes signal from the whole brain but we only use the data from the visual cortex for this study.

Our model generates CLIP image embeddings from fMRI signals and those image embeddings can be used for retrieval (using cosine similarity for example) or passed into a pretrained diffusion model that takes in CLIP image embeddings and generates an image (it's a bit more complicated than that but that's the gist, read the blog post for more info).

So we are doing both reconstruction and retrieval.

The reconstruction achieves SOTA results. The retrieval demonstrates that the image embeddings contain fine-grained information, not just saying it's just a picture of a teddy bear and then the diffusion model just generates a random teddy bear picture.

I think the zebra example really highlights that. The image embedding generated matches the exact zebra image that was seen by the person. If the model only could say it's just a zebra picture, it wouldn't be able to do that. But the model is picking up on fine-grained info present in the fMRI signal.

The blog post has more information and the paper itself has even more information so please check it out! :)

Yes we've been looking into ControlNet as well, and I think there is one recent fMRI-to-image paper that also has tried ControlNet. Maybe we'll use ControlNet in MindEye v2 :)

We think it could be useful for clinical research and maybe even diagnostics. For example, you could imagine a person with depression(or other neurological disorders) may have a different perception of the same image than a healthy person. Now with the much higher fidelity that both more powerful MRI machines and better generative AI tools can provide, this may now be a very promising direction for future research.

I use GitHub Copilot very frequently for any programming I do.

I also use BingChat very often, I consider it to be a very underrated tool that people haven't fully explored. Of course, it's great for when you want your LLM queries augmented with search results. But BingChat can also view the current webpage, so for example you can pull up papers and ask BingChat questions about it. I've used it to help write abstracts for my papers. When I do this, BingChat literally searches up "how to write an abstract" which I found absolutely hilarious , it's like it's learning skills on the go as well. I didn't use the outputs directly, but it served as useful inspiration. I think BingChat uses a mix of models including GPT-4, all for free! Apparently more features are coming soon, including multimodal features.

Okay, but are you a diffusion model/generative AI researcher?

As a researcher in this field, I found the article fairly understandable and accessible. I suspect that was the target audience.

The domain of diffusion models have a lot of terminology and concepts that are not explained here, so maybe that's why it's harder to understand?

It isn't like Stable Diffusion, it's more like Google's Imagen model.

Three of the authors of this paper work for Stability AI currently.

One of them even said: "Unfortunately we cannot release the weights. That's why I joined @StabilityAI to work on OS video models"

- https://twitter.com/andi_blatt/status/1648598423526932483

There are plans for an LLM course but very early ideas, stay tuned!

If you're interested in taking this course, you can check the forums and/or Discord where people will organize study groups going through the course. Some of the study groups organized are in-person, some are virtual. These are usually great opportunities to study with some real people at the same time and get that camaraderie.

https://forums.fast.ai https://discord.gg/YHtEBwzV

I have worked on the course with @jph00 and it's been an incredible experience! This is one of the most cutting edge courses about deep learning and diffusion models. Highly recommend folks check it out!

Link for more info: https://www.fast.ai/posts/part2-2023.html

I have worked on the course with @jph00 and it's been an incredible experience! This is one of the most cutting edge courses about deep learning and diffusion models. Highly recommend folks check it out!

the actual platform is called "HuggingFace Hub". The company itself is called "HuggingFace" or "Hugging Face" (I have seen it referred to in both ways, I am unsure which is officially correct). There is no namespace collision.

Hello everyone, I'm Tanishq, the founding CEO of MedARC. MedARC is a new research organization for medical AI research. We are interested in building foundation models for medical AI and building interdisciplinary teams to solve clinical problems. We have a couple of ongoing projects right now (fMRI-image reconstruction and fine-tuning Stable Diffusion for Chest X-rays) and more projects to be announced soon. Check our website for more details. You can also join our Discord:

https://discord.com/invite/CqsMthnauZ

Happy to answer any questions!

> 4) their own large language model LaMDA, that is apparently so good it convinced a senior AI researcher that it was sentient

Oh come on, that guy would be convinced that ELIZA is sentient

> 5) researchers that invented Transformers that GPT is modeled after Only one researcher from that paper remains at Google, the rest have gone on to work at or found their own startups. The so-called "Google Brain drain" is certainly a concern (although there still are many great researchers there).

No it hasn't.

I love lucidrains and he's a bona-fide genius, but I feel obligated to point out this is just code for training a ChatGPT-like model and not an actual trained model you can use.

Mixture of expert denoisers (basically multiple diffusion models for different noise levels rather than a single model for the entire diffusion process) might become the next big thing in diffusion model literature: https://twitter.com/iScienceLuvr/status/1587973173932195840

HuggingFace is a company that mainly builds open-source libraries and platforms to support open-source ML projects. They started out with their famous Transformers library and have many other libraries including the diffusion model one that is actually what this application here is using. They also have this model/dataset hub and interactive application platform known as "Spaces". Their goal is to be the "GitHub of machine learning".

Their business model is basically supporting enterprise and private use-cases. For example, getting expert support for using these libraries, or hosting models and datasets privately. You can see more information about the pricing here: https://huggingface.co/pricing

They reached a $2 billion valuation after a recent round of funding so overall they're probably pretty flush with cash lol

Good question!

I think a major reason for this is because of transfer learning. For computer vision, there are many good pretrained models that were trained on huge datasets (like ImageNet) that can be fine-tuned for custom tasks. Other fields often do not have such pretrained models and huge datasets to work on, so it turns out transforming a dataset into an image dataset and fine-tuning a pretrained model works better than training from scratch.

I don't think you understand what is going here...

None of your data is going to Salesforce.

All this video is showing is how to download and use an already trained model by the Salesforce Research team...

The link submitted here is Woit's link.

Mathpix is great, but they have become much more restrictive.

I had an old version of this software which I think allowed for unlimited snips but now I am limited to 100 for the free version. Plus this is also a closed-source proprietary product. I will look into the open-source alternative linked in this thread.

He left Discord lol