HNHacker News
TopNewBestAskShowJobs

amilios

650 karma · joined January 14, 2014

submissionscomments
amilios··on Stable Diffusion XL 1.0
I always wondered why the vision models don't seem to be following the whole "scale up as much as possible" mantra that has defined the language models of the past few years (to the same extent). Even 3.5 billion parameters is absolutely nothing compared to the likes of GPT-3, 3.5, 4, or even the larger open-source language models (e.g. LLaMA-65B). Is it just an engineering challenge that no one has stepped up for yet? Is it a matter of finding enough training data for the scaling up to make sense?
amilios··on Llama 2
Yeah unfortunately I think this is the result of the stochasticity of sampling from the LLMs with non-zero temperature, it'll give a different answer every time and some answers might trigger the guardrails and others might not. I am curious if the greedy-sampling answer contains the guardrails or not...
amilios··on Llama 2
Interesting. It worked for me!

Response:

Early stage venture capital firms typically have a few non-investor roles that are critical to their operations. Here are some of the most common non-investor roles at early stage venture capital firms:

Entrepreneur-in-Residence (EIR): An EIR is a seasoned entrepreneur who works closely with the venture capital firm to help identify, develop, and support startups. They often provide guidance and mentorship to the startups and help them navigate the challenges of building and scaling a business. Venture Partner: A venture partner is a seasoned professional who works closely with the venture capital firm to identify, evaluate, and invest in startups. They often have a deep understanding of the industry and market trends and can provide valuable insights to the firm.

Investment Analyst: An investment analyst is responsible for evaluating potential investments, conducting due diligence, and providing recommendations to the investment team. They often have a strong financial background and are skilled at analyzing data and identifying trends.

Marketing and Communications Manager: A marketing and communications manager is responsible for promoting the venture capital firm's brand, building relationships with the media and other stakeholders, and creating content that showcases the firm's investments and expertise.

Business Development Manager: A business development manager is responsible for identifying and pursuing new business opportunities for the venture capital firm. They often work closely with the investment team to identify potential investments and build relationships with entrepreneurs and other key stakeholders.

Portfolio Manager: A portfolio manager is responsible for managing the venture capital firm's investments and providing support to the startups in the portfolio. They often work closely with the entrepreneurs to help them navigate the challenges of building and scaling a business.

Operations Manager: An operations manager is responsible for managing the day-to-day operations of the venture capital firm. They often work closely with the investment team to ensure that the firm's investments are properly managed and that the firm's operations run smoothly.

These are just a few examples of the most common non-investor roles at early stage venture capital firms. Each role plays a critical function in supporting the firm's investment activities and helping to build successful startups.

amilios··on The Rise of the AI Engineer
Personally I think it's a combination of

a) much better resources being available than in the past, which explain concepts clearly and in relatively simple terms, and

b) much better tooling existing now than ever before, so many things that you used to need to do "by hand" are now taken care of by relatively standardized tooling. Even just automatic differentiation engines are a huge deal, not having to backprop "by hand" (first release of TensorFlow was 7 years ago, so if you looked at it a decade ago I'm assuming you had to implement backprop yourself). Beyond that another jump was from Tensorflow's delayed execution/compilation model (it really was a headache to work around its APIs/set up the computation graph) as well as just having a generally ugly API, to PyTorch's "literal" setup, where it feels like you're just writing regular Python code and performing operations "normally"

amilios··on Sail 7B: New Fine Tuned LLM Outperforms ChatGPT and Vicuna with Search
At this point a lot of people use "ChatGPT" to refer to the ChatGPT model with the GPT-3.5 backend
amilios··on Ask HN: What's the best self hosted/local alternative to GPT-4?
Is this with the 4-bit quantization? The only issue with it is that inference is incredibly slow with it on right now, but that should be fixed up in the next few weeks I think.
amilios··on Ask HN: What's the best self hosted/local alternative to GPT-4?
How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora. But as other comments mention, it's still noticeably worse in my experience.
amilios··on QLoRA: Efficient Finetuning of Quantized LLMs
It probably does have some sort of simplified internal model of the world, who's to say it doesn't?
amilios··on QLoRA: Efficient Finetuning of Quantized LLMs
Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.
amilios··on A Winelike Sea
The phrase in the original homeric greek is "οίνωψ πόντος". "οίνωψ" is a combination of "οίνος" = wine, and ωψ = "view/look" (related to the modern όψη). So it's not wine-"colored", per se, but it is specified to be "wine-like", specifically in a visual sense. to be fair, this could be other qualities than color, e.g. as others mentioned, perhaps opaqueness.
amilios··on There is no A.I.
You want a plausible scenario for destruction via AI? Here's one. Some idiot governments put a neural network behind missile defense or nukes. Because ANNs are uninterpretable black boxes, at some point the system does a counterdefense against nothing or grossly overreacts to some stimulus. Tada, absolute destruction and chaos. This might seem far fetched, but is it really that out-there of a scenario?
amilios··on The Last Week in AI
https://www.reddit.com/r/OpenAI/comments/12696oq/im_dating_a... This one is wild. I expected that Black Mirror episode to become real eventually, but definitely not so soon...
amilios··on OpenAI Tokenizer
Yes, the model has something like an EOF token which it emits for the output to end. It is part of the probability distribution that the model predicts.
amilios··on The teen mental illness epidemic is international – Part 1: The Anglosphere
Compare housing costs in the 1970s vs now.
amilios··on The teen mental illness epidemic is international – Part 1: The Anglosphere
(Seemingly) Inevitable climate change and ecological destruction, growing income inequality and falling standards of living, a deathly pandemic that's still killing people while we pretend everything is back to normal (+ long covid probably bringing about a disability crisis in 3-5 years), increasing tensions between various nuclear-capable states (Russia, China) and the West, changes in the age distribution in most western states meaning that most young people will probably work until they die (retirement? lol) etc. etc.

This is a multi-faceted issue and let's face it, part of the problem is that the world is just getting depressing as hell for young people who see less and less of a future for themselves.

amilios··on The teen mental illness epidemic is international – Part 1: The Anglosphere
eating disorder
amilios··on Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models
Comparing the 13B model here https://huggingface.co/cerebras/Cerebras-GPT-13B to LLaMA-13B https://github.com/facebookresearch/llama/blob/main/MODEL_CA... you can see that in all of the reasoning tasks Cerebras-GPT lags behind. Any reason to use Cerebras instead of LLaMA? Doesn't seem like it.
amilios··on GPT-4
We don't know, OpenAI refused to publish any details about the architecture in the technical report. We don't know parameters, we don't know depth, we don't know how exactly it's integrating image data (ViT-style maybe?), we don't even know anything about the training data. Right now it's a giant black box.
amilios··on OpenAI is now everything it promised not to be: closed-source and for-profit
This is mostly true, yeah. Although in certain cases OpenAI is indeed pushing research forward themselves (not only using existing algorithms/architectures just at scale) and therefore has first mover's advantage, e.g. with reinforcement-learning-based fine-tuning based on human preferences (Reinforcement Learning from Human Feedback, RLHF), which is basically the secret sauce which turned the original relatively-dumb GPT-3 (pure language model, "document autocomplete", what is the next most likely token based on my training data) into ChatGPT (what token would a human prefer to come next).
amilios··on Sea life bounced back fast after the ‘mother of mass extinctions’
OK sure life won't cease as a whole for humans, but living standards for the vast majority of us will enter freefall and stabilize at a MUCH much lower level than we're used to.
amilios··on Sea life bounced back fast after the ‘mother of mass extinctions’
I think the issue is feedback loops. You're right that eventually governments will be "forced" to do the "right thing", but the problem is that it is likely going to be too little too late, and the climate feedback loops already set in motion may indeed "finish the job" and make the planet too inhospitable for any humans to feasibly live on.
amilios··on Graph showing level of Covid, flu and RSV in US wastewater systems, by city
This is an absolute myth. There is no evidence of "immunity debt", and serious public officials parrotting this nonsense are doing real damage. It's a tempting explanation because it's simple and seems to make intuitive sense, but it's just wrong. The explanation that has much more evidence is that everyone and their dog caught COVID, and COVID has been directly linked to lasting immune system damage, making everyone more susceptible to catching these other viruses.

https://globalnews.ca/news/9272293/immunity-debt-covid-19-mi...

amilios··on Graph showing level of Covid, flu and RSV in US wastewater systems, by city
Unfortunately I'm quite sure it's the latter. Even in Canada, in MTL where I live, on public transit it's maybe 5-6% that is masking. For most people the pandemic is simply "over", regardless of what the data says or how overwhelmed hospitals are.
amilios··on How does GPT obtain its ability? Tracing emergent abilities of language models
It's not, not in the ChatGPT web interface at least. It's just a chat prompt, that's it. Can't control any sampling parameters at all, not temperature, nor top-P or top-K-based selection, nothing. I guess OpenAI is aiming to keep things simple for now. And to be fair, this choice probably does open the demo up to more users who would otherwise be confused at all the knobs and dials. Although they could just hide it by default or something
amilios··on Ask HN: How does ChatGPT work?
In the same way that it predicts a given token like 'we' or 'write', it predicts a special end-output token, something analogous to 'EOF'. So at some point as it's generating new tokens and coming up with probability distributions for the next token (conditioned on its own output so far), selecting from this distribution (either by greedily picking the highest probability token or sampling), it will select this 'EOF'-style token, and end its own output. Does that make sense?

So to answer your question about knowing how much text it has output so far, not exactly. It is not 'stateful' exactly, it's simply conditioned on previous text and gives a new probability distribution over its vocabulary for the next token. Whether the 'previous text' is its own output, or provided by a human user (a 'prompt'), it isn't necessarily aware of this by default (unless the user vs. system responses are delineated somehow with special tokens for example, which with ChatGPT they may be/are likely to be. My point is that these models don't come with this awareness built in, you have to add it. Fundamentally they simply condition their probability distribution on a piece of existing text).

amilios··on People tricking ChatGPT “like watching an Asimov novel come to life”
The "knowing what it doesn't know" problem is trickier even than it might seem, too: right now we don't even know how what it does know is encoded in its neurons, other than some vague notion that the knowledge is somehow "distributed" across a ton of distinct neurons (there is no specific neuron/set of neurons that are dedicated to memorizing X or Y fact), so figuring out what it doesn't is not trivial at all
amilios··on Ticketmaster's tech problem isn't a tech problem
The concern with Ticketmaster is the monopoly though, you are sidestepping the issue. It's about what kind of value Ticketmaster adds (minimal) to justify the exhorbitant fees they charge and can charge, because they have no competition due to all their exclusive contracts with the venues.
amilios··on Smolnes: A NES emulator in less than 5000 significant bytes of C++
Sometimes I think I'm good at programming, then I find something like this and I'm put back in my place.
amilios··on AI translates Hokkien, an unwritten language, for the first time
It's written, it just doesn't have a standardized written form, it's written informally in standard Chinese character but it's modified from the spoken form in order to conform better to standard Mandarin forms, AFAIK. From Wikipedia:

"if "pure, unadulterated spoken vernacular Taiwanese" were written exclusively in Chinese characters, with minimal use of Mandarin phrases, over 25% of morphemes would have no character, about 25% would have arbitrarily selected (yet more or less conventionally accepted) characters that are homophones or near-homophones, 10% would be written using characters exclusive to Hokkien, and 40% would be written with characters that have the correct sound and meaning. However, in more colloquial styles of Taiwanese Hokkien, the proportion of morphemes written with conventionally accepted characters would drop even lower than 40%."

amilios··on Amazon “suicide kits” have led to teen deaths
The article says that the Amazon recommendation algorithm was also recommending along with the sodium nitrite, "a scale to measure a lethal dose, a drug to prevent vomiting, and a book with instructions on how to use the chemical to attempt suicide". This is beyond just suggesting the chemical itself, this is almost suggesting to certain users to commit suicide.
← PreviousPage 4 of 7Next →