HNHacker News
TopNewBestAskShowJobs

heliophobicdude

689 karma · joined December 31, 2015

GH:afloresescarcega

E-mail: alex %{dot} floresescar %{@gmail}

Curious about Gaussian Splatting.

Most recently started work on splatinit.c It initializes 3D Gaussian Splats from an image and a depth map. It's fast and outputs to a typical 3D Gaussian Splat .ply file.

https://github.com/afloresescarcega/splatinit

submissionscomments
heliophobicdude··on Apple's game porting toolkit is fantastic. Cyberpunk 2077 at Ultra on an M1 MBP
Do keep in mind this script was marketed as a way for game developers to judge the viability to port to using native apis and native isa.

I would imagine that not running through this codeweavers patch and through rosetta would have better performance.

heliophobicdude··on Apple releasing segmentation/pose for humans and animals, embedding for 27 lang
Turns out they have already [1]

https://developer.apple.com/documentation/naturallanguage/nl...

heliophobicdude··on Apple releasing segmentation/pose for humans and animals, embedding for 27 lang
I would be interested to know of a consistent, on-board embeddings model. Trying to reduce latency and dependence on API calls for simple vector database search will go a long wag
heliophobicdude··on OpenAI Employee: GPT-4 has been static since March
His role is Developer Relations fwiw
heliophobicdude··on OpenAI Employee: GPT-4 has been static since March
The models for the api's could have not changed but the web app's might have too
heliophobicdude··on OpenAI Employee: GPT-4 has been static since March
Assuming you mean the base model, no not right now. GPT-4 base model is not public.
heliophobicdude··on Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?
Supposedly the API hasn't changed.

https://twitter.com/officiallogank/status/166393494793189785...

heliophobicdude··on OpenAI isn’t doing enough to make ChatGPT’s limitations clear
This is great!

Addressing the model limitations a bit: in the demonstration data that is provided to the base model, we should prevent computed or "looked up" answers.

I've seen some of the demonstration data that people are using to train instruction-tuned models and are being taught to respond by making up answers to solutions it shouldn't try to compute. Btw, the output is wrong.

{ "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }, [1]

The UI note for now would get us very far but by filtering out demonstrations that retrieve or compute information we can go even farther.

Symbol tuning [2] is addressing the quality of demonstrations but we can take it further by removing retrievals and computations altogether.

Bonus: we can demonstrate how to make it respond so that the user/agent be informed of how to compute or retrieve.

1: https://github.com/sahil280114/codealpaca/commit/0d265112c70...

2: https://arxiv.org/abs/2305.08298

heliophobicdude··on Is Apple too quiet regarding AI?
Seems that they are hiring for a bunch of roles [1]. Also keep in mind that they release and reveal via large events. Hopefully next week they show us their AI strategy.

1:https://jobs.apple.com/en-us/search?location=united-states-U...

heliophobicdude··on How to Finetune GPT-Like Large Language Models on a Custom Dataset
Ah! That makes sense! That's a neat strategy!
heliophobicdude··on How to Finetune GPT-Like Large Language Models on a Custom Dataset
Cheers!
heliophobicdude··on How to Finetune GPT-Like Large Language Models on a Custom Dataset
One argument is that it can contaminate training data from output of itself or other models.

We have already documented evidence of the effect of this. In the GPT-4 technical report [1], they reported contamination of humaneval data in the training data.

They did measure against a "non-contaminated" training set but no idea if that can still be trusted.

Why would this matter? We can have seemingly strong benchmarks for containments but measures poorly against new and quarantined information. Classic over fitting.

Another argument is that data being put out there could very much be wrong and the amounts of it amplified by other models. Take a look at this sample of demonstration data for codealpaca [2]. Not only is its output wrong but bad practices like,making up a random computation without it having access to a place to run a calculation, teaches the model these type of responses are ok.

{ "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }

1: https://cdn.openai.com/papers/gpt-4.pdf 2: https://github.com/sahil280114/codealpaca/commit/0d265112c70...

heliophobicdude··on How to Finetune GPT-Like Large Language Models on a Custom Dataset
I think these are two very separate concepts.

What we are mostly seeing when it comes to fine-tuning is making a model promptable. Models like LLaMA or the original GPT3 weren't promptable. They were fine-tuned with demonstration data that looks like a prompt input, prompt output.

See below: { "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }, [1]

Prompt engineering is really just carefully designing what inputs and outputs on a prompt-ready model work best.

I highly recommend skimming this RLHF article and looking for the parts where it talks about demonstration data [2]

1: https://github.com/sahil280114/codealpaca/blob/master/data/c...

2: https://huyenchip.com/2023/05/02/rlhf.html

heliophobicdude··on How to Finetune GPT-Like Large Language Models on a Custom Dataset
Assuming you would want to fine-tune over a codebase or set of documents, I would argue vector databases and fine-tuning are completely different tools.

I would strongly recommend against fine-tuning over a set of documents as this is a very lossy information system retrieval system. LLMs are not well suited for information retrieval like databases and search engines.

The applications of fine-tuning that we are seeing have a lot of success is making completion models like LLaMA or original GPT3 become prompt-able. In essence, prompt-tuning or instruction-tuning. That is, giving it the ability to respond with a user prompt, llm output chat interface.

Vector databases, for now, are a great way to store mappings of embeddings of documents with the documents themselves for relevant-document information retrieval.

I would highly recommend skimming this RLHF paper for how demonstration data was used to make a model prompt-able [1]. Keep in mind RLHF is another concept all together and we might be seeing a revolution where it might become optional (thanks to LIMA)!

1: https://huyenchip.com/2023/05/02/rlhf.html

heliophobicdude··on QLoRA: Efficient Finetuning of Quantized LLMs
Yes and no.

Someone pointed out that some of the answers are using OpenAI's famous "As an AI...". Soooo you can roughly say that RLHF might still have had an impact here through the training data that came from an RLHF model.

But what we are seeing is a revolution in picking quality demonstration data that might make RLHF an optional last step for fine-tuned models.

I'm bullish on a new technique where the quality of instruction-tuned code model output are measured and evaluated automatically by a machine. I'm calling it RL<machine>F

Code Model outputs a test harness, an implementation, and how to run the implementation for a users inputs. Have RLMF evaluate implementations against the synthetic test harness and against hidden user inputs.

heliophobicdude··on Facebook parent in talks with Magic Leap over augmented reality deal
https://archive.is/KkZVQ
heliophobicdude··on Ask HN: Those with success using GPT-4 for programming – what are you doing?
Coding with LLMs was easier after understanding their limitations.

1. They don't know what they are saying until they have said it.

2. Your inputs and its outputs help make the next message.

3. LLMs are not suited for information retrieval like databases and search engines.

LLMs excel at reasoning and predicting subsequent text based on given context. Their strength lies in their ability to generate relevant and cohesive responses.

To optimize results, outline clear rules, strategies, or ideas for the LLM to follow. This helps the model craft, revise, or build upon the established context.

Starting with a precise query and introducing rules or constraints incrementally can help steer the model's output in the desired direction.

Avoid zero-shot queries as these can lead to the model generating unexpected or unrelated responses.

Be cautious while seeking pre-calculated or non-derived answers. Some instruction-tuned models might output incorrect solutions, as they are trained to respond to certain queries without proper context or information.

also, this is my biggest gripe no fault of ours of course: don't seek pre-calculated or non-derived answers. I've seen some of the demonstration data that people are using to train instruction-tuned models and are being taught to respond by making up answers to solutions it shouldn't try to compute. Btw, the output is wrong.

{ "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." },

https://github.com/sahil280114/codealpaca/commit/0d265112c70...

heliophobicdude··on The bill of materials for the Apple headset leaked
Solid state, active cooling system

https://www.froresystems.com/

heliophobicdude··on The bill of materials for the Apple headset leaked
I was wondering if Apple is the partner this solid state, active cooling system was referring to at CES 2023

https://www.froresystems.com/

heliophobicdude··on Apple dives into display-making to cut reliance on Samsung
Couple of questions. Is Apple intending to use microOLED from Samsung after buying Sony's microOLED for this upcoming AR HMD?

Also are they intending to use Samsung's new purchase of eMagine optics for the higher quality microOLED micro displays with no color filter?

If so, are they then just trying to cut reliance on specific types of displays?

Also microLED != microOLED.

heliophobicdude··on ChatGPT app for iOS
None so far! It hasn't even asked for my clipboard access.
heliophobicdude··on StarCoder and StarCoderBase: 15.5B parameter models with 8K context length
I think we need a different strategy to instruction tuning for coding LLMs.

I don't think StarCoderBase is instruction-tuned off the bat but would serve as a good starting point for a new technique.

RLHF is fine for things that are hard to measure and evaluate, but code is runnable and testable.

I propose we try Reinforcement Learning Machine Feedback or RLMF.

Prompts and responses are evaluated by how accurate the response evaluated to. We can then train a reward model to help refine StarCoder.

heliophobicdude··on StarCoder and StarCoderBase: 15.5B parameter models with 8K context length
Refined training is usually updating the weights of usually what's called a foundational model with well structured and numerous data. It's can be expensive but most importantly disturb the usefulness of having all the generalizations baked in from training data [1]. While LLMs can generate code based on a wide range of inputs, they're not designed to retrieve specific pieces of information in the same way that a database or a search engine would. It's just very lossy. Perhaps it wouldn't be the best use for single code base fine tuning right now.

Can you please share more about the merging context across levels? This sounds interesting!

1: "Language Models are Few-Shot Learners" Brown et al. https://arxiv.org/pdf/2005.14165.pdf

heliophobicdude··on Anthropic's 100k context is now available in the web UI
Perhaps. I don't have those issues from the direct account I have with them.
heliophobicdude··on Ask HN: What's your favorite GPT powered tool?
I've been using the api in S-GPT [1]. An iOS shortcut that I use to primarily summarize the body text content of a webpage I'm on from the share-sheet. When it can fit the text in the context, I'm pretty happy with the the results.

It also supports sending it your clipboard contents when you launch it and it parses the words "clipboard". Good for when in a pinch.

I mapped some repetitive prompts to my text replacement on my phone that auto-expand when I type in the input field `*<some shortcut here>`

1: https://www.macstories.net/ios/introducing-s-gpt-a-shortcut-...

heliophobicdude··on Apple to occupy 90% of TSMC 3nm capacity in 2023
What are some broken-down reasons why gaming on Mac didn't take off. I'm sure it's a few different reasons all together
heliophobicdude··on AR Demands Peripherals
I want touch-type on a physical keyboard at my arms and hands resting position. I want pocket keyboards for AR for these reasons below:

Allows you to input data without needing to look at the keyboard.

The positioning of the keyboard wherever your hands and arms rest comfortably, improving ergonomics.

Haptic Feedback

Multi-tasking. Comfortable typing while not disrupting my AR experience

heliophobicdude··on PaLM 2 Technical Report [pdf]
Humaneval needs careful consideration though.

In the GPT-4 technical report, they reported contamination of humaneval data in the training data.

They did measure against a "non-contaminated" training set but no idea if that can still be trusted.

https://cdn.openai.com/papers/gpt-4.pdf

heliophobicdude··on It looks like GPT-4-32k is rolling out
LoRA probably does not affect the models biggest bottle neck. The attention mechanism. Original transformer was O(n^2d) where n is the query length and d was the cardinal of all the tokens
heliophobicdude··on It looks like GPT-4-32k is rolling out
Can you please elaborate on how you prune?
← PreviousPage 3 of 6Next →