bradfox2

I don't have the arxiv link bookmarked, but there was a paper written on pretraining with Lora. It involved merging adapters back every n steps with good results.

We own and operate 40+ data center GPUs (v100s, a100s, and ax000s) in a private cluster and use vast to rent unused capacity.

What would make you better than vast is extremely easy spot leasing and job prioritization.

I want to be able to have one of our training jobs finish, and then have the capacity immediately transition to a lease. With vast, we are renting in week long blocks.

The only experience I have is first hand, what my company is doing for our client base. We are doing continuous pretraining and the rest of the alignment stack training on about 10B private tokens + private customer data to produce private custom models for companies in the 500 to 3000 employee range. We built and operate a single rack cluster that cost mid 6 figures in order to be able to do this.

These models get combined with rag for highly specific technical doc authoring and other uses.

Qlora won't work well to add knowledge via private data.

Parameter efficient methods are not useful for these cases at the 8b scale without a more complex training procedure that periodically merges back adapters. Maybe at the 70B scale.

Our 2013 carpet does it, very annoying. Happens under the accelerator pedal.

Probably to avoid being arrested and/or retain the ability to pass a background check while finding alternate employment in the future.

Then they should quit if their personal beliefs no longer align with the company. The expectation that they can do anything outside of working for the company on company objectives while on company time and not face consequences is insane to anyone that's worked at a non-SV tech company.

What is a 'real' vacation?

If it's laying on the beach for a week in Hawaii, or traveling through France, those are luxuries that should not be annual expectations.

Vacation just means having time to periodically not work for a living. Anything above that historically has been earned as a rare experience, but today is unrightfully expected.

Like OP, we went camping or to visit extended family once a year. Instagram vacations to the beach were what the rich families did.

So what? Plenty of model trainers are fine tuning to remove all the alignment and bias crap anyway. The concept of 'jailbreak' to a freely distributed open source model doesn't really apply.

You are looking for Kairos. They are iterating and succeeding. Most of the hurdles are not engineering but licensing and regulatory.

There's a long gh issues thread with technium struggling with Mistral 7 and loss spikes. Easy to find googling.

My startup hosts our own training servers in a colo-ed space 10 min from our office. Took less than 40 hours to get moved in, with most of the time tinkering with fortigate network appliance settings.

Cloudflare zero trust for free is a huge timesaver

I think he means not needing to have a great search system to identify rag chunks. Just throw everything in.

Tesla isn't janky compared to a Chinese EV with no reliable charging infrastructure.

You want a specialized trade. HV Electrician at a utility or something like that.

An off the shelf commercial or residential trade sounds like a nightmare, especially in an area with bad weather part of the year.

That said, if you can deal with employees of that type as a business owner, you can clean up.

We've been daily driving tesla exclusively since 2015.

Too much uncertainty in a new area to rent one when on business travel with strict deadlines.

This is just poor anti car attitude infecting actual judgement of the technology.

Waymo doesn't require a human behind the wheel. They roam all over metro Phoenix without drivers and without issues.

Tesla FSD works fine with appropriate monitoring on city streets and virtually autonomously on freeway driving. Comma and open pilot aren't far behind.

The US is built for cars and thinking that is going to change is a pipedream.

Whistlindiesel

Are the 30x claims comparing full update training vs qlora/4bit weights?

These are significant claims locked behind a paid 'pro' version. Red flags.

Have you ever used either of the connectors? The advantage of NACS with water cooled cables is very obvious.

This is insane when react query is available and so easy to use.

This likely has a tangent to quantization research from Dettmers et al that corresponds to outlier weight values in models >6B params. Might allude to why the outliers emerge.

4 miles per hour or so if you are referring to a nema 5-15.

Someone else had the boxes, but that is what we were using!

Have you ever connected with the folks behind XBConnect?

I played competitive HaloCE through 3 at a professional level. XBConnect is what people playing competitive CE used prior to halo 2 and xbox live. Game was playable up to about 60ms, with <40 greatly preferred. Hosts had a large advantage, but off host was very predictable, and required leading the targets.

The same advantage existed on LAN, with competitive tournaments setup to trade host advantage for team color advantage. The last in person CE lan I attended, we were playing on modded Xboxes with a patched version of the game that allowed a third box as a neutral host, and added an on screen timer. Happy to expand on the details if anyone is interested.

You really want gpu rdma though. It's a bit of a pain to get setup but it's worth it.

100gbe mellanox connect x cards are not actually that expensive though.

My MSE dept was also colocated with mining engineering. Evolved from the metallurgy dept of a past era. We seemed to have a foot in manufacturing, computational materials, theory and chem lab.