Thanks, I’m actually pretty far in the interview process with a bunch of companies now but would be open to learning more.
My contact info is in my bio.
4,703 karma · joined February 20, 2012
My contact info is in my bio.
A ton of big tech companies opened large offices in Poland in the past few years. It’s starting to look like a pretty major tech hub. If you look at Google or Box job boards most of their open roles in Europe are in Warsaw.
I’m interviewing with like 10 companies right now and it feels like a full time job, I have full days of interviews lined up for almost every day in the first 3 weeks on January.
I have an Ivy league degree, worked in deep learning since alexnet at a leading startup in the space, was a CTO of a startup that got acquired and have referrals from very senior people at the top FANG companies and still struggled to get interviews.
I also have research scientist friends with Neurips papers, ones that solved long standing open math problems and even they are struggling to get hired.
What me and my friends heard from a lot of people at the large companies was that many of them are no longer hiring in the US, but in India, Poland and Brazil instead, and that the roles they have listed in the US are for internal transfers. I've had a referral for Google for months and did not get an interview for NYC based roles, but when I went to an ML conference in Warsaw a few months ago I learned that Google is looking to hire 2000 people there, but with people in that office making ~1/4th of my friends in the US.
On top of that you have a huge pool of bootcamp grads and foreign applicants so any role posted gets 1000s of applications in the first few hours, making it impossible for recruiters to look over all of them.
And if that wasn't enough we're going through a huge hiring downturn post the COVID bump, see: https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE
Things will get even worse as scammy companies start flooding the web with LLM generated content pushing their products to bias LLMs to increase the probability of outputing their name for keywords related to their business.
Strong organizations are usually bottom up, with a lot of ownership and direct contact between people doing the work and ones steering the ship.
I haven't worked in medical imaging in a while but VLMs make for much better diagnostic tools than task specific classifiers or segmentation models which tend to find hacks in the data to cheat on the objective that they're optimized for.
The next token objective turns our to give us much better vision supervision than things like CLIP or classification losses. (ex: https://arxiv.org/abs/2411.14402)
I spent the last few years working on large scale food recognition models and my multi label classification models had no chance of competing with GPT4 Vision, which was trained on all of the internet and has an amazing prior thanks to it's vast knowledge of facts about food (recipes, menus, ingredients and etc).
Same goes for other areas like robotics, we've seen very little progress outside of simulation up until about a year ago, when people took pretrained VLMs and tuned them to predict robot actions, beating all previous methods by a large margin (google Vision-Language-Action models). It turns out you need good foundational model with a core understanding of the world before you can train a robot to do general tasks.
I also wasted a lot of time building complex OCR pipelines that required dewarping / image normalization, detection, bounding box alignment, text recognition, layout analysis, etc and now open models like Qwen VL obliterate them with an end to end transformer model that can be defined in like 300 lines of pytorch code.
In all cases the models trained on a lot of this feedback were more consistent and accurate than individual expert annotators.
These were professional radiologists with years of experience and still came to different conclusions for fairly common conditions that we were trying to detect.
So yes, LLMs will make mistakes, but humans do too, and if these models do so less often at a much lower cost it’s hard to not use them.
We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible. The outcome of this venture is uncertain and the work is difficult, but we believe the goal and the structure are right.”
…
“ We’re hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies.”
o3 is just o1 scaled up, the main takeaway from this line of work that people should walk away with is that we now have a proven way to RL our way to super human performance on tasks where it’s cheap to sample and easy to verify the final output. Programming falls in that category, they focused on known benchmarks but the same process can be done for normal programs, using parsers, compilers, existing functions and unit tests as verifiers.
Pre o1 we only really had next token prediction, which required high quality human produced data, with o1 you optimize for success instead of MLE of next token. Explained in simpler terms, it means it can get reward for any implementation of a function that reproduces the expected result, instead of the exact implementation in the training set.
Put another way, it’s just like RLHF but instead of optimizing against learned human preferences, the model is trained to satisfy a verifier.
This should work just as well in VLA models for robotics, self driving and computer agents.
Doing this allows them to negotiate a lower price from the manufacturer or get the manufacturer to adjust the ingredients to lower the COGS. So either it increases their own margins or allows them to have lower prices than the competition.
EDIT: to make it easier here's a list of actions from perplexity:
Here are more explicit actions taken by the FTC under Lina Khan's leadership:
Lawsuit against Amazon (2023): The FTC filed a landmark antitrust case accusing Amazon of monopolistic practices in its online marketplace and Prime subscription service.
Meta (Facebook) lawsuit (2023): The FTC sued Meta to block its acquisition of virtual reality app maker Within Unlimited, citing concerns about monopolization in the VR market.
Microsoft-Activision merger challenge (2023): The FTC attempted to block Microsoft's $69 billion acquisition of Activision Blizzard, though it ultimately failed.
Kroger-Albertsons merger: A U.S. district court judge ruled in favor of the FTC to block the proposed $25 billion merger between these two major supermarket chains
Nvidia's acquisition of Arm: The FTC sued to block this merger, though it's not explicitly mentioned in the search results
Amazon's acquisition of iRobot: While not explicitly mentioned in the search results, this is another high-profile merger that the FTC has challenged under Khan's leadership.
Enforcement against data brokers (2022-2023): The FTC took action against several data brokers for selling precise geolocation data that could be used to track people's movements.
Zoom settlement (2021): The FTC finalized a settlement with Zoom over allegations of deceptive security practices.
Right to Repair initiative (2021): Khan's FTC unanimously voted to ramp up law enforcement against repair restrictions that prevent small businesses, workers, and consumers from fixing their own products.
Made in USA labeling rule (2021): The FTC finalized a new rule cracking down on marketers who make false, unqualified claims that their products are Made in the USA.
Penalties for fake reviews (2022): The FTC imposed multi-million dollar penalties on companies for using fake reviews and suppressing negative reviews.
Action against "dark patterns" (2021-2023): The FTC has taken action against companies using deceptive design practices known as "dark patterns" to trick consumers.
Increased use of Penalty Offense Authority: The FTC has revived its Penalty Offense Authority to seek civil penalties for violations of FTC administrative orders.
Ban on hidden junk fees: The FTC announced a rule requiring companies to show full prices for items like hotel rooms, concert tickets, and sporting events upfront, rather than hiding fees until the end of the checkout process
Changes to merger review process: The FTC has altered principles, practices, and policies of merger review that had been in place for decades
Expanded scope of enforcement: The FTC has taken a more holistic approach to identifying harms affecting workers, independent businesses, and consumers, with a focus on addressing power asymmetries and unlawful practices
Rulemaking changes: Chair Khan has orchestrated wholesale changes in FTC rulemaking practices and policies
Proposed ban on noncompete clauses: The FTC has proposed banning noncompete clauses in employer agreements
Increased focus on data privacy: The FTC has sued multiple companies for allegedly sharing customer data and warned about the "hidden impacts" of advertising tools like third-party tracking pixels
2. I'm sure LLMs are already way better at detecting vulnerabilities than the average engineer. (when asked to do so explicitly)
Old school desktop software takes very little maintenance. Once you get rid of user tracking, AB testing, monitoring, CICD pipelines, microservices, SOC, multi tenant distributed databases, network calls and all the other crap things get pretty simple.
A ton of huge business full of Sr Principal Architect SCRUM masters are about to get disrupted by 80 line ChatGPT wrappers hacked together by a few kids in their dorm room.
Yes I've been a software engineer working in deep learning for over 10 years, including as an early employee at a leading computer vision company and a founder / CTO of another startup that built multiple large products that ended up getting acquired.
> I apologize for misinterpreting what you were saying -- I was clearly taking "for the cost of an hour of a software engineer" to mean something that you didn't intend.
I meant that unlike a software engineer, the LLM can do a lot more iterations on the problem given the same budget. So if your boss comes and says build me new dashboard page it can generate 1000s of iterations and use a human aligned reward model to rank them based on which one your boss might like best. (that's what the test time compute / sampling at inference does).
> This is not relevant to our discussion. Again, I'm reasonably sure that I'm not going to be convinced by any research demonstrating that X new tech can increase Y metric by Z%.
These are not just research papers, people are reproducing these results all over the place. Another example from a few minutes ago: https://x.com/DimitrisPapail/status/1868710703793873144
> This is much more relevant to our discussion. Do you honestly feel this is an accurate representation of how you'd define the requirements for the pipeline you outlined in your post above? Keep in mind that we're talking about having LLMs work on already-existing large codebases,
I'm saying this will be solved pretty soon, working with large codebases doesn't work well right now because last years models had shorter context and were not trained to deal with anything longer than a few thousand tokens. Training these models is expensive so all of the coding assistant tools like cursor / devin are sitting around and waiting for the next iteration of models from Anthropic / OpenAI / Google to fix this issue. We will most likely have announcements of new long context LLMs in the next 1-2 weeks from Google / OpenAI / Deepseek / Qwen that will make major improvements on large code bases.
I'd also add that we probably don't want huge sprawling code bases, when the cost of a small custom app that solves just your problem goes to 0 we'll have way more tiny apps / microservices that are much easier to maintain and replace when needed.
As an example, huggigface just posted an article showing this for math, where with some sampling you can get a 3B model to outperform a 70B one: https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling...
Formalizing the criteria is not as hard as you're making it out to be. You can have an LLM listen to a conversation with the "customer", ask follow up questions and define a clear spec just like a normal engineer. If you doubt it open up chatGPT, tell it you're working on X and ask it to ask you clarifying questions, then come up with a few proposal plans and then tell it which plan to follow.
Here's a recipe for a human level LLM software engineer:
1. Pretrain an LLM on as much code and text as you can (done already)
2. Fine tune it on synthetic code specific tasks like: (a) given a function, hide the body, make the model implement it and validate that it's functionally equivalent to the target function (output matching), can also have an objective to optimize the runtime of the implementation (b) introduce bugs in existing code and make the LLM fix it, (c) make LLM make up problems, write tests / spec for it, then have it attempt to implement it many times until it comes up with a method that passes the tests, (d-z) a lot of other similar tasks that use linters, parsers, AST modifications, compilers, unit tests, specs validated by LLMs, profilers to check that the produced code is valid
3. Distill this success / failure criteria validator to a value function that can predict probability of success at each token to give immediate reward instead of requiring full roll out, then optimize the LLM on that.
4. At test time use this final LLM to produce multiple versions until one passes the criteria, for the cost of an hour of a software engineer you can have an LLM produce millions of different implementations.
See papers like: https://arxiv.org/abs/2409.15254 or slides from NeurIPS that I mentioned here https://news.ycombinator.com/item?id=42431382
Long context is practically a solved problem and there's a ton of work now on test time reasoning motivated by o1 showing that it's not that hard to RL a model into superhuman performance as long as the task is easy / cheap to validate (and there's works showing that if you can define the problem you can use an LLM to validate against your criteria).