HNHacker News
TopNewBestAskShowJobs

gas9S9zw3P9c

852 karma · joined April 27, 2020

submissionscomments
gas9S9zw3P9c··on Ask HN: What are the Implications of GPT-3?
Preface: I'd wait a few months before declaring it a game changer. There currently is a huge OpenAI marketing campaign going on touting how amazing GPT-3 is. Almost all the GPT-3 press comes from OpenAI's friends and network of YC founders who got privileged access and promised to build apps on top of the new API. You can see that by looking at the most popular tweets and posts. It's usually ex-YC people who are now working on new GPT-3 powered apps. Their examples and demos are clearly cherry picked. It's the typical Silicon Valley VC pitch deck smoke demo approach that you would expect from SV startup people. Let the hype and marketing campaign settle down.

That being said, if it all works as promised and the model becomes widely available it's quite amazing and has the potential to change a few things. The obvious one is that it becomes much easier to automatically create plausible-looking content such as news stories, comments, etc. This will create many more bots and spam than you are used to seeing. The other obvious one is to act as a natural-language based search engine or database, where you can ask questions and get facts as answers. This would be restricted to non-subjective things that are in the training data of course.

On a longer time scale, it could drive the adoption of technologies around fact and identify verification. As it becomes so much easier to automatically generate content, we need to better ways to establish trust. Safety and bias is yet another, since anything generated will obviously be biased to whatever is most common in the training data.

gas9S9zw3P9c··on Ask HN: What is the next hype train after blockchains and AI/ML?
My bet is on large-scale databases and data engineering. AI is hyped, but the sad truth is that a lot of the recent advances come from bigger models and making use of more data. Historically the same thing has happened a few times [0]. It's no different this time. Recent advances like big transformers, self-supervised learning, etc, are more examples of that. Even the initial convnet breakthrough in 2012 was possible only because we figured out how to process a large dataset efficiently on a GPU.

Companies will realize that they don't yet need Data Scientists or AI researchers. They need engineers that are really good at building data pipelines, event stores, feature stores, and databases specific to various query patterns. Or perhaps one to rule them all.

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

gas9S9zw3P9c··on TensorFlow, Keras and deep learning, without a PhD
Thanks for sharing your learning path and congrats to your success. There are so many great resources out there that it's possible for anyone to become an expert. Unfortunately, people like you who are willing to put in the hard work seem to be the minority. All of your success is well-deserved and props to you. Just like you said, most gravitate towards the easy-to-understand videos and blogs instead of confronting their gaps in knowledge. I've had the same experience with giving advice - anything that looks like it requires focused work (solving textbook problems!) or is unsexy is readily ignored in favor of the latest hype demo or high-level framework.

> It was primarily driven by the need to solve problems for autonomous systems I was creating.

I wonder if your success also had something to do with the fact that you had a specific problem you were trying to solve. Did you feel that your specific problem put everything you were learning into context and made it easier to stay motivated?

gas9S9zw3P9c··on TensorFlow, Keras and deep learning, without a PhD
The peer review system is so broken, especially with the number of submissions that the top journals get in ML now, that I don't even pay attention to what's accepted where anymore. It's all the same as arXiv to me. The best way to figure out what's useful and what's not is to wait and see which papers pass the test of time. But if you're an academic you don't necessarily have that luxury.
gas9S9zw3P9c··on TensorFlow, Keras and deep learning, without a PhD
Oh yes, I don't think advanced math is required in any way. However, there is a difference in that researchers who have worked with these models for many years (often including having done some of the math) have a very good intuitive understanding of these models. Once you have that, it's fine to be driven by gut feeling. Just like many engineers are driven by gut feeling that come from tacit knowledge through experience.

What is dangerous is reading a 30 minute blog post and getting the illusion of having some kind of understanding, when in reality it can take years to develop that. It's like cloning the postgres Github repository, compiling it, running a few queries, and then saying you've built databases and being hired to become the "database expert" at some company, spreading wrong knowledge left and right.

That's why the popularization of these quick immediate reward tutorials is dangerous. It takes time and effort to become knowledgable at something. Of course, many people are smart enough to know that these tutorials (or cloning the postgres repo) is just the first step on a longer learning journey, and in that case it's totally fine, but there are also many people who start thinking they are experts ready to work on research or production models after going through such things, not being aware of the many things they don't know [0]

[0] https://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Du...

gas9S9zw3P9c··on TensorFlow, Keras and deep learning, without a PhD
I think you're absolutely right. In my experience

1. There is much more supply than demand in terms of Machine Learning and Data Science. There aren't actually that many jobs outside of research, it's just that the hype makes it look that way. Now that all these PhDs seem to be starting in ML, I wonder what they will do in 4-5 years when they finish the degree. I don't think a market for them will exist.

2. Many companies don't know what they are doing. They are hiring ML people because they want to put AI into their marketing materials. In reality, they don't need ML, they just need someone collecting data, building a database, and running a query. They just don't realize that's the case. The same happened with "Data Science" and "Big Data" - What most companies needed were software engineers building infrastructure and data collection, not people running sklearn.

gas9S9zw3P9c··on TensorFlow, Keras and deep learning, without a PhD
Definitely. I came from a software engineering background into AI 6+ years ago, and I was lucky to be there at the right time. From purely a career perspective, leaving aside that this may be your life's passion, I believe that right now is the worst time to get started in AI. These is so much noise and competition but very few actual jobs or value created. It's just a research PR machine. Just as with Data Science, companies that think they need AI often just need better data collection, pipeline and infrastructure engineering. Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling.
gas9S9zw3P9c··on TensorFlow, Keras and deep learning, without a PhD
As a researcher in the field I am not quite sure how I feel about these kind of resources. I am all for making research accessible to a wider audience and I believe that you don't need a PhD, or any degree, to do meaningful work.

At the same time, the low barrier of entry and hype has resulted in a huge amount of people downloading Keras, copying a bunch of code, tuning a few parameters, and then putting their result on arXiv so they can put AI research on their resume. This has resulted in so much noise and low quality work that it really hurts the field.

You don't need a degree, but I think you do need to spend some time to get a deep enough understanding of what's going on under the hood, which often includes some math and takes time. This can be made accessible and there are plenty of good resources for that. But all these "become an AI pro by looking at some visualizations and copying this code" is maybe hurting more than it helps because it gives the illusion of understanding when it's actually not there. I wouldn't want people learning (solely) from this touching my production systems, writing blogs, or putting papers on arXiv.

gas9S9zw3P9c··on Why is infrastructure as code such a big deal?
Another benefit that hasn't been mentioned in the comments is that it serves as documentation. As new developers come into your team, you need to describe the infrastructure setup. Having your infrastructure defined in a clean repository is one of the best ways to do that.
gas9S9zw3P9c··on Ask HN: Which book helped you understand the world?
I enjoyed Fooled By Randomness. Taleb's other books are good too, but Fooled By Randomness was my favorite.
gas9S9zw3P9c··on Using GPT-3 to generate user interfaces
Well, I think it comes down to how you define search. People are amazed and how good GPT-3 is at generating text, but looking at it from a search lens, you could argue that rather than "generating" it searches for the most relevant pieces to the input, combines them, and then enforces correct grammatical rules on the output. A recent example I saw on Twitter was someone asking GPT-3 questions about human evolution, and the model answers with a bunch of book references and links - that's something you would have done in Google before.

The fact that the Transformer attention mechanisms works based on matching queries and keys/values is probably why the model is so good at finding relevant information to your input - that's kind of what it was trained to do.

gas9S9zw3P9c··on Using GPT-3 to generate user interfaces
As someone working in the field I have a few thoughts:

1. The way I think about these huge models is no longer as something that makes predictions, but rather as a kind of huge knowledge base (compressed training data as representations) with smart query capabilities, which come from the fact that the model is constrained to create syntactically correct output. This could be the next generation of search. There is a fine line between search and creating novel outputs, since most novelty is just a combination of old things with constraints on correct syntax.

2. The shown results are likely a result of massive cherry-picking in the typical "pitch deck demo" fashion. You'll see a lot more failures when trying to use this and you need to get the query "just right" for it to produce good output, which a skill in itself. Also let's not forget that the model must be fed fine-tuning examples, which also need to be "just right.". That's why I believe these kinds of models aren't that useful for fully automating things - for something to be used in production it should be 100% correct - but they are very useful as a search function and alternative to e.g. StackOverflow in this case. Querying the model will give you good enough results that you can use as a starting point for your own use case.

gas9S9zw3P9c··on Snorkel AI: Putting Data First in ML Development
There are some case studies on the website [0], but I would be wary of them. My experience from working in these kind of research environments is that the case studies are consulting jobs where a few PhD students do work for a company, and in return you can use the company name and write on your website whatever you want. They are probably not actually using the product. I'd be interested to hear about actual users and their experience.

[0] https://www.snorkel.ai/case-studies

gas9S9zw3P9c··on Snorkel AI: Putting Data First in ML Development
I also have problems understanding what exactly it does. I just briefly skimmed the paper, but it seems like the idea is as follows. Assuming you don't have ground truth labels for your data:

1. Generate many different nosiy labels for your data by writing functions. These don't need to be correct, but they should make uncorrelated errors. They are basically domain knowledge you have of your data.

2. Snorkel takes the output of these functions, and based on their (dis)agreement, builds a generative probabilistic model to uncorrelate your labels, which may have had some overlap in the errors.

3. You train your final discriminative model on the output of that probabilistic model.

So, the main idea is to create many noisy labels instead of relying on a single high-quality label and Snorkel does the hard work of figuring out how to smartly combine these labels so you can train on something clean.

gas9S9zw3P9c··on Ask HN: What are some good sites or blogs about the mechanics behind trading?
I used to work as a professional trader. I see this question so often, but it's just a strange question to ask. You are asking something like "I want to learn programming, you know, to make stuff like Apple or Google" - If you ask a dozen developers they will give you a dozen different answers depending on what their background is. Some will tell you to learn Javascript and React, some will tell you to understand infrastructure and Go, some will tell you to look at REST and Ruby on Rails, and some will tell you to look at Flutter for making mobile apps, and so on. None of them is wrong, but you're getting all these wildly different answers because you're asking an imprecise question.

Trading is the same. There are dozens of different types of "trading" and everyone you talk to will have a slightly different take, depending on their background. They may talk about portfolio management, directional trading, market making, HFT, arbitrage, options trading, exotics, long-term investing, all of this over different time horizons, technical analysis, fundamental analysis, and so on. None of them are wrong and all of these things are related (just like REST and JS may be related), but you need to be more precise in what you are asking. Someone trying to manage a diverse portfolio and trying to get a low-risk return over several years is using a completely different set of technologies and methods than someone working in HFT. It's like comparing someone writing low-level C++ driver code to someone iterating on HTML and CSS. It's just different, but both are programming.

As a side note: 99% of what you find online are gurus trying to sell you something. They are making a living selling and re-purposing educational content, but have never built anything profitable themselves. And even if they have, what worked 10-20 years ago is no longer relevant today. Books that teach fundamentals are generally of higher quality. [0] and [1] are a good starting point, but again, they are quite specific to certain types of trading.

[0] https://www.amazon.com/Trading-Exchanges-Microstructure-Prac...

[1] https://www.amazon.com/Concepts-Practice-Mathematical-Financ...

gas9S9zw3P9c··on Ask HN: How will NLP models affect online discussion?
I think this is true, but I don't think most people are aware of this. As the world becomes swamped with fake news and content of all forms, more people will realize that humans are not the rational beings they think they were. Perhaps that will result in more services that optimize for emotional well-being, i.e. the opposite of the Instagram and Facebook's.
gas9S9zw3P9c··on Ask HN: How will NLP models affect online discussion?
A few thoughts:

1. This has been possible for a while. GPT-3 is impressive, but I don't see it enabling too many fundamentally new things because many interesting applications are in the tails of the distribution where you don't have data. You can spam social media using simpler techniques based on smaller generative models (well before GPT, but still recent as in ~5 years ago), paraphrasing, and crowdsourcing. Running GPT-3 inference is quite expensive right now, but of course costs will go down over time. I'm not sure how much the lower budget will matter though - politics and large companies have had a big enough budget to do this anyway and it probably has been happening. The costs for this have been low enough for a while.

2. What makes an argument made by a machine less valid than an argument made by a human? If the machine can make a valid argument by pattern-matching your text to historical data and responses, why is that bad? That's what humans do as well, just more unconsciously.

3. I believe the culture around this will and has been changing. There are already bots everywhere, and people are slowly getting used to it. I believe there will be places where you can interact with verified humans, but not to argue or get facts (why would you want to talk to humans for that? They're bad at it.), but rather to foster relationships and get emotional support.

gas9S9zw3P9c··on Pulsar – an open-source distributed pub-sub messaging platform
Yeah, more or less. I'm not complaining about the JVM directly, but rather about what typical Java software, of which Kafka is one example, looks like, at least in my mind. Whenever I see JVM I think: Horrible user experience and tooling, bad memory footprint, slow to start and deploy, complex codebase where it's impossible to debug issues, huge Docker images, complicated configuration and tuning options that doesn't work out of the box, dependency hell, and so on. It's the typical enterprise package for managers that don't want to get fired - Nobody got ever fired for buying IBM.

When I see something like NATS - A few thousand lines of code and a 40MB image, that makes me happy. It's a lightweight thing that "just works."

That's why I stay away from anything JVM. And I acknowledge that I am totally biased and may be wrong in some cases, but that has been my experience.

gas9S9zw3P9c··on The stock market and economy have parted ways
I'm curious - why do you call the hedge fund world evil? I hear this quite often but where does it come from?

I've worked in both a hedge fund and in tech, and I found the former to be less evil and pretentious. People in finance think and say "I want to make a ton of money" while people in tech say "I want to save the world" while thinking "I want to make a ton of money" - resulting in misaligned incentives and imposter syndrome everywhere.

gas9S9zw3P9c··on The stock market and economy have parted ways
> I am curious: based on your experience, do you think that, in principle, an omniscient observer of all transactions on stock exchanges and who made them could answer questions like this in many cases? If so, what concrete things would they hypothetically want to look at to figure it out?

Not necessarily, because the order flow they see goes through brokers, is anonymized, and order execution is optimized algorithmically as to not reveal any patterns. You can't easily tell what is retail flow and what isn't, however you want to define that. What you would need is not just to exchange data, but also the internal user data from companies like robinhood, and an understanding of how exactly algorithms from hedge funds and HFT firms behave. In theory this would be possible if you had complete access to and a full understanding of all of this. In practice it's impossible.

gas9S9zw3P9c··on The stock market and economy have parted ways
You're asking a great question. I used to work as a professional trader and the answer is that probably nobody really knows. Like you said, any explanation you hear about prices moving is just a narrative fallacy.

People love hearing simple explanations to complex questions like "why did X price move" when in reality there is no such thing. The stock market is an incredibly complex interconnected system, with so many types of participants and algorithms optimizing for different outcomes over different time horizons.

Retail flow can certainly influence the market, but in my experience the influence is more indirect than direct, as algorithms may jump onto the retail order flow signals to profit from it, which may then create positive feedback loops. But who knows, the fact is that a lot of volume is being bought, and that's all you can conclude.

I'd go as far as saying that anyone who tries to connect the stock market with economic events doesn't understand a thing about trading. They're just not connected, except in a few special instruments. They're only connected in so far that people believe they are connected and invest based on that, just like Bitcoin is only worth what people believe it to be worth.

gas9S9zw3P9c··on Pulsar – an open-source distributed pub-sub messaging platform
Yep. I'm using containers and Kubernetes in production so I'm trying to stay far away from anything JVM-based. On top of being slow to start, memory hungry, and plagued by GC issues, JVM docker images are always huge. I don't want two layers of virtualization. If Docker and k8s become more mainstream that's probably bad news for a lot of JVM projects. Of course, Pulsar itself is quite old (2013?) so it was built well before any of the containerization stuff took off.
gas9S9zw3P9c··on Pulsar – an open-source distributed pub-sub messaging platform
I've used Kafka a lot in the past and I've been looking at Pulsar. The broker being stateless is great. I've had issues with moving partitions around in Kafka.

That being said, I'm getting tired of managing these clunky memory-hungry JVM-based systems that rely on external dependencies like Apache Zookeeper and BookKeeper. They may be necessary if you're Yahoo, but I would argue that for the vast majority of companies, these complex systems create so many administration headaches that the net productivity impact is negative. I've spend days or weeks debugging Kafka issues. I almost feel like "big clunky heavy JVM enterprise software" has become synonymous with Apache projects.

If you don't absolutely need the messaging guarantees I would recommend looking into NATS (https://nats.io/) - it's a brokered systems that's significantly more lightweight and super easy to deploy and play around with. You can get persistence and delivery guarantees with NATS streaming, but that part is optional and a bit more early stage.

gas9S9zw3P9c··on Ask HN: What do you want to see in a resume / GitHub profile?
The problem with this is that there is no context. It's impossible to verify or evaluate what happened. Who knows what "helped" means. It could be fixing a bug in an open source software product that was eventually used to reduce CAC. Who knows. It's vague.

The world is complex. There is rarely such a thing as a single person writing a few lines of code, and that code directly leading to $ savings. There are complex interactions and dependencies. Also, not all engineers are working on code that has measureable bottom-line impact, so these statements are only possible for a few. For example, an engineer refactoring old code can save a company millions over the lifespan of that code, and it would never show up as dollar values anywhere.

On the other hand, someone can change a cell in an Excel spreadsheet that had a typo in it and save the company $100,000. Should I hire the person because they found a typo? These kind of statements with dollar values are meaningless because the reader is missing context. More often than not, they are vanity statements and come across as disingenuous. I'd rather have you describe what you actually did.

A better example would be "Architected and implemented new real-time event messaging infrastructure using technologies X and Y that allowed us to move of expensive SaaS service Z and save $$.", or something like that. At least that tells me what you did, but even here the $ value is meaningless.

gas9S9zw3P9c··on Ask HN: What do you want to see in a resume / GitHub profile?
You're right, so I'd rather not so these claims on a resume at all. But, for example, if you've managed a team for several years or given a lot of talks, it's more likely that you're good at communicating ideas, simply because you've had a lot of practice and you have probably gotten some feedback over time.
gas9S9zw3P9c··on Ask HN: What do you want to see in a resume / GitHub profile?
Let me give you a slightly different take. There are no specific things I'm looking for. But here are a few red flags that have proven to be good indicators of who NOT to hire. Don't

- Use high-level vague marketing speech like "helped reduce customers acquisition cost by $X" - The world is complex, who knows what that means in practice.

- List MOOCs, certificates, or free online courses on your resume. 99% of these require no commitment and don't teach you anything. Listing them signals that you care about cheap credentials more than learning. If you wanted to learn, you would've created a project out of it, tried to implement something, or written a blog post, etc. I'd rather see "I privately read books about X in my free time" than cheap MOOC credential signaling. That'd be more impressive and I could ask about it in an interview.

- List software like Excel or Google spreadsheets. I mean, if you're an engineer I'm assuming you can use a computer. Listing this means that you're living in a different world.

- List fluffy soft skills like "I am a good communicator" without any backup. Results speak louder than words.

- Too many spelling mistakes.

These are the things off the top of my head, I may add to it later.

gas9S9zw3P9c··on Nvidia is now worth more than Intel
Well done! The only company that successfully managed to ride both the blockchain and AI hype :)
gas9S9zw3P9c··on The Hitchhiker’s Guide to Ethereum
The article lost me at "According to LinkedIn" :) I think this perfectly captures how Blockchain is something you put on your LinkedIn profile purely for signaling to recruiters and non-technical PR and marketing people that use LinkedIn, which is full of spam and pyramid schemes.
gas9S9zw3P9c··on Ask HN: Who Earns a Living of Betting?
I think stock market is a vague term. It depends on what "market" you're trading in. If you're trading highly liquid stocks with large market caps you'll be in against large hedge funds employing hundreds of people. If you're trading less liquid stocks, small caps, complex derivatives, or other smaller markets, you may have less competition and it'll be easier to find an edge.

I'd say the same is true for betting. Your competition will vary depending on the market size and liquidity. Small unknown markets are typically not worth going after for large teams of highly-paid individuals. For example, you probably won't find as much competition betting on eSports events than betting on horses [0] (in jurisdictions where it's legal). So yeah, I think you're right that the professional competition in betting is probably a lot less than in the stock market.

Finding a market with the right characteristics is IMO the most important thing. If you're an individual, you ideally want to find small unknown and relatively illiquid markets where you can make enough money for yourself, but where it's not worth for large players to get into.

[0] https://www.bloomberg.com/news/features/2018-05-03/the-gambl...

gas9S9zw3P9c··on Ask HN: Who Earns a Living of Betting?
Not myself, but I know someone who was making a living from sports betting a few years back. It's not that different from "gambling" in the stock market. The same principles apply. You need some kind of egde, which often is 1. Data or 2. Market Access, less often the statistical model. Better data beats better models, and modeling is commoditized these days. Also, you need a pretty decent bankroll so that you can handle the volatility.

It's an interesting space, but don't expect to be profitable without investing 1-2 years of your time into building good models, infrastructure and data access. This already assumes you have that kind of freedom/capital and are able to stick with it through tough times. Also, edges can be fleeting, so it's a constant race against time and other players to find new opportunities. A bit similar to online poker. It was possible to be highly profitable with online poker 5-10 years ago, but over the last few years it has become very hard to impossible to beat the odds, so a lot of poker people have moved on to gambling on other things, sports betting included.

← PreviousPage 2 of 4Next →