1,130 karma · joined March 9, 2017
- another stupid question: how do you about loading data from millions of pages simultaneously
- here is my silly thought process for this:
- get multiprocessing library in python
- combine that with asyncio and aiohttp
- send a whole bunch of requests and save raw html to file storage?
- some big questions...
- how often do you scan the same website
- what headers do you need to add in order to make it not look like some bot or should you actually reveal that you are a search engine bot
- do you need rotating proxies? is something like brightdata or residential proxies used or am I overthinking this?
- I am thinking of taking a small subset of 400 billion pages (like maybe just every blog ever listed on HN) and vector embedding all the text
- what do you think the cloud infra side specifically on AWS (since I am highly familiar with it) would look like?
- when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time
- Basically I speak in my tone but the output is one of your GPT voices
- Anyone know of a library or plugin that can accomplish this in real time
- let us say i want to index all 4 billion pages to generate vector embeddings and make a new search engine
- what does my AWS infra look like?
- how much storage on s3?
- what size of an ec2 instance?
- the argument these guys come up with all the time is "I dont need to"
- Big assumption there buddy, big assumption. Lets play both cases shall we.
- Case 1: LLMs infinitely improve and nobody has to code anymore. Yea well, writing a paragraph spec isnt that hard for me bro, I already do it for every project while not using an LLM
- Case 2: LLMS go bust completely for whatever reason. We have a whole generation of mass programmers and 99% of them cant add 2 numbers in c++. Guess what? I am now one of the most sought after programmers in the entire world and part of an absolute minority
- a whole generation of you guys are going to regret this decision very seriously 5 years down the line
- mark my words
- just like how studies are being published currently on how meta algorithms are designed to have you hooked and causes brainfart, 5 yrs down the line , studies ll come out showing how LLMs have caused degradation in critical thinking and coding for programmers
- A whole batch of people ll be forced to go back to the basics is how this ll end
- mind explaining it like how you would do to a 5yr old kid
- https://www.reddit.com/r/PostgreSQL/comments/1vbo5j8/raw_xml...
- your post did not have a single word on XML hence my comment
- you could simply spawn a docker container for each programming language and run user submitted code inside no?
- stop ageing and make people jellyfish immmortal
- think you can pull that off dario?
- i went through your list of posts and there is nothing but one opinion after the other with no hard facts to back any of the claims
- i see what you guys are doing there and even here on HN "riding the AI wave"
- honestly i would appreciate if someone could give far more balanced takes with actual facts to back them up
- your point "ai is accelerating....." explain to me how the same architecture transformers is accelerating? infact after the leading frontier labs pull all their gimmicks in this year and maybe the next, they are going to run out of gas unless they invent a new architecture
- i use them regularly for small stuff
- did you change the architecture magically somehow?
- Everyone and their mother using an LLM is slowly and steadily destroying their brain cells and abilities in the process while creating a learned dependence
- Give it about 5 yrs, mark my words. The studies ll come out saying "95% programmers cannot write a hello world without an LLM now"
- Now someone smart ll hop in on my comment and bring calculators into the argument and I have answer waiting for them that those do (not able to put a direct link here for some reason)
- it only the probability of the next word, next sentence, next paragraph however you wanna put it
- you are dreaming if you think the transformers architecture is going to give you AGI eventually
- every post is locked https://www.byhand.ai/archive
- sorry bruh, not gonna bother with a paywall that big
- amaze me first, i am happy to pay
- other half have no visualizations whatsoever
- i have posted far better resources than this in the past collecting everything HN has over a 10 yr period on these subjects https://news.ycombinator.com/item?id=49183198
- if you wanted to build a 1nm process cpu from scratch without china and taiwan, where do you start
- what do you need?
- what is the timespan
- how much money?
- what kind of expertise?
- if i wanted to build an aggregator say 500k tickers real time , ohlc from every exchange on the planet and every known asset to mankind...
- where would i start?