HNHacker News
TopNewBestAskShowJobs

lopuhin

156 karma · joined September 10, 2015

I work at https://www.zyte.com/ on https://www.zyte.com/automatic-extraction/

Github: https://github.com/lopuhin/ Kaggle: https://www.kaggle.com/lopuhin/

submissionscomments
lopuhin··on Exponentially faster language modelling
There are two issues here -- for one, in big transformers, more compute is in the attention layers, while this work improves only feed-forward layers, which are more important for smaller models and smaller sequence lengths. Second, in many typical scenarios LLM inference is memory bandwidth bound, I'm not sure if it's possible to utilize their approach to reduce required memory bandwidth.
lopuhin··on The first room-temperature ambient-pressure superconductor?
I think it's even from 2021-08-25, see https://patents.google.com/patent/KR20230030188A/en?oq=WO202...
lopuhin··on U-Net CNN in APL: Exploring Zero-Framework, Zero-Library Machine Learning
Impressively concise implementation, really interesting paper! Benchmark looks quite questionable though -- e.g. they use fp64 (while any sane person would use at least f32, if not f16), batch 1 (while normally one would try to get the max batch size which fits into memory, and it would reach 1 only for much bigger models or inputs), and also measure the time including transfer to/from GPU (while it would normally be interleaved with GPU operations). Not sure what results would look like in a more realistic setup, but still getting within 2x of PyTorch even in such a setting looks impressive!
lopuhin··on Ask HN: Open-source ChatGPT alternatives?
They released different sizes from "small" to "xxl", and at least "base" should be small enough to fine-tune virtually anywhere.
lopuhin··on Ask HN: Open-source ChatGPT alternatives?
In terms of models which are reasonably fast to run and easy to install, I think Flan-T5 is one of the best: https://huggingface.co/google/flan-t5-xxl - although out of the box it's more focused on giving short answers and it's very far from ChatGPT.
lopuhin··on Apparent size of Andromeda galaxy compared to the Moon
A nitpick on the image Andromeda image: when viewed with telescope and in most images you can find online, the core of the galaxy and the region around the core is much brighter than the outer branches, while on the composite image it looks like the core is dimmed to make the outer branches stand out more.

So the actual bright part of the Andromeda, seen with naked eye under dark skies, is much smaller.

lopuhin··on Congratulations: We now have opinions on your open source contributions
> they require a physical device instead of a TOTP app. Yeah, that's the line between reasonable and wtf.

No, a TOTP app is also allowed.

lopuhin··on Teaching GPT-3 to reverse words
Not 'char' - because it's using BPE (byte pair encoding), so after tokenization you might get ["Transform", "ers"] instead of ["T", "r", "a", ...]. This is relevant to how it struggles to reverse words. Not 'largest' because there are larger models like Pathways Language Model (PaLM) with 540 Billion parameters.
lopuhin··on Teaching GPT-3 to reverse words
Technically it's not the largest, not char, not rnn... but it's close :)
lopuhin··on Rustenstein 3D: Game programming like it's 1992
There is a very simple voxel terrian renderer: https://github.com/s-macke/VoxelSpace
lopuhin··on Meta AI: Fist high-performance self-supervised algorithm for multiple modalities
No, they don't need a pre-trained model, they start from scratch. Also it's only the approach which is common between modalities, they train them separately.
lopuhin··on Meta AI: Fist high-performance self-supervised algorithm for multiple modalities
That's not quite how I understood it

> Do the cutout in feature space, not the original input space.

I think they do the cutout in the original input space, based on examples they show, e.g. they mask parts of text and grey out parts of an image.

> and make the network predict the missing part

I think predict the latent representation (as you say in (2)), but not of the missing part, but of the whole corrupted sample, and require it to be close to latent representation of the original input done by the teacher.

lopuhin··on Ask HN: Who is hiring? (January 2022)
Zyte | Senior Data Scientist | REMOTE | Full-time | https://www.zyte.com/

At Zyte (formerly Scrapinghub), our goal is to help you get the data from the web, so we develop services such as smart rotating proxies, browser rendering API, data extraction API, a cloud for running your crawling jobs, etc.

At the data science team, the main project we work on is data extraction API, which can extract articles, products, job postings and other data types from any website, and also do automatic crawling and discovery. We approach this as a machine learning problem, with a deep learning model combining web page screenshot, text, node information and other features, trained on hundreds of thousands of web pages. We work on improving the quality of extraction and increasing coverage of attributes and data types. I find this problem really fascinating to work on, as on one side, you get to work on a neural network which uses both image, text and graphs as inputs and can find inspiration from current ML literature, but on the other hand, web extraction is not so well studied, and a great deal of experimentation is required. Our tech stack on the ML side is Python and PyTorch. We love Open Source: Zyte founders are authors of a popular Scrapy framework, and we open source many libraries we heavily rely on internally, such as dateparser and extruct.

The company has been fully remote since the start, and hires from a large number of countries. Please check more details at https://wrkbl.ink/iNsRDym, and feel free to check other positions at https://apply.workable.com/zyte/

lopuhin··on Ask HN: What are you predictions for 2022?
> Carrington Event

Not a great time to predict this, at this point in the solar cycle we're close to a minimum of solar activity, the max should be between 2023 and 2026, but it's not expected to be too big either - cycle 25 is expected to be similar to 24 which was quite modest.

lopuhin··on Stealth bomber in flight on Google Maps
A small roll of the place at the time the photo was taken could also explain this difference.
lopuhin··on Rust maintainer perfectionism, or, the tragedy of Alacritty (2020)
I prefer a tab for each machine, including local, and then each tab has a tmux or ssh + tmux.
lopuhin··on Benchmarking the Apple M1 Max
Probably they would be close - M1 still needs to use memory for the OS and other stuff, while 3090 can use fp16/mixed precision, which in many cases almost doubles effective memory. Also if we talk about training, then a more mature CUDA implementation of things like batch normalization and optimizers can also result in lower memory usage compared to a likely less mature TF Metal support.
lopuhin··on Benchmarking the Apple M1 Max
Yes, but then they brought it back here https://github.com/numpy/numpy/pull/18874 - apparently Apple developers fixed these issues.
lopuhin··on EvalAI: An Open-Source Alternative to Kaggle
Kaggle has two parts - more popular "featured" competitions which have most of those features, but cost money to host. And then also self-hosted competitions which indeed lack all those features and are more relevant for this comparison IMO.
lopuhin··on A dwarf planet coming within 11 AU of the sun over the next 10 years
I don't think Saturn comes that close, should be 8 AU at best, given that perihelion is at 9 AU.
lopuhin··on What did Einstein mean by “spooky action at a distance”?
I think the second paragraph in the wiki page about Bell's theorem explains that: https://en.wikipedia.org/wiki/Bell%27s_theorem
lopuhin··on Launch HN: Noya (YC W21) – Direct air capture of CO2 using cooling towers
How large is the CO2 market relative to the amount of CO2 currently injected into the atmosphere?
lopuhin··on Pip has dropped support for Python 2
The first option was by far the most popular and was used for years (including pip), only recently packages started dropping python 2 support. I'm not even aware of any packages which went with the second.
lopuhin··on Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
> I chose MobileNetV2 to make iteration faster. When I tried ResNet50 or other larger models the gap between the M1 and Nvidia grew wider.

(and that's on CIFAR-10). But why not report these results and also test on a more realistic datasets? The internet is full of M1 TF brenchmarks on CIFAR or MNIST, has anyone seen something different?

lopuhin··on Article extraction benchmark: open-source libraries and commercial services
The service itself is already released, you can play with the demo without registration at https://www.scrapinghub.com/data-api-news (scroll to "Try it out here for yourself"), and we also have a free API trial. Would be curious to know how you plan to use it.

In terms of the approach, the whole page is rendered in a headless browser, we extract the whole page screenshot, text and other features, and feed them into one neural network where all modalities are joined and which handles the extraction.

lopuhin··on Article extraction benchmark: open-source libraries and commercial services
Yeah I don't think it's trivial. If an algorithm made predictions on the level of html elements - whether an element should be a part of article body or not, it would be possible to combine probabilities or at least vote. But (a) it's probably a non-trivial modification (b) a lot of methods would also use some heuristics/postprocessing/have other quirks which make combining results difficult.
lopuhin··on Article extraction benchmark: open-source libraries and commercial services
Great question! I'm an active Pocket user myself, would love to know what they use on the backend. From seeing their failures - when they think something is not an article or excluding some relevant stuff, I would guess they use something working on pure html and more similar to current open source solutions - wheres for example Diffbot failures looked quite similar to ours as we seem to use a similar approach (and it's quite rare to miss a large chunk of the article). I imagine Pocket margins must be quite slim so they can't throw a headless browser + neural network on every page. Maybe they can use higher quality and more expensive extractors for popular articles.

Browser extensions are in an interesting position here as they can probably have access to much richer features from the browser context (element size, position, CSS properties), but still want to be low overhead. I think I saw such an implementation, maybe even from Mozilla, but can't find it right now.

lopuhin··on Article extraction benchmark: open-source libraries and commercial services
Author here, ready to answer any questions.

This is an evaluation of article body extraction for AutoExtract (ours), Diffbot, newspaper3k, readability-lxml, dragnet, boilerpipe and html-text. Since we're evaluating the quality of our system as well, we tried to be extra careful to be fair and transparent, releasing dataset, evaluation scripts and all details in the technical report.

lopuhin··on How Long Was Venus Habitable?
Check out https://en.wikipedia.org/wiki/Immanuel_Velikovsky#Ideas
lopuhin··on Single Headed Attention RNN
Good point, assuming some extent of collapse is crucial, and the question is if different perplexities due to tokenization can happen in principle. You are right that in "Alice" vs. "A|lice" example we get the same perplexity after re-normalization, I can't come up with an example where it would be different right now.
← PreviousPage 2 of 3Next →