HNHacker News
TopNewBestAskShowJobs

benwills

259 karma · joined July 13, 2011

submissionscomments
benwills··on Ask HN: Do you ever go back and admire a piece of code you wrote?
This sounds awesome. C is my preferred language and I use it whenever I can and when it makes sense. I'd love to see this code. Is there a public repo for it?
benwills··on Ask HN: Name one idea that changed your life
I've taken a slightly similar approach:

If the information won't affect my behavior/actions, I choose not to invest time into it.

This allows for me peruse headlines to be aware of what's happening (and to consider looking further into something), but keeps me uninvested enough to avoid becoming emotionally consumed by things I won't be taking some action toward/against.

benwills··on Reusing Old Hardware
For a lot of basic operations, this is often false.

I have an application I'm working on and tested it across three different generations of CPUS going ~10 years back.

The code was written in C and the fundamental steps were to: mmap() two arrays of structs from a sata 3 SSD, compare the two, and write the intersection to a block of memory. The data size of each array was multiple gigabytes and the operation took more than a full second.

The ~10 year old low powered Xeon L5640 beat the higher-speed ~5 year old Xeon E5-2667 v2, and was only slightly beaten by the 3 year old i7-4790k. In short, they all ran at roughly the same speed.

I would say that this is because of the limits of the SSDs, but then I tested by holding that data in memory rather than disk, then ran the intersection. Same result.

This is a very basic operation that tests the performance of the CPU and memory. And for operations like these (ie: not using SSE2, AVX, etc), the perceived performance difference to the vast majority of people for a secondary home machine may be negligible.

benwills··on Reusing Old Hardware
I've spent a fair amount of time this year with liquid cooling designs, and final parts are coming in to do something along the lines of what you're talking about. Do you have any blog posts/videos walking through this? It sounds really interesting and I'd love to see it.
benwills··on Show HN: Sublime Merge – A Git client from the makers of Sublime Text
Straight to payment for me. Sublime Text is one of the few pieces of software I've just felt was done totally right at so many levels. Entirely happy to give you more of my money.
benwills··on Ask HN: Favorite teachers on YouTube?
For electronics, GreatScott! : https://www.youtube.com/channel/UC6mIxFTvXkWQVEHPsEdflzQ

Electronics had always been a big black box to me. His stuff has helped me wrap my mind around it by incrementally walking through so many use cases. Highly recommended.

benwills··on Uncovering multiscale order in the prime numbers via scattering
From the abstract: "Our analysis leads to an algorithm that enables one to predict primes with high accuracy."

From my very, very basic understanding of/curiosity with primes, predicting primes with any sort of accuracy has been elusive. If this does, in fact, help to predict primes, it opens up all sorts of possibilities. That's my guess as to the importance of this article.

benwills··on Page Cache, the Affair Between Memory and Files (2009)
What Every Programmer Should Know About Memory: https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
benwills··on A new kernel polling interface
Wow. When I saw your code, I thought to myself "yeah, but does it work when the socket is for a client?" Turns out, it does.

My only guess is that I was botching something else when I was digging into this a few years ago.

I can't tell you how much I appreciate you putting that together. This seriously changes a lot of what I work on. Thank you.

The client code I wrote: https://gist.github.com/benwills/95ee844853d3b18588fb3df7d56...

benwills··on A new kernel polling interface
I dealt with all of this in 2015, and spent three months trying to get epoll to perform similarly to kqueue; namely, to avoid returning on any and every packet received and, preferably, to abide by some sort of minimum bytes received (or connection closing) before returning.

Unless I'm missing something (and I could very well be missing something), I'm not seeing how that is any different than anything I tried...and I went through every socket option I could find, even if it seemed irrelevant.

Have you written code that actually performs this way? Or are you speculating? I really do wish I could do this with epoll, but I (and several others) just never were able to figure it out. And kqueue simply performed at least an order of magnitude better of any variation I could hack together with epoll.

benwills··on A new kernel polling interface
Do you have a link to either an announcement of this change, or C code that demonstrates it? If it has changed, I might start doing some things differently...
benwills··on A new kernel polling interface
Behind the scenes, Python may have another layer in there to check the socket options, etc, before returning. ie: Just because it looks like it's working on the Python level doesn't mean it's working the same way underneath.
benwills··on A new kernel polling interface
My experience is across several years writing high performance http clients, writing in pure C.
benwills··on A new kernel polling interface
It's also worth noting that kqueue() also doesn't use the socket options for this. Instead, it uses a flag called EVFILT_READ. More here: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2
benwills··on A new kernel polling interface
Given that I wasn't asserting that it is, I'm not sure what you're getting at. I was responding to the parent comment.
benwills··on A new kernel polling interface
Correct. You can set those socket options and change the triggering behavior of epoll. But those socket options do not actually affect how epoll handles the data when it comes in. They are completely ignored by epoll.
benwills··on A new kernel polling interface
There are other major benefits of kqueue() when using sockets as a client that have essentially "forced" me to use bsd instead of Linux/epoll...

The way epoll() is written triggers so many syscalls that things like socket clients can get triggered for every single packet. Whereas kqueue coordinates with other socket options set on the descriptor, such as not returning until x bytes have been received and/or the server has closed the connection.

...not to mention how much more it's tied to across the IO subsystem.

benwills··on The final step for huge-page swapping
My interpretation wasn't that it was specifically that it's non-volatile. Rather, that it's the performance:price of NVME that's "changing the equation."

The combination of NVME's IOPS and throughput, particularly for sequential reads/writes like you'd have with something like huge pages, begins making it a reasonable consideration for caching/secondary memory in certain situations.

edit: s/NVME/non-volatile/

benwills··on Introverts, Emotional Processing, Self-Esteem and Salary Negotiations
My guess is that you don't intend to be insensitive, but the awareness you're demonstrating here shows a lack of empathy and true understanding/acceptance of the emotional sensitivities to others, particularly this author. To minimize someone's emotional state/response isn't only insensitive (intended or not), it's also abrasive. I'd imagine it's also likely causing problems for you in other areas of your life, if you look and/or solicit candid feedback from people around you that aren't quite like you are.

If you're (or anyone else) interested in talking more about it, feel free to let me know. I had to do a lot of similar work starting from a similar place as you're describing. You (and anyone else seeing nothing "wrong" with the author's response) might also consider reading Karla McLaren's _The_Language_of_Emotions_.

benwills··on Linux kernel coding style
I would disagree with the reason being "because we've always done it this way."

I started programming C full time about 2.5 years ago (after many more with PHP, et al). There's an aesthetic to writing C that I don't find with other programming languages. I believe this is, in part, due to the generalized use of shorter variables and keeping everything as compact and simple as possible.

Since deciding to keep everything at an 80-char width, I can say that it has helped me maintain a certain kind of readability I don't find with other languages. And, while aesthetics don't matter once the compiler takes over, I can say that good aesthetics do improve my ability to program more effectively and efficiently.

benwills··on Is Ontolo the fastest crawler, parser, indexer?
I think that everything I'm willing to list publicly is in that post. Understand that almost all of it is written in pure C. [Edited to add: I do not use Java in any part of ontolo.]

An example is that I do use Redis for some queueing operations. But there's a bottleneck in part of the process (not with Redis, but the nature of being unable to POP multiple items off of a list at once using the blocking pop requests, requiring multiple requests over a socket, or two separate data stores; one to notify, another to hold the data). So I'm in the process of writing my own queue for the crawler system.

I'm guessing that your question comes from a curiosity or desire to build something similar, or something similarly performant. I don't know if this is a useful perspective or not, but it's mine: that every time I don't know what's happening to a byte of data and how it's processed, there's the possibility of some unknown amount of inefficiency there, and that inefficiency could be small or large. But the not knowing is the problem. So I've tried to eliminate as many of those as possible. I don't always get it right, and I'm sure there are many areas for improvement. But if I find a performance bottleneck, I go in and edit my own code, rather than having to learn someone else's codebase that is written to handle edge- and use-cases I'll never need. I then either have to modify their code - and have to maintain future updates as well, etc - or I write my own, and keep refining it.

I'm certainly in no position to give any sort of programming advice to anyone, but that's just the philosophy I've had from the beginning while I worked on this. And I think it's an important one that's helped me get the performance I've achieved with it. It's a conversation I have often with other friends who are programmers and who make more liberal use of tech stacks. But those are often problems of a very different nature than the ones I'm interested in solving.

And if you do hope to build something similar, or something different but similarly performant, be sure to test everything. You don't need amazing tests, just consistent tests with consistent units of measure. Learn where the compiler is optimizing things out and how to make sure that doesn't happen, etc (I will often insert an integer that gets incremented by a random number generated by a fast xorshift prng at the critical place of measure, in order to increase the odds the compiler won't optimize it out), to ensure a level of quality in your tests. And when you're working at the byte level, keep refining the operations that run the most frequently and test all possibly ways you could imagine doing it.

benwills··on Is Ontolo the fastest crawler, parser, indexer?
That makes sense. I don't know if this factors in, but I didn't post the link here. And if I ever did post a link here to ontolo, the HN post title would be very different.

And I agree that few, if any, folks that talk about their own stuff are able to do so objectively.

benwills··on Is Ontolo the fastest crawler, parser, indexer?
It's my pleasure. I love talking about this sort of stuff.

As for the rate of requests, I just couldn't figure out how to max out the gigabit connection on Linux with epoll. I ended up moving to FreeBSD and kqueue, and when I did, that's when I was finally able to comfortably max out the connection. I could have just been doing something wrong with epoll, but I wasn't figuring it out, and neither did a friend who looked at it. My guess is that some of the limitations of the api that are resolved in kqueue, just weren't cutting it for that much data coming into epoll.

I briefly considered writing my own tcp stack to run on raw sockets, but I'm quite happy with FreeBSD's network stack, which seems to be more than sufficient.

As for the size of the project, toward the end, it kept shrinking as I better eliminated redundant logic. There might be more ways to improve it, but I'm pretty happy with where it is.

benwills··on Is Ontolo the fastest crawler, parser, indexer?
Hi Joe. I just saw this post here on HN. I'm the sole author of ontolo.

Your comment about the ontolo domain and assuming it was a PR piece, is that because you know of ontolo as a marketing tool, or because it wasn't a more generalized site like Medium?

Also, if you know of any other crawlers, etc, that are faster than ontolo (in the public domain, or public claims), let me know. There aren't a lot of folks I know who are working on this sort of thing and not also working at search engines. It'd be neat to meet some other folks who are interested in the problem.

benwills··on Is Ontolo the fastest crawler, parser, indexer?
I just saw this post. I'm the sole author of ontolo.

As for requirements, they're very low. The machines running the crawler+parser+indexer, which achieve those speeds, are dual Xeon 5639s with 72GB ram. When the indexer isn't running, the crawler and parser use 6 cores, but only "need" 4 of the 24 total on the box. The indexer generally uses very little CPU. All of those cores are actually needed only for queries. So, the limitation on throughput, at this point, is the 1gbps connection to the box. I say 'box' singular, as the 400,000,000/day is per box.

As for maintaining the codebase, it's about 10k lines, plus another 10k of data that's referenced throughout. As I've written everything in the crawler and parser by hand. There are no magic boxes or libraries that it goes into, so debugging is working around only the logic I've constructed, as opposed to someone else's logic.

That said, I've designed it to use as few system resources as possible, unless it would run faster using a uint32_t vs a bit field, for example. If someone else were to begin working with my code, I'm unsure of what that would be like for them. I'd like to believe it's generally straightforward, but I can't say until that happens.

That said, I've also designed it to be easily updated as the way that HTML is written continues to evolve. I know that's a bit hand-wavey, but it's as much as I'm comfortable describing how it works behind the scenes, as no one else I know of (other than common search engines) parses HTML like ontolo does.

If you have any other questions, just let me know. I'll be monitoring this post today.

- Ben

benwills··on Ask HN: What are you working on and why is it awesome? Please include URL
nope. it's all in sphinx. sphinx has its own sort of data store called 'attributes', which are analyzed after a full-text match is performed.

this doesn't work well for relational searches (e.g. sql), where you might look for results that match a specific location or metric (e.g. star-rating) first. but if the text-based search query is the most important part of the query, attributes can then be used to refine the results even further. fortunately for ontolo, this is the case. and it lets me eliminate managing another software package and data store. though that might change in the future, depending on how the complexity and amount of data evolves over time.

properly designing the index here for size and speed was one of the biggest challenges i faced; deciding what to keep in ram, what to keep on disk (attributes), and how to organize so much data that it didn't kill the disk, but still allowed everything you could want to be retrieved and searched against. this might have been the most time-consuming part of the entire project, in terms of thinking-time.

as for xxhash, it's amazing in a ton of ways. i've taken a sort of special interest in hash functions, designed my own suite of testing tools, and written many of my own. for every metric i've put xxhash up against, it has performed beautifully. the only time i'm ever able to write anything faster is when the quality of the hash is severely compromised in order to make a very customized hash for a very specific and narrow set of data. and compared to other 64-bit hashes out there, it consistently outperforms them in terms of speed and distribution, across many hardware architectures. yann collet really made something amazing there (in addition to several of his other projects).

[edited to add]: i forgot to mention that part of storing data in sphinx as attributes is that you can store plain text data (or json, etc) that is returned in the query result. this is how we return urls, page titles, etc to the user in the browser and exports, thus eliminating the need for a raw data store.

benwills··on Ask HN: What are you working on and why is it awesome? Please include URL
the parser is all hand-written c. the only library i use outside of libc, is xxhash.

documents are read and parsed byte by byte, where "sections" (e.g. side nav) and "contexts" (e.g. comment author) are maintained in a stack of states. part of the speed comes from writing a custom hashing tool that creates compact, very fast hash tables with few or no collisions. so identifying html tags, etc, is very fast and keeps everything as close to l1 as possible.

data is stored exclusively in sphinx. i don't know if it's a secret or not, but i've seen no one talk about it: elasticsearch crumbled horribly under this kind of load. sphinx performs beautifully.

depending on the path i take from here, creating my own index and engine is becoming a serious consideration.

benwills··on Ask HN: What are you working on and why is it awesome? Please include URL
ontolo.com : a fast, customizable search engine for marketing opportunities.

Written entirely in C, except for the front-end. Took 18 months. Didn't know C before and was a hack programmer. Just released a major update yesterday.

The crawler, parser, and indexer can process over 250,000 web pages per minute on commodity hardware and a gigabit connection. The parser is unlike any other I've seen, in that it parsers a document into about 200 sections.

This allows you to search for very specific things like "web pages that mention Donald Trump in the comments, where Bernie Sanders is mentioned in the main article, and there are Adsense ads on the page."

Details at : https://ontolo.com/app/guide/

benwills··on Show HN: Find the best recipe for any two foods, based on over 600k recipes
This is great. It's one of those totally obvious ideas that everyone else just missed. Stick with it and keep refining and improving. The line of thinking is fantastic.
benwills··on Virtual Memory with 256 Bytes of RAM
Not sure if others are having the same experience with it, but I'm finding it just beyond my comprehension, in the best way possible. So I'm not sure how I'd comment on it.

And what I mean by 'just beyond my comprehension' is that it's far enough ahead (and also in the direction I'm headed), that I'll be reading it several times before it all really sinks in.

In short, I love it. I think this is really fantastic and I wish more folks took the time you did to really walk through more intricate concepts like this. Really, really appreciate the time you put in. Thank you.

← PreviousPage 3 of 4Next →