On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk.
Or are the datasets large enough to fit on neither?
SAX is harder but has at least two key strongly related benefits: 1. Can handle a continuous firehose 2. Processing can start before the load is completed (because it might never be completed) so the time to first useful action can be greatly reduced.
This is not true, unless you’re referring to swap (which is a configuration of the system and may not be big enough to actually fit it either, many people run with only a small amount of swap or disable it altogether.)
You may be referring to mmap(2), which will map the on-disk dataset to a region of memory that is paged in on-demand, but somehow I doubt that’s what OP was referring to either.
If you just read() the file into memory and work on it, you’re going to be using a ton of RAM. The OS will only put “the rest on disk” if it swaps, which is a degenerate performance case, and it may not even be the dataset itself that gets swapped (the kernel may opt to swap everything else on the system to fit your dataset into RAM. All pages are equal in the eyes of the virtual memory layer, and the ranking algorithm is basically an LRU cache.)
read() is not mmap(). You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way.
which it would do, if you have just downloaded the file.
> But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space
i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here.
> You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way.
i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also.
You come in and say something to the effect of “but it may not have to read from disk because it’s cached”, which… has nothing to do with what was being discussed. We’re not talking about whether it incurs a disk read, we’re talking about whether it will run your system out of memory trying to load it into RAM.
> i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here.
What in god’s holy name are you blathering about? If I “just downloaded it from the network”, it’s on-disk. If I mmap() the disk contents, there’s no copy going on, it’s “mapped” to disk. If I read() the contents, which is what you said I should do, then another copy of the data is now sitting in a buffer in my process’s heap. This extra copy is now "using" memory, and if I keep doing this, I will run the system out of RAM. This is characteristically different from mmap(), where a region of memory maps to a file on-disk, and contents are faulted into memory as I read them. The reason this is an extremely important distinction, is that in the mmap scenario, the kernel is free to free the read-in pages any time it wants, and they will be faulted back again in if I try to read them again. Contrast this with using read(), which makes it so the kernel can't free the pages, because they're buffers in my process's heap, and are not considered file-backed from the kernel's perspective.
> i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also.
Swap is disabled on my system. Even if it wasn’t, I’d only have so much of it. Even if I had a ton of it, read()’ing 100GB of data and relying on swap to save me is going to grind the rest of the system to a halt as the kernel tries to make room for it (because the data is in my heap, and thus isn’t file-backed, so the kernel can’t just free the pages and read them back from the file I read it from.) read() is not mmap(). Please don’t conflate them.
Yep I do the same. If I have a server with hundreds of GB or even TB of RAM (not uncommon these days) I'm not setting up swap. If you're exhausting that much RAM, swap is only going to delay the inevitable. Fix your program.
Nobody made that claim except in your head.
Why don't you read it now:
Just curious: why would you not load the entire dataset "into memory" ("into memory" from a Python perspective)?
Look carefully: There's no mention of the word file. For all you or I know the programmer is imagining something like this:
>>> data=loaddata("https://...")
On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk.
You know exactly this what is meant by swap: We just confirmed that. And you know it is enabled on every average desktop server system, because you
> Swap is disabled on my system
are the sort of person who disables the average configuration! Can you not see you aren't arguing with anything but your own fantasies?
> If I “just downloaded it from the network”, it’s on-disk.
That's nonsense. It's in ram. That's the block cache you were just talking about.
> If I mmap() the disk contents, there’s no copy going on, it’s “mapped” to disk
Every word of that is nonsense. The disk is attached to a serial bus. Even if you're using fancy nvme "disks" there's a queue that operates in a (reasonably) serial fashion. The reason mmap() is referred to zero-copy is because it can reuse the block cache if it has been recently downloaded -- but if the data is paged out, there is absolutely a copy and it's more expensive than just read() by a long way.
> Even if it wasn’t, I’d only have so much of it. Even if I had a ton of it, read()’ing 100GB of data and relying on swap to save me is going to grind the rest of the system to a halt as the kernel tries to make room for it
You only have so much storage, this is life, but I can tell you as someone who does operate a 1tb of ram machine that downloads 300gb of logfiles every day, read() and write() work just fine -- I just can't speak for python (or why python people don't do it) because i don't like python.
> Every word of that is nonsense. The disk is attached to a serial bus. Even if you're using fancy nvme "disks" there's a queue that operates in a (reasonably) serial fashion. The reason mmap() is referred to zero-copy is because it can reuse the block cache if it has been recently downloaded -- but if the data is paged out, there is absolutely a copy and it's more expensive than just read() by a long way.
Please do a basic web search for what "virtual memory" means. You seem to think that handwavey nonsense about disk cache means that read() doesn't copy the data into your working set. You should look at the manpage for read() and maybe ponder why it requires you to pass your own buffer. This buffer would have to be something you've malloc()'d ahead of time. Hence why you're using more memory by using read() than you would using mmap()
> You only have so much storage, this is life, but I can tell you as someone who does operate a 1tb of ram machine that downloads 300gb of logfiles every day, read() and write() work just fine -- I just can't speak for python (or why python people don't do it) because i don't like python.
You should definitely learn what the mmap syscall does and why it exists. You really don't need to use 300gb of RAM to read a 300gb log file. You should probably look up how text editors like vim actually work, and why you can seek to the end of a 300gb log file without vim taking up a bunch of RAM. Maybe you've never been curious about this before, I dunno.
Try making a 20gb file called "bigfile", and run this C program:
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
char *mmap_buffer;
int fd;
struct stat sb;
ssize_t s;
fd = open("./bigfile", O_RDONLY);
fstat(fd, &sb); // get the size
// do the mmap
mmap_buffer = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
uint8_t sum;
// Ensure the whole buffer is touched by doing a dumb math operation on every byte
for (size_t i = 0; i < sb.st_size; ++i) {
sum += mmap_buffer[i]; // overflow is fine, sorta a dumb checksum
}
fprintf(stderr, "done! sum=%d\n", sum);
sleep(1000);
}
While it's sleeping, check /proc/<pid>/status and take note of the memory stats. You'll see that VmSize is as big as the file you read in (for me, bigfile is more than 20GB):
VmSize: 20974160 kB
VmRSS: 968 kB
How is this possible? Because mmap lets you do this. The kernel will read in pages from bigfile on demand, but is also free to free them at any point. There is no controversy here, every modern operating system has supported this for decades.
Compare this to a similar program using read():
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
int fd;
struct stat sb;
ssize_t s;
char *read_buffer;
fd = open("./bigfile", O_RDONLY);
fstat(fd, &sb); // get the size
// do the read
read_buffer = malloc(sb.st_size);
read(fd, read_buffer, sb.st_size);
uint8_t sum;
// Ensure the whole buffer is touched by doing a dumb math operation on every byte
for (size_t i = 0; i < sb.st_size; ++i) {
sum += read_buffer[i]; // overflow is fine, sorta a dumb checksum
}
fprintf(stderr, "done! sum=%d\n", sum);
sleep(1000);
}
VmPeak: 2099928 kB
VmSize: 2099928 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 2098056 kB
VmRSS: 2098056 kB
This is why you shouldn't read() in large datasets. Your strategy is to have servers with 1TB of ram and massive amounts of swap, and I'm telling you you don't need this to process this big of files. mmap() does so without requiring things to be read into RAM ahead of time.
Oh, and guess what: Take out the `sleep(1000)`, and the mmap version is faster than the read() version:
$ time ./mmap
done! sum=225
./mmap 6.89s user 0.28s system 99% cpu 7.180 total
$ time ./read
done! sum=246
./read 6.86s user 1.72s system 99% cpu 8.612 total
I don’t do much python but I used to do a lot of ruby, and it was rare to see anyone mmap’ing anything, most people just did File.read(path) and called it a day. If the norm in the python ecosystem is to mmap things, then you’re probably right.
Not to mention, if the code is not explicitly designed to work with a streaming approach, even for local data, might mean that early steps accidentally end up touching the whole data (e.g. they look for a closing } in something like a 10GB json document) in unexpected places, costing orders of magnitude more than they should.