Load half a webpage sadly...
The only thing I learned is apparently default Python JSON lib sucks on speed
The only thing I learned is apparently default Python JSON lib sucks on speed
-> ᛯ ls -s -h big.json
101M big.json
[11:53:30] ^ [/tmp]
-> ᛯ cat /tmp/1.pl
#!/usr/bin/perl
use v5.24;
use JSON::XS;
use File::Slurp;
my $j = decode_json(read_file('/tmp/big.json'));
say scalar @{$j->{'a'}}
[11:53:39] ^ [/tmp]
-> ᛯ time /tmp/1.pl
34952534
/tmp/1.pl 0,81s user 0,13s system 99% cpu 0,939 total
Benchmarking json_read_struct: Collecting 100 samples in estimated 5.2012 s (81k iterations)
json_read_struct time: [64.126 µs 64.376 µs 64.660 µs]
change: [+0.2646% +0.7525% +1.2538%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
6 (6.00%) high mild
1 (1.00%) high severe
[0] https://gist.github.com/Measter/acbae474ba8e1451946630da2a2c...
- you don’t seem to specify the size of the input. This is the most important omission
- you are constructing an optimised representation (in this case, strict with fields in the right places) instead of a generic ‘dumb’ representation that is more like a tree of python dicts
- rust is not a ‘moderately fast language’ imo (though this is not a very important point. It’s more about how optimised the parser is, and I suspect that serde_json is written in an optimised way, but I didn’t look very hard).
I found[1], which gives serde_json to a dom 300-400MB/s on a somewhat old laptop cpu. A simpler implementation runs at 100-200, a very optimised implementation gets 400-800. But I don’t think this does that much to confirm what I said in the comment you replied to. The numbers for simd json are a bit lower than I expected (maybe due to the ‘dom’ part). I think my 50MB/a number was probably a bit off but maybe the python implementation converts json to some C object and then converts that C object to python objects. That might half your throughput (my guess is that this is what the ‘strict parse’ case for rustc_serialise is roughly doing).