HNHacker News
TopNewBestAskShowJobs

afiodorov

784 karma · joined June 21, 2019

Likes Computer Science, Mathematics, AI, ML
submissionscomments
afiodorov··on Practical SQL for Data Analysis
I have been working with data for more than 5 years now and my 2 cents is that Pandas shines with exploratory analysis. Because you have the data in memory it's easy to empirically arrive at the exact data transformation you need and figure out whether your analysis is worth pursuing further. At this stage of the development your ability to iterate and experiment quickly is most valuable. I see no value in worrying about the deployment of the work you are doing if you haven't got any results just yet.

Now when the analysis is done and there's a conclusion about putting some of it in production, I am yet to see a single company that relies on pandas to do data transforms "online", i.e. on the live production data stream. Typically in my experience I would end up rewriting my data transforms in either a compiled language or SQL - i.e. something much more performant and maintainable.

As a result as you progress in your career you become very proficient in both pandas and SQL - and you also start glancing at the source code of some of the transforms performed by pandas when you need to port them to the language of choice of the company you're working for.

afiodorov··on Ask HN: Should I learn C++ in 2021?
I learned C++ at the start of my software career. I am glad I did; but I am also glad I never had to write it commercially.

Learning C++ makes learning other languages easier. It's C++ that made me grok the difference between stack and heap allocations; it taught me about moving semantics, dynamic arrays, reference counting garbage collection, data structure alignments, dynamic/static linking, segmentation faults and many more things.

The above things made me understand better how other programming languages work under the hood.

At times I find C++ code that is of intrinsic interest to me. Bitcoin, for example, was authored in C++. However I dread writing C++ because I find it very complex and it has outdated coding environments. Thus I am very happy I don't have to write it.

afiodorov··on Where Are All the Successful Rationalists?
I have been heavily influenced by lesswrong. After the initial stage of encorporating the rationality identity and arguing about everything with everyone, I tried becoming more of an applied rationalist when I realised the first approch isn't leading me to winning anything. Shortly after I read: https://www.lesswrong.com/posts/ZbgCx2ntD5eu8Cno9/how-to-be-...

I made a series of life changes. I can't attribute the blog post alone to these changes, but it was something that pushed me towards a more meaningful life. I stopped spending money on things and started buying experiences more often. I bought a bycicle (a thing but gives you many experiences) and started cycling to/from work in an effort to a) exercise b) spend more time outside. I broke up with my girlfriend of 9 years because I figured it just wasn't making me any happier. Cycling in London, UK for years made me very satisfied but I also got tired of the climate and I ended up finding a remote role and relocating to a climate where I can be outside every day. Finally I have started strength training - and now I am in a much better shape after doing it on a regular basis for a few years. It's hard to say that I am happier exactly, but there is a sense of liberty I feel that is hard to describe. The life I am living feels like my own, I feel the sense of control and I am liberated. I would like to think many of these things count as "winning".

afiodorov··on Running Awk in parallel to process 256M records
Alternatively one can load the data into Google BigQuery and start running queries without indexing the data. It's not free though, however can save a lot of time. BigQuery supports import of JSON and CSV files from Google Cloud Storage and it can even infer the schema.
afiodorov··on The State of the AWK
I work in data processing and I use awk occasionally to work with csv files often gigabytes in size.

I join csv files that each have a header with

  awk '(NR == 1) || (FNR > 1)' *.csv > joined.csv
Note this only works if your csv files don't contain new lines. However if they do, I recommend using https://github.com/dbro/csvquote to circumvent the issue.

Yesterday I used awk as a QA tool. I had to subtract a sum of values in the last column of one csv file from another, and I produced a

  expr $(tail -n+2 file1.csv | awk -F, '{s+=$(NF)} END {print s}') - $(tail -n+2 file2.csv | awk -F, '{s+=$(NF)} END {print s}')
beauty. This allowed me to quickly check whether my computation was correct. Doing same in pandas would require loading both files into RAM and writing more code.

However I avoid writing awk programs that are longer than a few lines. I am not too familiar with the development environment of awk, and I stick to either Python or Go (for speed) where I know how to debug, jump to definition, write unit tests and read documentation.

afiodorov··on Show HN: Radix sort big files in memory
>If IO is a bottleneck it would be much worse while trying to swap around text lines in a giant memory mapped file.

There's no swapping of lines in a giant memory mapped file, the byte array of the file remains intact.

The program only swaps the line markers.

> Just convert the date to 64 or 32 bit integer

This would require extra memory for each line, complicate user interface (data formats vary a lot) and make the utility less generic, since you don't always want to sort chronologically. The current idea is to sort using as little extra memory as possible. This is because when files are already so big fitting them into RAM is a challenge already.

---

Finally, there's no evidence why the proposed solution would be faster. Empirically I have found that sorting chunks and merging is considerably slower than sorting in memory, which makes sense given how much faster RAM is than HDD.

afiodorov··on Show HN: Radix sort big files in memory
It's a dataset related to balance changes of bitcoin addresses downsampled to daily resolution.

You could extract it from BigQuery's bitcoin public data.

afiodorov··on Show HN: Radix sort big files in memory
I had to reduce the batch-size to 512, because otherwise sort complained with batch-size too large. The timings are:

  real    30m54.557s
  user    82m9.055s
  sys     3m1.723s
no improvement over sort without the batch-size...
afiodorov··on Show HN: Radix sort big files in memory
I agree that a significant proportion of time is spent on IO. Only 8m38s is actually spent sorting (out of 19m37s). However in the past my experiments have shown that using `sort -m` to sort chunks is much much slower than using `sort -S100%`.
afiodorov··on Show HN: Radix sort big files in memory
this would work, but once I start sorting lines I'll need a copy of original line breaks array; so in the end I'd need 16 bytes per line anyway
afiodorov··on Show HN: Radix sort big files in memory
as I mentioned in the parent comment I tried saving just the start of each line, and then during sorting scan from the start until '\n' character, but this turned out to be much slower
afiodorov··on Show HN: Radix sort big files in memory
Could you elaborate more why this could be faster? These experiments take some time to complete. Also should I not use --parallel flag?
afiodorov··on Show HN: Radix sort big files in memory
Exact same file has been taking more than 35 minutes to sort already, so it's slower without splitting.

Edit: it finished!

real 35m28.370s

user 40m17.129s

sys 4m31.081s

afiodorov··on Show HN: Radix sort big files in memory
Indeed I load the entire file, but I think provided I have RAM to load the file with all the columns, the approach of loading everything should give me optimal performance. However I agree with your approach when there isn't enough RAM.
afiodorov··on Show HN: Radix sort big files in memory
>- Saving the start and end position of a string that represents a date with 16 bytes is silly.

I am not saving the start and end positions of a string that represents a data; I am saving start and end position of a line, since I need to split the loaded file into lines. This requires extra memory. I tried sorting without caching the end position of the line (and always reading from start till \n), but it was slower to sort this way.

← PreviousPage 5 of 5