145 karma · joined October 15, 2019
Maybe OP can make an example with high frequencies removed?
Is it necessary to do it for x and y separated? Couldn't you just take x for the real part and y for the imaginary part of the time series and do the Fourier transform on that?
edit: monday morning typos.
Still love pythons syntax in general.
(Intereseting fact: A lot of malware quits immediately if Little Snitch is installed. So running Little Snitch alone prevents some malware from infecting your system)
However I didn't get around to learning the principles to use it regularly. And I fear if you have a certain chart in your head with some special stuff you end up tinkering around eventually get a lot of code.
This is a huge potential procrastination catalyst :-)
I have a question. Although the article clearly states that this is not an epidemiological problem, I wonder about the following: gravity and electric fields are 1/r^2 type problems. I imagine that disease transmission depending on distance is a rather different thing, such as "high in immediate vicinity to the other point and then falling off quickly".
Do we know if the algorithm is accurate enough for this kind of "field"? Is it in general enough accurate or only for 1/r^2 type fields?
(e.g. freedom of opinion, freedom of press, separation of powers a.k.a. checks and balances, Rule of Law etc.)
Keep in mind that pandas (and probably also R?) internally uses optimized structures based on numpy. So a 10 GB csv, depending on the content, might end up with a much smnaller memory footprint inside pandas.
If you have 10 GB csv, I think you will be happy working with pandas locally even on a Laptop. If you go to csv files with tens of GB, a cloud vm with corresponding memory might serve you well. If you need to handle big-data-scale csvs (hundreds of GB or even >TB), a scalable parallel solution like Spark will be your thing. Before you scale up however, maybe your task allows to pre-filter the data and reduces the amount by orders of magnitude... often, thinking the problem through reduces the amount of metal one needs to throw at the problem...