Sure, if you need to continually access the same values then you'd want to save them but whether you need to reuse the same values or just do a one off calculation, you still need to compute the value the first time in some way.
For Fibonacci numbers, expressing terms of the sequence as a 2x2 matrix exponentiatied and then exponentiating by repeated squaring let's you get the n-th Fibonacci number in O(log n) multiplications. It's straightforward, if you get an input n and handle arbitrary precision integer arithmetic, you can calculate F_1000 quickly and without much hassle. Compare that to Binet's formula: How many digits of precision will you need to compute the golden ratio to ensure your result from Binet's formula is correct? And how will you compute it to that many digits? And that's not even worrying about the fact that floating point multiplication doesn't correspond exactly to multiplication of real numbers, so you may need to switch to arbitrary precision floating point arithmetic.
Not to say anything about the central thesis of the post, but the two examples of how deeper mathematical knowledge can lead to better programming are just terrible examples. As someone whose background was originally mathematics and starting programming relatively late, I can very much relate to thinking Binet's formula for Fibonacci numbers or gamma function for factorials might be faster than recursive methods. But when you actually learn about how ints, floats and their arithmetic is implemented at a low level, how cache and memory works etc, you realise that recursive methods for those two problems are actually your best bet, both for speed and accuracy. So ironically, these are examples where knowing much about low level coding and computing architecture helps you more than knowing more advanced mathematics.
> That's a poor assertion because even if detivative-free algorithms weren't already a thing, there are a myriad of smoothing techniques that enable nondifferentiable objective functions to be approximated by differentiable functions.
The function you wish to optimise may be non differentiable because the domain is discrete. E.g. to perform feature selection one can binary encode which features are included and which are excluded, with the fitness function being the accuracy of a model using the included features as inputs and trained for X iterations.
You shouldn't view GA as a competitor to gradient descent. Whenever gradient descent applies to a problem, it'll probably be better than GA. But GAs are flexible and applicable to more problems, and can be used in conjunction with other optimisation methods.
A few days ago I used GA to perform feature selection (binary encoding with 0 indicating a feature is excluded, 1 if included, fitness function was the accuracy of a NN taking the included features as inputs and trained with gradient descent for x epochs). This made my final network smaller, train faster, etc.
In a somewhat similar vein, I've always found it odd that engineering students commonly refer to themselves as engineers. I agree with much of the linked article and believe the title of engineer implies that one has already gone through a rigorous testing and certification process and that you conform with continual professional development and regulation. So it irks me slightly to hear a first year student call themselves an engineer. It feels like a medical student calling themselves a doctor.
In other fields where the title most designates that you regularly do an activity and does not imply certification and strict regulation I think it's fine to call yourself by that title even if you are a student. Many CS students are programmers, or PhD candidates in mathematics are mathematicians.
I didn't mean you were the one implying an elementary proof is unobtainable, that was directed at some of the other replies to your comment like "oh really?". I agree that it's enlightening to give these things a try.
Man HR is harsh as hell. While yes, if a proof of RH is found it will probably use advanced tools, ever finding a proof with elementary tools is not totally out of the question. Regarding the Prime Number Theorem, which can be seen as a baby version of RH, experts in the field such as Hardy have been quoted as saying elementary proofs would not exist. Then 50 years after Hadamard and Poussin found the original proofs, Selberg and Erdos published proofs based on elementary methods. No one can say that it's worthless for someone to try the same for RH.
Good practitioners do both. Whatever knowledge about the weights you can predetermine, you set those as the initial values of your weights. Then you improve on that by training. It really depends on the problem and size of your network how many weights you'll have a good estimate for but yes, you should try to estimate good weights yourself first.
This practice is well known but here's a concrete source from Andrej Karpathy:
"init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias."
Perhaps not at the hardware level, I'm not sure. I downloaded InSpectre (a windows application) which informs me that I had been protected against Spectre and Meltdown attacks through Windows updates. I also used the option to disable the protection, which is how I figured out how much of a performance hit I was taking.
I turn Spectre/Meltdown protection off (download InSpectre if you run Windows). My Haswell machine gets a noticeable boost (e.g. about 5% increase in Intel XTU benchmark), but newer architectures are less negatively affected by the patches so you may not find the effort/risk to be worth it.
People sometimes talk about waiting for a Spectre "fix" at a hardware or software level, but it seems that the only fix is to lose Speculative execution entirely (and with it, its performance gains). On my Haswell CPU, the performance hit was about 5% in various benchmarks, although newer architectures suffer less. Given that Spectre attacks still appear to be very difficult to perform in practice, I wonder if we will start to see Speculative execution at the hardware level brought back as an option for high performance and closed environment settings.
I don't understand the "Though" part of this. The only reason any compiled language may be faster than another is because a compiler knows/assumes more about the code (either through users specification, or restrictions of the language).
I'm currently getting a Masters in CS (specialising in ML) after leaving a pure math PhD. I'd love a research type job from anywhere, let alone Amazon, but every job posting for roles with 'research' in the title says they require a PhD. Was your success a case of simply applying anyway? I know they say to apply anyway but it feels like a long shot to me. Do many of your peers have a similar background to you?
E.g "To install PyTorch via Anaconda, and you are using CUDA 9.0, use the following conda command". If they are shipping with CUDA perhaps that should be phrased more like "and you want to use CUDA 9.0". And of course you do indeed need your own CUDA installation if you want to build PyTorch from source yourself.
Its been considered ML ever since experts started writing books about ML (and before that, "pattern recognition"). Most introductory ML books cover it in some detail.
Good point. The precision loss will be pretty bad if the numbers added above have greatly different magnitudes, which could occur if the next number (x) is very different to the current mean (mu), or if the list is very long. The first issue could be partially mitigated by sorting the list first. If the list is very long, the above algorithm will necessarily add numbers of quite different magnitudes, but employing something like Kahan summation - https://en.wikipedia.org/wiki/Kahan_summation_algorithm , could partially mitigate that issue as well.
I think OP was saying he would be suspicious of a coder who was not interested in at least one area outside their professional domain, not any given area. If that's the case then I have to say I agree with them.
This document is amazing. I'm interested in working in quant trading when I graduate, and I can't believe I haven't found this before. Do you think if I wrote my own back tester and test strategy and put it onto GitHub it would be a positive in applications/interviews, or is that too basic to be seen as interesting/substantial?
> All blockers except uBlock Origin are available as JavaScript libraries which can be loaded in Node.js
I may be incorrect, but I think their testing methodology involved loading a JavaScript version of the Brave adblocker and ran benchmarks inside the Node.js runtime, and indeed the results may be different if they had used the Brave browser itself written in C++.
I'm sure this would be clarified in the book, but it seems their tips are best for gaining proficiency in quickly and accurately recollecting things you have already seen before, and not necessarily for learning something for the first time. I've certainty studied subjects where it takes ~20 minutes to recollect last lessons ideas into my head before I can proceed to new things, and for those types of subjects I think the 3 hour study block would be superior to the 30 minute blocks.
Very much agreed. See my comment in this thread. The value of being able to build decent ML models is decreasing rapidly, so the difference maker will be acquiring better/more data (which you often have little control over) or better feature engineering (which is basically gaining domain knowledge through better skills in mathematics, problem solving and reasoning). You showed that you were the better problem solver against 5 other consulting companies with teams that became used to wow-ing naive companies with little effort.
I have a background in mathematics and am currently studying Machine Learning in a Master's program. Soon after I started I came to the same conclusion as the author of the article. It is becoming very easy for someone with some basic coding ability and shaky mathematical foundations to do some online courses, emulate a few tutorials or maybe even sign up for a bootcamp and, voilà! 6 months later they are more than competent enough to quickly put together ML solutions that are close to an 'optimal' solution which requires much more in-depth knowledge and engineering time. Close enough that the quality/amount of data and feature engineering (basically, domain knowledge gained through general problem solving skills rather than ML skill) is far more important than the details of any model.
This is why, when I start applying for jobs soon, I plan to maintain and frame my skill set as a quantitative problem solver, not an ML specialist (as I had thought I would do when I started this Master's degree). Soon the value of being able to build a good ML model will decline, but hopefully the knowledge and skill set of those with a proper training and understanding of this field (mathematical ability, computer science/programming ability and problem solving ability) will still be valuable.
That is indeed how the subject has been developed historically and often introduced in optimisation courses. First one studies Linear Programs (minimize a linear function with affine equality and affine inequality constraints), then Quadratic Programs (quadratic functions with affine constraints), Quadratically Constrained Quadratic Programs, etc. So we often assume very specific properties of not just the objective but also the constraints, and we get some extra juice in these cases. Through this process of seeing what results we can prove in more generality, we finally arrived at getting a good body of theory and algorithms for minimizing convex functions with convex constraints. In some courses they start with the general theory, as the top down approach can save some time and brain space.
Lagrange multipliers can be applied to solve optimization problems with equality constraints. KKT generalizes Lagrange multipliers to allow inequality constraints as well.