It is absolutely the interface to excel that causes errors. I can take your excel spreadsheet, format cells, add things, change a reference and generally fuck it up and hand it back to you and you would never know. Whereas, if I change a text file then you can see what's changed. Similar errors are much less common in a programming language.d
So just to give you an opposing perspective of what you just said: "we have __one__ person that does all...". So apart from the fact he can't access a database appropriately, he's the _only_ person who does this work and none of it is documented or reviewable because that's the nature of Excel. There's some serious operational risk in that. This guy disappears tomorrow and what do you do?
This guy has three options: 1) short 2) go long 3) do nothing. Given he knows about this vulnerability, what is the "ethical" thing to do. But the stock? Don't engage?
I think, in fact, he has a duty to short the stock.
I genuinely think these type of thoughts come from having extensive experience as a programmer, that can consider building out systems that might reach the performance ceiling of R. For 99% of people multicore/distributed architecture will never even be a consideration. But I'm with you, in that having these things from a system engineers perspective would be incredible. There are other implementations of R out there (not just R GNU): Hadley Wickham's discusses them in Advanced R somewhere.
Very good, just look at sparklyr and the likes. People seem to think there is some hang up here. R has some of the best developed packages for working out of memory.
Multidimensional scaling is designed to preserve the most variance (or some variation of that idea) among points in a lower dimensional space. So for the most part, the cluster memberships assigned by clustering in an n-dimensional space are the same in the lower dimensional projection.
There's often no need to cluster in a higher space because the outcome is the same, and clustering is more expensive when there's more dimensions.
The straight forward answer you'll get is to study a pure/applied science in university up to MS or PhD level. This is probably half of the answer. The other half is being voraciously curious in many many areas and actively self teaching (deliberately or not) through reading, practicing and generally immersing yourself in whatever you're interested in. Working on "big problems" requires experts across fields with unique perspectives and knowledge. Getting a PhD for XYZ might mean your smart, but not necessarily unique.
There seems to be some bizarre skepticism by some HNers of financial institutions' AML practices. But the thing is, AML is really difficult and expensive for banks, hiring floors of consultants dedicated to AML. This is a billion dollar problem (or more?). Whatever about training a CNN to classify images, an effective ML model/pipeline to tackle AML will break the back of a huge problem.
From Ireland, so I know what you're talking about. I suppose my point was really that Lidl and Aldi have swept into an market over the last 10 years, where customers are hard fought for, seemingly with ease. The Tescos of the world definitely can't compete at with own brand quality or price.
I don't know about the UK, but in Ireland Aldi and Lidl are proliferating across the country. The growth is so extreme, they're hiring grads at €60k plus an Audi in order to train them up as shop managers. I mentioned case study because they've taken a fundamentally different approach and it's working. Their low cost structures across supply chain, product base, building construction and shop design are how they're carrying out this coup.
Aldi and Lidl are blasting through other super market chains with their low cost, high quality goods. If there were ever perfect case studies for usurping incumbents in a sharply competitive market Aldi and Lidl are it.
The only cost tied to these issues is the salary costs for a statistician/data scientist. The best tools of the trade are open source. You won't find lowering costs until more people are educated or trained up in these fields.
I agree and I think it's clear TPOT and similar tools, are the first generation. Genetic algorithms might be slow/costly but the concept is there with an integrated, implementable solution. If it's as useful as I think it could be, there will be a flood of modules, libraries and packages developed with efficiency, UI and domain specific improvements.
There's a key statement here that most will either miss or never reach. The second last paragraph
> Machine Learning Automation
Tuning models is a pain, and if you don't understand the model and all its parameters (like most software engineers who's job is the write software and not build models!) it takes time and can be immensely frustrating. Randal Olson[1] just announced TPOT[2], a Python tool that "automatically creates and optimized machine learning pipelines using genetic programming". This is going to be a huge lever for engineers wanting to experiment/implement with ML algorithms.
This article's attempt to contribute Leicester's success this season to "statistics" is just painful. As if every other team in the Premier League and three tiers down don't use extensive analysis in every aspect of selection, performance and scouting. Ranieri's style isn't bizarre or unseen-before. This isn't Moneyball.
I understand where you're coming from, but these two lines are exactly 'taking a list of libraries for import'.
packages is a character vector of package names, lapply is by definition 'list apply'. We're taking a list of packages and applying the library function on them.
This seems complicated if you're not used to it but R is a functional language. Approaching R from this perspective makes it a powerful, flexible.
In terms of managing packages in session, readability and keystrokes, I think it definitely wins out over successive library calls. It's common to have >5 packages in any one script, especially if you avoid base R like many do.
This is a really interesting post. I have to say the data science team at Stitchfix are clearly doing really good, applied work that is central to their business. It's so cool to see.
Here's a tip for any R users who read through the code and (like me) is pained by repetition. Instead of using
This book is extremely practical - I would definitely recommend it for doing actual time series analysis. That said, it's not a learn time series book. It's a do basic time series in R book.
Well funnily enough I started back at what a matrix really is. What does it mean to multiply them, add them, invert them and so on. The first good resource I used was Better Explained! I also took out some Linear Algebra text books from the library. First was Linear Algebra for Dummies (don't knock it!) and then an undergraduate text book. I've still a ways to go but it was a good start.
Just a heads up: Jupyter Notebook is not an open source alternative to Mathematica. Originally, Jupyter was iPython notebook, an IDE of sorts for data science and analysis in Python, by writing code and markdown together in a more coherent and integrated way. Then they incorporated a host of other popular open source languages for computational science such as R, Julia, F#, ect., so that we could use the best tools for their task, all in one document.