NumberCruncher

>> someone with a bit of a spectrum disorder.

> I'm surprised I had to scroll this far to see someone suggesting this, since it seems extremely obvious to me that this is part of the case.

The whole thread has currently 1471 points and 681 comments. I seem the be the 4th person realising, that the whole discussion is about "communication advices given by someone fighting the spectrum disorder". I don't know wat to say.

Small remark: I think you have a typo there. Jancsi = Johnny in Hungarian.

> I have no hate for classic statistics, sorry if I gave that impression.

You didn't, but a lot of HNlers do. Maybe I should rant on them, that's true.

> many base assumptions made by e.g. statistics will not help you at all with neural networks

There is lately a lot of hate against classic statistics on HN. I don't know why. Does it help to understand why and how NNs work? Not yet. But saying that it is utterly useless and won't provide any useful insights in the future sounds to me like telling the young Steve Jobs that dropping out of college and taking calligraphy classes instead of is utterly useless. And still, I am writing this on an Apple product, which set the standards for digital typography...

> Rand implies that you have to do your own studies [...] which is 100% wrong - Facebook will do them for you

So FB, who earns on my spending, offers me to measure for me whether my spending makes sense for me. Why would FB ever tell me to spend less? Don't they like money?

> My feature store hasn't supported time-travel yet. But many SaaS implementations do, including Tecton, Hopswork, Splice Machine, et

Good to know, I wasn't aware of them. At the end of the day the make or buy decision is influenced by the cost factor. In our case the development costed ca. 50k€ (obviously no SV salary), we had a user base of 10 data-scientists (incl. myself) and someone in house who knows the ins and outs of the system and can implement new stuff. I have to admit I was sharing my office with our DB architect, it would be hard to put a price tag on his support. I don't know how the pricing of the SaaS solutions compares to this.

Almost all features (maybe with the exception of gender) were aggregations over time like "share of emails the customer opened in the last X days", or "did the customer use service Y in the past: y/n", etc. If you use a pseudo SQL code like:

CREATE OR REPLACE target_table AS

SELECT

i.customer_id

... some aggregation processing data from t ...

FROM init_table i

LEFT JOIN transactional_data t

WHERE i.... = t....

AND t.time_stamp BETWEEN i.reference_date - (X days) AND i.reference_date

GROUP BY i.customer_id

than depending on what you put into the init_table, you can obtain:

- in the prod environment: actual feature value(s) for one ore more customers (reference_date = current_date)

- in the model-dev environment: historic feature value(s) for one or more customers for one or more past dates (reference_date = [past_date1, past_date2, ...])

In different parts of the system (prod, model-dev) you can use the same SQL code, you only have to replace the init_table and the target_table. Implementing a new feature system-wide basically means putting a new aggregation into the SQL code, only once.

I used PL/SQL as glue code, like someone would use python. I.e. executing SQL code in a controlled manner, creating dynamic SQL the right way (out of the box Oracle solution) and the wrong way too (sticking SQL fragments as strings together), writing code-generators, managing indexes/partitions/views/materialized views/etc. for performance tuning, implementing centralized monitoring for bug and performance tracking (and to know how the projects of my colleges are going forward), and last but not least, for packaging. At the end of the day you have a library/modul (PL/SQL package) exposing functions (PL/SQL procedures). The "library" is partially type-checked on compilation on the DB, and than you grant the necessary execute rights to the DB users (distribution to the data-scientists) and they work with the historic results on their own VM (DB schema). Current code is searchable in the DB, all the users use the same (actual) version, code history is checked in into bitbucket. If a user wants to "fork" and create new features, he/she can just write plain SQL against the prod DB on his/her own and the new code will be migrated through pair programming into the prod environment.

Sorry for the formatting.

> What I find missing in the internet is that people who have been doing this for years are not writing about this.

In my last job I implemented a feature store from scratch with ca. 500 hand crafted and ca 2.500 with code generator automatically generated features. It didn't only serve the current value of the features, but the data scientists could populate an 'init' table manually with (customer_id, reference_date, target_value) tuples, and the pipeline re-calculated the historic feature values for the given customer and reference_date. So if the data scientists came up with a new fature definition, after implementation (5 mins - 2 hours per feature), he - and all other data scientists - immediatelly got access to the features's history. We had so many features, that I had to implement an automatic feature-prunning, otherwise the users got lost. We could train, test, validate and deploy models within 24 hours (model fitting run over-night). When I left the company, we had ca 40 models in production, managed by 1 person (by me) in part-time (3-4 hours a week).

This was in an off-line business, so we didn't had to deal with latency by feature serving and didn't had to be able to change a feature's value during the day, so everything could run batch based over night.

Why I didn't write about it? Because it was implemented in PL/SQL running on Oracle ExaData and in SAS. No one cares about feature stores implemented with tech like that. People care about models trained in python, ported to Scala by Java devs, running in docker on k8s, features coming from HiveQL, sqoop, oozie or Spark and stored on cassandra, MySQL or Elasticsearch. But do they have a feature store with built in time-travel functionality?

They are also handy at dimming/turning lights on/off or starting Spotify playlists while in the shower. Nothing essential TBH.

[Edit] Google Home seems to be pretty good at separating your voice from background noises, like shower/TV.

> Maybe we can develop healthier forms of social media.

Sounds like "let's develop a healthier form of crack" to me.

There was a saying before 2008: "How do you start an AI based FOREX trading fund? You pretend to be the russian prodigy developing the algo, while I do the insider trading".

I know at least one newly opened semiconductor plant.

Link: https://www.infineon.com/cms/en/about-infineon/press/press-r...

There is a difference between an academic career in a field which has some real life applications, like CS, chemistry, economy, etc. and an academic career in a field which doesn't. The second group is basically nothing else than a "hobby for rich kids", where someone else has to pay your bills while you study, and high probably afterwards too.

If you aren't a rich kid, go for the first option.

The "the right tool for the right job" applies for ML topics too.

If the job involves "looking smart and innovative" for whatever reasons, people tend to err on the side of overly complex solutions.

On the other hand if the advice "let's just go with an SQL query built on a stack of egregiously oversimplifying assumptions" comes from someone, who doesn't know how SQL and linear regression / logistic regression with binning/bucketing / simple decision trees work, I would ask for a second opinion. Because a huge part of the retail banking, non-life insurance and marketing business is running on this simple stack. Obviously profitable.

If the same advice comes from someone, who knows when to use deep learning instead of XGBoost and why, I would go with his/her advice. And I would try to keep him happy and on my team.

> Is this normal?

It is. At my last job the first ticket assigned to me was about fixing the bug "average is greater than 80% percentile". When I told them that this isn't a bug, they thought I am just kidding.

I used to work in a team where everyone except me had studied at the now closed Central European University. My current personal manager also sudied at the CUE. Very clever and open minded people. And what is "your experience" based on again?

In the video you posted you say "I don't want to work with blurry images". Is that not an image a human could work with (drive), maybe by reducing the driving speed to 50-33% and having more time to inspect his surroundings?

There is even a series about them: https://en.m.wikipedia.org/wiki/Bad_Blood_(TV_series)

Second comment on the video: would of been nice to see inside of the car.. its easy to say "there was a human driving!"

I mean, I am working at a start-up too, fake it till you make it is our daily bread...

> very least it'd be handy to have self-driving at least on, say, highways

Me too, like driving home after a long day on the highway, being sleepy, in the night, while raining. Good luck with that: [1]

[1] https://9gag.com/gag/azMv16z

> Right, that was pretty much the point of the article.

Agree. Nevertheless there is a flood of articles showing videos recorded under similar conditions but missing this point.

> riding around Miami Beach in self-driving test cars several days a week

I watched the video. 2+1+2 lanes, no trucks parking on the side, no bicycles, no bicycles-lanes. No pedestrians crossing the road, no pedestrians at all. Clear blue sky. The car can take a left turn alone. That's cool, until you realise that the majority of the cities outside of the USA doesn't look like Miami Beach. I highly doubt that any "self-driving" car could travel through Munich today without killing at least a couple of pedestrians or cyclists. And still, Munich is like a dream compared to Rome or Ho-Chi-Minh-City.

BTW, only because sometimes I feel safe enough to ride my bike without hands on, it doesn't means I have a self-riding-bike.

Every time a surveillance system and violation of privacy rights is advertised in the EU as a solution against child abuse and trafficking I ask myself how such a system could have changed the outcome of a case like Dutroux. Would have been the dozens of witnesses and police officers involved in the investigation suicided a way sooner, later, more silently, or at all? We will never know...

> You'd expect it to complete to "O Romeo, Romeo, wherefore art thou Romeo?" - a very famous quote.

What about completing it to "O Romeo, Romeo, brave Mercutio is dead", based on the context, as advertised?

Hmm... I should have written making up "blockers" disguised as policies stem from logic. I consider the loop closed. QED

I used to work in the public sector myself, so who am I to judge your way of making a living? But still, I don't like if public resources (work-time and tax money) are wasted just because of political reasons. But that is entirely my problem.

My Excel skills are a bit rusty but I remember the times when it was advised to not open .doc or .xls files from an unknown source because harmful VBA macros could be executed on opening the file. Did this changed lately? If not, how is opening an .xls file different from installing a non-tested python package from security perspective?

Let's agree that we disagree. Like the German public news and the online media disagree on this topic.

> I don't believe anyone involved

I like the version "I don't believe anyone if money and/or politics is involved".

> It's too hard to get access to GIS software

It took me a whole 5 seconds to google [1] up. As far as I can remember Anaconda can be installed without admin permissions on any PC and pulling the needed packages is a matter of minutes. Or a Tableau Online account with Tableau Creator desktop app having similar GIS functionality costs a whole 70 bucks a month, OMG. The public sector seems to excel in making up "blockers" to justify why shit doesn't get done.

[1]: https://blog.jupyter.org/interactive-gis-in-jupyter-with-ipy...

There are so many other places on the internet to discuss topics like this.