mziel

I don't see how shadow profiles can be justified under GDPR and ePrivacy. If have identifiable information (just "some" ID will do, but also IP and various other fingerprints) then you need to allow for deletion/takeout/opt-out. Current strategy of implied consent ("if you're on this site you agree") is strictly not allowed.

You also need to make sure you implement true deletion instead of a DB flag (and still use that stuff on the backend).

That's not true in the UK, there's no ID card here. IMO it's quite a nuisance, for example you need to always bring 2 documents to verify your identity and address for a variety of situations (opening a bank account, even opening a new savings account at the SAME bank, renting a new place, new job, mortgage, requesting information from the government).

This is the case with most organisations. You have a finite amount of resources and attention, therefore you need to prioritise.

Most GDPR chatter started picking up only in the last few months (of course big orgs have been preparing for the May deadline for a while already).

IP addresses are identified as personal data in GDPR. They're not exposed in the frontend, but HN might use them e.g. for logging.

Also things like deletion, takeout and consent/opt-out need to be supported (provided that HN falls under GDPR).

Regarding "they got the users’ permission initially" this is true for the users that signed up for it, not anybody in their social graph. GDPR treats data about a user as data belonging to this user. Those people have definitely not consented to having their data mined for this use case.

Next (as I understand) the consent was for research purposes, not for the CA targeting. So under GDPR Cambridge Analytica could be fined 4% of global revenue or €20M - whichever is HIGHER [1]

[1] https://www.gdpreu.org/compliance/fines-and-penalties/

You're supposed to enumerate all uses of the data (and they need to be sufficiently detailed and specific). The user has a choice to opt-in/out of each of them separately.

There is currently no detailed description as to what the definition of "sufficiently" is. For example:

- can I use your data to build a targeting machine learning model?

- can I use it to target you?

- do I need specific opt-in for every model?

Most things in GDPR are not specified in order to both give flexibility to the sites and to reduce the number of loopholes (which are technically legal but against the spirit of the law). You need to decide on the implementation and be ready to defend it in case of an audit.

You can read more about the cookie law here: https://www.cookielaw.org/the-cookie-law/

Basically EU wanted sites to obtain consent to use users' cookies (and for the users to give/take away that consent). However, pretty much all the sites just decided to provide you with a banner saying something like "if you're using this site you agree to our cookie policy". Therefore the law became ineffective and just a nuisance to the users.

This notion of "implied consent" is being actively fought with GDPR. You have to provide explicit consent to the usage of your data. And more importantly you can revoke it (at any point) and the site can't deny or degrade the service (unless the data is strictly necessary for a specific action related to the service).

With ePrivacy this will go one step further. Right now you only need to provide opt-out, which means most people will likely leave it as it. Going forward those additional services (marketing purposes, ad tracking) will need to be strictly opt-in (and there's already internal research done in some companies showing that marketing/ad opt-in rates will be 10-12% at best).

Looking forward to May (when GDPR officially comes into force). Provided that it doesn't end up like the cookie law (and there are explicit provisions in GDPR and ePrivacy to avoid that) this might shake up the ad industry:

* Explicit consent for non-essential data use, you always need to provide opt-out without degrading the service

* Opt-in/out separately for every activity (no more "research purposes")

* Data deletion and takeout. Maybe in the future EU will also introduce some standards for the takeout, which will allow us to migrate between services much easier (as we now can switch between banks or telcos in a semi-automatic way)

GDPR has been there for 2 years [1] and will start to be enforced come May. Facebook has a presence in the EU, since they're selling data about European users to European companies. Therefore they need to comply with European law.

[1] http://eur-lex.europa.eu/eli/reg/2016/679/oj

| they're there for convenience only. They can close them next month

Citation needed? Facebook gets revenue from their ad network, which is used by European business customers and targets European users. Therefore they need to comply with European law.

I will repeat my comment as you seem to be repeating the same argument.

It's not "consent" as understood by GPDR and ePrivacy. You had no recourse not to give it, therefore it was not willing and informed. Implied consent ("agree or leave") is not deemed sufficient by GDPR. According to the law you can't condition the service you're providing on collecting unrelated (to that service) data.

Again, this is not deemed sufficient by GDPR. According to the law you can't condition the service you're providing on collecting unrelated (to that service) data.

It's not "consent" as understood by GPDR and ePrivacy. You had no recourse not to give it, therefore it was not willing and informed.

It's not "consent" as understood by GPDR and ePrivacy. You had no recourse not to give it, therefore it was not willing and informed.

I find the same thing even with critically acclaimed "Thinking Fast and Slow" or "Power of habit". On the other hand sometimes blogposts may not be exhaustive enough. IMO roughly 60-100 pages is the perfect format – you don't need to overly simplify the topic (like in blogpost, newspaper article) but you don't need to repeat yourself 10 times over (like in 300-400 page book).

The article has it wrong, as I mentioned in another comment. Polish word for tea is "herbata" which comes from "herbal tea".

There's a lot of very bad pop-sci books on this topic. Do you mind sharing the ones you read and recommend?

Specifically in Poland: https://pl.wikipedia.org/wiki/Pu-erh

Translating the article: Pu-erh (chin. 普洱茶, pǔ’ěr chá) – a kind of tea that is classified as red tea in Poland (black tea in China since the Chinese classify the tea according to the colour of the brew, as opposed to Europeans who classify tea according to the colour of dried leaves).

Interestingly pu-erh is often referred to as "red tea" in Western countries.

It comes from "herba thea" (herbal tea) so you could argue that it should still be classified under "tea" origin.

Good intuition :) https://en.wikipedia.org/wiki/Interquartile_range

Do you have a write-up of your experience by any chance?

We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.

GraphFrames are a graph-based model. Spark DataFrames is the model directly inspired by R/Pandas.

There are ways to account for that. A model can be fit to race, and then you only predict "on top" of race (meaning residuals). You use that model, which is independent of race.

While obviously the answer is libraries and data types (DataFrame is a very powerful semantic, replicated later in Python, Julia, Scala/Spark), R definitely IS a functional language:

- closures, lambdas

- high-order functions

- a lot R processing is map/fold-like (even though it's lapply instead of map)

hetero-004

algs4partI-010

algs4partII-007

I think R's data frames is by far the most intuitive abstraction. There's a reason why it was duplicated in Python, Spark or Julia. And R has a lot of utilities for converting input strings, dates, JSON etc.

I can see that Wes is a reporter for Spark issue (https://issues.apache.org/jira/browse/SPARK-13534), what are the plans (if any) for tighter Spark/Python/R DataFrames integration?

It only works once. My usual response to such practices is to not return to the store again (and probably buy online). They lose in the long run.