349 karma · joined May 5, 2018
this is based on a very dubious assumption that bringing compute closer is the only path for innovation.
and even that is not true, you could imagine compute being even closer with a direct brain interface (actually you could consider google glasses to be an attempt at bringing compute closer)
while I generally agree with your idea, a little counterpoint: it depends on a way you consume those materials. If you do that thoroughly, actually read and analyze source codes or examples, try things on your own as instructed, complete excercises and so on there is absolutely no point in doing multiple thins on the same topic. But if your preferred method is fast skimming, skipping some excercises and so on then skimming two or three books/videos/whatever on the same subject could make some sense. The latter method is very chaotic and is probably not the best idea to apply it for learning the fundamentals, but I just wanted to point out that such a learning technique does exist and sometimes makes sense, especially if just the general understanding is your goal.
On the other hand their business requires the ability to discover candidates by HR people so I guess that completely disabling search/discovery is out of question. Of course a simple limit to a number of queries or their reach would still be a huge problem for the scraper while not being a problem for most of the users and therefore not hurting the business.
Then considering that such massive scraping is probably already illegal and additionally the operation is being done by some intelligence agency meaning that legality is not an issue we can do a lot more that simple scraping using some proxies. This could include use of botnets (free resources, mich wider and more realistic pool of IPs) and/or hacked accounts (to scrape as a verified reputable user).
This all of course makes such a scrape a lot harder and probably not something that a single person with just a personal budget could do,but I believe this is still within the reach of even a small organization. And I'm 100% certain that this does not require multi-billion black budgets or large datacenters hidden underground.
>a) Not having a profile isn't definitive. You could have missed it, it could have been locked down, or the person joined late.
of course you are right with that, but then I could have full-scrapes being done once a year or even more often. While missing a profile once is obviously quite realistic and actually expected I assume that it would be unlikely that the same profile is ommited 20 times in a row given that the scraping has generally been proven to be effective.
Additionally I was initially thinking about using such data as one of the metrics not as a definitive spy-detector. Your account missing in my 2009-2017 scrapes and appearing just recently does not make you a spy but does increase a likelihood of you being so.
>You can't go back to build your baseline. You had to have the foresight to scrape it then or count on one of the breaches to establish who had accounts when.
Thats true. And even with data available from breaches might not be accurate or even be intentionally altered. But then again not everyone runs an intelligence agency
if I actually scraped all the profiles in 2009, then in 2014 and then in 2019 I could tell whether an account is a 10+-year old account by simply checking if it is available in my 2009 snapshot. Does not matter if the social network displays or leaks profile age in one way or another. If it's not in my 2009 and not in 2014 snapshots then that profile is 5- years old. With frequent enough snapshots I would get even better timing resolotion. Now given that it's neither that hard nor that expensive to scrape or store that amounts of data, such an approach would actually be feasible.
1MB memory usage would not hurt most of the low-resource users while having 1MB memory for each of the millions of computational units in a brute-force rig would hurt the attacker a lot more. I assume that it is relatively easy to have a huge number of brute-force cracking units in a simple GPU/FPGA/ASIC rig but if each of them would additionally need lets say 1MB of RAM that would present a serious problem for the constructor of such rig.
I do get that AES will happily run on a 20-year old smart-card simplified cpu without a problem while my fictional 1meg-requiring cipher would be problematic, but then again it's 2019 and 1MB is not that of a problem for the least powerful smartphones or even embedded systems.
you could imagine a cipher with a computational/memory[0] intensity so high that the brute force attack could be practically infeasible even at lower key-sizes. I am not advocating using smaller keys, just pointing out that there is more to brute-force resistance that simply just the key-length number. (That is probably the numerology the GP was referring to)
[0]Since memory intensity instead of pure computational intensity seems to be employed by some hashing alghorithms I wonder is that would be possible block ciphers as well. The way I understand this, it makes ASIC/FPGA brute-forcing a lot harder/expensive
Given how simple and relatively cheap that is I would be very suprised if they don't. This is a kind of operation that a few people with their private budgets could pull off (I mean just scraping the contents and storing a single copy , not the analysis part) so don't expect powerful and not-so-powerful organizations or goverments restraining themselves from doing so. This is neither expensive nor difficult while being extremely valuable
the argument I was trying to make is more in the sense "is it worth to take (even a minimal) risk optimizing performance given that this very performance gain is not (that) important overall"
That being said, I do understand that there are a lot of areas where performance/energy budget is tight and this actually makes a (huge) difference
B) very,very debatable, especially this piece: "understand the space well-enough"
C) I tend to agree, but then (especially with hardware-supported crypto) there is not much to be gained by reducing the number of rounds
I laughed at this way louder than I'm willing to admit
Then if they also happen to anyhow benefit from the existence of those fake accounts then it becomes even harder to justify costly filters and leads to a sufficient-enough-to-not-look-obviously-weak grade filtering
From my very limited experience there are two main categories of websites:
1. Using curl and/or [library of your choice] in a [scripting language of choice] is enough
2. Forget about not being detected without a full blown browser unless you want to spend endless hours trying to emulate whatever is needed to be emulated and also willing to burn some accounts and IPs in the process.
[0] dedicated in a sense of lets include some script specifically designed to detect Selenium. [1] for example it's kind of unlikely that non-bot visitor is using IP from a range used by amazon instances. not sure how often this is used but I assume that most bot-detection systems would use that information at least as one of the metrics
Also there is a more common scenario where updating one thing requires updating other packages and through a long chain of denependencies one of the pieces being updated has something missing in the new version (that was available in the previous version) and anything that relies on that will stop working.
Anyway, even the best case scenario where everything is perfectly fine after the updates still requires detailed testing to ensure that really everything is as OK as it seems. So even then this is not totally free
But then of course, it may still be the easiest path for improving the performance.
On the other hand, when I'm later visiting pornhub I'm unlikely to consider purchase of those tools even though they may appear relevant to my interests based on my previous browsing history. On the other hand, I am more likely to be interested in meeting hot moms from my area then - which is again solely context based.
while this is orthogonal to Shopify API or your post at all, since you mentioned the need of updates, I just wanted to use that opportunity to vent my frustration with the constant push to update everything all the time and judging any piece of software by using "when was the last update" as a metric.
The problem I see is that not all apps or libs need (frequent) updates, many (maybe most) do need them, but some don't. They provide some functionality, they do that well and you could call them "complete". Maybe some security fix could be needed from time to time, but with a mature code being in use for many years event those are not frequent.
For example, consider something like ping utility. It does what it does for many decades. There was a need to add IPv6 support, but that was almost two decades ago. Why would anyone need to update it? I do not want any additional functionality, I don't want it to send emails or have social media share button. I want it to send ICMP echo requests and receive ICMP echo replies and nothing more. Aside from some security fixes no updates should be needed for 10+ years. This utility is done. It should not be thrown upon just because there were no updates for many years.
While of course neither ecommerce or Shopify platform are "done" and they get many updates now and will get updates in future it does not mean that some functionalities could have reached "done" stage.
For a "complete and done" addon, there could be a need for a security fix from time to time. There could be a need for some adjustments if a major browser introduces a new deviation from JS/CSS/HTML standards and forces everyone to update their code. But those events happen from time to time, possibly not that frequently. This means that some addon/plugin would not require any updates during the periods between those events and those periods could be many months/years long. But hey: "this addon did not receive any updates for 13 months, it must be really bad and should be avoided". This leads to a situation where a competing solution with tons of bugs will look better just because it receives two updates a week.
not only that. One insecure system in the area could be infected and then controlled to attack other networks in the area. With a high density of IoT devices you could have malware spreading wirelessly device-to-device. Also with a wide IoT adoption with such devices being used by public/utilities companies (smart street lamps, leakage monitoring in pipes, smart electric grid etc) you don't even need a densely populated area to have that problem.
>I am more concerned about the bigger threat, global attacks.
I once read something among the lines that IT security needs its own Pearl Harbor event. I can neither quote it exactly or attribute it to any source