WebXray
webxray.org
webxray.org
Re SQLite, he replied: "@coleifer: there was a sqlite branch, but it doesn't scale well to many-million record sets which is what I have been doing. the design of the software allows drop-in db replacement, it just lacks the code. I can't decide to go back and sqlite or to just make a web front-end."
Comments by some (not all) new accounts are killed by default when they look like possible spam or troll activity. It's not a great solution because it leads to false positives like this one. On the other hand, doing nothing is worse. It's a hard problem, and it certainly doesn't mean that the user is banned.
We're about to release software to let the community unkill these, which we hope will be a much better solution.
Naturally this means a different user agent and finger print which could ultimately mean the script is fed a different page altogether. The odds of that are probably low but still; someone could have a really shitty website that uses hundreds of trackers but could serve WebXray something completely without them.
I would like to see this type of stuff as web browser extensions. That way we can get the exact, most correct information possible. Also would simplify a semi-convoluted build process that seems to have tripped up a few readers.
https://www.ghostery.com/en/faq/how-does-ghostery-make-money...
Also, see the TED talk 'Tracking our online trackers' by Gary Kovacs - http://www.ted.com/talks/gary_kovacs_tracking_the_trackers?l....
since I am hell banned I can only edit this comment to reply.
as for video, I discuss research here: https://www.youtube.com/watch?v=OqW8erWi1Wo
but if you mean screencast of the software I haven't had time.
---
@coleifer: there was a sqlite branch, but it doesn't scale well to many-million record sets which is what I have been doing. the design of the software allows drop-in db replacement, it just lacks the code. I can't decide to go back and sqlite or to just make a web front-end.
---
@captn3m0 I have an academic paper in revision that is an analysis of the alexa 1M list, I also have other projects i development.
---
@snorrah: this was my first python project, so I went with the newest version. it's made a lot of things very difficult, especially porting to a web version.
---
@linuxlizard: proxy is problem when you want to do a lot of concurrent tests, I usually load about 64 pages in tandem to get good speeds on large sets.
---
@radmuzon: webdxray runs large batch jobs, so you can get lunch, and come back with all of pages analyzed. I know it does work on windows, and I apologize for not being able to provide directions...see comments above.
---
@TeMPOraL: thanks!
---
@pearjuice: yeah, I wish there was an easier way to get python3 to talk to mysql, that's the biggest PITA.
Here's a dashboard showing third party usage for The Guardian over the last 30 days: https://speedcurve.com/demo/thirdparty/1/1/chrome/1/30/39tfn...
Great to see an open source list of domains linked to organizations. We've built our own list as well and we'll look at contributing them to this project.
(Disclaimer: I'm the founder of SpeedCurve)
---
I see the hell ban, I'm posting all my replies at the bottom in a comment I'm editing - could somebody note this above?
Do you have a technique your using to match domains to organisations sometimes it can be hard to discover.
I was about to ask why isn't there a simple, unified build tool for ANYTHING, but I think that is what got us here in the first place...
It used to be that being a scripter was a stepping stone on the way to developers, mostly because back then scripting could only get you so far. Now, you can apparently make an entire career being a scripter, if said code academies are correct in the promise that they can find you a job with the extremely shallow curriculum I've seen them provide.
This isn't a bad thing, it's pretty amazing that it doesn't take a decade of dedicated study to do so much anymore. It's just that in our current culture we lump them in with developers because they are clearly more than just users. We still expect a set of resources put into a Github repository to have a significant amount of new programming, rather than just being glue code between a few commonly available libraries. But that's more of a problem of our lack of ability to differentiate between large, greenfield projects and small, configuration-oriented projects at-a-glance than it is a problem of programming being "too easy".
Though, if we recast scripters as users instead of developers, then it's a terrible thing. It means that the real software developers of the world have written a bunch of software with a really, really shitty user interface.
Either way, it's not the scripter's fault.
I can understand the thought behind ad blocking (it fucks up the experience), but a tangential referral code? Who cares?
Think the hellban must just be a glitch - so.. Amongst other things, Tilbert wrote:
"
... my hope with open-source is people will help add to it. if you want to help, I'd love that!
---
I see the hell ban, I'm posting all my replies at the bottom in a comment I'm editing - could somebody note this above?
---
I think you guys/gals have a point [about the referral link to DigitalOcean] the referral link is gone. ;-)
---
@coleifer: there was a sqlite branch, but it doesn't scale well to many-million record sets which is what I have been doing. the design of the software allows drop-in db replacement, it just lacks the code. I can't decide to go back and sqlite or to just make a web front-end.
---
@captn3m0 I have an academic paper in revision that is an analysis of the alexa 1M list, I also have other projects i development.
---
@snorrah: this was my first python project, so I went with the newest version. it's made a lot of things very difficult, especially porting to a web version.
---
@linuxlizard: proxy is problem when you want to do a lot of concurrent tests, I usually load about 64 pages in tandem to get good speeds on large sets.
---
@radmuzon: webdxray runs large batch jobs, so you can get lunch, and come back with all of pages analyzed. I know it does work on windows, and I apologize for not being able to provide directions...see comments above
" * There's more. Enable 'ShowDead' in Profile to see rest of his comments.
A request is the dynamic, transient action which occurs when a client such as a browser initiates a connection to a server and presents a command like GET or POST.
I suggest an opening paragraph along the following lines:
"WebXray" is a sort of web crawler which analyzes a given cluster of pages for their relationships with each other, as well as external pages which they link to.
It provides information about pages which direct a user's browser to various sites for the purposes of tracking, using tricks like hidden images.
(Do I have that approximately right?)
onResourceRequested
Introduced: PhantomJS 1.2
This callback is invoked when the page requests a resource. The first argument to the callback is the requestData metadata object. The second argument is the networkRequest object itself.
The requestData metadata object contains these properties:
id : the number of the requested resource method : http method url : the URL of the requested resource time : Date object containing the date of the request headers : list of http headers The networkRequest object contains these functions:
abort() : aborts the current network request. Aborting the current network request will invoke onResourceError callback. changeUrl(newUrl) : changes the current URL of the network request. By calling networkRequest.changeUrl(newUrl), we can change the request url to the new url. This is an excellent and only way to provide alternative implementation of a remote resource. (see Example-2) setHeader(key, value)
Windows Specific Instructions
Get a linux cloud server (which cost fractions of a cent per hour these days).
Ubuntu is the easiest flavor of Linux to get started with and the directions
above will serve you well. Seriously, this is your best option. You can do it.
I'm both confident in your abilities and proud of you for taking this important
step in life.
Or is it more about not wanting to track down how to install software on Windows?
(EDIT: and as others have pointed out, it's kind of cheesy to use the moment to plug your referral code for DigitalOcean)
in response to referral link comment: I host the site on my own dime so I didn't think getting $.25 from DO would make me a bad person...also, I really don't have time to learn windows to make instructions. I'm a grad student and make slightly above minimum wage annually, if I were to be evil and greedy I'd probably not be doing what I do with my life. I'm giving away thousands of hours of work and data I could make money off of doing evil things.
---
I think you guys/gals have a point, the referral link is gone. ;-)
Is that a sign that developers are starting to not care about Windows?
It's really no worse than having adverts or Amazon referrals. In fact it's less intrusive than the former.
I fully agree. I wouldn't remove it at all, just add a a mention that it is your referral link.
This is free software folks. If the coffers aren't held out with donation links, I see nothing wrong with him getting a cut elsewhere.
In fact, the nature of this software is unthanked work and relatively unknown to the general public. It addresses a real issue, and it addresses it for free.
"Cheesy" is criticizing him for the referral link.
Yeah, things that are free need to make money somehow, hence ads and ref links. It's asinine to think that a person should just give up their own cash to provide a free service. It's the epitome of using someone as a means to an end; and you're the one getting used.
any sane person can see that releasing my intellectual property for free wasn't an evil scheme to make $.25 from digital ocean. ;-)
>in response to referral link comment: I host the site on my own dime so I didn't think getting $.25 from DO would make me a bad person...also, I really don't have time to learn windows to make instructions. I'm a grad student and make slightly above minimum wage annually, if I were to be evil and greedy I'd probably not be doing what I do with my life. I'm giving away thousands of hours of work and data I could make money off of doing evil things.
>its not worth putting the link there and have people judge you / your intentions.
This makes no sense to me. His intentions are clear. Far more clear than the intentions of third party HTTP requests on the big websites you visit. And you didn't even need to hear it from him himself to know that he hosts his site and needs a few shekels to support it. Of course, you're entitled to your own opinion, but I could really care less if people "judge" me for a DO link, and I hope the tilbert feels the same way.
I don't understand in general this hate for people who post referral links. Do you not hover over a link before you click it? See all the garbage appended to the query string in the URL? If I don't want to click it, I don't. I don't hate the person for adding a referral link. I don't understand the hate for the people who do. Again, you're entitled to your own opinion but I'm not a fan of this minutiae being hammered in this HN thread. I'll just go to the website on a clean window and search the same product/website/service.
Finally, the third party advertisers are judging him far more harshly for this software, anyways. Their corrupt model is (in a small way) subverted. If he was worried about "judgment" he wouldn't have offered this software in the first place.
It was far less strong than the initial attack on his adding a referral link in the first place.
A referral link is (arguably, again, opinions here ;) ) much cleaner and more transparent than having your analytics data being pushed out to a dossier tracking your behavior, emotions, and extrapolating patterns on the two.
You can hover your cursor on a referral link to see what it is. You can even open up a Javascript console and see where the true location of any given link is taking you to, should you click on it.
You can't see what's being done with third party gathering information about you.
And funnily (read: ironically) enough, its the entire point of this software in the first place. I repeat that I'm confused about the concern over this minutiae considering the fact that when you go look up something online, someone is getting paid in some way, shape, or form. I don't mean to come off strong, and I apologize if you took it in that way.
A referrer gets paid in creation of a consumer whereas an advertising company gets paid as a spy, essentially. Using Amazon as an example, isn't the purpose of the service to sell you something? Instead of selling you as a product?[0]
[0] https://medium.com/@unsetbit/dear-amazon-you-dropped-somethi...
On the third hand, if the task was trivial, then there would be no reason to complain about a lack of bespoke Windows instructions, since they would almost write themselves. Of course if bespoke Windows instructions are non-trivial, then obviously it's hard to see merit in the complaint.