tlibert

personally, I feel the real magic of VMs is I can play around, keep notes, royally screw up my system doing something idiotic, trash the machine, spin up a new one, and resume from my notes. I'm old enough to remember the dark days of RHL CDs running on actual computers...

thank you.

this was my first python project, so I went with the newest version. it's made a lot of things very difficult, especially porting to a web version.

I removed it, I thought it was obvious once you clicked on the link. again, I'm a PhD student, I live at the poverty line and host this project myself.

thanks TeMPOraL!

I certainly am.

I have an academic paper in revision that is an analysis of the alexa 1M list, I also have other projects in development.

as for video, I discuss research here: https://www.youtube.com/watch?v=OqW8erWi1Wo but if you mean screencast of the software I haven't had time.

also, the main extensions use black-lists, webXray grabs all the requests.

it seems the gods have forgiven me.

it is non-trivial for me as don't use windows, and I already spent a fair amount of blood, sweat, and tears developing the code.

I also wish it were easier, the major pain point is python3/mysql.

webdxray runs large batch jobs, so you can get lunch, and come back with all of pages analyzed. I know it does work on windows, and I apologize for not being able to provide directions...see comments above.

proxy is problem when you want to do a lot of concurrent tests, I usually load about 64 pages in tandem to get good speeds on large sets.

see comments below, I don't know windows so can't write directions. I'm poor so just wanted to cover my hosting, but I removed the referral regardless as I didn't realize I would be attacked for it.

I have done a few, but it's not my forte. vice included some I made in an article they did about my work a few months back: http://motherboard.vice.com/read/looking-up-symptoms-online-...

this is the biggest issues, depending on the cloud ip block you can get banned.

yeah, I already spoof the UA string, I've done a bunch of testing to verify it is working and getting me the right code. more likely way to get banned is I'm hammering ad networks from the same IP addr.

I don't use any extra python libraries, it's all to get python3 to talk to mysql. (also I'm not a skiddie...)

I haven't used windows in 15 years and couldn't find anybody to write the directions for me, so I went with snarky over spending time figuring out how to install python3 and mysql on windows. ;-)

in response to referral link comment: I host the site on my own dime so I didn't think getting $.25 from DO would make me a bad person...also, I really don't have time to learn windows to make instructions. I'm a grad student and make slightly above minimum wage annually, if I were to be evil and greedy I'd probably not be doing what I do with my life. I'm giving away thousands of hours of work and data I could make money off of doing evil things.

---

I think you guys/gals have a point, the referral link is gone. ;-)

constantly being updated, my hope with open-source is people will help add to it. if you want to help, I'd love that!

---

I see the hell ban, I'm posting all my replies at the bottom in a comment I'm editing - could somebody note this above?

hey, the captcha system here is an f'n nightmare - anyway I wrote the software! happy to answer questions will try to do so below

since I am hell banned I can only edit this comment to reply.

as for video, I discuss research here: https://www.youtube.com/watch?v=OqW8erWi1Wo

but if you mean screencast of the software I haven't had time.

---

@coleifer: there was a sqlite branch, but it doesn't scale well to many-million record sets which is what I have been doing. the design of the software allows drop-in db replacement, it just lacks the code. I can't decide to go back and sqlite or to just make a web front-end.

---

@captn3m0 I have an academic paper in revision that is an analysis of the alexa 1M list, I also have other projects i development.

---

@snorrah: this was my first python project, so I went with the newest version. it's made a lot of things very difficult, especially porting to a web version.

---

@linuxlizard: proxy is problem when you want to do a lot of concurrent tests, I usually load about 64 pages in tandem to get good speeds on large sets.

---

@radmuzon: webdxray runs large batch jobs, so you can get lunch, and come back with all of pages analyzed. I know it does work on windows, and I apologize for not being able to provide directions...see comments above.

---

@TeMPOraL: thanks!

---

@pearjuice: yeah, I wish there was an easier way to get python3 to talk to mysql, that's the biggest PITA.