Ask HN: Does this 'local web' tool exist yet? | Hacker News Reader

Ask HN: Does this 'local web' tool exist yet?

6 pointsnewman8r15 comments

If something like this exists, I want to use use it. If not, maybe I want to open source something. Pasted from my idea log as follows:

premise: a local internet that has a lot of curated resources -If there was an apocalyptic event, this is the database you would want

-it is possible today to have an entirely local version of the web that would amaze anyone from early 2000. -premise is there is no worry about ip/copyright - it's a hypothetical -Can be updated, added to, shared, can run with minimal components, can run on its own os -can store massive amounts of data on a tape drive if needed, index it well and copy blocks to a large hard disk or solid state drive. Lots of ram (64gb+) for fast searching of active blocks

• Encyclopedic data • edu sites • original articles/newspaper articles • backups of abstracts from as many studies as possible • stack overflow, other similar sites • text of reddit, submissions with scores above 10 • copies of classic books and technical books/nonfiction • Top 200 movies, best tv shows full series - various levels of compression as needed • most useful youtube videos, ted talks, best documentaries • Links to the most useful software, highly compressed files,etc • Compilers and IDEs for all common programming languages and learning resources • datasheets for components by number, etc. • Top music by year, genre, album, etc. roms and emulators for many systems

These stuff exists, but it usefulness depends on the administrators.

I was really surprised when a couple of years ago visiting Tigre, a city north of Buenos Aires, argentina:

There is a huge river delta there, the Tigre delta and the people living in the small islands formed by the many branches of the river have a sort of local internet with their own news sites, wiki, etc... as well as access to the worldwide internet.

Another interesting fact is that some of this island houses have electricity but no drinkable water, they have to carry it from the city docks

This is fascinating. I've heard of these local internets in remote areas. I'd love something along those lines for my own personal enjoyment and to give access to my friends.

I might go to def con this weekend and hit up the data duplication village

There used to be a program called Teleport Pro, which can download an entire web site.

I used it rather extensively to download interesting sites for local storage, so i got a large selection of pages, a lot of which do not exist any more.

Then i moved from dial-up to something faster around 2007, and the habit to download the sites gradually died out. Most of them are pre-2009, and the latest one is from 2014.

I suppose making a new/free program like that might be of use today - the old one can't parse modern sites too well, i.e. Wikipedia.

Yeah I remember these types of offline browsers that were popular back in the day - data usage is still a big concern for some people (pre web 2.0 websites were easier to download. A shotgun approach using something like ScraPy or via wget/curl as mentioned isn't all that useful)

Since modern websites are very heavy on javascript, high res images, media, ads, comment sections, nav menus, etc - I think you just crawl portions of websites, cache the relevant data and keep a link to the original source. Instead of trying to crawl every obscure blog, I would focus on crawling the big/information dense sites really well. Many of them, like wikimedia sites, have complete data dumps available.

If you systematically ignore all the garbage on the web and distill a version that is mostly valuable knowledge, there's some interesting possibilities. I have read a few times that most people browse the same,predictable subsets of websites anyway. Most of the important sites we log into provide APIs to access them. In general I'd prefer not to connect via web browser or cell phone app constantly.

Try the recursive options of wget. It should be pretty much what you are talking about.

Huh, i was going to say that wget won't fix the links in the downloaded files, but turns out they thought of it too.

Nice, that means the tool does exist already.

Check my response above, feedback appreciated. downloading entire websites via wget isn't what I was getting at although that type of tech is certainly something that would be integrated into the hypothetical system.

What I am thinking of would really go beyond what you could achieve with wget. No doubt offline browsing and personal crawlers have been around for a while. Think of a 10TB+ tivo-like device that comes pre loaded with everything from wikipedia to musicBrainz. People would't need to figure out what to download, it would already be there, and they could augment it with anything.

Just a thought - I am going to keep looking into it and will create a public repo for it if it makes sense.

Something relevant https://www.indiegogo.com/projects/lantern-one-device-free-d...

Which sources content from https://outernet.is/

cool concept, nice find. I want something like this that I can host, no satellites needed, I can customize, let friends connect remotely, be able to do really cool searches using regex, have some big databases, etc.

I guess it's along the lines of scraping/crawling/local browsing, etc. Basically creating my own internet that I can invite people to, annotate, collaborate in, etc.

I'm working on an open communications platform at the moment and am contemplating creating a microservice to enable something like this - but I don't want to reinvent the wheel if someone has already done it well.

I think you should definitely do it unless it takes years to ship the first version. The first in market doesn't mean the first to succeed.

That's true. I'm not concerned about being first to market or making any profit on this type of project, would just be open source and release under GPL or whatever is appropriate.

I just didn't want to spend the time or effort doing it if a similar alternative already existed. My primary motivation is essentially just wanting the tool for myself and friends.

Not sure how long it would take but I might be able to release a useable proof of concept after a 3-day-weekend hackathon.

Wikipedia seems to fulfill a fair amount of your requirements. Add in Project Gutenberg...

Yeah wikipedia would be part of something like this. I already have all their data dumps and I have a lot of other cool DBs like musicBrainz, lots of stack exchange stuff, some "top" movies lists, lots of books, etc. I like to hoard data and hd space is cheap.

I don't know of any such curated collection, but a good tool to build what you are describing would be IPFS: https://ipfs.io

I had no idea this even existed. This would be extremely helpful.

Reply on news.ycombinator.com