10165

"... into actual structured data."

Would there be some particular structure that everyone would agree on?

Alternatively, what is the desired structure you want?

Because the current format is so messy, I just focus on what I believe is most important: titles and externallinks. IMO, often the most interesting content in an article is lifted from content found via the external links. I also would like to capture the talk pages. Maybe just the contributing usernames and IP addresses.

Opinions or explanations that have no supporting reference are inexpensive. One can always these for free on the web. No problem recruiting "contributors" for that sort of "content".

Back to the question: I am curious what structure would you envision would be best for Wikipedia data? Assume hypothetically that a "perfect" parser has been written for you to do the transformation.

"That gives you Wikitext encapsulated in XML."

avar: "The goal of Wikipedia should be to spread the content as far & wide as possible, the way OpenStreetMap operates is a better model."

I am confused.

Doesn't OSM data come encapsulated in XML or some binary format?

As for dispersion of content, I could have sworn I have seen Wikipedia content on non-Wikipedia websites. Is there some restriction that prohibits this?

I have seen Wikipedia data offered in DNS TXT records as well.

No one outside of WMF is downloading and storing the dumps they provide to the public?

In the past, I have used these for parsing and database setup practice, making CSV and local databases that require no network access. I am mainly interested in the reference urls found in wikipedia pages.

There is no need for keep-alive for a single HEAD request. Why use HTTP/1.1?

   cat << eof |nc -vv nautil.us 80
   HEAD / HTTP/1.0^M
   Host: nautil.us^M
   User-Agent: curl/7.47.0^M
   Accept: */*^M
   Connection: close^M
   ^M
   eof

Anyway, it looks like they fixed the problem or I was mistaken.

I will need to find another example.

Meanwhile looking on stackexchange one can still see people running websites asking whether to block or "turn off" HEAD as recently as last year.

If a user expects every website to respond properly to a HEAD request, then the user might be occasionally "surprised". This is because not every person running a website understands or agrees how HEAD can be useful. Sadly, GET is the only method that a user can expect to work on across all websites.

1) nautil.us

2) Yes.

I was not implying one should do otherwise. I was just pointing out that servers that respond to GET will not always respond to HEAD as expected. Some sites treat it the same as GET. Others may not allow it. For example, Amazon responds with 405 MethodNotAllowed.

"And it isn't like they were doing HTTP 'HEAD' requests -- no, they were doing 'GET' requests."

Some httpds treat them the same -- they still send the file after HEAD requests instead of only the headers.

Is it true that carriers are increasingly switching to calling over WiFi? I understand the new iPhones have this feature, with switchover happening automatically.

Did the popularity of Facetime, WhatsApp, etc. have anything to do with this? I have seen claims from carriers that WiFi calling "improves coverage".

At some stage, will anyone question what is the point of the cellular network? Especially in urban areas.

What is the difference between

a. a "smartphone" and

b. a portable, pocket-sized computer with rechargeable battery where the user chooses what to install and can remove any pre-installed software, where user gets a choice of 1. using pre-installed software and default settings or 2. using her own bootloader, kernel and userland, and where the user can easily open the case and tinker.

Does a computer need to have any association with the company selling internet access? Today's "phones" are manufactured for carriers (who are often ISP's, too), not for users. The carriers in turn sell these customized "phones" to users.

What is the purpose of an "application firewall" in an open source world?

If I have the source code and I am curious I just read it. I have yet to see any source code that attempted to obfuscate opening sockets.

Sometimes I run programs with ktrace (strace for Linux folks I guess) and look at the calls.

But truthfully in most cases controlling DNS catches most if not all of today's applications' attempts contact the mothership.

I do not use a third party cache, I maintain a custom root and can use it to block wildcarded domains, something that cannot be done with /etc/hosts. I do make extensive use of the HOSTS file but not for blocking.

I can also redirect traffic to localhost servers where I can log requests and analyze the captured packets. For instance if I want to reverse engineer the protocols used.

In my opinion an "application firewall" is a misnomer. IMHO a "firewall" operates on incoming packets from other computers, not packets originating from the computer running the firewall.

If a Windows user wants to run a "firewall" then IMO they need an additional computer, e.g., a gateway they control. This is the way to stop the telemetry, IMHO.

I think Microsoft puts some application they call a "firewall" on Windows. IMHO that is misleading, but not surprising considering the source.

In an open source world, in the event a particular user does not "have the time" to read source code or even grep it for clues, chances are that some other user does have the time and compiles all his programs from source. It is always possible that an application's "phoning home" behaviour could be documented by such users in public forums, etc.

What if an application hard codes an IP address? Answer: I see it in the source code. NB: I very rarely see this in practice. Apparently few people can maintain a stable IP.

The article begins with looking back many years. In those days there were technical limitations and clever workarounds. One MB was enormous.

Today, what are the technical limitations?

PCWorld's summary of by the comments in the talk lacks much insight. The article itself is misleading because it seems to suggest the goal is a "consumer PC". As I understand it the goal is a build machine, a computer whose purpose is to run a compiler. This is not a computer primarily for consumption. Graphics are optional.

Here is how I understood the comments:

1. Need expandable memory. Not everyone will require the same amount of memory. For example, the kernels I compile only need about 200MB of RAM, max. But this might not be true for other users. Back in the PC days, users could add their own RAM. This concept has been lost on mobile phone manufacturers. Not everyone needs the same amount of RAM.

2. Need better secondary storage. I would guess the reason is not to supplement RAM i.e., swap space, but because source trees have become so large. This is a pet peeve of mine. I keep writing half-baked tools to reduce the size of source trees to only what I will need. If the trees were smaller, perhaps more customized, then I could fit them in RAM (mfs or tmpfs). On my RPi I never use SD cards as writeable storage. They are just where I store a read-only copy of the kernel and userland which I can load into memory. I pull out the card after booting. I/O via secondary storage is just too slow.

Looking back at times past there was some incredible creativity to find ways to make due with limited memory. Today we have enormous amounts of RAM (and computing power) but I see little effort or creativity in getting systems to fit into these "constraints" (sheesh).

The systems of the past fit easily in today's "minimum" amounts of RAM. The systems of today are still used by a majority of users for many of the same boring tasks as in times past but cannot fit into GB of RAM?

Software is like a gas. It expands to fill space.

Forcing one to keep purchasing new hardware. I guess that is the point.