textfiles

You should turn it on when browsing archive.org - you'll really need it.

Hi, jacquesm. Can we talk about mirroring reocities?

We hugged it out, all is good. Apologies for the derail; the "artiste" is hypersensitive about his children.

I'm hoping that's the case, and I'm sure I'm oversensitive on this topic. It's just that subject was particularly important to me, and any impression that I didn't care so much about capturing that story before the chance to get the story was over just makes me over react.

Nope, I'm going to do it, I'm going to focus right on one sentence and ignore all the compliments.

The BBS documentary was shot across 4 years on my own dime (no crowdfunding, my family helped pay for the camera) and traveling to 20 states, from 2001-2004, and then released in 2005. (10 years ago). This means some footage was shot as long as 14 years ago.

"Production Value" is a very specific term - it is almost always used to indicate skill or attention to the final product. All along the way for the BBS Documentary, I had to make choices. Remember, the Canon XL1 I shot with cost me $4000 in 2000, when it was bought. And instead of going with a handful of people, say, under 15, to tell the whole story, I chose instead to interview 201 people. This meant that sometimes in a single day I'd drive to up to 4 different homes, with miles between them, set up all the equipment myself, conduct the pre-interview and interview, and then move on to the next location. In one case I drove 500 miles to grab an hour interview with a figure who had popped up, and then drove back.

This was before DSLR became prominent, before HD was the norm. I could have gotten a full crew, but everything else would have suffered: the breadth of subjects, the locations, the variant voices. It's a choice I'm fine with.

I recognized, after BBS Documentary was done, that I'd want to move to HD, but in 2006, when I started filming GET LAMP, that cost me $10,000. I paid larger numbers for the new movies - $20,000 for the DSLR equipment. (Both GET LAMP and the new films are crowdsourced, based on the reputation of BBS Documentary.)

Choose your words carefully, please. I'll take "Dated", I'll take "SD", I'll take "Videotaped", but please don't imply I didn't sweat bullets over every aspect of the production, and that everything in there wasn't a best-of-all-circumstances choice made with the intent of finishing a project that many (at the time) thought was impossible.

May I suggest you click on the little "X" in the upper right of the window of your browser with this story, which is short for "I am Xtremely beyond this"

There has been no problem - we were just focused on getting the oldest, earliest browsers we can get running. It's not about getting a fully standards compliant browser made 20 minutes ago to run in a browser made 10 minutes ago. It's about providing easy, hopefully pain-free access to web history.

The goal is to add more browsers when this finally goes on the Internet Archive, yes.

There's a number of problems, which folks like jwz have dealt with, and which we'll look at. Basically, you don't get https (of course) and the response codes of web servers have shifted enough that older browsers don't know what to do. Proxy injection of needed helper material will help this a bit.

Bear in mind, though, it will ALWAYS be insecure and it will ALWAYS be more of a "try this out" than popping on your unicycle, starting up your bagpipes and riding down the Information Superhighway permanently.

Well, the bespoke vintage internet experience is kind of what we're up to at the Internet Archive, so you'll see us move into that more, I guess.

It's all the same assholes.

Before you go too crazy on the packing, we very, very specifically wrote a loader that would use .zip files (and you have to declare which file in the .zip is the first-executed one). That way you're not translating everything to that crazy intermediate.

Answer One: Archive.org does the right thing. Answer Two: Long ago agreed to this policy: http://web.archive.org/web/20130628205733/http://www2.sims.b...

The irony that the policy can only be viewed through the wayback machine is not lost on me.

Just for the record, we are an archive, just one with some policies (that I myself have been working to have tuned) that you might not like. I understand that you don't like the policy.

It can play on phones, but I don't overly recommend it.

I think the dev team would be surprised to hear that getting to see the new site is a security vulnerability!

The overarching plan, as I expect it, will be to have a number of ways to interact with the system. So there'll be this Pinterest-like visual browsing, more "lists" oriented, and maybe other versions, like research-oriented and so on. For now, there's visual and lists.

I perhaps wasn't clear. There's two different major projects that I've been interacting with - the front UI design, and the python client for interacting with the Internet Archive's APIs.

I've been definitely "eating the dogfood" as regards the Python client for over a year, with dozens of changes, fixes and improvements being done.

I've been "eating the dogfood" in terms of the V2 UI by using it, and making additions and changes to my uploads to work better in the new interface, as well as submitting dozens of requests to the dev team about things I've encountered.

So I'm pretty full of dog food here.

If you aren't getting the form, you can write info@archive.org with any thoughts or comments you have and they'll be forwarded to the dev team, who are gleefully swimming in mail as we speak.

Graceful degredation is definitely a plan.

Anyone can generate an S3-like/API key. They have the same rights and restrictions as someone using other methodologies. So, for example, you can upload into the general Audio or Texts collections, but you can't upload, say, right into the Grateful Dead archive or the CD-ROM collections we have.

In the future, we hope to have it that accounts will be assigned and de-assigned by some credential different than the current somewhat-binary approach we have now, but that functionality doesn't exist yet.

So basically, upload is constricted like before. Download, however, is as unconstricted like before.

The team will see this thread. I'll leave it to them to take in the note.

If you go to the site cold, you do not get the new site. You get the old site. If, however, you've chosen to look at the new site, a cookie is set so you are looking at the new site. If you click on "exit beta" in the corner, it will go back to the old site and the old site after that.

So it's still an option.

Jason Scott, here. Disclaimer: I work for the Internet Archive (although I don't speak for the entire Archive) and I'm vaguely gung-ho on the place: https://twitter.com/textfiles/status/527549181175427072

I am also, as part of my job there, one of the largest individual uploaders of data to archive.org - I've added hundreds of thousands of individual items (texts, movies, music, websites) since I started working there in 2011.

So, moving on.

Welcome to the new user interface beta. I'm glad to see people toying with it, and the commentary and complaints are very, very welcome. As a smallish organization with a lot going on, the responses from people really digging down in the beta are very appreciated.

First, I'll say that the Beta interface is a "true" beta - it's the result of a lot of internal work, arguments, and discussions, but nothing is 100% set in stone. This isn't a beta like Gmail or a FPS trying to determine the rate of firing of the chaingun weapon: this is a lot of best-approach attempts at a whole range of goals. There's bound to be lots of responses from a lot of camps that are now coming forward. (For example, the new site has accessibility issues that need to be addressed.) If the term "beta" has been wrecked, stick with "prototype".

The internal name was V2, so I tend to keep calling it that.

V2 is the first major redesign of the main archive.org site in over a decade. And part of the conditions of this project (done by a handful of people) were to keep the old site (retroactively called V1) running, and mostly unchanged. That was a whole bucket of headache that isn't even obvious when you come into the site. (Anyone who has done this knows how it can be). With over 20 petabytes of data on the site, and millions of items and objects, spanning the whole environment without downtime is a feat in itself. So there's a whole range of philosophies being approached, but just getting the backend into a shape where it could sustain a new interface to it was a lot of non-obvious work.

Moving to the site as it is now.

Definitely slow. Definitely a shock. Definitely some great choices, some which might seem like head-scratchers. There is a designer, with a vision (his name is David) and there's been approaches to all the intended known shortcomings of V1 Internet Archive in this prototype.

One of the issues with Archive.org that's been an issue is non-responsiveness for different platforms - you got one site and that was it. Another was a lack of visual interface as an option. Now there is one.

The tagging and metadata efforts were spotty before now, because you were not really rewarded for doing so. The V2 site uses these tags and metadata extensively, and will continue to. This has been a nightmare for me, frankly - I've had to add logos to the 1,200 collections of items I've been uploading, and I'm doing descriptions as well as tags. But under the new system, the chance for finding things has increased exponentially.

There are definitely cases where I have to swap back to V1 to get kinds of "work done", because as an intense power-user, I do all sorts of grandiose work. But then again, 99% of my interaction with maintaining and adding content to the Internet Archive, I do through the API, and specifically through a python command-line interface we've had a developer working on for over a year:

https://pypi.python.org/pypi/internetarchive

I've uploaded many thousands of items, analyzed and upgraded their metadata, and done search-and-modify runs by the hundreds with this tool. It's being constantly updated.

In the future, I expect us to see multiple improvements to the interface - one which is much more bandwidth and processor friendly, a version of the "view" (we have image and list right now) that is optimum for researchers, and so on. But I'll stress again:

- This is a prototype which was done with a pretty small team who had to keep the old site running as smoothly as possible, while doing essentially a decade of upgrade in one swoop; - Now that it's "proven" that it works, refinement by the truckload needs to happen - Your comments are not just welcome but encouraged - Increased interest in the archive and the materials, and working together to find ways to access the petabytes of data in a meaningful way is not just a nice side benefit, but a vital core of the Archive's mission

Thanks for reading.

Sorry, was tired. I fixed that and a couple other typos. Thanks for reading, everyone.

Sorry for not being clear. Out of the 1.1 petabyte, something like 50-60% of that has zero views. Archive Team duplicated everything with 10-infinity views, which was about 10tb.

Since people are interested... a fact that may be obscured from the conversation and statistics is that somrthing like 550gb of those videos have ZERO views.

Kudos to archive team!

Naturally, you should wait and check on the credit union periodically to see how their project is going, and when they come out of Beta.

Appreciated.

Very appreciated.