k1m

I haven't seen guidelines lately, but Readability.com, before it was shut down, did have a page that's archived here: http://web.archive.org/web/20160301180825/https://readabilit...

Prune instructs the parser to remove any elements within the extracted article block that look superfluous. This can result in false positives, so we tend to disable it when we've gone to the trouble of creating site-specific extraction rules.

Tidy determines if the source HTML should be cleaned up first with HTML Tidy - https://github.com/htacg/tidy-html5. If you're parsing the source HTML with an HTML 5 parser, as we are now, it shouldn't be necessary any more (I think we actually ignore it now). We used it more before when we relied on libxml parsing, which often trips up on modern HTML.

Thank you! That's really nice to hear. (Appreciate the support too.)

In our case, we try to match using the XPath selectors that we have for the site. If we don't have any, or they fail to match anything for the title, author, or body, we then go to Readability and let it do its thing to try and extract whatever we're missing.

Thanks for mentioning Instant View, I hadn't come across that. We actually maintain something similar here: https://github.com/fivefilters/ftr-site-config

We use these in our own tools and also get contributions from others, including Wallabag users: https://github.com/wallabag/wallabag

Before it was sold, Instapaper used to have something similar. A public database of its site-specific extraction templates. We used that as the starting point for our repository.

Any developers who'd like to contribute to improving how article content is extracted from web pages should check out Mozilla's Readability repository: https://github.com/mozilla/readability

I'm currently trying to bring the PHP port up to speed here: https://github.com/fivefilters/readability.php

We use an older version as part of our article extraction for Push to Kindle: https://www.fivefilters.org/push-to-kindle/

For anyone interested in following, Kevin Gosztola is live tweeting today's preliminary appeal hearing: https://twitter.com/kgosztola/status/1425382572230590466

If it happens by itself (no interaction from you) it's likely a fault. I've had that happen on a Kindle before and had to send it back.

But the most common cause for me is when trying to tap the right part of the screen to move to the next page, I end up tapping a small link instead that jumps deeper into the book. Very annoying when it happens.

> The whole "jigsaw identification" thing is gish galloping. His supporters have made up a nonsense talking point to defend him.

I'm not going to continue this as it's not going anywhere. But as you're so adamant that he wasn't charged with jigsaw identification, and that it's only something his supporters claim (even though it's literally in the document you posted), you might want to take it upon yourself to reach out to the following publications and ask them to correct the record:

* The Guardian: "Ex-diplomat found in contempt of court over ‘jigsaw identification’" [1]

* BBC: "the retired diplomat breached the order and published pieces of information on his blog which, if pieced together like a jigsaw, could lead readers to establish the identities of the women" [2]

* The Independent: "Former diplomat hands himself in at police station to begin sentence over ‘jigsaw ID’ of Salmond accusers" [3]

* Daily Mail: "Lady Dorrian said Murray deliberately risked what is known as 'jigsaw identification'." [4]

* Scottish Legal News: "He published information on his blog which enabled jigsaw identification of the women in the case." [5]

[1] https://www.theguardian.com/politics/2021/aug/01/alex-salmon...

[2] https://www.bbc.com/news/uk-scotland-58018127

[3] https://www.independent.co.uk/news/uk/home-news/craig-murray...

[4] https://www.dailymail.co.uk/news/article-9849607/Britains-ex...

[5] https://www.scottishlegal.com/article/craig-murray-to-be-jai...

> They’re absolutely not claiming Murray did something exceptional - that has nothing to do with whether he broke the law.

Fine, but don't expect people to take you or your law seriously if out of a bunch of people who break the law in a similar way, you choose to punish and make an example of only one of them. The one who's had the least impact in identifying the people you claim to want to protect, and the one who's been a thorn in your side for many years.

> No jigsaw identification necessary at all

This makes no sense. First you claim they're "absolutely not claiming Murray did something exceptional", and in the next line you write he did something so unique that no jigsaw identification was necessary at all. So then why is he being charged with jigsaw identification? The first person ever to be jailed for it.

> It’s illegal to publish information identifying the accusers it’s not illegal to figure out who the accusers were.

This whole case has supposedly been about protecting the identities of people involved in the Salmond case by preventing "jigsaw identification".

The court document you posted has this:

"Jigsaw identification can occur through the combination of separate pieces of information or details about inter alia the victims, nature and timing of the offences and locations. ...Seemingly innocuous facts published can still pose the risk of jigsaw identification when viewed within the existing matrix of information..."

> Meanwhile the attempt to go out and try and find people who figured out the identity through other means is entirely immaterial to the case. It’s just not relevant because that’s not what the law is about.

Why is it immaterial? They're claiming Murray did something exceptional. But when you ask (via a standard public poll, not "finding people who figured out the identity through other means" as you claim), those who were able to identify the people involved nearly all point to other media, not Murray, as their source.

You can claim that's immaterial and the court has no reason to look at how other reporters revealed identities via jigsaw identification, but I don't think that's very convincing to the general public. And when you take into account Murray's history (as Jonathan Cook does in his article), it looks to many people like he's being singled out and punished for being a thorn in their side for too long.

Not to mention that no one has been jailed for jigsaw identification ever before.

Craig Murray addresses this in his latest post:

> I want to touch on one other aspect of liberty in my own imprisonment that appears not understood, or perhaps simply neglected, because somehow the very notion of liberty is slipping from our political culture. One point that features plainly in the troll talking points to be used against me, recurring continually on social media, is that I was ordered to take down material from my blog and refused.

> There is an extremely important point here. I have always instantly complied with any order of a court to remove material. What I have not done is comply with instructions from the Crown or Procurator Fiscal to remove material. Because it is over 330 years since the Crown had the right of censorship in Scotland without the intervention of a judge.

...

> What is even more terrifying in my case is that the Court explicitly states that I should have followed the directions of the Crown Office in what I did and did not publish, and my failure to not publish as the Crown ordered is an aggravating factor in my sentencing.

> If the Crown thinks something I write is in contempt and I think it is not, the Crown and I should stand as equals in court and argue our cases. There should be no presumption I ought to have obeyed the Crown in the first place. That Scottish “justice” has lost sight of this is disastrous, though perhaps as much from stupidity as malice.

https://www.craigmurray.org.uk/archives/2021/08/keeping-free...

He's being jailed for 8 months for his writing. Hardly escaping legal liability. I don't find that funny, no.

As for Jonathan Cook's piece, the double standard he writes about is the fact that those who claim they learned of the identities of the individuals claim they did so not from Craig Murray's writing but from other reporters. That's been completely ignored by this ruling. Do you think we'll see a court case targeting BBC reporters or the writer Dani Garavelli, cited by most respondants as the main source? Isn't it a double standard if they escape jail but Murray doesn't? Certainly looks to me like someone's got an axe to grind when they fixate on the ways Murray may have let readers learn about identities but gives other reporters a pass - especially when public surveys reveal those other reporters to be even more culpable than Murray.

What you don't get by reading this ruling is the full context given by Jonathan Cook and the double standards he highlights.

> The whole thing is just pretty funny

Glad you can laught it off. Meanwhile, Reporters Without Borders have expressed grave concerns: "Craig Murray's prison sentence on charges related to his blogging is disproportionate and highly concerning. RSF emphasises that journalistic activity should not lead to prison sentences anywhere; imprisonment in connection with any journalistic activity should only ever be a measure of absolute last resort - if at all. It is certainly not what we would expect in a country committed to protecting media freedom and the safety of journalists." - https://rsf.org/en/news/uk-blogger-craig-murray-jailed-eight...

Craig Murray is being jailed because he allegedly helped reveal the identities of individuals through his reporting. There's nothing indicating his reporting revealed any more information about those identities than reporters from the BBC and other outlets. In fact, surveys show those other reporters did far more to help people identify those individuals.

But they're not being jailed. Craig Murray has been singled out. Nothing ethical about that.

> The court order Murray broke applies to everybody.

It clearly doesn't as two surveys showed those who claim they were able to identify individuals from the reporting of the case, said they were able to do so from reports from the BBC and other corporate media. None of those reporters are facing jail.

Again, what Jonathan Cook is pointing out is exactly the opposite of what you claim. The court order does not apply to everybody: "Murray's writings appear to have had very little impact on the identification of any of the accusers. Among named individual journalists, Dani Garavelli, who wrote about the trial for Scotland on Sunday and the London Review of Books, was cited 15 times more often by respondents than Murray as helping them to identify Salmond's accusers."

Jigsaw identification isn't same as doxxing. The issue here is the double standards, as Jonathan Cook points out:

> In finding Murray guilty of so-called "jigsaw identification", Lady Dorrian did not make a distinction between what Murray wrote about the Salmond case and what approved, corporate journalists wrote.

> That is for good reason. Two surveys have shown that most of those following the Salmond trial who believe they identified one or more of his accusers did so from the coverage of the corporate media, especially the BBC. Murray's writings appear to have had very little impact on the identification of any of the accusers. Among named individual journalists, Dani Garavelli, who wrote about the trial for Scotland on Sunday and the London Review of Books, was cited 15 times more often by respondents than Murray as helping them to identify Salmond's accusers.

> Rather, Lady Dorrian’s distinction was between who gets protected when identification occurs. Write for the Times or the Guardian, or broadcast on the BBC, where the audience reach is enormous, and the courts will protect you from prosecution. Write about the same issues for a blog, and you risk being hounded into prison.

Yes, I should have added that I see nothing wrong with the replacement of key with keystone here. Not sure what the big issue is. I was just surprised that someone who does object to it assumes a HN posting has been deliberately crafted to mislead when it could simply have been an innocent error.

> why would you choose to change a shorter, more common word for a longer, less common one if they mean exactly the same?

I'm not the OP, but you're assuming this was copied and pasted and the word changed deliberately. I can think of a number of reasons why the different word was used, that has nothing to do with an attempt to mislead: e.g. autocorrect if typing it on mobile, genuine error (thinking it was actually 'keystone' in the original).

> Unless I'm missing something, this is a minor story ... in order to strengthen an already robust case, not the slam dunk people are making it out to be.

I don't think this was a robust case at all. A couple of things to note here:

1. The witness from Iceland whose testimony against Assange has now revealed to have been lies wasn't part of the original case the US put forward. They added him much later into the proceedings. So if this was a "robust case" as you put it, why would they decide to add new accusations?

2. You might say, well, they didn't need to include him and shouldn't have. But you have the judge (Baraitser) citing these new accusations when siding with the US, as Stundin reported: "Baraitser sided with the arguments of the American legal team, including citing the specific samples from Iceland which are now seriously called into question." So clearly they were significant enough for the judge.

3. Craig Murray has written about why he thinks the witness was included late into the proceedings:

https://www.craigmurray.org.uk/archives/2021/06/fbi-fabricat...

> The hearings on the Assange extradition in January 2020 did not seem to be going well for the US government. The arguments that political extradition is specifically banned by the UK/US extradition treaty, and that the publisher was not responsible for Chelsea Manning's whistleblowing on war crimes, appeared to be strong. The US Justice Department had decided that it therefore needed a new tack and to discover some "crimes" by Assange that seemed less noble than the Manning revelations.

> To achieve this, the FBI turned to an informant in Iceland, Sigi Thordarson, who was willing to testify that Assange had been involved with him in, inter alia, hacking private banking information and tracking Icelandic police vehicles. This was of course much easier to portray as crime, as opposed to journalism, so the second superseding indictment was produced based on Thordarson's story, which was elaborated with Thordarson by an FBI team.

> The difficulty was that Thordarson was hardly a reliable witness. He had already been convicted in Iceland for stealing approximately $50,000 from Wikileaks and with impersonating Julian Assange online, not to mention the inconvenient fact he is a registered sex offender for online activities with under-age boys. The FBI team was in fact expelled from Iceland by the Icelandic government, who viewed what the FBI was doing with Thordarson as wholly illegitimate.

> [...] While Baraitser's eventual decision barred extradition on the grounds of Assange's health and US inhumane prison conditions, the second superseding indictment and Thordarson's accusations were accepted as a valid basis for extradition.

Given all that, I think it's fair to describe him as a key witness.

> There's no doubt that the US is acting vindictively here, but the pro-Assange camp does little for its credibility by using the same tactics.

Revelations about an important witness lying about Assange to get immunity from the US is blanked by corporate media. How is reporting that this important story was blanked "acting vindictively"?

Sorry, but you've got that the wrong way round. Assange was the one who wanted careful redaction and it was his media partners at the Guardian and Der Spiegel who were impatient and wanted to publish early. In the end, it was the recklessness (perhaps malice) of the Guardian journalists which resulted in the unredacted cables being released. They published the password to decrypt the unredacted archives in their book (unbelievable, but true). Jonathan Cook (ex-Guardian) has written about this:

https://www.jonathan-cook.net/blog/2020-09-26/guardian-assan...

> The Guardian book let the cat out of the bag. Once it gave away Assange's password, the Old Bailey hearings have heard, there was no going back.

> Any security service in the world could now unlock the file containing the cables. And as they homed in on where the file was hidden at the end of the summer, Assange was forced into a desperate damage limitation operation. In September 2011 he published the unredacted cables so that anyone named in them would have advance warning and could go into hiding – before any hostile security services came looking for them.

> Yes, Assange published the cables unredacted but he did so – was forced to do so – by the unforgivable actions of Leigh and the Guardian.

Here's another Jonathan Cook article citing people intimately involved with the redaction process which contradicts your version of events:

https://www.jonathan-cook.net/blog/2020-09-22/guardian-silen...

> ...Goetz [Der Spiegel], as well as Nicky Hager, an investigative journalist from New Zealand, and Professor John Sloboda, of Iraq Body Count, all of whom worked with Wikileaks to redact names at different times, have testified that Assange was meticulous about the redaction process. Goetz admitted that he had been personally exasperated by the delays imposed by Assange to carry out redactions:

>> At that time, I remember being very, very irritated by the constant, unending reminders by Assange that we needed to be secure, that we needed to encrypt things, that we needed to use encrypted chats. … The amount of precautions around the safety of the material were enormous. I thought it was paranoid and crazy but it later became standard journalistic practice.

> Prof Sloboda noted that, as Goetz had implied in his testimony, the pressure to cut corners on redaction came not from Assange but from Wikileaks' "media partners", who were desperate to get on with publication. One of the most prominent of those partners, of course, was the Guardian. According to the account of proceedings at the Old Bailey by former UK ambassador Craig Murray:

>> Goetz [of Der Spiegel] recalled an email from David Leigh of The Guardian stating that publication of some stories was delayed because of the amount of time WikiLeaks were devoting to the redaction process to get rid of the "bad stuff."

> When confronted by US counsel with Leigh's [Guardian] claim in the book about the restaurant conversation, Hager observed witheringly: "I would not regard that [Leigh and Harding's book] as a reliable source." Under oath, he ascribed Leigh's account of the events of that time to "animosity".

But how many people have heard this version of events from the NYT, The Washington Post, the Guardian or all the other corporate media outlets that have been all to happy to pour scorn on Assange?

Your linked Washington Post article was published on 8th July. It's actually a good demonstration of the animosity towards Assange in the corporate press. It wasn't published in response to the major Sundin story (26th June) which as the OP's article (published 7 July, before your Post piece) shows, was blanked by all corporate press at the time:

> ...as of Friday, July 2, there has been literally zero coverage of it in corporate media; not one word in the New York Times, Washington Post, CNN, NBC News, Fox News or NPR. A search online for either “Assange” or “Thordarson” will elicit zero relevant articles from establishment sources, either US or elsewhere in the Anglosphere, even in tech-focused platforms like the Verge, Wired or Gizmodo,” FAIR says.

The Post mainly sides with the US and only mentions the Sundin reporting in passing, diminishing its importance and omitting important parts, as the other replies here point out. A great example of power-friendly reporting when it comes to Assange.

That's been my experience too, as well as encountering a lot of legacy HTML markup without many useful class/id attributes. The random values are likely result of modern build tools like the one you mention, but there's also the growing popularity of utility-first CSS frameworks like Tailwind which also make finding useful selectors difficult.

In these situations it might be best to rely on position of elements and other CSS selectors rather than attribute values. Unfortunately the :nth-child and :nth-of-type selectors still trip many people up. In Feed Creator we borrowed from XPath to make selecting by position a little easier. We've got a comparison here: https://help.fivefilters.org/feed-creator/css-selectors.html...

The more the merrier! :) And always nice to see more in the RSS space.

Feed Creator is in PHP and unlike yours doesn't produce a static file. The CSS selectors, URL and other parameters are all embedded in the URL, e.g.

  https://createfeed.fivefilters.org/extract.php?url=http%3A%2F%2Fjohnpilger.com%2Farticles&item=.entry&item_desc=.intro&item_date=.entry-date

The biggest issue I've found is people struggling to work out which CSS selectors to use. I wrote a blog post not that long ago to help people use the browser's developer tools to do that. Might help too with anyone trying to use this (although perhaps most of the HN audience doesn't need help here): https://www.fivefilters.org/2021/how-to-turn-a-webpage-into-...

Very nice! I work on Feed Creator - https://createfeed.fivefilters.org - which is similar, although unlike yours doesn't use a headless browser, so selecting Javacript-inserted elements isn't supported.

Yes, FeedBurner does this[1], and we do it too with Full-Text RSS[2]. The feeds should render as a regular webpage in your browser, but if you view source on those feed URLs, you'll see regular RSS output (XML), but they both contain XSL to transform the XML to HTML in the browser:

FeedBurner:

    <?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/rss2full.xsl"?>

Full-Text RSS:

   <?xml-stylesheet type="text/xsl" href="css/feed-free.xsl"?>

For anyone attempting this, note that there's an escaping issue on some browsers that requires Javascript to fix[3].

[1] http://feeds.feedburner.com/johnpilger?format=xml

[2] http://ftr.fivefilters.net/makefulltextfeed.php?url=http%3A%...

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=98168#c80

If you're trying to build one yourself, have a look at the open source Readability code[1]. It was originally developed by Arc90 and is now used by Apple and Mozilla in their browser reader views. The code has been ported to a number of different languages.

I work on a service called Full-Text RSS[2] that used a PHP port of Readability, coupled with site-specific extraction rules[3] to identify and extract article content from each feed item. It then produces a full-text version of the given feed. The idea is you subscribe to the full-text version in whichever feed reader you use and it will give you full-text articles where you had partial content before.

[1] https://github.com/mozilla/readability

[2] https://www.fivefilters.org/full-text-rss/

[3] https://github.com/fivefilters/ftr-site-config

> That charge could be completely removed from the case and he would still be on trial for the other charges.

That charge was completely absent from the case until the US decided it needed to strengthen its case and added it last-minute. So clearly they disagree with you, otherwise they would have kept it out (knowing the background of the individual involved). And again, why would the judge refer to it specifically in siding with the US if it were so unimportant?

You're implying this is a minor development. It's really not.

Edward Snowden said: "This is the end of the case against Julian Assange."

https://twitter.com/Snowden/status/1408847450656415751

I find it hard to believe the media would be silent if similar revelations came to light about anyone else.

I agree with Media Lens:

> in a sane world, Stundin’s revelations about a key Assange witness – that Thordarson lied in exchange for immunity from prosecution – would have been headline news everywhere, with extensive media coverage on BBC News at Six and Ten, ITV News, Channel 4 News, front-page stories in the Times, Telegraph, the Guardian and more. The silence is quite extraordinary; and disturbing.