Show HN: A single, searchable API for all social networks, news, and blogs | Hacker News Reader

Show HN: A single, searchable API for all social networks, news, and blogs

streamified.me

55 pointshollandaise45 comments

You lost me here: "We think that formats like RSS don't make sense for today's internet".

And honestly.. another API? So it only works as long as you are in business (which is based on what by the way?) or decide to kill or change it? And also posting to my social media will be going through your machines? Seriously?

If you'd offer me an opensource script that I could call like "localhost/post?message="hello"&where="twitter,blog,facebook".. that would be different story. But hey, good luck locking others in your new service.

RSS is full of embedded HTML. It's practically impossible to make a good news reader without doing some heavy manipulation on it and trying to extract/modify ugly elements. We have a full blog post on the topic here: http://blog.streamified.com/post/45846467833/the-future-of-s...

There's a discussion below this one why it makes sense to have a single API for this, instead of open-sourcing the code. The benefits are numerous (we're hooking into many paid APIs and covering the costs, for example).

RSS format sucks because XML sucks.

I agree that it would be awesome if it was opensource so you could mitigate any risks by having it run on your own hardware, but really it's a valuable service and they deserve to make some money, which will play into keeping the service alive.

Thanks!

The stack which is required for the Streamified API is really quite large, too. Between node, mongo, elasticsearch, and a few other pieces of software... plus a few dozen node modules and even some custom linux configuration, it's really not feasible to expect devs to install it on their own machines ;)

You could open source it anyway, though, or parts of it. NewsBlur does this: https://github.com/samuelclay/NewsBlur

One could run their own NewsBlur, but Samuel has stated it requires a lot of resources to pull RSS feeds, and so in practice it's much easier to pay them $2/month. (This may not apply if you plan on having business customers who are paying orders of magnitude more than that.) But it alleviates any fear that the service users depend on may disappear; in the worst case, run it yourself.

Fair points, thanks. I'd have to discuss with the team if this is an option, ultimately. I can promise that if we ever did go away, I would certainly aim to opensource it (why not?) But as a company w/ investors, it's not a decision I can make myself right now ;)

Your welcome! I get that the service your providing is more than the sum of its code. Someone is doing the website, marketing, documentation, fixing bugs, keeping up with changes etc... not to mention paying the bills to reliably run the servers.

If you want my opinion, I would recommend finding a small area of your tech stack that you wish was better and starting an opensource project to solve that specific problem. No one expects twitter to give away their source code for their core product, but they do appreciate bootstrap and bower.js a whole lot.

That's a really good point -- I'm going to look for a way to do that.

I would agree if we were talking about a SOAP protocol that could be restful or something. But RSS doesn't bother me at all. You can just pass it through a converter to suit your fancy, and the bloat for an average use case of 10 results can't be that much. It's a minor miracle anybody decided to publish their sites through RSS so I'm happy they used the format that was hot at the time and got adoption.

I love RSS, I couldn't have built a ton of cool stuff over the years without it. For example I built @twuoted using twitter's search feed 4 years ago with Sweetcron, but that has long since been replaced by twitters API. I don't hate on RSS, I think it was a wonderful invention for it's time and was great for showing how standards can work.

But I still want it to die and JSON to become the new king in data transfer. That could be due to my work being in the web development arena and using node.js, but I honestly think it's just a better solution.

I could have used this about two months ago. Instead I built a simple npm module – https://github.com/switz/melt

This is far better, though.

Nice! That's where we started, then got carried away... very, very carried away ;) Hope you find it useful; we'd love to hear from you!

I'd definitely consider implementing it into https://tweezer.io – send me an email!

This is a nice announcement to see, especially your support of searchable article feeds. Quick question: I see in https://streamified.me/developer/#concepts that we can search news articles, is there a list of sites that you cover, or is it safe to assume that you crawl over all major news services (tech, politics, money, etc)?

Also like your support for ADN. I have an account there that is sitting empty, might build something up over the weekend.

Thanks!

We should cover all major news sources. We have at least 250,000 different sources we're scraping, plus we also "crowdsource" news discovery. That is: organically (stream-discovered) content gets indexed as well.

Hope to hear from you! Zane@Streamified.com

I try this with js sample app + twitter auth. I get Error: {"type":"Authentication","status":401,"details":{"message":"OAuth Authentication Failure: invalid oauth_token syntax"}}

In the other hand, this could be handy for my news reader app for ipad. This notify of new content? Is this suitable for manage several users, each one with their own rss subscriptions?

Hey mamcx,

Would you mind shooting me an email to Zane@Streamified.com so we can discuss?

And yes, you could easily manage many users' RSS feeds :)

Looks nice, just a note: the redirect from http://streamified.me is broken.

    $ curl -I http://streamified.me
    HTTP/1.0 302 Found
    Cache-Control: no-cache
    Connection: close
    Location: https:\/\/streamified.me

I'm not following... how is this broken? The headers return a 302 code, pointing to the proper HTTPS site. If you click the link you pasted, your browser should properly redirect (tested in FF, Chrome).

Those slashes are escaped with backslashes in the response

Oh, yikes! Weird, wonder why it works on browsers, then. Hm. I'll have to dig into this ;)

Fixed!

The idea is good. But clarifying it further, how many API calls per month do you offer or plan ? Correct me if I am wrong, searchable means topic/keyword parsing with context analysis (i.e. intention analysis)?

Our courtesy limit is 1k requests per day to the API. Beyond that, we're happy to discuss what makes sense, we obviously just need to prevent abuse.

Yes, our searching capacities include everything from categories to sentiment to tags to media... you name it. ElasticSearch powers our backend, which is powered by Lucene, so anything in an article can be searched. Check out the articles/search endpoint: https://streamified.me/developer/#endpoint=GET%20articles%2F...

Really love this idea. I've started something similar for my content platform Amplifize.com, but am still in very early stages.

How can folks help build this out? Could it be an open service?

Thanks! I'm one of the devs on the project; you can contact me directly at Zane@Streamified.com.

To your questions: one of the major benefits of the service is the "crowdsourced" aspect of it. That is: the more people using the same API, the more news articles get found and indexed. Furthermore, the API actually hooks into other paid APIs, so it actually wouldn't work very well for you to run it on your own servers unless you wanted to also pay for them. For all these reasons and more, we think that a hosted API is the best solution.

Totally agree with all of the above. I'll send you an email to keep the conversation going.

If this depends on the individual APIs (especially twitter) and sees any sort of traction, isn't there a massive risk that one of the providers could block access to streamified?

Per our "whitelabeling" philosophy, Streamified actually uses your own app IDs (not ours). From Twitter's perspective, we're just acting as a middleman to help you use their API better; any blockage, then, would be a result of your implementation of their API (throttling, etc).

Sweet, looks like something I could really dig into for unscatter.com

I'm curious, are you proxying apis or crawling and keeping the information local to serve?

Thanks! Be sure to drop us a line if you want to dig a bit deeper into integration.

For social networks, we're proxying the APIs (essentially). For articles and news, we both aggregate organically discovered articles from streams, as well as scrape 250k+ RSS feeds and other news sources. So, articles are stored local to the server (which is why we have such a large collection of them), but sensitive social information is a pass-thru to the social network.

However, in both cases, we're doing lots of analysis on top of the data per our "normalization" philosophy.

sure I have a couple questions, I'll shoot you an email

Hey just wanted to say that I appreciate the API and it has very nice docs! Ignore the naysayers, it's not for everyone but I can already imagine a ton of uses.

Thanks!

Who's behind this? There's not a single name anywhere on the .me or .com sites. It's hard to trust a platform service without knowing who is building it.

Streamified, Inc.: Zane Claes and David Lee You're welcome to contact us directly @ zane@streamified.com and david@streamified.com

This sounds a lot like http://singly.com/ - how is it different?

My thoughts exactly.

Sure, we both cover social login and sharing. However, Streamified supports a whole bunch of news features, and a lot more social endpoints, which Singly doesn't begin to touch on.

This is a fantastic resource... I'm finding myself trying to come up with a project to use it on.

Totally not readable on a mobile. Left sidebar uses nearly the full screen even if you zoom in.

Sorry, but this is a developer portal -- not a news source or blog -- and really isn't meant to be used on a mobile device. I acknowledge that this may not be the answer you're looking for, but we didn't design the site for mobile because what it needs to do isn't really appropriate (or feasible) on mobile -- eg, the API console.

Well it is a news source, called hacker _news_. You telling me a developer doesn't read stuff on a cell is just complete BS. I never said i want a mobile site. Just a css that doesn't completely break on small resolutions. If you don't want to fix it fine. But don't give me such a BS answer.

I acknowledge this might not be the answer you were looking for.

I'm sorry, I think I misspoke. What I meant is: you're right. What we should have done, though, was link to a blog post for you to read, rather than the dev portal itself. It just wasn't realistic to build a mobile enabled dev portal, so we should have posted something mobile friendly instead.

The rage is uncalled for. Can you be civil?

Reply on news.ycombinator.com