Hacker News Reader

jikon Ask HN: Intermittent EC2 DNS failures at 1 minute past the hour

Here's the script I'm running from cron both inside AWS and outside it at one minute past the hour (with our internal DNS domain replaced with example.com):

  #!/bin/bash

  tf=/tmp/out.$$

  for turn in $(yes | head -60); do
      start=$(date)
      if ! host $(uuidgen).example.com 2>&1 | tee $tf | grep -q -s -w NXDOMAIN
      then
          end=$(date)
          echo "$cmd failed from $start to $end:"
          cat $tf
      fi
      sleep 1
  done

  rm -f $tf

jikon Ask HN: Who is hiring? (September 2014)

Quantopian (https://www.quantopian.com/about), Boston, MA

Quantopian is hiring software and operations engineers to help us disrupt the world of finance. We've built the world's first web-based algorithmic trading platform, and we're growing our team to support our rapidly expanding user base and to build everything on our ambitious product roadmap.

We're well-financed by some of the savviest investors in the business (Spark Capital, Khosla Ventures, GETCO, and various individuals).

We work on interesting problems, such as:

* Running arbitrary user code on our own servers, with all the associated security implications

* Designing and building intuitive and powerful research and development tools and APIs for our users Designing data stores to model real-world financial data and optimizing them for high throughput when running trading simulations

* Communicating with broker APIs to trade real money on our users' behalf

* Metering and autoscaling our cloud infrastructure to respond to varying load

* Designing and building an amazing user experience, both on desktop and mobile

So far, we've built Quantopian with Python and Ruby on Rails (and a little Node.js) on AWS and Heroku. We depend heavily on Redis, and our Docker usage is rapidly growing. However, we are very pragmatic and our highest priority is shipping user-delighting features built with the most sensible technologies. Be sure to check out Zipline (https://github.com/quantopian/zipline), our open-source backtester. We have several other projects that we plan on open-sourcing as well.

We've assembled a top-notch product and engineering team here in Boston, and we're growing fast. We're open to all levels of experience, from still-in-college to seasoned veteran. A financial background is not necessary at all (but is always nice to have).

We're still small enough that you will have a material impact on our company's trajectory. Our small size and ambitious goals dictate our approach to talent acquisition and retention: we believe in hiring the best engineers and giving them hard problems to solve and the autonomy to solve them.

Ideally, you:

* have a solid background in computer science fundamentals

* have good communication skills

* thrive on designing, building, and shipping mind-blowing features that delight our users

We provide:

* All the standard stuff (fully paid health and dental for you and your family, 5% 401(k) matching, gym and phone reimbursement, pretax transit, weekly catered lunch).

* Your choice of computer equipment

* A beautiful new office (in October) in downtown Boston

* Fun and intellectually stimulating work environment. We ship early and often. We have lives outside of work. We like each other.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

I'm wondering if this is too specific. The other alternative proposal that someone floated in response to my proposal -- a file containing just CVE numbers and timestamps for when the site was no longer vulnerable to each of them -- seems more generalized and covers more than just passwords.

I'd extend that proposal to allow a CVE to be listed with a special token indicating "never vulnerable" (perhaps "1970-01-01T00:00:00Z" as the patched-at timestamp) and perhaps to allow a CVE listed with two timestamps indicating the range of time during which the site was vulnerable.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

MITM attacks require far more sophistication and many more moving parts than just stealing data from a web server's memory. Just because you have the SSL cert's key doesn't mean you have the ability to use it to MITM a site.

I'm not saying what you propose is impossible; I'm just saying it's quite unlikely. Security is all about layers. You don't throw away a layer that could be effective at increasing security in the majority of cases just because there are some cases where it wouldn't work.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

I could live with simplifying the proposal to just list CVE numbers and timestamps.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

As others have pointed out, if they have metasploit installed they're just going to scan your site for all known vulnerabilities in a matter of minutes. Making it easier for the good guys to find out whether you've patched Heartbleed is not going to make things any easier for the bad guys.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

>What site operator would install something which turns users away?

The point of this proposal is mostly to enable sites to tell users that they _have_ been fixed (and when), not to tell users that they _haven't_.

Having said that, site operators pay to install valid SSL certificates from valid CAs because they know that if they won't, the browsers will increase the friction necessary for users to access their site.

If something like what I am proposing is widely adopted, then not patching their sites and notifying users that they have done so will similarly increase friction, so site operators will have an incentive to do it.

>Not every site operator is very security-savvy, and to a first approximation no users are.

The penetration of password management apps like LastPass continues to increase. The fact that many users do not choose to avail themselves of better security is no justification for not implementing better for security for the users who do.

If that were not the case then SSL would not exist.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

We use CloudFlare at work, but I had not previously foreseen the possibility that it might be needed for my blog.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

>Your proposal may want to follow RFC 5785 aka "/.well-known/"

This is addressed in a comment below the blog posting.

>That said, I don't see what problem you're solving or why the solution you propose should help.

I don't know how to explain it better than I already have. Sorry.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

Well, yes, frankly, they should. ;-)

Frankly, it doesn't matter whether they do or not, because if a vulnerability is known and there are exploits for it in the wild, then hackers are simply going to try to exploit it, not check /heartbleed.txt.

Aside from all that, sites who don't have the courage to list their unpatched vulnerabilities can simply only list the ones that are already patched.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

It's obviously going to be a lot easier to convince maintainers of web sites with wildly varying stacks to drop a plain-text file into the root directory of their site, then it is going to be to convince them to implement a universal password change API.

If Mozilla, Microsoft, and Google added support for /heartbleed.txt to their browsers then a whole lot of people would check it. Even if not, if password managers are modified to check it, and this leads to even a minor increase in the penetration of the user of password managers by users of the internet, that would be a good thing.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

That requires keeping a huge amount of state and doing a lot more complicated programming then just checking a few fields in a YAML file.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

Great, and how do the users keep track of which of the hundreds of sites they use have done that? With the idea I'm proposing, it's possible to automate this _in the client_, such that users can proactively defend themselves against sites that haven't patched themselves or even indicated whether they were ever vulnerable in the first place.

"Do you want to visit this web site? It may by vulnerable to the CVE-2014-0160 (a.k.a. 'the Heartbleed bug')." or the like.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

Sorry. My blog doesn't usually get this much traffic. ;-)

Trying to increase the number of servers but httpd isn't cooperating.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

Yeah, I added an update to the bottom of the posting mentioning the possibility of generalizing it.

jikon We need a “/heartbleed.txt” standard, and we need it ASAP

I hardly think that a security hole, no matter how large, at a single web site, is on the same magnitude as one security hole that probably impacts the majority of web sites on the internet and could have been taken advantage of completely invisibly for years.

jikon DDOS on Namecheap Free DNS and Default DNS V2

According to whois for namecheap.com, the DNS for that domain is hosted on dynect.net, and "host status.namecheap.com" resolves just fine (to 204.232.212.56), so it does not appear to be a DNS issue that is preventing your status page from working.

jikon DDOS on Namecheap Free DNS and Default DNS V2

If you're as big as NameCheap, you can afford to pay CloudFlare or somebody like it to protect your status page.

jikon DDOS on Namecheap Free DNS and Default DNS V2

Just for future reference, it's usually considered a good idea to put your status page on completely independent infrastructure so that it stays up even when the rest of your stuff goes down. A status page that doesn't work during an outage isn't particularly useful.

jikon Ask HN: Who is hiring? (January 2014)

Quantopian - Boston, MA - Operations Engineer

https://www.quantopian.com/about#jobs jobs@quantopian.com

At Quantopian, we’re building the world’s first algorithmic trading platform in the cloud. Our engineers are all generalists who write, test, and deploy code across our stack. Each of us also has an area of deep, focused excellence. If you like keeping the gears turning smoothly, crushing hard problems, and working with really smart people, then you may be the operations engineer we’re looking for to help us change the world of finance.

As a Quantopian Operations Engineer, you’ll report directly to the Vice President of Operations and play a key role in expanding and maintaining our cloud infrastructure. Your primary responsibilities will include:

- designing and implementing improvements to our application infrastructure and continuous delivery pipeline;

- identifying gaps in our application monitoring infrastructure and propose and implement solutions;

- monitoring the performance and availability of our application, analyze and troubleshoot issues, and propose and implement improvements to prevent them from recurring; and

- participating in our on-call rotation and providing occasional after-hours support for production issues.

Skills & Requirements

The ideal candidate will have experience in most of these areas:

- Hands-on management of Linux systems (Ubuntu preferred), 3+ years

- Python, Ruby automation scripting / application development

- MongoDB / other NoSQL databases

- Management and deployment of cloud-based infrastructure services such as Heroku, AWS, RightScale, Papertrail, PagerDuty

- Internet infrastructure services such as DNS, SMTP

- Development infrastructure services such as Github, Jenkins

Here are some of the Operations projects we’ve worked on recently.

- a Python class which uses off-the-shelf tools to manipulate the Interactive Brokers API Gateway, so that we can transparently log our users into their stock trading accounts through our application;

- Geckoboard status boards with custom push widgets that give us real-time visibility into site activity, with smart alerts about anomalies, displayed on a heads-up display in our office that’s driven by a Raspberry Pi; and

- strong encryption of our users’ intellectual property in our database, implemented using mongoid-encrypted-fields in our Ruby layer and a compatible Python class we built ourselves and integrated with Schematics.

About Quantopian

We’re still a small company (our daily standups include every employee!), so you’ll play a major role in shaping our direction and enjoy a complete lack of bureaucracy.

Our small size and ambitious goals dictate our approach to talent acquisition and retention: we believe in hiring the best engineers and giving them hard problems to solve and the autonomy to solve them.

Our office is conveniently located in Downtown Crossing.

We offer flexible work hours and a deeply ingrained respect for a healthy work/life balance.

We offer generous benefits, including 401(k) matching, stock options, and health insurance with no employee premium.

FOSS is the foundation of our product, and we believe in giving back to the community. We’ve open-sourced Zipline, our backtesting engine (>1,200 stars and >200 forks!), as well as other code we’ve written, and we plan on opening up more of our code in the future.

jikon Ask HN: Who is hiring? (December 2013)

Quantopian - Boston, MA - Operations Engineer

https://www.quantopian.com/about#jobs jobs@quantopian.com

At Quantopian, we’re building the world’s first algorithmic trading platform in the cloud. Our engineers are all generalists who write, test, and deploy code across our stack. Each of us also has an area of deep, focused excellence. If you like keeping the gears turning smoothly, crushing hard problems, and working with really smart people, then you may be the operations engineer we’re looking for to help us change the world of finance.

As a Quantopian Operations Engineer, you’ll report directly to the Vice President of Operations and play a key role in expanding and maintaining our cloud infrastructure. Your primary responsibilities will include:

- designing and implementing improvements to our application infrastructure and continuous delivery pipeline;

- identifying gaps in our application monitoring infrastructure and propose and implement solutions;

- monitoring the performance and availability of our application, analyze and troubleshoot issues, and propose and implement improvements to prevent them from recurring; and

- participating in our on-call rotation and providing occasional after-hours support for production issues.

Skills & Requirements

The ideal candidate will have experience in most of these areas:

- Hands-on management of Linux systems (Ubuntu preferred), 3+ years

- Python, Ruby automation scripting / application development

- MongoDB / other NoSQL databases

- Management and deployment of cloud-based infrastructure services such as Heroku, AWS, RightScale, Papertrail, PagerDuty

- Internet infrastructure services such as DNS, SMTP

- Development infrastructure services such as Github, Jenkins

Here are some of the Operations projects we’ve worked on recently.

- a Python class which uses off-the-shelf tools to manipulate the Interactive Brokers API Gateway, so that we can transparently log our users into their stock trading accounts through our application;

- Geckoboard status boards with custom push widgets that give us real-time visibility into site activity, with smart alerts about anomalies, displayed on a heads-up display in our office that’s driven by a Raspberry Pi; and

- strong encryption of our users’ intellectual property in our database, implemented using mongoid-encrypted-fields in our Ruby layer and a compatible Python class we built ourselves and integrated with Schematics.

About Quantopian

We’re still a small company (our daily standups include every employee!), so you’ll play a major role in shaping our direction and enjoy a complete lack of bureaucracy.

Our small size and ambitious goals dictate our approach to talent acquisition and retention: we believe in hiring the best engineers and giving them hard problems to solve and the autonomy to solve them.

Our office is conveniently located in Downtown Crossing.

We offer flexible work hours and a deeply ingrained respect for a healthy work/life balance.

We offer generous benefits, including 401(k) matching, stock options, and health insurance with no employee premium.

FOSS is the foundation of our product, and we believe in giving back to the community. We’ve open-sourced Zipline, our backtesting engine (>1,000 stars and >180 forks!), as well as other code we’ve written, and we plan on opening up more of our code in the future.

jikon Ask HN: Who is hiring? (October 2013)

Quantopian - Boston, MA - Operations Engineer

https://www.quantopian.com/about#jobs jobs@quantopian.com

At Quantopian, we’re building the world’s first algorithmic trading platform in the cloud. Our engineers are all generalists who write, test, and deploy code across our stack. Each of us also has an area of deep, focused excellence. If you like keeping the gears turning smoothly, crushing hard problems, and working with really smart people, then you may be the operations engineer we’re looking for to help us change the world of finance.

As a Quantopian Operations Engineer, you’ll report directly to the Vice President of Operations and play a key role in expanding and maintaining our cloud infrastructure. Your primary responsibilities will include:

- designing and implementing improvements to our application infrastructure and continuous delivery pipeline;

- identifying gaps in our application monitoring infrastructure and propose and implement solutions;

- monitoring the performance and availability of our application, analyze and troubleshoot issues, and propose and implement improvements to prevent them from recurring; and

- participating in our on-call rotation and providing occasional after-hours support for production issues.

Skills & Requirements

The ideal candidate will have experience in most of these areas:

- Hands-on management of Linux systems (Ubuntu preferred), 3+ years

- Python, Ruby automation scripting / application development

- MongoDB / other NoSQL databases

- Management and deployment of cloud-based infrastructure services such as Heroku, AWS, RightScale, Papertrail, PagerDuty

- Internet infrastructure services such as DNS, SMTP

- Development infrastructure services such as Github, Jenkins

Here are some of the Operations projects we’ve worked on recently.

- a Python class which uses off-the-shelf tools to manipulate the Interactive Brokers API Gateway, so that we can transparently log our users into their stock trading accounts through our application;

- Geckoboard status boards with custom push widgets that give us real-time visibility into site activity, with smart alerts about anomalies, displayed on a heads-up display in our office that’s driven by a Raspberry Pi; and

- strong encryption of our users’ intellectual property in our database, implemented using mongoid-encrypted-fields in our Ruby layer and a compatible Python class we built ourselves and integrated with Schematics.

About Quantopian

We’re still a small company (our daily standups include every employee!), so you’ll play a major role in shaping our direction and enjoy a complete lack of bureaucracy.

Our small size and ambitious goals dictate our approach to talent acquisition and retention: we believe in hiring the best engineers and giving them hard problems to solve and the autonomy to solve them.

Our office is conveniently located in Downtown Crossing.

We offer flexible work hours and a deeply ingrained respect for a healthy work/life balance.

We offer generous benefits, including 401(k) matching, stock options, and health insurance with no employee premium.

FOSS is the foundation of our product, and we believe in giving back to the community. We’ve open-sourced Zipline, our backtesting engine (>1,000 stars and >180 forks!), as well as other code we’ve written, and we plan on opening up more of our code in the future.

jikon Show HN: Zipline - algorithmic trading in open source Python

You'll have to roll your own backtester though when you build your own historical DB.

Or you could just PR a new data source class for Zipline that pulls data from IB. Ah, the wonders of open source!

jikon Quantopian Brings Algorithmic Trading To The Masses

You are not doing this for "the greater good" you are doing this to make money, as is GETCO et al.

Yes, we're doing this to make money. However, making money and doing good in the world are not mutually exclusive, and making money and stealing are not inextricably linked.

We believe we can make money at this, or we wouldn't be doing it. We intend to make money in an ethical way, giving our users the tools they want and respecting and protecting the privacy of their intellectual property.

Stealing algorithms and protection of algorithms are fundamental issues in the industry.

I entirely agree, but it does not follow that everyone doing algorithmic trading is a thief or that the only way to make money from algorithmic trading is to be a thief.

jikon Show HN: CGI script to produce RSS feed of comments on HN post

Perhaps I just don't grok the zen of HN, but I just don't get why the site doesn't provide RSS feeds of comments on postings.

I also don't get why nobody has written something to provide this. Or, at least, if it has been written, I couldn't find it (perhaps my Google mojo just isn't high enough).

http://api.ihackernews.com/ gets part of the way there, but it's an API, not an RSS feed, and it's broken and has been for quite a while.

So I wrote a script (gist at the link above) to generate such a feed.

You can see it in action at http://jik3.kamens.brookline.ma.us/cgi-bin/hn-comment-feed.c... (which I will have to remove if there is so much interest in this Show HN that people clicking on the link overwhelms my tiny little VPS).

jikon Quantopian Brings Algorithmic Trading To The Masses

HFT isn't.

We're not doing HFT. We're doing algorithmic investing. There's a big difference.

(I work for Quantopian.)

jikon Quantopian Brings Algorithmic Trading To The Masses

The main reason is because the function to make a trade will only place market orders.

We (I work for Quantopian) will surely support other order types when we support live trading.

Also, I don't see how the backtester avoids look-ahead bias.

We provide over ten years of historical minute bar data for U.S. equities, with no survivorship bias. This means two things:

1. The amount of data we provide is sufficiently large that if you test your algorithm against a bunch of stocks over that entire period of time and it performs reasonably well, it's unlikely that it's overfitted to the data in a way that is going to bite you on the ass in live trading.

2. But just to be even more paranoid, the smart algorithm writer will do just what you describe -- divide the available data into lots of subsets, randomly pick which subsets of the data to test again each time you backtest, and don't start live trading an algorithm until you've confirmed that it performs well on random subsets of data that you haven't previously tested it on. Right now on Quantopian you'd have to do all that data segmentation and selection by hand, but I suspect that we will eventually add features to make it easier to do automatically.

jikon Quantopian Brings Algorithmic Trading To The Masses

...but for the last 10% you will need something more powerful like C or matlab or excel.

Pandas can stand up to any of those. And you make the (incorrect) assumption that the features that Quantopian is offering now are the same features we (I work for Quantopian) will be offering in six months or a year. The product will grow. We will give users what they want and need to be successful.

oftentimes strategies don't directly translate to production profits, and to a great extent they depend on simulator assumptions...

No more or less true of Quantopian than of any other platform, off-the-shelf or home-grown, for quantative trading.

jikon Quantopian Brings Algorithmic Trading To The Masses

There are all sorts of reasons why GETCO might have chosen to back us (I work for Quantopian), but I can say with certainty that "Steal algorithms from the people who wrote them" wasn't one of them.

GETCO really doesn't need to steal other people's algorithms.