I believe the points shown are upvotes minus downvotes, but the ratio of upvotes to downvotes probably affects the ranking of a story too.
Also, IIRC there's a (hidden) modifier based on domains that artificially moves submissions up or down the rankings. Suffice to say that article ranking isn't just a function of time and votes.