Hacker News Reader

Comment by srean | Hacker News Reader

Let me add another often-overlooked tweak that you can use with a random forest: applying random rotations before one learns the trees.

This is extremely powerful. To give you some intuition on why this is so, recall that simple trees do axis parallel splits of the space. If the trees have to approximate an inclined line, they have to do a high resolution stair stepping to approximate it, -- like so --

    __
      |
      ------
            |
             ---------
                      |_______

this ends up requiring a lot of trees or a very deep one. Now if you equip the algorithm with a few inclined splits, which is exactly what a random rotations would do, you gain a lot of power.

View on HN

How is that different from allowing non-axis-aligned splits?

That is a very interesting question and one can go deep into the woods with that, not everything is known about this question. There are two distinct effects at play here, computational cost and learning efficiency.

A brutally shortened response would be that searching for non-axis aligned splits in the nodes can be very expensive CPU wise. This addresses the first effect at play. Applying a random rotation on the other hand is quite cheap.

The second effect at play is that ensembling loses its power when you optimize over oblique splits because of lost diversity.

It comes down to a trade off.

its all much of a muchness. You can use a linear function at the nodes to project the space before applying a >. However, all these tweaks tend to just consume CPU for marginal benefit. You can still appoximate a non axis aligned decision boundary with a series of stepwise aligned splits, albeit less compactly. But if your aim is to just generate a ton of non linear distance functions ... a la random forest, focusing on tree expressivity is the wrong thing. Why not spend your CPU cycles on more trees instead instead, which leads to smoother decision boundaries in general when mixing their responses. There is a ton of tweaks you can do, but unless you are expressly researching ML algorithms, the pragmatist should not care and just try a few options from whatever comes in scipy.

Reply on news.ycombinator.com