Rational behavior is behavior with purpose. It is acting out of self-fulfillment / self-interest / self-preservation in a logical (or internally consistent) way. "I want this, so I am going to do these things because they will help me achieve this goal".
Irrational behavior is when your behavior does not serve to fulfill any sort of goal whatsoever, or if the behavior is totally at odds with your desired goal.
A dog is rational when he wags his tail to entice gifts of food (because his desire is food). A dog is irrational when he bites the hand that feeds him (because he is a dog and still wants food).
Using this formulation of "rational behavior", mentally ill people frequently fall into the "irrational" group, as they will frequently do things for inexplicable and/or random reasons, which do nothing to serve their purported goals. Obviously this depends massively on what type of mental illness we are talking about, but I think Terry's schizophrenia/bipolar disorder generally qualified, as it led to frequent internally-inconsistent behavior.