I had no Lisp experience whatsoever before I started with Clojure, and the only real difficulties I experienced were (in descending order):
a) translating my imperative-style thinking into functional style - this would happen to me in Racket as well.
b) macros - but you don't need to grasp macros as a beginner, that's a topic by itself anymay.
(and NO, the bracket syntax was no problem at all)
YMMV if you have an other background. It is true: The more programming languages you already know, the easier it will be to learn a new one (iff it doesn't conceptionally diverge too much). But if you don't have that background, why not learn the desired language right away?