Compared to? Perspective matters a lot. It is definitely bigger than Erlang but I would still classify it as considerably smaller than languages like Scala and C++ and still smaller than Ruby and Python.
> It is essentially a big pile of macros
Since "a big pile of macros" is not quantifiable in anyway, I have decided to check for macros and function definitions in Elixir source and I got 1687 functions and 169 macros throughout the whole standard library. It is roughly 10%.
The Kernel module (imported by default) has 60 functions, 78 macros, which is, for example, considerably smaller than Haskell's prelude (about 300 last time I checked?).
Considering things like `defmodule`, `def`, `defp`, `case`, `&&`, `in` are regular macros in Elixir instead of syntax keywords (as they would be in many languages), I wouldn't personally call it a big pile. Although I agree that's a matter of perspective and that's why I decided to measure it).
> the barrier and number of concepts that you have to get familiar with is huge
I definitely agree with this. There are functional aspects, like pattern matching and immutability, the concurrency aspects and the OTP bits. There is definitely a learning curve although most patterns, as you said, are necessary for building the kind of systems Erlang/Elixir enable you to.