Yeah we struggled to keep our memory usage under 500M per worker. Sidekiq helped because it was able to use threads effectively, but especially when we were running resque workers, we could only handle tens of tasks per second, and often fell behind. I think Rails' autoload-the-world philosophy was a big part of the problem and we spent some time trying to untangle dependencies, but it was swimming against the stream.
I'm not sure if these same problems do or don't come up with Phoenix, but when I briefly used it, it did seem to have a smaller memory footprint.