In Java you will have to deal with an array with a million pointers to a million objects and in C++ you will deal with a sequential segment of memory that is a million times the size of your object - and mostly in your CPUs cache. That alone will be much, much faster.
But you can't do that in Java at all.
When you want, for example, an array of (float x, float y) “objects” in Java, you just keep them as two arrays (float[] x, float[] y) — and I guarantee it will use the same amount of memory and will have nearly the same speed as C++ (depends on what cache locality you need for a specific task, it might even be faster to have two separate arrays for some tasks).
So yeah, depending on the language, a slightly different implementation is more efficient, so one has to know its tools.
Also, as far as I know, when project Valhalla extension goes live, you will have an option to do packed structs the same way as C++ or C# does.
Edit: Oops, there was another reply about the same in the thread. I should remember to refresh the threads before answering in the morning.
And if you are doing something like nearest cluster mapping, or anything else where you process your data in linear order you are not going to get the optimized cache locality. Plus you can't read the file with mmap if your data is from there.
I would go further and say that if your job is analyzing terabytes of memory-mapped binary files, maybe C++ is not optimal either, and investing in specialized languages like Shakti/K would pay off quickly.