Regardless, even without these unobtainium big machines, the cardinality of the largest array for a platform (by any number of dimensions) must be representable in size_t. A too large of an array would not fit in RAM and you should get a compile/link/out of memory error.
Except we're talking about variable sized arrays, where the wanted size is not known until runtime (e.g. from user input). We can't check this at compile time.
This also isn't the same problem as an out of memory error; we need to check that nn didn't overflow the size of size_t, as well as that the memory allocation is successful. Something like:
int n = /* run-time value */
size_t sz = (size_t)n;
// Check n fits in size_t.
if ((int)sz != n) <error>
sz *= sz * sizeof(float);
// Check against overflow.
if (((sz / (size_t)n) / sizeof(float)) != (size_t)n) <error>
float *matrix = malloc(sz);
// Check allocation succeeded.
if (matrix == NULL) <error>