I was once thinking of implementing such an online judge. Looking through the existing implementations, most restrict the syscalls available to the untrusted process, and run in a chroot.

On freebsd, you'd use something like capsicum. On linux, something like seccomp. Although, nowadays I imagine people would think of just running in a VM or a container.