Who is "they"? There is not some scuba sheriff who magically appears and stops you from going as deep as you want. Get some gear, swim out, and dive. How deep you go is 100% up to you.
Also, dive computers make the whole thing very safe by tracking exactly how long and how deep you can stay without needing any lengthy nitrogen releasing safety stops. As a recreational diver with a bit of experience, you can feel very confident going down to 160 feet (maybe even 200?) as long as you listen to your dive computer.