It doesn't talk out loud if you hold the phone up to your ear. To use Siri with minimal noise (aside from your own speech):

Push the power button to turn on the screen. Hold the phone to your ear. Wait for the be-beep sound, played quietly through the ears-only speaker. Speak. Keep the phone against your ear. Siri will respond through the ears-only speaker.