As soon as some random website's PHP script publishes your IPv6 address, there goes your security.
Assuming your IP address will remain secret seems naive.
Also, this assumes your IP address within your /48 is randomly chosen. Common user choices (or router implementations) might not default to random choices, or the randomness might not actually be very random.