You likely want to move your environment variables to ~/.pam_environment instead of setting them in fish configuration.