Can't you simply decouple your lambda project into two different parts where you have public lambda(s) calling your private/VPC lambda function(s) when required?

Public Lambdas can invoke VPC Lambdas (AFAIK, the reverse is not possible without a VPC endpoint).