No need for them to be babies. That will work as long as they're younger than 12, and probably for a few years after. But if the kids have already learned to speak another language, they will hate this approach.
The way to fix this is to find them things that only exist in that language and not in English. It could be other family members, friends, TV, music, time abroad, etc.
Kids, like adults, need motivation as the primary factor that determines success learning a language.
I'm living in the US: I always speak Korean to my teen kids and they mostly answer in Korean, so they have no problem understanding phrases like "When is dinner today?" However, because they've never been to school in Korea, they basically have zero knowledge of Korean words for, say, "descendant", "ambassador", or "independence", while being totally familiar with their English counterparts.
It doesn't mean your points are invalid, but in general people like your kids are just going to be way better at learning language.
IMHO, it's almost cruel that many colleges expect all students to fulfill a 2 year foreign language requirement, even if they have no prior exposure or education. You end up with language learning that is watered down and generally a waste of the learner's time, in addition to making their education more expensive.
I agree it might often be a failure, in say, achieving fluency. But certainly no a waste. Exposure to foreign languages, and their cultural contexts, can be very enriching, especially the more distant they are from your own culture and language family. And the contrast with your own language and culture can teach you much about your own.
Most people would get way more out of studying a culture and its context by directly studying that and in way less time. This time saved one could spend as 4 semesters of specialized cultural studies, one full cultural study abroad semester, or learning about other things (comparative linguistics?).
There's also the idea that you can only fully appreciate works of literature in their original language, but learning Russian to appreciate Dostoevsky is overkill and many people love his work who don't know the language. Learning about other cultures in translation is arguably going to be superior since any translation you can do is with skills way below fluency.
Speaking to them in your native tongue isn't going to make them worse at English if they're growing up in an English-speaking environment.
Kids have an incredible ability to learn languages, given motivation, and they don't need to go to school to learn words like "accountant" reading books will do.
To me, giving your children a second language is an incredible gift. Setting the pathways in the brain that separate concepts from the written and spoken representation, the abstractions of different grammars, etc, is very valuable. And something that I have really valued as I grew older.
But I (superficially, of course) understand perhaps why you'd like curate their experience of something you took fairly extreme steps to escape. That might well be the greater gift.
Appreciate your perspective -- thanks for sharing.