Not sure about the video lectures, but I'm fairly certain the original course was taught to MIT undergrads as an intro to CS and EE. See https://people.eecs.berkeley.edu/~bh/sicp.html and https://mitpress.mit.edu/sites/default/files/sicp/course.htm....
I don't totally disagree with you btw. Pretty sure in that interview with Hal Abelson someone mentioned above, he admitted the book leaned a little heavily on calculus because all MIT freshmen had to take that course, so it seemed a natural source for examples. Perhaps not as natural to someone who hasn't taken a calculus course in a decade (me). I still got a lot out of the book though.