It wasn't as common as today, and it were isolated cases, but it was certainly happening.

https://en.wikipedia.org/wiki/XMODEM#:~:text=XMODEM%20is%20a....

My dad mentioned that he did his graduate work from home in the early 1980s (maybe late 70s?) using a 32-column terminal that he made from an Intel 8008 and a TV. He apparently bypassed the color demodulator in the TV to get a crisper B&W image. He dialed in to the University computer with a 300 baud modem.

Yes, except management did not approve because they couldn't check to make sure you were wearing your required white shirt, dark tie, and work-appropriate slacks.

Even in the office they used terminals connected to mainframes over serial cables.

Absolutely it happened, but it was uncommon. My dad had a Teletype at home in the 1970s, with a 110-baud acoustic coupler. Later upgraded to an ADM-3A and speed went to 300-baud.

I credit this for sparking some interest in programming, as I was able to use these to play around with small FORTRAN and BASIC programs.

The Bell Labs folks had phone lines to their homes, so they could have terminal access from home I believe. Or at least I think Brian Kernighan mentioned he did.