I'm not sure how many 'layman' are comfortable with Linear Algebra and Calculus ;).

Give it a shot; there's a fair number of non-mathy digressions that are worth reading, and you sound like you have the background to handle the mathier parts.

No, definitely not. Try to find some book in the topic that you are interested in. If it is electromagnetism, go for the black book with four equations printed. If it is quantum mechanics, I like the Sakurai book.

Yes, if you supplement it with problems -- you'll have to find them elsewhere or make some up yourself. (There's a companion book of exercises recently published, not on the website, which I haven't read.)

Of course different books help most for different people. This one's not the best if you want to pass a course and put it behind you. I think it's very much worth checking out if you want to be challenged and inspired to tackle problems for yourself and perhaps pick up a bit of how Feynman thought. There are very few other introductory texts by physicists of his stature: Matter and Motion by Maxwell, and, well, anything else? (Not counting popularizations; it's a fuzzy distinction, but these are securely on the textbook side.)

Parts are and parts definitely aren't. I have a four year physics degree and found the treatment of certain sections very challenging. I would not recommend it as an introductory text

Yes, this series is fantastic. It offers a number of side-discussions that should also prove useful if you decide to go on to read more specialized physics material (e.g. to follow a more "traditional" academic physics curriculum).

These books were created from Feynman's lectures to freshman and sophomore Caltech students. I have not looked at the books in detail, but I think that a dedicated reader who is comfortable with calculus should be able to understand them.

As a very first physics textbook I really liked Halliday's. But I did not read very extensively back then, the focus was really on understanding and solving problems.

Definitely. But there aren't any problems and problem-solving is essential to learning physics so you need to have other textbooks at hand as well.