> angle of attack etc
This is wrong.
The angle of attack is not a parameter of the stall speed, it is the cause of the stall (for a given configuration, assuming well below transonic speed). This is why for example, for precise handling you should use an angle-of-attack indicator (a large majority of fighter jets and more generally military aicraft have it).
> you should also give yourself some margin since you don’t ever really know what the stall speed is until you stall
The manufacter speed already take into account the afore-mentioned parameters into account, and the resulting speed is the worst case scenario, if not told otherwise (usually max weight, max forward CG).
One should not fly with an arbitrary speed margin, but instead use well-known speedq (1.3Vs, 1.45Vs, where Vs=stall speed in the given config) depending on the flight phase, and remember well the bank angle limit associated with them.