True, but there's many parts that you do have to copy into your source code:
* magic numbers at the start of the file format
* all sorts of enums that you just have to know, names and numbers and the correspondence between them
* names of data types and classes are often needed
* there are often many "standard" things that will be in both the original source and the duplicate because it's the industry standard (things like bold text in any word format or parallel constraints in cad files). They will mostly match in all programs of the same time
And that's just the beginning. You could make an argument that all of those fall under copyright. And they shouldn't. So it really isn't that far from an API.