A bit of fluff unrelated to the main story might diminish the reliability of the narrator a bit, but I don't think you can conclude the entire story is fictional as a result.