In my opinion, for a book to be considered non-fiction it must be true. Sure you can stretch the truth on dialogue because you’re obviously not going to remember word for word what was said. But if you stretch the truth too far by saying you killed someone when really you only we’re deeply affected by the death, then you should just consider it fiction. When you add characters that never existed and events that never happened into a non-fiction story, you take away from the credibility and readers start wondering what really happened.
I think half-truths are okay if the author makes note of it. They should tell you that some events in the story are stretched out and not entirely true, so that they can avoid a situation like James Frey had. It would have been wise for James Frey to come out and say that his story was stretched in areas when he first published the book. He could have even said his story was more fiction-like but inspired by true events.
When it comes to labeling things between fiction and non-fiction, it only matters a little bit. The reader should know if what they are reading is either true or fake, but whether the book is good or not is the main point. I could care less if we didn’t distinguish between other genres like romance, mystery, and comedy because in reality, most stories contain a little bit of each. In the end, all that matters to me is if what I am reading is a good book.