There are probably as many ways of sorting books as there are readers. I suppose most readers with a couple hundred books or more try to follow some kind of order in their shelves and it usually matches up with libraries and bookstores in terms of fiction/non-fiction – but then what? Depends a lot on how many books of what kind you have, I suppose – your preferences. But even back in the fiction vs nonfiction classification we have some issues.
I suppose this boils down to how you define “fiction” because you can’t define “non-fiction” without that. There are at least definitions of “fiction.” One is that it is an un-truth (and there we have another negative definition so a further definition is needed – a definition of “truth.”). The other definition, much easier, is a category of the written word – novels and short stories are included.
Within fiction there are lots and lots and lots of genres or categories and more can be invented any time. There are library genres and marketing genres and so on. These don’t always match up. Dewey Decimal only goes so far as the fiction of a nationality so within US fiction Robert Pirsig, Raymond Chandler, Zane Grey and Maurice Sendak would all be lumped together in alpha order of author.
Some Marketing Categories used by publishers and bookstores include Horror, Gothic, Western, Science Fiction and Fantasy, Romance, War, Mystery, and Historical. Children’s Fiction is another one and has its own age-specific divisions. Libraries have sometimes used these for patron convenience. Both stores and libraries sometimes have “New Fiction” shelves as well.
Books of myths and legends also pose a sort of problem. These fall into at least two main categories: those treating them as ancient religions, and those treating them as collections of stories. Books of the latter sort should, it seems to me, be placed with fantasy and science fiction, or with children’s fiction, and not with books on religion (as libraries do) or “Classics and Literature” (as most bookstores do). Of course, there is a fair amount of overlap between the two categories, and perhaps all books on mythology should be shelved together, since it is sometimes difficult to tell which class a book falls into without reading the whole thing, which librarians and bookstore managers scarcely have time for.
One category used by bookstores is “Classics and Literature”. It commonly includes most of the oldest fiction, most of the poetry, recent fiction praised by literary critics, and sometimes old books on history, philosophy and miscellaneous subjects. A more appropriate unifying title might be “Books Likely to be Assigned for School Reading,” though that might be a bit unwieldy. Perhaps old books which are not fiction should not be placed here.
Publishers have an advantage in that they can place an entry for a book under several headings in the catalog when there is some uncertainty about which category it fits best in. Libraries have a similar advantage with their card catalog, in which a book has at least three entries (for author, title, and subject) and sometimes more (if by multiple authors or on multiple subjects).