Im not writing books, but check out bookstack

https://www.literatureandlatte.com/scrivener/overview it’s proprietary, but it has a lot of features geared towards writing novels/screenplays/etc

I’ve been using Obsidian lately. Proprietary with an open plugin ecosystem. Works well, makes it easy for me to integrate with other notes and such, but I haven’t figured out a good workflow for exporting work for submission. That said, it’s all markdown and there are lots of plugins for stuff like that, so it’s probably mostly just that I haven’t tried very hard.

In the past I’ve used Google Docs (proprietary), Scrivener (proprietary), Manuskript (open), Zim (open), and probably a few I’m forgetting. Really it just comes down to what you’re looking for out of the software, there are lots of options.

The biggest thing to keep in mind from a self-hosting perspective is local storage and easy backups under your own control. I use syncthing to keep my whole Obsidian vault synced across a few devices; for some software that’s easier or harder due to file formats and accessibility.

Obsidian is really good for me for writing. Though I enjoy writing in markdown/plain text. I use italics for comments and bold for changes to make. It’s proprietary, but there’s a community plugin downloader built right in, with lots of open source support. I have my own database set up for syncing using the Self-hosted Livesync plugin and it works really well. There are other plugins specifically for writers as well.

I don’t write novels, but lately found apostrophe (gnome) and ghostwriter (KDE) which are intended to write using markdown, and have a UI intended to allow you to focus on writing. You can later use git to manage versions and backups (in a remote repository).

If you want something more focused on relationships, and regarding the answer from another user suggesting Obsidian, you might use also logseq, but I didn’t use it yet (but hear a lot of positive vibes around it).