I’ve been using Obsidian lately. Proprietary with an open plugin ecosystem. Works well, makes it easy for me to integrate with other notes and such, but I haven’t figured out a good workflow for exporting work for submission. That said, it’s all markdown and there are lots of plugins for stuff like that, so it’s probably mostly just that I haven’t tried very hard.

In the past I’ve used Google Docs (proprietary), Scrivener (proprietary), Manuskript (open), Zim (open), and probably a few I’m forgetting. Really it just comes down to what you’re looking for out of the software, there are lots of options.

The biggest thing to keep in mind from a self-hosting perspective is local storage and easy backups under your own control. I use syncthing to keep my whole Obsidian vault synced across a few devices; for some software that’s easier or harder due to file formats and accessibility.

https://www.literatureandlatte.com/scrivener/overview it’s proprietary, but it has a lot of features geared towards writing novels/screenplays/etc

I don’t write books but I’ve helped written a couple textbooks which used LaTeX. I personally use TeXstudio, but there’s many clients out there. If you appreciate beautiful typesetting, you’ll likely enjoy TeX despite its learning curve.

I take the approach of doing content first and styling second. For content I don’t need anything more sophisticated than a plain text editor. I like it because it removes decisions that I really don’t need to be making at that point.

It may be overkill for most—it’s not the easiest thing to set up and it’s got a high learning curve—but for heavy research and world-building I’ve found Semantic MediaWiki revolutionary.

You can create auto-generated and auto-updating maps, timelines, tables, etc., and make live queries that pull information from all relevant pages. (For instance, if you write pages for a bunch of events and annotate them with dates, locations, and which characters are involved, you can create a map and itinerary for each character and a list of all the characters they’ve met or interacted with. If two characters meet in a particular place, you can generate a list of the most recent events that happened to each character, recent events at that location, past events where both characters were present, people and places they know in common, etc. And if you decide to shuffle events around, everything updates accordingly.) It’s also great for collaborative writing, it can be accessed through the web from any device, and it has automatic versioning. It’s almost insanely powerful, and of course it’s FOSS.