I really want to run ceph because it fits a number of criteria I have: gradually adding storage, mismatched disks, fault tolerance, erasure encoding, encryption, support out-of-the-box from other software (like Incus).
But then I look at the hardware suggestions, and they seem like an up-front investment and ongoing cost to keep at least three machines evenly matched on RAM and physical storage. I also want more of a single-box NAS.
Would it be idiotic to put a ceph setup all on one machine? I could run three mons on it with separate physical device backing each so I don’t lose everything from a disk failure with those. I’m not too concerned about speed or network partitioning, this would be lukewarm storage for me.
You end up wasting a ton of space though because each vdev has its own parity drives.
What you lose in space, you gain in redundancy. As long as you’re not looking for the absolute least redundant setup, it’s not a bad tradeoff. Typically running a large stripe array with a single redundancy disk isn’t a great idea. And if you’re running mirrors anyway, you don’t lose any additional space to redundancy.
