This checks out after what they recently did to the ZLUDA project. As an owner of an AMD gpu I agree that ROCM support is really bad. It works half of the time and fairly poorly.
Even worse on Linux. Even worse on more exotic distros like Bazzite where I still can’t get koboldcpp to run, which was already kind of a hassle on my previous distro.
Yeah I hear ya. How do you happen to be running it? I use NixOS and its a challenge there for me but I found atleast some success using docker since the dependencies are so out of control for AI at the moment.
Also give Ollama-rocm + Open Webgui a shot as an alternative to koboldcpp if you cant get atleast some text generation to work because that is the only thing I have got working with rocm.
The trick to nixos, in this instance, is to use a python venv. Python dependencies are fickle and nasty in the first place, triply so when talking about fast-churning AI code, I tried specifying everything with nix, I succeeded, and then you have random comfyui plugins assuming they can get a writeable location by constructing a path from comfyui’s main.py. It’s not worth it: Let python be the only dependency you feed in, let pip and general python jank do the rest.
At the moment I just don’t. I got kobolcpp to run through distrobox / boxbuddy but I can’t get it to compile with rocm, so I can only use CPU generation, which is abysmally slow. Might go back to NovelAI when they release their new model if I can’t find a solution.
