Once we have super fast reliable internet we’ll likely have the whole computer as a service. We’ll just have access terminals basically and a subscription with a login, except for the nerds who want their own physical machine.
Given the digital literacy of many “regular people” (e.g. my father, and seemingly every other of my friends), the idea is appealing. Especially, as most of them don’t care about privacy. Give them decent availability, and they will throw money at you. And if you also give them support, I will, too.
It’ll never be fast enough. An SSD is orders of magnitude slower than RAM, which is orders of magnitude slower than cache. Internet speed is orders of magnitude slower than the slowest of hard drives, which is still way too slow to be used for anything that needs memory relatively soon.
A SATA SSD has ballpark 500MB/s, a 10g ethernet link 1250MB/s. Which means that it can indeed be faster to swap to the RAM of another box on the LAN that to your local SSD.
A Crucial P5 has a bit over 3GB/s but then there’s 25g ethernet. Let’s not speak of 400g direct attach.
It will crash as soon as it needs to touch the swap due to the relatively insane latency difference.
For those too young to remember, the A:\ drive was for the hard 3" floppy disks and B:\ drive was for the soft 5.25" floppy disks. The C:\ drive was for the new HDDs that came out, and for whatever reason the C:\ drive became the standard after that.
wait, didn’t some tech youtubers like LTT try using cloud storage as swap/RAM? afaik they failed because of latency
This guy used ICMP data payload as a hard drive. It kinda worked.
I remember using ICMP data to bypass my high school’s firewall. TCP and UDP were very locked down, but they allowed pings. It was slow though - I think I managed to get a few KB per sec. Maybe there’s faster/fancier firewall bypass methods these days. This was back in the 2000s when an entire school would have a single OC-1 fiber connection.
I wonder if there would be a speed boost by setting 2 gdrive as raid 0 for off site backups
So is pretty much all of the cloud services the average user already subscribes to. People still use them though.
Agreed. This is especially bad, though, because if it’s compromised they basically have hardware-level access to your machine. Unless you’re using encrypted swap, and I’m not sure how standard that is.
Obviously you should set up device mapper to encrypt the gdrive device then put the swap on the encrypted mapper device.
Once upon a time, Google offered unlimited drive storage as part of some GSuite tiers. They stopped offering it a while ago and have kicked most/all legacy users off of it in the past few months. It was glorious while it lasted 😢
Guess they ran everyone out of business that they needed to, so now the premium features get yanked and your choice of alternatives is curtailed. Hooray for enshittification.
Even better:
Free cloud storage that doesn’t require an account and provides no limit to the volume of data stored
Protip: Put swapfile on ramdisk for highest speed
Don’t do boy zram dirty, it has a ton of utility when you have ample spare compute and limited RAM.
Doesn’t it compress the contents that it’s storing to help kind of get the best of both worlds?
You get faster storage because it’s in ram still, but with it being compressed there’s also “more” available?
I could be completely mistaken though
You are correct, although zram uses more cpu power since it compresses things. It’s not really an issue if you’re not using a potato :=)
Hopefully that swap is on an SSD, otherwise that query may not ever finish lol
Once you’re deep into swap, things can get so slow that there’s no recovering from it.
WHAT FUCKING QUERY ARE YOU RUNNING TO USE UP THAT MUCH MEMORY DAMN
In a database course I took, the teacher told a story about a company that would take three days to insert a single order. Thing was, they were the sort of company that took in one or two orders every year. When it’s your whole revenue on the line, you want to make sure everything is correct. The relations in that database were checked to hell and back, and they didn’t care if it took a week.
Though that would have been in the 90s, so it’d go a lot faster now.
We have a company like that here somewhere. When they have one job a year, they have to reduce hours, if they have two, they are doing OK, and if they have three, they have to work overtime like mad. Don’t ask me what they are selling, though. It is big, runs on tracks, and fixes roads.
Exactly how I plan to deploy LLMs on my desktop 😹
You should be able to fit a model like LLaMa2 in 64GB RAM, but output will be pretty slow if it’s CPU-only. GPUs are a lot faster but you’d need at least 48GB of VRAM, for example two 3090s.
Amazon had some promotion in the summer and they had a cheap 3060 so I grabbed that and for Stable Diffusion it was more than enough, so I thought oh… I’ll try out llama as well. After 2 days of dicking around, trying to load a whack of models, I spent a couple bucks and spooled up a runpod instance. It was more affordable then I thought, definitely cheaper than buying another video card.
As far as I know, Stable Diffusion is a far smaller model than Llama. The fact that a model as large as LLaMa can even run on consumer hardware is a big achievement.
I need it just for the initial load on transformers based models to then run them in 8 bit. It is ideal for that situation
