I wonder how much that high cost could be reduced by modern manufacturing. Same/similar designs, but modern tooling and logistics.

I mean, they did not have CNC mills back then.

Fairly significant factor when building really large systems. If we do the math, there ends up being some relationships between

• disk speed
• targets for ”resilver” time / risk acceptance
• disk size
• failure domain size (how many drives do you have per server)
• network speed

Basically, for a given risk acceptance and total system size there is usually a sweet spot for disk sizes.

Say you want 16TB of usable space, and you want to be able to lose 2 drives from your array (fairly common requirement in small systems), then these are some options:

• 3x16TB triple mirror
• 4x8TB Raid6/RaidZ2
• 6x4TB Raid6/RaidZ2

The more drives you have, the better recovery speed you get and the less usable space you lose to replication. You also get more usable performance with more drives. Additionally, smaller drives are usually cheaper per TB (down to a limit).

This means that 140TB drives become interesting if you are building large storage systems (probably at least a few PB), with low performance requirements (archives), but there we already have tape robots dominating.

The other interesting use case is huge systems, large number of petabytes, up into exabytes. More modern schemes for redundancy and caching mitigate some of the issues described above, but they are usually onlu relevant when building really large systems.

tl;dr: arrays of 6-8 drives at 4-12TB is probably the sweet spot for most data hoarders.

How else would you be webscale?

Well, a few issues:

• For hosting or training large models you want high bandwidth between GPUs. PCIe is too slow, NVLink has literally a magnitude more bandwidth. See what Nvidia is doing with NVLink and AMD is doing with InfinityFabric. Only available if you pay the premium, and if you need the bandwidth, you are most likely happy to pay.
• Same thing as above, but with memory bandwidth. The HBM-chips in a H200 will run in circles around the GDDR-garbage they hand out to the poor people with filthy consumer cards. By the way, your inference and training is most likely bottlenecked by memory bandwidth, not available compute.
• Commercially supported cooling of gaming GPUs in rack servers? Lol. Good luck getting any reputable hardware vendor to sell you that, and definitely not at the power densities you want in a data center.
• TFLOP16 isn’t enough. Look at 4 and 8 bit tensor numbers, that’s where the expensive silicon is used.
• Nvidias licensing agreements basically prohibit gaming cards in servers. No one will sell it to you at any scale.

For fun, home use, research or small time hacking? Sure, buy all the gaming cards you can. If you actually need support and have a commercial use case? Pony up. Either way, benchmark your workload, don’t look at marketing numbers.

Is it a scam? Of course, but you can’t avoid it.

For you? No. For most people? Nope, not even close.

However, it mitigates certain threat vectors both on Windows and Linux, especially when paired with a TPM and disk encryption. Basically, you can no longer (terms and conditions apply) physically unscrew the storage and inject malware and then pop it back in. Nor can you just read data off the drive.

The threat vector is basically ”our employees keep leaving their laptops unattended in public”.

(Does LUKS with a password mitigate most of this? Yes. But normal people can’t be trusted with passwords and need the TPM to do it for them. And that basically requires SecureBoot to do properly.)

This will be so much fun for people with legacy systems

I’ll bite. It’s getting better, but still a long way to go.

• No commercially viable remote desktop or thin client solutions. I’m not talking about just VNC, take a look at for example ThinLinc to see what I’m looking for - a complete solution. (Also, it took like ten rough years before basic unencrypted single user VNC was available at all.) Free multimillion dollar business idea right here folks!
• Related to the above point - software rendered wayland is painful. To experience this yourselves, install any distro in VirtualBox or VMWare or whatever and compare the usability between a Xorg DE (with compositing turned off) and the same Wayland DE. Just look at the click-to-photon latency and weep. I’ve seen X11 perform better with VNC over WAN.
• ”We don’t need network transparency, VNC will save us”. See points above.
• ”Every frame is perfect” went just as well as can be expected, there is a reason VSYNC is an option in games and professional graphics applications. Thanks Valve.
• I’m assuming wlroots still won’t work on Nvidia, and that the Gnome/KDE implementations are still a hodgepodge, and that Nvidia will still ask me to install the supported Xorg drivers. If I’m wrong, it only took a decade or so to get a desktop working on hardware from the dominant GPU vendor. (Tangentially related - historically the only vendor with product lines specifically for serving GPU-accelerated desktops to thin clients)
• After over a decade of struggles, we can finally (mostly) share out screens in Zoom. Or so I’m told.

But what do I know, I’ve only deployed and managed desktop linux for a few thousand people. People were screaming about these design flaws back in 2008 when this all started. The criticisms above were known and dismissed as FUD, and here we are. A few architectural changes back then, and we could have done this migration a decade faster. Just imagine, screen sharing during the pandemic!

As an example, see Arcan, a small research project with an impressively large subset of features from both X11 and Wayland (including working screen sharing, network transparency and a functioning security model). I wouldn’t use it in production, but if it was more than one guy in a basement working on it, it would probably be very usable fairly fast, compared to the decade and half that RedHat and friends have poured into Wayland thus far. Using a good architecture from the start would have done wonders. And Wayland isn’t even close to a good architecture. It’s just what we have to work with now.

Hopefully Xorg can die at some point, a decade or so from now. I’m just glad I don’t work with desktops anymore, the swap to Wayland will be painful for a lot of organisations.

I have a mac I use for some specific tasks. I’ll agree the Apple is, ehh, Apple.

But mounting network fileshares is dead simple. My SMB share pops right up, authentication works fine, the user interface for it is fine. If I wanted to use it remotely, I’d just export it over my tailnet.

’sshfs’ is good for short stints of brief use, but ultimately it breaks on a protocol level as soon as your socket dies, on any OS.

Used to? It’s standard practice like everywhere.

If you can have NAND-gates, a clock and some wires, you can build anything.

Go visit https://nandgame.com/ to try it out yourself!

Apparently AMD couldn’t make the signal integrity work out with socketed RAM. (source: LTT video with Framework CEO)

IMHO: Up until now, using soldered RAM was lazy and cheap bullshit. But I do think we are at the limit of what’s reasonable to do over socketed RAM. In high performance datacenter applications, socketed RAM is on it’s way out (see: MI300A, Grace-{Hopper,Blackwell},Xeon Max), with onboard memory gaining ground. I think we’ll see the same trend on consumer stuff as well. Requirements on memory bandwidth and latency are going up with recent trends like powerful integrated graphics and AI-slop, and socketed RAM simply won’t work.

It’s sad, but in a few generations I think only the lower end consumer CPUs will be possible to use with socketed RAM. I’m betting the high performance consumer CPUs will require not only soldered, but on-board RAM.

Finally, some Grace Hopper to make everyone happy: https://youtube.com/watch?v=gYqF6-h9Cvg

#define yeet throw
#define let const auto
#define mut &
#define skibidi exit(1)

The future is now!