MudMan

A quick look at US Amazon spits out that the only 24Gb card in stock is a 3090 for 1500 USD. A look at the European storefront shows 2400EUR for a 4090. Looking at other assorted stores shows a bunch of out of stock notices.

It’s quite competitive, I’m afraid. Things are very stupid at this point and for obvious reasons seem poised to get even dumber.

Yeah, for sure. That I was aware of.

We were focusing on the Mini instead because… well, if the OP is fretting about going for a big GPU I’m assuming we’re talking user-level costs here. The Mini’s reputation comes from starting at 600 bucks for 16 gigs of fast shared RAM, which is competitive with consumer GPUs as a standalone system. I wanted to correct the record about the 24Gig starter speccing up to 64 because the 64 gig one is still in the 2K range, which is lower than the realistic market prices of 4090s and 5090s, so if my priority was running LLMs there would be some thinking to do about which option makes most sense in the 500-2K price range.

I am much less aware of larger options and their relative cost to performance because… well, I may not hate LLMs as much as is popular around the Internet, but I’m no roaming cryptobro, either, and I assume neither is anybody else in this conversation.

You didn’t, I did. The starting models cap at 24, but you can spec up the biggest one up to 64GB. I should have clicked through to the customization page before reporting what was available.

That is still cheaper than a 5090, so it’s not that clear cut. I think it depends on what you’re trying to set up and how much money you’re willing to burn. Sometimes literally, the Mac will also be more power efficient than a honker of an Nvidia 90 class card.

Honestly, all I have for recommendations is that I’d rather scale up than down. I mean, unless you also want to play kickass games at insane framerates with path tracing or something. Then go nuts with your big boy GPUs, who cares.

But for LLM stuff strictly I’d start by repurposing what I have around, hitting a speed limit and then scaling up to maybe something with a lot of shared RAM (including a Mac Mini if you’re into those) and keep rinsing and repeating. I don’t know that I personally am in the market for AI-specific muti-thousand APUs with a hundred plus gigs of RAM yet.

Thing is, you can trade off speed for quality. For coding support you can settle for Llama 3.2 or a smaller deepseek-r1 and still get most of what you need on a smaller GPU, then scale up to a bigger model that will run slower if you need something cleaner. I’ve had a small laptop with 16 GB of total memory and a 4060 mobile serving as a makeshift home server with a LLM and a few other things and… well, it’s not instant, but I can get the sort of thing you need out of it.

Sure, if I’m digging in and want something faster I can run something else in my bigger PC GPU, but a lot of the time I don’t have to.

Like I said below, though, I’m in the process of trying to move that to an Arc A770 with 16 GB of VRAM that I had just lying around because I saw it on sale for a couple hundred bucks and I needed a temporary GPU replacement for a smaller PC. I’ve tried running LLMs on it before and it’s not… super fast, but it’ll do what you want for 14B models just fine. That’s going to be your sweet spot on home GPUs anyway, anything larger than 16GB and you’re talking 3090, 4090 or 5090, pretty much exclusively.

This is… mostly right, but I have to say, macs with 16 gigs of shared memory aren’t all that, you can get many other alternatives with similar memory distributions, although not as fast.

A bunch of vendors are starting to lean on this by providing small, weaker PCs with a BIG cache of shared RAM. That new Framework desktop with an AMD APU specs up to 128 GB of shared memory, while the mac minis everybody is hyping up for this cap at 24 GB instead.

I’d strongly recommend starting with a mid-sized GPU on a desktop PC. Intel ships the A770 with 16GB of RAM and the B580 with 12 and they’re both dirt cheap. You can still get a 3060 with 12 GB for similar prices, too. I’m not sure how they benchmark relative to each other on LLM tasks, but I’m sure one can look it up. Cheap as the entry level mac mini is, all of those are cheaper if you already have a PC up and running, and the total amount of dedicated RAM you get is very comparable.

The culture goop in the splash zone is.

It’s extra not fun when you’re not in the US, April 1st isn’t a particularly relevant date and you still have to endure all the “fun” bleeding into real world news because you’re in the cultural imperialism splash zone.

I think that’s fair. And for a whole bunch of device types, it’s obviously the default (and often only) option. I think Linux users get too stuck on desktop usage viability. Linux is best when tuned for specific hardware and function, not trying to be everything for everybody at the user level.

Hey, at least you’re honest about it.

I don’t shill for software, man. Not for free, anyway.

But, you know, I talk to enough people about tech stuff to know that Linux getting name dropped generates at most some brief flicker of recognition in like 95% of adults, not some half-remembered decades-old stereotypes. There just isn’t enough awareness to support misconception here. And some of the misconception isn’t that “mis” in the first place, for the standards of non-technical normies.

FWIW, I’d love a free, usable mainstream OS alternative to Apple and Microsoft. I don’t think Linux as currently designed is built to be that effectively, but it’d sure be nice if somebody figured it out. Someone that isn’t Google trying to open yet another revenue stream for ads.

Man, scale is such a hard thing to get intuitively.

I mean, yeah, Linus Sebastian has a huge following. It’s a huge following of self-selected nerds, though. Most people have no idea who he is. Wouldn’t even know what he’s talking about if you showed it to them.

And that was one thing that he did once. That mostly nobody cared about unless they are an active Linux fan. Which is itself a tiny niche.

Humans just have a hard time parsing when things are big or small, particularly if it’s things they are a part of. This is not stupidity, it’s just how human perception works. It works both ways, too. A lot of mondern media is about having these parasocial relationships with huge media personalities and thinking you’ve found some hidden gem only to find out that your grandma follows them already.

It’s not that we’re dumb as a species, it’s that we’ve created this ecosystem built specifically to exploit human perceptual limits for profit and now it’s all we have. It kinda sucks.

Sorry, I went places there, but this whole thread (and honestly, the entire Lemmy linux community) makes me think about this constantly.

I don’t think they expect gaming on Linux to be anything. I don’t think they consider gaming on Linux at all.

Here’s my alternative proposal for their thought process:

“Linux?”

Alternately “I wonder what’s for dinner”.

People really overestimate how top of mind Linux is for the average Windows user.

Wait, which of these Windows users are aware enough of Linux to have a formed expectation of Steam working under it or not? Like, they don’t know enough to understand what it is, but they’re super concerned that Steam, of all things, won’t run on it?

Are these Very Real Windows Users from Canada? Did they approach the poster with tears in their eyes?

Have you tried acting egregiously WITH strategy? Or is this a case of “I’ve tried nothing and I’m all out of ideas”?

You could ABSOLUTELY have voted your way out. Others have.

You just didn’t.

And now you have to dig your way out through other ways. The article does contain some interesting examples of how the fascist right seized power in accessible, local platforms. Granted, those will be much harder under the fascist regime you all enabled, but at least it’s more practical than “we are cooked”.

Cool.

So?

I mean, you are assuming “decentralized” is good, but it’s only as good as what it gets you. On paper, and until proven otherwise, I may choose less decentralized and more “capable of proper, effective moderation” instead. Especially if “less decentralized” is actually “somewhat decentralized”. I haven’t seen a case that fundamental decentralization trumps all so far.

See, but as I was saying above about the privacy stuff, the perception is supposed to be that this is somehow “the alogrithm’s fault” or caused on purpose by corporate media to boost engagement.

Even your take is letting Fedi design off the hook, IMO. The answer here isn’t “oh, well, what can you do?” it’s designing proper moderation tools.

I know people get mad when you praise Bluesky around these parts, but they have an actually good block system, compared to Masto, Lemmy and Fedi in general. It really helps cut this crap short.

Well, where are you all when the Fedi cheerleading squad keeps posting about how bad it is that this or that competitor stores this or that information and how secure and private and great it is in Fedi servers because they don’t store anything?

Because I’ve spent years chiming in to explain these things in those and it normally just gets people angry and complaining that you’re shilling for corporate social media or whatever. The image being projected, both accidentally and on purpose is that no centralized data collection means your data on Fedi is private when it is extremely not.

Did I ask anybody to work with me/us?

See, that “oh, libs are discouraging the left from being excited to support them” crap was all well and good when people were being entitled and disingenuous about domestic politics. It may well have ended US democracy, but hey, it was their prerogative.

But nobody is asking anymore. The US is now a hostile force threatening to invade multiple countries, waging an active global trade war and upending international relations. Nobody in their right mind gives a crap about whether the US populace “wants to work” with anybody else.

I mean, if they want to, great. If they ever get another shot at electing their leaders I would genuinely encourage not electing overt fascists to run every branch of government next time. Good luck with that.

But until then? Again, this is not about you anymore.

I don’t, with all due respect, care how depressed you are. At least when it comes to this conversation.

At this point in time you are one of two things: an active oppositor, and then you have my support, or not that, and then you’re part of the problem.

If you feel like there’s no solution and it’s all a bummer and you’re just going to sit there “making the best of life that you can”, then you’re part of the regime. Authoritarian regimes don’t just run on militants. Granted, nobody has the obligation to be a hero or a martyr, but it’s also true that nobody else has the obligation to prioritize the anonymous foreign masses that keep an antagonistic regime running over their own interests.

Which is a long-winded way to say this is very much, extremely not about you anymore, as far as the rest of us are concerned. That ship sailed when you allowed these idiots to win the election.