Rendered at 04:45:06 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mips_avatar 11 hours ago [-]
I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.
anthonypasq 8 hours ago [-]
if all the competition pushes down margins on tokens to 10-20%, i dont see how the inherent scale advantages of cloud inference wont be way more to account for the 20% cheaper tokens youd be getting running locally. i dont see how local will ever be more economical
4 hours ago [-]
cyanydeez 11 hours ago [-]
sure, if there werent capitalists digging their moat. in 1 year local inferwnce doubled cost from ram alone because cloud vendors cornered markets on their circular cash flow expectations.
mips_avatar 11 hours ago [-]
Yeah but Samsung/Micron/SK hynix are building new fabs, and Chinese Ram will be able to fully supply the lower end ddr5 ram market within 2 years. The crunch is temporary and on the other side of it there will be incredible hardware at the $10-20k price point.
NuclearPM 9 hours ago [-]
There’s a shitload of money to be made in RAM fabrication. Costs will come down a ton.
Analemma_ 10 hours ago [-]
You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.
mips_avatar 9 hours ago [-]
The marginal cost of making RAM is zero and it's clear to everyone that there's near infinite demand, so the fab capacity is coming. Maybe the high end won't expand fast enough, but you can do a lot with Chinese ddr5.
Legend2440 10 hours ago [-]
DRAM prices have been highly cyclical for a long time. They'll come down again.
draygonia 10 hours ago [-]
From what I'm reading on the shortage, they are very aware of this fact (prices being highly cyclical) and they're trying to avoid overproducing RAM to avoid the oversupply glut. That's probably going to hurt the chance of prices coming down anytime soon.
mips_avatar 9 hours ago [-]
Maybe micron/samsung/sk won't expand enough, but trust me the Chinese will. It might mean we don't get access to HBM for a while, but you can do a lot with infinite cheap DDR5
organsnyder 9 hours ago [-]
Prices will come down when the stupid money isn't all going to datacenter buildouts.
nekusar 12 hours ago [-]
1. Slot machine
You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.
And it gets tantalizingly close, but for 100% gotta pull the arm again!
shshsjsj 9 hours ago [-]
that is way too accurate and ofc hn would downvote something funny and true.
techgnosis 5 hours ago [-]
it just seems off-topic. we see the slot machine analogy a lot but i dont see how it has anything to do with this long, thoughtful essay about pricing power
nekusar 3 hours ago [-]
Pricing in the article is the difference between a $.25 slot machine with a range up to the $10 slot machine.
Its always a probabilistic gamble that you get what you describe. And it's a pull of the lever each time. And you pay no matter what, for good or bad results.
You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.
And it gets tantalizingly close, but for 100% gotta pull the arm again!
Its always a probabilistic gamble that you get what you describe. And it's a pull of the lever each time. And you pay no matter what, for good or bad results.
Its gambling. Either your time or time+money.