Rendered at 06:39:17 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dghlsakjg 3 hours ago [-]
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
arjie 2 hours ago [-]
Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.
apimade 1 hours ago [-]
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today.
Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
xbmcuser 2 minutes ago [-]
We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
cududa 15 minutes ago [-]
Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
hedora 2 hours ago [-]
AMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.
gpt5 17 minutes ago [-]
Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
AHASIC 3 hours ago [-]
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
greggsy 59 minutes ago [-]
The innovation curve doesn’t always reward the innovators.
It pays to let others test the waters before you.
Mistletoe 2 hours ago [-]
What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
CircuitSeuss 2 hours ago [-]
A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
I could definitely image Apple embedding a kind of LLM-optimized FPGA: slow to load (update) an LLM, but blazing fast at computing tokens.
Who needs memory when your model is set in silicon ?
ethersteeds 56 minutes ago [-]
I think a major factor is memory bandwidth. Apple has raised it steadily for each M series generation, and that hasn't plateaued.
Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.
bobbylarrybobby 2 hours ago [-]
The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.
swiftcoder 19 minutes ago [-]
Agreed. We've just seen DeepSeek post-train their ~300 billion parameter flash model to outperform their 1.6 trillion parameter pro model, in the space of a few months. There would seem to still be quite a few opportunities on the table to bring big model smarts down to the smaller models
sudo_cowsay 2 hours ago [-]
It could be on software side too. OpenAI has certainly not plateaued.
adrianco 57 minutes ago [-]
This looks useful, you can increase the RAM cache so if you have a Mac with 24-32GB it should speed up a lot and still run models that wouldn’t normally fit. I’m going to run some tests…
CyLith 2 hours ago [-]
I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
zamadatix 2 hours ago [-]
That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
ianmurrays 2 hours ago [-]
I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.
brrrrrm 3 hours ago [-]
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.
half an hour to process 10k tokens on an M5 seems... not great
kennywinker 3 hours ago [-]
Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well
selcuka 18 minutes ago [-]
I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.
jbird99 3 hours ago [-]
At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
kennywinker 3 hours ago [-]
It says very prominently in the post: 4.5-5t/s for 80b on an M5
wat10000 3 hours ago [-]
Isn’t it only writes that kill drives?
petu 50 minutes ago [-]
There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours.
Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
sudo_cowsay 2 hours ago [-]
Yeah, that's why most of these comments seem weird to me.
Alpha3031 2 hours ago [-]
Yes for NAND, and I suppose nobody is using mechanical hard drives for this.
crossroadsguy 52 minutes ago [-]
I see this at the end of the README
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with <xyz> autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?
iamflimflam1 25 minutes ago [-]
It means they used Claude code to write the software.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
[1] https://youtu.be/x26A28DoT-w?t=605
Who needs memory when your model is set in silicon ?
Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.
half an hour to process 10k tokens on an M5 seems... not great
Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with <xyz> autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?