Find business ideas backed by real demand.

Running Qwen 3.8 next on 16vram+32ram - A useful/fun post for the gpu poors Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community. I have yet to benchmark this specific REAP version versus Qwen 3.8 27B QK4, but my assumption is that it will do much better, despite the hemorrhaged world knowledge. # System Specs * **GPU:** NVIDIA GeForce RTX 5060 Ti (16 GB VRAM) * **CPU:** AMD Ryzen 7 7840HS (8 cores / 16 threads) * **RAM:** 32 GB DDR5 (\~30 GB OS-visible) * **iGPU:** AMD Radeon 780M (RDNA3) * **Swap:** 8 GB zram As you can see, we have about 44.5 GB of actually addressable system and VRAM available. The iGPU is taking care of the OS to make sure the GPU is totally free—but still, this is barely enough to hold everything together. This config actually totally fails with any of the Unsloth quants—no, I needed something more aggressive. I found the perfect thing—this REAP: [https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF](https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF) What's so great is the total size—a cool \~68.95 GB. The couple of Gigs we have shaved are absolutely key for making this all work. # Model Weight Breakdown Here is the breakdown of the model weights. We have the famous new n-gram portion, the experts, the active layers, the attention/SSM layers, and the extra space needed for the KV cache: |Component|Weight (Approx)|Notes| |:-|:-|:-| |**N-gram / PLE Embedding**|\~29.48 GB|The massive lookup table| |**MoE Routed Experts (320)**|\~34.89 GB|The main expert slab (pruned from 512)| |**Attention / SSM / Router**|\~4.33 GB|Core architecture weights| |**KV Cache**|\[TBD\]|Context memory overhead| Obviously, running this model over SSD would make the speeds notoriously bad. Turning on `mmap` means that `llama.cpp` won't actually try to keep the model in RAM at all (it ...

Luna: That sounds incredibly capital intensive though. How are they funding the fab costs? Lucas: By leveraging the 'distillation' angle that Y Combinator’s Garry Tan has been pushing hard lately. The thesis is simple: if you can compress a massive frontier model into a smaller, efficient open-weight version that runs on custom hardware, your margin profile looks nothing like an API reseller. You look like a hardware company with high margins. Luna: So essentially they are skipping the training race entirely and winning on inference efficiency. Lucas: Exactly. Training is a winner-take-all game where you need fifty thousand H100s sitting in a warehouse.

It's also about the complexity of forecasting your cloud spend. Now you have to estimate per-gigabyte processing, which is variable. That makes budgeting harder. Luna: And with the economy the way it is in 2026, every line item is under scrutiny. Lucas: Absolutely. We're seeing more and more companies adopting FinOps practices precisely because of these kinds of changes. They need visibility into where their money is going, and granular billing forces that. Luna: So what's the takeaway for our listeners? Should they be worried about this? Lucas: I wouldn't say worried, but aware. If you're running a NAT gateway, check your bill this month and see if there's a new line item.

Right. And attackers are going to use, the local models are going to get better and they're going to use them to find vulnerabilities. And so, we know this is coming. You know, the hard part is what do we do about it.