BeefAndPoultry@lemmus.org to LocalLLaMA@sh.itjust.worksEnglish · 13 days agounsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Facehuggingface.coexternal-linkmessage-square8fedilinkarrow-up125arrow-down11file-text
arrow-up124arrow-down1external-linkunsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Facehuggingface.coBeefAndPoultry@lemmus.org to LocalLLaMA@sh.itjust.worksEnglish · 13 days agomessage-square8fedilinkfile-text
minus-squarehummingbird@lemmy.worldlinkfedilinkEnglisharrow-up6·12 days agoHuh the unsloth Q4_K_XL variant running with 5-6 t/s on a 16GB 9070 XT, 128 GB RAM and an NVME ssd. This is shockingly usable 🤯
minus-squareBeefAndPoultry@lemmus.orgOPlinkfedilinkEnglisharrow-up3·12 days agomake sure that holds up with large context, you might need to step down to Q3 (which I’ve heard is still good for this model, many people are even using IQ2)
minus-squareDacoTaco@lemmy.worldlinkfedilinkEnglisharrow-up3·12 days agoDid some more testing on it? This isnt a small model and i havent found the space to save it on my machines yet haha
Huh the unsloth Q4_K_XL variant running with 5-6 t/s on a 16GB 9070 XT, 128 GB RAM and an NVME ssd. This is shockingly usable 🤯
make sure that holds up with large context, you might need to step down to Q3 (which I’ve heard is still good for this model, many people are even using IQ2)
Did some more testing on it? This isnt a small model and i havent found the space to save it on my machines yet haha