LocalLlama [Reddit]
Articles about LocalLlama [Reddit]
Local AI solutions for businesses
Hi everyone, How many of you are building local AI solutions for businesses or other individuals? If you are building AI solutions for businesses, what are the biggest challenges you're facing? And what is driving businesses to decide to host their AI models themselves? submitted by /u/forevergeeks [link] [comments]
Occamy-1.0 by Accio Lab
submitted by /u/No-Name-Person111 [link] [comments]
It's Tuesday already with no new model drops
submitted by /u/Turtlesaur [link] [comments]
A question regarding DS V4.1 flash architecture
I have troubles understanding something and I think a lot are in the situation. Can someone eli5 what does v4.1 flash with its 500B+ weights and biases and its 190B+ engram look up table bring to the table? I understand it has a crazy prefill optimization, I understand it's kv cache is of ridiculous size. But I'm wondering what to do with this engram thing, can we update it for knowledge without fine tuning the model? can we make it grow? What's up with that thing? submitted by /u/No_Afternoon_4260 [link] [comments]
Closed source AI is more dangerous than open source AI
Security through obscurity is no form of security. submitted by /u/keequalshalfmvsqrd [link] [comments]
DeepSeek V4.1F Q4 on M3 Ultra with native DSpark MTP (40tps / 800tps)
I liked DeepSeek V4.1 Flash as an agent model but at 16 t/s on ds4 it was painful to sit through a real turn. So I forked antirez/ds4 and optimized it for V4.1 Flash on a 512 GB M3 Ultra. The screenshot is a real 91 minute agent turn in my UI: 101k tokens decoded, 4.6M tokens prefilled at 99.5% cache hit, 56 tool calls (no mistakes), context growing from 3k to 127k.
Qwen3.8 27B with embeddings
As you know, Qwen3.8 27B has built-in support for embeddings. It gives pretty decent results in llama.cpp with the following parameters:
Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF
it be like that submitted by /u/Every-Walrus [link] [comments]
CrofAI "cheapest inference provider in the world" gets exposed as an OpenRouter wrapper, routing requests to smaller, cheaper models at up to 20x markup. CrofAI responds to Wire Fraud allegations by denying everything, then backtracking, then 3 hours later wiping their entire online presence
Disclaimer: no AI was used whatsoever to write this post Cautionary tale about chasing cheap tokens. exposé: https://kendell.dev/blog/crofaifalse/ reaction by nahcrof, announcing the shutdown of the service: https://x.com/nahcrof/status/2099552389434900643 - now deleted, archive picture: https://i.imgur.com/teOQngH.png NahCrofAI (crof.ai, nahcrof.com) was an inference provider which had all the latest models at the cheapest price, often significantly below the lowest alternative on OpenRouter.
Best hardware for qwen 3.8
So I would like to run qwen 3.8 27b locally for my ai agents, maybe even in parallel with other small LLMs (such as qwen3.5 9b, oss 20b etc..). What is the best hardware to do this? Not a video card but I mean as “mini pc ai”. Thank you 🙏 submitted by /u/tombino104 [link] [comments]
Jinfer: An open-source AI inference engine for the JVM. Finally, AI in jar
For years, the JVM has watched the AI revolution from the bench. Every model, AI framework, every breakthrough, built with/for Python.
Voodoo Dynamic Quant - Now MIT Licensed
Two months ago I announced I had found a new dynamic quant method called Voodoo Quant which was SOTA for the most aggressive quant levels on some smaller Qwen3.5 GGUF models. I kept the methodology private at the time, but I've seen too many requests for dyn quants for various models lately, so I decided to give my method to the community since I don't have the time to scale this into something that could do it justice. Hopefully it will also inspire some researchers to find out more about it and improve it as I am just scratching the surface.