Release notes from llama.cpp
Articles about Release notes from llama.cpp
B11479: cuda: update uncoalesced memory reads in pool2d (#29425)
cuda: update uncoalesced memory reads in pool2d cuda: Added perf and restrict in pool2d Update POOL2D_WARP_KERNEL_MIN_WINDOW macro Co-authored-by: Georgi Gerganov ggerganov@gmail.com cuda: Fix compiler bugs Co-authored-by: Georgi Gerganov ggerganov@gmail.com
B11477: llama: share the nextn tensor flags between models (#30097)
llama: share the nextn tensor flags between models Follow-up of the TODO in glm5-next: move the trunk-only and MTP-only detection that each model copied into a nextn_flags helper of llama_model_base. It probes the first trunk layer and the first NextN layer, and adds TENSOR_SKIP when MTP is not loaded. qwen4exp probes hc_attn_norm since it has no attn_norm. deepseek4, nemotron-h, qwen35, qwen35moe, qwen3next and qwen4exp now also accept a trunk-only file, like the other models.
Release b11476 · ggml-org/llama.cpp
The generic few-row MMA kernel works for any type with a 16-weight dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and the IQ types. Each type starts at the row count where it beats the current kernels on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3 for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the others. test-backend-ops perf -o MUL_MAT, m=4096, k=14336, M3 Ultra, time of this change over master (mean of two interleaved runs each): 0.23 to 0.98 from
Release b11475 · ggml-org/llama.cpp
metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT
Release b11471 · ggml-org/llama.cpp
vocab : add plamo fim tokens (#30090) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/53567889 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11474 · ggml-org/llama.cpp
llama : add GLM5-Next NextN (MTP) graph Build the GLM5-Next multi-token-prediction head as graph_mtp: the NextN block embeds enorm(tok)+hnorm(h) through eh_proj, runs one plain DSA layer and the shared lm_head, reusing the trunk's builders through the no_build tag ctor. llama_memory_recurrent also tolerates a partial seq_rm when the context holds no recurrent layers, which is what the MTP draft context needs. Assisted-by: Claude llama : glm5-next: skip dead compute in headless NextN forwards
Release b11472 · ggml-org/llama.cpp
sampling : use greedy selection for eligible temperature-zero chains (#29797) sampling : use greedy selection for eligible temperature-zero chains Assisted-by: OpenAI Codex Apply suggestion from
Release b11469 · ggml-org/llama.cpp
musa: use the tile lightning indexer kernel (#30080) Signed-off-by: Xiaodong Ye xiaodong.ye@mthreads.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestati
Release b11468 · ggml-org/llama.cpp
vendor : update cpp-httplib to 0.60.0 (#30081)
B11467: imatrix : include clocale for std::setlocale (#30079)
Co-authored-by: yarik dazzywi@github.com
Release b11466 · ggml-org/llama.cpp
ggml-webgpu: fix flash_attn supports_op check for overlapping KV (#28
Release b11465 · ggml-org/llama.cpp
tests : retain the anchor when testing recurrent rollback (#29923)
B11464: [SYCL] fix the issue in mixed different model GPUs in FA (#29071)
fix mixed different model GPUs issue Update ggml/src/ggml-sycl/ggml-sycl.cpp Co-authored-by: Georgi Gerganov ggerganov@gmail.com Co-authored-by: Georgi Gerganov ggerganov@gmail.com
Release b11463 · ggml-org/llama.cpp
sycl: accelerate GLM MLA prefill with MKL flash attention GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape because the normal MKL flash-attention gate requires matching K/V widths and caps the head dimension at 512, so prompt processing falls back to the substantially slower TILE kernel. Admit only the validated 576/576/512, GQA-20 F16 shape to the existing MKL pipeline. Keep all other mismatched K/V shapes on their current fallback paths. Handle GLM'
Release b11462 · ggml-org/llama.cpp
sycl : fattn_kv_buffers cleanup (#27689) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/53521541 macOS/iOS: macOS Apple Silicon (arm64) macOS Appl
Release b11461 · ggml-org/llama.cpp
vulkan: fix amd iGPU slow checkpoint read (#30049) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/53465294 macOS/iOS: macOS Apple Silicon (arm64)
Release b11460 · ggml-org/llama.cpp
sycl: add IQ3_S multi-column MMVQ (#29500) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/53455527 macOS/iOS: macOS Apple Silicon (arm64) macOS Ap
Release b11459 · ggml-org/llama.cpp
ggml-webgpu: no dawn native features on wasi (#27069)
Backup/allozaur-ui-shell-polish: ui : polish the shell
The secondary button gets its own look back and the chat add button its own light surface. Pressable elements get a pointer cursor again, the font rendering smooths on the app shell, the root layout resolves its props probe through the providers' api url, and the agent skills stay out of prettier's way. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
Backup/allozaur-ui-models-manager: test : give the switch test an open chat
The pane offers "Use in this chat" only when a chat is open, so without one the test looked for a button that was never there. Assisted-by: pi (zai-org/GLM-5.3-Flash)
B11435: llama: fix k-pool scatter data race on shared sequences (#29994)
llama: re-pool each shared k-pool rep once With shared cells every pool is re-pooled, and since the pooled keys are always scattered, the pools a seq_cp shares between sequences wrote the same rep row from several scatter entries, a data race on the CPU backend. Mark each rep once: the sharing sequences read the same row through pool_cells. llama: assert whole-sequence seq_cp in the hybrid idx memory The recurrent state is always copied whole whatever the range, and a
B11434: hexagon: ssm-conv updates (#29971)
hexagon: ssm-conv double-buffered DMA for prefill and decode restructuring hex-ssm-conv: remove divs from loops and fix trace events hex-dma: improved SSM_CONV dma pipeline and streamlined dma_queue Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com
Release b11433 · ggml-org/llama.cpp
hexagon: add pool_2d support hexagon: add pool_1d support hex-pool: dma changes hex-pool: Optimize HTP pooling boundaries and DMA pipelining hex-pool: code cleanup and correctness fixes hex-pool: re-write the DMA pipeline hex-pool: pool chunking support hex-pool: remove/vectorize all scalar paths hex-pool: simplify chunk solver (no need for a loop) hex-pool: remove redundant checks Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com
Release v0.6.0 · ggml-org/llama.cpp
Overview llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API (with llama_process) for mixed token/embedding inputs and MTP/deepstack state embeddings, adds support for the GLM-5.3-Flash (GLM5-Next) 320B hybrid model, the Clef decision model (text and vision) and MTP speculative decoding for Qwen4Exp, ships a new /v1/systemone server API for decision models (laya, julia-1, lev, openjev, kev, nimble), overhauls the Web UI with a Hugging Face Hub data layer and model download pipeline,
Release b11429 · ggml-org/llama.cpp
llama.cpp : bump version to 0.6.0 (#29997) llama.cpp : bump version to 0.6.0 scripts : update summary prompt (#0) Website: https://llama.app Attestations: https://github.com/ggml-org/llam
Release b11425 · ggml-org/llama.cpp
CUDA: make the alloc_deps check batch independent (#29986) Fixes #29980 Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52874307 macOS/iOS: macOS A
Release b11424 · ggml-org/llama.cpp
vulkan: fix Flash Attention shmem write out of bounds (#29988)
B11422: cuda: use the vector lightning indexer kernel on MUSA (#29990)
cuda: stage the lightning indexer queries in head passes for MUSA MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile kernel staged the queries of all four heads next to the key tile for 33 KB. The queries are now staged in passes of LIGHTNING_INDEXER_TILE_HEADS_PER_PASS heads: two on MUSA for 25 KB, four elsewhere where the single pass folds to the previous kernel. cuda: use the vector lightning indexer kernel on MUSA Address review from am17an: the tile kernel stays off MUSA, whose archs
B11415: server: reject partial media truncation (#24076)
server: reject partial media truncation server: keep only the keep_first fix Drop the mtmd test helper change, which no longer builds since clip_image_f32_batch stores its entries by value, and drop the vision test: no test fixture reaches a cut between two adjacent media chunks with a reused cache (tinygemma3 uses SWA and wraps images in text tokens, tinyopenjev and small-test are recurrent), so the test passed or failed independently of the fix. Co-authored-by: Pascal admin@serveurperso.com
Release b11414 · ggml-org/llama.cpp
vulkan: fix stale prealloc_y reuse across flash attention and soft_ma
Release b11413 · ggml-org/llama.cpp
vulkan: sparse flash attention for quantized K/V Assisted-by: Claude vulkan: single-scan sparse FA index compaction The compaction ran one workgroup per mask row and walked the row in BLOCK_SIZE chunks, with a workgroup scan per chunk. For decode that is one workgroup doing KV/1024 barrier-bound iterations, so at 128k cells it cost more than the sparse attention it feeds. Split the row into contiguous segments instead: one per subgroup with ballot counting over coalesced loads, or one per thread without subgroups.
B11412: llama : fix unexpected graph reallocation in the k-pool models (#29958)
llama : fix unexpected graph reallocation in the k-pool models Both k-pool models built a graph shape that depends on state the full-context reserve cannot know: qwen4exp branched on inp->cache_safe, which turns false as soon as llama_memory_seq_cp shares cells (e.g. batched-bench -pps): the QSA layers swapped scatter+gather for fill+concat and dropped the new_pool_rep leaf, so the decode graph had 12 fewer nodes than the reserved one glm5-next branched on gather = n_tokens n_sel, so the
Release b11411 · ggml-org/llama.cpp
kv-cache: fix restoring mismatched KV cache rotation by saving exact rotation metadata (#28498) kv-cache: save exact KV rotation metadata, reject restoring mismatched rotation tests : move the
Release b11410 · ggml-org/llama.cpp
ci : winget urls must be separate strings (#29978)
Release b11405 · ggml-org/llama.cpp
cuda: tile the lightning indexer over keys and tokens for 4 heads (#29901) cuda: tile the lightning indexer over keys and tokens for 4 heads With too few heads for a wmma tile, a block scores 64 keys against 8 tokens: the keys are staged once in half precision, the queries one head at a time, and each thread owns one key for two tokens, so no dot product needs a cross thread reduction. Batches smaller than a token tile keep the vector kernel. test-backend-ops measures 4 heads. cuda: multiply the lightning indexer tile in float
Release b11404 · ggml-org/llama.cpp
metal : few-row MMA mat-mul and batched copies for speculative decoding Speculative decoding verifies a few draft tokens per step. Without the tensor API, Metal ran these mat-muls with the mat-vec kernels, whose time grows with every src1 row, so DFlash2 decoding on an M3 Ultra was slower than serial decoding.
Release b11403 · ggml-org/llama.cpp
CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size Signed-off-by: ynankani ynankani@nvidia.com adjust kernel selection logic Signed-off-by: ynankani ynankani@nvidia.com Co-authored-by: Johannes Gäßler johannesg@5d6.de
Release b11402 · ggml-org/llama.cpp
CUDA: prefer whole-tile FlashAttention scheduling for efficient two-stage kernels (#29435) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52707115
Release b11401 · ggml-org/llama.cpp
log, server: self contained colors, split child commands from logs in
Release b11400 · ggml-org/llama.cpp
llama: support both embd + raw tokens in batch add to test-llama-archs also check case llm_arch_supports_mixed_batch = false constant graph topology have dedicated input for mixed case rm set_tensor_backend is_embd --> type consolidate m-rope pos handling into one place nits
Release b11399 · ggml-org/llama.cpp
CUDA: refactor swizzling code (#29612) CUDA: refactor swizzling code fix templates/loop bounds Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/
B11398: ggml-cpu: support BF16/FP16/FP32 K tails in tinyBLAS on x86 (#29806)
ggml-cpu: vectorize BF16 K tails in tinyBLAS tests: Skip tinyBLAS when use_ref is enabled so CPU tests compare against the vec_dot path. ggml-cpu: vectorize tinyBLAS F16/F32 tails
B11397: cuda : move neu_padded to where it is used (#29940)
Signed-off-by: Adrien Gallouët angt@huggingface.co
Release b11396 · ggml-org/llama.cpp
ci : windows llvm build requires ninja multi-config (#29959)
B11395: ci : add windows arm64 vulkan release (#29954)
add windows vulkan arm64 release add link
Release b11393 · ggml-org/llama.cpp
chat-peg-parser : clear current_tool when pending_tool_call is reset
Release b11392 · ggml-org/llama.cpp
ci : set default permissions (#29945)
B11391: cuda : move blocks_per_col to where it is used (#29939)
Signed-off-by: Adrien Gallouët angt@huggingface.co
Release b11390 · ggml-org/llama.cpp
CUDA: fix MMQ memory fault if n_expert >> n_ubatch (#29941)
Release b11389 · ggml-org/llama.cpp
vulkan: fix rdna4 mat_vec tuning (#29934)
Release b11388 · ggml-org/llama.cpp
imatrix: calculate activation-based statistics for new format (GGUF)
B11387: spec : fix n-gram drafts rejected at temp > 0 after truncation (#29924)
Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com
Release b11386 · ggml-org/llama.cpp
common : prepare load_from_models_dir() for path conversion (#29674) This is part of the fs::path modernization series. That was also the opportunity to remove fs_list(). Signed-off-by: Adrien Gall
Release b11385 · ggml-org/llama.cpp
server : fix dead LLAMA_ARG_HF_REPO_FILE key in preset allow-list (#29938) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-or
Release b11384 · ggml-org/llama.cpp
ci : pushing tag needs deploy key (#29937)
Release b11382 · ggml-org/llama.cpp
webgpu: add f16 support to fill/set_rows (#29897) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52497832 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11381 · ggml-org/llama.cpp
mtmd : fix deprecated strdup warning on Windows (#29863) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52479259 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11380 · ggml-org/llama.cpp
vendor : update cpp-httplib to 0.59.0 (#29886) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52476117 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11379 · ggml-org/llama.cpp
server : fix laya abort by limiting n_batch to n_ubatch (#29903) server : fix laya abort by limiting n_batch to n_ubatch Fixes #29902 Assisted-by: Claude fix(review) : rm tests, embeddings cond Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52435853 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)
Release b11378 · ggml-org/llama.cpp
common : add common_is_tty() helper and fix deprecated warnings on Windows (#29860) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52429962 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11377 · ggml-org/llama.cpp
chat : honor json_schema in Ling 3.0 parser (#29813) chat : honor json_schema in Ling 3.0 parser Ling 3.0 only built a grammar for tool calls and did not handle inputs.json_schema, so response_format requests were left unconstrained. Add an eager response-format grammar path with precedence over tools, following the existing parser patterns. Require before JSON when thinking is enabled and do not allow trailing prose after the JSON response.
Release b11376 · ggml-org/llama.cpp
ci: fix flaky ADD_ADD f16 by using the fused ADD tolerance (#29904) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52418791 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11375 · ggml-org/llama.cpp
graph: gather the recurrent states once so the reserve covers every split (#29856) build_rs gathered the extra states (n_rs - n_seqs rows) with their own get_rows. The worst-case reserve has n_rs == n_seqs, so that node was sized at zero rows, and any ubatch whose cells are not contiguous forced a graph reallocation at an unchanged node count, which aborts under GGML_SCHED_NO_REALLOC. A single get_rows now gathers the n_rs states: the ubatch states and the extra states are views of it, and its size only depends on n_rs, which
Release b11374 · ggml-org/llama.cpp
ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. (#29852) ggml-openvino : Qwen3.5 MoE perf (#312) Squash of ravi9#312: ggml-openvino: add detailed inference profiling (Yu, Zijun) ggml-openvino: use remote output tensors by default (Yu, Zijun) ggml-openvino: optimize single-sequence recurrent state (Yu, Zijun) opt1: remove recurrent reset for single sequence, opt2: direct gdn outputs (break parallel sequence) (Yu, Zijun) fix parallel sequences (Yu, Zijun)
Release b11372 · ggml-org/llama.cpp
qwen4exp : halve the indexer score memory (#29825) qwen4exp : halve the indexer score memory The indexer scored all heads in one product and rectified a copy of it, so two [n_pool, n_idx_h, n_tokens] f32 tensors were live at once, the largest buffers of the graph at long context. Each head now gets its own product, rectified and summed in place into one [n_pool, n_tokens] score. qwen4exp: let the allocator reuse the indexer score buffers Address review from CISC: use plain ggml_add and ggml_relu in the indexer head loop.
Release b11370 · ggml-org/llama.cpp
CUDA: fuse shared experts into MMVQ (#29184) CUDA: fuse shared experts into MMVQ check if buffer is null move stride_col_dst to fusion args Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52371643 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11368 · ggml-org/llama.cpp
spec : add probabilistic sampling for simple draft and MTP (#27694) Make the drafter probabilistic and the target verify by rejection sampling Drop stale spec_draft_q before drafting Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy. Support grammar-constrained requests in rejection sampling Fix - renormalize distribution after masking copy rng on sampler copy and re-accept drafted tokens on replay
Release b11366 · ggml-org/llama.cpp
ggml-quants : avoid invalid rounding in qkx3 scale search (#29817) ggml-quants : avoid invalid rounding in qkx3 scale search The imatrix scale search can produce an infinite, NaN, or otherwise out-of-range value when the fitted minimum collapses to the maximum or makes the range extremely small. That value is then passed to nearest_int and can trip its assertion in Debug builds.
Release b11365 · ggml-org/llama.cpp
ggml-cpu : fix soft_max_back wrong output when dst aliases src1 (#27096) ggml-cpu : fix soft_max_back wrong output when dst aliases src1 GGML_OP_SOFT_MAX_BACK is listed in ggml_op_can_inplace, so the graph allocator may assign dst to alias either src0 (dy) or src1 (y). The result was built in several steps: When dst aliases src1, the first step overwrites y and the third step then reads the overwritten values, so the output is silently wrong. Aliasing dst with src0 is unaffected. The CUDA kernel completes its
Release b11364 · ggml-org/llama.cpp
model: support nimble decision model (#29844) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52332773 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11371 · ggml-org/llama.cpp
init support for clef (text only) more static graph clean up nits nits 2 Update gguf-py/gguf/constants.py Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co
Release b11362 · ggml-org/llama.cpp
metal : add tensor API flash attention kernel for F16 KV (#29570) metal : add tensor API flash attention kernel for F16 KV cont : add tensor FA kernels for DK=DV=512 and DK=576, DV=512 cont : support attention sinks, ALiBi and logit softcap in the tensor FA kernel cont : add tensor FA kernel for DK=192, DV=128 Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52326292 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11361 · ggml-org/llama.cpp
llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) (#29818) init conversion convert: ok model loaded add server code improve conversion script support shared prompt prefix add docs, imorove UX a bit add vision support add openjev tiny model for testing add dev docs support lev & kev clean up fix lev noul fix py lint nits docs clarify about not supporting date_facts Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52320438 macOS/iOS:
Release b11355 · ggml-org/llama.cpp
vulkan: disable large matmul tile on Samsung GPUs with 32KB shared memory (#28531) Assisted-by: Claude Opus Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52313922 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11352 · ggml-org/llama.cpp
qwen4exp : optimize mask constructions (#29824) qwen4exp : optimize mask constructions cont : apply the same change for GLM5-next Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52303821 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11351 · ggml-org/llama.cpp
ggml : add alloc_buffer_n to buffer type interface (#23671) ggml : add alloc_buffer_n to buffer type interface Add alloc_buffer_n method to ggml_backend_buffer_type_i interface, with a public API ggml_backend_buft_alloc_buffer_n. Default implementation in ggml-backend.cpp handles multi-buffer splitting and tensor allocation via ggml_tallocr Meta buffer type provides custom implementation that creates per-device sub-contexts and delegates to simple buffer types ggml_backend_alloc_ctx_tensors_from_buft now collects tensors
Release b11349 · ggml-org/llama.cpp
vulkan: add logging to pipeline compile issues (#29794) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52203370 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11346 · ggml-org/llama.cpp
qwen4exp: fix tests (#29819) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52169567 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11345 · ggml-org/llama.cpp
hexagon: add q2_k and q3_k quant type support (#29717) hexagon: add q2_k and q3_k quant type support hex-qk: consistent allocation of src1_row_size Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52157708 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11344 · ggml-org/llama.cpp
CUDA: fix 2 broken Volta FA cases (#29803) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52093363 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11342 · ggml-org/llama.cpp
common,rpc : fix cache dir creation through symlinks on buggy libstdc++ (#29816) See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101510 Close #29759 Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52050509 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)
Release b11339 · ggml-org/llama.cpp
llama : clamp kpool re-pool bound to existing pools (#29805) tests : simplify function signature llama : clamp kpool re-pool bound to existing pools The n_tokens/kpool + n_seqs_unq bound on n_new_g overshoots when a batch fills the whole cache: n_ctx tokens complete exactly n_ctx/kpool pools, so the +1 pads new_pool_idxs/new_pool_rep one entry past n_pool_real. Graph reserve only covers n_pool_real entries, so the first full-context decode builds bigger tensors than reserved and ggml-alloc demands a graph
Release b11338 · ggml-org/llama.cpp
hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA (#29685) hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA hex-cpy: various fixes on top of the concat optimizations Removed CONCAT_DMA_MIN_ROW logic, it was broken with 64-bit DMA. While it's kinda silly to use DMA for tiny stuff if that tensor gets mapped to an extended buffer the only way to read it is DMA. Added missing dma_queue_flush() calls. Added additional guards for conditions we don't support.
Release b11337 · ggml-org/llama.cpp
server: return HTTP 400 for invalid embedding requests (#29060) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52037093 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11335 · ggml-org/llama.cpp
cuda : route sm70 to the Turing MMVQ nwarps table (#29753) cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares TURING's tuning: the K-quant vec_dot prefers nwarps=2 there. Route sm_70 to the existing MMVQ_PARAMETERS_TURING table in both the device and the host table selector. Measured on one Tesla V100 32GB PCIe (PG500-216, driver 580.178.04,
Release b11334 · ggml-org/llama.cpp
metal : release temporary private transfer buffers (#29777) metal : release temporary private transfer buffers Assisted-by: OpenAI Codex metal : fix order and formatting Co-authored-by: Niklas Wenzel dev@nikwen.de Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52026582 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11347 · ggml-org/llama.cpp
hexagon: install rebuilt HTP skels (#29828) hexagon: install rebuilt HTP skels Assisted-by: OpenCode hexagon: fix HTP skel catalog dependencies Assisted-by: OpenCode Website: https://llama.ap
Release b11333 · ggml-org/llama.cpp
webgpu: add bfloat16 support for MUL_MAT/MUL_MAT_ID/GET_ROWS- #29358 (#29358) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52019658 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
Release b11332 · ggml-org/llama.cpp
llama : fix invalid assert in recurrent memory (#29799) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52014664 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11331 · ggml-org/llama.cpp
CUDA: Handle compute type for NVFP4 on cublass path (#29173) CUDA: Handle compute type for NVFP4 on cublass path Signed-off-by: ynankani ynankani@nvidia.com Use BF16 compute type for quantized models if HW allows Signed-off-by: ynankani ynankani@nvidia.com Set acc prec to bf16 for nvfp4 as it needs atleast bf16 range Signed-off-by: ynankani ynankani@nvidia.com Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler johannesg@5d6.de preserve op_params for per-expert matmul Signed-off-by: ynankani ynankani@nvidia.com
Release b11330 · ggml-org/llama.cpp
Qwen4Exp: add MTP (#29761) Qwen4Exp: add MTP remove has_state member, check via ctx_bufs being non-empty consistent naming + less verbose comments cont : clean-up recurrent memory cont : clean-up comments Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52003136 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel
Release b11327 · ggml-org/llama.cpp
mtmd: cap max_image to ubatch for non_causal models (#29773) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51957111 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11326 · ggml-org/llama.cpp
meta: clear inactive AllReduce shards with FILL, not SCALE (#29793) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51949614 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11325 · ggml-org/llama.cpp
jinja : skip copying loop scope unless a loop filter needs it (#29776) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51940374 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11324 · ggml-org/llama.cpp
llama-mmap : avoid a second full-size copy of each tensor with direct-io (#29749) Assisted-by: Claude Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51929570 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11323 · ggml-org/llama.cpp
HIP: avoid treating CDNA as dgx spark for gqa_ratio 20 in fattn_mma dqk 576 (#29572) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51919562 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11322 · ggml-org/llama.cpp
hex-workqueue: fix race condition in seqn getting out of sync with idx_read/write (#29785) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51906249 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11321 · ggml-org/llama.cpp
BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS (#29640) BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path AOCL-BLAS doc : Note on ZenDNN Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51838813 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11320 · ggml-org/llama.cpp
common : add LLM-jp-4.1 Harmony dialect handler (#29681) LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space after every special token and parallel tool calls are separated by . The GPT-OSS handler rejects this output, so add a dedicated handler, selected by the chat_format=llm-jp-harmony-v1 declaration in the chat template. Assisted-by: Claude Fable 5.1 Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51789325 macOS/iOS: macOS Apple Silicon (arm64)
Release b11319 · ggml-org/llama.cpp
opencl: mark vec subgroup bcast as supproted for Adreno E17 compiler (#29698) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51780568 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11318 · ggml-org/llama.cpp
vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 (#29734) vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 The original tokenizer configs for PLaMo-2 and PLaMo-3 have add_bos_token: true and add_eos_token: false , but _set_vocab_plamo() did not write the BOS/EOS metadata. The PLAMO2 tokenizer path also ignored add_bos/add_eos during tokenization. Write the settings from tokenizer_config.json and honor them in the PLAMO2 tokenization path. GGUFs without these keys keep the previous behavior. Update conversion/base.py
Release b11317 · ggml-org/llama.cpp
llama-bench : fix verbosity filter to show GGML_LOG_ERROR (#28229) bench : fix verbosity filter to show GGML_LOG_ERROR (#28107) bench: remove dead variables Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51761023 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)
Release b11316 · ggml-org/llama.cpp
metal : use bf16 math for mxfp4 mul-mat (#29770) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51752863 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11313 · ggml-org/llama.cpp
model : re-enable -sm tensor for qwen4exp (#28569) #27941 disabled -sm tensor for qwen4exp because test-llama-archs asserted on the Meta device once the fixture carried a PLE layer: GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476. With host-resident embeddings the PLE gather is a CPU node and hc_init (the REPEAT that fans the embedding out to the hc streams) was first reached through layer 0's PLE path, after that gather. ggml_backend_sched_split_graph pass 2 expands a device
Release b11312 · ggml-org/llama.cpp
webgpu: fix SSM_SCAN binding aliasing (#29750) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51685637 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11311 · ggml-org/llama.cpp
ggml-opencl : replace alloca() with std::vector (#29765) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51681384 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11310 · ggml-org/llama.cpp
cuda: guard the iq4_nl dequantize row kernel against short rows (#29683) dequantize_block_iq4_nl writes QK_K values per block, but a row can be shorter than that (an IQ4_NL row is only guaranteed to be a multiple of QK4_NL). Threads whose 32-value sub-block starts at or past k currently read and write past the end of the row. Skip those sub-blocks; for rows that are a multiple of QK_K the check never fires. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51674818 macOS/iOS:
Release b11309 · ggml-org/llama.cpp
Hexagon: optimize ALLREDUCE with support for safe scatter mode (#29757) hex-allreduce: add support for safe scatter mode hex-allreduce: pare down excessive comments hex-allreduce: re-write to remove register spills Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51669892 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11308 · ggml-org/llama.cpp
args: fix cli download mmproj arg (#28977) tests: add tests for cli download arg parsing args: fix cli download mmproj arg Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51664692 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11307 · ggml-org/llama.cpp
llama : preserve original batch order for speculative decoding layer inputs (#29019) llama: preserve original batch order for layer inputs Assisted-by: Codex tests: cover layer-input order across KV layouts Assisted-by: Codex tests: exercise layer-input ordering on CUDA devices Assisted-by: Codex llama: make layer input reordering compatible with tensor split Copy each microbatch tensor from offset zero and restore original row order after synchronization.
Release b11306 · ggml-org/llama.cpp
test-llama-archs : toggle causal_attn to catch graph shape changes (#29724) After the device decode, flip causal_attn off, decode n_ubatch/2 then n_ubatch tokens. Both have the same node count, so a shape that depends on the flag makes the second reallocate at an unchanged graph size, which aborts under GGML_SCHED_NO_REALLOC. Skipped for the encode archs. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51649651 macOS/iOS: macOS Apple Silicon (arm64)
Release b11304 · ggml-org/llama.cpp
llama: properly handle KV on training (#28520) llama: properly handle KV on training improve Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51644581 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11303 · ggml-org/llama.cpp
batch: migrate the rest of examples to llama_batch_ext (#29601) migrate the rest test-thread-safety rm common_batch_staged Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51638011 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11302 · ggml-org/llama.cpp
glm5-next: give dead indexer slots unique scatter rows (#29745) The sparse indexer mask is built with a set_rows scatter. Padded pools, absent sequences and missing tail cells all pointed to the same n_kv sentinel row, and invisible pools picked by top_k to fill the selection overlap the tail cells of the token, so several CPU threads wrote the same element (ThreadSanitizer data race in the sanitize CI). Allocate the slot mask for both selection paths and route every dead slot to its own dump row n_kv + slot.
Release b11301 · ggml-org/llama.cpp
ggml/gguf : fix integer overflow (#29384) ggml: fix integer overflow guard for zero-element tensors ggml: validate number of elements in tensor to prevent integer overflow ggml: fix error print Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51618843 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11299 · ggml-org/llama.cpp
cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast (#29722) cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast On Windows the simple input reader sends CTRL_C_EVENT to every process attached to the console when stdin reaches EOF, killing unrelated processes such as a supervising agent. The CLI only stopped on EOF because of that self inflicted SIGINT; on POSIX, and with the advanced reader, it spins forever printing prompts. Drop the broadcast so both platforms just return an empty read, and
Release b11298 · ggml-org/llama.cpp
mimo : support dflash (convert + feature extraction) (#29650) convert : update to support dflash cont : fix Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51602210 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Int
Release b11297 · ggml-org/llama.cpp
jinja : support coerced array attributes (#29574) support coerced array attributes add tests Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51590476 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11295 · ggml-org/llama.cpp
ci : fix Models Backend Check by shortening the hrm_text fixture (#29744) The fixture recycles its two blocks over 8 cache slots, so the fp16 error builds up past the 1e-4 NMSE bound on the Vulkan T4 and WebGPU jobs of Models Backend. Two l-cycles keep every branch of the cycle loop and halve the error. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51576946 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11294 · ggml-org/llama.cpp
llama: llama_prefetch_rows (#29599) llama: llama_prefetch_rows llama: support row prefetch on Windows Apply the Windows port contributed by @praneshgo unchanged. Source: #29599 (comment) avoid exposing llama-mmap in model code, route via llama-impl add windows check, only prefetch in lazy mode cont : clean-up cont : fix build cont : clarify padding token for gemma4 Co-authored-by: Pranesh Gonegandla
Release b11293 · ggml-org/llama.cpp
ggml : add BF16 unary, GLU, binary and scale ops (CPU, CUDA) (#29675) ggml : add BF16 unary, GLU, binary and scale ops (CPU, CUDA) ggml-cpu : use per-op _bf16 functions for BF16 unary and GLU ops Assisted-by: Claude Opus 5.5 CUDA: use ggml_cuda_cast in binbcast and unary kernels to fix the HIP bf16 build ggml-openvino : reject BF16 SCALE and mixed-type BF16 ADD/MUL/SUB Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51554823 macOS/iOS: macOS Apple Silicon (arm64)
Release b11292 · ggml-org/llama.cpp
cpu: accept BF16 in src1 of mul_mat (#28937) cpu: accept BF16 in src1 of mul_mat ggml_conv_1d_dw builds its im2col in F32 when the kernel is BF16, then calls ggml_mul_mat(im2col, kernel), which puts F32 in src0 and BF16 in src1. The CPU backend refused that combination, so it was reported as unsupported on every backend and never compared against anything. Widen BF16 into the F32 work buffer, next to the existing packing of F32 into vec_dot_type. This is the arithmetic the Metal mat vec kernel
Release b11284 · ggml-org/llama.cpp
openvino: serve GET_ROWS on a weight view from the base Constant (#28381) openvino: serve GET_ROWS on a weight view from the base Constant Resolve view_src when collecting weight Constants so a view over a quantized weight no longer becomes a dynamic typed Parameter, and fold the row offset of the view into the gather indices instead of slicing the dequantization subgraph. openvino: lift the quantized GET_ROWS view rejection The supports_op rejection of a quantized src0 view with a nonzero
Release b11282 · ggml-org/llama.cpp
musa : define CUDA_ARCH for device passes (#29508) The MUSA vendor header never defined CUDA_ARCH, so every architecture test in the shared ggml-cuda sources evaluated to 0. Kernel bodies gated on the architecture therefore compiled to nothing, for example the q8_0 -> f16 dequantization kernel in convert.cu, whose NO_DEVICE_CODE fallback expands to an empty body in host code. Report the newest architecture like the HIP backend does and exclude the NVIDIA-only features explicitly, as they are not usable on MUSA. Define it
Release b11280 · ggml-org/llama.cpp
SYCL: reduce tensor allreduce sync with pinned host buffers (#29604) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51446793 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11279 · ggml-org/llama.cpp
add GLM-5.3-Flash (GLM5-Next) support (#27773) Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx Add initial MTP support Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements. Review driven changes, remove env vars, protect tensors Strip MTP for initial PR Clean up after mtp strip Clean up after mtp strip Update speculative.cpp Update llama-context.h Clean up after mtp strip Fix tokenizer ignore merges
Release b11278 · ggml-org/llama.cpp
vendor: update BoringSSL to 0.20260929.0 (#29669) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51432581 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11276 · ggml-org/llama.cpp
Hexagon f16 activation ops (#29209) hexagon: add F16 support for activation ops (SILU/GELU/GELU_QUICK/GEGLU/SWIGLU) Widens ggml_hexagon_supported_activations() to accept F16 (src0/dst/src1 must agree on type), and adds F16 per-thread worker functions in act-ops.c mirroring the existing F32 workers, backed by new HVX f16 kernels (hvx_sigmoid_f16_aa, hvx_tanh_f16_aa, hvx_mul_mul_f16_aa, hvx_min_scalar_f16 family). SILU, GELU, GELU_QUICK, GEGLU, and SWIGLU are verified correct on-device
Release b11275 · ggml-org/llama.cpp
gguf : reject tensor size that wraps after padding (#26979) GGML_PAD(nbytes, alignment) wraps to 0 when nbytes is within (alignment - 1) of SIZE_MAX, which silently bypassed the size overflow guard in gguf_init_from_reader. Reject the tensor before padding when nbytes + (alignment - 1) would overflow. Adds a test-gguf handcrafted case (F32, ne = [4, 2^30-1, 2^30+1, 1]) whose ggml_nbytes = 2^64 - 16 lands in the wrap window. Fails on master, passes with the guard. Website: https://llama.app Attestations:
Release b11274 · ggml-org/llama.cpp
ggml : check row bounds in get_rows_back (#29575) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51392411 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11273 · ggml-org/llama.cpp
model : support classifier_pooling for rerankers (#29627) model : support classifier_pooling for ModernBERT rerankers Assisted-by: Claude Opus 5.5 model : read classifier pooling type in load_hparams Write classifier.pooling_type from _try_set_pooling_type whenever the config has classifier_pooling, and read it in llama_model_base::load_hparams. ModernBERT falls back to mean when it is unspecified. Assisted-by: Claude Opus 5.5 conversion : only accept cls and mean for classifier_pooling
Release b11272 · ggml-org/llama.cpp
hexagon: optimize concat op (#29673) hex-concat: reduce pkts in gather/transpose hot loop gather directly into dst buffer, use special instruction for gather sync hex-concat: use fastdiv replace calls to sw divide with fastpath hex-concat: optimize DMA-HVX pipeline and add transpose helpers Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51367895 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11277 · ggml-org/llama.cpp
ggml-zdnn: impl buffer reset, fix memory leaks (#29637) cont: fix code style Signed-off-by: Aaron Teo aaron.teo1@ibm.com Website: https://llama.app Attestations: https://github.com/ggml-org/lla
Release b11270 · ggml-org/llama.cpp
ggml-cuda: HIP: optimize packed byte subtraction (__vsubss4 -> __vsub4) (#29478) ggml-cuda: HIP: optimize non-saturating packed byte subtraction (__vsubss4) CI: ignore 1 spilled vgpr in fattn_vec Co-authored-by: Carl Philipp Klemm carl@uvos.xyz Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51344086 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11269 · ggml-org/llama.cpp
ci: add zdnn backend build but not test (#29541) ci: add zdnn backend build but not test Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci: attempt to run a ubuntu 26.04 container Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci: set shell to bash Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci: clean up comments Signed-off-by: Aaron Teo aaron.teo1@ibm.com ggml-zdnn: fix compiler errors Signed-off-by: Aaron Teo aaron.teo1@ibm.com vendor: attempt to ignore warnings from vendored files Signed-off-by: Aaron Teo aaron.teo1@ibm.com
Release b11268 · ggml-org/llama.cpp
opencl: fix get_tensor for q5_K adreno gemm_nonshuffle kernel (#29555) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51311810 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11267 · ggml-org/llama.cpp
vulkan: Tune GDN kernel, fix Intel performance (#29476) vulkan: tune GDN shader tune for Intel Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51306598 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11266 · ggml-org/llama.cpp
vulkan : Load F32 A matrix 2 at a time when its 2-aligned (#29254) It turns out Intel doesn't particularly like loading F32s one at a time and we already have the _2aliagned load logic in mul_mat_vec, so here we use it. While we do already check all the requirements to load elements 4 at a time across [B]F16 and F32, it turns out [B]F16 loading 4 at a time is sometimes slower on very specific shapes on Intel BMG. Loading 4 at a time is a bit faster on F32, but its not material and I assume might be slower on other platforms.
Release b11265 · ggml-org/llama.cpp
vulkan: MOE aware mat_mul_id tile selection (#29182) mut_mul_id selected its matmul tile with total token count. For MoE dispatch grid the true N per workgroup is per-expert rows. At pp128 on Sarvam 30B that is 6, not 128, so the picker took the l-tile for ~6 live rows. Most workers in each group had nothing to do. This wasted time. The slow part was 55% of the whole job. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51296894 macOS/iOS: macOS Apple Silicon (arm64)
Release b11264 · ggml-org/llama.cpp
ggml : fix c++ odr by properly using GGML_COMMON_DECL_CPP (#29504) fix c++ odr by properly using GGML_COMMON_DECL_CPP using actual field rather than macro Co-authored-by: Georgi Gerganov ggerganov@gmail.com Co-authored-by: XZiar xziar@xziar.xziar Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51291025 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
Release b11263 · ggml-org/llama.cpp
vocab : keep NORMAL in PLaMo-2 and PLaMo-3 (#29580) vocab : keep NORMAL in PLaMo-2 and PLaMo-3 The PLaMo-2 and PLaMo-3 vocabularies mark as NORMAL. Current EOG token heuristic matched it by text and added its attribute to CONTROL. Skip this heuristic for the PLAMO2 vocab type so stays NORMAL and is not treated as EOG. use for detection Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51285607 macOS/iOS: macOS Apple Silicon (arm64)
Release b11262 · ggml-org/llama.cpp
ggml : accumulate f16 dot products in f32 on AVX512-FP16 (#29545) Supersedes #29530 Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51274261 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLE
Release b11261 · ggml-org/llama.cpp
ggml : require input tensors to be GGML_OP_NONE (#29647) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51266318 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11260 · ggml-org/llama.cpp
hexagon: add FP32 GELU_ERF and GEGLU_ERF support (#29631) hexagon: add FP32 GELU_ERF and GEGLU_ERF support hex-erf: reduce register pressure in kernels Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51243904 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11259 · ggml-org/llama.cpp
common : stop accepting draft tokens at EOG (#29638) common : stop accepting draft tokens at EOG cont : remove the test Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51199980 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11258 · ggml-org/llama.cpp
server : remove the built-in UI's service worker when the UI is not served (#29565) With --path or --no-ui, /sw.js returned 404, and a 404 does not remove a service worker, so browsers kept showing the cached built-in UI. Serve a worker that unregisters itself, clears its caches and reloads open tabs. A sw.js in the --path folder is still served first. Assisted-by: Claude Opus 5.5 Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51186244 macOS/iOS: macOS Apple Silicon (arm64)
Release b11257 · ggml-org/llama.cpp
common : use fs::path for config dir (#29649) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51176378 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11256 · ggml-org/llama.cpp
tests : adjust server string regex to also match m2 utlra results (#29648) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51122498 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11255 · ggml-org/llama.cpp
common : add fs_write_atomic() (#29642) Check for buffered write errors when closing downloaded files. Use UTF-8 paths when writing ETag files on Windows. Write in binary mode on Windows. Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51111496 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11254 · ggml-org/llama.cpp
ggml : collect all input tensors into graph_inputs (#29634) graph_inputs was populated while splitting the graph, so it only contained the inputs that are used as srcs of some node. With pipeline parallelism (n_copies > 1), each graph input contributes n_copies leafs to graph_copy, so switching between batches that consume different inputs (e.g. token batches that do not use the embeddings input vs image batches that do) changed the graph composition. This shifted the
Release b11249 · ggml-org/llama.cpp
llama : fix init in several tools/examples (#29632) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51066139 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11247 · ggml-org/llama.cpp
vulkan : reuse descriptor sets when bindings are constant (#29280) vulkan : reuse descriptor sets when bindings are constant vulkan : bump buffer_destroy_count before destroying the buffer Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51033526 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11246 · ggml-org/llama.cpp
chat : fix Muse Glimmer ignoring response_format json_schema with --jinja (#29615) chat : fix Muse Glimmer ignoring response_format json_schema with --jinja Fixes #29613 chat : accept json fences and clean up chat : fix choice parenthesis Co-authored-by: Alde Rojas hello@alde.dev Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51023533 macOS/iOS: macOS Apple Silicon (arm64)
Release b11245 · ggml-org/llama.cpp
common : use fs::path for cache dirs (#29595) Avoid useless string conversions on Windows. No need for BSD or emscripten special cases. Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51016898 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11243 · ggml-org/llama.cpp
ci : update the oneAPI toolkit to 2026.1 (#29273) ci : update the oneAPI toolkit to 2026.1 oneDNN is removed from Intel Deep Learning Essentials in 2026.0, so staying on the deep-learning-essentials path would silently lose oneDNN support when the toolkit version is updated. Switch both the Ubuntu and Windows CI jobs to the new unified Intel oneAPI Toolkit installer, which still includes oneDNN (until 2027.0) and keeps the component IDs unchanged for the Windows install script.
Release b11242 · ggml-org/llama.cpp
mtmd: fix GCC 15 stringop-overflow in decode_embd_batch (#29607) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50949667 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11240 · ggml-org/llama.cpp
server : support typed content (vision/audio/video) input for /v1/embeddings endpoint (#29556) server : support multimodal input for /v1/embeddings (Qwen3-VL-Embedding) Accept the OpenAI-style wrapped content array format for multimodal embedding requests. Each {"content": [...]} object is one input that produces one embedding; text parts are concatenated and image_url parts are decoded via handle_media then spliced with process_mtmd_prompt. The legacy formats (plain string, token arrays, mixed arrays, and the
Release b11239 · ggml-org/llama.cpp
vulkan: include functional header (#29597) Fixes the compile error: no template named 'function' in namespace 'std' Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50927029 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11238 · ggml-org/llama.cpp
models: pad on the left with ggml_pad_ext (#29567) models: pad on the left with ggml_pad_ext The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders build a left padding as a right pad followed by a roll, and DFlash2 concatenates a zero filled block in front of the previous tokens. ggml_pad_ext does both in one node now that every backend supports a left padding. The Gemma 4 audio embeddings are bit identical. models: skip the DFlash2 taps that only read padding A tap at or past block_size shifts every row out of the block,
Release b11237 · ggml-org/llama.cpp
ggml-openvino: mark unaligned batch-stride views unsupported (#29603) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50908876 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11236 · ggml-org/llama.cpp
batch: migrate speculative, mtmd and server to batch_ext (#29385) adapt common add common_batch wip wip: spec cont common_speculative_process server_batch to use common_batch rm some stale calls Assisted-by: Claude Fable 5.1 migrate mtmd handle imrope, handle return val of add()/add_embd() add spec zeros vector add warning on zero fill path Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50870264 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)
Release b11235 · ggml-org/llama.cpp
common : fix HF cache paths on Windows (#29475) Supersedes #29158 Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50839831 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
Release b11234 · ggml-org/llama.cpp
webgpu: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor (#29471) Fix: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor Clang formatting Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50830975 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11233 · ggml-org/llama.cpp
tests : refactor test-recurrent-state-rollback (#29426) tests : use llama_context_ptr in test-recurrent-state-rollback Replace raw llama_context pointers with llama_context_ptr and drop the manual llama_free calls and cleanup lambda. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL tests : run test-recurrent-state-rollback over all dummy models Add a --models DIR mode that mirrors test-save-load-state: iterate every dummy model, report PASS/FAIL/SKIP in a table and fail only when a model fails.
Release b11232 · ggml-org/llama.cpp
ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 (#29423) ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 add AVX2 support for masked loading and storing in simd_gemm_ukernel_tail ggml-cpu: fix FA softcap handling for padded KV tiles Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50796641 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11229 · ggml-org/llama.cpp
HIP: fix template skip for DKQ > 256 mfma kernels (#29559) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50767985 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11228 · ggml-org/llama.cpp
metal: support left and circular padding in GGML_OP_PAD (#29561) metal: support left and circular padding in GGML_OP_PAD Align Metal with CPU, CUDA and Vulkan: shift the source coordinates by the left paddings, wrap them around with the same wrap_around when circular, and read the source through nb00, which also fixes a right padding of a permuted source. A test case covers it. Drop the f32_4 kernel: its selection is disabled as slower, and it fails two pad cases once enabled.
Release b11227 · ggml-org/llama.cpp
context : do not re-reserve the scheduler when toggling causal_attn (#28751) context : do not re-reserve the scheduler when toggling causal_attn llama_context::set_causal_attn() marks the scheduler to do a full re-reserve on every change of the flag. For vision inputs, this flag is flipped twice around each non-causal image chunk for Gemma models, resulting in two expensive sched_reserve() passes per image. This is especially slow for multi-image or video inputs.
Release b11226 · ggml-org/llama.cpp
Enables Windows ARM64 build with MSVC cl.exe (#28362) can reproduce the issue vlad sees fix fma issue drop volatile fix volatile runtime task add arm flag if needed fix hsum compile error fix syntax in quants strengthen sve probing make the syntax fixes one liners remove debug code formatting remove macro for float drive down gcc instruction count support armec fix CI comments address CI comments fix cross compile issue remove warning fix style and fix fma probing fix style add documentation update documentation
Release b11225 · ggml-org/llama.cpp
tests : fix ggml init (#29554) tests : init ggml for test-recurrent-state-rollback cont : same for test-save-load-state cont : add to test-state-restore-fragmented + add TODOs Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50700507 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11224 · ggml-org/llama.cpp
vulkan: fix wrong results when a mul_mat reads a slice of a larger cache (#28956) vulkan: read the batch stride of an in place src0 from nb[2] A dim01 contiguous tensor can still be a view whose batches are strided by more than ne[1] rows, the first rows of a KV cache for example. Both the mat-vec and the matrix paths read such a tensor in place but passed ne00*ne01 as the batch stride, so every head past the first read the wrong rows. The same applies to src1. The stride now comes from nb[2] whenever the tensor is used in place;
Release b11223 · ggml-org/llama.cpp
server : allow RANK pooling batch splitting for causal LLM rerankers (ie. Qwen3 and Qwen3-VL) (#28876) server : allow splitting RANK pooling for causal LLM rerankers Rerank models fall into two categories: bidirectional cross-encoders (BERT, etc.) that require all tokens in a single physical batch, and causal LLMs repurposed as rerankers (Qwen3, Qwen3-VL) that can use chunked prefill like any other decoder. Previously the server rejected all RANK-pooling inputs larger than
Release b11222 · ggml-org/llama.cpp
common : avoid side effects around params parsing (#29537) register --rpc unconditionally and call llama_supports_rpc() only from its handler print server "initialization ..." log after args are parsed Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50580981 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCF
Release b11221 · ggml-org/llama.cpp
common : make string_split throw on invalid values (#29518) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50573055 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11218 · ggml-org/llama.cpp
jinja : add support for dict builtin (#29477) add support for dict builtin add tests Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50563209 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11217 · ggml-org/llama.cpp
opencl: refine bin kernel loading condition (#29503) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50560500 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11216 · ggml-org/llama.cpp
sycl: FWHT kernels for block widths above 512 (#29243) The SYCL FWHT covers 64 to 512 via the standard butterfly network, plus 384/640/768/1280 via the Kronecker/Paley construction added separately in Hadamard hint can produce (1024, 2048, 4096, 8192); those still fall through to the default case and run as a dense GEMM against the materialized rotation tensor, correct but O(n^2) instead of O(n log n). fwht_kernel_wide runs one row per work-group instead of per sub-group, so each work-item keeps N/NT values rather than N/WARP_SIZE.
Release b11215 · ggml-org/llama.cpp
CUDA: tune fp16 tile FlashAttention configs for head sizes 40-112 (#26289) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50554486 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11214 · ggml-org/llama.cpp
HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes (#28907) HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes CI: hip-quality-check: ignore spills for very large mfma mma kernels Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50549501 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramewo
Release b11213 · ggml-org/llama.cpp
vulkan: fix argsort kernel selection for Adreno (#29469) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50544436 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11212 · ggml-org/llama.cpp
common : throw instead of abort on grammar without llguidance (#29516) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50541746 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11211 · ggml-org/llama.cpp
RPC: use RDMA completion channel to not spin (#29440) RPC: use RDMA completion queue to not spin add TODO for apple RDMA Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50534798 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11209 · ggml-org/llama.cpp
llama-bench : fix OOB access of hf_file (#29515) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50528155 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11208 · ggml-org/llama.cpp
hrm : fix layer placement of z_l_init weight (#29512) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50522371 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11207 · ggml-org/llama.cpp
hexagon: support tiled Q4_0 and Q8_0 GET_ROWS (#29511) hexagon: support tiled Q4_0 and Q8_0 GET_ROWS hex-get-rows: fix macros hex-get-rows: use tiled HVX dequantization Assisted-by: OpenCode hex-get-rows: fix register spills and clean up checks for unsupported ops hex-get-rows: improve dma pipeline hex-get-rows: improve/simplify kernel selection logic hex-build: reenable vectorizer, didnt notice the regression earlier in the sampler update Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com Website: https://llama.app
Release b11206 · ggml-org/llama.cpp
hexagon: support for backend sampler (#29502) hex-topk: trying to improve/cleanup the pipeline hex-sampling: add STEP op hex-sampler: add SUM op hex-sampler: update CPY to support sampling cases hex-binary: add support for chunking to handle large logits hex-argmax: super basic version of ARGMAX hex-binary: support for scalars in extended buffers hex-binary: fix wrong indexing for dim 1 broadcasts across dim 2 slices hex-argsort: fix missing header hex-sampler: cleanup dma usage in the sampler related ops, and binary
Release b11205 · ggml-org/llama.cpp
cuda: support Nemotron 3 Puzzle state size 96 for ssm scan (#28717) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50461180 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11203 · ggml-org/llama.cpp
cuda: add F16 input to the FWHT (#29096) cuda: add F16 input to the FWHT The CUDA FWHT accepts F32 input only. This makes the source type a template parameter, so the kernel reads an F16 source directly instead of requiring a converted copy. The F32 path is unchanged. supports_op accepts an F16 src1 against an F32 src0 for the Hadamard hint. Every other F16 src1 against a non-F16 src0 is still refused. ggml_cuda_op_mul_mat_use_fwht is the single predicate both supports_op and
Release b11202 · ggml-org/llama.cpp
server : fix wake_fd warning on Windows (#29479) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50455005 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11201 · ggml-org/llama.cpp
Revert "Change max context length for auto-fitting with unified KV (#28849)" (#29437) This reverts commit b04d4e5. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50433369 macOS/iOS: macOS Apple Silicon (arm64) ma
Release b11200 · ggml-org/llama.cpp
jinja : implement sameas test (#29448) implement sameas test add tests Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50397675 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11199 · ggml-org/llama.cpp
jinja : fix compile error (#29468) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50395469 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11195 · ggml-org/llama.cpp
ggml-cpu: tiled mul_mat for k-quants (#27851) Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16x16 tiles before writing out 256x256 float reults to main memory. Tests/benches in tests/test-tiled-mulmat.cpp. 3-6x speed improvement for large matmul, break even at 4096x64 * 64x4096, 80% performance (net loss) for GEMV. Error rates trivial (order of 1-e04 max, 1-e05 rmse). Fixes for ARM/windows builds more windows fixes, ggml-cpu.h isn'
Release b11194 · ggml-org/llama.cpp
opencl: add A8 Q8_0 non-MoE dp4a binary kernel (#29439) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50382689 macOS/iOS: macOS Apple Silicon (ar
Release b11193 · ggml-org/llama.cpp
hexagon: find software divide calls using binary inspection tool (#29449) hex-scripts: fix table alignment hex-scripts: find sw div calls using binary inspection tool Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50359675 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11192 · ggml-org/llama.cpp
vendor : update cpp-httplib to 0.58.0 (#29407) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50337762 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11191 · ggml-org/llama.cpp
common,rpc : simplify fs_create_directory_with_parents() (#29432) The original function was broken on Windows for some unicode paths Paths without a trailing separator now create the last directory too, matching the function name. All current callers already include a trailing separator, so this change does not affect them. Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50291720 macOS/iOS: macOS Apple Silicon (arm64)
Release b11190 · ggml-org/llama.cpp
mtmd: fix mel preprocessor in LFM2 audio (#29403) which resulted in different greedy transcripts for 4.5% of English and 6.5% of Japanese test utterances. In Japanese, some differences changed entire words. This change: uses log(x + 2^-24) instead of clamping to the log floor uses a symmetric Hann window, equivalent to torch.hann_window(periodic=False) adds the normalization epsilon to the standard deviation instead of inside the square root Only the lfm2a preprocessor opts into these behaviors. Other audio preprocessors are unchanged.
Release b11189 · ggml-org/llama.cpp
opencl: add bin kernel kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin, kernel_gemm_noshuffle_q5_k_q8_1_dp4a_ila_a8_bin (#29401) opencl: add A8 Q5_K non-MoE non dp4a + dp4a binary kernel opencl: fix s transpose - s only transposed for bin kernels Co-authored-by: Li He lih@qti.qualcomm.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50281199 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED ma
Release b11188 · ggml-org/llama.cpp
Fixing the vulkan build issue of legacy GLSLC version that has no cooperativeMatrix API support (#29373) (#29409) vulkan : fix build issue of legacy glslc version by adding GGML_VULKAN_COOPMAT_GLSLC_SUPPORT macro check for Intel FA shader compiling vulkan : add preprocess condition to filter out unsupported FA 2 phases kernels before creation. vulkan : move lock_guard for Intel FA shader pointer creation under CM1 compiling preprocessor Website: https://llama.app Attestations:
Release b11185 · ggml-org/llama.cpp
common : extract shared unicode path/string helpers (#29415) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50264913 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11184 · ggml-org/llama.cpp
metal: FWHT kernels for block widths above 512 (#29095) metal: FWHT kernels for block widths above 512 The Metal FWHT covers widths 64 to 512, one row per simdgroup with N/32 values per lane. Wider blocks need more registers per lane than that layout allows. kernel_fwht_tg runs one row per threadgroup with 256 threads, so each thread keeps N/256 values. Butterflies below the simdgroup width still shuffle, those up to the threadgroup width go through threadgroup memory, and the rest stay in registers.
Release b11183 · ggml-org/llama.cpp
metal : split fa kernels into per-dtype libraries (#29329) metal : split fa kernels into per-dtype libraries Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp cont : minor fix comment Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50249107 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11182 · ggml-org/llama.cpp
llama : add llama_prec_policy + model-driven W4A4 path (#24364) Rebase and update based on #26675 Signed-off-by: ynankani ynankani@nvidia.com CI failure fix(launh_bounds overload on HIP) and cleanup Signed-off-by: ynankani ynankani@nvidia.com Address review comments Signed-off-by: ynankani ynankani@nvidia.com Use ggml tensor instead of name in act policy map Signed-off-by: ynankani ynankani@nvidia.com Address review comments and cleanup Signed-off-by: ynankani ynankani@nvidia.com Address review comments Signed-off-by: ynankani
Release b11181 · ggml-org/llama.cpp
HIP: bump HIP_VERSION requried for fp8 to avoid missing __hip_fp8_e4m3 support in 6.2 (#29231) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50216799 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11180 · ggml-org/llama.cpp
rpc: include nb in the get_alloc_size cache key and floor the result at ggml_nbytes (#29283) rpc : include nb in the get_alloc_size cache key and floor the result at ggml_nbytes cont : remove redundant comment cont : add TODO Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50207987 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11179 · ggml-org/llama.cpp
[SYCL] support sparse FA (#28796) fix conflict fix format issue rm unused code Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50188435 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11178 · ggml-org/llama.cpp
musa: fix PH1 (MTT S5000) operator failures and build issues (#29193) musa: use 16-byte copies for MUSA like sm_70+ ggml_cuda_get_max_cpy_bytes() derives the copy width from CUDA_ARCH. mcc never defines it, so MUSA fell into the generic branch and returned 8 bytes instead of the 16 bytes that every sm_70+ target gets. The value sizes the per-thread copy unit of the FlashAttention K/V staging code (fattn-common, fattn-vec, fattn-tile, fattn-mma-f16 shared-memory loads) and of mmq-vec-dot,
Release b11177 · ggml-org/llama.cpp
CUDA: fuse RMS_NORM + SCALE into one kernel (#29393) #28068 builds the GDN q/k l2norm as ggml_scale(ggml_rms_norm(x, eps/n), 1/sqrt(n)). This adds 2 SCALE nodes per GDN layer, 96 extra kernel launches per ubatch on Qwen3.8-27B (48 GDN layers). The extra kernels take no measurable GPU time, but each launch has a host/driver cost. It is small with plain batch processing and about 10x larger with draft-mtp speculative decoding.
Release b11176 · ggml-org/llama.cpp
llama : fix tensor split for fused qkv with uneven K/V head sizes (#2
Release b11175 · ggml-org/llama.cpp
hexagon: add q5_k quant type support (#29123) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50064037 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11174 · ggml-org/llama.cpp
hexagon: use DMA for contiguous dim1 CONCAT (#29404) hexagon: use DMA for contiguous dim1 CONCAT Assisted-by: OpenCode hexagon: update CONCAT DMA for DMA64 Assisted-by: OpenCode Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50060105 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11173 · ggml-org/llama.cpp
metal : fix graph capture and handle empty graphs (#29390) return early when the graph has no nodes drop the redundant reset of capture_compute: the decrement at the top of the function already transitions the counter from 0 to -1, so a capture happens exactly once hint at METAL_CAPTURE_ENABLED=1 in the capture error message pass capture_compute == 0 (not the raw counter) as use_capture to ggml_metal_op_init, so GPU debug-group markers are only emitted on the captured compute Assisted-by: pi:llama.cpp/Qwen3.8-27B Website:
Release b11172 · ggml-org/llama.cpp
metal : optimize sparse FA + clean-up (#29377) metal : cache sparse FA indices in shared memory Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp metal : simplify shared memory size calculation Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp pi : update general metal : unroll sparse index load Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50026590 macOS/iOS: macOS Apple Silicon (arm64)
Release b11171 · ggml-org/llama.cpp
sync : ggml (#29396) ggml : bump version to 0.25.2 (ggml/1642) ggml : fix ubsan error in ggml_graph_nbytes (ggml/1644) ggml : bump version to 0.25.3 (ggml/1645) sync : ggml Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50020953 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11170 · ggml-org/llama.cpp
hexagon: handle multi-sequence in concat_2d (#29344) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50014220 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11169 · ggml-org/llama.cpp
llama-grammar: fix numeric truncation for token_id parsing (#29382) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50007794 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11168 · ggml-org/llama.cpp
hexagon: dynamic quantizer improvements (#29395) hexagon: fix accuracy issue in Q8_0 N=1 MUL_MAT hex-quant: fix register spills hex-mm: use dma for all dyn.quant paths Co-authored-by: Aparna M P aparmp@qti.qualcomm.com hex-mm: remove obsolete run_quant_task hex-mm: update tracing to properly wrap the events hex-mm: use act for activation data in all paths hex-mm: use act_ instead of src1_ to avoid confusion in fused kernels hex-mm: remove/reroute the rest of the non-DMA act (aka src1) logic
Release b11167 · ggml-org/llama.cpp
hexagon: support I32 CPY and CONT (#29379) Assisted-by: OpenCode Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49994729 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11166 · ggml-org/llama.cpp
cuda : add F16 kernel support for CONV_2D_DW (#29064) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49985945 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11165 · ggml-org/llama.cpp
test: flush status (#28352) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49978694 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11163 · ggml-org/llama.cpp
llama: add llama_batch_ext (#24669) (wip) add llama_batch_ext wip updated design updated impl change signature unused var demo common_prompt_batch_decode fix pos tmp disable test-batch-alloc fix compat nits: add const no more pos_max add comment about llama_batch_ext_set_embd_state handle n_embd_out properly rename api --> embd_token llama_embd stub llama_batch_ext_set_embd_state support both token + embd + state in batch llama_batch_ext_add_embd upstream some changes nits fix test-batch-alloc add test for compat
Release b11160 · ggml-org/llama.cpp
vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 (#27952) vulkan: add int8 coopmat quantized matmul shader apply scales inline use scalar sums probe and directly access coopmat values instead of going through shmem add q8_0 support add BK_STEP to shader, default to 2 use larger workgroups double buffering preload scales coopmat load first, then wmma use float for scales add faster RDNA int->float conversion workgroup scheduling for cache proximity clean up use wave32 restructure for vgpr use
Release b11159 · ggml-org/llama.cpp
vulkan: handle misalignment in conv_2d and conv_3d (#29365) vulkan: handle misalignment in conv_2d and conv_3d fix test-backend-ops print Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49846182 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11158 · ggml-org/llama.cpp
vulkan: tune KHR cooperative matrix support for Adreno GPUs (#29328) Enable coopmat support for Vulkan backend Fixed the mul_mat_s Removed the debug statement Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49831613 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11157 · ggml-org/llama.cpp
cuda : add conv3d with implicit GEMM (#29137) cuda : add conv3d with implicit GEMM cuda : refine conv3d implicit GEMM and handle empty kernels Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49789544 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11156 · ggml-org/llama.cpp
model : add Ling 3.0 VL support (#29151) model : fold Ling 3.0 VL into the BailingMoeV3 architecture Assisted-by: Scout model : keep shared NORM rope list intact when gating bailingmoe3 on mrope sections Co-authored-by: aetherbird aetherbird@users.noreply.github.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49780861 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11155 · ggml-org/llama.cpp
server,common : fix the GCC 12 stringop-overread false positive (again) (#29325) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49775037 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11154 · ggml-org/llama.cpp
test-save-load-state : print a per-model results table in --models mode (#29316) test-save-load-state : print a per-model results table in --models mode in --models mode the output was very heavy: every model printed its token dumps, per-test headers and PASS lines. instead, silence all logging except the table itself (common_log_set_verbosity_thold(0) leaves only LOG / LOG_LEVEL_OUTPUT) and print one row per model with one column per test, colored PASS/FAIL/SKIP cells, row by row.
Release b11153 · ggml-org/llama.cpp
hexagon: reject MUL_MAT_ID when src1 precision is F32 (#29348) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49757881 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release v0.5.0 · ggml-org/llama.cpp
Overview This release focuses on backend performance and correctness, broader model coverage, and more robust server/router operation. It adds HRM-Text (DFM Mimir 1B) support, MiMo-V2.6 and HunyuanOCR conversion support, ggml 0.25.0 backend improvements, multi-address HTTP binding, image outputs from function calls, and several chat parser/UI fixes.
Release b11147 · ggml-org/llama.cpp
opencl: add A8 Q6_K non-MoE dp4a binary kernel (#29057) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49630180 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11146 · ggml-org/llama.cpp
llama.cpp : bump version to 0.5.0 (#29333) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49623059 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11140 · ggml-org/llama.cpp
CUDA: enable sparse-fa for dsv4 prefill (again) (#29298) CUDA: enable sparse-fa for dsv4 prefill (again) CUDA: unroll the query loop of the sparse mask scan The query loop of flash_attn_mask_to_sparse_indices has a runtime trip count, which keeps the unrolled scan over the values of a lane from issuing its loads together. Template the kernel on ncols1 so the loop is bounded at compile time: batch one decodes compile to straight line code and the scan drops from 46 to 17 us at 49k columns on sparse decode shapes.
Release b11139 · ggml-org/llama.cpp
server: fix token counting API crash on sleep (#29309) server: wake up sleeping server correctly server: wake up sleeping server correctly (local aliases removed) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49582694 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11138 · ggml-org/llama.cpp
jinja : parse unary +/- before variables (#29244) jinja : parse unary +/- before variables Lexer already emits unary_operator for -n / +n, and runtime executes unary -. Parse them at multiplicative precedence so slices like items[:-n] and GigaChat indent[:-indent_factor] work. jinja : keep filters/tests outside unary operands Unary +/- must bind only the primary/postfix operand so -n|abs is (-n)|abs, not -(n|abs). Add unary + and filter/test regression coverage. Signed-off-by: sinksilk 785976238@qq.com
Release b11136 · ggml-org/llama.cpp
server: accept OpenAI video_url content type and data: video URIs (#27921) The OpenAI chat completions API specifies content part type "video_url" with a {"url": ...} object, and clients typically send data: URIs (e.g. data:video/mp4;base64,...). The llama-server only accepted the non-standard "input_video" type and rejected data: URIs for video (accept_base64_uri=false), so any OpenAI-conformant client failed with "unsupported content[].type" or "Invalid uri format". accept "video_url" as an alias of "input_video"
Release b11135 · ggml-org/llama.cpp
server: Dedup the draft HF model via dedup-cache-models (#27934) server: Dedup the draft HF model via dedup-cache-models Fixes #27846 server: avoid capturing structured binding in lambda Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49545978 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)
Release b11132 · ggml-org/llama.cpp
model : support Gemma4 DSpark draft backbone (#29226) dspark: add Gemma 4 draft support Add GGUF conversion and runtime support for full-attention and SWA Gemma 4 DSpark drafts, including tied output weights and boolean backbone metadata. Assisted-by: Codex dflash: infer Gemma draft features from metadata Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49534734 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11130 · ggml-org/llama.cpp
make-release : update summary prompt Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49525233 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU)
Release b11126 · ggml-org/llama.cpp
vulkan: add IQ4_XS MMQ/MMV matmul kernels (#28415) vulkan: optimize IQ4_XS matmul kernels Assisted-by: OpenAI Codex vulkan: address IQ4_XS review nits drop the dead LOAD_VEC_A != 8 branch in the IQ4_XS shmem load; iq4_xs is in lut_load_vec_a()'s "8" list, so that path is never generated disable MMVQ for IQ4_XS on Intel (27.3% tg regression on A770) remove a stray empty line in types.glsl Assisted-By: Claude Opus 5 noreply@anthropic.com Website: https://llama.app Attestations:
Release b11125 · ggml-org/llama.cpp
ggml-meta: resolve multi buffer views (#29266) ggml-meta: resolve multi buffer views add TODO to revisit if graph allocator gets refactored Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49502431 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11124 · ggml-org/llama.cpp
cuda: top-k MoE should always fire (#28432) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49495074 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11123 · ggml-org/llama.cpp
sycl : support new UT case for mul_mat_hadamard fp16 (#29218) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49484593 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11122 · ggml-org/llama.cpp
sycl: extend MMVQ GLU fusion, add rms_norm+scale and ssm_conv+silu fusions (#28931) sycl : extend MMVQ GLU fusion to mixed quant types; add rms_norm+scale and ssm_conv+silu fusions fixing spacing issue and macro converted to template function Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49468532 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11121 · ggml-org/llama.cpp
sycl : support op get_rows_back, only support fp32/fp16 (#25266) resovle confict support gedt_rows_back, update the ops.md Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49462636 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11120 · ggml-org/llama.cpp
vulkan: hide internal symbols to prevent duplicate-dlopen state destruction (#29139) Since #28732 our internal symbols are exported. A duplicate copy dlopened and dlclosed by ggml_backend_load_all() then interposes them, so its destructors destroy the live vk_instance and later device queries hit the GGML_ASSERT on vk_instance.device_indices. Hidden visibility exports only GGML_BACKEND_API, as before #28732. Fixes
Release b11119 · ggml-org/llama.cpp
sampler: reduce the size of the probe (#29285) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49443624 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11118 · ggml-org/llama.cpp
hex-dma: introduce direct-mapped DMA cache that is better suited for HVX FA mask handling (#29282) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49404271 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (
Release b11117 · ggml-org/llama.cpp
HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR (#27962) HIP : use bit manipulation for __vcmpne4 HIP : use bit manipulation for __vsub4 Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49401157 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11115 · ggml-org/llama.cpp
opencl: add bin kernel kernel_gemm_noshuffle_q4_k_q8_1_dp4a_ila_a8_bin (#29056) opencl: add A8 Q4_K non-MoE dp4a binary kernel opencl: rename binary kernel selection helpers Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49361383 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11114 · ggml-org/llama.cpp
server: fix router eviction races with the existing queue (#29217) server: route every model load through the queue A model loaded by the fast path has no queue entry, so tick() evicts it at its LOADED transition before its own request is proxied. Every load now joins the queue, whose entry protects the model until its waiters leave. server: do not admit requests into a stopping model A request for a model that is being stopped still sees it LOADED and is proxied into the dying child. Such a request now joins the queue
Release b11113 · ggml-org/llama.cpp
server: do not pass log file to children (#29212) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49344105 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11112 · ggml-org/llama.cpp
server: support input_image in function_call_output (#20663) (#22575) server: support input_image in function_call_output (#20663) server: fix if statement spacing server: avoid repeated type lookup Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/4933
Release b11111 · ggml-org/llama.cpp
vulkan: add Intel Xe flash attention optimization kernels (2/3, Xe-LPG Plus/Xe2/Xe3) (#24406) vulkan : Intel FA kernel optimization for split k path vulkan : Host code update for Intel split k FA kernel path selection, fix A770 Linux op test failures vulkan : use symmetric coopMatMulAdd() in flash_attn_decode_phase_1 shader to resolve test op failre on A770 Linux with 26.2.3 mesa driver vulkan : fix editorconfig issue in flash_attn_decode_phase_2.comp Co-authored-by: Liu, Russell russell.liu@intel.com Website: https://llama.app
Release b11110 · ggml-org/llama.cpp
mtmd: add various sanity checks (#29276) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49308522 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11109 · ggml-org/llama.cpp
metal : gate mul_mm_id src1 rescale behind ggml_prec (#29029) metal : gate mul_mm_id src1 rescale behind ggml_prec Assisted-by: Claude Fable 5.1 ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32 cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32 fix supports_op to return false for failing backends when the specified src1 precision is f32 Assisted-by: Claude Fable 5.1 Co-authored-by: yomaytk yoshimura.masashi.frbs@gmail.com Website: https://llama.app Attestations:
Release b11108 · ggml-org/llama.cpp
ggml : IQ1_M build prefix sums once per block (#28706) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49294443 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11105 · ggml-org/llama.cpp
jinja: use const for statement::execute and ::visit (#29271) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49279983 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11104 · ggml-org/llama.cpp
server: Add support for binding to multiple addresses (#28690) Add support for binding llama-server to multiple addresses Assisted-by: Codex remove redundant thread handler make it clear about overlapping addr reject --port 0 with multiple tcp addr improve arg handler nits fix test nits 2 nits nits 2 Co-authored-by: Xuan Son Nguyen son@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49267001 macOS/iOS: macOS Apple Silicon (arm64)
Release b11103 · ggml-org/llama.cpp
spec : support DFlash for HunyuanOCR (#28890) model : add DFlash layer-input taps for HunyuanVL DFlash speculative decoding needs the target graph to expose the residual stream entering each layer (res->t_layer_inp[il]) - the draft model reads those tensors to build its cross-context. Qwen3 and the other DFlash-capable targets register them, but the Hunyuan graphs do not, so serving a DFlash draft against a HunyuanOCR target aborts during the first graph build: GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")
Release b11102 · ggml-org/llama.cpp
convert: add MiMo-V2.6 support (#29257) convert: add MiMo-V2.6 support Hoist the K3 mxfp4 conversion repack into base.py so it can be reused Remove decoder from mmproj convert Update conversion/mimo.py fix: use autoparser Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-by: Piotr Wilkin piotr.wilkin@syndatis.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49229124 macOS/iOS: macOS Apple Silicon (arm64)
Release b11101 · ggml-org/llama.cpp
cmake : allow repeated find_package calls for llama (#29228) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49213556 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11100 · ggml-org/llama.cpp
chat : Fix Muse Glimmer tool-call first parser error (#29242) Fix Muse Glimmer tool-call first parser error Add test to verify Condense patterns remove test for trigger patterns Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49158756 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11099 · ggml-org/llama.cpp
ci : publish snapdragon builds in release workflow (#29007) The snapdragon CI builds packages only to feed the QDC device tests, so Hexagon NPU binaries never reached the releases page. Build both targets in release.yml and attach them as release assets. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49151847 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11097 · ggml-org/llama.cpp
opencl: add A8 Q4_0 non-MoE dp4a binary kernel (#29055) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49141307 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11096 · ggml-org/llama.cpp
ci : update Level Zero SDK to v1.33.1 and enable the L0/oneDNN CMake
Release b11095 · ggml-org/llama.cpp
hexagon: new HMX-optimized GATED_DELTA_NET (#29199) hex-gdn: start putting together HMX support for GDN hex-gdn: working hmx but not-pipelined and slow for now hex-gdn: re-write vtcm layout handling and prep for pipelining hex-gdn: starting to pipeline hmx and dmas hex-gdn: add hvx threading for most pipeline stages hex-gdb: add detailed trace events hex-gdn: vectorize expfs and use aligned hvx reads/writes hex-gnd: vectorize the rest of expf hex-gdn: optimize tail processing (pad partial chunks)
Release b11094 · ggml-org/llama.cpp
vendor : update cpp-httplib to 0.57.1 (#29239) Signed-off-by: Adrien Gallouët adrien@gallouet.fr Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49092442 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11093 · ggml-org/llama.cpp
metal : fix mask bounds in flash attention block pre-pass (#29220) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49087452 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11090 · ggml-org/llama.cpp
cuda: fix sm_70 tile compilation error (#29224) The 5-argument load_ldmatrix added in 1884824 only defines tile, so the Volta tile does not match. See #29222 for details. Building on 1884824, generalize the tile shape of the 5-argument load_ldmatrix from to , so the non-swizzle branch forwards to the 3-argument loade
Release b11081 · ggml-org/llama.cpp
test-llama-archs : make tensor data stdev configurable and improve help (#29133) test-llama-archs : make tensor data stdev configurable Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp test-llama-archs : expand usage and add examples Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp test-llama-archs : fail on unknown args and log usage Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp test-llama-archs : add test run summary Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
Release b11080 · ggml-org/llama.cpp
tests/test-backend-ops : allow regex entries in the -o filter (#29204) tests/test-backend-ops : allow regex entries in the -o filter so far -o only accepted a comma separated list of exact op names or full test case strings. entries that are not plain op names are now treated as regexes matched against the op name (e.g. "MUL_MAT.*"), while plain names keep their exact-matching behavior so that "-o ADD" does not match ADD_EX etc. Assisted-by: pi:llama.cpp/Qwen3.8-27B cont : don't print the FA vec slice log when not needed
Release b11078 · ggml-org/llama.cpp
args: add env vars for temperature, top-p, min-p and penalties (#27380) Allow configuring --temp, --top-p, --min-p, --repeat-penalty, --presence-penalty and --frequency-penalty via LLAMA_ARG_* so llama-server can be fully controlled from an EnvironmentFile (e.g. systemd on Debian). Use llama-gen-docs to regenerate the readme files. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49020823 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
Release b11077 · ggml-org/llama.cpp
server : do not forward --api-key-file to router-spawned child instances (#28938) In router mode, authentication belongs to the router. unset_reserved_args() already unset LLAMA_API_KEY, but did not unset LLAMA_ARG_API_KEY_FILE. When --api-key-file was passed, children re-validated against file keys only, causing clients using --api-key to 401 on chat completions (#28820). In addition, router internal calls without auth headers (such as
Release b11076 · ggml-org/llama.cpp
tests : remove stale comment (#29140) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48998612 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11075 · ggml-org/llama.cpp
ggml-metal : simplify fusion pattern op list declaration (#29206) ggml-metal : derive non-empty fusion ops from ops_all Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp ggml-metal : drop _all suffix from fusion op pattern vectors Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48991487 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11074 · ggml-org/llama.cpp
json: Fixed json enum handling (#28518) Fixed json enum handling Added common_json_value handling for enum values. Added tests/test-json.cpp to cover testing of some aspects of common_json. Removed tests as requested. Applied recommended style and simplification Simplified by delegating enum constructor to the constructor of the underlying type Matched style of surrounding templating code Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48980281 macOS/iOS: macOS Apple Silicon (arm64)
Release b11073 · ggml-org/llama.cpp
sycl : coalesce MKL-FA softmax loads instead of one work-item per row (#28918) sycl : coalesce MKL-FA softmax loads instead of one work-item per row better human readable variable name Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48971633 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11071 · ggml-org/llama.cpp
ci : Upgrade CUDA to 13.4 for Ubuntu CUDA Release Builds (#29202) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48951751 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11070 · ggml-org/llama.cpp
hexagon: overhaul of buffer and DMA handling to support 64bit mappings + improvements (#29197) hex-dma64: enable support extended buffer mappings and 64bit dma hex-dma64: expand binary ops to support more DMA scenarios hex-dma64: add binary-ops.h hex-dma64: add --hex-dma64 to run.py and fix minor issues hex-dma64: update SSM_CONV to use dma with proper support for 64bit hex-ops: remove obsolete gate for % 128 in binary ops hex-l2: dont check weight tensors against dirty ranges hex-dma64: most binary ops now support dma
Release b11069 · ggml-org/llama.cpp
cuda : tune MMVQ to MMQ crossover for SM70 (Volta) (#28912) tune MMVQ to MMQ crossover for SM70 (Volta) Signed-off-by: Yangyu Chen cyy@cyyself.name Apply suggestion from @JohannesGaessler Apply suggestion from @JohannesGaessler Apply suggestion from @JohannesGaessler Signed-off-by: Yangyu Chen cyy@cyyself.name Co-authored-by: Johannes Gäßler
Release b11068 · ggml-org/llama.cpp
metal : fix deprecation warnings from macOS 27 SDK (#29136) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48890258 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11067 · ggml-org/llama.cpp
webgpu : add fused gdn + cpy (#28976) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48873164 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11065 · ggml-org/llama.cpp
CUDA: tune FA for Gemma 4 on Ampere or newer (#29152) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48802880 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11064 · ggml-org/llama.cpp
metal : support arbitrary hc in dsv4_hc_pre (#29169) the dsv4_hc_pre kernels hardcoded hc = 4 via a constexpr used with simd_shuffle, so the op was rejected by supports_op for any other hc and fell back to CPU. Kimi-K3 uses dsv4_hc_pre with hc equal to the number of banked checkpoints in the cross-layer residual stack, which grows with the layer index. pass n_hc as a function constant (FC_DSV4_HC) with per-n_hc pipeline variants, and loop over it in both pre kernels with direct loads
Release b11063 · ggml-org/llama.cpp
common/peg : handle invalid utf-8 sequences in the AST (#29161) common/peg : handle invalid utf-8 sequences in the AST cont : return maximal subpart per Unicode recommendations cont : remove strict argument Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48751230 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11062 · ggml-org/llama.cpp
CUDA: enable sparse fa for qwen4 (#28770) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48739244 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11060 · ggml-org/llama.cpp
fix(mamba) : make time-step projection input contiguous (#28832) mamba : make time-step projection input contiguous Assisted-by: ChatGPT mamba : skip contiguous copy after normalization Assisted-by: ChatGPT Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48722326 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11059 · ggml-org/llama.cpp
metal: add F16 input to the FWHT (#29094) metal: add F16 input to the FWHT The Metal FWHT kernel accepts F32 input only. This change makes the source type a template parameter, so the kernel reads an F16 source directly instead of requiring a converted copy. The F32 instantiations are unchanged. The pipeline name now carries the source type, and supports_op accepts an F16 src1 for the Hadamard hint at the four sizes the kernels cover. Every other F16 src1 path still goes through ggml_metal_supports_mul_mat_op.
Release b11058 · ggml-org/llama.cpp
chat : fix gemma4 required tool grammar (#29115)
Release b11056 · ggml-org/llama.cpp
hexagon: enable I32 GET_ROWS (#29116) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48662388 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11055 · ggml-org/llama.cpp
hexagon: add support for GEGLU_QUICK (#29114) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48660160 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11054 · ggml-org/llama.cpp
hexagon: enable support for TOP_K op (#29113) hexagon: enable support for TOP_K op hex-topk: thread single-row TOP_K, raise VTCM-based size cap hex-topk: fix TOP_K mdev row partitioning hex-topk: optimize TOP_K large-row selection hexagon: clean up comment formatting hex-docs: update TOP_K support listings Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48657197 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
Release b11053 · ggml-org/llama.cpp
server : improve startup log messages (#29125) server-models : show source per model in log Show [source] tag (preset/models_dir/cache) per model instead of cryptic * marker Show HF hub cache path in the 'Loaded cached model presets' log Add hf_cache::get_cache_dir() public accessor Assisted-by: pi:llama.cpp/Qwen3.8-27B cont : pad log Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48640116 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED m
Release b11052 · ggml-org/llama.cpp
json-schema : accept escaped hyphen in regex patterns (#29127) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48635641 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11050 · ggml-org/llama.cpp
metal : fix FA support checks (#29122) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48626162 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11049 · ggml-org/llama.cpp
test-llama-archs : generate dummy test vocab (#29084) Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48623939 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11048 · ggml-org/llama.cpp
metal : support qwen4exp hc ops (#29000) Add support for the new DSV4 HC op variants used by qwen4exp: hc_pre with per-element sigmoid gate (gated variant) hc_post with identity mixing (comb == nullptr) Assisted-by: pi:llama.cpp/Qwen3.8-27B Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48621858 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XC
Release b11047 · ggml-org/llama.cpp
cuda : fix CUB argsort corruption caused by in-place keys (#28389) argsort_f32_i32_cuda_cub called the one-shot DeviceRadixSort::SortPairs API with d_keys_in == d_keys_out (temp_keys, temp_keys). CUB's internal double-buffer ping-pong requires distinct key buffers: with aliased buffers the sort partially overwrites its own input mid-pass and emits a corrupted permutation, surfacing as intermittent garbage indices (e.g. backend top_k over a 248k-column vocab on Maxwell/CUDA 12.5/CCCL 2.x,
Release b11046 · ggml-org/llama.cpp
opencl: add support for bin kernel flash_attn_f32_f16_bin (#29046) opencl: add flash_attn_f32_f16_bin opencl: guarded prefill fa Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48577094 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11045 · ggml-org/llama.cpp
hexagon: add ROLL op support (#29105) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48565469 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11044 · ggml-org/llama.cpp
hexagon: im2col update (#29103) ggml-hexagon: accept 1D and padded IM2COL ops ggml-hexagon: make pure-DDR IM2COL kernel is_2D-aware ggml-hexagon: extend IM2COL DMA patch-embed fast path to 1D ggml-hexagon: add blocked-staging general IM2COL DMA kernel Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48558320 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11043 · ggml-org/llama.cpp
hexagon: HMX flash-attention head_dim padding (support DK=DV=72) (#26539) Allow HMX flash-attention to run with head_dim not a multiple of 64 (e.g. SigLIP head_dim=72), by operating on DK/DV rounded up to 64 with zero-filled tail lanes. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48546365 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11042 · ggml-org/llama.cpp
opencl: add bin kernel kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a8_bin (#28678) opencl: add A8 Q6_K non-MoE binary kernel opencl: fix layout compatibility Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48517412 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11040 · ggml-org/llama.cpp
ggml : check for allocation failures to prevent crashes (#28149) ggml : check for allocation failures to prevent crashes wording Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48473967 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11039 · ggml-org/llama.cpp
Model-Saver: Write the SWA pattern, 15 more architectures roundtrip (#29042) llama: read the SWA pattern as a period or a per-layer array Add llama_model_base::load_swa_pattern(), which reads sliding_window_pattern either as one flag per layer or as a period expanded by set_swa_pattern(), and use it in every loader that reads the key as a period. These loaders silently ignored an array and applied their default period, although the converters of olmo2, gemma3n and exaone4 write arrays.
Release b11037 · ggml-org/llama.cpp
ggml-webgpu: fix supports_op condition for GET_ROWS (#28978) fix get_rows vec4 handling Add src strides checking to vec4_aligned of get_rows and the new test case. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48438627 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11036 · ggml-org/llama.cpp
ggml : handle graph buffer reservation failure (#26070) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48431150 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11035 · ggml-org/llama.cpp
vulkan: add IQ3_S MMQ matmul kernels (#28822) vulkan: add IQ3_S MMQ matmul kernels Make block_a_to_shmem do 2-byte loads (110 bytes is divisible by 2) Align the check, IQ3_S is also using K tile size Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48426453 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework
Release b11034 · ggml-org/llama.cpp
vocab : add ufakzeka pre-tokenizer (#29033) vocab : add ufakzeka pre-tokenizer vocab : move ufakzeka to the models list and regenerate the hash mapping Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48421591 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11033 · ggml-org/llama.cpp
cmake : fix build when GGML_CPU=OFF and GGML_CUDA=ON (#29026) fix: build fails when GGML_CPU=OFF and GGML_CUDA=ON fix: eol in examples/convert-llama2c-to-ggml/CMakeLists.txt file Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48413612 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11030 · ggml-org/llama.cpp
ci : bump android-actions/setup-android to 4.0.4 (#29065) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48395907 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11029 · ggml-org/llama.cpp
vulkan: raise the hoisted row-id limit for mul_mat_id from 256 to 1024 experts (#28501) vulkan: raise the hoisted row-id limit for mul_mat_id to 512 experts The expert-count shader (count_experts.comp) sizes its shared arrays with BLOCK_SIZE, which is 256. Because of that, row-id hoisting is switched off for any model with more than 256 experts, and every mul_mat_id workgroup has to rescan the whole ids tensor on its own. Qwen3.8-Flash-Next has 512 experts and was quietly running on that slow path.
Release b11028 · ggml-org/llama.cpp
ci : add missing evict-old-files (#29041) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48252257 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11027 · ggml-org/llama.cpp
rpc : skip ACCEL devices (#29020) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48232585 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11026 · ggml-org/llama.cpp
model : skip gate_up_exps if TENSOR_SKIP is set (#29014) required for qwen35moe if MTP tensors are fused but not loaded Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48194938 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11025 · ggml-org/llama.cpp
model : extend Nemotron MTP support (#29018) first fix removed unnecessary declarations Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48188464 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11024 · ggml-org/llama.cpp
openvino : Update OpenVINO to 2026.4;fix clangd,MSVC warnings; (#29009) Update to openvino-2026.4 Update OV docs ggml-openvino : fix clangd and MSVC warnings fix int to ptr cast, more internal linkage enforcement, and avoiding duplicate switch case Co-authored-by: Mostafa Faheem mostafaaafaheem@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48180864 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
Release b11022 · ggml-org/llama.cpp
vulkan: split buffers and debug code into separate files, add shared headers (#28732) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48170206 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11020 · ggml-org/llama.cpp
chat : add message delimiters to the DeepSeek V3.2/V4 parser (#29008) chat : add message delimiters to the DeepSeek V3.2/V4 parser Assisted-by: Claude Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48164235 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11019 · ggml-org/llama.cpp
gguf : align the data section relative to the GGUF start, not the file (#28993) gguf : align the data section relative to the GGUF start, not the file gguf_init_from_file_ptr reads a GGUF from the current file position, but padded the data section from file offset 0, so a GGUF embedded at an offset that is not a multiple of the alignment loaded without error and returned wrong tensor data. Also adds llama_adapter_lora_init_from_file_ptr, and disables mmap with a warning
Release b11018 · ggml-org/llama.cpp
sycl : fix the B70 mem allocate error when >19.3GB (#28953) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48150237 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11017 · ggml-org/llama.cpp
vulkan: skip unneeded MoE work in mul_mm coopmat1 path (#25483) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48144237 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11016 · ggml-org/llama.cpp
sycl: ssm_conv: fuse the SiLU epilogue into the ssm_conv kernel (#28929) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48138242 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11015 · ggml-org/llama.cpp
opencl: fix various warnings (#28984) opencl: fix warnings opencl: fix warnings for non adreno Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48132480 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11012 · ggml-org/llama.cpp
vulkan: support qwen4exp hc ops (#28988) vulkan: support qwen4exp hc ops fix stale comment [no-ci] Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48109146 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11011 · ggml-org/llama.cpp
[SYCL] Fix function signature for ggml_backend_sycl_split_buffer_type (#28981) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48082811 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11010 · ggml-org/llama.cpp
vulkan: work around NV bug with argsort_large.comp (#28975) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48059150 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)
Release b11009 · ggml-org/llama.cpp
TP: fix split state and granularity for fused QKV gemma4, qwen35 (#28965) model: calculate split states for attn_qkv from n_head * n_embd_head_k required for gemma4 with --fuse-qkv, where n_embd is 5376 but Q is 8192. model: handle fused full attention layers for qwen35/qwen35moe model: add TODO: [TAG_SPLIT_QGATE_QWEN] Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/48008214 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11007 · ggml-org/llama.cpp
Enable CUDA graph for MTP draft (#28549) Improve CUDA graph usage for MTP Rename field Address review feedback Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47982528 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux:
Release b11006 · ggml-org/llama.cpp
hexagon: Support for K-Quants Q4_K and Q6_K (#28994) implement q6k/q4k kernels Squashed from: feat: implement q6k kernel hex-q6k: improve unpack accuracy hex-q4_k: add support for Q4_K kernels Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47977588 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11005 · ggml-org/llama.cpp
hexagon: accept the zeroed rope probe in supports_op (#28995) llama probes weight placement with a rope where all params are 0, so rejecting n_dims == 0 or freq_base == 0 puts rope_freqs on the CPU. That splits the decode graph at every full-attention layer (gemma-4-E2B: 5 splits instead of 2). Assisted-by: Claude Opus 5 Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47971289 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)
Release b11003 · ggml-org/llama.cpp
model : add support for HrmTextForCausalLM (DFM Mimir 1B) (#27625) model : add support for HrmTextForCausalLM (DFM Mimir 1B) HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions. conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor
Release b11002 · ggml-org/llama.cpp
CUDA/HIP: improve access patterns in im2col (#28013) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47937267 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)