mike-ravkine (Mike Ravkine)

posted an update about 14 hours ago

Post

111

Happy 2026 everyone!

I've been busy working on some new ranking/position methodologies and excited to start sharing some results.

Plot legends:

- X = truncation rate (low = good)
- ? = confusion rate (low = good)
- blue bars = average completion tokens (low = good)
- black diamonds = CI-banded performance (high = good)
- cluster squares = models inside this group are equivalent

openai/gpt-oss-120b remains the king in all dimensions of interest: truncation rates, completion lengths and performance. If I had but one complaint it's the reason_effort does not seem to actually work - more on this soon.

Second is a 3-way tie in performance between the Qwen3-235B-2507 we all know and love with an unexpected entrant - ByteDance-Seed/Seed-OSS-36B-Instruct

This is a very capable model and it's reasoning effort controls actually works, but you should absolutely not leave it on the default "unlimited" - enable a sensible limit (4k works well for 8k context length).

Third place is another 3-way tie, this one between Seed-OSS-36B (it straddles the CI boundary between 2nd and 3rd place), Qwen/Qwen3-Next-80B-A3B-Instruct (demonstrating that full attention may be overrated after all and gated is the way to go) and the newly released zai-org/GLM-4.7 which offers excellent across the board performance with some of the shortest reasoning traces I've seen so far.

posted an update about 1 month ago

Post

1091

Applying Hazard and Entropy Analysis to LLMs

Here's an example of a model that behaves perfectly well up to 8k, smoothly increasing its entropy before going into a struggle zone, collapsing, seeing a region of recovery and finally falling down hard at the 16k wall.

Is your model implementation behaving badly like this?

Would you know if it was? 👀

posted an update about 2 months ago

Post

285

sunday = lab day

goal: understand how GGUF compression works - what exactly is being lost?

approach: quantize/dequantize some images and look at error maps

spent 80% of the time tracing down what turns out to be a data distribution assumption: real LLM weights are symmetric and their mean is 0 so our test image MUST retain these properties or the results turn into a kind of nonsense soup where Q5_1 beats Q8

with that issue solved, we have some fun results! from left to right:

- test pattern image (mean value is around 0.01)

- q8 error (almost nothing - some light banding in the gradients)

- q5km error (starting to see the 'blocks' around the circles)

- q4_0 error (this is why q4_1 is 'preferred')

- q3k error. q3k is a really interesting set of trade-offs: it does not have a block-offset so it really leans into the 0-mean assumption HARD, if you violate it locally the results are BAD

- q2k error: q2k has a block-offset so for certain patterns the errors are actually less then q3k (a rather counter-intuitive result)

looking at mxfp4, i-quants and the other stuff that's possible inside gguf remains future work.. aiming to clean up this repo and push it this week, feel free to ping me if you want to play sooner.

reacted to mitkox's post with 🔥 about 2 months ago

Post

4155

I just threw Qwen3-0.6B in BF16 into an on device AI drag race on AMD Strix Halo with vLLM:

564 tokens/sec on short 100-token sprints
96 tokens/sec on 8K-token marathons

TL;DR You don't just run AI on AMD. You negotiate with it.

The hardware absolutely delivers. Spoiler alert; there is exactly ONE configuration where vLLM + ROCm + Triton + PyTorch + Drivers + Ubuntu Kernel to work at the same time. Finding it required the patience of a saint

Consumer AMD for AI inference is the ultimate "budget warrior" play, insane performance-per-euro, but you need hardcore technical skills that would make a senior sysadmin nod in quiet respect.

1 reply

·

reacted to danielhanchen's post with ❤️ about 2 months ago

Post

4375

You can now run Kimi K2 Thinking locally with our Dynamic 1-bit GGUFs: unsloth/Kimi-K2-Thinking-GGUF

We shrank the 1T model to 245GB (-62%) & retained ~85% of accuracy on Aider Polyglot. Run on >247GB RAM for fast inference.

We also collaborated with the Moonshot AI Kimi team on a system prompt fix! 🥰

Guide + fix details: https://docs.unsloth.ai/models/kimi-k2-thinking-how-to-run-locally

replied to their post about 2 months ago

The way the coolers on these cards are designed is VERY unusual - this photo should come in the box! If you're having trouble in a closed case, see if you have space to add an intake on the bottom beside the PCIe edge - if all air is coming from the front like in this pic, the rear blower steals it all!

replied to their post about 2 months ago

I am running 3x140mm 89CFM as intakes on the front and directly underneath the in-blowers of those two 3090FE is the main trick - a dedicated 140mm 140CFM blowing upwards to feed the intake blowers at the front/pcie edge of these coolers. The cross-blower air passes through both cards and then has space to vent their heat on the right.

At 280W load the 'outside' card is bored and sitting at 50C with the blower fans at 50% while the 'inside' card maintains 60-65C with it's blowers closer to 70-80%.

The main thing to be careful about with the FE dual-blower coolers is do not try to pump air into the 'front' (pcie side) of them! I see this configuration on many mining rigs and its fine for air cooled cards but the FEs actually have a blower venting out this pci-slot side so you HAVE to feed them from either the rear or underneath.

When I had the 4-slot bridge installed across them, the rear-feed alone was sufficient.

posted an update about 2 months ago

Post

1442

measuring the information content of a reasoning trace seems like a straightforward reasoning LLM KPI, but how can we achieve this?

what if we keep it simple: gzip the resulting text and take the length of the compressed stream... "compressed bytes of information per output token" becomes the KPI

if we split across correct answers vs incorrect answers vs truncated answers and group by difficulty, a whole new world of analysis becomes not just possible but visually intuitive and almost trivial:

1) what is the model's overall reasoning efficiency? this is the slope of the scatterplot curve segments (there may be more then one..)

2) is the model able to apply more test-time compute towards more difficult variations of the task? the two on the left are not, the two on the right are.

3) when applying more test-time compute, is that compute useful? this is the curvature of the scatterplot trends - the two in the middle are 'losing their mojo' as answers get longer the information content falls down

4) is the model applying multiple approaches to the task? (right) do those approaches change with difficulty?

5) are truncations because we don't have enough context budget (left) or because the model has lost its mind and gone into a repeat loop (middle two) and does this happen across the board (middle left) or only when the problem is more difficult (middle right)

would love to hear your guys feedback on this kind of analysis, is anyone doing similar work?

this approach generates 12 plots per model (one for each task) so quite a bit of data and i've been hesitant to publish it so far, consider this post a toe tip.

replied to their post about 2 months ago

The top high density rack is successfully hitting 65C targets at 1.15kW TDP.

Really impressive coolers on the FEs, there is barely 10mm of space but the sandwiched one still keeps up to unobstructed air coolers.

We are back, baby 🏇

posted an update about 2 months ago

Post

2770

There is no anxiety quite like powering up 2KW of basement compute after rewiring it all. Small bit of trouble with the horizontal 3090 because I misread my motherboard manual, but otherwise so far so good.. Next we see if I've built up enough cooling to hit my target TDP on those 3-slot nvlinked cards especially. The 4-slot bridges are much easier to work with but their prices went bananas and I couldn't acquire a second, so gotta get a little creative with intakes.

5 replies

·

posted an update 2 months ago

Post

188

How many 3090 is too many 3090? Trick question - never enough 🤖.

Working on rewiring for a 5th this weekend so need the quad in 4U to make this happen. Definitely pushing air cooling to its limit but NVLink requires aggressive spacing between the cards.

Fortunately up here in Canada our power is quite cheap, so stacking compute this way works. While being annoying in terms of physical constraint the NVlink bridges really helps squeeze these cards - on smaller models at big batch the difference at -tp 2 can be as much as 50%. It's not bandwidth, it's latency!

1 reply

·

reacted to onekq's post with 👀 2 months ago

Post

2351

Kimi K2 is a bit disappointing by my expectations. It is on a par with Codex mini.

onekq-ai/WebApp1K-models-leaderboard

1 reply

·

posted an update 2 months ago

Post

226

I also like fried chicken and beer but they forgot to invite me for some reason 😔😆

https://www.cnn.com/2025/10/31/tech/south-korea-nvidia-apec-chicken-intl-hnk

1 reply

·

reacted to sergiopaniego's post with 👀 2 months ago

Post

436

On-Policy distillation is trendy! and super useful!

HuggingFaceH4/on-policy-distillation

posted an update 2 months ago

Post

281

Exploring the differences between models in the same family is always extra interesting as we get to isolate what at least on the surface appears to be a single variable - the number of parameters.

But is this really a single variable?

If we pick a specific example (lets say google/gemma-3-4b-it vs google/gemma-3-12b-it vs google/gemma-3-27b-it) we find that this single variable is actually a plethora of differences:

- The hidden_size, intermediate_size and num_hidden_layers are all different: not only are there more layers as the model grows larger, the layers themselves grow larger.
- num_attention_heads and num_key_value_heads are also all different: the larger models split the attention into more groups.

What is the final outcome of all these differences? This answer turns out to be task specific but has 4 archetypes worth noting:

Cars, Objects, Shapes: Everyone passes - smaller can do the job.

Arithmetic, Brackets: Everyone fails - model architecture is ill-suited for this task

Boolean, Sort: There's a threshold of "big enough" you have to cross, any smaller and it fails.

Objects, Shuffle: Bigger models demonstrate robust scaling in performance especially as complexity grows.

The practical lesson here is that since size is not really a single variable, *bigger is not necessarily better*! There's a surprising number of domains where you can 'downgrade' to a smaller model and gain improvements in latency and throughput without sacrificing your downstream task performance.

reacted to onekq's post with 👍 2 months ago

Post

3666

Context rot is such a catchy phrase, but the problem has been identified 2+ years ago, called attention decay.
Lost in the Middle: How Language Models Use Long Contexts (2307.03172)

I spotted the same problem in coding tasks, and documented in my book (https://www.amazon.com/dp/9999331130).

Why did this problem become hot again? This is because many of us thought the problem has been solved by long context models, which is not true.

Here we were misled by benchmarks. Most long-context benchmarks build around the QA scenario, i.e. "finding needle in haystack". But in agentic scenarios, the model needs to find EVERYTHING in the haystack, and just can't afford enough attention for this challenge.

6 replies

·

posted an update 2 months ago

Post

330

Spooky season is coming 👻 and there's nothing scarier then poor LLM evaluation results, right?

The ReasonScape m12x dataset, explorer and leaderboard been updated with 12 additional models to bring up the total to *54* models and over *7.1B thinking tokens* and the groups filter has been split into two: family and size, which lets us take a peek at our first small-models-only reasoning result!

The top performer, Qwen/Qwen3-4B-Thinking-2507 has an enormous overthink problem so I would actually give this crown to Qwen/Qwen3-4B-Instruct-2507 - simply ask it to think Step-by-Step to bring out it's latent hybrid-COT behavior and enjoy stellar performance at a fraction of the tokens!

Right below the Qwen3-4B set and sitting just above Qwen/Qwen3-1.7B is a model from a family that is somewhat underrated: tencent/Hunyuan-4B-Instruct Note that vllm 0.11.0 has trouble with Hunyuan dense models, it seems to only support the MoE variants, I used 0.10.2 for my evaluations.

In 7th place and worth a mention is HuggingFaceTB/SmolLM3-3B which is a very efficient smaller thinker, but it's achilees heel was the strawberry test: it fails both letter counting and word sorting.

The last model worth discussing in this context is google/gemma-3-4b-it which is obviously not a reasoning model but when asked to think step-by-step it demonstrated tolerable performance across several tasks with incredibly low token utilization compared to most of these little guys.

Would love to hear from the community! Do you use any of these models in your day-to-day? Did I miss any? Let me know!

Full leaderboard @ https://reasonscape.com/m12x/leaderboard/

reacted to DmitryRyumin's post with 🔥 2 months ago

Post

4810

🚀🤖🌟 New Research Alert - ICCV 2025 (Oral)! 🌟🤖🚀
📄 Title: Variance-based Pruning for Accelerating and Compressing Trained Networks 🔝

📝 Description: The one-shot pruning method efficiently compresses networks, reducing computation and memory usage while retaining almost full performance and requiring minimal fine-tuning.

👥 Authors: Uranik Berisha, Jens Mehnert, and Alexandru Paul Condurache

📅 Conference: ICCV, 19 – 23 Oct, 2025 | Honolulu, Hawai'i, USA 🇺🇸

📄 Paper: Variance-Based Pruning for Accelerating and Compressing Trained Networks (2507.12988)

🚀 ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers

🚀 Added to the Efficient Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/efficient-learning.md

📚 More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🔍 Keywords: #VarianceBasedPruning #NetworkCompression #ModelAcceleration #EfficientDeepLearning #VisionTransformers #AI #ICCV2025 #ResearchHighlight

posted an update 2 months ago

Post

2528

Spatial reasoning is a domain where LLMs struggle surprisingly hard.

A new paper, "Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models" compares performance on a handful of spacial reasoning tasks and finds all SOTA LLMs breaking down and hallucinating their faces off when the grids get large.

https://arxiv.org/html/2510.20198v1

The word search task is especially revealing: notice the bias towards detecting "horizontal" while struggling with "vertical" - LLMs only understand simple, linear relationships.. add a stride for 2D and it's basically over.

posted an update 2 months ago

Post

290

"Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinking"

https://arxiv.org/abs/2510.07880

> Our analysis reveals two major patterns for open-weight thinking models -- Explorer and Late Landing. This finding provides evidence that over-verification and over-exploration are the primary drivers of overthinking in LLMs. Grounded in thought structures, we propose a utility-based definition of overthinking, which moves beyond length-based metrics. This revised definition offers a more insightful understanding of LLMs' thought progression, as well as practical guidelines for principled overthinking management.

Really cool paper! Unfortunately no code, may need to attempt to recreate my own TRACE 🤔

Mike Ravkine PRO

AI & ML interests

Recent Activity

Organizations

Mike Ravkine PRO

AI & ML interests

Recent Activity

Organizations

mike-ravkine's activity