Kian Kyars

Yes, it's done for each transformer block in an LM because each transformer block has different attention heads. If you do it for only one transformer block across all blocks, then you don't get the same representation.

I think I got lost around the standard inference versus Kv caching section because I couldn't understand the matmuls happening based on each flashing repetition of those yellow blocks. But perhaps I just need to go through the blog post once again to try to better understand it.

I didn't understand the explanation

This technique is not a better lesson pilled at all. Waste of time when the model will just learn to do this anyways.

thanks!

makes sense

For the reasoning mid-training we used the ChatML template (so no system prompt), for SFT we use SmolLM3's final chat template.

ChatML does have a system prompt, can you please elaborate?

is it not possible to do on-policy distillation?

nice thoughts

tied embeddings were just used because llama uses it or was there an ablation?

Let's do this with Muon.