Post
1085
Applying Hazard and Entropy Analysis to LLMs
Here's an example of a model that behaves perfectly well up to 8k, smoothly increasing its entropy before going into a struggle zone, collapsing, seeing a region of recovery and finally falling down hard at the 16k wall.
Is your model implementation behaving badly like this?
Would you know if it was? 👀
Here's an example of a model that behaves perfectly well up to 8k, smoothly increasing its entropy before going into a struggle zone, collapsing, seeing a region of recovery and finally falling down hard at the 16k wall.
Is your model implementation behaving badly like this?
Would you know if it was? 👀