Authored Library of Layers transformer architecture allowing dynamic reordering of transformer blocks and discovered quantization time-bomb loss spikes in BF16 training.
Referenced in coverage
“Discovered delayed loss spikes driven by quantization error in the LM head during BF16 training.”
“Authored Library of Layers transformer architecture allowing dynamic reordering of transformer blocks and discovered quantization time-bomb loss spikes in BF16 training.”