https://arstechnica.com/information-technology/2024/06/researchers-upend-ai-status-quo-by-eliminating-matrix-multiplication-in-llms/
Running AI models without floating point matrix math could mean far less power consumption.
The peer review process is way too noisy to be meaningful. Better to just read it and judge the work for yourself.
https://arxiv.org/abs/2406.02528
Or wait for follow up work that can corroborate their claims.
They claim it's more efficient, which presumably means it requires less electricity to run the same process. Which means that if this is legit, the companies that do AI should implement it organically as a way to shave costs. If they have any brains whatsoever.
This is the best summary I could come up with:
Researchers claim to have developed a new way to run AI language models more efficiently by eliminating matrix multiplication from the process.
The technique has not yet been peer-reviewed, but the researchers—Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason Eshraghian—claim that their work challenges the prevailing paradigm that matrix multiplication operations are indispensable for building high-performing language models.
They argue that their approach could make large language models more accessible, efficient, and sustainable, particularly for deployment on resource-constrained hardware like smartphones.
In the paper, the researchers mention BitNet (the so-called "1-bit" transformer technique that made the rounds as a preprint in October) as an important precursor to their work.
According to the authors, BitNet demonstrated the viability of using binary and ternary weights in language models, successfully scaling up to 3 billion parameters while maintaining competitive performance.
Limitations of BitNet served as a motivation for the current study, pushing them to develop a completely "MatMul-free" architecture that could maintain performance while eliminating matrix multiplications even in the attention mechanism.
The original article contains 412 words, the summary contains 177 words. Saved 57%. I'm a bot and I'm open source!
What were the limitations they overcame with BitNet, and what are some unresolved issues mentioned in the reprint?