Microsoft LASERs away LLM inaccuracies
During the January Microsoft Research Forum, Dipendra Misra, a senior researcher at Microsoft Research Lab NYC and AI Frontiers, explained how Layer-Selective Rank Reduction (or LASER) can make large language models more accurate.
With LASER, researchers can “intervene” and replace one weight matrix with an approximate smaller one. Weights are the contextual connections models make. The heavier the weight, the more the model relies on it. So, does replacing something with more correlations and contexts make the model less accurate? Based on their test results, the answer, surprisingly, is no.
“We are doing intervention using LASER on the LLM, so one would expect that the model loss should go up as we are doing more approximation, meaning that the model is going to perform bad, right, because we are throwing out information from an LLM, which is trained on large amounts of data,” Misra said. “But to our surprise, we find that if the right type of LASER intervention is performed, the model loss doesn’t go up but actually goes down.”
Misra said his team successfully used LASER on three different open-source models: RoBERTa, Llama 2, and Eleuther’s GPT-J. He said, at times, model improvement increased by 20 to 30 percentage points. For example, the performance of GPT-J for gender prediction based on biographies went from 70.9 percent accuracy to 97.5 percent after a LASER intervention.
During the January Microsoft Research Forum, Dipendra Misra, a senior researcher at Microsoft Research Lab NYC and AI Frontiers, explained how Layer-Selective Rank Reduction (or LASER) can make large language models more accurate. With LASER, researchers can “intervene” and replace one weight matrix with an approximate smaller one. Weights are…
Recent Posts
- Network specialist debuts free tool that promises to solve VPN and ZTNA connectivity issues for good
- This pink Nvidia RTX 4060 GPU features adorable cats and pugs, oh my!
- Peacock may increase the cost of its monthly and annual plans again
- Is your MSI Claw struggling with performance issues? Download the latest GPU and BIOS update ASAP
- Quordle today – hints and answers for Wednesday, May 1 (game #828)
Archives
- May 2024
- April 2024
- March 2024
- February 2024
- January 2024
- December 2023
- November 2023
- October 2023
- September 2023
- August 2023
- July 2023
- June 2023
- May 2023
- April 2023
- March 2023
- February 2023
- January 2023
- December 2022
- November 2022
- October 2022
- September 2022
- August 2022
- July 2022
- June 2022
- May 2022
- April 2022
- March 2022
- February 2022
- January 2022
- December 2021
- November 2021
- October 2021
- September 2021
- August 2021
- July 2021
- June 2021
- May 2021
- April 2021
- March 2021
- February 2021
- January 2021
- December 2020
- November 2020
- October 2020
- September 2020
- August 2020
- July 2020
- June 2020
- May 2020
- April 2020
- March 2020
- February 2020
- January 2020
- December 2019
- November 2019
- December 2011