Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?
By Razan El Mais · Paper · cs.LG
Differentially Private Stochastic Gradient Descent (DP-SGD) is a leading approach for privacy-preserving fine-tuning of large language models (LLMs). Many decoder-only LLMs employ weight tying between input and output embeddings, a design choice originally introduced for paramete
Tags: Cs.lg