Various studies report that Tb³⁺/Yb³⁺ co-doped materials can split one UV or 488 nm (visible) photon in two near infrared (NIR) photons at 980 nm by an energy-transfer process involving one Tb³⁺ and two Yb³⁺ ions. Additionally, it was demonstrated that Ag multimers can provide an efficient optical sensitizing effect for rare earth ions (RE³⁺ ions), resulting in a broadband enhanced excitation, which could have a significant technological impact, overcoming their limited spectral absorptions and small excitation cross sections. However, a systematic and detailed investigation of the down-conversion process enhanced by Ag nanoaggregates is still lacking, which is the focus of this paper. Specifically, a step by step analysis of the energy-transfer quantum-cutting chain in Ag-exchanged Tb³⁺/Yb³⁺ co-doped glasses and glass-ceramics is presented. Moreover, the direct Ag-Yb³⁺ energy-transfer is also considered. Results of structural, compositional, and optical characterizations are given, providing quantitative data for the efficient broadband Ag-sensitization of Tb³⁺/Yb³⁺ quantum cutting. A deeper understanding of the physical processes beneath the optical properties of the developed materials will allow a wiser realization of more efficient energy-related devices, such as spectral converters for silicon solar cells and light-emitting devices (LEDs) in the visible and NIR spectral regions.