Symptoms of vestibular dysfunction such as dizziness and vertigo are common after sports-related concussions (SRC) and associated with a worse outcome and a prolonged recovery. Vestibular dysfunction after SRC can be because of an impairment of the peripheral or central neural parts of the vestibular system. The aim of the present study was to establish the cause of vestibular impairment in athletes with SRC who have persisting post-concussive symptoms (PPCS). We recruited 42 participants—21 athletes with previous SRCs and PPCS ≥6 months and 21 healthy athletic age- and sex-matched controls—who underwent symptom rating, a detailed test battery of vestibular function and 7T magnetic resonance imaging with diffusion tensor imaging (DTI) and diffusion kurtosis imaging (DKI) of cerebellar white matter tracts, and T1-weighted imaging for cerebellar volumetrics. Vestibular dysfunction was observed in 13 SRC athletes and three controls (p = 0.001). Athletes with vestibular dysfunction reported more pronounced symptoms on the Dizziness Handicap Inventory (DHI; p < 0.001) and the Hospital Anxiety and Depression Scale (HADS; p < 0.001). No significant differences in DTI metrics were found, while in DKI two metrics were observed in the superior and/or inferior cerebellar tracts. Cerebellar gray and white matter volumes were similar in athletes with SRC and controls. Compared with controls, pathological video head impulse test results (vHIT; p < 0.001) and cervical vestibular evoked myogenic potentials (cVEMP; p = 0.002) were observed in athletes with SRC, indicating peripheral vestibular dysfunction and specifically suggesting injury to the inferior vestibular nerve. In athletes with persisting symptoms after SRC, vestibular dysfunction is associated with injury to the inferior vestibular nerve.