Referring to the well-replicated finding that the presence of to-be-ignored sound disrupts short-term memory for serially-presented visual items, the irrelevant sound effect (ISE) is an important benchmark finding within cognitive psychology. The ISE has proven useful in evaluating the structure, function and development of short-term memory. This preregistered report focused on a methodological examination of the paradigm typically used to study the ISE and sought to determine whether the ISE can be reliably studied using the increasingly popular method of online testing. Comparing Psychology students tested online, in-person and participants from an online panel, results demonstrated successful reproduction of the key signature effects of auditory distraction (the changing-state effect and the steady-state effect), albeit smaller effects with the online panel. Our results confirmed the viability of online data collection for auditory distraction research and provided important insights for the accumulation and maintenance of high data quality in internet-based experimentation.