Defective rail wheels can cause severe damages to both track and vehicle components, which in worst case can lead to derailments with extensive losses. Wheel impact detection systems are intended to prevent railway damages and its related losses, through recognition of wheel defects and generation of alarms. At commissioning of a wheel impact detection system in Sweden, the commissioning study showed that the system was reliable, without generating any false alarms. However, during operation some of the system's alarms resulted in No Fault Found (NFF) events at subsequent manual inspections. Hence, a verification study was launched in order to determine if the cause of the NFF events were false alarms generated by the system, or the inability of subsequent manual inspection to replicate the system's test result. The verification study was expected to support an exclusion of the manual inspection, which was perceived to be the cause of the NFF events. However, an exclusion of the manual inspection would require the wheel impact detection system to be highly reliable, since it would be the primary foundation for decisions about both operation and maintenance. This study reports on vital experiences from the verification study and links these to experiences from other countries. The experiences highlight the importance of data accuracy and appropriate information management, in order to achieve the potential benefits and avoid the drawbacks of condition monitoring.