The Internet of Drones (IoD) provides a coordinated access to Unmanned Aerial Vehicles (UAVs) that are referred as drones. The on-going miniaturization of sensors, actuators, and processors with ubiquitous wireless connectivity makes drones to be used in a wide range of applications ranging from military to civilian. Since most of the applications involved in the IoD are real-time based, the users are generally interested in accessing real-time information from drones belonging to a particular fly zone. This happens if we allow users to directly access real-time data from flying drones inside IoD environment and not from the server. This is a serious security breach which may deteriorate performance of any implemented solution in this IoD environment. To address this important issue in IoD, we propose a novel lightweight user authentication scheme in which a user in the IoD environment needs to access data directly from a drone provided that the user is authorized to access the data from that drone. The formal security verification using the broadly-accepted Automated Validation of Internet Security Protocols and Applications (AVISPA) tool along with informal security analysis show that our scheme is secure against several known attacks. The performance comparison demonstrates that our scheme is efficient with respect to various parameters, and it provides better security as compared to those for the related existing schemes. Finally, the practical demonstration of our scheme is done using the widely-accepted NS2 simulation.