This article presents an obstacle avoidance framework for unmanned aerial vehicles (UAVs), with a focus on providing safe and stable local navigation in critical scenarios. The framework is based on enhanced artificial potential field (APF) concepts, and is paired with a nonlinear model predictive controller (NMPC) for complete local reactive navigation. This paper will consider a series of additions to the classical artificial potential field that addresses UAV-specific challenges, allows for smooth navigation in tightly constrained environments, and ensures safe human-robot interactions. The APF formulation is fundamentally based on using raw LiDAR pointcloud data as input to decouple the safe robot navigation problem from the reliance on any map or obstacle detection software, resulting in a very resilient and fail-safe framework that can be used a san additional safety layer for any 3D-LiDAR equipped UAV in any environment or mission scenario. We evaluate the scheme in both laboratory experiments and field trials, and also placea large emphasis on realistic scenarios for safe human-robot interactions.