Summary In this study, we propose an extremum-seeking approach for the approximation of optimal control problems for a class of unknown nonlinear dynamical systems. The technique combines a phasor extremum-seeking controller with a reinforcement learning strategy. The learning approach is used to estimate the value function of an optimal control problem of interest. The phasor extremum-seeking controller implements the approximate optimal controller. The approach is shown to provide reasonable approximations of optimal control problems without the need for a parameterization of the nonlinear system’s dynamics. A simulation example is provided to demonstrate the effectiveness of the technique.