In this study, we propose an extremum-seeking approach for the approximation of optimal control problems for unknown nonlinear dynamical systems. The technique combines a phasor extremum seeking controller with an reinforcement learning strategy. The learning approach is used to estimate the value function of an optimal control problem of interest. The phasor extremum seeking controller implements the approximate optimal controller. The approach is shown to provide reasonable approximations of optimal control problems without the need for a parameterization of the nonlinear control system. A simulation example are provided to demonstrate the effectiveness of the technique