Heavy winds hinder the execution of essential tasks for Rotary Unmanned Aerial Vehicles (RUAVs) such as mountain rescue operations or civil engineering assessments during typical Northern Irish winters. This study uses Reinforcement Learning (RL) methods to select controller gains, enhancing RUAV stability under challenging wind conditions, employing a Deep Deterministic Policy Gradient (DDPG) agent over conventional and optimal controllers. The proposed DDPG agent enables the controller to be built as a “Black Box” approach, where the agent can adapt to slight changes or model uncertainty in a real system enabling a more robust controller. Simulations carried out on Full State Feedback, Full State Compensator and Linear Quadratic Gaussian controllers tuned by a variety of techniques revealed that RL out-performed conventional manual tuning by 26% and Particle Swarm Optimization by 19% in performance measured in settling time.