This paper proposes an multiusability reinforcement learning controller design method in low-level control of multi-axis rotor configuration unmanned aerial vehicle (UAV). In other reinforcement learning applications, the trained neural network controllers were designed to a model with specific dynamic properties and it usually cannot play a role while the system configuration changed. To fix this problem, we use a six-degree-of-freedom (6-DOF) dynamic model as the prototype of various vehicles to learn the force and moment required for stability during training, and employs a deep neural network directly mapping the states to the actuator commands. The force and moment can be adjusted according to different conditions and applied to different axis multi-rotor. Using reinforcement learning, this paper demonstrate the flight control of quadrotor and hexrotor using trained policy in simulator to present the stability on different multi-rotor structure, and compared the performance with the one previously introduced by trained quadrotor.