Rotor imbalance in helicopters can lead to a flight quality degradation and a significant reduction in the lifespan of affected components. Considering the fundamental importance of safety and helicopter control being dependent on rotors, the propagation of this issue can result in undesirable events and catastrophic failures. Therefore, early detection of mass imbalance and misalignment plays a crucial role in reducing events and costs, and researchers in this field have recognized its significance. This paper focuses on experimental studies conducted on a coaxial rotor model helicopter to detect the presence of mass imbalance, misalignment, and a combination of them. A total of 320 tests were performed for eight conditions, including healthy, misalignment in the blades, and mass imbalances at various points on the blade's tip and center, within a rotational speed range of 880 to 1050 rotations per minute. Based on the time and frequency response of the system, signal features were extracted in the form of time-domain, frequency-domain, and time-frequency (using wavelet transform) graphs. In the next step, using Principal Component Analysis (PCA) in conjunction with the Ant Colony Optimization algorithm, 15 features containing more information were selected and inputted for classification into a neural network. According to the empirical evaluation, the proposed algorithm classifies and identifies the faults with a high-efficiency ratio. Thus, it can be utilized in condition-based maintenance of the rotorcrafts.