This study concerns the cooling of a mechanical resonator to the ground-state in the unresolved sideband regime theoretically. To this aim, the optical modes of an optomechanical cavity are coupled with an atomic ensemble and an auxiliary cavity. Then, a coherent feedback loop is applied via a controllable beam splitter, which reflects a fraction of the output field to the input mirror of the optomechanical cavity. Considering the proposed feedback scheme, the optical response of the cavity is analyzed for weak optomechanical coupling to obtain the rate equations. Utilizing the electromagnetically-induced-transparency-like shape of the fluctuation spectrum of the optical force, optimal cooling conditions are calculated to place the peaks and dip of the spectrum at the desired frequencies to maximize the difference between cooling and heating rates. It is shown that the coherent feedback loop enhances the cooling effect while the heating rate is not affected. Moreover, by utilizing two coupled auxiliary systems, the effect of heating transitions is better suppressed compared to the case with one auxiliary system. As a result, not only lower limits for cooling but also larger values of net cooling rate are achieved. The results show that the proposed feedback cooling scheme significantly improves the cooling capability of the hybrid system, and the mechanical resonator can be cooled near the quantum limit. Furthermore, it is shown that the proposed method performs well in a wide range of system parameters.
