The rock mass deformation modulus (RMDM) plays a crucial role in dam and tunnel design. This study introduces advanced machine-learning (ML) models to predict RMDM using rock mass rating (RMR) and the Q-system at the Khersan-2 dam site in southwestern Iran. Through the analysis of exploratory boreholes, the engineering geological properties of the samples, Q, RMR, RMDM, geological strength index (GSI), Hoek-Brown, and shear strength constants of the rock mass were determined. Subsequently, several effective ML models, namely random forest, multilayer perceptron backpropagation artificial neural network, Gaussian process regression, K-nearest neighbor, simple regression, and multiple linear and non-linear regression approaches, were utilized to estimate RMDM. Based on classification systems, the site was rated as having good RMR and Q categories. A new empirical relationship with high accuracy was established between Q and RMR89. Furthermore, RMDM demonstrated a strong correlation with Q and RMR, as supported by statistical analysis. The results showed the relative superiority of non-linear regression models compared to linear ones. The employed ML techniques displayed remarkable accuracy in estimating RMDM, achieving a coefficient of determination (R2 ) greater than 97%. Notably, Gaussian process regression with a squared exponential kernel function stood out as the most effective approach, yielding outstanding performance in predicting RMDM with an impressive R2=0.99 and RMSE=0.01 compared to all other investigated methods.