Stochastic finite element method (StFEM) is a robust tool for uncertainty quantification of engineering systems having random properties. Nevertheless, the matrices involved in this method are very large compared to their deterministic counterparts. Thus, the computational aspects of StFEM are of great importance to be optimized. In this paper, an efficient StFEM is developed for analysis of structures. For this purpose, a method based on graph concepts is presented and extended to StFEM and recently developed stochastic spectral finite element method (StSFEM) procedures. Here, mathematical remedies are incorporated to enhance the analysis performance. Firstly, a graph theoretical method is presented for swift numerical solution of Fredholm integral equation arising from Karhunen–Loève expansion, which greatly reduces the existing computational cost, and can even be applied to the domain without symmetry. Secondly, a preconditioner is applied to decompose the matrices to Kronecker products of submatrices, and then graph product rules are utilized to solve the governing linear equation of cyclically symmetric models without inversing the final matrix, while only a small matrix is inversed instead. The proposed method provides significant improvement in the stochastic structural analysis. Illustrative examples demonstrate the efficiency and accuracy of the present method as a swift analysis.