Assessing the reliability of safety-critical systems is an important and challenging task because even a minor failure in these systems may result in catastrophic consequences, like losing human life. A well-known and fully automatic technique in reliability assessing approaches is model checking. However, applying this technique to verify some properties such as safety may lead to the state space explosion problem in which all reachable states cannot be checked due to computational limitations. In such situations that the verification of a safety property is infeasible, it is possible to refute the safety property by searching a reachable state in which a special configuration (e.g., an error or an undesirable behaviour) occurs. Therefore, checking reachability can be done instead of refuting the corresponding safety property. Finding such reachable states, in the worst case, may cause the state space explosion problem again. Hence, using evolutionary algorithms to explore the state space efficiently can be a promising idea. In this paper, at first, we propose an evolutionary algorithm to check reachability properties and refute safety ones in software systems specified formally through graph transformations. Since the accuracy and convergence speed of the proposed approach can still be improved, we employ the Bayesian Optimization Algorithm (BOA) to propose another approach. In BOA, a Bayesian network is learnt from the population and then sampled to generate new solutions. The proposed approaches can be used to analyse the reachability and safety properties. The proposed approaches are implemented in GROOVE which is an open source toolset for designing and model checking graph transformation systems. To evaluate the efficiency of the proposed approaches, different benchmark problems are employed. Experimental results show that the proposed approaches are faster and more accurate than the existing methods.
