Safety is the most important factor when developing software for safety-critical systems. Traditional approachesattempted to achieve safety through testing the software. However, there might be some bugs in the software notrevealed in the test procedure. Formal verification is a new trend in developing safe software. In this paper, wepropose a multi-phase formal approach for safety management in safety-critical software. We use timed transitionPetri-net as a formal means to specify the properties of the model and their relations in each component of thesoftware. In addition, we use the Z language to specify textual and mathematical specifications of the model, as arepresentative model to evaluate the proposed approach; we chose continuous infusion insulin pump (CIIP).