Increased number of malware and advanced cyber-attacks have become a serious problem. Several different methods have been introduced to deal with this problem. Recently, deep learning methods have attracted the attention of researchers in this regard. In this study, in order to detect 25 classes of malware, with the aim of increasing the detection accuracy, the pre-trained convolutional neural network optimized with the firefly algorithm. In fact, Alex-Net convolutional neural network automatically extracted 1000 feature vectors for each input image using the convolutional layer in its architecture. In the next step, we used the transfer learning method to classify the extracted features. In this work, we optimized the weight and bias values of the neural network by Firefly meta-heuristic algorithm. Finally, we were able to achieve 99.8% accuracy, which was superior to other existing methods in terms of accuracy.
