Open Access Open Access  Restricted Access Subscription or Fee Access

Systematic Literature Review on Search Based Software Testing

(*) Corresponding author

Authors' affiliations



The use of random search is very poor at finding solutions when those solutions occupy a very small part of the overall search space. Test data may be found faster and more reliably if the search is given some guidance. This work is a paper that explains the application of metaheuristic techniques in search-based software testing. The paper systematically review 47 papers selected randomly from online databases and conference proceeding based on the metaheuristic search techniques that have been most widely applied to problem solving, the different fitness function used for test data selection in each of the metaheuristic technique, and the limitation in the use of each search-based technique for software testing. It was found that GA outperformed its counterparts SA, HC, GP and random search approaches in generating test data automatically, different approaches were used to make sure that test data are selected within shorter period of time and also with wider coverage of the paths based on the fitness function, and most of the limitations of the articles are the handling of complex data types, like array, object types, and branch coverage. The paper also provides areas of possible future work on the use of metaheuristic techniques in search-based software testing.
Copyright © 2017 Praise Worthy Prize - All rights reserved.


Metaheuristic Techniques; Search-Based Software Testing; Software Testing; Systematic Literature Review

Full Text:



S. P. Roger, Software Engineering: A practitioner’s Approach, Fifth Edition (McGraw-Hill Publisher, New York, America).

X. Jifeng, et al, A Random Walk Based Algorithm for Structural Test Case Generation, School of Mathematical Sciences

Dalian University of Technology Dalian, China, 2007.

K. Barbara et al, Guidelines for Performing Systematic Literature Reviews in Software Engineering, EBSE Technical Report, EBSE-2007-01

Z. Yuan, C. John A.“A search-based framework for automatic testing of MATLAB/Simulink models”, The Journal of Systems and Software 81 (2008), pp. 262–285.

A. Wasif, T. Richard, F. Robert, A systematic review of search-based testing for non-functional system properties, Journal of Information and Software Technology 51,2009,pp.957–976.

L. Jailton, G. C. Celso, V. Auri, R. Cassio,An Elitist Evolutionary Algorithm for Automatically Generating Test Data, World Congress on Computational Intelligence (WCCI2012) IEEE June, 10-15, 2012 - Brisbane, Australia.

M. Phil, An identification of program factors that impact crossover performance in evolutionary test input generation for the branch coverage of C programs Journal of Information and Software Technology 55 (2013), pp. 153–172.

H. Mark, M. Afshin Search Based Software Engineering: Introduction to the Special Issue of the IEEE Transactions on Software Engineering, IEEE Transactions on Software Engineering, Vol. 36, NO. 6, November/December 2010.

L. Kiran, H. Mark, G. Hamilton, AUSTIN: An open source tool for search based software testing of C programs, Journal of Information and Software Technology 55 (2013), pp. 112–125.

R. S. Praveen, K. Tai-hoon, Application of Genetic Algoritm in Software Testing, Journal of Software Engineering and Applications 3 (4), 2009, pp. 87-94.

H. Mark, Automated Test Data Generation using Search Based Software Engineering, Second International Workshop on Automation of Software Test (AST'07) 2007.

A. Nadia, H. Mark, Automated Web Application Testing Using Search Based Software Engineering, IEEE, ASE 2011, Lawrence, KS, USA.

S. Ramon, Y. Xin, Handling Constraints for Search Based Software Test Data Generation, IEEE International Conference on Software Testing Verification and Validation Workshop (ICSTW’08).

H. Mark, G. K. Sung, L.Kiran, M. Phil, Y. Shin, Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost problem, King’s College London, CREST centre, Strand, London, WC2R 2LS, UK, 2008.

A. S. Anastasis, S. A. Andreas, Automatic, evolutionary test data generation for dynamic software testing, Journal of Systems and Software 81 (2008), pp. 1883–1898.

L. Raluca, I. Florentin, A comparative landscape analysis of fitness functions forsearch-based testing, Proc. Of the 2008 International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 08), 2008, pp 201-208.

L. Andreas,O. Manuel, Z. Andreas, C. Ilinca, M. Bertrand Efficient Unit Test Minimzation, ASE 2007, Atlanta, Georgia, USA.

B. Oliver, W. Joachim, Evolutionary functional testing, Computers & Operations Research, 35 (2008) pp. 3144 – 3160.

L. Raluca, I. Florentin, Functional Search-based Testing from State Machines, International Conference on Software Testing, Verification, and Validation, 2008.

A. A. Moataz, H. Irman, GA-based multiple paths test data generator, Computers &Operations Research 35 (2008), pp. 3107 – 3124.

S. Liu, Z. Wang, Genetic Algorithm and its Application in the path-oriented test data automatic generation, Procedia Engineering 15 (2011), pp. 1186 – 1190.

J. Yue, H. Mark, Higher Order Mutation Testing, Journal of Information and Software Technology 51 (2009), pp. 1379– 1393.

B. Jorge, M. G. Paula, N. C. C. Ana, T. H. Guilherme, Systematic Review in Software Engineering, Technical Report RT-ES 679/05.

A. Andrea, It Does Matter How You Normalise the Branch Distance in Search BasedSoftware Testing, Third International Conference on Software Testing, Verification and Validation,2010, pp. 205-214.

J. Tao, G. Nicolas, H. Mark, L. Zheng, Locating dependence structures using search-based slicing, Information and Software Technology 50 (2008) pp. 1189–1209.

P. Mike, and M. Nicos, Mutation based test case generation via a path selection strategy, Information and Software Technology 54 (2012) pp. 915–932.

A. Enrique, and C. Francisco, Observations in using parallel and sequential evolutionary algorithms for automatic software testing, Journal of Computers & Operations Research 35 (2008) pp. 3161 – 3183

A. Wasif, T. Richard, On the application of genetic programming for software engineering predictive modeling: A systematic review, Journal of Expert System with Applications (2011), doi:10.1016/j.eswa.2011.03.041.

A. Andrea, On the Automation of Fixing Software Bugs, ICSE’08, May 10–18, 2008, Leipzig, Germany, pp. 1003-1006.

Y. Shin, H. Mark, Pareto Efficient Multi-Objective Test Case Selection, ISSTA’07,July 9–12, 2007, London, England, United Kingdom.

G. Vahid, Empirical Analysis of a Genetic Algorithm-basedStress Test Technique, GECCO’08, July 12–16, 2008, Atlanta, Georgia, USA, pp. 1743-1750.

M. Phil, Search-Based Software Testing: Past, Present and Future, IEEE2011.

M. Phil, Search Based Failure Discovery using Testability Transformations to Generate Pseudo Oracles, GECCO’09, July 8–12, 2009, Montr´ealQu´ebec, Canada.

A. Giuliano, Search Based Software Testing for Software Security: Breaking Code to Make it Safer, IEEE International Conference on Software Testing Verification and Validation Workshops, DOI 10.1109/ICSTW.2009.12.

A. Andrea, Y. Xin, Search based software testing of object-oriented containers, Journal of Information Sciences 178 (2008) pp. 3075–3095.

M. Phil, S. Muzammil, S. Mark, Search-Based Test Input Generation for String DataTypes Using the Results of Web Queries, 2012, Regent Court, 211 Portobello, Sheffield, UK.

F. Gordon, A. Andrea, The Seed is Strong: Seeding Strategies in Search-Based Software Testing, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.DOI 10.1109/ICST.2012.24, pp. 121-130.

J. A. Clark, et al., Semantic mutation testing, Science of Computer Programming (2011), doi:10.1016/j.scico.2011.03.011.

C. B. R. José, A. Z. Mário, F. V. Francisco, Test Case Evaluationand Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software, Journal of Information and Software Technology 51 (2009) pp. 1534–1548.

G. Dunei, Z. Wanqui, Y. Xiangjuan, Evolutionary generation of test data for many paths coverage based on grouping, Journal of Systems and Software 84 (2011), pp.2222-2233.

S. Alin, I. Florentin, L. Raluca, T. Cristina, Towards Search-ased Testing for Event-B Models, 2011 Fourth International Conference on Software Testing, Verification and Validation Workshops DOI 10.1109/ICSTW.2011.41.

G. Kamran, A. C. John, Widening the Goal Posts: Program Stretching to Aid Search Based Software Testing, 2009 International Symposium on Search Based Software Engineering DOI 10.1109/SSBSE.2009.26.

P. Manisha, P. J .Nikumbh, Pair-wise Testing Using Simulated Annealing, Procedia Technology 4 (2012)

doi: 10.1016/j.protcy.2012.05.127, pp. 778 – 782.

T. Jose, R. Eduardo, New bounds for binary covering arrays using simulated annealing, Journal of Information Sciences 185(2012) pp. 137–152.

P. M. S. Bueno et al., Diversity oriented test data generation using metaheuristic search Techniques, Inform. Sci. (2011), doi:10.1016/j.ins.2011.01.025.

M. P. Reza, A. G. Abdul Azim, A. Rusli, A. Rodziah, Empirical evaluation of the fault detection effectiveness and test effort efficiency of the automated AOP testing approaches, Information and Software Technology 53 (2011) pp. 1062–1083.

A. Bouchachia , R. Mittermeir, P. Sielecky, S. Stafiej, M. Zieminski, Nature-inspired techniques for conformance testing of object-oriented software, Applied Soft Computing 10 (2010) pp. 730–745.

D. Eugenia, T. Javier, B. Raquel, J. D. José, A tabu search algorithm for structural software testing, Journal of Computers & Operations Research 35 (2008) pp. 3052 – 3072.

F. Nilgun, M. K. Mieczyslaw, Self Controlling Tabu Search algorithm for the Quadratic Assignment Problem, Journal of Computers & Industrial Engineering 60 (2011) pp. 310–319.

A. W. II, Robert, J. C. Charles, Tabu search for covering arrays using permutation vectors, Journal of Statistical Planning and Inference 139 (2009) pp. 69 – 80.

M. Caserta, A. M. Uribe, Tabu search-based metaheuristic algorithm for software system reliability problems, Journal of Computers & Operations Research 36 (2009) pp. 811 – 822.

M. Brototi, D. Kousik, D. Paramartha, Load Balancing in Cloud Computing using Stochastic Hill Climbing-A Soft Computing Approach, Procedia Technology 4 (2012) pp. 783–789.


  • There are currently no refbacks.

Please send any question about this web site to
Copyright © 2005-2024 Praise Worthy Prize