Identifying Key Learning Algorithm Parameter of Forward Feature Selection to Integrate with Ensemble Learning for Customer Churn Prediction

Authors

DOI:

https://doi.org/10.21015/vtse.v12i2.1811

Abstract

The Telecommunication has been facing fierce growth of customer data and competition in the market for a couple of decades. Due to this situation, an analytical strategy of proactive anticipation about customer churn and their profitable retention is inevitable for Telecommunication companies. To nip this problem in the bud, a lot of research work has been conducted in the past, but still the previously introduced churn prediction models possess their own limitations, such as high dimensional data with poor information and class imbalance, which turn into barriers while being implicated in real life to attain accurate and improved predictions. This study has been conducted, basically, to identify the key Learning Algorithm parameter of Forward Feature Selection (FFS) for dimensionality reduction which can be further integrated with class Imbalance Handling Technique and Ensemble Learning (EL) to attain improved accuracy. The core objective of this study is to turn an imbalanced dataset into a balanced one for Ensemble Learning (EL) Model of Customer Churn Prediction (CCP). This study concluded that Logistic Regression (LR) based Forward Feature Selection (FFS) can outperform with Oversampling Class Imbalance Handling Techniques and Ensemble Learning (EL) by scoring 0.96% accuracy, which is the highest accuracy against benchmark studies. The resulting methodology has been named as the Logistic Regression Learning based Forward Feature Selection for ensemble Learning (LRLFFSEL) and applied over Orange dataset with 20 features and 3333 instances. In future this methodology can be evaluated over a bigger dataset and combined with some data optimization techniques to improve its accuracy.

References

A. K. Ahmad, A. Jafar, and K. Aljoumaa, "Customer churn prediction in telecom using machine learning in big data platform," Journal of Big Data, vol. 6, no. 1, pp. 1-24, 2019.

F. Naz et al., "Role of service quality in improving customer loyalty towards telecom companies in Hungary during the COVID-19 pandemic," Economies, vol. 9, no. 4, p. 200, 2021.

N. Akter, "Service quality and customer retention in COVID-19 period at telecom industry," Social Science Research Network (SSRN), p. 9, 2022.

A. Shahzad et al., "Antecedents of customer loyalty and performance improvement: Evidence from Pakistan’s telecommunications sector," Utilitie

R. Andrews, “Churn prediction in telecom sector using machine learning,” International Journal of Information Systems and Computer Sciences, vol. 8, no. 2, pp. 132-134, 2019.

R. Srinivasan, D. Rajeswari, and G. Elangovan, “Customer churn prediction using machine learning approaches,” pp. 1-6, 2023.

Y. Fareniuk et al., “Customer churn prediction model: a case of the telecommunication market,” Economics, vol. 10, no. 2, pp. 109-130, 2022.

S. Saha, “Customer retention and churn prediction in the telecommunication industry: a case study on a Danish university,” SN Applied Sciences, vol. 5, no. 173, pp. 1-14, 2023.

R. Zebari, A. Abdulazeez, D. Zeebaree, D. Zebari, and J. Saeed, “A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction,” Journal of Applied Science and Technology Trends, vol. 1, no. 1, pp. 56-70, 2020.

S. Velliangiri, S. Alagumuthukrishnan, et al., “A review of dimensionality reduction techniques for efficient computation,” Procedia Computer Science, vol. 165, pp. 104-111, 2019.

U. M. Khaire and R. Dhanalakshmi, “Stability of feature selection algorithm: A review,” Journal of King Saud University-Computer and Information Sciences, vol. 34, no. 4, pp. 1060-1073, 2022.

M. I. Prasetiyowati, N. U. Maulidevi, and K. Surendro, “Determining threshold value on information gain feature selection to increase speed and prediction accuracy of random forest,” Journal of Big Data, vol. 8, no. 1, p. 84, 2021.

F. Z. Janane, T. Ouaderhman, and H. Chamlal, “A filter feature selection for high-dimensional data,” Journal of Algorithms & Computational Technology, vol. 17, p. 17483026231184171, 2023.

S. Roy, S. Mondal, A. Ekbal, and M. S. Desarkar, “Dispersion ratio based decision tree model for classification,” Expert Systems with Applications, vol. 116, pp. 1-9, 2019.

L.-j. Cai, S. Lv, and K.-b. Shi, “Application of an improved chi feature selection algorithm,” Discrete Dynamics in Nature and Society, vol. 2021, pp. 1-8, 2021.

L. Sun, T. Wang, W. Ding, J. Xu, and Y. Lin, “Feature selection using fisher score and multilabel neighborhood rough sets for multilabel classification,” Information Sciences, vol. 578, pp. 887-912, 2021.

Doreswamy, M. K. Hooshmand, and I. Gad, “Feature selection approach using ensemble learning for network anomaly detection,” CAAI Transactions on Intelligence Technology, vol. 5, no. 4, pp. 283-293, 2020.

S. Nersisyan, V. Novosad, A. Galatenko, A. Sokolov, G. Bokov, A. Konovalov, D. Alekseev, and A. Tonevitsky, “Exhaufs: exhaustive search-based feature selection for classification and survival regression,” PeerJ, vol. 10, p. e13200, 2022.

R. Jain and W. Xu, “Artificial intelligence based wrapper for high dimensional feature selection,” BMC Bioinformatics, vol. 24, no. 1, p. 392, 2023.

S. Zhou, T. Li, and Y. Li, “Recursive feature elimination based feature selection in modulation classification for MIMO systems,” Chinese Journal of Electronics, vol. 32, no. 4, pp. 785-792, 2023.

W. Guodong, S. Lanxiang, W. Wei, C. Tong, G. Meiting, and P. Zhang, “A feature selection method combined with ridge regression and recursive feature elimination in quantitative analysis of laser induced breakdown spectroscopy,” Plasma Science and Technology, vol. 22, no. 7, p. 074002, 2020.

J. Li, H. Zhang, J. Zhao, X. Guo, W. Rihan, and G. Deng, “Embedded feature selection and machine learning methods for flash flood susceptibility-mapping in the mainstream Songhua river basin, China,” Remote Sensing, vol. 14, no. 21, p. 5523, 2022.

B. Bencsik, I. Reményi, M. Szemenyei, and J. Botzheim, “Designing an embedded feature selection algorithm for a drowsiness detector model based on electroencephalogram data,” Sensors, vol. 23, no. 4, p. 1874, 2023.

N. Pudjihartono, T. Fadason, A. W. Kempa-Liehr, and J. M. O’Sullivan, “A review of feature selection methods for machine learning-based disease risk prediction,” Frontiers in Bioinformatics, vol. 2, p. 927312, 2022.

A. Amin, F. Al-Obeidat, B. Shah, A. Adnan, J. Loo, and S. Anwar, “Customer churn prediction in telecommunication industry using data certainty,” Journal of Business Research, vol. 94, pp. 290-301, 2019. DOI: https://doi.org/10.1016/j.jbusres.2018.03.003

T. W. Cenggoro, R. A. Wirastari, E. Rudianto, M. I. Mohadi, D. Ratj, and B. Pardamean, “Deep learning as a vector embedding model for customer churn,” Procedia Computer Science, vol. 179, pp. 624-631, 2021.

R. A. Pratiwi, S. Nurmaini, D. P. Rini, M. N. Rachmatullah, and A. Darmawahyuni, “Deep ensemble learning for skin lesions classification with convolutional neural network,” IAES International Journal of Artificial Intelligence, vol. 10, no. 3, p. 563, 2021.

G. ApurvaSree, S. Ashika, S. Karthi, V. Sathesh, M. Shankar, and J. Pamina, “Churn prediction in telecom using classification algorithms,” International Journal of Scientific Research and Engineering Development, vol. 5, pp. 19-28, 2019.

S. W. Fujo, S. Subramanian, M. A. Khder, et al., “Customer churn prediction in telecommunication industry using deep learning,” Information Sciences Letters, vol. 11, no. 1, p. 24, 2022.

M. Saghir, Z. Bibi, S. Bashir, and F. H. Khan, “Churn prediction using neural network based individual and ensemble models,” in 2019 16th International Bhurban Conference on Applied Sciences and Technology (IBCAST), pp. 634-639, IEEE, 2019.

S. Maldonado, G. Domínguez, D. Olaya, and W. Verbeke, “Profit-driven churn prediction for the mutual fund industry: A multisegment approach,” Omega, vol. 100, p. 102380, 2021.

S. Kumar and M. Kumar, “Predicting customer churn using artificial neural network,” in Engineering Applications of Neural Networks: 20th International Conference, EANN 2019, Xersonisos, Crete, Greece, May 24-26, 2019, Proceedings 20, pp. 299-306, Springer, 2019.

A. Amin, F. Al-Obeidat, B. Shah, M. A. Tae, C. Khan, H. U. R. Durrani, and S. Anwar, “Just-in-time customer churn prediction in the telecommunication sector,” The Journal of Supercomputing, vol. 76, pp. 3924-3948, 2020. DOI: https://doi.org/10.1007/s11227-017-2149-9

I. Brandusoiu, G. Toderean, et al., “Churn prediction in the telecommunications sector using support vector machines,” Margin, vol. 1, no. 1, 2013. DOI: https://doi.org/10.15660/AUOFMTE.2013-1.2772

A. Mishra and U. S. Reddy, “A comparative study of customer churn prediction in telecom industry using ensemble based classifiers,” in 2017 International conference on inventive computing and informatics (ICICI), pp. 721-725, IEEE, 2017. DOI: https://doi.org/10.1109/ICICI.2017.8365230

R. M. Wahul, A. P. Kale, and P. N. Kota, “An ensemble learning approach to enhance customer churn prediction in telecom industry,” International Journal of Intelligent Systems and Applications in Engineering, vol. 11, no. 9s, pp. 258-266, 2023.

H. Jain, A. Khunteta, and S. Srivastava, “Churn prediction in telecommunication using logistic regression and logit boost,” Procedia Computer Science, vol. 167, pp. 101-112, 2020.

I. Ullah, B. Raza, A. K. Malik, M. Imran, S. U. Islam, and S. W. Kim, “A churn prediction model using random forest: analysis of machine learning techniques for churn prediction and factor identification in telecom sector,” IEEE Access, vol. 7, pp. 60134-60149, 2019.

Z. Lian-Ying, D. M. Amoh, L. K. Boateng, and A. A. Okine, “Combined appetency and upselling prediction scheme in telecommunication sector using support vector machines,” International Journal of Modern Education and Computer Science, vol. 10, no. 6, p. 1, 2019.

S. F. Bilal, A. A. Almazroi, S. Bashir, F. H. Khan, and A. A. Almazroi, “An ensemble based approach using a combination of clustering and classification algorithms to enhance customer churn prediction in telecom industry,” PeerJ Computer Science, vol. 8, p. e854, 2022.

K. Maharana, S. Mondal, and B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Global Transitions Proceedings, vol. 3, no. 1, pp. 91-99, 2022.

D. A. A. Prakash, “Pre-processing techniques for preparing clean and high-quality data for diabetes prediction,” International Journal of Research Publication and Reviews, vol. 5, no. 2, pp. 458-465, 2024.

R. Jagdhuber, M. Lang, A. Stenzl, J. Neuhaus, and J. Rahnenführer, “Cost-constrained feature selection in binary classification: adaptations for greedy forward selection and genetic algorithms,” BMC Bioinformatics, vol. 21, pp. 1-21, 2020.

K. Dissanayake and M. G. Md Johar, “Comparative study on heart disease prediction using feature selection techniques on classification algorithms,” International Journal of Engineering Trends and Technology, vol. 69, no. 4, pp. 233-238, 2021.

M. Z. I. Chowdhury and T. C. Turin, “Variable selection strategies and its importance in clinical prediction modelling,” Family Medicine and Community Health, vol. 8, no. 1, 2020.

G. Borboudakis and I. Tsamardinos, “Forward-backward selection with early dropping,” Journal of Machine Learning Research, vol. 20, no. 8, pp. 1-39, 2019.

M. Suyal and P. Goyal, “A review on analysis of k-nearest neighbor classification machine learning algorithms based on supervised learning,” International Journal of Engineering Trends and Technology, vol. 70, no. 7, pp. 43-48, 2022.

S. Uddin, I. Haque, H. Lu, M. A. Moni, and E. Gide, “Comparative performance analysis of k-nearest neighbour (KNN) algorithm and its different variants for disease prediction,” Scientific Reports, vol. 12, no. 1, p. 6256, 2022.

M. Schonlau and R. Y. Zou, “The random forest algorithm for statistical learning,” The Stata Journal, vol. 20, no. 1, pp. 3-29, 2020.

N. M. A. M. Abdulazeez, “Machine learning classification based on random forest algorithm: A review,” International Journal of Science and Business, IJSAB International, vol. 5, no. 2, pp. 128-142, 2021.

M. Savargiv, B. Masoumi, and M. R. Keyvanpour, “A new random forest algorithm based on learning automata,” Computational Intelligence and Neuroscience, vol. 2021, no. 1, p. 5572781, 2021.

Z. Yang, J. Ren, Z. Zhang, Y. Sun, C. Zhang, M. Wang, and L. Wang, “A new three-way incremental naive Bayes classifier,” Electronics, vol. 12, no. 7, p. 1730, 2023.

T. Rymarczyk, E. Kozłowski, G. Kłosowski, and K. Niderla, “Logistic regression for machine learning in process tomography,” Sensors, vol. 19, no. 15, p. 3400, 2019.

G. Aliman, T. F. S. Nivera, J. C. A. Olazo, D. J. P. Ramos, C. D. B. Sanchez, T. M. Amado, N. M. Arago, R. L. Jorda Jr, G. C. Virrey, and I. C. Valenzuela, “Sentiment analysis using logistic regression,” Journal of Computational Innovations and Engineering Applications, vol. 7, no. 1, pp. 35-40, 2022.

B. X. Liew, F. M. Kovacs, D. Rügamer, and A. Royuela, “Machine learning versus logistic regression for prognostic modelling in individuals with non-specific neck pain,” European Spine Journal, vol. 31, no. 8, pp. 2082-2091, 2022.

R. P. A. Murti, S. M. Putra, S. A. Kurniawan, and Y. R. Nugraha, “Naïve Bayes classifier for journal quartile classification,” 2019.

I. Wickramasinghe and H. Kalutarage, “Naive Bayes: applications, variations and vulnerabilities: a review of literature with code snippets for implementation,” Soft Computing, vol. 25, no. 3, pp. 2277-2293, 2021.

H. Chen, S. Hu, R. Hua, and X. Zhao, “Improved naive Bayes classification algorithm for traffic risk management,” EURASIP Journal on Advances in Signal Processing, vol. 2021, no. 1, p. 30, 2021.

B. Gaye, D. Zhang, and A. Wulamu, “Improvement of support vector machine algorithm in big data background,” Mathematical Problems in Engineering, vol. 2021, no. 1, p. 5594899, 2021.

S. U. Ahsaan, H. Kaur, A. K. Mourya, and S. Naaz, “A hybrid support vector machine algorithm for big data heterogeneity using machine learning,” Symmetry, vol. 14, no. 11, p. 2344, 2022.

Z. Noroozi, A. Orooji, and L. Erfannia, “Analyzing the impact of feature selection methods on machine learning algorithms for heart disease prediction,” Scientific Reports, vol. 13, no. 1, p. 22588, 2023.

R. Ghorbani and R. Ghousi, “Comparing different resampling methods in predicting students’ performance using machine learning techniques,” IEEE Access, vol. 8, pp. 67899-67911, 2020.

T. Sasada, Z. Liu, T. Baba, K. Hatano, and Y. Kimura, “A resampling method for imbalanced datasets considering noise and overlap,” Procedia Computer Science, vol. 176, pp. 420-429, 2020.

A. M. Elsobky, A. E. Keshk, and M. G. Malhat, “A comparative study for different resampling techniques for imbalanced datasets,” IJCI. International Journal of Computers and Information, vol. 10, no. 3, pp. 147-156, 2023.

A. Kim and I. Jung, “Optimal selection of resampling methods for imbalanced data with high complexity,” PLOS ONE, vol. 18, no. 7, p. e0288540, 2023.

D. Elreedy and A. F. Atiya, “A comprehensive analysis of synthetic minority oversampling technique (SMOTE) for handling class imbalance,” Information Sciences, vol. 505, pp. 32–64, 2019.

K. Li and Y. Hu, “Research on unbalanced training samples based on SMOTE algorithm,” in Journal of Physics: Conference Series, vol. 1303, p. 012095, IOP Publishing, 2019.

U. G. Mohammad, S. Imtiaz, M. Shakya, A. Almadhor, and F. Anwar, “An optimized feature selection method using ensemble classifiers in software defect prediction for healthcare systems,” Wireless Communications and Mobile Computing, vol. 2022, no. 1, p. 1028175, 2022.

D. Dablain, B. Krawczyk, and N. V. Chawla, “DeepSMOTE: Fusing deep learning and SMOTE for imbalanced data,” IEEE Transactions on Neural Networks and Learning Systems, 2022.

V. Chakraborty and M. Sundaram, “An efficient SMOTE-based model for dyslexia prediction,” International Journal of Information Engineering & Electronic Business, vol. 13, no. 6, 2021.

S. A.-S. Tahfim and Y. Chen, “Comparison of cluster-based sampling approaches for imbalanced data of crashes involving large trucks,” Information, vol. 15, no. 3, p. 145, 2024.

F. Prastyasari and T. Shinoda, “Near miss detection for encountering ships in Sunda Strait,” in IOP Conference Series: Earth and Environmental Science, vol. 557, p. 012039, IOP Publishing, 2020.

A. Tanimoto, S. Yamada, T. Takenouchi, M. Sugiyama, and H. Kashima, “Improving imbalanced classification using near-miss instances,” Expert Systems with Applications, vol. 201, p. 117130, 2022.

H. H. Hairani and D. D. Priyanto, “A new approach of hybrid sampling SMOTE and ENN to the accuracy of machine learning methods on unbalanced diabetes disease data,” vol. 14, no. 8, pp. 585–590, 2023.

K. Z. Wang, X. N. Wu, “SMOTE-Tomek-based resampling for personality recognition,” IEEE Access, vol. 7, pp. 129678–129689, 2019.

Z. Fang, L. P. Wang, “A comparative study of heterogeneous ensemble-learning techniques for landslide susceptibility mapping,” International Journal of Geographical Information Science, vol. 35, no. 2, pp. 321–347, 2020.

H. Alsawalqah, N. Hijazi, “Software defect prediction using heterogeneous ensemble classification based on segmented patterns,” Applied Sciences, vol. 10, no. 5, p. 1745, 2020.

Z. Yang, J. Ren, Z. Zhang, Y. Sun, C. Zhang, M. Wang, L. Wang, “A new three-way incremental naive Bayes classifier,” Electronics, vol. 12, no. 7, p. 1730, 2023.

S. Saleh, T. Alkhalifah, “Heterogeneous ensemble deep learning model for enhanced Arabic sentiment analysis,” Sensors (Basel), vol. 22, no. 10, p. 3707, 2022.

Y. Park, U. Yun, “A stacking heterogeneous ensemble learning method for the prediction of building construction project costs,” Applied Sciences, vol. 12, no. 19, p. 9729, 2022.

Y. Zhang, Z. Wang, Z. Zhang, “A new ensemble learning method for multiple fusion weighted evidential reasoning rule,” Journal of Electrical and Computer Engineering, vol. 2023, no. 3, pp. 1–15, 2023.

H. Alsawalqah, N. Hijazi, “Software defect prediction using heterogeneous ensemble classification based on segmented patterns,” Applied Sciences, vol. 10, no. 5, pp. 2076–3417, 2020.

H. Alsawalqah, N. Hijazi, “DeepTLF: Robust deep neural networks for heterogeneous tabular data,” International Journal of Data Science and Analytics, vol. 16, no. 1, pp. 85–100, 2022.

D. Gao, X. Yao, “A survey on heterogeneous federated learning,” arXiv preprint arXiv:2210.04505, pp. 85–100, 2022.

L. Nanni, S. Brahnam, “Heterogeneous ensemble for medical data classification,” Analytics, vol. 2, no. 3, pp. 676–693, 2023.

S. Kundu, S. K. Biswas, “A review on rainfall forecasting using ensemble learning techniques,” e-Prime - Advances in Electrical Engineering, Electronics and Energy, vol. 6, p. 100296, 2023.

R. Gao, “An improved adaboost algorithm for hyperparameter optimization,” Journal of Physics: Conference Series, vol. 1631, no. 1, p. 012048, 2020.

M. A. Gotardo, “Using decision tree algorithm to predict student performance,” Indian Journal of Science and Technology, vol. 12, no. 8, pp. 1–8, 2020.

S. R. Jiao, J. Song, “A review of decision tree classification algorithms for continuous variables,” Journal of Physics: Conference Series, vol. 1651, no. 1, p. 012083, 2020. DOI: https://doi.org/10.1088/1742-6596/1651/1/012083

Y. Zhang, J. Liu, “A review of ensemble learning algorithms used in remote sensing applications,” Applied Sciences, vol. 17, no. 12, p. 8654, 2022.

I. Domor Mienye, Y. Sun, “Prediction performance of improved decision tree-based algorithms: a review,” 2nd International Conference on Sustainable Materials Processing and Manufacturing (SMPM 2019), vol. 35, pp. 698–703, 2019.

E. Demirović, A. Lukina, “Murtree optimal decision trees via dynamic programming and search,” Journal of Machine Learning Research, vol. 23, pp. 1–47, 2022.

S. Liu, Y. Yang, “Application of decision tree-based classification algorithm on content marketing,” Journal of Mathematics, vol. 2022, pp. 1–10, 2022.

H. Blockeel, L. Devos, “Decision trees: from efficient prediction to responsible AI,” Frontiers in Artificial Intelligence, vol. 6, p. 1124553, 2023.

M. R. S. Abdulmajeed Atiah Alharbi, “Classification performance analysis of decision tree-based algorithms with noisy class variable,” Discrete Dynamics in Nature and Society, vol. 2024, pp. 1–10, 2024.

J. Song, Y. Gao, “The random forest model has the best accuracy among the four pressure ulcer prediction models using machine learning algorithms,” Risk Management and Healthcare Policy, vol. 14, pp. 1175–1187, 2021.

Y. Xiahou, “Customer churn prediction using adaboost classifier and BP neural network techniques in the e-commerce industry,” American Journal of Industrial and Business Management, vol. 12, no. 03, pp. 277–293, 2022.

Y. Xiahou, “An improved adaboost algorithm for highly imbalanced datasets in the co-authorship recommendation problem,” IEEE Access, vol. 11, pp. 89107–89123, 2023.

P. B. P. V. R. K. Alireza Ghasemieh, Alston Lloyed, “A novel machine learning model with stacking ensemble learner for predicting emergency readmission of heart-disease patients,” Decision Analytics Journal, vol. 7, p. 100242, 2023.

A. H. Z. C. C. Begum Ay Ture, Akhan Akbulut, “Stacking-based ensemble learning for remaining useful life estimation,” Soft Computing, vol. 28, no. 2, pp. 1337–1349, 2023.

F. El Barakaz, O. Boutkhoum, “Minimizing the overlapping degree to improve class-imbalanced learning under sparse feature selection: Application to fraud detection,” IEEE Access, vol. 9, pp. 28101–28110, 2021.

Downloads

Published

2024-06-11

How to Cite

Tasneem, S., Younas, M., & Shafiq, Q. (2024). Identifying Key Learning Algorithm Parameter of Forward Feature Selection to Integrate with Ensemble Learning for Customer Churn Prediction. VFAST Transactions on Software Engineering, 12(2), 56–75. https://doi.org/10.21015/vtse.v12i2.1811

Issue

Section

Articles