An Enhanced Deep Learning Approach for High-Dimensional and Complex Datasets

Authors

DOI:

https://doi.org/10.21015/vtse.v14i1.2365

Abstract

The increasing availability of extensive datasets with many features in contemporary scenarios imposes serious limitations on traditional machine learning and deep learning models due to such challenges as feature redundancy, noise sensitivity, and scalability restrictions. Therefore, this paper presents a powerful deep learning approach that, through its adaptive feature learning, integrated regularization, and efficient optimization strategies, is capable of overcoming these difficulties. The proposed method performs the feature selection and focus on the most informative features automatically, while the redundant and irrelevant ones are eliminated. As a result, the representational quality and generalization ability can be significantly enhanced. Extensive results obtained on benchmark as well as real-world datasets hardly any cases have been proposed to prove not only the superiority of the newly developed method over the baseline machine learning methods and even the latest deep learning models in terms of accuracy and robustness but also its application potential in the practice. Further, the component removal experiments provide support for the significance of the different parts of the design and the statistical significance tests justify the reliability of the performance gains observed. Also, the suggested technique results in improved computational performance, and as a result, it could be implemented in scenarios associated with a large amount of data and high-dimensional data processing. To sum up, their results highlight that the proposed model provides an effective and scalable solution to tackle complicated datasets.

References

A. C. Djedouboum, A. A. Abba Ari, A. M. Gueroui, A. Mohamadou, and Z. Aliouat, "Big data collection in large-scale wireless sensor networks," Sensors, vol. 18, no. 12, p. 4474, 2018.

K. Adnan and R. Akbar, "An analytical study of information extraction from unstructured and multidimensional big data," Journal of Big Data, vol. 6, no. 1, pp. 1–38, 2019.

R. Krishnan, D. Liang, and M. Hoffman, "On the challenges of learning with inference networks on sparse, high-dimensional data," in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), Mar. 2018, pp. 143–151.

V. Divya, S. Sendil Kumar, V. Gokula Krishnan, and M. Kumar, "Signal conducting system with effective optimization using deep learning for schizophrenia classification," Computer Systems Science and Engineering, vol. 45, no. 2, pp. 1869–1886, 2023.

S. Dargan, M. Kumar, M. R. Ayyagari, and G. Kumar, "A survey of deep learning and its applications: a new paradigm to machine learning," Archives of Computational Methods in Engineering, vol. 27, no. 4, pp. 1071–1092, 2020.

Y. Huang et al., "Advancing transformer architecture in long-context large language models: A comprehensive survey," arXiv preprint, vol. arXiv:2311.12351, 2023.

Z. Allen-Zhu, Y. Li, and Y. Liang, "Learning and generalization in overparameterized neural networks: Going beyond two layers," in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.

Y. Ding, L. Wang, B. Liang, S. Liang, Y. Wang, and F. Chen, "Domain generalization by learning and removing domain-specific features," in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 24226–24239.

A. A. Wani, "Comprehensive review of dimensionality reduction algorithms: Challenges, limitations, and innovative solutions," PeerJ Computer Science, vol. 11, p. e3025, 2025.

S. Gyamerah, G. T. Soori, D. R. Korda, J. K. Tawiah, E. A. Akolgo, and E. O. Dapaah, "Comparative analysis of feature extraction of high dimensional data reduction using machine learning techniques," American Journal of Electrical and Computer Engineering, vol. 7, no. 2, pp. 27–39, 2023.

L. Ashok Kumar, M. R. E. Jebarani, and V. Gokula Krishnan, "Optimized deep belief neural network for semantic change detection in multi-temporal image," International Journal on Recent and Innovation Trends in Computing and Communication, vol. 11, no. 2, pp. 86–93, 2023.

H. A. Salman, A. Kalakech, and A. Steiti, "Random forest algorithm overview," Babylonian Journal of Machine Learning, pp. 69–79, 2024.

K. L. Du et al., "Exploring kernel machines and support vector machines," Mathematics, vol. 12, no. 24, p. 3935, 2024.

E. Dritsas and M. Trigka, "Remote sensing and geospatial analysis in the big data era," Remote Sensing, vol. 17, no. 3, p. 550, 2025.

S. A. De Caires et al., "Advancing soil mapping and management," Discover Soil, vol. 2, no. 1, p. 53, 2025.

D. Hao et al., "Unveiling nitrogen fertilizer in medicinal plant cultivation," Agronomy, vol. 14, no. 8, p. 1647, 2024.

A. Momeni et al., "Training of physical neural networks," Nature, vol. 645, no. 8079, pp. 53–61, 2025.

B. B. McShane et al., "Statistical significance and reporting," Journal of Marketing, vol. 88, no. 3, pp. 1–19, 2024.

B. B. Jia et al., "Pairwise statistical comparisons of multiple algorithms," Frontiers of Computer Science, vol. 19, no. 12, pp. 1–3, 2025.

S. Prasad et al., "Plant-based food industry: Overview and trends," 2025.

L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry, "Adversarially robust generalization requires more data," in Advances in Neural Information Processing Systems (NeurIPS), vol. 31, 2018.

L. Gao, J. Song, X. Liu, J. Shao, J. Liu, and J. Shao, "Learning in high-dimensional multimedia data: The state of the art," Multimedia Systems, vol. 23, no. 3, pp. 303–313, 2017.

M. Sheykhmousa, M. Mahdianpari, H. Ghanbari, F. Mohammadimanesh, P. Ghamisi, and S. Homayouni, "Support vector machine versus random forest for remote sensing image classification: A meta-analysis and systematic review," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 6308–6325, 2020.

V. G. Krishnan, B. V. S. Rao, J. R. Prasad, P. Pushpa, and S. Kumari, "Sugarcane yield prediction using noa-based swin transformer model in iot smart agriculture," Journal of Applied Biology & Biotechnology, vol. 12, no. 2, pp. 239–247, 2024.

K. Sri Vijaya, V. Gokula Krishnan, D. Arul Kumar, B. Prathusha Laxmi, and B. Yasaswi, "Aoa-based masked region-cnn model for detection of parking space in iot environment," International Research Journal of Multidisciplinary Technovation, vol. 6, no. 1, pp. 97–108, 2024.

Downloads

Published

2026-03-24

How to Cite

Dharani, D. L., & G. R, P. (2026). An Enhanced Deep Learning Approach for High-Dimensional and Complex Datasets. VFAST Transactions on Software Engineering, 14(1), 225–239. https://doi.org/10.21015/vtse.v14i1.2365