Email this article 
Open Access
Subscription or Fee Access
Optimizing Deep Learning Methods in Neural Network Architectures
(*) Corresponding author
DOI: https://doi.org/10.15866/ireaco.v14i2.20591
Abstract
Deep neural networks are a powerful tool for computer-assisted learning and have achieved significant success in numerous computer vision and image processing tasks. This paper discusses several new neural network structures that have better performance than the traditional feedforward neural network structure. A method of network structure optimization based on gradient descent and heavy-ball algorithms has been proposed. Furthermore, an approach based on the concept of sparse representation for simultaneous training and optimizing the network structure has been presented. According to CIFAR-10 and CIFAR-100 dataset classification tasks and experimental results, the optimization of ResNet and DenseNet structures using gradient descent and heavy-ball algorithms, accordingly, has been shown to result in better performance with increased depth of neural network. A neural network based on a sparse representation is shown to have the highest performance in all datasets. This strategy encourages quick data adaptation at each iteration. The results obtained can be used to design deeper neural networks with no loss of precision and computing speed.
Copyright © 2021 Praise Worthy Prize - All rights reserved.
Keywords
References
A. Bashar, Survey on evolving deep learning neural network architectures, Journal of Artificial Intelligence, Vol. 1(Issue 2): 73-82, 2019.
https://doi.org/10.36548/jaicn.2019.2.003
W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F.E. Alsaadi, A survey of deep neural network architectures and their applications, Neurocomputing, Vol. 234: 11-26, 2017.
https://doi.org/10.1016/j.neucom.2016.12.038
A. Gómez-Ríos, S. Tabik, J. Luengo, A.S.M. Shihavuddin, B. Krawczyk, and F. Herrera, Towards highly accurate coral texture images classification using deep convolutional neural networks and data augmentation, Expert Systems with Applications, Vol. 118: 315-328, 2019.
https://doi.org/10.1016/j.eswa.2018.10.010
Q. Guan, Y. Wang, B. Ping, D. Li, J. Du, Y. Qin, H. Lu, X. Wan, and J. Xiang, Deep convolutional neural network VGG-16 model for differential diagnosing of papillary thyroid carcinomas in cytological images: a pilot study, Journal of Cancer, Vol. 10(Issue 20): 4876, 2019.
https://doi.org/10.7150/jca.28769
W. Tarnowski, P. Warchoł, S. Jastrzȩbski, J. Tabor, and M. Nowak. Dynamical isometry is achieved in residual networks in a universal way for any activation function, in The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2221-2230, PMLR, 2019.
D. Singh, V. Kumar, and M. Kaur, Densely connected convolutional networks-based COVID-19 screening model, Applied Intelligence, Vol. 51(Issue 5): 3044-3051, 2021.
https://doi.org/10.1007/s10489-020-02149-6
Nariman-zadeh, N., Haghgoo, E., Jamali, A., Pareto Optimization of GMDH-Type Neural Networks for Modelling and Prediction of Hoop Strain in Explosive Forming Process, (2020) International Review of Chemical Engineering (IRECHE), 12 (1), pp. 1-11.
https://doi.org/10.15866/ireche.v12i1.19518
Ebhota, V., Srivastava, V., Modeling Environmental Effects on Electromagnetic Signal Propagation Using Multi-Layer Perceptron Artificial Neural Network, (2020) International Journal on Communications Antenna and Propagation (IRECAP), 10 (3), pp. 175-182.
https://doi.org/10.15866/irecap.v10i3.18135
Idroes, R., Noviandy, T., Maulana, A., Suhendra, R., Sasmita, N., Muslem, M., Idroes, G., Kemala, P., Irvanizam, I., Application of Genetic Algorithm-Multiple Linear Regression and Artificial Neural Network Determinations for Prediction of Kovats Retention Index, (2021) International Review on Modelling and Simulations (IREMOS), 14 (2), pp. 137-145.
https://doi.org/10.15866/iremos.v14i2.20460
D.X. Zhou, Universality of deep convolutional neural networks, Applied and computational harmonic analysis, Vol. 48(Issue 2): 787-794, 2020.
https://doi.org/10.1016/j.acha.2019.06.004
Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, Nsga-net: neural architecture search using multi-objective genetic algorithm, in Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419-427, 2019.
https://doi.org/10.1145/3321707.3321729
Bataineh, A., Batayneh, W., Okour, M., Intelligent Control Strategies for Three Degree of Freedom Active Suspension System, (2021) International Review of Automatic Control (IREACO), 14 (1), pp. 17-27.
https://doi.org/10.15866/ireaco.v14i1.20057
J. Ma, F. Lin, S. Wesarg, and M. Erdt, A novel bayesian model incorporating deep neural network and statistical shape model for pancreas segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, pp. 480- 487, 2018.
https://doi.org/10.1007/978-3-030-00937-3_55
S. Singaravel, J. Suykens, and P. Geyer, Deep-learning neural-network architectures and methods: Using component-based models in building-design energy prediction, Advanced Engineering Informatics, Vol. 38: 81-90, 2018.
https://doi.org/10.1016/j.aei.2018.06.004
Jiménez-Moreno, R., Pinzón-Arenas, J., New Hybrid Fuzzy-CNN Architecture for Human-Robot Interaction, (2019) International Review of Automatic Control (IREACO), 12 (5), pp. 236-241.
https://doi.org/10.15866/ireaco.v12i5.17816
S. Akcay, M.E. Kundegorski, C.G. Willcocks, and T.P. Breckon, Using deep convolutional neural network architectures for object classification and detection within x-ray baggage security imagery, IEEE Transactions on Information Forensics and Security, Vol. 13(Issue 9): 2203-2215, 2018.
https://doi.org/10.1109/tifs.2018.2812196
S. Ramjee, S. Ju, D. Yang, X. Liu, A.E. Gamal, and Y.C. Eldar, Fast deep learning for automatic modulation classification, arXiv preprint arXiv:1901.05850, 2019.
Y. Jaafra, J.L. Laurent, A. Deruyver, and M.S. Naceur, Reinforcement learning for neural architecture search: A review, Image and Vision Computing, Vol. 89: 57-66, 2019.
https://doi.org/10.1016/j.imavis.2019.06.005
C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, Progressive neural architecture search, in Proceedings of the European conference on computer vision (ECCV), pp. 19-34, 2018.
https://doi.org/10.1007/978-3-030-01246-5_2
Wang, Z., Al Said, N., Analog Computing and a Hybrid Approach to the Element Base of Artificial Intelligence Applications, (2020) International Review of Automatic Control (IREACO), 13 (5), pp. 206-213.
https://doi.org/10.15866/ireaco.v13i5.19142
H. Guo, J. Zhou, M. Koopialipoor, D.J. Armaghani, and M.M. Tahir, Deep neural network and whale optimization algorithm to assess flyrock induced by blasting, Engineering with Computers, Vol. 37(Issue 1): 173-186, 2021.
https://doi.org/10.1007/s00366-019-00816-y
S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney, Recurrent experience replay in distributed reinforcement learning, International Conference on Learning Representations, 2018.
S. Du, J. Lee, H. Li, L. Wang, and X. Zhai, Gradient descent finds global minima of deep neural networks, International Conference on Machine Learning, PMLR, pp. 1675-1685, 2019.
Z. Chen, H. Cai, Y. Zhang, C. Wu, M. Mu, Z. Li, and M.A. Sotelo, A novel sparse representation model for pedestrian abnormal trajectory understanding, Expert Systems with Applications, Vol. 138: 112753, 2019.
https://doi.org/10.1016/j.eswa.2019.06.041
T. Bouwmans, S. Javed, M. Sultana, and S.K. Jung, Deep neural network concepts for background subtraction: A systematic review and comparative evaluation. Neural Networks, Vol. 117: 8-66, 2019.
https://doi.org/10.1016/j.neunet.2019.04.024
Y. Guo, Y. Liu, T. Georgiou, M.S. Lew, A review of semantic segmentation using deep neural networks, International Journal of Multimedia Information Retrieval, Vol. 7(Issue 2): 87-93, 2018.
https://doi.org/10.1007/s13735-017-0141-z
A. Krizhevsky, I. Sutskever, and G.E. Hinton, ImageNet classification with deep convolutional neural networks, Communications of the ACM, vol. 60(issue 6): 84-90, 2017.
https://doi.org/10.1145/3065386
Z. Meng, L. Li, X. Tang, Z. Feng, L. Jiao, and M. Liang, Multipath residual network for spectral-spatial hyperspectral image classification, Remote Sensing, Vol. 11(Issue 16): 1896, 2019.
https://doi.org/10.3390/rs11161896
G. Li, C. Zhang, R. Lei, X. Zhang, Z. Ye, and X. Li, Hyperspectral remote sensing image classification using three-dimensional-squeeze-and-excitation-DenseNet (3D-SE-DenseNet), Remote Sensing Letters, Vol. 11(Issue 2): 195-203, 2020.
https://doi.org/10.1080/2150704x.2019.1697001
Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying, A mean field analysis of deep ResNet and beyond: Towards provably optimization via overparameterization from depth, in International Conference on Machine Learning, pp. 6426-6436. PMLR, 2020.
J. Schmidt-Hieber, Nonparametric regression using deep neural networks with ReLU activation function, Annals of Statistics, Vol. 48(Issue 4): 1875-1897, 2020.
https://doi.org/10.1214/19-aos1875
A. Onan, Sentiment analysis on product reviews based on weighted word embeddings and deep neural networks, Concurrency and Computation: Practice and Experience, Vol. 1: e5909, 2020.
https://doi.org/10.1002/cpe.5909
S. Zeng, and Y. Huang, A Hybrid-Pipelined Architecture for FPGA-based Binary Weight DenseNet with High Performance-Efficiency, 2020 IEEE High Performance Extreme Computing Conference (HPEC), IEEE, pp. 1-5, 2020.
https://doi.org/10.1109/hpec43674.2020.9286185
S.S. Roy, R. Chopra, K.C. Lee, C. Spampinato, and B. Mohammadi-ivatlood, Random forest, gradient boosted machines and deep neural network for stock price forecasting: a comparative analysis on South Korean companies, International Journal of Ad Hoc and Ubiquitous Computing, Vol. 33(Issue 1): 62-71, 2020.
https://doi.org/10.1504/ijahuc.2020.10026453
S. Hwang, and H. Ikeda, Force balance controls the relaxation time of the gradient descent algorithm in the satisfiable phase, Physical Review E, Vol. 101(Issue 5): 052308, 2020.
https://doi.org/10.1103/physreve.101.052308
S. Gadat, F. Panloup, and S. Saadane, Stochastic heavy ball, Electronic Journal of Statistics, Vol. 12(Issue 1): 461-529, 2018.
https://doi.org/10.1214/18-ejs1395
W.S. Hu, H.C. Li, L. Pan, W. Li, R. Tao, and Q. Du, Spatial–spectral feature extraction via deep ConvLSTM neural networks for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing, Vol. 58(Issue 6): 4237-4250, 2020.
https://doi.org/10.1109/tgrs.2019.2961947
Z. Song, Y. Liu, R. Song, Z. Chen, J. Yang, C. Zhang, and Q. Jiang, A sparsity-based stochastic pooling mechanism for deep convolutional neural networks, Neural Networks, Vol. 105: 340-345, 2018.
https://doi.org/10.1016/j.neunet.2018.05.015
S. Shirakawa, Y. Iwata, and Y. Akimoto, Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32(Issue 1): 4074-4082, 2018.
J. Yoon, E. Gong, I. Chatnuntawech, B. Bilgic, J. Lee, W. Jung, J. Ko, H. Jung, K. Setsompop, G. Zaharchuk, E.Y. Kim, J. Pauly, and J. Lee, Quantitative susceptibility mapping using deep neural network: QSMnet, Neuroimage, Vol. 179: 199-206, 2018.
https://doi.org/10.1016/j.neuroimage.2018.06.030
H. Nakahara, H. Yonekawa, T. Fujii, M. Shimoda, and S. Sato, GUINNESS: A GUI based binarized deep neural network framework for software programmers, IEICE TRANSACTIONS on Information and Systems, Vol. 102(Issue 5): 1003-1011, 2019.
https://doi.org/10.1587/transinf.2018rcp0002
Y. Tu, and Y. Lin, Deep neural network compression technique towards efficient digital signal modulation recognition in edge device, IEEE Access, Vol. 7: 58113-58119, 2019.
https://doi.org/10.1109/access.2019.2913945
Z. Chen, Z. Chen, J. Lin, S. Liu, and W. Li, Deep neural network acceleration based on low-rank approximated channel pruning, IEEE Transactions on Circuits and Systems I: Regular Papers, Vol. 67(Issue 4): 1232-1244, 2020.
https://doi.org/10.1109/tcsi.2019.2958937
Refbacks
- There are currently no refbacks.
Please send any question about this web site to info@praiseworthyprize.com
Copyright © 2005-2024 Praise Worthy Prize
