A Review of Neural Network Lightweighting Techniques
Keywords:
Lightweighting Techniques for Neural Networks, Model Pruning, Network Structure Design, Convolutional Structure OptimizationAbstract
The application of portable devices based on deep learning has become increasingly widespread, which has made the deployment of complex neural networks on embedded devices a hot research topic. Neural network lightweighting is one of the key technologies for applying neural networks to embedded devices. This paper elaborates and analyzes neural network lightweighting techniques from two aspects: model pruning and network structure design. For model pruning, a comparison of methods from different periods is conducted, highlighting their advantages and limitations. Regarding network structure design, the principles of four classical lightweight network designs are described from a mathematical perspective, and the latest optimization methods for these networks are reviewed. Finally, potential research directions for lightweight neural network pruning and structure design optimization are discussed.
Downloads
References
Ge, D., Li, H., Zhang, L., et al. Survey of lightweight neural network. Journal of Software, 2020, 31: 2627-2653.
Kumari, A., Sharma, N. A Review on Convolutional Neural Networks for Skin Lesion Classification. International Conference on Secure Cyber Computing and Communications. IEEE, 2021. https://doi.org/10.1109/icsccc51823.2021.9478151 DOI: https://doi.org/10.1109/ICSCCC51823.2021.9478151
Bouguettaya, A., Kechida, A., TABERKIT, A. M. A survey on lightweight CNN-based object detection algorithms for platforms with limited computational resources. International Journal of Informatics and Applied Mathematics, 2019, 2(2): 28-44.
Jinlin M A, Yu Z, Ziping M A, et al. Research Progress of Lightweight Neural Network Convolution Design. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 512-528. https://doi.org/10.3778/j.issn.1673-9418.2107056
Shen, X., Yi, B., Liu, H., et al. Deep variational matrix factorization with knowledge embedding for recommendation system, IEEE Transactions on Knowledge and Data Engineering, 2019, 33(5): 1906-1918. https://doi.org/10.1109/tkde.2019.2952849 DOI: https://doi.org/10.1109/TKDE.2019.2952849
Li, Z., Li, H., Meng, L. Model Compression for Deep Neural Networks: A Survey. Computers, 2023, 12(3): 60. https://doi.org/10.3390/computers12030060 DOI: https://doi.org/10.3390/computers12030060
Zeng, Y., Xiong, N., Park, J. H., et al. An emergency-adaptive routing scheme for wireless sensor networks for building fire hazard monitoring. Sensors, 2010, 10(6): 6128-6148. https://doi.org/10.3390/s100606128 DOI: https://doi.org/10.3390/s100606128
Li, Y., Liu, J., & Wang, L. Lightweight network research based on deep learning: A review. In 2018 37th Chinese control conference (CCC), IEEE, July, 2018. https://doi.org/10.23919/chicc.2018.8483963 DOI: https://doi.org/10.23919/ChiCC.2018.8483963
Zheng, M., Tian, Y., Chen, H., et al. Lightweight network research based on deep learning. International Conference on Computer Graphics, Artificial Intelligence, and Data Processing (ICCAID 2021). SPIE, 2022, 12168: 333-338. https://doi.org/10.1117/12.2631211 DOI: https://doi.org/10.1117/12.2631211
Xiao, Y., Tian, Z., Yu, J., et al. A review of object detection based on deep learning. Multimedia Tools and Applications, 2020, 79: 23729-23791. https://doi.org/10.1007/s11042-020-08976-6 DOI: https://doi.org/10.1007/s11042-020-08976-6
Wang, C., Huang, K., Yao, Y., et al. Lightweight deep learning: An overview. IEEE Consumer Electronics Magazine, 2022. https://doi.org/10.1109/MCE.2022.3181759 DOI: https://doi.org/10.1109/MCE.2022.3181759
Kang, L., Chen, R., Xiong, N., et al. Selecting hyper-parameters of Gaussian process regression based on non-inertial particle swarm optimization in Internet of Things. IEEE Access, 2019, 7: 59504-59513. https://doi.org/10.1109/access.2019.2913757 DOI: https://doi.org/10.1109/ACCESS.2019.2913757
Zhao, J., Huang, J., Xiong, N. An effective exponential-based trust and reputation evaluation system in wireless sensor networks. IEEE Access, 2019, 7: 33859-33869. https://doi.org/10.1109/access.2019.2904544 DOI: https://doi.org/10.1109/ACCESS.2019.2904544
Yao, J., Li, P., Kang, X., et al. A pruning method based on the dissimilarity of angle among channels and filters. 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2022: 528-532. https://doi.org/10.1109/ictai56018.2022.00084 DOI: https://doi.org/10.1109/ICTAI56018.2022.00084
Cong, S., Zhou, Y. A review of convolutional neural network architectures and their optimizations. Artificial Intelligence Review, 2023, 56(3): 1905-1969. https://doi.org/10.1007/s10462-022-10213-5 DOI: https://doi.org/10.1007/s10462-022-10213-5
Hu, W., Fan, J., Du, Y., et al. MDFC–ResNet: an agricultural IoT system to accurately recognize crop diseases. IEEE Access, 2020, 8: 115287-115298. https://doi.org/10.1109/ACCESS.2020.3001237 DOI: https://doi.org/10.1109/ACCESS.2020.3001237
Huang, S., Zeng, Z., Ota, K., et al. An intelligent collaboration trust interconnections system for mobile information control in ubiquitous 5G networks. IEEE transactions on network science and engineering, 2020, 8(1): 347-365. https://doi.org/10.1109/tnse.2020.3038454 DOI: https://doi.org/10.1109/TNSE.2020.3038454
Anwar, S., Hwang, K., Sung, W. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 2017, 13(3): 1-18. https://doi.org/10.1145/3005348 DOI: https://doi.org/10.1145/3005348
LeCun, Y., Denker, J., Solla, S. Optimal brain damage. Advances in neural information processing systems, 1989, 2: 598-605. https://doi.org/http://dx.doi.org/
Hassibi, B., Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 1992, 5.
Thimm, G., Fiesler, E. Evaluating pruning methods. Proceedings of the International Symposium on Artificial neural networks. 1995: 20-25.
Srinivas, S., Babu, R. V. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149, 2015. https://doi.org/10.5244/c.29.31 DOI: https://doi.org/10.5244/C.29.31
Han, S., Pool, J., Tran, J., et al. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 2015, 28. https://doi.org/10.48550/arXiv.1506.02626
Han, S., Mao, H., Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint , 2015. https://doi.org/10.48550/arXiv.1510.00149
Han, S., Liu, X., Mao, H., et al. EIE: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 2016, 44(3): 243-254. https://doi.org/10.1109/isca.2016.30 DOI: https://doi.org/10.1145/3007787.3001163
Guo, Y., Yao, A., Chen, Y. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 2016, 29. https://doi.org/10.48550/arXiv.1608.04493
Hu, H., Peng, R., Tai, Y., et al. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint, 2016. https://doi.org/10.48550/arXiv.1607.03250
Louizos, C., Welling, M., Kingma, D. P. Learning sparse neural networks through regularization. arXiv preprint, 2017. https://doi.org/10.48550/arXiv.1712.01312
Ye, M., Gong, C., Nie, L., et al. Good subnetworks provably exist: Pruning via greedy forward selection. International Conference on Machine Learning. PMLR, 2020: 10820-10830. https://doi.org/10.48550/arXiv.2003.01794
Frankle, J., Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint, 2018. https://doi.org/10.48550/arXiv.1803.03635
Wang, C., Zhang, G., Grosse, R. Picking winning tickets before training by preserving gradient flow. arXiv preprint, 2020. https://doi.org/10.48550/arXiv.2002.07376
Zhang, T., Ye, S., Zhang, K., et al. StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs. 2018. https://doi.org/10.48550/arXiv.1807.11091
Xue, W., Bai, J., Sun, S., et al. Hierarchical Non-Structured Pruning for Computing-In-Memory Accelerators with Reduced ADC Resolution Requirement. 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2023: 1-6. https://doi.org/10.23919/date56975.2023.10136975 DOI: https://doi.org/10.23919/DATE56975.2023.10136975
Laurent, C., Ballas, C., George, T., et al. Revisiting loss modelling for unstructured pruning. arXiv preprint, 2020. https://doi.org/10.48550/arXiv.2006.12279
Vahidian, S., Morafah, M., Lin, B. Personalized federated learning by structured and unstructured pruning under data heterogeneity. 2021 IEEE 41st international conference on distributed computing systems workshops (ICDCSW). IEEE, 2021: 27-34. https://doi.org/10.48550/arXiv.2105.00562 DOI: https://doi.org/10.1109/ICDCSW53096.2021.00012
Chen, X., Zhu, J., Jiang, J., et al. Tight compression: compressing CNN model tightly through unstructured pruning and simulated annealing based permutation. 2020 57th ACM/IEEE Design Automation Conference (DAC). IEEE, 2020: 1-6. https://doi.org/10.1109/dac18072.2020.9218701 DOI: https://doi.org/10.1109/DAC18072.2020.9218701
Molchanov, P., Tyree, S., Karras, T., et al. Pruning convolutional neural networks for resource efficient inference. arXiv preprint, 2016. https://doi.org/10.48550/arXiv.1611.06440
Molchanov, P., Mallya, A., Tyree, S., et al. Importance estimation for neural network pruning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 11264-11272. https://doi.org/10.1109/cvpr.2019.01152 DOI: https://doi.org/10.1109/CVPR.2019.01152
Luo, J., Wu, J., Lin, W. Thinet: A filter level pruning method for deep neural network compression. Proceedings of the IEEE international conference on computer vision. 2017: 5058-5066. https://doi.org/10.1109/ICCV.2017.541 DOI: https://doi.org/10.1109/ICCV.2017.541
Mondal, M., Das, B., Roy, S. D., et al. Adaptive CNN filter pruning using global importance metric. Computer Vision and Image Understanding, 2022, 222: 103511. https://doi.org/10.1016/j.cviu.2022.103511
Fletcher, P. T., Venkatasubramanian, S., Joshi, S. Robust statistics on Riemannian manifolds via the geometric median. 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2008: 1-8. https://doi.org/10.1109/CVPR.2008.4587747 DOI: https://doi.org/10.1109/CVPR.2008.4587747
Ding, X., Ding, G., Guo, Y., et al. Approximated oracle filter pruning for destructive cnn width optimization. International Conference on Machine Learning. PMLR, 2019: 1607-1616. https://doi.org/10.48550/arXiv.1905.04748
Lin, S., Ji, R., Yan, C., et al. Towards optimal structured cnn pruning via generative adversarial learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2790-2799. https://doi.org/10.1109/cvpr.2019.00290 DOI: https://doi.org/10.1109/CVPR.2019.00290
Gao, X., Zhao, Y., Dudziak, Ł., et al. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint, 2018. https://doi.org/10.48550/arXiv.1810.05331
Wang, Y., Zhang, X., Hu, X., et al. Dynamic network pruning with interpretable layerwise channel selection. Proceedings of the AAAI conference on artificial intelligence. 2020, 34(04): 6299-6306. https://doi.org/10.1609/aaai.v34i04.6098 DOI: https://doi.org/10.1609/aaai.v34i04.6098
Liu, Z., Mu, H., Zhang, X., et al. Metapruning: Meta learning for automatic neural network channel pruning. Proceedings of the IEEE/CVF international conference on computer vision. 2019: 3296-3305. https://doi.org/10.1109/iccv.2019.00339 DOI: https://doi.org/10.1109/ICCV.2019.00339
Li, H., Kadav, A., Durdanovic, I., et al. Pruning filters for efficient convnets. arXiv preprint, 2016. https://doi.org/10.48550/arXiv.1608.08710
Chen, Y., Wen, X., Zhang, Y., et al. CCPrune: Collaborative channel pruning for learning compact convolutional networks. Neurocomputing, 2021, 451: 35-45. https://doi.org/10.1016/j.neucom.2021.04.063 DOI: https://doi.org/10.1016/j.neucom.2021.04.063
Mondal, M., Das, B., Roy, S. D., et al. Adaptive CNN filter pruning using global importance metric. Computer Vision and Image Understanding, 2022, 222: 103511. https://doi.org/10.1016/j.cviu.2022.103511 DOI: https://doi.org/10.1016/j.cviu.2022.103511
Tang, Y., Wang, Y., Xu, Y., et al. Manifold regularized dynamic network pruning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 5018-5028. https://doi.org/10.1109/cvpr46437.2021.00498 DOI: https://doi.org/10.1109/CVPR46437.2021.00498
Lin, M., Ji, R., Wang, Y., et al. Hrank: Filter pruning using high-rank feature map. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 1529-1538. https://doi.org/10.1109/cvpr42600.2020.00160 DOI: https://doi.org/10.1109/CVPR42600.2020.00160
Polyak, A., Wolf, L. Channel-level acceleration of deep face representations. IEEE Access, 2015, 3: 2163-2175. https://doi.org/10.1109/access.2015.2494536 DOI: https://doi.org/10.1109/ACCESS.2015.2494536
He, Y., Zhang, X., Sun, J. Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE international conference on computer vision. 2017: 1389-1397. https://doi.org/10.1109/ICCV.2017.155 DOI: https://doi.org/10.1109/ICCV.2017.155
Yu, R., Li, A., Chen, C., et al. NISP: Pruning Networks Using Neuron Importance Score Propagation. IEEE, 2018. https://doi.org/10.1109/CVPR.2018.00958 DOI: https://doi.org/10.1109/CVPR.2018.00958
Liu, Z., Li, J., Shen, Z.,et al. Learning Efficient Convolutional Networks through Network Slimming. IEEE, 2017. https://doi.org/10.1109/ICCV.2017.298 DOI: https://doi.org/10.1109/ICCV.2017.298
Huang, Z., Wang, N. Data-driven sparse structure selection for deep neural networks. Proceedings of the European conference on computer vision (ECCV). 2018: 304-320. https://doi.org/10.1007/978-3-030-01270-0_19 DOI: https://doi.org/10.1007/978-3-030-01270-0_19
Zhuang, Z., Tan, M., Zhuang, B., et al. Discrimination-aware channel pruning for deep neural networks. Advances in neural information processing systems, 2018: 31. https://doi.org/10.48550/arXiv.1810.11809
Ye, J., Lu, X., Lin, Z., et al. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint, 2018. https://doi.org/10.48550/arXiv.1802.00124
Ye, Y., You, G., Fwu, J. K., et al. Channel pruning via optimal thresholding. Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18–22, 2020, Proceedings, Part V 27. Springer International Publishing, 2020: 508-516. https://doi.org/10.1007/978-3-030-63823-8_58 DOI: https://doi.org/10.1007/978-3-030-63823-8_58
Li, Y., Adamczewski, K., Li, W., et al. Revisiting random channel pruning for neural network compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 191-201. https://doi.org/10.1109/CVPR52688.2022.00029 DOI: https://doi.org/10.1109/CVPR52688.2022.00029
Yang, C., Liu, H. Channel pruning based on convolutional neural network sensitivity. Neurocomputing, 2022, 507: 97-106. https://doi.org/10.1016/j.neucom.2022.07.051 DOI: https://doi.org/10.1016/j.neucom.2022.07.051
Liu, N., Ma, X., Xu, Z., et al. Autocompress: An automatic dnn structured pruning framework for ultra-high compression rates. Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 4876-4883. https://doi.org/10.1609/aaai.v34i04.5924 DOI: https://doi.org/10.1609/aaai.v34i04.5924
Zhou, Y., Zhang, Y., Liu, H., et al. A bare-metal and asymmetric partitioning approach to client virtualization. IEEE Transactions on Services Computing, 2012, 7(1): 40-53. https://doi.org/10.1109/TSC.2012.32 DOI: https://doi.org/10.1109/TSC.2012.32
Wang, H., Fu, Y. Trainability preserving neural structured pruning. arXiv preprint arXiv:2207.12534, 2022.
Xiong, N., Han, W., Vandenberg, A. Green cloud computing schemes based on networks: a survey. Iet Communications, 2012, 6(18): 3294-3300. https://doi.org/10.1049/iet-com.2011.0293 DOI: https://doi.org/10.1049/iet-com.2011.0293
Fang, G., Ma, X., Song, M., et al. Depgraph: Towards any structural pruning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 16091-16101. https://doi.org/10.1109/cvpr52729.2023.01544 DOI: https://doi.org/10.1109/CVPR52729.2023.01544
Hanson, E., Li, S., Li, H., et al. Cascading structured pruning: enabling high data reuse for sparse DNN accelerators. Proceedings of the 49th Annual International Symposium on Computer Architecture. 2022: 522-535. https://doi.org/10.1145/3470496.3527419 DOI: https://doi.org/10.1145/3470496.3527419
Bhalgaonkar, S. A., Munot, M. V., Anuse, A. D. Pruning for compression of visual pattern recognition networks: a survey from deep neural networks perspective. Pattern recognition and data analysis with applications, 2022: 675-687. https://doi.org/10.1007/978-981-19-1520-8_55 DOI: https://doi.org/10.1007/978-981-19-1520-8_55
Choudhary, T., Mishra, V., Goswami, A., et al. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, 2020, 53: 5113-5155. https://doi.org/10.1007/s10462-020-09816-7 DOI: https://doi.org/10.1007/s10462-020-09816-7
Wang, J., Jin, C., Tang, Q., et al. Intelligent ubiquitous network accessibility for wireless-powered MEC in UAV-assisted B5G, IEEE Transactions on Network Science and Engineering, 2020, 8 (4): 2801-2813. https://doi.org/10.1109/TNSE.2020.3029048 DOI: https://doi.org/10.1109/TNSE.2020.3029048
Zhang, W., Zhu, S., Tang, J., et al. A novel trust management scheme based on Dempster–Shafer evidence theory for malicious nodes detection in wireless sensor networks, The Journal of Supercomputing, 2018, 74 (4): 1779-1801. https://doi.org/10.1007/s11227-017-2150-3 DOI: https://doi.org/10.1007/s11227-017-2150-3
Simonyan, K., Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint, 2014. https://doi.org/10.48550/arXiv.1409.1556
Huang, G., Liu, Z., Pleiss, G., et al. Convolutional networks with dense connectivity. IEEE transactions on pattern analysis and machine intelligence, 2019, 44(12): 8704-8716. https://doi.org/10.1109/TPAMI.2019.2918284 DOI: https://doi.org/10.1109/TPAMI.2019.2918284
Han, D., Kim, J., Kim, J. Deep pyramidal residual networks. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 5927-5935. https://doi.org/10.1109/cvpr.2017.668 DOI: https://doi.org/10.1109/CVPR.2017.668
Iandola, F. N., Han, S., Moskewicz, M. W., et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size. arXiv preprint, 2016. https://doi.org/10.48550/arXiv.1602.07360
Krizhevsky, A., Sutskever, I., Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012: 25. https://doi.org/10.1145/3065386 DOI: https://doi.org/10.1145/3065386
Gholami, A., Kwon, K., Wu, B., et al. Squeezenext: Hardware-aware neural network design. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018: 1638-1647. https://doi.org/10.1109/CVPRW.2018.00215 DOI: https://doi.org/10.1109/CVPRW.2018.00215
MS, M., SS, S. R. Optimal Squeeze Net with Deep Neural Network-Based Arial Image Classification Model in Unmanned Aerial Vehicles. Traitement du Signal, 2022, 39(1): 275-281. https://doi.org/10.18280/ts.390128 DOI: https://doi.org/10.18280/ts.390128
Pierezan, J., Coelho, L. D. S. Coyote optimization algorithm: a new metaheuristic for global optimization problems. 2018 IEEE congress on evolutionary computation (CEC). IEEE, 2018: 1-8. https://doi.org/10.1109/CEC.2018.8477769 DOI: https://doi.org/10.1109/CEC.2018.8477769
Bernardo, L. S., Damaševičius, R., Ling, S., et al. Modified squeezenet architecture for parkinson’s disease detection based on keypress data. Biomedicines, 2022, 10(11): 2746. https://doi.org/10.3390/biomedicines10112746 DOI: https://doi.org/10.3390/biomedicines10112746
Nirmalapriya, G., Maram, B., Lakshmanan, R., et al. ASCA-squeeze net: Aquila sine cosine algorithm enabled hybrid deep learning networks for digital image forgery detection. Computers & Security, 2023, 128: 103155. https://doi.org/10.1016/j.cose.2023.103155 DOI: https://doi.org/10.1016/j.cose.2023.103155
Han, K., Wang, Y., Tian, Q., et al. Ghostnet: More features from cheap operations. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 1580-1589. https://doi.org/10.1109/CVPR42600.2020.00165 DOI: https://doi.org/10.1109/CVPR42600.2020.00165
Howard, A., Sandler, M., Chu, G., et al. Searching for mobilenetv3. Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1314-1324. https://doi.org/10.1109/ICCV.2019.00140 DOI: https://doi.org/10.1109/ICCV.2019.00140
Yuan, X., Li, D., Sun, P., et al. Real-Time Counting and Height Measurement of Nursery Seedlings Based on Ghostnet–YoloV4 Network and Binocular Vision Technology. Forests. 2022, 13(9):1459. https://doi.org/10.3390/f13091459 DOI: https://doi.org/10.3390/f13091459
Chi, J., Guo, S., Zhang, H., et al. L-GhostNet: Extract Better Quality Features. IEEE Access, 2023, 11: 2361-2374. https://doi.org/10.1109/access.2023.3234108 DOI: https://doi.org/10.1109/ACCESS.2023.3234108
Ke, X., Hou, W., Meng, L. Research on Pet Recognition Algorithm With Dual Attention GhostNet-SSD and Edge Devices. IEEE Access, 2022, 10: 131469-131480. https://doi.org/10.1109/ACCESS.2022.3228808 DOI: https://doi.org/10.1109/ACCESS.2022.3228808
Wang, X., Kan, M., Shan, S., et al. Fully learnable group convolution for acceleration of deep neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 9049-9058. https://doi.org/10.1109/CVPR.2019.00926 DOI: https://doi.org/10.1109/CVPR.2019.00926
Cohen, T., Welling, M. Group equivariant convolutional networks. International conference on machine learning. PMLR, 2016: 2990-2999. https://doi.org/10.48550/arXiv.1602.07576
Zhang, J., Zhao, H., Yao, A., et al. Efficient semantic scene completion network with spatial group convolution. Proceedings of the European Conference on Computer Vision (ECCV). 2018: 733-749. https://doi.org/10.1007/978-3-030-01258-8_45 DOI: https://doi.org/10.1007/978-3-030-01258-8_45
Zhang, X., Zhou, X., Lin, M., et al. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 6848-6856. https://doi.org/10.1109/CVPR.2018.00716 DOI: https://doi.org/10.1109/CVPR.2018.00716
Ma, N., Zhang, X., Zheng, H., et al. Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European conference on computer vision (ECCV). 2018: 116-131. https://doi.org/10.1007/978-3-030-01264-9_8 DOI: https://doi.org/10.1007/978-3-030-01264-9_8
Vu, D. Q., Le, N. T., Wang, J. (2+ 1) D Distilled ShuffleNet: A Lightweight Unsupervised Distillation Network for Human Action Recognition. 2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022: 3197-3203. https://doi.org/10.1109/icpr56361.2022.9956634 DOI: https://doi.org/10.1109/ICPR56361.2022.9956634
Chen, Z., Yang, J., Chen, L., et al. Garbage classification system based on improved ShuffleNet v2. Resources, Conservation and Recycling, 2022, 178: 106090. https://doi.org/10.1016/j.resconrec.2021.106090 DOI: https://doi.org/10.1016/j.resconrec.2021.106090
Wang, Y., Xu, X., Wang, Z., et al. ShuffleNet-Triplet: A lightweight RE-identification network for dairy cows in natural scenes. Computers and Electronics in Agriculture, 2023, 205: 107632. https://doi.org/10.2139/ssrn.4227546 DOI: https://doi.org/10.1016/j.compag.2023.107632
Howard, A. G., Zhu, M., Chen, B., et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint, 2017. https://doi.org/10.48550/arXiv.1704.04861
Sandler, M., Howard, A., Zhu, M., et al. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4510-4520. https://doi.org/10.1109/CVPR.2018.00474 DOI: https://doi.org/10.1109/CVPR.2018.00474
Chen, Y., Dai, X., Chen, D., et al. Mobile-former: Bridging mobilenet and transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 5270-5279. https://doi.org/10.1109/CVPR52688.2022.00520 DOI: https://doi.org/10.1109/CVPR52688.2022.00520
Nan, Y., Ju, J., Hua, Q., et al. A-MobileNet: An approach of facial expression recognition. Alexandria Engineering Journal, 2022, 61(6): 4435-4444. https://doi.org/10.1016/j.aej.2021.09.066 DOI: https://doi.org/10.1016/j.aej.2021.09.066
Huang, J., Mei, L., Long, M., et al. Bm-net: Cnn-based mobilenet-v3 and bilinear structure for breast cancer detection in whole slide images. Bioengineering, 2022, 9(6): 261. https://doi.org/10.3390/bioengineering9060261 DOI: https://doi.org/10.3390/bioengineering9060261
Zhang, K., Cheng, K., Li, J., et al. A channel pruning algorithm based on depth-wise separable convolution unit. IEEE Access, 2019, 7: 173294-173309. https://doi.org/10.1109/ACCESS.2019.2956976 DOI: https://doi.org/10.1109/ACCESS.2019.2956976
Shen, Y., Fang, Z., Gao, Y., et al., Coronary arteries segmentation based on 3D FCN with attention gate and level set function, Ieee Access , 2019,7: 42826-42835. https://doi.org/10.1109/ACCESS.2019.2908039 DOI: https://doi.org/10.1109/ACCESS.2019.2908039
Hung, K. W., Zhang, Z., Jiang, J. Real-time image super-resolution using recursive depthwise separable convolution network. IEEE Access, 2019, 7: 99804-99816. https://doi.org/10.1109/ACCESS.2019.2929223 DOI: https://doi.org/10.1109/ACCESS.2019.2929223
Wang, G., Ding, H., Li, B., et al. Trident‐YOLO: Improving the precision and speed of mobile device object detection. IET Image Processing, 2022, 16(1): 145-157. https://doi.org/10.1049/ipr2.12340 DOI: https://doi.org/10.1049/ipr2.12340
Wan, R., Xiong, N., Hu, Q., et al. Similarity-aware data aggregation using fuzzy c-means approach for wireless sensor networks, EURASIP Journal on Wireless Communications and Networking, 2019: 1-11. https://doi.org/10.1186/s13638-019-1374-8 DOI: https://doi.org/10.1186/s13638-019-1374-8
Yang, S., Xing, Z., Wang, H., et al. Maize-YOLO: a new high-precision and real-time method for maize pest detection. Insects, 2023, 14(3): 278. https://doi.org/10.3390/insects14030278 DOI: https://doi.org/10.3390/insects14030278
Tan, M., Chen, B., Pang, R., et al. Mnasnet: Platform-aware neural architecture search for mobile. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2820-2828. https://doi.org/10.48550/arXiv.1807.11626 DOI: https://doi.org/10.1109/CVPR.2019.00293
Huang, G., Liu, S., Maaten, L. V., et al. Condensenet: An efficient densenet using learned group convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 2752-2761. https://doi.org/10.1109/CVPR.2018.00291 DOI: https://doi.org/10.1109/CVPR.2018.00291
Mehta, S., Rastegari, M., Caspi, A., et al. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. Proceedings of the european conference on computer vision (ECCV). 2018: 552-568. https://doi.org/10.1007/978-3-030-01249-6_34 DOI: https://doi.org/10.1007/978-3-030-01249-6_34
Mehta, S., Rastegari, M., Shapiro, L., et al. Espnetv2: A light-weight, power efficient, and general purpose convolutional neural network. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 9190-9200. https://doi.org/10.1109/CVPR.2019.00941 DOI: https://doi.org/10.1109/CVPR.2019.00941
Gao, H., Wang, Z., Ji, S. Channelnets: Compact and efficient convolutional neural networks via channel-wise convolutions. Advances in neural information processing systems, 2018, 31. https://doi.org/10.1109/TPAMI.2020.2975796 DOI: https://doi.org/10.1109/TPAMI.2020.2975796
Wang, R., Li, X., Ling, C. Pelee: A real-time object detection system on mobile devices. Advances in neural information processing systems, 2018, 31. https://doi.org/10.48550/arXiv.1804.06882
Zhang, T., Qi, G., Xiao, B., et al. Interleaved group convolutions. Proceedings of the IEEE international conference on computer vision. 2017: 4373-4382. https://doi.org/10.1109/ICCV.2017.469 DOI: https://doi.org/10.1109/ICCV.2017.469
Xie, G., Wang, J., Zhang, T., et al. Interleaved structured sparse convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8847-8856. https://doi.org/10.48550/arXiv.1804.06202 DOI: https://doi.org/10.1109/CVPR.2018.00922
Sun, K., Li, M., Liu, D., et al. Igcv3: Interleaved low-rank group convolutions for efficient deep neural networks. arXiv preprint, 2018. https://doi.org/10.48550/arXiv.1806.00178
Wu, B., Dai, X., Zhang, P., et al. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 10734-10742. https://doi.org/10.1109/CVPR.2019.01099 DOI: https://doi.org/10.1109/CVPR.2019.01099
Wan, A., Dai, X., Zhang, P., et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 12965-12974. https://doi.org/10.1109/cvpr42600.2020.01298 DOI: https://doi.org/10.1109/CVPR42600.2020.01298
Dai, X., Wan, A., Zhang, P., et al. Fbnetv3: Joint architecture-recipe search using predictor pretraining. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 16276-16285. https://doi.org/10.1109/cvpr46437.2021.01601 DOI: https://doi.org/10.1109/CVPR46437.2021.01601
Koonce, B. EfficientNet. Convolutional Neural Networks with Swift for Tensorflow: Image Recognition and Dataset Categorization, 2021: 109-123. https://doi.org/10.1007/978-1-4842-6168-2 DOI: https://doi.org/10.1007/978-1-4842-6168-2_10
Tan, M., Le, Q. Efficientnetv2: Smaller models and faster training. International conference on machine learning. PMLR, 2021: 10096-10106. https://doi.org/10.48550/arXiv.2104.00298
Ma, N., Zhang, X., Huang, J., et al. Weightnet: Revisiting the design space of weight networks. European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 776-792. https://doi.org/10.1007/978-3-030-58555-6_46 DOI: https://doi.org/10.1007/978-3-030-58555-6_46
Li, Y., Chen, Y., Dai, X., et al. Micronet: Improving image recognition with extremely low flops. Proceedings of the IEEE/CVF International conference on computer vision. 2021: 468-477. https://doi.org/10.48550/arXiv.2108.05894 DOI: https://doi.org/10.1109/ICCV48922.2021.00052
Tsivgoulis, M., Papastergiou, T., Megalooikonomou, V. An improved SqueezeNet model for the diagnosis of lung cancer in CT scans. Machine Learning with Applications, 2022, 10: 100399. https://doi.org/10.1016/j.mlwa.2022.100399 DOI: https://doi.org/10.1016/j.mlwa.2022.100399
Mishra, D., Singh, S. K., Singh, R. K. Deep architectures for image compression: a critical review. Signal Processing, 2022, 191: 108346. https://doi.org/10.1016/j.sigpro.2021.108346 DOI: https://doi.org/10.1016/j.sigpro.2021.108346
Wang, Y., Fang, W., Ding, Y., et al. Computation offloading optimization for UAV-assisted mobile edge computing: a deep deterministic policy gradient approach, Wireless Networks, 2021, 27 (4): 2991-3006. https://doi.org/10.1007/s11276-021-02632-z DOI: https://doi.org/10.1007/s11276-021-02632-z
Veza, I., Afzal, A., Mujtaba, M. A., et al. Review of artificial neural networks for gasoline, diesel and homogeneous charge compression ignition engine. Alexandria Engineering Journal, 2022, 61(11): 8363-8391. https://doi.org/10.1016/j.aej.2022.01.072 DOI: https://doi.org/10.1016/j.aej.2022.01.072
Liu, Z., Sun, M., Zhou, T., et al. Rethinking the value of network pruning. arXiv preprint, 2018. https://doi.org/10.48550/arXiv.1810.05270
Wang, W., Chen, M., Zhao, S., et al. Accelerate cnns from three dimensions: A comprehensive pruning framework. International Conference on Machine Learning. PMLR, 2021: 10717-10726. https://doi.org/10.48550/arXiv.2010.04879
Zhou, J., Cui, G., Hu, S., et al. Graph neural networks: A review of methods and applications. AI open, 2020, 1: 57-81. https://doi.org/10.1016/j.aiopen.2021.01.001 DOI: https://doi.org/10.1016/j.aiopen.2021.01.001
Wu, Z., Pan, S., Chen, F., et al. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020, 32(1): 4-24. https://doi.org/10.1109/TNNLS.2020.2978386 DOI: https://doi.org/10.1109/TNNLS.2020.2978386
Scarselli, F., Gori, M., Tsoi, A. C., et al. The graph neural network model. IEEE transactions on neural networks, 2008, 20(1): 61-80. https://doi.org/10.1109/TNN.2008.2005605 DOI: https://doi.org/10.1109/TNN.2008.2005605
Han, K., Wang, Y., Chen, H., et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(1): 87-110. https://doi.org/10.1109/TPAMI.2022.3152247 DOI: https://doi.org/10.1109/TPAMI.2022.3152247
Zhou, D., Kang, B., Jin, X., et al. Deepvit: Towards deeper vision transformer. arXiv preprint, 2021. https://doi.org/10.48550/arXiv.2103.11886
Khan, S., Naseer, M., Hayat, M., et al. Transformers in vision: A survey. ACM computing surveys (CSUR), 2022, 54(10s): 1-41. https://doi.org/10.48550/arXiv.2101.01169 DOI: https://doi.org/10.1145/3505244
Liang, W., Xie, S., Cai, J., et al. Novel private data access control scheme suitable for mobile edge computing. China Communications, 2021, 18(11): 92-103. https://doi.org/10.23919/jcc.2021.11.007 DOI: https://doi.org/10.23919/JCC.2021.11.007
Liang, W., Li, Y., Xie, K., et al. Spatial-temporal aware inductive graph neural network for C-ITS data recovery. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(8): 8431–8442. https://doi.org/10.1109/tits.2022.3156266 DOI: https://doi.org/10.1109/TITS.2022.3156266
Published
How to Cite
Issue
Section
Copyright (c) 2023 Ziyi Gong, Huifu Zhang, Hao Yang, Fangjun Liu, Fan Luo
This work is licensed under a Creative Commons Attribution 4.0 International License.
