切换至 "中华医学电子期刊资源库"

中华医学超声杂志(电子版) ›› 2024, Vol. 21 ›› Issue (02) : 128 -136. doi: 10.3877/cma.j.issn.1672-6448.2024.02.004

心血管超声影像学

基于深度学习的超声心动图动态图像切面识别研究
成汉林1, 史中青2, 戚占如2, 王小贤2, 曾子炀3, 单淳劼1, 钱隼南4, 罗守华1, 姚静2,()   
  1. 1. 210096 南京,东南大学生物科学与医学工程学院
    2. 210008 南京,南京大学医学院附属鼓楼医院超声医学科;210008 南京,南京大学医学院附属鼓楼医院医学影像中心;211400 扬州,南京鼓楼医院集团仪征医院
    3. 215123 苏州,东南大学苏州联合研究院
    4. 210009 南京,江苏省省级机关医院信息处
  • 收稿日期:2023-06-18 出版日期:2024-02-01
  • 通信作者: 姚静
  • 基金资助:
    国家自然科学基金(61871126); 江苏省重点研发计划(BE2022828); 江苏省前沿引领技术基础研究专项(BK20222002); 江苏省卫生健康委2022年度医学科研项目(281); 南京鼓楼医院临床研究专项(2022-YXZX-YX-01)

Deep learning-based two-dimensional echocardiographic dynamic image view recognition

Hanlin Cheng1, Zhongqing Shi2, Zhanru Qi2, Xiaoxian Wang2, Ziyang Zeng3, Chunjie Shan1, Sunnan Qian4, Shouhua Luo1, Jing Yao2,()   

  1. 1. School of Biological Sciences and Medical Engineering, Southeast University, Nanjing 210096, China
    2. Department of Ultrasound Medicine, Affiliated Hospital of Medical School, Nanjing University, Nanjing 210008, China;Medical Imaging Centre, Affiliated Hospital of Medical School, Nanjing University, Nanjing 210008, China;Yizheng Hospital of Nanjing Drum Tower Hospital Group, Yangzhou 211400, China
    3. Suzhou Joint Research Institute, Southeastern University, Suzhou 215123, China
    4. Department of Information Office, Jiangsu Province Official Hospital, Nanjing 210009, China
  • Received:2023-06-18 Published:2024-02-01
  • Corresponding author: Jing Yao
引用本文:

成汉林, 史中青, 戚占如, 王小贤, 曾子炀, 单淳劼, 钱隼南, 罗守华, 姚静. 基于深度学习的超声心动图动态图像切面识别研究[J/OL]. 中华医学超声杂志(电子版), 2024, 21(02): 128-136.

Hanlin Cheng, Zhongqing Shi, Zhanru Qi, Xiaoxian Wang, Ziyang Zeng, Chunjie Shan, Sunnan Qian, Shouhua Luo, Jing Yao. Deep learning-based two-dimensional echocardiographic dynamic image view recognition[J/OL]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(02): 128-136.

目的

提出一种基于深度学习的切面识别模型SlowFast-Echo,进行二维经胸超声心动图动态图像的切面类型自动识别。

方法

选取2022年8月至12月在南京大学医学院附属鼓楼医院超声医学科完成二维经胸超声心动图检查的722例受检者(含心尖二腔、心尖三腔与心尖四腔等9类临床检查常用切面,共2243个动态图像),各类切面图像按照5∶2∶3的比例划分为训练集、验证集和测试集。进行SlowFast-Echo模型的训练和验证后,以准确率、精度、召回率、F1分数对模型的切面识别性能进行定量评价,以类激活映射图对模型的可解释性进行定性评价,以模型实地部署到超声医学科后的表现进行实用性评价。

结果

SlowFast-Echo模型对测试集动态图像切面类型预测的整体准确率、精度、召回率与F1分数分别为0.9866、0.9847、0.9872与0.9859;显著性热力图表明模型关注区域与超声科医师基本一致,如模型准确地定位到了肋骨旁短轴大血管水平切面(PSAXGV)显著的主动脉及主动脉瓣、胸骨旁短轴二尖瓣水平切面(PSAXMV)的二尖瓣与胸骨旁短轴乳头肌水平切面(PSAXPM)的乳头肌。实地部署后模型切面识别的整体准确率、精度、召回率与F1分数分别为0.9903、0.9865、0.9868与0.9865;在RTX 3060 GPU上单个动态图像的平均推理时间平均值为(303.2±119.3)ms,基本满足采图后即时处理的临床需求。

结论

本研究提出的SlowFast-Echo模型有着良好的二维经胸超声心动图动态图像切面识别性能与推理实时性,实用性较强,具有较好的应用前景。

Objective

To propose a deep learning-based view recognition model, SlowFast-Echo, for the automatic view recognition of two-dimensional (2D) transthoracic echocardiographic dynamic images.

Methods

From August to December 2022, 722 patients who underwent 2D transthoracic echocardiography at the Department of Ultrasound Medicine, Affiliated Hospital of Medical School, Nanjing University (9 types of clinically commonly used views [including apical two-chamber, apical three-chamber, and apical four-chamber views], with a total of 2243 dynamic images) were selected, and the images of each view were divided into training set, validation set, and test set in a ratio of 5:2:3. After training and validation of the SlowFast-Echo model, the performance of the model was evaluated quantitatively in terms of accuracy, precision, recall, and F1 score, qualitatively in terms of the interpretability of the model with regard to class activation mapping, and practically in terms of the performance of the model after field deployment to the ultrasound medicine department.

Results

The overall accuracy, precision, recall, and F1 score of the SlowFast-Echo model for dynamic image view recognition in the test set were 0.9866, 0.9847, 0.9872, and 0.9859, respectively, and the significance heatmap indicated that the model's regions of interest were generally consistent with those drawn by the physicians; e.g., the model accurately pinpointed the significant aorta and aortic valve in parasternal short axis view of great vessel (PSAXGV) view, mitral valve in parasternal short axis view of left ventricle at mitral value level (PSAXMV) view, and papillary muscles in parasternal short axis view of left ventricle at papillary muscle level (PSAXPM) view. The overall accuracy, precision, recall, and F1 score of the model for view recognition after deployment were 0.9903, 0.9865, 0.9868, and 0.9865, respectively, and the average inference time on RTX 3060 GPU for a single dynamic image was (303.2±119.3) ms, which basically meets the clinical demand for immediate processing after image acquisition.

Conclusion

The SlowFast-Echo model proposed in this study has good performance in view recognition of 2D transthoracic echocardiographic dynamic images and inference in real time, which is practically useful.

图1 本研究自动识别的二维超声心动图9类切面示意图 注:A2C为心尖二腔切面;A3C为心尖三腔切面;A4C为心尖四腔切面;A5C为心尖五腔切面;PLAX为胸骨旁长轴左心室切面;PSAXGV为肋骨旁短轴大血管水平切面;PSAXMV为胸骨旁短轴二尖瓣水平切面;PSAXPM为胸骨旁短轴乳头肌水平切面;PSAXA为胸骨旁短轴心尖水平切面
表1 9类切面数据情况
图2 SlowFast-Echo模型示意图
表2 各类切面视频测试集识别性能表现
图3 各类切面视频识别结果混淆矩阵 注:A2C为心尖二腔切面;A3C为心尖三腔切面;A4C为心尖四腔切面;A5C为心尖五腔切面;PLAX为胸骨旁长轴左心室切面;PSAXGV为肋骨旁短轴大血管水平切面;PSAXMV为胸骨旁短轴二尖瓣水平切面;PSAXPM为胸骨旁短轴乳头肌水平切面;PSAXA为胸骨旁短轴心尖水平切面
图4 各类切面动态图像与类别显著性热力图合成图 注:A2C为心尖二腔切面;A3C为心尖三腔切面;A4C为心尖四腔切面;A5C为心尖五腔切面;PLAX为胸骨旁长轴左心室切面;PSAXGV为肋骨旁短轴大血管水平切面;PSAXMV为胸骨旁短轴二尖瓣水平切面;PSAXPM为胸骨旁短轴乳头肌水平切面;PSAXA为胸骨旁短轴心尖水平切面
表3 实地部署后各切面识别性能表现与推理耗时情况
1
Cleve J, McCulloch ML. Conducting a cardiac ultrasound examination [J]. Echocardiography, 2018: 33-42.
2
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks [J]. Commun ACM, 2017, 60(6): 84-90.
3
陶攀, 付忠良, 朱锴, 等.基于深度学习的超声心动图切面识别方法 [J]. 计算机应用, 2017, 37(5): 1434-1438.
4
Madani A, Arnaout R, Mofrad M, et al. Fast and accurate view classification of echocardiograms using deep learning [J]. NPJ Digital Med, 2018, 1: 6.
5
Zhang J, Gajjala S, Agrawal P, et al. Fully automated echocardiogram interpretation in clinical practice: feasibility and diagnostic accuracy [J]. Circulation, 2018, 138(16): 1623-1635.
6
Østvik A, Smistad E, Aase SA, et al. Real-time standard view classification in transthoracic echocardiography using convolutional neural networks [J]. Ultrasound Med Biol, 2019, 45(2): 374-384.
7
Kusunose K, Haga A, Inoue M, et al. Clinically feasible and accurate view classification of echocardiographic images using deep learning [J]. Biomolecules, 2020, 10(5): 665.
8
Santosh Kumar BP, Haq MA, Sreenivasulu P, et al. Fine-tuned convolutional neural network for different cardiac view classification [J]. J Supercomput, 2022, 78(16): 18318-18335.
9
Gao X, Li W, Loomes M, et al. A fused deep learning architecture for viewpoint classification of echocardiography [J]. Inform Fusion, 2017, 36: 103-113.
10
Howard JP, Tan J, Shun-Shin MJ, et al. Improving ultrasound video classification: an evaluation of novel deep learning methods in echocardiography [J]. J Med Artif Intell, 2020, 3: 4.
11
Feichtenhofer C, Fan H, Malik J, et al. Slowfast networks for video recognition [C]. Proceedings of the IEEE/CVF international conference on computer vision, Seoul, Korea (South), 2019: 6202-6211. Piscataway, NJ: IEEE Computer Society, 2019.
12
Cubuk ED, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space [C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA, 2020: 702-703. Piscataway, NJ: IEEE Computer Society, 2020.
13
Zhong Z, Zheng L, Kang G, et al. Random erasing data augmentation [C]. Proceedings of the AAAI conference on artificial intelligence, New York, USA, 2020, 34(7): 13001-13008. Menlo Park, CA: AAAI, 2020.
14
He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners [C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022: 16000-16009. Piscataway, NJ: IEEE Computer Society, 2022.
15
Wang X, Girshick R, Gupta A, et al. Non-local neural networks [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA, 2018: 7794-7803. Piscataway, NJ: IEEE Computer Society, 2018.
16
He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 2016: 770-778. Piscataway, NJ: IEEE Computer Society, 2016.
17
Vanholder H. Efficient inference with tensorrt [C]. GPU Technology Conference, Sunny San Jose, California, USA, 2016, 1: 2. Santa Clara, CA: Nvidia, 2016.
18
Zhou B, Khosla A, Lapedriza A, et al. Learning deep features for discriminative localization [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 2016: 2921-2929. Piscataway, NJ: IEEE Computer Society, 2016.
19
姜玉新, 李建初, 王红燕, 等.信息化技术助力超声医学质量控制新发展 [J/OL].中华医学超声杂志(电子版),2021,18(7): 625-628.
20
Huang MS, Wang CS, Chiang JH, et al. Automated recognition of regional wall motion abnormalities through deep neural network interpretation of transthoracic echocardiography [J]. Circulation, 2020, 142(16): 1510-1520.
21
Huang KC, Huang CS, Su MY, et al. Artificial intelligence aids cardiac image quality assessment for improving precision in strain measurements [J]. JACC Cardiovasc Imaging, 2021, 14(2): 335-345.
22
Lane ES, Azarmehr N, Jevsikov J, et al. Multibeat echocardiographic phase detection using deep neural networks [J]. Comput Biol Med, 2021, 133: 104373.
23
吴洋, 张红梅, 尹立雪, 等.超声心动图心尖四腔心切面图像质量智能评分研究[J/OL].中华医学超声杂志(电子版), 2023, 20(1): 97-102.
24
Hasani R, Lechner M, Amini A, et al. Liquid time-constant networks [C]. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, Canada, 2021, 35(9): 7657-7666. Menlo Park, CA: AAAI, 2021.
[1] 李晓妮, 卫青, 孟庆龙, 牛丽莉, 田月, 吴伟春, 朱振辉, 王浩. 超声心动图在孤立性左心室心尖发育不良疾病中的应用价值[J/OL]. 中华医学超声杂志(电子版), 2024, 21(10): 937-942.
[2] 陈慧, 姚静, 张宁, 刘磊, 马秀玲, 王小贤, 方爱娟, 管静静. 超声心动图在多发性骨髓瘤心脏淀粉样变中的诊断价值[J/OL]. 中华医学超声杂志(电子版), 2024, 21(10): 943-949.
[3] 戴飞, 赵博文, 潘美, 彭晓慧, 陈冉, 田园诗, 狄敏. 胎儿心脏超声定量多参数对主动脉缩窄胎儿心脏结构及功能的诊断价值[J/OL]. 中华医学超声杂志(电子版), 2024, 21(10): 950-958.
[4] 王益佳, 周青, 曹省, 袁芳洁, 周妍, 张梅. 中国经胸超声心动图检查存图及报告质控现状分析[J/OL]. 中华医学超声杂志(电子版), 2024, 21(07): 657-663.
[5] 莫莹, 李文秀, 李刚, 王霄芳, 王强, 丁文虹. 超声心动图在三尖瓣下移畸形中的临床应用价值[J/OL]. 中华医学超声杂志(电子版), 2024, 21(07): 702-708.
[6] 王秋莲, 张莹, 李春敏, 徐树明, 张玉奇. 胎儿主动脉弓部梗阻伴发复杂心内畸形的产前超声诊断及漏误诊分析[J/OL]. 中华医学超声杂志(电子版), 2024, 21(07): 718-725.
[7] 李洋, 蔡金玉, 党晓智, 常婉英, 巨艳, 高毅, 宋宏萍. 基于深度学习的乳腺超声应变弹性图像生成模型的应用研究[J/OL]. 中华医学超声杂志(电子版), 2024, 21(06): 563-570.
[8] 夏靖涵, 林凤娇, 王胰, 丁戈琦, 张清凤, 张红梅, 谢盛华, 李明星, 尹立雪, 李文华. 二尖瓣空间变化联合左心房应变对肥厚型心肌病合并左心室流出道梗阻的预测价值[J/OL]. 中华医学超声杂志(电子版), 2024, 21(06): 585-592.
[9] 王岚, 徐斌胜, 谢乐. 肥厚型心肌病的经胸超声心动图诊断与心电图表现特征[J/OL]. 中华医学超声杂志(电子版), 2024, 21(06): 593-596.
[10] 曹雨欣, 毛卓君, 梁嘉赫, 伊江浦, 张泽凯, 马文帅, 陈云涛, 李晓倩, 张宇新, 曹铁生, 袁丽君. 3D打印心脏模型在模拟左心耳封堵术临床教学中的应用价值[J/OL]. 中华医学超声杂志(电子版), 2024, 21(06): 602-607.
[11] 叶莉, 杜宇. 深度学习在牙髓根尖周病临床诊疗中的应用[J/OL]. 中华口腔医学研究杂志(电子版), 2024, 18(06): 351-356.
[12] 熊鹰, 林敬莱, 白奇, 郭剑明, 王烁. 肾癌自动化病理诊断:AI离临床还有多远?[J/OL]. 中华腔镜泌尿外科杂志(电子版), 2024, 18(06): 535-540.
[13] 李伟, 宋子健, 赖衍成, 周睿, 吴涵, 邓龙昕, 陈锐. 人工智能应用于前列腺癌患者预后预测的研究现状及展望[J/OL]. 中华腔镜泌尿外科杂志(电子版), 2024, 18(06): 541-546.
[14] 黄俊龙, 李文双, 李晓阳, 刘柏隆, 陈逸龙, 丘惠平, 周祥福. 基于盆底彩超的人工智能模型在女性压力性尿失禁分度诊断中的应用[J/OL]. 中华腔镜泌尿外科杂志(电子版), 2024, 18(06): 597-605.
[15] 孙铭远, 褚恒, 徐海滨, 张哲. 人工智能应用于多发性肺结节诊断的研究进展[J/OL]. 中华临床医师杂志(电子版), 2024, 18(08): 785-790.
阅读次数
全文


摘要


AI


AI小编
你好!我是《中华医学电子期刊资源库》AI小编,有什么可以帮您的吗?