切换至 "中华医学电子期刊资源库"
高级检索
中华医学超声杂志（电子版）

中华医学超声杂志（电子版） ›› 2024, Vol. 21 ›› Issue (02) : 128 -136. doi: 10.3877/cma.j.issn.1672-6448.2024.02.004

心血管超声影像学

基于深度学习的超声心动图动态图像切面识别研究
成汉林1, 史中青2, 戚占如2, 王小贤2, 曾子炀3, 单淳劼1, 钱隼南4, 罗守华1, 姚静2,†()   
  1. 1. 210096　南京，东南大学生物科学与医学工程学院
    2. 210008　南京，南京大学医学院附属鼓楼医院超声医学科;210008　南京，南京大学医学院附属鼓楼医院医学影像中心;211400　扬州，南京鼓楼医院集团仪征医院
    3. 215123　苏州，东南大学苏州联合研究院
    4. 210009　南京，江苏省省级机关医院信息处
  • 收稿日期:2023-06-18 出版日期:2024-02-01
  • 通信作者: 姚静
  • 基金资助:
    国家自然科学基金(61871126); 江苏省重点研发计划(BE2022828); 江苏省前沿引领技术基础研究专项(BK20222002); 江苏省卫生健康委2022年度医学科研项目(281); 南京鼓楼医院临床研究专项(2022-YXZX-YX-01)

Deep learning-based two-dimensional echocardiographic dynamic image view recognition

Hanlin Cheng1, Zhongqing Shi2, Zhanru Qi2, Xiaoxian Wang2, Ziyang Zeng3, Chunjie Shan1, Sunnan Qian4, Shouhua Luo1, Jing Yao2,†()   

  1. 1. School of Biological Sciences and Medical Engineering, Southeast University, Nanjing 210096, China
    2. Department of Ultrasound Medicine, Affiliated Hospital of Medical School, Nanjing University, Nanjing 210008, China;Medical Imaging Centre, Affiliated Hospital of Medical School, Nanjing University, Nanjing 210008, China;Yizheng Hospital of Nanjing Drum Tower Hospital Group, Yangzhou 211400, China
    3. Suzhou Joint Research Institute, Southeastern University, Suzhou 215123, China
    4. Department of Information Office, Jiangsu Province Official Hospital, Nanjing 210009, China
  • Received:2023-06-18 Published:2024-02-01
  • Corresponding author: Jing Yao
全文 ( ) 下载PDF ( ) 导出引用
引用本文：

成汉林, 史中青, 戚占如, 王小贤, 曾子炀, 单淳劼, 钱隼南, 罗守华, 姚静. 基于深度学习的超声心动图动态图像切面识别研究[J]. 中华医学超声杂志（电子版）, 2024, 21(02): 128-136.

Hanlin Cheng, Zhongqing Shi, Zhanru Qi, Xiaoxian Wang, Ziyang Zeng, Chunjie Shan, Sunnan Qian, Shouhua Luo, Jing Yao. Deep learning-based two-dimensional echocardiographic dynamic image view recognition[J]. Chinese Journal of Medical Ultrasound (Electronic Edition), 2024, 21(02): 128-136.

目的

提出一种基于深度学习的切面识别模型SlowFast-Echo，进行二维经胸超声心动图动态图像的切面类型自动识别。

方法

选取2022年8月至12月在南京大学医学院附属鼓楼医院超声医学科完成二维经胸超声心动图检查的722例受检者（含心尖二腔、心尖三腔与心尖四腔等9类临床检查常用切面，共2243个动态图像），各类切面图像按照5∶2∶3的比例划分为训练集、验证集和测试集。进行SlowFast-Echo模型的训练和验证后，以准确率、精度、召回率、F1分数对模型的切面识别性能进行定量评价，以类激活映射图对模型的可解释性进行定性评价，以模型实地部署到超声医学科后的表现进行实用性评价。

结果

SlowFast-Echo模型对测试集动态图像切面类型预测的整体准确率、精度、召回率与F1分数分别为0.9866、0.9847、0.9872与0.9859；显著性热力图表明模型关注区域与超声科医师基本一致，如模型准确地定位到了肋骨旁短轴大血管水平切面（PSAXGV）显著的主动脉及主动脉瓣、胸骨旁短轴二尖瓣水平切面（PSAXMV）的二尖瓣与胸骨旁短轴乳头肌水平切面（PSAXPM）的乳头肌。实地部署后模型切面识别的整体准确率、精度、召回率与F1分数分别为0.9903、0.9865、0.9868与0.9865；在RTX 3060 GPU上单个动态图像的平均推理时间平均值为（303.2±119.3）ms，基本满足采图后即时处理的临床需求。

结论

本研究提出的SlowFast-Echo模型有着良好的二维经胸超声心动图动态图像切面识别性能与推理实时性，实用性较强，具有较好的应用前景。

关键词: 超声心动图, 切面识别, 深度学习, 人工智能
Objective

To propose a deep learning-based view recognition model, SlowFast-Echo, for the automatic view recognition of two-dimensional (2D) transthoracic echocardiographic dynamic images.

Methods

From August to December 2022, 722 patients who underwent 2D transthoracic echocardiography at the Department of Ultrasound Medicine, Affiliated Hospital of Medical School, Nanjing University (9 types of clinically commonly used views [including apical two-chamber, apical three-chamber, and apical four-chamber views], with a total of 2243 dynamic images) were selected, and the images of each view were divided into training set, validation set, and test set in a ratio of 5:2:3. After training and validation of the SlowFast-Echo model, the performance of the model was evaluated quantitatively in terms of accuracy, precision, recall, and F1 score, qualitatively in terms of the interpretability of the model with regard to class activation mapping, and practically in terms of the performance of the model after field deployment to the ultrasound medicine department.

Results

The overall accuracy, precision, recall, and F1 score of the SlowFast-Echo model for dynamic image view recognition in the test set were 0.9866, 0.9847, 0.9872, and 0.9859, respectively, and the significance heatmap indicated that the model's regions of interest were generally consistent with those drawn by the physicians; e.g., the model accurately pinpointed the significant aorta and aortic valve in parasternal short axis view of great vessel (PSAXGV) view, mitral valve in parasternal short axis view of left ventricle at mitral value level (PSAXMV) view, and papillary muscles in parasternal short axis view of left ventricle at papillary muscle level (PSAXPM) view. The overall accuracy, precision, recall, and F1 score of the model for view recognition after deployment were 0.9903, 0.9865, 0.9868, and 0.9865, respectively, and the average inference time on RTX 3060 GPU for a single dynamic image was (303.2±119.3) ms, which basically meets the clinical demand for immediate processing after image acquisition.

Conclusion

The SlowFast-Echo model proposed in this study has good performance in view recognition of 2D transthoracic echocardiographic dynamic images and inference in real time, which is practically useful.

Key words: Echocardiography, View recognition, Deep learning, Artificial intelligence
图1 本研究自动识别的二维超声心动图9类切面示意图 注：A2C为心尖二腔切面；A3C为心尖三腔切面；A4C为心尖四腔切面；A5C为心尖五腔切面；PLAX为胸骨旁长轴左心室切面；PSAXGV为肋骨旁短轴大血管水平切面；PSAXMV为胸骨旁短轴二尖瓣水平切面；PSAXPM为胸骨旁短轴乳头肌水平切面；PSAXA为胸骨旁短轴心尖水平切面
表1 9类切面数据情况
视频数目与帧数信息 A2C A3C A4C A5C PLAX PSAXGV PSAXMV PSAXPM PSAXA
视频总数 183 269 333 134 281 313 292 232 206
训练集视频数 91 134 166 67 140 156 145 115 102
验证集视频数 37 54 67 27 57 63 59 47 42
测试集视频数 55 81 100 40 84 94 88 70 62
总帧数 18963 27653 35889 15063 32366 38683 33107 26662 22477
帧数范围 41~342 33~285 26~205 41~172 16~401 47~367 41~243 29~184 41~241
图2 SlowFast-Echo模型示意图
表2 各类切面视频测试集识别性能表现
切面类型 测试视频数 准确率 精度 召回率 F1分数
A2C 55 1.0000 1.0000 1.0000 1.0000
A3C 81 1.0000 0.9878 1.0000 0.9939
A4C 100 0.9900 1.0000 0.9900 0.9950
A5C 40 1.0000 0.9756 1.0000 0.9877
PLAX 84 0.9762 1.0000 0.9881 0.9940
PSAXGV 94 1.0000 1.0000 1.0000 1.0000
PSAXMV 88 0.9659 0.9884 0.9659 0.9770
PSAXPM 70 0.9571 0.9571 0.9571 0.9571
PSAXA 62 0.9839 0.9531 0.9839 0.9683
合计 674 0.9866 0.9847 0.9872 0.9859
图3 各类切面视频识别结果混淆矩阵 注：A2C为心尖二腔切面；A3C为心尖三腔切面；A4C为心尖四腔切面；A5C为心尖五腔切面；PLAX为胸骨旁长轴左心室切面；PSAXGV为肋骨旁短轴大血管水平切面；PSAXMV为胸骨旁短轴二尖瓣水平切面；PSAXPM为胸骨旁短轴乳头肌水平切面；PSAXA为胸骨旁短轴心尖水平切面
图4 各类切面动态图像与类别显著性热力图合成图 注：A2C为心尖二腔切面；A3C为心尖三腔切面；A4C为心尖四腔切面；A5C为心尖五腔切面；PLAX为胸骨旁长轴左心室切面；PSAXGV为肋骨旁短轴大血管水平切面；PSAXMV为胸骨旁短轴二尖瓣水平切面；PSAXPM为胸骨旁短轴乳头肌水平切面；PSAXA为胸骨旁短轴心尖水平切面
表3 实地部署后各切面识别性能表现与推理耗时情况
切面类型 动态图像数目 准确率 精度 召回率 F1分数 推理耗时（ms，）
A2C 90 1.0000 1.0000 1.0000 1.0000 333.2±104.7
A3C 48 0.9792 1.0000 0.9792 0.9895 339.4±107.9
A4C 147 0.9932 0.9932 0.9932 0.9932 330.5±106.0
A5C 25 1.0000 0.9615 1.0000 0.9804 338.5±100.7
PLAX 93 1.0000 1.0000 1.0000 1.0000 304.6±125.8
PSAXGV 135 1.0000 1.0000 1.0000 1.0000 268.0±109.0
PSAXMV 60 0.9833 0.9833 0.9833 0.9833 308.2±101.3
PSAXPM 49 0.9388 0.9787 0.9388 0.9583 314.2±129.5
PSAXA 76 0.9868 0.9615 0.9868 0.9740 239.8±138.6
合计 723 0.9903 0.9865 0.9868 0.9865 303.2±119.3
1
Cleve J, McCulloch ML. Conducting a cardiac ultrasound examination [J]. Echocardiography, 2018: 33-42.
2
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks [J]. Commun ACM, 2017, 60(6): 84-90.
3
陶攀, 付忠良, 朱锴, 等.基于深度学习的超声心动图切面识别方法 [J]. 计算机应用, 2017, 37(5): 1434-1438.
4
Madani A, Arnaout R, Mofrad M, et al. Fast and accurate view classification of echocardiograms using deep learning [J]. NPJ Digital Med, 2018, 1: 6.
5
Zhang J, Gajjala S, Agrawal P, et al. Fully automated echocardiogram interpretation in clinical practice: feasibility and diagnostic accuracy [J]. Circulation, 2018, 138(16): 1623-1635.
6
Østvik A, Smistad E, Aase SA, et al. Real-time standard view classification in transthoracic echocardiography using convolutional neural networks [J]. Ultrasound Med Biol, 2019, 45(2): 374-384.
7
Kusunose K, Haga A, Inoue M, et al. Clinically feasible and accurate view classification of echocardiographic images using deep learning [J]. Biomolecules, 2020, 10(5): 665.
8
Santosh Kumar BP, Haq MA, Sreenivasulu P, et al. Fine-tuned convolutional neural network for different cardiac view classification [J]. J Supercomput, 2022, 78(16): 18318-18335.
9
Gao X, Li W, Loomes M, et al. A fused deep learning architecture for viewpoint classification of echocardiography [J]. Inform Fusion, 2017, 36: 103-113.
10
Howard JP, Tan J, Shun-Shin MJ, et al. Improving ultrasound video classification: an evaluation of novel deep learning methods in echocardiography [J]. J Med Artif Intell, 2020, 3: 4.
11
Feichtenhofer C, Fan H, Malik J, et al. Slowfast networks for video recognition [C]. Proceedings of the IEEE/CVF international conference on computer vision, Seoul, Korea (South), 2019: 6202-6211. Piscataway, NJ: IEEE Computer Society, 2019.
12
Cubuk ED, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space [C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA, 2020: 702-703. Piscataway, NJ: IEEE Computer Society, 2020.
13
Zhong Z, Zheng L, Kang G, et al. Random erasing data augmentation [C]. Proceedings of the AAAI conference on artificial intelligence, New York, USA, 2020, 34(7): 13001-13008. Menlo Park, CA: AAAI, 2020.
14
He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners [C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022: 16000-16009. Piscataway, NJ: IEEE Computer Society, 2022.
15
Wang X, Girshick R, Gupta A, et al. Non-local neural networks [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA, 2018: 7794-7803. Piscataway, NJ: IEEE Computer Society, 2018.
16
He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 2016: 770-778. Piscataway, NJ: IEEE Computer Society, 2016.
17
Vanholder H. Efficient inference with tensorrt [C]. GPU Technology Conference, Sunny San Jose, California, USA, 2016, 1: 2. Santa Clara, CA: Nvidia, 2016.
18
Zhou B, Khosla A, Lapedriza A, et al. Learning deep features for discriminative localization [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 2016: 2921-2929. Piscataway, NJ: IEEE Computer Society, 2016.
19
姜玉新, 李建初, 王红燕, 等.信息化技术助力超声医学质量控制新发展 [J/OL].中华医学超声杂志(电子版),2021,18(7): 625-628.
20
Huang MS, Wang CS, Chiang JH, et al. Automated recognition of regional wall motion abnormalities through deep neural network interpretation of transthoracic echocardiography [J]. Circulation, 2020, 142(16): 1510-1520.
21
Huang KC, Huang CS, Su MY, et al. Artificial intelligence aids cardiac image quality assessment for improving precision in strain measurements [J]. JACC Cardiovasc Imaging, 2021, 14(2): 335-345.
22
Lane ES, Azarmehr N, Jevsikov J, et al. Multibeat echocardiographic phase detection using deep neural networks [J]. Comput Biol Med, 2021, 133: 104373.
23
吴洋, 张红梅, 尹立雪, 等.超声心动图心尖四腔心切面图像质量智能评分研究[J/OL].中华医学超声杂志(电子版), 2023, 20(1): 97-102.
24
Hasani R, Lechner M, Amini A, et al. Liquid time-constant networks [C]. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, Canada, 2021, 35(9): 7657-7666. Menlo Park, CA: AAAI, 2021.
[1] 张胜男, 苗雅敬, 周虹, 韩高洁, 王静, 仝巧立, 张旭倩, 尹洪宁. 左心耳三维经食管超声测量与Watchman左心耳封堵器大小的相关性研究[J]. 中华医学超声杂志（电子版）, 2024, 21(02): 107-113.
[2] 刘韩, 王胰, 舒庆兰, 彭博, 尹立雪, 谢盛华. 基于深度学习的超声心动图三尖瓣反流严重程度智能评估方法研究[J]. 中华医学超声杂志（电子版）, 2024, 21(02): 121-127.
[3] 肖莉莉, 吴道珠, 陈晓乐, 李秀云, 寇红菊. 胎儿心脏参数对胎儿宫内生长受限的预测价值[J]. 中华医学超声杂志（电子版）, 2024, 21(01): 24-31.
[4] 谭芳, 杨娇娇, 范思涵, 叶彩玲, 纪学芹. 产前超声心动图在先天性血管环诊断中的价值[J]. 中华医学超声杂志（电子版）, 2024, 21(01): 37-41.
[5] 梁越, 董晓秋, 李奇默, 李岩, 姚金来, 朴雪梅. 孕11~13+6周子宫动脉与左心室参数对子痫前期的预测模型构建与验证[J]. 中华医学超声杂志（电子版）, 2024, 21(01): 42-48.
[6] 陈红, 阮骊韬, 师桃, 郭锋伟, 郝军军, 闫炀, 尚佳楠, 宋艳. 右心室心肌应变与左心室辅助装置植入后早期右心衰竭的相关性研究[J]. 中华危重症医学杂志(电子版), 2024, 17(01): 26-31.
[7] 王瑞, 张嘉炜, 张克诚, 周增丁. 基于深度学习的皮肤烧烫伤创面图像分割与分类及检测的研究进展[J]. 中华损伤与修复杂志(电子版), 2024, 19(02): 172-175.
[8] 钟佩芝, 杜宇. 龋病诊断方法的研究进展[J]. 中华口腔医学研究杂志(电子版), 2024, 18(02): 73-79.
[9] 戴雨霖, 张新春. 人工智能在口腔修复诊疗中的应用与进展[J]. 中华口腔医学研究杂志(电子版), 2024, 18(01): 65-69.
[10] 杨龙雨禾, 王跃强, 招云亮, 金溪, 卫娜, 杨智明, 张贵福. 人工智能辅助临床决策在泌尿系肿瘤的应用进展[J]. 中华腔镜泌尿外科杂志(电子版), 2024, 18(02): 178-182.
[11] 朱先理, 王守森. 大语言模型在创伤性脑损伤病历书写的应用前景[J]. 中华神经创伤外科电子杂志, 2024, 10(01): 55-57.
[12] 卢梦诗, 刘威, 马加威, 嵇丹丹, 贾璇, 詹心萍, 罗亮. 人工智能在急性呼吸窘迫综合征领域的应用进展[J]. 中华重症医学电子杂志, 2024, 10(01): 66-71.
[13] 陈健, 张子豪, 卢勇达, 夏开建, 王甘红, 刘罗杰, 徐晓丹. 基于深度学习构建结直肠息肉诊断自动分类模型[J]. 中华诊断学电子杂志, 2024, 12(01): 9-17.
[14] 王靖玺, 赵丽, 吕滨. 人工智能在肺栓塞CT检查中的临床研究进展[J]. 中华心脏与心律电子杂志, 2024, 12(01): 26-31.
[15] 胡陈玥, 葛贤秀, 邓雪婷, 姚家楠, 缪林. 图像增强放大内镜诊断胃癌前病变及早期胃癌[J]. 中华胃肠内镜电子杂志, 2024, 11(01): 47-51.
阅读次数
全文


摘要

版权所有 中华医学会 中华医学电子音像出版社有限责任公司  京ICP备14006079号-1  网络出版服务许可证：(署)网出证(京)字第075号