To assess and compare the diagnostic performance of ultrasound-based radiomics models constructed by different machine learning (ML) algorithms for identifying breast cancer.

Between January 2017 and April 2019, 828 consecutive patients with pathologically confirmed breast lesions at the First Affiliated Hospital of Wannan Medical College were included in this study. The patients were divided into a training set (n=526) and a validation set (n=302) according to the cutoff date of August 31, 2018. Radiomics features were extracted from the gray-scale ultrasound images. After features selection, five ML models based on five ML algorithms including k-nearest neighbor (kNN), logistics regression (LR), naive Bayes (NB), random forest (RF), and support vector machine (SVM) were constructed using the training set. Internal validation was performed using repeated k-fold cross-validation. Diagnostic metrics including sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for comparison. Furthermore, model discrimination and calibration were assessed in the external validation via ROC and calibration curve analysis.

Nineteen from 109 radiomics features were selected as effective features, and five ML prediction models were established. All the diagnostic metrics were statistically different across the models in the internal validation (P＜0.001). The median specificity, PPV, and NPV of the LR model were 0.769, 0.816, and 0.778, respectively; the median sensitivity was 0.824, which was higher than that of the kNN, RF, and SVM models. The median specificity, PPV, and NPV of SVM model were 0.706, 0.774, and 0.759, respectively, which were lower than those of the LR model, but higher than those of the other three models. The LR, SVM, RF, kNN, and NB models demonstrated good discrimination in external validation, with area under the curve (AUC) values of 0.890, 0.832, 0.821, 0.746, and 0.703, respectively. Of these, a significant difference was observed in AUC values between the LR and SVM models (P=0.012). While all the models did not perform consistently, the calibration curves for LR and SVM models indicated that the actual probability and the predicted probability agreed well.

The five ultrasound-based radiomics models developed based on different ML algorithms all demonstrate high diagnostic performance, but the LR model stands out for its superior performance. By applying an appropriate ML algorithm, the diagnostic performance of the final model could be further enhanced.