A Dual-Selective Channel Attention Network for Osteoporosis Prediction in Computed Tomography Images of Lumbar Spine

Osteoporosis is a common and systemic bone disease, and its early symptoms are not obvious. Most patients with osteoporosis undergo relevant examinations when complications arise, which are usually in the late stage and bring a lot of inconveniences and even death [1]. Therefore, early screening is crucial for the timely prevention and treatment of osteoporosis fractures [2]. Meanwhile, all kinds of orthopedic surgery need to refer to a bone status evaluation to formulate a better surgical plan. As the gold standard for bone mass measurement, DXA testing is expensive. Even in many developed countries, the opportunity to use DXA is still insufficient. CT image examination has a large number and clear images, which is of great significance in early screening for the prevention and treatment of osteoporotic fractures [3].

In order to reduce this preventable injury and subsequent complications, more and more researchers are focusing on methods that combine computer-aided detection and machine learning with radionics to assist clinicians with osteoporosis prediction. For example, Aouache et al. [4] designed a fuzzy decision tree (FDT) model to identify osteoporosis by identifying patients' cervical spine images. Devikanniga and Raj [5] proposed an artificial neural network optimized for monarch butterflies, which identified osteoporosis and normal subjects by recognizing hip X-ray images of patients and combining them with demographic attributes. However, their algorithms were based on small data sets and required complex data processing before feature extraction, causing inaccuracy in medical image processing.

In recent years, convolutional neural networks (CNNs) have achieved high performance in visual recognition tasks. Instead, this technique can automatically locate the region of interest (ROI) and extract features, avoiding empirical errors in manual feature extraction [6]. Therefore, the application of deep learning in medical imaging diagnosis has received extensive attention for osteoporosis classification [7], [8], [9], [10] and bone mineral density (BMD) prediction [11], [12], [13]. For instance, Pan et al. [14] developed a deep learning-based system for bone mineral density (BMD) classification in chest CT images. To extract the density information of trabecular bone more accurately, they firstly segmented and labeled all vertebral bodies, then their system can automatically extract the rectangular area in the center of the trabecular bone as the ROI, thereby realizing automatic measure BMD and make two classifications predictions of osteoporosis and osteopenia. Yasaka et al. [15] used an improved CNN model to extract the features of the manually labeled central circular region of trabecular bone from unenhanced abdominal CT images, achieving a correlation of 0.840 between the CNN network and the corresponding DXA results. Lee et al. [16] proposed a method based on the combination of the VGG network and random forest to classify of normal and abnormal BMD in spinal X-ray images. To extract features more accurately, they selected the central region of the fourth lumbar spine as ROI and achieved a 71% accuracy for the two-category classification. Gonzalez et al. [17] selected chest CT images as input data, proving that deep neural network has an excellent performance in osteoporosis identification.

Despite the above studies yielded exciting results in identifying osteoporosis, they still left significant limitations on the diagnostic ability of clinical osteoporosis recognition. Firstly, those designed network models were limited to two classifications: Osteoporosis and osteopenia, which could not meet the clinical requirements of various conditions of bone condition. Moreover, these studies mostly selected the trabecular structure as ROI, while ignoring the cortical bone thickness which plays an important role in the real discrimination scenarios of bone by clinical experts. Finally, the methods proposed in these studies were not effective in bone image texture recognition due to the insufficient ability of feature extraction, which affected the prediction accuracy.

In view of the limitations of CNN-based methods for accurate osteoporosis prediction, we proposed an improved network based on Faster R-CNN, in which a new channel attention module was integrated to enhance the texture, shape and other features of vertebral trabecular bone in CT images. The specific analysis is as follows: 1) We have achieved a three-category classification of osteoporosis, osteopenia and normal bone mineral density, which brings greater applicability to professional doctors in clinical diagnosis; 2) According to the actual clinical needs, the whole vertebral body including cortical bone and cancellous bone was tested for the region of interest; and 3) We adopt an improved channel attention mechanism, named DS module, to improve the interrelation among feature map channels and deepen the learning of extracted feature information.

This paper aims to improve the osteoporosis detection performance for the lumbar spine CT images based on an improved Faster R-CNN network.

The general framework of the proposed DSNet is based on the mainstream detection network of Faster R-CNN due to the high performance in many visual detection tasks [18]. In Faster R-CNN, the backbone network extracts feature from input images using ResNet [19], LeNet5 [20], AlexNet [21] or GoogLeNet [22]. The backbone network loads the officially trained model parameters to extract features, which can reduce the amount of model training data and speed up the training speed. Afterward, the target proposal boxes generated by the region proposal network (RPN) are projected to the feature map through the region of interest pooling layer (ROI pooling). Finally, the features are calculated by classification and bounding regression to achieve end-end detection. The framework of Faster R-CNN is shown in Figure 1.

In clinical practice, radiologists commonly distinguish normal, osteoporosis, and osteopenia BMD from the thickness of cortical bone and the sparsity of cancellous bone. For the automatic osteoporosis detection task, the network needs to pay more attention to the texture details of the whole vertebral body. With respect to these aspects, we pronounce a novel attention module and subsequently integrate it into the backbone of ResNet50 to improve the interconnection among feature map channels and deepen the learning of the extracted texture feature information. As a result, the proposed DSNet extracts more texture features of the entire vertebral body than the traditional ResNet50.

Channel attention is one of the most widely used attention mechanisms in various fields such as natural language processing (NLP), computer vision (CV) and speech recognition. The representative ones are SENet [23], SKNet [24], ECANet [25], etc. As channel attention modules, they can be easily embedded in the deep learning network to achieve improved performance. As shown in subgraph (a) of Figure 2, the input feature map with the size of H*W*C is mapped to the feature map with the size of H1*W1*C1 through a transformation of a 3*3 convolution. Then the SE attention mechanism establishes the inter-dependence between feature channels through a global pooling and a full convolution. Its mechanism is that each channel’s feature map is assigned a weight, which represents the correlation between the channel and the key information. The larger the weight, the higher the correlation. In this way, it simulates the brain signal processing mechanism of human vision and filters useful channel information through the weight. The SK attention mechanism introduces a 5*5 spatial dimension on the basis of SE to fuse feature channels, as shown in subgraph (b) of Figure 2. The fusion feature map from two different scales captures attention through a global pooling and a full convolution. Then it assigns weights by introducing a softmax calculation to select adaptively different spatial scales of information without sacrificing the amount of computation. Although SKNet captures target features at different scales, it fuses feature maps before weight assignment and allocates the same attention weight to the feature map after the global pooling and the full convolution, thereby losing the interactive feature information from different scales and not adequately condensing the model's attention ability.

This paper proposes a new attention mechanism module named dual-selective, as shown in subgraph (c) of Figure 2. The feature map extracted by Backbone is taken as the input. Our attention mechanism module performs feature fusion from two different scales of kernel sizes 3 and 5. The global pooling layer and the full convolution layer are used to fuse the weights of each channel. Then the two fused branches are superimposed. In order to adaptively adjust the multi-scale feature information and capture more accurate information from the target objects with different scales, we adopt a dilated convolution with a 3×3 kernel and a dilation size of 2 to fuse the output features. Therefore, our DS module can fuse more resolution information in the convolutional feature map from different attention weights, and deepen the texture information of the feature map from different scales by the secondary weight calculation. It is conducive to paying more accurate attention to the feature information of the image texture.

In subgraph (c) of Figure 2, the computational steps can be mathematically expressed as follows. The input of the channel attention block is a feature map $x \in R^{W * H * C}$, in which W, H and C are the width, the height and the channel, respectively [26]. The feature map $x$ is conducted with two transformations with kernel sizes 3 and 5, whose results are represented as x_c. The x_c is embed the global information by a global pooling. The global pooling $F_P\left(x_c\right)$ is calculated as:

Further, the output feature map $x_o \in R^{1 * 1 * C}$ after the global pooling is the one-dimensional vector. The feature map x₀ achieves precise and adaptive selections by a full convolution. The full convolution $F_c\left(x_o\right)$ is calculated is:

where, $W_{i j} \in R^{i * j}$ is the weight matrix and $B \in R^{i * 1}$ is the one-dimensional bias [27].

Dual-selective channel attention network (DSNet) serves as the backbone of the improved Faster R-CNN, which is a fusion network of ResNet50 and DS module. As shown in Figure 3, the first stage has a relatively simple structure, consisting of a 7*7 convolutional kernel and a 3*3 max pooling layer. The second stage consists of a DS module and a residual layer. Stages 3 and 4 consist of one residual layer, respectively. The last stage consists of a residual layer and a subsequent DS module. Each residual layer is composed of 1*1, 3*1, and 1*1 convolution kernels in sequence. In this way, DSNet improves the network's ability to extract texture features and can use transfer learning to improve the stability and training speed of the model.

We used axial unenhanced lumbar spine CT images (L1 - L4) as input data and the DXA report of each lumbar spine as reference data. All data were manually annotated for the entire lumbar vertebrae and labeled with the bone condition category of osteoporosis, osteopenia and normal. The training set and test set were divided at a ratio of 4:1. The training set is used to train the network model. The performance of the network model is evaluated by the test set.

This study was approved by the ethical board of the affiliated hospital of Hebei university. We retrospectively collected personal data of lumbar spine CT scan (full bone window) and DXA examination between May 2016 and April 2020 from the Department of Orthopedics, Affiliated Hospital of Hebei University, with a total of 50,296 lumbar spine CT images and corresponding DXA detection results of 1,132 patients. Moreover, we conducted screening and disorientation processing for patient privacy. All patients underwent single-photon emission computed tomography (CT) scans of the lumbar spine (from L1 to L4). Their gender, height, and weight were recorded, and the original images of the cases were in DICOM format. For all data, we performed rigorous data cleaning by learning from medical professionals: (1) Exclude images of poor quality, such as artifacts produced by spinal implants (instruments) implanted during surgery; (2) Exclude non-lumbar regions or images showing lumbar L1-L4 insufficiency, leaving only the lumbar cones Complete images; (3) Image data that has undergone lumbar spine surgery, such as bone nails and bone cement filling; (4) Individuals with a history of fractures, history of spinal surgery, primary or metastatic tumors, bone hyperplasia, or vertebral bodies are excluded. Finally, we achieved 4805 images (including any of L1, L2, L3 or L4 for the training dataset; including all L1, L2, L3 and L4 for the test dataset) of 133 patients (men, n=29, women, n=104). The whole vertebral body (including cortical bone and cancellous bone) was manually annotated by experienced physicians.

According to the criteria for diagnosis of osteoporosis defined by WHO [28]. We divided the subjects into three groups, namely the normal group with T value ≥ -1.0 SD, the osteopenia group with -2.5 SD < T value < -1.0 SD, and the osteoporosis group with T value ≤ -2.5 SD. Finally, we divide all the data at a ratio of 4:1, one part is used for the training of network parameters, namely the training set, which includes 3,844 CT images, and the other part is used to evaluate the generalization performance of the network, namely the test set, which includes 961 CT images.

A new convolutional neural network should be firstly initialized the weights. Otherwise, the activation layer function will fail to output during the training of deep neural networks, which will result in an explosion of the loss gradient and a prolonged convergence of the network. Therefore, we use the weight parameters learned from the COCO dataset as the initial network parameters, which can avoid effectively the problem of overfitting during the training process and improve the stability and training speed of the model.

Deep learning model training was performed on a computer equipped with a core I7-7800x 3.5-GHz central processing unit, 256 GB memory, and a 2080 Ti graphics processing unit. We used the programming language of python 3.7 and the deep learning framework of PyTorch. The loss function of the model was the dice loss, SGD was used as the optimizer of the model, the learning rate was set to 0.01, the batch size was 9, and the training epoch was 100.

To evaluate the performance of the proposed model, the metrics of mean average precision (Map), recall, accuracy, and frame per second (Fps) are adopted for object detection. The vertebral bodies of the lumbar CT images (IOU ≥ 0.5) are detected and classified correctly as correct predictions, otherwise, it is considered incorrect predictions.

Mean average precision (Map) is the proportion of the predictions for all categories that successfully predicted the true target, which is defined as:

Recall rate (Recall) represents the ability of the model to find all relevant targets, which is the number of true targets in the results predicted by the model. It is calculated as follows:

Accuracy (Acc) represents the proportion of all predictions that the model predicts correctly.

Frame per second (Fps) is defined as the number of pictures processed by the model per second. The higher the frame per second, the faster the model processing speed, and vice versa.

where, TP is the number of predicted bounding boxes with correct classification and correct bounding box coordinates. FP is the number of predicted bounding box classification errors or bounding box coordinates that do not meet the standard. FN is the number of ground truths falsely detected. Number (N) represents the number of categories for classification. Fn is the number of picture frames processed by the network. S represents the time.

In this study, we use 1,605 CT images of osteoporosis, 1,600 CT images of osteopenia, and 1,600 CT images of normal bone density for analysis. To estimate the detection performance of the proposed attention mechanism and the novel DSNet, we compared the effects of ResNet50, VGG16, and MobileNet as the backbone under different attention mechanisms.

We adopted Faster R-CNN as the main framework and used the backbone with different attentional mechanisms to extract features. We performed an ablation experiment using the modules of SK, SE, and our DS based on the traditional ResNet50 without the integration of the channel attention algorithm. The metrics of Map, Recall, Acc with the IOU of 0.5, and Fps were used to evaluate and compare the detection performances among different methods. As shown in Table 1, ResNet50 integrated with the attention module achieved increased Map and Recall, indicating that the attention module can improve the network’s performance. The integration of DS attention into ResNet50 achieved the best performance and the detection speed is the closest to the ResNet50, reaching a Map of 83.4%, a Recall of 90.0%, an Acc of 75.9%, and a Fps of 0.1846, indicating that our DS module has better performance on network feature extraction. Meanwhile, our DS module improved the shortcomings of other attention mechanisms on Acc, with a 1.8% improvement over Resnet50. Therefore, our model was effective at fusing multiscale attention weights.

To evaluate the detection performance of the network more intuitively, we performed a visual analysis to compare the results of different modules. Figure 4 shows the detection results of different backbone networks with the vertebrae’s boundary and a single probability value. The first column is the ground truth of normal bone density, osteopenia, and osteoporosis. The remaining columns are the visual detection results of different backbone networks, followed by Resnet50, Resnet50+SE, Resnet50+SK, and Resnet50+DS. All backbone networks accurately identified the normal class. But because the texture feature of osteopenia is more difficult to distinguish than another two classes, the Resnet50 and Resnet50+SE mistakenly identified osteopenia as normal. Meanwhile, the Resnet50 mistakenly identified osteoporosis as osteopenia, indicating that the attention module can improve the network’s detection performance. This can seriously affect the clinician's diagnosis and delay the patient's treatment. For the three categories, our Resnet50+DS module can accurately identify with the highest confidence. It can be seen that the ResNet50+DS has a higher detection efficiency for improving the network’s attention ability.

To verify the applicability of the proposed DS attention module, we tested the performance of different backbone networks with and without the fusion DS attention module. To be fair, these three sets of comparative experiments were tested under the same training conditions and the same data set (4,805 images of 133 patients). As shown in Table 2, ResNet50 with the DS attention module has improvements of 0.54%, 0.19%, and 0.18% in Map, Recall, and Acc with a 0.0003f/s decrease in Fps. We further compared the performance with or without the DS attention module on VGG [29], and MobileNet [30] and found that the integration of the DS attention module would also reduce their speed. The VGG16 integrated with the DS attention module achieved increased Map, Recall, and Acc. The integration of DS attention into MobileNet achieved better performance on Map and Recall with a 0.3% reduction in Acc. Among ResNet50 with the integration DS module has the best improvement. In general, our proposed DS module is also applicable to other backbones and the fusion method of ResNet50 had the best effect, which can effectively improve the feature extraction ability of the network.

Table 3 illustrated the comparative results between our improved network method and the other osteoporosis prediction networks. Our method produced remarkable recall and detection categories on the biggest number of images. The traditional osteoporosis prediction methods had the results of the two classifications. Our method added a detection category of osteopenia to alert patients before they reach the severity of osteoporosis. As we all know, as the detection category increases, the detection effect of the network decreases. Therefore, the Acc of our proposed method was reduced than the other two-classification tasks. But our proposed method achieved the highest Recall of 89.2% with a tri-classification identification of osteoporosis, osteopenia, and normal. (We have no the results of AUC due to the tri-classification detection task.) Thus, the overall detection and classification effect of our method achieved better performance and provide value for the prediction of three categories of bone status in the future [31].

This study proposes a new deep-learning method for the automatic detection of lumbar vertebrae and classification of bone status using 4,805 lumbar CT images of 133 patients. We train the network model with a large amount of clinical data and compare the test results with DXA diagnosis results. Meanwhile, the network model has been improved and optimized. At present, the prediction of osteoporosis is based on the measurement of bone density. However, bone density can only reflect about 70% of the degree of osteoporosis. The geometric characteristics of bone microstructure and the heterogeneity of density structure also have a certain impact on the diagnosis of osteoporosis. Therefore, the thickness of cortical bone and the thinning of cancellous bone are the basis for doctors to judge whether there is osteoporosis. Different from previous studies, our model is highly close to the doctor's diagnostic level. The entire lumbar vertebral body including cortical and cancellous bones is annotated as regions of interest, as the dataset for model training.

In this paper, we propose a new backbone integrating ResNet50 and DS module based on Faster R-CNN. In this network, the lumbar vertebral body is identified and detected first, and the detection results are marked in the form of rectangular boxes with predicted confidence. Then the vertebral body structure is extracted to class the categories of osteoporosis, osteopenia, and normal. The proposed module improves the performance of feature extraction and pays more attention to the texture features of cortical and cancellous bones. Compared with other attention mechanisms, our module achieves an improvement on Map, Recall, and Acc reaching 83.4% and 90.0%, and 75.9% respectively. The feasibility, effectiveness, and compatibility of the model are also verified.

This paper can expand the application of artificial intelligence-assisted diagnosis systems and contribute to the clinical identification of osteoporosis. To a certain extent, it can solve the problem that fracture osteoporosis cannot be detected in time to improve the treatment rate of osteoporosis so that it has important theoretical value and clinical significance.