Predicting Delays in Acute Ischemic Stroke Care with Machine Learning

Introduction

Machine learning with neural networks have demonstrated remarkable success in finding patterns in structured, as well as unstructured data. It has been applied to various aspects of health care including early detection and diagnosis, treatment, as well as outcome prediction and prognosis evaluation [1]. Process optimization is another application of machine learning, as it can find intricate trends from historical data. With 15-year historical data from the stroke thrombolysis program at a tertiary care center, an attempt was made to design a neural network with the aim of predicting team performance, in terms of door to needle time (defined as the time interval from arrival at the hospital to administration of the thrombolytic drug). Such information could be of help to warn of potential delay when a particular patient arrives, and the team armed with this information will be better enabled to pre-empt the delay [2].

Methods

Historical records of acute stroke thrombolysis were available from 2003 – 2018. The data was first split into training, validation sets. Data from 2003 – 201, consisting of 221 cases, was used as the training data, and the last 2 years’ data of 61 cases was used for validation. Identification of feature and target measures. (The inputs parameters fed to a machine learning model are called features, and the output it is trained to optimize on, the target). It has been reported from various centers that the door to needle time in acute ischemic stroke tends to have an inverse relation to duration of stroke at presentation [3], and this was replicated from data in our center as well. Similarly, regression and scatter plots were used to identify that: severity of stroke by NIH Stroke Score (NIHSS) scale, age of the patient in years, time of the day (rounded to the nearest hour) were also showing an association with the door to needle time, though individually the effects were weak (Figure 1). Thus, these four features were selected to train the network. The target parameter was the door to needle time, parametrized as above or below 45 minutes, as this was the cut off that showed the best predictive performance. Cases with delay <= 45 minutes were labelled as 0 and others as [1] (Table 1).

Table 1: Features and target.

Neural Network Architecture

A 3-layered fully connected architecture was used and implemented using the open source Tensor Flow framework [4]. The number of neurons in each layer were 4, 6 and 1, from input to output. The first two layers consisted of rectified linear activation units (Re LU), and the final unit used the sigmoid activation function, to give output in the range of 0 to 1. (Figure 1). The source code of the implementation is available at https://github.com/mlnoone/ nnstroketeamperf/

Training

The network was trained using batch gradient descent, with total iterations of 1500. Sigmoid cross entropy cost of the 45 min delay prediction (ranging 0 to 1) vs actual status (0 vs 1) was used as the minimization target. The learning rate was optimized based on performance on the training set and a value of 0.15 achieved the best result as shown in (Figures 2 & 3).

Figure 1: Scatter plots showing the (weak) correlations with door to needle time in minutes (on the vertical axis in all) of duration (A), Age(B), Severity (C) and Hour of day (D)

Figure 2: The neural network architecture.

Figure 3: Plot of cost vs iterations as the network was being trained at a learning rate of 0.15

Hyper Parameter Tuning

Based on performance in the training and the validation sets, the hyper parameters of the network were optimized. Performance on the validation set was assessed using the following parameters:
a) Accuracy, defined as percentage of delay > 45 min correctly predicted.
b) Sensitivity and positive predictive value (also called recall and precision in machine learning parlance) which were combined into the F1 score, defined as: 2 x precision x recall / (precision + recall).
These were then used to select the optimum hyper parameters for the network which, in addition to the learning rate of 0.15, included:
a) The number of neurons in each layer as mentioned above was set at 4, 6 and 1 which resulted in optimum performance on the training set (accuracy 60%, precision 68%, recall 71%, F1 0.68).
b) L2 Regularization was used to minimize overfitting which was observed initially, a regularization parameter (lambda) of 0.02 was chosen based on observing the result over a range of parameters to minimize error on the validation set.

Results

The network after optimization showed precision (positive predictive value) 63%, recall (sensitivity) 67% and F1 0.64 on the validation set, with an overall accuracy of 60% (Table 2).

Table 2: Performance measures on the validation set.

Discussion

Neural networks are capable of extracting intricate patterns from complex data, which are otherwise nearly impossible to delineate. They have been applied with remarkable success in fields like advertising and manufacturing to optimize performance based on structured data. Though the success of neural network with unstructured data like computer vision, speech and language are more widely appreciated, they are also remarkably effective in analyzing structured data. As seen here, with four weakly associated parameters it was possible to develop a reasonable prediction model for stroke team performance with 60% accuracy and 63% positive predictive value. Application of neural network in this context of optimizing acute care based on a predictive model has not so far been reported in literature. With increasing application of care pathways, neural networks could be a natural choice to build predictive models to identify potential for errors and delays.

For more Articles on: https://biomedres01.blogspot.com/