Generating rapid insights with PwC's Data Science Machine

To identify decisive features from a variety (53) of easily obtained blood markers, Yan et al.¹ fitted a multi-tree XGBoost machine learning model to the data of 70% of the 375 patients (the data of 110 patients was kept aside for testing purposes). Using XGBoost's built-in feature importance method, they were able to identify the top ten most important features. To pressure-test these findings, they tried fitting a single-tree XGBoost model to the data, starting with only the most important feature (as discovered by the previous method) and gradually adding the second-most important feature, then the third-most important and so on, until the performance of the model no longer improved on a validation set. This approach identified the following three features as the decisive features in the dataset:

• Lactate dehydrogenase
• Hypersensitive c-reactive protein
• Lymphocyte

The single-tree XGBoost model still performed very well in the validation and test sets and, as it only contained one tree, it produced a decision tree that was very minimal and easy to evaluate.

As Yan et al.¹ published the raw data and the scripts for pre-processing and evaluating the data on GitHub, we were able to analyse exactly the same data as them, but instead using the DSM as our accelerator.

The DSM is able to generate and fit hundreds of models within minutes, while performing feature optimisation at the same time. So, we let the DSM engine train a multitude (~10,000) of XGBoost models, randomly picking parameters for the XGBoost model and also randomly picking features to use from the dataset. In addition, we used a 70%/30% split between training and validation sets. Taking a fairly different approach and letting the DSM basically work unsupervised, this meant we were able to give further evidence on the importance of 'Lactate dehydrogenase' and 'Hypersensitive c-reactive protein' as important features for predicting COVID-19 outcomes.

In the table 1 below, we looked at all models with a performance higher than AUC = 85% and counted how often these models picked a certain feature. The features 'Hypersensitive c-reactive protein' and 'Lactate dehydrogenase' seem to correlate with good performance. Lymphocytes seem to be less important than the other two and only the count, not the percentage as suggested by Yan et al., shows up in the top ten. Again, this is in line with the paper, where the proposed, simple decision tree suggests that lymphocytes only act as a tiebreaker if the prognosis isn't clear from the other two features.

It’s also worth noting that in other papers discussing blood markers to determine the severity of COVID-19 infections, some of the other blood markers that we identified in the top ten were also found. Specifically, the features in this paper by Wu et al.² and this paper by Lu et al.³ overlapped significantly with the features in our top ten.

In this section, our aim is to verify whether or not the features (i.e. blood markers) identified as critical in the paper by Yan et al.¹ are of a similar critical nature within the models built by the DSM. To do this, we studied the performance of all DSM-generated models as a function of critical features used for different numbers of total features (figure 1). We observed a clear trend indicating that adding the first critical feature increases model performance, especially if there’s a low number of total features. In general model performance rises monotonously as a function of numbers of critical features used.

In general, models with good performance seem to correlate with the two features 'Lactate dehydrogenase' and 'Hypersensitive c-reactive protein'. The 'Lymphocyte' feature appears to be less important for having a model with good performance. This seems to be in line with the interpretable tree-based model that Yan et al. proposed, where 'Lymphocyte' was used more as a tiebreaker in cases where 'Lactate dehydrogenase' and 'Hypersensitive c-reactive protein' weren't decisive by themselves.