SAS MACRO for forward model selection in a logistic regression model with missing values

Missing data have long been a serious impediment in statistical analysis and modeling in research and business. The logistic regression model is a standard statistical method used in analysis when the response variables are dichotomous or binary^[1]. The present work aims to demonstrate the advantages or disadvantages of logistic regression modeling methods among SAS PROC LOGISTIC forward selection, SAS MI procedure, and the developed macro-based forward selection method, using the artificial binary response data with the presence of random missing values.

Missing data mechanisms

According to the generating mechanisms, missing data can be classified into three board types: missing completely at random (MCAR) if missing data are completely independent of any other factors; missing at random (MAR) describes the missingness based on other complete variables; not missing at random (NMAR) if the probability of missing data systematically depends on incomplete variables. According to missing data mechanisms, distribution assumptions are made in advance indicating the modeling methods. To specify, missing data does not represent the entire data. Thus, the modeling assumption could be MCAR, MAR, or NMAR depending on the variables of interest^[2−5]. MCAR is possible to test in practice empirically.

Deletion and single imputation

Traditionally, missing data are deleted or replaced with single imputations^[5]. Maximum likelihood and multiple imputation are advantageous^[6]. The deletion method includes two approaches: listwise deletion and pairwise deletion. In SAS, the default effect selection of the LOGISTIC procedure listwise deletes any observation with missing values in independent and response variables. The analysis is restricted to the remaining complete data set using standard procedures^[7]. Listwise deletion of missing values is under the assumption of MCAR and results in bias estimates since MCAR is rarely satisfied^[5]. In addition, deleting missing values severely reduces the analyzable sample size, particularly for data with a high proportion of missing values or missing in many variables. This results in the loss of power of the significance test^[8].

Single imputation is another traditional approach to processing missing data. The most widely used single imputation methods include mean imputation, regression imputation, and stochastic regression^[9]. Mean imputation replaces the missing values with arithmetic mean based on the available data of the corresponding variables. The added mean values are equivalent to a group of uncorrelated data, which results in an underestimation of the overall correlations. On the other hand, regression imputation will strengthen the variable correlation. It predicts a regression equation from available values and replaces the missing values with predicted regression outcomes^[6]. Mean imputation and regression imputation dramatically reduce the variability of data and result in bias. Stochastic regression imputation adds residual terms to regression imputation, which effectively restores the variability represented by the lost values^[3].

Multiple imputation

Multiple imputation (MI) is the procedure that replaces the missing value with more than one imputation method. Each missing value is replaced by m (m > 1) plausible imputed values given the observed values. The imputation creates m complete data sets which are analyzed separately with the same procedure, resulting in m sets of parameter estimates and standard errors. The multiple sets of results are subsequently merged into one set of results. These are the three steps of multiple imputation: imputation phase, analysis phase, and pooling phase^[10−13].

Imputation algorithms - Markov Chain Monte Carlo

Multiple imputation is regarded as a model-based imputation since the imputation model is constructed according to the distributional relationship between the missing values and the observed values^[14]. Data augmentation is the most often used algorithm for normally distributed data which belongs to the family of Markov Chain Monte Carlo (MCMC) procedure. MCMC draws pseudo-random samples with the Monte Carlo method using Markov chains^[10,11]. MCMC is a composite of two steps: The imputation step (I-step) and the Posterior step (P-step). Initial I-step is principally identical to stochastic imputation where model-based imputed values replace the missing values and the error terms are added to the estimated parameters to represent the uncertainty of observed data. In P-step, the I-step data perform as building blocks to generate new estimates of means and covariances under the Bayesian estimation principles. The new parameters randomly differ from those used in the previous I-step and are successively used to create imputed values in the preceding I-step. Repeating this two-step procedure for specific times yields multiple copies of complete data sets, each of which contains unique estimate parameters of the missing values^[15,16]. Multiple imputation shares the advantage of a single imputation and restores the variability of imputed complete data.

SAS MI procedure

To apply the MI procedure, the type and distribution of the data need to satisfy certain assumptions. MI assumes that the data are from a continuous multivariate distribution and contain missing values that may occur on any of the variables. It also assumes that the missing data are under the MAR mechanism^[17]. The next step is to analyze the existing missing data patterns, arbitrary or other specific patterns such as monotone. The PROC MI default MCMC method assumes that the multivariate normal data can be used. MCMC is an appropriate method for continuous arbitrary missing data^[18].

SAS LOGISTIC procedure forward selection provides an automated model selection for binary response. It computes and ranks the p-value of the Chi-square statistic for each effect not in the model. If the minimum p-value is smaller than α, this corresponding effect is added to the model and never removed. The above procedures are repeated until none of the remaining effects is significant at level α^[17]. However, forward selection is not a good method to process missing data, since the observations with missing values are entirely excluded from analysis. The huge loss of information represented by incomplete observations impairs the accuracy of effect selection and power of significance test.

To keep the maximum information in incomplete observations, the Macro-based Forward Model Selection (MFMS) method was developed, in which the macro controls the automated effect selection using the forward selection algorithms. The biggest advantage of MFMS is to process one variable each time while keeping the incomplete observations in other variables undeleted. MFMS can be decomposed into steps: p-values of the Chi-square score for all one-variable models are calculated and ranked using the SAS LOGISTIC procedure. The smallest p-value (P-min) is considered as the candidate for the first cycle. If P-min is smaller than α, it indicates that the corresponding variable is significant at level α and eligible to enter the model. Only if the addition of the first effect happens, we consider all the two-variable combinations consisting of the first enrolled variable and one of the remained variables not in the model. Again, the smallest p-value is compared with α. If significant, the effect not in the previous model will enter as the second effect. Otherwise, the program ends up with the previous model. Macro controls the sequential entry of significant effects one by one until the termination point and returns the final effects list. The final model will contain the best significant effect added in each cycle.

Fan^[19] reported the application of MFMS algorithms to analyze missing data in linear regression and concluded that MFMS is superior to traditional REG and MI procedures.

Simulation and model assumptions

Logistic regression is used to investigate the binary response, ordinal response, and nominal response. For binary response models, the response variable Y can take on one of two possible values, denoted for convenience by 1 and 2. Suppose vector X is the explanatory variable, and the response probability is:

The logic response model can be a linear form as:

A logistic regression model with 200 observations was simulated, each of which consists of a binary response Y and 20 independent variables X₁, X₂, …, X₂₀, as Table 1. In most cases, actual data usually have a small sample size and a complex set of variables. Additionally, the smaller the sample size, the larger the influence that the missingness could cause. So, the complete data was simulated with 200 samples and 20 X variables.

The model relies on several assumptions. Each X variable is independently drawn from a standard normal distribution as our complete explanatory variable matrix X. Y is the binary logistic function of X. Thus, our complete data set is a binary response with 20 multivariate normal-distributed explanatory variables. The assumption of our models, along with MCAR and MAR mechanisms, satisfy the requirements for the SA LOGISTIC and MI procedures.

The complete and missing data were simulated based on the following logistic function with descending coefficients:

The descending order of coefficients was designed to examine how the coefficients can affect the model selection.

First, the 100 simulated complete data sets were processed by traditional logistic procedures with forward selection and each of the final effects list was recorded. By counting the effect selected times in the 100 procedures, the numbers of each X selected out of 100 times were acquired. Similarly, the 5%, 10%, 15%, and 20% randomly deleted datasets were processed by the proposed MFMS method, SAS traditional logistic procedure, and MI procedure respectively. Merging the results from the first and the second steps by SAS SQL procedure, an overall list of each X variable selected numbers were achieved for the complete data and at each missing percentage as shown in Tables 2−7.

Results

The development of the MFMS method aims to circumvent the shortcomings of LOGISTIC forward selection when processing missing data of logistic regression. MI procedure MCMC method is the most popular logistic modeling procedure. The MFMS, PROC LOGISTIC, and PROC MI methods were compared by testing the selection frequency of each X variable. The significant level of entry for all methods was 0.1. Detailed methodologies are described in the Methods section. Several rules for model selection accuracy were established: (1) the true model variables could be selected 50 times or more; (2) the true model variables could be enrolled into models more frequently than other variables; (3) the results of model selection and parameter estimates should be close to those of true models or complete data. (4) less false positive effects. The methods that satisfy those criteria are believed to be reliable for processing missing data. Tables 2−5 show the selecting frequency of X variables at different missing levels among all methods. Tables 6 & 7 present the selecting frequency of X variables under different missing mechanisms.

Columns 2−4 in each table show the results of complete data model selection by PROC LOGISTIC at significance level 0.1 where X1, X2, and X3 are in the true model and X4−X20 are not in the model. Tables 2−5 show that with the current model setting, almost all three model variables, except X3 for 96 times, were selected to enter into the model 100 times. The frequencies of other non-model effects were far behind. In addition, the coefficient estimates of X1−X3 matched the true model with minor differences. The above indicated the accuracy of the LOGISTIC procedure in model selection for complete data.

To demonstrate the model selection efficiency and accuracy, the performance of the three modeling methods were analyzed at each missing level under MCAR.

At a 5% missing level (Table 2), the model selection pattern and the coefficient estimates obtained from the proposed MFMS method were highly consistent with the complete data. X1, X2, and X3 were found for 100, 99, and 87, respectively. This suggests the validity of the MFMS method at a 5% missing level. SAS PROC LOGISTIC did not perform as well as MFMS, which had a lower selection frequency and a larger difference compared to MFMS. However, it was still acceptable in the current setting. The MI also identified X1 and X2 100 times, while X3 was found 98 times. The coefficient estimates were also close to those of the complete data. However, after carefully examining the MI results, it was found that all the non-model variables were included in the model more than 50 times. This indicated that the MI procedure not only selected the true model variables but also incorrectly included the non-model variables at a high frequency. This increased the likelihood of achieving a biased conclusion based on the insignificant model. The issues inherent in the forward selection and the MI procedure worsened with increasing missingness.

To evaluate the performance of the three model selection methods under MAR data scenarios, datasets with a 5% missing level for MAR were established. The results, shown in Table 6, demonstrate consistent conclusions with those observed under MCAR. Since NMAR does not meet the assumptions required by the LOGISTIC and MI procedures, only the performance of the MFMS method under the NMAR mechanism were evaluated. The results, presented in Table 7, indicate that the MFMS method also performs well under NMAR conditions.

At a 10% missing level shown in Table 3, MFMS enrolled X1−X3 100, 97, and 79 times, respectively. The parameter estimates were close to the complete data with slight changes. This approved that MFMS was reliable at the 10% missing level. SAS LOGISTIC could not maintain its accuracy. It found X1 54 times, X2 35 times, and X3 25 times. The selection frequency dramatically decreased for true model variables. Only X1 entered the model more than 50 times, while X2 and X3 could not stand out from other non-model variables. It indicated that PROC LOGISTIC was no longer able to build up a correct logit model. This was because missingness at this level caused a huge loss of usable data. Although it was only 10% missing in each X variable. When considering 20 X variables, the listwise deletion algorithms resulted in a much higher percentage of overall missing. The complete subset for SAS LOGISTIC was much smaller than the 200 sample size. As seen with a 10% missing level, MI enrolled too many non-significant variables and was not a reliable method.

The results are shown in Table 4 for 15% missingness. MFMS sturdily selected the true model variables with a similar frequency of 10%. However, the estimates of coefficients moderately deviated from the true model. SAS LOGISTIC failed to select any significant variable. All three true model variables were found under 50 times. The coefficient estimates by MI started to deviate from the true model while the model selection was almost the same as previous missing levels.

As seen in Table 5, the 20% missing level where MFMS still found the true model was examined, but the parameter estimates largely deviated from the complete data. SAS LOGISTIC completely lost its accuracy of model selection. No significant variable was found. Parts of the X variables, including X3, X9, X14, X16, and X19, were not found even once. No parameter was estimated for those variables. Those demonstrated that SAS LOGISTIC was no longer able to analyze the data at a high missing level. The frequency of variable selection by MI seldom changed with increasing of missing level. The parameter estimates deviated further from the true model.

In summary, the MFMS satisfied the criteria to be a reliable logistic modeling method at 5% and 10% missing levels. MFMS exhibited a high consistency of model selection pattern compared to the complete data set. The three true model variables were successively selected at all missing levels. Additionally, the happening of false positive selection was efficiently forbidden, since MFMS only added the best significant effect with the smallest p-value each cycle. MFMS also extensively reduced the exclusion of incomplete observations. Only the missing values in the current analyzed variable were excluded and the other variable was untouched. It maintained the maximum information and uncertainty of the missing data. As proof, the parameter estimates at a 5% missing level were very close to those of complete sets. With the increase of missingness to 15%, the parameter estimates slightly deviated from complete data. However, compared to the SAS LOGISTIC and MI procedures, MFMS was still the best one. At a 20% missing level, MFMS was able to select the right model, but with incorrect parameter estimates due to the low events per variable.

SAS LOGISTIC excluded any observation that contained missing values. When missingness happens in multiple variables, the listwise deletion causes severe loss of information. The selecting frequency of true model variables dramatically decreased at 10% and higher missing levels. Finally, it completely failed to select the accurate model and estimate the parameters at the 15% and 20% missing levels.

The MI procedure could choose the true model variables at all missing levels 100 times. However, the selection frequency of non-significant variables was also around 50 times. MI automatically sampled the missing data with model-based imputation, which led to an overestimation of the logistic association and a narrower confidence interval. Those resulted in insignificant model selection and biased parameter estimates.

In summary, it was concluded that MFMS was the best reliable selection method for binary logistic models with independently normal-distributed multivariate for low-moderate level missing under MCAR, MAR, or NMAR assumption.

The World Health Organization estimates 12 million annual deaths from heart diseases globally, with half of all deaths in the United States and other developed countries attributed to cardiovascular diseases. Early detection enables high-risk individuals to adopt preventive lifestyle changes. A publicly available cardiovascular dataset from a study in Framingham, Massachusetts (USA), with over 4,000 records and 15 attributes, was analyzed to identify key risk factors for heart disease. The dataset is publicly available on the Kaggle website. Given its missing values, this dataset was used to compare SAS forward selection with the Macro-based selection method. Results are shown in Tables 8 & 9.

The results indicate that both methods identify key variables — age, systolic blood pressure, cigarettes smoked per day, gender, glucose level, and total cholesterol level — as significant predictors of cardiovascular disease risk. However, the macro-based selection method also includes an additional variable, the history of stroke, with a significant p-value (0.016), suggesting it may have identified an additional important predictor overlooked by the forward selection method.

Logit function parameters are estimated using the method of maximum likelihood^[17]. While sample size does not directly influence parameter estimates in logistic regression, the number of events is critical. However, unless the two levels of the binary response are highly imbalanced, such as 10% vs 90%, a reduction in sample size indirectly impacts parameter estimates by decreasing the number of events. Research indicates that biases in parameter estimates decrease as sample size increases in binary logistic models^[20]. Insufficient events can lead to infinite maximum likelihood estimates, resulting in parameter estimation failure. The minimum requirement for accurate estimation of logistic parameters is 10 events per explanatory variable (EPV)^[21].

The present simulation included 20 explanatory variables (X) and 200 samples. Under the assumptions of the model, the probability of each binary response was not extremely imbalanced. However, the EPV in the complete dataset was lower than 10, which explains the minor bias observed in parameter estimates. At low levels of missingness, the MFMS method was reliable for selecting significant variables and estimating parameters since the EPV did not drop substantially. As missingness increased to 20%, the EPV decreased dramatically. Although MFMS could still identify significant variables, parameter estimates were biased due to the lower EPV, which resulted in increased variance and wider confidence intervals.

When the sample size increased to 500, the accuracy of parameter estimates improved significantly compared to the estimates derived from 200 samples, as shown in Appendix I. However, given funding and time constraints, researchers often work with smaller sample sizes of around 200. To address parameter estimate bias at high levels of missingness, MFMS can first be applied to identify significant variables, such as X1–X3. A logistic model can then be fitted with only these significant variables, substantially increasing the EPV. Parameters can subsequently be re-estimated using PROC LOGISTIC or PROC MI. This approach was tested in the present model, and the re-estimated parameters, presented in Appendix II, were much closer to the true model values.

In the simulated logistic model, it was assumed that the X variables were independently normally distributed and considered only the main effects in the true model. Interactions between X variables were not accounted for. Independence among variables is rare in practical scenarios. Interactions can provide valuable information about relationships and associations between variables. Future research will explore the effects of interactions on model performance.

Another limitation of the present approach is its focus on continuous explanatory variables. Extending the procedure to accommodate mixed categorical and continuous variables would make it more applicable to a broader range of problems.

Additionally, the simulated data used in this study have the same sample size and the same percentage of missingness across all variables. In real-world data, the percentage of missingness and the sample size of the complete subset often vary between variables, leading to differences in the degrees of freedom (DOF) for each variable. MFMS probably could not find the criteria for statistics. Despite this, MFMS demonstrated robustness by successfully selecting significant variables one-by-one using the Chi-square-associated p-value. In this study, the impact of DOF differences was negligible.

The model currently focuses mainly on MCAR mechanisms. However, MAR scenarios are more prevalent in practice. Under MAR, the MI procedure often performs better. Future research could explore using MI to preprocess MAR data by identifying a subset of variables that fit the model. This subset could then undergo a LOGISTIC backward selection step to re-evaluate and remove false positives^[6,7,10]. However, MI is not always the optimal method for MAR or other complex data scenarios. Since MI involves random draws, the parameter estimates and test statistics can vary between runs on the same dataset, potentially leading to different conclusions from the same data and procedures^[22].

The MACRO code is provided in Appendix III. This MACRO can also handle categorical predictors and interaction terms. Additionally, the methodology in the MACRO can be adapted for model selection in other SAS procedures, such as GENMOD and MIXED, broadening its utility.