Statistical significance should not be conflated with explanatory importance: a comment on "China's solar expansion policy reduces bird diversity"

In Table 1, PSI is described as "the independent variable," whereas the other nine variables are described as "controls." This terminology gives the impression that the analysis estimates the effect of PSI after accounting for a set of secondary variables, and may lead readers to interpret PSI as the principal explanatory factor while regarding the other variables merely as adjustment variables.

In their article, Zhang et al.^[1] state:

"We estimated the relationship between policy stringency and bird diversity using a high-dimensional, two-way fixed-effects regression model. This approach isolates the policy effect by controlling for time-invariant county characteristics and common temporal shocks, alongside time-varying meteorological, geographic, and socioeconomic covariates (e.g., temperature, wind speed, population density, bird-watching duration, carbon emissions, and land cover proportions)."

However, this description does not accurately reflect the statistical roles of the variables in the fitted model. Using the raw data provided by the authors, we independently reproduced the analysis using fixest::feols() in R (Bergé et al.^[2], fixest packages version: 0.14.2). The raw data in R format and the R code used for the reproduction are provided in the Supplementary File 1. The corresponding code, which reproduces the results reported in the third column of Table 1, is as follows:

feols(ShannonBD ~ PSI + Temp + Wind + Pop + Duration + Carbon + Water + Green + Farm + Grass | county + year, data = d1,cluster = ~county)

The model can be expressed as:

ShannonBD = β₁PSI + β₂Temp + β₃Wind + β₄Pop + β₅Duration + β₆Carbon + β₇Water + β₈Green + β₉Farm + β₁₀Grass + α_i + γ_t + ε

where α_i denotes county fixed effects, γ_t denotes temporal fixed effects, and ε denotes the model residual term.

From a statistical modeling perspective, PSI and the other nine variables all enter the same linear predictor and therefore occupy the same statistical level as explanatory covariates in the fitted model. Although researchers may designate one variable as the variable of primary interest and refer to the remaining variables as controls for the purposes of a particular research question, such labels describe the analytical focus rather than creating intrinsically different statistical roles within the regression model. In a multivariable regression model, the predictors included in the linear predictor are jointly estimated, and their estimated coefficients represent associations conditional on the other variables included in the model. Thus, none of these variables is intrinsically an "independent variable" while the others are intrinsically "controls" merely because of how they are labeled in the table.

This distinction is particularly important for interpreting the results presented in Table 1. Describing PSI as "the independent variable" and the other nine variables as "controls" may lead readers to infer that PSI is the principal explanatory factor and that the remaining variables play merely a secondary adjustment role. Such wording may consequently give the impression that the model establishes a relatively large or otherwise distinctive contribution of PSI after the effects of the other variables have been removed. However, the fitted model itself does not support such an interpretation. All ten variables are simultaneously estimated within the same two-way fixed-effects regression framework and are subject to the same county and temporal fixed effects. The designation of PSI as "the independent variable" therefore reflects the authors' substantive focus rather than a distinct statistical property of the model. This distinction is especially important when communicating statistical results to a broad scientific audience. Given the visibility and broad interdisciplinary readership of Science, statistical terminology should be sufficiently precise to avoid implying a hierarchy among covariates that is not actually specified or estimated by the statistical model.

A more fundamental concern relates to the interpretation and presentation of the coefficient of determination (R²) reported in Table 1. Although the authors may not intend to use R² as a measure of the importance of PSI, reporting the overall R² without distinguishing it from the within-model explanatory variation (within R²) may lead readers to overinterpret the explanatory power of the focal variable. This issue is particularly important for a two-way fixed-effects model, in which a substantial proportion of the observed variation may be captured by the fixed effects rather than by the explicitly listed time-varying covariates.

The fitted model includes both county and temporal fixed effects:

| county + year

Consequently, the overall R² of the complete model reflects the model's ability to account for variation associated with both the fixed effects and the explicitly included time-varying covariates. It therefore cannot be interpreted as the proportion of variation explained by PSI and the other covariates alone. In our independent reproduction of the third-column in Table 1, for example, the overall R² is approximately 0.2999, whereas the corresponding Within R² is only 0.0398. The Within R² measures the proportion of the variation remaining after accounting for the relevant fixed effects that is explained by the time-varying covariates. In the present example, the Within R² of 0.0398 indicates that the ten explicitly listed covariates collectively account for only about 4.0% of the within-county variation in the response. Relative to the total variation in the outcome, this corresponds to approximately (0.0398/0.2999 = 0.1327), or 13.3%. This magnitude is not surprising and, in fact, indicates that substantial heterogeneity in bird diversity exists across counties and years that is captured by the fixed-effects structure rather than by the explicitly listed covariates.

More importantly, the contribution attributable specifically to PSI is considerably smaller. In the first-column specification, where PSI is the only time-varying explanatory variable, the within R² is approximately 0.0007, compared with an overall R² of 0.2715. Thus, the relevant explanatory variation associated with PSI should not be inferred from the overall R² of 0.2715. Rather, the Within R² of approximately 0.0007 indicates that PSI accounts for only a very small fraction of the variation remaining after the county and temporal fixed effects have been removed. Relative to the within R² of the full model (0.0398), this corresponds to approximately (0.0007/0.0398 = 0.0176), or only 1.8% of the within-model explanatory variation. It is therefore difficult to justify drawing such a strong ecological conclusion from an effect that accounts for only 1.8% of the explanatory variation captured by the full model.

The presentation of R² across the columns of Table 1 further illustrates why this distinction matters. The reported overall R² values increase from 0.2715 in the first specification, which includes PSI, to 0.2998 after additional covariates are introduced, and then to 0.2999 in the full specification. Viewed without reference to the fixed-effects structure or the within R², these values may give readers the impression that adding the additional covariates produces little change in the model's explanatory power and, by implication, that these variables contribute little to explaining bird diversity. However, this interpretation is not warranted from the reported overall R² values alone. The small changes in overall R² largely reflect the fact that a substantial portion of the total variation is already captured by the county and temporal fixed effects. What is more relevant for assessing the explanatory contribution of the time-varying covariates is the variation that remains after these fixed effects have been accounted for.

If the authors do not intend to use R² to support an argument about explanatory importance, then reporting the overall R² without clearly explaining its interpretation is potentially confusing. Conversely, if R² is presented as evidence relevant to the explanatory performance of the models, the distinction between overall and within R² should be made explicit. In a high-profile interdisciplinary journal such as Science, where statistical results are communicated to a broad scientific readership, such distinctions are particularly important. Clear reporting of the appropriate R² measure would help prevent readers from conflating the explanatory variation captured by the fixed effects with the explanatory contribution of the focal variable, PSI. More importantly, it would provide a more transparent basis for evaluating whether the statistically significant association between PSI and bird diversity is also substantively important in terms of explanatory contribution.

In ecological research, the magnitude and explanatory contribution of an effect are generally more informative for assessing its biological relevance than whether its P-value falls below an arbitrary significance threshold. A small P-value indicates that an association is statistically detectable, but it does not indicate that the effect is large or ecologically important. This distinction is particularly important in large datasets, where even very small effects can achieve high statistical significance because statistical power increases with sample size (Nakagawa & Cuthill^[3]; Sullivan & Feinn^[4]).

Although the coefficient of PSI is statistically significant (P < 0.01), its explanatory contribution is extremely small (0.0007). Thus, the statistical significance of PSI indicates that its association with bird diversity is detectable, but provides little evidence that PSI is an important explanatory driver of variation in bird diversity. Interpreting this highly significant coefficient without considering its very limited explanatory contribution may therefore give readers a misleading impression of the ecological importance of the effect. The distinction between statistical significance and biological importance is particularly important here because the title and central conclusion of the study imply a substantive effect of solar expansion policy on bird diversity, whereas the statistical results indicate that the contribution of PSI to the explained within-county variation is very limited.

If statistical significance alone were used to evaluate the relative importance of the explanatory variables, Duration would already stand out among the nine variables described as "controls." Based on our independent reproduction using the raw data, the coefficient for Duration is associated with a much smaller p-value (p < 2.2 × 10⁻¹⁶) than that reported for PSI (p = 0.00062). More importantly, the within R² we calculated for Duration is approximately 0.0388, which is about 55 times larger than that of PSI (0.0007). Thus, the explanatory contribution of PSI is less than 2% of that of Duration. From the perspective of the fitted model, Duration and PSI have the same statistical role as explanatory variables, even though PSI is the authors' focal variable. Yet the original study does not discuss the much stronger statistical and explanatory contribution of Duration, while extensively emphasizing the statistical significance and ecological implications of PSI. From an ecological perspective, this selective emphasis is problematic because it may lead readers to overestimate the relative importance of PSI in explaining variation in bird diversity.

The title and central conclusion of the paper state:

"China's solar expansion policy reduces bird diversity."

Our concern is that this conclusion places substantial emphasis on the statistical significance and negative sign of the PSI coefficient without adequately communicating its very small explanatory contribution relative to other variables included in exactly the same model.

We believe that these distinctions are essential for an accurate interpretation of the central findings. Without explicitly reporting and considering the within R² and the contributions of the explanatory variables, the current presentation of Table 1 risks overstating the explanatory importance of photovoltaic policy stringency and understating the much larger contribution of other variables included in the same model. The new statistical evidence presented here therefore calls for a substantial reassessment of whether the reported regression results, by themselves, adequately support the strength of the paper's central conclusion.