Which Climate Model Evaluation Methods Can Consistently Select Skillful Models from the CMIP6 Ensemble?

INTRODUCTION

At present, climate models are the most valuable tool in projecting future climate under different scenarios (Taylor et al. 2012; Stocker et al. 2014; Otero et al. 2018). Over the years, global climate models (GCMs) have been continuously developed by different modeling centers, incorporating diverse parameterizations. Consequently, future climate projections from these models can vary significantly, leading to substantial uncertainty in projected changes of climate variables (Knutti et al. 2010; Stocker et al. 2014). In addition to this considerable projection uncertainty, individual models often have significant biases. Therefore, a multi-model ensemble of GCM simulations is commonly employed in research, as it has been shown that ensemble means tend to cancel out individual model biases - i.e., the ensemble mean of a large group of models generally outperforms any single model in most cases (Gleckler et al. 2008; Knutti et al. 2010; Raju and Kumar 2020).

However, in the IPCC’s Sixth Assessment Report (AR6), climate models were assigned weightings when assessing future projections for the first time. This was due to the “hot model” problem: a subset of CMIP6 models exhibited climate sensitivities outside the range estimated from multiple lines of evidence (Sherwood et al. 2020). This issue has prompted efforts within the community to establish robust model selection criteria (Hausfather et al. 2022). Research indicates that selecting skillful GCMs can reduce uncertainty in ensemble projections compared to using the full set of dozens of models (Herger et al. 2018; Gnatiuk et al. 2020). As a result, various model selection criteria and evaluation methods have been introduced in the literature. A categorization mind map of these methods is provided in Appendix A.2.

One approach to reducing uncertainty is to select only those models that perform well in historical simulations when compared to observations. The primary assumption underpinning this approach is that model skill in historical simulations is a reliable predictor of model performance in future climate projections; that is, models that are skillful in historical simulations are also likely to be skillful in their response to forcing. Numerous studies have demonstrated that the response to forcing is sensitive to baseline climatology, which supports this assumption (Caballero and Huber 2013). However, if there is a substantial change in the relative importance of different processes shaping regional climate, then this assumption – that historical skill predicts future model performance – may not hold. Additionally, given the considerable internal variability within models, it is essential that the evaluation of model skill be conducted over a long period (several decades) to ensure a fair assessment of model performance (Jain et al. 2023). Furthermore, it is crucial that the criteria for model selection are not overly sensitive to the phase of natural variability within the climate system.

There is no consensus among researchers on best practice for climate model evaluation and selection (Knutti et al. 2010; Ahmadalipour et al. 2017; Herger et al. 2018; Calvin et al. 2023). This lack of agreement has resulted in a diverse range of approaches; for example, some studies employ only a single statistical metric for GCM evaluation (e.g., Walsh et al. 2008; Macadam et al. 2010; Sillmann et al. 2013; Agosta et al. 2015), while others utilize multiple statistical metrics (e.g., McMahon et al. 2015; Aghakhani Afshar et al. 2017; Ruan et al. 2019). For instance, the near-surface air temperature simulations from 17 GCMs were analyzed using just one statistical metric – root mean square error (RMSE) – and models were ranked from the lowest to the highest RMSE values (Reifen and Toumi 2009; Macadam et al. 2010). Herger et al. (2018) selected an optimal subset of 38 GCMs for surface air temperature and precipitation based on RMSE. RMSE was also employed for inter-model comparison and evaluation by Sillmann et al. (2013) and Zhou et al. (2014). Other statistical metrics have been used to assess GCM accuracy as well – for example, Kumar et al. (2013) evaluated 19 GCMs based on trends in temperature and precipitation across continental areas. Maxino et al. (2008) and Perkins et al. (2007) proposed a skill score that measures the common area between the probability density functions (PDFs) of modeled and observed data.

Many studies evaluating the accuracy of GCMs utilize the Taylor diagram, which combines three statistical criteria – standard deviation (STD), RMSE, and correlation coefficient (r) (Taylor 2001). This diagram is summarized into a single metric – the Taylor skill score (Taylor 2001; Inoue and Ueda 2011; Ogata et al. 2014; Sharmila et al. 2015; Kadel et al. 2018; Yang et al. 2020). It should be noted that even when employing the same evaluation methods, researchers often apply different thresholds for sub-ensemble selection, as there is frequently no clear criterion defining a model as “good” or “bad” within these evaluation frameworks. For example, the Taylor skill score has been utilized by Sharmila et al. (2015), Kadel et al. (2018), and Yang et al. (2020), but with varying thresholds.

Some studies evaluating GCMs have employed multiple statistical metrics. For example, McMahon et al. (2015) assessed 23 GCMs using RMSE, the Nash-Sutcliffe Efficiency coefficient (NSE), and the coefficient of determination (r²) for temperature and precipitation patterns. Kumar et al. (2015) considered bias, trend analysis, and Taylor diagrams to evaluate simulations of extreme winds from 15 GCMs across 22 regions. Aghakhani Afshar et al. (2017) evaluated 14 GCMs for precipitation using four statistical criteria: r², NSE, percent of bias (PBIAS), and the ratio of root mean square error to the standard deviation of measured data (CPI). Furthermore, Aghakhani Afshar et al. (2017) categorized the statistical metrics into four groups – very good, good, satisfactory, and unsatisfactory – using a threshold criterion, ultimately selecting models with scores between 75% and 100%, which were ranked as the very good group. Jiang et al. (2015) evaluated 31 GCMs for total precipitation and three indices (the fraction of total rainfall from events exceeding the long-term 95^th percentile, precipitation intensity, and maximum consecutive dry days) over China, employing a Taylor diagram and the Interannual Variability Skill Score (IVS; Chen et al. 2011). They further ranked the models using a Comprehensive Rating Metric. You et al. (2018) applied a similar analysis and model selection process as Jiang et al. (2015), but additionally analyzed trends and IVS for both the sub-ensemble and full ensemble across 16 temperature indices. This comprehensive rating metric has been utilized in many studies for model ranking, including those by Jiang et al. (2015), You et al. (2018), Rao et al. (2019), Ahmed et al. (2019, 2020), and Cai et al. (2021).

Other researchers have employed more complex methods involving multiple (up to seven) statistical criteria for evaluating the reliability of GCMs (e.g., Fu et al. 2013; Rupp et al. 2013; Ruan et al. 2019; Jia et al. 2019; Gnatiuk et al. 2020). These studies used different combinations of variables – such as air temperature, precipitation, wind speed, shortwave radiation, and nutrients – and a varying number of GCMs, ranging from 11 to 41. To compare the models, researchers ranked them based on their total scores and selected the relatively best GCMs: for example, 8 out of 41 models – around 20% (Rupp et al. 2013), the top 25% of models (Ruan et al. 2019; Gnatiuk et al. 2020), or the top 30% (Jia et al. 2019). All of these authors suggest the use of a method that incorporates multiple statistical criteria rather than relying on a single metric.

Furthermore, there is no universally accepted method for climate model evaluation and selection. A significant challenge is the trade-off associated with ensemble size: the stricter the filtering of non-skillful models, the smaller the ensemble becomes, which can increase the influence of individual model biases in the ensemble mean projections.

Any model selection criterion should also be robust – that is, it should consistently identify similar models as skillful across different time periods used for model evaluation. This task presents particular difficulties given the large natural variability and internal variability within models. Consequently, there has been a shift toward evaluating models based on processes rather than states (Eyring et al. 2019).

In summary, various methods are employed to evaluate and select appropriate GCMs for specific research questions (Herger et al. 2018; Raju and Kumar 2020; Calvin et al. 2023). Chai and Draxler (2014) recommend using several statistical metrics, including RMSE and mean absolute error (MAE). Raju and Kumar (2020) suggest considering the statistical metrics in a category-wise manner – for example, one metric for error, one for correlation, and one for skill score – and then computing the overall weight of the metrics, for instance, using a rating method. Fu et al. (2013), Ruan et al. (2019), Jia et al. (2019), and Gnatiuk et al. (2020) propose the application of multiple criteria for model evaluation.

Currently, there is no comprehensive comparison of these different methods for evaluating climate models; therefore, this study aims to fill this gap. We tested several widely used approaches, ranging from a single statistical metric to multiple metrics: (i) RMSE, (ii) spatial trends, (iii) TSS, (iv) PDF, (v) Taylor diagram, IVS, MR, (vi) Taylor diagram, MAE, trend, and (vii) a percentile-based method for evaluating CMIP6 surface air temperature (SAT) in the Arctic. To compare these methods with each other, we ranked GCMs according to their performance for each approach and selected the top 25%. Additionally, we conducted the analysis over two different periods – 1951-1980 and 1981-2010 – to assess the consistency of these methods.

MATERIALS AND METHODS

Study area and data

In this paper, we compare model evaluation methods based on surface air temperature in the Arctic (60-90° N). Simulations of historical surface air temperatures from 25 GCMs from the Coupled Model Intercomparison Project Phase 6 (CMIP6) were obtained from the Earth System Grid Federation portal¹. Information about these models is provided in Table A.1. The list of model names is presented in Fig. 1. The model data were compared to observations from the Berkeley Earth database (Rohde et al. 2013; Rohde and Hausfather 2020). The Berkeley Earth database is a comprehensive global land-ocean temperature record that integrates monthly land temperature data from over 40,000 weather stations with sea surface temperature data from HadSST3 (Hadley Centre Sea Surface Temperature dataset, version 3). Using kriging-based spatial interpolation, it provides extensive spatial coverage for the period spanning from 1850 to the present. It offers average temperatures in 1° x 1° latitude-longitude grid cells for each month.

We selected seven of the most frequently used methods. Table A.3 summarizes the published works employing these methods. Each method is described in detail below:

i) Method of model comparison by root mean square error

Root mean square error (RMSE) is a commonly used statistic to quantify differences between two fields of data (Eq. 1):

(1)

where Tm_i is the temperature from the model and To_i is the temperature from observations at time step, i, with n representing the number of measurements in the time series. The smaller the RMSE, the better the agreement between the two data fields.

ii) Method of model comparison by trends

Analysis of spatial trends using statistics such as the correlation coefficient (r), standard deviation (STD), and mean value (T) is proposed by Kumar et al. (2013). The correlation coefficient r is calculated as (Yang et al. 2020) (Eq. 2):

(2)

where Tm_i is the temperature from a given model and To_i is the temperature from observations at time step i, T m is the average temperature of the model, while T o is the average temperature of the observations, n represents the number of observations in the time series, STDm is the standard deviation of the model temperatures, and STDo is the standard deviation of the observed temperatures.

STD is calculated as (Eq. 3):

(3)

where T_i is the temperature at time step i, T is the mean temperature, and n is the number of data points in the time series.

Trend (Tr) was calculated using the least squares method (Eq. 4):

(4)

where T_i is the temperature at time step i, T is the average temperature, Y_i is the time step of the time series, is the mean of the time series, n is the number of data points in the series.

Each model was ranked based on its ability to represent observations across three metrics: r, the difference between the mean values of models and observations, and the difference between the STD of models and observations.

iii) Method of model comparison by Taylor skill score

The Taylor skill score (TSS) is calculated using r and STD statistical metrics (Taylor 2001) (Eq. 5):

(5)

where r is the correlation coefficient and STDN is the ratio of the model’s and observations’ STD. The TSS is bounded between 0 and 1, with 1 indicating a perfect fit between the model and observations.

iv) Method of model comparison by S_score

The skill score based on PDFs (S_score) was proposed by Perkins et al. (2007) as a robust metric for evaluating and ranking climate models because it is less affected by observational errors than the mean value and standard deviation. The S_score measures the common area between the PDFs of observed and model data. The formula for S_score is expressed as (Eq. 6):

(6)

where n is the number of temperature bins, Z_m is the frequency of model data values within the corresponding bin, and Z_o is the frequency of observed data values within that bin. Therefore, the S_score ranges between 0 and 1, with a score of 1 indicating a perfect match between the observed and model distributions and a score of 0 indicating no overlap between them. When applying this skill score metric, we used bins that are 1°C wide.

v) Comprehensive rating metric based on Taylor diagram and interannual variability skill score

The Comprehensive Rating Metric (MR) was proposed by Jiang et al. (2015) (Eq. 7):

(7)

where n is the total number of parameters, m is the number of models, and rank is the position of a given model based on its performance (with 1 being the best model). The metric ranges between 0 and 1, with higher MR values indicating better model skill. The MR was calculated for each of the metrics used in the Taylor diagram (r, STDN, RMSE), and an average MR value was derived for each metric.

Additionally, MR was calculated for the Interannual Variability Skill Score (IVS; Chen et al. 2011) (Eq. 8):

(8)

where STDm is the standard deviation from the model and STDo is the standard deviation of the observations. Smaller IVS values indicate that the simulated variability more closely matches the observed variability.

vi) Method of model comparison by Taylor Diagram, bias and trend

Kumar et al. (2015) proposed to evaluate models using Taylor diagram, bias, and trend statistics. Bias (B) is calculated as the mean of the differences between the model and observations (Eq. 9):

(9)

where n is the number of observations in the time series.

vii) Percentile-based method

A percentile score-based model ranking method introduced by Gnatiuk et al. (2020) includes the analysis of the mean spatially averaged climatology of the annual cycle, interannual variability of parameters using r, RMSE, STD, the Climate Prediction Index (CPI – ratio of RMSE to the STD of the observations) (Agosta et al. 2015), as well as the spatial trends and biases (at each grid point) between the model data and reanalysis/observations to illustrate how temperature varies across the study area. The range of the statistical indices for each model was divided into four categories: 0-25% – very good, 25-50% – good, 50-75% – satisfactory, and 75-100% – unsatisfactory. Each category was assigned a score from 3 to 0, respectively. For correlation, the scoring was reversed. These scores were then summed for each model to obtain a total skill score. Based on this total skill score, the top 25% of GCMs were selected as an optimal ensemble.

Approach for comparing model evaluation methods

SAT in the Arctic was analyzed for two periods, 1951-1980 and 1981-2010, to evaluate the consistency of model evaluation methods. We assume that if the models selected by each individual method are consistent across these two periods, then the method can reliably identify skillful models regardless of potential inconsistencies arising from different phases of natural and internal variability. For each period, the statistical metrics were normalized to a scale from 0 to 1 (with 1 indicating perfect performance). The original statistical metrics prior to normalization are provided in the Appendices (Fig. A.10-A.13, Table A.14-A.17). Models were ranked based on their ability to simulate SAT; if multiple metrics were used, their mean value was employed for the final ranking. Following ranking, the top 25% (in this case, six models) were selected as the optimal model ensemble for each period (Aghakhani Afshar et al. 2017; Ruan et al. 2019; Gnatiuk et al. 2020). Additionally, three quality groups (QGs) of models were distinguished based on their rankings: the first 25% of the models were classified as very good (QG I), the last 25% as unsatisfactory (QG III), and the remaining 50% as satisfactory (QG II).

The consistency of the model evaluation methods was analyzed using mean absolute differences of normalized statistical metrics and mean absolute differences of ranks between two time periods of all models. The consistency of a method is considered better when the specified absolute differences are closer to zero. Furthermore, the study examined whether a model belonged to the same quality group in both periods. If this was the case, the model’s ranking was defined as consistent. Finally, we summarized the percentage of models that were consistent according to each method. These values were then used to compare the evaluation methods.

RESULTS

Model evaluation according to each considered method

i) Method of model comparison by root mean square error (RMSE)

The results of the normalized values of RMSE along with the assigned model ranks and quality groups for two periods are presented in Fig. 1. Based on the ranking results, the following models were selected for the sub-ensemble (the best models are highlighted in green and belong to the first quality group, QG I):

. for the period 1951-1980: ACCESS-ESM1-5, AWI-CM-1-1-MR, CESM2-WACCM, GFDL-ESM4, MPI-ESM1-2-LR and NorESM2-LM;

. for the period 1981-2010: ACCESS-ESM1-5, CESM2-WACCM, EC-Earth3-Veg, FIO-ESM-2-0, MPI-ESM1-2-HR, and MPI-ESM1-2-LR.

In Fig. 1, the three far right columns display the evaluation of the method’s consistency. The intermodel mean difference in normalized RMSE between 1981-2010 and 1951-1980 is 0.06; the mean difference in model ranks is 3.0; and the results of the consistency assessment based on the model classified into the same quality group in both periods are 60%. Additionally, during both periods, 3 models out of 25 were classified as QG I, 8 as QG II, and 4 as QG III.

It should be noted that in this case, RMSE normalization involved converting values so that 1 indicates perfect performance and 0 indicates poor performance, allowing for comparison across all methods using the mean of the statistic. For original RMSE values, see Fig. A.10.

A similar ranking of models and assignment of each model to a quality group was performed for the other six model evaluation methods. Figures analogous to Fig. 1 for these additional evaluation methods are presented in Figs. A.4-A.9. The results of the comparison of the model evaluation methods are summarized in Table 1.

Intercomparison of the model evaluation methods

The summarized results of the consistency assessment for all methods are presented in Table 1 and Fig. 2. The first column in Table 1 indicates the number of models that were included in the selected sub-ensemble across both periods. The value in parentheses represents the percentage of the possible six sub-ensemble models, corresponding to the top 25% of the full ensemble. For example, for method iii, no models were included in the selected sub-ensemble in either period. The highest consistency was observed for methods i and vii, where three and four models, respectively, appeared in the selected sub-ensembles across both periods. The difference in mean normalized statistical metrics between the two periods (where 0 indicates perfect agreement) ranges from 0.06 (method i) to 0.36 (method iii). The most effective methods are (i) RMSE, (vi) Taylor diagram, bias and trend, and (vii) the percentile-based method. The difference in rank values between the two periods (where 0 indicates perfect agreement) varies from 3.0 (method i) to 8.9 (method ii). The best-performing methods are (i) RMSE, (vii) the percentile-based approach, and (iv) the S_score method. The last column in Table 1 and Fig. 2 (left) shows the percentage of models classified into the same quality group across both periods. The lowest consistency based on quality groups was observed for method iii at 40%, while the highest was for method vii at 72%. Thus, among the seven evaluated methods, the most effective model evaluation techniques are (vii) the percentile-based method and (i) RMSE.

Fig. 3 illustrates how many times each model was identified as the best model and included in the sub-ensemble (top 25%), as well as how many times it was defined as unsatisfactory (worst 25%) across 14 ranking cases (7 methods × 2 periods). The most frequently selected models for the sub-ensemble are ACCESS-ESM1-5, AWI-CM-1-1-MR, EC-Earth3-Veg, GFDL-ESM4, and MPI-ESM1-2-LR. Conversely, the GCMs most commonly identified as unsatisfactory are CAMS-CSM1-0, CIESM, FGOALS-g3, INM-CM4-8, and NESM3. Climate models such as AWI-CM-1-1-MR, BCC-CSM2-MR, INM-CM5-0, and MPI-ESM1-2-LR were never classified among the worst 25%. Similarly, models like CAMS-CSM1-0 and MIROC6 were never ranked in the top 25%.

Interannual variability of SAT over the Arctic for the periods 1951-1980 (left) and 1981-2010 (right), for observations, the full ensemble, and sub-ensembles using the seven model assessment methods, are presented in Figs. 4 and 5. In general, all sub-ensembles exhibit an interannual variability distribution pattern similar to that of the observations but with some errors (larger or smaller). The averaging of the full ensemble significantly smooths the interannual temperature amplitude. Overall, the SAT of the full ensemble and sub-ensembles based on methods iii and iv underestimate observations during the period 1951-1980; sub-ensembles based on methods i, ii, v, and vii better reproduce the SAT. During the period 1981-2010, models selected based on methods iii and ii underestimate SAT compared to observations; however, methods i, v, vi, and vii better reproduce the interannual variability of SAT.

Boxplots of the annual SAT over the Arctic for the periods 1951-1980 and 1981-2010 are shown in Fig. 5. The dashed line indicates the mean value of observations. In the first period, only the mean values for the sub-ensemble selected by method vii are close to the mean of observations. For other methods, deviations from the observational mean generally range from 0.5 to 1 degree Celsius. In the second period, the shape of the distribution for all methods is closer to that of observations, except for methods ii and iii, which significantly underestimate them.

Regarding the annual cycle (Fig. 6), greater agreement is watched during the warm period and less during the cold period between the different sub-ensembles and observations. The exception is method iii, which underestimates SAT in all months. However, for some methods (e.g., iv and vii), the agreement is better in winter than in summer and autumn.

Fig. 7 shows the results of GCMs ranking by seven methods for the two periods. Such a comparison allows us to evaluate the consistency of the methods. We observe considerable disagreement in the results, with some methods ranking certain models as among the best, while others classify them as among the worst – for example, ACCESS-CM2, CIESM, EC-Earth3, EC-Earth3-Veg, and KACE-1-0-G. However, some models are consistently classified as either “good” or “bad” by most methods – for instance, AWI-CM-1-1-MR, CASM-CSM1-0, and MPI-ESM1-2-LR. From Fig. 7, we can clearly see the consistency of each method; for example, method (i) identifies the ACCESS-CM2 model as unsatisfactory in both periods – showing high consistency. Conversely, method (ii) classifies the same model as very good across both periods. However, when analyzing these two methods (i and ii) across all models, it becomes evident that the consistency of method (ii) is significantly lower and it more frequently assigns incorrect categories to models.

DISCUSSION AND CONCLUSIONS

Seven different model evaluation methods for the selection of a sub-ensemble were tested for CMIP6 SAT over the Arctic. All model evaluation methods were analyzed for two periods, 1951-1980 and 1981-2010, to assess their consistency. Specifically, differences in mean values, model rankings, and the matching of assigned quality groups were examined. The ability of a model evaluation method to identify the climate model as superior, regardless of the time (e.g., warming or cooling), confirms its robustness. For each evaluation method, a sub-ensemble comprising the top 25% of models was selected based on ranking, which was 6 out of 25 GCMs.

The intercomparison results indicate superior performance for the methods (i) root mean square error and (vii) the percentile-based approach. The models selected for the sub-ensemble under the (i) root mean square error method for both periods are ACCESS-ESM1-5, CESM2-WACCM, and MPI-ESM1-2-LR. Similarly, under the (vii) the percentile-based approach, the models selected for both periods are ACCESS-ESM1-5, AWI-CM-1-1-MR, GFDL-ESM4, and MPI-ESM1-2-LR. The most frequently included models - appearing more than 4 times out of 14 - under the tested evaluation methods across both periods (1951-1980 and 1981-2010) are ACCESS-ESM1-5, AWI-CM-1-1-MR, EC-Earth3-Veg, GFDL-ESM4, and MPI-ESM1-2-LR. The GCMs most commonly identified as unsatisfactory include CAMS-CSM1-0, CIESM, FGOALS-g3, INM-CM4-8, and NESM3.

Considering that the simulated distribution of interannual variability is comparable to observations, albeit with some systematic errors (Fig. 2), we recommend applying bias correction to the CMIP6 temperature data.

In general, comparing the results obtained here with those from other studies is challenging due to the use of different sets of model input data (e.g., 22, 25, 30, 35 models, etc.), various study regions, and differing meteorological parameters. Furthermore, most studies employ only one method of model evaluation and selection without comparing it to alternative approaches. Given the wide range of such methods, comparing their effectiveness for evaluation, ranking, and selecting climate models remains an important task.

Among all the methods for estimating model skill examined in this article, we recommend using the following for temperature data: (i) root mean square error and (vii) a percentile-based method, as they produce the most consistent results. It should be noted that we did not assess the sensitivity of the choice of evaluation method to different variables; therefore, our findings should be applied with caution to other meteorological parameters. For example, methods based solely on RMSE may not be suitable for precipitation. When selecting climate models for other meteorological variables, more comprehensive approaches are likely to be more robust than those relying on a single statistical parameter.

APPENDICES

A.18 References to Table A2

Aghakhani Afshar A., Hasanzadeh Y., Besalatpour A.A., and Pourreza-Bilondi M. (2017). Climate change forecasting in a mountainous data scarce watershed using CMIP5 models under representative concentration pathways. Theoretical and Applied Climatology, 129, 683-699, DOI:10.1007/s00704-016-1908-5.

Ahmed K., Sachindra D., Shahid S., et al. (2020). Multi-model ensemble predictions of precipitation and temperature using machine learning algorithms. Atmospheric Research, 236, 104806, DOI:10.1016/j.atmosres.2019.104806.

Anandhi A. and Nanjundiah R.S. (2015). Performance evaluation of AR4 Climate Models in simulating daily precipitation over the Indian region using skill scores. Theoretical and Applied Climatology, 119, 551-566, DOI:10.1007/s00704-013-1043-5.

Anandhi A., Pierson D.C., and Frei A. (2019). Evaluation of Climate Model Performance for Water Supply Studies: Case Study for New York City. Journal of Water Resources Planning and Management, 145, 06019006, DOI:10.1061/(ASCE)WR.1943-5452.0001054.

Bannister D., Herzog M., Graf H.-F., et al. (2017). An Assessment of Recent and Future Temperature Change over the Sichuan Basin, China, Using CMIP5 Climate Models. Journal of Climate, 30, 6701-6722, DOI:10.1175/JCLI-D-16-0536.1.

Cai Z., You Q., Wu F., et al. (2021). Arctic Warming Revealed by Multiple CMIP6 Models: Evaluation of Historical Simulations and Quantification of Future Projection Uncertainties. Journal of Climate, 34, 4871-4892. DOI:10.1175/JCLI-D-20-0791.1.

Fu G., Liu Z., Charles S.P., et al. (2013). A score-based method for assessing the performance of GCMs: A case study of southeastern Australia. Journal of Geophysical Research-Atmospheres, 118, 4154-4167, DOI:10.1002/jgrd.50269.

Gnatiuk N., Radchenko I., Davy R., et al. (2020). Simulation of factors affecting Emiliania huxleyi blooms in Arctic and sub-Arctic seas by CMIP5 climate models: model validation and selection. Biogeosciences, 17, 1199-1212, DOI:10.5194/bg-17-1199-2020.

Herger N., Abramowitz G., Knutti R., et al. (2018). Selecting a climate model subset to optimise key ensemble properties. Earth System Dynamics, 9, 135-151, DOI:10.5194/esd-9-135-2018.

Inoue T. and Ueda H. (2011). Delay of the First Transition of Asian Summer Monsoon under Global Warming Condition. SOLA, 7, 81-84, DOI:10.2151/sola.2011-021.

Jia K., Ruan Y., Yang Y., and You Z. (2019). Assessment of CMIP5 GCM Simulation Performance for Temperature Projection in the Tibetan Plateau. Earth and Space Science, 6, 2362-2378, DOI:10.1029/2019EA000962.

Jiang Z., Li W., Xu J., and Li L. (2015). Extreme Precipitation Indices over China in CMIP5 Models. Part I: Model Evaluation. Journal of Climate, 28, 8603-8619, DOI:10.1175/JCLI-D-15-0099.1.

Kadel I., Yamazaki T., Iwasaki T., and Abdillah M. (2018). Projection of future monsoon precipitation over the central Himalayas by CMIP5 models under warming scenarios. Climate Research, 75, 1-21, DOI:10.3354/cr01497.

Kumar D., Mishra V., and Ganguly A.R. (2015). Evaluating wind extremes in CMIP5 climate models. Climate Dynamics, 45, 441-453, DOI:10.1007/s00382-014-2306-2.

Kumar S., Merwade V., Kinter J.L., and Niyogi D. (2013). Evaluation of Temperature and Precipitation Trends and Long-Term Persistence in CMIP5 Twentieth-Century Climate Simulations. Journal of Climate, 26, 4168-4185, DOI:10.1175/JCLI-D-12-00259.1.

Macadam I., Pitman A.J., Whetton P.H., and Abramowitz G. (2010). Ranking climate models by performance using actual values and anomalies: Implications for climate change impact assessments. Geophysical Research Letters, 37, 16704, DOI:10.1029/2010GL043877.

Maxino C.C., McAvaney B.J., Pitman A.J., and Perkins S.E. (2008). Ranking the AR4 climate models over the Murray-Darling Basin using simulated maximum temperature, minimum temperature and precipitation. International Journal of Climatology, 28, 1097-1112, DOI:10.1002/joc.1612.

McMahon T.A., Peel M.C., and Karoly D.J. (2015). Assessment of precipitation and temperature data from CMIP3 global climate models for hydrologic simulation. Hydrology and Earth System Sciences, 19, 361-377, DOI:10.5194/hess-19-361-2015.

Ogata T., Ueda H., Inoue T., et al. (2014). Projected Future Changes in the Asian Monsoon: A Comparison of CMIP3 and CMIP5 Model Results. Journal of the Meteorological Society of Japan, 92, 207-225, DOI:10.2151/jmsj.2014-302.

Ongoma V., Chen H., Gao C. (2019). Evaluation of CMIP5 twentieth century rainfall simulation over the equatorial East Africa. Theoretical and Applied Climatology, 135, 893-910, DOI:10.1007/s00704-018-2392-x.

Perkins S.E., Pitman A.J., Holbrook N.J., and McAneney J. (2007). Evaluation of the AR4 Climate Models’ Simulated Daily Maximum Temperature, Minimum Temperature, and Precipitation over Australia Using Probability Density Functions. Journal of Climate, 20, 4356-4376, DOI:10.1175/JCLI4253.1.

Rao X., Lu X., and Dong W. (2019). Evaluation and projection of extreme precipitation over northern China in CMIP5 models. Atmosphere, 10, 691, DOI:10.3390/atmos10110691.

Reifen C. and Toumi R. (2009). Climate projections: Past performance no guarantee of future skill? Geophysical Research Letters, 36, DOI:10.1029/2009GL038082.

Ruan Y., Liu Z., Wang R., and Yao Z. (2019). Assessing the Performance of CMIP5 GCMs for Projection of Future Temperature Change over the Lower Mekong Basin. Atmosphere, 10, 93, DOI:10.3390/atmos10020093.

Saha A., Ghosh S., Sahana A.S., and Rao E.P. (2014). Failure of CMIP5 climate models in simulating post-1950 decreasing trend of Indian monsoon. Geophysical Research Letters, 41, 7323-7330, DOI:10.1002/2014GL061573.

Sharmila S., Joseph S., Sahai A., et al. (2015). Future projection of Indian summer monsoon variability under climate change scenario: An assessment from CMIP5 climate models. Global and Planetary Change, 124, 62-78, DOI:10.1016/j.gloplacha.2014.11.004.

Sillmann J., Kharin V.V., Zwiers F.W., et al. (2013). Climate extremes indices in the CMIP5 multimodel ensemble: Part 2. Future climate projections. Journal of Geophysical Research-Atmospheres, 118, 2473-2493, DOI:10.1002/JGRD.50188.

Sun Q., Miao C., and Duan Q. (2015). Comparative analysis of CMIP3 and CMIP5 global climate models for simulating the daily mean, maximum, and minimum temperatures and daily precipitation over China. Journal of Geophysical Research-Atmospheres, 120, 4806-4824, DOI:10.1002/2014JD022994.

Walsh J.E., Chapman W.L., Romanovsky V., et al. (2008). Global Climate Model Performance over Alaska and Greenland. Journal of Climate, 21, 6156-6174, DOI:10.1175/2008JCLI2163.1.

Wang B., Liu D.L., Macadam I., et al. (2016). Multi-model ensemble projections of future extreme temperature change using a statistical downscaling method in south eastern Australia. Climatic Change, 138, 85-98, DOI:10.1007/s10584-016-1726-x.

Yang X., Yu X., Wang Y., et al. (2020). The Optimal Multimodel Ensemble of Bias-Corrected CMIP5 Climate Models over China. Journal of Hydrometeorology, 21, 845-863, DOI:10.1175/JHM-D-19-0141.1.

You Q., Jiang Z., Wang D., et al. (2018). Simulation of temperature extremes in the Tibetan Plateau from CMIP5 models and comparison with gridded observations. Climate Dynamics, 51, 355-369, DOI:10.1007/s00382-017-3928-y.

Zhou B., Wen Q.H., Xu Y., et al (2014). Projected Changes in Temperature and Precipitation Extremes in China by the CMIP5 Multimodel Ensembles. Journal of Climate, 27, 6591-6611, DOI:10.1175/JCLI-D-13-00761.1.