Study reference

(first author, publication year)

Appropriate and clearly focused question? ¹

Yes/no

Comprehensive and systematic literature search?²

Yes/partial yes/ no

Description of included and excluded studies?³

Yes/partial yes/ no

Description of relevant characteristics of included studies?⁴

Yes/partial yes/no

Assessment of scientific quality of included studies?⁵

Yes/partial yes/no

Enough similarities between studies to make combining them reasonable?⁶

Yes/partial yes/ no

Potential risk of publication bias taken into account?⁷

Yes/partial yes/ no

Potential conflicts of interest reported?⁸

Yes/partial yes/ no

Zakhem, 2023

Yes

Partial yes

Partial

Partial yes

Yes

No

No

Yes

DOMAIN 1: Participants

A. Risk of Bias

Describe the sources of data and criteria for participant selection:
Data from all patients with a histopathologically confirmed primary cSCC with diagnosis and excision between January 1, 2013, and December 31, 2015, were included from the National Disease Registration Service, England. Patient inclusion and metastasis selection procedures have been described previously. Shortly, patients with metastatic cSCC (cases) were identified using an extensive algorithm, and all identified reports were reviewed by Z.C.V. with the second opinion from B.R. in ambiguous cases. Thereafter, controls were randomly selected from patients with a cSCC in 2013 with no metastasis occurrence until the end of follow-up (ie, December 31, 2015).

Replication (the Netherlands).
The cases and controls were selected from a Dutch nationwide cohort of patients with a histopathologically confirmed first primary cSCC in 2007/ 2008, as registered by the Netherlands Cancer Registry, which has been linked to the nationwide network and registry of histopathology and cytopathology (PALGA) for the retrieval of subsequent and metastatic cSCCs up to December 31, 2018. The selection of metastatic cSCCs has been described before. Shortly, metastases were identified using an algorithm based on the pathology reports. Thereafter, all selected reports from potential cases were reviewed manually, and nonmetastatic controls were selected from the remaining patients.

Dev

Val

1.1 Were appropriate data sources used, e.g. cohort, RCT or nested case-control study data?

Yes

Yes

1.2 Were all inclusions and exclusions of participants appropriate?

Yes

Yes

Risk of bias introduced by selection of participants

RISK:

(low/ high/ unclear)

Low

Low

Rationale of bias rating: appropriate data sources and inclusion of patients.

B. Applicability

Describe included participants, setting and dates: patients form England and the Netherlands were included (n=1774, n=434, respectively)

Concern that the included participants and setting do not match the review question

CONCERN:

(low/ high/ unclear)

Low

Low

Rationale of applicability rating: patients match the research question.

DOMAIN 2: Predictors

A. Risk of Bias

List and describe predictors included in the final model, e.g. definition and timing of assessment:
Diameter of the tumor, tumor thickness, poor differentiation, invasion in and beyond subcutaneous fat, perineural/lymphovascular invasion, male sex, and facial localization were determined to be the important risk factors for metastasis.

Dev

Val

2.1 Were predictors defined and assessed in a similar way for all participants?

Yes

Yes

2.2 Were predictor assessments made without knowledge of outcome data?

No

No

2.3 Are all predictors available at the time the model is intended to be used?

Yes

Yes

Risk of bias introduced by predictors or their assessment

RISK:

(low/ high/ unclear)

Unclear

Unclear

Rationale of bias rating:

B. Applicability

Concern that the definition, assessment or timing of predictors in the model do not match the review question

CONCERN:

(low/ high/ unclear)

Low

Low

Rationale of applicability rating:

DOMAIN 3: Outcome

A. Risk of Bias

Describe the outcome, how it was defined and determined, and the time interval between predictor assessment and outcome determination: primary outcome was metastasis, interval: 10-11 years.

Dev

Val

3.1 Was the outcome determined appropriately?

No

No

3.2 Was a pre-specified or standard outcome definition used?

Yes

Yes

3.3 Were predictors excluded from the outcome definition?

Yes

Yes

3.4 Was the outcome defined and determined in a similar way for all participants?

Yes

Yes

3.5 Was the outcome determined without knowledge of predictor information?

No

No

3.6 Was the time interval between predictor assessment and outcome determination appropriate?

Yes

Yes

Risk of bias introduced by the outcome or its determination:

RISK:

(low/ high/ unclear)

Low-high

Low-high

Rationale of bias rating: 3.1: Pathology records could not be reassessed.

B. Applicability

At what time point was the outcome determined: Primary outcome was metastasis, interval: 10-11 years.

Concern that the outcome, its definition, timing or

determination do not match the review question

CONCERN:

(low/ high/ unclear)

Low

Low

Rationale of applicability rating: We do not expect the potential risk of bias to interfere.

DOMAIN 4: Analysis

Risk of Bias

Describe numbers of participants, number of candidate predictors, outcome events and events per candidate predictor:
In total, 887 metastatic cases and 887 nonmetastatic controls (n = 1774) were included from the

English data set for the main analyses, and 217 cases and 217 controls (n = 434) were included from the Dutch data set for the replication analyses.

See ‘Table 1 Descriptive characteristics of the English (n = 1774) and Dutch (n = 434) data sets, stratified by metastasis outcome’ from the original article.

Describe how the model was developed (for example in regards to modelling technique (e.g. survival or logistic modelling), predictor selection, and risk group definition):

Conditional logistic regression analyses with backward stepwise selection identified the set of statistically significant metastasis risk factors in the English data set. A 2-sided statistical significance level of P = .10 was used in the backward stepwise selection to reduce optimism and selection bias. Variance inflation factors were calculated, with no evidence for multicollinearity. Missing values for covariates were imputed 20 times using multivariate imputation by chained equations. The imputation model included all covariates, the outcome, and, for the English data set, ethnicity and deprivation as auxiliary variables. For the continuous variables age, the number of previous cSCCs, diameter, and thickness, restricted cubic splines with 3 knots were used to evaluate a possible nonlinear relationship with the metastasis outcome. To facilitate interpretation, nonlinear variables were categorized into clinically relevant categories on the basis of 2 criteria: (1) increase of at least 2 odds ratio (OR) points per category per variable and (2) as little as possible overlap between the confidence intervals (CIs) of the categories within a variable. For comparison purposes, we categorized diameter and thickness following the American Joint Committee on Cancer eighth edition (AJCC8)18 criteria and, for diameter, using the Brigham and Women’s Hospital (BWH)19 staging system. Since the BWH classification does not specify any criteria for thickness, this was included as a continuous variable. The discriminative ability of the final set of risk factors was assessed by Harrell’s concordance index (c-index) in the main analyses. The final risk factors were replicated in the Dutch data set to investigate whether similar ORs would be found. The c-indices of the models could not be compared because no absolute risk model was available. Our model fit was compared with that of the AJCC8 and BWH using Nagelkerke’s pseudo R2 measure, which explains the improvement in model likelihood over a null model and can be used to compare different models using the same data set.20 This measure ranges from 0 to 1, and the higher the value, the better the model predicts the outcome. Ethical approval and informed consent were not required for analyzing data from the National Disease Registration Service following section 251 of the National Health Service act 2006.21 Approval was obtained from the scientific committees of the Netherlands Cancer Registry, PALGA, Dutch Transplant Foundation, and a waiver of informed consent was granted by the Erasmus Medical Center (MEC-2020-0147). Statistical analyses were performed using SPSS 25.0 statistical software (SPSS Inc) and R statistical software version 3.4.1

Describe whether and how the model was validated, either internally (e.g. bootstrapping, cross validation, random split sample) or externally (e.g. temporal validation, geographical validation, different setting, different type of participants):

Replication (the Netherlands). The cases and controls were selected from a Dutch nationwide cohort of patients with a histopathologically confirmed first primary cSCC in 2007/ 2008, as registered by the Netherlands Cancer Registry,13 which has been linked to the nationwide network and registry of histopathology and cytopathology (PALGA) for the retrieval of subsequent and metastatic cSCCs up to December 31, 2018.14 The selection of metastatic cSCCs has been described before.10 Shortly, metastases were identified using an algorithm based on the pathology reports. Thereafter, all selected reports from potential cases were reviewed manually, and nonmetastatic controls were selected from the remaining patients.

Describe the performance measures of the model, e.g. (re)calibration, discrimination, (re)classification, net benefit, and whether they were adjusted for optimism:

Using 2 large nationwide data sets with a total of 1104 metastatic cSCCs, we identified patient- and tumor-based risk factors with a c-index of 0.96 in the development data set. Comparison of our final set of risk factors with the AJCC8 and BWH showed higher pseudo R2 measures in both data sets. Following tumor diameter and thickness, poor differentiation proved an important risk factor for metastasis, despite being omitted from the AJCC8, thereby emphasizing the importance of reviewing and refining current staging systems.

A 2-sided statistical significance level of P = .10 was used in the backward stepwise selection to reduce optimism and selection bias.

Describe any participants who were excluded from the analysis:

Metastases from potential sources other than skin or unknown origin were excluded.

Describe missing data on predictors and outcomes as well as methods used for missing data:

Missing values for covariates were imputed 20 times using multivariate imputation by chained equations.

Dev

Val

4.1 Were there a reasonable number of participants with the outcome?

Yes 

Yes

4.2 Were continuous and categorical predictors handled appropriately?

Yes 

Yes

4.3 Were all enrolled participants included in the analysis?

Yes

Yes

4.4 Were participants with missing data handled appropriately?

Yes

Yes

4.5 Was selection of predictors based on univariable analysis avoided?

Yes

4.6 Were complexities in the data (e.g. censoring, competing risks, sampling of controls) accounted for appropriately?

Yes

Yes

4.7 Were relevant model performance measures evaluated appropriately?

Yes

Yes

4.8 Were model overfitting and optimism in model performance accounted for?

Yes

4.9 Do predictors and their assigned weights in the final model correspond to the results from multivariable analysis?

Yes

Risk of bias introduced by the analysis

RISK:

(low/ high/ unclear)

Low

Low

DOMAIN 1: Participants

A. Risk of Bias

Describe the sources of data and criteria for participant selection:
From the Netherlands Cancer Registry (NCR), we selected all patients with a histopathologically confirmed first primary cSCC in 2007 or 2008 (12,325 patients). All patients were linked to the nationwide network and registry of histo- and cytopathology (PALGA)17 for retrieval of subsequent and metastatic cSCCs up to 26 August 2020, an update of the cohort described in Tokez et al. 2022.

The model was validated in a subset of a previously described cohort,8,20 consisting of a NCC study derived from patients with a primary cSCC registered in the National Disease Registration Service in England, between January 1st 2013 and December 31st 2015. Matching was performed based on follow up time: each case with a metastatic event at a given time t was matched to a control with a follow-up time greater or equal to t. To guarantee a reasonable follow-up time, only patients with a primary cSCC in 2013 were considered for the NCC cohort.

Dev

Val

1.1 Were appropriate data sources used, e.g. cohort, RCT or nested case-control study data?

Yes

Yes

1.2 Were all inclusions and exclusions of participants appropriate?

Yes

Yes

Risk of bias introduced by selection of participants

RISK:

(low/ high/ unclear)

Low

Low

Rationale of bias rating: appropriate data sources and inclusion of patients.

B. Applicability

Describe included participants, setting and dates: The model was developed in a NCC cohort with 390 patients diagnosed with a first cSCC in 2007/2008 in the Netherlands (195 metastatic cases and 195 nonmetastatic controls), and externally validated in a NCC cohort of 696 patients diagnosed with a primary cSCC in 2013 in England (348 cases and 348 controls). In the cohort from the Netherlands, the cumulative incidence of metastasis at 5 years is 1.6% (1.4–1.8). In the cohort from England, the cumulative incidence of metastasis at 3 years is 1.5% (1.4–1.7). Controls included in the NCC cohorts are similar to the full nationwide cohorts (Table 1). The median follow-up time in the development cohort (9 years) is longer than in the validation cohort (2.4 years). In general, a higher proportion of cases had high risk clinico-pathological characteristics, but this difference was not reflected in current staging systems.

Concern that the included participants and setting do not match the review question

CONCERN:

(low/ high/ unclear)

Low

Low

Rationale of applicability rating: patients match the research question.

DOMAIN 2: Predictors

A. Risk of Bias

List and describe predictors included in the final model, e.g. definition and timing of assessment:
Age, sex, number of prior cSCC, tumour location, tumour diameter, tissue involvement, differentiation, perineural of lymphovascular invasion. 

Dev

Val

2.1 Were predictors defined and assessed in a similar way for all participants?

Yes

Yes

2.2 Were predictor assessments made without knowledge of outcome data?

No

No

2.3 Are all predictors available at the time the model is intended to be used?

Yes

Yes

Risk of bias introduced by predictors or their assessment

RISK:

(low/ high/ unclear)

Unclear

Unclear

Rationale of bias rating:

B. Applicability

Concern that the definition, assessment or timing of predictors in the model do not match the review question

CONCERN:

(low/ high/ unclear)

Low

Low

Rationale of applicability rating:

DOMAIN 3: Outcome

A. Risk of Bias

Describe the outcome, how it was defined and determined, and the time interval between predictor assessment and outcome determination: primary outcome was metastasis, interval: 5 years.

Dev

Val

3.1 Was the outcome determined appropriately?

No

No

3.2 Was a pre-specified or standard outcome definition used?

Yes

Yes

3.3 Were predictors excluded from the outcome definition?

Yes

Yes

3.4 Was the outcome defined and determined in a similar way for all participants?

Yes

Yes

3.5 Was the outcome determined without knowledge of predictor information?

No

No

3.6 Was the time interval between predictor assessment and outcome determination appropriate?

Yes

Yes

Risk of bias introduced by the outcome or its determination:

RISK:

(low/ high/ unclear)

Low-high

Low-high

Rationale of bias rating: 3.1: Pathology records could not be reassessed.

B. Applicability

At what time point was the outcome determined: Primary outcome was metastasis within 5 years

Concern that the outcome, its definition, timing or

determination do not match the review question

CONCERN:

(low/ high/ unclear)

Low

Low

Rationale of applicability rating: We do not expect the potential risk of bias to interfere.

DOMAIN 4: Analysis

Risk of Bias

Describe included participants, setting and dates: The model was developed in a NCC cohort with 390 patients diagnosed with a first cSCC in 2007/2008 in the Netherlands (195 metastatic cases and 195 nonmetastatic controls), and externally validated in a NCC cohort of 696 patients diagnosed with a primary cSCC in 2013 in England (348 cases and 348 controls). In the cohort from the Netherlands, the cumulative incidence of metastasis at 5 years is 1.6% (1.4–1.8). In the cohort from England, the cumulative incidence of metastasis at 3 years is 1.5% (1.4–1.7). Controls included in the NCC cohorts are similar to the full nationwide cohorts (Table 1). The median follow-up time in the development cohort (9 years) is longer than in the validation cohort (2.4 years). In general, a higher proportion of cases had high risk clinico-pathological characteristics, but this difference was not reflected in current staging systems.

Describe whether and how the model was validated, either internally (e.g. bootstrapping, cross validation, random split sample) or externally (e.g. temporal validation, geographical validation, different setting, different type of participants):

The model was validated in a subset of a previously described cohort,8,20 consisting of a NCC study derived from patients with a primary cSCC registered in the National Disease Registration Service in England, between January 1st 2013 and December 31st 2015. Matching was performed based on follow up time: each case with a metastatic event at a given time t was matched to a control with a follow-up time greater or equal to t. To guarantee a reasonable follow-up time, only patients with a primary cSCC in 2013 were considered for the NCC cohort.

Describe the performance measures of the model, e.g. (re)calibration, discrimination, (re)classification, net benefit, and whether they were adjusted for optimism:

In the validation cohort, the C-index of our absolute risk model was 0.84 (95% CI 0.81–0.87) (Table 4), which is higher than that of the BWH and the AJCC8 (0.82 (95% CI 0.79–0.85, p-value = 0.09) and 0.79 (95% CI 0.75–0.82, p-valuee = 0.09) and 0.79 (95% CI 0.75–0.82, p-value <0.01), respectively). The model was well-calibrated: the 3-year calibration slope was 0.96 (95% CI 0.76–1.16). The observed to expected ratio (O/E ratio) of 0.82 (95% CI 0.58–1.06) indicates that the model slightly overestimated metastatic events in this cohort, particularly for higher ranges of metastatic risk (Fig. 1). However, BWH staging is less well calibrated, with an O/E ratio of 0.56 (95% CI 0.40–0.71). Similar conclusions hold for performance metrics at 1 year (Supplementary Table S4) and for case–control sets without any missing data (Supplementary Table S5). The clinical utility of the model was evaluated using decision curve analysis and was compared to current staging systems (Fig. 2).

Describe any participants who were excluded from the analysis:

Metastases from potential sources other than skin or unknown origin were excluded.

Describe missing data on predictors and outcomes as well as methods used for missing data:

Missing values were assumed to be missing at random, and were imputed using multiple imputation using chained equations, using the mice R package (v3.13.0).

Dev

Val

4.1 Were there a reasonable number of participants with the outcome?

Yes 

Yes

4.2 Were continuous and categorical predictors handled appropriately?

Yes 

Yes

4.3 Were all enrolled participants included in the analysis?

Yes

Yes

4.4 Were participants with missing data handled appropriately?

Yes

Yes

4.5 Was selection of predictors based on univariable analysis avoided?

Yes

4.6 Were complexities in the data (e.g. censoring, competing risks, sampling of controls) accounted for appropriately?

Yes

Yes

4.7 Were relevant model performance measures evaluated appropriately?

Yes

Yes

4.8 Were model overfitting and optimism in model performance accounted for?

Yes

4.9 Do predictors and their assigned weights in the final model correspond to the results from multivariable analysis?

Yes

Risk of bias introduced by the analysis

RISK:

(low/ high/ unclear)

Low

Low

Rationale of bias rating: