Evaluation of the IRR rates for assignment of the HCT-CI scores
| Study phases . | Evaluators . | Patients, n . | Agreement on individual scores, % . | Kw* (standard error) . |
|---|---|---|---|---|
| Initial assessment phase | Principal investigator vs evaluator 1 | 80 | 38 | 0.585 (0.114) |
| Principal investigator vs multiple evaluators | 80 | 37 | 0.552 (0.065) | |
| Evaluator 1 vs multiple evaluators | 80 | 38 | 0.433 (0.076) | |
| Initial validation phase | Principal investigator vs evaluator 1 | 90 | 78 | 0.910 (0.03) |
| Principal investigator vs evaluator 2 | 90 | 78 | 0.890 (0.03) | |
| Principal investigator vs evaluator 3 | 90 | 70 | 0.910 (0.03) | |
| Final validation phase | Principal investigator vs evaluator 4 | 30 | 83 | 0.970 (0.03) |
| Evaluator 1 vs evaluator 4 | 30 | 77 | 0.920 (0.05) | |
| Evaluator 2 vs evaluator 4 | 30 | 77 | 0.910 (0.05) | |
| Evaluator 3 vs evaluator 4 | 30 | 79 | 0.930 (0.05) |
| Study phases . | Evaluators . | Patients, n . | Agreement on individual scores, % . | Kw* (standard error) . |
|---|---|---|---|---|
| Initial assessment phase | Principal investigator vs evaluator 1 | 80 | 38 | 0.585 (0.114) |
| Principal investigator vs multiple evaluators | 80 | 37 | 0.552 (0.065) | |
| Evaluator 1 vs multiple evaluators | 80 | 38 | 0.433 (0.076) | |
| Initial validation phase | Principal investigator vs evaluator 1 | 90 | 78 | 0.910 (0.03) |
| Principal investigator vs evaluator 2 | 90 | 78 | 0.890 (0.03) | |
| Principal investigator vs evaluator 3 | 90 | 70 | 0.910 (0.03) | |
| Final validation phase | Principal investigator vs evaluator 4 | 30 | 83 | 0.970 (0.03) |
| Evaluator 1 vs evaluator 4 | 30 | 77 | 0.920 (0.05) | |
| Evaluator 2 vs evaluator 4 | 30 | 77 | 0.910 (0.05) | |
| Evaluator 3 vs evaluator 4 | 30 | 79 | 0.930 (0.05) |
Weighted κ statistic (Kw) was calculated for agreement on HCT-CI risk groups of 0 to 1, 2, 3, and 4 or more.