Adjudicating Bleeding Outcomes in a Large Thromboprophylaxis Trial in Critical Illness.

Adjudication in clinical trials can confirm or refute eligibility, describe cointerventions, judge appropriateness of care, or assess the severity of morbidity outcomes.

To refine the adjudication process, calibrate 4 adjudicators, and measure agreement on bleeding severity in an international trial of heparin thromboprophylaxis (PROTECT).

Independently and blinded to each others' ratings and study drug, 4 adjudicators used web-based methods to examine 40 charts of patients assessed by local research coordinators to have either major (20 patients) or minor (20 patients) bleeding. We discussed reasons for disagreement after the first 20 charts to remediate and recalibrate. We calculated crude agreement, chance-corrected agreement (kappa), and chance-independent agreement (phi).

For 45 events in 40 patients, pair-wise crude agreement ranged from 86.7-93.3% (average 82.2%). Overall kappa was 0.81. Phi (which can only analyze pair-wise values) ranged from 0.75-0.87. We resolved all disagreements. During adjudication discussions, we 1) addressed methodological issues (e.g., handling recurrent bleeds), 2) added a category (e.g., no bleed), 3) expanded a category (e.g., a major bleed did not have to be overt if other criteria were fulfilled), and 4) divorced procedure-grounded definitions (such as the need for transfusion or therapeutic interventions) from bleeding severity criteria (e.g., the patient could still be classified as having no bleed or a minor bleed if 2 units of PRBCs were transfused for anemia).

After independent quadruplicate review of 45 bleeding events, we documented satisfactory agreement for bleeding outcomes, refined the adjudication process, and calibrated adjudicators for the remainder of the trial. Henceforth, charts will be randomly allocated to pairs of adjudicators for blinded review.

No relevant conflicts of interest to declare.