Decision theoretical foundations of clinical practice guidelines: an extension of the ASH thrombophilia guidelines

Current evidence–based clinical practice guidelines suffer from several deficiencies¹ including the following: “black-box” operation, a process with defined inputs and outputs but without complete understanding of its internal workings²; and what is referred to as “the integration problem,” which entails a lack of a framework for explicit integration of patient preferences and trade-offs between treatment benefits and harms.² We have previously argued that the solution of the “black-box” and the “integration” problems is only possible within a decision-analytical framework.^1,3-7 Importantly, such a framework enables not only the logical and transparent integration of patient’s values and preferences (V&P) and trade-offs between treatment benefits and harms^8,9 but protects against violation of the principles of rational decision-making.^3,4 

Recently, the influential American Society of Hematology (ASH) thrombophilia guidelines¹⁰ has made a significant step forward in demonstrating how decision modeling integrated within Grading of Recommendations Assessment, Development, and Evaluation (GRADE)¹¹ methodology can advance the field of guideline development. The panel developed transparent and easily understandable models that are important to end users.¹⁰ However, perhaps in a desire to simplify its presentation, the panel may have chosen a less-than-optimal decision model, leading to what we refer as the “omitted choice bias”¹² that we illustrate below.

Using the state-of-the-art guidelines methodology GRADE,¹¹ the ASH panel developed 23 recommendations (R1-R23) for testing for thrombophilia in various circumstances leading to “provoked” vs “unprovoked” venous thromboembolism (VTE).^10,Figure 1A presents the conceptual thrombophilia model used by the ASH panel.¹⁰ The model represents a truncated version of a decision tree, which we converted into the full decision-analytical model shown in Figure 1B. The panel considered the population of patients who were treated for initial VTE episodes for 3 to 6 months,¹³ after which it compared VTE recurrence and major bleeding rates of the following management strategies: perform thrombophilia testing and administer a long-term, indefinite anticoagulation only to those patients who tested positive vs do not conduct testing for thrombophilia (Figure 1A) and discontinue treatment for all (as in the case of “provoked” VTE) or continue treatment indefinitely for all (as in the case of "unprovoked" VTE).

To calculate VTE recurrence and bleeding rates with each strategy, the ASH panel estimated thrombophilia prevalence and the risk ratio (RR) for recurrent VTE in patients with thrombophilia vs patients without thrombophilia (RR_t).¹³ The panel relied on the ASH guidelines for the management of VTE¹³ to estimate the effects (ie, RR_rx and major bleeding RR [RR_bleed]) of anticoagulant treatment compared with stopping anticoagulant therapy after completion of primary treatment for the initial VTE. For most recommendations, the ASH thrombophilia panel used the following input parameters: the median prevalence (P) of any thrombophilia was 38.0% (minimum 21.6%; maximum 59.5%); RR_t of 1.65 (95% confidence interval [CI], 1.28-2.47); RR_rx of recurrent VTE of 0.15 (95% CI, 0.10-0.23; relative risk reduction = 1–RR_rx); RR_bleed of major bleeding on indefinite anticoagulant treatment was estimated at 2.17 (95% CI, 1.40-3.35), with the baseline major bleeding rate of 5 per 1000 patients (0.5%) at low risk and 15 per 1000 patients (1.5%) at high risk of bleeding per year.

To illustrate our approach, we focused on the first 6 thrombophilia panel recommendations in patients with low vs high risk of major bleeding (12 total recommendations). To address the superiority of a given strategy, the panel needed to estimate the overall risk of VTE recurrence without treatment (here denoted as p; not to be confused with P, the prevalence of thrombophilia shown in Figure 1A) after unprovoked VTE (R1), provoked after surgery (R2), provoked after a nonsurgical major transient risk factor, pregnancy, or associated with the use of oral contraceptives (R3-R5) or not specified as provoked or unprovoked VTE (R6). The ASH panel estimated the overall risk (probability) of VTE recurrence without treatment (p) to range from an average of 100 cases per 1000 patients (10%; scenario R1); 10 cases per 1000 patients (1%; R2); 50 cases per 1000 patients (5%; R3-R5); and 75 cases per 1000 patients (7.5%; R6).

After 3 to 6 months of index VTE, the panel modeled a comparison of thrombophilia testing (A) vs treat none (discontinuing anticoagulants; B) and testing (A) vs treat all (recommend indefinite anticoagulation to all patients; C). The panel considered 2 models: test (A) vs treat none (B) and test (A) vs treat all (C), separately instead of considering all relevant options simultaneously (A vs B vs C) in a single model. However, considering the comparisons separately instead of simultaneously can lead to what can be referred as the “omitted choice bias,” named after the well-known “omitted variable bias,”¹² when not considering a relevant option can skew the results and lead to incorrect conclusions. A reader must not confuse omission choice bias we are referring to, with tendency of decision makers to prefer nonaction (omission) to action, which is also sometimes referred as “omission choice bias.”

Although the authors presented an explicit model structure (Figure 1A), they did not leverage the entire apparatus of decision analysis^3,14 to generate recommendations. Instead, the panel relied on intuitive judgment, presumably informed by their model, to determine the optimal management approach for each clinical situation. Specifically, the panel stated the following¹⁰: “The following thresholds were used to judge the reduction in VTE (first time or recurrence): trivial, ≤5 events per 1000 patient-years; small, 5 to 20 per 1000 patient-years; and moderate, 20 to 50 per 1000 patient-years.” The management strategy resulting in VTE recurrence below these thresholds was considered superior and thus recommended. However, how the panel weighed the trade-offs between VTE and major bleeding rates is unclear.

We converted 2 ASH models into a coherent single decision tree (Figure 1B). Many theoretical frameworks exist for solving a decision tree,³ but most decision analyses use expected utility theory (EUT),³ as in this study. EUT is the only theory of choice that satisfies all mathematical axioms of rational decision-making, ensuring that the choices are consistent with the deciders’ V&P and trade-offs between treatment benefits and harms.³ When choosing between different options, the most rational choice is that with the highest expected utility, regardless of statistical significance.¹⁵ 

From a decision-making perspective, the task is to determine the probability of VTE recurrence (threshold, P_t) above which we should commit to treatment. By “treatment,” we refer to a commitment to a course of action that may include management consisting of treatment or diagnostic testing. We are indifferent between acting in favor of 1 management strategy over another when the net benefits and harms and decision-makers’ V&P between these 2 strategies are identical.^3,6,16-18 Importantly, P_t depends only on benefits and harms (and decision-makers’ V&P).^3,14,18 When considering only strategies for administering or stopping treatment, the threshold serves in the following way: if the probability of VTE recurrence is greater than P_t, we should give treatment (ie, anticoagulants in our case); if the probability of VTE recurrence is less than P_t, we should not give treatment.^3,14,18 When considering 3 possible strategies, continuing the treatment, administering the test and acting according to the results of the test, or stopping the treatment, we have 2 additional thresholds: the testing threshold P_tt and the treatment threshold P_rx. So, in total, we consider 5 possible management choices.

P_tt (test–no treatment threshold) refers to the pretest (prior) probability of VTE recurrence at which we are indifferent between no treatment and testing.^3,14,18 P_rx (test-treatment threshold) refers to the pretest (prior) probability of VTE recurrence at which we are indifferent between testing and treatment.^3,14,18 If the prior probability of VTE recurrence is less than P_tt, this guarantees that the posttest probability of VTE recurrence will always be less than treatment threshold, P_t, regardless of the test results.^3,14,18 Hence, we should not test and should withhold treatment under these circumstances. If the prior probability of VTE recurrence is greater than P_rx, this guarantees that the posttest probability of VTE recurrence will always be greater than treatment threshold, P_t, regardless of the test results.^3,14,18 If this relationship holds, we can give treatment without further testing.

Note how formal threshold equations 1 to 3 effectively capture everyday clinical intuition (Table 1).

Thus, in the case of thrombophilia recommendations, we contrast the overall risk (probability) of VTE recurrences (p) against these thresholds. According to our model, the thrombophilia testing should only be done for P_tt<p < P_rx. No testing/no treatment should be recommended for < P_tt. Treatment with anticoagulants should be recommended for p > P_rx.

The ASH panel presented its results as VTE and bleeding rates, counted separately. As explained below, such an approach introduces bias. Nevertheless, sometimes, we may wish to count events descriptively. If so, our model can be easily used to this effect.

Appendix 3 Table 1 in Appendix 3 illustrates using our model based on the data from the ASH report¹⁰ to reproduce the panel's calculations for R1 (an identical approach can be used to reproduce the calculations for other recommendations).

Figure 2A displays these results graphically, comparing all management strategies (for R1). Figure 2B-D show the results of VTE and major bleeding rates for the remaining recommendations (R2, R3-R5, and R6) in low-risk bleeding settings. Figure 3A-D shows the same results assuming high-risk bleeding. Notably, even though the ASH report referred to the threshold in issuing its recommendations,¹⁰ it is impossible to derive the VTE threshold from the method used by the ASH panel. The thresholds for the prevalence of VTE recurrence are 0, and the thresholds for the bleeding rates are undefined (see Appendix 2 for proof).

Table 2 displays calculations of the decision thresholds based on equations 1 to 3. These default calculations reflect the ASH thrombophilia evidence report and assume RV is 1, that is, that patients are equally concerned by the burden of VTE vs major bleeding. We explore this issue in detail below.

As explained above, to determine whether we should recommend thrombophilia testing, we contrast the probability of VTE recurrence (p) against the decision thresholds. The ASH panel estimated the probability of VTE recurrence (p) of 0.01 (1%) in the scenario guiding derivation of R2, 0.05 (5%) for R3 to R5, 0.075 (7.5%) to develop R6, and 0.1 (10%) in the clinical scenario resulting in R1. Figure 4 shows the results. Table 3 shows the ASH panel’s R1 to R6 thrombophilia recommendations compared with the recommendations according to our threshold model.

For R1, the probability of VTE recurrence (p) of 10% is greater than P_rx = of 0.85% (low bleeding risk) and 2.57% (high bleeding risk), which means that the patients should be offered long-term treatment with anticoagulation without thrombophilia testing. This agrees with the ASH R1 recommendation.

For R2, in patients at low bleeding risk, p of 1% is greater than P_rx = of 0.85%. However, in the high bleeding risk, p of 1% is lower than P_rx = of 2.57% and P_tt of 1.56%. Thus, no thrombophilia testing should be offered in the low-risk bleeding scenario, but extended anticoagulant treatment should be recommended (Figure 4A). In the high–bleeding risk scenario, p of 1% is less than P_tt = of 1.56%, and no treatment nor thrombophilia testing should be offered to these patients. Thus, our analysis agrees with the ASH R2 recommendation only in the case of high bleeding risk (Figure 4B).

For R3 to R5, p of 5% is greater than P_rx = of 0.85% (low bleeding risk) and P_rx = of 2.57% (high bleeding risk), meaning the patients should be offered long-term treatment with anticoagulation without thrombophilia testing. This result does not agree with the ASH R3 to R5 recommendations, which recommend thrombophilia testing and administering indefinite anticoagulation only to those patients who tested positive for thrombophilia (Table 3).

Finally, for R6, the probability of VTE recurrence (p) of 7.5% is greater than P_rx = of 0.85% (low bleeding risk) and P_rx = of 2.57% (high bleeding risk), indicating that the patients should be recommended long-term treatment with anticoagulation without thrombophilia testing. This conclusion also agrees with the ASH R6 recommendation (Table 3).

The analysis above assumed an RV of 1. When sensitivity analysis for RV was performed (ie, when RV ≠ 1), the results may differ. Table 4 shows the effect of RV on thrombophilia recommendations. For example, if we assume that the patient places twice as much importance on avoiding major bleeding than VTE recurrence (RV = 2), then recommendation 2 is consistent with the ASH panel’s R2 (“do not test for thrombophilia; discontinue anticoagulants”¹⁰; Table 3). Interestingly, in the high–bleeding risk scenarios, the same conclusion (“don’t test/don’t treat”) holds for RV ≥0.64, whereas testing and treatment strategy are recommended for the patients’ with RV <0.641 and <0.388, respectively; that is, when the patient prefers avoiding VTE 1.5 (=1/0.641) and 2.5 (=1/0.388) times over bleeding. The ASH panel reports that the patients generally prefer avoiding VTE recurrence over bleeding.¹⁰ In the low-risk bleeding scenario, we found that RV, in most cases, is unrealistically high (and never <1 for all recommendations) to affect our default recommendations for RV of 1. Nevertheless, it is conceivable that some patients fear the consequences of bleeding far more than they do of VTE. For instance, in case of R3 to R5, in low- and high-risk bleeding scenarios, patients might prefer to avoid major bleeding 5.8 to 9.6 times and 1.9 to 3.2 times more often than avoiding VTE, respectively. Under these conditions, our model aligns with the ASH recommendations (see “Discussion”).

In this study, we reanalyzed 6 × 2 recommendations made by the ASH thrombophilia panel¹⁰ for various “provoked” vs “unprovoked” VTE clinical scenarios used as an example to illustrate the need to improve the broader field of decision-making and guideline development. By incorporating decision modeling within GRADE methods, the ASH thrombophilia panel has taken a substantial stride in advancing the guidelines development methodology and tackling “black-box” operation and “integration” challenges.² Although we believe that the application of decision modeling is the only logical and transparent method to facilitate the integration of all relevant ingredients required to make recommendations,³ it is also essential to choose the correct model. The thrombophilia testing model developed by the ASH panel¹⁰ generated 4 of 12 recommendations that proved inaccurate when judged against using a fully developed threshold model (Table 3).³ Unlike the ASH panel, our model, assuming that patients placed equal value on avoiding VTE and bleeding, generated recommendations against thrombophilia testing in all scenarios considered.

The discrepancy occurred because of the omitted choice bias¹²: the panel considered 2 models separately instead of considering all relevant options simultaneously. This resulted in rationally incoherent recommendations when judged against an appropriate decision model. Nevertheless, because the certainty of evidence was judged to be very low, the panel issued conditional (weak) recommendations, meaning that “most individuals in this situation would want the suggested course of action, but many would not.”¹⁰ Thus, on the surface, the ASH panel’s judgment appears consistent with its recommendations, even if it disagreed with its model. Unfortunately, the panel never explained its deviation from the proposed model that apparently guided all the panel’s recommendations. Why engage in modeling or develop guidelines if very low certainty of evidence almost always (barring some exceptions¹⁹) generates uncertain recommendations that may or may not be coherent with the underlying model structure?

The ASH panel has not performed any sensitivity analyses to assess the uncertainty range at which their recommendations could possibly switch. We have argued that it is precisely in these circumstances that modeling followed by judicious deliberation of the panel is most useful because it combines the explicitness and transparency of decision modeling with the considered panel’s judgments.¹ 

Somewhat surprisingly, we also concluded that extended anticoagulation should be offered to patients with low bleeding risk who developed surgery-related VTE. This conclusion disagreed with most current guidelines recommending discontinuing oral anticoagulation after 3 months of surgery-related VTE.²⁰ Nonetheless, the UK NICE guidelines state that “in low bleeding risk patients, the benefits of continuing anticoagulation treatment are likely to outweigh the risks.”²¹ The reason for this recommendation can be related to the extraordinarily high efficacy (RRR)/major bleeding (0.85/0.00585 = 145.3) ratio in the low-risk scenarios calculated based on the evidence presented in the ASH thrombophilia guidelines.¹⁰ Such a high benefit/harms ratio generated a low test-treatment threshold (P_rx) = of 0.85%, which is below the 1% probability of VTE recurrence after surgery estimated by the ASH guidelines.¹⁰ According to EUT, as long as the EU of 1 strategy is higher than the other, regardless of whether the differences are trivial or large, we should select that management option.¹⁵ The earlier studies indicated a 0% risk of VTE after surgery,²² but more recent studies suggested a risk of ∼3%, noting that after a provoked VTE, the risk may not return to the population baseline of 0.1 to 0.2 per 100 patients per year.^20,23 Therefore, under the assumed ASH risk of VTE of 1% after 3 months of anticoagulation for VTE provoked by surgery,¹⁰ recommending extended anticoagulation is logically justifiable. Of course, the recommendations depend on the trustworthiness of these estimates. The ASH panel judged that the evidence used in their (and consequently our calculations) is of very low certainty because they were based on calculations with serious indirectness and imprecision of the estimates.¹⁰ Such evidence can be equally right or wrong.²⁴ It is quite possible that different assumptions would have resulted in different recommendations. However, the panel presented the best evidence on the topic to date, often with a range of estimates. Unfortunately, in the case of overall risk for VTE recurrence, the panel was able to provide only point estimates (eg, the risk of unprovoked VTE was 100 per 1000 patients in the first year).¹⁰ 

Another disagreement between our models relates to recommendations R3 to R5 (Figure 4; Table 3). The reason that the strategy “test and treat only positive” remains inferior to the strategy of “treating all” patients according to the threshold model is because patients with false negative tests would have incorrectly not received treatment.

Our model is based on the well-known threshold model,³ but equations 2 and 3 are novel derivations using the input parameters specified by the ASH thrombophilia panel.¹⁰ The unavailability of these threshold formulas may have been a reason why the ASH thrombophilia panel did not use the full scope of decision modeling as outlined in our paper. If so, we urge the panel to update its recommendations accordingly.

One limitation that affected both our and ASH thrombophilia models is that it is based on the “average” data obtained from the literature, including the overall average and estimates of the probability of VTE recurrence. Indeed, individual patients are at different risks of VTE recurrence. As a result, we have called for developing more individualized recommendations by the guidelines panels.^1,25 This can be accomplished by integrating the best evidence from systematic reviews/meta-analyses on the average treatment effects with predictive models to estimate individualized disease risks or outcomes and threshold decision models.^1,25,26 Despite the plethora of models, some better validated than others,²⁷ the use of predictive models to help individualize the guideline recommendations has not been widely promoted in the VTE field. Indeed, some experts favor recommendations based on intuitive, holistic assessment over predictive models.²⁰ The GRADE method refers not only to the benefits and harms of the management and patients' V&P but also to resource use, feasibility, acceptability, and equity.¹¹ However, all thrombophilia recommendations, both ASH’s and ours, were driven by decision modeling without formally considering these other factors. Clearly considering, articulating, and transparently displaying such issues would be desirable. Indeed, the whole idea of combining decision analysis with the GRADE methodology is to generate recommendations using explicit, easily understood decision models based on the best existing evidence following time-honored cognitive scientists’ advice: “to value formal principles of rationality” but then reflect on the appropriateness of further adjustments consistent with our explicit and implicit reasoning.^28-30 Still, the experience from other fields suggests that statistical rules typically outperform experts who rely on intuitive judgments.³¹ Intuitive approaches to integrating complex elements such as synthesis of treatment benefits and harms with patients’ V&P often do not agree with EUT models.^4,32 Under these conditions, people may rely on non-EUT decision strategies,^4,32 such as anticipating regret of being wrong to drive their decisions.^33-37 This is also true for guideline panels. For example, we have previously demonstrated that ASH guideline panel for the management of pulmonary embolism relied on several decision theoretical approaches to formulate their recommendations, some of which were based on non-EUT constructs.⁴ Which approach to take will largely depend on the type of problem, the consequences and the likelihood of being wrong, and contextual issues such as V&P, time, and available resources, etc.³⁸ However, there is a general consensus that we should always start with the EUT threshold model based on the best available evidence and further adjust it depending on the other elements considered essential for decision-making.³⁸ As we showed, our model clearly indicates the importance of consulting the patient’s V&P, which may include consideration of costs and other burdens specified within the GRADE system. Nevertheless, it can be argued that “the most optimal decisions may be those that achieve coherence at both the normative and intuitive levels.”²⁹ But, when these 2 types of knowledge do not agree, maximum efforts should be undertaken to reconcile the differences by exploring various theoretical approaches. This may be the most crucial reason why the ASH thrombophilia model should be updated. Providing a transparent and explicit explanation of the reasoning and analytical process through decision modeling, a solution to overcoming the challenges of integration and the “black-box” issue, when closely integrated with the GRADE methodology, can likely produce more coherent and accurate recommendations than relying solely on either decision models or the GRADE process alone. Adding the methodology described in this and other papers^{1,3-5,16,25,26} can bring us to the goal of transparent, trustworthy, easily accessible, understandable, and highly accurate guidelines. The method we described in response to the ASH thrombophilia guidelines can be easily applied to other clinical problems and holds promise to improve the current guidelines' methods without requiring additional resources that complex decision modeling does.

This project was supported in part by grant number R01HS024917 from the Agency for Healthcare Research and Quality (PI: B.D.).

Contribution: B.D. developed the conceptual idea and wrote the first draft of the manuscript; I.H. solved the model and developed program code; and G.G. revised the manuscript and improved its intellectual and research content

Conflict-of-interest disclosure: The authors declare no competing financial interests.

Correspondence: Benjamin Djulbegovic, Division of Medical Hematology and Oncology, Department of Medicine, Medical University of South Carolina, 39 Sabin St, MSC 635, Charleston, SC 29425; email: djulbegov@musc.edu.