Table 1. Prompt engineering with 1000... | American Society of Hematology

Table 1.

Prompt engineering with 1000 labeled examples

Prompt only with no labeled examples (zero-shot)
Prompt 1 (shortest prompt) Ask the model to roleplay as a physician and classify reports directly	Act as an expert radiologist and identify reports that describe acute pulmonary embolism as positive. Output ‘positive’ or ‘negative’ at the end of your response, in a single line that starts with ‘Label:’
Prompt 2 (longer prompt) Ask the model to roleplay as a physician and classify reports directly, with additional direction	Please review the radiology report and identify which ones describe an acute pulmonary embolism. If there is an acute pulmonary embolism, label the report positive. If not, the report should be labeled negative. Output ‘positive’ or ‘negative’ at the end of your response, in a single line that starts with ‘Label:’
Prompt 3 (longest prompt) Ask the model to roleplay as a physician and classify reports via chain-of-thought prompting, with additional direction and examples	You are a physician reviewing a radiology report. Please isolate any sentences that describe pulmonary embolism. If the radiology report describes an acute pulmonary embolism, label the report as ‘positive.’ All other reports are labeled ‘negative.’ If there is no pulmonary embolism, chronic pulmonary embolism only, or the findings are equivocal, label the report as ‘negative.’ Other types of findings such as other thrombi or masses are labeled ‘negative.’ Output ‘positive’ or ‘negative’ at the very end of the response, in a single line that starts with ‘Label:’
Best prompt from above with labeled examples (in-context learning)
4 labeled examples	Acute PE (positive) Negative PE (negative) Equivocal PE (negative) Chronic PE (negative)
5 labeled examples	Acute PE (positive) Subsegmental PE (positive) Negative PE (negative) Equivocal PE (negative) Chronic PE (negative)
6 labeled examples	Acute PE (positive) Subsegmental PE (positive) Negative PE (negative) Equivocal PE (negative) Chronic PE (negative) Chronic PE with other findings (negative)
Best prompt from above with maximum number of labeled examples and explanation
Label only	Label: negative
Label and relevant sentence(s)	Label: negative Relevant sentence: “Chronic appearing pulmonary emboli involving the distal right main pulmonary artery, and multiple lobar, segmental and subsegmental branches of the bilateral upper and lower lobes.”
Label and relevant sentence(s) & explanation	Label: negative Relevant sentence: “Chronic appearing pulmonary emboli involving the distal right main pulmonary artery, and multiple lobar, segmental and subsegmental branches of the bilateral upper and lower lobes.” Explanation: This report states that there are chronic pulmonary emboli only with no acute pulmonary emboli. We consider these cases negative.

Prompt only with no labeled examples (zero-shot)
Prompt 1 (shortest prompt) Ask the model to roleplay as a physician and classify reports directly	Act as an expert radiologist and identify reports that describe acute pulmonary embolism as positive. Output ‘positive’ or ‘negative’ at the end of your response, in a single line that starts with ‘Label:’
Prompt 2 (longer prompt) Ask the model to roleplay as a physician and classify reports directly, with additional direction	Please review the radiology report and identify which ones describe an acute pulmonary embolism. If there is an acute pulmonary embolism, label the report positive. If not, the report should be labeled negative. Output ‘positive’ or ‘negative’ at the end of your response, in a single line that starts with ‘Label:’
Prompt 3 (longest prompt) Ask the model to roleplay as a physician and classify reports via chain-of-thought prompting, with additional direction and examples	You are a physician reviewing a radiology report. Please isolate any sentences that describe pulmonary embolism. If the radiology report describes an acute pulmonary embolism, label the report as ‘positive.’ All other reports are labeled ‘negative.’ If there is no pulmonary embolism, chronic pulmonary embolism only, or the findings are equivocal, label the report as ‘negative.’ Other types of findings such as other thrombi or masses are labeled ‘negative.’ Output ‘positive’ or ‘negative’ at the very end of the response, in a single line that starts with ‘Label:’
Best prompt from above with labeled examples (in-context learning)
4 labeled examples	Acute PE (positive) Negative PE (negative) Equivocal PE (negative) Chronic PE (negative)
5 labeled examples	Acute PE (positive) Subsegmental PE (positive) Negative PE (negative) Equivocal PE (negative) Chronic PE (negative)
6 labeled examples	Acute PE (positive) Subsegmental PE (positive) Negative PE (negative) Equivocal PE (negative) Chronic PE (negative) Chronic PE with other findings (negative)
Best prompt from above with maximum number of labeled examples and explanation
Label only	Label: negative
Label and relevant sentence(s)	Label: negative Relevant sentence: “Chronic appearing pulmonary emboli involving the distal right main pulmonary artery, and multiple lobar, segmental and subsegmental branches of the bilateral upper and lower lobes.”
Label and relevant sentence(s) & explanation	Label: negative Relevant sentence: “Chronic appearing pulmonary emboli involving the distal right main pulmonary artery, and multiple lobar, segmental and subsegmental branches of the bilateral upper and lower lobes.” Explanation: This report states that there are chronic pulmonary emboli only with no acute pulmonary emboli. We consider these cases negative.