A hotwash captures immediate participant experience; an after-action report validates and explains findings; an improvement plan turns those findings into accountable actions. The process is complete only when changes are implemented and their effectiveness is checked in later training, exercises or operations.
The common failure is to produce a polished report that is not connected to ownership, resources or follow-up. Improvement planning should begin during exercise design, not after the event.
Prepare the evidence structure in advance
Link every evaluation question to an objective and identify the evidence an evaluator should collect. Provide a consistent observation format: time, condition, action, result, relevant plan or standard, and factors that affected performance.
Assign evaluators to specific functions or objectives. Brief them on what is inside and outside scope. This reduces broad comments such as “communications were poor” that are difficult to validate or act upon.
Run a focused hotwash
Hold the hotwash soon after play while decisions are fresh. Create a learning environment rather than a performance. Ask:
What was expected to happen?
What actually happened?
What helped the team?
What made the work difficult?
Which information or assumption shaped the decision?
What should be preserved, changed or examined further?
Separate immediate impressions from confirmed findings. A participant may experience an information gap while the simulation log shows the message was sent to the wrong role; both perspectives are useful, but the cause still needs analysis.
Consolidate and validate observations
Bring together evaluator notes, participant feedback, system records, plans and exercise-control information. Combine duplicates and resolve contradictions. Confirm that the condition was not created by an unintended exercise artifact such as an inject delivered late or an unavailable tool that participants were told would work.
Write findings around performance and consequence. A useful statement explains the capability, the observed condition, the effect and the likely contributing factors. Avoid naming individuals unless personnel-specific coaching is the defined purpose and handled through the appropriate process.
Find the cause that can be changed
“More training” is often proposed before the cause is understood. Ask whether the issue came from knowledge, practice, authority, staffing, workflow, technology, information, policy or partner agreement. Several factors may interact.
For example, a late situation report might reflect unfamiliar software, unclear responsibility for validation, conflicting deadlines or no agreed information requirements. Each cause calls for a different action.
Write improvement actions that can be verified
An improvement action should specify the change, owner, expected result, target date and method of verification. Compare:
Weak: Improve communications.
Stronger: The EOC planning lead will revise the shift-briefing checklist to require unresolved resource requests, brief all rostered section leads, and test the checklist during the October shift-change simulation.
Not every observation needs an action. Preserve strengths, close issues that have no material consequence and combine actions that address the same cause. prioritize by risk, operational value, dependencies and available authority.
Assign ownership at the right level
The owner must be able to coordinate the work, but may need support from policy, technology, facilities, training or partner organizations. Record a responsible owner and contributing parties. If an action depends on executive approval or funding, make that dependency visible.
Set review dates that reflect the work rather than defaulting every action to the same deadline. Use interim milestones for complex changes.
Track closure and effectiveness separately
An action can be administratively complete but operationally unproven. Updating a document, configuring a system or delivering a briefing closes an implementation step. A later observation must show whether the change produced the intended result.
Maintain two fields: implementation status and effectiveness status. This simple distinction prevents “closed” findings from disappearing before anyone checks them.
Retest deliberately
Retesting does not always require another large exercise. A short browser simulation can test a revised information workflow. A focused tabletop can verify authority. A drill can check a corrected physical handoff. Choose the least burdensome method that produces credible evidence.
Carry unresolved actions into future exercise objectives and planning meetings. If the same finding recurs, reassess the assumed cause and the practicality of the chosen action.
The value of an exercise is not the number of observations it generates. It is the quality of the decisions an organization makes about what to preserve, what to change and how to know the change worked.
Sources
FEMA Improvement Planning Resources: https://preptoolkit.fema.gov/web/hseep-resources/improvement-planning
FEMA Homeland Security Exercise and Evaluation Program: https://training.fema.gov/programs/nsec/hseep/
FEMA Exercise Evaluation Guides: https://preptoolkit.fema.gov/web/hseep-resources/eegs
FEMA Continuous Improvement Technical Assistance Program: https://preptoolkit.fema.gov/web/cip-citap
Continue the conversation
Every training environment has different constraints. If you are working through how to make scenarios more repeatable, measurable, or accessible, share your training challenge with us.
A hotwash captures immediate participant experience; an after-action report validates and explains findings; an improvement plan turns those findings into accountable actions. The process is complete only when changes are implemented and their effectiveness is checked in later training, exercises or operations.
The common failure is to produce a polished report that is not connected to ownership, resources or follow-up. Improvement planning should begin during exercise design, not after the event.
Prepare the evidence structure in advance
Link every evaluation question to an objective and identify the evidence an evaluator should collect. Provide a consistent observation format: time, condition, action, result, relevant plan or standard, and factors that affected performance.
Assign evaluators to specific functions or objectives. Brief them on what is inside and outside scope. This reduces broad comments such as “communications were poor” that are difficult to validate or act upon.
Run a focused hotwash
Hold the hotwash soon after play while decisions are fresh. Create a learning environment rather than a performance. Ask:
What was expected to happen?
What actually happened?
What helped the team?
What made the work difficult?
Which information or assumption shaped the decision?
What should be preserved, changed or examined further?
Separate immediate impressions from confirmed findings. A participant may experience an information gap while the simulation log shows the message was sent to the wrong role; both perspectives are useful, but the cause still needs analysis.
Consolidate and validate observations
Bring together evaluator notes, participant feedback, system records, plans and exercise-control information. Combine duplicates and resolve contradictions. Confirm that the condition was not created by an unintended exercise artifact such as an inject delivered late or an unavailable tool that participants were told would work.
Write findings around performance and consequence. A useful statement explains the capability, the observed condition, the effect and the likely contributing factors. Avoid naming individuals unless personnel-specific coaching is the defined purpose and handled through the appropriate process.
Find the cause that can be changed
“More training” is often proposed before the cause is understood. Ask whether the issue came from knowledge, practice, authority, staffing, workflow, technology, information, policy or partner agreement. Several factors may interact.
For example, a late situation report might reflect unfamiliar software, unclear responsibility for validation, conflicting deadlines or no agreed information requirements. Each cause calls for a different action.
Write improvement actions that can be verified
An improvement action should specify the change, owner, expected result, target date and method of verification. Compare:
Weak: Improve communications.
Stronger: The EOC planning lead will revise the shift-briefing checklist to require unresolved resource requests, brief all rostered section leads, and test the checklist during the October shift-change simulation.
Not every observation needs an action. Preserve strengths, close issues that have no material consequence and combine actions that address the same cause. prioritize by risk, operational value, dependencies and available authority.
Assign ownership at the right level
The owner must be able to coordinate the work, but may need support from policy, technology, facilities, training or partner organizations. Record a responsible owner and contributing parties. If an action depends on executive approval or funding, make that dependency visible.
Set review dates that reflect the work rather than defaulting every action to the same deadline. Use interim milestones for complex changes.
Track closure and effectiveness separately
An action can be administratively complete but operationally unproven. Updating a document, configuring a system or delivering a briefing closes an implementation step. A later observation must show whether the change produced the intended result.
Maintain two fields: implementation status and effectiveness status. This simple distinction prevents “closed” findings from disappearing before anyone checks them.
Retest deliberately
Retesting does not always require another large exercise. A short browser simulation can test a revised information workflow. A focused tabletop can verify authority. A drill can check a corrected physical handoff. Choose the least burdensome method that produces credible evidence.
Carry unresolved actions into future exercise objectives and planning meetings. If the same finding recurs, reassess the assumed cause and the practicality of the chosen action.
The value of an exercise is not the number of observations it generates. It is the quality of the decisions an organization makes about what to preserve, what to change and how to know the change worked.
Sources
FEMA Improvement Planning Resources: https://preptoolkit.fema.gov/web/hseep-resources/improvement-planning
FEMA Homeland Security Exercise and Evaluation Program: https://training.fema.gov/programs/nsec/hseep/
FEMA Exercise Evaluation Guides: https://preptoolkit.fema.gov/web/hseep-resources/eegs
FEMA Continuous Improvement Technical Assistance Program: https://preptoolkit.fema.gov/web/cip-citap
Continue the conversation
Every training environment has different constraints. If you are working through how to make scenarios more repeatable, measurable, or accessible, share your training challenge with us.