Posted: May 1st, 2025
Final Acceptance is by the Exercise Design Team
Final Acceptance is by the Exercise Design Team.
Major Content.
Additional Exercise Program Planning.
Changes to Plans, Polices, and Procedures.
capability development and refinement.
priority focus of limited resources.
AAR / IP should always be considered an “FOUO” and the distribution of it should be agreed upon by the stakeholders.
Emergency Management
Exercise Design
Session 5
Exercise Evaluation
Exercise Evaluation
Homeland Security:
The act of reviewing or observing and recording exercise activity or conduct,
assessing behaviors or activities against exercise objectives, and noting strengths,
weaknesses, deficiencies, or other observations
Should be included incorporated into the exercise planning and design
Is used as the basis for improvement planning
The Homeland Security Exercise &
Evaluation Program Evaluation Process
Evaluation Planning
The process of evaluating an exercise begins with the exercise planning and
design team
The Lead Evaluator should be selected in the initial phases of the exercise
process and participate throughout the exercise design
Should be an experienced senior level individual with knowledge of;
The exercise planning, conduct and evaluation process
Applicable response plans and policies
The Incident Command System / NIMS
The ability to evaluate must be considered when crafting the exercise
objectives
Lead Evaluator
Before The Exercise
Development of the evaluation tools (EEGs, Etc.)
Identifying and training evaluators
Determining the evaluation plan
Assembling resources necessary for the Evauators
Maps
Plans
EEGs
Etc.
Lead Evaluator
During the Exercise
Assures that evaluators are properly located
Provides the evaluators with documentation and other necessary supplies
Makes assistance available to evaluators is/if needed
Lead Evaluator
After The Exercise
Lead the data analysis
Coordinate the evaluation meetings
Be principally responsible for the development of the After Action Report
Coordinate the Improvement Planning Activities
TCL / UTL
Target Capabilities List
The Federal Government recognizes 37 capabilities that government entities at all
levels are expected to support; such as:
Medical Surge
Emergency Public Information and Warning
Search and Rescue
Universal Task List
A comprehensive list of tasks which must be performed in response to an
emergency
The breadth of tasks performed increases with higher levels of government
Can be a reference for development of exercises and objectives
Exercise Evaluation Guides
HSEEP Standardized documents published to provide a consistent tool for
evaluation
Linked to Target Capabilities
Tools for documenting activities and observations
https://preptoolkit.fema.gov/web/hseep-resources/eegs
Evaluators
Evaluators should be non-players, ideally with independent expertise, and not
associated with an agency participating in the exercise
In addition to knowledge of response and preparedness, evaluators should
have experience in conducting an evaluation
Observing and recording the actions and interactions of and between players
Analyzing actions against expected actions and objectives
Assisting in the preparation of the After Action Report
Ideally, Municipalities should consider establishing a cadre of trained exercise
evaluators.
Evaluators
Should be assigned;
To each of the exercise play areas
In accordance with their knowledge and expertise
Must be thoroughly familiar with the exercise, including the objectives, the
exercise plan and the anticipated activities including all injects
Should be issued a Sit Man or a copy of the ExPlan, a copy of the C/E
handbook, the assingments and contact info of other evaluators and should
have access local plans, policies and procedures prior to the exercise.
C/E Briefing
Conducted shortly before the exercise.
A review of the exercise activites
A review of the exercise goal & objectives, and the the evaluation process
Affords an opportunity to ask questions
May include a visit to all play areas
Systematic Observation
Descriptive Reporting: A direct reporting of actions taken and observed
Inferential Reporting: Requires that the evaluator reach a conclusion based on
observations: “Response was timely”, “action was adequate”
Evaluative Reporting: Requires evaluation against a range performance.
Most exercise evaluation is accomplished through descriptive reporting
Levels of Performance Analysis
Task Level
Activity Level
Focus on direct actions
Involve groups of related tesks
Capability Level
High level Actions
Usually involves multiple participants from different agencies
Common Evaluation Issues
Observer Drift – Occurs when the evaluator loses interest
Errors of Leniency – Tendency to rate all points as positive
Errors of Central Tendency – Tendency to rate as aspects as “middle of the
road”
Halo Effect – All exercise activities rated positively based upon the early
positive impression of a person or group
Common Evaluation Issues
Hypercritical Effect – Occurs when the evaluator(s) believe that they are
supposed to find fault
Contamination – Involves evaluator knowledge of other, prior, performance of
the same task(s)
Evaluator bias – Evaluation is affected by the personal boas of the evaluator.
The Evaluator Effect
The mere presence…
Arrive Prior
Document Later
Adjusting actions
They way they should vs. they way they would
Intervening
For Safety
otherwise
Only when absolutely necessary
To gather crucial information not otherwise available
To clarify evaluator confusion
When Intervening
Use common language
Delay if practicable
Minimize time of intervention
Avoid leading or prompting questions
Avoid acting as an advisor
Reducing the Evaluator Effect
Use experienced / trained evaluators
Include training on evaluation issues
Ensure Evaluators are familiar with the documentation prior to the exercise
Evaluators should not hesitate to seek assistance / clarification from the Lead
Evaluator or other Evaluators
Avoid judgments during the exercise
Avoid projecting a negative impression
Avoid conversations about the progress of the exercise
Exercise Evaluation Guides
For Operations based exercises
Facilitate the recording of
What actions occurred
Who performed the actions or made the decision
Where the action or decision took place
When the action or decision took place
Why an action was performed or a decision was made
How players performed the action or made the decision
Collecting information
Participant Feedback forms
Logs (EOC, etc.)
Video or Audio recordings
Recorded phone lines
Player notes
Incident documentation
ICS Documentation
Controller / facilitator notes
Photos (Anyone)
Player Hotwash
Should include
Players, Exercise Design team, Evaluators, Controllers
An opportunity for players to express their impression of the exercise event;
including the response and the exercise design.
As soon as practical after EndEx
Allows Evaluators to seek clarification
May be more than one, should be convenient to all locations of play
Essentially a self-assessment, but should focus on strengths as well as
opportunities for improvement
Should be led by an individual experienced in leading group discussions
Preliminary Analysis
Each Evaluator should attempt to complete a preliminary analysis of the
objectives to which they were assigned.
Where possible, the evaluation should be at the activity and task level.
It should include documentation (description) of significant events occurring
during the exercise
Should include time references
Controller / Evaluator Debrief
Allows Evaluators to review the results of the Hotwash and obtain Controller
input / clarification
Allows Evaluators to compare notes
Allows revision of EEGs or other evaluation documents as / if needed
Facilitates evaluators assigned to the same function area(s) developing an
overall assessment.
Analyzing the Data
1) Identify Issues
Comparing actual performance to expected actions (objectives)
Construct an exercise timeline of events
2) Determining Root Cause of discrepancies
Use the 5 Whys!
The 5 Whys!
Analyzing The Data
3) Develop Recommendations for Improvement
Identify areas to improve
Consider both long term and short term solutions
Be consistent among all recommendations
Identify sources for assistance
Consider including How and possibly even Who in the recommendation
Examples of RFI
Changes to plans or procedures
Changes to organization structures
Changes to leadership or management processes
Additional or alternate training
Changes to resources or capabilities
Lessons Learned
In addition the formal analysis of the capabilities and tasks against the
objectives, Lessons Learned should be identified
Knowledge gained as a result of an exercise that identifies a practice or
existing knowledge as a tool for use in approaching a similar problem should it
occur in the future.
Should be specifically included in the AAR.
Consider DHS – LLIS
Emergency Management
Exercise Design
Session 5
Exercise Evaluation
Exercise Evaluation
Homeland Security:
The act of reviewing or observing and recording exercise activity or conduct,
assessing behaviors or activities against exercise objectives, and noting strengths,
weaknesses, deficiencies, or other observations
Should be included incorporated into the exercise planning and design
Is used as the basis for improvement planning
The Homeland Security Exercise &
Evaluation Program Evaluation Process
Evaluation Planning
The process of evaluating an exercise begins with the exercise planning and
design team
The Lead Evaluator should be selected in the initial phases of the exercise
process and participate throughout the exercise design
Should be an experienced senior level individual with knowledge of;
The exercise planning, conduct and evaluation process
Applicable response plans and policies
The Incident Command System / NIMS
The ability to evaluate must be considered when crafting the exercise
objectives
Lead Evaluator
Before The Exercise
Development of the evaluation tools (EEGs, Etc.)
Identifying and training evaluators
Determining the evaluation plan
Assembling resources necessary for the Evauators
Maps
Plans
EEGs
Etc.
Lead Evaluator
During the Exercise
Assures that evaluators are properly located
Provides the evaluators with documentation and other necessary supplies
Makes assistance available to evaluators is/if needed
Lead Evaluator
After The Exercise
Lead the data analysis
Coordinate the evaluation meetings
Be principally responsible for the development of the After Action Report
Coordinate the Improvement Planning Activities
TCL / UTL
Target Capabilities List
The Federal Government recognizes 37 capabilities that government entities at all
levels are expected to support; such as:
Medical Surge
Emergency Public Information and Warning
Search and Rescue
Universal Task List
A comprehensive list of tasks which must be performed in response to an
emergency
The breadth of tasks performed increases with higher levels of government
Can be a reference for development of exercises and objectives
Exercise Evaluation Guides
HSEEP Standardized documents published to provide a consistent tool for
evaluation
Linked to Target Capabilities
Tools for documenting activities and observations
https://preptoolkit.fema.gov/web/hseep-resources/eegs
Evaluators
Evaluators should be non-players, ideally with independent expertise, and not
associated with an agency participating in the exercise
In addition to knowledge of response and preparedness, evaluators should
have experience in conducting an evaluation
Observing and recording the actions and interactions of and between players
Analyzing actions against expected actions and objectives
Assisting in the preparation of the After Action Report
Ideally, Municipalities should consider establishing a cadre of trained exercise
evaluators.
Evaluators
Should be assigned;
To each of the exercise play areas
In accordance with their knowledge and expertise
Must be thoroughly familiar with the exercise, including the objectives, the
exercise plan and the anticipated activities including all injects
Should be issued a Sit Man or a copy of the ExPlan, a copy of the C/E
handbook, the assingments and contact info of other evaluators and should
have access local plans, policies and procedures prior to the exercise.
C/E Briefing
Conducted shortly before the exercise.
A review of the exercise activites
A review of the exercise goal & objectives, and the the evaluation process
Affords an opportunity to ask questions
May include a visit to all play areas
Systematic Observation
Descriptive Reporting: A direct reporting of actions taken and observed
Inferential Reporting: Requires that the evaluator reach a conclusion based on
observations: “Response was timely”, “action was adequate”
Evaluative Reporting: Requires evaluation against a range performance.
Most exercise evaluation is accomplished through descriptive reporting
Levels of Performance Analysis
Task Level
Activity Level
Focus on direct actions
Involve groups of related tesks
Capability Level
High level Actions
Usually involves multiple participants from different agencies
Common Evaluation Issues
Observer Drift – Occurs when the evaluator loses interest
Errors of Leniency – Tendency to rate all points as positive
Errors of Central Tendency – Tendency to rate as aspects as “middle of the
road”
Halo Effect – All exercise activities rated positively based upon the early
positive impression of a person or group
Common Evaluation Issues
Hypercritical Effect – Occurs when the evaluator(s) believe that they are
supposed to find fault
Contamination – Involves evaluator knowledge of other, prior, performance of
the same task(s)
Evaluator bias – Evaluation is affected by the personal boas of the evaluator.
The Evaluator Effect
The mere presence…
Arrive Prior
Document Later
Adjusting actions
They way they should vs. they way they would
Intervening
For Safety
otherwise
Only when absolutely necessary
To gather crucial information not otherwise available
To clarify evaluator confusion
When Intervening
Use common language
Delay if practicable
Minimize time of intervention
Avoid leading or prompting questions
Avoid acting as an advisor
Reducing the Evaluator Effect
Use experienced / trained evaluators
Include training on evaluation issues
Ensure Evaluators are familiar with the documentation prior to the exercise
Evaluators should not hesitate to seek assistance / clarification from the Lead
Evaluator or other Evaluators
Avoid judgments during the exercise
Avoid projecting a negative impression
Avoid conversations about the progress of the exercise
Exercise Evaluation Guides
For Operations based exercises
Facilitate the recording of
What actions occurred
Who performed the actions or made the decision
Where the action or decision took place
When the action or decision took place
Why an action was performed or a decision was made
How players performed the action or made the decision
Collecting information
Participant Feedback forms
Logs (EOC, etc.)
Video or Audio recordings
Recorded phone lines
Player notes
Incident documentation
ICS Documentation
Controller / facilitator notes
Photos (Anyone)
Player Hotwash
Should include
Players, Exercise Design team, Evaluators, Controllers
An opportunity for players to express their impression of the exercise event;
including the response and the exercise design.
As soon as practical after EndEx
Allows Evaluators to seek clarification
May be more than one, should be convenient to all locations of play
Essentially a self-assessment, but should focus on strengths as well as
opportunities for improvement
Should be led by an individual experienced in leading group discussions
Preliminary Analysis
Each Evaluator should attempt to complete a preliminary analysis of the
objectives to which they were assigned.
Where possible, the evaluation should be at the activity and task level.
It should include documentation (description) of significant events occurring
during the exercise
Should include time references
Controller / Evaluator Debrief
Allows Evaluators to review the results of the Hotwash and obtain Controller
input / clarification
Allows Evaluators to compare notes
Allows revision of EEGs or other evaluation documents as / if needed
Facilitates evaluators assigned to the same function area(s) developing an
overall assessment.
Analyzing the Data
1) Identify Issues
Comparing actual performance to expected actions (objectives)
Construct an exercise timeline of events
2) Determining Root Cause of discrepancies
Use the 5 Whys!
The 5 Whys!
Analyzing The Data
3) Develop Recommendations for Improvement
Identify areas to improve
Consider both long term and short term solutions
Be consistent among all recommendations
Identify sources for assistance
Consider including How and possibly even Who in the recommendation
Examples of RFI
Changes to plans or procedures
Changes to organization structures
Changes to leadership or management processes
Additional or alternate training
Changes to resources or capabilities
Lessons Learned
In addition the formal analysis of the capabilities and tasks against the
objectives, Lessons Learned should be identified
Knowledge gained as a result of an exercise that identifies a practice or
existing knowledge as a tool for use in approaching a similar problem should it
occur in the future.
Should be specifically included in the AAR.
Consider DHS – LLIS
Emergency Management
Exercise Design
Session 6
After Action Reporting
After Action Report
Records what occurred during the exercise
Provides feedback on the achievement, or not, of capabilities and tasks
associated with the objectives
Suggests recommendations for improvement
Developed initially in draft form
After Action Report
Executive Summary
Executive Overview
Exercise Goal(s) and Objectives
Analysis of Capabilities Demonstrated
Conclusion
Improvement Plan
The After Action Conference
For review and review and revision of the draft AAR
Includes the Exercise Planning team, Evaluation Team, and stakeholders
Begin Development of the Improvement Plan
Includes corrective actions and timelines
Produces the final AAR/IP
Corrective Actions
May initially just be first steps
Must accept realistic appreciation of limited resources
Should be both short term and long term
Should have benchmarks
…Number trained
…items purchased
…plans revised
…MOUs completed
Final AAR / IP
Final Acceptance is by the Exercise Design Team
Major Content
Additional Exercise Program Planning
Changes to Plans, Polices and Procedures
Capability development and refinement
Priority focus of limited resources
AAR / IP should always be considered an “FOUO” and the distribution of it
should be agreed upon by the stakeholders
For next week, please prepare one or more Exercise Evaluation Guidelines for your
exercise and please prepare a blank After Action Report / Improvement Plan
template, likewise for your exercise.
You can get EEG templates by searching for “HSEEP EEG Template”
Please package these together with the rest of your exercise and prepare one
printed copy to give me next week.
be prepared to present your exercise to me and the class as if you were an
emergency manager presenting to a mayor and or a city council seeking approval
to conduct the exercise.
if you would like to also prepare a powerpoint to help brief your exercise, that
would be fine.
Expert paper writers are just a few clicks away
Place an order in 3 easy steps. Takes less than 5 mins.