Posted: May 1st, 2025

Prepare the Exercise Evaluation Guidelines for my exercise attached

Prepare
the Exercise Evaluation Guidelines for my exercise attached as the (
Response_Infrastructure Systems_HSEEP Template), which are attached, according
to the two presentations attached. Also, please prepare a blank After Action
Report / Improvement Plan template, likewise for my exercise.

Description of Home Locality
My home locality is the Jazan region, situated in the southwestern part of Saudi
Arabia. Jazan is known for its lush green landscapes, diverse culture, and significant
agricultural activities. The region is characterized by its tropical climate, with hot
summers and mild winters. The topography of the area includes both coastal areas
along the Red Sea and mountainous terrain inland. Jazan is primarily an agricultural
region, with various crops such as dates, grains, and fruits cultivated in the fertile
soils.
Emergency and Disaster Response Resources
In Jazan, emergency and disaster response resources are primarily managed at both
the national and local levels. At the national level, Saudi Arabia has established
various governmental agencies and organizations responsible for disaster
management, including the Saudi Red Crescent Authority and the General Directorate
of Civil Defense. These entities oversee preparedness, response, and recovery efforts
for various emergencies and disasters nationwide.
At the local level, within the Jazan region, there are emergency response teams,
including fire departments, medical services, and civil defense units. These local
resources work in coordination with national agencies to address emergencies
effectively. Additionally, community-based organizations and volunteers often play a
crucial role in providing support during disasters, offering assistance such as shelter,
food, and medical aid.
Potential Hazards Faced by the Locality
Jazan region faces various potential hazards, including natural, human-caused, and
technical events.
Natural Hazards: The region is prone to heavy rainfall, especially during the
monsoon season, which can lead to flooding and landslides, as observed in the case of
continuous rain for more than two days. Additionally, Jazan is susceptible to tropical
cyclones originating from the Red Sea, which can cause extensive damage to
infrastructure and agriculture.
Human-Caused Events: Industrial accidents, transportation incidents, and structural
failures are potential human-caused hazards in the area. Jazan hosts several industrial
facilities, including oil refineries and petrochemical plants, increasing the risk of
accidents such as chemical spills or explosions.
Hazmat Events: Given the presence of industrial complexes, the risk of hazardous
materials incidents is a concern. Chemical leaks or releases from industrial sites could
pose significant risks to public health and the environment.
Technical Hazards: Infrastructure failures, such as those related to water drainage
systems or transportation networks, can occur, especially during extreme weather
events like heavy rainfall. Poorly maintained drainage systems can lead to water
accumulation in streets, exacerbating flooding and causing disruption to daily life.
Addressing the challenges of continuous rain and poor drainage requires a comprehensive
and proactive approach. Collaboration between national, regional, and local authorities is
essential for effective emergency planning and response in Jazan. This collaboration should
involve allocating resources, training programs, and public awareness campaigns to enhance
community resilience.
Investing in resilient infrastructure, including drainage systems, is essential to
mitigate the impact of heavy rainfall and minimize flooding. Public education
initiatives on disaster preparedness and evacuation procedures can empower residents
to take proactive measures and respond effectively to emergencies.
A comprehensive understanding of these potential hazards is essential for effective
emergency planning and response in the Jazan region. Cooperation between national,
regional, and local authorities and community engagement is vital for mitigating risks
and enhancing resilience to disasters.
Emergency Exercise.
Communities and Agencies Involved.
Jazan City, in Saudi Arabia, experiences the constant threat of flooding due to
climatic conditions and geographical location. The exercise will include the city’s
main agencies and stakeholders tasked with emergency management and response.
They include, Jazan Municipality, Ministry of Health, Water Authority, The Police
Department, Saudi Red Crescent Authority, Local Community Organizations and
Volunteer Groups
Exercise Goal
The exercise intends to improve local emergency management agencies’ and
stakeholders’ readiness and response abilities in managing urban flooding
emergencies.
Type of Exercise.
This is a Full-Scale Exercise since it offers a realistic simulation of an actual
emergency scenario, and due to the nature of urban flooding and its effect on public
safety, infrastructure, and health, it would facilitate the deployment of personnel,
mobilization of resources, and assessing of response processes in realistic
environments. It also allows the assessment of response abilities in various agencies
involved in emergency management. Incorporating different stakeholders and
simulating a complicated scenario enables the exercise to evaluate communication,
coordination, and decision-making procedures. Considering that urban flooding
situations necessitate coordination among various entities like emergency responders,
municipal authorities, utility services, and public health officials, the exercise enables
combining knowledge and resources from multiple entities, resulting in a more
effective and coordinated effort.
SMART Exercise Objectives.
SMART Objective 1: Acquire knowledge of the responsibilities and roles of the
relevant agencies when tackling urban flooding emergencies.
Detailed SMART Objective 1: The project will explain what actions and duties of
those involved in urban flooding problems are and how they are performed. This
involves assessing the level of communication and coordination of agencies in a
tabletop exercise or simulation session. Agencies of about 20 individuals will undergo
training on 5th April and be equipped with resources to be able to do their duties to
reduce the consequences of emergencies. This will also help to make sure the set
goals pay attention to urban flooding, and the evaluation will be conducted within
three weeks.
Figure 1: SMART Objective 1
Specific
Who: Participating Agencies
What: Know agency responsibilities in the event of an urban flooding emergency
Measurable
How many: 20 individuals
Documented by: Assessment reports
Accountable to: Exercise coordinators
Attainable
Justification: Train and supply resources critical for role fulfillment
Relevant
Rationale: Set exercise objectives in the context of urban flooding situations
Time-based
When: The agency’s performance review will be finished within three weeks
SMART Objective 2: Evaluate the efficiency of the plans for emergency situation.
Detailed SMART Objective 2: The performance of the agencies engaged in the
process will be determined by the degree to which their emergency plan has been
implemented and the extent to which it has been employed. The targeted number of
emergency plans to be evaluated is 20 per day. This is done to mitigate public safety
and infrastructure disruptions that may have been caused by flooding. The process
involves reviewing simulations responses, identifying things that need to be
improved, and develop alternative response approaches. It will be employed in 10th
April timeframe.
Figure 2: SMART Objective 2
Specific
Who: Participating Agencies
What: Evaluating the efficiency of emergency plans
Measurable
How many: 20 emergency plan
Documented by: Assessment reports
Accountable to: Exercise coordinators
Attainable
Justification: Specify the scope for improvement and refinement of the given strategies
Relevant
Rationale: Find a way to handle the most urgent needs of the suffering population
Time-based
When: 10th April
SMART Objective 3: Improve interagency interoperability and coordination.
Detailed SMART Objective 3: The project will provide interagency interoperability
and collaboration by joint exercises and resources. The targeted participant is 30
individuals both male and female. The evaluation shall be conducted through
observation of communication and resources integration shared among agencies
over the course of the exercise and it will be done on monthly basis from AprilSeptember 2024. Creating an atmosphere of cooperation among organizations
through networking opportunities is essential in achieving the task.
Figure 3: SMART Objective 3
Specific
Who: Participating Agencies
What: Foster effective coordination and collaboration on an interagency level
Measurable
How many: 30 individuals both male and female
Documented by: Assessment reports
Accountable to: Exercise coordinators
Attainable
Justification: Aim to establish networking and partnerships
Relevant
Rationale: Take into account the intricacy of urban flooding emergencies
Time-based
When: April-September 2024
SMART Objective 4: Increase community preparedness awareness and public
attention to urban flooding.
Detailed SMART Objective 4: The project’s aim is to foster readiness in community,
by reaching out and educating people about urban flooding and everything related to
that. This will be evaluated by tracking the efficacy of outreach programs aiming at
developing the preparedness and response of the communities in emergency
situations. The institution will be 5 in each of the urban center with a minimum of 10
instructors. In addition, achievability is possible by creating educational resources
and communication messages that tell communities the dangers of flooding and the
steps to take for their own safety. The program will take 4 months from 5th April 2024
to 5th August 2024.
Figure 4: SMART Objective 4:
Specific
Who: Participating Agencies
What: Increase the proportion of community preparedness and spread public
consciousness
Measurable
How many: 5 institutions in each of the urban center with a minimum of 10 instructors
Documented by: Assessment reports
Accountable to: Outreach coordinators
Attainable
Justification: Develop educational content and communication strategies
Relevant
Rationale: Urge the community to participate in guarding against flooding incidents
Time-based
When: 4 months from 5th April 2024 to 5th August 2024.
SMART Objective 5: Analyze the stability and operability of the critical infrastructure
system.
Detailed SMART Objective 5: The project will measure the stability and efficiency of
the infrastructure systems that include a flood mitigation system and drainage
networks. It is estimated to target 80% of the urban centers with poor infrastructure
systems. This entails testing the performance of critical infrastructure in the simulated
environments, spotting vulnerable parts, and working closely with the federal, state,
and local agencies and utility providers, throughout the evaluation process. Essential
services will be initiated on 10th April and the plan will help make infrastructure
repairs and improvement, including pre-determined arrangements.
SMART Objective 5:
Specific
Who: Participating Agencies and Utility Companies
What: Evaluate the essential elements and systems’ ability and extent to continue without
disruptions
Measurable
How many: 80% of the urban centers
Documented by: Assessment reports
Accountable to: Infrastructure evaluation coordinators
Attainable
Justification: Work with agencies and companies for evaluations
Relevant
Rationale: Create sustainable arrangements to enable the continuity of crucial services
during calamities
Time-based
When: 10th April
SMART Objective 6: Enhancing the proficiency of the emergency responders for
search and rescue mission.
Detailed SMART Objective 6: The project manager will accomplish emergency
response team’s ability to conduct search and rescue mission as well as medical aid
in flood periods. Also, the number of responders trained for the program will be 10
both male and female from the 5-training institution. In addition, the emergency
response incorporates drowning and flood situations into the assessment of search
and rescue mission, and medical protocols. Also, it points out the needed training
and resources for rescue team to increase the responder’s skills and readiness. The
trained individuals will start their rehearsals on 4th April 2024 to familiarize
themselves with the program’s aim
SMART Objective 6
Specific
Who: Emergency Responders
What: Develop search and rescue operations and provide medical assistance in times of
flooding
Measurable
How many: 10 both male and female from the 5-training institution
Documented by: Assessment reports
Accountable to: Training coordinators
Attainable
Justification: Provide training and resources for skill enhancement
Relevant
Rationale: Address challenges and risks in urban flooding situations
Time-based
When: 4th April 2024
Staffing Plan
Design Team.
The team is tasked with the exercise’s planning and development. It comprises
experts and representatives from the main agencies. The team includes;
● Emergency Management Coordinator from the municipality
● Emergency Response Planner also from the municipality
● Public Health Specialist from the Ministry of Health
● Infrastructure Engineer from the Water Authority
● Training and Exercise Coordinator from the Local Emergency Management
Office.
Conduct Team.
Exercise Director.
They will have authority over the exercise and will ensure the exercise goals are
accomplished and progress smoothly.
Controllers.
They will manage the exercise setup. This entails controlling the exercise’s pace,
implementing the scenario, and constantly interacting with participants.
Facilitators.
They will be important in discussion-based exercises by guiding group discussions,
ensuring participants are actively involved, and meeting exercise goals.
Lead Evaluator.
They will lead the evaluation procedures. This entails coordinating with evaluators,
gathering data, and assessing exercise outcomes to develop an After Action Report.
Evaluators.
They will assess agencies and participants’ performances during the exercise.
Predefined evaluation criteria will be used to develop feedback on strengths,
weaknesses, and areas for improvement.
Role Players.
They will act as different community members and stakeholders impacted by the
flooding scenario. They will develop realistic situations and environments for
participants.
Budget.
Personnel Costs. 1,500,000
On-Duty Personnel.
● Emergency Management Coordinator (Jazan Municipality), 100,000 SAR
● Public Health Specialist (Ministry of Health) 100,000 SAR
● Infrastructure Engineer (Water Authority) 100,000 SAR
● Training and Exercise Coordinator (Local Emergency Management Office)
100,000 SAR
● Safety Officer 100,000 SAR
● Medical Team (Paramedics) 100,000 SAR
● Controllers 100,000 SAR
● Facilitators 100,000 SAR
● Lead Evaluator 100,000 SAR
● Evaluators 100,000 SAR
● Role Players 100,000 SAR
● Simulators 100,000 SAR
● Communications Officer 100,000 SAR
● Logistics Coordinator 100,000 SAR
● Observers 100,000 SAR
Backfill Costs. 200,000
In situations that require on-duty personnel to be replaced, the backfill expenses will
entail additional staffing or overtime expenses their respective agencies incurred.
Other Expenses. 1,300,000
● Venue Rental: Expenses from renting an appropriate venue for doing the
exercise. This includes setup and cleanup costs. 100,000 SAR
● Equipment and Supplies: This results from simulation materials like rescue
boats and flood water pumps. 100,000 SAR
● Communication devices like megaphones and radios. 100,000 SAR
● Medical supplies and first aid kits. 100,000 SAR
● Exercise materials like scenario documents and maps. 100,000 SAR
● Safety gear like life jackets and helmets. 100,000 SAR
● Moulage supplies for simulating injuries. 100,000 SAR
● Water and refreshment for participants. 100,000 SAR
● Transportation costs for moving personnel and equipment to and from the
venue. 100,000 SAR
● Documentation and reporting will be involved since exercise manuals,
evaluation forms, and after-action reports will be printed. 100,000 SAR
● Administrative costs resulting from compiling the documentation. 100,000
SAR
● Training sessions and pre-exercise briefings for participants will result in
additional expenses. 100,000 SAR
● Miscellaneous costs resulting from unexpected emergencies or expenses.
100,000 SAR
Total Estimated Budget.
The budget for the urban flooding emergency exercise is 3,000,000 SAR.
Resource List for Urban Flooding Emergency Exercise
1. Resources for Planning
People
 Emergency Management Coordinator
 Municipality’s Emergency Response Planner
 Health Officer from the Ministry of Health
 Water Infrastructure Engineer.
 Training and Exercise Coordinator
Information
 Flood incidents data from historical records of Jazan City.
 The geographic information system (GIS) data for flood-prone urban zones.
 Demographic data of the communities projected to be affected by flood.
Things
 Templates for planning emergency drills
 Communication tools (such as email video conferencing)
 Project management software
 Budgeting tools for financial planning.
Places
 Space for holding planning meetings and briefings.
 Community centers at the local level.
2. Resources for Conduct
People
 Exercise Director
 Controllers
 Facilitators
 Lead Evaluator
 Safety Officer
 Medical Team
 Communications Officer
 Logistics Coordinator
Information
 Exercise guidelines for the siting scenario.
 Plans of emergency response with procedures for participating agencies
 Local agencies and authority’s emergency phone numbers
Things
 Rescue boats, floodwater pumps, and moulage supplies.
 A megaphone, a walkie-talkie, or a mobile device.
 Life jackets, helmets, and first aid kits.
Places
 Outdoors settings.
 Indoor venues.
 Medical centers.
3. Resources for Simulation
People
 Simulation trainees.
 Trained responders.
 Technical support team.
Information
 Detailed guidelines and roadmaps to follow.
 The creation of weather forecasts to resemble the conditions in real-time.
 Feedback forms.
Things
 Floodwater tanks, sound effects, and lighting.
 Infrastructure and vehicles flooded in the flood.
Places
 Exercise hall.
 Control rooms
4. Resources for Exercise Play
People
 Emergency responders and volunteers.
 Coordinators.
 Support team.
Information
 Protocols and plans for emergency response of the participants involved
in the exercise.
 Maps and models of the exercise area and area of the scenarios.
 The post-exercise report guides for recording and analyzing outcomes
and lessons learned.
Things
 Signage, barricades, and markers for designated lanes or spaces.
 Snacks and refreshments.
 The equipment for exercise records and debriefing sessions.
Places
 Labeled areas where the exercise phases occur, inside and outside the
building.
 Chat rooms for post-exercise feedback and assessment.
 Spaces for storage of exercise equipment and supplies.
The MSEL and Exercise Injects
Increase in Speed of Events Injects
Decrease in Speed of Events Injects
Refocus Player Actions Injects
Response: Infrastructure Systems
[Exercise Name]
Exercise Evaluation Guide
Exercise Name: [Insert Exercise Name]
Exercise Date: [Insert Exercise Date]
Jurisdiction/Organization: [Insert Jurisdiction or Organization]
Venue: [Insert Venue Name]
Response
Exercise Objective: [Insert exercise objective]
Core Capability: Infrastructure Systems
Stabilize critical infrastructure functions, minimize health and safety threats, and efficiently restore and
revitalize systems and services to support a viable, resilient community.
Organizational Capability Target 1: [Insert customized target based on plans and assessments]
Critical Task: [Insert task from frameworks, plans, or Standard Operating Procedures (SOPs)]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Source(s): [Insert name of plan, policy, procedure, or reference]
Organizational Capability Target 2: [Insert customized target based on plans and assessments]
Critical Task: [Insert task from frameworks, plans, or Standard Operating Procedures (SOPs)]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Source(s): [Insert name of plan, policy, procedure, or reference]
Organizational Capability Target 3: [Insert customized target based on plans and assessments]
Critical Task: [Insert task from frameworks, plans, or Standard Operating Procedures (SOPs)]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Critical Task: [Insert task from frameworks, plans, or SOPs]
Source(s): [Insert name of plan, policy, procedure, or reference]
EEG-Resp-IS
Homeland Security Exercise and Evaluation Program
[PROTECTIVE MARKING, AS APPROPRIATE]
1
[Sponsor Organization]
Rev. 2020 508
Response: Infrastructure Systems
[Exercise Name Continued (if needed)]
Organizational
Capability Target
Associated Critical Tasks
Observation Notes and Explanation of Rating
Target
Rating
[Insert Organizational
Capability Target 1
from page 1]
• [Insert Organizational
Capability Target 1 Critical
Tasks from page 1]
[Observation notes and explanation of rating]
[Target
rating]
[Insert Organizational
Capability Target 2
from page 1]
• [Insert Organizational
Capability Target 2 Critical
Tasks from page 1]
[Observation notes and explanation of rating]
[Target
rating]
[Insert Organizational
Capability Target 3
from page 1]
• [Insert Organizational
Capability Target 3 Critical
Tasks from page]
[Observation notes and explanation of rating]
[Target
rating]
Final Core Capability Rating: [Enter Total Rating here]
Evaluator Information
Evaluator Name: [Insert]
Evaluator Email: [Insert]
Evaluator Phone: [Insert]
Ratings Key
P: Performed without challenges
S: Performed with some challenges
M: Performed with major challenges
U: Unable to be performed
EEG- Resp-IS
Homeland Security Exercise and Evaluation Program
[Sponsor Organization]
Rev. 2020 508
[PROTECTIVE MARKING, AS APPROPRIATE]
2
Response: Infrastructure Systems
[Exercise Name]
Ratings Definitions
Performed without
Challenges (P)
The targets and critical tasks associated with the core capability were
completed in a manner that achieved the objective(s) and did not
negatively impact the performance of other activities. Performance of
this activity did not contribute to additional health and/or safety risks
for the public or for emergency workers, and it was conducted in
accordance with applicable plans, policies, procedures, regulations,
and laws.
Performed with Some
Challenges (S)
The targets and critical tasks associated with the core capability were
completed in a manner that achieved the objective(s) and did not
negatively impact the performance of other activities. Performance of
this activity did not contribute to additional health and/or safety risks
for the public or for emergency workers, and it was conducted in
accordance with applicable plans, policies, procedures, regulations,
and laws. However, opportunities to enhance effectiveness and/or
efficiency were identified.
Performed with Major
Challenges (M)
The targets and critical tasks associated with the core capability were
completed in a manner that achieved the objective(s), but some or all
of the following were observed: demonstrated performance had a
negative impact on the performance of other activities; contributed to
additional health and/or safety risks for the public or for emergency
workers; and/or was not conducted in accordance with applicable
plans, policies, procedures, regulations, and laws.
Unable to be
Performed (U)
The targets and critical tasks associated with the core capability were
not performed in a manner that achieved the objective(s).
EEG-Resp-IS
Homeland Security Exercise and Evaluation Program
[PROTECTIVE MARKING, AS APPROPRIATE]
3
[Sponsor Organization]
Rev. 2020 508
Emergency Management
Exercise Design
Session 5
Exercise Evaluation
Exercise Evaluation

Homeland Security:

The act of reviewing or observing and recording exercise activity or conduct,
assessing behaviors or activities against exercise objectives, and noting strengths,
weaknesses, deficiencies, or other observations

Should be included incorporated into the exercise planning and design

Is used as the basis for improvement planning
The Homeland Security Exercise &
Evaluation Program Evaluation Process
Evaluation Planning

The process of evaluating an exercise begins with the exercise planning and
design team

The Lead Evaluator should be selected in the initial phases of the exercise
process and participate throughout the exercise design


Should be an experienced senior level individual with knowledge of;

The exercise planning, conduct and evaluation process

Applicable response plans and policies

The Incident Command System / NIMS
The ability to evaluate must be considered when crafting the exercise
objectives
Lead Evaluator
Before The Exercise

Development of the evaluation tools (EEGs, Etc.)

Identifying and training evaluators

Determining the evaluation plan

Assembling resources necessary for the Evauators

Maps

Plans

EEGs

Etc.
Lead Evaluator
During the Exercise

Assures that evaluators are properly located

Provides the evaluators with documentation and other necessary supplies

Makes assistance available to evaluators is/if needed
Lead Evaluator
After The Exercise

Lead the data analysis

Coordinate the evaluation meetings

Be principally responsible for the development of the After Action Report

Coordinate the Improvement Planning Activities
TCL / UTL

Target Capabilities List


The Federal Government recognizes 37 capabilities that government entities at all
levels are expected to support; such as:

Medical Surge

Emergency Public Information and Warning

Search and Rescue
Universal Task List

A comprehensive list of tasks which must be performed in response to an
emergency

The breadth of tasks performed increases with higher levels of government

Can be a reference for development of exercises and objectives
Exercise Evaluation Guides

HSEEP Standardized documents published to provide a consistent tool for
evaluation

Linked to Target Capabilities

Tools for documenting activities and observations

https://preptoolkit.fema.gov/web/hseep-resources/eegs
Evaluators

Evaluators should be non-players, ideally with independent expertise, and not
associated with an agency participating in the exercise

In addition to knowledge of response and preparedness, evaluators should
have experience in conducting an evaluation


Observing and recording the actions and interactions of and between players

Analyzing actions against expected actions and objectives

Assisting in the preparation of the After Action Report
Ideally, Municipalities should consider establishing a cadre of trained exercise
evaluators.
Evaluators

Should be assigned;

To each of the exercise play areas

In accordance with their knowledge and expertise

Must be thoroughly familiar with the exercise, including the objectives, the
exercise plan and the anticipated activities including all injects

Should be issued a Sit Man or a copy of the ExPlan, a copy of the C/E
handbook, the assingments and contact info of other evaluators and should
have access local plans, policies and procedures prior to the exercise.
C/E Briefing

Conducted shortly before the exercise.

A review of the exercise activites

A review of the exercise goal & objectives, and the the evaluation process

Affords an opportunity to ask questions

May include a visit to all play areas
Systematic Observation

Descriptive Reporting: A direct reporting of actions taken and observed

Inferential Reporting: Requires that the evaluator reach a conclusion based on
observations: “Response was timely”, “action was adequate”

Evaluative Reporting: Requires evaluation against a range performance.

Most exercise evaluation is accomplished through descriptive reporting
Levels of Performance Analysis

Task Level


Activity Level


Focus on direct actions
Involve groups of related tesks
Capability Level

High level Actions

Usually involves multiple participants from different agencies
Common Evaluation Issues

Observer Drift – Occurs when the evaluator loses interest

Errors of Leniency – Tendency to rate all points as positive

Errors of Central Tendency – Tendency to rate as aspects as “middle of the
road”

Halo Effect – All exercise activities rated positively based upon the early
positive impression of a person or group
Common Evaluation Issues

Hypercritical Effect – Occurs when the evaluator(s) believe that they are
supposed to find fault

Contamination – Involves evaluator knowledge of other, prior, performance of
the same task(s)

Evaluator bias – Evaluation is affected by the personal boas of the evaluator.
The Evaluator Effect


The mere presence…

Arrive Prior

Document Later
Adjusting actions

They way they should vs. they way they would
Intervening

For Safety
otherwise

Only when absolutely necessary

To gather crucial information not otherwise available

To clarify evaluator confusion
When Intervening

Use common language

Delay if practicable

Minimize time of intervention

Avoid leading or prompting questions

Avoid acting as an advisor
Reducing the Evaluator Effect

Use experienced / trained evaluators

Include training on evaluation issues

Ensure Evaluators are familiar with the documentation prior to the exercise

Evaluators should not hesitate to seek assistance / clarification from the Lead
Evaluator or other Evaluators

Avoid judgments during the exercise

Avoid projecting a negative impression

Avoid conversations about the progress of the exercise
Exercise Evaluation Guides

For Operations based exercises

Facilitate the recording of
 What actions occurred
 Who performed the actions or made the decision
 Where the action or decision took place
 When the action or decision took place
 Why an action was performed or a decision was made
 How players performed the action or made the decision
Collecting information

Participant Feedback forms

Logs (EOC, etc.)

Video or Audio recordings

Recorded phone lines

Player notes

Incident documentation

ICS Documentation

Controller / facilitator notes

Photos (Anyone)
Player Hotwash

Should include


Players, Exercise Design team, Evaluators, Controllers
An opportunity for players to express their impression of the exercise event;
including the response and the exercise design.

As soon as practical after EndEx

Allows Evaluators to seek clarification

May be more than one, should be convenient to all locations of play

Essentially a self-assessment, but should focus on strengths as well as
opportunities for improvement

Should be led by an individual experienced in leading group discussions
Preliminary Analysis

Each Evaluator should attempt to complete a preliminary analysis of the
objectives to which they were assigned.

Where possible, the evaluation should be at the activity and task level.

It should include documentation (description) of significant events occurring
during the exercise

Should include time references
Controller / Evaluator Debrief

Allows Evaluators to review the results of the Hotwash and obtain Controller
input / clarification

Allows Evaluators to compare notes

Allows revision of EEGs or other evaluation documents as / if needed

Facilitates evaluators assigned to the same function area(s) developing an
overall assessment.
Analyzing the Data


1) Identify Issues

Comparing actual performance to expected actions (objectives)

Construct an exercise timeline of events
2) Determining Root Cause of discrepancies
 Use the 5 Whys!
The 5 Whys!
Analyzing The Data

3) Develop Recommendations for Improvement

Identify areas to improve

Consider both long term and short term solutions

Be consistent among all recommendations

Identify sources for assistance

Consider including How and possibly even Who in the recommendation
Examples of RFI

Changes to plans or procedures

Changes to organization structures

Changes to leadership or management processes

Additional or alternate training

Changes to resources or capabilities
Lessons Learned

In addition the formal analysis of the capabilities and tasks against the
objectives, Lessons Learned should be identified

Knowledge gained as a result of an exercise that identifies a practice or
existing knowledge as a tool for use in approaching a similar problem should it
occur in the future.

Should be specifically included in the AAR.

Consider DHS – LLIS
Emergency Management
Exercise Design
Session 6
After Action Reporting
After Action Report

Records what occurred during the exercise

Provides feedback on the achievement, or not, of capabilities and tasks
associated with the objectives

Suggests recommendations for improvement

Developed initially in draft form
After Action Report

Executive Summary

Executive Overview

Exercise Goal(s) and Objectives

Analysis of Capabilities Demonstrated

Conclusion

Improvement Plan
The After Action Conference

For review and review and revision of the draft AAR

Includes the Exercise Planning team, Evaluation Team, and stakeholders

Begin Development of the Improvement Plan


Includes corrective actions and timelines
Produces the final AAR/IP
Corrective Actions

May initially just be first steps

Must accept realistic appreciation of limited resources

Should be both short term and long term

Should have benchmarks

…Number trained

…items purchased

…plans revised

…MOUs completed
Final AAR / IP

Final Acceptance is by the Exercise Design Team

Major Content


Additional Exercise Program Planning

Changes to Plans, Polices and Procedures

Capability development and refinement

Priority focus of limited resources
AAR / IP should always be considered an “FOUO” and the distribution of it
should be agreed upon by the stakeholders