Posted: April 24th, 2025

week one evaluation

 

All of the discussions and written assignments in this course will focus on a program evaluation of one specific non-profit agency.  Each week’s discussion thread will focus on the issues that will be relevant to that week’s written assignment.  Use the discussion as a means to clarify your understanding of the issues related to the topics each week.  This week, you will begin by selecting an agency, interviewing key personnel at the agency regarding the programs they provide, and developing a logic model that provides a graphic overview of the program’s purpose, services, inputs, activities, outputs, and outcomes that will be evaluated. 

Please note that, due to the very short duration of this course, it will likely not be possible to actually collect and analyze data from that agency.  You will have the option to collect agency data (with the permission of the agency, of course) or to create data files using hypothetical data in future weeks.

Review the PowerPoint presentation of a program evaluation conducted at

Program Evaluation for Broward Partnership for the Homeless.pptx

In this discussion thread, please identify the following. Items 1, 2, and 3 are due by Wednesday; item 4 is due by Sunday.

1. The name of the agency that you will be evaluating in this course NOTE: If the agency prefers, you may use a pseudonym for confidentiality purposes. The agency should provide psychological and/or social services and be a not-for-profit agency. 
2. Discuss the ethical issues involved with being a consultant for a social services agency.  How will the confidentiality of the clients be maintained throughout your program evaluation project?  Are there other ethical or legal issues that need to be considered?
3. By the end of the week, you will need to interview at least one key personnel at the agency.  When deciding who to interview, consider the following:

   Who can provide the best information about the services delivered by the agency?
   Who can provide the best information about how the agency currently measures outcomes related to effectiveness of their programs?
   Review the components that are required for your logic model.  The person(s) interviewed should be able to explain the inputs, activities, outputs, and desired short-term, intermediate-term, and long-term outcomes.

Broward Partnership for the Homeless, Inc. (BPHI)

Program Evaluation Study

Single-site program evaluation
Homeless shelter outcomes
1 ½ year duration
2001 – 2003

BPHI
Homeless shelter in Broward County, FL
Approximately 1700 residents per year
Up to one-year stay
Average six months
200 beds
10 family units

BPHI – Services
Case Management
Psychiatric Treatment
Substance abuse, mental health counseling
Job training
Resume Writing
Education
GED
Referrals for educational services
Employment Services

BPHI – Services
Three meals a day
Medical evaluation and treatment
Dental office
Child care
Library
Clothing
Hair salon
Message service

BPHI – Need for Evaluation
Funding agencies require evidence of treatment effectiveness

Donors often require evidence

No data to support effectiveness of services

BPHI – Program Evaluation
Grant from Health Foundation of South Florida

Goals
Increase behavioral health services
Increase engagement in center residents
Improve behavioral stability in center residents
Reduce symptom severity
Increase functionality

Provide evidence of results achieved

Research Design Considerations
Pre-Post design with control group

What is the main purpose of assessment?

Who will do the assessments?

What type of assessment?
Is diagnostic interview necessary?
Assessment versus screening

BPHI – Measures
Public Domain = Free!

CAGE for drug/alcohol screening

Mental Health Screening Form (MHSF)

SOFAS

GAF

BPHI – CAGE
1. Have you felt you ought to cut down on your drinking or drug use?

2. Have people annoyed you by criticizing your drinking or drug use?

3. Have you felt bad or guilty about your drinking or drug use?

4. Have you ever had a drink or used drugs first thing in the morning to steady your nerves or to get rid of a hangover (eye-opener)?

BPHI – MHSF
Mental Health Screening Form (MHSF; Carroll & McGinley, 2000).
Screen for mental health symptoms

Designed for use with co-occurring disorders

17 items

Instrument available
here

BPHI – SOFAS
Social and Occupational Functioning Scale (SOFAS; American Psychiatric Association, 2000)

Focus on social, occupational functioning

Impairment not exclusively due symptom severity

Includes general medical conditions

Case Manager rated at intake and discharge

BPHI – GAF
Global Assessment of Functioning (GAF; American Psychiatric Association, 2000)

Impairment due to symptom severity

Impairment not due to general medical conditions

Rated by therapist/psychiatrist each session

BPHI – Issues
Training

Oversight

Supervision

Data Collection

Data Entry

Results

BPHI

Compared those treated to not treated

↓ Substance abuse symptoms

↓ Psychiatric symptoms

↑ SOFAS scores

↑ GAF scores

↑ attained housing

References
American Psychiatric Association. (2000).  Diagnostic and statistical manual of mental disorders (4th ed., text revision).  Washington, DC:  Author.

Carroll, J.F.X., & McGinley, J.J. (2000). Mental Health Screening Form III. Tampa, FL: Project Return Foundation. Retrieved from http://smchealth.org/sites/default/files/docs/1309587941MentalHealthScreeningFormIII

media1.m4a

image2

image3

media2.m4a

media3.m4a

media4.m4a

media5.m4a

media6.m4a

media7.m4a

media8.m4a

media9.m4a

media10.m4a

media11.m4a

media12.m4a

media13.m4a

media14.m4a

media15.m4a

media16.m4a

Evaluating BPHI’s Impact on Homelessness

Student’s name

Instructor

Course

Date

Evaluating BPHI’s Impact on Homelessness

The purpose of this course is to conduct a program evaluation of Broward Partnership for the Homeless, Inc. (BPHI), a nonprofit agency providing services to homeless individuals in Broward County, Florida. Some of the services provided by BPHI include case management services, psychiatric care, vocational training, and educational referrals. A part of the assessment was to develop a logic model that describes the program structure of BPHI, including the magnitude, activities, outputs, and outcomes of the program or organization (Fink, 2015). In addition, hypothetical data would be used to represent the actual data because of limited time. The model will demonstrate that the systematic approach shows how the resources and actions of BPHI result in measurable impacts through a systematic approach that can provide real effectiveness about program performance and areas for potential improvement.

A number of ethical issues emerge throughout the course of a program evaluation in a social service organization. The consultants have to convince the clients that confidentiality of information is assured throughout the project, and data collected is not of a nature to reveal identities of particular individuals or clients. In this respect, the evaluation of BPHI uses measures for which data is available to reflect effectiveness without compromising privacy. More than that, informed consent from both the agency and clients, should any client information be involved, needs to be obtained by the evaluators in line with guidelines on the respect of the rights of the clients and policies of the agency.

Further ethical considerations are control of likely biases or conflicts of interests that may arise in the process of evaluation. Evaluators have to be objective and avoid any action that might alter results or damage the reputation of the agency. This is borrowed from Fink, who posited that the review to be carried out should be free from blemish, value-free, and supplemented, which appreciates openness in neutrality when confirming the effectiveness of a specific program. The other type of concerns present could be legal; for example, data protection legislation. These are significant in implementing guidelines for sensitive information about customers and in upholding ethical concern.

I will interview a key personnel member with an in-depth knowledge of the agency’s services and outcome measurement practices that will give insight into how the agency operates. This person will be able to provide information on the program structure, inputs, and intended outcomes necessary to develop a comprehensive logic model. In developing an appropriate evaluation framework, consultation with knowledgeable personnel is indispensable since they are the ones who would add nuances to program effectiveness and areas that may warrant change according to Fink (2015). If information is integrated into the logic model, the study would represent the agency’s goals by highlighting immediate outputs to long-term impacts on client wellbeing.

References

Fink, A. (2015).
Evaluation fundamentals: Insights into program effectiveness, quality, and value (3rd ed.). Thousand Oaks, CA: Sage.

Evaluating BPHI’s Impact on Homelessness

Student’s name

Instructor

Course

Date

Evaluating BPHI’s Impact on Homelessness

The purpose of this course is to conduct a program evaluation of Broward Partnership for the Homeless, Inc. (BPHI), a nonprofit agency providing services to homeless individuals in Broward County, Florida. Some of the services provided by BPHI include case management services, psychiatric care, vocational training, and educational referrals. A part of the assessment was to develop a logic model that describes the program structure of BPHI, including the magnitude, activities, outputs, and outcomes of the program or organization (Fink, 2015). In addition, hypothetical data would be used to represent the actual data because of limited time. The model will demonstrate that the systematic approach shows how the resources and actions of BPHI result in measurable impacts through a systematic approach that can provide real effectiveness about program performance and areas for potential improvement.

A number of ethical issues emerge throughout the course of a program evaluation in a social service organization. The consultants have to convince the clients that confidentiality of information is assured throughout the project, and data collected is not of a nature to reveal identities of particular individuals or clients. In this respect, the evaluation of BPHI uses measures for which data is available to reflect effectiveness without compromising privacy. More than that, informed consent from both the agency and clients, should any client information be involved, needs to be obtained by the evaluators in line with guidelines on the respect of the rights of the clients and policies of the agency.

Further ethical considerations are control of likely biases or conflicts of interests that may arise in the process of evaluation. Evaluators have to be objective and avoid any action that might alter results or damage the reputation of the agency. This is borrowed from Fink, who posited that the review to be carried out should be free from blemish, value-free, and supplemented, which appreciates openness in neutrality when confirming the effectiveness of a specific program. The other type of concerns present could be legal; for example, data protection legislation. These are significant in implementing guidelines for sensitive information about customers and in upholding ethical concern.

I will interview a key personnel member with an in-depth knowledge of the agency’s services and outcome measurement practices that will give insight into how the agency operates. This person will be able to provide information on the program structure, inputs, and intended outcomes necessary to develop a comprehensive logic model. In developing an appropriate evaluation framework, consultation with knowledgeable personnel is indispensable since they are the ones who would add nuances to program effectiveness and areas that may warrant change according to Fink (2015). If information is integrated into the logic model, the study would represent the agency’s goals by highlighting immediate outputs to long-term impacts on client wellbeing.

References

Fink, A. (2015).
Evaluation fundamentals: Insights into program effectiveness, quality, and value (3rd ed.). Thousand Oaks, CA: Sage.