A Programme Evaluation PROCESS EVALUATION of the E.A.S.Y. Programme.

Guidance Councillors and parents/caregivers were required to be a part of the programme. Parents were expected to provide additional background information and to assist with the re-socialization process, while Guidance Councillors would be instrumental in ensuring that the boys attended the sessions and also in assessing their progress as the programme went on.

The students in the programme would be assessed both before and after the delivery of the programme. Intervention methods were developed through the use of tests and interviews.

The E.A.S.Y. Programme was administered over the course of 3 months, between January and March 2004. The 24 students were brought to St. Georges College where the intervention would take place on two levels. On one level, the students would participate in seminars on topics such as anger, aggression, thought controlling techniques, behaviour controlling techniques, effective communication, enhancing coping skills, and how to prevent relapse. Within these seminars students were asked to complete personal assignments, and do activities such as modelling, role-play, teamwork, self- and peer-evaluation, and constant participant involvement. The objective was to teach and promote new behaviours to effect lasting change of the inappropriate and unproductive thought and behaviour patterns.

On the other level of the programme, the students were subjected to psychometric tests. The Adolescent Psychopathology Scale was the main test employed. This instrument is designed to evaluate the presence, and severity, of important domains of psychopathology in adolescents. The test has scales that have a high level of content validity with respect to the DSM-IV for the following disorders:

* Conduct Disorder (CND)

* Oppositional Defiant Disorder (OPD)

* Major Depression (DEP)

* Generalised Anxiety Disorder (GAD)

* Post-traumatic Stress Disorder (PTS)

* Substance Abuse Disorder (SUB)

The JobBank Personality Profile was also administered so as to ascertain the individual's personality type, thereby allowing further insight into the forces driving the individual's behaviour.

The JobBank PERSONALITY PROFILE measures the following:

l. PSYCHOLOGICAL TYPE AS RELATES TO SELF, OTHERS AND WORK

By measuring how people perceive, and judge the world around them, we can reliably type people as

EXTROVERT INTROVERT

SENSING INTUITIVE

THINKING FEELING

PERCEIVING JUDGING

The SRA(r) Reading Index (RAI) is a test of general reading and computational achievement for adults and for young people over fourteen years of age. The RAI-12 was administered to attempt to predict school and job performance.

The SRA(r) Reading measure distinct levels of development. The Reading Index progresses through the following six levels:

* Picture-Word Association

* Word Decoding

* Phrase Comprehension

* Sentence Comprehension

* Paragraph Comprehension I

* Paragraph Comprehension II

Bar-On-Emotional Quotient Inventory was used to measure an array of emotional, personal and interpersonal abilities that may influence the adolescent's ability to cope with environmental demands and pressures. This was used because the Baron EQ is an important factor in determining one's ability to succeed in life and directly influences one's general emotional well being. The well-functioning, successful and emotionally healthy individual is one who possesses a sufficient degree of emotional intelligence.

Dr. Leahcim Semaj, the programmes creator, lead these sessions with the input of JobBank staff members and the Guidance councillors from the schools that participated in the exercise. Participants signed a contract agreeing to participate in the programme or to accept the consequences of the escalating antisocial behaviours. Parents, Care givers and/or siblings were also mandated to participate and to provide additional background information and to assist with the resocialization process.

THE EVALUATION

The process evaluation for the E.A.S.Y. programme is a comprehensive look at the characteristics and experience; student participation, exposure, and reaction to the programme; student perceptions of barriers to and successes of the program; written reports about the project; perceptions of problems, difficulties, and barriers; facilitator training evaluation; and observer documentation of session content. The various measures were collected through a variety of instruments and methods, both qualitative and quantitative. Data for this paper came from the four primary sources: 1) self-administered questionnaires, 2) Questionnaires completed by Guidance Councillors and other observers, 3) Observation of the sessions by Dr. Semaj and his team 4) Results of psychometric tests

Dr. Semaj and his team subjected the participants to various assessments before the seminars in the programme began. This was to ascertain what level the students were at as it regards General reading and computational ability (SRA(r) Reading Index) Personality Type ( The JobBank Personality Profile) , Domains of Psychopathology ( Adolescent Psychopathology Scale), Emotional, personal and interpersonal abilities that influence would influence the students ability to cope with environmental demands and pressures(Bar-on-Emotional Quotient inventory). These same assessments were administered at the end of the programme so that they could compare results Pre exposure to the programme against Post exposure.

Facilitator Questionnaires

Each facilitator was asked to complete a questionnaire requesting background information (e.g., age, level in school, ethnicity, and experience with adolescents), prior experience with and attitudes toward the curriculum topics (e.g., alcohol and drug use), and their perceived self-efficacy about influencing others. Analysis of background data provided a description of the facilitator population. The facilitations included trained Guidance Councillors along with Dr. Semaj and first degree level psychology university graduates.

Students Questionnaires

The students were given questionnaires to assess themselves before and after the programme. This questionnaire was similar to the one that the guidance councillors completed so that comparison could be done between the students and the facilitators views on the effects the programme had been having.

Training Evaluation

The facilitators were talked to by Dr. Semaj who gave them specific instructions on how to talk to the students and the possible problem they may encounter. He instructed facilitators to be cogizant of the background of the students and the reasons they were there. Some of the participants were thought to be suffering from depression and other emotional disorders so it was important for facilitators to always are conscious of the words they used and for them to look out for behaviour that the questionnaires and tests would not record. Sessions were poorly attended by the guidance councillors and when they did attend was frequently guilty of being late.

Observation of Session Content

The sessions were observed on two occasions. Observation was based on:

. Gaining a better understanding of the way in which the programme was administered and

2. Observing student behaviour including group behaviour, attention, willingness to participate, anger management and discussion of concepts such as understanding external and internal emotional triggers, how thoughts help create anger, and the costs and benefits of anger and aggression

FINDINGS FROM THE EVALUATION

All the students in the programme completed the psychometric tests SRA(r) Reading Index, The JobBank Personality Profile, A.P.S., and Bar-on-E-Qi, Both at the beginning of the programme and again when it had come to an end.

Results from the Adolescent Psychopathology Scale (APS) are found in Table 1. The test results of the students before and after the programme indicated that on eight (8) of nine (9) disorders the average scores had been reduced. The results from the Bar-on EQ-I are found in Table 2. The students had an average EQ of 92.3 which increased to 98.0 after the programme. The table also shows that the emotional quotient of the students increased on all dimensions.

Implementation Reported by Guidance Counsellors

Guidance Councillors from the schools involved were required to be a par1t of the programme. As such they were asked to fill out 2 evaluation forms, one to assess the individual students and their behaviour with others in the programme and the other for the programme overall. As an example, Tables 3-6 show areas of the Guidance Counsellors analysis including Improvements in Anger Management and Changes in Student Behaviour. The Guidance Councillors note that there was an improvement in behaviour for 58.3% of the students. Also they note that other people noted an improvement in the behaviour of 75% of the students.

Implementation Reported by the Students

A third method of programme assessment had a survey being administered to the students to determine what they thought of the program and its impact on them. From the survey we found that 85.7% of the students believed that their anger gets them in trouble. Table 7 shows this.

IMPLICATIONS OF THE EVALUATION

Analysis of the E.A.S.Y. Programme over the course of its pilot year has shown that it is a thoroughly thought out programme but it has also left questions to be answered relating to its structure among other things. Although Guidance Councillors were given training and were spoken to more than once about their importance to the programme, their turnout to the sessions was far below what it should have been. Also the counsellors were expected to be instrumental in ensuring that the students came to the sessions. However one of the major problems with the programme was the serious lack of committed participation by the students. While the programme had admitted 24 students, only 8 boys attended more than 10 sessions out of the 24. On the parents' side, only 3 parents attended any sessions at all and the3 only attended one session each.

The data showed concretely that the programme had scope for success but that this success was directly related to the number of sessions attended by the students. The students with the highest rating of improvement were those who attended more than 10 sessions', followed by those who attended 5 - 9 sessions. None of those boys who attended fewer than 4 sessions had evidenced any significant improvement in their behaviour or their anger management skills.

Another reason that some content/activity areas were executed poorly was because sessions were scheduled in the days when students had classes so in many instances students did not want to miss their classes and therefore did not attend E.A.S.Y. sessions. Even though all the sessions were essential to the programme students tended to act as if the activities were optional, so they would attend when they wanted to.

Implications for the E.A.S.Y. Programme

Process evaluation can provide information on whether strategically selected programme constructs were actually emphasized in the implementation as intended. For the EASY programme, anger management was the construct of primary importance, and in every instance it was implemented in the group sessions. In the sessions attended and in the student and facilitator reports anger and controlling this emotion were at the forefront of the efforts. Students and councillors alike report on the effectiveness of this emphasis on anger management. Table 6 shows that 60 % of the facilitators see an improvement in the anger management of the students in the programme and Table7 shows that 85.7% of the boys admitted that anger gets them in trouble. Therefore it is the belief of the evaluator, based on the data, that reducing anger is a good way to go in correcting the deviant behaviour. Also the data would suggest that this method, properly executed is a good method.

CONCLUSION

After analyzing all of the data it seems as if the focus on anger management has been successful in reducing the delinquency and deviancy which was the main goal. Qualitative methods, such as what the facilitators believe may have been possible barriers and contributors to implementation, should have been included.(n10) Sessions should have been kept at a time out of school hours so that students would not have to miss class time to attend.

All of the observations, measured quantity of implementation, and not quality of implementation.(n18) For example, great variance in results from student to student due to differences in factors beyond the control of the facilitators.(e.g. some students may have just wanted to change their behaviour, therefore attending more sessions etc.) Training for session observers and facilitators should have been more formalized and had a specific structure. Students should be mandated to attend these sessions by the Ministry since data showed that success was higher for students who attended more sessions.

The programme itself needs to conduct a programme evaluation so that it can improve on itself in the future. It must look at what is thought to be crucial to improving outcomes and working on it. Though process evaluation can be often time tedious, and time-consuming, the insights gained and the benefits that can simply be derived can easily outweigh the costs.

APPENDIX

Table 1: Number of students with Disorders before and after the intervention

Table 1: Number of Students With Disorder

Disorders

Pre-Test

Post-Test

Students

Student %

Students

Student%

Major Depression

(DEP)

2

52.2%

3

37.5%

Conduct Disorder

(CND)

1

47.8%

3

37.5%

Substance Abuse Disorder

(SUB)

9

39.1%

2

25.0%

Anger/Violence Proneness

(AVP)

9

39.1%

2

25.0%

Interpersonal Problems

(IPP)

9

39.1%

2

25.0%

Academic Problems

(ADP)

6

26.1%

2.5%

Self-Concept

(SCP)

5

21.7%

0

0.0%

Oppositional Defiant Disorder

(OPD)

4

7.4%

3

37.5%

Generalized Anxiety Disorder

(GAD)

4

7.4%

2.5%

Table 2: Mean Baron EQ Scores

Mean

Pre-Test

Post-Test

Total EQ

92.3

98.0

INTRA-PERSONAL

92.9

98.3

INTER-PERSONAL

82.7

86.2

STRESS MANAGEMENT

99.4

04.8

ADAPT-ABILITY

92.2

97.6

Table 3: Guidance Counsellor noted Improvement in Students' Behaviour

Table 4: Others noted Improvement in Student's Behaviour

Table 5: Effect of Programme on Student's Participation

Table 6: Effect of Programme on Anger Management

Table 7: Belief that Anger Gets Him in Trouble.

The E.A.S.Y. Programme

Educating, Assessing & Socializing the Youth

HELLO! Please complete the following questions so that we will be better able to assess the progress of the student in the programme. Answer all questions unless otherwise instructed.

STUDENT'S NAME: SCHOOL:

. Comment on the relationship between the student and the following person(s) over the last two (2) months?

Select relevant options

* Teachers ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

* Principal/Vice-Principal ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

* Prefects ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

* Class Monitor ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

* Other Classmates ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

* Younger Students ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

* Older Students ? Very Poor ? Poor ? No Relationship ? Good ? Very Good

Why?

2. Have there been any improvements in his relationship with any of the following persons in the last two (2) months? Select relevant options

Teachers ? Yes ? No

Principal/Vice-Principal ? Yes ? No

Prefects ? Yes ? No

Class Monitor ? Yes ? No

Other Classmates ? Yes ? No

Younger Students ? Yes ? No

Older Students ? Yes ? No

3. How would you best describe the student in question?

? Reclusive ? Friendly ? Aggressive ? Mild Mannered ? Immature ? Mature

? Other

4. Is the student a member of any gang? ? Yes ? No

IF YES, what is the name of the gang?

5. Are the student's friends considered 'bad company'? ? Yes ? No

Has he continued to hang-out with them over the last two (2) months? ? Yes ? No

6. Is he a member of any school club/organization or sports team? ? Yes ? No

Name the club(s) or sport(s)

7. Has the student been reported for the use of any of the following substances in the last two (2) months:

* Marijuana (ganja) ? Yes ? No

* Cigarette ? Yes ? No

* Cocaine (Coke) ? Yes ? No

* Crack ? Yes ? No

* Alcohol ? Yes ? No IF YES, what?

Other

8. Have you noticed any improvements in his behaviour in the last two (2) months? ? Yes ? No

IF YES, please explain.

9. Has the student's participation in class activities/discussions changed in the last two (2) months? ? Increased ? No Change ? Decreased

0. Has anyone noted any behavioural improvement in this student? ? Yes ? No

1. How Often does he get into fights at school since attending Sessions?

? Very Often ? Sometimes ? A Few Times ? Never

Answers to Question 12 should be based on the student's last report card

2. What is the student's average grade? (%)

In what subject does the student perform the BEST? Grade: (%)

What subject(s) has the student failed on his last report?

3. Has the student been suspended from school since the start of the E.A.S.Y. programme? ? Yes ? No

IF YES: How many times?

For what reason(s)?

4. In your opinion, have there been improvements with how the student deals with his/her anger? ? Yes ? No

IF YES, How so?

Name: Post:

Date:

u

References

(n1.) Steckler A, McLeroy KR, Goodman RM, Bird ST, McCormick L. Toward integrating qualitative and quantitative methods: an introduction. Health Educ Q. 1992;19:1-9.

(n2.) Israel BA, Cummings KM, Dignan MB. Evaluation of health education programs: current assessment and future directions. Health Educ Q. 1995;22:364-389.

(n3.) McGraw SA, Stone EJ, Osganian SK, et al. Design of process evaluation within the Child and Adolescent Trial for Cardiovascular Health (CATCH). Health Educ Q. 1994;2(suppl):5-26.

Leahcim Semaj-Interview

(n4.) Helitzer DL, Davis SM, Gittelsohn J, et al. Process evaluation in a multisite, primary obesity-prevention trial in American Indian schoolchildren. Am J Clin Nutr. 1999;69(suppl):816-824.

(n5.) Dehar M, Casswell S, Duigan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17:204-220.

(n6.) Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87-108.

(n7.) Chen H. Theory-Driven Evaluations. Newbury Park, Calif: Sage Publications; 1990.

(n8.) Weiss CH. How can theory-based evaluation make greater headway? Eval Rev. 1997;21(4):501-524.

(n9.) Weiss CH. Theory-based evaluation: past, present, and future. New Dir Evaluation. 1997;76:41-55.

(n10.) Rossi PH, Freeman HE. Evaluation: A Systematic Approach. 5th ed. Newbury Park, Calif: Sage Publications; 1993.

(n11.) Roccella. Considerations for evaluating community high blood pressure control programmes. J Hum Hypertens. 1996;1(suppl): 17-19.

(n12.) Scheirer MA, Shediac MC, Cassady CE. Measuring the implementation of health promotion programs: the case of the Breast and Cervical Cancer Program in Maryland. Health Educ Res. 1995;10(1):11-25.

(n13.) van Assema P, Steenbakkers M, Eriksen M, Kok G. The process evaluation of a Dutch community health project. Int Q Commun Health Educ. 1994-5; 15(2): 187-207.

(n14.) Eggert LL, Thompson EA, Herting JR, Nicholas LJ, Dicker BG. Preventing adolescent drug abuse and high school dropout through an intensive school-based social network development program. Am J Health Promo. 1994;8(3):202-215.

(n15.) Scott DM, Merkel PA, Barlow TW. Process evaluation of Nebraska's team training project. J Drug Educ. 1994;24(3):269-279.

(n16.) Elder JP, McGraw SA, Stone EJ, Harsha D, Wambsgans K, Greene T. CATCH: process evaluation of tobacco policies, food services, and secular trends in CATCH. Health Educ Q. 1994;2(suppl): 107-127.

(n17.) Elder JP, Woodruff SI, Sallis JF, de Moor C, Edwards C, Wildey MB. Effects of health facilitator performance and attendance at training sessions on the acquisition of tobacco refusal skills among multi-ethnic, high-risk adolescents. Health Educ Res. 1994;9(2):225-233.

(n18.) Basen-Engquist K, O'Hara-Tompkins N, Lovato CY, Lewis MJ, Parcel GS, Gingiss, P. The effect of two types of teacher training on implementation of Smart Choices: a tobacco prevention curriculum. J Sch Health. 1994;64(8):334-339.

(n19.) Wickizer TM, Von Korff M, Cheadle A, Maeser J, Wagner EH. Activating communities for health promotion: a process evaluation method. Am J Public Health. 1993;83(4):561-567.

(n20.) Lester C, Donnelly P, Weston C. Is peer tutoring beneficial in the context of school resuscitation training? Health Educ Res. 1997;12(3):347-354.

(n21.) Kraiger K. Integrating training research. Training Res J. 1996;1(1):5-16.

(n22.) Gagne RM. Learning processes and instruction. Learning Res J. 1996;1(1):17-28.

(n23.) Quinones MA, Ford JK, Sego DJ, Smith EM. The effects of individual and transfer environment characteristics on the opportunity to perform trained tasks. Training Res J. 1996;1(1):29-48.

(n24.) Drug Free Schools and Communities Program of NM's Children, Youth Fa

milies Dept, Risk Reduction Services Division. Albuquerque Public Schools Summary - Tobacco, Alcohol and Drug Survey. Albuquerque, NM: Research & Polling, Inc.; 1994.

(n25.) Wallerstein N, Bernstein E. Empowerment education: Freire's ideas adapted to health education. Health Educ Q. 1988;15:379-394.

(n26.) Wallerstein N, Sanchez-Merki V. Freirian praxis in health education: research results from an adolescent prevention program. Health Educ Res. 1994;9(1):105-118.

(n27.) Hall GE, Hord SM. Change in Schools: Facilitating the Process. Albany, NY: State University of New York Press; 1987.

~~~~~~~~

By Deborah Helitzer; Soo-Jin Yoon; Nina Wallerstein and Lily Dow y Garcia-Velarde

Deborah Helitzer, ScD, Research Assistant Professor/Director, Office of Evaluation; or <[email protected]>; Soo-Jin Yoon, MA, Research Scientist, Office of Evaluation; and Nina Wallerstein, DrPH, Associate Professor/Director, Masters in Public Health Program, Dept. of Family and Community Medicine, University of New Mexico, 2400 Tucker NE, Family Practice Center, Room 176, Albuquerque, NM 87131; and Lily Dow y Garcia-Velarde, PhD, Assistant Professor, College of Health and Social Services, Dept. of Health Science, New Mexico Southern University, P.O. Box 30001, Las Cruces, NM 88003. This research was supported by NIAAA Grant #1 RO1 AA08943-01A3. This article was submitted September 8, 1999, and revised and accepted for publication November 22, 1999.

National Institute of Justice. Learning from Demonstration Programs. Washington, DC: Paper

prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates Inc. p.12.