Construction of a questionnaire to measure
Construction of a questionnaire to measure the personality characteristic openness to experience, in relation to the job of a probation officer.
Abstract
Questionnaire constructed to measure the personality characteristic openness, as defined by Costa & McCrae (1977), to aid an occupational psychologist screening applicants for the job of a Probation Officer. Questionnaire consisted of 30 statements, with responses indicted on a 5 point likert scale. 50 Participants selected using an opportunity sample from the University of Teesside. 15 of the 50 were re-tested, and completed Goldberg's (1990) openness questionnaire. Reliability was tested for, using Cronbach's Alpha, giving a value of 0.8212 for the final scale at the 0.8 level. The test-retest reliability correlation coefficient was 0.93, and the concurrent validity was not significant at the 0.75 level, with a value of 0.541. The factor analysis suggested that the questionnaire had one underlying factor, which was interpreted as 'Openness'.
Introduction
The job that has been chosen by the researchers is a Probation Officer. The job entails working with offenders to decrease their chance of re-offending; protecting the public against criminal offences; to assist in community punishments; making offenders aware of the effects the crime committed has on others; and rehabilitating offenders. The personality characteristic that has been related to the job is openness to experience.
Two different professionals have defined openness to experience in two different ways; Goldberg (1990) measured it from an intelligence point; While Costa and McCrae (1977) argued that openness to experience consists of six factors; fantasy, aesthetics, feelings, action, ideas, and values. Fantasy is defined as imaginative; aesthetic in philosophical terms means art and beauty; and feelings are mental and physical awareness's of emotional issues. Actions are ways of successfully completing tasks; ideas are "a persons perception of something" (Webster's 1999 pg161); and values are an individuals moral principles.
Openness to experience is essential to the chosen job, as the probation service core values are centred around treating people fairly, openly, and with respect. Other important issues are valuing diversity; having a strong belief in the principle that people can change; to learn from experience; have the ability to relate to others; and to work in difficult situations with sensitivity. A probation officer must be able to successfully work as a member of a group, be "none discriminatory, and avoid being influenced by personal bias" (The Probation Service 2004 pg 7). They must also be open to new ideas and flexible to change, and be able to consider others when solving problems. As is clear from this job description from the North West Probation Service (http://www.probation-northwest.co.uk/role.htm), the majority of the Probation Service's values are linked back to openness to experience in relation to Costa and McCrae's (1977) six factors. Without having openness to experience, a probation officer would be biased towards people different to themselves and have problems understanding other people's life experiences, which could prove fatal when rehabilitating offenders. When researching for empirical studies in this area, a major lack of research was discovered. No studies could be found which directly related openness to experience and probation officers.
The researchers have therefore decided that research is needed to support the claims of the probation service when recruiting new officers. However, for the purposes of this study the researchers have chosen to take the description given by the probation service as sufficient evidence to associate openness with the probation officer.
The aim of the study was to screen applicants for jobs in Probation, by constructing a questionnaire testing for openness to experience which an occupational psychologist could use during the screening process. Interviews and other questionnaires would also be used to aid the decision process.
Statistical tests will be used to ensure the questionnaire is both reliable and valid. An internal consistency test will be used to ensure that each item is measuring the intended personality trait. A Cronbach's alpha level is generated, which must be above 0.8 to be reliable. Test-retest reliability will also be carried ...
This is a preview of the whole essay
The aim of the study was to screen applicants for jobs in Probation, by constructing a questionnaire testing for openness to experience which an occupational psychologist could use during the screening process. Interviews and other questionnaires would also be used to aid the decision process.
Statistical tests will be used to ensure the questionnaire is both reliable and valid. An internal consistency test will be used to ensure that each item is measuring the intended personality trait. A Cronbach's alpha level is generated, which must be above 0.8 to be reliable. Test-retest reliability will also be carried out to ensure the questionnaire is reliable. This involves the questionnaire being reissued two weeks later to 15 participants (out of the original 50), and their responses being correlated against their original scores. This will be tested using Pearson's correlation co-efficient, and a value of 0.8 or above will be required for a significant correlation.
Two measures will be used to test for validity; content validity and concurrent validity. Content validity will be tested for by a professional in the subject area, as they have the knowledge to decide if the items relate to openness. The test for concurrent validity involves correlating the responses against the responses given in an established questionnaire. Goldberg's (1990) questionnaire for openness to experience will be used for this test. A high correlation of above 0.75 demonstrates that both questionnaires are testing for the same thing; however there is no way to tell what is being tested without performing a factor analysis. A factor analysis will identify any similarities the statements may have. Through this the researchers will be able to identify a linking theme, and conclude whether the questionnaire is testing for the intended characteristic.
Participants
The participants were students from the University of Teesside, selected using an opportunity sample. There were 50 participants (n=50), 15 of which took part in a re-test two weeks after initially taking part (n=15). The same15 re-tested participants also completed Goldberg's (1990) questionnaire. In total there were 22 female participants (n=22), who ranged from ages 18 to 40. The mean age of the female participants was 22. There were 28 male participants (n=28), with a mean age of 28. The age range for the male participants was 18 to 40.
Materials.
The pilot questionnaire was created by the researchers to screen potential probation officers. The personality characteristic of openness was researched, and the researchers then formed statements which indicated a person's level of openness. Openness was defined as comprising of six factors; fantasy, aesthetics, feelings, action, ideas, and values. A person who scored highly on each of these areas would be described as being open to experience. The statements were screened both by the researchers and by a professional in the area, and were formed around the six components of the characteristic.
Equal pools of statements were formed for different facets of the characteristic. The questionnaire contained equal numbers of positive and negative statements to balance the pool of potential results. The response format used was a 5 - point likert scale with responses ranging from Strongly Agree to Strongly Disagree.
The pilot was to be compared to Goldberg's (1990) questionnaire for openness in order to test for validity. The scale used in Goldberg's questionnaire was a 10-point numbered scale, with bi-polar adjectives at each extreme. The scores for each participant would be totalled, with high scores demonstrating closedness, and low scores indicating openness. The Cronbach's Alpha of Goldberg's scale for openness is 0.88.
Procedure
The pilot questionnaire was given to 50 participants from the University of Teesside, using an opportunity sample. Participants were debriefed after the questionnaire was completed.
The rights of the participant were protected as they had the opportunity to withdraw at any time, and their details were kept confidential. They were given an identification number and a researcher's email address. If the participant wished to withdraw, they could email the participant the number and their information would be removed from the database, without having to give any personal details.
Results
Item Analysis and Reliability
In order to discover how reliable the questionnaire was in testing for openness, an item analysis and internal consistency analyses were carried out. Internal consistency was conducted to ensure that all items were all testing for the same variable. This was done using Cronbach's alpha, where the criteria set for an acceptable internal consistency was a minimum of 0.8.
Items were removed from the scale if they failed to meet either of the following;
Firstly, an item-total correlation that was positive and equal to or greater than 0.3;
and a standard deviation of 0.5 or greater. The item-total correlation
should be equal to or greater than 0.3, as it shows that each statement is
measuring the same variable. The standard deviation should be over 0.5 as it
shows that the distribution of scores for an item are spread out, showing
variability in responses.
After the first analysis, the Cronbach's Alpha value was 0.7075, which is under
the specified acceptable level. Also, several of the statements showed
item-total correlations of less than 0.3. After carrying out the item analysis and
studying the item-total correlations, 19 statements were removed leaving the statements
which met the above criteria. Following this, the Cronbach's Alpha value was 0.8212,
which is above the 0.8 level, leaving the final questionnaire with 11 items.
The following diagram demonstrates the possible range of scores for the final scale;
Low Openness
1 33 55
High Openness
The table below shows the mean scores for all 50 participants for the final scale. Included are the statements included in the final questionnaire for reference purposes.
As is shown in the table, participants tended to score above the mid point, meaning the average participant indicated a high level of openness.
Mean Scores
Standard Deviation
. I feel it is important to be honest to others and myself
4.4800
0.6141
2. I tend to disagree with people who have different opinions to my own
2.9000
.0926
3. I prefer to be around people similar to myself
2.6800
0.9134
4. I am always open to new ideas
3.9600
0.6688
5. I prefer not to see public displays of affection
3.5800
0.9278
6. I enjoy new challenges
4.0200
0.6543
7. I am always interested in hearing other peoples experiences
3.8000
0.8571
8. We can all learn something from people who have different opinions to that of our own
4.0400
0.8071
9. I tend to bend the truth to spare peoples feelings
2.6800
0.9134
0. I believe that there is potential in everyone and everyone has something to offer.
4.1400
0.7001
1. I have difficultly relating to people who have different opinions to myself.
3.5200
0.9739
Total Mean Score
39.800
The final Questionnaire was then re-tested for reliability, using Pearson's correlation co-efficient. Again, a level was set at 0.8 to indicate an acceptable correlation between the two scores for each participant. There was a significant positive correlation between participants responses on the two occasions, (r=+0.93, df=13, p<0.001, two-tailed). This exceeds the level for a significant correlation, therefore indicating that the scale is reliable.
Validity
The questionnaire was tested for two types of validity; content validity and concurrent validity. The content validity of the questionnaire was checked by a professional in the area, as they had the knowledge to identify any possibly problematic statements.
Next, the questionnaire was tested for concurrent validity, using Goldberg's (1990) openness questionnaire. The total scores for the final questionnaire were tested against the responses of 15 participants to Goldberg's questionnaire, using Pearson's correlation co-efficient to identify any relationship between the two.
The level set to indicate an acceptable correlation was 0.75. From the analysis, it is clear that there was no significant positive correlation between participants responses on the new questionnaire and Goldberg's openness scale, (r=+0.541, df=13, p<0.001, two-tailed). This value of 0.541 is less than the set level, meaning the questionnaires are testing for two different variables.
Factor Analysis
A factor analysis was carried out in order to identify how many factors the items in the final questionnaire had. Principal axis factoring was used, with a direct oblimin rotation.
The table below shows the factors contained in the factor solution;
Factor
Eigenvalues
Percentage Variance Explained
Cumulative Percentage Variance Explained
Factor 1
3.624
32.947
32.9247
Factor 2
.040
9.451
42.398
It is clear from the table that there is one major factor involved in this questionnaire, with another relatively minor factor possibly emerging. It is also necessary to analyze the Scree plot to ensure only the dominant factors are included in the analysis.
It is possible that both methods are valid for discovering the number of factors in the questionnaire, however for the purposes of this research the scree plot will be used in the analysis.
It is clear from the scree plot that there is one definite change in the gradient of the line, indicating only one factor.
In order to analyse the given factor, and what weight the factor has, it was important to look at the table of summarised factor loadings to discover which statements had more of an influence. The table below shows the key statements involved in the generation of the given factor,
Statement
Factor Loading
1. I am always open to new ideas.
0.725
30. I have difficultly relating to people who have different opinions to myself
0.703
8. I tend to disagree with people who have different opinions to my own.
0.660
2. I prefer not to see public displays of affection.
0.655
7. I am always interested in hearing other people's experiences.
0.577
0. I prefer to be around people similar to myself.
0.547
Through analyzing the content of the items which contributed to the generation of the factor, the given factor is named "Openness", as each of the statements in the table for factor loadings relates in some way to the criteria set in the initial stages of questionnaire construction.
Discussion
The results showed the questionnaire to be reliable, with a Cronbach's Alpha of 0.8212. However, it is not valid in terms of Goldberg's openness questionnaire which could be the result of the researchers using differing definitions of 'openness'. The questionnaire was found to have content validity, as this was the opinion of the professional in the field.
The questionnaire was shown to have one factor on the scree plot, however in the table there were two factors with eigenvalues of above 1. The researchers chose to use the scree plot when analysing the factors, as the gradient was much more significant for the first than the second. The researchers named the factor 'openness' after finding it to be the link between the statements; as they all related in some way to the definition of openness given prior to the questionnaire's construction.
In terms of the data collection, the researchers faced some difficulty in acquiring adequate numbers of participants to take part. This was probably due to personal commitments beyond the researcher's control.
More consideration could be put into the questionnaire itself, as the high number of statements which had to be withdrawn from the pilot study limited the data collected from the participant's. If this had been more successful, the study would have given more reliable results and greater chance of replicating the findings.
Although a space was left at the bottom of the questionnaire, none of the participants left any further comments.
As this questionnaire was constructed as an aid for the occupational psychologist, it is important to recognise alternative characteristics required to become a successful probation officer. These other personality characteristics include an ability to relate to others; an ability to work comfortably within both group and individual situations; and to be able to make confident assessments of the lives of offenders. When choosing the most suitable applicant, along with the above, it is important for them to possess the appropriate social work skills and have experience within the field.
The scale constructed within this research could be applied to careers within the field of social work, for example teachers, nurses and prison officers.
The use of personality trait theories as part of the job selection process has been widely criticised. Most importantly, it has been widely documented that the tests used to measure personality traits are often misused and do not predict behaviour very well (Burger, cited in Prager 2002). This is relevant to the research being discussed here, as this questionnaire did not significantly correlate with Goldberg's scale, although both were meant to be measuring the same characteristic. However openness is undoubtedly an asset to a probation officer.
Conclusion
The research shows that openness to experience is an important aspect to the chosen career, and this can be seen and supported by the Probation Service itself.
The final questionnaire was reliable and showed content validity, therefore would be beneficial to an occupational psychologist when recruiting new probation officers.
References
Coolican, H.C. (1999), Research Methods and Statistics in Psychology, 3rd Edition, Hodder and
Stoughton, England.
Costa and McCrae (1977) cited in Engler, B. (2003). Personality Theories: An Introduction. (6th
Edition) Houghton Mifflin Company: London.
Goldberg, L. R. (1992) The Development of Markes for the Big-Five Factor Structure,
Psychological Assessment, 4, 1, 26 - 42.
Goldberg, L. R. (1990) cited in Cooper, C. (2002) Individual Differences. (2nd Edition) Oxford
University Press
Home Office (2004) Probation North West Recruitment, Home Office Publication:
www.probation-northwest.co.uk Dec 14th 2004
- 1 -
