It was decided to pilot and validate the assessment battery using data collected from employees of the BT Commercial and Brands business unit. The aims of the pilot were to tailor the assessment tools’ norms to fit BT’s employee population, as well as to optimize the tools’ ability to predict successful job performance. An additional aim of the pilot was to study the differences between the different job families within Major Business. Specifically, the pilot aimed to check whether differential norms are required, and whether different combination of competencies assessment provide better predictors for the different job families.
This document presents the results of the pilot and validation study.
Method and Procedures
Sample
- 567 active BT sales personnel took CareerHarmony’s battery of assessment tools between August and November, 2004. The assessments were taken online, by each employee using their own computer.
- The average age of the participants was 41 years old (SD=7.2), ranging from 22 to 58 years.
- The subjects had an average of 4.1 years of experience in their current positions (SD=3.5), ranging from just a few months to 24 years.
- There were 5 BT job categories in the current sample: Bid Management (N=29); Sales Engineer (N=71); Sales Farmer (N=157); Sales Hunter (N=113); Specialist Sales (N=174). Approximately 23 subjects’ job families were not labelled.
-
Roughly 83% of the subjects were males (N=460), and 17% females (N=94). 13 subjects did not identify their gender.
-
90% of the subjects were Whites (N=551), and 10% non-Whites (N=53; Asian: 6%, Black 1%, Mixed 3%). 16 subjects did not identify their race.
-
96% of the subjects (N=543) were native English speakers. The remaining listed Persian, Indian, Gujarati, French, Italian and Punjabi as their native languages, with approximately equal frequencies between them.
- Only 1% of the subjects (N=6) reported themselves to be disabled in some respect.
Tools
A total of 5 assessment tools were administered, consisting of both ability and personality measurements. These tools were selected from CareerHarmony’s suite of assessment tools in cooperation with the BT project team. The tests used were:
-
Graph and Table Comprehension (AAGT): This test presents a series of graphs and tables. Each graph or table is followed by a series of questions, which measure the candidate’s level of understanding of the information presented within the illustration.
-
Planning and Organizing Ability (AAPO): Each item in this test presents a scenario giving a general introduction to a situation in which the candidate needs to operate, followed by information detailing existing knowledge regarding the situation. After studying the information the candidate needs to answer a series of questions regarding future planning, based on the information provided.
-
Verbal Reasoning (ARV): This test contains a series of items; each consists of a pair of words or phrases that are related to one another. From a set of five additional pairs of words or phrases, the subject’s task is to select the pair that best expresses the relationship conveyed by the original pair.
-
Interpersonal Coping Style (PIC): This test contains a series of paired statements describing possible reactions towards interpersonal conflict situations are presented. For each pair of statements, the subject is forced to choose one that describes his/her behaviour. In this test, the subject is scored on a profile on five characteristic dimensions of coping with interpersonal conflicts: Accommodating, Avoiding, Collaborating, Competing, and Compromising.
-
JPPI: The JPPI is a unique personality inventory designed specifically for use in occupational settings. It was designed from the outset based on job analyses to match occupational personality competencies. Each item contains behavioural sentences for which the job applicant is required to indicate the degree to which the sentence is true or false for themselves. This test has 30 scales, including: Adaptable, Artistic Appreciation, Agreeable, Forgoing, Socially Attentive, Autonomous, Communicative, Cooperative, Dependable, Sociable, Instructive Ability, Emotionally Intelligent, Ambitious, Orderly, Persuasive, Resourceful, Responsive, Stress Tolerant, Planning and Coordinating, Decisive, Self-Confident, Energetic, Problem Solving, Self Developing, Extraverted, Abiding, Hasty, Leadership, Emotionally Stable, Enduring.
Performance Criteria
Two performance criteria were used:
-
DPR Score: this score is a summary of an individual's achievements against their job description requirements and individual objectives. Each employee’s DPR is discussed and agreed upon in a one to one meeting with their manager. The score is a simple ranking on a 9 point scale, in which a lower score denotes a higher performance.
-
Managers’ Ratings: managers were asked to rate each employee on a list of scales, related to their job performance in comparison to other similar employees. These ratings were then integrated into one overall average score.
Results
An initial analysis of the results included a process of choosing the most reliable and valid tests for the BT population. As a result, it was decided to include 19 of the total 30 scales of the JPPI, and the 3 ability tests. None of the Interpersonal Coping Style scales were found to be valid in this case, and were therefore not included in further analyses. The scales chosen were found to be more reliable and valid for the BT sales sample. These 19 scales were then included in the advanced stage of the data analysis. The technical procedures and analyses leading to the choice of the final JPPI scales are described in Appendix A.
The resulting norms and validity indices are described below.
System Norms
An initial aim of the study was to establish a norm-base for the BT population, while checking whether there is a need to establish differential norms for the different job families, or whether one norm group can be used across the BT sales community.
To test the differences between the results of the different job families, one-way ANOVAs (analysis of variance) were conducted with the scales scores as dependent variables and the job category as a dependent variable. Post hoc comparisons revealed slight differences between most scale means, whereby the Hunters were characterized by higher means than the Engineers. Other differences were less salient or consistent.
Overall, the differences between the means were not meaningful, and therefore do not justify implementing individually set norms for these groups.
The norms for the total sample are presented below. These have been implemented into the CareerHarmony system, and the individual reports that will be produced for each employee will be based on scores calculated by using these norms.
Norms per job group can be found in Appendix B.
System Validity
The aim of this stage was to design a composite score that would maximize the predictive power of the assessments. In this stage of the analyses, both the total sample and each job group were analyzed. An additional aim was to establish whether different combinations of assessment tools would provide better predictions for the different job families.
Linear regression analyses were used in order to design predictor composites, which would include both the relevant of the 19 JPPI scales and the 3 cognitive ability tests: Graphs and Tables Comprehension, Planning and Organizing, and Verbal Reasoning. (Please note that as mentioned above, based on detailed data analyses, 19 of the total 30 scales were found to be more reliable and valid for the BT sales sample).
In order to choose the most relevant scales to be entered into the regression analyses, certain scales were identified for each job group by choosing the scales that showed the highest correlation with the criteria (for either Managers’ Ratings or DPR scores).
In the table below, the scales showing the highest correlation with the criteria are marked in green, and the scales showing a good level of correlation are marked in yellow. All scales marked in green and yellow were entered into the regression analysis. The table below shows the validity indices of each of the 19 scales as well as those of the composite scores yielded by the linear regression (R).
As shown in the table, a greater validity was achieved overall for the Managers’ Ratings criterion (.27) than for the DPR score (.19), and a similar pattern was seen per job group, ranging from .27-.42 for the Managers’ Ratings, and .15-.25 for the DPR score.
*p<.05
The next table below presents the appropriate beta weights for approximately reproducing the linear regressions. The rationale followed here was a 50% weighting of the ability scores, and a 50% weighting of the personality scores. Within each score type (i.e., ability or personality) scales the more highly correlated scales (marked in green) were given a double relative weight, and all other scores were weighted equally. The sum of all scale weights is approximately equal to 1.
Beta Weights
Based on the results presented, it is possible to administer differential composite indices for each job group, or one general composite for the entire sample.
Across the job groups, all 3 ability tests were almost always included in the composite score, whereas the personality scales differed.
Across all jobs, Adaptable, Ambitious, Communicative, Dependable, Decisive, Emotionally Stable, Leadership, Problem Solving, Resourceful, Self Confident, and Self Developing were the more predictive personality scales.
Analysis results per job group found the following personality scales to be most predictive:
-
The Bid Management group: Adaptable, Ambitious, Communicative, Dependable and Problem Solving, were the more predictive personality scales. Emotionally Stable, Leadership, Resourceful, Self Confident and Stress Tolerant were also predictive.
-
The Sales Engineer group: - Leadership, Problem Solving, Resourceful, Sociable, and Stress Tolerant were the more predictive personality scales. Adaptable, Communicative, Decisive, Emotionally Stable, Extraversion and Self Developing were also predictive.
-
The Sales Farmer group: Agreeable, Ambitious, Communicative, Leadership and Self Confident were all the relatively more predictive personality scales.
-
The Sales Hunter group: Communicative, Decisive, Leadership, Resourceful, and Self Developing were the more predictive personality scales. Adaptable, Ambitious, Artistic, Emotionally Stable, Extraversion, Problem Solving and Self Confident were also predictive.
-
The Sales Specialist group: Ambitious, Dependable and Leadership were the more predictive personality scales. Adaptable, Decisive, Emotionally Stable, Problem Solving and Resourceful were also predictive.
Conclusions
Overall, this study empirically demonstrated the CareerHarmony assessment system to be an effective tool for assessing and predicting performance success within the Major Business unit of Commercial & Brands. It also highlighted minor variation in the optimal profiles of the different job groups. Specific conclusions are:
-
Norms: A general norm group for the entire BT sales population should be used.
-
Assessment Validity: The data collected indicates that the CareerHarmony system provides a valid assessment tool, which predicts successful job performance, based on both the DPR criterion and the managers’ assessments.
-
Differences between job families: The analysis of the data supports the assumption that differential profiles can be created for assessing and predicting success in the different job families.
The cognitive abilities and a group of personality characteristics were found to be important across all job families, however some personality characteristics were found to be more important for specific job families only.
A single assessment covering all these roles can be created so that it provides a score for each job group, enabling managers to see the relative fit of an employee or candidate to each position.
-
Issues for further investigation: Further analysis should be carried out on the DPR process, and on the Sales Farmer population.
-
Extending sample: BT should ensure that it extends the sample before drawing conclusions about other business units.
-
Profiling future performers: This analysis provides a foundation which if coupled with further analyses can be used as a basis for creating the profiles of performers in the future organisation.
-
Preparing for the future: Once future and current performers have been profiled, an exercise can be undertaken to determine how selective recruitment and employee development can help BT ensure it has good performers - both today and in the future.
Appendix A – Procedures and Analyses
Analysis of the JPPI scales was carried out in 2 stages. In the first stage, the weakest of the 30 total scales were eliminated, and in the second stage the remaining scales were cross validated.
Stage 1 – Eliminating Weaker Scales for This Sample (N=295)
Item analysis of the JPPI was conducted on the first sample of 295 test takers (209 of whom had Managers’ Ratings). 2 items were eliminated from each scale, leaving 10 items per scale, based on the following method: scales with an alpha coefficient of less than .70 (marked in yellow) or low validities for either Managers’ Ratings or DPR scores (marked in green), were eliminated. 19 scales remained. It should be noted that while several other job criteria were available as well (e.g., self ratings, sales scores, etc.), the 2 criteria chosen were considered to best reflect subjective and objective ratings of overall performance.
* p<.05; +DPR is an inverted scale i.e., the lower the score the better
Stage 2: Cross Validation (N=273)
The 19 scales remaining from the first stage were then re-examined in a new sample of the next 273 applicants (226 of whom had Managers’ Ratings). The table below presents reliabilities and validities of these scales.
* p<.05 **p<.01; + DPR is an inversed scale i.e., the lower the score the better
Appendix B - Norms per Job Group
Bid Management
Sales Engineer
Sales Farmer
Sales Hunter
Sales Specialist
