Critiquing research papers: methods, methodologies, theoretical perspectives and epistemologies.

HIB1003 – Research Methods in Psychology. 5/10/2007 .

Assignment 1 - Critiquing research papers.

Zorkina and Nalbone (2003) did a study to see if the confidence levels of college students according to what instructions they were given had an effect on their performance on a cognitive ability test.

Zorkina and Nalbone (2003) tested out the hypothesis by placing college students into 2 groups. Each group was given different instructions in order to induce different levels of confidence in them, high level and low level. The participants in both groups were given the same cognitive ability test and the scores were compared for both groups. Peer evaluations were also done on participants’ confidence and academic performance for external validity. It was found, that induced level of confidence did have an effect on college students and a bivariate correlation indicated that the results were significant. r(30) = 0.54, p = 0.002. As academic confidence increased, academic performance also increased.

The aim of this assignment is to critique the research by Zorkina and Nalbone taking into account the epistemological position of the paper, the theoretical perspective, the methodology and the method of data collection the researcher has used.

Epistemology

The research paper takes an objective epistemological position of the paper. According to Crotty objectivism is based on the assumption that attitudes exists as meaningful entities, both truth and meaning residing within them. (1998). The theory around objectivism assumes that both the level of confidence and the performance on the test can be objectively measured through the use of carefully planned, reliable and valid methods.

A constructionist position of the paper would however, argue that performance and confidence are terms that are socially constructed and cannot be directly observed. Constructionism assumes that the meanings of these terms are decided by humans and interpreted by people, therefore questioning whether confidence can be objectively measured. Subjectivism assumes that the meaning is imposed on the object, in this case the term confidence is decided by humans and the term might be interpreted differently by different people and therefore there is no valid way of measuring these constructs.

Theoretical perspective

The theoretical position the paper takes is positivism mainly because positivism is objective by definition and is seen as a natural science. According to smith (1998) for a study to fit into the positivist framework, it has to qualify to a few assumptions. Yates (2004) has explored these assumptions. The main assumption of positivism is that of naturalism, which assumes that behavior and social phenomena can be measured in the same way as scientific reactions to produce objective and valid results. The current study has been based on the assumption that induced levels of confidence will have an effect on performance, both of which can be objectively measured.

The idea of naturalism is consistent with objectivism in the sense that they both assume that it is possible to produce objective facts about things by conducting research in a systematic way.

A second assumption of positivism is the distinction between facts and values. Values involve subjective interpretations whereas facts are seen as scientific statements which are quantifiable. The use of statistical data in the current research allows for objectivity and enables the claims to be backed up.

Another assumption of positivist is that of phenominalism, this is based on the assumption that only knowledge gained ...

This is a preview of the whole essay

The idea of naturalism is consistent with objectivism in the sense that they both assume that it is possible to produce objective facts about things by conducting research in a systematic way.

Another assumption of positivist is that of phenominalism, this is based on the assumption that only knowledge gained from direct observation can be taken seriously. This view is also shared by nominalism that, concepts have no use other than as names and area meaningless entities. Phenominalism and nominalism would therefore argue that since confidence cannot be observed directly it can’t be measured. It also argues that confidence is just a term made by humans and interpreted differently by all respondents and therefore it isn’t measurable. However positivist would assume that although confidence cannot be observed it can be measured through valid and reliable methods and the study is based on the assumption that term confidence has the same meaning for all respondents. Yates (2004).

Methodology and method.

The piece of research uses an experimental methodology with the aid of questionnaires as the method to test the hypothesis. Experimental methods most clearly adhere to the assumption of naturalism. Experimental methods rely upon quantitative and numerical evidence and the process of conducting research is often categorized as positivist ideas. Positivism and objectivism both believes that social science can measure social phenomena and a key element for both is the term ‘natural’ science. An experimental methodology achieves this assumption as it has the ability to control extraneous variables through the use of ‘closed systems’

‘Closed systems are ones where no ‘external factors’ influence the ways in which the system functions.’ Yates (2004) pg 10. A closed system allows us to test out predictions about the relationship between various elements within the system without having to worry about other factors that may influence the results. All external factors can be discounted for by operationalising the variables and then isolating the IV from all other factors affecting it.

One of the ways in which objectivism is achieved in this study is through the operationalising of variables. The research question of the study is the ‘effect of induced level of confidence on college students’ performance on a cognitive ability test’, the IV being the induced level of confidence, high level of confidence and low level of confidence and the DV being operationalised as the performance on a cognitive ability test. The induced level of confidence depended on what instructions the participants were given. Opereationalising variables effectively, allow for discriminant validity as what is being measured is clearly defined and any ambiguity regarding the variables is minimalised.

As well as participants doing an academic self confidence questionnaire they also did a general self confidence test, this was done in order to differentiate between induced and non induced academic self confidence. The test found that there was no difference in the general confidence between the two groups. An independent groups t- test showed the results to be significant. t(28) = -1.75, p = 0.09. The test was done in order to ensure internal validity that only the induced level of confidence (the instructions they were given) had an effect on their performance.

Although the research is seen to be valid it can be argued that they are not genralisable to the target population and are not representative of all college students. The aim was to see if the induced level of confidence had an effect on College students. The sample was very small with only 30 participants, therefore limiting the statistical power and all participants were voluntary recruits from a Liberal Arts College. Although this is convenient in the sense that it is less time consuming, easier and cheaper to recruit the participants, there is the possibility that all participants acquiring similar characteristics and abilities. However Zorkina and Nalbone have acknowledged this problem in the discussion section and do state that for stronger results and greater generalisability, a much larger and diverse sample is needed.

The importance of closed systems in this study is further shown through the use of statistical data. Statistical data are used to provide greater objectivity and are used to ensure that any random or external factors that are unknown or not accounted for can be discounted.

All the results in the results section have been backed up by statistical data, for example the use of t-tests, which provide us with quantifiable data therefore fitting into the assumption of providing factual objective data.

The study aimed to measure the effect of induced level of confidence on the performance on a cognitive test. To test whether the participants in both groups actually had a difference in academic confidence a test was done to measure it. The IV – the induced level of confidence was tested between the two groups through the academic confidence questionnaire. It was found that the mean academic confidence was higher for the high confidence group, M = 3.15 than for the low confidence group, M = 2.47. An independent group’s t-test found the results to be significant. t(28) = -2.86, p = 0.08.

One of the main questions regarding validity is that of face validity. Face validity questions whether the study measures what it aims to measure. (Bryman, 2001). In this study the researcher has aimed to measure the performance of respondents through a cognitive ability test which the researcher made up. It could be argued that the researchers test fails to achieve content validity and fails to measure performance. The researcher would have been better off using a test which has been previously used for similar studies for example an IQ test. Also using an existing test would allow the researcher to compare her results with that of existing ones and test for concurrent validity.

The method used by the study is through the use of questionnaires. The use of questionnaires for the research question is the most appropriate method as questionnaires ‘tend to attempt to ‘control’ the aspect of human behaviour being studied by creating ‘closed systems’’ (Smith, 1998). However a few problems faced by using a questionnaire for this study is that it breaks the assumption of phenominalism and nominalism. Nominalism because respondents will have different ideas of what is confidence and phenominalism because the questions about confidence are asking respondents about their beliefs, feelings and personal values, respondents could view themselves as more/ less confident than they really are. This would reduce the internal validity of the study. This problem was however overcome by doing peer evaluations of participants’ academic confidence levels and then comparing them with the respondents self report of academic confidence. The results showed that self report of academic confidence and peer evaluations of academic confidence was highly correlated, ensuring the assumptions of phenominalism and nominalism are met and also proving the study to be internally reliable and also demonstrating convergent validity.

The participants who took part in the study were given the tests to do as test conditions with minimal distractions, however the participants were tested individually in there own dorm rooms. This is a good idea as other variables such as pressure or stress are minimalised, it is also a good way of maintaining ecological validity. According to Cooligan ‘a study has higher ecological validity if it generalises beyond the laboratory’. (1999). The participant would be at ease and can be seen as applicable to everyday natural settings.

Conclusion.

Zorkina and Nalbone’s study is very objective all the way through, they have used a lot of statistical evidence to back up all the claims they make. Also they have tried to ensure validity throughout through the use of a Multitrait-Multimethod Matrix. This method allows for construct validity and helps ensure that only the academic confidence has an effect on the performance and other factors are isolated and removed.

Although the method they have used has maintained objectivity, this doesn’t reflect the whole research. The claims stated in the abstract are not presented as objective facts but more as value laden subjective interpretations. The abstract only claim the hypothesis has been supported, but fails to give any statistical evidence to back up the claims.

The studies mentioned in the introduction are consistent with the theoretical perspective, positivism. The studies mentioned has all adopted objective methods of data collection. The studies have assumed that traits such as self efficiency, academic confidence and general confidence can all be objectively measured with the aid of statistical tests.

One of the strengths of the discussion section of the research is that Zorkina and Nalbone have critically evaluated their own study and have tried to come up with improvements. One of the things they have mentioned is that a control group with no instructions given should be added. This would allow the researcher to see whether no instructions had an effect on the performance.

Overall the study has been effect in measuring what it aimed to measure and the main strengths of this research are that it has tried to achieve validity of the study through the use of objective methods and statistical testing. Also the way in which the researchers have presented the procedure is very clear and precise making it easily replicable. However the limitations of the study are that of genaeralisability, due to the process of sample selection. One of the main questions that could arise from the study is that is it representative of the target population and can it be generalized to other people, places or times.

References.

Bryman, A. (2001). Social Research Methods. Oxford.
Cooligan, H. (1999). Research Methods and Statistics in Psychology. (3Rd Ed). Hodder and Stoughton.
Crotty, M. (1998). The Foundations of Social Research. Sage Publications.
Smith, M. (1998). Social science in question. London: Sage
Yates, S.J. (2004). Doing Social Science Research. Sage Publications.
Zorkina, Y and Nalbone, D, P. (2003). Effect of induced level of confidence on college students’ performance on a cognitive test. Current research in social psychology. 8 (11)

Bibliography.

Jonathan Howell et al (1997). Overview: Reliability and Validity. Colorado State University. (10.12.04).
Trochim, W.M (2002). Research Methods Knowledge Base. Cornell University. (10.12.04).