There are a number of reasons why connectionism has failed to deliver. Firstly there are some fundamental differences between a neural net and the brain. Secondly algorithm that is able to learn or process information of sufficient complexity for connectionism to be considered as a genuine means of explaining human cognition.
Thirdly a number of more fundamental arguments, criticising connectionism's claim to offer an alternative to classical cognition, have been forwarded. It is these arguments that I will examine in this essay, as they focus solely on connectionism and cognition.
The theory of connectionism is advancing at a considerable pace, and whether connectionism has the potential to create a cognitive network has been the subject of a fierce debate. This debate is important because much of connectionism's significance lies in its claim to represent a Kuhnian shift in our understanding of learning and cognition. If it could be demonstrated that connectionism, as it is now understood, does not have the capacity to cognitize, then as a concept it would be robbed of much of its allure.
The debate about whether connectionism has the capacity to provide a new explanation for cognition began with a paper by Fodor and Pylyshyn (1988)(1). In their paper Fodor and Pylyshyn (hereafter F + P) introduced the 'systematicity' argument. This states that, almost by definition, a functioning cognitive system must demonstrate systematicity. Systematicity, argue F + P is demonstrated by both humans and lesser organisms.
1 - Fodor, J.A. & Pylyshyn, Z.W. (1988).
Connectionism and cognitive architecture: A critical
analysis. Cognition, 28, 3-71.
For example a pigeon that can be conditioned to respond to a blue circle must have the capacity to be conditioned to respond to a red square. Likewise in humans it is inconceivable that someone would be able to think the thought, "John loves the girl", without being able to think, "the girl loves John". Such systematicity, argued F + P, is only possible through syntactic structure and effective syntax. A syntactic structure is one that employs representations that are used in structure sensitive ways, ie. it is a structure that is based on some form of language. As F + P put it, "an empirically adequate cognitive theory must recognise.relations of syntactic and semantic constituency".
Early connectionist systems did not attempt to utilise syntactic structure representations. Rather each node represented some part of the input. Again using F and P's example, the input, 'John loves the girl' would result in the 'John', 'the girl' and 'love' nodes becoming active, but the system would not be able to differentiate between John loving the girl, and the girl loving John. As such the system has no concept of context, of object and subject. Such systems are
described by F and P as merely associative.
The classical artificial intelligence (A.I.) concept of mentality and cognition, a concept referred to as classicism, is an excellent example of systematic cognition. Classicism asserts that cognition is a form of rule-governed symbol manipulation. In a classical system each input is tokened with a symbol, which can then be manipulated according to the rules, or program, which the system is using. Whilst classicism adequately demonstrates systematicity, it does not offer an adequate explanation for human cognition. This is because, according to Smolensky, classicisms commitment to programmable, representation level rules contradicts the evidence that human cognition conforms to a representation without rules concept. Smolensky goes so far as to describe classicism as being in a Kuhnian crisis.
F + P do not claim that neural nets are incapable of systematicity through syntactic structure and effective syntax. Rather they claim that the only way in whichsyntactic structure is possible in a neural net is through an implementation of the classical concept of cognition. As such all connectionist systems are doomed to fail because they will either be merely associative, or a mere implementation of classicism. The charge that a connectionist system is a mere implementation of classical theory is a serious one. Connectionism's claim that it marks a Kuhnian shift and offers a viable new way to understand cognition relies on its status as something novel. If it can be proven that connectionism is really nothing but a rehash of the theory it is supposed to replace, then it will relegate connectionism to a place in the shadow of classicism.
Those who criticise F + P's analyse accept the systematicity aspect of their argument, and as such the only way for the critics to challenge F + P's conclusions is to show that an effective syntactic structure is possible in a connectionist system. Acceptance of the systematicity argument also has considerable implications for the role of language in connectionism. It raises language above being a mere example of a higher cognitive function. It makes language, in the loosest sense of the word, a prerequisite for genuine cognitive thought.
Implementing language in a connectionist system is not simple. Language is, almost by definition, structured, whilst a connectionist system, if it to avoid the charge of being an implementation of classicism, has to remain largely unstructured. Smolensky has written widely on the implementation of language in connectionist systems, and he was the first to respond to F + P's analysis. Smolensky answered F + P's analysis on two grounds, which he termed the distributed (weakly compositional) case and the distributed (stongly compositional) case. The weakly compositional case is a response to F + P's description of connectionist inputs as represented in the activity of an individual neurone. Smolensky says this is a naive observation that doesn't relate to modern connectionist systems. Smolensky's (1) alternative form of representation utilises micro-features of the original input, creating a distributed representation. Smolensky described it as, "a family of distributed activity patterns". The example that Smolensky gives is a full cup of coffee. This would be represented by the activity in a number of nodes, each of which would represent a microfeature of the cup of coffee. Relevant microfeatures could include such things as 'hot-liquid', 'porcelain curved surface', and 'finger sized handle'.
1 - Smolensky, P. Connectionism, constituency, and the
language of thought.
Smolensky argues that this offers an alternative to classical representation because an input is not tokened in the classical way. The representation of coffee is context-independent, there is no single representation of coffee. Also the micro-features that are activated following the input of a cup of coffee might also be active when another input, such as a tea-pot is presented. This theory correlates with what we know about human perception, in which a stimulus is broken down to its constituent elements in the brain, before being reconstructed into what we perceive. However the theory has been widely criticised, Quinlan (1) describes it as, "both unworkable and untenable".
The weakly compositional case is an explanation of within-level processing, this means that it doesn't have to provide for the problem of a lack of role; of subject or object. However there must be a form of between-level processing that would provide for context if the theory is to answer F + P's criticisms. Smolensky's answer to this is tensor-product representations. In a tensor-product representation one vector represents the 'filler' (the constituent), and a second vector represents the role. The vectors are tensor-multiplied, and the result is a representation for both the filler and the role.
For example a tensor-product representation of 'John loves in the role of verb, and the girl in the role of object. Expressed another way, [v(John) x v(subj)] +[v(loves) x v(verb)] + [v(the girl) x v(obj)]
Tensor-product representations remain largely theoretical, and opinions on their value are varied. Fodor and McLaughlin (1990)(2) (hereafter referred to as F + M) responded to Smolensky's theories. F + M did not fundamentally alter the position expressed by F + P. They maintained that Smolensky had not proved that structure sensitive processing, essential if a system is to operate with effective syntax, is possible without classical symbolic representations. As F + M put it, "[Smolensky] provides no suggestion as to how mental processes could be structure sensitive unless mental representations have classical constituents".
1 - Quinlan, P. (1991) Connectionism and Psychology.
2 - Fodor, Y. & McLaughlin, B.P. (1990) Connectionism and the problem of systematicity: Why Smolensky's solution doesn't work. Cognition 35, pp 183-204.
Horgan and Tienson (1992) (hereafter referred to as H + T) have provided an excellent critique (1) of the various arguments put forward by F + P, Smolensky, and F + T. In this they reserve judgement on whether tensor-product representations demonstrate that systematicity is possible without classical symbolic representations, concluding that neither Smolensky, nor F + T have proven their case. H + T disagree with F + T that structure sensitivity requires symbolically tokened constituents, using as an example a mono recording of a string quartet.
On the tape the four instruments are not tokened individually, yet when the music is replayed a listener will be able to distinguish between the instruments, identifying the music's structure.
H + T also make some fascinating observations on the nature of the whole dispute. Tensor-product representations proceeded on the assumption that symbolic tokening is a,if not the, defining characteristic of classicism. H + T dispute this, arguing that, " the commitment to classical syntax is not really an essential, or definitive, tenet of classicism". They go on to argue the difference between classical and non-classical constituents is merely implementational. As such Smolensky's connectionist system, even if it did prove successful, would be, "an implementation of classical cognitive architecture after all - if you like a non-canonical implementation". H + T argue that, rather than concentrating on the constituents of the syntax, connectionism should be aiming to distinguish itself totally from classical cognition by abandoning its central tenet. That is connectionism should attempt to develop 'a version of cognitive connectionism that repudiates the 'rules' component of the classicism package deal while still retaining effective syntax'. Such a version would have to abandon the hard rules that Smolensky uses in his tensor-product representations, in favour of 'soft laws'.
Representations without rules, as F + L term their concept of processing, would be a radical departure from traditional connectionism, and would probably be treated with equal incredulity by both Fodor and Smolensky. However the attractions of such a radical approach are obvious. The dispute over the possibility of effective non-classical constituents appears to be going nowhere fast, and as H + T point out, even if it is resolved, it will not be a definitive rejection of classicism.
1 - Horgan, T. & Tienson, J. (1992) Structured representations in connectionist systems? From Davies, S. Connectionism: Theory and practice. Oxford U.P
Representation without rules on the hand offers the possibility of a genuinely new concept of cognition and artificial intelligence, and the prospect of a scientific revolution. It might also go some way to identifying the real problems that connectionism must overcome if it is to achieve genuine replication of higher cognitive functions. As it is, the debate is at risk of becoming bogged down in the niceties of a number of pet theories.
I do not believe that connectionism has really begun to tackle the problems that replicating higher cognition will entail.
References
Brunak, S. & Lautrup, B. (1990) Neural networks, computers with intuition. World Scientific: Singapore.
Click, F. (1989) The recent excitement about neural networks. Nature 337, pp 129-133.
Fodor, Y. & McLaughlin, B.P. (1990) Connectionism and the problem of systematicity: Why Smolensky's solution doesn't work. Cognition 35, pp 183-204.
Fodor, J.A. & Pylyshyn, Z.W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition
28, 3-71.
Hinton, G.E. (1992) How neural networks learn from experience. Scientific American 287, pp 105-109.
Horgan, T. & Tienson, J. (1992) Structured representations in connectionist systems? From Davies, S. Connectionism: Theory and practice. Oxford U.P.
Quinlan, P. (1991) Connectionism and Psychology. Harvester Wheatsheaf: London.
Smolensky, P. (1988) On the proper treatment of connectionism. Behavioural and Brain Sciences 11, pp 1-74.
