The semiotic function and the genesis of pictorial meaning

Göran Sonesson

in Tarasti, Eero, ed.,Center/Periphery in representations and institutions. Proceedings from the 3rd Annual Meeting and Congress of The International Semiotics Insitute, Imatra, Finland, July 16-21, 1990.: International Semiotics Institute, Imatra 1992; ss 211-256.

One way of approaching the subject matter of semiotics may be to consider what is implied by the notion of semiotic function. There are two classical contexts for the use of this phrase. One is found in the work of the linguist and proto-semiotician Louis Hjelmslev writing in the forties. To him, the semiotic function simply means that any sign must involve an expression serving as vehicle for a content. But as soon as we look a little closer into his notion of sign, things start out to be very complicated, so it will be more convenient to begin at the other end. The second classical locus appears spread out all other the numerous writings of the cognitive psychologist Jean Piaget. Therefore, we will first attend to Piaget, his admirers and critics, to both of which I happen to belong. But before we can begin to deepen our insights into semiotics by way of a discussion of the semiotic function, something must be said, in rough outline, and at a more abstract level, about semiotics in general and pictorial semiotics in particular.

On semiotics in general and pictorial semiotics in particular

According to Ferdinand de Saussure, one of its reputed initiators, semiotics (or semiology as he called it) was to study “the life of signs in society”; and the second mythical founding-father, Charles Sanders Peirce, as well as his forerunner John Locke, conceived of semiotics as being the “doctrine of signs”. Later in life, however, Peirce come to prefer the wider term “mediation” as a description of the subject matter of semiotics (cf. Parmentier 1985). And Saussure actually argued that in the semiotic sciences, there was no object to be studied except for the point of view which we adopt on other objects (see Sonesson 1989a,I.1.4.). More recently, Greimas has rejected the notion of sign, and his followers Floch (1984a) and Thürlemann (1982a: 1990) have argued the case in the domain of pictorial semiotics. In a similar fashion, Umberto Eco (1976), at the end of his tortuous critique of iconicity, substituted the notion of sign process for the traditional sign concept.

The sign, then (synonymous, it would appear so far, with the semiotic function), is not comprehensive enough to delimit the field of semiotics: rather, the domain of semiotics is meaning (or “mediation”), in some wider, yet to be specified sense. But since everything, or almost everything, may be endowed with meaning, we must insist that if any object whatsoever (or almost) may enter into the domain of semiotics, then this is so only to the extent that it is studied precisely as far as its capacity for conveying meaning is concerned.

Nor should we adopt the popular preconception, according to which the semiotic field is inhabited simply by the followers of Peirce and Saussure. In the first place, there would be no reason (more than a superficial terminological coincidence) to amalgamate two such dissimilar doctrines as those represented by the elaborate but fragmentary philosophy of Peirce, and the marginal, if suggestive, annotations of Saussure. But, more importantly, in adopting this point of view, we would be unable to account, not only for the semiotical work accomplished well before the time of our two cultural heroes, be it that of the stoics, Augustin, the scholastics, Locke, Leibniz, or the ideologues, but also for much of contemporary semiotics, some parts of which are not particularly indebted to any of the forefathers. Some of the complexities of semiotic history may be grasped from figure 1.

Fig. 1. Fragments of semiotic history

The point of view of semiotics may be applied to any phenomenon produced by the human race (or perhaps we should say, more broadly, which originates in the animal kingdom). This point of view consists, in Saussurean terms, in an investigation of the point of view itself, which is equivalent, in Peircean terms, to the study of mediation. In other words, semiotics is concerned with the different forms and conformations given to the means through which humankind believe itself to have access to “the world”. It tries to emulate the point of view of humankind itself (and of its different fractions), but it must also go beyond it, to explain the workings of such operative, albeit tacit, knowledge which underlies the behaviour of all culture-bearing human groups. Moreover, semiotics takes an interest in these phenomena in their qualitative aspects rather than the quantitative ones, and it is geared to rules and regularities, instead of unique objects. It is not restricted to any single method, but is known to have used analysis of concrete texts as well as classical experimental technique and imaginary variation reminiscent of the one found in philosophy. Moreover, it is not dependant on a model taken over from linguistics, as is often believed, although it remains a peculiarity of the approach to construct models which then guide its practitioners in their effort to bring about adequate analyses, instead of simply relying on the power of the “innocent eye”, as is usual in the human sciences. After having borrowed its models from linguistics, philosophy, medicine, and mathematics, semiotics is now well on its way to the elaboration of its proper models.

Semiotics has a long history, mainly as a part of other disciplines; and, of course, meaning continues to be discussed, even today, inside most of the human and social sciences. The creation of a particular discipline, centred around the notion of meaning, and often, but not always, termed semiotics or semiology, has been announced many times in history, but its career has been episodic, until the recent period. It is only during the last thirty years that semiotics has gained a large following and, more importantly, entered the domain of hard institutional facts. In recent years, research institutes, national and international associations, revues, congresses, and so on, seem to be springing up everywhere. Eco (1977) has rightly observed that the emergence, in our time, of semiotics as a particular discipline must be at least partially due to the present profusion of mass-media and other means of communication. It should be added, however, that as a result of these secondary or tertiary layers of mediation which have recently accrued to our everyday experience, we have been forced to realise how deeply mediated is also our ordinary, unreflected life in the unquestioned, sociocultural, Lifeworld (cf. Sonesson 1987c: 1990a).

You will undoubtedly have noted the similarity of the preceding account to the one which could be given of artificial intelligence and, more broadly, cognitive science. There is at the present time a remarkable convergence between the two disciplines, with an incipient interest for the other domain being noticeable on both sides, and some practitioners being ascribed to both camps. In his closing statement, at the meeting of the International Semiotics Institute in Imatra in July 1990, Thomas Sebeok suggested that cognitive science was simply semiotics with money. I would add, however, that semiotics is cognitive science with additional intellectual sophistication. Marcelo Dascal (1978; 1983) once suggested that artificial intelligence could have avoided many of its errors and confusions by studying the history of semiotics, including the proto-history of the 17th and 18th centuries. Indeed, I will argue that the wide and vague use of such terms as “sign” and “symbol” current in present-day cognitive science (for instance, Johnson-Laird 1988) is precisely the same at that found in medieval semiotics, before the semiotic function was isolated as particular instance.

However, I do not want to claim that semiotics has nothing to learn from cognitive science. It may learn from computer science how to make its models more explicit. And it certainly has to absorb a lot of factual knowledge and some notions of theory from cognitive psychology and the psychology of perception. The latter is particularly true of pictorial semiotics. No doubt, pictorial semiotics may be said to take its origin from a small article by Roland Barthes, written in 1964, in which a analysis of a publicity picture boosting the delights of Panzani spaghetti is attempted, using a few ill-understood linguistic terms taken over from Saussure and Hjelmslev (cf. Sonesson 1989a,II.1). Since then, the field has has been flourishing, with the most important contributions having been made so far, to my mind, by Floch and Thürlemann of the Greimas school, by the Belgian Groupe µ, and by Fernande Saint-Martin and her followers in Canada (cf. fig 2.) Last year, a review specialized in pictorial semiotics began to be published, and the International Association of Visual Semiotics (of which I happen to be one of the initiators) was founded in Blois, France.

Fig. 2. Context of pictorial semiotics

At is present stage, however, pictorial semiotics may well have less in common with Barthes’ Panzani analysis than with .that “linguistics of the visual image” invoked by the art historian E.H. Gombrich, or that “science of depiction” called for by the psychologist James Gibson; as well as with the studies of pictorial meaning initiated in philosophy by, for instance, Edmund Husserl, Ludwig Wittgenstein, Nelson Goodman, and Richard Wollheim. The most relevant reference, however, as will be seen shortly, may well be that to Gibson, who, together with such disciples and colleagues as Julian Hochberg, John Kennedy, and Margaret Hagen, has started to elaborate a psychology of picture perception. – but psycholinguistics cannot do without linguistics, and, by the same token, we need to establish a more general, theoretical, framework for the study of the picture sign (cf. Sonesson 1989a).

Yet another preliminary consideration has to be made at this point. Many exponents of contemporary semiotics, like those assembled around A.J. Greimas in France, claims that semiotics must be a pure, or autonomous science, such as was once the ideal of structural linguistics. Other researchers, notably in the United States, tend to look upon semiotics as merely a meeting-place of many different sciences, a kind of interdisciplinary framework common to the humanities and the social sciences, including, on some accounts, biology and neurology..The point of view taken here, and applied in my book, is different from both these approaches: I will take the results of all disciplines involved with the same subject matter (that is, in the present case, with pictures) to be relevant to semiotics, but only once they have been reviewed, redefined and complemented from a specifically semiotic viewpoint.

Pictorial semiotics, we shall take it, is that part of the science of signification which is particularly concerned to understand the nature and specificity of such meanings (or vehicles of meaning) which are colloquially identified by the term “picture”. Thus, the assignments of such a speciality must involve, at the very least, a demonstration of the semiotic character of pictures, as well as a study of the peculiarities which differentiate pictorial meanings from other kinds of signification, and a assessment of the ways (from some or other point of view) in which pictorial meanings are apt to differ from each other while still remaining pictorial in kind. In differentiating pictorial meaning form other meanings, we should in fact be particularly interested in knowing how they are distinguished from other kinds of visual signification, such as sculpture, architecture, gesture, and even writing; or how they differ from other iconic signs, that its, from other signs motivated by similarity or identity. And the further analysis of the picture category may lead on to the tasks of characterizing photography and drawing, the advertisement picture and the art picture, the picture post card and the poster.

In order to elucidate some of these issues, we will now turn to the semiotic function as conceived by Piaget, and ask to what extent, and in what way, it is embodied in the picture.

The semiotic function according to Piaget

To Piaget, the semiotic function (which, in the early writings, was less adequately termed the symbolic function) is a capacity acquired by the child at around 18 to 24 months of age, which enables him to imitate something outside the direct presence of the model, to use language, make drawings, play “symbolically”, and have access to mental imagery and memory..The common factor underlying all these phenomena, according to Piaget, is the ability to represent reality by means of a signifier which is distinct from the signified. Indeed, Piaget argues that the child’s experience of meaning antedates the semiotic function, but that is does not then suppose a differentiation of signifier and signified in the sign.

In the numerous passages in which he introduces this notion of semiotic function, Piaget goes on to point out that “indices” and “signals” obviously are possible long before the age of 18 months, but then they do not really suppose any differentiation between expression and content. The signifier of the index is, Piaget says, “an objective aspect of the signified”; thus, for instance, the visible butt of an almost entirely hidden object is the signifier of the object for the baby; and the tracks in the snow stand for the prey to the hunter, just as any effect stands for its cause. But when the child uses a pebble to signify candy, he is well aware of the difference between them, which implies, as Piaget tells us, “a differentiation, from the subject’s own point of view, between the signifier and the signified”.

Piaget is, I believe, quite right in distinguishing the manifestation of the semiotic function from other ways of “connecting significations”, to employ his own terms. Nevertheless, it is important to note that while the signifier of the index is said to be an objective aspect of the signifier, we are told that in the sign and the symbol (i.e. in Piaget’s terminology, the conventional and the motivated variant of the semiotic function, respectively) expression and content are differentiated form the point of view of the subject. We could actually imagine this same child that in Piaget’s example uses a pebble to stand for a piece of candy have recourse instead to a feather in order to represent a bird, without therefore confusing the feather and the bird: then the child would be using the feature, which is objectively a part of the bird, while differentiating the former form the latter from his point of view. Only then would he be using an index, in the sense in which this term is employed (our should be employed) in semiotics. And obviously the hunter, who has recourse to the tracks to identify the animal, and to find out which direction is has followed, and who does this in order to catch the animal, does not, in his construal of the sign, confuse the tracks with the animal itself, in which case he would be satisfied with the former.

Both the child in our example and the hunter are using indices, or indexical signs. On the other hand, the child and the adult will fail to differentiate the perceptual adumbration in which he has access to the object from the object itself; indeed, they will identify them, as least until they decide to change their perspective and approach the object from another vantage point. And at least the adult will consider a branch jutting out behind a wall as something which is non-differentiated from the tree, to use Piaget’s example, in the rather different sense of being a proper part of it. Although we will return later to consider the precise nature of the indexical sign, it will be convenient to note, at this point, that an index is a sign, the relata of which are connected, independently of the sign function, by contiguity or by that kind of relation which obtains between a part and the whole (henceforth termed factorality). But of course contiguity and factorality are present everywhere in the perceptual world without as yet forming signs: we will say, in that case, that they are mere indexicalities. Perception is profused with indexicality.

Fig.3. The prototypical sign and other meanings

The phenomenological tradition stemming form Edmund Husserl and later amply developed by Alfred Schütz and Thomas Luckman has contributed some useful distinctions here. According to these thinkers, two or more items may enter into different kinds of “pairings”, from the “paired association” of two co-present items, over the “appresentative pairing” with one item present and the other indirectly given through the first, to the real sign relation, where again one item is directly present and the other only indirectly so, but where the indirectly presented member of the pair is the theme, i.e. the centre of attention for consciousness. This will be enough to distinguish the abductive context from the sign, if we suppose the former to always carry the theme in the directly presented part or to have it span the whole context. But this is by no means certain, and there seems to be many intermediate cases between a perfect sign and an abductive context (the poetic function, ostensive definitions, proto-indices, etc.).

This is where Piaget’s idea of the semiotic function supposing a differentiation turns out to be useful. Whereas the items forming the sign are conceived to be clearly differentiated entities and indeed as pertaining to different “realms” of reality, the “mental” and the “physical” in terms of naive consciousness, the items of the context continuously flow into each other, and are not felt to be different in nature. Before we go on to illustrate this, two things should be noted: First, both content and expression of the sign are actually “mental” or, perhaps better, “intersubjective”, as most linguists would insist; but we are interested in the respect in which the sign user conceive them to be different. In the second place, Piaget’s notion of differentiation is vague, and in fact multiply ambiguous, but, on the basis of his examples, I have introduced two interpretations for it: first, the sign user’s idea of the items pertaining to different basic categories of the common sense Lifeworld; and, in the second place, the impossibility of one of them going over into the other, following the flow of time or an extension in space.

Suppose that, turning around a corner of the forest path, we suddenly catch a glimpse of the wood-cutter lifting his axe other his shoulder and head. This experience perfectly illustrates the flow of indexicalities which do not stop to become signs: it is sufficient to observe the wood-cutter in one phase of his action to know what has gone before and what is to come: that he has just raised his tool from some base level, and that at the next moment, he is going to hit the trunk of the tree. If we take a snap-shot of one of the phases of the wood-cutter’s work, we could use it, like the well-known traffic sign meaning “roadworks ahead”, as a part for the whole or, more oddly perhaps, as a phase signifying contiguous phases. There has been a radical change from the flow of indexicalities occurring in reality, for not only is there now a separation of expression and content “from the point of view of the subject”, but this separation has been objectified in the picture. Not even a series of pictures will reconstitute the perceptual continuum, but a film may of course do so. However, when we ask the wood-cutter to stand still for a moment (like in a “tableau vivant”), his position as such, before it is transformed into the motive of a picture, is already a sign for the whole of the action, although the directly presented position does not seem to be non-thematized, continuity is only provisionally interrupted, and expression and content are felt to be of the same nature. We are somewhere in between the abductive context and the sign: this may be termed a proto-index.

Ever since the time of Gestalt psychology, there has been another meaning of meaning around, quite distinct from the sign: the whole which is perceived to be something more than the elements out of which it is observed to be constituted. Although nobody would nowadays accept the Gestalt psychological explanation for the emergence of the whole, the phenomena are still there to be accounted for, and are described in contemporary cognitive psychology as being some kind of perceptual prototypes. Meaning according to Gestalt psychology amounts to something being more than its parts; but the sign, according to Roman Jakobson’s formula, which we tried to explicate earlier, supposes there to be something standing for something quite distinct from itself.

There is certainly a wider sense of meaning, which may be related, as Lévi-Strauss once put it, to order, that is, organization, relatedness, indexicality. What is important here is the connecting of things together, and the selection of elements to connect from a wider field of possibilities. It is interesting to observe that it is not the sign function but the paradigm, the feature, and the phoneme, as metaphors for selection, and the syntagm and the index, as metaphors for connection, which have had an important role to play in the adoption of the linguistic model in semiotics, notably in the work of Barthes, Greimas, Lévi-Strauss, and the Peirceans. When Lévi-Strauss presents the myth as a sign function, this interpretation is contradicted by his own detailed description, which really manifests a second-order texture. And when Greimas claims that even the phoneme carries meaning, this can only be understood in the sense of its forming a whole, a category having its own limits.

As we have already observed, the picture is certainly a sign, in the sense of it having a signifier which is doubly differentiated from its signified, and which is non-thematic and directly given, while the signified is thematic and only indirectly present. On the other hand, the picture is made up of, and presupposes, a number of meanings which are more elementary than signs. We shall see the importance of this observation later. For the time being, let us simply note that these meanings are of two kinds, those which, in Piagetian terms, pertains to operativity (logic, classification, etc.), and those which have to do with figurativity (which, in Piaget, is a residue concept). According to serious brain research (which should not be confused with the dubious lore pertaining to brain hemispheres nowadays found in the weeklies), operativity and figurativity may be very roughly distributed into the left and the right half, respectively, of the brain. Indeed, putting together information from Gardner and Coffman, we end up with the following table (fig.5.), in which the importance to drawing ability of both operativity and figurativity is made clear: contours and global properties are on the side of figurativity, while details, inner elements, and richness of details are on the side of operativity. Put more generally, operativtiy seems to account for structure, in which the whole works on the parts to make them stand out more prominently, whereas figurativity explains the configuration, in which many elements are fused into a new whole.

Fig.5 Figurativty vs operativity (from Sonesson 1989a)

Before we return to the semiotic function, we should attend to another kind of meaning, if indeed that is what it is. I mentioned earlier that in cognitive science, terms like “sign”, “symbol”, and “representation” are used in a vastly more comprehensive sense than the one which I favour. The contents of consciousness are said to be “symbols” ,and so on, of things in the “real” world (see Johnson-Laird 1988). Interestingly, that is an employment of the term found also in John Locke, one of the first explicit semioticians, at the beginning of the 18th century. Even before that, however, Pedro Fonseca, in his treatise on signs from 1564, distinguished two types of signs: formal signs, by means of which we know the outside world, and instrumental signs, which lead to the cognition of something else, like the track of an animal, smoke, a statue, and the like (cf. Deely 1982). However, as recognized in philosophical phenomenology, and more recently in the ecological psychology of James Gibson, we do not ordinarily perceive signs of the world, but the world itself; and thus, if indeed meaning is involved, its relata cannot be differentiated, and there can be no semiotic function.

An interesting aspects of Piaget’s notion of semiotic function is that is puts the acquisition of language, and the development of drawing ability into parallel; thus, verbal and pictorial signs are treated as being of a kind. In addition, however, Piaget introduces a number of other concurrent manifestations of the semiotic function: to imitate something without the model being accessible for observation at the same time, to play “symbolically”, and to make use of mental imagery and memory. Now we should ask: are really all these phenomena manifestations of the semiotic function in our, somewhat revised sense, that is, are they signs? Indeed, do they embody the semiotic functions, even in Piaget’s more restricted sense of supposing a differentiation between the signifier and the signified?

The German semiotician Günter Bentele (1984) has written an interesting study concerned with the development of the sign function, mainly from the point of view of phylogenesis, that is, the emergence of the sign out of more elementary processes of meaning and signals used by the lower animals. He also attends, albeit more briefly, to ontogenesis, that is, to the way in which the child learns to master signs, and here is mainly concerned with Piaget’s theory. Interestingly, he observes, as I did before taking cognizance of his work, that the sign supposes the content to be something distinct form its expression. More importantly, perhaps, he argues that imitation does not manifest the semiotic function, but is a prerequisite for it: indeed, it will function as a sign only to the extent that it is taken to refer back to the imitated act, instead of just being another instance of the same kind. The same observation should apply to “symbolic” play, and is in fact made by Bentele in another context: the toy is a sign, only to the extent that the child takes it to represent the real thing, which cannot be true, for instance, in the case of a toy lion if the child has no experience of the real animal.

Bentele insists on the implication that signs must be intentional. However, since it is notoriously difficult to ascertain the presence of intentions, I would rather require them to manifest forms characteristic of social routines which are normally taken to be intentional. In any case, the important requirement here, I contend, is actually that the act which is repeated should function as the expression of which the act or object imitated is the content, not just as a repetition of it.

Imagery and memory are not mentioned by Bentele. However, it is not clear that they involve any real differentiation between expression and content; at least, Husserl would deny it (as discussed in Sonesson 1989,III.3.5-6.). In any case, since other psychologists doubt their very existence, or at least that they are not yet other instance of propositional thinking, it seems very hazardous to claim a particular period for their coming into being.

This leaves us with much fewer manifestations of the semiotic functions than Piaget suggests: verbal language, drawing, together with some instance of play and imitation. In addition, however, there are certainly many other embodiment of the semiotic function not mentioned by Piaget: gesture, for instance, perhaps music, and so on.