Language and information

Lecture 3: Information

3.6. The structure of information

[Audio recording—Opens a separate, blank window. Return to this window to read the transcript.]

Now, we began with information as an undefined thing, without knowing even if it had a structure. We found some constructions on relations in language having information—those two, the partial ordering and the likelihood differences. We found other constraints which had some relation to information, or no relation to information, but they did not create the substantive information. We of course have examples of unstructured things in language which carry information, like alliterative allusion, which certainly carries certain kinds of information—in language. There are things in speech, though perhaps not in language, which carry information, like the individual voice, which tells us who is speaking, or intonations of anger and surprise and so forth, which give us a lot of information, and hesitations. They cannot be fitted into language—into grammar, that is—though they are in speech. And of course there is information outside of language, in graphs and charts, in photographs, even when you hear a bird in a tree without seeing it you have information that a bird is in the tree. So there are certainly kinds of information which are outside the structure of language, but there is no question that there is an awful lot of information in the structure of language, and that it is a unique kind of information. Not only in amount, it is extremely specific—but of course the information in charts and graphs is also specific—but in any case it certainly is an important thing.

Now, for this information, the information in language, which is all I will speak about in the next few minutes, we find that there are structural contributions to the information. The set of contributions to information, meaning these two constraints which do the information, but over the whole language they are a subset of the set of individual constraints—meaning of every occurrence of every constraint in the language as a whole—meaning they are a subset of the redundancies of the language, the redundancies which are imposed by the first two constraints of partial order and likelihood, is a subset of the total redundancies in the language. The language carries an awful lot of redundancies, due to the reductions, due to even changes in the linearization, to various things, but these two redundancies are a subset of the total set. Now, if we look at the sentences in the base, which are the ones which have only this material, we see that they contain meanings: the meaning of the operator, and the meaning of the argument, but they also contain some additional meaning that the sentence contains over and above the meaning of the words. It is true that if you throw ten words together, without any grammar, it will still give a certain extremely fuzzy—fuzzy is too strong a word, or too weak a word—kind of meaning. It will be some penumbra of meaning, in any collection of words. But this is not what we get when we have a specific sentence, like John slept, or something like that. Then we have a very definite meaning, and this meaning is given by the organization, by the partial order, in other words. The partial order carries a meaning over and above the meaning of the participating words. What does this mean?

It means that the partial order, which is an imposition of redundancy, is added to the choice of words, which is also an imposition of redundancy—choosing one word or choosing another word. Throughout the language, one can argue that information is a redundancy on redundancies. It is not simply more than one redundancy. First of all, it is more than one redundancy, meaning there's more than one meaning. But it is more than that. At least speaking for language—and there are good reasons to argue this, once we have seen it for language, also for other cases of information, but I don't want to say that. In language we see that information consists in having redundancies on redundancies—redundancies acting on redundancies. And this is the main structural point about information, which I think comes out from the description of the relation of information to structure that we have seen so far.

Now, from this, I want to move to something very much as an aside. We see here that in the case of language, instead of describing language as something which nobody else in the world has, namely grammatical relations, or as something which perhaps some things do have, some things don't have, like predication, or something like that, some important, meaningful things that language has, we see that we were able to describe language by and large—there may be problems—in terms of departures from randomness of things, from randomness of, let's say, sounds. We see that phonemes are a departure from randomness of sounds, of allophones, and words are a departure from randomness of phonemes, and so on. And we have seen that not only is there a departure from randomness, that is to say there is redundancy, but that the whole of language can be described, rather closely, step by step, as contributions, as individual, itemizeable and characterizeable contributions to this redundancy. Then we saw that certain of these contributions to redundancy went hand-in-hand with their information. They gave the substantive information of the sentence. And that if the sentence was a redundancy on redundancy, the information was a redundancy on redundancy. The information—this kind of information at least—has a certain kind of structure of departures from randomness, not just the departure from randomness, but a structure of this. In information theory, the amount of information was studied, and it was studied as being limited by the total amount of the redundancy of the system, or the channel in which the information was being carried or expressed. Information theory of course did not deal with the relations between individual contributions to the total redundancy, and it did not characterize the individual items of information which were involved. In language we are able both to characterize the contributions and to get down to the items of information. Now, this raises afresh a certain point, because it makes it possible, perhaps, to assay the amount of structure, the structure of non-randomness of information with whatever is the structure of non-randomness in what it is talking about. Because the world, of course, is itself, naturally, a very complex system of non-randomnesses. To try to think about it or to speak about it is hopeless. It may, in certain very very limited sciences be a little bit more possible to describe. But we have here something which describes language and information, namely, these departures from randomness, but it also is a fact of the real world. And, while I am far from suggesting that we could connect the two things together, however, it does raise the question of what 'refer' means. Because we can say that language refers to things in the world; a sentence refers to an event; a paragraph or an article refers to a set of things that are closely interrelated. It does it by having non-randomnesses of words and of meanings, which create information. But if we can say that it doesn't really refer to it as some kind of a completely arbitrary vehicle 'it refers', but that there is some kind of a correspondence between the non-randomnesses that go into the language and information, and the non-randomness—the structure, no matter how abstract—that goes into what is being described. On may have something that explicates the relation of 'referring'. Certainly, in a very detailed science language, one sees that the nouns and verbs that one is dealing with, and the conjunctions even that one is dealing with—not in the forms in which they are in the grammar, but in the form in which you have to simplify them finally when you make the purest science language form—you see that they are really reasonably corresponding to the things in the laboratory, to the actions that the person does, and so forth. To something in the world, which this material is describing.

Now, a last thing. One might ask whether the whole analysis as it has been presented here to date is a theory of language or a theory of information. I'd like to summarize that. For language, this effort started as a theory of all constraints, something which was deemed necessary because [break; repaired with repetition] of the lack of a metalanguage. But it was found, first, that the initial combinatory constraints were included in everything else that happened to language, so that everything carried those two constraints, and second, that they carried all the information. That those two constraints carried all the information. This was from language to information. From the point of view of information, of the information in language, it was found that this information, was given step by step by the initial, basic constraints—these two constraints—and only by these. In this sense, the theory is both a theory of the basic structure of language, and also a theory of language information.