|
Hilbert College. |
Annotated Bibliography: Information Theory |
- Home - Students - Tutorials - Student Papers - |
|
Abramson, Norman. Introduction. Information Theory and Coding. New York: McGraw-Hill, 1963. Information theory is a questionable name to apply to a scientific discipline. Shannon's theory deals with the carriers of information, not with information itself. Symbols must obey certain laws if they are going to transit information. Shannon maintains that "semantic aspects of communication are irrelevant to the engineering problem" while Weaver claims that the engineering aspects of communication may be relevant to the semantic, psychological, and linguistic aspects. The importance of binary code is stressed, going back as far as Matthew 5:37 where a reference is made. Questions are raised about how far we can go, in terms of shortening the number of binits per message.
Ash, Robert. Preface. Information Theory. New York: Interscience Publishers, 1965. While communication theory is regarded as having been founded by Shannon and Wiener, their approaches are quite different. In the Shannon model, a randomly generated message is "encoded" before being transmitted through a channel. When the output is received a "decoding" operation is performed. The properties of the channel and concern for the different channel models are fundamental. In the Wiener model, however, a random signal is sent through a fixed channel model. The channel adds a randomly generated "noise". In this case, the design of the decoder is important.
Cherry Collin. On Human Communication 'A Review, a Survey, and a Criticism'. Cambridge: MIT Press, 1978. One of the most important technical developments that helped spawn the information theory is that of telegraphy. Alexander Graham Bell's concept of the telephone and the invention of the television have also greatly influenced the development of the theory. The 'noise problem' has always received much attention. In the technical sense, noise refers to any disturbance or interference present in the channel. Noise is the ultimate limiter of communication.
---. "The Communication of Information (An historical Overview)". American Scientist. Vol. 40, No 4 (October 1952): 640-645. The theoretical work of the information theory has early historic origins. "Communication" is defined as "… that which links any organism together". Communication involves language of symbolism. As the world becomes more technical, languages have increased in simplicity in order to avoid redundancy. Egyptian hieroglyphics were very redundant as each syllable had a number of different symbols. Semitic languages such as Hebrew recognized redundancy and thus had no vowels. Shannon estimated that English has a redundancy rate of 50%. Ciphering, or code, came out of redundancy. Used for military and diplomatic secrecy, it has many historic uses. Francis Bacon developed early binary code when he suggested that one could use two slightly different founts, proving what early societies had known - information could be coded into a two-symbol code.
Crowley, D.J. (1999) Understanding Communication - The Signifying Web. New York: Gordon and Breach, Science Publishers, Inc., 1982. Shannon and Weaver, two telephone engineers at the Bell Laboratories, produced the first formal transmission model of information. They suggested that the transmission of information from one place to another depended on three criteria: 1) the condition and capacity of the channel through which the communication occurs 2) the influence of noise from the general environment and 3) the amount of redundancy in the message itself. Wiener's idea of feedback became a fourth criterion. Weaver and Shannon stated repeatedly that the model did not apply to human communication. Nonetheless, Wilbur Schramm adapted it.
DeFleur, Melvin L. and Otto N. Larsen. The Flow of Information. New York: Harper Brothers, 1958. In a general model of communication, the operation performed by the transmitter is called encoding. The receiver reverses the operation of the transmission and reconverts the coded message by decoding it. In the process of decoding, there is always the possibility of error. The errors may occur in any stage of transit. All sources of error are put in the same category - noise.
Fano, R. M. Proceedings of the Symposium on Information Networks, April, 1954: Information Theory and Generalized Networks Microwave Research Institute Symposia Series Volume III (Brooklyn: Polytechnic Press, 1955). The information theory may be linked to the Network theory. When information moves through a network, a communication channel is created. This channel may, for instance, consist of a series of binary switches, which are controlled by the input.
Fiske, John. Introduction to Communication Studies. New York: Methuen and Company, 1982. Shannon and Weaver identified three levels of problems in the study of communication. Level A (technical problems) - How accurately can the symbols of communication be transmitted? Level B (semantic problems) - How precisely do the transmitted symbols convey the desired meaning? Level C (effectiveness problems) - How effectively does the received meaning affect conduct in the desired way?
Gallager, Robert G. Introduction. Information Theory and Reliable Communication. New York; John Wiley and Sons, Inc., 1968. The provision of block diagrams allows one to visualize the behavior of communication systems. The distinguishing characteristics of Shannon's theory are a greater emphasis on probability theory and a primary concern with the encoder and decoder. The theory became much more precise following its release. The splitting of the encoder and decoder is stressed since it makes the design of the channel encoder and decoder virtually independent of the source encoder and decoder, using binary date as an interface. This allows the use of different sources on the same channel. Related to the encoding of binary data is the measure of information of the letters of a source alphabet.
Hartley, R.V. L. "Transmission of Information", Bell System Technical Journal Vol. 7 (1928): 535-543. In order to communicate on any level, there must be a physical group of symbols such as words, dot and dashes, etc… The sender chooses a symbol and by some means relays it to the receiver. By filtering out all other symbols present in the line of communication, the receiver is able to make the information more precise and understand the intended message.
"Information Theory". Academic American Encyclopedia. 1996 ed. Also known as the theory of communication, the information theory is divided into two branches, one for continuous and the other for discrete information systems. The continuous theory deals with the amplitude, wavelength, and frequency of communication signals. The discrete theory is more easily applied and was developed for both noiseless and noisy channels. The latter theory can organize letters and words that approximate the English language. "Information Theory". Encyclopedia Americana. 1996 ed.There are a wide variety of applications for the information theory including pure and applied mathematics, communication theory, cybernetics, computer science, translating machines, genetics, psychology, and medical diagnosis. There is a mathematical formula for calculating the information content of a single symbol. It is also important to measure the average information content of an entire source, counting the totality of symbols for transmission. "Information Theory". Encyclopedia Britannica. 1997 ed. There are some central problems with the information theory. During transmission the signal may be distorted or changed. An example of this is snow on the television screen and static heard on radio channels. These disturbances are known as 'noise factors'.
Information theory is defined as the measurement of the quantity and quality of information. Shannon discovered the measurement for the entropy of a message. Frequent signs are coded with fewer signals while rare signs are coded with more signals. More frequent sings have a higher probability and therefore contain less information, while less frequent signs have a lower probability and therefore contain more information. Wiener claimed that information was the third basic component of our natural world, following matter and energy. Six different types of information have been labeled: structure approach; knowledge approach; message approach; meaning approach; effect approach; and process approach.
Shannon and Weaver created a diagrammatic representation for the message transmission theory. The model is geared towards the engineering aspects of communications. Throughout the years, the theory has been expanded by Shannon and Weaver and others.
Many have proclaimed a divorce between information theory and semantics, which is not a reasonable statement. Semantics find a place within the information theory when diagrams take into account the nature of these terminals as "goal-directed self-adaptive systems". Questions and commands can be analyzed in informational terms. The amount of information received by an organism can be measured by measuring the logical work that it does for the organism.
The biggest fundamental problem of communication is reproducing messages at one point that have been sent from another point. The message must be selected from a set of possible messages. The communication system must be designed to accommodate each selection - not just the one that is chosen. If there are a set number of messages in the set, then the number is a measure if the information produced when one message is chosen from the set. For many practical reasons, the logarithmic measure is more convenient.
In 1949 Warren Weaver offered a definition of communication. It states, "All of the procedures by which one mind may affect another." This may be applied to communication between machines and humans. Human communication may be summed up by two transformations. They consist in the following four phases: 1) The sender must make clear for himself exactly what to communicate 2) He must choose symbols that externalize the internal content (sound, gestures, body motions, words, intonation, etc. - the first transformation) 3) The receiver must assimilate these symbols despite disturbances. That is, hear the transmission, know the used language, interpret nonverbal information 4) He must thereafter integrate all received symbols and transform them into an internal content (the second transformation).
Van der Lubbe, Jan C. A. Preface. Information Theory. Cambridge: Cambridge University Press, 1997. There were several forerunners to Claude Shannon. H. Nyguist raised the question of how messages could be sent over a telegraph channel at the maximum speed without distortion. R. V. L. Hartley was the first to attempt to define "a measure of information". By incorporating the concept of chance and probability, Shannon laid the foundation for what today is known as the information theory.
When Wiener defined cybernetics, he classified communication and control together. He later regretted this decision because humans exercise their control over their environment through commands. In control and communication, we are always fighting nature's tendency to degrade the organized and destroy the meaningful. Cybernetics develops language and techniques that will enable the problem of control and communication to be attacked.
|
||