Course overview
This course will introduce the basic tools and concepts in information theory, and practical coding schemes for data compression and transmission over noisy channels. The first part of this course is about information theory. The course aims at answering two fundamental questions in data communications: What is the ultimate data compression? What is the ultimate transmission rate of a communication channel? To answer these questions, the basic tools and concepts in information theory are introduced. Practical coding schemes for data compression and transmission over noisy channels are introduced in the second part of this course. Review of Mathematical Background and Basic Concepts in Probability Theory. Introduction of Information Measures and their Properties: Entropy; Mutual Information; Conditional Entropy; Conditional Mutual Information; Kullback-Leibler Divergence; Log-sum inequality; Data Processing Inequality; Fano's inequality. Asymptotic Equipartition Property (AEP): Law of Large Numbers; Types and Typicality. Lossless source coding theorem and algorithms. Noisy Channel Models and Shannon's Channel Coding Theorem. Fundamentals of Linear Codes: Introduction to Error Detection and Correction; Types of Codes; Minimum Distance. Block Coding Principles: Generator Matrix Description, Systematic Codes, Detection and Correction Bounds. Convolutional Codes (Encoding): encoders; generator polynomials; constraint length; state diagrams; tree and trellis diagrams; distance measures. Convolutional Codes (Decoding): Viterbi algorithm; Soft-Decision Decoding; MAP Decoding. LDPC Codes: Code Density, Tanner Graphs, Belief Propagation. Special topics: Network Coding and Digital Fountain.
Course learning outcomes
- Examine mathematically the notion of information contained in data and the fundamental limits to compress data for efficient storage.
- Analyse and compare the roles of redundancy in combating the effect of noise in various communications media.
- Develop the connections between information-theoretic schemes and algebraic/convolutional/graph-based schemes for communicating data over discrete memoryless channels..
- Examine and derive the operations involved in different encoders and decoders, and be able to compare the advantages and disadvantages of various coding paradigms.
- Use problem-solving and computing skills to analyse and derive efficient means of representation, storage and transmission of information in communications networks.