Depicts autoencoders as end-to-end communication systems.
In essence, any communication system is comprised of a source of information, a transmitter, a channel, a receiver and a destination.
Mathematically, any communication system can be discribed by using the following formalism. The source of information, Bob, wants to transmit a uniformly chosen message S, such that
,
in n discrete uses of the channel. To this end, a transmiter operates on the message to create a signal suitable for transmission and it generates a lower level representation of message S (i.e., converts the message S of size M to a message T of size n). As a result, the communication rate of the message is R = k/n [bits per channel use], where
.
The transmiter imposes a power constraint on the message T (per bit, or on average), and the message T is then transmitted over the channel. The channel is the transmission medium and it is corrupted by Gaussian noise. The ratio between the input power and the power of the Gaussian noise is called the signal-to-noise ratio (SNR). As a result of the channel noise, the receiver receives a distorted version of the message T, denoted by T'. The receiver operates on the received message and generates a higher level representation of message T' (i.e., converts the message T' of size n to a message S' of size M). The destination, Alice, then decodes the transmitted message.
This project mimics a communication system with an autoencoder in TensorFlow.
The input layer of the autoencoder is of size M, and it takes in a one-hot representation of the uniformly chosen message S. The first hidden layer, of size n, creates a lower level representation of S. The output of this hidden layer is then corrupted by Gaussian noise and is used as an input to the second hiddel layer of size n. The output layer, of size M, is a higher level representation of hidden layer 2. The decoded message is found as argmax(output layer).
The adopted metric is the bit-error-rate (BER), which quantifies how many bits of the one-hot representation of the output message differ from the one-hot representation of the input message. For higher SNR the BER is lower, and conversly, lower SNR means higher BER. For high enough SNR, the BER should be zero.
The code requires:
- TensorFlow
$ pip install TensorFlow
- Numpy
$ pip install numpy
Check out http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf for further info on communication systems, and http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/ for a very good and concise explination on autoencoders.