S NAGA MALLESWARA REDDY

 In this post we will discuss some fundamentals of source coding and in later posts we will write codes for some source coding algorithms. We will like to remind our readers that we are right now discussing the basics of simulation and soon we will go for OFDM and more advance topics. So lets hit it!

Mutual Information:

The information content provided by the occurrence of the event Y = yi about the event X = xi is defined as

Mutual Information

I(xi;yj) is called the mutual information between xiand yj.

Self Information:

when we talk about the information of a single event X= xi we called it self information and is denoted as

Self Information

it is a matter of fact, that , a high probability event conveys less information than a low probability event. That is, P(x) = 1 which means I(x) = 0.

Conditional Self Information:

In addition to self information and mutual information, conditional self information as

Conditional Self Information

Average Mutual Information:

the mutual information associated with the pair of the events (xi,yj), which are possible outcomes of the two random variables X and Y, we can obtain the average value of the mutual information by simply weighting I(xi,yj) by the probability of occurrence of the joint event and summing over all possible joint events.

Mathematically

Average Mutual Information

as a matter of fact, I(X;Y) = 0 when X and Y are statistically Independent. Also I(X;Y) ≥ 0.

Entropy:

Entropy or Average self information denoted as H(X) and defined as

Entropy

Conditional Entropy:

the average conditional self information is called conditional entropy

Conditional Entropy

we also get the relation from the above the equations

Relation Between Information and Entropy

Here is a plot of binary conditional entropy produced in matlab. In next post we will discuss some left over theorems of source coding like Kraft's inequality and will start some real coding work !

binary entropy function

GSM Public Land Mobile Network (PLMN)

GSM frequencies include 2 bands at 900 MHz and 1800 MHz.

GSM 900:

124 Radio Carriers, assigned in two sub bands of 25 MHz in 890 – 915 MHz and 935-960MHz ranges, channel widths 200kHz.

Each carrier is divided into frames of 8 time slots, frame duration 4.6ms.

DCS-1800:

2 sub bands of 75MHz in 1710 – 1785 MHz and 1805-1880MHz.

In evaluating the performance of a digital communication system, it is often necessary to determine the area under the tail of the PDF. there for there are uppr bound on this and they are mentioned below:

1. Chebyshev inequality

X is an arbitrary random variable with finite mean mx and finite variance σx2. For any positive number δ , the chebyshev inequality is defined by

Chebyshev inequality

2. Chernoff bound:

The Chebyshev bound involves area under the two tails of the PDF. In some applications we are interested only in the area under on tail either in the interval (δ, ꝏ) or in the interval (- ꝏ, δ). In such a case , we can obtain an extremely tight upper bound by over bounding g(Y) by an exponential having a parameter that can be optimized to yield as tight an upper bound as possible. By considering tail probability in the area (δ, ꝏ) , the function g(Y) is over bounded as

Chernoff Bound

where g(Y) is defined as

Chernoff Bound (equation 2)

and v >= 0 is the parameter to be optimized.

Central Limit Theorem:

"The sum of statistically independent and identically distributed random variables with finite mean and variance approaches a Gaussian CDF as n → ꝏ."

The proof of this theorem can be found any where in good text book. The matlab code for the proof will be uploaded if I will enough request. By the way the matlab code is very simple.

In the whole discussion so far we have not discussed about SAMPLING THEOREM. Which we will discuss in near future, under the title "Visiting back to Basics". From now onwards we will discuss some practical aspects of digital communication i.e. less theory (as usual )and more codes and results.

Binomial Distribution:

Let X be a discrete random variable that has two possible values says X = 1 or X = 0 with probabilities of p and 1-p respectively. Now suppose, we define Y as

pdf of Y will be

and CDF is

the above equation characterizes a binomially distributed random variable.

The first two moments of Y are :

E(Y) = np

E(Y2) = np(1-p) +n2p2

σ2 = np(1-p)

Uniform Distribution:

first two moments are:

E(X) = ½ (a + b)

E(X2)= 1/3(a2+b2+ab)

σ2 = 1/12 (a-b)2

Gaussian (normal) distribution:

PDF

CDF is

and erf(x) denotes the error function, defined as

in terms of erfc(x) CDF is given as

where erfc(x) = 1 – erf(x)

"The sum of n statistically independent Gaussian random variables is also Gaussian random variable"

Chi- Square (gamma) Distribution (with n degrees of freedom):

central chi square distribution

PDF

Cdf

First two moments are

E(Y) = nσ2

E(Y2) = 2nσ4 + n2σ4

σ2y = 2nσ4

for non central chi square distribution refer page nujmber 42 – 44 of Digital Communications 4/e John G. Proakis

Rayleigh Distribution

This is one of the most important probability distribution in Communicaiton Theory. This distribution is frequently used to model the statistics of signals transmitted through radio channels such as cellular radio. This distribution is closely related to the central chi-square distribution.

Let Y=X12 +X22 where X1 and X2 are zero mean statistically independent Gaussian random variables, each having a variance σ2 . It follows U is chi square distributed with two degrees of freedom. That means PDF of Y is

now, let define a new Random variable R which is

PDF of R is

CDF is

and moments are

E(Rk) = (2σ2)k/2Γ(1 + ½ k)

and the variance is

σ2r = (2 – ½ π)σ2

Lognormal Distribution:

a random variable X is normally distributed with mean m and variance σ2. Let us define a new random variable R which is related to X through X = ln R . Then the pdf of R is

The lognormal distribution is suitable for modeling the effect of shadowing of the signal due to large obstructions, such as tall building, in mobile radio communications.

Matlab Functions and Codes:

In this post, we will discuss about the probability functions and we will see the important characteristics of them along with the equivalent matlab functions.

Random Variables:

Given an experiment having sample spaces S and elements s ɛ
S, we define a function X(s) whose domain is S and range is a set of numbers on the real line. The function X(s) is called random variable.

Probability Distribution Function / Cumulative Distribution Function (CDF)

Given a random variable X, let us consider the event {X ≤ x} where x is any real number in the interval (-∞, ∞). We write the probability of this event as P(X≤ x) and denote it simply by F(x), i.e.

F(x) = P(X≤ x)     –∞ < x < ∞

Function F(x) is called Probability Distribution Function, also known asCumulative Distribution Function (CDF).

Probability Density Function (PDF)

The derivative of CDF is known as probability density function (PDF) and it is denoted as p(x). Mathematically:

Probability Density Function PDF

Equivalently

Probability Density Function PDF

This is for continuous variables, but in case of discrete random variables PDF is expressed as

Discrete Probability Density Function PDF

Statistical Averages of Random variables:

The first and second moments of a single random variable and the joint moments such are correlation and covariance between any pair of random variables in a multidimensional set of random variables are of particular interests.

For all the cases below: X is single random variable and its PDF is p(x).

Mean or Expected value of X:

This is first moment of the random variable X. E() denotes expectation (statistical averaging)

Mean or Expected Value of X

The n-th moment is defined as

nth Moment

if mx is the mean value of the random variable X, the nth central moment is defined as

nth Central Moment

Variance:

when n = 2 the central moment is called the variance of the random variable and denoted as σ

Variance

variance provides the measure of the dispersion of the random variable X.

Variance

Joint moment:

in case of two random variables X1 and X2 with joint PDF p(x1,x2) we define joint moment as:

Joint Moment

joint central moment as:

Joint Central Moment

when k = n = 1, we get correlation and covariance of the random variables X1and X2 i.e. Correlation between Xi and Xjis given by joint moment

Correlation

and covariance of Xi and Xj is

Covariance

the n x n matrix with the elements µij is called the covariance matrix of the random variables Xi, i = 1,2,……..n

The two random variables are said to be uncorrelated if

E(XiXj) = E(Xi)E(Xj) = mimj

which means µij =0.

i.e. Xi and Xj are statistically independent then they are uncorrelated but vice versa is not true.

Two random variables are said to be orthogonal is

E(XiXj) = 0

this condition holds when Xi and Xj are uncorrelated and either one or both of the random variables have zero mean

The equivalent Matlab functions so far:

Matlab Tools