Demonstration of Usability of Theory of Large Deviations

Abstract

Extreme and rare events keep on happening and this makes them not so rare. In the course of the thesis, the major causes for the occurrence of rare events are studied. Out of all the reasons, estimation of probability turns out to be important and is further studied. It is sometimes the key factor which leaves people ill prepared to deal with the extremes. A study of theory of large deviation is done and its usability in the estimation of probability is shown. An algorithm to estimate rate function, which deals with the tails of probability distributions is presented. The results are compared with the existing data of stock market and it has been shown that empirical and normal distributions are not sufficient to deal with the probability of extremely rare events as they tends to underestimate it. Theory of large deviations provide a way to deal with the tails in a separate way, leaving aside the behavior at the mean to the parent distribution. It has been shown that theory of large deviation provides an upper bound of the probability at the tails both analytically and numerically, also the mean square error of the estimates of large deviation theory as compared to normal approximations is less. Along with this, a measure for quality of a classifier (in machine learning) is created which is inspired by theory of large deviation. The measure is created to estimate the performance of a classifier for which the training data is imbalanced. It is further compared with existing measure and is shown to estimate something new.