Presentation
Markov chains are stochastic processes characterized by a property of "independence of the past and the future given the present". Markov chains have many applications as statistical models of real-world processes.
One can cite performance evaluation (of queuing systems), signal processing (audio, video), digital communications (error correcting codes), control of stochastic systems, stochastic optimization, biology (genome modeling), or mathematical finance.
The course will start with an introduction to Markov chains. We will use the newly learned formalism to study queuing systems and popular MCMC methods. Then the Markov chain model will be extended to hidden Markov models, or Markov chains with noise. To end we will see how a stochastic system can be controlled optimally with a Markov decision process.
Prerequisites
Probabilities, scientific programming with Python.
Duration:
42h
Content
- Markov chains
- Poisson process and exponential distribution
- Queuing systems
- Monte Carlo methods and Advanced Monte Carlo methods (MCMC)
- Hidden Markov Models
- Markov Decision Processes
Organization
Examination
Final Exam and Practicals.
Scheduled activities
- C1-C2 (3h) Probabilities and Markov Chains
- C3-C4 (3h) Queuing Systems
- PC1-2 (3h) Markov Chains, Token Bucket
- C5 (1h30) Queuing Systems
- PC3 (1h30) PageRank Algorithm
- TP1 (3h) M/M/1
- PC4-5 (3h) M/G/1, MMPP/M/1
- C6-C7 (3h) Monte Carlo experiments
- C8-C9 (3h) Advanced Monte Carlo methods
- TP3 (3h) Stochastic Simulation (I)
- TP4 (3h) Stochastic Simulation (II)
- C10-C11 (3h) Parametric/Non parametric modeling
- TP5 (3h) Hidden Markov Models
- C12 (1h30) Optimal Control of a Stochastic System
- TP6 (3h) Markov Decision Processes
- Exam (1h30) Final Exam
Team
|
