Before going to contextual bandit problem, let us discus what the multi-armed bandit problem is.