Critique

I think your use of pseudo code is good and I can see the contribution of the second paper by making a DP variant! It worked well as a unit too.

Paper 1

• Should it be $r(s_{i},\pi (s,h),h)$ or can ${\displaystyle }$ be any number of time steps?
• I was wondering how SAH is computed if S is a set of states and A is a set of actions.
• What $\phi$ represents is not explicitly mentioned.

Paper 2

• Related to paper 1, Q-functions and Q-learning are mentioned and a link is given but it is not really elaborated on. I think it may help to write a few sentences on Q-learning in the background information.

Minor Corrections

Abstract

• as well intro the -> as well as the pseudo-code of the two algorithms and their PAC and regret guarantees

Paper 1

• S a finite set of states -> S is a finite set of states
• s.t. could be written out as such that
• for all time and state -> for all times and states

Notible -> Notable

Under Intuition of JDP, there is a [todo] remaining.

Should it be $r(s,\pi (s,h),h)$ or can $i$ be any number of time steps?*