MDP 与 ADP:非终止状态分布的构造
整理自印象笔记导出(2023)。原始导出:2023印象笔记导出
Constructing the distribution of non-terminal states for Approximate Dynamic Programming (ADP) algorithms is a crucial but challenging task. Your provided information outlines different strategies based on the type of ADP algorithm and the nature of the Markov Processes involved. Let's delve deeper into this:
Approximate Policy Evaluation and Value Iteration:
-
Using Uniform Policy:
- When you have an MDP, evaluate it with a uniform policy (each action has equal probability).
- Convert the resulting MDP to its Markov Reward Process (MRP).
- Determine the stationary distribution of this MRP, which becomes your distribution of non-terminal states.
-
Special Properties:
- For finite-state Markov Processes, you might utilize matrix operations to infer the stationary distribution. This means, by repeatedly applying the transition matrix, one can converge to a stationary distribution.
-
Simulation:
- When the aforementioned methods are not feasible, a simulation-based method becomes a practical alternative.
- Simulate the MRP using methods like
simulateinMarkovRewardProcess. - After a large number of steps, gather the distribution of non-terminal states. This is a form of estimating the stationary distribution.
-
Uniform Distribution:
- If other methods are computationally heavy or infeasible, use a uniform distribution across all non-terminal states.
Backward Induction ADP Algorithms for Finite-Horizon MDPs/MRPs:
-
Mathematical Structure:
- Leverage the inherent structure of the Markov Process. If it has a specific structure that can be mathematically analyzed (like being derived from a stochastic differential equation), then solving these equations can yield analytical distributions.
-
Sampling:
- When the mathematical analysis isn't feasible, sampling can help.
- Sample incrementally from the state-transition probabilities of the MRPs at each time step.
- If working on a control problem with an MDP, first convert it to its corresponding MRP using a uniform policy.
- The states reached during sampling at specific time steps form the SampledDistribution for that step.
-
Uniform Distribution for Finite Horizon:
- For each time step in the horizon, if other methods are not feasible, use a uniform distribution across non-terminal states.
Cross-Disciplinary Insights:
-
Philosophy: The choice of distribution speaks to our beliefs and understanding of the problem. A uniform policy assumes all actions are equally likely, reflecting an epistemic humility about the dynamics of the environment.
-
Sociology: In societal decision-making, policies often emerge from a combination of historical data (akin to the stationary distribution) and heuristic choices (akin to the uniform distribution).
-
Psychology: Human decisions often rely on a blend of experience (previous outcomes) and heuristics (simple rules). The simulation-based method is reminiscent of how people project potential futures based on their current actions.
Relevance to Your Goal:
Given your interest in automated trading systems using reinforcement learning, understanding how to construct these distributions is fundamental. The distribution you choose can greatly influence the learning and eventual performance of your model. Sampling methods might be especially relevant in trading where the dynamics of the market can be complex, and deriving an analytical solution might be difficult.