How a biological and artificial decision makers represent their inputs is critical for their performance. However, the representational needs of uninformed policies in the early phase of learning are quite different than those of more optimised ones later on - and the early representational choices mustn’t prevent the policy evolving into the latter. This problem is often addressed using heuristics in machine reinforcement learning. We propose to formulate choosing representational trajectories as meta-cognitive decision making, and to use planning in the space of representations to construct a normative solution. We explore the resulting representational dynamics in the simple setting of finite contextual bandits, and demonstrate that particular decisions to fine-grain or coarse-grain the agent’s representation depend not only on temporal discounting, but also on the constraints on the computational resources available to the agent. Finally, we propose an experimental paradigm to test the predictions of representational planning regarding human behaviour.