The second condition will be partially answered in this chapter, by showing how much more efficient the planner is than those that either use a logical, rather than a probabilistic belief model, and those that use no user model whatsoever. However, in designing a planner that adapts itself to a model of the user, reinforcement learning must be considered as a competitor (see Section 2.9.1). Reinforcement learning systems use data about dialogues with users to reinforce dialogue strategies, and so they can be said to adapt to the user. Due to the difficulty of implementing both the planner and a reinforcement learning system, this has been deferred to future work, but is an important comparison to make. It is expected that in situations where there is little training material, the planner would perform better than a reinforcement learning system, especially where the plans are novel. Such plans require the intelligent application of planning knowledge, and reinforcement learning fails in this respect, relying instead on the brute force of training data. Novel dialogue planning might occur in, for example, a meta-level dialogue about a complex domain-level plan. There will be further discussion of this question in the future work chapter, Chapter 6.
In summary, the chapter will describe a simulation method for evaluating the system, and compare this method with the preferable, but impractical one of direct evaluation with users. The evaluation makes use of two examples that are typical of the kind of problem where strategies create a decision surface, that is, the choice between them depends on the belief state of the agent. A user model is of benefit only for problems that have a decision surface.