Theoretical studies of pathogens frequently focus on the time scale of epidemiological and evolutionary processes at the host population level. In viruses such as influenza and human immunodeficiency virus (HIV) however, mutations are generated sufficiently quickly that phenotypic change can occur over the shorter time scale of a single infection. I will describe some stochastic process models for this within-host evolution, both in the specific cases of influenza and HIV, as well as in a general framework where selection acts concurrently at both within-host and between-host levels. The modeling techniques used range from stochastic simulations for the development of broadly neutralizing anti-HIV antibodies, to the theory of measure-valued processes applied to a ball-and-urn formulation of multilevel selection. In the case of seasonal influenza and HIV, these studies suggest improvements on current or planned vaccination strategies which may not have been identified without a specific within-host model. At a more abstract level, they also suggest general rules of thumb for the evolution of pathogens that experience antagonistic selection between and within hosts.