Attention can be voluntarily directed to a location or automatically summoned to a location by a salient stimulus. We compared the effects of voluntary and stimulus-driven shifts of spatial attention on the blood oxygenation level-dependent signal in humans, using a method that separated preparatory activity related to the initial shift of attention from the subsequent activity caused by target presentation. Voluntary shifts produced greater preparatory activity than stimulus-driven shifts in the frontal eye field (FEF) and intraparietal sulcus, core regions of the dorsal frontoparietal attention network, demonstrating their special role in the voluntary control of attention. Stimulus-driven attentional shifts to salient color singletons recruited occipitotemporal regions, sensitive to color information and part of the dorsal network, including the FEF, suggesting a partly overlapping circuit for endogenous and exogenous orienting. The right temporoparietal junction (TPJ), a core region of the ventral frontoparietal attention network, was strongly modulated by stimulus-driven attentional shifts to behaviorally relevant stimuli, such as targets at unattended locations. However, the TPJ did not respond to salient, task-irrelevant color singletons, indicating that behavioral relevance is critical for TPJ modulation during stimulus-driven orienting. Finally, both ventral and dorsal regions were modulated during reorienting but significantly only by reorienting after voluntary shifts, suggesting the importance of a mismatch between expectation and sensory input.