The world we perceive is delayed in relation to its flowing content, as well as the outcome of our actions on the world in relation to the moment we decide to act. This mosaic of different latencies permeating both perception and action has to be taken into account critically in order for us to cope with the temporal challenges constantly imposed by the environment. Fundamental notions, such as the sense of agency and causality, depend on the temporal relationship of events occurring in well-defined windows of time. Here, we offer a broad, yet abridged, historical view of some thought-provoking issues concerning the time of perception and action. From the pioneering work of Wundt, Titchener, and Libet to recent findings and ideas related to the employment of visual illusions as psychophysical probes (such as the flash-lag effect), we have tried to expose some problems inherent to the act of measuring the time of both perception and action, and devise possible solutions as well.