Upwelling of cold, nutrient-rich water from the oxygen minimum zone (OMZ) off Peru sustains the world’s highest production of forage fish, mostly composed of anchovy (Engraulis ringens). However, the potential impacts of climate change on upwelling dynamics and thus fish productivity in the near future are uncertain. Here, we reconstruct past changes in fish populations during the last 25,000 years to unravel their response to changes in OMZ intensity and productivity. We quantified and identified fish scales and bones deposited in laminated sediments from Pisco (Peru) with an average sampling resolution of 20.4 years (±7.1). The records span the Last Glacial Maximum to the recent Holocene and thus encompass a variety of combinations of productivity, oxygen, and global temperature. Our results reveal that productivity appears to be the main factor controlling small pelagic fish abundance, while sub-surface oxygenation affects mainly anchovy and likely sardine populations. Lower productivity and higher oxygen concentrations during the glacial resulted in lower total fish productivity, whereas higher productivity and a stronger OMZ in some time intervals during the Holocene resulted in higher fish abundances. A variety of different conditions between these two oceanographic end members indicate preferred environmental conditions for a variety of small pelagic fishes. There is no evidence in our record for an out of phase relationship between anchovy and sardine at the timescales examined in the present study. Anchovy have been the predominant small pelagic fish throughout the record, at least over centennial to millennial timescales. Its abundance reached a maximum during the Current Warm Period, an era characterized by high productivity and intense OMZ conditions. Thus, industrial fisheries developed during a period of exceptional productivity in relation to that of the last 25 kyr. The records reveal that dramatic decreases in pelagic fish abundances have occurred in response to past large-scale climate changes than those observed in the instrumental period, which suggests that future climate change may result in substantial changes in ecosystem structure.