Direct photons do not interact strongly with the medium produced in relativistic heavy-ion collisions, which makes them an exceptional observable: at intermediate pT direct photons provide information on the early stage of heavy-ion collisions. At high pT they constitute a crucial reference for the medium modification of high-pT hadron production which has been observed. PHENIX has found a significant excess of direct photons above the background from decay photons in Au+Au collisions at 200 GeV at both high and intermediate pT. Furthermore, direct photon production in p+p and d+Au collisions has been measured, allowing a fundamental test of QCD as well as constraining the effects of cold nuclear matter. A systematic study of direct photon production will be presented and put into the context of theoretical models.