The embryonic development of the pineal organ, a neuroendocrine gland on top of the diencephalon, remains enigmatic. Classic fate-mapping studies suggested that pineal progenitors originate from the lateral border of the anterior neural plate. We show here, using gene expression and fate mapping/lineage tracing in zebrafish, that pineal progenitors originate, at least in part, from the non-neural ectoderm. Gene expression in chick indicates that this non-neural origin of pineal progenitors is conserved in amniotes. Genetic repression of placodal, but not neural crest, cell fate results in pineal hypoplasia in zebrafish, while mis-expression of transcription factors known to specify placodal identity during gastrulation promotes the formation of ectopic pineal progenitors. We also demonstrate that fibroblast growth factors (FGFs) position the pineal progenitor domain within the non-neural border by repressing pineal fate and that the Otx transcription factors promote pinealogenesis by inhibiting this FGF activity. The non-neural origin of the pineal organ reveals an underlying similarity in the formation of the pineal and pituitary glands, and suggests that all CNS neuroendocrine organs may require a non-neural contribution to form neurosecretory cells.