In spite of decades of linkage and association studies and its potential impact on human health, reliable prediction of an individual’s risk for heritable disease remains difficult. Large numbers of mapped loci do not explain substantial fractions of the heritable variation, leaving an open question of whether accurate complex trait predictions can be achieved in practice. Here, we use a full genome sequenced population of 7396 yeast strains of varying relatedness, and predict growth traits from family information, effects of segregating genetic variants, and growth measurements in other environments with an average coefficient of determination R2 of 0.91. This accuracy exceeds narrow-sense heritability, approaches limits imposed by measurement repeatability, and is higher than achieved with a single replicate assay in the lab. We find that both relatedness and variant-based predictions are greatly aided by availability of closer relatives, while information from a large number of more distant relatives does not improve predictive performance when close relatives can be used. Our results prove that very accurate prediction of heritable traits is possible, and recommend prioritizing collection of deeper family-based data over large reference cohorts.