Genomic instability causes cancers to acquire hundreds to thousands of mutations and chromosomal alterations during their somatic evolution. Most of these mutations and alterations are termed passengers because they do not confer cancer phenotypes. Evolutionary simulations and cancer genomic studies suggested that mildly-deleterious passengers accumulate, collectively slow cancer progression, reduce the fitness of cancer cells and enhance the effects of therapeutics. However, these effects of passengers and their impact on clinical variables remain limited to genomic analysis. Here, to assess passengers’ effect on cell fitness and cancer, we specifically introduced increasing passenger loads into human cell lines and mouse models. We found that passenger load dramatically reduced cancer cell’s fitness in every model investigated. Passengers’ average fitness cost of ~3% per MB, indicates that genomic instability in cancer in patients can slow tumor growth and prevent metastatic progression. We conclude that genomic instability in cancer is a double-edged sword: it accelerates the accumulation of adaptive drivers, yet incurs a harmful passenger load that can outweigh drivers’ benefit. Passenger load could be a useful biomarker for tumor aggressiveness and response to mutagenic or passenger-exacerbating therapies, including anti-tumor immunity.