The impact of environment on human health is dramatic, with major risk factors including substance use, diet and exercise. However, identifying interactions between the environment and an individual’s genetic background (GxE) has been hampered by statistical and computational challenges. By combining RNA sequencing of whole blood and extensive environmental annotations collected from 922 individuals, we have evaluated GxE interactions at a cellular level. We have developed EAGLE, a hierarchical Bayesian model for identifying GxE interactions based on association between environment and allele-specific expression (ASE). EAGLE increases power by leveraging the controlled, within-sample comparison of environmental impact on different genetic backgrounds provided by ASE, while also taking into account technical covariates and over-dispersion of sequencing read counts. EAGLE identifies 35 GxE interactions, a substantial increase over standard GxE testing. Among EAGLE hits are variants that modulate response to smoking, exercise and blood pressure medication. Further, application of EAGLE identifies GxE interactions to infection response that replicate results reported in vitro, demonstrating the power of EAGLE to accurately identify GxE candidates from large RNA sequencing studies.