Genetic epistasis and dominance are nonlinear effects

In my genetics class, I define epistasis as (1) type of gene interaction, and (2) one gene “masks” the phenotypic effect of the other gene. In classical Mendelian genetics, a single gene determines a single trait. Epistasis moves us closer to real world genetics as an example of gene interaction in which two or more genes "determine" (in classical Mendelian sense) a trait. 

But the second part of the definition is opaque: What does it mean for one gene to mask the phenotypic effect of the other? More clarity is provided by the definition of epistasis in an advanced quantitative or human genetics course (Lynch and Walsh, 1998): "Epistasis describes the nonadditivity of effects between loci (genes)". In a linear model of genotypic effects, you sum the phenotypic contribution of each allele for all the genes that contribute to the phenotype.

Epistasis and dominance represent the deviation (residual error) from this linear model. In other words, the nonlinear (nonadditive) interaction between alleles of a single gene is termed dominance, and the nonlinear interaction between alleles of different genes is epistasis. The "masking" part of the epistasis definition refers to the fact that there are fewer phenotypic classes than genotypic classes. In a linear (additive) model, each genotypic class should give rise to a distinct phenotypic class (i.e. the genetic factors sum up to a different value), but with nonlinear interaction some of the genotypic classes will produce the same phenotype. In an analogous fashion, Mendelian dominance gives rise to fewer phenotypic classes than genotypic classes. For example, the heterozygous genotype Aa and the homozygous gentoype AA exhibit the same dominant phenotype given that the A allele is dominant to the a allele. 

I have observed that this "masking" concept does not make a lot of sense to the students, and they tend to use the terms epistasis and gene interaction interchangeably. This is one example in which the more advanced quantitative definition is better and should be used if the students have been taught the concept of a linear (additive) model, which is essentially fitting a line (i.e. $y = ax + b$) to data points. 

The Lynch and Walsh textbook, Genetics and Analysis of Quantitative Traits, nicely explains dominance and epistasis in Chapters 5 and 6. Starting with the Fundamental Law of Genetics (slightly modified version from previous post), we have that the phenotypic value z equals the genotypic value $G$ (contribution of genetic factors) plus the environmental deviation (residual) $E$: $z = G + E$. $G$ in turn is the sum of the effects of the various relevant alleles on the phenotypic value. For example, my genotypic value for systolic blood pressure may be 110 mmHg, and then an unhealthier than average diet may contribute another 20 mmHg for a measured blood pressure of 130 mmHg.

Say there is a gene B that affects blood pressure and this gene has two alleles $B_1$ and $B_2$. We can propose a linear (additive) model in which each copy of allele $B_1$ contributes $\alpha_1$ to the blood pressure (above or below the mean), and each copy of allele $B_2$ contributes $\alpha_2$ to the blood pressure. Then in this example, the genotypic value $G = \mu_G + \alpha_1 N_1 + \alpha_2 N_2$ in which $\mu_G$ is the mean genotypic value in population, $N_1$ represents the number of $B_1$ alleles and $N_2$ represents the number of $B_2$ alleles in the individual. Focusing for the moment on the $B_2$ allele in Figure 1, one can plot our linear model for $G$ versus $N_2$ (black line with slope $\alpha_2$).

Fig 1. The linear dependence of the genotypic value $G$ on the number of alleles. The linear model (i.e. line) produces the estimate $\hat{G}$ (unfilled circles). Unfortunately life being what it is, the observed phenotypic data $G$ (filled circles) do not reside on the line. The deviation $G - \hat{G}$ is referred to as the dominance $\delta$. This plot just happens to depict Mendelian dominance, but other nonlinear relationships can give rise to dominance effects. Adapted from Figure 4.6 of Lynch and Walsh.

Let’s say that $B_2$ is dominant over $B_1$ in the classical Mendelian sense (and causes higher blood pressure). Then the genotypic values of $B_1 B_2$ (1 allele of $B_2$) and $B_2 B_2$ (2 alleles of $B_2$) should be the same (black dots located at genotypic value of dominant phenotype), whereas $B_1 B_1$ possesses a lower value (recessive phenotype). These data points cannot be fit by a line, and the deviation from the linear model is represented by $\delta$ which is referred to as the dominance term. As a reminder, in Mendelian genetics there is a dominant allele (B) and a recessive allele (b). Both the heterozygote Bb and the homozygote BB express the dominant phenotype. The homozygous recessive bb exhibits the recessive phenotype. 

To repeat myself, the residual error (dominance) $\delta = G - \hat{G}$ is the deviation between the observed genotypic value $G$ and the linear estimate $\hat{G}$. Thus, the full one gene equation is $G = \mu_G + (\alpha_1 N_1 + \alpha_2 N_2 + \delta_{12}$). Classical Mendelian dominance can give rise to this deviation, but really any nonlinear interaction between alleles of the same gene can cause nonadditive "dominance" effects.

Now let’s consider the situation in which two genes (A and B) influence the same trait (i.e. an example of gene interaction). We can write this as $G_{ijkl} = \mu_G + (\alpha_i N_i + \alpha_j N_j + \delta_{ij}) + (\alpha_k N_k + \alpha_l N_l + \delta_{kl})$ in which $i$ and $j$ represent the alleles for gene A and $k$ and $l$ represent the alleles for gene B. The total genotypic value is the sum of the contributions from each gene. Visually, one can picture plotting $G$ on the y-axis and the right-hand side of the equation on the x-axis which will result in a line. 

As before, the observed phenotypic data will most likely not reside on the line, and the deviation between the observed data and the estimated genotypic value is denoted $\epsilon_{ijkl}$ which stands for the epistasis. As defined above, epistasis represents the nonadditivity between the two genes. Thus, our final equation for $G$ in the two gene case is:

$G_{ijkl} = \mu_G + (\alpha_i N_i + \alpha_j N_j + \delta_{ij}) + (\alpha_k N_k + \alpha_l N_l + \delta_{kl}) + \epsilon_{ijkl}$

We have the additive terms ($\alpha$'s), dominance terms ($\delta$'s), and epistasis term ($\epsilon$) with $\mu_G$ being the population mean. The genotypic value is the sum of the effects of gene A in the first parenthesis and gene B in the second parenthesis plus the epistasis, which is not accounted for by adding together the individual gene contributions.

TL;DR: The quantitative genetic models describing phenotypes are fundamentally linear models in which nonlinear effects such as epistasis and dominance are “error” terms, and this formulation may predispose researchers to drop them from further analyses.