TY - JOUR
T1 - Short branch attraction in phylogenomic inference under the multispecies coalescent
AU - Liu, Liang
AU - Yu, Lili
AU - Wu, Shaoyuan
AU - Arnold, Jonathan
AU - Whalen, Christopher
AU - Davis, Charles
AU - Edwards, Scott
N1 - Publisher Copyright:
Copyright © 2023 Liu, Yu, Wu, Arnold, Whalen, Davis and Edwards.
PY - 2023
Y1 - 2023
N2 - Accurate reconstruction of species trees often relies on the quality of input gene trees estimated from molecular sequences. Previous studies suggested that if the sequence length is fixed, the maximum likelihood may produce biased gene trees which subsequently mislead inference of species trees. Two key questions need to be answered in this context: what are the scenarios that may result in consistently biased gene trees? and for those scenarios, are there any remedies that may remove or at least reduce the misleading effects of consistently biased gene trees? In this article, we establish a theoretical framework to address these questions. Considering a scenario where the true gene tree is a 4-taxon star tree (Formula presented.) with two short branches leading to the species S1 and S2, we demonstrate that maximum likelihood significantly favors the wrong bifurcating tree [(S1, S2), S3, S4] grouping the two species S1 and S2 with short branches. We name this inconsistent behavior short branch attraction, which may occur in real-world data involving a 4-taxon bifurcating gene tree with a short internal branch. If no mutation occurs along the internal branch, which is likely if the internal branch is short, the 4-taxon bifurcating tree is equivalent to the 4-taxon star tree and thus will suffer the same misleading effect of short branch attraction. Theoretical and simulation results further demonstrate that short branch attraction may occur in gene trees and species trees of arbitrary size. Moreover, short branch attraction is primarily caused by a lack of phylogenetic information in sequence data, suggesting that converting short internal branches to polytomies in the estimated gene trees can significantly reduce artifacts induced by short branch attraction.
AB - Accurate reconstruction of species trees often relies on the quality of input gene trees estimated from molecular sequences. Previous studies suggested that if the sequence length is fixed, the maximum likelihood may produce biased gene trees which subsequently mislead inference of species trees. Two key questions need to be answered in this context: what are the scenarios that may result in consistently biased gene trees? and for those scenarios, are there any remedies that may remove or at least reduce the misleading effects of consistently biased gene trees? In this article, we establish a theoretical framework to address these questions. Considering a scenario where the true gene tree is a 4-taxon star tree (Formula presented.) with two short branches leading to the species S1 and S2, we demonstrate that maximum likelihood significantly favors the wrong bifurcating tree [(S1, S2), S3, S4] grouping the two species S1 and S2 with short branches. We name this inconsistent behavior short branch attraction, which may occur in real-world data involving a 4-taxon bifurcating gene tree with a short internal branch. If no mutation occurs along the internal branch, which is likely if the internal branch is short, the 4-taxon bifurcating tree is equivalent to the 4-taxon star tree and thus will suffer the same misleading effect of short branch attraction. Theoretical and simulation results further demonstrate that short branch attraction may occur in gene trees and species trees of arbitrary size. Moreover, short branch attraction is primarily caused by a lack of phylogenetic information in sequence data, suggesting that converting short internal branches to polytomies in the estimated gene trees can significantly reduce artifacts induced by short branch attraction.
KW - coalescent methods
KW - gene trees
KW - long branch attraction
KW - multispecies coalescent model
KW - short branch attraction
KW - species trees
UR - http://www.scopus.com/inward/record.url?scp=85164971667&partnerID=8YFLogxK
U2 - 10.3389/fevo.2023.1134764
DO - 10.3389/fevo.2023.1134764
M3 - Article
AN - SCOPUS:85164971667
SN - 2296-701X
VL - 11
JO - Frontiers in Ecology and Evolution
JF - Frontiers in Ecology and Evolution
M1 - 1134764
ER -