Comparing Bayes factors and hierarchical inference for testing general relativity with gravitational waves

In the context of testing general relativity with gravitational waves, constraints obtained with multiple events are typically combined either through a hierarchical formalism or though a combined multiplicative Bayes factor. We show that the well-known dependence of Bayes factors on the analysis priors in regions of the parameter space without likelihood support can lead to strong confidence in favor of incorrect conclusions when one employs the multiplicative Bayes factor. Bayes factors O(1) are ambivalent as they depend sensitively on the analysis priors, which are rarely set in a principled way; additionally, combined Bayes factors > O(10³) can be obtained in favor of the incorrect conclusion depending on the analysis priors when many O(1) Bayes factors are multiplied, and specifically when the priors are much wider than the underlying population. The hierarchical analysis that instead infers the ensemble distribution of the individual beyond-general-relativity constraints does not suffer from this problem, and generically converges to favor the correct conclusion. Rather than a naive multiplication, a more reliable Bayes factor can be computed from the hierarchical analysis. We present a number of toy models showing that the practice of multiplying Bayes factors can lead to incorrect conclusions.