Statistician Brian Blais has attempted to criticize Tim's and my use of a full partition between R and ~R (the resurrection and no resurrection) in evaluating the evidence for the resurrection. He has called this "inflating the Bayes factor with nothing." I intend to do a video series on my Youtube channel on various skeptical attempts to avoid using a partition, which is the actually confusing approach to modeling the impact of evidence. That series will talk about other skeptics who have tried something similar and what's epistemologically wrong with doing so.
Meanwhile, I thought I'd rescue from obscurity in a comments thread some comments I made in immediate response when Blais posted his criticisms. These are a little unorganized, but here they are for now.
There actually is no problem with this analysis (using a full partition), significant or otherwise. You simply prefer, for unknown reasons, to analyze an empirical inference in what to me seems like a confusingly roundabout way, in which one evaluates the specific evidence set E for some salient hypothesis H (e.g., testimony to some event, alleged video of that event, or what-not) by comparing the prior probability of H to the prior probability of some highly gerrymandered theory, ~H1, which is a subhypothesis of ~H specifically generated to give probability to the entirety of the set E that is equal to its conditional probability on H.
It is not even clear that there always exists such a ~H1 (unless one simply considers a non-explanatory declaration "~H and E, somehow" to be a hypothesis), and such a procedure is absolutely riddled with opportunities for epistemic confusion. In contrast, the odds form of Bayes' Theorem, so far from being misleading, yields the correct answer even in your "exercise for the reader" and does so with ease.
There you create a distribution in which the prior odds are 99/1
against H and the Bayes factor is 39/1 in favor of H from some evidence or set
of evidence E. The use of the odds form shows immediately that of course E
increases the probability of H but is not sufficient to swamp the prior odds.
We have there posterior odds of 39/99 (H over ~H), and an extremely simple
calculation (39/(99 + 39)) yields the posterior probability of H as approximately
.28.
I have actually seen a professional probability theorist confused by not using a partition in evaluating the evidence for miracles, to
the point that he switched between a naturalistic hypothesis
which was the conjunction of a set of theories specifically evolved to account
for the evidence for the resurrection (and frankly, even so it didn't do a good job!) and the individual conjuncts, or naturalism more broadly. In other
words, he treated the statement that a miracle has a low prior probability as
equivalent to saying that any naturalistic hypothesis, however convoluted and
consisting of however many independent parts conjoined together, evolved to
account for the evidence for the miracle, will always have a higher prior
probability than that of the miracle.
Needless to say, "Naturalism" is not the same thing as "this incredibly specific conjunctive naturalistic hypothesis that we dreamed up in a blatantly ad hoc manner so as to try to account for the evidence for this specific miracle." I believe that the failure to think in terms of partitions is at least partly to blame for this sort of error, which is rife.
Finally, I stress that I deem the full partition Bayes factor analysis to be the most natural way to conceive of evidence for any specific event for which there is specific evidence. When we say, "The evidence for a moon landing is very strong," a very natural way to think of that is as a reference to the specific evidence (videos, interviews, etc.) that man actually landed on the moon, not to more general evidence such as a knowledge of the state of technology at the time and whether it would have been good enough for a moon landing. The latter is relevant to the prior. The former is the specific evidence for the event.
I submit that considering a Bayes factor using a partition makes a lot more sense than instead saying, "How does the prior probability of a moon landing compare to the prior probability of no moon landing plus a conspiracy hypothesis specifically gerrymandered to account for this whole set of evidence that we have?" If there actually were a subhypothesis of the negation that gave identical probability to the whole of the specific set E to that given by H (and I question that), and if one were able to think about it with perfect clarity (also unlikely) and deduct from the prior of the subhypothesis all that needed to be deducted for the various ad hoc embellishments one would need to make, the final result would be the same. But I can't imagine why anyone would try to go about it in such a way, and I submit that if we were considering a variety of mundane events for which we have specific evidence, no one would think of preferring such a method.
One more point: There is a longstanding and lively debate on the best way to measure evidential force. Various methods are suggested, such as measuring the difference between the prior and the posterior of H--P(H|E) - P(H). Another proposed measure, sometimes called the r measure, is P(H|E)/P(H), and there are more candidates. Tim and I favor the likelihood ratio, sometimes called the L measure, which is P(E|H)/P(E|~H). Sometimes probability theorists use the log of the L measure.
In any event, I don't know of anyone who suggests that we should measure the force of evidence in the round-the-barn fashion that Brian is suggesting: "Take a highly ad hoc version of ~H, assuming that you can find one, that gives precisely the same conditional probability to the set of evidence E that H gives to E, while leaving the probability of E given all the rest of the partition as 0, then somehow or other calculate the prior probability of this subhypothesis ~H1, then calculate the posteriors of H and ~H as constituting a new partition in the same ratio as the priors of H and ~H1. Good luck with that."
The whole point of finding a good measure of the force of specific evidence E is to figure out the most intuitive way to measure what E actually does for or against H qua evidence, not to try as hard as possible to make it look like E isn't doing anything. Part of the reason we favor the L measure (an extremely well-known measure, favored by plenty of people who wouldn't be caught dead being Christians) is because it separates out the power of the specific evidence so cleanly from separate considerations that feed into the priors. In any event, our use of the L measure has precisely nothing to do with weird religious apologists engaging in some sort of probabilistic skulduggery.
Posts
43 comments:
Please tell me how I’m wrong here because I don’t think this post addresses it. First, the only probability measure guaranteed to follow the postulates as described by ET Jaynes and others is the posterior (not the likelihood or the prior, or the bayes factor or prior odds). This includes the postulate that “equivalent states of knowledge are given equivalent probability assignments”. Second, you state my analysis is correct where I show that you can arbitrarily increase the bayes factor by insisting on a partition and including as many models as you’d like that don’t even attempt to address the hypothesis. This process doesn’t change the posterior — because it involves equivalent states of knowledge — but does affect the Bayes factor. I don’t see how the bayes factor can be an interesting measure if this property is true, and I definitely don’t see it as productive to focus on Bayes factors over posteriors. Where am I going wrong here? why should I care about “a good measure of the force of specific evidence E” as opposed to the probability that my hypothesis is true given the data?
I'll be honest: I have *no idea* what you mean by saying that the posterior is guaranteed to follow probability axioms (I assume this is what you mean by postulates) but that the priors or Bayes factors do not. Seriously, I don't know what you could mean by that. To a really subjective Bayesian (which I'm not), like say Richard Jeffrey, *all* of these would end up being subjective, with the only constraints being a) consistency within a given distribution and b) proper application of updating by Bayes's Theorem and/or Jeffrey Conditioning. And JC requires the rigidity of the posteriors, so it is also rule-governed as an updating rule and still follows the same axioms. But since (to a subjective Bayesian) the priors can be fairly subjective, as long as they produce a coherent distribution, the posteriors will be as well.
To a more objective Bayesian like me, all of these should be true to the actual state of the evidence, though of course we humans are fallible in our applications and estimates.
But all of these agree: There is nothing more rule-governed or objective or true to probability about the posteriors than about the priors or likelihoods. In fact, the priors and likelihoods work together to produce the posteriors. These are all inextricably linked.
In all arguments for or against a historical event (be it the moon landing, the Battle of Gettysburg, the resurrection of Jesus, the Covid lockdowns in Michigan, etc. etc.) there will be a body of evidence that *purports* to indicate that this event happened. That's what allows us to talk about it, even if it's something that didn't happen (e.g., the Donation of Constantine). And it will be at least roughly possible to separate out this evidence that draws our attention to the alleged event (that is prima facie evidence for it) from the myriad other considerations that would previously have been relevant to the prior.
Most specific historical events are *improbable* in the prior distribution.
It is therefore completely relevant and indeed very important to evaluate the nature and force of that *particular* evidence in order to have a well-grounded posterior. Even a person who previously thought that man would never land on the moon should be reasonably convinced now, in virtue of the available evidence, that man did land on the moon. Why? Because the *specific evidence* is very strong.
It's impossible to overleap the historical question of the force of the specific evidence and come to a reasonable conclusion about the posteriors.
And the force of the specific evidence *for or against that event* cannot be rightly understood if you don't look at it using a full partition. What you are calling irrelevant hypotheses or "models that don't even attempt to address the hypothesis" are in fact merely broad descriptions of portions of ~H that factor into the *actual force of the evidence*, which is *unavoidable* in evaluating the posterior.
No, you can't "arbitrarily increase the Bayes factor" by including those. That's...just wrong. If H is *confirmed*, it is confirmed *overall*--i.e. vs. ~H. H goes up if and only if ~H goes down, and vice versa. Emphasizing the fact (if it is a fact in a particular case) that if H didn't occur it is overwhelmingly improbable that we'd have this evidence at all is a way of getting at the fact that the evidence very strongly favors H and can swamp a low prior of such-and-such. (The backsolving we did in our 2009 paper.)
Responding to your last comment first. In the post I was referring to, I don’t think I have an example with a BF of 2/1. Most of the post was analytical and doesn’t use numbers. The “exercise for the student” at the bottom has a BF of 39 tipped toward H but a posterior ratio of 2.5 against. If you change the prior for the nothing model to p(N)=.994 and the others .005 and .001 you’ll see that the BF goes up but the posterior is unchanged because we haven’t changed our state of knowledge about our hypotheses by increasing the weight of a model that is irrelevant.
Meant to put the link. Not sure if those go through: https://bblais.github.io/posts/2026/May/14/how-to-inflate-your-bayes-factor-with-nothing/
For the postulates/axioms/desiserata (whatever you want to call them) of probability I find the paper by Tom Loredo From Laplace to Supernova to be the shortest but clearest writing on this.
Jeffreys is ok but he’s a bit old. I prefer the work of E T Jaynes. He has a lot about subjective vs objective, etc… all his works are easily found online and my perspective is best described by Jaynes. Hope this helps!
Yes, I'm looking right at your post. I put the link to it in the o.p., above. Not that it matters, but E.T. Jaynes and Richard Jeffrey were almost exactly the same age. Again, that doesn't matter. Perhaps you thought I meant Harold Jeffreys?
Anyway. throwing around E.T. Jayes's name isn't really helping you. Or anybody else's name. Look: There is *nothing* about a posterior distribution that somehow is guaranteed to follow objective rules of probability whereas the likelihoods are not. If anything it has sometimes been thought (I don't actually agree with this, btw) by someone like Elliot Sober and others, that the likelihoods are *more* objective, easier to get a grip on, or whatever. As I say, I don't actually agree with that, especially not in history where it's not like we're dealing with urns or machines with known properties.
Jaynes was a pioneer of maxent Bayesianism. I've always been fascinated by maxent Bayesianism and at least I do support putting a flat distribution down over a finite number of possibilities in the prior distribution where these are epistemically symmetrical. One can also use epistemic symmetry for setting likelihoods in some cases (as I said, urns, machines with known mechanisms). Things aren't often that neat, but one should try anyway where there is epistemic symmetry. Again, this has nothing to do with the posterior being guaranteed to follow probabilistic principles better than the prior or better than the likelihoods.
The posterior distribution is *just a distribution*, just as the prior distribution is *just a distribution*. When there is some E and some H in which we are interested (as in a case of an alleged historical incident), the posterior, which is P(H|E) arises from the prior P(H) and the likelihoods on the partition P(E|H), P(E|~H). There is *nothing* about the posterior P(H|E) that is more "guaranteed" to follow probabilistic principles or anything of that kind than any of these other quantities, or even the ratio P(E|H)/P(E|~H) (which Tim and I think is sometimes easier to access psychologically than the separate numerator and denominator).
In over twenty years of working in probability theory, I have never known anyone, in any paper, to even suggest such a thing about the posterior. What is it (specifically) do you think it is in E.T. Jaynes that supports your statement? Quote and explanation, please.
I hate to have to say this so bluntly, but it looks to me like you are just confused and even sort of flailing.
Of course, you're welcome to set up a distribution in which the Bayes Factor *is* 39/1. Go for it. And if you envisage a prior for H of .01, it will (again) be easy to read right off those numbers that the posterior of H will not be greater than .5. It will not be swamped. because 39/1 BF in favor of H obviously can't swamp or even match prior odds of 99/1 against. This is all pretty straightforward, and there is literally nothing wrong with the method we use in the paper on this point. That's why we did the backsolving: Here's what we think the BF is; this could swamp a prior as low as ____ leaving a posterior of _____.
I had not previously read Laredo's paper. I'm reading it now. Notice that he (of course correctly) gives a version of Bayes' Theorem as (translating into my notation) P(H|E) = P(H)P(E|H)/P(E). Equation 5 on p. 87. I note that the total prior probability of E must use a full partition of the distribution, because the prior probability of E equals the sum of the following terms: P(H1)P(E|H1) for all propositions H1 through Hi which make up the full partition. H will be confirmed to some degree or other (though it could be a large or small degree as far as just this information goes) just in case P(E|H) is greater than P(E|~H), where ~H refers to the disjunction of all propositions in the partition other than H. Note that when you have a partition of more than one hypothesis, multiple hypotheses can be confirmed or disconfirmed, but that if H is confirmed, then ~H must be disconfirmed.
Again, there is absolutely nothing in this that says or means or implies that the posteriors are guaranteed to follow probability postulates but the likelihoods or priors are not. That would make no sense at all.
Terminologically, I also note that Laredo is at times considering Bayes factors that do not involve a full partition--that is, a factor showing only relative odds of the likelihoods as between some H1 and H2 that are not jointly exhaustive. One can do that, but doing so will not provide a posterior. In fact, where H1 and H2 are a non-partition, it is possible for both of them to be confirmed or both to be disconfirmed by evidence E.
Okay, apologies for the comments above (now deleted) in which I incorrectly stated that the BF is 2/1 in your exercise. Interestingly, I did not make that mistake in the o.p. Funny, don't know why I slipped up like that in the comments but not in the o.p.
As you state, and as I state in the o.p., your BF is in fact 39/1. And, once again, it's easy to see that given the stated prior, this BF will not swamp it, because 39/1 in favor can't swamp 99/1 against.
To your point about raising the BF and having no effect upon the posterior: This is because you have lowered the prior of H by the same degree to which you have increased the Bayes factor! Obviously if you do that, the posterior will end up the same. That's because the posterior is an effect of combining the prior and the likelihood. A higher Bayes factor (using a partition) indicates greater confirmation. But when you put together greater confirmation with a lower prior, set lower to exactly such a degree as to offset the greater confirmation, then of course you end up with the same posterior!
If you held constant P(H) at .01 and increased what you call the "nothing" portion at the expense merely of the hypothesis that gives .5 probability to E, then the posterior of H would be higher.
> Perhaps you thought I meant Harold Jeffreys?
Yes indeed! Thanks for the correction. đŸ˜€
> throwing around E.T. Jayes's name isn't really helping you
My bringing up Jaynes is not some argument from authority or “throwing the name around”. It’s just to point out where I’m coming from since his writings are readily available. The phrase “equivalent states of knowledge are given equivalent probability assignments” I’m taking from Jaynes’s writing although others have used it.
When I say that likelihoods do not follow the postulates of probability it’s at least in the violation of the equivalent states of knowledge principle. How do I know this? Because I can construct a very small and simple case where I get different bayes factors for equivalent problems, but that the posterior and posterior ratios give the same value.
> To your point about raising the BF and having no effect upon the posterior: This is because you have lowered the prior of H by the same degree to which you have increased the Bayes factor!
You phrase this as if I’ve set the prior myself to cancel the bayes factor. I didn’t. Look at my derivation in the “The answers have to be the same” section. The scaling term appears in both the likelihood and prior terms automatically due to the partition and the properties of the nothing model. I make the comment “This last equation matches my solution, and the Bayes Factor in the previous equation matches the McGrew's Bayes Factor”. The entire derivation shows that my bayes factor matches yours, and by changing only the weight of the nothing model I can make it as big as I want without changing the posterior for the primary model.
> Is it possible: Could you be conflating "Some Hi gives probability 0 to evidence E" with "E and Hi are probabilistically irrelevant to each other"? I hesitate to suggest this, since it's clearly incorrect (in fact, E entails the *falsehood* of that Hi), but since you have used the word "irrelevant" I can't help wondering.
I wish there were equation numbers in the html. I’ll have to look into that. đŸ˜€ In the first section, last equation, you do see that any p(data|{anything}, N)~0 where the nothing model predicts nearly anything, so the specific evidence/data is unlikely. However there is this lingering term from the part of ~H that is occupied by the gerrymandered model as you call it (which is a small part) the bulk of that space occupied by the nothing model. This seems to match your bayes factor in the examples you give. This same term is inverted in the prior automatically as the last section shows.
This is the simplest example I could think of that seems to reproduce both the bayes factor you describe with the partition and my calculations of the posteriors. Is there a better example?
"Because I can construct a very small and simple case where I get different bayes factors for equivalent problems, but that the posterior and posterior ratios give the same value."
What? You are here referring to your case suggested in the comment above, I take it, where the prior of H is .001 instead of .01? But the prior ratio in this new set-up of P(H)/P(~H) is 999/1, whereas the prior ratio in your exercise case is 99/1. So the prior ratios are not the same. Again, the Bayes factor makes no pretense to represent the priors. Of course not. It represents the force of the evidence.
There is no particular *necessity* in what you have set up here in this new case as far as the priors are concerned. Of course you can't keep H at a prior of .01 as in the original exercise, because you've chosen to make N so much higher, and there isn't space for anything else to have a prior of .01 once N has a prior of .994. (The priors do have to sum to 1.) But, for example, you could set the new, higher prior of what you're calling N more "at the expense of" what you're calling M than you have chosen to do.
E.g. We could set P(N) = .994 (as you now suggest), P(M) = .003, P(H) = .003. These priors sum to one as of course they must. Keep the likelihoods for the individual hypotheses the same as you set them in the exercise: P(E|N) = 0, P(E|M) = .5, P(E|H) = 1. Then the overall P(E) = .0045. And of course the Bayes Factor is higher because, as you say, the proportion of the ~H space that gives the evidence a probability of 0 has been increased.
And if I'm doing my math right (which I might not be doing at this time of night, but I think I am), I'm coming up with P(H|E) as approximately .67.
By the way, I'm not entirely sure what you mean by saying that you're using the McGrews' Bayes factor. Perhaps I'm just missing something, but our suggested BF in the paper for the testimony of the disciples is not 39. It's 10 to the 39th power. There's a pretty big difference there.
I should say the prior ratio in the new example is 999/1 against H--P(~H)/P(H).
I'm sitting here just staring at your derivation. I cannot for the life of me see how you get that final attempted reduction after the words, "Which reduces to." (Unfortunately I can't copy and paste the equations over here into the combox, but that's the only time that the phrase "which reduces to" occurs.) That doesn't reduce to that at all, as far as I can see. The ratio of the priors between H and ~H should always be P(H)/P(~H) *regardless* of the fact that some portion of ~H (which you've labeled N) gives probability 0 to the data. That absolutely doesn't matter at all. It's not like you should just make that part of the prior of ~H disappear because the P(E|N) = 0. Why in the world would anybody think such a thing? That's completely wrong. If, for example, the prior of H is .01 (as in your first exercise) then P(H)/P(~H) = .01/.99 equals 1/99. (In other words, 99/1 against.) It makes not the slightest difference to this that somewhere else in our distribution we're setting the probability of E given some segment of ~H as 0. And similarly, if P(H) is .001, as in your more recent example here in this thread, then P(H)/P~H) = .001/.999, i.e. 999/1 against H.
The ratio of H to ~H doesn't magically change into the ratio of H to M just because M is the only portion of ~H (as you've set up the distribution) that gives a non-zero probability to E.
So something seems to have gone very far wrong with this derivation at that step.
sure, the prior ratio of P(H)/P(~H) has changed, but that's not surprising -- the nothing model is going to be way more probably a-priori than any of the specific models we choose, either H or M, by quite a lot! my first numbers were probably not even close to real-case values, but were used for instructional purposes. the point here is that it **shouldn't matter**. Models that don't predict the data at all do not contribute to our state of knowledge, and thus should not contribute to our probability assignments. the prior ratio of H to the actual alternative M here has not changed -- still 5 to 1, and the explanatory value of those two models has also not changed -- M is still 2 times worse. The posterior has also not changed -- still 2.5 against H. Changing the weight of the nothing model should not concern us in the slightest -- it disappears in the posterior calculations I did, and it cancels in the bayes-factor x prior ratio that I showed. The fact that it is a concern to you, focusing on the bayes factor alone, seems to be a bug not a feature.
Actually, we backsolve in the article for a prior (a very low prior) that could be swamped by the Bayes factor we suggest.
And once again: There is no *necessity* that if you increase the prior probability of what you're calling "the nothing model" that that has to be done in some precise way that doesn't penalize the M model any more than it penalizes H. There is no reason why it should. if N is .994, H is .003, and M is .003, of course we get a very different result.
Since you acknowledge that in the changed situation you describe, the Bayes factor (with a full partition) has changed (as of course it has), then you have to look at the fact that the prior ratio *with a full partition* has also changed. You can't mix and match, saying, "I kept the prior ratio the same" (speaking of the ratio with the two non-partition hypotheses) "But the Bayes factor did change" (now speaking of the BF ratio on the full partition) "But the posterior was the same, so therefore there must be something probabilistically screwy about the Bayes factor ratio with a full partition." No, if you consistently look at the BF with a full partition and the prior with a full partition, you can see that of course the higher Bayes factor (with full partition) can't overtake the now-lower prior for H.
Or, if we were consistent in the other direction, the Bayes factor *without* a partition *would* be 2/1 in both cases, and that would not change either!
Look, this whole statement that if a part of the partition gives a value of 0 to the data, that does not "contribute to the state of our knowledge" is wrong.
You can see this in an urn set-up. Suppose you have three urns, and the ball you receive must come from one of those three. Urn 1 has all black balls. Urn 2 has 1/2 black balls. Urn 3 has no black balls at all. The decision about which urn to draw from will be made by a process such that the probability of drawing from Urn 1 is .01, the probability of drawing from Urn 2 is .05, and the probability of drawing from Urn 3 is .94. A ball is drawn, and it is black. What is the probability that it came from Urn 1?
Knowing that the prior of coming from urn 1 is .01 is obviously relevant to this calculation, as is knowing that the highest-prior urn also has zero black balls.
If we are then told that the prior choice procedure has now changed and that urn 3 now has a probability of .994 of being drawn from, there is *absolutely no reason* why this implies that the prior ratio between Urn 1 and Urn 2 must remain the same. Maybe it does; maybe it doesn't. If the procedure has changed, for all we know the prior ratio between those two might have changed as well. Moreover, if we're told that *in fact* the prior of Urn 1 is now .001 and of urn 2 is now .005, it is then no surprise that the higher Bayes factor (using a full partition--P(E|Urn 1/P(E|~Urn 1)) is insufficient to swamp this yet-lower prior of Urn 1--in fact, if this happens to be how the total prior situation has changed, they are perfectly offset from one another. But they *need not have been*. There is nothing in either probability or epistemology that dictates that such a perfect offset must occur if the prior of N changes.
In our paper we used the Bayes factor with a full partition, and in our backsolving we were describing a very low prior that could be completely swamped by such a high overall Bayes factor.
When you set up a BF (with a full partition) of 39/1 and a prior for H of .01, it is *completely obvious* that that Bayes factor (which is much lower than the one we suggest for the evidence for the resurrection) cannot offset that prior. The full prior odds are 99/1 against in that example. These examples therefore do not show anything wrong with our method. I just don't know how to be clearer on that.
btw I think I see now what you're doing in your derivation. The problem then seems to come in what I call mixing and matching up above. It kind of blows my mind that you're under the impression that the BF with full partition somehow doesn't follow the probability axioms or that there must be something wrong with it probabilistically. The odds form of BT is inter-derivable from the non-odds form (the latter used by the way in the paper by Laredo). So there's no inconsistency or probabilistic problem with using the BF with full partition. Apparently you're thinking that there is some problem because you're noting that the prior ratio *without* a partition (as one moves from one of your exercises to the next) remains the same (because you've set it up that way) but that the Bayes factor *using* a full partition is different between the two examples.. But that doesn't point to a problem. It just means that you're not sticking to the same way of setting the ratios at each point in the odds form. And that's crucial. You either have to do them all with a partition or all without a partition, or it's wrong.
Comment by Brian Blais, posted by me with his permission from a chat. He quoted my initial urn example. Then:
Great analogy. I’ll modify it slightly but I think you’ll see it is equivalent. I’ll refer to urn H for the primary model, urn M for the alternative and urn N for the nothing model — your urns 1, 2, and 3 — just to keep the same letters as my original post. The contents of the urns will be:
- urn H: 10 black balls
- urn M: 25 black balls and 25 white balls
- urn N: 940 red balls
These are the same ratios as what you list and given your priors you will arrive at the same posterior ratio of 1 to 2.5 against urn H: p(H|black)/p(M|black)=1/2.5. And this will match exactly the calculation that I did in my post.
BB continued:
If we don’t want to keep track of priors separately here we could stipulate that the 10 black balls in the urn H are labeled also with an H, all the balls in M are labeled M and the balls in N are labeled N. For good measure all the balls from M and N are also labeled ~H. We then throw all the balls into one big urn and make our draw. The result is a black ball drawn and we want to know what is the probability that that ball is labeled H given the data of one black ball -- p(H|black), the probability of the ball is labeled M given the data of one black ball --p(M|black) and the probability of the ball labeled N given the data of one black ball --p(N|black). Then we just count balls for our results. You can see that the same result will emerge.
Given this set up all of the things that I have been saying, follow naturally:
- The quantity that we are actually interested in is the following posterior ratio: p(H|black)/p(M|black)=1/2.5
- Comparing likelihood, ratios and prior ratios, ignoring any kind of partition which includes urn N Gives the same result: p(black|H)/p(black|M) x p(H)/p(M)= (10/10)/(25/50) x 10/50 = 1/0.5 x 1/5= 1/2.5
- Looking at the BF without the prior, including the partition with the nothing model, gives a distorted confidence in H because the BF is highly top heavy
p(black|H)/p(black|~H) = p(black|H)/(p(black|M,~H)p(M|~H) + p(black|N,~H)p(N|~H))
= 1/(0.5 x 50/990 + 0 x 940/990)= 39.6 = p(black|H)/p(black|~H)
- Looking at the posterior ratio with the partition = BF x prior ratio eliminates the effect of including the nothing model at all
p(H|black)/p(M|black)= p(black|H)/p(black|~H) x p(H)/p(~H) = 39.6 x 10/990 = 1/2.5
BB continued: None of this is controversial and I think that Lydia McGrew would agree with everything I've said here.
## Summary
To summarize, we got here because in Lydia and Than's 9-part video (all the links listed in my first response https://bblais.github.io/posts/2026/May/13/response-bayes-skeptics-and-the-resurrection/), McGrew makes the claim that it is a mistake to not include the partition, and even uses an example where the H and M models as I have named them have equal likelihoods and complains that the BF is then uninformative. McGrew -- at least in that video -- does not mention priors, and uses the top-heavy BF in a way to seemingly imply that the model H is stronger than it is.
When I say that model N is irrelevant it is in the sense that I can add any amount of red balls to the urn and not affect the posterior p(H|black) in any way. When I modified my example to have the prior for N be larger, like p(N)=0.994, it was in a way that doesn't change the information about the other models -- it would be like adding 9000 more red balls -- it doesn't change the number of balls for the other models, and again doesn't change the posterior p(H|black) in any way. McGrew's complaint that "If we are then told that the prior choice procedure has now changed and that urn 3 now has a probability of .994 of being drawn from, there is *absolutely no reason* why this implies that the prior ratio between Urn 1 and Urn 2 must remain the same." is seen to be incorrect. I have only changed my information about model N -- that it occupies more space. This should not affect my information about H or M, the number of balls from those is constant, thus the posterior is constant.
BB continued: My complaint comes down to the following two points:
- insisting that one needs a full partition, where that partition includes irrelevant models is not correct. one can do it, of course, but it is more work and doesn't change the posteriors. criticizing others for ignoring those irrelevant models is not correct.
- writing out the partition form of the BF -- without in the same paragraph of text/minute of video referring to the cancelation property of the prior is seriously misleading. one can achieve an arbitrarily high BF by including these nothing models, but the inflation is canceled automatically in the prior. Exclaiming, in the example above, that the evidence is 40 times more likely on H than on "not H" is a misleading statement -- even if mathematically correct.
Lydia McGrew comment around the same time as the above: The bottom line is this: We're not doing anything probabilistically screwy or illicit, and there's certainly nothing wrong with using the ratio of the likelihoods using a full partition, even when one part of the probability space gives a probability of 0 to the evidence.
One just has to be consistent then, as we were in our paper: Ratio of the posteriors equals ratio of the likelihoods times ratio of the priors. We used ratio of the likelihoods with a full partition, which then has to be multiplied by ratio of the priors with a full partition, which we did.
If you're going to set it up with three hypotheses, one of which gives 0 to the evidence, and then that hypothesis is going to disappear as one continues, then the paradox that you suggest when you move from one case to another doesn't arise. You can *choose* whether or not to keep the ratio of the non-partition priors the same or not when raising the prior of the "nothing" portion. If you do keep that ratio the same, then of course the posterior will be the same, and so will the non-partition likelihoods. Same will produce same. But if you're going to look at the fact that the Bayes factor *with* a partition changes in that case, then you should do the ratio of the priors with a full partition as well, and you'll see that that has also changed, so there isn't any paradox.
I think if you focus on our backsolving move in the paper and the ginormous likelihood ratio we suggest (10 to the 39, not 39), it will be clearer that there's nothing wrong with our *probability*.
To be perfectly frank, what you *should* try to say instead (which is doubtless what you think) is that we've simply misevaluated the nature of the evidence and that this is a case of "garbage in, garbage out." Nothing wrong with our probability, nothing sneaky or illicit about it. But just that (in your opinion) that ginormous Bayes factor doesn't rightly represent the state of the evidence. But again, nothing to do with "inflating the Bayes factor with nothing."
It looks like in part of one of the comments (I didn't read the whole comment) on your post, one of your commenters said this: He said that in the case of the moon landing and the Battle of Gettysburg, our type of modeling would represent the epistemic situation, but that he just thinks we're wrong in our evaluation of the strength of the evidence. Something to that effect.
On my side, I will argue that there simply is no gerrymandered hypothesis under ~R that gathers up all the probability of E on the ~R side, gives exactly the same probability as R to the evidence, and has a higher prior than R. In other words, that the situation *is* similar to the moon landing or the Battle of Gettysburg.
In other words, we differ about the empirical case and the epistemological evaluation. We shouldn't be differing about the probability theory per se.
LM: I really think you shouldn't double down on the "is misleading" claim. In fact, I find it kind of surprising that you would insist on going that route.
BB: The misleading is that I have heard others — like Than and others who I don’t think knows the math as well as you — repeat the top heavy BF in an argument for the resurrection. And this wasn’t just from evidence - it was the result of the partition. We’ll probably agree to disagree on the evidence part, as your paper with Tim shows. My issue was insisting on a partition but not mentioning the prior. Doing that def gives others the impression that the primary model is better than it is whether you yourself can see the full picture.
LM: But...as long as you use the partition for the priors as well, it comes to exactly the same thing. You just have to be consistent. I don't see why this should be thought of as misleading for even a moment. After all, *I'm* not complaining (per se) about the fact that a skeptic thinks the prior ratio with a partition is very strong *against* H.
BB: Is misleading a wrong word here? I agree with what you just said although it’s not that the skeptic is taking the prior smaller on purpose. Including nothing models in the BF pushes the opposite factor into the prior automatically.
BB: Also do you mention priors at all in the discussion of the partition in the 9-part series? I don’t recall but I may have missed it.
LM: Oh, yeah, we discussed your statement that the prior has to be incredibly low because it would be contrary to all the known laws of nature, or words to that effect. I think we addressed several things about the prior.
LM: Well, you claimed in the combox thread that somehow the BF with partition doesn't follow the axioms of probability because you could keep the non-partition BF and non-partition prior ratios the same and get the same posterior. But that doesn't mean that the BF with the partition doesn't follow the axioms of probability! There only appears to be a paradox if you use the BF with partition and the prior ratio without a partition in the same instance of the odds form. But nobody would do that, because it's probabilistically wrong, and in fact Tim and I don't do that, ever.
LM: Here, let me quote our backsolving statement from the article: "But our estimated Bayes factors for these pieces of evidence were, respectively, 10^2, 10^39, and 10^3. Sheer multiplication through gives a Bayes factor of 10^44, a weight of evidence that would be sufficient to overcome a prior probability (or rather improbability) of 10^-40 for R and leave us with a posterior probability in excess of .9999."
Let me address here what I think is a crux where you're still making just a mistake that is apparently still causing you to think there is something misleading about the BF with a partition. [Quotes Blais's discussion of the urn model and my statement that there's no reason to carry over the ratio of the priors of Urn 1 and Urn 2 into a new scenario. Says that this appears to be incorrect.] No, that complaint is not incorrect at all. Here's how you can see that it isn't incorrect. Let's stick with the three urns for now and the decision procedure. If you want, we can use the letters H, M, and N for the urns. That is no problem. First you're in the scenario as in the original "exercise for the reader" in your published blog post. You do the calculation and of course come up with the conclusion that the posterior probability that it was drawn from H is .28-something. (I think it was .2857.
Okay, now you're standing there. An official running the scenario walks in. He says, "We're going to draw again. But this is a new scenario: We changed the machine settings. We kept the proportions of black balls in each urn the same. But now N has a .994 prior probability of being drawn from."
Full stop. He doesn't say anything else. He doesn't tell you what the priors are now on the new machine settings for H and M.
Since these are new settings and just a new scenario, before you can figure out what the posterior of H will be if a black ball is drawn, you *have to* find out how these new settings affected the priors of H and M. The official *did not* say this: "And by the way, when we changed that prior for N, we did it in such a way that the relative prior odds of H and M remain the same." And since they are tinkering with the settings of the machine that tells them which urn to draw from, there is *no reason* to treat the prior ratio of H and M as if it "carries over" from the previous scenario. This is just a new scenario, new machine settings that create new priors. There is no rule that says that you *have to* change the prior of N in such a way that it affects H and M in such a specific way.
So my complaint stands. There is nothing un-probabilistic or "doesn't follow the postulates of probability" or anything like that about the BF with partition.
LM: You know, the whole idea of gathering all the probability for E into M and giving E zero in N is artificial (especially in historical cases) anyway. After all, you could just as easily say that 1/2 of M is also a "nothing model" as say that P(E|M) = .5. You can do that for *anything* short of a likelihood of 1. You could say that you're "inflating the likelihood ratio" in your first exercise case "with nothing" by setting P(E|M) = .5 and P(M) = .05. Instead, someone could try to insist, you're obligated to talk only about P(E|M1) = 1 and P(E|M2) = 0 and then, voila! The likelihood ratio is totally uninformative. Why are we obligated to do anything of the kind? And similarly: It's perfectly fine and not misleading at all to talk about the extremely low probability of the moon landing evidence *if there was no moon landing* overall and to say "The probability of this set of evidence that we have is orders of magnitude greater if there was a moon landing than if there was no moon landing." We aren't either epistemologically or probabilistically obligated to somehow sort out the "no moon landing" side with an artificial "nothing" hypothesis and then get rid of that and do everything with a non-partition of the rest of the no moon landing space. As long as it's legitimate to compare likelihoods other than 1 (which it obviously is), it's legitimate to do so even when using a full partition of H and ~H and even when one of these is very, very low.
I think we've gone in a few different directions, so let me see if I can pull it together a little bit. My original post (https://bblais.github.io/posts/2026/May/14/how-to-inflate-your-bayes-factor-with-nothing/) I had a specific complaint about the 9-part Than-McGrew video. I quoted another paper of McGrew's (https://lydiaswebpage.blogspot.com/2025/12/the-resurrection-independence-and.html) because it had a clear statement of the sentiment stated in the video:
> It's surprising how many people don't know this: When considering evidence for some event, such as the resurrection or the Battle of Gettysburg or the moon landing or anything else, and comparing how well that event explains the evidence and how well its negation explains the evidence, you _must not confine yourself_ to thinking _only_ of alternatives that have some hope of explaining the evidence.
In this sentence (and the paragraphs that follow it) we get the following two points:
1. to do inference on some event, it is *necessary* to include alternative hypotheses that don't even have any hope of explaining the evidence (which I call the "nothing models") -- to form a partition
2. The justification for this point that McGrew provides is the effect on Bayes Factor by not including this model -- the BF becomes uninformative -- and McGrew stresses that one must include the partition to avoid this problem
At no point did McGrew mention in her video anything about the prior-with-the-partition, nor the role of the posterior, even though her responses here show she is aware of them. My challenge in my original post was that:
1. it is completely unnecessary to include alternative hypotheses that don't even have any hope of explaining the evidence
2. including such models has the odd property that it magnifies the BF while simultaneously reducing the prior *by an identical amount*.
Notice this has nothing to do with the "backsolving", where one calculates a likelihood or BF and then infers the value of the prior necessary to overcome that likelihood or BF. This is a specific effect of adding "nothing" models, which has a direct consequence on the prior to cancel its effect on the likelihood -- a consequence that is hard to see by eye. This prior change also has nothing to do with any skeptic's choice of low priors -- it is not a subjective effect in any way, it is a direct mathematical consequence of adding models that don't even attempt to address the evidence.
Following Jaynes, "equivalent states of knowledge are assigned equivalent probability values", adding "nothing" models produces situations that have equivalent states of knowledge. Unsurprisingly, the posterior doesn't change when one does this. However, likelihoods do -- and so do Bayes Factors. The BF *by itself* does not follow this principle of probability, so it is easy to make a BF as large or as small as one wants just by including "nothing" models at various weights.
Saying that "the BF *by itself* does not follow this principle of probability" may be an obvious statement, because McGrew repeatedly says that the partition effect on the BF is handled with the prior with the partition -- I agree -- but that is never mentioned in the original video. It is also never pointed out that by including the "nothing" model, the increased BF is immediately and automatically canceled by the prior without any effort by the skeptic. The implication of the video is that the top heavy BF was some indication that we should prefer the primary hypothesis over its negation, which it certainly does not do.
"1. it is completely unnecessary to include alternative hypotheses that don't even have any hope of explaining the evidence
2. including such models has the odd property that it magnifies the BF while simultaneously reducing the prior *by an identical amount*."
If E provides some evidence for H, the portion of the probability space on some ~H in which the event in question doesn't happen is just the reason that E provides evidence for H. In that sense, it isn't a "model" or a "hypothesis" at all. Or to look at it differently, whether or not E provides evidence for H, as long as the probability given H and/or ~H is less than 1, there is always some portion of the probability space in which E doesn't happen!
Using a full partition in the Bayes factor doesn't really tell us anything, good or bad, about the prior of H. The prior could be high or low. One of the great epistemological virtues of using the odds form with a partition is that it is especially helpful for modeling cases where what goes into the prior is fairly clearly separable from what one is considering to be E. In a historical case, for example, there is whatever document or documents, testimonies, videos, etc., ostensibly record H. That's the specific evidence.
But you can have strong specific evidence for some H that has a high prior as well! Maybe everybody already knows that a particular horse will win a race based on the prior records of the horses involved, but the video evidence also tells us what happened on the day itself.
All that doing the odds form with a full partition means is that it requires us to also use the prior odds *whatever those may be* with a full partition, in order to be consistent.
There is nothing at all strange about this. Indeed, there would often be artificiality in talking only about the non-partition Bayes factor, and this is obvious in a great many cases. For example, in a medical test, when we talk about the false positive rate, we're talking about the probability of getting a positive result if the patient doesn't have the disease. This is P(E|~H). Nobody should insist that we're *obligated* to talk about "the probability of a positive result of this test given that you don't have the disease and there is this specific other thing going on that makes a false positive result likely." That would merely muddy the waters. And considering the Bayes factor composed of the false positive, true positive, false negative, and true negative rates (all of which are related to H or ~H overall, not to some gerrymandered subhypothesis of either) doesn't *in any way* imply that the base rate is unimportant to the calculation of the final probability that you have the disease, given that you got a positive result on the test.
The same is true for locutions like, "The probability is incredibly low that we'd have all this evidence if man never landed on the moon" and "This evidence is much better explained by the view that man did land on the moon than that he didn't." That's both a natural way to speak and to think, and is modeled well by the Bayes factor for the moon landing evidence using the whole of ~H in the denominator.
(continued) "This prior change also has nothing to do with any skeptic's choice of low priors -- it is not a subjective effect in any way, it is a direct mathematical consequence of adding models that don't even attempt to address the evidence."
Once again: *Whatever* the prior is of H (which automatically gives us a prior of ~H, since they form a partition), using the odds form with a full partition merely requires us to use the prior odds with a partition as well, whether the prior of H be low, very low, medium, high, or whatever. In this particular case (of a miracle), we are dealing with a situation where skeptics think the prior of H is overwhelmingly low. I myself think it's moderately low prior to, say, our possessing the Gospels, and very low if we assume them to be *unreliable*. But there's nothing about using a partition for the Bayes factor that somehow "causes" the prior of H to be necessarily low. If it *is* very low, then of course that is reflected in the prior odds with a partition. And yes, this point *does* have to do with our backsolving quote--it makes it quite clear that we weren't trying any kind of fancy maneuver to make the evidence look stronger. Indeed, I have no idea why anybody should think that Tim and I, or Than and I, or anybody, was mixing and matching a Bayes factor with a partition with prior odds using no partition. In our paper we *consistently* talk about the odds form with a full partition.
I have to say bluntly, Brian, that it is *you* in *this very thread* who upthread tried to say that there is something probabilistically wrong with the Bayes factor with partition, when there isn't anything wrong, and when this claim of yours could be explained only by mixing and matching within the odds form--*something we never do*.
You said, "When I say that likelihoods do not follow the postulates of probability it’s at least in the violation of the equivalent states of knowledge principle. How do I know this? Because I can construct a very small and simple case where I get different bayes factors for equivalent problems, but that the posterior and posterior ratios give the same value."
As I pointed out, when one consistently uses the odds factor without a partition in the cases you chose to construct, along with the prior odds without a partition in those cases, both were the same. And when one consistently uses the odds factor and prior factor *with* a partition, the priors are different as well, so there is no paradox of the kind you suggest in this quote. That faulty notion of a probabilistic problem with the Bayes factor was *your* problem; it was never mine. Nor is there any necessity when moving from one set of priors to a totally new case with a 3-part partition that we keep the prior odds between two of the subhypotheses the same. I also pointed this out.
That we should prefer using the primary hypothesis over its negation (in *needless to say* a consistent fashion, as we do) is a result of a variety of epistemological considerations in historical cases and others (e.g., medical testing, as I discussed above). This has to do with the artificiality and confusingness of treating the empty space as a "model" and breaking it out separately, and the arbitrariness of such a requirement. The fact that we'd likely not have the evidence at all given ~H in some historical cases and that this "nothing" space is much larger in ~H (corresponding to a true negative rate) than in H (corresponding to a false negative rate) *just is* a big part of the explanation for a high Bayes factor favoring H. We ignore that point at our epistemological peril.
You seem to have a huge issue with my phrasing that the BF doesn't follow the postulates of probability. Fair enough. Let me go with this phrasing, and perhaps we can ignore the previous phrasing so that we can move on from semantics: *The value of the BF can be different for equivalent states of knowledge, whereas the posterior probability will never be different for equivalent states of knowledge*. Does that work for you?
The medical test example is a bad example, only because there are only two possible models -- you have the disease or you don't. To match my example, one can modify this to H=you have the rare disease "H" which always gives the positive test result, M=you have this more common disease "M" which sometimes gives the same positive test, and N=you have nothing, which never gives the positive result. And you're interested in what you have: P(N|positive), P(M|positive), P(N|positive). You can immediately tell that any model or any part of a model that entails the negative test result will not be relevant to these probabilities.
> "The fact that we'd likely not have the evidence at all given ~H in some historical cases and that this "nothing" space is much larger in ~H (corresponding to a true negative rate) than in H (corresponding to a false negative rate) _just is_ a big part of the explanation for a high Bayes factor favoring H. We ignore that point at our epistemological peril."
I agree that the partition is "a big part of the explanation for a high Bayes factor favoring H" -- it's been my point all along. I just don't care about the BF alone. Let me state as clearly as I can -- *I only care about the posterior probabilities for my hypotheses given the data/evidence* because I only care about which explanations of the data are true, or probably true. If the BF is a mathematical stepping stone to get to that, then that's fine, but I never care about the BF by itself. I don't care if the BF is high favoring H if the posterior is high biased against H.
So here is the challenge for you: please show me an example where models or parts of models that entail that *we didn't see the evidence we observed* actually affects the posterior. My original example showed at least one case where including a model that didn't entail the evidence at all simply cancels when looking at the posterior, no matter its prior probability or how much of the ~H space it took up. Can you show me a counter example? I don't think it exists.
"So here is the challenge for you: please show me an example where models or parts of models that entail that *we didn't see the evidence we observed* actually affects the posterior." This is, once more, semantics as can be seen by the odd use of "or parts of models."
I am inclined to reply that *any time* the probability of E given H or the probability of E given ~H is less than 1, even if it's very high, there is a "part of the model" in which "we didn't see the evidence we observed." That's why the probability is less than 1!!! Even if P(E|H) or P(E|~H) is .9 or something, there's a "part of" the space in which the evidence doesn't occur. That's what makes the probability less than 1. In that sense, *any* scenario in which it is legitimate to compare the likelihoods of two hypotheses, either one of which gives a probability less than 1 to the evidence, is a model where a "part of the model entails that we didn't see the evidence we observe" "actually affects the posterior."
You are just arbitrarily making a fuss about this in cases where P(E|~H) is very low, where you are trying to insist that we are somehow obligated to segregate a part of the zero space (not even all of it--you didn't seem to have a problem with an H2 that gave the evidence a probability of .5), refer to it as a "nothing model," no matter how strange this concept is in the concrete instance in question, and then reckon without it.
I don't admit any obligation to do that. And you're special pleading about the disease example. It *is* a counterexample to your insistence in a historical case. "No moon landing and no evidence" is in no ordinary sense a "model."
By the way, Laredo uses a partition. And I may have missed it, but I didn't see anywhere where he argued that we're for some reason "not allowed" to have any portion of the probability space that gives E probability zero. Probably because insisting on that makes no sense probabilistically. As long as E itself has probability less than 1, we can *always* carve things up in such a way that there's a portion of the probaiblity space in which E has probability zero!
"*The value of the BF can be different for equivalent states of knowledge, whereas the posterior probability will never be different for equivalent states of knowledge*. Does that work for you?"
The value of the BF with a full partition, which is *what we used in the paper and what Than and I were talking about* can never be objectively different for equivalent states of knowledge; nor can the value of the prior ratio with a full partition. I feel like a weird kind of projection is going on here. I've never tried to compare a very high BF with a partition to a more favorable prior ratio without a partition.
A big part of my interest in all of this is the way that ad hocness works--trying to account for some evidence under ~H whittles away at any prior advantage that ~H might have had without that being noticed. This is the point that Tim and I make about the low probability of such an ad hoc alternative given ~H as a whole. It's also crucial to my published paper on a Bayesian analysis of ad hocness.
Ironically too (as I'll be discussing in a video series I'm doing), at times even a proposed ad hoc hypothesis does a pretty bad job accounting for the E anyway, especially when E is a set of cumulative, complex evidence that favors H. I've certainly noticed this in proposed skeptical theories meant to account for the evidence for R.
Let me try to forestall a further confusion: A case where the prior of H3 is .94 and where the non-partition likelihood ratio between H1 and H2 is 2/1 and the non-partition prior ratio between H1 and H2 is 1/5 is not an "equivalent state of knowledge" to a case where the prior of H3 is .994 and both the non-partition prior ratio (between the other two) and the non-partition likelihood ratio have been held constant. This is evident from the fact that the prior probability of H1, in your examples, is .01 in the first example and .001 in the second example. Obviously if all the evidence other than E really gives a prior probability ten times lower to H1 in one scenario than in another, there is not an equivalent state of knowledge. Since you are very insistent on objectivity here (as am I), you ought to acknowledge this.
Consider this: By principles of objectivity, different states of information about the same propositions should yield different probabilities. Moreover, that our credences should be apportioned to the actual state of the evidence is true *for different times*. I should not be more concerned that my credences are correctly linked to the evidence *before* I observe E than *after* I observe E.
Now, in our urn case, or in our medical test case, or in our historical cases, in the prior distribution E has not yet been observed. Now consider your two scenarios that were carefully set up to yield the same posterior for H1. Let t1 (for time 1) refer to the time when you have not yet observed E. Your scenarios have a different set of priors for *all* of the following: H1, H2, H3, and E itself! E.g. In scenario A, H1 is ten times more probable at t1 than in Scenario B. The probability of E is also higher at t1 in Scenario A than in scenario B.
Surely, your concern for objective truth and the proper apportioning of confidences should be just as applicable to t1 as to t2! And surely you should be just as interested in having the correct probability for H1, or for E itself, in your state of knowledge at t1 as at t2!
Very well, then. It is obvious that Scenario A and Scenario B do not represent equivalent states of knowledge, for at t1 in Scenario A you should have quite different credences for several specific propositions than you should have at t1 in Scenario B.
Therefore, the fact that Scenario A and Scenario B have been set up in a particular fashion (which, as I have pointed out repeatedly, there is *no necessity for them to be*) such that, at t2, after E has been observed (and now is treated as having probability 1), it happens to be the case that you have the same posterior probabilities for H1, H2, H3, and E, *does not mean* that the two scenarios represent "equivalent states of knowledge."
E might not even be observed.
Prior probabilities represent states of knowledge as well, relative to particular bodies of data that don't include the certainty of E. Therefore, these different scenarios do not represent equivalent states of knowledge. QED
> "all the evidence other than E really gives a prior probability ten times lower to H1 in one scenario than in another, there is not an equivalent state of knowledge"
I don't think this phrasing is correct -- since the prior doesn't include the evidence at all -- but the sentiment is something that I'd acknowledge. It's true, before the evidence is observed, changing from a prior of 0.01 to 0.001 would represent a change in the state of knowledge for H1. However, what I am concerned with is the state of knowledge represented by the posterior: p(H1|E). In that case, introducing any model that entails ~E will be an equivalent state of knowledge and thus an equivalent posterior. It's like in my urn example, where I have 10 black balls labeled H, 25 black and 25 white labeled M, and 940 red labeled N. Throwing more red balls into the urn will of course change the probability of pulling a ball labeled H a-priori. However, if we have data of a draw of a black ball, throwing more red balls into N will not change p(H|black) and be considered equivalent situations. Changing the red balls does not change our state of knowledge about any case where we know we've drawn a black ball.
> _any_ scenario in which it is legitimate to compare the likelihoods of two hypotheses, either one of which gives a probability less than 1 to the evidence, is a model where a "part of the model entails that we didn't see the evidence we observe" "actually affects the posterior."
This is mathematically false. My example shows that -- the likelihood for ~H in my urn example with a black draw is p(black|~H) = 25/(50+940) which is clearly less than 1 yet the inclusion of the N model doesn't affect the posterior. The BF=(50+940)/25 ~ 39 in favor of H, includes the effect of the N model, even though the posterior is unaffected. So I repeat the challenge: show me an example where models or parts of models that entail that _we didn't see the evidence we observed_ actually affects the posterior.
"So I repeat the challenge: show me an example where models or parts of models that entail that _we didn't see the evidence we observed_ actually affects the posterior."
I have already met your "challenge" by my point concerning any case in which the probability of E is less than 1, or the probability of E given some Hi is less than one. E.g. P(E|H2) = .5 and P(H2) = .05 (which you have allowed). Any time that this is the case, someone could make *exactly* the same demand that you have made to *conceive of* this by *further* subdividing it into a so-called "nothing model"--e.g. P(E|H2*) = 0 and P(H2*) = .025--and a model that gives probability 1 to E--i.e., in this example P(E|H2**) = 1 and P(H2**) = .025.
If such a demand to subdivide artificially is legitimate for "no resurrection" and "no moon landing" and so forth, it is equally legitimate in any other case where a conditional probability for E is less than 1. Since there are obviously cases (as tacitly admitted by your setting up a case where one of the conditional probabilities is .5) in which such a demand for further subdivision in order to sequester all of the space in which E doesn't occur into a particular so-called "model" is not binding, there is no reason to consider such a demand to be binding in the case of "no resurrection" and "no moon landing" and so forth.
At this point we're just repeating ourselves, so I'll leave things here for any readers who have followed us thus far.
"I don't think this phrasing is correct -- since the prior doesn't include the evidence at all"
This is just the sheerest wordplay on the word "the evidence." Of course, by definition, you haven't already conditionalized on a particular piece of evidence E in the prior distribution. But if the probabilities in this distribution are not merely made up (which presumably you don't want to say, especially not in the case of the low probability of a miracle!!), they are based on something-or-other. In a historical case, where we can fairly neatly think of a prior in terms of evidence we've already acquired, before we acquire the evidence we're treating as E, these prior probabilities are based on the other evidence we have at that earlier point. (Or in a medical test, on the previous clinical examination, before we have obtained a positive result on the test in question.)
So once again, since you're very concerned about objectivity, the fact that the prior distribution in your two scenarios represents *different* states of knowledge, even though they have been carefully and unnecessarily set up so that the posterior probabilities are the same, the BF using the full partition does not violate the principle that equivalent sets of knowledge should yield equivalent probabilities.
This is just absolutely clear.
"throwing more red balls into N will not change p(H|black) and be considered equivalent situations. Changing the red balls does not change our state of knowledge about any case where we know we've drawn a black ball."
This should instead refer to throwing more red balls in *while definitely not changing the relative proportions of balls which have the other properties.* As I have pointed out (at this point ad nauseum), being told that the prior of Urn 3 (or N, or H3, or however one labels it) has changed *while being given no information about how the relative priors of the other two have changed* is simply an incomplete state of information for calculating the posterior for H1 given a draw of a black ball. Indeed, if a lot rides on getting that calculation right, you would be very well-advised to ask explicitly, "Okay, so is that all you did? What are the priors of the other two now?" And so forth. Of course if you *set up* the *completely different* scenario in a carefully balanced fashion so that you are increasing the prior probability of H3 and this is *equally affecting* the priors of the others, and you are leaving all the likelihoods the same, then *by that setup* the posterior of H1 upon drawing a black ball will be the same. But this fact *in no way* means that the likelihood ratio using a partition violates a principle that equivalent states of knowledge should yield equivalent probabilities. That's just a nonsensical accusation. The two scenarios are just different scenarios. Being told that the prior for H3 has changed (or that more red balls have been added) while being given no additional information about how this affects the other two is just incompletely specifying the relevant quantities in the new scenario. It neither entails nor implies that the relative prior odds of the other two possibilities have remained the same. As I have pointed out again and again, you could increase the prior of H3 to .994 and change the priors of the other two to .003 and .003, and there is *nothing at all* that tells you that the people dealing with the urn(s) haven't done this *unless they actually tell you that*.
Post a Comment