Last year (2023) I posted an article in which I estimated the effect of Thanksgiving on gratitude. Feel free to have a quick read if you want to get into the details regarding the psychological benefits of gratitude (and catch up on what I did there.) In that study, I compared participants’ emotion ratings (gratitude, thankfulness, appreciativeness) on Thanksgiving to a few days before and after Thanksgiving. Overall, I found a positive effect of doing the survey on Thanksgiving on gratitude (probably not a huge surprise to everyone.) However, at the end of the article, I noted that simply comparing responses on Thanksgiving to responses on days around the holiday isn’t a rigorous approach to answering the question.
Difference-in-Differences
For 2024, I managed to pull off a more rigorous approach to this question — I leveraged the fact that Canadians celebrate Thanksgiving in October, and therefore used them as a comparison group! I sent out a survey at 6pm EST on the 25th-28th of November which asked participants to rate how they were feeling in that moment (emotions included grateful, thankful, appreciative, happy, sad, and others) on a scale from 1: Not at all to 5: Very much. The approach I took here is a quasi-experimental design called difference-in-differences — basically we use a comparison that we hope has similar gratitude trends prior to Thanksgiving as a control. Here’s a look at what the best-case scenario would look like:
Figure 1. We rely on the assumption that the pre-Thanksgiving differences from the 25-27 would have persisted on the 28th. Since we have the real value for the comparison, we use that to estimate a counterfactual value that would have continued that trend. The difference between the observed value and the counterfactual is our estimate! (It never looks this perfect in real life…sigh)
This design (if the assumptions work out), gives a way to account for temporal trends that might explain away our effect. For example, it could be that Thanksgiving engenders gratitude, but it could also be that people are just more grateful on Thursdays than on earlier days of the week because the weekend is right around the corner! If we assume Canadians have similar emotional variation across the week, we can use their ratings to account for any usual weekly changes in gratitude and tease out the effect of the U.S. holiday!
Checking assumptions
First, I checked to see if the pre-Thanksgiving trends were reasonably parallel. In other words, we want the distance between average ratings in the U.S. and Canada to be pretty similar over time — and then if this holds, we infer (read: cross our fingers) that barring the holiday, this trend would persist. As you can see in Figure 2, the two trends are reasonably parallel — Canadians appear to show a bit of a jump in gratitude on the 27th, but it isn’t dramatic enough to violate our assumptions. Figure 3 provides a zoomed in view that just looks at the difference between the US/Canada difference on the 27th vs. all the other days (we’re looking for the averages for the 25th and the 26th to be near zero.)
Figure 2. This is more like reality… the distance between Canada and USA from the 25-27th are not identical, but they’re close enough to go ahead with this approach.
For stats nerds: Posterior marginal effects from a Bayesian linear regression. I also did a joint hypothesis test for the day X country effect for the 25th and 26th (reference was the 27th), in which the 95% credible interval for the linear combination of the terms was between -.39 and .85.
Figure 3. An event plot simply shows you the estimates of how much larger (or smaller) the difference between countries is than the difference on the 27th (which is the day before the treatment (in this case — Thanksgiving.) This is a standard plot that economists use for evaluating assumptions in difference-in-difference designs.
no observed effect of Thanksgiving on gratitude
You could probably tell from the figures above, but when compared the average ratings across the 3 days prior to Thanksgiving to Thanksgiving, we aren’t really picking up an increase on gratitude on Thanksgiving. We do see gratitude ratings rise on Thanksgiving for Americans, but we also see a corresponding jump for Canadians, and moreover, the difference is actually bigger on the 25th and 26th!
When I run the difference-in-differences model, the estimated average effect of Thanksgiving (i.e., the parameter reflecting the interaction of being American and it being Thanksgiving) is -0.01 95% CI [-.47, .46]. Needless to say, this confirms what we can see with our eyeballs — we aren’t observing a clear effect. That being said, there’s a 95% probability that the effect lies between -.47, .46 (our Credible Interval (CI)) — which means we are still highly uncertain.
In the study, I measured 3 different emotion ratings which capture gratitude — thankfulness, gratitude, and appreciativeness. To keep things simple, I’ve plotted the average estimated treatment effects of Thanksgiving (or in this case, the lack thereof…) in figure 4 below. The quick summary of what you’re seeing is:
Based on this alone, we cannot conclude that Thanksgiving affects gratitude.
We also can’t really be super confident that it doesn’t either… there’s a lot of uncertainty in these estimates. The 95% credibility interval for the grateful measure ranges from almost -0.5 to 0.5 (which could be a whopping 13% increase or decrease!) (These little surveys are self-funded, and I have a toddler at home, so we get what we get.)
That being said, if we came into this believing that Thanksgiving does increase gratitude, we should probably temper that belief by just a tiny bit given this information (if you want to get a sense for how much we should potentially temper it, read “Fun with Bayes!” below)
Figure 4. A simple way to interpret this is to think of the yellow bell curves as all the possible values that the effect of Thanksgiving could be. More specifically, this figure shows the posterior distributions (in yellow) for the effect of Thanksgiving for each of the 3 gratitude emotions. The red dots are the median of the posterior distribution and the red bars show the credible intervals (the thick part is where we think there’s a 66% chance the effect lies and the thinner part is a 95% chance.
For stats nerds: I estimated these with a Bayesian multivariate linear regression, in which I also modeled the correlation between measures, which were r(thankful, grateful) = 0.84, 95% CI [0.8, 0.86]; r(thankful, appreciative) = 0.78, 95% CI [0.74, 0.82]; r(grateful, appreciative) = 0.81, 95% CI [0.78, 0.84].
what if I believe that thanksgiving does increase gratitude?
A weakness and strength of these sorts of studies is that one must specify the context under which the question is examined. It’s a strength because you are forced to be precise with your methods and your measurement. It’s a weakness because there are potentially endless configurations for a study and you have to pick one (or if you’re lucky, a few.) This means the generalizability of your conclusions will be limited. In the case of this study, I used online survey-takers on a platform called Cloud Research, I used Canadians as a comparison, I chose 3 days prior to Thanksgiving to measure (with smaller than desired sample sizes), I used an emotion rating scale that focused on how participants felt in the moment, and I started the surveys at 6pm EST each day.* I’m sure you can generate a laundry list of objections to this design, but c'est la vie — I also have lots of reasons (some under my control, many not) for why I did it this way. So when I say that it looks like Thanksgiving isn’t increasing gratitude, I can’t possibly claim that this is true for all possible configurations of the question.
So did we learn anything?? (This is the existential crisis portion of the blog.) I (predictably) argue that despite all of the weaknesses, we still learned something useful — we learned that we should at least be open to the possibility that Thanksgiving isn’t the gratitude-inducing holiday we thought it was. In other words, maybe this result doesn’t change our overall belief (and to be clear, I’m still team “Thanksgiving increases gratitude”), but perhaps it should nudge us towards a bit less confidence in that belief (as now I’m a bit less certain in my belief.) The other benefit (in my view) is that it adds to the corpus of other work in this area — it is one small brick in the massive construction project that is answering this question! This study didn’t have enough data to definitively get at an answer, but I suspect I’ll keep doing this type of thing every year (perhaps one in October for Canadians and one in November for Americans), and eventually, we might have ourselves a pretty solid idea!
[I get a bit more technical after this, so if you don’t want that, you can stop reading. That being said, I do think you might find the rest interesting!]
Fun with Bayes!
incorporating belief into the analysis
The benefit of a Bayesian approach is that you can actually see how much your prior belief should change in light of new evidence! All of the analyses above assumed that we had no strong prior belief either way, which means we essentially let the data drive the conclusions. What we can do here is say, “hey, I actually believe that Thanksgiving does increase gratitude and I’d like the analysis to reflect that belief“. Quantifying a belief like this is pretty tricky, but luckily, there is prior work on this question (including my study from last year) that can guide us!
Let’s start with a belief grounded in prior work on gratitude increasing interventions, suggesting somewhere around a 5% increase in gratitude. This would put us from around a 3.64 to 3.822 (a 0.182 increase.) We also need to know how uncertain you are about that 5% increase, so to keep things simple, let’s say you believe that there’s a ~95% probability that gratitude increases (in other words, if you are pretty dang confident.) If I rerun the Bayesian model, incorporating this prior belief into the model, then once I take into account the data from this study, my belief shifts from thinking there’s a 5% (or .182) increase to thinking that there’s a 4% (or ~.15) increase. This means I went from believing that there’s a 95% probability that Thanksgiving increases gratitude to believing there’s a 93% probability — so I clearly still believe that it has a positive effect, but my belief in the size of that effect has shifted downward by just a bit!
We can run the same analysis, but perhaps assume that a person believes that on average there’s a 5% increase, but is much less confident, say, believing that there’s a 66% (or 2-in-3) chance that it increases gratitude. In this example, their belief shifts to thinking perhaps there’s a 1.3% increase (or 0.05), and believing now that there’s only a 60% chance overall that it increases gratitude.
If it helps, I’ve visualized what this looks like in Figure 5 below!
The simplest way to think about this figure is:
The dashed line represents zero effect of Thanksgiving on gratitude.
The proportion of the distribution to the right of that dashed line reflects the probability that Thanksgiving has some positive effect.
Figure 5. A visual for how someone with a strong belief in Thanksgiving increasing gratitude might change their belief in the face of new data (left) vs. someone with a weaker belief (right). The top (red) reflects their belief prior to reading this blog, the data (yellow) are the results of the study, and the new belief (yellow + red = orange) is their updated belief after incorporating this new data into their prior!
On the left, you have someone who starts off believing that there’s a 95% probability that Thanksgiving increases gratitude (Strong Prior). Since they have a pretty strong prior belief, and the data contains a lot of uncertainty, you see little change in their new belief after taking the data into account. You can see that someone with a weak belief (around 66% chance), has a much flatter distribution of belief (meaning they’re open to many more possibilities.) The data has a stronger effect on them, reducing their confidence (and ruling out larger possible effects.)
This seems like I can only test whether there is a positive or negative effect… what about no effect at all?
The conclusion I draw from this study is we can’t say there is an effect, it kind of looks like there isn’t one, but also our estimates are just too noisy to be sure (in other words, we need more data.) So far, I’ve been simply looking at the posterior distribution of the effect and drawing general conclusions. However, we can rely on a standard approach to test the hypothesis — namely, setting a region of practical equivalence (ROPE), which serves as a boundary around zero which we would consider the effect to be, for all practical purposes, the same as no effect at all. To get our ROPE estimate, we calculate the percentage of our posterior distribution that lies within that boundary! (One recommendation is to calculate the % that lies within the 89% credible interval, but to keep it simple, we’ll use the whole posterior distribution.) In order to definitively conclude that there is no effect, we are going to want to see a high percentage (a typical standard would be something like 97.5%.) Conversely, to conclude that there is an effect, we’d want to see a really small % — something like 2.5%.
For our gratitude measure, let’s say that we really don’t care about any effects within 1/10 of a standard deviation of zero — so effects between -0.12 and 0.12 (the standard deviation of our gratitude measure is 1.2, meaning 68% of responses were within 1.2 points on the scale.) To put this another way, if Thanksgiving did lead to an average increase in gratitude of 0.12, then there would be a 52.8% chance that a person picked at random from the Thanksgiving group would have a higher score than a person picked at random from the no-Thanksgiving group (all else being equal.)
It turns out that ~37% of our distribution lies within our ROPE (see figure 6). That is certainly less than 97.5% (and definitely larger than 2.5%), so we can’t really conclude that there is no effect of Thanksgiving on gratitude any more than we can conclude that there is one!
Figure 6. The posterior distribution of our effect of Thanksgiving on gratitude (the yellow part reflects ±2.5% of the distribution. The blue bar reflects our ROPE. This pretty clearly demonstrates that we should still be pretty darn uncertain!
Taking a look at % choosing each option
For the sake of simplicity, I spent this post treating the outcome variable as a continuous ratio-level variable (I ran linear regressions) — largely because that’s the typical approach for difference-in-differences in economics (ordinary least-squares approaches tend to do pretty well even when assumptions are violated and they make for more easily interpreted coefficients as there aren’t complex relationships between parameters to think about.) That being said, the outcome in this survey is not at the ratio level — it is ordinal (from 1: Not at all to 5: Very much). So a more appropriate way to think about these responses is to estimate the probability of choosing each option — which I’ve done in Figure 7 below!
Figure 7. The dots reflect the best estimate of the % of respondents choosing that option — the bars reflect uncertainty (95% CIs).
For stats nerds: I ran a Bayesian ordinal regression (cumulative link function). To test for an effect of Thanksgiving, I simplified things and just ran a model with country, Thanksgiving/Not Thanksving, and the interaction term (which captures the effect.) The beta for the interaction was 0.05 95% CI [-0.64, 0.77], so needless to say, my general conclusions from above remain unchanged.
This can be kind of difficult to understand, so a general guideline is to look at the shape of the responses. Across every day, you see kind of a upside down hockey-stick, with a majority of people (somewhere between 30-40%) choosing “Somewhat grateful” and the fewest choosing “Not at all grateful” (somewhere between 5-15%). You can also see the trend we saw in the previous sections if you focus on “Very much grateful”, with a bump for both Americans and Canadians on Thanksgiving day (the bump being a bit more gradual for Canadians from the 26-28 and more localized to Thanksgiving for Americans.) That being said, you can also tell by the widths of the bars that we still have lots of uncertainty, and therefore, need a lot more data before we can draw any serious conclusions.
Survey details
Sample size
407 participants from Cloud Research Crowdsourcing platform
Nov. 25: 29 Canadians, 28 Americans
Nov. 26: 28 Canadians, 27 Americans
Nov. 27: 55 Canadians, 60 Americans
Nov. 28: 86 Canadians, 94 Americans
*Actually there were 50 Canadians who completed it on the 29th, but for simplicity, I binned them on the 28th. There wasn’t any real difference if I left them out.
Materials
Used the gratitude measurement from https://osf.io/preprints/psyarxiv/29ebh.
Like the article? Consider supporting for more like it!
Click here to check out my Patreon page!