I believe I do have a timeline of events that you might find revelatory.
The Nightingale family are celebrating Christmas with a family friend, Monty James.
His son, Oswald, and daughter, Gally, both protest that he should wear the hat. Even his wife, Lettie, points out that Mr. James is wearing the hat that he won from his own cracker. Patrick is unmoved and does not wear his hat.
Gally blames the rip on the large size of Oswald’s head, but suggests that Oswald should take their father’s unused hat. Patrick refutes the idea, saying that Oswald should have taken better care of his hat.
Seeing an opportunity to lighten the mood, Mr. James seizes Patrick’s hat and rips it. The family laughs as Patrick then rips Mr. James’ hat and Gally rips her mother’s one.
Patrick describes this round of hat ripping as “fun in its proper place”.
As the laughter dies down, Gally tells Oswald to let their dog, Toby, out.
At the end of the meal, Patrick reveals to the children that Mr. James is known for his ghost stories, and that he has kindly consented to tell them one now.