Perhaps because he has cared for the Young Lady since she was a child, he still sees her as just a baby—at least, that's what he tells himself. Even when she throws tantrums saying she hates this or that, he just smiles and handles her like a child. On days when he occasionally realizes how much she has grown (in more ways than one), he chokes up alone... and mutters curses at himself for having such strange thoughts. He used to be fine with minor physical affection with her, but now... he tries to avoid it. You probably know why without him saying it.