I also focus on remembering that sitting is VERY difficult, and that I am grateful to everyone who sits there for hour after hour with me sharing the joy and the agony of zazen.
Have you sat a sesshin with this person? That might help.
True story: I resented a priest for 10 years, then worked with him during work-period this June. He was a totally different guy from the imagined person I resented. I talked with him in dokusan and told him of my feelings, and he revealed to me he often felt that way himself about how he has been treated. I realized I had been unjust and unkind to him, and I felt ashamed of myself, and grateful he trusted me enough to share his side.
Now we are friends and have lunch together every couple weeks or a month if I’m away.