hold it against

to like someone less because they have done something wrong or behaved badly in the past:

He made a mistake but I don’t hold it against him - we all make mistakes.