Should I take him off for paying child support?

My childs father is mad at me and said that since he has to pay a heap in child support that he never wants to see my retarded a$$ face or my sons. I asked him to mediate in court and he never showed up so the judge made the amount A LOT! he works for the sheriffs dept and it also kinda worries me what he is capable of....I really cannot afford it as I work full time and school part time, but as soon as I am done with school I can get a second job, but that means i'd rarely see my child :(