I want to go to a new youth group...But I'm not sure if I should?

Instead of leaving, get together with your pastor, and this youth group leader and address what you just said. That will get to the root of the problem, dispel any rumors, and bring the situation into the open. If the pastor of your church ignores the problem, then there is a problem. Never be afraid to confront situations in church, it is every Christians responsibility and authority to address and confront abuses in the ministry - but you must do so through the proper authority of the church, as directed. Be careful not to start rumors and stir up dissension or discord - even if you are right, spreading discord is one of the seven deadly sins.