Instead of manually removing the events, you can mark them as having a weak reference. Weak reference listeners does not count on the garbage collection process. Do this in right in the addEventListener function:
object.addEventListener(eventType, function, bubbles, priority, useWeakReference);
Set it as true and the weak reference will not count towards as reference when doing garbage collecion.