The collider is only destroyed in ondestroy, its suspicious if it gets destroyed otherwise. Add some code to track it, eg. disable and immediately check if its gone missing, etc. Polygon colliders are shared between sprites, so if you destroy one that is instantiated from a scene object, it can destroy other instances of it.