Build WILL be expensive if you have colliders. As the docs say: "Avoid building tilemaps every frame if you use Unity colliders as it will likely be too slow for runtime use.". If you need to build stuff at runtime, disable colliders and manage it yourself, eg. building a "skin" of box colliders pulled out of a cache is one way of doing this efficiently.