Loading time isn't something you can optimize directly - however, you could try using the async load functions to load in the background. You'll still get a pause at some point, but it won't be as bad as directly loading it.
You can try calling UnloadUnusedAssets before loading another scene, but if you're getting occasional crashes, it could be because of any number of things - eg. out of mono memory, etc. You need to identify why its crashing - eg. what memory its running out of, before trying to fix this.