Does it consistently do nothing / crash? The memory footprint on Windows & OSX is slightly different. Unity is a 32 bit app, and as soon as memory usage hits around 3.5GB it will run out and die. It sounds like a lot, but Unity does some things super inefficiently when it comes to textures, eg. each texture is stored more than once in ram, etc. I suspect you're very very close to the limit, and it happens to just work on PC and is over the limit on OSX.
How many textures (rough dimensions & count) and are you using atlas spanning?