The scale isn't related to dimensions at all. Its meant to be 1, 1 at unless you want it to be bigger or smaller than its native size. If you create your camera at 1 pixel per meter, you'd probably want to set up your sprite collections at the same size. If your sprites and camera are set up with the same settings, you'll find that the scale will be 1,1,1 for it to be pixel perfect.
A scale of 320x320 on it suggests your sprite collection is set up with different parameters, likely 640 height and 1 ortho size. Are you using one of the demo sprites?
If you want the sprite to fit a particular "size" at 1 pixel per meter regardless of configuration, you can use something similar to this:
Vector2 yourSizeInPixels;
var nativeSize = sprite.CurrentSprite.GetUntrimmedBounds().size;
sprite.scale = new Vector2( yourSizeInPixels.x / nativeSize.x, yourSizeInPixels.y / nativeSize.y, 1);
This will get you the scale you need to apply to resize it to fit the required bounds.