When loading one image with mips, eg. 4x with mips, will be naturally bigger than 4x without mips. (1.3x bigger more or less).
The second issue with QualitySettings, is that the mipmaps aren't generated programmatically and the atlases aren't generated with the required constraints.
For downscaling to work properly (not even considering compression here), it will require sprites to exist at downscaleable boundaries, depending on the number of mips required otherwise you'll get unwanted bleeding across sprites. If you think about a 2x2 sprite at 0, 0 and another at 2, 0 - when downscaled twice, you'll end up with the 2 sprites blended. Now the same case, with the 2x2 at 1,1 and the other at 3,1 - they will blend on the first downscale.
Add compression to the mix and things start getting a bit more complex...
But, if it works for you, go for it. Whether you'll see any of the above in your textures isn't certain, so its definitely worth a try, since its easy enough to try it.