Fixed a subpixel precision calculation of mask sizes

Now the size of the mask is calculated depending on the fractional offset.
This is the second version of a commit 2c30521 which was reverted due to
the problems with the autobrush. Now the auto brush calculates the center
of the dab correctly using the hotspot information which gives the most
stable and correct results.