Fix Color Smudge op in Dulling mode

We must not store reading random accessors when someone else in writing to
the same device, because the cache of the random accessor will become
outdated.