Fix changerecorder assert further.

Since 022d1045118 in akonadi, when the result of merging two ModifyFlags
notifications is a no-op notification, both are dropped. So we must catch
removals also when nothing gets appended.