Fix infinite loop that can happen in some cases where colspan=0 or rowspan=0
because accessing empty structures. Tracking for not empty qmap by using
additional flag is redundant and buggy in this case.

(cherry picked from commit 543c2751da42c1664171685e25fbce594024d9cd)