tree 8095286e62844a46566a52e409dc0b9807c2a512
parent 64965900c4fc4a33fe97bc1db6fa9ba946621b31
author luochen01 <cluo8@uci.edu> 1507508624 -0700
committer Luo Chen <cluo8@uci.edu> 1507650156 -0700

[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
Currently CorrelatedMergePolicy uses component Ids to ensure disk
components of primary and secondary indexes are merged together,
but without synchronization. However, this results in too many disk
components for secondary InvertedIndex. The reason is that secondary
index could miss some round of merges, if the merge policy finds out
the corresponding secondary components are not available (either being
merged or being flushed). Even though flow-control on secondary indexes
can guarantee the secondary index would catch up the next time, it is
still possible that the primary component is finialized, which leaves
the secondary components which miss this round of merge are never merged
again.

This patch fixes this bug by:
- Add the mechanism of depending operations to LSM IO operation. An
operation finishes only after all depending operations have finished.
- For correlated merge policy, the flush/merge of the primary index depends
on all flushes/merges of secondary indexes. This ensures when the
correlated policy schedules merge, all related components of all indexes
are available to merge.

Change-Id: Ib6c06ee23f3bfd16b758802388389c00e29780b1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2018
Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>