tree cf6eef9dbb3eb14793df6861d5af8a310f57e319
parent 1a6de0b9125256b57438e920f06165538e8ecbac
author Abdullah Alamoudi <bamousaa@gmail.com> 1526859175 +0300
committer abdullah alamoudi <bamousaa@gmail.com> 1526873620 -0700

[NO ISSUE][STO] Misc Storage Fixes and Improvements

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
- This change introduces some improvements to storage
  operations.
- Local RecoveryManager is now extensible.
- Bulk loaders now call the IO callback similar to
  Flushes, making them less special and creating a
  unified lifecycle for adding an index component.
- As a result, The IndexCheckpointManager doesn't need
  to have a special treatment for components loaded
  through the bulk load operation.
- Component Id have been added to the index checkpoint
  files.
- Cleanup for the code of local recovery for failed flush
  operations.
- Ensure that after local recovery of flushes, primary
  and secondary indexes have the same index for mutable
  memory component.
- The use of WAIT logs to ensure in-flight flushes
  are scheduled didn't work as expected. A new log type
  WAIT_FOR_FLUSHES was introduced to acheive the expected
  behavior.
- The local test framework was made Extensible to support
  more use cases.
- Test cases were added for component ids in checkpoint files.
  The following scenarios were covered:
  - Primary and secondary both have values when a flush is
    shceduled.
  - Primary have values but not secondary when a flush is
    scheduled.
  - Primary is empty and an index is created through bulk
    load.
  - Primary has a single component and secondary is created
    through bulk load.
  - Primary has multiple components and secondary is created
    through bulk load.
- Each primary opTracker now keeps a list of ongoing flushes.
- FlushDataset now waits only for flushes only and
  not all io operations.
- Previously, we had many flushes scheduled on open datasets.
  This was not detected but after this change, a failure
  is thrown in such cases.
- Flush operations dont need to extend the comparable
  interface anymore since they are FIFO per index.

Change-Id: If24c9baaac2b79e7d1acf47fa2601767388ce988
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2632
Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <mhubail@apache.org>