Added replicate operator with materialization
be more aggressive to find shared plans in ExtractCommonOperatorRule
- find all the isomorphic subgraphs instead of just the ones on join build branches
- while expanding candidates handle the operators with multiple inputs
- analyze the DAG to find all the operators that can be co-scheduled, and infer the dependencies between clusters
- based on the dependencies, decide which outputs of a replicate operator needs materialization
- if the shared branch needs materialization, and it consists of only trivial operators (such as assign, unnest, datasource scan), that branch is discarded from the candidates
- modified the replicate operator descriptor to materialize the input if needed, and read from the materialized file for the outputs that requires materialization
- removed redundant decor variables in group-by
- fixed a bug on computing live variables for unnest-map operator: if the operator does not propagate inputs, those input variables should not be live anymore
- fixed a bug in ComplexUnnestToProductRule
Change-Id: If221d1507844f9409bf1163f93b0c04ef5848578
Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/86
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
45 files changed