tree 9225a0a42c5a1fc51e0a309d18cdf38f70468b12
parent 934ce9d903d36990d22febd69e08a81d0c8e40b2
author Rui Guo <ruig2@uci.edu> 1586797459 -0700
committer Ian Maxon <imaxon@uci.edu> 1587108413 +0000

[NO ISSUE] Remove out-of-date tokenizer

The string-based Tokenizer should be replaced with the array-based
IBinaryTokenizer. The Tokenizer is not used in the codebase in a
meaningful way, so let's remove it to make things clear.

Change-Id: I483604bf2a5e20c18f6224ac2a153667828dabfb
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/5763
Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@uci.edu>