| Ian Maxon | a1cc51b | 2020-08-07 13:11:35 -0700 | [diff] [blame^] | 1 | <!DOCTYPE html> |
| 2 | <!-- |
| 3 | | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/sqlpp/fulltext.md at 2020-07-30 |
| 4 | | Rendered using Apache Maven Fluido Skin 1.7 |
| 5 | --> |
| 6 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| 7 | <head> |
| 8 | <meta charset="UTF-8" /> |
| 9 | <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| 10 | <meta name="Date-Revision-yyyymmdd" content="20200730" /> |
| 11 | <meta http-equiv="Content-Language" content="en" /> |
| 12 | <title>AsterixDB – AsterixDB Support of Full-text search queries</title> |
| 13 | <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> |
| 14 | <link rel="stylesheet" href="../css/site.css" /> |
| 15 | <link rel="stylesheet" href="../css/print.css" media="print" /> |
| 16 | <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script> |
| 17 | |
| 18 | </head> |
| 19 | <body class="topBarDisabled"> |
| 20 | <div class="container-fluid"> |
| 21 | <div id="banner"> |
| 22 | <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div> |
| 23 | <div class="pull-right"></div> |
| 24 | <div class="clear"><hr/></div> |
| 25 | </div> |
| 26 | |
| 27 | <div id="breadcrumbs"> |
| 28 | <ul class="breadcrumb"> |
| 29 | <li id="publishDate">Last Published: 2020-07-30</li> |
| 30 | <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li> |
| 31 | <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li> |
| 32 | </ul> |
| 33 | </div> |
| 34 | <div class="row-fluid"> |
| 35 | <div id="leftColumn" class="span2"> |
| 36 | <div class="well sidebar-nav"> |
| 37 | <ul class="nav nav-list"> |
| 38 | <li class="nav-header">Get Started - Installation</li> |
| 39 | <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li> |
| 40 | <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li> |
| 41 | <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li> |
| 42 | <li class="nav-header">AsterixDB Primer</li> |
| 43 | <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li> |
| 44 | <li class="nav-header">Data Model</li> |
| 45 | <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li> |
| 46 | <li class="nav-header">Queries</li> |
| 47 | <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li> |
| 48 | <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li> |
| 49 | <li class="nav-header">API/SDK</li> |
| 50 | <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li> |
| 51 | <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li> |
| 52 | <li class="nav-header">Advanced Features</li> |
| 53 | <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li> |
| 54 | <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li> |
| 55 | <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li> |
| 56 | <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li> |
| 57 | <li class="active"><a href="#"><span class="none"></span>Support of Full-text Queries</a></li> |
| 58 | <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li> |
| 59 | <li class="nav-header">Deprecated</li> |
| 60 | <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li> |
| 61 | <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li> |
| 62 | <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li> |
| 63 | </ul> |
| 64 | <hr /> |
| 65 | <div id="poweredBy"> |
| 66 | <div class="clear"></div> |
| 67 | <div class="clear"></div> |
| 68 | <div class="clear"></div> |
| 69 | <div class="clear"></div> |
| 70 | <a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a> |
| 71 | </div> |
| 72 | </div> |
| 73 | </div> |
| 74 | <div id="bodyColumn" class="span10" > |
| 75 | <!-- |
| 76 | ! Licensed to the Apache Software Foundation (ASF) under one |
| 77 | ! or more contributor license agreements. See the NOTICE file |
| 78 | ! distributed with this work for additional information |
| 79 | ! regarding copyright ownership. The ASF licenses this file |
| 80 | ! to you under the Apache License, Version 2.0 (the |
| 81 | ! "License"); you may not use this file except in compliance |
| 82 | ! with the License. You may obtain a copy of the License at |
| 83 | ! |
| 84 | ! http://www.apache.org/licenses/LICENSE-2.0 |
| 85 | ! |
| 86 | ! Unless required by applicable law or agreed to in writing, |
| 87 | ! software distributed under the License is distributed on an |
| 88 | ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 89 | ! KIND, either express or implied. See the License for the |
| 90 | ! specific language governing permissions and limitations |
| 91 | ! under the License. |
| 92 | !--> |
| 93 | <h1>AsterixDB Support of Full-text search queries</h1> |
| 94 | <div class="section"> |
| 95 | <h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2> |
| 96 | <ul> |
| 97 | |
| 98 | <li><a href="#Motivation">Motivation</a></li> |
| 99 | <li><a href="#Syntax">Syntax</a></li> |
| 100 | <li><a href="#FulltextIndex">Creating and utilizing a Full-text index</a></li> |
| 101 | </ul></div> |
| 102 | <div class="section"> |
| 103 | <h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> |
| 104 | <p>Full-Text Search (FTS) queries are widely used in applications where users need to find records that satisfy an FTS predicate, i.e., where simple string-based matching is not sufficient. These queries are important when finding documents that contain a certain keyword is crucial. FTS queries are different from substring matching queries in that FTS queries find their query predicates as exact keywords in the given string, rather than treating a query predicate as a sequence of characters. For example, an FTS query that finds “rain” correctly returns a document when it contains “rain” as a word. However, a substring-matching query returns a document whenever it contains “rain” as a substring, for instance, a document with “brain” or “training” would be returned as well.</p></div> |
| 105 | <div class="section"> |
| 106 | <h2><a name="Syntax_.5BBack_to_TOC.5D"></a><a name="Syntax" id="Syntax">Syntax</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> |
| 107 | <p>The syntax of AsterixDB FTS follows a portion of the XQuery FullText Search syntax. Two basic forms are as follows:</p> |
| 108 | |
| 109 | <div> |
| 110 | <div> |
| 111 | <pre class="source"> ftcontains(Expression1, Expression2, {FullTextOption}) |
| 112 | ftcontains(Expression1, Expression2) |
| 113 | </pre></div></div> |
| 114 | |
| 115 | <p>For example, we can execute the following query to find Chirp messages where the <tt>messageText</tt> field includes “voice” as a word. Please note that an FTS search is case-insensitive. Thus, “Voice” or “voice” will be evaluated as the same word.</p> |
| 116 | |
| 117 | <div> |
| 118 | <div> |
| 119 | <pre class="source"> use TinySocial; |
| 120 | |
| 121 | select element {"chirpId": msg.chirpId} |
| 122 | from ChirpMessages msg |
| 123 | where ftcontains(msg.messageText, "voice", {"mode":"any"}); |
| 124 | </pre></div></div> |
| 125 | |
| 126 | <p>The DDL and DML of TinySocial can be found in <a href="../sqlpp/primer-sqlpp.html#ADM:_Modeling_Semistructured_Data_in_AsterixDB">ADM: Modeling Semistructured Data in AsterixDB</a>.</p> |
| 127 | <p>The <tt>Expression1</tt> is an expression that should be evaluable as a string at runtime as in the above example where <tt>msg.messageText</tt> is a string field. The <tt>Expression2</tt> can be a string, an (un)ordered list of string value(s), or an expression. In the last case, the given expression should be evaluable into one of the first two types, i.e., into a string value or an (un)ordered list of string value(s).</p> |
| 128 | <p>The following examples are all valid expressions.</p> |
| 129 | |
| 130 | <div> |
| 131 | <div> |
| 132 | <pre class="source"> ... where ftcontains(msg.messageText, "sound") |
| 133 | ... where ftcontains(msg.messageText, "sound", {"mode":"any"}) |
| 134 | ... where ftcontains(msg.messageText, ["sound", "system"], {"mode":"any"}) |
| 135 | ... where ftcontains(msg.messageText, {{"speed", "stand", "customization"}}, {"mode":"all"}) |
| 136 | </pre></div></div> |
| 137 | |
| 138 | <p>The last <tt>FullTextOption</tt> parameter clarifies the given FTS request. If you omit the <tt>FullTextOption</tt> parameter, then the default value will be set for each possible option. Currently, we only have one option named <tt>mode</tt>. And as we extend the FTS feature, more options will be added. Please note that the format of <tt>FullTextOption</tt> is a record, thus you need to put the option(s) in a record <tt>{}</tt>. The <tt>mode</tt> option indicates whether the given FTS query is a conjunctive (AND) or disjunctive (OR) search request. This option can be either <tt>“all”</tt> (AND) or <tt>“any”</tt> (OR). The default value for <tt>mode</tt> is <tt>“all”</tt>. If one specifies <tt>“any”</tt>, a disjunctive search will be conducted. For example, the following query will find documents whose <tt>messageText</tt> field contains “sound” or “system”, so a document will be returned if it contains either “sound”, “system”, or both of the keywords.</p> |
| 139 | |
| 140 | <div> |
| 141 | <div> |
| 142 | <pre class="source"> ... where ftcontains(msg.messageText, ["sound", "system"], {"mode":"any"}) |
| 143 | </pre></div></div> |
| 144 | |
| 145 | <p>The other option parameter,<tt>“all”</tt>, specifies a conjunctive search. The following examples will find the documents whose <tt>messageText</tt> field contains both “sound” and “system”. If a document contains only “sound” or “system” but not both, it will not be returned.</p> |
| 146 | |
| 147 | <div> |
| 148 | <div> |
| 149 | <pre class="source"> ... where ftcontains(msg.messageText, ["sound", "system"], {"mode":"all"}) |
| 150 | ... where ftcontains(msg.messageText, ["sound", "system"]) |
| 151 | </pre></div></div> |
| 152 | |
| 153 | <p>Currently AsterixDB doesn’t (yet) support phrase searches, so the following query will not work.</p> |
| 154 | |
| 155 | <div> |
| 156 | <div> |
| 157 | <pre class="source"> ... where ftcontains(msg.messageText, "sound system", {"mode":"any"}) |
| 158 | </pre></div></div> |
| 159 | |
| 160 | <p>As a workaround solution, the following query can be used to achieve a roughly similar goal. The difference is that the following queries will find documents where <tt>msg.messageText</tt> contains both “sound” and “system”, but the order and adjacency of “sound” and “system” are not checked, unlike in a phrase search. As a result, the query below would also return documents with “sound system can be installed.”, “system sound is perfect.”, or “sound is not clear. You may need to install a new system.”</p> |
| 161 | |
| 162 | <div> |
| 163 | <div> |
| 164 | <pre class="source"> ... where ftcontains(msg.messageText, ["sound", "system"], {"mode":"all"}) |
| 165 | ... where ftcontains(msg.messageText, ["sound", "system"]) |
| 166 | </pre></div></div> |
| 167 | </div> |
| 168 | <div class="section"> |
| 169 | <h2><a name="Creating_and_utilizing_a_Full-text_index_.5BBack_to_TOC.5D"></a><a name="FulltextIndex" id="FulltextIndex">Creating and utilizing a Full-text index</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2> |
| 170 | <p>When there is a full-text index on the field that is being searched, rather than scanning all records, AsterixDB can utilize that index to expedite the execution of a FTS query. To create a full-text index, you need to specify the index type as <tt>fulltext</tt> in your DDL statement. For instance, the following DDL statement create a full-text index on the <tt>GleambookMessages.message</tt> attribute. Note that a full-text index cannot be built on a dataset with the variable-length primary key (e.g., string).</p> |
| 171 | |
| 172 | <div> |
| 173 | <div> |
| 174 | <pre class="source">use TinySocial; |
| 175 | |
| 176 | create index messageFTSIdx on GleambookMessages(message) type fulltext; |
| 177 | </pre></div></div></div> |
| 178 | </div> |
| 179 | </div> |
| 180 | </div> |
| 181 | <hr/> |
| 182 | <footer> |
| 183 | <div class="container-fluid"> |
| 184 | <div class="row-fluid"> |
| 185 | <div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache |
| 186 | feather logo, and the Apache AsterixDB project logo are either |
| 187 | registered trademarks or trademarks of The Apache Software |
| 188 | Foundation in the United States and other countries. |
| 189 | All other marks mentioned may be trademarks or registered |
| 190 | trademarks of their respective owners. |
| 191 | </div> |
| 192 | </div> |
| 193 | </div> |
| 194 | </footer> |
| 195 | </body> |
| 196 | </html> |