Update site for release
Change-Id: I8aa5dce0b9a9c51df2fcafa91d2e4f47d9fa4cf3
Reviewed-on: https://asterix-gerrit.ics.uci.edu/3254
Reviewed-by: Ian Maxon <imaxon@uci.edu>
diff --git a/content/docs/0.9.4/ansible.html b/content/docs/0.9.4/ansible.html
index 4722841..6758e08 100644
--- a/content/docs/0.9.4/ansible.html
+++ b/content/docs/0.9.4/ansible.html
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/ansible.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/ansible.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
- <title>AsterixDB – Installation using Ansible</title>
+ <title>AsterixDB – </title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="./css/site.css" />
<link rel="stylesheet" href="./css/print.css" media="print" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li class="active"><a href="#"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
@@ -90,9 +90,6 @@
! specific language governing permissions and limitations
! under the License.
!-->
-<h1>Installation using Ansible</h1>
-<div class="section">
-<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
<ul>
<li><a href="#Introduction">Introduction</a></li>
@@ -117,7 +114,7 @@
! specific language governing permissions and limitations
! under the License.
!-->
-</div>
+
<div class="section">
<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
<p>This installation option provides several wrapped <a class="externalLink" href="https://www.ansible.com/">Ansible</a>-based scripts to deploy, start, stop, and erase an AsterixDB instance on a multi-node cluster without requiring users to interact with each individual node in the cluster.</p></div>
diff --git a/content/docs/0.9.4/api.html b/content/docs/0.9.4/api.html
index 2de7ec2..4ef672d 100644
--- a/content/docs/0.9.4/api.html
+++ b/content/docs/0.9.4/api.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/api.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/api.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – HTTP API to AsterixDB</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li class="active"><a href="#"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/aql/builtins.html b/content/docs/0.9.4/aql/builtins.html
index 43116f4..b04749d 100644
--- a/content/docs/0.9.4/aql/builtins.html
+++ b/content/docs/0.9.4/aql/builtins.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aql/builtins.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aql/builtins.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – Builtin Functions</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li class="active"><a href="#"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/aql/externaldata.html b/content/docs/0.9.4/aql/externaldata.html
index 5cbe512..b13dac8 100644
--- a/content/docs/0.9.4/aql/externaldata.html
+++ b/content/docs/0.9.4/aql/externaldata.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/externaldata.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/externaldata.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – Accessing External Data in AsterixDB</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li class="active"><a href="#"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/aql/filters.html b/content/docs/0.9.4/aql/filters.html
new file mode 100644
index 0000000..ec88bd6
--- /dev/null
+++ b/content/docs/0.9.4/aql/filters.html
@@ -0,0 +1,142 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/filters.md at 2019-03-07
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>AsterixDB – Filter-Based LSM Index Acceleration</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script>
+
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
+ <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">Get Started - Installation</li>
+ <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
+ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
+ <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
+ <li class="nav-header">AsterixDB Primer</li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
+ <li class="nav-header">Data Model</li>
+ <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
+ <li class="nav-header">Queries - SQL++</li>
+ <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
+ <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">API/SDK</li>
+ <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
+ <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
+ <li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
+ <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
+ <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+ ! Licensed to the Apache Software Foundation (ASF) under one
+ ! or more contributor license agreements. See the NOTICE file
+ ! distributed with this work for additional information
+ ! regarding copyright ownership. The ASF licenses this file
+ ! to you under the Apache License, Version 2.0 (the
+ ! "License"); you may not use this file except in compliance
+ ! with the License. You may obtain a copy of the License at
+ !
+ ! http://www.apache.org/licenses/LICENSE-2.0
+ !
+ ! Unless required by applicable law or agreed to in writing,
+ ! software distributed under the License is distributed on an
+ ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ! KIND, either express or implied. See the License for the
+ ! specific language governing permissions and limitations
+ ! under the License.
+ !-->
+<h1>Filter-Based LSM Index Acceleration</h1>
+<div class="section">
+<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2>
+<ul>
+
+<li><a href="#Motivation">Motivation</a></li>
+<li><a href="#FiltersInAsterixDB">Filters in AsterixDB</a></li>
+<li><a href="#FiltersAndMergePolicies">Filters and Merge Policies</a></li>
+</ul></div>
+<div class="section">
+<h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>Traditional relational databases usually employ conventional index structures such as B+ trees due to their low read latency. However, such traditional index structures use in-place writes to perform updates, resulting in costly random writes to disk. Today’s emerging applications often involve insert-intensive workloads for which the cost of random writes prohibits efficient ingestion of data. Consequently, popular NoSQL systems such as Cassandra, HBase, LevelDB, BigTable, etc. have adopted Log-Structured Merge (LSM) Trees as their storage structure. LSM-trees avoids the cost of random writes by batching updates into a component of the index that resides in main memory – an <i>in-memory component</i>. When the space occupancy of the in-memory component exceeds a specified threshold, its entries are <i>flushed</i> to disk forming a new component – a <i>disk component</i>. As disk components accumulate on disk, they are periodically merged together subject to a <i>merge policy</i> that decides when and what to merge. The benefit of the LSM-trees comes at the cost of possibly sacrificing read efficiency, but, it has been shown in previous studies that these inefficiencies can be mostly mitigated.</p>
+<p>AsterixDB has also embraced LSM-trees, not just by using them as primary indexes, but also by using the same LSM-ification technique for all of its secondary index structures. In particular, AsterixDB adopted a generic framework for converting a class of indexes (that includes conventional B+ trees, R trees, and inverted indexes) into LSM-based secondary indexes, allowing higher data ingestion rates. In fact, for certain index structures, our results have shown that using an LSM-based version of an index can be made to significantly outperform its conventional counterpart for <i>both</i> ingestion and query speed (an example of such an index being the R-tree for spatial data).</p>
+<p>Since an LSM-based index naturally partitions data into multiple disk components, it is possible, when answering certain queries, to exploit partitioning to only access some components and safely filter out the remaining components, thus reducing query times. For instance, referring to our <a href="primer.html#ADM:_Modeling_Semistructed_Data_in_AsterixDB">TinySocial</a> example, suppose a user always retrieves tweets from the <tt>TweetMessages</tt> dataset based on the <tt>send-time</tt> field (e.g., tweets posted in the last 24 hours). Since there is not a secondary index on the <tt>send-time</tt> field, the only available option for AsterixDB would be to scan the whole <tt>TweetMessages</tt> dataset and then apply the predicate as a post-processing step. However, if disk components of the primary index were tagged with the minimum and maximum timestamp values of the objects they contain, we could utilize the tagged information to directly access the primary index and prune components that do not match the query predicate. Thus, we could save substantial cost by avoiding scanning the whole dataset and only access the relevant components. We simply call such tagging information that are associated with components, filters. (Note that even if there were a secondary index on <tt>send-time</tt> field, using filters could save substantial cost by avoiding accessing the secondary index, followed by probing the primary index for every fetched entry.) Moreover, the same filtering technique can also be used with any secondary LSM index (e.g., an LSM R-tree), in case the query contains multiple predicates (e.g., spatial and temporal predicates), to obtain similar pruning power.</p></div>
+<div class="section">
+<h2><a name="Filters_in_AsterixDB_.5BBack_to_TOC.5D"></a><a name="FiltersInAsterixDB" id="FiltersInAsterixDB">Filters in AsterixDB</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>We have added support for LSM-based filters to all of AsterixDB’s index types. To enable the use of filters, the user must specify the filter’s key when creating a dataset, as shown below:</p>
+<div class="section">
+<div class="section">
+<h4><a name="Creating_a_Dataset_with_a_Filter"></a>Creating a Dataset with a Filter</h4>
+
+<div>
+<div>
+<pre class="source"> create dataset Tweets(TweetType) primary key tweetid with filter on send-time;
+</pre></div></div>
+
+<p>Filters can be created on any totally ordered datatype (i.e., any field that can be indexed using a B+ -tree), such as integers, doubles, floats, UUIDs, datetimes, etc.</p>
+<p>When a dataset with a filter is created, the name of the filter’s key field is persisted in the <tt>Metadata.Dataset</tt> dataset (which is the metadata dataset that stores the details of each dataset in an AsterixDB instance) so that DML operations against the dataset can recognize the existence of filters and can update them or utilize them accordingly. Creating a dataset with a filter in AsterixDB implies that the primary and all secondary indexes of that dataset will maintain filters on their disk components. Once a filtered dataset is created, the user can use the dataset normally (just like any other dataset). AsterixDB will automatically maintain the filters and will leverage them to efficiently answer queries whenever possible (i.e., when a query has predicates on the filter’s key).</p></div></div></div>
+<div class="section">
+<h2><a name="Filters_and_Merge_Policies_.5BBack_to_TOC.5D"></a><a name="FiltersAndMergePolicies" id="FiltersAndMergePolicies">Filters and Merge Policies</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>The AsterixDB default merge policy, the prefix merge policy, relies on component sizes and the number of components to decide which components to merge. This merge policy has proven to provide excellent performance for both ingestion and queries. However, when evaluating our filtering solution with the prefix policy, we observed a behavior that can reduce filter effectiveness. In particular, we noticed that under the prefix merge policy, the disk components of a secondary index tend to be constantly merged into a single component. This is because the prefix policy relies on a single size parameter for all of the indexes of a dataset. This parameter is typically chosen based on the sizes of the disk components of the primary index, which tend to be much larger than the sizes of the secondary indexes’ disk components. This difference caused the prefix merge policy to behave similarly to the constant merge policy (i.e., relatively poorly) when applied to secondary indexes in the sense that the secondary indexes are constantly merged into a single disk component. Consequently, the effectiveness of filters on secondary indexes was greatly reduced under the prefix-merge policy, but they were still effective when probing the primary index. Based on this behavior, we developed a new merge policy, an improved version of the prefix policy, called the correlated-prefix policy. The basic idea of this policy is that it delegates the decision of merging the disk components of all the indexes in a dataset to the primary index. When the policy decides that the primary index needs to be merged (using the same decision criteria as for the prefix policy), then it will issue successive merge requests to the I/O scheduler on behalf of all other indexes associated with the same dataset. The end result is that secondary indexes will always have the same number of disk components as their primary index under the correlated-prefix merge policy. This has improved query performance, since disk components of secondary indexes now have a much better chance of being pruned.</p></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
+ feather logo, and the Apache AsterixDB project logo are either
+ registered trademarks or trademarks of The Apache Software
+ Foundation in the United States and other countries.
+ All other marks mentioned may be trademarks or registered
+ trademarks of their respective owners.
+ </div>
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/content/docs/0.9.4/aql/fulltext.html b/content/docs/0.9.4/aql/fulltext.html
new file mode 100644
index 0000000..17da151
--- /dev/null
+++ b/content/docs/0.9.4/aql/fulltext.html
@@ -0,0 +1,208 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/fulltext.md at 2019-03-07
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>AsterixDB – AsterixDB Support of Full-text search queries</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script>
+
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
+ <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">Get Started - Installation</li>
+ <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
+ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
+ <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
+ <li class="nav-header">AsterixDB Primer</li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
+ <li class="nav-header">Data Model</li>
+ <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
+ <li class="nav-header">Queries - SQL++</li>
+ <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
+ <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">API/SDK</li>
+ <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
+ <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
+ <li class="nav-header">Advanced Features</li>
+ <li class="active"><a href="#"><span class="none"></span>Support of Full-text Queries</a></li>
+ <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
+ <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+ ! Licensed to the Apache Software Foundation (ASF) under one
+ ! or more contributor license agreements. See the NOTICE file
+ ! distributed with this work for additional information
+ ! regarding copyright ownership. The ASF licenses this file
+ ! to you under the Apache License, Version 2.0 (the
+ ! "License"); you may not use this file except in compliance
+ ! with the License. You may obtain a copy of the License at
+ !
+ ! http://www.apache.org/licenses/LICENSE-2.0
+ !
+ ! Unless required by applicable law or agreed to in writing,
+ ! software distributed under the License is distributed on an
+ ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ! KIND, either express or implied. See the License for the
+ ! specific language governing permissions and limitations
+ ! under the License.
+ !-->
+<h1>AsterixDB Support of Full-text search queries</h1>
+<div class="section">
+<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2>
+<ul>
+
+<li><a href="#Motivation">Motivation</a></li>
+<li><a href="#Syntax">Syntax</a></li>
+<li><a href="#FulltextIndex">Creating and utilizing a Full-text index</a></li>
+</ul></div>
+<div class="section">
+<h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>Full-Text Search (FTS) queries are widely used in applications where users need to find records that satisfy an FTS predicate, i.e., where simple string-based matching is not sufficient. These queries are important when finding documents that contain a certain keyword is crucial. FTS queries are different from substring matching queries in that FTS queries find their query predicates as exact keywords in the given string, rather than treating a query predicate as a sequence of characters. For example, an FTS query that finds “rain” correctly returns a document when it contains “rain” as a word. However, a substring-matching query returns a document whenever it contains “rain” as a substring, for instance, a document with “brain” or “training” would be returned as well.</p></div>
+<div class="section">
+<h2><a name="Syntax_.5BBack_to_TOC.5D"></a><a name="Syntax" id="Syntax">Syntax</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>The syntax of AsterixDB FTS follows a portion of the XQuery FullText Search syntax. Two basic forms are as follows:</p>
+
+<div>
+<div>
+<pre class="source"> ftcontains(Expression1, Expression2, {FullTextOption})
+ ftcontains(Expression1, Expression2)
+</pre></div></div>
+
+<p>For example, we can execute the following query to find tweet messages where the <tt>message-text</tt> field includes “voice” as a word. Please note that an FTS search is case-insensitive. Thus, “Voice” or “voice” will be evaluated as the same word.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $msg in dataset TweetMessages
+ where ftcontains($msg.message-text, "voice", {"mode":"any"})
+ return {"id": $msg.id}
+</pre></div></div>
+
+<p>The DDL and DML of TinySocial can be found in <a href="primer.html#ADM:_Modeling_Semistructed_Data_in_AsterixDB">ADM: Modeling Semistructed Data in AsterixDB</a>.</p>
+<p>The same query can be also expressed in the SQL++.</p>
+
+<div>
+<div>
+<pre class="source"> use TinySocial;
+
+ select element {"id":msg.id}
+ from TweetMessages as msg
+ where TinySocial.ftcontains(msg.`message-text`, "voice", {"mode":"any"})
+</pre></div></div>
+
+<p>The <tt>Expression1</tt> is an expression that should be evaluable as a string at runtime as in the above example where <tt>$msg.message-text</tt> is a string field. The <tt>Expression2</tt> can be a string, an (un)ordered list of string value(s), or an expression. In the last case, the given expression should be evaluable into one of the first two types, i.e., into a string value or an (un)ordered list of string value(s).</p>
+<p>The following examples are all valid expressions.</p>
+
+<div>
+<div>
+<pre class="source"> ... where ftcontains($msg.message-text, "sound")
+ ... where ftcontains($msg.message-text, "sound", {"mode":"any"})
+ ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"any"})
+ ... where ftcontains($msg.message-text, {{"speed", "stand", "customization"}}, {"mode":"all"})
+ ... where ftcontains($msg.message-text, let $keyword_list := ["voice", "system"] return $keyword_list, {"mode":"all"})
+ ... where ftcontains($msg.message-text, $keyword_list, {"mode":"any"})
+</pre></div></div>
+
+<p>In the last example above, <tt>$keyword_list</tt> should evaluate to a string or an (un)ordered list of string value(s).</p>
+<p>The last <tt>FullTextOption</tt> parameter clarifies the given FTS request. If you omit the <tt>FullTextOption</tt> parameter, then the default value will be set for each possible option. Currently, we only have one option named <tt>mode</tt>. And as we extend the FTS feature, more options will be added. Please note that the format of <tt>FullTextOption</tt> is a record, thus you need to put the option(s) in a record <tt>{}</tt>. The <tt>mode</tt> option indicates whether the given FTS query is a conjunctive (AND) or disjunctive (OR) search request. This option can be either <tt>“any”</tt> or <tt>“all”</tt>. The default value for <tt>mode</tt> is <tt>“all”</tt>. If one specifies <tt>“any”</tt>, a disjunctive search will be conducted. For example, the following query will find documents whose <tt>message-text</tt> field contains “sound” or “system”, so a document will be returned if it contains either “sound”, “system”, or both of the keywords.</p>
+
+<div>
+<div>
+<pre class="source"> ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"any"})
+</pre></div></div>
+
+<p>The other option parameter,<tt>“all”</tt>, specifies a conjunctive search. The following examples will find the documents whose <tt>message-text</tt> field contains both “sound” and “system”. If a document contains only “sound” or “system” but not both, it will not be returned.</p>
+
+<div>
+<div>
+<pre class="source"> ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"all"})
+ ... where ftcontains($msg.message-text, ["sound", "system"])
+</pre></div></div>
+
+<p>Currently AsterixDB doesn’t (yet) support phrase searches, so the following query will not work.</p>
+
+<div>
+<div>
+<pre class="source"> ... where ftcontains($msg.message-text, "sound system", {"mode":"any"})
+</pre></div></div>
+
+<p>As a workaround solution, the following query can be used to achieve a roughly similar goal. The difference is that the following queries will find documents where <tt>$msg.message-text</tt> contains both “sound” and “system”, but the order and adjacency of “sound” and “system” are not checked, unlike in a phrase search. As a result, the query below would also return documents with “sound system can be installed.”, “system sound is perfect.”, or “sound is not clear. You may need to install a new system.”</p>
+
+<div>
+<div>
+<pre class="source"> ... where ftcontains($msg.message-text, ["sound", "system"], {"mode":"all"})
+ ... where ftcontains($msg.message-text, ["sound", "system"])
+</pre></div></div>
+</div>
+<div class="section">
+<h2><a name="Creating_and_utilizing_a_Full-text_index_.5BBack_to_TOC.5D"></a><a name="FulltextIndex" id="FulltextIndex">Creating and utilizing a Full-text index</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>When there is a full-text index on the field that is being searched, rather than scanning all records, AsterixDB can utilize that index to expedite the execution of a FTS query. To create a full-text index, you need to specify the index type as <tt>fulltext</tt> in your DDL statement. For instance, the following DDL statement create a full-text index on the TweetMessages.message-text attribute.</p>
+
+<div>
+<div>
+<pre class="source">create index messageFTSIdx on TweetMessages(message-text) type fulltext;
+</pre></div></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
+ feather logo, and the Apache AsterixDB project logo are either
+ registered trademarks or trademarks of The Apache Software
+ Foundation in the United States and other countries.
+ All other marks mentioned may be trademarks or registered
+ trademarks of their respective owners.
+ </div>
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/content/docs/0.9.4/aql/js-sdk.html b/content/docs/0.9.4/aql/js-sdk.html
index c5902e4..ab93443 100644
--- a/content/docs/0.9.4/aql/js-sdk.html
+++ b/content/docs/0.9.4/aql/js-sdk.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/js-sdk.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/js-sdk.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – AsterixDB Javascript SDK</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/aql/manual.html b/content/docs/0.9.4/aql/manual.html
index ce8e842..e69a88f 100644
--- a/content/docs/0.9.4/aql/manual.html
+++ b/content/docs/0.9.4/aql/manual.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/manual.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/manual.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – The Asterix Query Language, Version 1.0</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li class="active"><a href="#"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li class="active"><a href="#"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/aql/primer.html b/content/docs/0.9.4/aql/primer.html
index 36d59dd..00ae29f 100644
--- a/content/docs/0.9.4/aql/primer.html
+++ b/content/docs/0.9.4/aql/primer.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/primer.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/primer.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – AsterixDB 101: An ADM and AQL Primer</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li class="active"><a href="#"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/aql/similarity.html b/content/docs/0.9.4/aql/similarity.html
new file mode 100644
index 0000000..f5558ee
--- /dev/null
+++ b/content/docs/0.9.4/aql/similarity.html
@@ -0,0 +1,310 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/aql/similarity.md at 2019-03-07
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>AsterixDB – AsterixDB Support of Similarity Queries</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script>
+
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
+ <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">Get Started - Installation</li>
+ <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
+ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
+ <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
+ <li class="nav-header">AsterixDB Primer</li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
+ <li class="nav-header">Data Model</li>
+ <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
+ <li class="nav-header">Queries - SQL++</li>
+ <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
+ <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">API/SDK</li>
+ <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
+ <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
+ <li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
+ <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
+ <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Support of Similarity Queries</a></li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+ ! Licensed to the Apache Software Foundation (ASF) under one
+ ! or more contributor license agreements. See the NOTICE file
+ ! distributed with this work for additional information
+ ! regarding copyright ownership. The ASF licenses this file
+ ! to you under the Apache License, Version 2.0 (the
+ ! "License"); you may not use this file except in compliance
+ ! with the License. You may obtain a copy of the License at
+ !
+ ! http://www.apache.org/licenses/LICENSE-2.0
+ !
+ ! Unless required by applicable law or agreed to in writing,
+ ! software distributed under the License is distributed on an
+ ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ! KIND, either express or implied. See the License for the
+ ! specific language governing permissions and limitations
+ ! under the License.
+ !-->
+<h1>AsterixDB Support of Similarity Queries</h1>
+<div class="section">
+<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2>
+<ul>
+
+<li><a href="#Motivation">Motivation</a></li>
+<li><a href="#DataTypesAndSimilarityFunctions">Data Types and Similarity Functions</a></li>
+<li><a href="#SimilaritySelectionQueries">Similarity Selection Queries</a></li>
+<li><a href="#SimilarityJoinQueries">Similarity Join Queries</a></li>
+<li><a href="#UsingIndexesToSupportSimilarityQueries">Using Indexes to Support Similarity Queries</a></li>
+</ul></div>
+<div class="section">
+<h2><a name="Motivation_.5BBack_to_TOC.5D"></a><a name="Motivation" id="Motivation">Motivation</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>Similarity queries are widely used in applications where users need to find objects that satisfy a similarity predicate, while exact matching is not sufficient. These queries are especially important for social and Web applications, where errors, abbreviations, and inconsistencies are common. As an example, we may want to find all the movies starring Schwarzenegger, while we don’t know the exact spelling of his last name (despite his popularity in both the movie industry and politics :-)). As another example, we want to find all the Facebook users who have similar friends. To meet this type of needs, AsterixDB supports similarity queries using efficient indexes and algorithms.</p></div>
+<div class="section">
+<h2><a name="Data_Types_and_Similarity_Functions_.5BBack_to_TOC.5D"></a><a name="DataTypesAndSimilarityFunctions" id="DataTypesAndSimilarityFunctions">Data Types and Similarity Functions</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>AsterixDB supports <a class="externalLink" href="http://en.wikipedia.org/wiki/Levenshtein_distance">edit distance</a> (on strings) and <a class="externalLink" href="http://en.wikipedia.org/wiki/Jaccard_index">Jaccard</a> (on sets). For instance, in our <a href="primer.html#ADM:_Modeling_Semistructed_Data_in_AsterixDB">TinySocial</a> example, the <tt>friend-ids</tt> of a Facebook user forms a set of friends, and we can define a similarity between the sets of friends of two users. We can also convert a string to a set of grams of a length “n” (called “n-grams”) and define the Jaccard similarity between the two gram sets of the two strings. Formally, the “n-grams” of a string are its substrings of length “n”. For instance, the 3-grams of the string <tt>schwarzenegger</tt> are <tt>sch</tt>, <tt>chw</tt>, <tt>hwa</tt>, …, <tt>ger</tt>.</p>
+<p>AsterixDB provides <a href="functions.html#Tokenizing_Functions">tokenization functions</a> to convert strings to sets, and the <a href="functions.html#Similarity_Functions">similarity functions</a>.</p></div>
+<div class="section">
+<h2><a name="Similarity_Selection_Queries_.5BBack_to_TOC.5D"></a><a name="SimilaritySelectionQueries" id="SimilaritySelectionQueries">Similarity Selection Queries</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>The following query asks for all the Facebook users whose name is similar to <tt>Suzanna Tilson</tt>, i.e., their edit distance is at most 2.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $user in dataset('FacebookUsers')
+ let $ed := edit-distance($user.name, "Suzanna Tilson")
+ where $ed <= 2
+ return $user
+</pre></div></div>
+
+<p>The following query asks for all the Facebook users whose set of friend ids is similar to <tt>[1,5,9,10]</tt>, i.e., their Jaccard similarity is at least 0.6.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $user in dataset('FacebookUsers')
+ let $sim := similarity-jaccard($user.friend-ids, [1,5,9,10])
+ where $sim >= 0.6f
+ return $user
+</pre></div></div>
+
+<p>AsterixDB allows a user to use a similarity operator <tt>~=</tt> to express a condition by defining the similarity function and threshold using “set” statements earlier. For instance, the above query can be equivalently written as:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ set simfunction "jaccard";
+ set simthreshold "0.6f";
+
+ for $user in dataset('FacebookUsers')
+ where $user.friend-ids ~= [1,5,9,10]
+ return $user
+</pre></div></div>
+
+<p>In this query, we first declare Jaccard as the similarity function using <tt>simfunction</tt> and then specify the threshold <tt>0.6f</tt> using <tt>simthreshold</tt>.</p></div>
+<div class="section">
+<h2><a name="Similarity_Join_Queries_.5BBack_to_TOC.5D"></a><a name="SimilarityJoinQueries" id="SimilarityJoinQueries">Similarity Join Queries</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>AsterixDB supports fuzzy joins between two sets. The following <a href="primer.html#Query_5_-_Fuzzy_Join">query</a> finds, for each Facebook user, all Twitter users with names similar to their name based on the edit distance.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ set simfunction "edit-distance";
+ set simthreshold "3";
+
+ for $fbu in dataset FacebookUsers
+ return {
+ "id": $fbu.id,
+ "name": $fbu.name,
+ "similar-users": for $t in dataset TweetMessages
+ let $tu := $t.user
+ where $tu.name ~= $fbu.name
+ return {
+ "twitter-screenname": $tu.screen-name,
+ "twitter-name": $tu.name
+ }
+ };
+</pre></div></div>
+</div>
+<div class="section">
+<h2><a name="Using_Indexes_to_Support_Similarity_Queries_.5BBack_to_TOC.5D"></a><a name="UsingIndexesToSupportSimilarityQueries" id="UsingIndexesToSupportSimilarityQueries">Using Indexes to Support Similarity Queries</a> <font size="4"><a href="#toc">[Back to TOC]</a></font></h2>
+<p>AsterixDB uses two types of indexes to support similarity queries, namely “ngram index” and “keyword index”.</p>
+<div class="section">
+<h3><a name="NGram_Index"></a>NGram Index</h3>
+<p>An “ngram index” is constructed on a set of strings. We generate n-grams for each string, and build an inverted list for each n-gram that includes the ids of the strings with this gram. A similarity query can be answered efficiently by accessing the inverted lists of the grams in the query and counting the number of occurrences of the string ids on these inverted lists. The similar idea can be used to answer queries with Jaccard similarity. A detailed description of these techniques is available at this <a class="externalLink" href="http://www.ics.uci.edu/~chenli/pub/icde2009-memreducer.pdf">paper</a>.</p>
+<p>For instance, the following DDL statements create an ngram index on the <tt>FacebookUsers.name</tt> attribute using an inverted index of 3-grams.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ create index fbUserIdx on FacebookUsers(name) type ngram(3);
+</pre></div></div>
+
+<p>The number “3” in “ngram(3)” is the length “n” in the grams. This index can be used to optimize similarity queries on this attribute using <a href="functions.html#edit-distance">edit-distance</a>, <a href="functions.html#edit-distance-check">edit-distance-check</a>, <a href="functions.html#similarity-jaccard">similarity-jaccard</a>, or <a href="functions.html#similarity-jaccard-check">similarity-jaccard-check</a> queries on this attribute where the similarity is defined on sets of 3-grams. This index can also be used to optimize queries with the “<a href="(functions.html#contains">contains()</a>” predicate (i.e., substring matching) since it can be also be solved by counting on the inverted lists of the grams in the query string.</p>
+<div class="section">
+<h4><a name="NGram_Index_usage_case_-_edit-distance"></a>NGram Index usage case - <a href="functions.html#edit-distance">edit-distance</a></h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $user in dataset('FacebookUsers')
+ let $ed := edit-distance($user.name, "Suzanna Tilson")
+ where $ed <= 2
+ return $user
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="NGram_Index_usage_case_-_edit-distance-check"></a>NGram Index usage case - <a href="functions.html#edit-distance-check">edit-distance-check</a></h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $user in dataset('FacebookUsers')
+ let $ed := edit-distance-check($user.name, "Suzanna Tilson", 2)
+ where $ed[0]
+ return $ed[1]
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="NGram_Index_usage_case_-_similarity-jaccard"></a>NGram Index usage case - <a href="functions.html#similarity-jaccard">similarity-jaccard</a></h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $user in dataset('FacebookUsers')
+ let $sim := similarity-jaccard($user.friend-ids, [1,5,9,10])
+ where $sim >= 0.6f
+ return $user
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="NGram_Index_usage_case_-_similarity-jaccard-check"></a>NGram Index usage case - <a href="functions.html#similarity-jaccard-check">similarity-jaccard-check</a></h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $user in dataset('FacebookUsers')
+ let $sim := similarity-jaccard-check($user.friend-ids, [1,5,9,10], 0.6f)
+ where $sim[0]
+ return $user
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="NGram_Index_usage_case_-_contains.28.29"></a>NGram Index usage case - <a href="(functions.html#contains">contains()</a></h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ for $i in dataset('FacebookMessages')
+ where contains($i.message, "phone")
+ return {"mid": $i.message-id, "message": $i.message}
+</pre></div></div>
+</div></div>
+<div class="section">
+<h3><a name="Keyword_Index"></a>Keyword Index</h3>
+<p>A “keyword index” is constructed on a set of strings or sets (e.g., OrderedList, UnorderedList). Instead of generating grams as in an ngram index, we generate tokens (e.g., words) and for each token, construct an inverted list that includes the ids of the objects with this token. The following two examples show how to create keyword index on two different types:</p>
+<div class="section">
+<h4><a name="Keyword_Index_on_String_Type"></a>Keyword Index on String Type</h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ drop index FacebookMessages.fbMessageIdx if exists;
+ create index fbMessageIdx on FacebookMessages(message) type keyword;
+
+ for $o in dataset('FacebookMessages')
+ let $jacc := similarity-jaccard-check(word-tokens($o.message), word-tokens("love like ccast"), 0.2f)
+ where $jacc[0]
+ return $o
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="Keyword_Index_on_UnorderedList_Type"></a>Keyword Index on UnorderedList Type</h4>
+
+<div>
+<div>
+<pre class="source"> use dataverse TinySocial;
+
+ create index fbUserIdx_fids on FacebookUsers(friend-ids) type keyword;
+
+ for $c in dataset('FacebookUsers')
+ let $jacc := similarity-jaccard-check($c.friend-ids, {{3,10}}, 0.5f)
+ where $jacc[0]
+ return $c
+</pre></div></div>
+
+<p>As shown above, keyword index can be used to optimize queries with token-based similarity predicates, including <a href="functions.html#similarity-jaccard">similarity-jaccard</a> and <a href="functions.html#similarity-jaccard-check">similarity-jaccard-check</a>.</p></div></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
+ feather logo, and the Apache AsterixDB project logo are either
+ registered trademarks or trademarks of The Apache Software
+ Foundation in the United States and other countries.
+ All other marks mentioned may be trademarks or registered
+ trademarks of their respective owners.
+ </div>
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/content/docs/0.9.4/aws.html b/content/docs/0.9.4/aws.html
index 977e782..1e0f142 100644
--- a/content/docs/0.9.4/aws.html
+++ b/content/docs/0.9.4/aws.html
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aws.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aws.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
- <title>AsterixDB – Installation using Amazon Web Services</title>
+ <title>AsterixDB – </title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="./css/site.css" />
<link rel="stylesheet" href="./css/print.css" media="print" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li class="active"><a href="#"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
@@ -90,9 +90,6 @@
! specific language governing permissions and limitations
! under the License.
!-->
-<h1>Installation using Amazon Web Services</h1>
-<div class="section">
-<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
<ul>
<li><a href="#Introduction">Introduction</a></li>
@@ -117,7 +114,7 @@
! specific language governing permissions and limitations
! under the License.
!-->
-</div>
+
<div class="section">
<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p>
diff --git a/content/docs/0.9.4/csv.html b/content/docs/0.9.4/csv.html
index 02a9830..d37087d 100644
--- a/content/docs/0.9.4/csv.html
+++ b/content/docs/0.9.4/csv.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/csv.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/csv.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – CSV Support in AsterixDB</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li class="active"><a href="#"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/datamodel.html b/content/docs/0.9.4/datamodel.html
index 4682c66..16116d5 100644
--- a/content/docs/0.9.4/datamodel.html
+++ b/content/docs/0.9.4/datamodel.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/datamodel.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/datamodel.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – The Asterix Data Model (ADM)</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li class="active"><a href="#"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/feeds/tutorial.html b/content/docs/0.9.4/feeds/tutorial.html
new file mode 100644
index 0000000..924cd86
--- /dev/null
+++ b/content/docs/0.9.4/feeds/tutorial.html
@@ -0,0 +1,450 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/feeds/tutorial.md at 2019-03-07
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>AsterixDB – Support for Data Ingestion in AsterixDB</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script>
+
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href=".././" id="bannerLeft"><img src="../images/asterixlogo.png" alt="AsterixDB"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
+ <li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">Get Started - Installation</li>
+ <li><a href="../ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
+ <li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
+ <li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
+ <li class="nav-header">AsterixDB Primer</li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
+ <li class="nav-header">Data Model</li>
+ <li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
+ <li class="nav-header">Queries - SQL++</li>
+ <li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
+ <li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">API/SDK</li>
+ <li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
+ <li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
+ <li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
+ <li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Support for Data Ingestion</a></li>
+ <li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href=".././" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="../images/asterixlogo.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+ ! Licensed to the Apache Software Foundation (ASF) under one
+ ! or more contributor license agreements. See the NOTICE file
+ ! distributed with this work for additional information
+ ! regarding copyright ownership. The ASF licenses this file
+ ! to you under the Apache License, Version 2.0 (the
+ ! "License"); you may not use this file except in compliance
+ ! with the License. You may obtain a copy of the License at
+ !
+ ! http://www.apache.org/licenses/LICENSE-2.0
+ !
+ ! Unless required by applicable law or agreed to in writing,
+ ! software distributed under the License is distributed on an
+ ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ! KIND, either express or implied. See the License for the
+ ! specific language governing permissions and limitations
+ ! under the License.
+ !-->
+<h1>Support for Data Ingestion in AsterixDB</h1>
+<div class="section">
+<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
+<ul>
+
+<li><a href="#Introduction">Introduction</a></li>
+<li><a href="#FeedAdapters">Feed Adapters</a></li>
+</ul><!-- * [Feed Policies](#FeedPolicies) -->
+</div>
+<div class="section">
+<h2><a name="Introduction">Introduction</a></h2>
+<p>In this document, we describe the support for data ingestion in AsterixDB. Data feeds are a new mechanism for having continuous data arrive into a BDMS from external sources and incrementally populate a persisted dataset and associated indexes. We add a new BDMS architectural component, called a data feed, that makes a Big Data system the caretaker for functionality that used to live outside, and we show how it improves users’ lives and system performance.</p></div>
+<div class="section">
+<h2><a name="Feed_Adapters"></a><a name="FeedAdapters">Feed Adapters</a></h2>
+<p>The functionality of establishing a connection with a data source and receiving, parsing and translating its data into ADM objects (for storage inside AsterixDB) is contained in a feed adapter. A feed adapter is an implementation of an interface and its details are specific to a given data source. An adapter may optionally be given parameters to configure its runtime behavior. Depending upon the data transfer protocol/APIs offered by the data source, a feed adapter may operate in a push or a pull mode. Push mode involves just one initial request by the adapter to the data source for setting up the connection. Once a connection is authorized, the data source “pushes” data to the adapter without any subsequent requests by the adapter. In contrast, when operating in a pull mode, the adapter makes a separate request each time to receive data. AsterixDB currently provides built-in adapters for several popular data sources such as Twitter and RSS feeds. AsterixDB additionally provides a generic socket-based adapter that can be used to ingest data that is directed at a prescribed socket.</p>
+<p>In this tutorial, we shall describe building two example data ingestion pipelines that cover the popular scenarios of ingesting data from (a) Twitter (b) RSS (c) Socket Feed source.</p>
+<div class="section">
+<div class="section">
+<h4><a name="Ingesting_Twitter_Stream"></a>Ingesting Twitter Stream</h4>
+<p>We shall use the built-in push-based Twitter adapter. As a pre-requisite, we must define a Tweet using the AsterixDB Data Model (ADM) and the AsterixDB Query Language (AQL). Given below are the type definitions in AQL that create a Tweet datatype which is representative of a real tweet as obtained from Twitter.</p>
+
+<div>
+<div>
+<pre class="source"> create dataverse feeds;
+ use dataverse feeds;
+
+ create type TwitterUser as closed {
+ screen_name: string,
+ lang: string,
+ friends_count: int32,
+ statuses_count: int32
+ };
+
+ create type Tweet as open {
+ id: int64,
+ user: TwitterUser
+ }
+
+ create dataset Tweets (Tweet)
+ primary key id;
+</pre></div></div>
+
+<p>We also create a dataset that we shall use to persist the tweets in AsterixDB. Next we make use of the <tt>create feed</tt> AQL statement to define our example data feed.</p>
+<div class="section">
+<h5><a name="Using_the_.E2.80.9Cpush_twitter.E2.80.9D_feed_adapter"></a>Using the “push_twitter” feed adapter</h5>
+<p>The “push_twitter” adapter requires setting up an application account with Twitter. To retrieve tweets, Twitter requires registering an application. Registration involves providing a name and a brief description for the application. Each application has associated OAuth authentication credentials that include OAuth keys and tokens. Accessing the Twitter API requires providing the following. 1. Consumer Key (API Key) 2. Consumer Secret (API Secret) 3. Access Token 4. Access Token Secret</p>
+<p>The “push_twitter” adapter takes as configuration the above mentioned parameters. End users are required to obtain the above authentication credentials prior to using the “push_twitter” adapter. For further information on obtaining OAuth keys and tokens and registering an application with Twitter, please visit <a class="externalLink" href="http://apps.twitter.com">http://apps.twitter.com</a></p>
+<p>Given below is an example AQL statement that creates a feed called “TwitterFeed” by using the “push_twitter” adapter.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ create feed TwitterFeed if not exists using "push_twitter"
+ (("type-name"="Tweet"),
+ ("format"="twitter-status"),
+ ("consumer.key"="************"),
+ ("consumer.secret"="**************"),
+ ("access.token"="**********"),
+ ("access.token.secret"="*************"));
+</pre></div></div>
+
+<p>It is required that the above authentication parameters are provided valid. Note that the <tt>create feed</tt> statement does not initiate the flow of data from Twitter into the AsterixDB instance. Instead, the <tt>create feed</tt> statement only results in registering the feed with the instance. The flow of data along a feed is initiated when it is connected to a target dataset using the connect feed statement and activated using the start feed statement.</p>
+<p>The Twitter adapter also supports several Twitter streaming APIs as follow:</p>
+<ol style="list-style-type: decimal">
+
+<li>Track filter (“keywords”=“AsterixDB, Apache”)</li>
+<li>Locations filter (“locations”=“-29.7, 79.2, 36.7, 72.0; -124.848974,-66.885444, 24.396308, 49.384358”)</li>
+<li>Language filter (“language”=“en”)</li>
+<li>Filter level (“filter-level”=“low”)</li>
+</ol>
+<p>An example of Twitter adapter tracking tweets with keyword “news” can be described using following ddl:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ create feed TwitterFeed if not exists using "push_twitter"
+ (("type-name"="Tweet"),
+ ("format"="twitter-status"),
+ ("consumer.key"="************"),
+ ("consumer.secret"="**************"),
+ ("access.token"="**********"),
+ ("access.token.secret"="*************"),
+ ("keywords"="news"));
+</pre></div></div>
+
+<p>For more details about these APIs, please visit <a class="externalLink" href="https://dev.twitter.com/streaming/overview/request-parameters">https://dev.twitter.com/streaming/overview/request-parameters</a></p></div></div>
+<div class="section">
+<h4><a name="Lifecycle_of_a_Feed"></a>Lifecycle of a Feed</h4>
+<p>A feed is a logical artifact that is brought to life (i.e., its data flow is initiated) only when it is activated using the <tt>start feed</tt> statement. Before we active a feed, we need to designate the dataset where the data to be persisted using <tt>connect feed</tt> statement. Subsequent to a <tt>connect feed</tt> statement, the feed is said to be in the connected state. After that, <tt>start feed</tt> statement will activate the feed, and start the dataflow from feed to its connected dataset. Multiple feeds can simultaneously be connected to a dataset such that the contents of the dataset represent the union of the connected feeds. Also one feed can be simultaneously connected to multiple target datasets.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ connect feed TwitterFeed to dataset Tweets;
+
+ start feed TwitterFeed;
+</pre></div></div>
+
+<p>The <tt>connect feed</tt> statement above directs AsterixDB to persist the data from <tt>TwitterFeed</tt> feed into the <tt>Tweets</tt> dataset. The <tt>start feed</tt> statement will activate the feed and start the dataflow. If it is required (by the high-level application) to also retain the raw tweets obtained from Twitter, the end user may additionally choose to connect TwitterFeed to a different dataset.</p>
+<p>Let the feed run for a minute, then run the following query to see the latest tweets that are stored into the data set.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ for $i in dataset Tweets limit 10 return $i;
+</pre></div></div>
+
+<p>The dataflow of data from a feed can be terminated explicitly by <tt>stop feed</tt> statement.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ stop feed TwitterFeed;
+</pre></div></div>
+
+<p>The <tt>disconnnect statement</tt> can be used to disconnect the feed from certain dataset.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ disconnect feed TwitterFeed from dataset Tweets;
+</pre></div></div>
+</div></div>
+<div class="section">
+<h3><a name="Ingesting_with_Other_Adapters"></a>Ingesting with Other Adapters</h3>
+<p>AsterixDB has several builtin feed adapters for data ingestion. User can also implement their own adapters and plug them into AsterixDB. Here we introduce <tt>rss_feed</tt>, <tt>socket_adapter</tt> and <tt>localfs</tt> feed adapter that cover most of the common application scenarios.</p>
+<div class="section">
+<div class="section">
+<h5><a name="Using_the_.E2.80.9Crss_feed.E2.80.9D_feed_adapter"></a>Using the “rss_feed” feed adapter</h5>
+<p><tt>rss_feed</tt> adapter allows retrieving data given a collection of RSS end point URLs. As observed in the case of ingesting tweets, it is required to model an RSS data item using AQL.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ create type Rss if not exists as open {
+ id: string,
+ title: string,
+ description: string,
+ link: string
+ };
+
+ create dataset RssDataset (Rss)
+ primary key id;
+</pre></div></div>
+
+<p>Next, we define an RSS feed using our built-in adapter “rss_feed”.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ create feed my_feed using
+ rss_feed (
+ ("type-name"="Rss"),
+ ("format"="rss"),
+ ("url"="http://rss.cnn.com/rss/edition.rss")
+ );
+</pre></div></div>
+
+<p>In the above definition, the configuration parameter “url” can be a comma-separated list that reflects a collection of RSS URLs, where each URL corresponds to an RSS endpoint or an RSS feed. The “rss_feed” retrieves data from each of the specified RSS URLs (comma separated values) in parallel.</p>
+<p>The following statements connect the feed into the <tt>RssDataset</tt>:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ connect feed my_feed to dataset RssDataset;
+</pre></div></div>
+
+<p>The following statements activate the feed and start the dataflow:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ start feed my_feed;
+</pre></div></div>
+
+<p>The following statements show the latest data from the data set, stop the feed, and disconnect the feed from the data set.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ for $i in dataset RssDataset limit 10 return $i;
+
+ stop feed my_feed
+
+ disconnect feed my_feed from dataset RssDataset;
+</pre></div></div>
+</div>
+<div class="section">
+<h5><a name="Using_the_.E2.80.9Csocket_adapter.E2.80.9D_feed_adapter"></a>Using the “socket_adapter” feed adapter</h5>
+<p><tt>socket_adapter</tt> feed opens a web socket on the given node which allows user to push data into AsterixDB directly. Here is an example:</p>
+
+<div>
+<div>
+<pre class="source"> drop dataverse feeds if exists;
+ create dataverse feeds;
+ use dataverse feeds;
+
+ create type TestDataType as open {
+ screenName: string
+ }
+
+ create dataset TestDataset(TestDataType) primary key screenName;
+
+ create feed TestSocketFeed using socket_adapter
+ (
+ ("sockets"="127.0.0.1:10001"),
+ ("address-type"="IP"),
+ ("type-name"="TestDataType"),
+ ("format"="adm")
+ );
+
+ connect feed TestSocketFeed to dataset TestDataset;
+
+ use dataverse feeds;
+ start feed TestSocketFeed;
+</pre></div></div>
+
+<p>The above statements create a socket feed which is listening to “10001” port of the host machine. This feed accepts data records in “adm” format. As an example, you can download the sample dataset <a href="../data/chu.adm">Chirp Users</a> and push them line by line into the socket feed using any socket client you like. Following is a socket client example in Python:</p>
+
+<div>
+<div>
+<pre class="source"> from socket import socket
+
+ ip = '127.0.0.1'
+ port1 = 10001
+ filePath = 'chu.adm'
+
+ sock1 = socket()
+ sock1.connect((ip, port1))
+
+ with open(filePath) as inputData:
+ for line in inputData:
+ sock1.sendall(line)
+ sock1.close()
+</pre></div></div>
+</div></div>
+<div class="section">
+<h4><a name="Using_the_.E2.80.9Clocalfs.E2.80.9D_feed_adapter"></a>Using the “localfs” feed adapter</h4>
+<p><tt>localfs</tt> adapter enables data ingestion from local file system. It allows user to feed data records on local disk into a dataset. A DDL example for creating a <tt>localfs</tt> feed is given as follow:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ create type TweetType as closed {
+ id: string,
+ username : string,
+ location : string,
+ text : string,
+ timestamp : string
+ }
+
+ create dataset Tweets(TweetType)
+ primary key id;
+
+ create feed TweetFeed
+ using localfs
+ (("type-name"="TweetType"),("path"="HOSTNAME://LOCAL_FILE_PATH"),("format"="adm"))
+</pre></div></div>
+
+<p>Similar to previous examples, we need to define the datatype and dataset this feed uses. The “path” parameter refers to the local datafile that we want to ingest data from. <tt>HOSTNAME</tt> can either be the IP address or node name of the machine which holds the file. <tt>LOCAL_FILE_PATH</tt> indicates the absolute path to the file on that machine. Similarly to <tt>socket_adapter</tt>, this feed takes <tt>adm</tt> formatted data records.</p></div></div>
+<div class="section">
+<h3><a name="Datatype_for_feed_and_target_dataset"></a>Datatype for feed and target dataset</h3>
+<p>The “type-name” parameter in create feed statement defines the <tt>datatype</tt> of the datasource. In most use cases, feed will have the same <tt>datatype</tt> as the target dataset. However, if we want to perform certain preprocess before the data records gets into the target dataset (append autogenerated key, apply user defined functions, etc.), we will need to define the datatypes for feed and dataset separately.</p>
+<div class="section">
+<h4><a name="Ingestion_with_autogenerated_key"></a>Ingestion with autogenerated key</h4>
+<p>AsterixDB supports using autogenerated uuid as the primary key for dataset. When we use this feature, we will need to define a datatype with the primary key field, and specify that field to be autogenerated when creating the dataset. Use that same datatype in feed definition will cause a type discrepancy since there is no such field in the datasource. Thus, we will need to define two separate datatypes for feed and dataset:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ create type DBLPFeedType as closed {
+ dblpid: string,
+ title: string,
+ authors: string,
+ misc: string
+ }
+
+ create type DBLPDataSetType as open {
+ id: uuid,
+ dblpid: string,
+ title: string,
+ authors: string,
+ misc: string
+ }
+ create dataset DBLPDataset(DBLPDataSetType) primary key id autogenerated;
+
+ create feed DBLPFeed using socket_adapter
+ (
+ ("sockets"="127.0.0.1:10001"),
+ ("address-type"="IP"),
+ ("type-name"="DBLPFeedType"),
+ ("format"="adm")
+ );
+
+ connect feed DBLPFeed to dataset DBLPDataset;
+
+ start feed DBLPFeed;
+</pre></div></div>
+</div></div></div>
+<div class="section">
+<h2><a name="Policies_for_Feed_Ingestion"></a><a name="FeedPolicies">Policies for Feed Ingestion</a></h2>
+<p>Multiple feeds may be concurrently operational on an AsterixDB cluster, each competing for resources (CPU cycles, network bandwidth, disk IO) to maintain pace with their respective data sources. As a data management system, AsterixDB is able to manage a set of concurrent feeds and make dynamic decisions related to the allocation of resources, resolving resource bottlenecks and the handling of failures. Each feed has its own set of constraints, influenced largely by the nature of its data source and the applications that intend to consume and process the ingested data. Consider an application that intends to discover the trending topics on Twitter by analyzing tweets that are being processed. Losing a few tweets may be acceptable. In contrast, when ingesting from a data source that provides a click-stream of ad clicks, losing data would translate to a loss of revenue for an application that tracks revenue by charging advertisers per click.</p>
+<p>AsterixDB allows a data feed to have an associated ingestion policy that is expressed as a collection of parameters and associated values. An ingestion policy dictates the runtime behavior of the feed in response to resource bottlenecks and failures. AsterixDB provides a set of policies that help customize the system’s runtime behavior when handling excess objects.</p>
+<div class="section">
+<div class="section">
+<h4><a name="Policies"></a>Policies</h4>
+<ul>
+
+<li>
+
+<p><i>Spill</i>: Objects that cannot be processed by an operator for lack of resources (referred to as excess objects hereafter) should be persisted to the local disk for deferred processing.</p>
+</li>
+<li>
+
+<p><i>Discard</i>: Excess objects should be discarded.</p>
+</li>
+</ul>
+<p>Note that the end user may choose to form a custom policy. For example, it is possible in AsterixDB to create a custom policy that spills excess objects to disk and subsequently resorts to throttling if the spillage crosses a configured threshold. In all cases, the desired ingestion policy is specified as part of the <tt>connect feed</tt> statement or else the “Basic” policy will be chosen as the default.</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ connect feed TwitterFeed to dataset Tweets
+ using policy Basic;
+</pre></div></div></div></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
+ feather logo, and the Apache AsterixDB project logo are either
+ registered trademarks or trademarks of The Apache Software
+ Foundation in the United States and other countries.
+ All other marks mentioned may be trademarks or registered
+ trademarks of their respective owners.
+ </div>
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/content/docs/0.9.4/index.html b/content/docs/0.9.4/index.html
index 02923b1..3a9ddff 100644
--- a/content/docs/0.9.4/index.html
+++ b/content/docs/0.9.4/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/index.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/index.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – AsterixDB</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/ncservice.html b/content/docs/0.9.4/ncservice.html
index dd36797..9288ec6 100644
--- a/content/docs/0.9.4/ncservice.html
+++ b/content/docs/0.9.4/ncservice.html
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/ncservice.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/ncservice.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
- <title>AsterixDB – Installation using NCService</title>
+ <title>AsterixDB – Table of Contents</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="./css/site.css" />
<link rel="stylesheet" href="./css/print.css" media="print" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
@@ -90,7 +90,6 @@
! specific language governing permissions and limitations
! under the License.
!-->
-<h1>Installation using NCService</h1>
<div class="section">
<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2>
<ul>
diff --git a/content/docs/0.9.4/overview.html b/content/docs/0.9.4/overview.html
index ba0ef11..707945c 100644
--- a/content/docs/0.9.4/overview.html
+++ b/content/docs/0.9.4/overview.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/overview.md at 2018-10-11
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/overview.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181011" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – AsterixDB: A Big Data Management System</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -26,8 +26,8 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-11</li>
- <li id="projectVersion" class="pull-right">Version: 0.9.5-SNAPSHOT</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
</div>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/sqlpp/builtins.html b/content/docs/0.9.4/sqlpp/builtins.html
index 2ace95a..b3b73bc 100644
--- a/content/docs/0.9.4/sqlpp/builtins.html
+++ b/content/docs/0.9.4/sqlpp/builtins.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/sqlpp/builtins.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/sqlpp/builtins.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – Builtin Functions</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li class="active"><a href="#"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/sqlpp/manual.html b/content/docs/0.9.4/sqlpp/manual.html
index 8d8b5df..93e124f 100644
--- a/content/docs/0.9.4/sqlpp/manual.html
+++ b/content/docs/0.9.4/sqlpp/manual.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/sqlpp/manual.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/sqlpp/manual.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – The SQL++ Query Language</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="../sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="../sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li class="active"><a href="#"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
diff --git a/content/docs/0.9.4/sqlpp/primer-sqlpp.html b/content/docs/0.9.4/sqlpp/primer-sqlpp.html
index 1a935a1..98d10dc 100644
--- a/content/docs/0.9.4/sqlpp/primer-sqlpp.html
+++ b/content/docs/0.9.4/sqlpp/primer-sqlpp.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/sqlpp/primer-sqlpp.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/sqlpp/primer-sqlpp.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
<title>AsterixDB – AsterixDB 101: An ADM and SQL++ Primer</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="../index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="../ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="../aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li class="active"><a href="#"><span class="none"></span>Using SQL++</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="../aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="../datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="../sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="../sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="../aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="../aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="../api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="../csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="../aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="../aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="../feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="../feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li><a href="../udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="../sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="../sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="../sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="../aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="../aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="../aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="../aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="../aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
@@ -768,36 +768,7 @@
{ "chirp": { "chirpId": "5", "user": { "screenName": "NathanGiesen@211", "lang": "en", "friendsCount": 39339, "statusesCount": 473, "name": "Nathan Giesen", "followersCount": 49416 }, "senderLocation": point("40.09,92.69"), "sendTime": datetime("2006-08-04T10:10:00.000Z"), "referredTopics": {{ "product-w", "speed" }}, "messageText": " can't stand product-w its speed is terrible:(" }, "similarChirps": [ ] }
{ "chirp": { "chirpId": "8", "user": { "screenName": "NathanGiesen@211", "lang": "en", "friendsCount": 39339, "statusesCount": 473, "name": "Nathan Giesen", "followersCount": 49416 }, "senderLocation": point("46.05,93.34"), "sendTime": datetime("2005-10-14T10:10:00.000Z"), "referredTopics": {{ "product-z", "shortcut-menu" }}, "messageText": " like product-z the shortcut-menu is awesome:)" }, "similarChirps": [ ] }
</pre></div></div>
-</div></div>
-<div class="section">
-<h2><a name="Inserting_New_Data"></a>Inserting New Data</h2>
-<p>In addition to loading and querying data, AsterixDB supports incremental additions to datasets via the SQL++ <i>INSERT</i> statement.</p>
-<p>The following example adds a new chirp by user “NathanGiesen@211” to the ChirpMessages dataset. (An astute reader may notice that this chirp was issued a half an hour after his last chirp, so his counts have all gone up in the interim, although he appears not to have moved in the last half hour.)</p>
-
-<div>
-<div>
-<pre class="source"> USE TinySocial;
-
- INSERT INTO ChirpMessages
- (
- {"chirpId": "13",
- "user":
- {"screenName": "NathanGiesen@211",
- "lang": "en",
- "friendsCount": 39345,
- "statusesCount": 479,
- "name": "Nathan Giesen",
- "followersCount": 49420
- },
- "senderLocation": point("47.44,80.65"),
- "sendTime": datetime("2008-04-26T10:10:35"),
- "referredTopics": {{"chirping"}},
- "messageText": "chirpy chirp, my fellow chirpers!"
- }
- );
-</pre></div></div>
-
-<p>In general, the data to be inserted may be specified using any valid SQL++ query expression. The insertion of a single object instance, as in this example, is just a special case where the query expression happens to be a object constructor involving only constants.</p>
+</div>
<div class="section">
<h3><a name="Deleting_Existing_Data"></a>Deleting Existing Data</h3>
<p>In addition to inserting new data, AsterixDB supports deletion from datasets via the SQL++ <i>DELETE</i> statement. The statement supports “searched delete” semantics, and its <i>WHERE</i> clause can involve any valid XQuery expression.</p>
diff --git a/content/docs/0.9.4/udf.html b/content/docs/0.9.4/udf.html
index 65bc7bb..e5816d4 100644
--- a/content/docs/0.9.4/udf.html
+++ b/content/docs/0.9.4/udf.html
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/udf.md at 2018-10-12
+ | Generated by Apache Maven Doxia Site Renderer 1.8.1 from src/site/markdown/udf.md at 2019-03-07
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20181012" />
+ <meta name="Date-Revision-yyyymmdd" content="20190307" />
<meta http-equiv="Content-Language" content="en" />
- <title>AsterixDB – User-defined Functions</title>
+ <title>AsterixDB – Support for User Defined Functions in AsterixDB</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="./css/site.css" />
<link rel="stylesheet" href="./css/print.css" media="print" />
@@ -26,7 +26,7 @@
<div id="breadcrumbs">
<ul class="breadcrumb">
- <li id="publishDate">Last Published: 2018-10-12</li>
+ <li id="publishDate">Last Published: 2019-03-07</li>
<li id="projectVersion" class="pull-right">Version: 0.9.4</li>
<li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
</ul>
@@ -40,26 +40,26 @@
<li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
<li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
<li class="nav-header">AsterixDB Primer</li>
- <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
+ <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
+ <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
<li class="nav-header">Data Model</li>
<li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
- <li class="nav-header">Queries</li>
+ <li class="nav-header">Queries - SQL++</li>
<li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
<li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
+ <li class="nav-header">Queries - AQL</li>
+ <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
+ <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
<li class="nav-header">API/SDK</li>
<li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
<li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
<li class="nav-header">Advanced Features</li>
+ <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
<li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
- <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
+ <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
<li class="active"><a href="#"><span class="none"></span>User Defined Functions</a></li>
- <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
- <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
- <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
- <li class="nav-header">Deprecated</li>
- <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
- <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
- <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
+ <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
+ <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
</ul>
<hr />
<div id="poweredBy">
@@ -90,173 +90,160 @@
! specific language governing permissions and limitations
! under the License.
!-->
-<h1>User-defined Functions</h1>
+<h1>Support for User Defined Functions in AsterixDB</h1>
<div class="section">
<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
<ul>
-<li><a href="#introduction">Introduction</a></li>
-<li><a href="#installingUDF">Installing an UDF Library</a></li>
-<li><a href="#UDFOnFeeds">Attaching an UDF on Data Feeds</a></li>
-<li><a href="#udfConfiguration">A quick look of the UDF configuration</a></li>
-<li><a href="#uninstall">Unstalling an UDF Library</a><!--
-! Licensed to the Apache Software Foundation (ASF) under one
-! or more contributor license agreements. See the NOTICE file
-! distributed with this work for additional information
-! regarding copyright ownership. The ASF licenses this file
-! to you under the Apache License, Version 2.0 (the
-! "License"); you may not use this file except in compliance
-! with the License. You may obtain a copy of the License at
-!
-! http://www.apache.org/licenses/LICENSE-2.0
-!
-! Unless required by applicable law or agreed to in writing,
-! software distributed under the License is distributed on an
-! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-! KIND, either express or implied. See the License for the
-! specific language governing permissions and limitations
-! under the License.
-!--></li>
-</ul></div>
+<li><a href="#PreprocessingCollectedData">Using UDF to preprocess feed-collected data</a></li>
+<li><a href="#WritingAnExternalUDF">Writing an External UDF</a></li>
+<li><a href="#CreatingAnAsterixDBLibrary">Creating an AsterixDB Library</a></li>
+<li><a href="#installingUDF">Installing an AsterixDB Library</a></li>
+</ul>
+<p>In this document, we describe the support for implementing, using, and installing user-defined functions (UDF) in AsterixDB. We will explain how we can use UDFs to preprocess, e.g., data collected using feeds (see the <a href="feeds/tutorial.html">feeds tutorial</a>).</p>
<div class="section">
-<h2><a name="Introduction"></a><a name="introduction">Introduction</a></h2>
-<p>Apache AsterixDB supports two languages for writing user-defined functions (UDFs): SQL++ and Java. A user can encapsulate data processing logic into a UDF and invoke it later repeatedly. For SQL++ functions, a user can refer to <a href="sqlpp/manual.html#Functions">SQL++ Functions</a> for their usages. In this document, we focus on how to install/invoke/uninstall a Java function library using the Ansible script that we provide.</p></div>
-<div class="section">
-<h2><a name="Installing_an_UDF_Library"></a><a name="installingUDF">Installing an UDF Library</a></h2>
-<p>UDFs have to be installed offline. This section describes the process assuming that you have followed the preceding <a href="ansible.html">ansible installation instructions</a> to deploy an AsterixDB instance on your local machine or cluster. Here are the instructions to install an UDF library:</p>
+<h3><a name="Installing_an_AsterixDB_Library"></a><a name="installingUDF">Installing an AsterixDB Library</a></h3>
+<p>We assume you have followed the <a href="../install.html">installation instructions</a> to set up a running AsterixDB instance. Let us refer your AsterixDB instance by the name “my_asterix”.</p>
<ul>
<li>
-<p>Step 1: Stop the AsterixDB instance if it is ACTIVE.</p>
+<p>Step 1: Stop the AsterixDB instance if it is in the ACTIVE state.</p>
<div>
<div>
-<pre class="source">$ bin/stop.sh
+<pre class="source">$ managix stop -n my_asterix
</pre></div></div>
</li>
<li>
-<p>Step 2: Deploy the UDF package.</p>
+<p>Step 2: Install the library using Managix install command. Just to illustrate, we use the help command to look up the syntax</p>
<div>
<div>
-<pre class="source">$ bin/udf.sh -m i -d DATAVERSE_NAME -l LIBRARY_NAME -p UDF_PACKAGE_PATH
-</pre></div></div>
-</li>
-<li>
-
-<p>Step 3: Start AsterixDB</p>
-
-<div>
-<div>
-<pre class="source">$ bin/start.sh
+<pre class="source">$ managix help -cmd install
+Installs a library to an asterix instance.
+Options
+n Name of Asterix Instance
+d Name of the dataverse under which the library will be installed
+l Name of the library
+p Path to library zip bundle
</pre></div></div>
</li>
</ul>
-<p>After AsterixDB starts, you can use the following query to check whether your UDFs have been sucessfully registered with the system.</p>
+<p>Above is a sample output and explains the usage and the required parameters. Each library has a name and is installed under a dataverse. Recall that we had created a dataverse by the name - “feeds” prior to creating our datatypes and dataset. We shall name our library - “testlib”.</p>
+<p>We assume you have a library zip bundle that needs to be installed. To install the library, use the Managix install command. An example is shown below.</p>
<div>
<div>
-<pre class="source"> SELECT * FROM Metadata.`Function`;
+<pre class="source"> $ managix install -n my_asterix -d feeds -l testlib -p extlibs/asterix-external-data-0.8.7-binary-assembly.zip
</pre></div></div>
-<p>In the AsterixDB source release, we provide several sample UDFs that you can try out. You need to build the AsterixDB source to get the compiled UDF package. It can be found under the <tt>asterixdb-external</tt> sub-project. Assuming that these UDFs have been installed into the <tt>udfs</tt> dataverse and <tt>testlib</tt> library, here is an example that uses the sample UDF <tt>mysum</tt> to compute the sum of two input integers.</p>
+<p>You should see the following message:</p>
<div>
<div>
-<pre class="source"> use udfs;
-
- testlib#mysum(3,4);
+<pre class="source"> INFO: Installed library testlib
</pre></div></div>
-</div>
+
+<p>We shall next start our AsterixDB instance using the start command as shown below.</p>
+
+<div>
+<div>
+<pre class="source"> $ managix start -n my_asterix
+</pre></div></div>
+
+<p>You may now use the AsterixDB library in AQL statements and queries. To look at the installed artifacts, you may execute the following query at the AsterixDB web-console.</p>
+
+<div>
+<div>
+<pre class="source"> for $x in dataset Metadata.Function
+ return $x
+
+ for $x in dataset Metadata.Library
+ return $x
+</pre></div></div>
+
+<p>Our library is now installed and is ready to be used.</p></div></div>
<div class="section">
-<h2><a name="Attaching_a_UDF_on_Data_Feeds"></a><a name="UDFOnFeeds" id="UDFOnFeeds">Attaching a UDF on Data Feeds</a></h2>
-<p>In <a href="feeds.html">Data Ingestion using feeds</a>, we introduced an efficient way for users to get data into AsterixDB. In some use cases, users may want to pre-process the incoming data before storing it into the dataset. To meet this need, AsterixDB allows the user to attach a UDF onto the ingestion pipeline. Following the example in <a href="feeds.html">Data Ingestion</a>, here we show an example of how to attach a UDF that extracts the user names mentioned from the incoming Tweet text, storing the processed Tweets into a dataset.</p>
-<p>We start by creating the datatype and dataset that will be used for the feed and UDF. One thing to keep in mind is that data flows from the feed to the UDF and then to the dataset. This means that the feed’s datatype should be the same as the input type of the UDF, and the output datatype of the UDF should be the same as the dataset’s datatype. Thus, users should make sure that their datatypes are consistent in the UDF configuration. Users can also take advantage of open datatypes in AsterixDB by creating a minimum description of the data for simplicity. Here we use open datatypes:</p>
+<h2><a name="Preprocessing_Collected_Data"></a><a name="PreprocessingCollectedData" id="PreprocessingCollectedData">Preprocessing Collected Data</a></h2>
+<p>In the following we assume that you already created the <tt>TwitterFeed</tt> and its corresponding data types and dataset following the instruction explained in the <a href="feeds/tutorial.html">feeds tutorial</a>.</p>
+<p>A feed definition may optionally include the specification of a user-defined function that is to be applied to each feed object prior to persistence. Examples of pre-processing might include adding attributes, filtering out objects, sampling, sentiment analysis, feature extraction, etc. We can express a UDF, which can be defined in AQL or in a programming language such as Java, to perform such pre-processing. An AQL UDF is a good fit when pre-processing a object requires the result of a query (join or aggregate) over data contained in AsterixDB datasets. More sophisticated processing such as sentiment analysis of text is better handled by providing a Java UDF. A Java UDF has an initialization phase that allows the UDF to access any resources it may need to initialize itself prior to being used in a data flow. It is assumed by the AsterixDB compiler to be stateless and thus usable as an embarrassingly parallel black box. In contrast, the AsterixDB compiler can reason about an AQL UDF and involve the use of indexes during its invocation.</p>
+<p>We consider an example transformation of a raw tweet into its lightweight version called <tt>ProcessedTweet</tt>, which is defined next.</p>
<div>
<div>
-<pre class="source"> use udfs;
+<pre class="source"> use dataverse feeds;
- create type TweetType if not exists as open {
- id: int64
+ create type ProcessedTweet if not exists as open {
+ id: string,
+ user_name:string,
+ location:point,
+ created_at:string,
+ message_text:string,
+ country: string,
+ topics: {{string}}
};
- create dataset ProcessedTweets(TweetType) primary key id;
+ create dataset ProcessedTweets(ProcessedTweet)
+ primary key id;
</pre></div></div>
-<p>As the <tt>TweetType</tt> is an open datatype, processed Tweets can be stored into the dataset after they are annotated with an extra attribute. Given the datatype and dataset above, we can create a Twitter Feed with the same datatype. Please refer to section <a href="feeds.html">Data Ingestion</a> if you have any trouble in creating feeds.</p>
+<p>The processing required in transforming a collected tweet to its lighter version of type <tt>ProcessedTweet</tt> involves extracting the topics or hash-tags (if any) in a tweet and collecting them in the referred “topics” attribute for the tweet. Additionally, the latitude and longitude values (doubles) are combined into the spatial point type. Note that spatial data types are considered as first-class citizens that come with the support for creating indexes. Next we show a revised version of our example TwitterFeed that involves the use of a UDF. We assume that the UDF that contains the transformation logic into a “ProcessedTweet” is available as a Java UDF inside an AsterixDB library named ‘testlib’. We defer the writing of a Java UDF and its installation as part of an AsterixDB library to a later section of this document.</p>
<div>
<div>
-<pre class="source"> use udfs;
+<pre class="source"> use dataverse feeds;
- create feed TwitterFeed with {
- "adapter-name": "push_twitter",
- "type-name": "TweetType",
- "format": "twitter-status",
- "consumer.key": "************",
- "consumer.secret": "************",
- "access.token": "**********",
- "access.token.secret": "*************"
- };
+ create feed ProcessedTwitterFeed if not exists
+ using "push_twitter"
+ (("type-name"="Tweet"),
+ ("consumer.key"="************"),
+ ("consumer.secret"="**************"),
+ ("access.token"="**********"),
+ ("access.token.secret"="*************"))
+
+ apply function testlib#addHashTagsInPlace;
</pre></div></div>
-<p>After creating the feed, we attach the UDF onto the feed pipeline and start the feed with following statements:</p>
-
-<div>
-<div>
-<pre class="source"> use udfs;
-
- connect feed TwitterFeed to dataset ProcessedTweets apply function udfs#addMentionedUsers;
-
- start feed TwitterFeed;
-</pre></div></div>
-
-<p>You can check the annotated Tweets by querying the <tt>ProcessedTweets</tt> dataset:</p>
-
-<div>
-<div>
-<pre class="source"> SELECT * FROM ProcessedTweets LIMIT 10;
-</pre></div></div>
-</div>
+<p>Note that a feed adaptor and a UDF act as pluggable components. These contribute towards providing a generic “plug-and-play” model where custom implementations can be provided to cater to specific requirements.</p>
<div class="section">
-<h2><a name="A_quick_look_of_the_UDF_configuration"></a><a name="udfConfiguration">A quick look of the UDF configuration</a></h2>
-<p>AsterixDB uses an XML configuration file to describe the UDFs. A user can use it to define and reuse their compiled UDFs for different purposes. Here is a snippet of the configuration used in our <a href="#UDFOnFeeds">previous example</a>:</p>
-
-<div>
-<div>
-<pre class="source"> <libraryFunction>
- <name>addMentionedUsers</name>
- <function_type>SCALAR</function_type>
- <argument_type>TweetType</argument_type>
- <return_type>TweetType</return_type>
- <definition>org.apache.asterix.external.library.AddMentionedUsersFactory</definition>
- <parameters>text</parameters>
- </libraryFunction>
-</pre></div></div>
-
-<p>Here are the explanations of the fields in the configuration file:</p>
-
-<div>
-<div>
-<pre class="source"> name: The proper name that is used for invoke the function.
- function_type: The type of the function.
- argument_type: The datatype of the arguments passed in. If there is more than one parameter, separate them with comma(s), e.g., `AINT32,AINT32`.
- return_type: The datatype of the returning value.
- definition: A reference to the function factory.
- parameters: The parameters passed into the function.
-</pre></div></div>
-
-<p>In our feeds example, we passed in <tt>"text"</tt> as a parameter to the function so it knows which field to look at to get the Tweet text. If the Twitter API were to change its field names in the future, we can accommodate that change by simply modifying the configuration file instead of recompiling the whole UDF package. This feature can be further utilized in use cases where a user has a Machine Learning algorithm with different trained model files. If you are interested, You can find more examples <a class="externalLink" href="https://github.com/apache/asterixdb/tree/master/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library">here</a></p></div>
<div class="section">
-<h2><a name="Unstalling_an_UDF_Library"></a><a name="uninstall">Unstalling an UDF Library</a></h2>
-<p>If you want to uninstall the UDF library, put AsterixDB into <tt>INACTVIVE</tt> mode and run following command:</p>
+<h4><a name="Building_a_Cascade_Network_of_Feeds"></a>Building a Cascade Network of Feeds</h4>
+<p>Multiple high-level applications may wish to consume the data ingested from a data feed. Each such application might perceive the feed in a different way and require the arriving data to be processed and/or persisted differently. Building a separate flow of data from the external source for each application is wasteful of resources as the pre-processing or transformations required by each application might overlap and could be done together in an incremental fashion to avoid redundancy. A single flow of data from the external source could provide data for multiple applications. To achieve this, we introduce the notion of primary and secondary feeds in AsterixDB.</p>
+<p>A feed in AsterixDB is considered to be a primary feed if it gets its data from an external data source. The objects contained in a feed (subsequent to any pre-processing) are directed to a designated AsterixDB dataset. Alternatively or additionally, these objects can be used to derive other feeds known as secondary feeds. A secondary feed is similar to its parent feed in every other aspect; it can have an associated UDF to allow for any subsequent processing, can be persisted into a dataset, and/or can be made to derive other secondary feeds to form a cascade network. A primary feed and a dependent secondary feed form a hierarchy. As an example, we next show an example AQL statement that redefines the previous feed “ProcessedTwitterFeed” in terms of their respective parent feed (TwitterFeed).</p>
<div>
<div>
-<pre class="source"> $ bin/udf.sh -m u -d DATAVERSE_NAME -l LIBRARY_NAME
+<pre class="source"> use dataverse feeds;
+
+ drop feed ProcessedTwitterFeed if exists;
+
+ create secondary feed ProcessedTwitterFeed from feed TwitterFeed
+ apply function testlib#addHashTags;
+
+ connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
+</pre></div></div>
+
+<p>The <tt>addHashTags</tt> function is already provided in the example UDF.To see what objects are being inserted into the dataset, we can perform a simple dataset scan after allowing a few moments for the feed to start ingesting data:</p>
+
+<div>
+<div>
+<pre class="source"> use dataverse feeds;
+
+ for $i in dataset ProcessedTweets limit 10 return $i;
+</pre></div></div>
+
+<p>For an example of how to write a Java UDF from scratch, the source for the example UDF that has been used in this tutorial is available [here] (<a class="externalLink" href="https://github.com/apache/asterixdb/tree/master/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library">https://github.com/apache/asterixdb/tree/master/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library</a>)</p></div></div></div>
+<div class="section">
+<h2><a name="Unstalling_an_AsterixDB_Library"></a><a name="installingUDF">Unstalling an AsterixDB Library</a></h2>
+<p>To uninstall a library, use the Managix uninstall command as follows:</p>
+
+<div>
+<div>
+<pre class="source"> $ managix stop -n my_asterix
+
+ $ managix uninstall -n my_asterix -d feeds -l testlib
</pre></div></div></div>
</div>
</div>