Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / d07e25e874f97f9031fe8fbbfd9fa003e085529c / . / docs / 0.9.4 / feeds.html
blob: 93086c18e28fae0ea23c2836d2e5dba956ccb0fc [file] [log] [blame]
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]1<!DOCTYPE html>
2<!--
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]3 | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/feeds.md at 2018-10-12
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]4 | Rendered using Apache Maven Fluido Skin 1.7
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]10 <meta name="Date-Revision-yyyymmdd" content="20181012" />
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]11 <meta http-equiv="Content-Language" content="en" />
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]12 <title>AsterixDB &#x2013; Data Ingestion with Feeds</title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16 <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]17
18 </head>
19 <body class="topBarDisabled">
20 <div class="container-fluid">
21 <div id="banner">
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]22 <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]23 <div class="pull-right"></div>
24 <div class="clear"><hr/></div>
25 </div>
26
27 <div id="breadcrumbs">
28 <ul class="breadcrumb">
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]29 <li id="publishDate">Last Published: 2018-10-12</li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]30 <li id="projectVersion" class="pull-right">Version: 0.9.4</li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]31 <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]32 </ul>
33 </div>
34 <div class="row-fluid">
35 <div id="leftColumn" class="span2">
36 <div class="well sidebar-nav">
37 <ul class="nav nav-list">
38 <li class="nav-header">Get Started - Installation</li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]39 <li><a href="ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
40 <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
41 <li><a href="aws.html" title="Option 3: using Amazon Web Services"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]42 <li class="nav-header">AsterixDB Primer</li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]43 <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]44 <li class="nav-header">Data Model</li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]45 <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
46 <li class="nav-header">Queries</li>
47 <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
48 <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]49 <li class="nav-header">API/SDK</li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]50 <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
51 <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]52 <li class="nav-header">Advanced Features</li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]53 <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
54 <li class="active"><a href="#"><span class="none"></span>Data Ingestion with Feeds</a></li>
55 <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
56 <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
57 <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
58 <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
59 <li class="nav-header">Deprecated</li>
60 <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
61 <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
62 <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]63</ul>
64 <hr />
65 <div id="poweredBy">
66 <div class="clear"></div>
67 <div class="clear"></div>
68 <div class="clear"></div>
69 <div class="clear"></div>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]70<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]71 </div>
72 </div>
73 </div>
74 <div id="bodyColumn" class="span10" >
75<!--
76 ! Licensed to the Apache Software Foundation (ASF) under one
77 ! or more contributor license agreements. See the NOTICE file
78 ! distributed with this work for additional information
79 ! regarding copyright ownership. The ASF licenses this file
80 ! to you under the Apache License, Version 2.0 (the
81 ! "License"); you may not use this file except in compliance
82 ! with the License. You may obtain a copy of the License at
83 !
84 ! http://www.apache.org/licenses/LICENSE-2.0
85 !
86 ! Unless required by applicable law or agreed to in writing,
87 ! software distributed under the License is distributed on an
88 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
89 ! KIND, either express or implied. See the License for the
90 ! specific language governing permissions and limitations
91 ! under the License.
92 !-->
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]93<h1>Data Ingestion with Feeds</h1>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]94<div class="section">
95<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
96<ul>
97
98<li><a href="#Introduction">Introduction</a></li>
99<li><a href="#FeedAdapters">Feed Adapters</a></li>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]100<li><a href="#FeedPolicies">Feed Policies</a><!--
101! Licensed to the Apache Software Foundation (ASF) under one
102! or more contributor license agreements. See the NOTICE file
103! distributed with this work for additional information
104! regarding copyright ownership. The ASF licenses this file
105! to you under the Apache License, Version 2.0 (the
106! "License"); you may not use this file except in compliance
107! with the License. You may obtain a copy of the License at
108!
109! http://www.apache.org/licenses/LICENSE-2.0
110!
111! Unless required by applicable law or agreed to in writing,
112! software distributed under the License is distributed on an
113! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
114! KIND, either express or implied. See the License for the
115! specific language governing permissions and limitations
116! under the License.
117!--></li>
118</ul></div>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]119<div class="section">
120<h2><a name="Introduction">Introduction</a></h2>
121<p>In this document, we describe the support for data ingestion in AsterixDB. Data feeds are a new mechanism for having continuous data arrive into a BDMS from external sources and incrementally populate a persisted dataset and associated indexes. We add a new BDMS architectural component, called a data feed, that makes a Big Data system the caretaker for functionality that used to live outside, and we show how it improves users&#x2019; lives and system performance.</p></div>
122<div class="section">
123<h2><a name="Feed_Adapters"></a><a name="FeedAdapters">Feed Adapters</a></h2>
124<p>The functionality of establishing a connection with a data source and receiving, parsing and translating its data into ADM objects (for storage inside AsterixDB) is contained in a feed adapter. A feed adapter is an implementation of an interface and its details are specific to a given data source. An adapter may optionally be given parameters to configure its runtime behavior. Depending upon the data transfer protocol/APIs offered by the data source, a feed adapter may operate in a push or a pull mode. Push mode involves just one initial request by the adapter to the data source for setting up the connection. Once a connection is authorized, the data source &#x201c;pushes&#x201d; data to the adapter without any subsequent requests by the adapter. In contrast, when operating in a pull mode, the adapter makes a separate request each time to receive data. AsterixDB currently provides built-in adapters for several popular data sources such as Twitter and RSS feeds. AsterixDB additionally provides a generic socket-based adapter that can be used to ingest data that is directed at a prescribed socket.</p>
125<p>In this tutorial, we shall describe building two example data ingestion pipelines that cover the popular scenarios of ingesting data from (a) Twitter (b) RSS (c) Socket Feed source.</p>
126<div class="section">
127<div class="section">
128<h4><a name="Ingesting_Twitter_Stream"></a>Ingesting Twitter Stream</h4>
129<p>We shall use the built-in push-based Twitter adapter. As a pre-requisite, we must define a Tweet using the AsterixDB Data Model (ADM) and the AsterixDB Query Language (AQL). Given below are the type definitions in AQL that create a Tweet datatype which is representative of a real tweet as obtained from Twitter.</p>
130
131<div>
132<div>
133<pre class="source"> create dataverse feeds;
134 use dataverse feeds;
135
136 create type TwitterUser as closed {
137 screen_name: string,
138 lang: string,
139 friends_count: int32,
140 statuses_count: int32
141 };
142
143 create type Tweet as open {
144 id: int64,
145 user: TwitterUser
146 }
147
148 create dataset Tweets (Tweet)
149 primary key id;
150</pre></div></div>
151
152<p>We also create a dataset that we shall use to persist the tweets in AsterixDB. Next we make use of the <tt>create feed</tt> AQL statement to define our example data feed.</p>
153<div class="section">
154<h5><a name="Using_the_.E2.80.9Cpush_twitter.E2.80.9D_feed_adapter"></a>Using the &#x201c;push_twitter&#x201d; feed adapter</h5>
155<p>The &#x201c;push_twitter&#x201d; adapter requires setting up an application account with Twitter. To retrieve tweets, Twitter requires registering an application. Registration involves providing a name and a brief description for the application. Each application has associated OAuth authentication credentials that include OAuth keys and tokens. Accessing the Twitter API requires providing the following. 1. Consumer Key (API Key) 2. Consumer Secret (API Secret) 3. Access Token 4. Access Token Secret</p>
Ian Maxonb2f1d3e2018-10-12 14:42:34 -0700[diff] [blame]156<p>The &#x201c;push_twitter&#x201d; adapter takes as configuration the above mentioned parameters. End users are required to obtain the above authentication credentials prior to using the &#x201c;push_twitter&#x201d; adapter. For further information on obtaining OAuth keys and tokens and registering an application with Twitter, please visit <a class="externalLink" href="http://apps.twitter.com">http://apps.twitter.com</a>.</p>
157<p>Note that AsterixDB uses the Twitter4J API for getting data from Twitter. Due to a license conflict, Apache AsterixDB cannot ship the Twitter4J library. To use the Twitter adapter in AsterixDB, please download the necessary dependencies (<tt>twitter4j-core-4.0.x.jar</tt> and <tt>twitter4j-stream-4.0.x.jar</tt>) and drop them into the <tt>repo/</tt> directory before AsterixDB starts.</p>
Ian Maxond00eca82018-10-05 17:29:55 -0700[diff] [blame]158<p>Given below is an example AQL statement that creates a feed called &#x201c;TwitterFeed&#x201d; by using the &#x201c;push_twitter&#x201d; adapter.</p>
159
160<div>
161<div>
162<pre class="source"> use dataverse feeds;
163
164 create feed TwitterFeed if not exists using &quot;push_twitter&quot;
165 ((&quot;type-name&quot;=&quot;Tweet&quot;),
166 (&quot;format&quot;=&quot;twitter-status&quot;),
167 (&quot;consumer.key&quot;=&quot;************&quot;),
168 (&quot;consumer.secret&quot;=&quot;**************&quot;),
169 (&quot;access.token&quot;=&quot;**********&quot;),
170 (&quot;access.token.secret&quot;=&quot;*************&quot;));
171</pre></div></div>
172
173<p>It is required that the above authentication parameters are provided valid. Note that the <tt>create feed</tt> statement does not initiate the flow of data from Twitter into the AsterixDB instance. Instead, the <tt>create feed</tt> statement only results in registering the feed with the instance. The flow of data along a feed is initiated when it is connected to a target dataset using the connect feed statement and activated using the start feed statement.</p>
174<p>The Twitter adapter also supports several Twitter streaming APIs as follow:</p>
175<ol style="list-style-type: decimal">
176
177<li>Track filter (&#x201c;keywords&#x201d;=&#x201c;AsterixDB, Apache&#x201d;)</li>
178<li>Locations filter (&#x201c;locations&#x201d;=&#x201c;-29.7, 79.2, 36.7, 72.0; -124.848974,-66.885444, 24.396308, 49.384358&#x201d;)</li>
179<li>Language filter (&#x201c;language&#x201d;=&#x201c;en&#x201d;)</li>
180<li>Filter level (&#x201c;filter-level&#x201d;=&#x201c;low&#x201d;)</li>
181</ol>
182<p>An example of Twitter adapter tracking tweets with keyword &#x201c;news&#x201d; can be described using following ddl:</p>
183
184<div>
185<div>
186<pre class="source"> use dataverse feeds;
187
188 create feed TwitterFeed if not exists using &quot;push_twitter&quot;
189 ((&quot;type-name&quot;=&quot;Tweet&quot;),
190 (&quot;format&quot;=&quot;twitter-status&quot;),
191 (&quot;consumer.key&quot;=&quot;************&quot;),
192 (&quot;consumer.secret&quot;=&quot;**************&quot;),
193 (&quot;access.token&quot;=&quot;**********&quot;),
194 (&quot;access.token.secret&quot;=&quot;*************&quot;),
195 (&quot;keywords&quot;=&quot;news&quot;));
196</pre></div></div>
197
198<p>For more details about these APIs, please visit <a class="externalLink" href="https://dev.twitter.com/streaming/overview/request-parameters">https://dev.twitter.com/streaming/overview/request-parameters</a></p></div></div>
199<div class="section">
200<h4><a name="Lifecycle_of_a_Feed"></a>Lifecycle of a Feed</h4>
201<p>A feed is a logical artifact that is brought to life (i.e., its data flow is initiated) only when it is activated using the <tt>start feed</tt> statement. Before we active a feed, we need to designate the dataset where the data to be persisted using <tt>connect feed</tt> statement. Subsequent to a <tt>connect feed</tt> statement, the feed is said to be in the connected state. After that, <tt>start feed</tt> statement will activate the feed, and start the dataflow from feed to its connected dataset. Multiple feeds can simultaneously be connected to a dataset such that the contents of the dataset represent the union of the connected feeds. Also one feed can be simultaneously connected to multiple target datasets.</p>
202
203<div>
204<div>
205<pre class="source"> use dataverse feeds;
206
207 connect feed TwitterFeed to dataset Tweets;
208
209 start feed TwitterFeed;
210</pre></div></div>
211
212<p>The <tt>connect feed</tt> statement above directs AsterixDB to persist the data from <tt>TwitterFeed</tt> feed into the <tt>Tweets</tt> dataset. The <tt>start feed</tt> statement will activate the feed and start the dataflow. If it is required (by the high-level application) to also retain the raw tweets obtained from Twitter, the end user may additionally choose to connect TwitterFeed to a different dataset.</p>
213<p>Let the feed run for a minute, then run the following query to see the latest tweets that are stored into the data set.</p>
214
215<div>
216<div>
217<pre class="source"> use dataverse feeds;
218
219 for $i in dataset Tweets limit 10 return $i;
220</pre></div></div>
221
222<p>The dataflow of data from a feed can be terminated explicitly by <tt>stop feed</tt> statement.</p>
223
224<div>
225<div>
226<pre class="source"> use dataverse feeds;
227
228 stop feed TwitterFeed;
229</pre></div></div>
230
231<p>The <tt>disconnnect statement</tt> can be used to disconnect the feed from certain dataset.</p>
232
233<div>
234<div>
235<pre class="source"> use dataverse feeds;
236
237 disconnect feed TwitterFeed from dataset Tweets;
238</pre></div></div>
239</div></div>
240<div class="section">
241<h3><a name="Ingesting_with_Other_Adapters"></a>Ingesting with Other Adapters</h3>
242<p>AsterixDB has several builtin feed adapters for data ingestion. User can also implement their own adapters and plug them into AsterixDB. Here we introduce <tt>rss_feed</tt>, <tt>socket_adapter</tt> and <tt>localfs</tt> feed adapter that cover most of the common application scenarios.</p>
243<div class="section">
244<div class="section">
245<h5><a name="Using_the_.E2.80.9Crss_feed.E2.80.9D_feed_adapter"></a>Using the &#x201c;rss_feed&#x201d; feed adapter</h5>
246<p><tt>rss_feed</tt> adapter allows retrieving data given a collection of RSS end point URLs. As observed in the case of ingesting tweets, it is required to model an RSS data item using AQL.</p>
247
248<div>
249<div>
250<pre class="source"> use dataverse feeds;
251
252 create type Rss if not exists as open {
253 id: string,
254 title: string,
255 description: string,
256 link: string
257 };
258
259 create dataset RssDataset (Rss)
260 primary key id;
261</pre></div></div>
262
263<p>Next, we define an RSS feed using our built-in adapter &#x201c;rss_feed&#x201d;.</p>
264
265<div>
266<div>
267<pre class="source"> use dataverse feeds;
268
269 create feed my_feed using
270 rss_feed (
271 (&quot;type-name&quot;=&quot;Rss&quot;),
272 (&quot;format&quot;=&quot;rss&quot;),
273 (&quot;url&quot;=&quot;http://rss.cnn.com/rss/edition.rss&quot;)
274 );
275</pre></div></div>
276
277<p>In the above definition, the configuration parameter &#x201c;url&#x201d; can be a comma-separated list that reflects a collection of RSS URLs, where each URL corresponds to an RSS endpoint or an RSS feed. The &#x201c;rss_feed&#x201d; retrieves data from each of the specified RSS URLs (comma separated values) in parallel.</p>
278<p>The following statements connect the feed into the <tt>RssDataset</tt>:</p>
279
280<div>
281<div>
282<pre class="source"> use dataverse feeds;
283
284 connect feed my_feed to dataset RssDataset;
285</pre></div></div>
286
287<p>The following statements activate the feed and start the dataflow:</p>
288
289<div>
290<div>
291<pre class="source"> use dataverse feeds;
292
293 start feed my_feed;
294</pre></div></div>
295
296<p>The following statements show the latest data from the data set, stop the feed, and disconnect the feed from the data set.</p>
297
298<div>
299<div>
300<pre class="source"> use dataverse feeds;
301
302 for $i in dataset RssDataset limit 10 return $i;
303
304 stop feed my_feed
305
306 disconnect feed my_feed from dataset RssDataset;
307</pre></div></div>
308</div>
309<div class="section">
310<h5><a name="Using_the_.E2.80.9Csocket_adapter.E2.80.9D_feed_adapter"></a>Using the &#x201c;socket_adapter&#x201d; feed adapter</h5>
311<p><tt>socket_adapter</tt> feed opens a web socket on the given node which allows user to push data into AsterixDB directly. Here is an example:</p>
312
313<div>
314<div>
315<pre class="source"> drop dataverse feeds if exists;
316 create dataverse feeds;
317 use dataverse feeds;
318
319 create type TestDataType as open {
320 screenName: string
321 }
322
323 create dataset TestDataset(TestDataType) primary key screenName;
324
325 create feed TestSocketFeed using socket_adapter
326 (
327 (&quot;sockets&quot;=&quot;127.0.0.1:10001&quot;),
328 (&quot;address-type&quot;=&quot;IP&quot;),
329 (&quot;type-name&quot;=&quot;TestDataType&quot;),
330 (&quot;format&quot;=&quot;adm&quot;)
331 );
332
333 connect feed TestSocketFeed to dataset TestDataset;
334
335 use dataverse feeds;
336 start feed TestSocketFeed;
337</pre></div></div>
338
339<p>The above statements create a socket feed which is listening to &#x201c;10001&#x201d; port of the host machine. This feed accepts data records in &#x201c;adm&#x201d; format. As an example, you can download the sample dataset <a href="../data/chu.adm">Chirp Users</a> and push them line by line into the socket feed using any socket client you like. Following is a socket client example in Python:</p>
340
341<div>
342<div>
343<pre class="source"> from socket import socket
344
345 ip = '127.0.0.1'
346 port1 = 10001
347 filePath = 'chu.adm'
348
349 sock1 = socket()
350 sock1.connect((ip, port1))
351
352 with open(filePath) as inputData:
353 for line in inputData:
354 sock1.sendall(line)
355 sock1.close()
356</pre></div></div>
357</div></div>
358<div class="section">
359<h4><a name="Using_the_.E2.80.9Clocalfs.E2.80.9D_feed_adapter"></a>Using the &#x201c;localfs&#x201d; feed adapter</h4>
360<p><tt>localfs</tt> adapter enables data ingestion from local file system. It allows user to feed data records on local disk into a dataset. A DDL example for creating a <tt>localfs</tt> feed is given as follow:</p>
361
362<div>
363<div>
364<pre class="source"> use dataverse feeds;
365
366 create type TweetType as closed {
367 id: string,
368 username : string,
369 location : string,
370 text : string,
371 timestamp : string
372 }
373
374 create dataset Tweets(TweetType)
375 primary key id;
376
377 create feed TweetFeed
378 using localfs
379 ((&quot;type-name&quot;=&quot;TweetType&quot;),(&quot;path&quot;=&quot;HOSTNAME://LOCAL_FILE_PATH&quot;),(&quot;format&quot;=&quot;adm&quot;))
380</pre></div></div>
381
382<p>Similar to previous examples, we need to define the datatype and dataset this feed uses. The &#x201c;path&#x201d; parameter refers to the local datafile that we want to ingest data from. <tt>HOSTNAME</tt> can either be the IP address or node name of the machine which holds the file. <tt>LOCAL_FILE_PATH</tt> indicates the absolute path to the file on that machine. Similarly to <tt>socket_adapter</tt>, this feed takes <tt>adm</tt> formatted data records.</p></div></div>
383<div class="section">
384<h3><a name="Datatype_for_feed_and_target_dataset"></a>Datatype for feed and target dataset</h3>
385<p>The &#x201c;type-name&#x201d; parameter in create feed statement defines the <tt>datatype</tt> of the datasource. In most use cases, feed will have the same <tt>datatype</tt> as the target dataset. However, if we want to perform certain preprocess before the data records gets into the target dataset (append autogenerated key, apply user defined functions, etc.), we will need to define the datatypes for feed and dataset separately.</p>
386<div class="section">
387<h4><a name="Ingestion_with_autogenerated_key"></a>Ingestion with autogenerated key</h4>
388<p>AsterixDB supports using autogenerated uuid as the primary key for dataset. When we use this feature, we will need to define a datatype with the primary key field, and specify that field to be autogenerated when creating the dataset. Use that same datatype in feed definition will cause a type discrepancy since there is no such field in the datasource. Thus, we will need to define two separate datatypes for feed and dataset:</p>
389
390<div>
391<div>
392<pre class="source"> use dataverse feeds;
393
394 create type DBLPFeedType as closed {
395 dblpid: string,
396 title: string,
397 authors: string,
398 misc: string
399 }
400
401 create type DBLPDataSetType as open {
402 id: uuid,
403 dblpid: string,
404 title: string,
405 authors: string,
406 misc: string
407 }
408 create dataset DBLPDataset(DBLPDataSetType) primary key id autogenerated;
409
410 create feed DBLPFeed using socket_adapter
411 (
412 (&quot;sockets&quot;=&quot;127.0.0.1:10001&quot;),
413 (&quot;address-type&quot;=&quot;IP&quot;),
414 (&quot;type-name&quot;=&quot;DBLPFeedType&quot;),
415 (&quot;format&quot;=&quot;adm&quot;)
416 );
417
418 connect feed DBLPFeed to dataset DBLPDataset;
419
420 start feed DBLPFeed;
421</pre></div></div>
422</div></div></div>
423<div class="section">
424<h2><a name="Policies_for_Feed_Ingestion"></a><a name="FeedPolicies">Policies for Feed Ingestion</a></h2>
425<p>Multiple feeds may be concurrently operational on an AsterixDB cluster, each competing for resources (CPU cycles, network bandwidth, disk IO) to maintain pace with their respective data sources. As a data management system, AsterixDB is able to manage a set of concurrent feeds and make dynamic decisions related to the allocation of resources, resolving resource bottlenecks and the handling of failures. Each feed has its own set of constraints, influenced largely by the nature of its data source and the applications that intend to consume and process the ingested data. Consider an application that intends to discover the trending topics on Twitter by analyzing tweets that are being processed. Losing a few tweets may be acceptable. In contrast, when ingesting from a data source that provides a click-stream of ad clicks, losing data would translate to a loss of revenue for an application that tracks revenue by charging advertisers per click.</p>
426<p>AsterixDB allows a data feed to have an associated ingestion policy that is expressed as a collection of parameters and associated values. An ingestion policy dictates the runtime behavior of the feed in response to resource bottlenecks and failures. AsterixDB provides a set of policies that help customize the system&#x2019;s runtime behavior when handling excess objects.</p>
427<div class="section">
428<div class="section">
429<h4><a name="Policies"></a>Policies</h4>
430<ul>
431
432<li>
433
434<p><i>Spill</i>: Objects that cannot be processed by an operator for lack of resources (referred to as excess objects hereafter) should be persisted to the local disk for deferred processing.</p>
435</li>
436<li>
437
438<p><i>Discard</i>: Excess objects should be discarded.</p>
439</li>
440</ul>
441<p>Note that the end user may choose to form a custom policy. For example, it is possible in AsterixDB to create a custom policy that spills excess objects to disk and subsequently resorts to throttling if the spillage crosses a configured threshold. In all cases, the desired ingestion policy is specified as part of the <tt>connect feed</tt> statement or else the &#x201c;Basic&#x201d; policy will be chosen as the default.</p>
442
443<div>
444<div>
445<pre class="source"> use dataverse feeds;
446
447 connect feed TwitterFeed to dataset Tweets
448 using policy Basic;
449</pre></div></div></div></div></div>
450 </div>
451 </div>
452 </div>
453 <hr/>
454 <footer>
455 <div class="container-fluid">
456 <div class="row-fluid">
457<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
458 feather logo, and the Apache AsterixDB project logo are either
459 registered trademarks or trademarks of The Apache Software
460 Foundation in the United States and other countries.
461 All other marks mentioned may be trademarks or registered
462 trademarks of their respective owners.
463 </div>
464 </div>
465 </div>
466 </footer>
467 </body>
468</html>