Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / a1cc51bd9021157f09c3328554de0b7eed5a026d / . / content / docs / 0.9.3 / ncservice.html
blob: 8eb3b823264e942f4b68600bd178be4be644c37d [file] [log] [blame]
Ian Maxon9c40a662018-02-09 12:42:56 -0800[diff] [blame]1<!DOCTYPE html>
2<!--
3 | Generated by Apache Maven Doxia at 2018-02-09
4 | Rendered using Apache Maven Fluido Skin 1.3.0
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
10 <meta name="Date-Revision-yyyymmdd" content="20180209" />
11 <meta http-equiv="Content-Language" content="en" />
12 <title>AsterixDB &#x2013; <a id="toc">Table of Contents</a></title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16
17
18 <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
19
20
21
22<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
23 (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
24 m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
25 })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
26
27 ga('create', 'UA-41536543-1', 'uci.edu');
28 ga('send', 'pageview');</script>
29
30 </head>
31 <body class="topBarDisabled">
32
33
34
35
36 <div class="container-fluid">
37 <div id="banner">
38 <div class="pull-left">
39 <a href="./" id="bannerLeft">
40 <img src="images/asterixlogo.png" alt="AsterixDB"/>
41 </a>
42 </div>
43 <div class="pull-right"> </div>
44 <div class="clear"><hr/></div>
45 </div>
46
47 <div id="breadcrumbs">
48 <ul class="breadcrumb">
49
50
51 <li id="publishDate">Last Published: 2018-02-09</li>
52
53
54
55 <li id="projectVersion" class="pull-right">Version: 0.9.3</li>
56
57 <li class="divider pull-right">|</li>
58
59 <li class="pull-right"> <a href="index.html" title="Documentation Home">
60 Documentation Home</a>
61 </li>
62
63 </ul>
64 </div>
65
66
67 <div class="row-fluid">
68 <div id="leftColumn" class="span3">
69 <div class="well sidebar-nav">
70
71
72 <ul class="nav nav-list">
73 <li class="nav-header">Get Started - Installation</li>
74
75 <li class="active">
76
77 <a href="#"><i class="none"></i>Option 1: using NCService</a>
78 </li>
79
80 <li>
81
82 <a href="ansible.html" title="Option 2: using Ansible">
83 <i class="none"></i>
84 Option 2: using Ansible</a>
85 </li>
86
87 <li>
88
89 <a href="aws.html" title="Option 3: using Amazon Web Services">
90 <i class="none"></i>
91 Option 3: using Amazon Web Services</a>
92 </li>
93
94 <li>
95
96 <a href="yarn.html" title="Option 4: using YARN">
97 <i class="none"></i>
98 Option 4: using YARN</a>
99 </li>
100
101 <li>
102
103 <a href="install.html" title="Option 5: using Managix (deprecated)">
104 <i class="none"></i>
105 Option 5: using Managix (deprecated)</a>
106 </li>
107 <li class="nav-header">AsterixDB Primer</li>
108
109 <li>
110
111 <a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++">
112 <i class="none"></i>
113 Option 1: using SQL++</a>
114 </li>
115
116 <li>
117
118 <a href="aql/primer.html" title="Option 2: using AQL">
119 <i class="none"></i>
120 Option 2: using AQL</a>
121 </li>
122 <li class="nav-header">Data Model</li>
123
124 <li>
125
126 <a href="datamodel.html" title="The Asterix Data Model">
127 <i class="none"></i>
128 The Asterix Data Model</a>
129 </li>
130 <li class="nav-header">Queries - SQL++</li>
131
132 <li>
133
134 <a href="sqlpp/manual.html" title="The SQL++ Query Language">
135 <i class="none"></i>
136 The SQL++ Query Language</a>
137 </li>
138
139 <li>
140
141 <a href="sqlpp/builtins.html" title="Builtin Functions">
142 <i class="none"></i>
143 Builtin Functions</a>
144 </li>
145 <li class="nav-header">Queries - AQL</li>
146
147 <li>
148
149 <a href="aql/manual.html" title="The Asterix Query Language (AQL)">
150 <i class="none"></i>
151 The Asterix Query Language (AQL)</a>
152 </li>
153
154 <li>
155
156 <a href="aql/builtins.html" title="Builtin Functions">
157 <i class="none"></i>
158 Builtin Functions</a>
159 </li>
160 <li class="nav-header">API/SDK</li>
161
162 <li>
163
164 <a href="api.html" title="HTTP API">
165 <i class="none"></i>
166 HTTP API</a>
167 </li>
168
169 <li>
170
171 <a href="csv.html" title="CSV Output">
172 <i class="none"></i>
173 CSV Output</a>
174 </li>
175 <li class="nav-header">Advanced Features</li>
176
177 <li>
178
179 <a href="aql/fulltext.html" title="Support of Full-text Queries">
180 <i class="none"></i>
181 Support of Full-text Queries</a>
182 </li>
183
184 <li>
185
186 <a href="aql/externaldata.html" title="Accessing External Data">
187 <i class="none"></i>
188 Accessing External Data</a>
189 </li>
190
191 <li>
192
193 <a href="feeds/tutorial.html" title="Support for Data Ingestion">
194 <i class="none"></i>
195 Support for Data Ingestion</a>
196 </li>
197
198 <li>
199
200 <a href="udf.html" title="User Defined Functions">
201 <i class="none"></i>
202 User Defined Functions</a>
203 </li>
204
205 <li>
206
207 <a href="aql/filters.html" title="Filter-Based LSM Index Acceleration">
208 <i class="none"></i>
209 Filter-Based LSM Index Acceleration</a>
210 </li>
211
212 <li>
213
214 <a href="aql/similarity.html" title="Support of Similarity Queries">
215 <i class="none"></i>
216 Support of Similarity Queries</a>
217 </li>
218 </ul>
219
220
221
222 <hr class="divider" />
223
224 <div id="poweredBy">
225 <div class="clear"></div>
226 <div class="clear"></div>
227 <div class="clear"></div>
228 <a href="./" title="AsterixDB" class="builtBy">
229 <img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" />
230 </a>
231 </div>
232 </div>
233 </div>
234
235
236 <div id="bodyColumn" class="span9" >
237
238 <!-- ! Licensed to the Apache Software Foundation (ASF) under one
239 ! or more contributor license agreements. See the NOTICE file
240 ! distributed with this work for additional information
241 ! regarding copyright ownership. The ASF licenses this file
242 ! to you under the Apache License, Version 2.0 (the
243 ! "License"); you may not use this file except in compliance
244 ! with the License. You may obtain a copy of the License at
245 !
246 ! http://www.apache.org/licenses/LICENSE-2.0
247 !
248 ! Unless required by applicable law or agreed to in writing,
249 ! software distributed under the License is distributed on an
250 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
251 ! KIND, either express or implied. See the License for the
252 ! specific language governing permissions and limitations
253 ! under the License.
254 ! --><div class="section">
255<h2><a name="Table_of_Contents"></a><a name="toc" id="toc">Table of Contents</a></h2>
256
257<ul>
258
259<li><a href="#quickstart">Quick Start</a></li>
260
261<li><a href="#Small_cluster">Starting a small single-machine cluster using the NCService</a></li>
262
263<li><a href="#Multi_machine">Deploying AsterixDB via NCService in a multi-machine setup</a></li>
264
265<li><a href="#Parameters">Available Configuration Parameters</a></li>
266</ul>
267<h1><a name="quickstart" id="quickstart">Quick Start</a></h1>
268<p>The fastest way to get set up with a single-machine sample instance of AsterixDB is to use the included sample helper scripts. To do so, in the extracted <tt>asterix-server</tt> directory, navigate to <tt>opt/local/bin/</tt></p>
269
270<div class="source">
271<div class="source">
272<pre>user@localhost:~/
273$cd asterix-server/
274user@localhost:~/asterix-server
275$cd opt/local/bin
276</pre></div></div>
277<p>This folder should contain 4 scripts, two pairs of <tt>.sh</tt> and <tt>.bat</tt> files respectively. <tt>start-sample-cluster.sh</tt> will simply start a basic sample cluster using the configuration files located in <tt>opt/local/conf/</tt>.</p>
278
279<div class="source">
280<div class="source">
281<pre>user@localhost:~/a/o/l/bin
282$./start-sample-cluster.sh
283CLUSTERDIR=/home/user/asterix-server/opt/local
284INSTALLDIR=/home/user/asterix-server
285LOGSDIR=/home/user/asterix-server/samples/opt/logs
286
287INFO: Starting sample cluster...
288INFO: Waiting up to 30 seconds for cluster 127.0.0.1:19002 to be available.
289INFO: Cluster started and is ACTIVE.
290user@localhost:~/a/o/l/bin
291$
292</pre></div></div>
293<p>Now, there should be a running AsterixDB cluster on the machine. To go to the Web Interface, visit <a class="externalLink" href="http://localhost:19001">http://localhost:19001</a></p>
294
295<div class="source">
296
297<div class="source">
298<pre>
299<img src="images/asterixdb_interface.png" alt="The AsterixDB Web Interface" />
300<i>Fig. 1</i>: The AsterixDB Web Interface
301</pre></div>
302</div>
303<h1><a name="Small_cluster" id="Small_cluster">Starting a small single-machine cluster using NCService</a></h1>
304<p>The above cluster was started using a script, but below is a description in detail of how precisely this was achieved. The config files here are analagous to the ones within <tt>samples/local/conf</tt>.</p>
305<p>When running a cluster using the <tt>NCService</tt> there are 3 different kinds of processes involved:</p>
306
307<ul>
308
309<li><tt>NCDriver</tt>, also known as the Node Controller or NC for short. This is the process that does the actual work of queries and data management within the AsterixDB cluster</li>
310
311<li><tt>NCService</tt> configures and starts the <tt>NCDriver</tt> process. It is a simple daemon whose only purpose is to wait for the <tt>CCDriver</tt> process to call upon it to initiate cluster bootup.</li>
312
313<li><tt>CCDriver</tt>, which is the Cluster Controller process, also known as the CC. This process manages the distribution of tasks to all NCs, as well as providing the parameters of each NC to the NCService upon cluster startup. It also hosts the Web interface and query compiler and optimizer.</li>
314</ul>
315<p>The cluster startup follows a particular sequence, which is as follows:</p>
316
317<ol style="list-style-type: decimal">
318
319<li>Each host on which an NC is desired and is mentioned in the configuration, the <tt>NCService</tt> daemon is started first. It will listen and wait for the CC to contact it.</li>
320
321<li>The one host on which the CC is to be placed is started with an appropriate configuration file.</li>
322
323<li>The CC contacts all <tt>NCService</tt> daemons and the <tt>NCService</tt> subsequently starts and <tt>NCDriver</tt> process with the configration supplied by the <tt>CC</tt></li>
324
325<li>Each <tt>NCDriver</tt> then contacts the CC to register itself as started</li>
326</ol>
327<p>This process is briefly illustrated in the diagram below:</p>
328
329<div class="source">
330
331<div class="source">
332<pre>
333<img src="images/ncservice.png" alt="The AsterixDB Web Interface" />
334<i>Fig. 2</i>: NCService startup sequence
335</pre></div>
336</div>
337<p>To start a small cluster consisting of 2 NodeControllers (<tt>red</tt> and <tt>blue</tt>) and 1 ClusterController (<tt>cc</tt>) on a single machine only 2 configuration files are required. The first one is</p>
338<p><tt>blue.conf</tt>:</p>
339
340<div class="source">
341<div class="source">
342<pre>[ncservice]
343port=9091
344</pre></div></div>
345<p>It is a configuration file for the second <tt>NCService</tt>. This contains only the port that the <tt>NCService</tt> of the second NodeControllers listens to as it is non-standard. The first <tt>NCService</tt> does not need a configuration file, as it only uses default parameters. In a distributed environment with 1 NodeController per machine, no <tt>NCService</tt> needs a configuration file.</p>
346<p>The second configuration file is</p>
347<p><tt>cc.conf</tt>:</p>
348
349<div class="source">
350<div class="source">
351<pre>[nc/red]
352txn.log.dir=/tmp/asterix/red/txnlog
353core.dump.dir=/tmp/asterix/red/coredump
354iodevices=/tmp/asterix/red
355
356[nc/blue]
357port=9091
358txn.log.dir=/tmp/asterix/blue/txnlog
359core.dump.dir=/tmp/asterix/blue/coredump
360iodevices=/tmp/asterix/blue
361
362[nc]
363app.class=org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint
364storage.subdir=storage
365address=127.0.0.1
366command=asterixnc
367
368[cc]
369address = 127.0.0.1
370console.listen.port = 12345
371</pre></div></div>
372<p>This is the configuration file for the cluster and it contains information that each <tt>NCService</tt> will use when starting the corresponding <tt>NCDriver</tt> as well as information for the <tt>CCDriver</tt>.</p>
373<p>To start the cluster simply use the following steps</p>
374
375<ol style="list-style-type: decimal">
376
377<li>
378<p>Change directory into the asterix-server binary folder</p>
379
380<div class="source">
381<div class="source">
382<pre>user@localhost:~/
383$cd asterix-server/
384user@localhost:~/asterix-server
385$cd samples/local/bin
386</pre></div></div></li>
387
388<li>
389<p>Start the 2 <tt>NCServices</tt> for <tt>red</tt> and <tt>blue</tt>.</p>
390
391<div class="source">
392<div class="source">
393<pre>user@localhost:~/asterix-server
394$bin/asterixncservice -config-file blue.conf &gt; blue-service.log 2&gt;&amp;1 &amp;
395user@localhost:~/asterix-server
396$bin/asterixncservice &gt;red-service.log 2&gt;&amp;1 &amp;
397</pre></div></div></li>
398
399<li>
400<p>Start the <tt>CCDriver</tt>.</p>
401
402<div class="source">
403<div class="source">
404<pre>user@localhost:~/asterix-server
405$bin/asterixcc -config-file cc.conf &gt; cc.log 2&gt;&amp;1 &amp;
406</pre></div></div></li>
407</ol>
408<p>The <tt>CCDriver</tt> will connect to the <tt>NCServices</tt> and thus initiate the configuration and the start of the <tt>NCDrivers</tt>. After running these scripts, <tt>jps</tt> should show a result similar to this:</p>
409
410<div class="source">
411<div class="source">
412<pre>user@localhost:~/asterix-server
413$jps
41413184 NCService
41513200 NCDriver
41613185 NCService
41713186 CCDriver
41813533 Jps
41913198 NCDriver
420</pre></div></div>
421<p>The logs for the <tt>NCDrivers</tt> will be in <tt>$BASEDIR/logs</tt>.</p>
422<p>To stop the cluster again simply run</p>
423
424<div class="source">
425<div class="source">
426<pre>$ kill `jps | egrep '(CDriver|NCService)' | awk '{print $1}'`
427</pre></div></div>
428<p>to kill all processes.</p>
429<h1><a name="Multi_machine" id="Multi_machine">Deploying AsterixDB via NCService in a multi-machine setup</a></h1>
430<p>Deploying on multiple machines only differs in the configuration file and where each process is actually resident. Take for example a deployment on 3 machines, <tt>cacofonix-1</tt>,<tt>cacofonix-2</tt>,and <tt>cacofonix-3</tt>. <tt>cacofonix-1</tt> will be the CC, and <tt>cacofonix-2</tt> and <tt>cacofonix-3</tt> will be the two NCs, respectively. The configuration would be as follows:</p>
431<p><tt>cc.conf</tt>:</p>
432
433<div class="source">
434<div class="source">
435<pre>[nc/red]
436txn.log.dir=/lv_scratch/asterix/red/txnlog
437core.dump.dir=/lv_scratch/asterix/red/coredump
438iodevices=/lv_scratch/asterix/red
439address=cacofonix-2
440
441[nc/blue]
442txn.log.dir=/lv_scratch/asterix/blue/txnlog
443core.dump.dir=/lv_scratch/asterix/blue/coredump
444iodevices=/lv_scratch/asterix/blue
445address=cacofonix-3
446
447[nc]
448app.class=org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint
449storagedir=storage
450command=asterixnc
451
452[cc]
453address = cacofonix-1
454</pre></div></div>
455<p>To deploy, first the <tt>asterix-server</tt> binary must be present on each machine. Any method to transfer the archive to each machine will work, but here <tt>scp</tt> will be used for simplicity&#x2019;s sake.</p>
456
457<div class="source">
458<div class="source">
459<pre>user@localhost:~
460$for f in {1,2,3}; scp asterix-server.zip cacofonix-$f:~/; end
461</pre></div></div>
462<p>Then unzip the binary on each machine. First, start the <tt>NCService</tt> processes on each of the slave machines. Any way of getting a shell on the machine is fine, be it physical or via <tt>ssh</tt>.</p>
463
464<div class="source">
465<div class="source">
466<pre>user@cacofonix-2 12:41:42 ~/asterix-server/
467$ bin/asterixncservice &gt; red-service.log 2&gt;&amp;1 &amp;
468
469
470user@cacofonix-3 12:41:42 ~/asterix-server/
471$ bin/asterixncservice &gt; blue-service.log 2&gt;&amp;1 &amp;
472</pre></div></div>
473<p>Now that each <tt>NCService</tt> is waiting, the CC can be started.</p>
474
475<div class="source">
476<div class="source">
477<pre>user@cacofonix-1 12:41:42 ~/asterix-server/
478$ bin/asterixcc -config-file cc.conf &gt; cc.log 2&gt;&amp;1 &amp;
479</pre></div></div>
480<p>The cluster should now be started and the Web UI available on the CC host at the default port.</p>
481<h1><a name="Parameters" id="Parameters">Available Configuration Parameters</a></h1>
482<p>The following parameters are for the master process, under the &#x201c;[cc]&#x201d; section.</p>
483
484<table border="0" class="table table-striped">
485 <thead>
486
487<tr class="a">
488
489<th>Section </th>
490
491<th>Parameter </th>
492
493<th>Meaning </th>
494
495<th>Default </th>
496 </tr>
497 </thead>
498 <tbody>
499
500<tr class="b">
501
502<td>cc </td>
503
504<td>active.port </td>
505
506<td>The listen port of the active server </td>
507
508<td>19003 </td>
509 </tr>
510
511<tr class="a">
512
513<td>cc </td>
514
515<td>address </td>
516
517<td>Default bind address for all services on this cluster controller </td>
518
519<td>127.0.0.1 </td>
520 </tr>
521
522<tr class="b">
523
524<td>cc </td>
525
526<td>api.port </td>
527
528<td>The listen port of the API server </td>
529
530<td>19002 </td>
531 </tr>
532
533<tr class="a">
534
535<td>cc </td>
536
537<td>app.class </td>
538
539<td>Application CC main class </td>
540
541<td>org.apache.asterix.hyracks.bootstrap.CCApplication </td>
542 </tr>
543
544<tr class="b">
545
546<td>cc </td>
547
548<td>client.listen.address </td>
549
550<td>Sets the IP Address to listen for connections from clients </td>
551
552<td>same as address </td>
553 </tr>
554
555<tr class="a">
556
557<td>cc </td>
558
559<td>client.listen.port </td>
560
561<td>Sets the port to listen for connections from clients </td>
562
563<td>1098 </td>
564 </tr>
565
566<tr class="b">
567
568<td>cc </td>
569
570<td>cluster.listen.address </td>
571
572<td>Sets the IP Address to listen for connections from NCs </td>
573
574<td>same as address </td>
575 </tr>
576
577<tr class="a">
578
579<td>cc </td>
580
581<td>cluster.listen.port </td>
582
583<td>Sets the port to listen for connections from node controllers </td>
584
585<td>1099 </td>
586 </tr>
587
588<tr class="b">
589
590<td>cc </td>
591
592<td>cluster.public.address </td>
593
594<td>Address that NCs should use to contact this CC </td>
595
596<td>same as cluster.listen.address </td>
597 </tr>
598
599<tr class="a">
600
601<td>cc </td>
602
603<td>cluster.public.port </td>
604
605<td>Port that NCs should use to contact this CC </td>
606
607<td>same as cluster.listen.port </td>
608 </tr>
609
610<tr class="b">
611
612<td>cc </td>
613
614<td>cluster.topology </td>
615
616<td>Sets the XML file that defines the cluster topology </td>
617
618<td>&lt;undefined&gt; </td>
619 </tr>
620
621<tr class="a">
622
623<td>cc </td>
624
625<td>console.listen.address </td>
626
627<td>Sets the listen address for the Cluster Controller </td>
628
629<td>same as address </td>
630 </tr>
631
632<tr class="b">
633
634<td>cc </td>
635
636<td>console.listen.port </td>
637
638<td>Sets the http port for the Cluster Controller) </td>
639
640<td>16001 </td>
641 </tr>
642
643<tr class="a">
644
645<td>cc </td>
646
647<td>heartbeat.max.misses </td>
648
649<td>Sets the maximum number of missed heartbeats before a node is marked as dead </td>
650
651<td>5 </td>
652 </tr>
653
654<tr class="b">
655
656<td>cc </td>
657
658<td>heartbeat.period </td>
659
660<td>Sets the time duration between two heartbeats from each node controller in milliseconds </td>
661
662<td>10000 </td>
663 </tr>
664
665<tr class="a">
666
667<td>cc </td>
668
669<td>job.history.size </td>
670
671<td>Limits the number of historical jobs remembered by the system to the specified value </td>
672
673<td>10 </td>
674 </tr>
675
676<tr class="b">
677
678<td>cc </td>
679
680<td>job.manager.class </td>
681
682<td>Specify the implementation class name for the job manager </td>
683
684<td>org.apache.hyracks.control.cc.job.JobManager </td>
685 </tr>
686
687<tr class="a">
688
689<td>cc </td>
690
691<td>job.queue.capacity </td>
692
693<td>The maximum number of jobs to queue before rejecting new jobs </td>
694
695<td>4096 </td>
696 </tr>
697
698<tr class="b">
699
700<td>cc </td>
701
702<td>job.queue.class </td>
703
704<td>Specify the implementation class name for the job queue </td>
705
706<td>org.apache.hyracks.control.cc.scheduler.FIFOJobQueue </td>
707 </tr>
708
709<tr class="a">
710
711<td>cc </td>
712
713<td>profile.dump.period </td>
714
715<td>Sets the time duration between two profile dumps from each node controller in milliseconds; 0 to disable </td>
716
717<td>0 </td>
718 </tr>
719
720<tr class="b">
721
722<td>cc </td>
723
724<td>result.sweep.threshold </td>
725
726<td>The duration within which an instance of the result cleanup should be invoked in milliseconds </td>
727
728<td>60000 </td>
729 </tr>
730
731<tr class="a">
732
733<td>cc </td>
734
735<td>result.ttl </td>
736
737<td>Limits the amount of time results for asynchronous jobs should be retained by the system in milliseconds </td>
738
739<td>86400000 </td>
740 </tr>
741
742<tr class="b">
743
744<td>cc </td>
745
746<td>root.dir </td>
747
748<td>Sets the root folder used for file operations </td>
749
750<td>${java.io.tmpdir}/asterixdb/ClusterControllerService </td>
751 </tr>
752
753<tr class="a">
754
755<td>cc </td>
756
757<td>web.port </td>
758
759<td>The listen port of the legacy query interface </td>
760
761<td>19001 </td>
762 </tr>
763
764<tr class="b">
765
766<td>cc </td>
767
768<td>web.queryinterface.port </td>
769
770<td>The listen port of the query web interface </td>
771
772<td>19006 </td>
773 </tr>
774 </tbody>
775</table>
776<p>The following parameters for slave processes, under &#x201c;[nc]&#x201d; sections.</p>
777
778<table border="0" class="table table-striped">
779 <thead>
780
781<tr class="a">
782
783<th>Section </th>
784
785<th>Parameter </th>
786
787<th>Meaning </th>
788
789<th>Default </th>
790 </tr>
791 </thead>
792 <tbody>
793
794<tr class="b">
795
796<td>nc </td>
797
798<td>address </td>
799
800<td>Default IP Address to bind listeners on this NC. All services will bind on this address unless a service-specific listen address is supplied. </td>
801
802<td>127.0.0.1 </td>
803 </tr>
804
805<tr class="a">
806
807<td>nc </td>
808
809<td>app.class </td>
810
811<td>Application NC Main Class </td>
812
813<td>org.apache.asterix.hyracks.bootstrap.NCApplication </td>
814 </tr>
815
816<tr class="b">
817
818<td>nc </td>
819
820<td>cluster.address </td>
821
822<td>Cluster Controller address (required unless specified in config file) </td>
823
824<td>&lt;undefined&gt; </td>
825 </tr>
826
827<tr class="a">
828
829<td>nc </td>
830
831<td>cluster.connect.retries </td>
832
833<td>Number of attempts to contact CC before giving up </td>
834
835<td>5 </td>
836 </tr>
837
838<tr class="b">
839
840<td>nc </td>
841
842<td>cluster.listen.address </td>
843
844<td>IP Address to bind cluster listener on this NC </td>
845
846<td>same as address </td>
847 </tr>
848
849<tr class="a">
850
851<td>nc </td>
852
853<td>cluster.listen.port </td>
854
855<td>IP port to bind cluster listener </td>
856
857<td>0 </td>
858 </tr>
859
860<tr class="b">
861
862<td>nc </td>
863
864<td>cluster.port </td>
865
866<td>Cluster Controller port </td>
867
868<td>1099 </td>
869 </tr>
870
871<tr class="a">
872
873<td>nc </td>
874
875<td>cluster.public.address </td>
876
877<td>Public IP Address to announce cluster listener </td>
878
879<td>same as public.address </td>
880 </tr>
881
882<tr class="b">
883
884<td>nc </td>
885
886<td>cluster.public.port </td>
887
888<td>Public IP port to announce cluster listener </td>
889
890<td>same as cluster.listen.port </td>
891 </tr>
892
893<tr class="a">
894
895<td>nc </td>
896
897<td>command </td>
898
899<td>Command NCService should invoke to start the NCDriver </td>
900
901<td>hyracksnc </td>
902 </tr>
903
904<tr class="b">
905
906<td>nc </td>
907
908<td>core.dump.dir </td>
909
910<td>The directory where node core dumps should be written </td>
911
912<td>${java.io.tmpdir}/asterixdb/coredump </td>
913 </tr>
914
915<tr class="a">
916
917<td>nc </td>
918
919<td>data.listen.address </td>
920
921<td>IP Address to bind data listener </td>
922
923<td>same as address </td>
924 </tr>
925
926<tr class="b">
927
928<td>nc </td>
929
930<td>data.listen.port </td>
931
932<td>IP port to bind data listener </td>
933
934<td>0 </td>
935 </tr>
936
937<tr class="a">
938
939<td>nc </td>
940
941<td>data.public.address </td>
942
943<td>Public IP Address to announce data listener </td>
944
945<td>same as public.address </td>
946 </tr>
947
948<tr class="b">
949
950<td>nc </td>
951
952<td>data.public.port </td>
953
954<td>Public IP port to announce data listener </td>
955
956<td>same as data.listen.port </td>
957 </tr>
958
959<tr class="a">
960
961<td>nc </td>
962
963<td>iodevices </td>
964
965<td>Comma separated list of IO Device mount points </td>
966
967<td>${java.io.tmpdir}/asterixdb/iodevice </td>
968 </tr>
969
970<tr class="b">
971
972<td>nc </td>
973
974<td>jvm.args </td>
975
976<td>JVM args to pass to the NCDriver </td>
977
978<td>&lt;undefined&gt; </td>
979 </tr>
980
981<tr class="a">
982
983<td>nc </td>
984
985<td>messaging.listen.address </td>
986
987<td>IP Address to bind messaging listener </td>
988
989<td>same as address </td>
990 </tr>
991
992<tr class="b">
993
994<td>nc </td>
995
996<td>messaging.listen.port </td>
997
998<td>IP port to bind messaging listener </td>
999
1000<td>0 </td>
1001 </tr>
1002
1003<tr class="a">
1004
1005<td>nc </td>
1006
1007<td>messaging.public.address </td>
1008
1009<td>Public IP Address to announce messaging listener </td>
1010
1011<td>same as public.address </td>
1012 </tr>
1013
1014<tr class="b">
1015
1016<td>nc </td>
1017
1018<td>messaging.public.port </td>
1019
1020<td>Public IP port to announce messaging listener </td>
1021
1022<td>same as messaging.listen.port </td>
1023 </tr>
1024
1025<tr class="a">
1026
1027<td>nc </td>
1028
1029<td>ncservice.address </td>
1030
1031<td>Address the CC should use to contact the NCService associated with this NC </td>
1032
1033<td>same as public.address </td>
1034 </tr>
1035
1036<tr class="b">
1037
1038<td>nc </td>
1039
1040<td>ncservice.pid </td>
1041
1042<td>PID of the NCService which launched this NCDriver </td>
1043
1044<td>-1 </td>
1045 </tr>
1046
1047<tr class="a">
1048
1049<td>nc </td>
1050
1051<td>ncservice.port </td>
1052
1053<td>Port the CC should use to contact the NCService associated with this NC </td>
1054
1055<td>9090 </td>
1056 </tr>
1057
1058<tr class="b">
1059
1060<td>nc </td>
1061
1062<td>net.buffer.count </td>
1063
1064<td>Number of network buffers per input/output channel </td>
1065
1066<td>1 </td>
1067 </tr>
1068
1069<tr class="a">
1070
1071<td>nc </td>
1072
1073<td>net.thread.count </td>
1074
1075<td>Number of threads to use for Network I/O </td>
1076
1077<td>1 </td>
1078 </tr>
1079
1080<tr class="b">
1081
1082<td>nc </td>
1083
1084<td>public.address </td>
1085
1086<td>Default public address that other processes should use to contact this NC. All services will advertise this address unless a service-specific public address is supplied. </td>
1087
1088<td>same as address </td>
1089 </tr>
1090
1091<tr class="a">
1092
1093<td>nc </td>
1094
1095<td>result.listen.address </td>
1096
1097<td>IP Address to bind dataset result distribution listener </td>
1098
1099<td>same as address </td>
1100 </tr>
1101
1102<tr class="b">
1103
1104<td>nc </td>
1105
1106<td>result.listen.port </td>
1107
1108<td>IP port to bind dataset result distribution listener </td>
1109
1110<td>0 </td>
1111 </tr>
1112
1113<tr class="a">
1114
1115<td>nc </td>
1116
1117<td>result.manager.memory </td>
1118
1119<td>Memory usable for result caching at this Node Controller in bytes </td>
1120
1121<td>-1 (-1 B) </td>
1122 </tr>
1123
1124<tr class="b">
1125
1126<td>nc </td>
1127
1128<td>result.public.address </td>
1129
1130<td>Public IP Address to announce dataset result distribution listener </td>
1131
1132<td>same as public.address </td>
1133 </tr>
1134
1135<tr class="a">
1136
1137<td>nc </td>
1138
1139<td>result.public.port </td>
1140
1141<td>Public IP port to announce dataset result distribution listener </td>
1142
1143<td>same as result.listen.port </td>
1144 </tr>
1145
1146<tr class="b">
1147
1148<td>nc </td>
1149
1150<td>result.sweep.threshold </td>
1151
1152<td>The duration within which an instance of the result cleanup should be invoked in milliseconds </td>
1153
1154<td>60000 </td>
1155 </tr>
1156
1157<tr class="a">
1158
1159<td>nc </td>
1160
1161<td>result.ttl </td>
1162
1163<td>Limits the amount of time results for asynchronous jobs should be retained by the system in milliseconds </td>
1164
1165<td>86400000 </td>
1166 </tr>
1167
1168<tr class="b">
1169
1170<td>nc </td>
1171
1172<td>storage.buffercache.maxopenfiles </td>
1173
1174<td>The maximum number of open files in the buffer cache </td>
1175
1176<td>2147483647 </td>
1177 </tr>
1178
1179<tr class="a">
1180
1181<td>nc </td>
1182
1183<td>storage.buffercache.pagesize </td>
1184
1185<td>The page size in bytes for pages in the buffer cache </td>
1186
1187<td>131072 (128 kB) </td>
1188 </tr>
1189
1190<tr class="b">
1191
1192<td>nc </td>
1193
1194<td>storage.buffercache.size </td>
1195
1196<td>The size of memory allocated to the disk buffer cache. The value should be a multiple of the buffer cache page size. </td>
1197
1198<td>715915264 (682.75 MB) </td>
1199 </tr>
1200
1201<tr class="a">
1202
1203<td>nc </td>
1204
1205<td>storage.lsm.bloomfilter.falsepositiverate </td>
1206
1207<td>The maximum acceptable false positive rate for bloom filters associated with LSM indexes </td>
1208
1209<td>0.01 </td>
1210 </tr>
1211
1212<tr class="b">
1213
1214<td>nc </td>
1215
1216<td>storage.memorycomponent.globalbudget </td>
1217
1218<td>The size of memory allocated to the memory components. The value should be a multiple of the memory component page size </td>
1219
1220<td>715915264 (682.75 MB) </td>
1221 </tr>
1222
1223<tr class="a">
1224
1225<td>nc </td>
1226
1227<td>storage.memorycomponent.numcomponents </td>
1228
1229<td>The number of memory components to be used per lsm index </td>
1230
1231<td>2 </td>
1232 </tr>
1233
1234<tr class="b">
1235
1236<td>nc </td>
1237
1238<td>storage.memorycomponent.numpages </td>
1239
1240<td>The number of pages to allocate for a memory component. This budget is shared by all the memory components of the primary index and all its secondary indexes across all I/O devices on a node. Note: in-memory components usually has fill factor of 75% since the pages are 75% full and the remaining 25% is un-utilized </td>
1241
1242<td>1/16th of the storage.memorycomponent.globalbudget value </td>
1243 </tr>
1244
1245<tr class="a">
1246
1247<td>nc </td>
1248
1249<td>storage.memorycomponent.pagesize </td>
1250
1251<td>The page size in bytes for pages allocated to memory components </td>
1252
1253<td>131072 (128 kB) </td>
1254 </tr>
1255
1256<tr class="b">
1257
1258<td>nc </td>
1259
1260<td>storage.metadata.memorycomponent.numpages </td>
1261
1262<td>The number of pages to allocate for a metadata memory component </td>
1263
1264<td>1/64th of the storage.memorycomponent.globalbudget value or 256, whichever is larger </td>
1265 </tr>
1266
1267<tr class="a">
1268
1269<td>nc </td>
1270
1271<td>storage.subdir </td>
1272
1273<td>The subdirectory name under each iodevice used for storage </td>
1274
1275<td>storage </td>
1276 </tr>
1277
1278<tr class="b">
1279
1280<td>nc </td>
1281
1282<td>txn.log.dir </td>
1283
1284<td>The directory where transaction logs should be stored </td>
1285
1286<td>${java.io.tmpdir}/asterixdb/txn-log </td>
1287 </tr>
1288 </tbody>
1289</table>
1290<p>The following parameters are configured under the &#x201c;[common]&#x201d; section.</p>
1291
1292<table border="0" class="table table-striped">
1293 <thead>
1294
1295<tr class="a">
1296
1297<th>Section </th>
1298
1299<th>Parameter </th>
1300
1301<th>Meaning </th>
1302
1303<th>Default </th>
1304 </tr>
1305 </thead>
1306 <tbody>
1307
1308<tr class="b">
1309
1310<td>common </td>
1311
1312<td>active.memory.global.budget </td>
1313
1314<td>The memory budget (in bytes) for the active runtime </td>
1315
1316<td>67108864 (64 MB) </td>
1317 </tr>
1318
1319<tr class="a">
1320
1321<td>common </td>
1322
1323<td>compiler.framesize </td>
1324
1325<td>The page size (in bytes) for computation </td>
1326
1327<td>32768 (32 kB) </td>
1328 </tr>
1329
1330<tr class="b">
1331
1332<td>common </td>
1333
1334<td>compiler.groupmemory </td>
1335
1336<td>The memory budget (in bytes) for a group by operator instance in a partition </td>
1337
1338<td>33554432 (32 MB) </td>
1339 </tr>
1340
1341<tr class="a">
1342
1343<td>common </td>
1344
1345<td>compiler.joinmemory </td>
1346
1347<td>The memory budget (in bytes) for a join operator instance in a partition </td>
1348
1349<td>33554432 (32 MB) </td>
1350 </tr>
1351
1352<tr class="b">
1353
1354<td>common </td>
1355
1356<td>compiler.parallelism </td>
1357
1358<td>The degree of parallelism for query execution. Zero means to use the storage parallelism as the query execution parallelism, while other integer values dictate the number of query execution parallel partitions. The system will fall back to use the number of all available CPU cores in the cluster as the degree of parallelism if the number set by a user is too large or too small </td>
1359
1360<td>0 </td>
1361 </tr>
1362
1363<tr class="a">
1364
1365<td>common </td>
1366
1367<td>compiler.sortmemory </td>
1368
1369<td>The memory budget (in bytes) for a sort operator instance in a partition </td>
1370
1371<td>33554432 (32 MB) </td>
1372 </tr>
1373
1374<tr class="b">
1375
1376<td>common </td>
1377
1378<td>instance.name </td>
1379
1380<td>The name of this cluster instance </td>
1381
1382<td>DEFAULT_INSTANCE </td>
1383 </tr>
1384
1385<tr class="a">
1386
1387<td>common </td>
1388
1389<td>log.level </td>
1390
1391<td>The logging level for master and slave processes </td>
1392
1393<td>WARNING </td>
1394 </tr>
1395
1396<tr class="b">
1397
1398<td>common </td>
1399
1400<td>max.wait.active.cluster </td>
1401
1402<td>The max pending time (in seconds) for cluster startup. After the threshold, if the cluster still is not up and running, it is considered unavailable </td>
1403
1404<td>60 </td>
1405 </tr>
1406
1407<tr class="a">
1408
1409<td>common </td>
1410
1411<td>messaging.frame.count </td>
1412
1413<td>Number of reusable frames for NC to NC messaging </td>
1414
1415<td>512 </td>
1416 </tr>
1417
1418<tr class="b">
1419
1420<td>common </td>
1421
1422<td>messaging.frame.size </td>
1423
1424<td>The frame size to be used for NC to NC messaging </td>
1425
1426<td>4096 (4 kB) </td>
1427 </tr>
1428
1429<tr class="a">
1430
1431<td>common </td>
1432
1433<td>metadata.callback.port </td>
1434
1435<td>IP port to bind metadata callback listener (0 = random port) </td>
1436
1437<td>0 </td>
1438 </tr>
1439
1440<tr class="b">
1441
1442<td>common </td>
1443
1444<td>metadata.listen.port </td>
1445
1446<td>IP port to bind metadata listener (0 = random port) </td>
1447
1448<td>0 </td>
1449 </tr>
1450
1451<tr class="a">
1452
1453<td>common </td>
1454
1455<td>metadata.node </td>
1456
1457<td>the node which should serve as the metadata node </td>
1458
1459<td>&lt;undefined&gt; </td>
1460 </tr>
1461
1462<tr class="b">
1463
1464<td>common </td>
1465
1466<td>metadata.registration.timeout.secs </td>
1467
1468<td>how long in seconds to wait for the metadata node to register with the CC </td>
1469
1470<td>60 </td>
1471 </tr>
1472
1473<tr class="a">
1474
1475<td>common </td>
1476
1477<td>replication.log.batchsize </td>
1478
1479<td>The size in bytes to replicate in each batch </td>
1480
1481<td>4096 (4 kB) </td>
1482 </tr>
1483
1484<tr class="b">
1485
1486<td>common </td>
1487
1488<td>replication.log.buffer.numpages </td>
1489
1490<td>The number of log buffer pages </td>
1491
1492<td>8 </td>
1493 </tr>
1494
1495<tr class="a">
1496
1497<td>common </td>
1498
1499<td>replication.log.buffer.pagesize </td>
1500
1501<td>The size in bytes of each log buffer page </td>
1502
1503<td>131072 (128 kB) </td>
1504 </tr>
1505
1506<tr class="b">
1507
1508<td>common </td>
1509
1510<td>replication.max.remote.recovery.attempts </td>
1511
1512<td>The maximum number of times to attempt to recover from a replica on failure before giving up </td>
1513
1514<td>5 </td>
1515 </tr>
1516
1517<tr class="a">
1518
1519<td>common </td>
1520
1521<td>replication.timeout </td>
1522
1523<td>The time in seconds to timeout when trying to contact a replica, before assuming it is dead </td>
1524
1525<td>15 </td>
1526 </tr>
1527
1528<tr class="b">
1529
1530<td>common </td>
1531
1532<td>txn.commitprofiler.enabled </td>
1533
1534<td>Enable output of commit profiler logs </td>
1535
1536<td>false </td>
1537 </tr>
1538
1539<tr class="a">
1540
1541<td>common </td>
1542
1543<td>txn.commitprofiler.reportinterval </td>
1544
1545<td>Interval (in seconds) to report commit profiler logs </td>
1546
1547<td>5 </td>
1548 </tr>
1549
1550<tr class="b">
1551
1552<td>common </td>
1553
1554<td>txn.job.recovery.memorysize </td>
1555
1556<td>The memory budget (in bytes) used for recovery </td>
1557
1558<td>67108864 (64 MB) </td>
1559 </tr>
1560
1561<tr class="a">
1562
1563<td>common </td>
1564
1565<td>txn.lock.escalationthreshold </td>
1566
1567<td>The maximum number of entity locks to obtain before upgrading to a dataset lock </td>
1568
1569<td>1000 </td>
1570 </tr>
1571
1572<tr class="b">
1573
1574<td>common </td>
1575
1576<td>txn.lock.shrinktimer </td>
1577
1578<td>The time (in milliseconds) where under utilization of resources will trigger a shrink phase </td>
1579
1580<td>5000 </td>
1581 </tr>
1582
1583<tr class="a">
1584
1585<td>common </td>
1586
1587<td>txn.lock.timeout.sweepthreshold </td>
1588
1589<td>Interval (in milliseconds) for checking lock timeout </td>
1590
1591<td>10000 </td>
1592 </tr>
1593
1594<tr class="b">
1595
1596<td>common </td>
1597
1598<td>txn.lock.timeout.waitthreshold </td>
1599
1600<td>Time out (in milliseconds) of waiting for a lock </td>
1601
1602<td>60000 </td>
1603 </tr>
1604
1605<tr class="a">
1606
1607<td>common </td>
1608
1609<td>txn.log.buffer.numpages </td>
1610
1611<td>The number of pages in the transaction log tail </td>
1612
1613<td>8 </td>
1614 </tr>
1615
1616<tr class="b">
1617
1618<td>common </td>
1619
1620<td>txn.log.buffer.pagesize </td>
1621
1622<td>The page size (in bytes) for transaction log buffer </td>
1623
1624<td>131072 (128 kB) </td>
1625 </tr>
1626
1627<tr class="a">
1628
1629<td>common </td>
1630
1631<td>txn.log.checkpoint.history </td>
1632
1633<td>The number of checkpoints to keep in the transaction log </td>
1634
1635<td>0 </td>
1636 </tr>
1637
1638<tr class="b">
1639
1640<td>common </td>
1641
1642<td>txn.log.checkpoint.lsnthreshold </td>
1643
1644<td>The checkpoint threshold (in terms of LSNs (log sequence numbers) that have been written to the transaction log, i.e., the length of the transaction log) for transaction logs </td>
1645
1646<td>67108864 (64 MB) </td>
1647 </tr>
1648
1649<tr class="a">
1650
1651<td>common </td>
1652
1653<td>txn.log.checkpoint.pollfrequency </td>
1654
1655<td>The frequency (in seconds) the checkpoint thread should check to see if a checkpoint should be written </td>
1656
1657<td>120 </td>
1658 </tr>
1659
1660<tr class="b">
1661
1662<td>common </td>
1663
1664<td>txn.log.partitionsize </td>
1665
1666<td>The maximum size (in bytes) of each transaction log file </td>
1667
1668<td>268435456 (256 MB) </td>
1669 </tr>
1670 </tbody>
1671</table>
1672<p>For the optional NCService process configuration file, the following parameters, under &#x201c;[ncservice]&#x201d; section.</p>
1673
1674<table border="0" class="table table-striped">
1675 <thead>
1676
1677<tr class="a">
1678
1679<th>Parameter </th>
1680
1681<th>Meaning </th>
1682
1683<th>Default </th>
1684 </tr>
1685 </thead>
1686 <tbody>
1687
1688<tr class="b">
1689
1690<td>address </td>
1691
1692<td>The address the NCService listens on for commands from the CC </td>
1693
1694<td>(all addresses) </td>
1695 </tr>
1696
1697<tr class="a">
1698
1699<td>port </td>
1700
1701<td>The port for the NCService listens on for commands from the CC </td>
1702
1703<td>9090 </td>
1704 </tr>
1705
1706<tr class="b">
1707
1708<td>logdir </td>
1709
1710<td>Directory where NCService logs should be written (&#x2018;-&#x2019; indicates that output should go to stdout) </td>
1711
1712<td>${app.home}/logs (${user.home} if &#x2018;app.home&#x2019; not present in NCService Java system properties. </td>
1713 </tr>
1714 </tbody>
1715</table></div>
1716 </div>
1717 </div>
1718 </div>
1719
1720 <hr/>
1721
1722 <footer>
1723 <div class="container-fluid">
1724 <div class="row span12">Copyright &copy; 2018
1725 <a href="https://www.apache.org/">The Apache Software Foundation</a>.
1726 All Rights Reserved.
1727
1728 </div>
1729
1730 <?xml version="1.0" encoding="UTF-8"?>
1731<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
1732 feather logo, and the Apache AsterixDB project logo are either
1733 registered trademarks or trademarks of The Apache Software
1734 Foundation in the United States and other countries.
1735 All other marks mentioned may be trademarks or registered
1736 trademarks of their respective owners.</div>
1737
1738
1739 </div>
1740 </footer>
1741 </body>
1742</html>