Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / 5f34114603bb17e7c3493d060d88bad69e418cda / . / content / docs / 0.9.2 / aws.html
blob: 514dfb0d25a24d9fda23f1151626ac867ef17c2f [file] [log] [blame]
Ian Maxon7a4bed92017-09-15 02:01:18 +0200[diff] [blame]1<!DOCTYPE html>
2<!--
3 | Generated by Apache Maven Doxia at 2017-09-14
4 | Rendered using Apache Maven Fluido Skin 1.3.0
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
10 <meta name="Date-Revision-yyyymmdd" content="20170914" />
11 <meta http-equiv="Content-Language" content="en" />
12 <title>AsterixDB &#x2013; </title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16
17
18 <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
19
20
21
Ian Maxon7a4bed92017-09-15 02:01:18 +0200[diff] [blame]22
Ian Maxon7a4bed92017-09-15 02:01:18 +0200[diff] [blame]23
24 </head>
25 <body class="topBarDisabled">
26
27
28
29
30 <div class="container-fluid">
31 <div id="banner">
32 <div class="pull-left">
33 <a href="./" id="bannerLeft">
34 <img src="images/asterixlogo.png" alt="AsterixDB"/>
35 </a>
36 </div>
37 <div class="pull-right"> </div>
38 <div class="clear"><hr/></div>
39 </div>
40
41 <div id="breadcrumbs">
42 <ul class="breadcrumb">
43
44
45 <li id="publishDate">Last Published: 2017-09-14</li>
46
47
48
49 <li id="projectVersion" class="pull-right">Version: 0.9.2</li>
50
51 <li class="divider pull-right">|</li>
52
53 <li class="pull-right"> <a href="index.html" title="Documentation Home">
54 Documentation Home</a>
55 </li>
56
57 </ul>
58 </div>
59
60
61 <div class="row-fluid">
62 <div id="leftColumn" class="span3">
63 <div class="well sidebar-nav">
64
65
66 <ul class="nav nav-list">
67 <li class="nav-header">Get Started - Installation</li>
68
69 <li>
70
71 <a href="ncservice.html" title="Option 1: using NCService">
72 <i class="none"></i>
73 Option 1: using NCService</a>
74 </li>
75
76 <li>
77
78 <a href="ansible.html" title="Option 2: using Ansible">
79 <i class="none"></i>
80 Option 2: using Ansible</a>
81 </li>
82
83 <li class="active">
84
85 <a href="#"><i class="none"></i>Option 3: using Amazon Web Services</a>
86 </li>
87
88 <li>
89
90 <a href="yarn.html" title="Option 4: using YARN">
91 <i class="none"></i>
92 Option 4: using YARN</a>
93 </li>
94
95 <li>
96
97 <a href="install.html" title="Option 5: using Managix (deprecated)">
98 <i class="none"></i>
99 Option 5: using Managix (deprecated)</a>
100 </li>
101 <li class="nav-header">AsterixDB Primer</li>
102
103 <li>
104
105 <a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++">
106 <i class="none"></i>
107 Option 1: using SQL++</a>
108 </li>
109
110 <li>
111
112 <a href="aql/primer.html" title="Option 2: using AQL">
113 <i class="none"></i>
114 Option 2: using AQL</a>
115 </li>
116 <li class="nav-header">Data Model</li>
117
118 <li>
119
120 <a href="datamodel.html" title="The Asterix Data Model">
121 <i class="none"></i>
122 The Asterix Data Model</a>
123 </li>
124 <li class="nav-header">Queries - SQL++</li>
125
126 <li>
127
128 <a href="sqlpp/manual.html" title="The SQL++ Query Language">
129 <i class="none"></i>
130 The SQL++ Query Language</a>
131 </li>
132
133 <li>
134
135 <a href="sqlpp/builtins.html" title="Builtin Functions">
136 <i class="none"></i>
137 Builtin Functions</a>
138 </li>
139 <li class="nav-header">Queries - AQL</li>
140
141 <li>
142
143 <a href="aql/manual.html" title="The Asterix Query Language (AQL)">
144 <i class="none"></i>
145 The Asterix Query Language (AQL)</a>
146 </li>
147
148 <li>
149
150 <a href="aql/builtins.html" title="Builtin Functions">
151 <i class="none"></i>
152 Builtin Functions</a>
153 </li>
154 <li class="nav-header">API/SDK</li>
155
156 <li>
157
158 <a href="api.html" title="HTTP API">
159 <i class="none"></i>
160 HTTP API</a>
161 </li>
162
163 <li>
164
165 <a href="csv.html" title="CSV Output">
166 <i class="none"></i>
167 CSV Output</a>
168 </li>
169 <li class="nav-header">Advanced Features</li>
170
171 <li>
172
173 <a href="aql/fulltext.html" title="Support of Full-text Queries">
174 <i class="none"></i>
175 Support of Full-text Queries</a>
176 </li>
177
178 <li>
179
180 <a href="aql/externaldata.html" title="Accessing External Data">
181 <i class="none"></i>
182 Accessing External Data</a>
183 </li>
184
185 <li>
186
187 <a href="feeds/tutorial.html" title="Support for Data Ingestion">
188 <i class="none"></i>
189 Support for Data Ingestion</a>
190 </li>
191
192 <li>
193
194 <a href="udf.html" title="User Defined Functions">
195 <i class="none"></i>
196 User Defined Functions</a>
197 </li>
198
199 <li>
200
201 <a href="aql/filters.html" title="Filter-Based LSM Index Acceleration">
202 <i class="none"></i>
203 Filter-Based LSM Index Acceleration</a>
204 </li>
205
206 <li>
207
208 <a href="aql/similarity.html" title="Support of Similarity Queries">
209 <i class="none"></i>
210 Support of Similarity Queries</a>
211 </li>
212 </ul>
213
214
215
216 <hr class="divider" />
217
218 <div id="poweredBy">
219 <div class="clear"></div>
220 <div class="clear"></div>
221 <div class="clear"></div>
222 <a href="./" title="AsterixDB" class="builtBy">
223 <img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" />
224 </a>
225 </div>
226 </div>
227 </div>
228
229
230 <div id="bodyColumn" class="span9" >
231
232 <!-- ! Licensed to the Apache Software Foundation (ASF) under one
233 ! or more contributor license agreements. See the NOTICE file
234 ! distributed with this work for additional information
235 ! regarding copyright ownership. The ASF licenses this file
236 ! to you under the Apache License, Version 2.0 (the
237 ! "License"); you may not use this file except in compliance
238 ! with the License. You may obtain a copy of the License at
239 !
240 ! http://www.apache.org/licenses/LICENSE-2.0
241 !
242 ! Unless required by applicable law or agreed to in writing,
243 ! software distributed under the License is distributed on an
244 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
245 ! KIND, either express or implied. See the License for the
246 ! specific language governing permissions and limitations
247 ! under the License.
248 ! -->
249<ul>
250
251<li><a href="#Introduction">Introduction</a></li>
252
253<li><a href="#Prerequisites">Prerequisites</a></li>
254
255<li><a href="#config">Cluster Configuration</a></li>
256
257<li><a href="#lifecycle">Cluster Lifecycle Management</a> <!-- ! Licensed to the Apache Software Foundation (ASF) under one
258 ! or more contributor license agreements. See the NOTICE file
259 ! distributed with this work for additional information
260 ! regarding copyright ownership. The ASF licenses this file
261 ! to you under the Apache License, Version 2.0 (the
262 ! "License"); you may not use this file except in compliance
263 ! with the License. You may obtain a copy of the License at
264 !
265 ! http://www.apache.org/licenses/LICENSE-2.0
266 !
267 ! Unless required by applicable law or agreed to in writing,
268 ! software distributed under the License is distributed on an
269 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
270 ! KIND, either express or implied. See the License for the
271 ! specific language governing permissions and limitations
272 ! under the License.
273 ! --></li>
274</ul>
275<div class="section">
276<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
277<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p>
278<p>However, via this installation option, we provide a combo solution for automating both AWS EC2 and AsterixDB, where you can run only one script to deploy, start, stop, and terminate an AsterixDB cluster on AWS.</p></div>
279<div class="section">
280<h2><a name="Prerequisites" id="Prerequisites">Prerequisites</a></h2>
281
282<ul>
283
284<li>
285<p>Supported operating systems for the client: <b>Linux</b> and <b>MacOS</b></p></li>
286
287<li>
288<p>Supported operating systems for Amazon Web Services instances: <b>Linux</b></p></li>
289
290<li>
291<p>Install pip on your client machine:</p>
292<p>CentOS</p>
293
294<div class="source">
295<div class="source">
296<pre> $ sudo yum install python-pip
297</pre></div></div>
298<p>Ubuntu</p>
299
300<div class="source">
301<div class="source">
302<pre> $ sudo apt-get install python-pip
303</pre></div></div>
304<p>macOS</p>
305
306<div class="source">
307<div class="source">
308<pre> $ brew install pip
309</pre></div></div></li>
310
311<li>
312<p>Install Ansible, boto, and boto3 on your client machine:</p>
313
314<div class="source">
315<div class="source">
316<pre> $ pip install ansible
317 $ pip install boto
318 $ pip install boto3
319</pre></div></div>
320<p>Note that you might need <tt>sudo</tt> depending on your system configuration.</p>
321<p><b>Make sure that the version of Ansible is no less than 2.2.1.0</b>:</p>
322
323<div class="source">
324<div class="source">
325<pre> $ ansible --version
326 ansible 2.2.1.0
327</pre></div></div>
328<p><b>For users with macOS 10.11+</b>, please create a user-level Ansible configuration file at:</p>
329
330<div class="source">
331<div class="source">
332<pre> ~/.ansible.cfg
333</pre></div></div>
334<p>and add the following configuration:</p>
335
336<div class="source">
337<div class="source">
338<pre> [ssh_connection]
339 control_path = %(directory)s/%%C
340</pre></div></div></li>
341
342<li>
343<p>Download the AsterixDB distribution package, unzip it, navigate to <tt>opt/aws/</tt></p>
344
345<div class="source">
346<div class="source">
347<pre> $ cd opt/aws
348</pre></div></div>
349<p>The following files and directories are in the directory <tt>opt/aws</tt>:</p>
350
351<div class="source">
352<div class="source">
353<pre> README bin conf yaml
354</pre></div></div>
355<p><tt>bin</tt> contains scripts that start and terminate an AWS-based cluster instance, according to the configuration specified in files under <tt>conf</tt>, and <tt>yaml</tt> contains internal Ansible scripts that the shell scripts in <tt>bin</tt> use.</p></li>
356
357<li>
358<p>Create an AWS account and an IAM user.</p>
359<p>Set up a security group that you&#x2019;d like to use for your AWS cluster. <b>The security group should at least allow all TCP connections from anywhere.</b> Provide the name of the security group as the value for the <tt>group</tt> field in <tt>conf/aws_settings.yml</tt>.</p></li>
360
361<li>
362<p>Retrieve your AWS EC2 key pair name and use that as the <tt>keypair</tt> in <tt>conf/aws_settings.yml</tt>;</p>
363<p>retrieve your AWS IAM <tt>access key ID</tt> and use that as the <tt>access_key_id</tt> in <tt>conf/aws_settings.yml</tt>;</p>
364<p>retrieve your AWS IAM <tt>secret access key</tt> and use that as the <tt>secret_access_key</tt> in <tt>conf/aws_settings.yml</tt>.</p>
365<p>Note that you can only read or download <tt>access key ID</tt> and <tt>secret access key</tt> once from your AWS console. If you forget them, you have to create new keys and delete the old ones.</p></li>
366
367<li>
368<p>Configure your ssh setting by editing <tt>~/.ssh/config</tt> and adding the following entry:</p>
369
370<div class="source">
371<div class="source">
372<pre> Host *.amazonaws.com
373 IdentityFile &lt;path_of_private_key&gt;
374</pre></div></div>
375<p>Note that &lt;path_of_private_key&gt; should be replaced by the path to the file that stores the private key for the key pair that you uploaded to AWS and used in <tt>conf/aws_settings</tt>. For example:</p>
376
377<div class="source">
378<div class="source">
379<pre> Host *.amazonaws.com
380 IdentityFile ~/.ssh/id_rsa
381</pre></div></div></li>
382</ul></div>
383<div class="section">
384<h2><a name="Cluster_Configuration"></a><a name="config" id="config">Cluster Configuration</a></h2>
385
386<ul>
387
388<li>
389<p><b>AWS settings</b>. Edit <tt>conf/instance_settings.yml</tt>. The meaning of each parameter is listed as follows:</p>
390
391<div class="source">
392<div class="source">
393<pre> # The OS image id for ec2 instances.
394 image: ami-76fa4116
395
396 # The data center region for ec2 instances.
397 region: us-west-2
398
399 # The tag for each ec2 machine. Use different tags for isolation.
400 tag: scale_test
401
402 # The name of a security group that appears in your AWS console.
403 group: default
404
405 # The name of a key pair that appears in your AWS console.
406 keypair: &lt;to be filled&gt;
407
408 # The AWS access key id for your IAM user.
409 access_key_id: &lt;to be filled&gt;
410
411 # The AWS secret key for your IAM user.
412 secret_access_key: &lt;to be filled&gt;
413
414 # The AWS instance type. A full list of available types are listed at:
415 # https://aws.amazon.com/ec2/instance-types/
416 instance_type: t2.micro
417
418 # The number of ec2 instances that construct a cluster.
419 count: 3
420
421 # The user name.
422 user: ec2-user
423
424 # Whether to reuse one slave machine to host the master process.
425 cc_on_nc: false
426</pre></div></div>
427<p><b>As described in <a href="#Prerequisites">prerequisites</a>, the following parameters must be customized:</b></p>
428
429<div class="source">
430<div class="source">
431<pre> # The tag for each ec2 machine. Use different tags for isolation.
432 tag: scale_test
433
434 # The name of a security group that appears in your AWS console.
435 group: default
436
437 # The name of a key pair that appears in your AWS console.
438 keypair: &lt;to be filled&gt;
439
440 # The AWS access key id for your IAM user.
441 access_key_id: &lt;to be filled&gt;
442
443 # The AWS secrety key for your IAM user.
444 secret_access_key: &lt;to be filled&gt;
445</pre></div></div></li>
446
447<li>
448<p><b>Remote working directories</b>. Edit <tt>conf/instance_settings.yml</tt> to change the remote binary directory (the variable &#x201c;binarydir&#x201d;) when necessary. By default, the binary directory will be under the home directory (as the value of Ansible builtin variable ansible_env.HOME) of the ssh user account on each node.</p></li>
449</ul></div>
450<div class="section">
451<h2><a name="Cluster_Lifecycle_Management"></a><a name="lifecycle" id="lifecycle">Cluster Lifecycle Management</a></h2>
452
453<ul>
454
455<li>
456<p>Allocate AWS EC2 nodes (the number of nodes is specified in <tt>conf/instance_settings.yml</tt>) and deploy the binary to all allocated EC2 nodes:</p>
457
458<div class="source">
459<div class="source">
460<pre> bin/deploy.sh
461</pre></div></div></li>
462
463<li>
464<p>Before starting the AsterixDB cluster, you the instance configuration file <tt>conf/instance/cc.conf</tt> can be modified with the exception of the IP addresses/DNS names which are are generated and cannot be changed. All available parameters and their usage can be found <a href="ncservice.html#Parameters">here</a>.</p></li>
465
466<li>
467<p>Launch your AsterixDB cluster on EC2:</p>
468
469<div class="source">
470<div class="source">
471<pre> bin/start.sh
472</pre></div></div>
473<p>Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node listed in <tt>conf/instance/inventory</tt> at port <tt>19001</tt> (which can be customized in <tt>conf/instance/cc.conf</tt>) in your browser.</p></li>
474
475<li>
476<p>If you want to stop the AWS-based AsterixDB cluster, run the following script:</p>
477
478<div class="source">
479<div class="source">
480<pre> bin/stop.sh
481</pre></div></div>
482<p>Note that this only stops AsterixDB but does not stop the EC2 nodes.</p></li>
483
484<li>
485<p>If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script:</p>
486
487<div class="source">
488<div class="source">
489<pre> bin/terminate.sh
490</pre></div></div>
491<p><b>Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes for the cluster.</b></p></li>
492</ul></div>
493 </div>
494 </div>
495 </div>
496
497 <hr/>
498
499 <footer>
500 <div class="container-fluid">
501 <div class="row span12">Copyright &copy; 2017
502 <a href="https://www.apache.org/">The Apache Software Foundation</a>.
503 All Rights Reserved.
504
505 </div>
506
507 <?xml version="1.0" encoding="UTF-8"?>
508<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
509 feather logo, and the Apache AsterixDB project logo are either
510 registered trademarks or trademarks of The Apache Software
511 Foundation in the United States and other countries.
512 All other marks mentioned may be trademarks or registered
513 trademarks of their respective owners.</div>
514
515
516 </div>
517 </footer>
518 </body>
519</html>