Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / a54b0716f0177e62ba4db58b5c74f923beec4688 / . / content / docs / 0.9.1 / aws.html
blob: f8dd1208a10549f5522a707d58b6623a2a393527 [file] [log] [blame]
Ian Maxon100cb802017-04-24 18:48:07 -0700[diff] [blame]1<!DOCTYPE html>
2<!--
3 | Generated by Apache Maven Doxia at 2017-04-24
4 | Rendered using Apache Maven Fluido Skin 1.3.0
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
10 <meta name="Date-Revision-yyyymmdd" content="20170424" />
11 <meta http-equiv="Content-Language" content="en" />
12 <title>AsterixDB &#x2013; </title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16
17
18 <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
19
20
21
22<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
23 (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
24 m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
25 })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
26
27 ga('create', 'UA-41536543-1', 'uci.edu');
28 ga('send', 'pageview');</script>
29
30 </head>
31 <body class="topBarDisabled">
32
33
34
35
36 <div class="container-fluid">
37 <div id="banner">
38 <div class="pull-left">
39 <a href="./" id="bannerLeft">
40 <img src="images/asterixlogo.png" alt="AsterixDB"/>
41 </a>
42 </div>
43 <div class="pull-right"> </div>
44 <div class="clear"><hr/></div>
45 </div>
46
47 <div id="breadcrumbs">
48 <ul class="breadcrumb">
49
50
51 <li id="publishDate">Last Published: 2017-04-24</li>
52
53
54
55 <li id="projectVersion" class="pull-right">Version: 0.9.1</li>
56
57 <li class="divider pull-right">|</li>
58
59 <li class="pull-right"> <a href="index.html" title="Documentation Home">
60 Documentation Home</a>
61 </li>
62
63 </ul>
64 </div>
65
66
67 <div class="row-fluid">
68 <div id="leftColumn" class="span3">
69 <div class="well sidebar-nav">
70
71
72 <ul class="nav nav-list">
73 <li class="nav-header">Get Started - Installation</li>
74
75 <li>
76
77 <a href="ncservice.html" title="Option 1: using NCService">
78 <i class="none"></i>
79 Option 1: using NCService</a>
80 </li>
81
82 <li>
83
84 <a href="ansible.html" title="Option 2: using Ansible">
85 <i class="none"></i>
86 Option 2: using Ansible</a>
87 </li>
88
89 <li class="active">
90
91 <a href="#"><i class="none"></i>Option 3: using Amazon Web Services</a>
92 </li>
93
94 <li>
95
96 <a href="yarn.html" title="Option 4: using YARN">
97 <i class="none"></i>
98 Option 4: using YARN</a>
99 </li>
100
101 <li>
102
103 <a href="install.html" title="Option 5: using Managix (deprecated)">
104 <i class="none"></i>
105 Option 5: using Managix (deprecated)</a>
106 </li>
107 <li class="nav-header">AsterixDB Primer</li>
108
109 <li>
110
111 <a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++">
112 <i class="none"></i>
113 Option 1: using SQL++</a>
114 </li>
115
116 <li>
117
118 <a href="aql/primer.html" title="Option 2: using AQL">
119 <i class="none"></i>
120 Option 2: using AQL</a>
121 </li>
122 <li class="nav-header">Data Model</li>
123
124 <li>
125
126 <a href="datamodel.html" title="The Asterix Data Model">
127 <i class="none"></i>
128 The Asterix Data Model</a>
129 </li>
130 <li class="nav-header">Queries - SQL++</li>
131
132 <li>
133
134 <a href="sqlpp/manual.html" title="The SQL++ Query Language">
135 <i class="none"></i>
136 The SQL++ Query Language</a>
137 </li>
138
139 <li>
140
141 <a href="sqlpp/builtins.html" title="Builtin Functions">
142 <i class="none"></i>
143 Builtin Functions</a>
144 </li>
145 <li class="nav-header">Queries - AQL</li>
146
147 <li>
148
149 <a href="aql/manual.html" title="The Asterix Query Language (AQL)">
150 <i class="none"></i>
151 The Asterix Query Language (AQL)</a>
152 </li>
153
154 <li>
155
156 <a href="aql/builtins.html" title="Builtin Functions">
157 <i class="none"></i>
158 Builtin Functions</a>
159 </li>
160 <li class="nav-header">API/SDK</li>
161
162 <li>
163
164 <a href="api.html" title="HTTP API">
165 <i class="none"></i>
166 HTTP API</a>
167 </li>
168
169 <li>
170
171 <a href="csv.html" title="CSV Output">
172 <i class="none"></i>
173 CSV Output</a>
174 </li>
175 <li class="nav-header">Advanced Features</li>
176
177 <li>
178
179 <a href="aql/fulltext.html" title="Support of Full-text Queries">
180 <i class="none"></i>
181 Support of Full-text Queries</a>
182 </li>
183
184 <li>
185
186 <a href="aql/externaldata.html" title="Accessing External Data">
187 <i class="none"></i>
188 Accessing External Data</a>
189 </li>
190
191 <li>
192
193 <a href="feeds/tutorial.html" title="Support for Data Ingestion">
194 <i class="none"></i>
195 Support for Data Ingestion</a>
196 </li>
197
198 <li>
199
200 <a href="udf.html" title="User Defined Functions">
201 <i class="none"></i>
202 User Defined Functions</a>
203 </li>
204
205 <li>
206
207 <a href="aql/filters.html" title="Filter-Based LSM Index Acceleration">
208 <i class="none"></i>
209 Filter-Based LSM Index Acceleration</a>
210 </li>
211
212 <li>
213
214 <a href="aql/similarity.html" title="Support of Similarity Queries">
215 <i class="none"></i>
216 Support of Similarity Queries</a>
217 </li>
218 </ul>
219
220
221
222 <hr class="divider" />
223
224 <div id="poweredBy">
225 <div class="clear"></div>
226 <div class="clear"></div>
227 <div class="clear"></div>
228 <a href="./" title="AsterixDB" class="builtBy">
229 <img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" />
230 </a>
231 </div>
232 </div>
233 </div>
234
235
236 <div id="bodyColumn" class="span9" >
237
238 <!-- ! Licensed to the Apache Software Foundation (ASF) under one
239 ! or more contributor license agreements. See the NOTICE file
240 ! distributed with this work for additional information
241 ! regarding copyright ownership. The ASF licenses this file
242 ! to you under the Apache License, Version 2.0 (the
243 ! "License"); you may not use this file except in compliance
244 ! with the License. You may obtain a copy of the License at
245 !
246 ! http://www.apache.org/licenses/LICENSE-2.0
247 !
248 ! Unless required by applicable law or agreed to in writing,
249 ! software distributed under the License is distributed on an
250 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
251 ! KIND, either express or implied. See the License for the
252 ! specific language governing permissions and limitations
253 ! under the License.
254 ! -->
255<ul>
256
257<li><a href="#Introduction">Introduction</a></li>
258
259<li><a href="#Prerequisites">Prerequisites</a></li>
260
261<li><a href="#config">Cluster Configuration</a></li>
262
263<li><a href="#lifecycle">Cluster Lifecycle Management</a> <!-- ! Licensed to the Apache Software Foundation (ASF) under one
264 ! or more contributor license agreements. See the NOTICE file
265 ! distributed with this work for additional information
266 ! regarding copyright ownership. The ASF licenses this file
267 ! to you under the Apache License, Version 2.0 (the
268 ! "License"); you may not use this file except in compliance
269 ! with the License. You may obtain a copy of the License at
270 !
271 ! http://www.apache.org/licenses/LICENSE-2.0
272 !
273 ! Unless required by applicable law or agreed to in writing,
274 ! software distributed under the License is distributed on an
275 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
276 ! KIND, either express or implied. See the License for the
277 ! specific language governing permissions and limitations
278 ! under the License.
279 ! --></li>
280</ul>
281<div class="section">
282<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
283<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p>
284<p>However, via this installation option, we provide a combo solution for automating both AWS EC2 and AsterixDB, where you can run only one script to deploy, start, stop, and terminate an AsterixDB cluster on AWS.</p></div>
285<div class="section">
286<h2><a name="Prerequisites" id="Prerequisites">Prerequisites</a></h2>
287
288<ul>
289
290<li>
291<p>Supported operating systems for the client: <b>Linux</b> and <b>MacOS</b></p></li>
292
293<li>
294<p>Supported operating systems for Amazon Web Services instances: <b>Linux</b></p></li>
295
296<li>
297<p>Install pip on your client machine:</p>
298<p>CentOS</p>
299
300<div class="source">
301<div class="source">
302<pre> $ sudo yum install python-pip
303</pre></div></div>
304<p>Ubuntu</p>
305
306<div class="source">
307<div class="source">
308<pre> $ sudo apt-get install python-pip
309</pre></div></div>
310<p>macOS</p>
311
312<div class="source">
313<div class="source">
314<pre> $ brew install pip
315</pre></div></div></li>
316
317<li>
318<p>Install Ansible, boto, and boto3 on your client machine:</p>
319
320<div class="source">
321<div class="source">
322<pre> $ pip install ansible
323 $ pip install boto
324 $ pip install boto3
325</pre></div></div>
326<p>Note that you might need <tt>sudo</tt> depending on your system configuration.</p>
327<p><b>Make sure that the version of Ansible is no less than 2.2.1.0</b>:</p>
328
329<div class="source">
330<div class="source">
331<pre> $ ansible --version
332 ansible 2.2.1.0
333</pre></div></div>
334<p><b>For users with macOS 10.11+</b>, please create a user-level Ansible configuration file at:</p>
335
336<div class="source">
337<div class="source">
338<pre> ~/.ansible.cfg
339</pre></div></div>
340<p>and add the following configuration:</p>
341
342<div class="source">
343<div class="source">
344<pre> [ssh_connection]
345 control_path = %(directory)s/%%C
346</pre></div></div></li>
347
348<li>
349<p>Download the AsterixDB distribution package, unzip it, navigate to <tt>opt/aws/</tt></p>
350
351<div class="source">
352<div class="source">
353<pre> $ cd opt/aws
354</pre></div></div>
355<p>The following files and directories are in the directory <tt>opt/aws</tt>:</p>
356
357<div class="source">
358<div class="source">
359<pre> README bin conf yaml
360</pre></div></div>
361<p><tt>bin</tt> contains scripts that start and terminate an AWS-based cluster instance, according to the configuration specified in files under <tt>conf</tt>, and <tt>yaml</tt> contains internal Ansible scripts that the shell scripts in <tt>bin</tt> use.</p></li>
362
363<li>
364<p>Create an AWS account and an IAM user.</p>
365<p>Set up a security group that you&#x2019;d like to use for your AWS cluster. <b>The security group should at least allow all TCP connections from anywhere.</b> Provide the name of the security group as the value for the <tt>group</tt> field in <tt>conf/aws_settings.yml</tt>.</p></li>
366
367<li>
368<p>Retrieve your AWS EC2 key pair name and use that as the <tt>keypair</tt> in <tt>conf/aws_settings.yml</tt>;</p>
369<p>retrieve your AWS IAM <tt>access key ID</tt> and use that as the <tt>access_key_id</tt> in <tt>conf/aws_settings.yml</tt>;</p>
370<p>retrieve your AWS IAM <tt>secret access key</tt> and use that as the <tt>secret_access_key</tt> in <tt>conf/aws_settings.yml</tt>.</p>
371<p>Note that you can only read or download <tt>access key ID</tt> and <tt>secret access key</tt> once from your AWS console. If you forget them, you have to create new keys and delete the old ones.</p></li>
372
373<li>
374<p>Configure your ssh setting by editing <tt>~/.ssh/config</tt> and adding the following entry:</p>
375
376<div class="source">
377<div class="source">
378<pre> Host *.amazonaws.com
379 IdentityFile &lt;path_of_private_key&gt;
380</pre></div></div>
381<p>Note that &lt;path_of_private_key&gt; should be replaced by the path to the file that stores the private key for the key pair that you uploaded to AWS and used in <tt>conf/aws_settings</tt>. For example:</p>
382
383<div class="source">
384<div class="source">
385<pre> Host *.amazonaws.com
386 IdentityFile ~/.ssh/id_rsa
387</pre></div></div></li>
388</ul></div>
389<div class="section">
390<h2><a name="Cluster_Configuration"></a><a name="config" id="config">Cluster Configuration</a></h2>
391
392<ul>
393
394<li>
395<p><b>AWS settings</b>. Edit <tt>conf/instance_settings.yml</tt>. The meaning of each parameter is listed as follows:</p>
396
397<div class="source">
398<div class="source">
399<pre> # The OS image id for ec2 instances.
400 image: ami-76fa4116
401
402 # The data center region for ec2 instances.
403 region: us-west-2
404
405 # The tag for each ec2 machine. Use different tags for isolation.
406 tag: scale_test
407
408 # The name of a security group that appears in your AWS console.
409 group: default
410
411 # The name of a key pair that appears in your AWS console.
412 keypair: &lt;to be filled&gt;
413
414 # The AWS access key id for your IAM user.
415 access_key_id: &lt;to be filled&gt;
416
417 # The AWS secret key for your IAM user.
418 secret_access_key: &lt;to be filled&gt;
419
420 # The AWS instance type. A full list of available types are listed at:
421 # https://aws.amazon.com/ec2/instance-types/
422 instance_type: t2.micro
423
424 # The number of ec2 instances that construct a cluster.
425 count: 3
426
427 # The user name.
428 user: ec2-user
429
430 # Whether to reuse one slave machine to host the master process.
431 cc_on_nc: false
432</pre></div></div>
433<p><b>As described in <a href="#Prerequisites">prerequisites</a>, the following parameters must be customized:</b></p>
434
435<div class="source">
436<div class="source">
437<pre> # The tag for each ec2 machine. Use different tags for isolation.
438 tag: scale_test
439
440 # The name of a security group that appears in your AWS console.
441 group: default
442
443 # The name of a key pair that appears in your AWS console.
444 keypair: &lt;to be filled&gt;
445
446 # The AWS access key id for your IAM user.
447 access_key_id: &lt;to be filled&gt;
448
449 # The AWS secrety key for your IAM user.
450 secret_access_key: &lt;to be filled&gt;
451</pre></div></div></li>
452
453<li>
454<p><b>Remote working directories</b>. Edit <tt>conf/instance_settings.yml</tt> to change the remote binary directory (the variable &#x201c;binarydir&#x201d;) when necessary. By default, the binary directory will be under the home directory (as the value of Ansible builtin variable ansible_env.HOME) of the ssh user account on each node.</p></li>
455</ul></div>
456<div class="section">
457<h2><a name="Cluster_Lifecycle_Management"></a><a name="lifecycle" id="lifecycle">Cluster Lifecycle Management</a></h2>
458
459<ul>
460
461<li>
462<p>Allocate AWS EC2 nodes (the number of nodes is specified in <tt>conf/instance_settings.yml</tt>) and deploy the binary to all allocated EC2 nodes:</p>
463
464<div class="source">
465<div class="source">
466<pre> bin/deploy.sh
467</pre></div></div></li>
468
469<li>
470<p>Before starting the AsterixDB cluster, you the instance configuration file <tt>conf/instance/cc.conf</tt> can be modified with the exception of the IP addresses/DNS names which are are generated and cannot be changed. All available parameters and their usage can be found <a href="ncservice.html#Parameters">here</a>.</p></li>
471
472<li>
473<p>Launch your AsterixDB cluster on EC2:</p>
474
475<div class="source">
476<div class="source">
477<pre> bin/start.sh
478</pre></div></div>
479<p>Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node listed in <tt>conf/instance/inventory</tt> at port <tt>19001</tt> (which can be customized in <tt>conf/instance/cc.conf</tt>) in your browser.</p></li>
480
481<li>
482<p>If you want to stop the AWS-based AsterixDB cluster, run the following script:</p>
483
484<div class="source">
485<div class="source">
486<pre> bin/stop.sh
487</pre></div></div>
488<p>Note that this only stops AsterixDB but does not stop the EC2 nodes.</p></li>
489
490<li>
491<p>If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script:</p>
492
493<div class="source">
494<div class="source">
495<pre> bin/terminate.sh
496</pre></div></div>
497<p><b>Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes for the cluster.</b></p></li>
498</ul></div>
499 </div>
500 </div>
501 </div>
502
503 <hr/>
504
505 <footer>
506 <div class="container-fluid">
507 <div class="row span12">Copyright &copy; 2017
508 <a href="https://www.apache.org/">The Apache Software Foundation</a>.
509 All Rights Reserved.
510
511 </div>
512
513 <?xml version="1.0" encoding="UTF-8"?>
514<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
515 feather logo, and the Apache AsterixDB project logo are either
516 registered trademarks or trademarks of The Apache Software
517 Foundation in the United States and other countries.
518 All other marks mentioned may be trademarks or registered
519 trademarks of their respective owners.</div>
520
521
522 </div>
523 </footer>
524 </body>
525</html>