Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / 3355d4c7a24401fa5f77a4200b9787c9f7c8a61b / . / content / docs / 0.9.4.1 / aws.html
blob: 449ac703224ed4e27400b48a21ccb803cf336b10 [file] [log] [blame]
Ian Maxon41b806c2019-03-07 15:58:20 -0800[diff] [blame]1<!DOCTYPE html>
2<!--
3 | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aws.md at 2019-03-07
4 | Rendered using Apache Maven Fluido Skin 1.7
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
10 <meta name="Date-Revision-yyyymmdd" content="20190307" />
11 <meta http-equiv="Content-Language" content="en" />
12 <title>AsterixDB &#x2013; </title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16 <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
17
18 </head>
19 <body class="topBarDisabled">
20 <div class="container-fluid">
21 <div id="banner">
22 <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div>
23 <div class="pull-right"></div>
24 <div class="clear"><hr/></div>
25 </div>
26
27 <div id="breadcrumbs">
28 <ul class="breadcrumb">
29 <li id="publishDate">Last Published: 2019-03-07</li>
30 <li id="projectVersion" class="pull-right">Version: 0.9.4.1</li>
31 <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
32 </ul>
33 </div>
34 <div class="row-fluid">
35 <div id="leftColumn" class="span2">
36 <div class="well sidebar-nav">
37 <ul class="nav nav-list">
38 <li class="nav-header">Get Started - Installation</li>
39 <li><a href="ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
40 <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
41 <li class="active"><a href="#"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
42 <li class="nav-header">AsterixDB Primer</li>
43 <li><a href="sqlpp/primer-sqlpp.html" title="Option 1: using SQL++"><span class="none"></span>Option 1: using SQL++</a></li>
44 <li><a href="aql/primer.html" title="Option 2: using AQL"><span class="none"></span>Option 2: using AQL</a></li>
45 <li class="nav-header">Data Model</li>
46 <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
47 <li class="nav-header">Queries - SQL++</li>
48 <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
49 <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
50 <li class="nav-header">Queries - AQL</li>
51 <li><a href="aql/manual.html" title="The Asterix Query Language (AQL)"><span class="none"></span>The Asterix Query Language (AQL)</a></li>
52 <li><a href="aql/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
53 <li class="nav-header">API/SDK</li>
54 <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
55 <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
56 <li class="nav-header">Advanced Features</li>
57 <li><a href="aql/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
58 <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
59 <li><a href="feeds/tutorial.html" title="Support for Data Ingestion"><span class="none"></span>Support for Data Ingestion</a></li>
60 <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
61 <li><a href="aql/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
62 <li><a href="aql/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
63</ul>
64 <hr />
65 <div id="poweredBy">
66 <div class="clear"></div>
67 <div class="clear"></div>
68 <div class="clear"></div>
69 <div class="clear"></div>
70<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a>
71 </div>
72 </div>
73 </div>
74 <div id="bodyColumn" class="span10" >
75<!--
76 ! Licensed to the Apache Software Foundation (ASF) under one
77 ! or more contributor license agreements. See the NOTICE file
78 ! distributed with this work for additional information
79 ! regarding copyright ownership. The ASF licenses this file
80 ! to you under the Apache License, Version 2.0 (the
81 ! "License"); you may not use this file except in compliance
82 ! with the License. You may obtain a copy of the License at
83 !
84 ! http://www.apache.org/licenses/LICENSE-2.0
85 !
86 ! Unless required by applicable law or agreed to in writing,
87 ! software distributed under the License is distributed on an
88 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
89 ! KIND, either express or implied. See the License for the
90 ! specific language governing permissions and limitations
91 ! under the License.
92 !-->
93<ul>
94
95<li><a href="#Introduction">Introduction</a></li>
96<li><a href="#Prerequisites">Prerequisites</a></li>
97<li><a href="#config">Cluster Configuration</a></li>
98<li><a href="#lifecycle">Cluster Lifecycle Management</a></li>
99</ul><!--
100 ! Licensed to the Apache Software Foundation (ASF) under one
101 ! or more contributor license agreements. See the NOTICE file
102 ! distributed with this work for additional information
103 ! regarding copyright ownership. The ASF licenses this file
104 ! to you under the Apache License, Version 2.0 (the
105 ! "License"); you may not use this file except in compliance
106 ! with the License. You may obtain a copy of the License at
107 !
108 ! http://www.apache.org/licenses/LICENSE-2.0
109 !
110 ! Unless required by applicable law or agreed to in writing,
111 ! software distributed under the License is distributed on an
112 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
113 ! KIND, either express or implied. See the License for the
114 ! specific language governing permissions and limitations
115 ! under the License.
116 !-->
117
118<div class="section">
119<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
120<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p>
121<p>However, via this installation option, we provide a combo solution for automating both AWS EC2 and AsterixDB, where you can run only one script to deploy, start, stop, and terminate an AsterixDB cluster on AWS.</p></div>
122<div class="section">
123<h2><a name="Prerequisites" id="Prerequisites">Prerequisites</a></h2>
124<ul>
125
126<li>
127
128<p>Supported operating systems for the client: <b>Linux</b> and <b>MacOS</b></p>
129</li>
130<li>
131
132<p>Supported operating systems for Amazon Web Services instances: <b>Linux</b></p>
133</li>
134<li>
135
136<p>Install pip on your client machine:</p>
137<p>CentOS</p>
138
139<div>
140<div>
141<pre class="source"> $ sudo yum install python-pip
142</pre></div></div>
143
144<p>Ubuntu</p>
145
146<div>
147<div>
148<pre class="source"> $ sudo apt-get install python-pip
149</pre></div></div>
150
151<p>macOS</p>
152
153<div>
154<div>
155<pre class="source"> $ brew install pip
156</pre></div></div>
157</li>
158<li>
159
160<p>Install Ansible, boto, and boto3 on your client machine:</p>
161
162<div>
163<div>
164<pre class="source"> $ pip install ansible
165 $ pip install boto
166 $ pip install boto3
167</pre></div></div>
168
169<p>Note that you might need <tt>sudo</tt> depending on your system configuration.</p>
170<p><b>Make sure that the version of Ansible is no less than 2.2.1.0</b>:</p>
171
172<div>
173<div>
174<pre class="source"> $ ansible --version
175 ansible 2.2.1.0
176</pre></div></div>
177
178<p><b>For users with macOS 10.11+</b>, please create a user-level Ansible configuration file at:</p>
179
180<div>
181<div>
182<pre class="source"> ~/.ansible.cfg
183</pre></div></div>
184
185<p>and add the following configuration:</p>
186
187<div>
188<div>
189<pre class="source"> [ssh_connection]
190 control_path = %(directory)s/%%C
191</pre></div></div>
192</li>
193<li>
194
195<p>Download the AsterixDB distribution package, unzip it, navigate to <tt>opt/aws/</tt></p>
196
197<div>
198<div>
199<pre class="source"> $ cd opt/aws
200</pre></div></div>
201
202<p>The following files and directories are in the directory <tt>opt/aws</tt>:</p>
203
204<div>
205<div>
206<pre class="source"> README bin conf yaml
207</pre></div></div>
208
209<p><tt>bin</tt> contains scripts that start and terminate an AWS-based cluster instance, according to the configuration specified in files under <tt>conf</tt>, and <tt>yaml</tt> contains internal Ansible scripts that the shell scripts in <tt>bin</tt> use.</p>
210</li>
211<li>
212
213<p>Create an AWS account and an IAM user.</p>
214<p>Set up a security group that you&#x2019;d like to use for your AWS cluster. <b>The security group should at least allow all TCP connections from anywhere.</b> Provide the name of the security group as the value for the <tt>group</tt> field in <tt>conf/aws_settings.yml</tt>.</p>
215</li>
216<li>
217
218<p>Retrieve your AWS EC2 key pair name and use that as the <tt>keypair</tt> in <tt>conf/aws_settings.yml</tt>;</p>
219<p>retrieve your AWS IAM <tt>access key ID</tt> and use that as the <tt>access_key_id</tt> in <tt>conf/aws_settings.yml</tt>;</p>
220<p>retrieve your AWS IAM <tt>secret access key</tt> and use that as the <tt>secret_access_key</tt> in <tt>conf/aws_settings.yml</tt>.</p>
221<p>Note that you can only read or download <tt>access key ID</tt> and <tt>secret access key</tt> once from your AWS console. If you forget them, you have to create new keys and delete the old ones.</p>
222</li>
223<li>
224
225<p>Configure your ssh setting by editing <tt>~/.ssh/config</tt> and adding the following entry:</p>
226
227<div>
228<div>
229<pre class="source"> Host *.amazonaws.com
230 IdentityFile &lt;path_of_private_key&gt;
231</pre></div></div>
232
233<p>Note that &lt;path_of_private_key&gt; should be replaced by the path to the file that stores the private key for the key pair that you uploaded to AWS and used in <tt>conf/aws_settings</tt>. For example:</p>
234
235<div>
236<div>
237<pre class="source"> Host *.amazonaws.com
238 IdentityFile ~/.ssh/id_rsa
239</pre></div></div>
240</li>
241</ul></div>
242<div class="section">
243<h2><a name="Cluster_Configuration"></a><a name="config" id="config">Cluster Configuration</a></h2>
244<ul>
245
246<li>
247
248<p><b>AWS settings</b>. Edit <tt>conf/instance_settings.yml</tt>. The meaning of each parameter is listed as follows:</p>
249
250<div>
251<div>
252<pre class="source"> # The OS image id for ec2 instances.
253 image: ami-76fa4116
254
255 # The data center region for ec2 instances.
256 region: us-west-2
257
258 # The tag for each ec2 machine. Use different tags for isolation.
259 tag: scale_test
260
261 # The name of a security group that appears in your AWS console.
262 group: default
263
264 # The name of a key pair that appears in your AWS console.
265 keypair: &lt;to be filled&gt;
266
267 # The AWS access key id for your IAM user.
268 access_key_id: &lt;to be filled&gt;
269
270 # The AWS secret key for your IAM user.
271 secret_access_key: &lt;to be filled&gt;
272
273 # The AWS instance type. A full list of available types are listed at:
274 # https://aws.amazon.com/ec2/instance-types/
275 instance_type: t2.micro
276
277 # The number of ec2 instances that construct a cluster.
278 count: 3
279
280 # The user name.
281 user: ec2-user
282
283 # Whether to reuse one slave machine to host the master process.
284 cc_on_nc: false
285</pre></div></div>
286
287<p><b>As described in <a href="#Prerequisites">prerequisites</a>, the following parameters must be customized:</b></p>
288
289<div>
290<div>
291<pre class="source"> # The tag for each ec2 machine. Use different tags for isolation.
292 tag: scale_test
293
294 # The name of a security group that appears in your AWS console.
295 group: default
296
297 # The name of a key pair that appears in your AWS console.
298 keypair: &lt;to be filled&gt;
299
300 # The AWS access key id for your IAM user.
301 access_key_id: &lt;to be filled&gt;
302
303 # The AWS secrety key for your IAM user.
304 secret_access_key: &lt;to be filled&gt;
305</pre></div></div>
306</li>
307<li>
308
309<p><b>Remote working directories</b>. Edit <tt>conf/instance_settings.yml</tt> to change the remote binary directory (the variable &#x201c;binarydir&#x201d;) when necessary. By default, the binary directory will be under the home directory (as the value of Ansible builtin variable ansible_env.HOME) of the ssh user account on each node.</p>
310</li>
311</ul></div>
312<div class="section">
313<h2><a name="Cluster_Lifecycle_Management"></a><a name="lifecycle" id="lifecycle">Cluster Lifecycle Management</a></h2>
314<ul>
315
316<li>
317
318<p>Allocate AWS EC2 nodes (the number of nodes is specified in <tt>conf/instance_settings.yml</tt>) and deploy the binary to all allocated EC2 nodes:</p>
319
320<div>
321<div>
322<pre class="source"> bin/deploy.sh
323</pre></div></div>
324</li>
325<li>
326
327<p>Before starting the AsterixDB cluster, you the instance configuration file <tt>conf/instance/cc.conf</tt> can be modified with the exception of the IP addresses/DNS names which are are generated and cannot be changed. All available parameters and their usage can be found <a href="ncservice.html#Parameters">here</a>.</p>
328</li>
329<li>
330
331<p>Launch your AsterixDB cluster on EC2:</p>
332
333<div>
334<div>
335<pre class="source"> bin/start.sh
336</pre></div></div>
337
338<p>Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node listed in <tt>conf/instance/inventory</tt> at port <tt>19001</tt> (which can be customized in <tt>conf/instance/cc.conf</tt>) in your browser.</p>
339</li>
340<li>
341
342<p>If you want to stop the AWS-based AsterixDB cluster, run the following script:</p>
343
344<div>
345<div>
346<pre class="source"> bin/stop.sh
347</pre></div></div>
348
349<p>Note that this only stops AsterixDB but does not stop the EC2 nodes.</p>
350</li>
351<li>
352
353<p>If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script:</p>
354
355<div>
356<div>
357<pre class="source"> bin/terminate.sh
358</pre></div></div>
359
360<p><b>Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes for the cluster.</b></p>
361</li>
362</ul></div>
363 </div>
364 </div>
365 </div>
366 <hr/>
367 <footer>
368 <div class="container-fluid">
369 <div class="row-fluid">
370<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
371 feather logo, and the Apache AsterixDB project logo are either
372 registered trademarks or trademarks of The Apache Software
373 Foundation in the United States and other countries.
374 All other marks mentioned may be trademarks or registered
375 trademarks of their respective owners.
376 </div>
377 </div>
378 </div>
379 </footer>
380 </body>
381</html>