Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / 340445830e94147ab24a8ca38dd91710ee172981 / . / content / docs / 0.9.5 / aws.html
blob: 63d8b342fef8534787d00fb7b4a6f8ff7e55242b [file] [log] [blame]
Ian Maxona1cc51b2020-08-07 13:11:35 -0700[diff] [blame]1<!DOCTYPE html>
2<!--
3 | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aws.md at 2020-08-07
4 | Rendered using Apache Maven Fluido Skin 1.7
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
10 <meta name="Date-Revision-yyyymmdd" content="20200807" />
11 <meta http-equiv="Content-Language" content="en" />
12 <title>AsterixDB &#x2013; Installation using Amazon Web Services</title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16 <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
17
18 </head>
19 <body class="topBarDisabled">
20 <div class="container-fluid">
21 <div id="banner">
22 <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div>
23 <div class="pull-right"></div>
24 <div class="clear"><hr/></div>
25 </div>
26
27 <div id="breadcrumbs">
28 <ul class="breadcrumb">
29 <li id="publishDate">Last Published: 2020-08-07</li>
30 <li id="projectVersion" class="pull-right">Version: 0.9.5</li>
31 <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
32 </ul>
33 </div>
34 <div class="row-fluid">
35 <div id="leftColumn" class="span2">
36 <div class="well sidebar-nav">
37 <ul class="nav nav-list">
38 <li class="nav-header">Get Started - Installation</li>
39 <li><a href="ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
40 <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
41 <li class="active"><a href="#"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
42 <li class="nav-header">AsterixDB Primer</li>
43 <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
44 <li class="nav-header">Data Model</li>
45 <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
46 <li class="nav-header">Queries</li>
47 <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
48 <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
49 <li class="nav-header">API/SDK</li>
50 <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
51 <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
52 <li class="nav-header">Advanced Features</li>
53 <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
54 <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
55 <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
56 <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
57 <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
58 <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
59 <li class="nav-header">Deprecated</li>
60 <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
61 <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
62 <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
63</ul>
64 <hr />
65 <div id="poweredBy">
66 <div class="clear"></div>
67 <div class="clear"></div>
68 <div class="clear"></div>
69 <div class="clear"></div>
70<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a>
71 </div>
72 </div>
73 </div>
74 <div id="bodyColumn" class="span10" >
75<!--
76 ! Licensed to the Apache Software Foundation (ASF) under one
77 ! or more contributor license agreements. See the NOTICE file
78 ! distributed with this work for additional information
79 ! regarding copyright ownership. The ASF licenses this file
80 ! to you under the Apache License, Version 2.0 (the
81 ! "License"); you may not use this file except in compliance
82 ! with the License. You may obtain a copy of the License at
83 !
84 ! http://www.apache.org/licenses/LICENSE-2.0
85 !
86 ! Unless required by applicable law or agreed to in writing,
87 ! software distributed under the License is distributed on an
88 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
89 ! KIND, either express or implied. See the License for the
90 ! specific language governing permissions and limitations
91 ! under the License.
92 !-->
93<h1>Installation using Amazon Web Services</h1>
94<div class="section">
95<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
96<ul>
97
98<li><a href="#Introduction">Introduction</a></li>
99<li><a href="#Prerequisites">Prerequisites</a></li>
100<li><a href="#config">Cluster Configuration</a></li>
101<li><a href="#lifecycle">Cluster Lifecycle Management</a></li>
102</ul><!--
103 ! Licensed to the Apache Software Foundation (ASF) under one
104 ! or more contributor license agreements. See the NOTICE file
105 ! distributed with this work for additional information
106 ! regarding copyright ownership. The ASF licenses this file
107 ! to you under the Apache License, Version 2.0 (the
108 ! "License"); you may not use this file except in compliance
109 ! with the License. You may obtain a copy of the License at
110 !
111 ! http://www.apache.org/licenses/LICENSE-2.0
112 !
113 ! Unless required by applicable law or agreed to in writing,
114 ! software distributed under the License is distributed on an
115 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
116 ! KIND, either express or implied. See the License for the
117 ! specific language governing permissions and limitations
118 ! under the License.
119 !-->
120</div>
121<div class="section">
122<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
123<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p>
124<p>However, via this installation option, we provide a combo solution for automating both AWS EC2 and AsterixDB, where you can run only one script to deploy, start, stop, and terminate an AsterixDB cluster on AWS.</p></div>
125<div class="section">
126<h2><a name="Prerequisites" id="Prerequisites">Prerequisites</a></h2>
127<ul>
128
129<li>
130
131<p>Supported operating systems for the client: <b>Linux</b> and <b>MacOS</b></p>
132</li>
133<li>
134
135<p>Supported operating systems for Amazon Web Services instances: <b>Linux</b></p>
136</li>
137<li>
138
139<p>Install pip on your client machine:</p>
140<p>CentOS</p>
141
142<div>
143<div>
144<pre class="source"> $ sudo yum install python-pip
145</pre></div></div>
146
147<p>Ubuntu</p>
148
149<div>
150<div>
151<pre class="source"> $ sudo apt-get install python-pip
152</pre></div></div>
153
154<p>macOS</p>
155
156<div>
157<div>
158<pre class="source"> $ brew install pip
159</pre></div></div>
160</li>
161<li>
162
163<p>Install Ansible, boto, and boto3 on your client machine:</p>
164
165<div>
166<div>
167<pre class="source"> $ pip install ansible
168 $ pip install boto
169 $ pip install boto3
170</pre></div></div>
171
172<p>Note that you might need <tt>sudo</tt> depending on your system configuration.</p>
173<p><b>Make sure that the version of Ansible is no less than 2.2.1.0</b>:</p>
174
175<div>
176<div>
177<pre class="source"> $ ansible --version
178 ansible 2.2.1.0
179</pre></div></div>
180
181<p><b>For users with macOS 10.11+</b>, please create a user-level Ansible configuration file at:</p>
182
183<div>
184<div>
185<pre class="source"> ~/.ansible.cfg
186</pre></div></div>
187
188<p>and add the following configuration:</p>
189
190<div>
191<div>
192<pre class="source"> [ssh_connection]
193 control_path = %(directory)s/%%C
194</pre></div></div>
195</li>
196<li>
197
198<p>Download the AsterixDB distribution package, unzip it, navigate to <tt>opt/aws/</tt></p>
199
200<div>
201<div>
202<pre class="source"> $ cd opt/aws
203</pre></div></div>
204
205<p>The following files and directories are in the directory <tt>opt/aws</tt>:</p>
206
207<div>
208<div>
209<pre class="source"> README bin conf yaml
210</pre></div></div>
211
212<p><tt>bin</tt> contains scripts that start and terminate an AWS-based cluster instance, according to the configuration specified in files under <tt>conf</tt>, and <tt>yaml</tt> contains internal Ansible scripts that the shell scripts in <tt>bin</tt> use.</p>
213</li>
214<li>
215
216<p>Create an AWS account and an IAM user.</p>
217<p>Set up a security group that you&#x2019;d like to use for your AWS cluster. <b>The security group should at least allow all TCP connections from anywhere.</b> Provide the name of the security group as the value for the <tt>group</tt> field in <tt>conf/aws_settings.yml</tt>.</p>
218</li>
219<li>
220
221<p>Retrieve your AWS EC2 key pair name and use that as the <tt>keypair</tt> in <tt>conf/aws_settings.yml</tt>;</p>
222<p>retrieve your AWS IAM <tt>access key ID</tt> and use that as the <tt>access_key_id</tt> in <tt>conf/aws_settings.yml</tt>;</p>
223<p>retrieve your AWS IAM <tt>secret access key</tt> and use that as the <tt>secret_access_key</tt> in <tt>conf/aws_settings.yml</tt>.</p>
224<p>Note that you can only read or download <tt>access key ID</tt> and <tt>secret access key</tt> once from your AWS console. If you forget them, you have to create new keys and delete the old ones.</p>
225</li>
226<li>
227
228<p>Configure your ssh setting by editing <tt>~/.ssh/config</tt> and adding the following entry:</p>
229
230<div>
231<div>
232<pre class="source"> Host *.amazonaws.com
233 IdentityFile &lt;path_of_private_key&gt;
234</pre></div></div>
235
236<p>Note that &lt;path_of_private_key&gt; should be replaced by the path to the file that stores the private key for the key pair that you uploaded to AWS and used in <tt>conf/aws_settings</tt>. For example:</p>
237
238<div>
239<div>
240<pre class="source"> Host *.amazonaws.com
241 IdentityFile ~/.ssh/id_rsa
242</pre></div></div>
243</li>
244</ul></div>
245<div class="section">
246<h2><a name="Cluster_Configuration"></a><a name="config" id="config">Cluster Configuration</a></h2>
247<ul>
248
249<li>
250
251<p><b>AWS settings</b>. Edit <tt>conf/instance_settings.yml</tt>. The meaning of each parameter is listed as follows:</p>
252
253<div>
254<div>
255<pre class="source"> # The OS image id for ec2 instances.
256 image: ami-76fa4116
257
258 # The data center region for ec2 instances.
259 region: us-west-2
260
261 # The tag for each ec2 machine. Use different tags for isolation.
262 tag: scale_test
263
264 # The name of a security group that appears in your AWS console.
265 group: default
266
267 # The name of a key pair that appears in your AWS console.
268 keypair: &lt;to be filled&gt;
269
270 # The AWS access key id for your IAM user.
271 access_key_id: &lt;to be filled&gt;
272
273 # The AWS secret key for your IAM user.
274 secret_access_key: &lt;to be filled&gt;
275
276 # The AWS instance type. A full list of available types are listed at:
277 # https://aws.amazon.com/ec2/instance-types/
278 instance_type: t2.micro
279
280 # The number of ec2 instances that construct a cluster.
281 count: 3
282
283 # The user name.
284 user: ec2-user
285
286 # Whether to reuse one slave machine to host the master process.
287 cc_on_nc: false
288</pre></div></div>
289
290<p><b>As described in <a href="#Prerequisites">prerequisites</a>, the following parameters must be customized:</b></p>
291
292<div>
293<div>
294<pre class="source"> # The tag for each ec2 machine. Use different tags for isolation.
295 tag: scale_test
296
297 # The name of a security group that appears in your AWS console.
298 group: default
299
300 # The name of a key pair that appears in your AWS console.
301 keypair: &lt;to be filled&gt;
302
303 # The AWS access key id for your IAM user.
304 access_key_id: &lt;to be filled&gt;
305
306 # The AWS secrety key for your IAM user.
307 secret_access_key: &lt;to be filled&gt;
308</pre></div></div>
309</li>
310<li>
311
312<p><b>Remote working directories</b>. Edit <tt>conf/instance_settings.yml</tt> to change the remote binary directory (the variable &#x201c;binarydir&#x201d;) when necessary. By default, the binary directory will be under the home directory (as the value of Ansible builtin variable ansible_env.HOME) of the ssh user account on each node.</p>
313</li>
314</ul></div>
315<div class="section">
316<h2><a name="Cluster_Lifecycle_Management"></a><a name="lifecycle" id="lifecycle">Cluster Lifecycle Management</a></h2>
317<ul>
318
319<li>
320
321<p>Allocate AWS EC2 nodes (the number of nodes is specified in <tt>conf/instance_settings.yml</tt>) and deploy the binary to all allocated EC2 nodes:</p>
322
323<div>
324<div>
325<pre class="source"> bin/deploy.sh
326</pre></div></div>
327</li>
328<li>
329
330<p>Before starting the AsterixDB cluster, you the instance configuration file <tt>conf/instance/cc.conf</tt> can be modified with the exception of the IP addresses/DNS names which are are generated and cannot be changed. All available parameters and their usage can be found <a href="ncservice.html#Parameters">here</a>.</p>
331</li>
332<li>
333
334<p>Launch your AsterixDB cluster on EC2:</p>
335
336<div>
337<div>
338<pre class="source"> bin/start.sh
339</pre></div></div>
340
341<p>Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node listed in <tt>conf/instance/inventory</tt> at port <tt>19001</tt> (which can be customized in <tt>conf/instance/cc.conf</tt>) in your browser.</p>
342</li>
343<li>
344
345<p>If you want to stop the AWS-based AsterixDB cluster, run the following script:</p>
346
347<div>
348<div>
349<pre class="source"> bin/stop.sh
350</pre></div></div>
351
352<p>Note that this only stops AsterixDB but does not stop the EC2 nodes.</p>
353</li>
354<li>
355
356<p>If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script:</p>
357
358<div>
359<div>
360<pre class="source"> bin/terminate.sh
361</pre></div></div>
362
363<p><b>Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes for the cluster.</b></p>
364</li>
365</ul></div>
366 </div>
367 </div>
368 </div>
369 <hr/>
370 <footer>
371 <div class="container-fluid">
372 <div class="row-fluid">
373<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
374 feather logo, and the Apache AsterixDB project logo are either
375 registered trademarks or trademarks of The Apache Software
376 Foundation in the United States and other countries.
377 All other marks mentioned may be trademarks or registered
378 trademarks of their respective owners.
379 </div>
380 </div>
381 </div>
382 </footer>
383 </body>
384</html>