Gitiles
Code Review Sign In
asterix-gerrit.ics.uci.edu / incubator-asterixdb-site / refs/heads/asf-site / . / content / docs / 0.9.8 / aws.html
blob: 56387cc5ad63171c7ada9f0d429d566a371c143c [file] [log] [blame]
Ian Maxon858061a2022-05-12 19:11:28 -0700[diff] [blame]1<!DOCTYPE html>
2<!--
3 | Generated by Apache Maven Doxia Site Renderer 1.8.1 from target/generated-site/markdown/aws.md at 2022-05-12
4 | Rendered using Apache Maven Fluido Skin 1.7
5-->
6<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta charset="UTF-8" />
9 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
10 <meta name="Date-Revision-yyyymmdd" content="20220512" />
11 <meta http-equiv="Content-Language" content="en" />
12 <title>AsterixDB &#x2013; Installation using Amazon Web Services</title>
13 <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
14 <link rel="stylesheet" href="./css/site.css" />
15 <link rel="stylesheet" href="./css/print.css" media="print" />
16 <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
17
18 </head>
19 <body class="topBarDisabled">
20 <div class="container-fluid">
21 <div id="banner">
22 <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/asterixlogo.png" alt="AsterixDB"/></a></div>
23 <div class="pull-right"></div>
24 <div class="clear"><hr/></div>
25 </div>
26
27 <div id="breadcrumbs">
28 <ul class="breadcrumb">
29 <li id="publishDate">Last Published: 2022-05-12</li>
30 <li id="projectVersion" class="pull-right">Version: 0.9.8</li>
31 <li class="pull-right"><a href="index.html" title="Documentation Home">Documentation Home</a></li>
32 </ul>
33 </div>
34 <div class="row-fluid">
35 <div id="leftColumn" class="span2">
36 <div class="well sidebar-nav">
37 <ul class="nav nav-list">
38 <li class="nav-header">Get Started - Installation</li>
39 <li><a href="ncservice.html" title="Option 1: using NCService"><span class="none"></span>Option 1: using NCService</a></li>
40 <li><a href="ansible.html" title="Option 2: using Ansible"><span class="none"></span>Option 2: using Ansible</a></li>
41 <li class="active"><a href="#"><span class="none"></span>Option 3: using Amazon Web Services</a></li>
42 <li class="nav-header">AsterixDB Primer</li>
43 <li><a href="sqlpp/primer-sqlpp.html" title="Using SQL++"><span class="none"></span>Using SQL++</a></li>
44 <li class="nav-header">Data Model</li>
45 <li><a href="datamodel.html" title="The Asterix Data Model"><span class="none"></span>The Asterix Data Model</a></li>
46 <li class="nav-header">Queries</li>
47 <li><a href="sqlpp/manual.html" title="The SQL++ Query Language"><span class="none"></span>The SQL++ Query Language</a></li>
48 <li><a href="SQLPP.html" title="Raw SQL++ Grammar"><span class="none"></span>Raw SQL++ Grammar</a></li>
49 <li><a href="sqlpp/builtins.html" title="Builtin Functions"><span class="none"></span>Builtin Functions</a></li>
50 <li class="nav-header">API/SDK</li>
51 <li><a href="api.html" title="HTTP API"><span class="none"></span>HTTP API</a></li>
52 <li><a href="csv.html" title="CSV Output"><span class="none"></span>CSV Output</a></li>
53 <li class="nav-header">Advanced Features</li>
54 <li><a href="aql/externaldata.html" title="Accessing External Data"><span class="none"></span>Accessing External Data</a></li>
55 <li><a href="feeds.html" title="Data Ingestion with Feeds"><span class="none"></span>Data Ingestion with Feeds</a></li>
56 <li><a href="udf.html" title="User Defined Functions"><span class="none"></span>User Defined Functions</a></li>
57 <li><a href="sqlpp/filters.html" title="Filter-Based LSM Index Acceleration"><span class="none"></span>Filter-Based LSM Index Acceleration</a></li>
58 <li><a href="sqlpp/fulltext.html" title="Support of Full-text Queries"><span class="none"></span>Support of Full-text Queries</a></li>
59 <li><a href="sqlpp/similarity.html" title="Support of Similarity Queries"><span class="none"></span>Support of Similarity Queries</a></li>
60 <li><a href="geo/quickstart.html" title="GIS Support Overview"><span class="none"></span>GIS Support Overview</a></li>
61 <li><a href="geo/functions.html" title="GIS Functions"><span class="none"></span>GIS Functions</a></li>
62 <li><a href="interval_join.html" title="Support of Interval Joins"><span class="none"></span>Support of Interval Joins</a></li>
63 <li><a href="spatial_join.html" title="Support of Spatial Joins"><span class="none"></span>Support of Spatial Joins</a></li>
64 <li><a href="sqlpp/arrayindex.html" title="Support of Array Indexes"><span class="none"></span>Support of Array Indexes</a></li>
65 <li class="nav-header">Deprecated</li>
66 <li><a href="aql/primer.html" title="AsterixDB Primer: Using AQL"><span class="none"></span>AsterixDB Primer: Using AQL</a></li>
67 <li><a href="aql/manual.html" title="Queries: The Asterix Query Language (AQL)"><span class="none"></span>Queries: The Asterix Query Language (AQL)</a></li>
68 <li><a href="aql/builtins.html" title="Queries: Builtin Functions (AQL)"><span class="none"></span>Queries: Builtin Functions (AQL)</a></li>
69</ul>
70 <hr />
71 <div id="poweredBy">
72 <div class="clear"></div>
73 <div class="clear"></div>
74 <div class="clear"></div>
75 <div class="clear"></div>
76<a href="./" title="AsterixDB" class="builtBy"><img class="builtBy" alt="AsterixDB" src="images/asterixlogo.png" /></a>
77 </div>
78 </div>
79 </div>
80 <div id="bodyColumn" class="span10" >
81<!--
82 ! Licensed to the Apache Software Foundation (ASF) under one
83 ! or more contributor license agreements. See the NOTICE file
84 ! distributed with this work for additional information
85 ! regarding copyright ownership. The ASF licenses this file
86 ! to you under the Apache License, Version 2.0 (the
87 ! "License"); you may not use this file except in compliance
88 ! with the License. You may obtain a copy of the License at
89 !
90 ! http://www.apache.org/licenses/LICENSE-2.0
91 !
92 ! Unless required by applicable law or agreed to in writing,
93 ! software distributed under the License is distributed on an
94 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
95 ! KIND, either express or implied. See the License for the
96 ! specific language governing permissions and limitations
97 ! under the License.
98 !-->
99<h1>Installation using Amazon Web Services</h1>
100<div class="section">
101<h2><a name="Table_of_Contents"></a><a name="atoc" id="#toc">Table of Contents</a></h2>
102<ul>
103
104<li><a href="#Introduction">Introduction</a></li>
105<li><a href="#Prerequisites">Prerequisites</a></li>
106<li><a href="#config">Cluster Configuration</a></li>
107<li><a href="#lifecycle">Cluster Lifecycle Management</a></li>
108</ul><!--
109 ! Licensed to the Apache Software Foundation (ASF) under one
110 ! or more contributor license agreements. See the NOTICE file
111 ! distributed with this work for additional information
112 ! regarding copyright ownership. The ASF licenses this file
113 ! to you under the Apache License, Version 2.0 (the
114 ! "License"); you may not use this file except in compliance
115 ! with the License. You may obtain a copy of the License at
116 !
117 ! http://www.apache.org/licenses/LICENSE-2.0
118 !
119 ! Unless required by applicable law or agreed to in writing,
120 ! software distributed under the License is distributed on an
121 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
122 ! KIND, either express or implied. See the License for the
123 ! specific language governing permissions and limitations
124 ! under the License.
125 !-->
126</div>
127<div class="section">
128<h2><a name="Introduction" id="Introduction">Introduction</a></h2>
129<p>Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the Ansible cluster installation scripts as described <a href="ansible.html">here</a> separately to manage the lifecycle of an AsterixDB cluster on those EC2 instances.</p>
130<p>However, via this installation option, we provide a combo solution for automating both AWS EC2 and AsterixDB, where you can run only one script to deploy, start, stop, and terminate an AsterixDB cluster on AWS.</p></div>
131<div class="section">
132<h2><a name="Prerequisites" id="Prerequisites">Prerequisites</a></h2>
133<ul>
134
135<li>
136
137<p>Supported operating systems for the client: <b>Linux</b> and <b>MacOS</b></p>
138</li>
139<li>
140
141<p>Supported operating systems for Amazon Web Services instances: <b>Linux</b></p>
142</li>
143<li>
144
145<p>Install pip on your client machine:</p>
146<p>CentOS</p>
147
148<div>
149<div>
150<pre class="source"> $ sudo yum install python-pip
151</pre></div></div>
152
153<p>Ubuntu</p>
154
155<div>
156<div>
157<pre class="source"> $ sudo apt-get install python-pip
158</pre></div></div>
159
160<p>macOS</p>
161
162<div>
163<div>
164<pre class="source"> $ brew install pip
165</pre></div></div>
166</li>
167<li>
168
169<p>Install Ansible, boto, and boto3 on your client machine:</p>
170
171<div>
172<div>
173<pre class="source"> $ pip install ansible
174 $ pip install boto
175 $ pip install boto3
176</pre></div></div>
177
178<p>Note that you might need <tt>sudo</tt> depending on your system configuration.</p>
179<p><b>Make sure that the version of Ansible is no less than 2.2.1.0</b>:</p>
180
181<div>
182<div>
183<pre class="source"> $ ansible --version
184 ansible 2.2.1.0
185</pre></div></div>
186
187<p><b>For users with macOS 10.11+</b>, please create a user-level Ansible configuration file at:</p>
188
189<div>
190<div>
191<pre class="source"> ~/.ansible.cfg
192</pre></div></div>
193
194<p>and add the following configuration:</p>
195
196<div>
197<div>
198<pre class="source"> [ssh_connection]
199 control_path = %(directory)s/%%C
200</pre></div></div>
201</li>
202<li>
203
204<p>Download the AsterixDB distribution package, unzip it, navigate to <tt>opt/aws/</tt></p>
205
206<div>
207<div>
208<pre class="source"> $ cd opt/aws
209</pre></div></div>
210
211<p>The following files and directories are in the directory <tt>opt/aws</tt>:</p>
212
213<div>
214<div>
215<pre class="source"> README bin conf yaml
216</pre></div></div>
217
218<p><tt>bin</tt> contains scripts that start and terminate an AWS-based cluster instance, according to the configuration specified in files under <tt>conf</tt>, and <tt>yaml</tt> contains internal Ansible scripts that the shell scripts in <tt>bin</tt> use.</p>
219</li>
220<li>
221
222<p>Create an AWS account and an IAM user.</p>
223<p>Set up a security group that you&#x2019;d like to use for your AWS cluster. <b>The security group should at least allow all TCP connections from anywhere.</b> Provide the name of the security group as the value for the <tt>group</tt> field in <tt>conf/aws_settings.yml</tt>.</p>
224</li>
225<li>
226
227<p>Retrieve your AWS EC2 key pair name and use that as the <tt>keypair</tt> in <tt>conf/aws_settings.yml</tt>;</p>
228<p>retrieve your AWS IAM <tt>access key ID</tt> and use that as the <tt>access_key_id</tt> in <tt>conf/aws_settings.yml</tt>;</p>
229<p>retrieve your AWS IAM <tt>secret access key</tt> and use that as the <tt>secret_access_key</tt> in <tt>conf/aws_settings.yml</tt>.</p>
230<p>Note that you can only read or download <tt>access key ID</tt> and <tt>secret access key</tt> once from your AWS console. If you forget them, you have to create new keys and delete the old ones.</p>
231</li>
232<li>
233
234<p>Configure your ssh setting by editing <tt>~/.ssh/config</tt> and adding the following entry:</p>
235
236<div>
237<div>
238<pre class="source"> Host *.amazonaws.com
239 IdentityFile &lt;path_of_private_key&gt;
240</pre></div></div>
241
242<p>Note that &lt;path_of_private_key&gt; should be replaced by the path to the file that stores the private key for the key pair that you uploaded to AWS and used in <tt>conf/aws_settings</tt>. For example:</p>
243
244<div>
245<div>
246<pre class="source"> Host *.amazonaws.com
247 IdentityFile ~/.ssh/id_rsa
248</pre></div></div>
249</li>
250</ul></div>
251<div class="section">
252<h2><a name="Cluster_Configuration"></a><a name="config" id="config">Cluster Configuration</a></h2>
253<ul>
254
255<li>
256
257<p><b>AWS settings</b>. Edit <tt>conf/instance_settings.yml</tt>. The meaning of each parameter is listed as follows:</p>
258
259<div>
260<div>
261<pre class="source"> # The OS image id for ec2 instances.
262 image: ami-76fa4116
263
264 # The data center region for ec2 instances.
265 region: us-west-2
266
267 # The tag for each ec2 machine. Use different tags for isolation.
268 tag: scale_test
269
270 # The name of a security group that appears in your AWS console.
271 group: default
272
273 # The name of a key pair that appears in your AWS console.
274 keypair: &lt;to be filled&gt;
275
276 # The AWS access key id for your IAM user.
277 access_key_id: &lt;to be filled&gt;
278
279 # The AWS secret key for your IAM user.
280 secret_access_key: &lt;to be filled&gt;
281
282 # The AWS instance type. A full list of available types are listed at:
283 # https://aws.amazon.com/ec2/instance-types/
284 instance_type: t2.micro
285
286 # The number of ec2 instances that construct a cluster.
287 count: 3
288
289 # The user name.
290 user: ec2-user
291
292 # Whether to reuse one slave machine to host the master process.
293 cc_on_nc: false
294</pre></div></div>
295
296<p><b>As described in <a href="#Prerequisites">prerequisites</a>, the following parameters must be customized:</b></p>
297
298<div>
299<div>
300<pre class="source"> # The tag for each ec2 machine. Use different tags for isolation.
301 tag: scale_test
302
303 # The name of a security group that appears in your AWS console.
304 group: default
305
306 # The name of a key pair that appears in your AWS console.
307 keypair: &lt;to be filled&gt;
308
309 # The AWS access key id for your IAM user.
310 access_key_id: &lt;to be filled&gt;
311
312 # The AWS secrety key for your IAM user.
313 secret_access_key: &lt;to be filled&gt;
314</pre></div></div>
315</li>
316<li>
317
318<p><b>Remote working directories</b>. Edit <tt>conf/instance_settings.yml</tt> to change the remote binary directory (the variable &#x201c;binarydir&#x201d;) when necessary. By default, the binary directory will be under the home directory (as the value of Ansible builtin variable ansible_env.HOME) of the ssh user account on each node.</p>
319</li>
320</ul></div>
321<div class="section">
322<h2><a name="Cluster_Lifecycle_Management"></a><a name="lifecycle" id="lifecycle">Cluster Lifecycle Management</a></h2>
323<ul>
324
325<li>
326
327<p>Allocate AWS EC2 nodes (the number of nodes is specified in <tt>conf/instance_settings.yml</tt>) and deploy the binary to all allocated EC2 nodes:</p>
328
329<div>
330<div>
331<pre class="source"> bin/deploy.sh
332</pre></div></div>
333</li>
334<li>
335
336<p>Before starting the AsterixDB cluster, you the instance configuration file <tt>conf/instance/cc.conf</tt> can be modified with the exception of the IP addresses/DNS names which are are generated and cannot be changed. All available parameters and their usage can be found <a href="ncservice.html#Parameters">here</a>.</p>
337</li>
338<li>
339
340<p>Launch your AsterixDB cluster on EC2:</p>
341
342<div>
343<div>
344<pre class="source"> bin/start.sh
345</pre></div></div>
346
347<p>Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node listed in <tt>conf/instance/inventory</tt> at port <tt>19001</tt> (which can be customized in <tt>conf/instance/cc.conf</tt>) in your browser.</p>
348</li>
349<li>
350
351<p>If you want to stop the AWS-based AsterixDB cluster, run the following script:</p>
352
353<div>
354<div>
355<pre class="source"> bin/stop.sh
356</pre></div></div>
357
358<p>Note that this only stops AsterixDB but does not stop the EC2 nodes.</p>
359</li>
360<li>
361
362<p>If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script:</p>
363
364<div>
365<div>
366<pre class="source"> bin/terminate.sh
367</pre></div></div>
368
369<p><b>Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes for the cluster.</b></p>
370</li>
371</ul></div>
372 </div>
373 </div>
374 </div>
375 <hr/>
376 <footer>
377 <div class="container-fluid">
378 <div class="row-fluid">
379<div class="row-fluid">Apache AsterixDB, AsterixDB, Apache, the Apache
380 feather logo, and the Apache AsterixDB project logo are either
381 registered trademarks or trademarks of The Apache Software
382 Foundation in the United States and other countries.
383 All other marks mentioned may be trademarks or registered
384 trademarks of their respective owners.
385 </div>
386 </div>
387 </div>
388 </footer>
389 </body>
390</html>