Ambari部署 Ambari部署

Ambari部署 Ambari简介： Apache Ambari是一种基于Web的工具，支持Apache Hadoop集群的供应、管理和监控。Ambari目前已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeper、Sqoop和Hcatalog等。
Apache Ambari 支持HDFS、MapReduce、Hive、Pig、Hbase、Zookeper、Sqoop和Hcatalog等的集中管理。也是5个顶级hadoop管理工具之一。

文章图片
Ambari界面预览部署环境：CentOS 6/7 部署步骤：一、环境准备 1.关闭防火墙 CentOS 6 :

chkconfig iptables off service iptables stop chkconfig --list iptables

CentOS 7：

systemctl disable firewalld.service systemctl stop firewalld.service

2.关闭selinux

setenforce 0(临时关闭) vi /etc/selinux/config SELINUX=disabled(永久关闭，重启后生效)

3.修改主机名 CentOS 6:

vi /etc/sysconfig/network NETWORKING=yes HOSTNAME=namenode-1

Centos 7:

hostnamectl set-hostname namenode-1

4.配置本地域名解析（以下为示例）

192.168.5.1namenode-1 192.168.5.2datanode-1 192.168.5.3web1 192.168.5.4web2 192.168.5.5datanode-2 192.168.5.6datanode-3 192.168.5.7datanode-4 192.168.5.8ambari

5.配置SSH（Ambari节点可无密码访问其他主机）

ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keysssh-copy-id -i /root/.ssh/id_rsa.pub root@namenode-1 ... ...

6.修改YUM源这里使用阿里云镜像已保证安装速度

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bk wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo yum makecache

7.安装JDK

HADOOP环境需要的JAVA为1.7以上版本的JDK，这里使用1.8版本的JAVA 上传1.8版本的JDK包 rpm -ivh jdk-8u131-linux-x64.rpm 配置环境变量 vim /etc/profile（文件末尾添加） export JAVA_HOME=/usr/java/jdk1.8.0_131 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 验证是否安装成功 java -version java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

8.安装时间同步服务（NTP），确保集群内部时间统一 CentOS 6:

yum install -y ntp chkconfig --list ntpd chkconfig ntpd on service ntpd start

CentOS 7:

yum install -y ntp chkconfig --list ntpd systemctl is-enabled ntpd systemctl enabled ntpd systemctl start ntpd

9.关闭THP服务（如果不关闭transparent_hugepage，Ambari在检查环境是也会进行警告，HDFS性能会因为THP服务未关闭受到严重的影响）永久关闭（重启后生效）：
CentOS 6:

vi /etc/grub.conf 添加 transparent_hugepage=never vi /etc/rc.local 添加： if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi exit 0

CentOS 7:

vi /etc/default/grub 在有CMDLINE一行添加 transparent_hugepage=never grub2-mkconfig -o /boot/grub2/grub.cfg systemctl disable tuned

临时关闭（不需要重启）：
CentOS 6:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never]

CentOS 7:

echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag

检查方法：

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled 有 [never]则表示THP被禁用

10.配置UMASK，设定用户创建目录的初始权限

umask 0022

11.禁止离线更新

vi /etc/yum/pluginconf.d/refresh-packagekit 添加enbaled=0

12.备份克隆（略）以上的环境配置是最基本的，每个节点都需要的配置。可以做一备份节点，用于将来创建子节点克隆使用，减少重复工作。当然克隆后还有小的调整，比如要重新配置hostname，还有SSH配置过程里，分发主节点里配置好的authorized_keys到各从节点，还是在克隆所有子节点后再操作。
二、安装Ambari集群 1.制作本地源使用官方默认源的话十分缓慢，因此建议使用本地源进行安装（本地源文件较大、下载较为缓慢，酌情考虑）
1).配置HTTP服务（yum实质上是使用http协议进行下载安装的）

chkconfig httpd on service httpd start/systemctl start httpd

2).安装本地源制作相关工具

yum -y install yum-utils createrepo yum-plugin-priorities vi /etc/yum/pluginconf.d/priorities.conf 添加gpgcheck=0

3).下载Ambari和HDP（下载时间1M带宽预计需要14小时左右，网络不稳定时容易断线） CentOS 6：

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos6.tar.gz http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos6-rpm.tar.gz http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6/HDP-UTILS-1.1.0.21-centos6.tar.gz

CentOS 7:

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos7.tar.gz http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos7-rpm.tar.gz http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

注：以下将按照CentOS 6进行部署演示，CentOS 7请依据实际情况进行改动
4).创建本地源（我在这里将源文件解压到了/var/www/html目录中）

tar zxvf /opt/ambari-2.5.0.3-centos6.tar.gz -C /var/www/html tar zxvf /opt/HDP-2.6.0.3-centos6-rpm.tar.gz -C /var/www/html tar zxvf /opt/HDP-UTILS-1.1.0.21-centos6.tar.gz -C /var/www/htmlcd /var/www/html/ createrepo./wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari.repo -O /etc/yum.repos.d/ambari.repo

修改ambari.repo,配置为本地源

vi /etc/yum.repos.d/ambari.repo [ambari-2.5.0.3] name=ambari Version - ambari-2.5.0.3 baseurl=http://192.168.5.8/ambari/centos6 gpgcheck=1 gpgkey=http://192.168.5.8/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1

priority=1

vi /etc/yum.repos.d/HDP.repo [HDP-2.6] name=HDP-2.6 baseurl=http://192.168.5.8/HDP/centos6/path=/ enabled=1 gpgcheck=0

vi /etc/yum.repos.d/HDP-UTILS.repo [HDP-UTILS-1.1.0.21] name=HDP-UTILS-1.1.0.21 baseurl=http://192.168.50.38/HDP-UTILS-1.1.0.20/path=/ enabled=1 gpgcheck=0

验证是否生效（如果不确定是否正确可以尝试浏览器访问，例如 http://192.168.5.8/ambari/centos6

yum clean all yum makecache yum repolist

2.安装数据库 Ambari使用的默认数据库是PostgreSQL,用于存储安装元数据，可以使用自己安装的MySQL作为Ambari元数据库（我是使用了源码安装的方法，不做过多赘述，这里只做Yum安装步骤）

yum -y install mysql mysql-server chkconfig mysqld on service mysqld start

3.安装Ambari

yum install -y ambari-server

执行yum install ambari-server成功后,针对mysql数据库再做一些工作：
将mysql-connector-java.jar复制到/usr/share/java目录下（在下载的文件中有mysql-connector-java.jar，具体位置为hdp-utils/mysql-connector-java-source/mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar）

mkdir /usr/share/java cp /root/hdp-utils/mysql-connector-java-source/mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar /usr/share/java/mysql-connector-java.jar cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar

编辑Ambari.properties

vi /etc/ambari-server/conf/ambari.properties 添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

在mysql中分别创建数据库ambari,hive,oozie和其相应用户，创建相应的表（如果不愿意将hive和oozie放在和ambari相同的数据库里，可以另外建立数据库）

CREATE DATABASE ambari; use ambari; grant all on *.* to ambari@'localhost' identified by 'password'; grant all on *.* to ambari@'hostname' identified by 'password'; grant all on *.* to ambari@'%' identified by 'password'; CREATE DATABASE hive; use hive; grant all on *.* to hive@'localhost' identified by 'password'; grant all on *.* to hive@'hostname' identified by 'password'; grant all on *.* to hive@'%' identified by 'password'; CREATE DATABASE oozie; use oozie; grant all on *.* to oozie@'localhost' identified by 'password'; grant all on *.* to oozie@'hostname' identified by 'password'; grant all on *.* to oozie@'%' identified by 'password';

4.配置Ambari 执行ambari-server setup

ambari-server setup 下面是配置执行流程，按照提示操作（1）提示是否自定义设置。输入：y Customize user account for ambari-server daemon [y/n] (n)? y （2）ambari-server 账号。 Enter user account for ambari-server daemon (root): 如果直接回车就是默认选择root用户如果输入已经创建的用户就会显示： Enter user account for ambari-server daemon (root):ambari Adjusting ambari-server permissions and ownership... （3）检查防火墙是否关闭 Adjusting ambari-server permissions and ownership... Checking firewall... WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports. OK to continue [y/n] (y)? 直接回车（4）设置JDK。输入：3 Checking JDK... Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 如果上面选择3自定义JDK,则需要设置JAVA_HOME。输入：/usr/java/jdk1.8.0_131 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /usr/java/jdk1.8.0_131 Validating JDK on Ambari Server...done. Completing setup... （5）数据库配置。选择：y Configuring database... Enter advanced database configuration [y/n] (n)? y （6）选择数据库类型。输入：3 Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere ============================================================================== Enter choice (3): 3 （7）设置数据库的具体配置信息，根据实际情况输入，如果和括号内相同，则可以直接回车。如果想重命名，就输入。 Hostname (localhost):hdp131.cancer.com Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Re-Enter password: （8）将Ambari数据库脚本导入到数据库 WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? 如果使用自己定义的数据库，必须在启动Ambari服务之前导入Ambari的sql脚本。这一点在安装Ambari环节提到过。

5.启动Ambari