CentOS|CentOS Hadoop完全分布式集群搭建 CentOSHadoop完全分布式集群搭建

java环境安装
现有三台主机,CentOS 静态IP设置后，CentOS 修改主机名，修改
hosts

vi /etc/hosts

192.168.31.xxx master 192.168.31.xxx slaver1 192.168.31.xxx slaver2

集群ssh无密匙登录设置

--- 在master,slave1,slave2中执行

$ ssh-keygen -t rsa //一路回车

--- 在 master（master角色）服务器上执行，将~/.ssh/下的id_rsa.pub公私作为认证发放到master,slave1,slave2的~/.ssh/下

# ssh-copy-id -i ~/.ssh/id_rsa.pub master # ssh-copy-id -i ~/.ssh/id_rsa.pub slave1 # ssh-copy-id -i ~/.ssh/id_rsa.pub slave2 设置完后，通过#ssh localhost测试，第一次登录会有如下提示： The authenticity of host 'localhost (127.0.0.1)' can't be established. RSA key fingerprint is a2:44:5f:79:00:c9:17:3b:b4:b5:47:cf:66:be:c4:0d. Are you sure you want to continue connecting (yes/no)? 输入yes后，之后就不需要了。(必须操作)

--- 在 master上登录其他Linux服务器不需要输入密码即成功

//不需要输入密码 # ssh slaver1 or # ssh slaver2

hadoop完全分布式集群文件配置和启动

第一步安装Hadoop

上传hadoop的安装包hadoop-2.6.4.tar.gz到服务器上去(自由选择路径）

$ tar -zxvf hadoop-2.6.4.tar.gz-C /home/cloud/

第二步配置hadoop

$ cd /home/cloud/hadoop-2.6.4/etc/hadoop

hadoop-2.6.4的搭建需要修改几个配置文件
第0个

$ vi/home/cloud/hadoop-2.6.4/etc/hadoop/slaves

这是设置从节点hostname的地方（这个文件表示从节点，只填写主机名）

master#姑且让grape0既作namenode又作datanode slave1 #datanode1 slave2 #datanode2

第一个：vi /home/cloud/hadoop-2.6.4/etc/hadoop/hadoop-env.sh

#第27行 (建议使用推荐的版本号，存放路径自定) export JAVA_HOME=/home/cloud/jdk1.8.0_101

第二个：vi /home/cloud/hadoop-2.6.4/etc/hadoop/core-site.xml(添加如下内容)

fs.defaultFS hdfs://master:9000 io.file.buffer.size 131072 hadoop.tmp.dir file:/home/cloud/hadoop-2.6.4/temp Abasefor other temporary directories.

第三个：vi /home/cloud/hadoop-2.6.4/etc/hadoop/hdfs-site.xml

dfs.namenode.secondary.http-address master:9001 dfs.namenode.name.dir file:/home/cloud/hadoop-2.6.4/dfs/name dfs.datanode.data.dir file:/home/cloud/hadoop-2.6.4/dfs/data dfs.replication 1 dfs.webhdfs.enabled true

新建文件夹/home/cloud/hadoop-2.6.4/dfs/name和/home/cloud/hadoop-2.6.4/dfs/data
第四个：mapred-site.xml (改名后得到的文件)

# mapred-site.xml.template重命名 (在/home/cloud/hadoop-2.6.4/etc/hadoop/路径下) $ mv mapred-site.xml.template mapred-site.xml $ vi mapred-site.xml

mapreduce.framework.name yarn mapreduce.jobhistory.address master:10020 mapreduce.jobhistory.webapp.address master:50030 mapreduce.reduce.memory.mb 4096

第五个：vi /home/cloud/hadoop-2.6.4/etc/hadoop/yarn-site.xml

【CentOS|CentOS Hadoop完全分布式集群搭建】yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.resourcemanager.address master:8032 yarn.resourcemanager.scheduler.address master:8030 yarn.resourcemanager.resource-tracker.address master:8031 yarn.resourcemanager.admin.address master:8033 yarn.resourcemanager.webapp.address master:8088