HA-Spark集群环境搭建(Yarn模式) _环境搭建

卧疾丰暇豫，翰墨时间作。这篇文章主要讲述HA-Spark集群环境搭建(Yarn模式)相关的知识，希望能为你提供帮助。

前置准备【HA-Spark集群环境搭建(Yarn模式)】CentOS7、jdk1.8、scala-2.11.12、spark-2.4.5、hadoop-2.7.7、zookeeper-3.5.7
想要完成本期视频中所有操作，需要以下准备：

一、集群规划
二、集群配置 2.1 spark-env.sh

[xiaokang@hadoop01 conf]$ cp spark-env.sh.template spark-env.sh

export java_HOME=/opt/moudle/jdk1.8.0_191
export SCALA_HOME=/opt/moudle/scala-2.11.12
YARN_CONF_DIR=/opt/software/hadoop-2.7.7/etc/hadoop
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop01:2181,hadoop02:2181,hadoop03:2181 -Dspark.deploy.zookeeper.dir=/ha-spark"
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=24 -Dspark.history.fs.logDirectory=hdfs://hadoop01:9000/spark-jobhistory"

2.2 spark-defaults.conf

[xiaokang@hadoop01 conf]$ cp spark-defaults.conf.template spark-defaults.conf

#spark.masterspark://hadoop01:7077
spark.masterspark://hadoop01:7077,hadoop02:7077
spark.eventLog.enabledtrue
spark.eventLog.dirhdfs://hadoop01:9000/spark-jobhistory
spark.yarn.historyServer.addresshadoop01:18080

2.3 slaves

[xiaokang@hadoop01 conf]$ cp slaves.template slaves

hadoop01
hadoop02
hadoop03

2.4 yarn-site.xml

< !-- 由于测试环境的虚拟机内存太少, 防止将来任务被意外杀死, 做如下配置 -->
< !--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
< property>
< name> yarn.nodemanager.pmem-check-enabled< /name>
< value> false< /value>
< /property>
< !--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
< property>
< name> yarn.nodemanager.vmem-check-enabled< /name>
< value> false< /value>
< /property>

< property>
< name> yarn.log.server.url< /name>
< value> http://hadoop01:19888/jobhistory/logs< /value>
< /property>

2.5 分发

[xiaokang@hadoop01 ~]$ distribution.sh /opt/software/spark-2.4.5
[xiaokang@hadoop01 ~]$ distribution.sh /opt/software/hadoop-2.7.7/etc/hadoop/yarn-site.xml

三、启动集群 3.1 启动ha-hadoop集群

[xiaokang@hadoop01 ~]$ ha-hadoop.sh start

3.2 在hadoop01上启动spark集群

# 进入/opt/software/spark-2.4.5/sbin目录并启动集群
[xiaokang@hadoop01 sbin]$ ./start-all.sh

3.3 在hadoop02上启动备Master

# 进入/opt/software/spark-2.4.5/sbin目录并启动备Master
[xiaokang@hadoop02 sbin]$ ./start-master.sh

3.4 在hadoop01上启动任务历史服务器

# 进入/opt/software/spark-2.4.5/sbin目录并启动任务历史服务器
[xiaokang@hadoop01 sbin]$ ./start-history-server.sh

四、查看集群 4.1 jps进程查看

[xiaokang@hadoop01 ~]$ call-cluster.sh jps
--------hadoop01--------
10881 NodeManager
11265 HistoryServer
13058 Master
10131 NameNode
10470 JournalNode
11176 Worker
10250 DataNode
10683 DFSZKFailoverController
10796 JobHistoryServer
12973 Jps
9935 QuorumPeerMain
--------hadoop02--------
7232 QuorumPeerMain
7648 DFSZKFailoverController
7505 JournalNode
7831 NodeManager
8183 Worker
7321 NameNode
7402 DataNode
9099 Jps
7743 ResourceManager
8255 Master
--------hadoop03--------
7313 DataNode
8273 Jps
7234 QuorumPeerMain
7843 Worker
7545 ResourceManager
7419 JournalNode
7663 NodeManager

4.2 Web UI查看

[xiaokang@hadoop01 sbin]$ cat /opt/software/spark-2.4.5/logs/spark-xiaokang-org.apache.spark.deploy.master.Master-1-hadoop01.out | grep MasterWebUI

20/05/31 08:32:08 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hadoop01:8081

通过启动日志可以看到hadoop01的??MasterWebUI??的端口号为8081

[xiaokang@hadoop02 sbin]$ cat /opt/software/spark-2.4.5/logs/spark-xiaokang-org.apache.spark.deploy.master.Master-1-hadoop02.out | grep MasterWebUI

20/05/31 08:32:43 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hadoop02:8082

通过启动日志可以看到hadoop02的??MasterWebUI??的端口号为8082

文章图片

hadoop01上的任务历史服务器端口号为18080

五、计算 PI (测试高可用)

[xiaokang@hadoop01 ~]$ spark-submit --master yarn --class org.apache.spark.examples.SparkPi /opt/software/spark-2.4.5/examples/jars/spark-examples_2.11-2.4.5.jar 1000

运行过程中将主Master给kill掉，测试是否高可用