1.Master主备切换机制的原理（图解）
文章图片
2.部分源码分析
【spark|Spark内核源码深度剖析（Master主备切换机制原理剖析与源码分析）】master.scala中的completeRecovery方法：
/*
* 完成Master的主备切换
*/
def completeRecovery() {
// Ensure "only-once" recovery semantics using a short synchronization period.
synchronized {
if (state != RecoveryState.RECOVERING) { return }
state = RecoveryState.COMPLETING_RECOVERY
}
/*
* 将Application和worker，过滤出来目前的状态还是UNKNOW的
* 然后遍历，分别调用removeWorker和finishApplication方法，
* 对可能已经出故障，或者甚至已经死掉的Application和Worker，进行清理
*/
// Kill off any workers and apps that didn't respond to us.
workers.filter(_.state == WorkerState.UNKNOWN).foreach(removeWorker)
apps.filter(_.state == ApplicationState.UNKNOWN).foreach(finishApplication)// Reschedule drivers which were not claimed by any workers
drivers.filter(_.worker.isEmpty).foreach { d =>
logWarning(s"Driver ${d.id} was not found after master recovery")
if (d.desc.supervise) {
logWarning(s"Re-launching ${d.id}")
relaunchDriver(d)
} else {
removeDriver(d.id, DriverState.ERROR, None)
logWarning(s"Did not re-launch ${d.id} because it was not supervised")
}
}state = RecoveryState.ALIVE
schedule()
logInfo("Recovery complete - resuming operations!")
}推荐阅读
- spark|Spark内核源码深度剖析（Master注册机制原理剖析与源码分析）
- spark|Spark内核源码深度剖析（SparkContext原理剖析与源码分析）
- spark|spark的基本工作原理和RDD
- Spark|基于Spark封装的二次开发工程edata-base，介绍
- spark|spark学习笔记（七）——sparkcore核心编程-RDD序列化/依赖关系/持久化/分区器/累加器/广播变量
- 技术干货|推荐系统-协同过滤在Spark中的实现
- mysql|大数据处理与开发课程设计——纽约出租车大数据分析
- pyspark_mllib_classifier—(SVM)
- 大数据|SPARK Parquet嵌套类型的向量化支持以及列索引(column index)