flink standalone模式部署【高可用】

注意事项：

高可用需要依赖zookeeper的故障恢复，因此要先准备好zookeeper集群，建议独立搭建zookeeper集群，不要用flink内置的单节点zookeeper

1.flink版本

flink-1.11.2-bin-scala_2.11.tgz
           

2.机器数量：3台

master：2台	'--对应jobmanager'
flink01
flink02

worker：3台	'--对应taskmanager'
flink01
flink02
flink03
           

3.配置flink-conf.yaml文件：vim conf/flink-conf.yaml

'Common【普通】'
------------------------------------------------------
# jobmanager配置
jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
jobmanager.memory.process.size: 1600m

# TaskManager配置
taskmanager.heap.size: 1024m
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 1

# 程序默认并行计算的个数
parallelism.default: 1
------------------------------------------------------

'High Availability【高可用】'
------------------------------------------------------
# 高可用方式：zookeeper
high-availability: zookeeper

# 高可用集群数据存储文件夹
high-availability.storageDir: file:///opt/soft/flink-1.11.2/ha

# 高可用集群的zookeeper地址
high-availability.zookeeper.quorum: flink01:2181,flink02:2181,flink03:2181

# flink zookeeper 根目录
high-availability.zookeeper.path.root: /opt/soft/flink-1.11.2/ha/zookeeper
------------------------------------------------------

'Fault tolerance and checkpointing【容错和检查点】'
------------------------------------------------------
# 状态后端将checkpoints存储在哪
state.backend: filesystem

# checkpoints：检查点目录
state.checkpoints.dir: file:///opt/soft/checkpoints

# 对于增量检查点，只存储上一个检查点的差异，而不是完整的检查点状态。某些状态后端可能不支持增量检查点并忽略此选项
state.backend.incremental: true

# Task故障恢复策略：参考官网（https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/task_failure_recovery.html#restart-all-failover-strategy）
# 1.重启策略
# （1）这里采用固定时间间隔重启策略
restart-strategy: fixed-delay
# （2）失败重启次数
restart-strategy.fixed-delay.attempts: 3
# （3）每次重启时间间隔
restart-strategy.fixed-delay.delay: 5 s

# 2.Task故障恢复策略
jobmanager.execution.failover-strategy: region
------------------------------------------------------
           

4.配置masters文件：vim conf/masters

flink01
flink02
           

5.配置workers文件：vim conf/workers

注意：此配置文件在不同版本的flink中名字可能不一样，此版本是workers，有些版本是slaves。

flink01
flink02
flink03
           

6.分发配置文件

scp conf/flink-conf.yaml flink02://opt/soft/flink-1.11.2/conf/
scp conf/flink-conf.yaml flink03://opt/soft/flink-1.11.2/conf/

scp conf/masters flink02://opt/soft/flink-1.11.2/conf/
scp conf/masters flink03://opt/soft/flink-1.11.2/conf/

scp conf/workers flink02://opt/soft/flink-1.11.2/conf/
scp conf/workers flink03://opt/soft/flink-1.11.2/conf/
           

7.启动flink standalone

在flink01上执行：
[[email protected] flink-1.11.2]# bin/start-cluster.sh 
Starting HA cluster with 2 masters.
Starting standalonesession daemon on host flink01.
Starting standalonesession daemon on host flink02.
Starting taskexecutor daemon on host flink01.
Starting taskexecutor daemon on host flink02.
Starting taskexecutor daemon on host flink03.
           

8.检查进程

[[email protected] ~]# jps
20582 Jps
1752 Kafka
1481 QuorumPeerMain
19786 StandaloneSessionClusterEntrypoint
20139 TaskManagerRunner

[[email protected] ~]# jps
9249 Jps
1481 QuorumPeerMain
9113 TaskManagerRunner
1754 Kafka
8783 StandaloneSessionClusterEntrypoint

[[email protected] ~]# jps
1745 Kafka
1478 QuorumPeerMain
5051 Jps
4925 TaskManagerRunner
           

至此，flink standalone模式部署完成！