技术交流

• 微信:thinktothings
• 微博:
• Flink版本为1.7.2
• 本站持续更新中.......2019-03-13.......

源码

中文文档

• Flink 1.7中文文档(官网英文翻译过来) 在线版:
• Flink 1.7中文文档(官网英文翻译过来) PDF版:

Flink 本地运行交互Shell

• start-scala-shell.sh local
• 参数说明: [local | remote | yarn]

benv.fromElements(1,2,3).map(i => i * i ).print

• 输出结果

149

运行 jar 到 Flink 集群

flink run -c  com.opensourceteams.module.bigdata.flink.example.stream.worldcount.nc.SocketWindowWordCount    ./flink-maven-scala-2-0.0.1.jar

创建flink java 项目

mvn archetype:generate                               \      -DarchetypeGroupId=org.apache.flink              \      -DarchetypeArtifactId=flink-quickstart-java     \      -DarchetypeVersion=1.7.1      -DgroupId=com.opensourceteams \      -DartifactId=flink-maven-java \      -Dversion=0.0.1 \      -Dpackage=com.opensourceteams.module.bigdata.flink  \      -DinteractiveMode=false

创建flink scala项目

mvn archetype:generate                               \      -DarchetypeGroupId=org.apache.flink              \      -DarchetypeArtifactId=flink-quickstart-scala     \      -DarchetypeVersion=1.7.1      -DgroupId=com.opensourceteams \      -DartifactId=flink-maven-scala-2 \      -Dversion=0.0.1 \      -Dpackage=com.opensourceteams.module.bigdata.flink  \      -DinteractiveMode=false

查看jar中文件列表

jar tvf test.jar

maven 运行某个类

mvn exec:java -Dexec.mainClass=wikiedits.WikipediaAnalysis

执行计划图

• 用Firefox 打开，显示的比较全(别浏览器有显示不全的现象)
• 地址:

//执行计划      //println(env.getExecutionPlan)      //StreamGraph     //println(env.getStreamGraph.getStreamingPlanAsJSON)

Execute Plan

{"nodes":[{"id":1,"type":"Source: Socket Stream","pact":"Data Source","contents":"Source: Socket Stream","parallelism":1},{"id":2,"type":"Flat Map","pact":"Operator","contents":"Flat Map","parallelism":1,"predecessors":[{"id":1,"ship_strategy":"FORWARD","side":"second"}]},{"id":3,"type":"Map","pact":"Operator","contents":"Map","parallelism":1,"predecessors":[{"id":2,"ship_strategy":"FORWARD","side":"second"}]},{"id":5,"type":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","pact":"Operator","contents":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","parallelism":1,"predecessors":[{"id":3,"ship_strategy":"HASH","side":"second"}]},{"id":6,"type":"Sink: Print to Std. Out","pact":"Data Sink","contents":"Sink: Print to Std. Out","parallelism":1,"predecessors":[{"id":5,"ship_strategy":"FORWARD","side":"second"}]}]}

StreamGraph Plan

{"nodes":[{"id":1,"type":"Source: Socket Stream","pact":"Data Source","contents":"Source: Socket Stream","parallelism":1},{"id":2,"type":"Flat Map","pact":"Operator","contents":"Flat Map","parallelism":1,"predecessors":[{"id":1,"ship_strategy":"FORWARD","side":"second"}]},{"id":3,"type":"Map","pact":"Operator","contents":"Map","parallelism":1,"predecessors":[{"id":2,"ship_strategy":"FORWARD","side":"second"}]},{"id":5,"type":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","pact":"Operator","contents":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","parallelism":1,"predecessors":[{"id":3,"ship_strategy":"HASH","side":"second"}]},{"id":6,"type":"Sink: Print to Std. Out","pact":"Data Sink","contents":"Sink: Print to Std. Out","parallelism":1,"predecessors":[{"id":5,"ship_strategy":"FORWARD","side":"second"}]}]}

Flink 环境，配置

• Flink 源码debug方法:
• Flink 名词术语 :
• Flink 源码编译:

example

• scala 版Flink WordCount单词统计 :

• wordCount Dataset批处理

  • start-scala-shell.sh local
  • 参数说明:[local | remote | yarn]

  benv.fromElements("a b a c").flatMap(x => x.split(" ")).map((_,1)).groupBy(0).sum(1).print

  • 输出结果

  (a,2)  (b,1)  (c,1)

• Flink 1.7.2 DataStream operator 示例 :
• Flink1.7.2 Dataset transformation示例:

Flink1.7.2 DataStream 源码分析(流处理)

• Flink MiniCluster 作业提交:
• Flink1.7.2 local WordCount源码分析:
• Flink Sink 接收数据的顺序(Window发送数据顺序):
• Flink Window 排序:
• Flink1.7.2 Source、Window数据交互源码分析:
• Flink1.7.2 并行计算源码分析:
• Flink 1.7.2 业务时间戳分析流式数据源码分析:
• Flink 005-source-operation-sink源码分析:

Flink1.7.2 Dataset 源码分析(批处理)

• Flink1.7.2 Dataset local 源码分析 :
• Flink1.7.2 Dataset 文件切片计算方式和切片数据读取源码分析:
• Flink1.7.2 Dataset 并行计算源码分析:

Flink1.7.2 时序图

• Flink 客户端提交程序到MiniCluster(时序图):
• Flink ExecutionGraph的构建和Execution.deploy之前(时序图):
• Flink Execution deploy和source数据读取(时序图):
• Flink OperatorChian计算source数据(时序图):

Flink 1.7.2 Error 收集

• Flink 1.7.2 Error 收集: