How to fix spark streaming java.lang.nullpointerexception occurred in RDD foreach
spark 2018. 11. 21. 13:43If execute above code via spark cluster, it will occur java.lang.NullPointerException like below.
ResultStage 1 (foreach at Streamer.java:52) failed in 0.257 s due to Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 10, 192.168.200.77, executor 4): java.lang.NullPointerException
at co.kr.rollupcat.consumer.streamer.Streamer.lambda$null$d469caef$1(Streamer.java:56)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:921)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:921)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
At first, I thought the reason is because the Connection object that used inside of hub.insert(record); is shared between other spark slave nodes, so I changed it to Connection pool but error is not fixed.
And then, I searched google and found this.
https://stackoverflow.com/questions/26582617/spark-nullpointerexception-inside-foreach-loop
Following answer of this, we should not use RDD Operations inside of RDD Operations. Well, so I followed like below.
It's working fine, hurray!
WRITTEN BY
- pf1999
프로그래밍 도중 삽질하다가 해결한 내용 공유용 블로그