If execute above code via spark cluster, it will occur java.lang.NullPointerException like below.


ResultStage 1 (foreach at Streamer.java:52) failed in 0.257 s due to Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 10, 192.168.200.77, executor 4): java.lang.NullPointerException

        at co.kr.rollupcat.consumer.streamer.Streamer.lambda$null$d469caef$1(Streamer.java:56)

        at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)

        at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)

        at scala.collection.Iterator$class.foreach(Iterator.scala:893)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

        at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:921)

        at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:921)

        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)

        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)

        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

        at org.apache.spark.scheduler.Task.run(Task.scala:109)

        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)


 At first, I thought the reason is because the Connection object that used inside of hub.insert(record); is shared between other spark slave nodes, so I changed it to Connection pool but error is not fixed.


 And then, I searched google and found this.


https://stackoverflow.com/questions/26582617/spark-nullpointerexception-inside-foreach-loop


 Following answer of this, we should not use RDD Operations inside of RDD Operations. Well, so I followed like below.


 It's working fine, hurray!


WRITTEN BY
pf1999
프로그래밍 도중 삽질하다가 해결한 내용 공유용 블로그

,