Spark executor lost because of time out even after setting quite long time out value 1000 seconds -

-

hi have written spark job seems working fine hour , after executor start getting lost because of timeout see following in log statement

15/08/16 12:26:46 warn spark.heartbeatreceiver: removing executor 10 no recent heartbeats: 1051638 ms exceeds timeout 1000000 ms

i dont see errors see above warning , because of executor gets removed yarn , see rpc client disassociated error , ioexception connection refused , fetchfailedexception

after executor gets removed see again getting added , starts working , other executors fails again. question is normal executor getting lost? happens task lost executors working on? spark job keeps on running since long around 4-5 hours have cluster 1.2 tb memory , no of cpu cores. solve above time out issue tried increase time spark.akka.timeout 1000 seconds no luck. using following command run spark job please guide new spark. using spark 1.4.1. in advance.

./spark-submit --class com.xyz.abc.mysparkjob  --conf "spark.executor.extrajavaoptions=-xx:maxpermsize=512m" --driver-java-options -xx:maxpermsize=512m --driver-memory 4g --master yarn-client --executor-memory 25g --executor-cores 8 --num-executors 5 --jars /path/to/spark-job.jar

what might happen slaves cannot launch executor anymore, due memory issue. following messages in master logs:

15/07/13 13:46:50 info master: removing executor app-20150713133347-0000/5 because exited 15/07/13 13:46:50 info master: launching executor app-20150713133347-0000/9 on worker worker-20150713153302-192.168.122.229-59013 15/07/13 13:46:50 debug master: [actor] handled message (2.247517 ms) executorstatechanged(app-20150713133347-0000,5,exited,some(command exited code 1),some(1)) actor[akka.tcp://sparkworker@192.168.122.229:59013/user/worker#-83763597]

you might find detailed java errors in worker's log directory, , maybe type of file: work/app-id/executor-id/hs_err_pid11865.log.

see http://pastebin.com/b4fbxvhr

this issue might resolved application management of rdd's, not increasing size of jvm's heap.


Comments

Post a Comment

Popular posts from this blog

dns - How To Use Custom Nameserver On Free Cloudflare? -

-
i have problem. i own small hosting company. trying add cloudflare site free cdn. problem when sign have change nameservers nameservers. have nameserver dns set through godaddy, how add them cloudflare? main site uses company's nameservers whenever add them site goes offline. when go cloudflare's dns records click on dropdown, click ns, , there's 2 boxes 'name' , 'nameserver'. mean input 'ns1' 'name' , 'ns1.domain.com' 'nameserver'? input server's ip addresses? is there way of accomplishing free version of cloudflare? that impossible. option have custom name servers available on business , enterprise. see https://www.cloudflare.com/plans . also, might mistaken. must change name servers in domain register settings, not on cloudflare.
Read more

python - Pygame screen.blit not working -

-
in program creating in python/pygame, screen.blit not working. in other programs have made, screen.blit works. error: traceback (most recent call last): file "c:...\main.py", line 46, in <module> screen.blit(sight.image, sight.rect) attributeerror: 'int' object has no attribute 'blit' the error in question caused screen variable being allocated integer instead of pygame.surface() object.perhaps accidentally rewrote variable somewhere? from can make of doing, seems stating position of blit full rect instead of coordinate. when do: screen.blit(sight.image,sight.rect) what program seeing is: screen.blit(sight.image,(sight.rect.x,sight.rect.y,sight.rect.width,sight.rect.height)) i assuming don't intend this. remember rect object consists of position, width , height. to correct code, put (sight.rect.x,sight.rect.y) location goes instead this: screen.blit(sight.image,(sight.rect.x,sight.rect.y)) the .x , .y corres
Read more

c# - Web API response xml language -

-
i having simple post url clients send post request. http://mydomain.com/requests/request post data: <request> <orderid>1e008921</orderid> <ordername>mc1</ordername> </request> to respond sending them xml <srequest> <requestedby>client 1 </requestedby> <requestname>test name</requestname> <requeststatus>success</requeststatus> <srequest> now client wants above xml in different language (say french ) how handle ? how send response language preference in web api ? specially post scenario ? (append language string ? or best practices ?) i suggest either include in url: http://mydomain.com/fr/requests/request or use http header accept-language . read more in answer to: good way changing language resources dynamically request
Read more