r - Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 -

-

please me solve issue in rmr2 (rhadoop integration)

i using cloudera-quickstart-vm-5.4.0-0-virtualbox

version details:

hadoop 2.6.0-cdh5.4.0 java version "1.7.0_67" r version 3.2.0  rmr 2.3.0

below r code , error:

sys.setenv("hadoop_home"="/usr/lib/hadoop") sys.setenv("hive_home"="/usr/lib/hive") sys.setenv("hadoop_cmd"="/usr/lib/hadoop/bin/hadoop") sys.setenv("hadoop_streaming"="/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.6.0-mr1-cdh5.4.0.jar") sys.getenv("hadoop_cmd") [1] "/usr/lib/hadoop/bin/hadoop" sys.setenv("rhive_fs_home"="/home/rhive") sys.setenv(java_home="/usr/java/jdk1.7.0_67-cloudera") library(rmr2) library(rhdfs) hdfs.init()  ## map function map <- function(k,lines) {   words.list <- strsplit(lines, '/t')   words <- unlist(words.list)   return( keyval(words, 1) ) }  ## reduce function reduce <- function(word, counts) {   keyval(word, sum(counts)) }  wordcount <- function (input, output=null) {   mapreduce(input=input, output=output, input.format="text",   map=map, reduce=reduce) } ## delete previous result if hdfs.root <- 'wordcount' hdfs.data <- file.path(hdfs.root, 'input/d.txt') hdfs.out <- file.path(hdfs.root, 'out') out <- wordcount(hdfs.data, hdfs.out)  15/08/10 05:16:40 warn streaming.streamjob: -file option deprecated, please use generic option -files instead. packagejobjar: [/tmp/rtmpqrizwy/rmr-local-env17807fd4e342, /tmp/rtmpqrizwy/rmr-global-env17807c276b72, /tmp/rtmpqrizwy/rmr-streaming-map17807a097c08, /tmp/rtmpqrizwy/rmr-streaming-reduce178012121630] [/usr/lib/hadoop-mapreduce/hadoop-streaming-2.6.0-cdh5.4.0.jar] /tmp/streamjob8275237671226206531.jar tmpdir=null 15/08/10 05:16:41 info client.rmproxy: connecting resourcemanager @ /0.0.0.0:8032 15/08/10 05:16:42 info client.rmproxy: connecting resourcemanager @ /0.0.0.0:8032 15/08/10 05:16:42 info mapred.fileinputformat: total input paths process : 1 15/08/10 05:16:42 info mapreduce.jobsubmitter: number of splits:2 15/08/10 05:16:43 info mapreduce.jobsubmitter: submitting tokens job: job_1439208338231_0002 15/08/10 05:16:43 info impl.yarnclientimpl: submitted application application_1439208338231_0002 15/08/10 05:16:43 info mapreduce.job: url track job: http://quickstart.cloudera:8088/proxy/application_1439208338231_0002/ 15/08/10 05:16:43 info mapreduce.job: running job: job_1439208338231_0002 15/08/10 05:16:49 info mapreduce.job: job job_1439208338231_0002 running in uber mode : false 15/08/10 05:16:49 info mapreduce.job:  map 0% reduce 0% 15/08/10 05:16:57 info mapreduce.job:  map 50% reduce 0% 15/08/10 05:16:57 info mapreduce.job: task id : attempt_1439208338231_0002_m_000000_0, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:322)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:535)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:61)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:453)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  15/08/10 05:16:57 info mapreduce.job: task id : attempt_1439208338231_0002_m_000001_0, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:322)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:535)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:61)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:453)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  15/08/10 05:16:58 info mapreduce.job:  map 0% reduce 0% 15/08/10 05:17:06 info mapreduce.job: task id : attempt_1439208338231_0002_m_000000_1, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:322)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:535)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:61)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:453)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  15/08/10 05:17:07 info mapreduce.job: task id : attempt_1439208338231_0002_m_000001_1, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:322)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:535)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:61)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:453)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  15/08/10 05:17:16 info mapreduce.job: task id : attempt_1439208338231_0002_m_000000_2, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:322)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:535)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:61)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:453)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  15/08/10 05:17:17 info mapreduce.job: task id : attempt_1439208338231_0002_m_000001_2, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:322)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:535)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:61)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:453)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:163)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1671)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:158)  15/08/10 05:17:25 info mapreduce.job:  map 100% reduce 100% 15/08/10 05:17:25 info mapreduce.job: job job_1439208338231_0002 failed state failed due to: task failed task_1439208338231_0002_m_000000 job failed tasks failed. failedmaps:1 failedreduces:0  15/08/10 05:17:25 info mapreduce.job: counters: 13     job counters          failed map tasks=7         killed map tasks=1         launched map tasks=8         other local map tasks=6         data-local map tasks=2         total time spent maps in occupied slots (ms)=53461         total time spent reduces in occupied slots (ms)=0         total time spent map tasks (ms)=53461         total vcore-seconds taken map tasks=53461         total megabyte-seconds taken map tasks=54744064     map-reduce framework         cpu time spent (ms)=0         physical memory (bytes) snapshot=0         virtual memory (bytes) snapshot=0 15/08/10 05:17:25 error streaming.streamjob: job not successful! streaming command failed! error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  :    hadoop streaming failed error code 1

you can display logs app running command:

yarn logs -applicationid application_${appid}

in case

yarn logs -applicationid application_1439208338231_0002

or can find them in locations like:

/yarn/apps/&{user_name}/logs/application_${appid}/

or

/app-logs/&{user_name}/logs/application_${appid}/

in general, localization of logs in hadoop: https://stackoverflow.com/a/21629631/3846521

in logs have stderr output code. if you'll still have problem solution paste log here, i'll try you.


Comments

Post a Comment

Popular posts from this blog

php - Admin SDK -- get information about the group -

-
please, help. https://www.googleapis.com/admin/directory/v1/groups/106274944935548788283 sending request obtain information group id 1062 ... here link method in google https://developers.google.com/admin-sdk/directory/v1/guides/manage-groups#get_group in response error - 401 unauthorized. oauth authenticate via user has received rights necessary request, included in console admin sdk. authorization successful, there token. not can not understand. if not hard - response. requested information: file_get_contents("https://www.googleapis.com/admin/directory/v1/groups/1062749‌​44935548788283?key=aizasybruhiomdf9nlx9lx3imxwesqmcrnhh4l4"); id group - "106274944935548788283" key applications in case - "aizasybruhiomdf9nlx9lx3imxwesqmcrnhh4l4" error: failed open stream: http request failed! http/1.0 401 unauthorized
Read more

dns - How To Use Custom Nameserver On Free Cloudflare? -

-
i have problem. i own small hosting company. trying add cloudflare site free cdn. problem when sign have change nameservers nameservers. have nameserver dns set through godaddy, how add them cloudflare? main site uses company's nameservers whenever add them site goes offline. when go cloudflare's dns records click on dropdown, click ns, , there's 2 boxes 'name' , 'nameserver'. mean input 'ns1' 'name' , 'ns1.domain.com' 'nameserver'? input server's ip addresses? is there way of accomplishing free version of cloudflare? that impossible. option have custom name servers available on business , enterprise. see https://www.cloudflare.com/plans . also, might mistaken. must change name servers in domain register settings, not on cloudflare.
Read more

Python Error - TypeError: input expected at most 1 arguments, got 3 -

-
can explain why can not use your_name in goal variable? my_name = "bryson" my_age = 29 your_name = input ("what name? ") your_age = input ("what age? ") print ("my name is", my_name,", , am", my_age, "years old.") print ("your name is", your_name,", , are", your_age,".") print("thank buying book,", your_name,"!") goal = input ("what favorite part of book,", your_name, "?") print("awesome!") the error is: goal = input ("what favorite part of book,", your_name, "?") typeerror: input expected @ 1 arguments, got 3 you got error because in fact gave 3 arguments input function when expecting 1 (namely, string prompt). in input ("what favorite part of book,", your_name, "?") ---------------------------------------- , ---------, --- the underlined parts comma-separated arguments: st...
Read more