天天看點

mapreduce JVM參數配置的問題

在mapred-site.xml中添加參數:

<property>

    <name>mapreduce.map.java.opts</name>

    <value>1536</value>

</property>

<property>

    <name>mapreduce.reduce.java.opts</name>

    <value>3072</value>

</property>

在hive中執行簡單查詢:

select count(1) from sample;

報錯:

Error during job, obtaining debugging information...

Examining task ID: task_1491368313881_0001_m_000000 (and more) from job job_1491368313881_0001

Task with the most failures(4): 

-----

Task ID:

  task_1491368313881_0001_m_000000

URL:

  http://ctdn01:8088/taskdetails.jsp?jobid=job_1491368313881_0001&tipid=task_1491368313881_0001_m_000000

-----

Diagnostic Messages for this Task:

Exception from container-launch.

Container id: container_1491368313881_0001_01_000005

Exit code: 1

Stack trace: ExitCodeException exitCode=1: 

at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)

at org.apache.hadoop.util.Shell.run(Shell.java:481)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

修改調整這兩個參數多次,均不能改正這個問題,現在忽略設定這兩個參數。