Hadoop(一)：CentOS 安装hadoop

我的CSDN博客地址：[url]http://blog.csdn.net/qidong7[/url]

[b]为Hadoop集群安装SSH[/b]

一、验证是否安装了ssh

[[email protected] /]# which ssh
/usr/bin/ssh
[[email protected] /]# which sshd
/usr/sbin/sshd
[[email protected] /]# which ssh-keygen
/usr/bin/ssh-keygen           

如果收到类似这样的错误信息

/usr/bin/which no ssh in (/usr/bin:/bin:/usr/bin...           

您可以通过Linux安装包管理器安装OpenSSH(www.openssh.com)或者直接下载源码。

二、生成SSH密钥对

务必要避免输入口令，否则，主节点每次试图访问其他节点时，你都得手动输入这个口令

[[email protected] /]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop-user/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop-user/.ssh/id_rsa.
Your public key has been saved in /home/hadoop-user/.ssh/id_rsa.pub.
The key fingerprint is:
2e:e1:91:cf:85:48:f9:92:8f:2d:e5:36:92:15:e0:1a [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . o         |
|    E + .        |
|     + = o       |
|    . B S .      |
|     . / .       |
|      B X        |
|       = .       |
|                 |
+-----------------+
           

生成密钥对之后，公钥的形式为：

[[email protected] /]# 
more  /home/hadoop-user/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0pmwcaf/xgUrDQfK4ZePQEqjXhYPSV2iUsRxunk20RxerLOpw5SNJ+3gb2jVSnuncAlABkS
yi8VCCJVmV4b0y6kCP6ap9jdcRJhxLB33zd5IyHMyo9zjdo8j2xRnFtv1SfaqkhIt8/By1E2+Si3dlIE61DhRGtdxGJNkVv0O4+8AANKcEk
C5LK8Hr+pEBBVlM9m9OHYJ9GdekQcMQQC/CtrZohzHznDm70tJ1EtnidYdUcBz31FuOfeJjuEapbkn6Az9HwDEQD8cemtED0QK/aW9FLjko
zid5nLjJ7ZJwmkmq5V3rLSE2AbqGnGtYSYCezJFFHmUpNyG35vYjpGj....
           

三、将密钥公布(本人的target为localhost)

[[email protected] hadoop-1.2.0]# scp ~/.ssh/id_rsa.pub [email protected]:~/master_key           

[[email protected] /]# ssh localhost

Last login: Sat Jun 1 18:12:59 2013 from 192.168.1.102

四、安装hadoop

1、 解压

tar xzvf hadoop-1.2.0.tar.gz

2、配置 Hadoop

2.1 . 配置JDK

[b]hadoop-env.sh[/b]

export JAVA_HOME=/usr/java/jdk1.7.0_17

2.2 . 配置conf/core-site.xml、conf/hdfs-site.xml、conf/mapred-site.xml文件

[b]core-site.xml[/b]

<configuration>     
<property>  
       <name>fs.default.name</name>  
       <value>hdfs://localhost:9000</value>  
</property>  
<property>
       <name>hadoop.tmp.dir</name>
        <value>/usr/app/hadoop-hdfs</value>
        </property>
</configuration>              

[b]hdfs-site.xml[/b]

<configuration>     
   <property>  
         <name>dfs.replication</name>  
         <value>1</value>  
    </property>  
</configuration>             

[b]mapred-site.xml[/b]

<configuration>     
    <property>  
         <name>mapred.job.tracker</name>  
         <value>localhost:9001</value>  
    </property>  
</configuration>             

3、启动hadoop

cd /home/brain/hadoop-1.2.0/bin

1.格式化文件系统

./hadoop namenode –format

2. 启动/关闭 hadoop

启动所有任务start-all.sh/stop-all.sh

启动关闭HDFS: start-dfs.sh/stop-dfs.sh

启动关闭MapReduce: start-mapred.sh/stop-mapred.sh

3. 用jps命令查看进程,确保有

namenode,

dataNode,

JobTracker,

TaskTracker

如果没有namenode ,

1、先运行stop-all.sh

2、格式化namdenode，不过在这之前要先删除原目录，即core-site.xml下配置 的<name>hadoop.tmp.dir</name>所指向的目录，然后运行 hadoop namenode -format

3、运行start-all.sh

4、检查执行

访问

http://192.168.1.77:50030 (MapReduce的Web页面)

http://192.168.1.77:50070 (HDfS的web页面)

本人安装过程中出现没有namenode，格式化后仍然没有，这时可以查看下日志，发现端口被占。

有问题查看日志 HADOOP_HOME/logs/

另外如果访问

http://192.168.1.77:50030 访问不到，可能是防火墙的原因，可以关闭防火墙，再次访问

service iptables stop

如果是，修改防火墙配置，对50030端口不监控；

修改防火墙监控端口

vi /etc/sysconfig/iptables

中加

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 50030 -j ACCEPT

就不对50030端口进行监控了

请关注微博[url]http://weibo.com/u/3272022684[/url]