Hadoop(一)：CentOS 安裝hadoop

我的CSDN部落格位址：[url]http://blog.csdn.net/qidong7[/url]

[b]為Hadoop叢集安裝SSH[/b]

一、驗證是否安裝了ssh

[[email protected] /]# which ssh
/usr/bin/ssh
[[email protected] /]# which sshd
/usr/sbin/sshd
[[email protected] /]# which ssh-keygen
/usr/bin/ssh-keygen           

如果收到類似這樣的錯誤資訊

/usr/bin/which no ssh in (/usr/bin:/bin:/usr/bin...           

您可以通過Linux安裝包管理器安裝OpenSSH(www.openssh.com)或者直接下載下傳源碼。

二、生成SSH密鑰對

務必要避免輸入密碼，否則，主節點每次試圖通路其他節點時，你都得手動輸入這個密碼

[[email protected] /]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop-user/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop-user/.ssh/id_rsa.
Your public key has been saved in /home/hadoop-user/.ssh/id_rsa.pub.
The key fingerprint is:
2e:e1:91:cf:85:48:f9:92:8f:2d:e5:36:92:15:e0:1a [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . o         |
|    E + .        |
|     + = o       |
|    . B S .      |
|     . / .       |
|      B X        |
|       = .       |
|                 |
+-----------------+
           

生成密鑰對之後，公鑰的形式為：

[[email protected] /]# 
more  /home/hadoop-user/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0pmwcaf/xgUrDQfK4ZePQEqjXhYPSV2iUsRxunk20RxerLOpw5SNJ+3gb2jVSnuncAlABkS
yi8VCCJVmV4b0y6kCP6ap9jdcRJhxLB33zd5IyHMyo9zjdo8j2xRnFtv1SfaqkhIt8/By1E2+Si3dlIE61DhRGtdxGJNkVv0O4+8AANKcEk
C5LK8Hr+pEBBVlM9m9OHYJ9GdekQcMQQC/CtrZohzHznDm70tJ1EtnidYdUcBz31FuOfeJjuEapbkn6Az9HwDEQD8cemtED0QK/aW9FLjko
zid5nLjJ7ZJwmkmq5V3rLSE2AbqGnGtYSYCezJFFHmUpNyG35vYjpGj....
           

三、将密鑰公布(本人的target為localhost)

[[email protected] hadoop-1.2.0]# scp ~/.ssh/id_rsa.pub [email protected]:~/master_key           

[[email protected] /]# ssh localhost

Last login: Sat Jun 1 18:12:59 2013 from 192.168.1.102

四、安裝hadoop

1、 解壓

tar xzvf hadoop-1.2.0.tar.gz

2、配置 Hadoop

2.1 . 配置JDK

[b]hadoop-env.sh[/b]

export JAVA_HOME=/usr/java/jdk1.7.0_17

2.2 . 配置conf/core-site.xml、conf/hdfs-site.xml、conf/mapred-site.xml檔案

[b]core-site.xml[/b]

<configuration>     
<property>  
       <name>fs.default.name</name>  
       <value>hdfs://localhost:9000</value>  
</property>  
<property>
       <name>hadoop.tmp.dir</name>
        <value>/usr/app/hadoop-hdfs</value>
        </property>
</configuration>              

[b]hdfs-site.xml[/b]

<configuration>     
   <property>  
         <name>dfs.replication</name>  
         <value>1</value>  
    </property>  
</configuration>             

[b]mapred-site.xml[/b]

<configuration>     
    <property>  
         <name>mapred.job.tracker</name>  
         <value>localhost:9001</value>  
    </property>  
</configuration>             

3、啟動hadoop

cd /home/brain/hadoop-1.2.0/bin

1.格式化檔案系統

./hadoop namenode –format

2. 啟動/關閉 hadoop

啟動所有任務start-all.sh/stop-all.sh

啟動關閉HDFS: start-dfs.sh/stop-dfs.sh

啟動關閉MapReduce: start-mapred.sh/stop-mapred.sh

3. 用jps指令檢視程序,確定有

namenode,

dataNode,

JobTracker,

TaskTracker

如果沒有namenode ,

1、先運作stop-all.sh

2、格式化namdenode，不過在這之前要先删除原目錄，即core-site.xml下配置 的<name>hadoop.tmp.dir</name>所指向的目錄，然後運作 hadoop namenode -format

3、運作start-all.sh

4、檢查執行

通路

http://192.168.1.77:50030 (MapReduce的Web頁面)

http://192.168.1.77:50070 (HDfS的web頁面)

本人安裝過程中出現沒有namenode，格式化後仍然沒有，這時可以檢視下日志，發現端口被占。

有問題檢視日志 HADOOP_HOME/logs/

另外如果通路

http://192.168.1.77:50030 通路不到，可能是防火牆的原因，可以關閉防火牆，再次通路

service iptables stop

如果是，修改防火牆配置，對50030端口不監控；

修改防火牆監控端口

vi /etc/sysconfig/iptables

中加

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 50030 -j ACCEPT

就不對50030端口進行監控了

請關注微網誌[url]http://weibo.com/u/3272022684[/url]