天天看點

Hadoop 離線安裝 CDH5.1 第二章:cloudera manager與agent 安裝

建立cloudera-scm使用者 (做了分行,這個是一條linux指令,)

[hadoop@master cloudera-manager-5.1.0]$ sudo useradd --system 
--home=/opt/cloudera-manager-5.1.0/run/cloudera-scm-server 
--no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm      

改/opt/cloudera-manager-5.1.0/etc/cloudera-scm-agent/config.ini 裡的 server_port 和 server_host ,把server_port

改成ClouderaManager運作的主機名或ip, 預設server_port 端口是 7182

[hadoop@master cloudera-scm-agent]$ sudo vim config.ini
[General]
# Hostname of the CM server.
server_host=localhost

# Port that the CM server is listening on.
server_port=7182      

為Cloudera Manager 建立資料庫,需要下載下傳Mysql的JDBC驅動,本次從搜狐鏡像上下載下傳最新版

http://mirrors.sohu.com/mysql/Connector-J/mysql-connector-java-5.1.32.tar.gz

解壓

之後找到mysql-connector-java-5.1.32-bin.jar放到/opt/cloudera-manager-5.1.0/share/cmf/lib/目錄下

參考連結:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_install_path_C.html

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_installing_configuring_dbs.html#cmig_topic_5_2_unique_1 資料庫配置方法

[hadoop@master mysql-connector-java-5.1.32]$ sudo cp mysql-connector-java-5.1.32-bin.jar

 /opt/cloudera-manager-5.1.0/share/cmf/lib/      

生成資料庫:(關于密碼這裡有個小小的坑,如果你遇到過不去的地方,私信我)

[hadoop@master schema]$ pwd
/opt/cloudera-manager-5.1.0/share/cmf/schema
[hadoop@master schema]$ sudo ./scm_prepare_database.sh mysql -h localhost -u root 
-phadoop --scm-host localhost scm scm scm       
Verifying that we can write to /opt/cloudera-manager-5.1.0/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cloudera-manager-5.1.0/etc/cloudera-scm-server
Executing:  /usr/java/jdk1.7.0_67/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager-5.1.0/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager-5.1.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!      

啟動SCM server

[hadoop@master init.d]$ sudo ./cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[hadoop@master init.d]$ pwd
/opt/cloudera-manager-5.1.0/etc/init.d      

設定成開機啟動

[hadoop@master init.d]$ sudo cp cloudera-scm-server /etc/init.d/cloudera-scm-server
[hadoop@master init.d]$ sudo chkconfig cloudera-scm-server on      

将用戶端腳本拷貝到客戶機上

先SCP 編好的目錄 到所有機器上 ,然後在每台機器上開啟Agents端:

[hadoop@master opt]$ sudo scp -r /opt/cloudera-manager-5.1.0/  [email protected]:/opt/
[hadoop@master opt]$ sudo scp -r /opt/cloudera-manager-5.1.0/  [email protected]:/opt/
[hadoop@master opt]$ sudo scp -r /opt/cloudera-manager-5.1.0/  [email protected]:/opt/
Starting cloudera-scm-agent:                               [FAILED]      

排錯中..... 看日志,發現少個什麼libpython的東西

/opt/cloudera-manager-5.1.0/lib64/cmf/agent/build/env/bin/python: error 
while loading shared libraries: libpython2.4.so.1.0: cannot open shared 
object file: No such file or directory      

×××,萬能的google,發現有人也在抱怨,因為RHEL6預設指向是python2.6,RHEL5是python2.4況,https://groups.google.com/a/cloudera.org/forum/#!msg/scm-users/Nz8O8eRn1RA/Jk0PUOAhQUkJ

Hi Simon,
On Wed, Dec 21, 2011 at 6:44 AM, Simon <[email protected]> wrote:
    Hello,
    during the installation on all hosts process I get a few installation
    errors:
    1. the master (runs the manager) installs all hadoop packages, but
    when it comes to Installing Clouder Manager agent package... it says
    that it requires libpython2.4.so.1.0()(64bit), of course python2.4 is
    not installed but 2.6 is, is there any workaround?
    This also occurs if I try to install the agent manually via yum
What OS are you running?  Is this an Amazon AMI?
Usually, when folks report this error, they're using RHEL/CentOS 6 (which has python 2.6) but their yum repos are pointed to our RHEL/CentOS 5 repos (5 has python 2.4).      

突然恍然大悟,我的問題應該也是版本的問題,翻到前面的步驟,果然,一個眼花我下載下傳的是el5版本的cdh5.1 

下載下傳離線的cloudera manager安裝包, 放置于/opt目錄,并解壓,改名

1

2

[hadoop@master opt]$ 

sudo

tar

-xzvf cloudera-manager-el5-cm5.1.0_x86_64.

tar

.gz

[hadoop@master opt]$ 

sudo

mv

cm-5.1.0 cloudera-manager-5.1.0

發現了原因就好辦了。删掉

[hadoop@master opt]$ sudo rm -rf /opt/cloudera
[hadoop@master opt]$ sudo rm -rf /opt/cloudera-manager-5.1.0/
[hadoop@master opt]$ sudo rm -rf /opt/cloudera-manager-el5-cm5.1.0_x86_64.tar.gz
[hadoop@master opt]$ ls      

再來一遍,這次一定要小心看清el6

Hadoop 離線安裝 CDH5.1 第二章:cloudera manager與agent 安裝

,下到master.hadoop的/opt/裡

wget 
http://archive-primary.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.1.1_x86_64.tar.gz      
[hadoop@master opt]$ sudo tar -xzvf cloudera-manager-el6-cm5.1.0_x86_64.tar.gz      

裝完檢視

[hadoop@master opt]$ ls
cloudera  cloudera-manager-el6-cm5.1.1_x86_64.tar.gz  cm-5.1.1      

删除我們之間建的用的cloudera-scm

#确認/etc/passwd裡如下内容,删除後消失
cloudera-scm:x:497:497:Cloudera SCM User:/opt/cloudera-manager-5.1.0/run/cloudera-scm-server:/bin/false

[hadoop@master opt]$ sudo userdel cloudera-scm
#重新建立cloudera-scm使用者
sudo useradd --system --home=/opt/cm-5.1.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm      

改改配置,解壓的mysql-connector拷貝進去

[hadoop@master cm-5.1.1]$ sudo vim etc/cloudera-scm-agent/config.ini 
[General]
# Hostname of the CM server.
server_host=master.hadoop

# Port that the CM server is listening on.
server_port=7182


[hadoop@master cm-5.1.1]$sudo cp /home/hadoop/mysql-connector-java-5.1.32/mysql-connector-java-5.1.32-bin.jar /opt/cm-5.1.1/share/cmf/lib/mysql-connector-java-5.1.32-bin.jar      
[hadoop@master cm-5.1.1]$ sudo ./etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]      
[hadoop@master cm-5.1.1]$ sudo rm /etc/init.d/cloudera-scm-server      
[hadoop@master init.d]$ sudo cp cloudera-scm-server /etc/init.d/cloudera-scm-server
[hadoop@master init.d]$ sudo chkconfig cloudera-scm-server on      
[hadoop@slave01 /]$ sudo /opt/cm-5.1.1/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[hadoop@slave01 /]$ sudo chkconfig /etc/init.d/cloudera-scm-agent on
error reading information on service /etc/init.d/cloudera-scm-agent: No such file or directory
[hadoop@slave01 /]$ sudo chkconfig cloudera-scm-agent on      

繼續閱讀