文章目錄
-
-
- 一、安裝flume-ng-1.6.0-cdh5.5.2
-
- 1.準備：
- 2.修改 flume-env.sh 配置檔案：
- 3.驗證是否安裝成功：
- 二、往hbase中導入資料
-
- 1.準備：
- 2.建立配置檔案：
- 3.啟動flume agent：
- 4.産生資料：
-
一、安裝flume-ng-1.6.0-cdh5.5.2
1.準備：
壓縮包下載下傳位址：http://archive.cloudera.com/cdh5/cdh/5/flume-ng-1.6.0-cdh5.5.2.tar.gz
[[email protected] ~]$ tar -zxvf flume-ng-1.6.0-cdh5.5.2.tar.gz
2.修改 flume-env.sh 配置檔案：
主要是JAVA_HOME變量設定：
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ cp conf/flume-env.sh.template conf/flume-env.sh
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ vi conf/flume-env.sh
# 添加：
export JAVA_HOME=/usr/jdk1.7.0_25
注：這裡添加的是你的Java安裝目錄，我這裡安裝的是jdk1.7.0_25，注意Java版本不要過低。如果不知道java的安裝路徑的話，可以在有java環境下執行
echo $JAVA_HOME
檢視。
3.驗證是否安裝成功：
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ bin/flume-ng version
Flume 1.6.0-cdh5.5.2
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: f65a722cd2d1e7ceeda972570a5a6ee01c3a0a3d
Compiled by jenkins on Mon Jan 25 16:38:11 PST 2016
From source with checksum 028a2c6b035a03df1dfa91a3feda3424
二、往hbase中導入資料
1.準備：
[[email protected] ~]$ cd hbase-1.0.0-cdh5.5.2/lib/
# 然後将以下檔案複制到flume中：
[[email protected] lib]$ cp protobuf-java-2.5.0.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-protocol-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-client-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-common-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-server-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-hadoop2-compat-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp hbase-hadoop-compat-1.0.0-cdh5.5.2.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
[[email protected] lib]$ cp htrace-core-3.2.0-incubating.jar /home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin/lib/
# 注：也可以直接把hbase-1.0.0-cdh5.5.2/lib下的jar包全部複制到flume的lib目錄下
確定test_idoall_org表在hbase中已經存在：
hbase(main):002:0> create 'test_idoall_org','uid','name'
0 row(s) in 0.6730 seconds
=> Hbase::Table - test_idoall_org
hbase(main):003:0> put 'test_idoall_org','10086','name:idoall','idoallvalue'
0 row(s) in 0.0960 seconds
2.建立配置檔案：
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ vi conf/hbase_simple.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/hadoop/data.txt
a1.sources.r1.port = 44444
a1.sources.r1.host = 192.168.8.71
a1.sources.r1.channels = c1
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.type = hbase
a1.sinks.k1.table = test_idoall_org
a1.sinks.k1.columnFamily = name
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k1.channel = memoryChannel
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
3.啟動flume agent：
[[email protected] apache-flume-1.6.0-cdh5.5.2-bin]$ bin/flume-ng agent -c . -f conf/hbase_simple.conf -n a1 -Dflume.root.logger=INFO,console
。。。。。。。。。（前面省略，太多了）
:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/logredactor-1.0.3.jar:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/commons-dbcp-1.4.jar:/home/hadoop/hive-1.1.0-cdh5.5.2/lib/jcommander-1.32.jar
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/home/hadoop/hadoop-2.6.0-cdh5.5.2/lib/native
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-194.el5
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop/apache-flume-1.6.0-cdh5.5.2-bin
12/12/13 00:21:08 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x1d630160x0, quorum=localhost:2181, baseZNode=/hbase
12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:47668, server: 127.0.0.1/127.0.0.1:2181
12/12/13 00:21:09 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x3b8fd59e830000, negotiated timeout = 90000
12/12/13 00:21:10 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
12/12/13 00:21:10 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.
12/12/13 00:21:10 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
（啟動成功。。。）
4.産生資料：
[[email protected] ~]$ touch data.txt
[[email protected] ~]$ echo "hello idoall.org from flume" >> data.txt
這時登入到hbase中，可以發現新資料已經插入：
hbase(main):005:0> scan 'test_idoall_org'
ROW COLUMN+CELL
10086 column=name:idoall, timestamp=1355329032253, value=idoallvalue
1355329550628-0EZpfeEvxG-0 column=name:payload, timestamp=1355329383396, value=hello idoall.org from flume
2 row(s) in 0.0140 seconds
![Flume架構介紹[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)