centos之Flume安装步骤

2020年06月15日 113点热度 0人点赞 0条评论
Flume是一个分布式、可靠、和高可用的海量日志采集、日志聚和  和 传输 的系统。
支持在日志系统中定制各类数据发送方,用于收集数据。
同时,Flume提供对数据进行简单处理,并具有写到各种数据接收方(HDFS、Hbase等)的功能。
其核心为Agent,是flume中最小的独立运行单位(JVM)。
核心组件为:source、channel、sink

0. 软件版本下载

http://mirror.bit.edu.cn/apache/flume/

1. 集群环境

Master 192.168.71.10
Slave1 192.168.71.11
Slave2 192.168.71.12

2. 下载软件包

#Master
wget http://mirror.bit.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
tar zxvf apache-flume-1.6.0-bin.tar.gz

3. 修改Flume配置

#NetCat
vim conf/flume-netcat.conf
# Name the components on this agent
agent.sources = r1
agent.sinks = k1
agent.channels = c1
# Describe/configuration the source
agent.sources.r1.type = netcat
agent.sources.r1.bind = 127.0.0.1
agent.sources.r1.port = 44444
# Describe the sink
agent.sinks.k1.type = logger
# Use a channel which buffers events in memory
agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1
#验证
#Server
bin/flume-ng agent --conf conf --conf-file conf/flume-netcat.conf --name=agent -Dflume.root.logger=INFO,console
#Client
telnet master 44444
#Exec
vim conf/flume-exec.conf
# Name the components on this agent
agent.sources = r1
agent.sinks = k1
agent.channels = c1
# Describe/configuration the source
agent.sources.r1.type = exec
agent.sources.r1.command = tail -f /data/hadoop/flume/test.txt
# Describe the sink
agent.sinks.k1.type = logger
# Use a channel which buffers events in memory
agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1
#验证
#Server
bin/flume-ng agent --conf conf --conf-file conf/flume-exec.conf --name=agent -Dflume.root.logger=INFO,console
#Client
while true;do echo `date` >> /data/hadoop/flume/test.txt ; sleep 1; done
#Avro
vim conf/flume-avro.conf
# Define a memory channel called c1 on agent
agent.channels.c1.type = memory
# Define an avro source alled r1 on agent and tell it
agent.sources.r1.channels = c1
agent.sources.r1.type = avro
agent.sources.r1.bind = 127.0.0.1
agent.sources.r1.port = 44444
# Describe/configuration the source
agent.sinks.k1.type = hdfs
agent.sinks.k1.channel = c1
agent.sinks.k1.hdfs.path = hdfs://master:9000/flume_data_pool
agent.sinks.k1.hdfs.filePrefix = events-
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.writeFormat = Text
agent.sinks.k1.hdfs.rollSize = 0
agent.sinks.k1.hdfs.rollCount= 600000
agent.sinks.k1.hdfs.rollInterval = 600
agent.channels = c1
agent.sources = r1
agent.sinks = k1
#验证
#Server
bin/flume-ng agent --conf conf --conf-file conf/flume-netcat.conf --name=agent -Dflume.root.logger=DEBUG,console
#Client
telnet master 44444

阿布

源自灵魂深处的自我救赎。

文章评论