Flume的开发

大数据相关技术学习

1.flume特点和主要作用

1. 动态采集，流式处理
2. 只能处理log data，（视频，音频等不能处理）
3. 将本地服务器日志文件上传到hdfs（不手动上传的作用就是：实时读取）

2.flume的使用案例1

• 使用flume监听端口

1. 编写配置文件

   # example.conf: A single-node Flume configuration
   # Name the components on this agent
   a1.sources = r1 
   a1.sinks = k1
   a1.channels = c1
   
   # Describe/configure the source
   a1.sources.r1.type = netcat
   a1.sources.r1.bind = localhost
   a1.sources.r1.port = 44444
   
   # Describe the sink
   a1.sinks.k1.type = logger
   
   # Use a channel which buffers events in memory
   a1.channels.c1.type = memory
   a1.channels.c1.capacity = 1000
   a1.channels.c1.transactionCapacity = 100
   
   # Bind the source and sink to the channel
   a1.sources.r1.channels = c1
   a1.sinks.k1.channel = c1

2. 进行启动

   # 第一种写法：
   [atguigu@hadoop102 flume]$ bin/flume-ng agent --conf conf/ --name 
   a1 --conf-file job/flume-netcat-logger.conf -Dflume.root.logger=INFO,console
   # 第二种写法：
   [atguigu@hadoop102 flume]$ bin/flume-ng agent -c conf/ -n a1 -f 
   job/flume-netcat-logger.conf -Dflume.root.logger=INFO,console

3. 借助nc开启客户端

   nc localhost 44444

4. 此时即可监控到传输信息，信息格式如下

   2024-03-11 19:38:04,410 INFO sink.LoggerSink: Event: { headers:{} body: 6E 69 68 61 6F 79 61                            nihaoya }