# 大数据-笔记-1

文章地址：(可能更好的阅读体验)

0. <https://ovo.dayi.ink/2023/11/09/big%E6%95%B0%E6%8D%AE/>
1. <https://www.cnblogs.com/rabbit-dayi/p/17822822.html>
2. </posts/0d24d613-663b-5e06-89ba-8c387a265799>
3. <https://blog.dayi.ink/?p=93>
4. <https://cmd.dayi.ink/-BShZBz_ThuNVoGepjYbsA> (本文原始链接，打开速度可能偏慢)


## 1. 安装

![](/media/87/873a43e25b3cdf3e8e9dd21ff54dea78224a9b5c987eaaf5816b87ef3c051c1e.png#vwid=1110&vhei=693)

## 2. 查看三台虚拟机IP

![](/media/aa/aac5fb42933366e2c4d0591b914aef9ec172459d8dd29ac20833b63ee936412b.png#vwid=800&vhei=648)

设置静态IP

![](/media/92/92893d37eebc6c12d5db27de6f1571cb7690f7e570fe41404d72bf3091d80bbd.png#vwid=1066&vhei=669)

## 3. 修改IPV4为manual

![](/media/ae/ae38326766333489c6a93662bdfd12df5b60950e6aa6067d1a697140dbcb99e8.png#vwid=1005&vhei=620)

IP 地址如下：

```bash
gataway : 192.168.83.2
mask : 24
master: 192.168.83.10
slv1: 192.168.83.11
slv2: 192.168.83.12
```

![](/media/17/17188e514657fd608fa494c787c0241367df2e836cb28215e75cf0cd2cddf543.png#vwid=1056&vhei=638)

## 3. 关闭防火墙

```bash
su
[输入密码:123456]
service iptable stop
service iptable status
```

![](/media/73/73a50d9450859a81839b55e39431bfc88d1c139ff238c2b355110954a483a5ee.png#vwid=721&vhei=552)

===

![](/media/c2/c228ecce378aaa08de971b4454dbdabea5e58a45baab02f2e1acb4ffa35839ca.png#vwid=585&vhei=213)

----

slave1 slave2:

![](/media/fd/fd9b2c6b16fcbf1eca7e8c33a2f61023bf84d97daea692a040cdf85e33c2839e.png#vwid=853&vhei=643)

![](/media/60/60f283cfa281a8327c272a84449f50beb2841b09e85da7d6a1c6ceccd7fcde68.png#vwid=656&vhei=577)

---

## 4.关闭防火墙自动运行

```bash
chkconfig iptables off
chkconfig --list | grep iptables
```

master:

![](/media/c5/c5eaf651192514f1d8c2deb4e02b518fcb52742fa211ff8732ac0e20c9acfd92.png#vwid=650&vhei=364)

slave1:

![](/media/61/619b79bbf94bb4bb3dc58468ff8befe016fc70ed61f64b4658fc313c005ab5f3.png#vwid=790&vhei=583)

slave2:

![](/media/b9/b92c6e460ad6ce5e79969ab5d6c29b20567df2ec754d22e12819489ccb77295f.png#vwid=915&vhei=667)

## 5. 配置主机名

```bash
#hostname 
vi /etc/sysconfig/network
cat /etc/sysconfig/network
hostname dayi-bigdata-master
hostname dayi-bigdata-salve-1
hostname dayi-bigdata-salve-2
```

输入 i 进入编辑模式
输入 :wq 保存文件

发现已经做了）

---

master:

![](/media/8c/8cf85570684e2583fbe97f0511e31856ece392e68d6b7332935f3642581d6fef.png#vwid=1204&vhei=590)

slave1:

![]![](/media/d2/d2a80f5d164553e066164214b51e8b4e632fce048e1230c0b7e9965d45e44077.png#vwid=994&vhei=630)

slave2:

![](/media/1a/1a2dce869855f4e7fa078b724c3755ff9069a8dd9c81120403fd637fde6ef4ae.png#vwid=898&vhei=623)

## 6. IP 地址 HOSTNAME 绑定

```bash
vi /etc/hosts
192.168.83.XX master
192.168.83.xx slave1
192.168.83.xx slave2
```

i : 进入编辑模式
:wq : 保存文件

![](/media/7a/7a2db4060568a1d5ab5c657a73a56343d72526443412d6c2deb5e9e8a9c2a1bd.png#vwid=868&vhei=582)

master:

![](/media/bd/bdd1b8c2b9e1d4d3bf996514da8db0d7a6df8b6a63d3a9677a3bd2ec8453ec71.png#vwid=1011&vhei=701)

slave1:

![](/media/43/43597e60545dd9786b6d06e969c6e8e1c5e99a2a5aa9e736fe3658f1a9ecfb33.png#vwid=962&vhei=674)

slave2
![](/media/1a/1a2dce869855f4e7fa078b724c3755ff9069a8dd9c81120403fd637fde6ef4ae.png#vwid=898&vhei=623)

chk1：
![](/media/16/16ffb892792556c7365e9011e564c4283b9a8f5cb2411b081e1b1a4315c08742.png#vwid=839&vhei=577)

## 7. 修改windows的host

打开文件

路径打开这个：

`C:\Windows\System32\drivers\etc\`

`C:\Windows\System32\drivers\etc\hosts`

先复制到桌面，然后修改完再复制回去（真就同步，我先想出来的pvp写的PVP）

![](/media/53/53390ad2b40e1b1feeeb2798da4c1c2ba9c2da14c7c47efac032387f460825ad.png#vwid=633&vhei=499)

![](/media/f5/f5534a3323aadb6608b758c8bd5effd85341fe9a5e2172498f26d4d782bef0d4.png#vwid=840&vhei=645)

## 8. 配置远程SSH登录

哦，直接登录

![](/media/d0/d029ea5bdf9c32ff2fe492a0fc9ea7e0209dccdf169918e468a9318fa8185b0c.png#vwid=723&vhei=592)

~~但是不是限制密钥登录..登不进去呀~~
哦哦，可以直接登录，那没事了

![](/media/ba/baa8a57e43b0ffa87da582c09c1bfaf4b0b06cde1388893e7283413ef022a619.png#vwid=672&vhei=470)

添加三个主机之后

![](/media/64/64c0f7f33d296852bccb1360a0a8f7a9e24735631acf3c670798337a5c7d6655.png#vwid=680&vhei=470)

ok）

![](/media/f1/f11f447e7dfbf6cc5475762916dffaaeb87e1337f6cdc2718e1816f3ac784312.png#vwid=700&vhei=528)

锟斤拷烫烫烫

问题解决：

chk:

1. windows 的hosts文件`C:\Windows\System32\drivers\etc\`
2. 虚拟机的IP地址是否对应

----

三个节点窗口：

查看 `->` 交互窗口 `->` 右键底部的窗口，发送交互到所有标签

因为虚拟机里套虚拟机不好截图，菜单截不下来（

![](/media/d4/d4e65be7888e91b92558454655498d77981a4c5af433251bda4ed163b2aebe35.png#vwid=759&vhei=673)

----

这个分割线没有用

我猜的

输入

ssh-keygen

生成RSA密钥：

![](/media/79/79456640cd50036a080e99755c592a50e86a1cf83190ef33f3b545f0a9a03d75.png#vwid=595&vhei=377)

复制公钥：

![](/media/4c/4c3f67f4c36dbf976b1553ef42ccb67658721e414d0165e1773588ef2b3170a9.png#vwid=590&vhei=156)

输入到虚拟机里的

`.ssh/authorized_keys`

---

## 9. 时钟同步

### 9.1 手动时间同步

```bash
su root
[输入密码:123456]
date #显示系统时间
hwclock --show #显示硬件时间
#如果时间不一样的话这样省事，直接NTP同步时间：
#设置硬件时钟调整为与本地时钟一致
timedatectl set-local-rtc 1
#设置时区为上海
timedatectl set-timezone Asia/Shanghai
ntpdate -u  pool.ntp.org
date
## 手动同步
date -s 20230907
date -s 9:40
hwclock --systohc
```

![](/media/5d/5d63d343d1f7e9fa20ad158c8ac699e46983885a79b4a640c01fce2d2a5d5231.png#vwid=759&vhei=673)

显示系统时间

![](/media/ed/ede7ce8072a04b08246e3233d273792eacdaf0dfbae2084fa96d53d553963341.png#vwid=486&vhei=219)

显示系统硬件时间：
![](/media/7f/7f8ede3509ef8a4e97cde6ba3b3be2944ac1b46ec4dc01a0e433ddd7ca24114a.png#vwid=580&vhei=246)

（全是锟斤拷）

NTP同步时间
![](/media/bf/bf61ba0cc1e700bf3a04ab30a8920521d8283120db2e3140ba999972fa5aa783.png#vwid=697&vhei=307)

### 9.2 自动时间同步

下班
> 2023年9月13日14:09:57
时间同步

```bash
timedatectl set-local-rtc 1
#设置时区为上海
timedatectl set-timezone Asia/Shanghai
ntpdate -u  ntp.aliyun.com
hwclock --systohc
```

![](/media/45/45e0e166487e5c7247fb99ac904178da08e0061fae5081d220c0da7aa4e15143.png#vwid=768&vhei=670)

### 9.3 配置自动时间同步

两个从节点

```bash
crontab -e 
# 输入i进行输入模式
# ctrl+shift+v 粘贴
0 1 * * * /usr/sbin/ntpdate ntp.aliyun.com
# 输入:wq 保存
```

## 10. 免密登录

```bash
#直接输入exit
# 或者
su dayi
```

![](/media/9e/9e72168dcfe2c1c3750f0844c23676371e133f23a2501c5e1a2d78f1c4395012.png#vwid=768&vhei=670)

### 10.1 设置SSH免密登录

```bash
ssh-keygen -t rsa
#然后回车3次就行了
```

![](/media/bc/bcffe8787b267c899c695f095595819ac75055750e3d2de5b3330a4ff23df529.png#vwid=635&vhei=292)

```bash
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
cat ~/.ssh/authorized_keys
```

![](/media/bb/bb30f79c5019e3d338bebd47de7479b15f397e9acf3dfe1d6d37f108601f277d.png#vwid=760&vhei=169)

我觉得这个公钥有点用：(应该没啥用了)
`ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6yR468UQBZ/9KSG71FD0UVlv9N0I6q2RfA94yLT7uGhja9vzBJSP9mDg8RF8+Z6p5+katfYE7YLzjtLNMtC5lTjkwW8WHFyCGUP0QEcAIH0ZdDVn3nwHG9k+b2XfpLNKOieWYqUoixRSzIecUd5iq3WDe4dUjgBmGhfouo+FtQob/q8OOg2iJszl86ad8dE9W2BRS3VU5q6/OZmPp8uJcfXAl4/bHJq56+FNPSwk9b+umAsiH+bqeVCkW6JJd/Tw7DGkhYACGxleF5onBtiLKwbMZ+RanWiFm9AZqod86rcmZ9IPYaWf/QXgyun5vrNBBgBT+a8CBsRoBpFk0X7CCw== dayi@dayi-bigdata-master`

```bash

ssh-copy-id dayi-bigdata-slave-1
ssh-copy-id dayi-bigdata-slave-2
```

![](/media/99/99223c2e80bb83c8f11449104fbc3f49ff58fc0ce180d65286ec987cd7186346.png#vwid=768&vhei=670)

### 10.2 测试是否免密登录

```bash
ssh dayi-bigdata-slave-1
# 退出 exit 或者 ctrl+d
ssh dayi-bigdata-slave-2
# 退出 exit 或者 ctrl+d
```

![](/media/25/250d4e8268e317534c9d661e96eb71506e12d8636a7cc343411eaa2f634d3455.png#vwid=621&vhei=200)

## 11. 安装JDK（普通用户）

新建文件夹

```
cd ~
mkdir -p ~/resources
cd ~/resources
```

复制文件
sftp://dayi-bigdata-master
sftp://dayi-bigdata-slave-1
sftp://dayi-bigdata-slave-2

![](/media/9f/9f6affb9b84b09ab8f83b868a20d515871edad986ce2352637ed57e14cd1a1d3.png#vwid=1171&vhei=788)

![](/media/0d/0d0026d390f407d5da4a87bc932f252897ae6e9ef4b6c2879579e3d60b2b5a69.png#vwid=1284&vhei=864)

查看下当前的文件

```
ls 

[dayi@HOSTNAME=dayi-bigdata-slave-2 resources]$ ls
jdk-7u71-linux-x64.gz
[dayi@HOSTNAME=dayi-bigdata-slave-2 resources]$ 

```

解压

```bash
tar -zxvf jdk-7u71-linux-x64.gz 
# 重命名
mv jdk1.7.0_71 jdk
```

### 11.1 配置环境变量

```bash
# (可选,出错请忽略）按老师目录移动文件
mv ~/Home/resources ~/resources
vim ~/.bash_profile
#按 i 输入
export JAVA_HOME=~/resources/jdk
export PATH=.:$JAVA_HOME/bin/:$PATH
#input: :wq (保存)
source .bash_profile
```

### 11.2 验证JDK安装成功

```
java -version
```

![](/media/ac/ac7f30fa3e3031578ff27b679a823db1be15f7af0d3521b3ceb56672a99c5f4b.png#vwid=537&vhei=146)

## 12. 安装hadoop(master 普通用户)

只做主节点。

```bash
tar -zxvf hadoop-2.6.4.tar.gz 
mv hadoop-2.6.4 hadoop
```

编辑bash配置 文件

```bash

# 记得在普通用户下
vim ~/.bash_profile

# i 输入

# 编辑模式下ctrl+shift+v
export HADOOP_HOME=~/resources/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

#:wq保存文件

```

### np++连接

- 打开notepad++

![](/media/a1/a1f7797988f6211f657199aa3d0045603b8bae14b6591d281b9545ac90c0f66a.png#vwid=403&vhei=193)

输入配置文件
![](/media/cb/cb600ce88a0c96e37a33c12ce8345d91846a485f740d0785d71326cfe79f0d37.png#vwid=531&vhei=426)

点这个
![](/media/f3/f321fafa133cad528a26edb96921e2aad981a58f93b4d4b66faa61801c07347a.png#vwid=232&vhei=125)

然后到这里来：

`/home/dayi/resoureces/hadoop/etc/hadoop`

![](/media/18/18c948ee5b51f9f3d7a3c40bbb0692a728e057a0baec2456e45622356bad40c4.png#vwid=481&vhei=448)

### 修改`hadoop-env.sh`

hadoop-env.sh 25 行

```bash
export JAVA_HOME=~/resources/jdk
```

![](/media/73/733bba01d12635e01d35ab35f1b94bc0e665f030c2fdc3d85f7a1ea1c89b00f4.png#vwid=580&vhei=571)

### 修改 `yarn-env.sh`

`yarn-env.sh`

- 23 行

```bash
export JAVA_HOME=~/resources/jdk
```

![](/media/b8/b83461abe5b1c951442d59c736fd244057f35631d9aaf770ab577147f855a8e7.png#vwid=737&vhei=715)

### 修改 `core-site.xml`

`core-site.xml`

```xml
<configuration>
  <property>
    <name>fs.defaultFS</name>     
    <value>hdfs://master:9000</value>   
  </property>
  <property>
    <name>hadoop.tmp.dir</name>     
    <value>/home/wg/resources/hadoopdata</value>   
  </property>
</configuration>
```

![](/media/d4/d431ead4919fadabd4b5c0716ac61fe316c023e645a6c8c3e32c9bbf3c8f6e03.png#vwid=920&vhei=531)

### 新建data文件夹

```bash
# 记得自己改
mkdir -p /home/dayi/resources/hadoopdata
```

### 修改`hdfs-site.xml`

```xml
<property>
   <name>dfs.replication</name>
   <value>2</value>
</property>
```

![](/media/0b/0b33c7464b8f080f2fecc65da790ac3f74c4de5815d47726b05cd1f2c416f0a7.png#vwid=764&vhei=585)

ctrl+s

### 修改`yarn-site.xml`

```xml
<configuration>
<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
   <name>yarn.resourcemanager.address</name>
   <value>master:18040</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:18030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:18025</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:18141</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:18088</value>
  </property> 
</configuration>
```

![](/media/4d/4d90430fe75c5de44d06f4511aefaded0be9fffff0f5b21188cc188889104f7f.png#vwid=762&vhei=682)

### 修改`mapred-site.xml`

- 打开`mapred-site.xml.template`

- 新建文件:

![](/media/0c/0c55bc2e2364ddf835b82cf0f52ec47ad20e5d2d8d6503ee5ce0bd97857ee7bc.png#vwid=474&vhei=324)

- 输入内容

```xml

<configuration>
 <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
```

![](/media/f7/f71e8c0fa687f7578d4f9837e2d0d3cf742f2fc84b52106ba8b6860791b96645.png#vwid=1360&vhei=826)

### 打开`slaves`

添加

```
slave1
slave2
```

![](/media/e1/e1f9ddc6eb72bbae691e6513d565f4fdea48e9745a708c856348dace4f2034fc.png#vwid=1380&vhei=403)

### 将配置好点hadoop发送到从节点

```bash
cd ~
cd resources
scp -r 

scp -r hadoop dayi-bigdata-slave-1:~/resources/hadoop/
scp -r hadoop dayi-bigdata-slave-2:~/resources/hadoop/


```

![](/media/1f/1fe6a3caa8146a30a1bed8a4ff0f31cb049bd996b6b552b0b54829806aa37c93.png#vwid=831&vhei=523)

## 13.启动hadoop

1. 格式化文件系统（master,普通用户

```bash
hadoop namenode -format
```

![](/media/2c/2cc0624e63d25e4ec43ae95af6ad376e19d0b5eb8ec12bc52aa34068cb3436f7.png#vwid=831&vhei=523)

![](/media/e4/e4f9a3b28f6e97ae3c713b8935de50a67df289bf266f9d6fa7eb84f28d7a564a.png#vwid=911&vhei=540)

2. 启动hadoop

```bash
cd ~/resources/hadoop/sbin/

./start-all.sh


~/resources/hadoop/sbin/./start-all.sh

```

![](/media/f8/f82635796ecf0cb38562342802434074880e3de9a43a8dcfc06fa20372efb7ba.png#vwid=1839&vhei=576)

3. check

```bash
jps

```

![](/media/bd/bd4b64fa6178045d19a9b7d88e0f51d1e07e8d15c26e56fe6a95ccaa354e9fe3.png#vwid=558&vhei=200)

![](/media/b2/b2f149ff09c2fc04e509ba987cc6e4d2c89d19d9443e847a79c75e48370f94e8.png#vwid=842&vhei=253)

![](/media/bc/bc57c13d55b007ccfb4a8f6fe9bab211ca9daec152414ec9f37ee71065e7f366.png#vwid=584&vhei=229)

<http://master:50070>

![](/media/36/363092aa0c0dd4499904fb5a1bf9bb5e3a0c46fef52229de32520fc8758fd8fd.png#vwid=1149&vhei=723)

![](/media/36/368c9b60eaa02843d649b06b5a31d5507f554b9c9668d8ba47d831f60f34f7f6.png#vwid=1207&vhei=689)

### 13.1 验证

```bash
#启动
~/resources/hadoop/sbin/./start-all.sh
```

<http://master:50070>

### 13.2 修复

```bash
~/resources/hadoop/sbin/./stop-all.sh

rm -rf ~/resources/hadoopdata
rm -rf ~/resources/hadoop/logs

hadoop namenode -format

~/resources/hadoop/sbin/./start-all.sh
```

结果：
![](/media/ff/ffe7bc7a6d84838c6d7bd9d86f8a9c2678cddd4b8ad2922b1cb901339cfd7e0a.png#vwid=1920&vhei=1048)

## 14 HDFS

先保证你的节点有两个以上哦
![](/media/1c/1cb0d0db2dd49a349c31bc4a6be4f0082125fa2b869d7b3c4ad3e5d3d8980a39.png#vwid=1064&vhei=782)

1. 浏览文件

```
hadoop fs -ls /
```

好像没什么东西

![](/media/b7/b77ee953dd07b4b0af52fa255e3e596d88b2a94fec2b114713dab737db793721.png#vwid=592&vhei=295)

这里好像也可以看：

<http://master:50070/explorer.html#/>

![](/media/e4/e43830d6677a2c0f467003e1d7eb0ca06c60f3d083a02335df9875e448f39ab3.png#vwid=1306&vhei=648)

2. 创建目录

```bash
# 创建目录
hadoop fs -mkdir /ovo
hadoop fs -mkdir /a

# 查看目录/
hadoop fs -ls /
```

![](/media/b7/b72b74353acf4fd349830b277ec6e61d16be7d7d5302541f4e9eac68edd16959.png#vwid=847&vhei=319)

主页查看文件
![](/media/d1/d11c2e1abdb2dcd17801e2045f98c1d837e7724b39d3da4efdb959f21c52b3a3.png#vwid=1401&vhei=639)

3. 上传文件？

1) 本地新建一个文件

    也可以直接图形界面去干

    ```bash
    cd ~
    mkdir b
    ls -alh b
    cd b
    echo ovo>>ovo.txt
    
    # i 输入  esc->:wq 保存
    vi test
    ```

    ![](/media/ea/eaef8d174e4bc78f6133d9a9899d64340a76ea41e218c77f35f0e0b8a1587005.png)

    查看目录和文件：

    ```bash
    ls -alh
    cat test
    ```

    ![](/media/a6/a653cb92670b680ee18a9c0b0114ff592414673c1d33cd69a86f52fe57535b70.png)

2) 将test上传到文件系统

```
cd ~/b
hadoop fs -put test /a
hadoop fs -ls /a
#或者主页打开
```

![](/media/65/652903d26ac7cc5848fdf2ef78c858df99146d357183b1cdf23d7bab2fb4ffe9.png#vwid=894&vhei=362)

![](/media/58/58606e42d3768621206eef89f301384ba1f110021bb40be1b98f1482d0ffc2c5.png#vwid=1659&vhei=643)

![](/media/90/90b2f57bf15aba43635b9ba8ec95b5e4d3dda1dbbb6db0fbf134430fb2cf8d3e.png#vwid=1072&vhei=789)

4. 查看文件内容

```bash
hadoop fs -cat /a/test
hadoop fs -text /a/text

```

![](/media/ca/ca3b5d51ef28e948182d4efdd0b00bfe1fe6fa73a9bca628700dbd91226375ac.png#vwid=811&vhei=683)

5. 下载文件

```
hadoop fs  -get /a/test test1
ls -alh
cat test1
```

![](/media/00/00dc29b4cc498199121b300a33cfb8f6fe780cf398ec8768fecae9d50f64013f.png#vwid=773&vhei=483)

6. 修改文件权限

```bash
hadoop fs -ls /a
hadoop fs -ls /

hadoop fs -chmod +? 
```

![](/media/60/60883f7f95532ebd1581077ad9171d16d8470ade0bf6da48f03fb96d5044fcb8.png#vwid=841&vhei=231)

![](/media/67/67a9fc78dfe7e655e90acea4420d59363ff3d46984f1fdcfd39eb28ccb878881.png#vwid=798&vhei=320)

```bash
-rw-r--r--

- 文件/目录


rwx : read write exce（执行）

rw- ： 用户权限 (u) 
r-- : 组权限 (g)
r-- : 其他人权限 (o)


```

增加执行权限:

```bash
hadoop fs -chmod u+x /a/test
hadoop fs -ls  /a
```

![](/media/5f/5f98537f90ccb0a1efb51b0c88b478ecbb26ac26a2f1a9ef8980122054076db6.png#vwid=885&vhei=211)

组权限+wx
other权限+wx

```bash
hadoop fs -chmod o+x /a/test
hadoop fs -chmod g+x /a/test
hadoop fs -ls  /a
```

![](/media/e8/e83710a0356cf48f61c10e812201e81a26e1cb1174b474f1554d145596b64823.png#vwid=899&vhei=371)

去除所有人的执行权限

```bash
hadoop fs -chmod a-x /a/test
hadoop fs -ls  /a
```

![](/media/6a/6a9b2d63cda6208d604a3eea67e614073f1bfa89eb0c0fb9342e265b192839c6.png)

数字：

```
000
rwx

自由组合，二进制转十进制

比如全部权限

111 = 2^0+2^1+2^2=1+2+4=7
rwx

```

权限

```bash
hadoop fs -chmod 644 /a/test

#很危险的操作哦）容易被黑掉
hadoop fs -chmod 777 /a/test

hadoop fs -ls  /a

hadoop fs -chmod 644 /a/test
```

![](/media/aa/aa99fc98e252638db454e0f734cf7ee038a4e99347984b2e9df1831167fc95ee.png)

7. 删除文件

```bash
hadoop fs -rm /a/test
```

![](/media/56/568183f03f7762c54caa341058b84f639e239939a428cbdf8f205d84746fb7e9.png)

8. 删除目录

```bash
hadoop fs -rm -r /a
hadoop fs -ls /
```

![](/media/77/77361b39e6a7b6faa2e2df0cad9f0104049930de1bc536193d5c695b14490936.png)

## 15. mysql安装？

1. 新建mydb文件

普通用户

```bash
mkdir ~/resources/mydb
```

2. 复制文件，把文件粘贴进去

> 能直接粘贴到虚拟机就不用这个

这里软件是filezila

![](/media/ae/ae8e99ed5048a899eac14b5494f89d9c6e8bb2e5539273dc58d5a0d240bb779a.png)

![](/media/b2/b2716832785f7682b576a89a542d281fdac430f25d6a806c8768c9355fefc45f.png)

如何连接：
![](/media/b9/b9370f4dcf002a71b58ea6f14629efda896222adf7fe6f79d4fa4f3d3dde3740.png)

3. 检查系统里是否有自带的mysql

```bash
su

rpm -qa | grep -i mysql
```

![](/media/7e/7ecb4eeb381a4b50a3fec31b98f4ca3890afe601767732d35f958870731c7bd9.png)

有一个诶

4. 有的话就删除

>建议打个快照先

```bash
rpm -e mysql-libs-5.1.71-1.el6.x86_64 --nodeps

#然后看看还有没有

rpm -qa | grep -i mysql
```

![](/media/d8/d8303bb94b931ed45bfd81af0a08563b89ba3db987cc20b950efb8ef9ac6b9e1.png)

5. 安装4个文件

common , libs , client, server

```bash
#(在普通用户下)
su dayi(按你用户名)
cd ~
su 
cd resources/mydb
ls -al
```

![](/media/42/42f1aedfd9d488d4838f5d86dbdb6affb8a11b5bb44be3b59acba37c27ca7a21.png)

安装：

```
rpm -ivh mysql-community-common-5.7.13-1.el6.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.13-1.el6.x86_64.rpm
rpm -ivh mysql-community-client-5.7.13-1.el6.x86_64.rpm
rpm -ivh mysql-community-server-5.7.13-1.el6.x86_64.rpm
```

![](/media/61/6187e91f9b06ba5cead79d687e52d15cdfcdd8108cc3ed0744c9820f21fceb92.png)

6. 启动server

```bash
service mysqld start 
```

![](/media/a3/a3cd2d9f0768b9bd4e925d6bb1029c5962c47158167a3a3b31531970366d6fe5.png)

###  修改mysql默认密码

```bash

service mysqld start
sudo cat /var/log/mysqld.log | grep 'temporary password' 

```

![](/media/d1/d1b8fd5ce4594ffed133d5c6cdcafe95b7c403ea9b53a04090ad0184feb81bad.png)

![](/media/55/5566dfe60314c910ec6c191f4e6a6589dd4eada47665906d6eb8d2cdf891cb33.png)

```bash
mysql -uroot -p"di<aPM7a&p/("
```

![](/media/91/9136d8e12e58dd63546bcc663a52659084378a678b52b82b51807c93c1ce8530.png)

修改密码：
用户名:root
密码:wg1%Bigdata

```sql
ALTER USER 'root'@'localhost' identified by 'wg1%Bigdata'; 
```

OK 就可以

![](/media/4e/4e2f8543453f8d195205ba0b0e055a7f4fed98387667fb7d7bafd27993ef6948.png)

### mysql中创建用户

用户名: hadoop
密码: hadoop1%Bigdata

```sql
Grant all on *.* to 'hadoop'@'%' identified by 'hadoop1%Bigdata';
grant all on *.* to hadoop@'localhost' identified by 'hadoop1%Bigdata';
grant all on *.* to hadoop@'master' identified by 'hadoop1%Bigdata';
flush privileges;
```

![](/media/d7/d77b278724a23ca6325b7329938a0bd68ffde1d4ac3043739ccb5908bdf789fa.png)

尝试登录

```bash
quit;
su dayi
mysql -uhadoop -p"hadoop1%Bigdata"
```

![](/media/6a/6ae0d78fc3dddaa4a4d5f6523ac3c136718d664880ccc04224547e9952d3d5c1.png)

### MYSQL数据库里创建hive元数据库，叫hive1

查看当前数据库

```sql
show databases;
```

![](/media/a9/a9e5a89460dd6fdfcc48a990555349443a0c759ba121eb9d09bdf443211e98bd.png)

创建数据库

```sql
create database hive1;
show databases;
quit;
```

![](/media/8b/8bd9b161792c7c110de16671cc584cc8a2c8707830eb5aac1ed76964d612cb4b.png)

## 16. 安装hive

### 1.复制hive文件到resources

![](/media/d8/d808921f7eec5feaf586d830c9eb38e0b42a47133780d897317496c321bd9c93.png)

### 2.解压缩

```bash
#进入目录
cd ~/resources
# 解压缩
tar -zxvf apache-hive-1.2.1-bin.tar.gz
# 重命名（移动）
mv apache-hive-1.2.1-bin hive
```

![](/media/d2/d2f7494977aae32e1e9bf4156fee11bd6c5f2989a3c2752611dcea7ffa310d14.png)

### 3. NPP改配置文件

连接：
![](/media/c6/c6c8bd8403604d969642a615aa7a11de6a46d4141b404a2e8a374aced92c92e2.png)

打开目录:`~/resources/hive/conf`
![](/media/e5/e5981472bebe1b14e505f93e4ec58b75cdf6616bdbced3c2c111376826250d00.png)

新建文件，右键文件夹`hive-site.xml`

![](/media/35/35dce3bc3ebbb156677a47e5653dd845feb1875c81d23f0ad9a231bb67a9329f.png)

内容如下：

```xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> 
  <property> 
    <name>hive.metastore.local</name> 
    <value>true</value> 
  </property> 
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://master:3306/hive1?characterEncoding=UTF-8</value>
   </property> 
   <property> 
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value> 
   </property> 
   <property>
      <name>javax.jdo.option.ConnectionUserName</name> 
      <value>hadoop</value> 
   </property> 
   <property> 
      <name>javax.jdo.option.ConnectionPassword</name> 
      <value>hadoop1%Bigdata</value> 
   </property> 
</configuration>
```

![](/media/30/30232a39877dab79b7cce7056a45b1e9c0ca502317015202b128b2ff6747e13e.png)

如果你的主机名不一样记得改这行：
`jdbc:mysql://master:3306/hive1?characterEncoding=UTF-8`

```bash
jdbc:mysql://master:3306/hive1?characterEncoding=UTF-8
driver      host    port database_name  
            
hadoop 用户名
hadoop1%Bigdata 密码
```

### 4. 复制驱动

`mysql-connector-java-5.1.42-bin.jar` -> `~/resources/hive/lib`

![](/media/06/0688c5ff55ac413d1d454b59e8146869811debc8253b152e8bcd88969a0acfbd.png)

### 5.配置bash

配置环境变量

- npp
![](/media/ba/ba2dd0a667fe3360cecc060fd1b52f060a1c36b701ae4d7cb4a16cd141fd4ae8.png)

```bash
vim ~/.bash_profile

#添加这两行
export HIVE_HOME=~/resources/hive
export PATH=$PATH:$HIVE_HOME/bin

# 重载配置文件
source ~/.bash_profile
```

![](/media/98/98a0d1d5cfe924c5f76f19d87d914502b17174186bf3c2a216943810fe096fd0.png)

然后查看hive版本，如果能查看就说明环境目录成功了

```bash
hive --version
```

![](/media/c3/c3105cb6227057decdcade0af41f565350a50750a23733170320c0923ae1f0f8.png)

搞得和预言家一样

### 6. 替换文件

```bash
rm ~/resources/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar
cp ~/resources/hive/lib/jline-2.12.jar ~/resources/hadoop/share/hadoop/yarn/lib/
```

![](/media/39/3955e5b05a505ec0e0206d611554366843e89f73fab0d2964a34da89a03378d7.png)

### 7. 启动hive

```bash
#1.下载原神(hadoop)
~/resources/hadoop/sbin/./start-all.sh

#2. 安装原神（mysqld）
service mysqld start

#3.启动原神
hive
```

![](/media/ea/eaeb6355efb1109c3bd048bf5cd55f0fe2687cb9558c52c282a67d78c65b9f23.png)

![](/media/01/01046c70171babe950b85fc91c7125eb8bdb9f523e65ebaf451b06cbf24f4781.png)

原神：
![](/media/ef/ef1640c22344438fa5f216753ccec68ae856eecaa233e113ffd542228278af1d.png)

![](/media/65/65a6703799901b0e5a4ba633c3212350e735bc8a04906b0c956e13eb1cffa654.png)

![](/media/5b/5bfa8c0170bcd7a1c7d0f664e2091262ef8fd57d116a0120cd61f369c23f1c6f.png)

### 8.启动

```bash
~/resources/hadoop/sbin/./start-all.sh
service mysqld start
hive
```

![](/media/79/79abf35d01baefff697b4fbf6ea008c70f4175a92a7d68c0604fbcf8524e8276.png)

![](/media/ad/adf9fc6874ce202ed77fdaa7ce6b62fd7fa25c60e2f788f44ecd00f04d437b36.png)

CRT 调整：

![](/media/55/5523907a50ce49fbeeb94949e8951a6b9b129f2f49799948654bad369eb7f4a4.png)

![](/media/9f/9f799adb59dd6debc3e55ca773263f434c7f66b2a5fcd467f356154faf932be7.png)

![](/media/a4/a48773e207f8306fd1d1252bf13be995971ba60422ec1d5eebfac7bb8db65726.png)

## 17.hive应用

```bash
hive>quit;
```

### 1. 新建文件

```bash
cd ~
mkdir cc
cd cc
vi stu
```

`i` 输入 `ESC`+`:wq`保存

![](/media/30/304178455e0902cc21dd0885817ef884edf25bb704e30cc6d7f95702ce5d2686.png)

查看文件

```bash
cat stu
```

![](/media/02/0291ee5d0ac62c679e014a849ea90ecd84c9bb68228d0ec2114a7a2b47c4614a.png)

### 2.启动hive

```bash
hive
```

命令:

- 查看数据库

    ```sql
    show databases;
    ```

    ![](/media/3b/3b7d52e9e2534365f79f6942c18734c0b81a8e60f5521b7e695f4236562f75c3.png)

- 创建数据库

    ```sql
    create database studb;
    show databases;
    ```

    ![](/media/01/017047aa97b272023306a4a0dd94c6ae45459b42babd6448dc76515e335b43bb.png)

- 切换数据库

    ```sql
    use studb;
    ```

    ![](/media/3a/3a9c48e09a723e457f3ba6e70d68dd763b4f9ce96ff4bbff1a27d6b6a18512ec.png)

- 查看当前数据库里的表

    ```sql
    show tables;
    ```

    ![](/media/c9/c9c09ea83f9ea40392ef945f1778cd5769791ca1ad3812498ff3a72270a964a2.png)

- 创建表
    `\t`来区分列

    ```sql
    create table if not exists stu
    (name string,age int,sex string)
    row format delimited fields terminated by '\t';
    show tables;
    ```

    ![](/media/f8/f86535aa4fc1240d6a77eec68c8c8af2e0935de7506ec21bb73d425b94002a68.png)

    查看文件：
    <http://master:50070/explorer.html#/user/hive/warehouse/studb.db/>
    ![](/media/98/98c6a4c505bc599b9563178d746ea8ce5edfa0e46f2f31ed2f5dbd6bc72043f8.png)

- 查看表的创建信息等

    ```sql
    show create table stu;
    ```

    ![](/media/cc/ccc9bfb785b407b05897720c014b6610a0947d50c7b90534926cf5684c5a0fcf.png)

- 查看表结构

    ```sql
    desc stu;
    ```

    ![](/media/1a/1a96f765c0614156069d6462ffd6768bef10f8ed971f86e41ce2bd1827b481bc.png)

### 3. 加载数据

> 在hive下

- 从文件加载数据

    ```sql
    load data local inpath '/home/dayi/cc/stu' into table stu;
    ```

    ![](/media/6b/6be2edec498d96a44c15c438f27654b9a9bd30f814d45f093368f680120c497f.png)

- 查询表

    ```sql
    select * from stu;
    ```

    ![](/media/03/03f2ac6e85d0e5af85c4b8e264eb761bcef1e9d9017309903fc3c0d744de6eec.png)

- 删除表

    ```sql
    drop table stu;
    ```

    ![](/media/a2/a280fcd44337e7d507ab22aac3fa420670d5aa9a58a513bb1e30e052bfc23111.png)

- 删库跑路
    > database没有表，为空

    ```sql
    drop database studb;
    ```

    ![](/media/d2/d271e301dd4b8d0d296eba6ef11459cabc453a9328e85dc2cf48b75f676e89fc.png)

    > 有表的情况下

    ```sql
    drop database studb cascade;
    ```

### 4. 大数据分析-hive应用2

搜狗
![](/media/48/4882f1d70eceb58e509a26c8b9d9e030450fcaaf9fbdf3231fb2a4d2a1b2ac43.png)

```bash
ts    搜索时间
uid    用户id
kw    用户搜索的关键字
rank    搜索结果排序
orders    点击次数
url    访问的路径
```

#### 创建

```sql
create database sogoudb;
show databases;
```

![](/media/2e/2ec2cf2054c56135597bb7af2c307b6ab4b1f83a4930bcb0ee0b2facb18153c8.png)

#### 使用数据库

```sql
use sogoudb;
```

#### 创建外部表

```sql
create external table if not exists sogou
(ts string,uid string,kw string,rank int,orders int,url string)
row format delimited fields terminated by '\t'
location '/s1/sogoudata';
```

![](/media/53/5334926282cd8632764c89178163e9b9a84524d0bbd330f5c464a01085e81767.png)

#### 查看表

```sql
show tables;
desc sogou;
```

![](/media/86/86710e3d7f333b44b724818d394ea904d22aa52bd1c7d9c43b0bd2307ae7f5f5.png)

#### 加载数据方式2

把这个500M文件(`sogou.500w.utf8`)塞到cc目录(`~/cc`)

![](/media/d4/d43fce1308d452b8360f7c5e874a9f97f26e82650048b3e85fb90e6059beaae9.png)

![](/media/1c/1ccfc07144f152f09c6f98f5e19a4d9f6d24efd759b71d07997a71f6a085278f.png)

#### 查看文件

```bash
cd ~/cc
ls -alh
```

![](/media/7b/7babcb50bfe310e729ec806830a3a29a110d91134c857f4ce7140b403933a805.png)

#### 上传文件

```bash
#上传文件（好慢）
hadoop fs -put sogou.500w.utf8 /s1/sogoudata
#查看文件
hadoop fs -ls /s1/sogoudata
```

![](/media/71/71ee3e4f970122fba0c79507d07d891fb0954c9fa43a7e9a14dc248a05793940.png)

<http://dayi-bigdata-master:50070/explorer.html#/s1/sogoudata>
![](/media/8e/8e74563b3cf2877d2010198a7892678e441af4e22af478473bbd1ce1f964da06.png)

#### 进入hive-导入

> 在hive里

```sql
use sogoudb;
```

#### 查询记录-前三条

```sql
select * from sogou limit 3;
```

![](/media/c3/c3f78b4056afcbe1da31d660f6f04af0e776b89cc14b8e991dbc37c09658eefe.png)

![](/media/57/5720566b3ce8d54209bb9408d80ffc8ffc1159a14644f9faf6ebe7772a4ae1d2.png)

#### 查询表里总共多少记录

```sql
select count(*) from sogou;
```

![](/media/b6/b6b6f833b386b4749f92cca78dd94494aa03e0797466e3f7db3a1570a249c106.png)

![](/media/b2/b27591903e82f32187939cc315092a7941313f7207d7220ad2e237814a6d9706.png)

超级慢

![](/media/fb/fb34dfde2c3d0120bf509cd2e765aa53751e5fc4425f2194e59021faf001a967.png)

![](/media/23/23e093c7a6135563603cc4cc849d5ec45e77d7d440a4ef7daa6725887444ee95.png)

![](/media/79/790b79e4e1bec9cfe2ae1face786bb8c0a497eb7c4b868890ba00291e6c05c9a.png)

193秒：
![](/media/f0/f050e676d419d3fc2f8a4cd13102bc31c2eab9e9c797d00d078457a82aeec0d2.png)

#### 查询UID总数

```sql
select count(distinct(uid)) from sogou;
```

还没执行完
![](/media/b7/b791c3758bf3f685322d5166b1ac95b24bfecb3e9c04fa9952080800f742bb3f.png)

177秒
![](/media/c9/c9206256715450f55de2570a915ebe67e9c27ca42f0d13ffdca0a0c4d4171879.png)

```sql
select count(uid) from sogou 
group by uid;
```

![](/media/b7/b774cb725f4a205eb62a071e09dc5d12aaa0d79371f6ecdcbab7c2918fd3c775.png)

出来了：
![](/media/6c/6c4a85cbb0ce4d2e18b033341e9669c30935af407d06a6270a036dd25e66c605.png)

#### 统计用户平均查询次数

$平均次数=总次数/用户个数$

建立临时表：

表`a`

| uid | cnt|
| -------- | --------  |
| a   | 10  |
| b   | 20  |

总次数 = sum(a.cnt)
用户个数 = count(a.uid)

临时表a

```sql
select uid,count(*) as cnt
from sogou
group by uid;
```

总表：

```sql
select sum(a.cnt)/count(a.uid) from
(select uid,count(*) as cnt
from sogou
group by uid)as a;
```

开炮：
![](/media/db/db84b676df3b7d4ee6f539609a5ca37b5f3416fba2a74112576dd1b28dd682a9.png)

![](/media/59/592df63a544280b0096850c512b6389a787c97526ac0109457531afecaf06292.png)

好慢：
![](/media/fc/fc30a7a580c9e34c9069352b7a0558c596ecc8e92a12932b6db8a6afa759791f.png)

出了
![](/media/22/220923d82683b59377050277d3830a35e68d890e07f44a552f3c179340ebd844.png)

3.6964094557111005

#### 查询频度最高的前10个关键字

```sql
SELECT kw, COUNT(kw) AS cnt
FROM sogou
GROUP BY kw
ORDER BY cnt DESC
LIMIT 10;
```

![](/media/4d/4d0e7b96bfec5b5cc1ed47e39f5755fa90e96d6c07eefa50abe4d0854a36d2df.png)

那个URL

![](/media/e6/e6c4b5d06f56e61e02d2ebb146fa6d60609ce12f13b4535f9d94dccef911865e.png)

![](/media/54/547f0e4a16c1c0886ffbcb16c78972769ec34adf57444265ef04997fa8d73fa9.png)

好慢hhhh:
![](/media/e5/e5d80ad38b337143d15317a60986dc7d640ab64376fd9ce1873e783d88439771.png)

出来了：
![](/media/db/db034197d9821d555d0a858d02d6b2a43ec04d47aac1ecbd401128259b4428b4.png)

## 18. zookeeper

### 1. 复制压缩包

```bash
cd resources
tar -zxvf zookeeper-3.4.6.tar.gz
mv zookeeper-3.4.6 zookeeper
```

![](/media/51/515dfd2f1f718a0bf8eb21cdeb63da6d65d8b68a1b4f57cd2eade9a84e87db2d.png)

recources/zookeeper/conf

文件`zoo.cfg`

```ini
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
```

`server.1` 服务器编号

`master` 服务器名

`2888` 服务端口

`3888` 选举端口

![](/media/0e/0e597ecc9b7c331eb47ce5ee859fbf5ef01e42deec88a2e682b4d146db46e74d.png)

### 3.创

数据节点表示文件myid

位置 dataDir=/tmp/zookeeper

```bash
#位置在内存盘里 重启会丢失
mkdir -pv /tmp/zookeeper
cd /tmp/zookeeper
vi myid
输入1
然后ESC + :wq

# 那样建议这样写

#master
mkdir -pv /tmp/zookeeper && cd /tmp/zookeeper && echo 1 > myid
cat /tmp/zookeeper/myid

#slave1
mkdir -pv /tmp/zookeeper && cd /tmp/zookeeper && echo 2 > myid
cat /tmp/zookeeper/myid
#slave2
mkdir -pv /tmp/zookeeper && cd /tmp/zookeeper && echo 3 >myid
cat /tmp/zookeeper/myid
```

![](/media/ea/eac5fc5d080e9b56b2b69c7345b72dd82ed268a63146cd29f97a66a2b796ea3a.png)

![](/media/fb/fb3d97ae92f7bdd569aabf83c42bdcc46a227d7d4dd1aca3925d837b3f04b55e.png)

~~服务器炸了虚拟机没了~~（![](/media/15/159175b0ec1a1adfc9d806d6b19db0fa2c2a57823eaaa92411886e56df4eb489.png)）

### 4. 发送zookeeper发送到从节点

```bash

cd ~/resources/
ls

scp -r zookeeper slave1:~/resources/
scp -r ~/resources/zookeeper slave2:~/resources/
```

### 5. 传送bash_profile

```bash
scp -r ~/.bash_profile slave1:~/
scp -r ~/.bash_profile slave2:~/


# 从节点分别执行：
source ~/.bash_profile
```

### 6. 启动zookeeper

```bash
cd ~/resources/zookeeper/bin/
ls 
./zkServer.sh start

#一键
~/resources/zookeeper/bin/./zkServer.sh start
```

![](/media/a7/a7118aaa37e3d673dd4ab285a1752eef652533fb3bd18a6d5ec5943fbc800bd3.png)

![](/media/7b/7be08c111d7e38c5899856fcef959571735eb47a5fce954e8339d3b06fd02215.png)

![](/media/95/95ab451471b0ce21683ac8323a13092e4d16bd6242f5701369a5c47560f00e08.png)

跟上了

![](/media/0f/0f69bd8fb8bd532631de21d28055a6e379a4b101db8b242fa5466e532f9886e2.png)

### 验证启动

```bash

./zkServer.sh status

```

这样是正常的

![](/media/2d/2dee3c3b2799fab123ab4fa2b4c0852f89c0784318d5b2c472805d3d9e1bcc14.png)

---

炸了

![](/media/02/0231c2adb11d5204a0e508db86fd07c983da1e8829b6b93b90f9dd9fc05ae685.png)

调试（`zkServer.sh print-cmd`)

```bash
"/home/dayi/resources/jdk/bin/java" -Dzookeeper.log.dir="." -Dzookeeper.root.logger="INFO,CONSOLE" -cp "/home/dayi/resources/zookeeper/bin/../build/classes:/home/dayi/resources/zookeeper/bin/../build/lib/*.jar:/home/dayi/resources/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/dayi/resources/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/home/dayi/resources/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/home/dayi/resources/zookeeper/bin/../lib/log4j-1.2.16.jar:/home/dayi/resources/zookeeper/bin/../lib/jline-0.9.94.jar:/home/dayi/resources/zookeeper/bin/../zookeeper-3.4.6.jar:/home/dayi/resources/zookeeper/bin/../src/java/lib/*.jar:/home/dayi/resources/zookeeper/bin/../conf:"  -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain "/home/dayi/resources/zookeeper/bin/../conf/zoo.cfg"

```

然后直接复制启动（你的跟我的不一样)

在这里看看报错是什么

![](/media/7c/7c002c7e07cd0d11083528d3150c859b68b3dfe7a03142f689871c15b486bb5f.png)

出现错误看看这个文件是否正常

```bash
/tmp/zookeeper/myid

```

## 19. 安装hbase

### 1. 复制文件解压缩

```bash
cd ~/resources/
tar -zxvf hbase-0.98.9-hadoop2-bin.tar.gz
mv hbase-0.98.9-hadoop2 hbase

```

![](/media/19/19c4a8a0ee88be2400a85756156733fecef5c8269cd59e280a0de9075fe9ec5d.png)

### 2. 修改配置文件

#### 文件1

`~/resources/hbase/conf/hbase-env.sh`

29 行：

![](/media/4b/4bdfe8ab4cd2559ce04e685fa027ef200949eade6108bf6f5b0cd700523752e7.png)

获得JAVA_HOME:

`echo $JAVA_HOME`

124行：

![](/media/a8/a81d1c90315cb1706c1e3be78860a79085530d7e22c1ede2f1733f68eb5f4418.png)

`true` : 使用habase自带的ZK（你启动失败了zookeeper）

`false` : 不使用hbase自带的ZK （上节课启动成功）

#### 文件2

`~/resources/hbase/conf/hbase-site.xml`

```xml
<configuration>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master:2181,slave1:2181,slave2:2181</value>
</property>
</configuration>
```

![](/media/4c/4c01198cb5e94abd377c06143ce4bc47edec343dbf56cad3c1430305b6444c37.png)

#### 文件3

`regionservers`

![](/media/a8/a8bf84fc942c3527f2d54414805050be278981dd87643117de1a18e161fb4e30.png)

### 3. 添加环境变量

```bash
export HBASE_HOME=~/resources/hbase
export PATH=$PATH:$HBASE_HOME/bin
export HADOOP_CLASSPATH=$HBASE_HOME/lib/*
```

![](/media/4a/4ad6ec8f9c5a9e9e92e654422f3f20782b8e129e137d08f61fa2f471a065b082.png)

保存后，sources一下

**两个从节点：**

```bash
source ~/.bash_profile
```

### 4. 传送文件

```bash
scp -r ~/resources/hbase slave1:~/resources/
scp -r ~/resources/hbase slave2:~/resources/
scp -r ~/.bash_profile slave1:~/
scp -r ~/.bash_profile slave2:~/
```

![](/media/3c/3c44047f824d0357f9b389a5008bc2b6ded8b1997495aee786166914fa408534.png)

![](/media/46/46af0fdd01a49ad63df9f01bd8381402d8a3d1ea9f905175acc5e4b0f925215e.png)

source一下

```bash
#slave1
source ~/.bash_profile
#slave2
source ~/.bash_profile
```

### 5. 原神启动

#### hadoop

```bash
~/resources/hadoop/sbin/./start-all.sh
```

验证：
<http://master:50070>

这种情况重新执行一下就行
![](/media/7b/7b4332f54a50ace625798558c7bb0b786fe85661fdc653abd0eec52b2a802cd1.png)

![](/media/48/480db5a2cc220c6c598997de780893f5e277bd6414416d47076413752f2cef49.png)

#### zookeeper

```bash
#master
mkdir -pv /tmp/zookeeper && cd /tmp/zookeeper && echo 1 > myid
~/resources/zookeeper/bin/./zkServer.sh start
#slave1
mkdir -pv /tmp/zookeeper && cd /tmp/zookeeper && echo 2 > myid
~/resources/zookeeper/bin/./zkServer.sh start
#slave2
mkdir -pv /tmp/zookeeper && cd /tmp/zookeeper && echo 3 >myid
~/resources/zookeeper/bin/./zkServer.sh start
```

验证:

```bash
~/resources/zookeeper/bin/./zkServer.sh status
```

![](/media/83/83ccddc36c16bc749d95026203ded66580d77b48567b07875727fe6babb46594.png)
lader or following

#### hbase

```bash
~/resources/hbase/bin/start-hbase.sh
```

验证：
<http://master:60010>

![](/media/b2/b20529fbfab6a0f5cd074c178953626a6814748911c8b37d9088154fdabce993.png)

![](/media/b0/b02e789ecb72227df6967f47a40f38d9140ccb954571ca5ef09b579802a50773.png)

## 20. hbase使用

### 1. shell

`hbase shell`

![](/media/3e/3e088fac1eb14e531fdbfdad843722dd788fe3d6fafa67280e8e2607acae8bab.png)

### 2. CRT

![](/media/b4/b4ebe84451175039c43ab6ddf602543930deabd16167151bbac13b384f86a684.png)

![](/media/39/398ceb4465daf5242daeb7f1e874e4b67befe2df840de541a3016a2614a93dbe.png)

![](/media/00/005e806e9b9cfe559ac1fa49ca0757216f42e15bb42db47d1389a657abf8a378.png)

### 3. 命令1 list

![](/media/53/5335d2a6f0f276e8b3c87e50b37c37df190c342effad2d239b1c380d697dbcb8.png)

### 4. 命令2 新建表

```bash
create 'stu','info','score'

        表名    列族1  列族2
```

![](/media/e4/e4e0dd1186882fccbc7cdeece76aeb76721290a91607fd4a30f6cffb2d0be1d3.png)

### 4. 命令3 写数据

尽量不要超过三列

```bash
put 'stu','1','info:name','John'
```

![](/media/db/db0ba923c05a0e4fd482f44d6033583ebcc92c23ba9a0c19e3194f9a0d7cb8ea.png)

```bash
put 'stu','2','info:age',20
put 'stu','3','info:name','Mary'
put 'stu','3','info:age',19


put 'stu','3','score:math',90
```

![](/media/c6/c67a7bf1ac1d1373bc53abe0eab52a79922187be90c0e82dd16dd06c33e685e9.png)

### 5. 读数据

两种方式

```bash
get 'stu','1'

get 'stu','3'
```

![](/media/51/5187edd29ff2df098ca8642c72a7c24488d336ffb9fe97ebfbf964da6b1f0120.png)

scan 读所有行

```bash
scan 'stu'
```

![](/media/a0/a00e66dec881d7b9707c1f389dae84e3f5d5c65e79758deb11c522f6351aaa0f.png)

查看表结构

```bash
describe 'stu'
```

![](/media/e7/e75e3c376e4cc3bc0968bd787385c5ddf1d5f41834e82acdefd6448901b45314.png)

```bash
Table stu is ENABLED                                                                                                                                                                                            
COLUMN FAMILIES DESCRIPTION                                                                                                                                                                                     
{NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}                                                                                                                                                 
{NAME => 'score', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}                                                                                                                                                
2 row(s) in 0.0680 seconds
```

### 6. 删除

#### 删单元格

`delete 'stu','3','score:Math'`

![](/media/d7/d7b4de9b79b38c7e467452891c34c8d7b68d29af27673a877470ed872f0d886d.png)

删除失败好像不会报错。

#### 删行

```sql
deleteall 'stu','2'
scan 'stu'
```

![](/media/c9/c9356dcca51fd42820bbddf355945a1b617e859fbe2f1b26277d31e4d646919a.png)

#### 删表

1. 先让表失效
2. 删除表

```sql
disable 'stu'
drop 'stu'
 list
 ```

![](/media/c7/c7b20648fc8ccd4103902ba5df93e27ff5b4b594d0a5aac7c9746e88c57623a2.png)

## 20 MapReduce

>肚子疼死了
>拉肚子拉的有点自闭，上课没跟上
>
### 1. 安装JDK

`jdk-8u102-windows-x64.exe`
安装目录不要带有中文，可以的话推荐选择
`C:\Program Files\Java`或者`D:\Program Files\Java`
有闸瓦环境了非1.8版本也建议再装一个。

![](/media/27/2722e8186c39df1b465e957fed641bd5cd4a06d634c6bda055da374754bac57c.png)

### 2. 解压缩hadoop

1. 想法子把这个文件`hadoop-2.6.4.tar.gz`解压了

![](/media/40/40ad6bf75ad1a295d9b23bb0a937b0ea6980842c38976bbfe60ce7c629743402.png)

2. 推荐放到目录：`D:\hadoop\hadoop-2.6.4`

![](/media/d0/d035df7f448407baef920611451fc23e8b2974df5460b49ea15619db5cdec142.png)

![](/media/3d/3df8b9650c6780c95aceb1524c78dd9a01decc7829a4a5bcfebbee6fc83c263f.png)

3. 然后修补下hadoop

需要替换文件直接替换即可。

`D:\hadoop\hadoop-2.6.4\bin`里面放下：

- `hadoop-common-2.2.0-bin-master.zip`里面的`bin`目录（替换文件）。
- `hadoop.dll` 放入 `D:\hadoop\hadoop-2.6.4\bin` （替换）
- `winutils.exe` 放入 `D:\hadoop\hadoop-2.6.4\bin` (替换)

![](/media/75/752b11f50c6786830a0eb1693de0769524891c0f2cf5935499f8d77d7669e203.png)

![](/media/38/381f3770218be0ed1f491ffabb0f210176ad3707ef7087f9784c20602bd15ffb.png)

### 3. 配置环境变量

> 这一步就是加4个（3个）环境变量，没那么复杂，但是会根据你电脑的实际情况有所不一样。所以篇幅比较大

> 如何修改环境变量？
> `此电脑/这台电脑/计算机（在文件管理器里，防止你桌面的是个快捷方式）`->`右键 属性`-> 找到`高级系统设置` —> `高级` -> `环境变量` ![](/media/11/1140e77f4401ad2dec155593e2646a023ff571f15386031cc6e0cbc316f2a6b1.png)
![最后的5](/media/ef/ef5896613abb3102290ac1041673cc3c05ccc54eb3547dec5e94416c4ec6aeb1.png)

> 按你实际修改目录，用户变量和系统变量都可以。

好像如果JAVA_HOME目录有空格的话会有些问题，

- `JAVA_HOME` : `C:\Program Files\Java\jdk1.8.0_102`
    ![](/media/27/271b218d9e062c703bc3077f0673281dbe710d57ac8eaafc07b1640a7102b7c9.png)

- `classpath` : `.;C:\Program Files\Java\jdk1.8.0_102\lib\dt.jar;C:\Program Files\Java\jdk1.8.0_102\lib\tools.jar;C:\Program Files\Java\jdk1.8.0_102\bin`
- `HADOOP_HOME`: `D:\hadoop\hadoop-2.6.4`
- `Path`(不要跟***天宇**一样直接删了再加，在原来的基础上修改，你的系统不应该没有这个环境变量参数(`Path`)，请先找找，这个环境变量是有顺序的，前面的会优先遍历) :

  - 这种的**添加**`;C:\Program Files\Java\jdk1.8.0_102\bin\;D:\hadoop\hadoop-2.6.4\bin`
    ![](/media/53/53233844d6f7fe75ac554e5e5f4a7518ac1489afcae64c5c432380ac09597849.png)
  - 这种的新建两个条目，移动到最上面去
    - `C:\Program Files\Java\jdk1.8.0_102\bin\`
    - `D:\hadoop\hadoop-2.6.4\bin`
    - ![](/media/04/04f0278f4138d94cefc7f2005168ac61a6ad4144f341e85b9d239fe4c04806f7.png)

![](/media/e6/e6edb0ecf59f341fcb65ed91ec1a3380a8e9e8894ada07c8d6ac7b2e121ec81b.png)

- 验证环境变量

```bash
java -version
echo %JAVA_HOME%
ECHO %HADOOP_HOME%
hadoop verison
```

D:\jdk\jdk1.8.0_102

![](/media/10/1004f444fe0d79e04b4d5362c2b5ca2e5a8a4e3d8dac73ceb9f9ad23a3914426.png)

遇到这个问题：
![](/media/b7/b7f2b5263c5d7ae78b5138394e0d91d87ea9e0120f634ccf91f97344485946b9.png)

我也不知道影不影响，原因是：`Program Files`中存在空格，然后他不能正常用，两个方法：

- **方法1**:把`JAVA_HOME`环境变量中的`Program Files`改为：`PROGRA~1`，比如这里我改为了：`C:\PROGRA~1\Java\jdk1.8.0_102`
![](/media/e9/e9efeafc2625ff6339559a9e184043379f3e4aafdada29f7e6a7f8ffc7954267.png)

- **方法2**(推荐）：把JDK复制一份到普通目录中（不带空格），然后修改`JAVA_HOME`到这里
    ![](/media/27/2767ab44d6dbcd0eab36f5942935093176efacdceebeb9f60da1e55ec158d938.png)
    ![](/media/69/69205a020c488530187d7af5c3a029514105b0d17be7ca1d6de92d60eeb86584.png)

- **方法三**：虽然我带着的空格，但是跑起来很正常，老师跟我说是这样（求告知）

>他这个JAVA_HOME的路径带空格好像会奇奇怪怪的
>![](/media/fb/fbb3ba04f636eb4ac00612be89990746dc4a3bca4379b7bb3d779f545c9da97f.png)
>改成JAVA_HOME = "C:/xxxxx/jre1.8_102"（带引号会这样）
>![](/media/ce/ceda26dd32cd50a9b6a8372c5a9b67a744b30be36ac1a9bb0aa157e860762317.png)
>复制了jdk到没空格的目录
>把hadoop的env
>![](/media/ff/ffcefaa3440b1c3f6d5a908391ddd457a77cfd5566fdae0867723fc3793c07f3.png)
>这样好像可以
>感觉不应该呀，带空格就识别不出来感觉不至于有这种低级问题bia
>![](/media/15/15446b92494a4f68f064f4b614f656293a28df19cc209bb2b184b976f53c3e16.png)

![](/media/64/648a6931808b55694c929bbb4765c97e72492255d22e83976942f030fc98e506.jpg)

- 也可以尝试修改这个文件的25行`D:\hadoop\hadoop-2.6.4\etc\hadoop\hadoop-env.cmd`
    ![](/media/23/238bc70ad506727c560d2a8bd4bd1337834fd793128c63fc356997ded8a2be8f.png)

总之：
执行：

```bash
java -version
echo %JAVA_HOME%
ECHO %HADOOP_HOME%
hadoop verison
```

第一行和最后一行的结果应该这样，但是你的如果有点多余的东西，应该也不是很影响？
![](/media/28/28182f76061d787915509df7d301f324ae53424e50b042dd01dbfe77d349bbf5.png)

### 4. 新建项目

#### 1. 配置Eclipse

> 感谢王策哥哥

`windows->preference->java->install JREs->add->standard VM`

![](/media/1c/1cd2714f8e339693df69f6f6697ec757050a1933762bfc8947b7f830d84fd9e8.png)

![](/media/b3/b3b73779ac0a410f7ec23acb11e8b398d8ac73a2ca6f5868b718743c1e95538b.png)

#### 2. 新建工程

> 感谢王策
`file->new project->java project->next->name->finish`

![](/media/ce/cee412f8792d1d491d29f91ff9a7025ad450754c63581e559b77a8e9aae5429b.png)

![](/media/e8/e8e970d63b9c4e6790888198026f9a8ef902867230b9380685f1ae842bfc5568.png)

#### 3. 导包

1. ALT+ENTER 打开项目设置
2. `JAVA BUILD PATH` -> 添加外部库
3. 导入hadoop环境jar包

```bash
D:\hadoop\hadoop-2.6.4\share\hadoop\common所有jar包
D:\hadoop\hadoop-2.6.4\share\hadoop\common\lib所有jar包

D:\hadoop\hadoop-2.6.4\share\hadoop\hdfs所有jar包
D:\hadoop\hadoop-2.6.4\share\hadoop\hdfs\lib所有jar包

D:\hadoop\hadoop-2.6.4\share\hadoop\mapreduce所有jar包
D:\hadoop\hadoop-2.6.4\share\hadoop\mapreduce\lib所有jar包

D:\hadoop\hadoop-2.6.4\share\hadoop\yarn所有jar包
D:\hadoop\hadoop-2.6.4\share\hadoop\yarn\lib所有jar包
```

![](/media/e4/e40e5f1accedf855f933e800a66a1c5e151607beaa8b2a85db3b9330ce4867ad.png)

![](/media/28/28b6d4456fd4bf292558c12434a59d80704b26fe6efbec800196399d56297dc6.png)

![](/media/4b/4bc017df36435d7b13133df971f208d511db32f8f1e2c7280ae43dea1a1891fd.png)

## 5. 新建包

### 新建`org.apache.hadoop.io.nativeio`

org.apache.hadoop.io.nativeio

![](/media/1a/1a46a67dcd5194ec0a720c3accc632cf0275a47fad187e4fd099de0bfceb9e05.png)

![](/media/8e/8ea53574f028212bf3968a10bc700b60fbb821253997af2ac47f290fcb01ec4b.png)

然后把`NativeIO.java`弄到包里

可以直接拖过来
![](/media/0e/0e6ac1fce8382391485fcd2ad170c54c08b7c49bfb21e153b7116a63d3bbe9ec.png)

![](/media/ef/ef865e1ad9a4d4e8e54223782d7274804b81ffa6cbde68cd2c4eace5d1da7848.png)

### 新建包 `my`

`my`

![](/media/22/22c0a7ed9e4d3abf423fa17b9e90b22ec7d87a93818ec51b4514dea139f0dae7.png)

同样的，把文件弄过去

如果需要修改的话这里：

![](/media/4d/4d0ba269567658c4bca59bd5a677841fb05452280f7db9212794e05694041d6f.png)

### 到master下修改下目录权限

master

```bash
#启动hadoop
~/resources/hadoop/sbin/./start-all.sh
```

看看网页：
<http://master:50070/dfshealth.html#tab-datanode>
![](/media/57/57c1f4cc6e574fd21d8b8a3c1fc700c25b7a8fed220f0875c34c1fcd5a054732.png)

正常的话，
设置777权限

```
hadoop fs -chmod 777 /
```

![](/media/93/934980ef0e9482a25c52afedd33d4d578902934dd3d612f9bb2510b0c01d329f.png)

## 6. 尝试新建目录

运行`Test.java`

![](/media/fc/fc4f9fa1c4de0f2ce9d1f2b0787145142d27d3544e3b70f035472c828bb4ef79.png)

这样就好啦

![](/media/c0/c06fbfed9a33dbd293a54c75d587c0855a05bbc130a51a25f03c0d24688beda4.png)

![](/media/69/69dc9345d44217337eb73548f35e879a30e187ffe25d1ef0707af2c7c1960143.png)

## 单词统计

![](/media/41/41e270ddf587a82a41213751cf1af2f3289647e4995aff6f45f048bab087c4f2.png)

- 21 行 可能需要重新配置

### 存放数据源

master

```bash
mkdir -pv ~/aa
cd ~/aa
vim ~/aa/words
cat ~/aa/words

[dayi@dayi-bigdata-master aa]$ cat ~/aa/words
ovo ovo
Hello words
Pig man
ZOOKEEPER
HADOOP
Hello words
ovo ovo
Zhangsan like pigman
Do u wanna build a snowman?
Come on let’s go and play!
I never see you anymore
come out the door
```

![](/media/00/00739e2481f1b6f069ab1ea829f49b9c3f4af2f629dbdb17eb394d00df36b3eb.png)

### 上传words

- 目录是否正常

```bash
hadoop fs -ls /

Found 6 items
drwxr-xr-x   - dayi          supergroup          0 2023-10-26 17:09 /hbase
drwxr-xr-x   - dayi          supergroup          0 2023-10-11 14:43 /ovo
drwxr-xr-x   - dayi          supergroup          0 2023-10-18 23:40 /s1
drwx-wx-wx   - dayi          supergroup          0 2023-10-18 23:57 /tmp
drwxr-xr-x   - dayi          supergroup          0 2023-10-18 23:01 /user
drwxr-xr-x   - Administrator supergroup          0 2023-11-01 23:37 /ww
[dayi@dayi-bigdata-master aa]$ 

要有这个/ww目录
没有的话

hadoop fs -mkdir /ww
hadoop fs -chmod 777 /ww
```

- 上传文件

```bash
hadoop fs -put ~/aa/words /ww
hadoop fs -cat /ww/words
```

![](/media/2b/2b1e992e44662b3b39abf50b95680256789b5055d50b9f7e695ef69dbee16a88.png)

### 运行程序

![](/media/08/08e39c513f90eac4974f358a584b371326502170bf391cc3ea8595c89fc3bf7c.png)

![](/media/4a/4ae64e8201400b56afef85a75a4cb83efaad8aa45d93d20a4c78ff25c2f1cbf9.png)

查看文件：

<http://master:50070/explorer.html#/ww/result>

![](/media/9a/9ac104a20cd715ec979e35d190ea32dfe6f852d60f7def1f0ad6dfd2ed84bb97.png)

![](/media/99/99ad6641afa034454fcf621849ac6fafeadc7c53f6cb6e8560a8966378ab093a.png)

## 21 mapperreduce 代码

### 读取文件

```java
package my;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class ReadData {
 public static void main(String[] args) throws Exception{
  String hdfs_addr = "hdfs://dayi-bigdata-master:9000";
  String uriSrting=hdfs_addr +"/ww/words";
  Configuration config = new Configuration();
  FSDataInputStream inputStream=null;
  try{
   FileSystem fileSystem = FileSystem.get(URI.create(hdfs_addr),config);
   inputStream = fileSystem.open(new Path(uriSrting));
   IOUtils.copyBytes(inputStream, System.out, 4096,false);
  }catch(Exception e){
   e.printStackTrace();
  }finally{
   IOUtils.closeStream(inputStream);
  }
 }
}

```

![](/media/4d/4da6f0129b9519e153b5d469ca70de0e262b3b7b72d86d38fe826da901f4e679.png)

### 新建目录

### 上传文件

```java
package my;

import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class PutFile {
  public static void main(String[] args) {
    String hdfs_addr = "hdfs://dayi-bigdata-master:9000";
    try{
      FileSystem fileSystem = FileSystem.get(URI.create(hdfs_addr),new Configuration());
      Path src = new Path("C:\\Windows\\System32\\drivers\\etc\\hosts");
      Path desc=new Path(hdfs_addr+"/ww/ff");
    fileSystem.copyFromLocalFile(src, desc);
    }catch (Exception e){
      e.printStackTrace();
    }
  }
}

```

### 下载文件

```bash
hadoop fs -chmod 777 /ww/ff
```

```java
package my;

import java.io.FileOutputStream;
import java.io.IOException;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class GetFile {
  public static void main(String[] args) throws IOException {
    String hdfs_addr = "hdfs://dayi-bigdata-master:9000";
    // FileSystem fileSystem = new FileSystem.get(URI.create(hdfs_addr),new Configuration());
    FileSystem fileSystem=FileSystem.get(URI.create(hdfs_addr), new Configuration());
    Path src = new Path(hdfs_addr+"/ww/ff");
  
    try{
      FileOutputStream outputStream=new FileOutputStream("D:\\c.txt");
      FSDataInputStream inputStream=fileSystem.open(src);
   IOUtils.copyBytes(inputStream, outputStream, 4096, false);
  } catch (Exception e) {
   // TODO: handle exception
   e.printStackTrace();
  }
    
  }
}

```

![](/media/cc/ccb52e348742248a0e14e282fb69954323b610491942b3f3aa4b95df6b319656.png)

### 列表

> 下发源文件

![](/media/fc/fc922b7a444a86c5b178c6f32df2d115e97e6df6bf56608c4a26ac74d99393b4.png)

### 删除文件

> 下发文件

![](/media/ea/ea80e1769caf2302567b0d5eccb339bec937e3af50405e235a9047747ecf055e.png)


### 结束

OVO