大数据平台|minio对象存储搭建大数据平台|大数据

一、ntp 服务器 1 安装ntp yum install ntp
1.1 服务端配置
vi /etc/ntp.conf

restrict 192.168.16.0 mask 255.255.255.0 nomodify notrapserver ntp1.aliyun.com server time1.aliyun.comrestrict ntp1.aliyun.com nomodify notrap noquery restrict time1.aliyun.com nomodify notrap noqueryserver127.127.1.0# local clock fudge127.127.1.0 stratum 10

systemctl start ntpd
ntpq -p
systemctl disable chronyd
systemctl enable ntpd
1.2 客户端配置
vi /etc/ntp.conf

server 192.168.16.41restrict 192.168.16.41 nomodify notrap noquery

ntpdate -u 192.168.16.41
systemctl restart ntpd
ntpq -p
systemctl disable chronyd
systemctl enable ntpd
二、Minio安装下载：https://dl.min.io/server/minio/release/linux-amd64/minio
1 创建数据目录 mkdir -p /data/miniodata
2 启动 chmod u+x minio
四节点，每个节点一个数据目录

export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=admin123 nohup ./minio server http://192.168.16.41/data/miniodata http://192.168.16.42/data/miniodata http://192.168.16.44/data/miniodata http://192.168.16.50/data/miniodata > minio.log 2>&1 & 其他节点相同

两节点，每个节点两个数据目录

export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=admin123 nohup ./minio server http://192.168.16.37/data/minio1 http://192.168.16.37/data/minio2 http://192.168.16.38/data/minio1 http://192.168.16.38/data/minio2 > minio.log 2>&1 & 其他节点相同

3 客户端下载：https://dl.min.io/client/mc/release/linux-amd64/mc
chmod u+x mc
添加连接
./mc config host add minio38 http://192.168.16.38:9000 admin admin123 --api s3v4
创建桶
./mc mb minio38/milvusdata
4. 开机启动 4.1 创建启动脚本
start_minio.sh

#!/bin/bash export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=admin123 /home/minio server --address=192.168.16.44:9002 http://192.168.16.41:9002/data/miniodata http://192.168.16.42:9002/data/miniodata http://192.168.16.44:9002/data/miniodata http://192.168.16.50:9002/data/miniodata

4.2 创建启动服务service
在/usr/lib/systemd/system下面新建minio.service文件，然后写入

[Unit] Description=Minio service Documentation=https://docs.minio.io/[Service] WorkingDirectory=/home/ ExecStart=/home/start_minio.shRestart=on-failure RestartSec=5[Install] WantedBy=multi-user.target

4.3 授权执行权限
chmod +x /usr/lib/systemd/system/minio.service
4.4 启动
systemctl daemon-reload
systemctl enable minio
systemctl start minio
三、挂载为本地目录 1. s3fs安装在milvus所在机器上
yum install -y epel-release
yum install -y s3fs-fuse
2. 创建密码文件 echo “admin:admin123” >> /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs
3. 启动 s3fs -o passwd_file=/etc/passwd-s3fs -o use_path_request_style -o url=http://192.168.16.37:9000 -o allow_other -o cipher_suites=AESGCM
-o kernel_cache
-o max_background=1000
-o max_stat_cache_size=100000
-o multipart_size=128
-o parallel_count=30
-o multireq_max=30
-o dbglevel=warn -o bucket=milvusdata /data/milvus
4. 创建开机自动挂载 【大数据平台|minio对象存储搭建】vim /etc/fstab

s3fs#milvusdata /data/milvus fuse _netdev,allow_other,use_path_request_style,nonempty,url=http://192.168.16.41:9000,max_background=1000,max_stat_cache_size=100000,multipart_size=128,parallel_count=30,multireq_max=30,dbglevel=warn 0 0