需要在信创平台(鲲鹏ARM CPU 和 银河麒麟V10操作系统)部署Apache Doris 存储日志数据,使用社区编译好的 ARM 安装包,同时配置好ARM 版本的JDK即可。Doris社区本身文档很齐全,整个部署过程就是在官方文档的指引下进行的,这里只是对过程的记录。
名称 | 版本 | 备注 |
---|---|---|
操作系统 | 银河麒麟 V10 SP3 | |
CPU型号 | 鲲鹏 920 | |
JDK | bisheng-jdk-8u382-linux-aarch64 | 毕昇JDK 8 |
Apache Doris | 2.0.2 | 这里是测试环境所以部署1个FE 和 3个BE |
所有节点都需要,默认在root用户下执行
yum -y remove java-1.8.0*
yum -y remove java-11*
sudo mkdir -p /opt/soft/java
cd /opt/soft/java/
wget https://mirrors.huaweicloud.com/kunpeng/archive/compiler/bisheng_jdk/bisheng-jdk-8u382-linux-aarch64.tar.gz
tar -zxf bisheng-jdk-8u382-linux-aarch64.tar.gz
rm bisheng-jdk-8u382-linux-aarch64.tar.gz
ln -snf bisheng-jdk1.8.0_382 java
cat << EOF | sudo tee /etc/profile.d/java.sh
export JAVA_HOME=/opt/soft/java/java
export PATH=\$PATH:\$JAVA_HOME/bin
EOF
source /etc/profile
所有节点都需要,默认在root用户下执行
gcc -v
vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
sed -i '/swap/s/^/#/' /etc/fstab
swapoff -a
Doris 的元数据要求时间精度要小于5000ms,所以所有集群所有机器要进行时钟同步,避免因为时钟问题引发的元数据不一致导致服务出现异常。
配置略
因为这里是 测试环境,所以直接选择了关闭防火墙
systemctl stop firewalld
所有节点都需要,默认在root用户下执行
adduser doris
# 安装目录,所有节点
mkdir -p /opt/soft/doris
chown -R doris:doris /opt/soft/doris
# FE 元数据目录 FE节点
mkdir -p /data/doris-meta
chown -R doris:doris /data/doris-meta
# BE 数据目录 BE 节点
mkdir -p /data/doris
chown -R doris:doris /data/doris
# 日志数据 所有节点
mkdir -p /var/log/doris/sys
mkdir -p /var/log/doris/audit
chown -R doris:doris /var/log/doris
sudo su - doris
wget https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.0.2-bin-arm64.tar.gz
tar -zxf apache-doris-2.0.2-bin-arm64.tar.gz -C /opt/soft/doris/
只需要在FE节点操作即可,在doris用户下执行
cd /opt/soft/doris
mv apache-doris-2.0.2-bin-arm64/fe /opt/soft/doris/fe-2.0.2
mv apache-doris-2.0.2-bin-arm64/extensions /opt/soft/doris/extensions-2.0.2
ln -snf fe-2.0.2 doris-fe
cat << EOF | tee /opt/soft/doris/doris-fe/conf/fe.conf
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#####################################################################
## The uppercase properties are read and exported by bin/start_fe.sh.
## To see all Frontend configurations,
## see fe/src/org/apache/doris/common/Config.java
#####################################################################
CUR_DATE=`date +%Y%m%d-%H%M%S`
# the output dir of stderr and stdout
LOG_DIR = /var/log/doris
JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:/var/log/doris/fe.gc.log.\$CUR_DATE"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:/var/log/doris/fe.gc.log.\$CUR_DATE:time"
##
## the lowercase properties are read by main program.
##
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
sys_log_mode = NORMAL
# store metadata, must be created before start FE.
# Default value is \${DORIS_HOME}/doris-meta
meta_dir = /data/doris-meta
# Default dirs to put jdbc drivers,default value is \${DORIS_HOME}/jdbc_drivers
# jdbc_drivers_dir = \${DORIS_HOME}/jdbc_drivers
http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24 or IP format, e.g. 10.10.10.1
# Default value is empty.
priority_networks = 192.168.91.0/24
# Advanced configurations
# log_roll_size_mb = 1024
sys_log_dir = /var/log/doris/sys
# sys_log_roll_num = 10
# sys_log_verbose_modules = org.apache.doris
audit_log_dir = /var/log/doris/audit
# audit_log_modules = slow_query, query
# audit_log_roll_num = 10
# meta_delay_toleration_second = 10
# qe_max_connection = 1024
# qe_query_timeout_second = 300
# qe_slow_log_ms = 5000
EOF
只需要在FE节点操作即可,在doris用户下执行
cd /opt/soft/doris/
mv apache-doris-2.0.2-bin-arm64/be /opt/soft/doris/be-2.0.2
mv apache-doris-2.0.2-bin-arm64/extensions /opt/soft/doris/extensions-2.0.2
ln -snf be-2.0.2 doris-be
cat << EOF | tee /opt/soft/doris/doris-be/conf/be.conf
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
CUR_DATE=`date +%Y%m%d-%H%M%S`
PPROF_TMPDIR="/var/log/doris/"
JAVA_OPTS="-Xmx1024m -DlogPath=/var/log/doris/jni.log -Xloggc:/var/log/doris/be.gc.log.\$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDLE_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Xmx1024m -DlogPath=/var/log/doris//jni.log -Xlog:gc:/var/log/doris//be.gc.log.\$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDLE_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"
# since 1.2, the JAVA_HOME need to be set to run BE process.
# JAVA_HOME=/path/to/jdk/
JAVA_HOME=/opt/soft/java/java
# https://github.com/apache/doris/blob/master/docs/zh-CN/community/developer-guide/debug-tool.md#jemalloc-heap-profile
# https://jemalloc.net/jemalloc.3.html
JEMALLOC_CONF="percpu_arena:percpu,background_thread:true,metadata_thp:auto,muzzy_decay_ms:15000,dirty_decay_ms:15000,oversize_threshold:0,lg_tcache_max:20,prof:false,lg_prof_interval:32,lg_prof_sample:19,prof_gdump:false,prof_accum:false,prof_leak:false,prof_final:false"
JEMALLOC_PROF_PRFIX=""
# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO
# ports for admin, web, heartbeat service
be_port = 9060
webserver_port = 8040
heartbeat_service_port = 9050
brpc_port = 8060
# HTTPS configures
enable_https = false
# path of certificate in PEM format.
ssl_certificate_path = "\$DORIS_HOME/conf/cert.pem"
# path of private key in PEM format.
ssl_private_key_path = "\$DORIS_HOME/conf/key.pem"
# enable auth check
enable_auth = false
# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24 or IP format, e.g. 10.10.10.1
# Default value is empty.
# priority_networks = 10.10.10.0/24;192.168.0.0/16
priority_networks = 192.168.91.0/24
# data root path, separate by ';'
# you can specify the storage medium of each root path, HDD or SSD
# you can add capacity limit at the end of each root path, separate by ','
# eg:
# storage_root_path = /home/disk1/doris.HDD,50;/home/disk2/doris.SSD,1;/home/disk2/doris
# /home/disk1/doris.HDD, capacity limit is 50GB, HDD;
# /home/disk2/doris.SSD, capacity limit is 1GB, SSD;
# /home/disk2/doris, capacity limit is disk capacity, HDD(default)
#
# you also can specify the properties by setting '<property>:<value>', separate by ','
# property 'medium' has a higher priority than the extension of path
#
# Default value is \${DORIS_HOME}/storage, you should create it by hand.
# storage_root_path = \${DORIS_HOME}/storage
storage_root_path = /data/doris,medium:hdd
# Default dirs to put jdbc drivers,default value is \${DORIS_HOME}/jdbc_drivers
# jdbc_drivers_dir = \${DORIS_HOME}/jdbc_drivers
# Advanced configurations
# sys_log_dir = \${DORIS_HOME}/log
sys_log_dir = /var/log/doris/sys
# sys_log_roll_mode = SIZE-MB-1024
# sys_log_roll_num = 10
# sys_log_verbose_modules = *
# log_buffer_level = -1
# palo_cgroups
EOF
只需要在FE节点操作即可,在doris用户下执行 。这里只有一个FE
/opt/soft/doris/doris-fe/bin/start_fe.sh --daemon
mysql -uroot -P9030 -h192.168.91.161
SHOW PROC '/frontends';
只需要在BE节点操作即可,在doris用户下执行 。
echo "vm.max_map_count=2000000" >> /etc/sysctl.conf
sudo sysctl -p
ALTER SYSTEM ADD BACKEND "192.168.91.162:9050";
ALTER SYSTEM ADD BACKEND "192.168.91.163:9050";
ALTER SYSTEM ADD BACKEND "192.168.91.164:9050";
/opt/soft/doris/doris-be/bin/start_be.sh --daemon
SHOW PROC '/backends';
http://192.168.91.161:8030
网站声明:如果转载,请联系本站管理员。否则一切后果自行承担。
添加我为好友,拉您入交流群!
请使用微信扫一扫!