重庆网站商城,酒店如何做网络营销,公众号开发者权限,南通高端网站建设公司本文以远程模式安装Hive2.1.1将hive的元数据放置在MySQL数据库中。 1 安装mysql数据库 sudo apt-get install mysql-server11 重启mysql服务使得配置文件生效 sudo service mysql restart11 创建hive专用账户 CREATE USER hive% IDENTIFIED BY 123456;11 给hive账户授予所有权限… 本文以远程模式安装Hive2.1.1将hive的元数据放置在MySQL数据库中。 1 安装mysql数据库 sudo apt-get install mysql-server11 重启mysql服务使得配置文件生效 sudo service mysql restart11 创建hive专用账户 CREATE USER hive% IDENTIFIED BY 123456;11 给hive账户授予所有权限 grant all privileges on *.* to hive% identified by 123456 with grant option;11 刷新系统权限表使配置生效 flush privileges;11 2 解压安装hive cd /usr/local
sudo tar -xvzf apache-hive-2.1.1-bin.tar.gz
sudo mv apache-hive-2.1.1-bin/ hive-2.1.1123123 配置系统环境变量 sudo gedit .bashrc
export HIVE_HOME/usr/local/hive-2.1.1
exportPATH$HIVE_HOME/bin:$HIVE_HOME/lib:$PATH123123 使得环境变量配置生效 source .bashrc11 3 配置hive 3.1 修改conf/hive-env.sh文件 cd /usr/local/hive-2.1.1/conf/
sudo cp hive-env.sh.template hive-env.sh
sudo chown hadoop:hadoop hive-env.sh
sudo vi hive-env.sh
HADOOP_HOME/usr/local/hadoop-2.7.3
export HIVE_CONF_DIR/usr/local/hive-2.1.1/conf
export HIVE_AUX_JARS_PATH/usr/local/hive-2.1.1/lib12345671234567 3.2 修改日志属性文件配置日志存储目录 修改hive-log4j2.properties sudo cp hive-log4j2.properties.template hive-log4j2.properties
sudo chown hadoop:hadoop hive-log4j2.properties
sudo vi hive-log4j2.properties
property.hive.log.dir /usr/local/hive-2.1.1/logs12341234 修改llap-cli-log4j2.properties property.hive.log.dir /usr/local/hive-2.1.1/logs
property.hive.log.file llap-cli.log1212 3.3 修改hive-site.xml配置文件主要修改如下配置项目 propertynamehive.exec.local.scratchdir/namevalue/usr/local/hive-2.1.1/tmp/valuedescriptionLocal scratch space for Hive jobs/description/propertypropertynamehive.downloaded.resources.dir/namevalue/usr/local/hive-2.1.1/tmp/${hive.session.id}_resources/valuedescriptionTemporary local directory for added resources in the remote file system./description/propertypropertynamehive.querylog.location/namevalue/usr/local/hive-2.1.1/logs/valuedescriptionLocation of Hive run time structured log file/description/propertypropertynamehive.server2.logging.operation.log.location/namevalue/usr/local/hive-2.1.1/logs/valuedescriptionTop level directory where operation logs are stored if logging functionality is enabled/description/propertypropertynamehive.metastore.warehouse.dir/namevalue/usr/hive/warehouse/valuedescriptionlocation of default database for the warehouse/description/property
propertynamehive.metastore.uris/namevaluethrift://192.168.80.130:9083/valuedescriptionThrift URI for the remote metastore. Used by metastore client to connect to remote metastore./description/property
propertynamehive.exec.scratchdir/namevalue/tmp/hive/valuedescriptionHDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/lt;usernamegt; is created, with ${hive.scratch.dir.permission}./description/propertypropertynamehive.exec.local.scratchdir/namevalue/usr/local/hive-2.1.1/tmp/valuedescriptionLocal scratch space for Hive jobs/description/propertypropertynamehive.downloaded.resources.dir/namevalue/usr/local/hive-2.1.1/tmp/${hive.session.id}_resources/valuedescriptionTemporary local directory for added resources in the remote file system./description/propertypropertynamejavax.jdo.option.ConnectionURL/namevaluejdbc:mysql://192.168.80.130:3306/metastore?createDatabaseIfNotExisttrueamp;useSSLfalse/valuedescriptionJDBC connect string for a JDBC metastore.To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.For example, jdbc:postgresql://myhost/db?ssltrue for postgres database./description/propertypropertynamejavax.jdo.option.ConnectionDriverName/namevaluecom.mysql.jdbc.Driver/valuedescriptionDriver class name for a JDBC metastore/description/propertypropertynamejavax.jdo.option.ConnectionUserName/namevaluehive/valuedescriptionUsername to use against metastore database/description/propertypropertynamejavax.jdo.option.ConnectionPassword/namevalue123456/valuedescriptionpassword to use against metastore database/description/property
propertynamehive.hwi.listen.host/namevalue0.0.0.0/valuedescriptionThis is the host address the Hive Web Interface will listen on/description/propertypropertynamehive.hwi.listen.port/namevalue9999/valuedescriptionThis is the port the Hive Web Interface will listen on/description/propertypropertynamehive.server2.thrift.bind.host/namevalue0.0.0.0/valuedescriptionBind host on which to run the HiveServer2 Thrift service./description/property123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384 4 拷贝mysql连接包到hive主目录下的lib中 sudo mv ~/下载/mysql-connector-java-5.1.40-bin.jar /usr/local/hive-1.2.1/lib/11 5 配置Hive的hwi网页访问方式 下载hive-2.1.1源码包 wget http://www-us.apache.org/dist/hive/hive-2.1.1/ apache-hive-1.2.1-src.tar.gz
tar -zxvf apache-hive-2.1.1-src.tar.gz
cd apache-hive-2.1.1-src
cd hwi/web
zip hive-hwi-2.1.1.zip ./*
mv hive-hwi-2.1.1.zip hive-hwi-2.1.1.war
mv hive-hwi-2.1.1.war $HIVE_HOME/lib12345671234567 拷贝tools包 sudo cp /usr/lib/jdk1.8.0_121/lib/tools.jar /usr/local/hive-2.1.1/lib11 删除lib下的ant-1.6.5.jar否则浏览hwi网页时会显示错误信息需要刷新两次才能看到网页。 6 初始化hive schematool -dbType mysql -initSchema11 7 启动hive服务 启动metaStore服务 hive --service metastore 11 启动hive web界面 hive --service hwi 11 启动thrift2服务 hive --service hiveserver2 11 启动hive shell hive11 hwi访问网址 http://localhost:9999/hwi/11 --------------------------------------------------------------------------------------------------------------------------------------------- 如果发现不行可以按照一下在来一遍。 第一步 下载最新的hive直接去apache 里面找hive2.1.0下载就行。 第二步解压到服务器 tar zxvf apache-hive-2.0.0-bin.tar.gz mv apache-hive-2.0.0-bin /home/hivecd /home/hive 123 第三步修改conf。这里只关心hadoop和hive的配置其他JAVA HBASE的配置根据自己来 vi /etc/profile #for hadoop
export HADOOP_HOME/home/hadoop/hadoop-2.7.3
export HADOOP_COMMON_LIB_NATIVE_DIR$HADOOP_HOME/lib/native
export HADOOP_OPTS-Djava.library.path$HADOOP_HOME/lib
export PATH$PATH:/home/hadoop/hadoop-2.7.3/bin
export PATH$PATH:/home/hadoop/hadoop-2.7.3/sbin #for hive
export HIVE_HOME/home/hive export PATH$HADOOP_HOME/bin:$JAVA_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH1234567891011 第四步下载并设置好jdbc connector 我这里使用了最新的mysql-connector-java-5.1.40.tar.gz 记住将解压出来的jar放入hive 的lib中 cp mysql-connector-java-5.1.36-bin.jar $HIVE_HOME/lib/ 1 第五步配置hive-site.xml文件 文件来源于hive-default.xml.template 即 cp hive-default.xml.template hive-site.xml1 然后找到 namejavax.jdo.option.ConnectionURL/name1 修改其value valuejdbc:mysql://139.196.xxx.xxx:3306/hive?characterEncodingUTF8amp;useSSLfalseamp;createDatabaseIfNotExisttrue/value1 同时注意修改对应数据库的账号密码否则会在执行hive时出错 propertynamejavax.jdo.option.ConnectionUserName/namevalueroot/value/propertypropertynamejavax.jdo.option.ConnectionPassword/namevalue123456/value/property12345678 第六步 运行hive客户端 cd /home/hive/bin
hive12 第七步初始化DB schematool -initSchema -dbType mysql 第八步查看成功后的元数据 可以看到对应数据库hive中有了各种初始的表 第九步启动masternode节点 启动单机 hive 启动集群 hive -hiveconf hbase.zookeeper.quorumslave1,slave2,slave3 ———————————————————————————————————————————— 以下是可能出现的错误 一如果执行hive时候出现报出账号密码的错误 那么记得修改hive-site.xml中账号密码参考第五步中的内容。 二如果出现db没有初始化如 Exception in thread main java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, dont forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExisttrue for mysql))1 这种错误的时候请先执行初始化DB。 三如果提示 SSL相关的内容请在配置jdbc链接的时候设置ssl为false jdbc:mysql://139.196.xxx.xxx:3306/hive?useSSLfalsecreateDatabaseIfNotExisttrue Wed Nov 30 14:24:50 CST 2016 WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to false. You need either to explicitly disable SSL by setting useSSLfalse, or set useSSLtrue and provide truststore for server certificate verification.
Wed Nov 30 14:24:55 CST 2016 WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to false. You need either to explicitly disable SSL by setting useSSLfalse, or set useSSLtrue and provide truststore for server certificate verification.12 四遇到hive出错的时候 [Fatal Error] hive-site.xml:26:5: The element type value must be terminated by the matching end-tag /value.
Exception in thread main java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/home/hive/conf/hive-site.xml; lineNumber: 26; columnNumber: 5; The element type value must be terminated by the matching end-tag /value.12 Logging initialized using configuration in jar:file:/home/hive/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: true
Exception in thread main java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D12 这里就是配置文件Hive-site.xml中修改system:java.io.tmpdir指定一个系统存在的目录即可。 这里追加两种启动方式方便各位用来进行hive测试。 hive提供了四种运行hive的方式,分别是: **Hive CLI HiveServer2 和 Beeline** HCatalog WebHCat (Templeton) 这里主要介绍前两种 第一种hive CLI 因为hive的bin目录已经添加了path变量, 因此, 可以直接使用hive命令启动: hive 输入完命令后可以直接可以进行hive操作。 第二种HiveServer2 和 Beeline beeline提供多用户, 更加安全的服务, 因此beeline用得比较多. hiveserver2启动时默认的地址是”localhost:10000”, 因此, 在使用beeline连接的时候, 需要使用” jdbc:hive2://localhost:10000”作为参数. 相关的命令如下: hiveserver2
beeline -u jdbc:hive2://localhost:1000012 同时也可以将 Beeline和HiveServer2在同一个进程里启动, 用于测试: beeline -u jdbc:hive2:// 但是这里如果用到自定义账号密码必须在配置文件hive-site.xml中进行相关配置。 上文第五步已经进行了相关介绍可以参照。 ---------------------------------------------------------------------------------------------------------------------------------------------------------- hive启动时报Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D解决办法 hive启动时遇到以下错误 Exception in thread mainjava.lang.RuntimeException: java.lang.IllegalArgumentException:java.net.URISyntaxException: Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D 解决办法 在hive下创建临时IO的tmp文件夹。然后将路径配置到hive-site.xml的下列参数中 property namehive.querylog.location/name value/usr/local/hive/iotmp/value descriptionLocation of Hive run time structured log file/description /property property namehive.exec.local.scratchdir/name value/usr/local/hive/iotmp/value descriptionLocal scratch space for Hive jobs/description /property property namehive.downloaded.resources.dir/name value/usr/local/hive/iotmp/value descriptionTemporary local directory for added resources in the remote file system./description /property 保存重启hive即可。 [rootmaster ~]# hive Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-1.2.0.jar!/hive-log4j.properties hive show databases; OK default Time taken: 3.684 seconds, Fetched: 1 row(s) hive hiveserver启动方式 1 hive 命令行模式直接输入/hive/bin/hive的执行程序或者输入 hive –service cli 用于linux平台命令行查询查询语句基本跟mysql查询语句类似 2 hive web界面的启动方式hive –service hwi 用于通过浏览器来访问hive感觉没多大用途 3 hive 远程服务 (端口号10000) 启动方式 hive –service hiveserver 用java等程序实现通过jdbc等驱动的访问hive就用这种起动方式了这个是程序员最需要的方式了 也可以自己指定端口 hive -service hiveserver -p 50000 (表示后台运行) 输入完这些指令后终端就在运行hiveserver了会卡住不动。其实已经在运行了不用担心。
