Hive on Spark 注意事项

1、hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl" target="_blank" rel="external nofollow" ?>
 <configuration>
	<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
	<description></description>
	</property>

	<property>
	    <name>javax.jdo.option.ConnectionDriverName</name>
	    <value>com.mysql.jdbc.Driver</value>
	    <description></description>
	</property>

	<property>
	    <name>javax.jdo.option.ConnectionUserName</name>
	    <value>root</value>
	    <description></description>
	</property>

	<property>
	    <name>javax.jdo.option.ConnectionPassword</name>
	    <value>root</value>
	    <description></description>
	</property>

	<property>  
                   <name>spark.master</name>  
                   <value>spark://pc2:7077</value>  
	</property> 
</configuration>
           

2、将此配置文件分发到集群中【每一个结点】的spark的conf目录下

3、将【每一个】spark结点下的jars目录中的hive开头的jar包删除

rm hive*

4、将hive的lib目录下的hive开头的jar包复制到【每一个】spark结点的

scp hive* [email protected]:/home/chenjie/spark-2.0.2-hadoop2.7

5、启动hive

6、hive命令 set hive.execution.engine=spark;

7、hive命令 set hive.execution.engine=spark;

如果要切换回mapreduce方式

set hive.execution.engine=mr;