# spark-lab

**Repository Path**: zacker330/spark-lab

## Basic Information

- **Project Name**: spark-lab
- **Description**: a vagrant environment for Spark
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 0
- **Created**: 2016-01-10
- **Last Updated**: 2021-06-20

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README


## 注意
在虚拟机中，请都使用 `spark`用户，密码也是`spark`

包含3台机器：

192.168.2.100 spark-master

192.168.2.101 spark-slave1

192.168.2.102 spark-slave2

tips: 

使用命令登录到虚拟机：vagrant ssh <spark-master | spark-slave1 | spark-slave2>


## 软件准备，下载并放到相应的位置

provision/files/spark http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz

provision/files/zookeeper http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/

provision/files/scala-2.11.7.tgz http://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.tgz

provision/files/jdk-8u66-linux-x64.tar.gz http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

## 启动虚拟机

> vagrant up


## 启动ZooKeeper

1. 分别ssh上这些虚拟机，执行 > /usr/local/zookeeper/bin/zkServer.sh start

2. 在每台机器上检查是否成功

	1. 使用jps命令查看：

		```
		spark@spark-master:/usr/local/zookeeper$ jps
		4996 QuorumPeerMain
		```
	2. > /usr/local/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181

## 启动 Spark集群

1. 登录 spark-master: /usr/local/spark/sbin/start-all.sh

2. 分别到每台机器检查spark有没启动成功:

		spark@spark-master:/usr/local$ jps
		6436 Jps
		6196 QuorumPeerMain
		6374 Worker
		6235 Master

3. 启动第二台master，防单点
	
	1. 登录spark-slave1 > /usr/local/spark/sbin/start-master.sh
	2. 检查： 

			spark@spark-slave1:/usr/local/spark$ jps
			7456 Worker
			7541 Master
			6617 QuorumPeerMain
			7595 Jps


## 提交一个simpleApp试验是否OK

> /usr/local/spark/bin/spark-submit  /home/spark/simple-project_2.10-1.0.jar /usr/local/spark/README.md  --Master spark://spark-master:7077