org.apache.spark

Class SparkEnv

Object

org.apache.spark.SparkEnv

All Implemented Interfaces:

Logging

public class SparkEnv
extends Object
implements Logging

:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, Akka actor system, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a global variable, so all the threads can access the same SparkEnv. It can be accessed by SparkEnv.get (e.g. after creating a SparkContext).

NOTE: This is not intended for external use. This is exposed for Shark and may be made private in a future release.

Constructor Summary

Constructors

Constructor and Description

SparkEnv(String executorId, akka.actor.ActorSystem actorSystem, Serializer serializer, Serializer closureSerializer, CacheManager cacheManager, MapOutputTracker mapOutputTracker, org.apache.spark.shuffle.ShuffleManager shuffleManager, BroadcastManager broadcastManager, org.apache.spark.network.BlockTransferService blockTransferService, BlockManager blockManager, SecurityManager securityManager, HttpFileServer httpFileServer, String sparkFilesDir, MetricsSystem metricsSystem, org.apache.spark.shuffle.ShuffleMemoryManager shuffleMemoryManager, SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
akka.actor.ActorSystem	actorSystem()
BlockManager	blockManager()
org.apache.spark.network.BlockTransferService	blockTransferService()
BroadcastManager	broadcastManager()
CacheManager	cacheManager()
Serializer	closureSerializer()
SparkConf	conf()
static SparkEnv	createDriverEnv(SparkConf conf, boolean isLocal, LiveListenerBus listenerBus) Create a SparkEnv for the driver.
static SparkEnv	createExecutorEnv(SparkConf conf, String executorId, String hostname, int port, int numCores, boolean isLocal, akka.actor.ActorSystem actorSystem) Create a SparkEnv for an executor.
java.net.Socket	createPythonWorker(String pythonExec, scala.collection.immutable.Map<String,String> envVars)
void	destroyPythonWorker(String pythonExec, scala.collection.immutable.Map<String,String> envVars, java.net.Socket worker)
static String	driverActorSystemName()
static scala.collection.immutable.Map<String,scala.collection.Seq<scala.Tuple2<String,String>>>	environmentDetails(SparkConf conf, String schedulingMode, scala.collection.Seq<String> addedJars, scala.collection.Seq<String> addedFiles) Return a map representation of jvm information, Spark properties, system properties, and class paths.
static String	executorActorSystemName()
String	executorId()
static SparkEnv	get() Returns the SparkEnv.
static SparkEnv	getThreadLocal() Returns the ThreadLocal SparkEnv.
java.util.concurrent.ConcurrentMap<String,Object>	hadoopJobMetadata()
HttpFileServer	httpFileServer()
boolean	isStopped()
MapOutputTracker	mapOutputTracker()
MetricsSystem	metricsSystem()
void	releasePythonWorker(String pythonExec, scala.collection.immutable.Map<String,String> envVars, java.net.Socket worker)
SecurityManager	securityManager()
Serializer	serializer()
static void	set(SparkEnv e)
org.apache.spark.shuffle.ShuffleManager	shuffleManager()
org.apache.spark.shuffle.ShuffleMemoryManager	shuffleMemoryManager()
String	sparkFilesDir()
void	stop()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

SparkEnv

public SparkEnv(String executorId,
        akka.actor.ActorSystem actorSystem,
        Serializer serializer,
        Serializer closureSerializer,
        CacheManager cacheManager,
        MapOutputTracker mapOutputTracker,
        org.apache.spark.shuffle.ShuffleManager shuffleManager,
        BroadcastManager broadcastManager,
        org.apache.spark.network.BlockTransferService blockTransferService,
        BlockManager blockManager,
        SecurityManager securityManager,
        HttpFileServer httpFileServer,
        String sparkFilesDir,
        MetricsSystem metricsSystem,
        org.apache.spark.shuffle.ShuffleMemoryManager shuffleMemoryManager,
        SparkConf conf)

Method Detail

driverActorSystemName

public static String driverActorSystemName()

executorActorSystemName

public static String executorActorSystemName()

set

public static void set(SparkEnv e)

get

public static SparkEnv get()

Returns the SparkEnv.

getThreadLocal

public static SparkEnv getThreadLocal()

Returns the ThreadLocal SparkEnv.

createDriverEnv

public static SparkEnv createDriverEnv(SparkConf conf,
                       boolean isLocal,
                       LiveListenerBus listenerBus)

Create a SparkEnv for the driver.

createExecutorEnv

public static SparkEnv createExecutorEnv(SparkConf conf,
                         String executorId,
                         String hostname,
                         int port,
                         int numCores,
                         boolean isLocal,
                         akka.actor.ActorSystem actorSystem)

Create a SparkEnv for an executor. In coarse-grained mode, the executor provides an actor system that is already instantiated.

environmentDetails

public static scala.collection.immutable.Map<String,scala.collection.Seq<scala.Tuple2<String,String>>> environmentDetails(SparkConf conf,
                                                                                                          String schedulingMode,
                                                                                                          scala.collection.Seq<String> addedJars,
                                                                                                          scala.collection.Seq<String> addedFiles)

Return a map representation of jvm information, Spark properties, system properties, and class paths. Map keys define the category, and map values represent the corresponding attributes as a sequence of KV pairs. This is used mainly for SparkListenerEnvironmentUpdate.

executorId

public String executorId()

actorSystem

public akka.actor.ActorSystem actorSystem()

serializer

public Serializer serializer()

closureSerializer

public Serializer closureSerializer()

cacheManager

public CacheManager cacheManager()

mapOutputTracker

public MapOutputTracker mapOutputTracker()

shuffleManager

public org.apache.spark.shuffle.ShuffleManager shuffleManager()

broadcastManager

public BroadcastManager broadcastManager()

blockTransferService

public org.apache.spark.network.BlockTransferService blockTransferService()

blockManager

public BlockManager blockManager()

securityManager

public SecurityManager securityManager()

httpFileServer

public HttpFileServer httpFileServer()

sparkFilesDir

public String sparkFilesDir()

metricsSystem

public MetricsSystem metricsSystem()

shuffleMemoryManager

public org.apache.spark.shuffle.ShuffleMemoryManager shuffleMemoryManager()

conf

public SparkConf conf()

isStopped

public boolean isStopped()

hadoopJobMetadata

public java.util.concurrent.ConcurrentMap<String,Object> hadoopJobMetadata()

stop

public void stop()

createPythonWorker

public java.net.Socket createPythonWorker(String pythonExec,
                                 scala.collection.immutable.Map<String,String> envVars)

destroyPythonWorker

public void destroyPythonWorker(String pythonExec,
                       scala.collection.immutable.Map<String,String> envVars,
                       java.net.Socket worker)

releasePythonWorker

public void releasePythonWorker(String pythonExec,
                       scala.collection.immutable.Map<String,String> envVars,
                       java.net.Socket worker)