org.apache.spark.sql

Interface SQLConf

All Known Implementing Classes:

HiveContext, LocalHiveContext, SQLContext, TestHive, TestHiveContext, TestSQLContext

public interface SQLConf

A trait that enables the setting and getting of mutable config parameters/hints.

In the presence of a SQLContext, these can be set and queried by passing SET commands into Spark SQL's query functions (i.e. sql()). Otherwise, users of this trait can modify the hints by programmatically calling the setters and getters of this trait.

SQLConf is thread-safe (internally synchronized, so safe to be used in multiple threads).

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	SQLConf.Deprecated$

Method Summary

Methods

Modifier and Type	Method and Description
int	autoBroadcastJoinThreshold() Upper bound on the sizes (in bytes) of the tables qualified for the auto conversion to a broadcast value during the physical executions of join operations.
void	clear()
boolean	codegenEnabled() When set to true, Spark SQL will use the Scala compiler at runtime to generate custom bytecode that evaluates expressions found in queries.
int	columnBatchSize() The number of rows that will be
String	columnNameOfCorruptRecord()
long	defaultSizeInBytes() The default size in bytes to assign to a logical operator's estimation statistics.
String	dialect() The SQL dialect that is used when parsing queries.
boolean	externalSortEnabled() When true the planner will use the external sort, which may spill to disk.
scala.collection.immutable.Map<String,String>	getAllConfs() Return all the configuration properties that have been set (i.e.
String	getConf(String key) Return the value of Spark SQL configuration property for the given key.
String	getConf(String key, String defaultValue) Return the value of Spark SQL configuration property for the given key.
boolean	inMemoryPartitionPruning() When set to true, partition pruning for in-memory columnar tables is enabled.
boolean	isParquetBinaryAsString() When set to true, we always treat byte arrays in Parquet files as strings.
int	numShufflePartitions() Number of partitions to use for shuffle operators.
String	parquetCompressionCodec() The compression codec for writing to a Parquetfile
boolean	parquetFilterPushDown() When true predicates will be passed to the parquet record reader when possible.
void	setConf(java.util.Properties props) Set Spark SQL configuration properties.
void	setConf(String key, String value) Set the given Spark SQL configuration property.
java.util.Map<String,String>	settings() Only low degree of contention is expected for conf, thus NOT using ConcurrentHashMap.
boolean	useCompression() When true tables cached using the in-memory columnar caching will be compressed.

Method Detail

settings

java.util.Map<String,String> settings()

Only low degree of contention is expected for conf, thus NOT using ConcurrentHashMap.

dialect

String dialect()

The SQL dialect that is used when parsing queries. This defaults to 'sql' which uses a simple SQL parser provided by Spark SQL. This is currently the only option for users of SQLContext.

When using a HiveContext, this value defaults to 'hiveql', which uses the Hive 0.12.0 HiveQL parser. Users can change this to 'sql' if they want to run queries that aren't supported by HiveQL (e.g., SELECT 1).

Note that the choice of dialect does not affect things like what tables are available or how query execution is performed.

useCompression

boolean useCompression()

When true tables cached using the in-memory columnar caching will be compressed.

parquetCompressionCodec

String parquetCompressionCodec()

The compression codec for writing to a Parquetfile

columnBatchSize

int columnBatchSize()

The number of rows that will be

numShufflePartitions

int numShufflePartitions()

Number of partitions to use for shuffle operators.

parquetFilterPushDown

boolean parquetFilterPushDown()

When true predicates will be passed to the parquet record reader when possible.

externalSortEnabled

boolean externalSortEnabled()

When true the planner will use the external sort, which may spill to disk.

codegenEnabled

boolean codegenEnabled()

When set to true, Spark SQL will use the Scala compiler at runtime to generate custom bytecode that evaluates expressions found in queries. In general this custom code runs much faster than interpreted evaluation, but there are significant start-up costs due to compilation. As a result codegen is only beneficial when queries run for a long time, or when the same expressions are used multiple times.

Defaults to false as this feature is currently experimental.

autoBroadcastJoinThreshold

int autoBroadcastJoinThreshold()

Upper bound on the sizes (in bytes) of the tables qualified for the auto conversion to a broadcast value during the physical executions of join operations. Setting this to -1 effectively disables auto conversion.

Hive setting: hive.auto.convert.join.noconditionaltask.size, whose default value is 10000.

defaultSizeInBytes

long defaultSizeInBytes()

The default size in bytes to assign to a logical operator's estimation statistics. By default, it is set to a larger value than autoBroadcastJoinThreshold, hence any logical operator without a properly implemented estimation of this statistic will not be incorrectly broadcasted in joins.

isParquetBinaryAsString

boolean isParquetBinaryAsString()

When set to true, we always treat byte arrays in Parquet files as strings.

inMemoryPartitionPruning

boolean inMemoryPartitionPruning()

When set to true, partition pruning for in-memory columnar tables is enabled.

columnNameOfCorruptRecord

String columnNameOfCorruptRecord()

setConf

void setConf(java.util.Properties props)

Set Spark SQL configuration properties.

setConf

void setConf(String key,
           String value)

Set the given Spark SQL configuration property.

getConf

String getConf(String key)

Return the value of Spark SQL configuration property for the given key.

getConf

String getConf(String key,
             String defaultValue)

Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

getAllConfs

scala.collection.immutable.Map<String,String> getAllConfs()

Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

clear

void clear()