org.apache.spark

Class TestUtils

Object

org.apache.spark.TestUtils

public class TestUtils
extends Object

Utilities for tests. Included in main codebase since it's used by multiple projects.

TODO: See if we can move this to the test codebase by specifying test dependencies between projects.

Constructor Summary

Constructors

Constructor and Description
TestUtils()

Method Summary

Modifier and Type	Method and Description
static void	assertExceptionMsg(Throwable exception, String msg, boolean ignoreCase) Asserts that exception message contains the message.
static void	assertNotSpilled(SparkContext sc, String identifier, scala.Function0<scala.runtime.BoxedUnit> body) Run some code involving jobs submitted to the given context and assert that the jobs did not spill.
static void	assertSpilled(SparkContext sc, String identifier, scala.Function0<scala.runtime.BoxedUnit> body) Run some code involving jobs submitted to the given context and assert that the jobs spilled.
static void	configTestLog4j(String level) config a log4j properties used for testsuite
static java.io.File	createCompiledClass(String className, java.io.File destDir, String toStringValue, String baseClass, scala.collection.Seq<java.net.URL> classpathUrls, scala.collection.Seq<String> implementsClasses, String extraCodeBody) Creates a compiled class with the given name.
static java.io.File	createCompiledClass(String className, java.io.File destDir, org.apache.spark.TestUtils.JavaSourceFromString sourceFile, scala.collection.Seq<java.net.URL> classpathUrls) Creates a compiled class with the source file.
static java.net.URL	createJar(scala.collection.Seq<java.io.File> files, java.io.File jarFile, scala.Option<String> directoryPrefix, scala.Option<String> mainClass) Create a jar file that contains this set of files.
static java.net.URL	createJarWithClasses(scala.collection.Seq<String> classNames, String toStringValue, scala.collection.Seq<scala.Tuple2<String,String>> classNamesWithBase, scala.collection.Seq<java.net.URL> classpathUrls) Create a jar that defines classes with the given names.
static java.net.URL	createJarWithFiles(scala.collection.immutable.Map<String,String> files, java.io.File dir) Create a jar file containing multiple files.
static String	createTempJsonFile(java.io.File dir, String prefix, org.json4s.JsonAST.JValue jsonValue) Creates a temp JSON file that contains the input JSON record.
static String	createTempScriptWithExpectedOutput(java.io.File dir, String prefix, String output) Creates a temp bash script that prints the given output.
static int	httpResponseCode(java.net.URL url, String method, scala.collection.Seq<scala.Tuple2<String,String>> headers) Returns the response code from an HTTP(S) URL.
static boolean	isPythonVersionAtLeast38()
static java.io.File[]	recursiveList(java.io.File f) Lists files recursively.
static boolean	testCommandAvailable(String command) Test if a command is available.
static <T> T	withHttpConnection(java.net.URL url, String method, scala.collection.Seq<scala.Tuple2<String,String>> headers, scala.Function1<java.net.HttpURLConnection,T> fn)
static <L extends SparkListener> void	withListener(SparkContext sc, L listener, scala.Function1<L,scala.runtime.BoxedUnit> body) Runs some code with the given listener installed in the SparkContext.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestUtils

public TestUtils()

Method Detail

createJarWithClasses

public static java.net.URL createJarWithClasses(scala.collection.Seq<String> classNames,
                                                String toStringValue,
                                                scala.collection.Seq<scala.Tuple2<String,String>> classNamesWithBase,
                                                scala.collection.Seq<java.net.URL> classpathUrls)

Create a jar that defines classes with the given names.

Note: if this is used during class loader tests, class names should be unique in order to avoid interference between tests.

Parameters:

classNames - (undocumented)

toStringValue - (undocumented)

classNamesWithBase - (undocumented)

classpathUrls - (undocumented)

Returns:

(undocumented)

createJarWithFiles

public static java.net.URL createJarWithFiles(scala.collection.immutable.Map<String,String> files,
                                              java.io.File dir)

Create a jar file containing multiple files. The files map contains a mapping of file names in the jar file to their contents.

Parameters:

files - (undocumented)

dir - (undocumented)

Returns:

(undocumented)

createJar

public static java.net.URL createJar(scala.collection.Seq<java.io.File> files,
                                     java.io.File jarFile,
                                     scala.Option<String> directoryPrefix,
                                     scala.Option<String> mainClass)

Create a jar file that contains this set of files. All files will be located in the specified directory or at the root of the jar.

Parameters:

files - (undocumented)

jarFile - (undocumented)

directoryPrefix - (undocumented)

mainClass - (undocumented)

Returns:

(undocumented)

createCompiledClass

public static java.io.File createCompiledClass(String className,
                                               java.io.File destDir,
                                               org.apache.spark.TestUtils.JavaSourceFromString sourceFile,
                                               scala.collection.Seq<java.net.URL> classpathUrls)

Creates a compiled class with the source file. Class file will be placed in destDir.

createCompiledClass

public static java.io.File createCompiledClass(String className,
                                               java.io.File destDir,
                                               String toStringValue,
                                               String baseClass,
                                               scala.collection.Seq<java.net.URL> classpathUrls,
                                               scala.collection.Seq<String> implementsClasses,
                                               String extraCodeBody)

Creates a compiled class with the given name. Class file will be placed in destDir.

assertSpilled

public static void assertSpilled(SparkContext sc,
                                 String identifier,
                                 scala.Function0<scala.runtime.BoxedUnit> body)

Run some code involving jobs submitted to the given context and assert that the jobs spilled.

Parameters:

sc - (undocumented)

identifier - (undocumented)

body - (undocumented)

assertNotSpilled

public static void assertNotSpilled(SparkContext sc,
                                    String identifier,
                                    scala.Function0<scala.runtime.BoxedUnit> body)

Run some code involving jobs submitted to the given context and assert that the jobs did not spill.

Parameters:

sc - (undocumented)

identifier - (undocumented)

body - (undocumented)

assertExceptionMsg

public static void assertExceptionMsg(Throwable exception,
                                      String msg,
                                      boolean ignoreCase)

Asserts that exception message contains the message. Please note this checks all exceptions in the tree.

Parameters:

exception - (undocumented)

msg - (undocumented)

ignoreCase - (undocumented)

testCommandAvailable

public static boolean testCommandAvailable(String command)

Test if a command is available.

Parameters:

command - (undocumented)

Returns:

(undocumented)

isPythonVersionAtLeast38

public static boolean isPythonVersionAtLeast38()

httpResponseCode

public static int httpResponseCode(java.net.URL url,
                                   String method,
                                   scala.collection.Seq<scala.Tuple2<String,String>> headers)

Returns the response code from an HTTP(S) URL.

Parameters:

url - (undocumented)

method - (undocumented)

headers - (undocumented)

Returns:

(undocumented)

withHttpConnection

public static <T> T withHttpConnection(java.net.URL url,
                                       String method,
                                       scala.collection.Seq<scala.Tuple2<String,String>> headers,
                                       scala.Function1<java.net.HttpURLConnection,T> fn)

withListener

public static <L extends SparkListener> void withListener(SparkContext sc,
                                                          L listener,
                                                          scala.Function1<L,scala.runtime.BoxedUnit> body)

Runs some code with the given listener installed in the SparkContext. After the code runs, this method will wait until all events posted to the listener bus are processed, and then remove the listener from the bus.

Parameters:

sc - (undocumented)

listener - (undocumented)

body - (undocumented)

configTestLog4j

public static void configTestLog4j(String level)

config a log4j properties used for testsuite

Parameters:

level - (undocumented)

recursiveList

public static java.io.File[] recursiveList(java.io.File f)

Lists files recursively.

Parameters:

f - (undocumented)

Returns:

(undocumented)

createTempJsonFile

public static String createTempJsonFile(java.io.File dir,
                                        String prefix,
                                        org.json4s.JsonAST.JValue jsonValue)

Creates a temp JSON file that contains the input JSON record.

createTempScriptWithExpectedOutput

public static String createTempScriptWithExpectedOutput(java.io.File dir,
                                                        String prefix,
                                                        String output)

Creates a temp bash script that prints the given output.