SparkHadoopUtil

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def addCredentials(conf: JobConf): Unit

   Add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster.

7. def addCurrentUserCredentials(creds: Credentials): Unit

8. def addSecretKeyToUserCredentials(key: String, secret: String): Unit

9. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

10. def clone(): AnyRef

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

11. val conf: Configuration

12. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

13. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

14. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

15. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

16. def getConfigurationFromJobContext(context: JobContext): Configuration

    Using reflection to get the Configuration from JobContext/TaskAttemptContext.

    Using reflection to get the Configuration from JobContext/TaskAttemptContext. If we directly call JobContext/TaskAttemptContext.getConfiguration, it will generate different byte codes for Hadoop 1.+ and Hadoop 2.+ because JobContext/TaskAttemptContext is class in Hadoop 1.+ while it's interface in Hadoop 2.+.

17. def getCurrentUserCredentials(): Credentials

18. def getSecretKeyFromUserCredentials(key: String): Array[Byte]

19. def hashCode(): Int

    Definition Classes

    AnyRef → Any

20. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

21. def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

22. def isYarnMode(): Boolean

23. def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

24. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

25. def logDebug(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

26. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

27. def logError(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

28. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

29. def logInfo(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

30. def logName: String

    Attributes

    protected

    Definition Classes

    Logging

31. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

32. def logTrace(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

33. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

34. def logWarning(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

35. def loginUserFromKeytab(principalName: String, keytabFilename: String): Unit

36. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

37. def newConfiguration(conf: SparkConf): Configuration

    Return an appropriate (subclass) of Configuration.

    Return an appropriate (subclass) of Configuration. Creating config can initializes some Hadoop subsystems.

38. def newConfiguration(): Configuration

    Annotations

    @Deprecated

39. final def notify(): Unit

    Definition Classes

    AnyRef

40. final def notifyAll(): Unit

    Definition Classes

    AnyRef

41. def runAsSparkUser(func: () ⇒ Unit): Unit

    Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.

    Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.

    IMPORTANT NOTE: If this function is going to be called repeated in the same process you need to look https://issues.apache.org/jira/browse/HDFS-3545 and possibly do a FileSystem.closeAllForUGI in order to avoid leaking Filesystems

42. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

43. def toString(): String

    Definition Classes

    AnyRef → Any

44. def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit

45. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

46. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

47. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )