pyspark.sql.streaming.DataStreamReader.load

DataStreamReader.load(path=None, format=None, schema=None, **options)

Loads a data stream from a data source and returns it as a DataFrame.

New in version 2.0.0.

Parameters:

pathstr, optional

optional string for file-system backed data sources.

formatstr, optional

optional string for format of the data source. Default to ‘parquet’.

schemapyspark.sql.types.StructType or str, optional

optional pyspark.sql.types.StructType for the input schema or a DDL-formatted string (For example col0 INT, col1 DOUBLE).

**optionsdict

all other string options

Notes

This API is evolving.

Examples

>>> json_sdf = spark.readStream.format("json") \
...     .schema(sdf_schema) \
...     .load(tempfile.mkdtemp())
>>> json_sdf.isStreaming
True
>>> json_sdf.schema == sdf_schema
True

pyspark.sql.streaming.DataStreamReader.json pyspark.sql.streaming.DataStreamReader.option