pyspark.sql.streaming.StreamingQuery.processAllAvailable#

StreamingQuery.processAllAvailable()[source]#

Blocks until all available data in the source has been processed and committed to the sink. This method is intended for testing.

New in version 2.0.0.

Changed in version 3.5.0: Supports Spark Connect.

Notes

In the case of continually arriving data, this method may block forever. Additionally, this method is only guaranteed to block until data that has been synchronously appended data to a stream source prior to invocation. (i.e. getOffset must immediately reflect the addition).

Examples

>>> sdf = spark.readStream.format("rate").load()
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()

Blocks query until all available data in the source has been processed and committed to the sink

>>> sq.processAllAvailable
<bound method StreamingQuery.processAllAvailable ...>

>>> sq.stop()