pyspark.sql.functions.variant_strip_nulls#

pyspark.sql.functions.variant_strip_nulls(v, include_arrays=True)[source]#

Recursively removes object fields and array elements whose value is a variant null, unless include_arrays is False, in which case null array elements are kept. Returns NULL if any argument is NULL.

Added in version 4.3.0.

Parameters:

vColumn or str: a variant column or column name
include_arraysbool, optional: whether null elements are also removed from arrays (default True).

Returns:

Column: a variant column with variant null fields/elements removed

Examples

>>> from pyspark.sql.functions import lit, parse_json, to_json, variant_strip_nulls
>>> df = spark.createDataFrame([{
...     'json': '''{ "a" : 1, "b" : null, "c" : [1, null], "d" : { "e" : null, "f" : 4 } }'''
... }])
>>> v = parse_json(df.json)
>>> df.select(to_json(variant_strip_nulls(v)).alias("r")).collect()
[Row(r='{"a":1,"c":[1],"d":{"f":4}}')]
>>> df.select(to_json(variant_strip_nulls(v, False)).alias("r")).collect()
[Row(r='{"a":1,"c":[1,null],"d":{"f":4}}')]
>>> df.select(variant_strip_nulls(lit(None)).alias("r")).collect()
[Row(r=None)]
>>> df2 = spark.createDataFrame([{'json': '{"a": null}'}, {'json': 'null'}])
>>> v2 = parse_json(df2.json)
>>> df2.select(to_json(variant_strip_nulls(v2)).alias("r")).collect()
[Row(r='{}'), Row(r='null')]