In addition, UDF allows the user to develop more complicated hash functions in pure Python or reuse the same function they have already developed. Spark support sha or md5 function natively, but UDF allows us to reuse the same hash and salt method on multiple columns.
Although Spark already supports plenty of mainstream functions which cover most of use cases, we might still want to build customized functions to transform data for migration existing scripts or for developers who are not familiar with Spark.įor example, let’s say we need a function to hash columns. Generally, all Spark-native functions applied on Spark DataFrame are vectorized, which takes advantage of Spark’s parallel processing. UDF is an abbreviation of “user defined function” in Spark. In that case, Pandas UDF is there to apply Python functions directly on Spark DataFrame which allows engineers or scientists to develop in pure Python and still take advantage of Spark’s parallel processing features at the same time. However, if developers develop in pure Python on Databricks, they barely take advantage of features (especially parallel processing for big data) from Spark. For example, while developing ML models, developers may depend on certain libraries available in Python which are not supported by Spark natively (like basic Scikit learn library, which cannot be applied on Spark DataFrame). However, Python has become the default language for data scientists to build ML models, where a huge number of toolkits and libraries can be very useful. When Spark engineers develop in Databricks, they use Spark DataFrame API to process or transform big data which are native Spark functions.
The idea of Pandas UDF is to narrow the gap between processing big data using Spark and developing in Python. Pandas UDF was introduced in Spark 2.3 and continues to be a useful technique for optimizing Spark jobs in Databricks.
0 kommentar(er)
