This is a function for comparing two results and reporting their differences. The function takes two input results, first and second. An option specifying one or more columns to use as a row identifier must also be provided. Given this input, the function will join the first and second input results using the id column and produce a result which for each joined row reports all changes from the first input to the second input. Added/removed rows will be reported as well. The function options are documented in CompareRowsOptions.

This function takes a single input. Each column of the input produce an output row containing a selection of properties describing values in the column. The specific reported properties depend on the data type of each column. For all data types, total number of values, distinct values, nulls, frequent and infrequent values and counts will be reported. For numeric types, various statistical measures and number of positive, negative and zeros will be reported as well. For string column the number of special values like empty strings, null-like strings will also be reported. The function options are documented in DescribeColumnsOptions.

This is a timeline-based function for monitoring the development of a result over time. The function takes a single input, imported on a timeline and an option specifying columns to use as a row identifier. The function then performs an operation resembling that of the ds_describe_columns function, but partitioned by the row identifier, so a result will be reported - not for each column - but for each combination of column and row identifier. For numeric columns the changes over time will be described using linear regression for each row id as well. The function options are documented in DescribeRowsOptions.

This is a function for placing a local transform result on an timeline. The function takes two parents as input. One parent must be the result of a source import using a timeline. The other parent must not contain timeline information. The execution takes the non-timeline parent, adorns it with timeline information (thus creating a new single-point timeline) and performs a unification of the two timelines. The function can be used to create a local timeline which includes results from the current pipeline and will typically be used as an intermediate result in timeline-oriented pipelines.

This is a function that uses a previously trained machine learning model for categorical prediction. The function takes two input results - one serving as a data result and one serving as a machine-learning model. The function applies the model to the data result, producing output containing the original data and model probabilities of all possible values for each row. The value with the highest probability is also included as the model prediction. The function is meant to be used in conjunction with the ds_train_random_forest_categorical function. Whereas the input to the former must contain the prediction column (for training), this should not. The function options are documented in ModelConsumerOptions.

This function is similar to ds_predict_random_forest_categorical, but requires a model that predicts a continuous data column.

This function is similar to ds_predict_random_forest_categorical, but requires the prediction column to be present in the input result. It includes the same output, but adds a probability score for how likely the prediction column value is under the model.

This function is similar to ds_test_random_forest_categorical, but requires a model that predicts a continuous data column.

This is a function that builds a machine learning model for categorical prediction. The function takes a single input and a reference to a column with categorical data in that input. Using the random forest algorithm a model is then trained to predict the categorical column based on the other columns. Internally the function leverages the random forest implementation of Apache Spark. The output of the function is a dataset with the models predictions on the training data - see ds_predict_random_forest_categorical. The trained model is stored internally, and is accessible to other transforms referencing this transform. The function options are documented in classes ModelOptions and RandomForestOptions.

This function is similar to ds_train_random_forest_categorical, but requires a column with continuous data as the prediction column, and will output a predicted continuous value for each row.