# R Interface to Apache Spark

## Documentation for package ‘sparklyr’ version 0.8.4

- DESCRIPTION file.

## Help Pages

A C D E F G H I J L M N R S T

## -- A --

augment.ml_model_generalized_linear_regression | Tidying methods for Spark ML linear models
augment.ml_model_linear_regression | Tidying methods for Spark ML linear models

## -- C --

checkpoint_directory | Set/Get Spark checkpoint directory
compile_package_jars | Compile Scala sources into a Java Archive (jar)
connection_config | Read configuration values for a connection
copy_to.spark_connection | Copy an R Data Frame to Spark

## -- D --

download_scalac | Downloads default Scala Compilers

## -- E --

ensure | Enforce Specific Structure for R Objects
ensure_scalar_boolean | Enforce Specific Structure for R Objects
ensure_scalar_character | Enforce Specific Structure for R Objects
ensure_scalar_double | Enforce Specific Structure for R Objects
ensure_scalar_integer | Enforce Specific Structure for R Objects

## -- F --

find_scalac | Discover the Scala Compiler
ft_binarizer | Feature Transformation - Binarizer (Transformer)
ft_bucketed_random_projection_lsh | Feature Tranformation - LSH (Estimator)
ft_bucketizer | Feature Transformation - Bucketizer (Transformer)
ft_chisq_selector | Feature Tranformation - ChiSqSelector (Estimator)
ft_count_vectorizer | Feature Tranformation - CountVectorizer (Estimator)
ft_dct | Feature Transformation - Discrete Cosine Transform (DCT) (Transformer)
ft_discrete_cosine_transform | Feature Transformation - Discrete Cosine Transform (DCT) (Transformer)
ft_dplyr_transformer | Feature Transformation - SQLTransformer
ft_elementwise_product | Feature Transformation - ElementwiseProduct (Transformer)
ft_feature_hasher | Feature Transformation - FeatureHasher (Transformer)
ft_hashing_tf | Feature Transformation - HashingTF (Transformer)
ft_idf | Feature Tranformation - IDF (Estimator)
ft_imputer | Feature Transformation - Imputer (Estimator)
ft_index_to_string | Feature Transformation - IndexToString (Transformer)
ft_interaction | Feature Transformation - Interaction (Transformer)
ft_lsh | Feature Tranformation - LSH (Estimator)
ft_lsh_utils | Utility functions for LSH models
ft_max_abs_scaler | Feature Tranformation - MaxAbsScaler (Estimator)
ft_minhash_lsh | Feature Tranformation - LSH (Estimator)
ft_min_max_scaler | Feature Tranformation - MinMaxScaler (Estimator)
ft_ngram | Feature Tranformation - NGram (Transformer)
ft_normalizer | Feature Transformation - Normalizer (Transformer)
ft_one_hot_encoder | Feature Transformation - OneHotEncoder (Transformer)
ft_pca | Feature Tranformation - PCA (Estimator)
ft_polynomial_expansion | Feature Transformation - PolynomialExpansion (Transformer)
ft_quantile_discretizer | Feature Transformation - QuantileDiscretizer (Estimator)
ft_regex_tokenizer | Feature Tranformation - RegexTokenizer (Transformer)
ft_r_formula | Feature Tranformation - RFormula (Estimator)
ft_sql_transformer | Feature Transformation - SQLTransformer
ft_standard_scaler | Feature Tranformation - StandardScaler (Estimator)
ft_stop_words_remover | Feature Tranformation - StopWordsRemover (Transformer)
ft_string_indexer | Feature Tranformation - StringIndexer (Estimator)
ft_string_indexer_model | Feature Tranformation - StringIndexer (Estimator)
ft_tokenizer | Feature Tranformation - Tokenizer (Transformer)
ft_vector_assembler | Feature Transformation - VectorAssembler (Transformer)
ft_vector_indexer | Feature Tranformation - VectorIndexer (Estimator)
ft_vector_slicer | Feature Transformation - VectorSlicer (Transformer)
ft_word2vec | Feature Tranformation - Word2Vec (Estimator)

## -- G --

glance.ml_model_generalized_linear_regression | Tidying methods for Spark ML linear models
glance.ml_model_linear_regression | Tidying methods for Spark ML linear models

## -- H --

hive_context | Access the Spark API
hive_context_config | Runtime configuration interface for Hive

## -- I --

invoke | Invoke a Method on a JVM Object
invoke_new | Invoke a Method on a JVM Object
invoke_static | Invoke a Method on a JVM Object
is_ml_estimator | Spark ML - Transform, fit, and predict methods (ml_ interface)
is_ml_transformer | Spark ML - Transform, fit, and predict methods (ml_ interface)

## -- J --

java_context | Access the Spark API

## -- L --

livy_config | Create a Spark Configuration for Livy
livy_service_start | Start Livy
livy_service_stop | Start Livy

## -- M --

ml-params | Spark ML - ML Params
ml-persistence | Spark ML - Model Persistence
ml-transform-methods | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml-tuning | Spark ML - Tuning
ml_aft_survival_regression | Spark ML - Survival Regression
ml_als | Spark ML - ALS
ml_als_factorization | Spark ML - ALS
ml_approx_nearest_neighbors | Utility functions for LSH models
ml_approx_similarity_join | Utility functions for LSH models
ml_association_rules | Frequent Pattern Mining - FPGrowth
ml_binary_classification_eval | Spark ML - Evaluators
ml_binary_classification_evaluator | Spark ML - Evaluators
ml_bisecting_kmeans | Spark ML - Bisecting K-Means Clustering
ml_chisquare_test | Chi-square hypothesis testing for categorical data.
ml_classification_eval | Spark ML - Evaluators
ml_clustering_evaluator | Spark ML - Clustering Evaluator
ml_compute_cost | Spark ML - K-Means Clustering
ml_corr | Compute correlation matrix
ml_cross_validator | Spark ML - Tuning
ml_decision_tree | Spark ML - Decision Trees
ml_decision_tree_classifier | Spark ML - Decision Trees
ml_decision_tree_regressor | Spark ML - Decision Trees
ml_default_stop_words | Default stop words
ml_describe_topics | Spark ML - Latent Dirichlet Allocation
ml_evaluate | Spark ML - Evaluate prediction frames with evaluators
ml_evaluator | Spark ML - Evaluators
ml_feature_importances | Spark ML - Feature Importance for Tree Models
ml_find_synonyms | Feature Tranformation - Word2Vec (Estimator)
ml_fit | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_fit_and_transform | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_fpgrowth | Frequent Pattern Mining - FPGrowth
ml_freq_itemsets | Frequent Pattern Mining - FPGrowth
ml_gaussian_mixture | Spark ML - Gaussian Mixture clustering.
ml_gbt_classifier | Spark ML - Gradient Boosted Trees
ml_gbt_regressor | Spark ML - Gradient Boosted Trees
ml_generalized_linear_regression | Spark ML - Generalized Linear Regression
ml_glm_tidiers | Tidying methods for Spark ML linear models
ml_gradient_boosted_trees | Spark ML - Gradient Boosted Trees
ml_isotonic_regression | Spark ML - Isotonic Regression
ml_is_set | Spark ML - ML Params
ml_kmeans | Spark ML - K-Means Clustering
ml_labels | Feature Tranformation - StringIndexer (Estimator)
ml_lda | Spark ML - Latent Dirichlet Allocation
ml_linear_regression | Spark ML - Linear Regression
ml_linear_svc | Spark ML - LinearSVC
ml_load | Spark ML - Model Persistence
ml_logistic_regression | Spark ML - Logistic Regression
ml_log_likelihood | Spark ML - Latent Dirichlet Allocation
ml_log_perplexity | Spark ML - Latent Dirichlet Allocation
ml_model_data | Extracts data associated with a Spark ML model
ml_multiclass_classification_evaluator | Spark ML - Evaluators
ml_multilayer_perceptron | Spark ML - Multilayer Perceptron
ml_multilayer_perceptron_classifier | Spark ML - Multilayer Perceptron
ml_naive_bayes | Spark ML - Naive-Bayes
ml_one_vs_rest | Spark ML - OneVsRest
ml_param | Spark ML - ML Params
ml_params | Spark ML - ML Params
ml_param_map | Spark ML - ML Params
ml_pca | Feature Tranformation - PCA (Estimator)
ml_pipeline | Spark ML - Pipelines
ml_predict | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_predict.ml_model_classification | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_random_forest | Spark ML - Random Forest
ml_random_forest_classifier | Spark ML - Random Forest
ml_random_forest_regressor | Spark ML - Random Forest
ml_recommend | Spark ML - ALS
ml_regression_evaluator | Spark ML - Evaluators
ml_save | Spark ML - Model Persistence
ml_save.ml_model | Spark ML - Model Persistence
ml_stage | Spark ML - Pipeline stage extraction
ml_stages | Spark ML - Pipeline stage extraction
ml_sub_models | Spark ML - Tuning
ml_summary | Spark ML - Extraction of summary metrics
ml_survival_regression | Spark ML - Survival Regression
ml_topics_matrix | Spark ML - Latent Dirichlet Allocation
ml_train_validation_split | Spark ML - Tuning
ml_transform | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_tree_feature_importance | Spark ML - Feature Importance for Tree Models
ml_uid | Spark ML - UID
ml_validation_metrics | Spark ML - Tuning
ml_vocabulary | Feature Tranformation - CountVectorizer (Estimator)

## -- N --

na.replace | Replace Missing Values in Objects

## -- R --

random_string | Random string generation
registered_extensions | Register a Package that Implements a Spark Extension
register_extension | Register a Package that Implements a Spark Extension

## -- S --

sdf-saveload | Save / Load a Spark DataFrame
sdf-transform-methods | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_along | Create DataFrame for along Object
sdf_bind | Bind multiple Spark DataFrames by row and column
sdf_bind_cols | Bind multiple Spark DataFrames by row and column
sdf_bind_rows | Bind multiple Spark DataFrames by row and column
sdf_broadcast | Broadcast hint
sdf_checkpoint | Checkpoint a Spark DataFrame
sdf_coalesce | Coalesces a Spark DataFrame
sdf_copy_to | Copy an Object into Spark
sdf_debug_string | Debug Info for Spark DataFrame
sdf_describe | Compute summary statistics for columns of a data frame
sdf_dim | Support for Dimension Operations
sdf_fit | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_fit_and_transform | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_import | Copy an Object into Spark
sdf_last_index | Returns the last index of a Spark DataFrame
sdf_len | Create DataFrame for Length
sdf_load_parquet | Save / Load a Spark DataFrame
sdf_load_table | Save / Load a Spark DataFrame
sdf_mutate | Mutate a Spark DataFrame
sdf_mutate_ | Mutate a Spark DataFrame
sdf_ncol | Support for Dimension Operations
sdf_nrow | Support for Dimension Operations
sdf_num_partitions | Gets number of partitions of a Spark DataFrame
sdf_partition | Partition a Spark Dataframe
sdf_persist | Persist a Spark DataFrame
sdf_pivot | Pivot a Spark DataFrame
sdf_predict | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_project | Project features onto principal components
sdf_quantile | Compute (Approximate) Quantiles with a Spark DataFrame
sdf_read_column | Read a Column from a Spark DataFrame
sdf_register | Register a Spark DataFrame
sdf_repartition | Repartition a Spark DataFrame
sdf_residuals | Model Residuals
sdf_residuals.ml_model_generalized_linear_regression | Model Residuals
sdf_residuals.ml_model_linear_regression | Model Residuals
sdf_sample | Randomly Sample Rows from a Spark DataFrame
sdf_save_parquet | Save / Load a Spark DataFrame
sdf_save_table | Save / Load a Spark DataFrame
sdf_schema | Read the Schema of a Spark DataFrame
sdf_separate_column | Separate a Vector Column into Scalar Columns
sdf_seq | Create DataFrame for Range
sdf_sort | Sort a Spark DataFrame
sdf_transform | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_with_sequential_id | Add a Sequential ID Column to a Spark DataFrame
sdf_with_unique_id | Add a Unique ID Column to a Spark DataFrame
spark-api | Access the Spark API
spark-connections | Manage Spark Connections
spark_apply | Apply an R Function in Spark
spark_apply_bundle | Create Bundle for Spark Apply
spark_apply_log | Log Writer for Spark Apply
spark_compilation_spec | Define a Spark Compilation Specification
spark_config | Read Spark Configuration
spark_connect | Manage Spark Connections
spark_connection | Retrieve the Spark Connection Associated with an R Object
spark_connection-class | spark_connection class
spark_connection_is_open | Manage Spark Connections
spark_context | Access the Spark API
spark_context_config | Runtime configuration interface for the Spark Context.
spark_dataframe | Retrieve a Spark DataFrame
spark_default_compilation_spec | Default Compilation Specification for Spark Extensions
spark_dependency | Define a Spark dependency
spark_disconnect | Manage Spark Connections
spark_disconnect_all | Manage Spark Connections
spark_get_checkpoint_dir | Set/Get Spark checkpoint directory
spark_home_set | Set the SPARK_HOME environment variable
spark_jobj | Retrieve a Spark JVM Object Reference
spark_jobj-class | spark_jobj class
spark_load_table | Reads from a Spark Table into a Spark DataFrame.
spark_log | View Entries in the Spark Log
spark_read_csv | Read a CSV file into a Spark DataFrame
spark_read_jdbc | Read from JDBC connection into a Spark DataFrame.
spark_read_json | Read a JSON file into a Spark DataFrame
spark_read_libsvm | Read libsvm file into a Spark DataFrame.
spark_read_parquet | Read a Parquet file into a Spark DataFrame
spark_read_source | Read from a generic source into a Spark DataFrame.
spark_read_table | Reads from a Spark Table into a Spark DataFrame.
spark_read_text | Read a Text file into a Spark DataFrame
spark_save_table | Saves a Spark DataFrame as a Spark table
spark_session | Access the Spark API
spark_session_config | Runtime configuration interface for the Spark Session
spark_set_checkpoint_dir | Set/Get Spark checkpoint directory
spark_table_name | Generate a Table Name from Expression
spark_version | Get the Spark Version Associated with a Spark Connection
spark_version_from_home | Get the Spark Version Associated with a Spark Installation
spark_web | Open the Spark web interface
spark_write_csv | Write a Spark DataFrame to a CSV
spark_write_jdbc | Writes a Spark DataFrame into a JDBC table
spark_write_json | Write a Spark DataFrame to a JSON file
spark_write_parquet | Write a Spark DataFrame to a Parquet file
spark_write_source | Writes a Spark DataFrame into a generic source
spark_write_table | Writes a Spark DataFrame into a Spark table
spark_write_text | Write a Spark DataFrame to a Text file
src_databases | Show database list

## -- T --

tbl_cache | Cache a Spark Table
tbl_change_db | Use specific database
tbl_uncache | Uncache a Spark Table
tidy.ml_model_generalized_linear_regression | Tidying methods for Spark ML linear models
tidy.ml_model_linear_regression | Tidying methods for Spark ML linear models