sdkagent

spark API reference

400 public APIs from spark (apache/spark) — 101 classes, 6 functions, 293 methods. Signatures extracted by static analysis of the actual source.

Repository: apache/spark

KindCount
Classes101
Functions6
Methods293

API list

classdev.merge_spark_pr.Title
Structured PR title: SPARK refs, component tags, and body.
methoddev.merge_spark_pr.Title.parse(raw:str) -> 'Title'
Parse a PR title string into a :class:`Title`.
funcdev.pr_merge_status.git(*check=True, *args)
Runs a git command.
classdev.sparktestsupport.modules.Module
A module is the basic abstraction in our test runner script.
methodpython.pyspark.accumulators.Accumulator.add(term:T) -> None
Adds a term to this accumulator's value
methodpython.pyspark.accumulators.Accumulator.value() -> T
Get the accumulator's value; only usable in driver program
methodpython.pyspark.cloudpickle.cloudpickle.Pickler.save_global(obj, name=None, pack=struct.pack)
Main dispatch method.
funcpython.pyspark.cloudpickle.cloudpickle.instance(cls)
Create a new instance of a class.
classpython.pyspark.conf.SparkConf
Configuration for a Spark application.
methodpython.pyspark.conf.SparkConf.contains(key:str) -> bool
Does this configuration contain a given key?
methodpython.pyspark.conf.SparkConf.getAll() -> List[Tuple[str, str]]
Get all values as a list of key-value pairs.
methodpython.pyspark.conf.SparkConf.set(key:str, value:str) -> 'SparkConf'
Set a configuration property.
methodpython.pyspark.conf.SparkConf.setAll(pairs:List[Tuple[str, str]]) -> 'SparkConf'
Set multiple parameters, passed as a list of key-value pairs.
methodpython.pyspark.conf.SparkConf.setAppName(value:str) -> 'SparkConf'
Set application name.
methodpython.pyspark.conf.SparkConf.setIfMissing(key:str, value:str) -> 'SparkConf'
Set a configuration property, if not already set.
methodpython.pyspark.conf.SparkConf.setMaster(value:str) -> 'SparkConf'
Set master URL to connect to.
methodpython.pyspark.conf.SparkConf.setSparkHome(value:str) -> 'SparkConf'
Set path where Spark is installed on worker nodes.
methodpython.pyspark.core.broadcast.Broadcast.destroy(blocking:bool=False) -> None
Destroy all data and metadata related to this broadcast variable.
methodpython.pyspark.core.broadcast.Broadcast.dump(value:T, f:BinaryIO) -> None
Write a pickled representation of value to the open file or socket.
methodpython.pyspark.core.broadcast.Broadcast.load(file:BinaryIO) -> T
Read a pickled representation of value from the open file or socket.
methodpython.pyspark.core.broadcast.Broadcast.unpersist(blocking:bool=False) -> None
Delete cached copies of this broadcast on the executors.
methodpython.pyspark.core.broadcast.Broadcast.value() -> T
Return the broadcasted value
classpython.pyspark.core.context.SparkContext
Main entry point for Spark functionality.
methodpython.pyspark.core.context.SparkContext.addArchive(path:str) -> None
Add an archive to be downloaded with this Spark job on every node.
methodpython.pyspark.core.context.SparkContext.addFile(path:str, recursive:bool=False) -> None
Add a file to be downloaded with this Spark job on every node.
methodpython.pyspark.core.context.SparkContext.addJobTag(tag:str) -> None
Add a tag to be assigned to all the jobs started by this thread.
methodpython.pyspark.core.context.SparkContext.applicationId() -> str
A unique identifier for the Spark application.
methodpython.pyspark.core.context.SparkContext.cancelJobGroup(groupId:str) -> None
Cancel active jobs for the specified group.
methodpython.pyspark.core.context.SparkContext.cancelJobsWithTag(tag:str) -> None
Cancel active jobs that have the specified tag.
methodpython.pyspark.core.context.SparkContext.clearJobTags() -> None
Clear the current thread's job tags.
methodpython.pyspark.core.context.SparkContext.dump_profiles(path:str) -> None
Dump the profile stats into directory `path` ..
methodpython.pyspark.core.context.SparkContext.getCheckpointDir() -> Optional[str]
Return the directory where RDDs are checkpointed.
methodpython.pyspark.core.context.SparkContext.getSystemProperty(key:str) -> str
Get a Java system property, such as `java.home`.
methodpython.pyspark.core.context.SparkContext.parallelize(c:Iterable[T], numSlices:Optional[int]=None) -> RDD[T]
Distribute a local Python collection to form an RDD.
methodpython.pyspark.core.context.SparkContext.setCheckpointDir(dirName:str) -> None
Set the directory under which RDDs are going to be checkpointed.
methodpython.pyspark.core.context.SparkContext.setJobDescription(value:str) -> None
Set a human readable description of the current job.
methodpython.pyspark.core.context.SparkContext.setLogLevel(logLevel:str) -> None
Control our logLevel.
methodpython.pyspark.core.context.SparkContext.setSystemProperty(key:str, value:str) -> None
Set a Java system property, such as `spark.executor.memory`.
methodpython.pyspark.core.context.SparkContext.show_profiles() -> None
Print the profile stats to stdout ..
methodpython.pyspark.core.context.SparkContext.sparkUser() -> str
Get SPARK_USER for user who is running SparkContext.
methodpython.pyspark.core.context.SparkContext.statusTracker() -> StatusTracker
Return :class:`StatusTracker` object ..
methodpython.pyspark.core.context.SparkContext.stop() -> None
Shut down the :class:`SparkContext`.
methodpython.pyspark.core.context.SparkContext.union(rdds:List[RDD[T]]) -> RDD[T]
Build the union of a list of RDDs.
classpython.pyspark.core.rdd.RDD
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
methodpython.pyspark.core.rdd.RDD.checkpoint() -> None
Mark this RDD for checkpointing.
methodpython.pyspark.core.rdd.RDD.cleanShuffleDependencies(blocking:bool=False) -> None
Removes an RDD's shuffles and it's non-persisted ancestors.
methodpython.pyspark.core.rdd.RDD.coalesce(numPartitions:int, shuffle:bool=False) -> 'RDD[T]'
Return a new RDD that is reduced into `numPartitions` partitions.
methodpython.pyspark.core.rdd.RDD.collect() -> List[T]
Return a list that contains all the elements in this RDD.
methodpython.pyspark.core.rdd.RDD.collectWithJobGroup(groupId:str, description:str, interruptOnCancel:bool=False) -> 'List[T]'
When collect rdd, use this method to specify job group.
methodpython.pyspark.core.rdd.RDD.context() -> 'SparkContext'
The :class:`SparkContext` that this RDD was created on.
methodpython.pyspark.core.rdd.RDD.count() -> int
Return the number of elements in this RDD.
methodpython.pyspark.core.rdd.RDD.countApproxDistinct(relativeSD:float=0.05) -> int
Return approximate number of distinct elements in the RDD.
methodpython.pyspark.core.rdd.RDD.distinct(numPartitions:Optional[int]=None) -> 'RDD[T]'
Return a new RDD containing the distinct elements in this RDD.
methodpython.pyspark.core.rdd.RDD.filter(f:Callable[[T], bool]) -> 'RDD[T]'
Return a new RDD containing only the elements that satisfy a predicate.
methodpython.pyspark.core.rdd.RDD.first() -> T
Return the first element in this RDD.
methodpython.pyspark.core.rdd.RDD.foreach(f:Callable[[T], None]) -> None
Applies a function to all elements of this RDD.
methodpython.pyspark.core.rdd.RDD.foreachPartition(f:Callable[[Iterable[T]], None]) -> None
Applies a function to each partition of this RDD.
methodpython.pyspark.core.rdd.RDD.getNumPartitions() -> int
Returns the number of partitions in RDD ..
methodpython.pyspark.core.rdd.RDD.getStorageLevel() -> StorageLevel
Get the RDD's current storage level.
methodpython.pyspark.core.rdd.RDD.groupBy(f:Callable[[T], K], numPartitions:Optional[int]=None, partitionFunc:Callable[[K], int]=portable_hash) -> 'RDD[Tuple[K, Iterable[T]]]'
Return an RDD of grouped items.
methodpython.pyspark.core.rdd.RDD.histogram(buckets:Union[int, List['S'], Tuple['S', ...]]) -> Tuple[Sequence['S'], List[int]]
Compute a histogram using the provided buckets.
methodpython.pyspark.core.rdd.RDD.id() -> int
A unique ID for this RDD (within its SparkContext).
methodpython.pyspark.core.rdd.RDD.intersection(other:'RDD[T]') -> 'RDD[T]'
Return the intersection of this RDD and another one.
methodpython.pyspark.core.rdd.RDD.keyBy(f:Callable[[T], K]) -> 'RDD[Tuple[K, T]]'
Creates tuples of the elements in this RDD by applying `f`.
methodpython.pyspark.core.rdd.RDD.keys() -> 'RDD[K]'
Return an RDD with the keys of each tuple.
methodpython.pyspark.core.rdd.RDD.leftOuterJoin(other:'RDD[Tuple[K, U]]', numPartitions:Optional[int]=None) -> 'RDD[Tuple[K, Tuple[V, Optional[U]]]]'
Perform a left outer join of `self` and `other`.
methodpython.pyspark.core.rdd.RDD.lookup(key:K) -> List[V]
Return the list of values in the RDD for key `key`.
methodpython.pyspark.core.rdd.RDD.map(f:Callable[[T], U], preservesPartitioning:bool=False) -> 'RDD[U]'
Return a new RDD by applying a function to each element of this RDD.
methodpython.pyspark.core.rdd.RDD.mapPartitions(f:Callable[[Iterable[T]], Iterable[U]], preservesPartitioning:bool=False) -> 'RDD[U]'
Return a new RDD by applying a function to each partition of this RDD.
methodpython.pyspark.core.rdd.RDD.mean() -> float
Compute the mean of this RDD's elements.
methodpython.pyspark.core.rdd.RDD.name() -> Optional[str]
Return the name of this RDD.
methodpython.pyspark.core.rdd.RDD.randomSplit(weights:Sequence[Union[int, float]], seed:Optional[int]=None) -> 'List[RDD[T]]'
Randomly splits this RDD with the provided weights.
methodpython.pyspark.core.rdd.RDD.repartition(numPartitions:int) -> 'RDD[T]'
Return a new RDD that has exactly numPartitions partitions.
methodpython.pyspark.core.rdd.RDD.sample(withReplacement:bool, fraction:float, seed:Optional[int]=None) -> 'RDD[T]'
Return a sampled subset of this RDD.
methodpython.pyspark.core.rdd.RDD.saveAsPickleFile(path:str, batchSize:int=10) -> None
Save this RDD as a SequenceFile of serialized objects.
methodpython.pyspark.core.rdd.RDD.setName(name:str) -> 'RDD[T]'
Assign a name to this RDD.
methodpython.pyspark.core.rdd.RDD.sortBy(keyfunc:Callable[[T], 'S'], ascending:bool=True, numPartitions:Optional[int]=None) -> 'RDD[T]'
Sorts this RDD by the given keyfunc ..
methodpython.pyspark.core.rdd.RDD.stdev() -> float
Compute the standard deviation of this RDD's elements.
methodpython.pyspark.core.rdd.RDD.subtract(other:'RDD[T]', numPartitions:Optional[int]=None) -> 'RDD[T]'
Return each value in `self` that is not contained in `other`.
methodpython.pyspark.core.rdd.RDD.sum() -> 'NumberOrArray'
Add up the elements in this RDD.
methodpython.pyspark.core.rdd.RDD.take(num:int) -> List[T]
Take the first num elements of the RDD.
methodpython.pyspark.core.rdd.RDD.takeSample(withReplacement:bool, num:int, seed:Optional[int]=None) -> List[T]
Return a fixed-size sampled subset of this RDD.
methodpython.pyspark.core.rdd.RDD.toLocalIterator(prefetchPartitions:bool=False) -> Iterator[T]
Return an iterator that contains all of the elements in this RDD.
methodpython.pyspark.core.rdd.RDD.treeReduce(f:Callable[[T, T], T], depth:int=2) -> T
Reduces the elements of this RDD in a multi-level tree pattern.
methodpython.pyspark.core.rdd.RDD.union(other:'RDD[U]') -> 'RDD[Union[T, U]]'
Return the union of this RDD and another one.
methodpython.pyspark.core.rdd.RDD.values() -> 'RDD[V]'
Return an RDD with the values of each tuple.
methodpython.pyspark.core.rdd.RDD.variance() -> float
Compute the variance of this RDD's elements.
methodpython.pyspark.core.rdd.RDD.zipWithIndex() -> 'RDD[Tuple[T, int]]'
Zips this RDD with its element indices.
methodpython.pyspark.core.rdd.RDD.zipWithUniqueId() -> 'RDD[Tuple[T, int]]'
Zips this RDD with generated unique Long ids.
classpython.pyspark.core.status.SparkExecutorInfo
Exposes information about Spark Executors.
classpython.pyspark.core.status.SparkJobInfo
Exposes information about Spark Jobs.
classpython.pyspark.core.status.SparkStageInfo
Exposes information about Spark Stages.
methodpython.pyspark.core.status.StatusTracker.getJobIdsForGroup(jobGroup:Optional[str]=None) -> List[int]
Return a list of all known jobs in a particular job group.
funcpython.pyspark.daemon.worker(sock:socket.socket, authenticated:bool) -> int
Called by a worker process after the fork().
classpython.pyspark.errors.exceptions.base.AnalysisException
Failed to analyze a SQL query plan.
classpython.pyspark.errors.exceptions.base.IllegalArgumentException
Passed an illegal or inappropriate argument.
classpython.pyspark.errors.exceptions.base.ParseException
Failed to parse a SQL command.
classpython.pyspark.errors.exceptions.base.PySparkAssertionError
Wrapper class for AssertionError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkException
Base Exception for handling errors generated from PySpark.
methodpython.pyspark.errors.exceptions.base.PySparkException.getCondition() -> Optional[str]
Returns an error condition.
methodpython.pyspark.errors.exceptions.base.PySparkException.getErrorClass() -> Optional[str]
Returns an error class as a string.
methodpython.pyspark.errors.exceptions.base.PySparkException.getMessage() -> str
Returns full error message.
methodpython.pyspark.errors.exceptions.base.PySparkException.getQueryContext() -> List['QueryContext']
Returns :class:`QueryContext`.
methodpython.pyspark.errors.exceptions.base.PySparkException.getSqlState() -> Optional[str]
Returns an SQLSTATE as a string.
classpython.pyspark.errors.exceptions.base.PySparkImportError
Wrapper class for ImportError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkIndexError
Wrapper class for IndexError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkKeyError
Wrapper class for KeyError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkNotImplementedError
Wrapper class for NotImplementedError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkRuntimeError
Wrapper class for RuntimeError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkTypeError
Wrapper class for TypeError to support error classes.
classpython.pyspark.errors.exceptions.base.PySparkValueError
Wrapper class for ValueError to support error classes.
classpython.pyspark.errors.exceptions.base.PythonException
Exceptions thrown from Python workers.
classpython.pyspark.errors.exceptions.base.QueryContext
Query context of a :class:`PySparkException`.
methodpython.pyspark.errors.exceptions.base.QueryContext.contextType() -> QueryContextType
The type of this query context.
methodpython.pyspark.errors.exceptions.base.QueryContext.fragment() -> str
The corresponding fragment of the query which throws the exception.
methodpython.pyspark.errors.exceptions.base.QueryContext.summary() -> str
Summary of the exception cause.
classpython.pyspark.errors.exceptions.base.QueryContextType
The type of :class:`QueryContext`.
classpython.pyspark.errors.exceptions.base.QueryExecutionException
Failed to execute a query.
classpython.pyspark.errors.exceptions.base.SparkUpgradeException
Exception thrown because of Spark upgrade.
classpython.pyspark.errors.exceptions.base.StreamingQueryException
Exception that stopped a :class:`StreamingQuery`.
classpython.pyspark.errors.exceptions.base.UnknownException
None of the other exceptions.
funcpython.pyspark.errors.exceptions.base.recover_python_exception(e:T) -> T
Recover Python exception stack trace.
classpython.pyspark.errors.exceptions.captured.AnalysisException
Failed to analyze a SQL query plan.
classpython.pyspark.errors.exceptions.captured.ArithmeticException
Arithmetic exception.
classpython.pyspark.errors.exceptions.captured.ArrayIndexOutOfBoundsException
Array index out of bounds exception.
classpython.pyspark.errors.exceptions.captured.DateTimeException
Datetime exception.
classpython.pyspark.errors.exceptions.captured.IllegalArgumentException
Passed an illegal or inappropriate argument.
classpython.pyspark.errors.exceptions.captured.NumberFormatException
Number format exception.
classpython.pyspark.errors.exceptions.captured.ParseException
Failed to parse a SQL command.
classpython.pyspark.errors.exceptions.captured.PythonException
Exceptions thrown from Python workers.
classpython.pyspark.errors.exceptions.captured.QueryExecutionException
Failed to execute a query.
classpython.pyspark.errors.exceptions.captured.SparkNoSuchElementException
No such element exception.
classpython.pyspark.errors.exceptions.captured.SparkRuntimeException
Runtime exception.
classpython.pyspark.errors.exceptions.captured.SparkUpgradeException
Exception thrown because of Spark upgrade.
classpython.pyspark.errors.exceptions.captured.StreamingQueryException
Exception that stopped a :class:`StreamingQuery`.
classpython.pyspark.errors.exceptions.captured.UnknownException
None of the other exceptions.
classpython.pyspark.errors.exceptions.captured.UnsupportedOperationException
Unsupported operation exception.
classpython.pyspark.errors.exceptions.connect.ArithmeticException
Arithmetic exception thrown from Spark Connect.
classpython.pyspark.errors.exceptions.connect.DateTimeException
Datetime exception thrown from Spark Connect.
classpython.pyspark.errors.exceptions.connect.InvalidPlanInput
Error thrown when a connect plan is not valid.
classpython.pyspark.errors.exceptions.connect.NumberFormatException
Number format exception thrown from Spark Connect.
classpython.pyspark.errors.exceptions.connect.PythonException
Exceptions thrown from Spark Connect.
classpython.pyspark.errors.exceptions.connect.SparkConnectException
Exception thrown from Spark Connect.
classpython.pyspark.errors.exceptions.connect.SparkConnectGrpcException
Base class to handle the errors from GRPC.
classpython.pyspark.errors.exceptions.connect.SparkNoSuchElementException
No such element exception.
classpython.pyspark.errors.exceptions.connect.SparkRuntimeException
Runtime exception thrown from Spark Connect.
classpython.pyspark.errors.exceptions.connect.UnknownException
Exception for unmapped errors in Spark Connect.
classpython.pyspark.errors.exceptions.tblib.Traceback
Class that wraps builtin Traceback objects.
methodpython.pyspark.errors.exceptions.tblib.Traceback.as_dict() -> dict
Converts to a dictionary representation.
methodpython.pyspark.errors.exceptions.tblib.Traceback.from_string(string:str, strict:bool=True) -> 'Traceback'
Creates an instance by parsing a stacktrace.
classpython.pyspark.errors.utils.ErrorClassesReader
A reader to load error information from error-conditions.json.
methodpython.pyspark.errors.utils.ErrorClassesReader.get_breaking_change_info(errorClass:Optional[str]) -> Optional[Dict[str, Any]]
Returns the breaking change info for an error if it is present.
methodpython.pyspark.errors.utils.ErrorClassesReader.get_sqlstate(errorClass:Optional[str]) -> Optional[str]
Returns the SQL state for the given error class.
classpython.pyspark.logger.logger.JSONFormatter
Custom JSON formatter for logging records.
methodpython.pyspark.logger.logger.JSONFormatter.format(record:logging.LogRecord) -> str
Format the specified record as a JSON string.
classpython.pyspark.messages.zero_copy_byte_stream.ZeroCopyByteStream
Accepts chunks of bytes as zero-copy memory views.
methodpython.pyspark.messages.zero_copy_byte_stream.ZeroCopyByteStream.add_next_chunk(chunk:memoryview) -> None
Adds the next chunk as a read source.
methodpython.pyspark.messages.zero_copy_byte_stream.ZeroCopyByteStream.finish() -> None
Marks the stream as ended.
methodpython.pyspark.messages.zero_copy_byte_stream.ZeroCopyByteStream.read(size:int) -> memoryview
Reads size bytes.
classpython.pyspark.ml.base.Estimator
Abstract class for estimators that fit models to data.
classpython.pyspark.ml.base.Model
Abstract class for models that are fitted by estimators.
classpython.pyspark.ml.base.PredictionModel
Model for prediction tasks (regression and classification).
methodpython.pyspark.ml.base.PredictionModel.predict(value:T) -> float
Predict label for the given features.
methodpython.pyspark.ml.base.PredictionModel.setFeaturesCol(value:str) -> P
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.base.PredictionModel.setPredictionCol(value:str) -> P
Sets the value of :py:attr:`predictionCol`.
classpython.pyspark.ml.base.Predictor
Estimator for prediction tasks (regression and classification).
methodpython.pyspark.ml.base.Predictor.setFeaturesCol(value:str) -> P
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.base.Predictor.setLabelCol(value:str) -> P
Sets the value of :py:attr:`labelCol`.
methodpython.pyspark.ml.base.Predictor.setPredictionCol(value:str) -> P
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.base.Transformer.transform(dataset:DataFrame, params:Optional['ParamMap']=None) -> DataFrame
Transforms the input dataset with optional parameters.
classpython.pyspark.ml.classification.BinaryLogisticRegressionSummary
Binary Logistic regression results for a given model.
classpython.pyspark.ml.classification.ClassificationModel
Model produced by a ``Classifier``.
methodpython.pyspark.ml.classification.ClassificationModel.predictRaw(value:Vector) -> Vector
Raw prediction for each possible label.
methodpython.pyspark.ml.classification.ClassificationModel.setRawPredictionCol(value:str) -> 'P'
Sets the value of :py:attr:`rawPredictionCol`.
classpython.pyspark.ml.classification.Classifier
Classifier for classification tasks.
methodpython.pyspark.ml.classification.Classifier.setRawPredictionCol(value:str) -> 'P'
Sets the value of :py:attr:`rawPredictionCol`.
classpython.pyspark.ml.classification.DecisionTreeClassificationModel
Model fitted by DecisionTreeClassifier.
methodpython.pyspark.ml.classification.DecisionTreeClassificationModel.featureImportances() -> Vector
Estimate of the importance of each feature.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setCacheNodeIds(value:bool) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`cacheNodeIds`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setCheckpointInterval(value:int) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`checkpointInterval`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setImpurity(value:str) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`impurity`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setMaxBins(value:int) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`maxBins`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setMaxDepth(value:int) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`maxDepth`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setMaxMemoryInMB(value:int) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`maxMemoryInMB`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setMinInfoGain(value:float) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`minInfoGain`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setMinInstancesPerNode(value:int) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`minInstancesPerNode`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setPruneTree(value:bool) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`pruneTree`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setSeed(value:int) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.classification.DecisionTreeClassifier.setWeightCol(value:str) -> 'DecisionTreeClassifier'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.classification.FMClassificationModel
Model fitted by :class:`FMClassifier`.
methodpython.pyspark.ml.classification.FMClassificationModel.evaluate(dataset:DataFrame) -> 'FMClassificationSummary'
Evaluates the model on a test dataset.
methodpython.pyspark.ml.classification.FMClassificationModel.factors() -> Matrix
Model factor term.
methodpython.pyspark.ml.classification.FMClassificationModel.intercept() -> float
Model intercept.
methodpython.pyspark.ml.classification.FMClassificationModel.linear() -> Vector
Model linear term.
classpython.pyspark.ml.classification.FMClassificationSummary
Abstraction for FMClassifier Results for a given model.
classpython.pyspark.ml.classification.FMClassificationTrainingSummary
Abstraction for FMClassifier Training results.
methodpython.pyspark.ml.classification.FMClassifier.setFactorSize(value:int) -> 'FMClassifier'
Sets the value of :py:attr:`factorSize`.
methodpython.pyspark.ml.classification.FMClassifier.setFitIntercept(value:bool) -> 'FMClassifier'
Sets the value of :py:attr:`fitIntercept`.
methodpython.pyspark.ml.classification.FMClassifier.setFitLinear(value:bool) -> 'FMClassifier'
Sets the value of :py:attr:`fitLinear`.
methodpython.pyspark.ml.classification.FMClassifier.setInitStd(value:float) -> 'FMClassifier'
Sets the value of :py:attr:`initStd`.
methodpython.pyspark.ml.classification.FMClassifier.setMaxIter(value:int) -> 'FMClassifier'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.classification.FMClassifier.setMiniBatchFraction(value:float) -> 'FMClassifier'
Sets the value of :py:attr:`miniBatchFraction`.
methodpython.pyspark.ml.classification.FMClassifier.setRegParam(value:float) -> 'FMClassifier'
Sets the value of :py:attr:`regParam`.
methodpython.pyspark.ml.classification.FMClassifier.setSeed(value:int) -> 'FMClassifier'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.classification.FMClassifier.setSolver(value:str) -> 'FMClassifier'
Sets the value of :py:attr:`solver`.
methodpython.pyspark.ml.classification.FMClassifier.setStepSize(value:float) -> 'FMClassifier'
Sets the value of :py:attr:`stepSize`.
methodpython.pyspark.ml.classification.FMClassifier.setTol(value:float) -> 'FMClassifier'
Sets the value of :py:attr:`tol`.
classpython.pyspark.ml.classification.GBTClassificationModel
Model fitted by GBTClassifier.
methodpython.pyspark.ml.classification.GBTClassificationModel.featureImportances() -> Vector
Estimate of the importance of each feature.
methodpython.pyspark.ml.classification.GBTClassificationModel.trees() -> List[DecisionTreeRegressionModel]
Trees in this ensemble.
methodpython.pyspark.ml.classification.LinearSVC.setAggregationDepth(value:int) -> 'LinearSVC'
Sets the value of :py:attr:`aggregationDepth`.
methodpython.pyspark.ml.classification.LinearSVC.setFitIntercept(value:bool) -> 'LinearSVC'
Sets the value of :py:attr:`fitIntercept`.
methodpython.pyspark.ml.classification.LinearSVC.setMaxBlockSizeInMB(value:float) -> 'LinearSVC'
Sets the value of :py:attr:`maxBlockSizeInMB`.
methodpython.pyspark.ml.classification.LinearSVC.setMaxIter(value:int) -> 'LinearSVC'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.classification.LinearSVC.setRegParam(value:float) -> 'LinearSVC'
Sets the value of :py:attr:`regParam`.
methodpython.pyspark.ml.classification.LinearSVC.setStandardization(value:bool) -> 'LinearSVC'
Sets the value of :py:attr:`standardization`.
methodpython.pyspark.ml.classification.LinearSVC.setThreshold(value:float) -> 'LinearSVC'
Sets the value of :py:attr:`threshold`.
methodpython.pyspark.ml.classification.LinearSVC.setTol(value:float) -> 'LinearSVC'
Sets the value of :py:attr:`tol`.
methodpython.pyspark.ml.classification.LinearSVC.setWeightCol(value:str) -> 'LinearSVC'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.classification.LinearSVCModel
Model fitted by LinearSVC.
methodpython.pyspark.ml.classification.LinearSVCModel.coefficients() -> Vector
Model coefficients of Linear SVM Classifier.
methodpython.pyspark.ml.classification.LinearSVCModel.evaluate(dataset:DataFrame) -> 'LinearSVCSummary'
Evaluates the model on a test dataset.
methodpython.pyspark.ml.classification.LinearSVCModel.intercept() -> float
Model intercept of Linear SVM Classifier.
methodpython.pyspark.ml.classification.LinearSVCModel.setThreshold(value:float) -> 'LinearSVCModel'
Sets the value of :py:attr:`threshold`.
classpython.pyspark.ml.classification.LinearSVCSummary
Abstraction for LinearSVC Results for a given model.
classpython.pyspark.ml.classification.LinearSVCTrainingSummary
Abstraction for LinearSVC Training results.
classpython.pyspark.ml.classification.LogisticRegression
Logistic regression.
methodpython.pyspark.ml.classification.LogisticRegression.setAggregationDepth(value:int) -> 'LogisticRegression'
Sets the value of :py:attr:`aggregationDepth`.
methodpython.pyspark.ml.classification.LogisticRegression.setElasticNetParam(value:float) -> 'LogisticRegression'
Sets the value of :py:attr:`elasticNetParam`.
methodpython.pyspark.ml.classification.LogisticRegression.setFamily(value:str) -> 'LogisticRegression'
Sets the value of :py:attr:`family`.
methodpython.pyspark.ml.classification.LogisticRegression.setFitIntercept(value:bool) -> 'LogisticRegression'
Sets the value of :py:attr:`fitIntercept`.
methodpython.pyspark.ml.classification.LogisticRegression.setMaxBlockSizeInMB(value:float) -> 'LogisticRegression'
Sets the value of :py:attr:`maxBlockSizeInMB`.
methodpython.pyspark.ml.classification.LogisticRegression.setMaxIter(value:int) -> 'LogisticRegression'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.classification.LogisticRegression.setRegParam(value:float) -> 'LogisticRegression'
Sets the value of :py:attr:`regParam`.
methodpython.pyspark.ml.classification.LogisticRegression.setStandardization(value:bool) -> 'LogisticRegression'
Sets the value of :py:attr:`standardization`.
methodpython.pyspark.ml.classification.LogisticRegression.setTol(value:float) -> 'LogisticRegression'
Sets the value of :py:attr:`tol`.
methodpython.pyspark.ml.classification.LogisticRegression.setWeightCol(value:str) -> 'LogisticRegression'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.classification.LogisticRegressionModel
Model fitted by LogisticRegression.
methodpython.pyspark.ml.classification.LogisticRegressionModel.coefficientMatrix() -> Matrix
Model coefficients.
methodpython.pyspark.ml.classification.LogisticRegressionModel.evaluate(dataset:DataFrame) -> 'LogisticRegressionSummary'
Evaluates the model on a test dataset.
methodpython.pyspark.ml.classification.LogisticRegressionModel.intercept() -> float
Model intercept of binomial logistic regression.
methodpython.pyspark.ml.classification.LogisticRegressionModel.interceptVector() -> Vector
Model intercept.
classpython.pyspark.ml.classification.MultilayerPerceptronClassificationModel
Model fitted by MultilayerPerceptronClassifier.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassificationModel.evaluate(dataset:DataFrame) -> 'MultilayerPerceptronClassificationSummary'
Evaluates the model on a test dataset.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassificationModel.weights() -> Vector
the weights of layers.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setBlockSize(value:int) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`blockSize`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setInitialWeights(value:Vector) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`initialWeights`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setLayers(value:List[int]) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`layers`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setMaxIter(value:int) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setSeed(value:int) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setSolver(value:str) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`solver`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setStepSize(value:float) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`stepSize`.
methodpython.pyspark.ml.classification.MultilayerPerceptronClassifier.setTol(value:float) -> 'MultilayerPerceptronClassifier'
Sets the value of :py:attr:`tol`.
classpython.pyspark.ml.classification.NaiveBayes
Naive Bayes Classifiers.
methodpython.pyspark.ml.classification.NaiveBayes.setModelType(value:str) -> 'NaiveBayes'
Sets the value of :py:attr:`modelType`.
methodpython.pyspark.ml.classification.NaiveBayes.setSmoothing(value:float) -> 'NaiveBayes'
Sets the value of :py:attr:`smoothing`.
methodpython.pyspark.ml.classification.NaiveBayes.setWeightCol(value:str) -> 'NaiveBayes'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.classification.NaiveBayesModel
Model fitted by NaiveBayes.
methodpython.pyspark.ml.classification.NaiveBayesModel.pi() -> Vector
log of class priors.
methodpython.pyspark.ml.classification.NaiveBayesModel.sigma() -> Matrix
variance of each feature.
methodpython.pyspark.ml.classification.NaiveBayesModel.theta() -> Matrix
log of class conditional probabilities.
classpython.pyspark.ml.classification.OneVsRestModel
Model fitted by OneVsRest.
methodpython.pyspark.ml.classification.OneVsRestModel.setFeaturesCol(value:str) -> 'OneVsRestModel'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.classification.OneVsRestModel.setPredictionCol(value:str) -> 'OneVsRestModel'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.classification.OneVsRestModel.setRawPredictionCol(value:str) -> 'OneVsRestModel'
Sets the value of :py:attr:`rawPredictionCol`.
classpython.pyspark.ml.classification.ProbabilisticClassificationModel
Model produced by a ``ProbabilisticClassifier``.
methodpython.pyspark.ml.classification.ProbabilisticClassificationModel.setProbabilityCol(value:str) -> CM
Sets the value of :py:attr:`probabilityCol`.
methodpython.pyspark.ml.classification.ProbabilisticClassificationModel.setThresholds(value:List[float]) -> CM
Sets the value of :py:attr:`thresholds`.
classpython.pyspark.ml.classification.ProbabilisticClassifier
Probabilistic Classifier for classification tasks.
methodpython.pyspark.ml.classification.ProbabilisticClassifier.setProbabilityCol(value:str) -> 'P'
Sets the value of :py:attr:`probabilityCol`.
methodpython.pyspark.ml.classification.ProbabilisticClassifier.setThresholds(value:List[float]) -> 'P'
Sets the value of :py:attr:`thresholds`.
classpython.pyspark.ml.classification.RandomForestClassificationModel
Model fitted by RandomForestClassifier.
methodpython.pyspark.ml.classification.RandomForestClassificationModel.evaluate(dataset:DataFrame) -> 'RandomForestClassificationSummary'
Evaluates the model on a test dataset.
methodpython.pyspark.ml.classification.RandomForestClassificationModel.featureImportances() -> Vector
Estimate of the importance of each feature.
methodpython.pyspark.ml.classification.RandomForestClassificationModel.trees() -> List[DecisionTreeClassificationModel]
Trees in this ensemble.
methodpython.pyspark.ml.classification.RandomForestClassifier.setBootstrap(value:bool) -> 'RandomForestClassifier'
Sets the value of :py:attr:`bootstrap`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setCacheNodeIds(value:bool) -> 'RandomForestClassifier'
Sets the value of :py:attr:`cacheNodeIds`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setCheckpointInterval(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`checkpointInterval`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setImpurity(value:str) -> 'RandomForestClassifier'
Sets the value of :py:attr:`impurity`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setMaxBins(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`maxBins`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setMaxDepth(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`maxDepth`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setMaxMemoryInMB(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`maxMemoryInMB`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setMinInfoGain(value:float) -> 'RandomForestClassifier'
Sets the value of :py:attr:`minInfoGain`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setMinInstancesPerNode(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`minInstancesPerNode`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setNumTrees(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`numTrees`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setPruneTree(value:bool) -> 'RandomForestClassifier'
Sets the value of :py:attr:`pruneTree`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setSeed(value:int) -> 'RandomForestClassifier'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setSubsamplingRate(value:float) -> 'RandomForestClassifier'
Sets the value of :py:attr:`subsamplingRate`.
methodpython.pyspark.ml.classification.RandomForestClassifier.setWeightCol(value:str) -> 'RandomForestClassifier'
Sets the value of :py:attr:`weightCol`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setDistanceMeasure(value:str) -> 'BisectingKMeans'
Sets the value of :py:attr:`distanceMeasure`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setFeaturesCol(value:str) -> 'BisectingKMeans'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setK(value:int) -> 'BisectingKMeans'
Sets the value of :py:attr:`k`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setMaxIter(value:int) -> 'BisectingKMeans'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setMinDivisibleClusterSize(value:float) -> 'BisectingKMeans'
Sets the value of :py:attr:`minDivisibleClusterSize`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setPredictionCol(value:str) -> 'BisectingKMeans'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setSeed(value:int) -> 'BisectingKMeans'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.clustering.BisectingKMeans.setWeightCol(value:str) -> 'BisectingKMeans'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.clustering.BisectingKMeansModel
Model fitted by BisectingKMeans.
methodpython.pyspark.ml.clustering.BisectingKMeansModel.predict(value:Vector) -> int
Predict label for the given features.
methodpython.pyspark.ml.clustering.BisectingKMeansModel.setFeaturesCol(value:str) -> 'BisectingKMeansModel'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.BisectingKMeansModel.setPredictionCol(value:str) -> 'BisectingKMeansModel'
Sets the value of :py:attr:`predictionCol`.
classpython.pyspark.ml.clustering.BisectingKMeansSummary
Bisecting KMeans clustering results for a given model.
classpython.pyspark.ml.clustering.ClusteringSummary
Clustering results for a given model.
methodpython.pyspark.ml.clustering.ClusteringSummary.clusterSizes() -> List[int]
Size of (number of data points in) each cluster.
methodpython.pyspark.ml.clustering.ClusteringSummary.featuresCol() -> str
Name for column of features in `predictions`.
methodpython.pyspark.ml.clustering.ClusteringSummary.k() -> int
The number of clusters the model was trained with.
methodpython.pyspark.ml.clustering.ClusteringSummary.numIter() -> int
Number of iterations.
classpython.pyspark.ml.clustering.DistributedLDAModel
Distributed model fitted by :py:class:`LDA`.
classpython.pyspark.ml.clustering.GaussianMixture
GaussianMixture clustering.
methodpython.pyspark.ml.clustering.GaussianMixture.setAggregationDepth(value:int) -> 'GaussianMixture'
Sets the value of :py:attr:`aggregationDepth`.
methodpython.pyspark.ml.clustering.GaussianMixture.setFeaturesCol(value:str) -> 'GaussianMixture'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.GaussianMixture.setK(value:int) -> 'GaussianMixture'
Sets the value of :py:attr:`k`.
methodpython.pyspark.ml.clustering.GaussianMixture.setMaxIter(value:int) -> 'GaussianMixture'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.clustering.GaussianMixture.setPredictionCol(value:str) -> 'GaussianMixture'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.clustering.GaussianMixture.setProbabilityCol(value:str) -> 'GaussianMixture'
Sets the value of :py:attr:`probabilityCol`.
methodpython.pyspark.ml.clustering.GaussianMixture.setSeed(value:int) -> 'GaussianMixture'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.clustering.GaussianMixture.setTol(value:float) -> 'GaussianMixture'
Sets the value of :py:attr:`tol`.
methodpython.pyspark.ml.clustering.GaussianMixture.setWeightCol(value:str) -> 'GaussianMixture'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.clustering.GaussianMixtureModel
Model fitted by GaussianMixture.
methodpython.pyspark.ml.clustering.GaussianMixtureModel.gaussiansDF() -> DataFrame
Retrieve Gaussian distributions as a DataFrame.
methodpython.pyspark.ml.clustering.GaussianMixtureModel.predict(value:Vector) -> int
Predict label for the given features.
methodpython.pyspark.ml.clustering.GaussianMixtureModel.predictProbability(value:Vector) -> Vector
Predict probability for the given features.
methodpython.pyspark.ml.clustering.GaussianMixtureModel.setFeaturesCol(value:str) -> 'GaussianMixtureModel'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.GaussianMixtureModel.setPredictionCol(value:str) -> 'GaussianMixtureModel'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.clustering.GaussianMixtureModel.setProbabilityCol(value:str) -> 'GaussianMixtureModel'
Sets the value of :py:attr:`probabilityCol`.
classpython.pyspark.ml.clustering.GaussianMixtureSummary
Gaussian mixture clustering results for a given model.
methodpython.pyspark.ml.clustering.KMeans.setDistanceMeasure(value:str) -> 'KMeans'
Sets the value of :py:attr:`distanceMeasure`.
methodpython.pyspark.ml.clustering.KMeans.setFeaturesCol(value:str) -> 'KMeans'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.KMeans.setInitMode(value:str) -> 'KMeans'
Sets the value of :py:attr:`initMode`.
methodpython.pyspark.ml.clustering.KMeans.setInitSteps(value:int) -> 'KMeans'
Sets the value of :py:attr:`initSteps`.
methodpython.pyspark.ml.clustering.KMeans.setIntermediateStorageLevel(value:str) -> 'KMeans'
Sets the value of :py:attr:`intermediateStorageLevel`.
methodpython.pyspark.ml.clustering.KMeans.setK(value:int) -> 'KMeans'
Sets the value of :py:attr:`k`.
methodpython.pyspark.ml.clustering.KMeans.setMaxBlockSizeInMB(value:float) -> 'KMeans'
Sets the value of :py:attr:`maxBlockSizeInMB`.
methodpython.pyspark.ml.clustering.KMeans.setMaxIter(value:int) -> 'KMeans'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.clustering.KMeans.setPredictionCol(value:str) -> 'KMeans'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.clustering.KMeans.setSeed(value:int) -> 'KMeans'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.clustering.KMeans.setSolver(value:str) -> 'KMeans'
Sets the value of :py:attr:`solver`.
methodpython.pyspark.ml.clustering.KMeans.setTol(value:float) -> 'KMeans'
Sets the value of :py:attr:`tol`.
methodpython.pyspark.ml.clustering.KMeans.setWeightCol(value:str) -> 'KMeans'
Sets the value of :py:attr:`weightCol`.
classpython.pyspark.ml.clustering.KMeansModel
Model fitted by KMeans.
methodpython.pyspark.ml.clustering.KMeansModel.predict(value:Vector) -> int
Predict label for the given features.
methodpython.pyspark.ml.clustering.KMeansModel.setFeaturesCol(value:str) -> 'KMeansModel'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.KMeansModel.setPredictionCol(value:str) -> 'KMeansModel'
Sets the value of :py:attr:`predictionCol`.
classpython.pyspark.ml.clustering.KMeansSummary
Summary of KMeans.
methodpython.pyspark.ml.clustering.LDA.setCheckpointInterval(value:int) -> 'LDA'
Sets the value of :py:attr:`checkpointInterval`.
methodpython.pyspark.ml.clustering.LDA.setDocConcentration(value:List[float]) -> 'LDA'
Sets the value of :py:attr:`docConcentration`.
methodpython.pyspark.ml.clustering.LDA.setFeaturesCol(value:str) -> 'LDA'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.LDA.setK(value:int) -> 'LDA'
Sets the value of :py:attr:`k`.
methodpython.pyspark.ml.clustering.LDA.setKeepLastCheckpoint(value:bool) -> 'LDA'
Sets the value of :py:attr:`keepLastCheckpoint`.
methodpython.pyspark.ml.clustering.LDA.setLearningDecay(value:float) -> 'LDA'
Sets the value of :py:attr:`learningDecay`.
methodpython.pyspark.ml.clustering.LDA.setLearningOffset(value:float) -> 'LDA'
Sets the value of :py:attr:`learningOffset`.
methodpython.pyspark.ml.clustering.LDA.setMaxIter(value:int) -> 'LDA'
Sets the value of :py:attr:`maxIter`.
methodpython.pyspark.ml.clustering.LDA.setOptimizeDocConcentration(value:bool) -> 'LDA'
Sets the value of :py:attr:`optimizeDocConcentration`.
methodpython.pyspark.ml.clustering.LDA.setOptimizer(value:str) -> 'LDA'
Sets the value of :py:attr:`optimizer`.
methodpython.pyspark.ml.clustering.LDA.setSeed(value:int) -> 'LDA'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.clustering.LDA.setSubsamplingRate(value:float) -> 'LDA'
Sets the value of :py:attr:`subsamplingRate`.
methodpython.pyspark.ml.clustering.LDA.setTopicConcentration(value:float) -> 'LDA'
Sets the value of :py:attr:`topicConcentration`.
methodpython.pyspark.ml.clustering.LDA.setTopicDistributionCol(value:str) -> 'LDA'
Sets the value of :py:attr:`topicDistributionCol`.
classpython.pyspark.ml.clustering.LDAModel
Latent Dirichlet Allocation (LDA) model.
methodpython.pyspark.ml.clustering.LDAModel.describeTopics(maxTermsPerTopic:int=10) -> DataFrame
Return the topics described by their top-weighted terms.
methodpython.pyspark.ml.clustering.LDAModel.logPerplexity(dataset:DataFrame) -> float
Calculate an upper bound on perplexity.
methodpython.pyspark.ml.clustering.LDAModel.setFeaturesCol(value:str) -> 'M'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.clustering.LDAModel.setSeed(value:int) -> 'M'
Sets the value of :py:attr:`seed`.
methodpython.pyspark.ml.clustering.LDAModel.setTopicDistributionCol(value:str) -> 'M'
Sets the value of :py:attr:`topicDistributionCol`.
classpython.pyspark.ml.clustering.LocalLDAModel
Local (non-distributed) model fitted by :py:class:`LDA`.
funcpython.pyspark.ml.common.callJavaFunc(sc:'pyspark.core.context.SparkContext', func:Callable[..., 'JavaObjectOrPickleDump'], *args:Any) -> 'JavaObjectOrPickleDump'
Call Java Function
funcpython.pyspark.ml.common.inherit_doc(cls:'C') -> 'C'
A decorator that makes a class inherit documentation from its parents.
classpython.pyspark.ml.connect.base.Estimator
Abstract class for estimators that fit models to data.
methodpython.pyspark.ml.connect.base.Evaluator.evaluate(dataset:DataFrame, params:Optional['ParamMap']=None) -> float
Evaluates the output with optional parameters.
classpython.pyspark.ml.connect.base.Model
Abstract class for models that are fitted by estimators.
methodpython.pyspark.ml.connect.base.PredictionModel.setFeaturesCol(value:str) -> 'PredictionModel'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.connect.base.PredictionModel.setPredictionCol(value:str) -> 'PredictionModel'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.connect.base.Predictor.setFeaturesCol(value:str) -> 'Predictor'
Sets the value of :py:attr:`featuresCol`.
methodpython.pyspark.ml.connect.base.Predictor.setLabelCol(value:str) -> 'Predictor'
Sets the value of :py:attr:`labelCol`.
methodpython.pyspark.ml.connect.base.Predictor.setPredictionCol(value:str) -> 'Predictor'
Sets the value of :py:attr:`predictionCol`.
methodpython.pyspark.ml.connect.base.Transformer.transform(dataset:Union[DataFrame, pd.DataFrame], params:Optional['ParamMap']=None) -> Union[DataFrame, pd.DataFrame]
Transforms the input dataset.
classpython.pyspark.ml.connect.classification.LogisticRegression
Logistic regression estimator.
classpython.pyspark.ml.connect.classification.LogisticRegressionModel
Model fitted by LogisticRegression.
classpython.pyspark.ml.connect.feature.MaxAbsScalerModel
Model fitted by MaxAbsScaler.
classpython.pyspark.ml.connect.feature.StandardScalerModel
Model fitted by StandardScaler.
classpython.pyspark.ml.connect.pipeline.Pipeline
A simple pipeline, which acts as an estimator.
methodpython.pyspark.ml.connect.pipeline.Pipeline.copy(extra:Optional['ParamMap']=None) -> 'Pipeline'
Creates a copy of this instance.
methodpython.pyspark.ml.connect.pipeline.Pipeline.getStages() -> List[Params]
Get pipeline stages.
methodpython.pyspark.ml.connect.pipeline.Pipeline.setParams(*stages:Optional[List[Params]]=None) -> 'Pipeline'
setParams(self, \*, stages=None) Sets params for Pipeline.
methodpython.pyspark.ml.connect.pipeline.Pipeline.setStages(value:List[Params]) -> 'Pipeline'
Set pipeline stages.
methodpython.pyspark.ml.evaluation.Evaluator.evaluate(dataset:DataFrame, params:Optional['ParamMap']=None) -> float
Evaluates the output with optional parameters.
classpython.pyspark.ml.feature.Binarizer
Binarize a column of continuous features given a threshold.
methodpython.pyspark.ml.feature.Binarizer.setInputCol(value:str) -> 'Binarizer'
Sets the value of :py:attr:`inputCol`.
methodpython.pyspark.ml.feature.Binarizer.setInputCols(value:List[str]) -> 'Binarizer'
Sets the value of :py:attr:`inputCols`.
methodpython.pyspark.ml.feature.Binarizer.setOutputCol(value:str) -> 'Binarizer'
Sets the value of :py:attr:`outputCol`.
methodpython.pyspark.ml.feature.Binarizer.setOutputCols(value:List[str]) -> 'Binarizer'
Sets the value of :py:attr:`outputCols`.
methodpython.pyspark.ml.feature.Binarizer.setThreshold(value:float) -> 'Binarizer'
Sets the value of :py:attr:`threshold`.
methodpython.pyspark.ml.feature.Binarizer.setThresholds(value:List[float]) -> 'Binarizer'
Sets the value of :py:attr:`thresholds`.
classpython.pyspark.ml.feature.BucketedRandomProjectionLSH
LSH class for Euclidean distance metrics.
methodpython.pyspark.ml.feature.BucketedRandomProjectionLSH.setBucketLength(value:float) -> 'BucketedRandomProjectionLSH'
Sets the value of :py:attr:`bucketLength`.
methodpython.pyspark.ml.feature.BucketedRandomProjectionLSH.setSeed(value:int) -> 'BucketedRandomProjectionLSH'
Sets the value of :py:attr:`seed`.
classpython.pyspark.ml.feature.ChiSqSelectorModel
Model fitted by :py:class:`ChiSqSelector`.
methodpython.pyspark.ml.feature.CountVectorizer.setBinary(value:bool) -> 'CountVectorizer'
Sets the value of :py:attr:`binary`.
methodpython.pyspark.ml.feature.CountVectorizer.setInputCol(value:str) -> 'CountVectorizer'
Sets the value of :py:attr:`inputCol`.
methodpython.pyspark.ml.feature.CountVectorizer.setMaxDF(value:float) -> 'CountVectorizer'
Sets the value of :py:attr:`maxDF`.

About this data

These signatures were extracted from the public source of apache/spark using Python's ast module. Argument names, default values, type annotations and return types are taken verbatim from the code. Implementation bodies are never stored. See how it works for details.

Back to all 805 libraries