mygenomics.cloud - Cluster Analysis — (Data points | Birds) of (classes | feathers) flock together….

Cluster Analysis — (Data points | Birds) of (classes | feathers) flock together…. – MyGenomics

Mygenomics.cloud Spined HTML

ClusterWringer— (Data points | Birds) of (classes | feathers) flock together…. – MyGenomics Skip to content MyGenomics Cloud Scale Genomic Analytics Home Technology Blog Contact About Priyanka Dangi ClusterWringer— (Data points | Birds) of (classes | feathers) flock together…. Posted on June 24, 2016January 23, 2017 pdangiPosted in ADAM, Clustering, K-Means, Machine Learning k-means Clustering Today’s blog will focus on k-means its variants, k-means++ and parallelized variant of k-means++, k-means|| (spark.mllib implementation includes k-means||). There is a surfeit of on-line material on Wikipedia and wonk research papers on these topics that do a far largest job probe into the mathematics & theoretical aspects of some of these algos that I don’t plan to imbricate today [most of it frankly, is still Greek to me (no pun intended) – as I’m trying to understand the Expectation Optimization schemes, Vector Quantization & Multi-Variate Gaussian Distributions]. I will take you through k-means|| implementation in Spark on a data set that consists of some of the leading NBA, NFL, Olympic Gymnasts and cluster them using k-means|| by {height, weight} features. Background on ClusterWringerFrom Wikipedia, “machine learning is the sub-field of computer science that gives computers the worthiness to learn without stuff explicitly programmed (Arthur Samuel, 1959)”. Clustering or cluster wringer is a form of machine learning algorithm where the characteristics or features of the data points are known superiority of time, however, the data points are not labeled or classified. This algorithm falls under the domain of “unsupervised learning” (Unsupervised learning sounds like an oxymoron to me. How does learning happen sans supervision? Definitely, not in our senior class.) Unsupervised learning is a type of machine learning algorithm used to yank inferences from datasets consisting of input data without labeled responses. Simply put, the goal of clustering is to infer some structure or similarity in a hodgepodge of unlabeled data. This stratum of similarity is calculated using some loftiness function( for example, standard Euclidean, Manhattan or Jacard), etc based on the nature of the data. Clustering applications are used extensively in various fields such as strained intelligence, pattern recognition, economics, ecology, psychiatry, genomics and marketing. Lloyd’s Algorithm for k-means The most famous clustering formulation is k-means. k-means is not an algorithm, it is a problem formulation. However, due to its ubiquity it is often referred to it as the k-means algorithm; but the standard algorithm for that formulation is the Lloyd’s algorithm. Let’s take a quick squint at the algorithm: Given a set of observations (x1, x2, …, xn), where each observation is a d-dimensional real vector, k-means clustering aims to partition the n observations into k (≤ n) sets S = {S1, S2, …, Sk} so as to minimize the within-cluster sum of squares (WCSS) (sum of loftiness functions of each point in the cluster to the K center). In other words, its objective is to find: where μi is the midpoint of points in Si. The algorithm then iterates between two steps: (I) Data work step: Each centroid defines one of the clusters. In this step, each data point is prescribed to its nearest centroid, based on the squared Euclidean distance.Increasinglyformally, if ci is the hodgepodge of centroids in set C, then each data point x is prescribed to a cluster based on where dist( · ) is the standard (L2) Euclidean distance. Let the set of data point assignments for each ith cluster centroid be Si. (II) Centroid update step: In this step, the centroids are recomputed. This is washed-up by taking the midpoint of all data points prescribed to that centroid’s cluster. The algorithm iterates between steps I and II until a stopping criteria is met (i.e., no data points transpiration clusters, the sum of the square of the distances is minimized, or some maximum number of iterations is reached). This algorithm is guaranteed to converge to a result. The result may be a local optimum (i.e. not necessarily the weightier possible outcome), meaning that assessing increasingly than one run of the algorithm with randomized starting centroids may requite a largest outcome. The k-means algorithm is one of the fastest heuristics clustering algorithms but suffers from verism (could fall in local minima) and is very sensitive to the initial choices of the number of clusters and may require several restarts. It is is the speed and simplicity that makes it attractive, however, adversarial placement of these centroids can heavily impact accuracy. Elbow Method provides some science overdue the art of deciding the value of k.Unelevatedis a plot that shows plot of WSSE forfeit v/s the number. The line orchestration looks like an arm and the “elbow point” on the arm is the value of k that is the weightier as the elbow usually represents where we start to have diminishing returns by increasing k. The forfeit is computed as follows: //Computing Within Set Sum of Squared Errors //Click unelevated image for data val WSSSE = clusters.computeCost(parsedData) k-means++ To gainsay this issue of no verism guarantees of Lloyd’s algorithm, k-means++ was introduced that provides both the speed and the required accuracy. It uses a randomized seeding technique that chooses centers at random from the data points but weighs the data points equal to their squared loftiness from the closest part-way once chosen. In simple terms, cluster centers are initially chosen at random from the set of input observation vectors, where the probability of choosing the data point x is upper if x is not near any previously chosen centers. Please refer k-means++ : The Advantages of Careful Seeding for remoter details and proofs. This paper moreover provides experiments that show k-means++ outperforms k-means in terms of both verism and speed, often by a substantial margin. k-means || The spark.mllib implementation includes a parallelized variant of the k-means++ method tabbed kmeans||. The implementation in spark.mllib has the pursuit parameters: k is the number of desired clusters. maxIterations is the maximum number of iterations to run. initializationMode specifies either random initialization or initialization via k-means||. runs is the number of times to run the k-means algorithm (k-means is not guaranteed to find a globally optimal solution, and when run multiple times on a given dataset, the algorithm returns the weightier clustering result). initializationSteps determines the number of steps in the k-means|| algorithm. epsilon determines the loftiness threshold within which we consider k-means to have converged. initialModel is an optional set of cluster centers used for initialization. If this parameter is supplied, only one run is performed. /* Trains a k-means model using the given set of parameters. Parameters: data - Training points as an RDD of Vector types. k - Number of clusters to create. maxIterations - Maximum number of iterations allowed. runs - This param has no effect since Spark 2.0.0. initializationMode - The initialization algorithm. This can either be "random" or "k-means||". (default: "k-means||") seed - Random seed for cluster initialization. Default is to generate seed based on system time. */ public static KMeansModel train(RDD data, int k, int maxIterations, int runs, String initializationMode, long seed) Scala Code for Spark Shell (Data File) //Imports from mlib scala> import org.apache.spark.mllib.linalg.{ Vector => MLVector, Vectors } scala> import org.apache.spark.mllib.linalg.{Vector, Vectors} scala> import org.apache.spark.mllib.clustering.{ KMeans, KMeansModel } scala> import org.apache.spark.ml.feature.VectorAssembler //User Defined Function scala> val toDouble = udf[Double, String]( _.toDouble) //Reading CSVs and converting to DF scala> val df = sqlContext.read.format("com.databricks.spark.csv").option("header","false").load("../data/ht_wto.csv") //df: org.apache.spark.sql.DataFrame = [C0: string, C1: string, C2: string] scala> val featureDf = df.withColumn("ht", toDouble(df("C1"))).withColumn("wt", toDouble(df("C2"))) //featureDf: org.apache.spark.sql.DataFrame = [C0: string, C1: string, C2: string, ht: double, wt: double] scala> val assembler = new VectorAssembler().setInputCols(Array("ht", "wt")).setOutputCol("features") //assembler: org.apache.spark.ml.feature.VectorAssembler = vecAssembler_c0e327ce0f55 scala> val output = assembler.transform(featureDf) //output: org.apache.spark.sql.DataFrame = [C0: string, C1: string, C2: string, ht: double, wt: double, features: vector] scala> val featVctrMap = output.rdd.map{row => (row.getString(0), row.getAs[Vector](5),row.getDouble(3),row.getDouble(4))} //featVctrMap: org.apache.spark.rdd.RDD[(String, org.apache.spark.mllib.linalg.Vector, Double, Double)] = MapPartitionsRDD[10] at map at :34 scala> featVctrMap.first res3: (String, org.apache.spark.mllib.linalg.Vector, Double, Double) = (Konz, Peter,[77.0,317.0],77.0,317.0) //Cache the RDD as it k-means is an iterative algorithm featVctrMap.cache //._2 --> picks the second item from the "tuple" scala> val model = KMeans.train(featVctrMap.map(_._2), 5, 1) scala> val predictions = featVctrMap.map(elt => {(elt._1, (model.predict(elt._2), elt._3, elt._4)) }) scala> model.computeCost(featVctrMap.map(_._2)) scala> for ((k, v)