talisman: (kMeans, cosine) wrong number of centroids (?)

I’m trying to cluster sentences using TF-IDF, cosine similarity and k-means algorithm. I calculate TF-IDF using natural library then I prepare vectors that look like:

[0, 0, 0, 0, 0, 1.24225242, 0, 0, 0, 0, 0] where every value represents a word and the whole vector represents a sentence (in case of this example it would be a sentence with one word).

Then I try to use talisman library to make clusters using created vectors and metrics/distance/cosine. The problem is that I get:

TypeError: Cannot read property 'length' of undefined
    at KMeans.cosine [as distance] (.../node_modules/talisman/metrics/distance/cosine.js:29:21)
    at KMeans.iterate (.../node_modules/talisman/clustering/k-means.js:164:22)
    at KMeans.run (.../node_modules/talisman/clustering/k-means.js:224:12)
    at kMeans (.../node_modules/talisman/clustering/k-means.js:243:21)
    at Object.<anonymous> (.../lev-kmeans.js:39:22)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

I tried looking into the code and the reason it fails is this piece of code (https://github.com/Yomguithereal/talisman/blob/master/src/clustering/k-means.js#L148).

for (let j = 0, m = this.dimensions; j < m; j++) {
     const d = this.distance(vector, this.centroids[j]);

     if (d < min) {
         min = d;
         minIndex = j;
     }
}

It tries to enter cosine.js functions using undefined as the second vector. For example for k equal 2 it should generate 2 centroids. It does. But this loop is trying to access centroids with very high indices like 181 (this.dimensions and thus m is equal 182).

It works fine when I tested it for simpler stuff using Euclidean distances.

To be complete I add the way I initialize the kMeans:

const clusters = kMeans({
    distance: cosine,
    k: 6,
}, vectors)

Where cosine is a const cosine = require('talisman/metrics/distance/cosine').

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 24 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Ok, the thing was that in the data you provided in the repo, the description key seems to always yield only three tokens.

So are we going in the good direction?