talisman: (kMeans, cosine) wrong number of centroids (?)
I’m trying to cluster sentences using TF-IDF, cosine similarity and k-means algorithm. I calculate TF-IDF using natural
library then I prepare vectors that look like:
[0, 0, 0, 0, 0, 1.24225242, 0, 0, 0, 0, 0]
where every value represents a word and the whole vector represents a sentence (in case of this example it would be a sentence with one word).
Then I try to use talisman
library to make clusters using created vectors and metrics/distance/cosine
. The problem is that I get:
TypeError: Cannot read property 'length' of undefined
at KMeans.cosine [as distance] (.../node_modules/talisman/metrics/distance/cosine.js:29:21)
at KMeans.iterate (.../node_modules/talisman/clustering/k-means.js:164:22)
at KMeans.run (.../node_modules/talisman/clustering/k-means.js:224:12)
at kMeans (.../node_modules/talisman/clustering/k-means.js:243:21)
at Object.<anonymous> (.../lev-kmeans.js:39:22)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
I tried looking into the code and the reason it fails is this piece of code (https://github.com/Yomguithereal/talisman/blob/master/src/clustering/k-means.js#L148).
for (let j = 0, m = this.dimensions; j < m; j++) {
const d = this.distance(vector, this.centroids[j]);
if (d < min) {
min = d;
minIndex = j;
}
}
It tries to enter cosine.js
functions using undefined
as the second vector. For example for k
equal 2 it should generate 2 centroids. It does. But this loop is trying to access centroids with very high indices like 181 (this.dimensions
and thus m
is equal 182).
It works fine when I tested it for simpler stuff using Euclidean distances.
To be complete I add the way I initialize the kMeans
:
const clusters = kMeans({
distance: cosine,
k: 6,
}, vectors)
Where cosine
is a const cosine = require('talisman/metrics/distance/cosine')
.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 24 (15 by maintainers)
Commits related to this issue
- Refs #103 - iterating on k rather than vector dimensions in k-means — committed to Yomguithereal/talisman by Yomguithereal 7 years ago
- Refs #103 - fixing & optimizing k-means clusterer — committed to Yomguithereal/talisman by Yomguithereal 7 years ago
Ok, the thing was that in the data you provided in the repo, the
description
key seems to always yield only three tokens.So are we going in the good direction?