ramda: uniq is slow

We had to replace:

var getIds = R.map(R.path(['id']));
var oneIds = getIds(listOne);
var twoIds = getIds(listTwo);
var ids = R.uniq(R.concat(oneIds, twoIds));

with:

var idsHash = {};

listOne.forEach(function(item) {
          idsHash[item.id] = 1;
});

listTwo.forEach(function(item) {
          idsHash[item.id] = 1;
});

var ids = Object.keys(idsHash);

The former for big amounts of data blocked node event loop for ridiculous amouts of time (something like 25minutes).

Did I misuse uniq?

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 34 (28 by maintainers)

Most upvoted comments

@Restuta: I would rather a new issue around fixing equals. Feel free to open one and point to this discussion.

CrossEye on Oct 12, 2017

@Restuta You can achieve similar times when using R.uniqWith((x, y) => x === y) instead.

ramda uniqWith: 6.982ms
4000
lodash: 4.592ms
4000

To emphasise the difference in functions, changing your randomNumbers to a constant but non-identical {x: 8} …

const randomNumbers = R.range(1, 4001).map(x => ({x: 8}))

… produces 1 unique element with R.uniq and 4000 unique elements for the other two:

ramda: 23.973ms
1
ramda uniqWith: 7.414ms
4000
lodash: 4.929ms
4000

If you happen to know something about the structure of the elements being compared, you can also use R.uniqBy such as R.uniqBy(x => x.x, randomNumbers), which should produce reasonable times too:

const randomNumbers = R.range(1, 4001)
  .map(x => ({x: Math.random()}))
  .concat(R.repeat({x: 8}, 100))
---
ramda uniqBy: 3.049ms
4001
lodash: 4.026ms
4001

const randomNumbers = R.range(1, 4001).map(x => ({x: 8}))
---
ramda uniqBy: 2.444ms
1
lodash: 4.133ms
4000

scott-christopher on Oct 5, 2017