Categories
arrays duplicates javascript object

How to remove all duplicates from an array of objects?

760

I have an object that contains an array of objects.

obj = {};

obj.arr = new Array();

obj.arr.push({place:"here",name:"stuff"});
obj.arr.push({place:"there",name:"morestuff"});
obj.arr.push({place:"there",name:"morestuff"});

I’m wondering what is the best method to remove duplicate objects from an array. So for example, obj.arr would become…

{place:"here",name:"stuff"},
{place:"there",name:"morestuff"}

4

  • Do you mean how do you stop a hashtable/object with all the same parameters being added to an array?

    Feb 8, 2010 at 0:46

  • 8

    Mathew -> If it is simpler to prevent a duplicate object from being added to the array in the first place, instead of filtering it out later, yes, that would be fine too.

    – Travis

    Feb 8, 2010 at 1:01

  • Suuuper long answers and yet MDN has possibly the shortest: arrayWithNoDuplicates = Array.from(new Set(myArray))

    – tonkatata

    Dec 6, 2021 at 21:47

  • 4

    @tonkatata This doesn’t work with array of objects.

    Dec 14, 2021 at 7:50

197

A primitive method would be:

const obj = {};

for (let i = 0, len = things.thing.length; i < len; i++) {
  obj[things.thing[i]['place']] = things.thing[i];
}

things.thing = new Array();

 for (const key in obj) { 
   things.thing.push(obj[key]);
}

6

  • 73

    You should never user the length in the for loop, because it will slow everything down calculating it on every iteration. Assign it to a variable outside the loop and pass the variable instead of the things.thing.length.

    – Nosebleed

    Aug 26, 2014 at 12:56


  • 16

    @aefxx I do not quite understand this function, how do you handle the situation that the “place” is same but name is different, should that be consider dup or not?

    – Kuan

    Jun 23, 2015 at 21:48

  • 2

    Though this works, it does not take care of a sorted array since fetching keys is never order guaranteed. So, you end up sorting it again. Now, suppose the array was not sorted but yet its order is important, there is no way you can make sure that order stays intact

    Apr 17, 2019 at 6:31

  • 3

    @DeepakGM You’re absolutely right. The answer won’t (necessarily) preserve a given order. If that is a requirement, one should look for another solution.

    – aefxx

    Apr 17, 2019 at 17:03

  • How could I modify the above to remove objects from an array that contain X as well as de-duped?

    – Ryan H

    Feb 9, 2020 at 12:36

422

One liners with filter ( Preserves order )

Find unique id‘s in an array.

arr.filter((v,i,a)=>a.findIndex(v2=>(v2.id===v.id))===i)

If the order is not important, map solutions will be faster: Solution with map


Unique by multiple properties ( place and name )

arr.filter((v,i,a)=>a.findIndex(v2=>['place','name'].every(k=>v2[k] ===v[k]))===i)

Unique by all properties (This will be slow for large arrays)

arr.filter((v,i,a)=>a.findIndex(v2=>(JSON.stringify(v2) === JSON.stringify(v)))===i)

Keep the last occurrence by replacing findIndex with findLastIndex.

arr.filter((v,i,a)=>a.findLastIndex(v2=>(v2.place === v.place))===i)

9

  • 44

    v,i,a == value, index, array

    – James B

    Oct 9, 2020 at 18:23


  • This worked great to find if the key,value pairs in my vue modal had duplicates. +1

    – Arriel

    Feb 1, 2021 at 22:23

  • 5

    arr.filter((v,i,a)=>a.findIndex(t=>(JSON.stringify(t) === JSON.stringify(v)))===i) this will not work if the keys are not in the same order

    Mar 19, 2021 at 14:11

  • what t on findIndex stands for?

    Jun 10, 2021 at 3:10

  • 1

    simply BEAUTIFUL

    Nov 30, 2021 at 15:15

232

Using ES6+ in a single line you can get a unique list of objects by key:

const unique = [...new Map(arr.map((item, key) => [item[key], item])).values()]

It can be put into a function:

function getUniqueListBy(arr, key) {
    return [...new Map(arr.map(item => [item[key], item])).values()]
}

Here is a working example:

const arr = [
    {place: "here",  name: "x", other: "other stuff1" },
    {place: "there", name: "x", other: "other stuff2" },
    {place: "here",  name: "y", other: "other stuff4" },
    {place: "here",  name: "z", other: "other stuff5" }
]

function getUniqueListBy(arr, key) {
    return [...new Map(arr.map(item => [item[key], item])).values()]
}

const arr1 = getUniqueListBy(arr, 'place')

console.log("Unique by place")
console.log(JSON.stringify(arr1))

console.log("\nUnique by name")
const arr2 = getUniqueListBy(arr, 'name')

console.log(JSON.stringify(arr2))

How does it work

First the array is remapped in a way that it can be used as an input for a Map.

arr.map(item => [item[key], item]);

which means each item of the array will be transformed in another array with 2 elements; the selected key as first element and the entire initial item as second element, this is called an entry (ex. array entries, map entries). And here is the official doc with an example showing how to add array entries in Map constructor.

Example when key is place:

[["here", {place: "here",  name: "x", other: "other stuff1" }], ...]

Secondly, we pass this modified array to the Map constructor and here is the magic happening. Map will eliminate the duplicate keys values, keeping only last inserted value of the same key.
Note: Map keeps the order of insertion. (check difference between Map and object)

new Map(entry array just mapped above)

Third we use the map values to retrieve the original items, but this time without duplicates.

new Map(mappedArr).values()

And last one is to add those values into a fresh new array so that it can look as the initial structure and return that:

return […new Map(mappedArr).values()]

8

  • This does not answer the original question as this is searches for an id. The question needs the entire object to be unique across all fields such as place and name

    Dec 10, 2019 at 18:54

  • Your ES6 function seems very concise and practical. Can you explain it a bit more? What is happening exactly? Are first or last duplicates removed? Or is it random, which duplicate gets removed? That would be helpful, thanks.

    Mar 25, 2020 at 17:15

  • As far as i can tell, a Map with the property value as key is created. But it is not 100% how or if the order of the array is preserved.

    Mar 25, 2020 at 18:16

  • 1

    Hi @DavidSchumann, I will update the answer and will explain how it works. But for short answer the order is preserved and the first one are removed… Just think about how it is inserted in the map… it checks if the key already exists it will update it, therfore the last one will remain

    – V. Sambor

    Mar 25, 2020 at 18:21


  • There is an error in case of null items or some items without the called key, any fix for this?

    Jun 4, 2021 at 9:17