5 ways to remove duplicate objects from an array based on a property

The other day I was working on a small side project where I had to retrieve a set of tweets and display them in a grid. In the hero section, I wanted to display the authors of the tweets so people can easily scan them.

But what happens when I share multiple tweets by the same person? I would end up showcasing the avatar of the person multiple times as well.

I wanted to find a way to remove the duplicate avatars and only display each other once, no matter how many tweets from that person. Let’s explore different technique to get this done.

Our test data

Here’s a simplified version of the data that I receive from my API call.

We would like to remove the duplicate authors and then loop through the data and display only the unique authors.

const data = [
  { tweet: { id: "101", author: { id: "1", name: "Michael" } } },
  { tweet: { id: "102", author: { id: "2", name: "Gob" } } },
  { tweet: { id: "103", author: { id: "1", name: "Michael" } } },
  { tweet: { id: "104", author: { id: "3", name: "Buster" } } },
  { tweet: { id: "105", author: { id: "2", name: "Gob" } } },
];

Our goal is to get to the following array which will allow us to simply loop through the array and display the authors.

const idealOutcome = [
  { tweet: { id: "101", author: { id: "1", name: "Michael" } } },
  { tweet: { id: "102", author: { id: "2", name: "Gob" } } },
  { tweet: { id: "104", author: { id: "3", name: "Buster" } } },
];

Technique #1: Use a reduce method

We can use the reduce method to go through every item and see if we already have an object added to the accumulator with the same author id as the current item.

const uniqueAuthors = data.reduce((accumulator, current) => {
  if (!accumulator.find((item) => item.tweet.author.id === current.tweet.author.id)) {
    accumulator.push(current);
  }
  return accumulator;
}, []);

✅
console.log(uniqueAuthors.map(item => item.tweet.author.id))
// ["1", "2", "3"]
console.log(idealOutcome.map(item => item.tweet.author.id))
// ["1", "2", "3"]

Technique #2: Use the Map object

Using the Map object is a more performant technique and takes less lines of code to write, but it could not preserve the order of the original array and it might not be as readable for other developers.

const uniqueAuthors = [...new Map(data.map(v => [v.tweet.author.id, v])).values()]

✅
console.log(uniqueAuthors.map(item => item.tweet.author.id))
// ["1", "2", "3"]
console.log(idealOutcome.map(item => item.tweet.author.id))
// ["1", "2", "3"]

Technique #3: Use lodash.uniqueWith()

If you are already using Lodash, uniqueWith() function might be a suitable and fast solution for you.

import _ from "lodash"

const uniqueAuthors = _.uniqWith(data, (arrVal, othVal) => arrVal.tweet.author.id === othVal.tweet.author.id)

✅
console.log(uniqueAuthors.map(item => item.tweet.author.id))
// ["1", "2", "3"]
console.log(idealOutcome.map(item => item.tweet.author.id))
// ["1", "2", "3"]

Technique #4: Filter + Set()

Another technique is to use the Set object and its add and has methods.

This is subjective opinion, but this is probably the most readable solution out of all other techniques in this article.

const seen = new Set();
const uniqueAuthors = data.filter(item => {
  const duplicate = seen.has(item.tweet.author.id);
  seen.add(item.tweet.author.id);
  return !duplicate;
});

✅
console.log(uniqueAuthors.map(item => item.tweet.author.id))
// ["1", "2", "3"]
console.log(idealOutcome.map(item => item.tweet.author.id))
// ["1", "2", "3"]

Technique #5: Filter + findIndex

A similar approach to Technique #4, but this time we are using the findIndex method. This is probably the least performant method out all since you are doing a loop inside a loop.

const uniqueAuthors = data.filter((value, index, self) =>
  index === self.findIndex((t) => (
    t.tweet.author.id === value.tweet.author.id
  ))
)

✅
console.log(uniqueAuthors.map(item => item.tweet.author.id))
// ["1", "2", "3"]
console.log(idealOutcome.map(item => item.tweet.author.id))
// ["1", "2", "3"]

Do you have any other techniques which you’d like to share? Drop a comment below and let us know!