Remove Duplicates From List in Python

John on November 03, 2021

In this tutorial, we will go through the different ways of removing duplicate element values from a list in Python.

Remove Duplicate List Elements in Python with a for Loop

The most efficient way to remove duplicate items from a list is to use a for loop. Loop through the original list and on each iteration check if the current element is in the new list and add it if that is the case.

items = [1, 3, 3, 4, 5, 1, 6]

unique_items = []

for i in items:
   if i not in unique_items:
       unique_items.append(i)
       
print(unique_items)

[1, 3, 4, 5, 6]

Remove Duplicates in a Python List with the set() Function

A set in Python is a collection of unique values. If we convert a list to a set all the duplicates will be removed in the process. This is helpful if you want to remove a duplicate element in the shortest amount of code possible.

items = [3, 3, 4, 5, 1, 6]

unique_items = set(items)

print(unique_items)

{1, 3, 4, 5, 6}

If the unique items need to be of the data type list, convert it back to a list after using the Python set() function like this:

items = [3, 3, 4, 5, 1, 6]

unique_items = list(set(items))

print(unique_items)

[1, 3, 4, 5, 6]

The only issue with using set() is it does not preserve the order of the original list.

Remove Duplicates & Keep Order in List with OrderedDict

In the example below we will convert a list to a unique list and maintain its original order in one line using the OrderedDict function from the collections Python package.

from collections import OrderedDict

items = [3, 3, 4, 5, 1, 6]

unique_items = list(OrderedDict.fromkeys(items))

print(items)

[3, 3, 4, 5, 1, 6]