{"id":2980748,"date":"2023-11-20T10:00:30","date_gmt":"2023-11-20T15:00:30","guid":{"rendered":"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/plato-data\/why-you-should-not-overuse-list-comprehensions-in-python-kdnuggets\/"},"modified":"2023-11-20T10:00:30","modified_gmt":"2023-11-20T15:00:30","slug":"why-you-should-not-overuse-list-comprehensions-in-python-kdnuggets","status":"publish","type":"station","link":"https:\/\/platodata.io\/plato-data\/why-you-should-not-overuse-list-comprehensions-in-python-kdnuggets\/","title":{"rendered":"Why You Should Not Overuse List Comprehensions in Python – KDnuggets"},"content":{"rendered":"
In Python, list comprehensions provide a concise syntax to create new lists from existing lists and other iterables. However, once you get used to list comprehensions you may be tempted to use them even when you shouldn’t.\u00a0<\/p>\n Remember, your goal is to write simple and maintainable code; not complex code. It\u2019s often helpful to revisit the Zen of Python<\/a>, a set of aphorisms for writing clean and elegant Python, especially the following:<\/p>\n In this tutorial, we\u2019ll code three examples\u2014each more complex than the previous one\u2014where list comprehensions make the code super difficult to maintain. We\u2019ll then try to write a more maintainable version of the same.\u00a0<\/p>\n So let\u2019s start coding!<\/p>\n Let’s start by reviewing list comprehensions in Python. Suppose you have an existing iterable such as a list or a string. And you\u2019d like to create a new list from it. You can loop through the iterable, process each item, and append the output to a new list like so:<\/p>\n \u00a0 <\/p>\n But less comprehensions provide a concise one-line alternative to do the same:<\/p>\n \u00a0 <\/p>\n In addition, you can also add filtering conditions.<\/p>\n The following snippet:<\/p>\n \u00a0 <\/p>\n Can be replaced by this list comprehension:<\/p>\n \u00a0 <\/p>\n So list comprehensions help you write Pythonic code\u2014often make your code cleaner by reducing visual noise.\u00a0<\/p>\n Now let’s take three examples to understand why you shouldn’t be using list comprehensions for tasks that require super complex expressions. Because in such cases, list comprehensions\u2014instead of making your code elegant\u2014make your code difficult to read and maintain.<\/p>\n Problem<\/strong>: Given a number You can break down this problem into two key ideas:<\/p>\n The list comprehension expression to do this is as shown:<\/p>\n \u00a0 <\/p>\n And here\u2019s the output:<\/p>\n \u00a0 <\/p>\n At first glance, it is difficult to see what is going on\u2026Let\u2019s make it better.<\/p>\n Perhaps, better?<\/p>\n \u00a0 <\/p>\n Easier to read, certainly. Now let\u2019s write a truly <\/i>better version.<\/p>\n Though a list comprehension is actually a good idea to solve this problem, the logic to check for primes in the list comprehension is making it noisy.<\/p>\n So let’s write a more maintainable version that moves the logic for checking if a number is prime to a separate function \u00a0 <\/p>\n \u00a0 <\/p>\n Is the better version good enough? <\/strong>This makes the comprehension expression much easier to understand. It’s now clear that the expression collects all numbers up to Problem<\/strong>: Given a matrix, find the following:<\/p>\n \u00a0<\/p>\n To flatten the matrix and collect the list of all prime numbers, we can\u00a0 use a logic similar to the previous example.\u00a0<\/p>\n However, to find the indices, we have another complex list comprehension expression (I\u2019ve formatted the code such that it is easy to read).\u00a0<\/p>\n You can combine checking for primes and getting their indices in a single comprehension. But that will not make things any simpler.\u00a0<\/p>\n Here\u2019s the code:<\/p>\n \u00a0 <\/p>\n And the corresponding output:<\/p>\n \u00a0 <\/p>\n So what is a better version?\u00a0<\/p>\n Now for the better version, we can define a series of functions to separate out concerns. So that if there\u2019s a problem, you know which function to go back to and fix the logic.<\/p>\n \u00a0 <\/p>\n This code also gives the same output as before.<\/p>\n \u00a0 <\/p>\n Is the better version good enough?<\/strong> While this works for a small matrix such as the one in this example, returning a static list is generally not recommended. And for generalizing to larger dimensions, you can use generators<\/a> instead.<\/p>\n Problem<\/strong>: Parse a given nested JSON string based on conditions and get a list of required values.<\/p>\n Parsing nested JSON strings is challenging because you have to account for the different levels of nesting, the dynamic nature of the JSON response, and diverse data types in your parsing logic.<\/p>\n Let’s take an example of parsing a given JSON string based on conditions to get a list of all values that are:<\/p>\n You can load a JSON string into a Python dictionary using the The list comprehension uses nested loops to iterate over the nested dictionary. For each value, it constructs a list based on the following conditions:<\/p>\n Have a look at the code snippet below:<\/p>\n \u00a0 <\/p>\n Here\u2019s the output:<\/p>\n \u00a0 <\/p>\n As seen, the list comprehension is very difficult to wrap your head around.\u00a0<\/p>\n Please do yourself and others on the team a favor by never writing such code.<\/i><\/p>\n I think the following snippet using nested for loops and if-elif ladder is better. Because it\u2019s easier to understand what\u2019s going on.\u00a0<\/p>\n \u00a0 <\/p>\n This gives the expected output, too:<\/p>\n \u00a0 <\/p>\n Is the better version good enough?<\/strong> Well, not really.\u00a0<\/p>\n Because if-elif ladders are often considered a code smell. You may repeat logic across branches and adding more conditions will only make the code more difficult to maintain.\u00a0<\/p>\n But for this example, the if-elif ladders and nested loops the version is easier to understand than the comprehension expression, though.<\/p>\n The examples we\u2019ve coded thus far should give you an idea of how overusing a Pythonic feature such as list comprehension can often become too much of a good thing. This is true not just for list comprehensions (they\u2019re the most frequently used, though) but also for dictionary and set comprehensions.<\/p>\n You should always write code that\u2019s easy to understand and maintain. So try to keep things simple even if it means not<\/strong> using some Pythonic features. Keep coding! Bala Priya C<\/a><\/strong><\/a><\/b> is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she’s working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.<\/p>\n
Image by Author<\/span>
\u00a0 <\/p>\n\n
new_list = []\nfor item in iterable:\n new_list.append(output)<\/code><\/pre>\n<\/div>\n
new_list = [output for item in iterable]<\/code><\/pre>\n<\/div>\n
new_list = []\nfor item in iterable:\n if condition:\n new_list.append(output)<\/code><\/pre>\n<\/div>\n
new_list = [output for item in iterable if condition]<\/code><\/pre>\n<\/div>\n
upper_limit<\/code>, generate a list of all the prime numbers up to that number.<\/p>\n
\n
import math\n\nupper_limit = 50 \n\nprimes = [x for x in range(2, upper_limit + 1) if x > 1 and all(x % i != 0 for i in range(2, int(math.sqrt(x)) + 1))]\n\nprint(primes)<\/code><\/pre>\n<\/div>\n
Output >>>\n[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]<\/code><\/pre>\n<\/div>\n
import math\n\nupper_limit = 50 \n\nprimes = [\n\tx\n\tfor x in range(2, upper_limit + 1)\n\tif x > 1 and all(x % i != 0 for i in range(2, int(math.sqrt(x)) + 1))\n]\n\nprint(primes)<\/code><\/pre>\n<\/div>\n
A Better Version<\/h2>\n
is_prime()<\/code>. And call the function
is_prime()<\/code> in the comprehension expression:<\/p>\n
import math\n\ndef is_prime(num):\n return num > 1 and all(num % i != 0 for i in range(2, int(math.sqrt(num)) + 1))\n\nupper_limit = 50 \n\nprimes = [\n\tx\n\tfor x in range(2, upper_limit + 1)\n\tif is_prime(x)\n]\n\nprint(primes)<\/code><\/pre>\n<\/div>\n
Output >>>\n[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]<\/code><\/pre>\n<\/div>\n
upper_limit<\/code> that are prime (where
is_prime()<\/code> returns True).<\/p>\n
\n
Image by Author<\/span>
\u00a0 <\/p>\nimport math\nfrom pprint import pprint\n\nmy_matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]\n\ndef is_prime(num):\n return num > 1 and all(num % i != 0 for i in range(2, int(math.sqrt(num)) + 1))\n\n\n# Flatten the matrix and filter to contain only prime numbers\nprimes = [\n\tx\n\tfor row in my_matrix\n\tfor x in row\n\tif is_prime(x)\n]\n\n# Find indices of prime numbers in the original matrix\nprime_indices = [\n\t(i, j)\n\tfor i, row in enumerate(my_matrix)\n\tfor j, x in enumerate(row)\n\tif x in primes\n]\n\n# Calculate the sum of prime numbers\nsum_of_primes = sum(primes)\n\n# Sort the prime numbers in descending order\nsorted_primes = sorted(primes, reverse=True)\n\n# Create a dictionary with the results\nresult = {\n\t\"primes\": primes,\n\t\"prime_indices\": prime_indices,\n\t\"sum_of_primes\": sum_of_primes,\n\t\"sorted_primes\": sorted_primes\n}\n\npprint(result)<\/code><\/pre>\n<\/div>\n
Output >>>\n\n{'primes': [2, 3, 5, 7],\n 'prime_indices': [(0, 1), (0, 2), (1, 1), (2, 0)],\n 'sum_of_primes': 17,\n 'sorted_primes': [7, 5, 3, 2]}<\/code><\/pre>\n<\/div>\n
A Better Version<\/h2>\n
import math\nfrom pprint import pprint\n\ndef is_prime(num):\n return num > 1 and all(n % i != 0 for i in range(2, int(math.sqrt(num)) + 1))\n\ndef flatten_matrix(matrix):\n flattened_matrix = []\n for row in matrix:\n for x in row:\n if is_prime(x):\n flattened_matrix.append(x)\n return flattened_matrix\n\ndef find_prime_indices(matrix, flattened_matrix):\n prime_indices = []\n for i, row in enumerate(matrix):\n for j, x in enumerate(row):\n if x in flattened_matrix:\n prime_indices.append((i, j))\n return prime_indices\n\ndef calculate_sum_of_primes(flattened_matrix):\n return sum(flattened_matrix)\n\ndef sort_primes(flattened_matrix):\n return sorted(flattened_matrix, reverse=True)\n\nmy_matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]\n\nprimes = flatten_matrix(my_matrix)\nprime_indices = find_prime_indices(my_matrix, primes)\nsum_of_primes = calculate_sum_of_primes(primes)\nsorted_primes = sort_primes(primes)\n\nresult = {\n\t\"primes\": primes,\n\t\"prime_indices\": prime_indices,\n\t\"sum_of_primes\": sum_of_primes,\n\t\"sorted_primes\": sorted_primes\n}\n\npprint(result)<\/code><\/pre>\n<\/div>\n
Output >>>\n\n{'primes': [2, 3, 5, 7],\n 'prime_indices': [(0, 1), (0, 2), (1, 1), (2, 0)],\n 'sum_of_primes': 17,\n 'sorted_primes': [7, 5, 3, 2]}<\/code><\/pre>\n<\/div>\n
\n
loads<\/code> function from the built-in json module<\/a>. So we\u2019ll have a nested dictionary over which we have a list comprehension.<\/p>\n
\n
[inner_item]<\/code>.\n<\/li>\n
[inner_item['sub_key']]<\/code>.\n<\/li>\n
[inner_item]<\/code>.\n<\/li>\n
list(inner_item.values())<\/code>.\n<\/li>\n<\/ul>\n
import json\n\njson_string = '{\"key1\": {\"inner_key1\": [1, 2, 3], \"inner_key2\": {\"sub_key\": \"value\"}}, \"key2\": {\"inner_key3\": \"text\"}}'\n\n# Parse the JSON string into a Python dictionary\ndata = json.loads(json_string)\n\n\nflattened_data = [\n\tvalue\n\tif isinstance(value, (int, str))\n\telse value\n\tif isinstance(value, list)\n\telse list(value)\n\tfor inner_dict in data.values()\n\tfor key, inner_item in inner_dict.items()\n\tfor value in (\n \t[inner_item]\n \tif not isinstance(inner_item, dict) and key.startswith(\"inner_key\")\n \telse [inner_item[\"sub_key\"]]\n \tif isinstance(inner_item, dict) and \"sub_key\" in inner_item\n \telse [inner_item]\n \tif isinstance(inner_item, (int, str))\n \telse list(inner_item.values())\n\t)\n]\n\nprint(f\"Values: {flattened_data}\")<\/code><\/pre>\n<\/div>\n
Output >>>\nValues: [[1, 2, 3], 'value', 'text']<\/code><\/pre>\n<\/div>\n
A Better Version<\/h2>\n
flattened_data = []\n\nfor inner_dict in data.values():\n for key, inner_item in inner_dict.items():\n if not isinstance(inner_item, dict) and key.startswith(\"inner_key\"):\n flattened_data.append(inner_item)\n elif isinstance(inner_item, dict) and \"sub_key\" in inner_item:\n flattened_data.append(inner_item[\"sub_key\"])\n elif isinstance(inner_item, (int, str)):\n flattened_data.append(inner_item)\n elif isinstance(inner_item, list):\n flattened_data.extend(inner_item)\n elif isinstance(inner_item, dict):\n flattened_data.extend(inner_item.values())\n\nprint(f\"Values: {flattened_data}\")<\/code><\/pre>\n<\/div>\n
Output >>>\nValues: [[1, 2, 3], 'value', 'text']<\/code><\/pre>\n<\/div>\n
\u00a0
\u00a0<\/p>\n