Object and Memory Handling in Python

It's well known that Python (and other languages too) do some smart things when it comes to handling memory. For example, if you create a variable and then assign it to another variable the language generally only keeps one copy of the data. This can cause some interesting challenges...

For ...reasons... I needed to initialise a list (that's an array for all the non-Python people out there) with a bunch of static data. I didn't want to use that data - it would get overwritten. For those who really want to know, I was creating a ring buffer and statically assigning all the values in it so that there were no memory operations needed later.

Consider the following code fragment:

>>> simpleList = [0]*20
>>> simpleList
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Pretty easy and it works just the way you expect. If you change one of the list elements, it's all good.

>>> simpleList[5] = 5
>>> simpleList[11] = 1234
>>> simpleList
[0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 1234, 0, 0, 0, 0, 0, 0, 0, 0]

But of course things are not that simple. I didn't want a list of integers, I wanted a list of dictionaries. That's easy, right? (Formatting changed to make things a little easier to read.)

>>> complexList = [{'firstName':'nothing', 'lastName':'alsonothing'}]*20
>>> complexList
[{'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'}]

Looks fine but it doesn't do what you expect.

>>> complexList[2]['firstName'] = 'myname'
>>> complexList
[{'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'},
 {'firstName': 'myname', 'lastName': 'alsonothing'}, {'firstName': 'myname', 'lastName': 'alsonothing'}]

Well that's amazingly inconvenient because I don't want all the elements to be changed. Python is trying to be efficient so it takes the object and just puts a bunch of pointers into the list. Along the way though it has forgotten about that and assumes that when I update one of the list elements I want to update all of them. Not the case.

You can't fix this using copy() or something like that because it just creates a pointer to the copied object that is then spread throughout the list. What can we do then?

>>> workingList = []
>>> for _ in range(0, 20):
...     workingList.append({'firstName':'nothing', 'lastName':'alsonothing'})
...
>>> workingList
[{'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'}]

Looks the same but does it actually work? Yes it does. It seems like it shouldn't because we're appending the same object each time but it does.

>>> workingList[2]['firstName'] = 'myupdatedname'
>>> workingList
[{'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'myupdatedname', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'},
 {'firstName': 'nothing', 'lastName': 'alsonothing'}, {'firstName': 'nothing', 'lastName': 'alsonothing'}]

Stuff was learned. There endeth the lesson.