A memory problem using the SPListItemCollection - How to avoid it ?

In the last few weeks my main core is tracking down memory leaks in client Sharepoint applications. This issue might not be happening to all of you who using this Object/Class. but it does might have something to do with you if you work on large lists or "future to be" large lists.

Ok, SPListItemCollection is a very strong class. Why ? Only with him you can get in flat way all the files you have in the doclib/list, regardless how deep they are located in the doclib ( in whatsoever folder in it ). I saw people that in order to get all the files which located in the sub folders in their doblibs, they were doing some serious digging ( including inefficient recursive iteration to get all those files ). But I'm sure you say - but if I need to to know in which subfolder this file was in ? Easy, you can track it down with the URL property.

So where the problem begins ? When you need just to do something very simple and very common - Automatically/pragmatically adding files\data into a list.
Let's say I want to add an item for my list, in order to do so I need to get the collection , right ? But what if my collection is huge ? above 2000 items in the list ( Above the recommendation in the whitepapers ) ?
Yes, now you start getting the problem. By calling this huge collection repeatedly Im using a lot of memory, till I might even get "Out of memory" exception.

Some code snippet to make sure you got the scenario -

 

            SPWeb web = site.OpenWeb();
            SPList list = web.Lists["Shared Documents"];
            //Getting all Items in the collection ,even though we dont really need it
            SPListItemCollection itemCol = list.Items;
            SPListItem listItem = itemCol.Add();
            //Working on the listItem we just added
            listItem["Field1"] = //adding metadata and so on
            listItem.Update();

So what we do just to workaround this scenario ? we get the collection from a certain view, which will be "thinner" version or in other words not as big as the default set of the collection. In fact, we made a special view only for that.

Short example to make sure you got the idea again -

 
            
            SPWeb web = site.OpenWeb();
            SPList list = web.Lists["Shared Documents"];
            //Getting only the view items - could be empty
            SPListItemCollection itemCol = list.GetItems(FakeView);
            SPListItem listItem = itemCol.Add();
            //Working on the listItem we just added
            listItem["Field1"] = //adding metadata again or whatsoever
            listItem.Update();

Apparently , This issue is very well known, it's much greater than I thought when I first saw this issue over one of my clients. Even in the MSDN they made a clearafaction about it.

Another disclaimer, one of my colleagues asked me right away, does it mean we added those items to that certain view ? Well, No. Since we still added the data to the list, the view might present those items if they fall in his view criteria. I personally made a "dummy/fake" view so it will always present nothing.

Enjoy.