UWP GridView slow down starting the second data binding with exact same data and ConditionalWeakTable doesn't go down and Memory keeps increasing.

Question

For UWP windows app,
I bind data to Gridview to show images.

<GridView>  
 <GridView.ItemTemplate>  
 <DataTemplate x:DataType="modelLocal:FileInfo">  
 <Grid Tag="{x:Bind Id, Mode=OneWay}">  
 <Grid>  
  <Image x:Phase="1" Source="{x:Bind Thumbnail, Mode=OneWay}"  
</DataTemplate>  
                            </GridView.ItemTemplate>  
                            <GridView.ItemsPanel>  
                                <ItemsPanelTemplate>  
                                    <ItemsWrapGrid Orientation="Horizontal"/>  
                                </ItemsPanelTemplate>  
                            </GridView.ItemsPanel>     

I bind new ObservableCollection<FileInfo>() to the GridView, and load 500 records.
When I start app and load first time, it is fast and smooth!
But, when I load second time and forth, it is slow and lagging.
I call UI thread to update loading count. First load show the counting so smooth. But starting second loading and forth, it's always lagging and skips and doesn't update UI well.

I bind the exact same data over and over for the first fast and slow second, third, ... tries.
On each loading, I clear the collection and call "gvFiles.ItemsSource = null;"

From my coding point of view, I have no idea why only first load is good.

I read "ListView and GridView UI optimization"
https://learn.microsoft.com/en-us/windows/uwp/debug-test-perf/optimize-gridview-and-listview
It talks about UI Virtualization. I assume GridView does't clean up things after first load. I couldn't find any clue from the object in runtime.
But with Diagnostic Tools, I noticed strange thing.

Bound objects are all cleaned and GC well, but "ConditionalWeakTable" object keep increase.

When I bind only 50 records after the first 500 records bound, my custom object bound reduced only 50 but "ConditionalWeakTable" doesn't go down.

I feel like GridView holds all previously bound elements somehow. Once 500 records are bound and rendered, it don't release the previously bound items even though small number of records are bound or even after all items (ItemsSource) are cleared.

This is so strange. I tested with DataGrid, it doesn't cause any kind of this issue. The performance is always same, the first and second loads.

How can I cleanup and initialize GridView properly to keep each data loading(binding) fast and smooth like the first loading?