You can find several lists by each condition, then combine the results. If you want the intersection (not union), then try something like this:
Dim files As List(Of FileInfo)
files = (From p In FileIO.FileSystem.GetFiles(path, FileIO.SearchOption.SearchAllSubDirectories)
Select New FileInfo(p)).ToList
Dim results As IEnumerable(Of FileInfo) = files
If compareLength Then
Dim byLength = From fi In files
Group By fi.Length Into g = Group
Where g.Count() > 1
From i In g
Select i
results = byLength
End If
If compareName Then
Dim byName = From fi In files
Group By fi.Name Into g = Group
Where g.Count() > 1
From i In g
Select i
results = results.Intersect(byName)
End If
If compareCreationTime Then
Dim byCreationTime = From fi In files
Group By fi.CreationTime Into g = Group
Where g.Count() > 1
From i In g
Select i
results = results.Intersect(byCreationTime)
End If
Adjust the code related to listbox. Use a single loop for results, which consists of FileInfo.