fwrite issues with large data write

Question

_{Tuesday, September 30, 2008 3:00 PM}

I am using fwrite to store the array of structure with large size . As I am in 64 bit development when I pass the number of records as 64 bit value then fwrite hangs..and it could not handle large number of records, is there any solution to make disk I/O for large number of records in 64 bit

For example i am trying to write the memory size 11489112868 bytes in one shot using fwrite

as follows

records = 114891128668/ 512
Size of record = 512 byte

fwrite(buffer,sizeof(record), records, FILE*)

now this takes very long time and almost hangs..I wonder as this fwrite accepts 64 bit for sizeof and number of records argument then why it hangs IF i use 64 bit value for number of records

All replies (8)

_{Tuesday, September 30, 2008 5:19 PM ✅Answered}

Right , I agree and it takes more and more time and no response..so I think the only way is split the buffer and write into disk ??

---

_{Tuesday, September 30, 2008 3:18 PM}

Can you Show your code here .
Thanx

---

Rupesh Shukla

---

_{Tuesday, September 30, 2008 3:21 PM}

Out of curiosity, how much memory does your machine have?

---

_{Tuesday, September 30, 2008 4:04 PM}

15 GB RAM and I am deciding the record size by available physical memory 

MEMORYSTATUSEX Mem_Stat;
  Mem_Stat.dwLength = sizeof(MEMORYSTATUSEX);
  DWORDLONG Memory_To_Use = RAM_GB * 1024 * 1024 * 1024 - 1;  //Translate Gigabytes to Bytes.
  if(GlobalMemoryStatusEx(&Mem_Stat))
  {
    Memory_To_Use = Mem_Stat.ullAvailPhys ;    //Amount of available physical memory.
    cout << "Available physical memory is " << Memory_To_Use/1024 << "K\n";
  }
  Memory_To_Use    /= 4;                            //We'll use 3/4 of it.
  Memory_To_Use    *= 3;

  Quotes_In_Sorting_Buffer     = Memory_To_Use / sizeof(OwnStructure);

Sorting_Buffer = new OwnStructure[Quotes_In_Sorting_Buffer];

fwrite(Sorting_Buffer, sizeof(OwnStructure), Quotes_In_Sorting_Buffer, Work_File1);

if the quotes_In_sorting_Buffer * sizeof(OwnStructure) goes more than the value 2^32 or 4GB then it hangs

---

_{Tuesday, September 30, 2008 4:12 PM}

Trying out your new PC?  It is still the same slow hard drive.  Writing 11.5 Gigabytes to a disk takes a good while, especially since it can never fit in the lazy write-back cache.

---

Hans Passant.

---

_{Monday, November 30, 2009 5:02 AM | 1 vote}

I think nobody ever tested fwrite with data more than 4GB as Microsoft code loops forever.

MSVC 2008, 64-bit project:

    fwrite( p, sizeof(int), num, fout );

num is 1024*1024*1024
sizeof(int) is 4

fwrite locks the stream and calls
size_t __cdecl _fwrite_nolock

there is nice loop there, where (bufsize is 4096)

nbytes = ( bufsize ? (unsigned)(count - count % bufsize) : (unsigned)count );

count at this point is 4*1024*1024*1024

so
nbytes = (unsigned)(4*1024*1024*1024) =0
it tries to write 0 bytes, subtructs it from count (no change of course) so infinite tight loop.

if I'm trying to write not 4GB of data but say 5GB then

nbytes = ( bufsize ? (unsigned)(count - count % bufsize) : (unsigned)count );

nbytes now 1GB, it writes 1GB, then count is 5GB-1GB = 4GB, (unsigned)4GB is 0 - infinite loop.

So no matter what the size is if it's above 4GB (i've tried to write 10GB originally) it writes out whatever is above closest multiple of 4GB (in the case of 10GB - it writes what is above 8GB hence 2GB) then it gets count as something in multiple of 4GB units (say 8GB) and does

nbytes = (unsigned)(8*1024*1024*1024) =0

and tight loop forever.

I suspect that fread may have the same issue....

---

_{Monday, October 27, 2014 8:10 AM}

so, what's the solution for writing file larger than 4GB using fwrite

am using fwrite and fread because it's easy to choose file writing mode for opening the same file as flush(fresh) mode(re-write) by using "wb" or append mode by using "rb+", but larger file size is depth issue for me.

---

_{Wednesday, February 11, 2015 7:04 AM}

You can write your own function to copy/delete/move the extra large file. MS default commands for large file will freeze or choke the network. In my project, I move large media files in small chunks at a time. For example I use 10MB of sub chunks ( in each iteration) of the 23+ GB media files regularly. You can also control the speed of the file transfer by adjusting the sub chunk size. Use a batch command window to stack up the incoming commands.