IIS Logs stops writing in cloud service

I was recently working on a case where the IIS logs stops writing on all Production boxes. There was 995GB on C:\ disk for logs, but logs writing starts only after some files were deleted. The Current IIS logs directory size was 2.91 GB (3,132,727,296 bytes).

So I looked at the diagnostic.wadcfg files.

<?xml version="1.0" encoding="utf-8"?>

<DiagnosticMonitorConfiguration configurationChangePollInterval="PT1M" overallQuotaInMB="4096" xmlns="https://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration">

  <DiagnosticInfrastructureLogs scheduledTransferPeriod="PT1M" />

  <Directories scheduledTransferPeriod="PT1M">

    <IISLogs container="wad-iis-logfiles" />

    <CrashDumps container="wad-crash-dumps" />

  </Directories>

  <Logs bufferQuotaInMB="1024" scheduledTransferPeriod="PT1M" />

  <PerformanceCounters bufferQuotaInMB="512" scheduledTransferPeriod="PT1M">

    <PerformanceCounterConfiguration counterSpecifier="\Memory\Available MBytes" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\ISAPI Extension Requests/sec" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Bytes Total/Sec" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\ASP.NET Applications(__Total__)\Requests/Sec" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\ASP.NET Applications(__Total__)\Errors Total/Sec" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\ASP.NET\Requests Queued" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\ASP.NET\Requests Rejected" sampleRate="PT3M" />

    <PerformanceCounterConfiguration counterSpecifier="\Processor(_Total)\% Processor Time" sampleRate="PT1S" />

  </PerformanceCounters>

  <WindowsEventLog bufferQuotaInMB="1024" scheduledTransferPeriod="PT1M" scheduledTransferLogLevelFilter="Error">

    <DataSource name="Application!*" />

    <DataSource name="System!*" />

  </WindowsEventLog>

</DiagnosticMonitorConfiguration>

 

I noticed that there is no bufferQuotaInMB mentioned for Directories and IIS logs.

 <Directories scheduledTransferPeriod="PT1M">

    <IISLogs container="wad-iis-logfiles" />

    <CrashDumps container="wad-crash-dumps" />

</Directories>

 

The attribute bufferQuotaInMB needs to be mentioned as per this article https://www.windowsazure.com/en-us/develop/net/common-tasks/diagnostics/ . Making this change led to the IIS logs getting moved to the Azure Storage BLOB location at scheduled intervals and making space for new IIS logs to be collected.

Since there was lot of disk space I further recommend the customer to increase the quota limits. The way it needs to be done is as below

a) Increase the Local Resource Size. If this is not done then irrespective of what value is allocated to OverallQuotaInMB the value it will get in 4GB.

 <LocalResources>

 <LocalStorage name="DiagnosticStore" cleanOnRoleRecycle="false"sizeInMB="8192" /> ==> Example of Setting 8GB DiagnosticStore

 </LocalResources>

b) Keep the “OverallQuotaInMB” at least 512 KB less than the value above. We need 512 KB for the GC to run and incase all the space is consumed the GC can’t run to move the logs.

c) The “BufferQuotaInMB” property sets the amount of local storage to allocate for a specified data buffer. You can specify the value of this property when configuring any of the diagnostic sources for a diagnostic monitor. By default, the BufferQuotaInMB property is set to 0 for each data buffer. Diagnostic data will be written to each data buffer until the OverallQuotaInMB limit is reached unless you set a value for this property. So the space left after the sum of the all the directory quota is deducted from OverallQuotaInMB is allocated to BufferQuota. 

d) Also change the “ScheduledTransferPeriodInMinutes” to a non 0 value. The minimum is 1 minute, if it’s set to 0 then the diagnostics will not be moved to storage and you might lose them if the VM is re-incarnated anytime due to a guest or host O.S update.

 

Angshuman Nayak, Cloud Integration Engineering