Configure automatic log upload for continuous reports

Note

Microsoft Defender for Cloud Apps (previously known as Microsoft Cloud App Security) is now part of Microsoft 365 Defender. The Microsoft 365 Defender portal allows security admins to perform their security tasks in one location. This will simplify workflows, and add the functionality of the other Microsoft 365 Defender services. Microsoft 365 Defender will be the home for monitoring and managing security across your Microsoft identities, data, devices, apps, and infrastructure. For more information about these changes, see Microsoft Defender for Cloud Apps in Microsoft 365 Defender.

Log collectors enable you to easily automate log upload from your network. The log collector runs on your network and receives logs over Syslog or FTP. Each log is automatically processed, compressed, and transmitted to the portal. FTP logs are uploaded to Microsoft Defender for Cloud Apps after the file finished the FTP transfer to the Log Collector. For Syslog, the Log Collector writes the received logs to the disk. Then the collector uploads the file to Defender for Cloud Apps when the file size is larger than 40 KB.

After a log is uploaded to Defender for Cloud Apps, it's moved to a backup directory. The backup directory stores the last 20 logs. When new logs arrive, the old ones are deleted. Whenever the log collector disk space is full, the log collector drops new logs until it has more free disk space (this shouldn't happen if prerequisites are properly met). You'll receive a warning on the Log collectors tab of the Upload logs automatically settings when this happens.

Before setting up automatic log file collection, verify your log matches the expected log type. You want to make sure Defender for Cloud Apps can parse your specific file. For more information, see Using traffic logs for Cloud Discovery.

Note

  • Defender for Cloud Apps provides support for forwarding logs from your SIEM server to the Log Collector assuming the logs are being forwarded in their original format. However, it is highly recommended that you integrate the log collector directly with your firewall and/or proxy.
  • The log collector compresses data before it is uploaded. The outbound traffic on the log collector will be 10% of the size of the traffic logs it receives.
  • If the log collector encounters issues, you will receive an alert after data wasn't received for 48 hours.

Prerequisites

  • Disk space 250 GB
  • CPU cores: 2
  • CPU Architecture: Intel® 64 and AMD 64
  • RAM: 4 GB
  • Set your firewall as described in Network requirements

Note

If you have an existing log collector and want to remove it before deploying it again, or if you simply want to remove it, run the following commands:

docker stop <collector_name>

docker rm <collector_name>

Log collector performance

The Log collector can successfully handle log capacity of up to 50 GB per hour. The main bottlenecks in the log collection process are:

  • Network bandwidth - Your network bandwidth determines the log upload speed.
  • I/O performance of the virtual machine - Determines the speed at which logs are written to the log collector's disk. The log collector has a built-in safety mechanism that monitors the rate at which logs arrive and compares it to the upload rate. In cases of congestion, the log collector starts to drop log files. If your setup typically exceeds 50 GB per hour, it's recommended that you split the traffic between multiple log collectors.

Deployment modes

The Log Collector supports the Container deployment mode. It runs as a Docker image on Windows, Ubuntu on-premises, Ubuntu in Azure, RHEL on-premises or CentOS.

Next steps