Configure Cloud Discovery in Microsoft Defender for Cloud Apps

Completed

Cloud Discovery analyzes your traffic logs against the Microsoft Defender for Cloud Apps catalog of over 25,000 cloud apps. It then ranks and scores the apps based on more than 90 risk factors. This design provides you with ongoing visibility into cloud use, Shadow IT, and the risk Shadow IT poses into your organization.

Cloud Discovery reports

Organizations can generate the following types of reports in Cloud Discovery:

• Snapshot reports. Provides ad-hoc visibility on a set of traffic logs you manually upload from your firewalls and proxies.
• Continuous reports. Cloud Discovery analyzes all log files forwarded from your network using Microsoft Defender for Cloud Apps. The logs provide improved visibility over all data. They automatically identify anomalous use using either the Machine Learning anomaly detection engine or by using custom policies that you define. The system can create these reports in the following ways:
  • Microsoft Defender for Endpoint integration. Microsoft Defender for Cloud Apps natively integrates with Microsoft Defender for Endpoint. This integration:
    • Simplifies rollout of Cloud Discovery.
    • Extends Cloud Discovery capabilities beyond your corporate network.
    • Enables machine-based investigation.
  • Log collector. Log collectors enable you to easily automate log upload from your network. The log collector runs on your network and receives logs over Syslog or FTP.
  • Secure Web Gateway (SWG). When an organization integrates Microsoft Defender for Cloud Apps and Secure Web Gateway, the integration provides seamless deployment of Cloud Discovery, automatic blocking of unsanctioned apps, and risk assessment directly in the SWG's portal. If you work with both Microsoft Defender for Cloud Apps and one of the following SWGs, you can integrate the products to enhance your security Cloud Discovery experience:
    • Zscaler integration
    • iboss integration
    • Corrata integration
    • Menlo Security integration
• Reports created using the Cloud Discovery API. Use the Cloud Discovery API to automate traffic log upload and generate an automated Cloud Discovery report and risk assessment. You can also use the API to generate block scripts and streamline app controls directly to your network appliance.

Log process flow: From raw data to risk assessment

The process of generating a risk assessment consists of the following steps.

1. Upload. Microsoft Defender for Cloud Apps uploads the web traffic logs from your network to the portal.

2. Parse. Microsoft Defender for Cloud Apps parses and extracts traffic data from the traffic logs with a dedicated parser for each data source.

3. Analyze. Cloud Discovery analyzes traffic data against the Cloud App Catalog. This process enables it to identify more than 25,000 cloud apps and to assess their risk score. This analysis also identifies active users and IP addresses.

   Note

   Cloud Discovery analyzes and updates the extracted log file data four times a day.

4. Generate report. Cloud Discovery generates a risk assessment report based on the extracted log file data.

The process takes between a few minutes to several hours depending on the amount of data processed.

Using traffic logs for Cloud Discovery

Cloud Discovery uses the data in your traffic logs. The more detailed your log, the better visibility you get. Cloud Discovery requires web-traffic data with the following attributes:

• Date of the transaction
• Source IP
• Source user - highly recommended
• Destination IP address
• Destination URL recommended (URLs provide higher accuracy for cloud app detection than IP addresses)
• Total amount of data (data information is highly valuable)
• Amount of uploaded or downloaded data (provides insights about the usage patterns of the cloud apps)
• Action taken (allowed/blocked)

Cloud Discovery can only show and analyze the attributes included in traffic logs. For example, Cisco ASA Firewall standard log format doesn't include the following attributes: the number of uploaded bytes per transaction, Username, and Target URL (only target IP). Therefore, the data in an organization's traffic logs, which Cloud Discovery uses, doesn't include these attributes. This lack of data limits the organization's visibility into its cloud apps. For Cisco ASA firewalls, it's necessary to set the information level to 6.

To successfully generate a Cloud Discovery report, your traffic logs must meet the following conditions:

• Cloud Discovery supports the data source of your traffic logs. If Cloud Discovery doesn't support your data source, you can define a custom parser that matches your format. For more information on supported data sources, see Set up Cloud Discovery.
• Log format matches the expected standard format (format checked upon upload by the Log tool).
• Events aren't more than 90 days old.
• The log file is valid and includes outbound traffic information.

Create snapshot Cloud Discovery reports

It's important to upload a log manually and let Microsoft Defender for Cloud Apps parse it before trying to use the automatic log collector. For information on how the log collector works and the expected log format, see Using traffic logs for Cloud Discovery.

If you want to see an example of what a log file looks like, but you don't yet have a log file, then download a sample log file. Perform the following procedure to see what your log should look like.

You must complete the following steps to create a snapshot report:

1. Collect log files from your firewall and proxy, through which users in your organization access the Internet.

   Important

   Make sure to gather logs during times of peak traffic that are representative of all user activity in your organization.

2. In the Microsoft Defender portal, select Settings in the left-hand navigation pane.

3. On the Settings page, in the list of settings, select Cloud Apps.

4. On the Settings | Cloud apps page, under the Cloud Discovery section in the middle navigation pane, select Snapshot reports.

5. On the Snapshot reports page, select +Create snapshot report. This option initiates the Create new Cloud Discovery snapshot report wizard.

   

6. In the Create new Cloud Discovery snapshot report wizard, on the Overview page, select Next.

7. On the Report Details page, enter a Report Name and an optional Description.

   

8. Select the Source from which you want to upload the log files.

9. Verify your log format to ensure proper formatting according to the sample log you can download. Under Verify your log format, select View log format then select Download sample log. Compare your log with the sample provided to make sure it's compatible.

   

   Snapshots and automated upload support the FTP log format. In comparison, the only data source that supports the syslog format is automated upload. Downloading a sample log downloads a sample FTP log.

10. On the Upload traffic logs page, select the Browse button and then select the log files to upload. You can upload up to 20 files at once. The upload process also supports compressed and zipped files.

    

11. Select Upload logs.

12. After the upload finishes, a status message appears at the top-right corner of your screen indicating a successful upload.

13. After you upload your log files, it can take some time for Cloud Discovery to parse and analyze them. After processing of your log files completes, you should receive an email notifying you that Cloud Discovery completed its analysis.

14. A notification banner appears in the status bar at the top of the page. The banner updates you with the processing status of your log files.

    

15. After the traffic logs successfully upload, you should see a notification letting you know that the log file processing completed successfully. At this point, you can view the report on the Snapshot reports page.

Configure automatic log upload for continuous reports

Log collectors enable you to easily automate log upload from your network. The log collector runs on your network and receives logs over Syslog or FTP. It then automatically processes, compresses, and transmits each log to the portal. The log collector uploads FTP logs to Microsoft Defender for Cloud Apps after the file finished the FTP transfer to the log collector. For Syslog, the log collector writes the received logs to the disk. Then the collector uploads the file to Microsoft Defender for Cloud Apps when the file size is larger than 40 KB.

Once the upload process completes, Microsoft Defender for Cloud Apps moves the log to a backup directory. The backup directory stores the last 20 logs. When new logs arrive, the system deletes the old ones. Whenever the log collector disk space is full, the log collector drops new logs until it has more free disk space. You should receive a warning on the Log collectors tab of the Upload logs automatically settings when this scenario occurs.

Before setting up automatic log file collection, verify your log matches the expected log type. You want to make sure Defender for Cloud Apps can parse your specific file. For more information, see Using traffic logs for Cloud Discovery.

Note

Microsoft Defender for Cloud Apps provides support for forwarding logs in their original format from your SIEM server to the log collector. However, Microsoft recommends that you integrate the log collector directly with your firewall and/or proxy.

The log collector compresses data before uploading it. The outbound traffic on the log collector is 10% of the size of the traffic logs it receives. If the log collector encounters issues, it might not upload data. If Microsoft Defender for Cloud Apps doesn't receive data for 48 hours, it sends you an alert.

Knowledge check

Choose the best response for the following question.

Check your knowledge

1.

Cloud Discovery uses the event data in an organization's traffic logs to generate a Cloud Discovery report. What's the maximum age a traffic log event can be to appear on a Cloud Discovery report?

30 days old

60 days old

90 days old

You must answer all questions before checking your work.

Continue