Import/Export Service Log File Format

 

When the Microsoft Azure Import/Export service performs an action on a drive as part of an import job or an export job, logs are written to block blobs in the storage account associated with that job.

There are two logs that may be written by the Import/Export service:

• The error log is always generated in the event of an error.

• The verbose log is not enabled by default, but may be enabled by setting the EnableVerboseLog property on a Put Job or Update Job Properties operation.

Log File Location

The logs are written to block blobs in the container or virtual directory specified by the ImportExportStatesPath setting, which you can set on a Put Job operation. The location to which the logs are written depends on how authentication is specified for the job, together with the value specified for ImportExportStatesPath. Authentication for the job may be specified via a storage account key, or a container SAS (shared access signature).

The name of the container or virtual directory may either be the default name of waimportexport, or another container or virtual directory name that you specify.

The table below shows the possible options:

Authentication Method	Value of ImportExportStatesPathElement	Location of Log Blobs
Storage account key	Default value	A container named waimportexport, which is the default container. For example: https://myaccount.blob.core.windows.net/waimportexport
Storage account key	User-specified value	A container named by the user. For example: https://myaccount.blob.core.windows.net/mylogcontainer
Container SAS	Default value	A virtual directory named waimportexport, which is the default name, beneath the container specified in the SAS. For example, if the SAS specified for the job is https://myaccount.blob.core.windows.net/mylogcontainer?sv=2012-02-12&se=2015-05-22T06%3A54%3A55Z&sr=c&sp=wl&sig=sigvalue, then the log location would be https://myaccount.blob.core.windows.net/mylogcontainer/waimportexport
Container SAS	User-specified value	A virtual directory named by the user, beneath the container specified in the SAS. For example, if the SAS specified for the job is https://myaccount.blob.core.windows.net/mylogcontainer?sv=2012-02-12&se=2015-05-22T06%3A54%3A55Z&sr=c&sp=wl&sig=sigvalue, and the specified virtual directory is named mylogblobs, then the log location would be https://myaccount.blob.core.windows.net/mylogcontainer/waimportexport/mylogblobs.

You can retrieve the URl for the error and verbose logs by calling the Get Job operation. The logs are available after processing of the drive is complete.

Log File Format

The format for both logs is the same: a blob containing XML descriptions of the events that occurred while copying blobs between the hard drive and the customer's account.

The verbose log contains complete information about the status of the copy operation for every blob (for an import job) or file (for an export job), whereas the error log contains only the information for blobs or files that encountered errors during the import or export job.

The verbose log format is shown below. The error log has the same structure, but filters out successful operations.

<DriveLog Version="2014-11-01">
    <DriveId>drive-id</DriveId>
    [<Blob Status="blob-status">
       <BlobPath>blob-path</BlobPath>
       <FilePath>file-path</FilePath>
       [<Snapshot>snapshot</Snapshot>]
       <Length>length</Length>
       [<LastModified>last-modified</LastModified>]
       [<ImportDisposition Status="import-disposition-status">import-disposition</ImportDisposition>]
       [page-range-list-or-block-list]
       [metadata-status]
       [properties-status]
    </Blob>]
    [<Blob>
    . . .
    </Blob>]
    <Status>drive-status</Status>
</DriveLog>

page-range-list-or-block-list ::= 
  page-range-list | block-list

page-range-list ::= 
<PageRangeList>
      [<PageRange Offset="page-range-offset" Length="page-range-length" 
       [Hash="md5-hash"] Status="page-range-status"/>]
      [<PageRange Offset="page-range-offset" Length="page-range-length" 
       [Hash="md5-hash"] Status="page-range-status"/>]
</PageRangeList>

block-list ::=
<BlockList>
      [<Block Offset="block-offset" Length="block-length" [Id="block-id"]
       [Hash="md5-hash"] Status="block-status"/>]
      [<Block Offset="block-offset" Length="block-length" [Id="block-id"] 
       [Hash="md5-hash"] Status="block-status"/>]
</BlockList>

metadata-status ::=
<Metadata Status="metadata-status">
   [<GlobalPath Hash="md5-hash">global-metadata-file-path</GlobalPath>]
   [<Path Hash="md5-hash">metadata-file-path</Path>]
</Metadata>

properties-status ::=
<Properties Status="properties-status">
   [<GlobalPath Hash="md5-hash">global-properties-file-path</GlobalPath>]
   [<Path Hash="md5-hash">properties-file-path</Path>]
</Properties>

The following table describes the elements of the log file.

XML Element	Type	Description
DriveLog	XML Element	Represents a drive log.
Version	Attribute, String	The version of the log format.
DriveId	String	The drive's hardware serial number.
Status	String	Status of the drive processing. See the Drive Status Codes table below for more information.
Blob	Nested XML element	Represents a blob.
Blob/BlobPath	String	The URI of the blob.
Blob/FilePath	String	The relative path to the file on the drive.
Blob/Snapshot	DateTime	The snapshot version of the blob, for an export job only.
Blob/Length	Integer	The total length of the blob in bytes.
Blob/LastModified	DateTime	The date/time that the blob was last modified, for an export job only.
Blob/ImportDisposition	String	The import disposition of the blob, for an import job only.
Blob/ImportDisposition/@Status	Attribute, String	The status of the import disposition.
PageRangeList	Nested XML element	Represents a list of page ranges for a page blob.
PageRange	XML element	Represents a page range.
PageRange/@Offset	Attribute, Integer	Starting offset of the page range in the blob.
PageRange/@Length	Attribute, Integer	Length in bytes of the page range.
PageRange/@Hash	Attribute, String	Base16-encoded MD5 hash of the page range.
PageRange/@Status	Attribute, String	Status of processing the page range.
BlockList	Nested XML element	Represents a list of blocks for a block blob.
Block	XML element	Represents a block.
Block/@Offset	Attribute, Integer	Starting offset of the block in the blob.
Block/@Length	Attribute, Integer	Length in bytes of the block.
Block/@Id	Attribute, String	The block ID.
Block/@Hash	Attribute, String	Base16-encoded MD5 hash of the block.
Block/@Status	Attribute, String	Status of processing the block.
Metadata	Nested XML element	Represents the blob's metadata.
Metadata/@Status	Attribute, String	Status of processing of the blob metadata.
Metadata/GlobalPath	String	Relative path to the global metadata file.
Metadata/GlobalPath/@Hash	Attribute, String	Base16-encoded MD5 hash of the global metadata file.
Metadata/Path	String	Relative path to the metadata file.
Metadata/Path/@Hash	Attribute, String	Base16-encoded MD5 hash of the metadata file.
Properties	Nested XML element	Represents the blob properties.
Properties/@Status	Attribute, String	Status of processing the blob properties, e.g. file not found, completed.
Properties/GlobalPath	String	Relative path to the global properties file.
Properties/GlobalPath/@Hash	Attribute, String	Base16-encoded MD5 hash of the global properties file.
Properties/Path	String	Relative path to the properties file.
Properties/Path/@Hash	Attribute, String	Base16-encoded MD5 hash of the properties file.
Blob/Status	String	Status of processing the blob.

Drive Status Codes

The following table lists the status codes for processing a drive.

Status code	Description
Completed	The drive has finished processing without any errors.
CompletedWithWarnings	The drive has finished processing with warnings in one or more blobs per the import dispositions specified for the blobs.
CompletedWithErrors	The drive has finished with errors in one or more blobs or chunks.
DiskNotFound	No disk is found on the drive.
VolumeNotNtfs	The first data volume on the disk is not in NTFS format.
DiskOperationFailed	An unknown failure occurred when performing operations on the drive.
BitLockerVolumeNotFound	No BitLocker encryptable volume is found.
BitLockerNotActivated	BitLocker is not enabled on the volume.
BitLockerProtectorNotFound	The numerical password key protector does not exist on the volume.
BitLockerKeyInvalid	The numerical password provided cannot unlock the volume.
BitLockerUnlockVolumeFailed	Unknown failure has happened when trying to unlock the volume.
BitLockerFailed	An unknown failure occurred while performing BitLocker operations.
ManifestNameInvalid	The manifest file name is invalid.
ManifestNameTooLong	The manifest file name is too long.
ManifestNotFound	The manifest file is not found.
ManifestAccessDenied	Access to the manifest file is denied.
ManifestCorrupted	The manifest file is corrupted (the content does not match its hash).
ManifestFormatInvalid	The manifest content does not conform to the required format.
ManifestDriveIdMismatch	The drive ID in the manifest file does not match the one read from the drive.
ReadManifestFailed	A disk I/O failure occurred while reading from the manifest.
BlobListFormatInvalid	The export blob list blob does not conform to the required format.
BlobRequestForbidden	Access to the blobs in the storage account is forbidden. This might be due to invalid storage account key or container SAS.
InternalError	And internal error occurred while processing the drive.

Blob Status Codes

The following table lists the status codes for processing a blob.

Status code	Description
Completed	The blob has finished processing without errors.
CompletedWithErrors	The blob has finished processing with errors in one or more page ranges or blocks, metadata, or properties.
FileNameInvalid	The file name is invalid.
FileNameTooLong	The file name is too long.
FileNotFound	The file is not found.
FileAccessDenied	Access to the file is denied.
BlobRequestFailed	The Blob service request to access the blob has failed.
BlobRequestForbidden	The Blob service request to access the blob is forbidden. This might be due to invalid storage account key or container SAS.
RenameFailed	Failed to rename the blob (for an import job) or the file (for an export job).
BlobUnexpectedChange	An unexpected change has occurred with the blob (for an export job).
LeasePresent	There is a lease present on the blob.
IOFailed	A disk or network I/O failure occurred while processing the blob.
Failed	An unknown failure occurred while processing the blob.

Import Disposition Status Codes

The following table lists the status codes for resolving an import disposition.

Status code	Description
Created	The blob has been created.
Renamed	The blob has been renamed per rename import disposition. The Blob/BlobPath element contains the URI for the renamed blob.
Skipped	The blob has been skipped per no-overwrite import disposition.
Overwritten	The blob has overwritten an existing blob per overwrite import disposition.
Cancelled	A prior failure has stopped further processing of the import disposition.

Page Range/Block Status Codes

The following table lists the status codes for processing a page range or a block.

Status code	Description
Completed	The page range or block has finished processing without any errors.
Committed	The block has been committed, but not in the full block list because other blocks have failed, or put full block list itself has failed.
Uncommitted	The block is uploaded but not committed.
Corrupted	The page range or block is corrupted (the content does not match its hash).
FileUnexpectedEnd	An unexpected end of file has been encountered.
BlobUnexpectedEnd	An unexpected end of blob has been encountered.
BlobRequestFailed	The Blob service request to access the page range or block has failed.
IOFailed	A disk or network I/O failure occurred while processing the page range or block.
Failed	An unknown failure occurred while processing the page range or block.
Cancelled	A prior failure has stopped further processing of the page range or block.

Metadata Status Codes

The following table lists the status codes for processing blob metadata.

Status code	Description
Completed	The metadata has finished processing without errors.
FileNameInvalid	The metadata file name is invalid.
FileNameTooLong	The metadata file name is too long.
FileNotFound	The metadata file is not found.
FileAccessDenied	Access to the metadata file is denied.
Corrupted	The metadata file is corrupted (the content does not match its hash).
XmlReadFailed	The metadata content does not conform to the required format.
XmlWriteFailed	Writing the metadata XML has failed.
BlobRequestFailed	The Blob service request to access the metadata has failed.
IOFailed	A disk or network I/O failure occurred while processing the metadata.
Failed	An unknown failure occurred while processing the metadata.
Cancelled	A prior failure has stopped further processing of the metadata.

Properties Status Codes

The following table lists the status codes for processing blob properties.

Status code	Description
Completed	The properties have finished processing without any errors.
FileNameInvalid	The properties file name is invalid.
FileNameTooLong	The properties file name is too long.
FileNotFound	The properties file is not found.
FileAccessDenied	Access to the properties file is denied.
Corrupted	The properties file is corrupted (the content does not match its hash).
XmlReadFailed	The properties content does not conform to the required format.
XmlWriteFailed	Writing the properties XML has failed.
BlobRequestFailed	The Blob service request to access the properties has failed.
IOFailed	A disk or network I/O failure occurred while processing the properties.
Failed	An unknown failure occurred while processing the properties.
Cancelled	A prior failure has stopped further processing of the properties.

Sample Logs

The following is an example of verbose log.

<?xml version="1.0" encoding="UTF-8"?>
<DriveLog Version="2014-11-01">
    <DriveId>WD-WMATV123456</DriveId>
    <Blob Status="Completed">
       <BlobPath>pictures/bob/wild/desert.jpg</BlobPath>
       <FilePath>\Users\bob\Pictures\wild\desert.jpg</FilePath>
       <Length>98304</Length>
       <ImportDisposition Status="Created">overwrite</ImportDisposition>
       <BlockList>
          <Block Offset="0" Length="65536" Id="AAAAAA==" Hash=" 9C8AE14A55241F98533C4D80D85CDC68" Status="Completed"/>
          <Block Offset="65536" Length="32768" Id="AQAAAA==" Hash=" DF54C531C9B3CA2570FDDDB3BCD0E27D" Status="Completed"/>
       </BlockList>
       <Metadata Status="Completed">
          <GlobalPath Hash=" E34F54B7086BCF4EC1601D056F4C7E37">\Users\bob\Pictures\wild\metadata.xml</GlobalPath>
       </Metadata>
    </Blob>
    <Blob Status="CompletedWithErrors">
       <BlobPath>pictures/bob/animals/koala.jpg</BlobPath>
       <FilePath>\Users\bob\Pictures\animals\koala.jpg</FilePath>
       <Length>163840</Length>
       <ImportDisposition Status="Overwritten">overwrite</ImportDisposition>
       <PageRangeList>
          <PageRange Offset="0" Length="65536" Hash="19701B8877418393CB3CB567F53EE225" Status="Completed"/>
          <PageRange Offset="65536" Length="65536" Hash="AA2585F6F6FD01C4AD4256E018240CD4" Status="Corrupted"/>
          <PageRange Offset="131072" Length="4096" Hash="9BA552E1C3EEAFFC91B42B979900A996" Status="Completed"/>
       </PageRangeList>
       <Properties Status="Completed">
          <Path Hash="38D7AE80653F47F63C0222FEE90EC4E7">\Users\bob\Pictures\animals\koala.jpg.properties</Path>
       </Properties>
    </Blob>
    <Status>CompletedWithErrors</Status>
</DriveLog>

The corresponding error log is shown below.

<?xml version="1.0" encoding="UTF-8"?>
<DriveLog Version="2014-11-01">
    <DriveId>WD-WMATV6965824</DriveId>
    <Blob Status="CompletedWithErrors">
       <BlobPath>pictures/bob/animals/koala.jpg</BlobPath>
       <FilePath>\Users\bob\Pictures\animals\koala.jpg</FilePath>
       <Length>163840</Length>
       <ImportDisposition Status="Overwritten">overwrite</ImportDisposition>
       <PageRangeList>
          <PageRange Offset="65536" Length="65536" Hash="AA2585F6F6FD01C4AD4256E018240CD4" Status="Corrupted"/>
       </PageRangeList>
    </Blob>
    <Status>CompletedWithErrors</Status>
</DriveLog>

The follow error log for an import job contains an error about a file not found on the import drive. Note that the status of subsequent components is Cancelled.

<?xml version="1.0" encoding="utf-8"?>
<DriveLog Version="2014-11-01">
  <DriveId>9WM35C2V</DriveId>
  <Blob Status="FileNotFound">
    <BlobPath>pictures/animals/koala.jpg</BlobPath>
    <FilePath>\animals\koala.jpg</FilePath>
    <Length>30310</Length>
    <ImportDisposition Status="Cancelled">rename</ImportDisposition>
    <BlockList>
      <Block Offset="0" Length="6062" Id="MD5/cAzn4h7VVSWXf696qp5Uaw==" Hash="700CE7E21ED55525977FAF7AAA9E546B" Status="Cancelled" />
      <Block Offset="6062" Length="6062" Id="MD5/PEnGwYOI8LPLNYdfKr7kAg==" Hash="3C49C6C18388F0B3CB35875F2ABEE402" Status="Cancelled" />
      <Block Offset="12124" Length="6062" Id="MD5/FG4WxqfZKuUWZ2nGTU2qVA==" Hash="146E16C6A7D92AE5166769C64D4DAA54" Status="Cancelled" />
      <Block Offset="18186" Length="6062" Id="MD5/ZzibNDzr3IRBQENRyegeXQ==" Hash="67389B343CEBDC8441404351C9E81E5D" Status="Cancelled" />
      <Block Offset="24248" Length="6062" Id="MD5/ZzibNDzr3IRBQENRyegeXQ==" Hash="67389B343CEBDC8441404351C9E81E5D" Status="Cancelled" />
    </BlockList>
  </Blob>
  <Status>CompletedWithErrors</Status>
</DriveLog>

The following error log for an export job indicates that the blob content has been successfully written to the drive, but that an error occurred while exporting the blob's properties.

<?xml version="1.0" encoding="utf-8"?>
<DriveLog Version="2014-11-01">
  <DriveId>9WM35C3U</DriveId>
  <Blob Status="CompletedWithErrors">
    <BlobPath>pictures/wild/canyon.jpg</BlobPath>
    <FilePath>\pictures\wild\canyon.jpg</FilePath>
    <LastModified>2012-09-18T23:47:08Z</LastModified>
    <Length>163840</Length>
    <BlockList />
    <Properties Status="Failed" />
  </Blob>
  <Status>CompletedWithErrors</Status>
</DriveLog>

See Also

Storage Import/Export Service REST API Reference