View Service Fabric health reports
Azure Service Fabric introduces a health model with health entities on which system components and watchdogs can report local conditions that they are monitoring. The health store aggregates all health data to determine whether entities are healthy.
The cluster is automatically populated with health reports sent by the system components. Read more at Use system health reports to troubleshoot.
Service Fabric provides multiple ways to get the aggregated health of the entities:
- Service Fabric Explorer or other visualization tools
- Health queries (through PowerShell, API, or REST)
- General queries that return a list of entities that have health as one of the properties (through PowerShell, API, or REST)
To demonstrate these options, let's use a local cluster with five nodes and the fabric:/WordCount application. The fabric:/WordCount application contains two default services, a stateful service of type WordCountServiceType, and a stateless service of type WordCountWebServiceType. I changed the ApplicationManifest.xml to require seven target replicas for the stateful service and one partition. Because there are only five nodes in the cluster, the system components report a warning on the service partition because it is below the target count.
<Service Name="WordCountService">
<StatefulService ServiceTypeName="WordCountServiceType" TargetReplicaSetSize="7" MinReplicaSetSize="2">
<UniformInt64Partition PartitionCount="[WordCountService_PartitionCount]" LowKey="1" HighKey="26" />
</StatefulService>
</Service>
Health in Service Fabric Explorer
Service Fabric Explorer provides a visual view of the cluster. In the image below, you can see that:
- The application fabric:/WordCount is red (in error) because it has an error event reported by MyWatchdog for the property Availability.
- One of its services, fabric:/WordCount/WordCountService is yellow (in warning). The service is configured with seven replicas and the cluster has five nodes, so two replicas can't be placed. Although it's not shown here, the service partition is yellow because of a system report from
System.FMsaying thatPartition is below target replica or instance count. The yellow partition triggers the yellow service. - The cluster is red because of the red application.
The evaluation uses default policies from the cluster manifest and application manifest. They are strict policies and do not tolerate any failure.
View of the cluster with Service Fabric Explorer:
Note
Read more about Service Fabric Explorer.
Health queries
Service Fabric exposes health queries for each of the supported entity types. They can be accessed through the API, using methods on FabricClient.HealthManager, PowerShell cmdlets, and REST. These queries return complete health information about the entity: the aggregated health state, entity health events, child health states (when applicable), unhealthy evaluations (when the entity is not healthy), and children health statistics (when applicable).
Note
A health entity is returned when it is fully populated in the health store. The entity must be active (not deleted) and have a system report. Its parent entities on the hierarchy chain must also have system reports. If any of these conditions are not satisfied, the health queries return a FabricException with FabricErrorCode FabricHealthEntityNotFound that shows why the entity is not returned.
The health queries must pass in the entity identifier, which depends on the entity type. The queries accept optional health policy parameters. If no health policies are specified, the health policies from the cluster or application manifest are used for evaluation. If the manifests don't contain a definition for health policies, the default health policies are used for evaluation. The default health policies do not tolerate any failures. The queries also accept filters for returning only partial children or events--the ones that respect the specified filters. Another filter allows excluding the children statistics.
Note
The output filters are applied on the server side, so the message reply size is reduced. We recommended that you use the output filters to limit the data returned, rather than apply filters on the client side.
An entity's health contains:
- The aggregated health state of the entity. Computed by the health store based on entity health reports, child health states (when applicable), and health policies. Read more about entity health evaluation.
- The health events on the entity.
- The collection of health states of all children for the entities that can have children. The health states contain entity identifiers and the aggregated health state. To get complete health for a child, call the query health for the child entity type and pass in the child identifier.
- The unhealthy evaluations that point to the report that triggered the state of the entity, if the entity is not healthy. The evaluations are recursive, containing the children health evaluations that triggered current health state. For example, a watchdog reported an error against a replica. The application health shows an unhealthy evaluation due to an unhealthy service; the service is unhealthy due to a partition in error; the partition is unhealthy due to a replica in error; the replica is unhealthy due to the watchdog error health report.
- The health statistics for all children types of the entities that have children. For example, cluster health shows the total number of applications, services, partitions, replicas, and deployed entities in the cluster. Service health shows the total number of partitions and replicas under the specified service.
Get cluster health
Returns the health of the cluster entity and contains the health states of applications and nodes (children of the cluster). Input:
- [Optional] The cluster health policy used to evaluate the nodes and the cluster events.
- [Optional] The application health policy map, with the health policies used to override the application manifest policies.
- [Optional] Filters for events, nodes, and applications that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events, nodes, and applications are used to evaluate the entity aggregated health, regardless of the filter.
- [Optional] Filter to exclude health statistics.
- [Optional] Filter to include fabric:/System health statistics in the health statistics. Only applicable when the health statistics are not excluded. By default, the health statistics include only statistics for user applications and not the System application.
API
To get cluster health, create a FabricClient and call the GetClusterHealthAsync method on its HealthManager.
The following call gets the cluster health:
ClusterHealth clusterHealth = await fabricClient.HealthManager.GetClusterHealthAsync();
The following code gets the cluster health by using a custom cluster health policy and filters for nodes and applications. It specifies that the health statistics include the fabric:/System statistics. It creates ClusterHealthQueryDescription, which contains the input information.
var policy = new ClusterHealthPolicy()
{
MaxPercentUnhealthyNodes = 20
};
var nodesFilter = new NodeHealthStatesFilter()
{
HealthStateFilterValue = HealthStateFilter.Error | HealthStateFilter.Warning
};
var applicationsFilter = new ApplicationHealthStatesFilter()
{
HealthStateFilterValue = HealthStateFilter.Error
};
var healthStatisticsFilter = new ClusterHealthStatisticsFilter()
{
ExcludeHealthStatistics = false,
IncludeSystemApplicationHealthStatistics = true
};
var queryDescription = new ClusterHealthQueryDescription()
{
HealthPolicy = policy,
ApplicationsFilter = applicationsFilter,
NodesFilter = nodesFilter,
HealthStatisticsFilter = healthStatisticsFilter
};
ClusterHealth clusterHealth = await fabricClient.HealthManager.GetClusterHealthAsync(queryDescription);
PowerShell
The cmdlet to get the cluster health is Get-ServiceFabricClusterHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet.
The state of the cluster is five nodes, the system application, and fabric:/WordCount configured as described.
The following cmdlet gets cluster health by using default health policies. The aggregated health state is warning, because the fabric:/WordCount application is in warning. Note how the unhealthy evaluations provide details on the conditions that triggered the aggregated health.
PS D:\ServiceFabric> Get-ServiceFabricClusterHealth
AggregatedHealthState : Warning
UnhealthyEvaluations :
Unhealthy applications: 100% (1/1), MaxPercentUnhealthyApplications=0%.
Unhealthy application: ApplicationName='fabric:/WordCount', AggregatedHealthState='Warning'.
Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%.
Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Warning'.
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Warning'.
Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false.
NodeHealthStates :
NodeName : _Node_4
AggregatedHealthState : Ok
NodeName : _Node_3
AggregatedHealthState : Ok
NodeName : _Node_2
AggregatedHealthState : Ok
NodeName : _Node_1
AggregatedHealthState : Ok
NodeName : _Node_0
AggregatedHealthState : Ok
ApplicationHealthStates :
ApplicationName : fabric:/System
AggregatedHealthState : Ok
ApplicationName : fabric:/WordCount
AggregatedHealthState : Warning
HealthEvents : None
HealthStatistics :
Node : 5 Ok, 0 Warning, 0 Error
Replica : 6 Ok, 0 Warning, 0 Error
Partition : 1 Ok, 1 Warning, 0 Error
Service : 1 Ok, 1 Warning, 0 Error
DeployedServicePackage : 6 Ok, 0 Warning, 0 Error
DeployedApplication : 5 Ok, 0 Warning, 0 Error
Application : 0 Ok, 1 Warning, 0 Error
The following PowerShell cmdlet gets the health of the cluster by using a custom application policy. It filters results to get only applications and nodes in error or warning. As a result, no nodes are returned, as they are all healthy. Only the fabric:/WordCount application respects the applications filter. Because the custom policy specifies to consider warnings as errors for the fabric:/WordCount application, the application is evaluated as in error, and so is the cluster.
PS D:\ServiceFabric> $appHealthPolicy = New-Object -TypeName System.Fabric.Health.ApplicationHealthPolicy
$appHealthPolicy.ConsiderWarningAsError = $true
$appHealthPolicyMap = New-Object -TypeName System.Fabric.Health.ApplicationHealthPolicyMap
$appUri1 = New-Object -TypeName System.Uri -ArgumentList "fabric:/WordCount"
$appHealthPolicyMap.Add($appUri1, $appHealthPolicy)
Get-ServiceFabricClusterHealth -ApplicationHealthPolicyMap $appHealthPolicyMap -ApplicationsFilter "Warning,Error" -NodesFilter "Warning,Error" -ExcludeHealthStatistics
AggregatedHealthState : Error
UnhealthyEvaluations :
Unhealthy applications: 100% (1/1), MaxPercentUnhealthyApplications=0%.
Unhealthy application: ApplicationName='fabric:/WordCount', AggregatedHealthState='Error'.
Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%.
Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'.
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Error'.
Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=true.
NodeHealthStates : None
ApplicationHealthStates :
ApplicationName : fabric:/WordCount
AggregatedHealthState : Error
HealthEvents : None
REST
You can get cluster health with a GET request or a POST request that includes health policies described in the body.
Get node health
Returns the health of a node entity and contains the health events reported on the node. Input:
- [Required] The node name that identifies the node.
- [Optional] The cluster health policy settings used to evaluate health.
- [Optional] Filters for events that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events are used to evaluate the entity aggregated health, regardless of the filter.
API
To get node health through the API, create a FabricClient and call the GetNodeHealthAsync method on its HealthManager.
The following code gets the node health for the specified node name:
NodeHealth nodeHealth = await fabricClient.HealthManager.GetNodeHealthAsync(nodeName);
The following code gets the node health for the specified node name and passes in events filter and custom policy through NodeHealthQueryDescription:
var queryDescription = new NodeHealthQueryDescription(nodeName)
{
HealthPolicy = new ClusterHealthPolicy() { ConsiderWarningAsError = true },
EventsFilter = new HealthEventsFilter() { HealthStateFilterValue = HealthStateFilter.Warning },
};
NodeHealth nodeHealth = await fabricClient.HealthManager.GetNodeHealthAsync(queryDescription);
PowerShell
The cmdlet to get the node health is Get-ServiceFabricNodeHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet. The following cmdlet gets the node health by using default health policies:
PS D:\ServiceFabric> Get-ServiceFabricNodeHealth _Node_1
NodeName : _Node_1
AggregatedHealthState : Ok
HealthEvents :
SourceId : System.FM
Property : State
HealthState : Ok
SequenceNumber : 3
SentAt : 7/13/2017 4:39:23 PM
ReceivedAt : 7/13/2017 4:40:47 PM
TTL : Infinite
Description : Fabric node is up.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 4:40:47 PM, LastWarning = 1/1/0001 12:00:00 AM
The following cmdlet gets the health of all nodes in the cluster:
PS D:\ServiceFabric> Get-ServiceFabricNode | Get-ServiceFabricNodeHealth | select NodeName, AggregatedHealthState | ft -AutoSize
NodeName AggregatedHealthState
-------- ---------------------
_Node_4 Ok
_Node_3 Ok
_Node_2 Ok
_Node_1 Ok
_Node_0 Ok
REST
You can get node health with a GET request or a POST request that includes health policies described in the body.
Get application health
Returns the health of an application entity. It contains the health states of the deployed application and service children. Input:
- [Required] The application name (URI) that identifies the application.
- [Optional] The application health policy used to override the application manifest policies.
- [Optional] Filters for events, services, and deployed applications that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events, services, and deployed applications are used to evaluate the entity aggregated health, regardless of the filter.
- [Optional] Filter to exclude the health statistics. If not specified, the health statistics include the ok, warning, and error count for all application children: services, partitions, replicas, deployed applications, and deployed service packages.
API
To get application health, create a FabricClient and call the GetApplicationHealthAsync method on its HealthManager.
The following code gets the application health for the specified application name (URI):
ApplicationHealth applicationHealth = await fabricClient.HealthManager.GetApplicationHealthAsync(applicationName);
The following code gets the application health for the specified application name (URI), with filters and custom policies specified via ApplicationHealthQueryDescription.
HealthStateFilter warningAndErrors = HealthStateFilter.Error | HealthStateFilter.Warning;
var serviceTypePolicy = new ServiceTypeHealthPolicy()
{
MaxPercentUnhealthyPartitionsPerService = 0,
MaxPercentUnhealthyReplicasPerPartition = 5,
MaxPercentUnhealthyServices = 0,
};
var policy = new ApplicationHealthPolicy()
{
ConsiderWarningAsError = false,
DefaultServiceTypeHealthPolicy = serviceTypePolicy,
MaxPercentUnhealthyDeployedApplications = 0,
};
var queryDescription = new ApplicationHealthQueryDescription(applicationName)
{
HealthPolicy = policy,
EventsFilter = new HealthEventsFilter() { HealthStateFilterValue = warningAndErrors },
ServicesFilter = new ServiceHealthStatesFilter() { HealthStateFilterValue = warningAndErrors },
DeployedApplicationsFilter = new DeployedApplicationHealthStatesFilter() { HealthStateFilterValue = warningAndErrors },
};
ApplicationHealth applicationHealth = await fabricClient.HealthManager.GetApplicationHealthAsync(queryDescription);
PowerShell
The cmdlet to get the application health is Get-ServiceFabricApplicationHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet.
The following cmdlet returns the health of the fabric:/WordCount application:
PS D:\ServiceFabric> Get-ServiceFabricApplicationHealth fabric:/WordCount
ApplicationName : fabric:/WordCount
AggregatedHealthState : Warning
UnhealthyEvaluations :
Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%.
Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Warning'.
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Warning'.
Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false.
ServiceHealthStates :
ServiceName : fabric:/WordCount/WordCountWebService
AggregatedHealthState : Ok
ServiceName : fabric:/WordCount/WordCountService
AggregatedHealthState : Warning
DeployedApplicationHealthStates :
ApplicationName : fabric:/WordCount
NodeName : _Node_4
AggregatedHealthState : Ok
ApplicationName : fabric:/WordCount
NodeName : _Node_3
AggregatedHealthState : Ok
ApplicationName : fabric:/WordCount
NodeName : _Node_0
AggregatedHealthState : Ok
ApplicationName : fabric:/WordCount
NodeName : _Node_2
AggregatedHealthState : Ok
ApplicationName : fabric:/WordCount
NodeName : _Node_1
AggregatedHealthState : Ok
HealthEvents :
SourceId : System.CM
Property : State
HealthState : Ok
SequenceNumber : 282
SentAt : 7/13/2017 5:57:05 PM
ReceivedAt : 7/13/2017 5:57:05 PM
TTL : Infinite
Description : Application has been created.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:05 PM, LastWarning = 1/1/0001 12:00:00 AM
HealthStatistics :
Replica : 6 Ok, 0 Warning, 0 Error
Partition : 1 Ok, 1 Warning, 0 Error
Service : 1 Ok, 1 Warning, 0 Error
DeployedServicePackage : 6 Ok, 0 Warning, 0 Error
DeployedApplication : 5 Ok, 0 Warning, 0 Error
The following PowerShell cmdlet passes in custom policies. It also filters children and events.
PS D:\ServiceFabric> Get-ServiceFabricApplicationHealth -ApplicationName fabric:/WordCount -ConsiderWarningAsError $true -ServicesFilter Error -EventsFilter Error -DeployedApplicationsFilter Error -ExcludeHealthStatistics
ApplicationName : fabric:/WordCount
AggregatedHealthState : Error
UnhealthyEvaluations :
Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%.
Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'.
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Error'.
Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=true.
ServiceHealthStates :
ServiceName : fabric:/WordCount/WordCountService
AggregatedHealthState : Error
DeployedApplicationHealthStates : None
HealthEvents : None
REST
You can get application health with a GET request or a POST request that includes health policies described in the body.
Get service health
Returns the health of a service entity. It contains the partition health states. Input:
- [Required] The service name (URI) that identifies the service.
- [Optional] The application health policy used to override the application manifest policy.
- [Optional] Filters for events and partitions that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events and partitions are used to evaluate the entity aggregated health, regardless of the filter.
- [Optional] Filter to exclude health statistics. If not specified, the health statistics show the ok, warning, and error count for all partitions and replicas of the service.
API
To get service health through the API, create a FabricClient and call the GetServiceHealthAsync method on its HealthManager.
The following example gets the health of a service with specified service name (URI):
ServiceHealth serviceHealth = await fabricClient.HealthManager.GetServiceHealthAsync(serviceName);
The following code gets the service health for the specified service name (URI), specifying filters and custom policy via ServiceHealthQueryDescription:
var queryDescription = new ServiceHealthQueryDescription(serviceName)
{
EventsFilter = new HealthEventsFilter() { HealthStateFilterValue = HealthStateFilter.All },
PartitionsFilter = new PartitionHealthStatesFilter() { HealthStateFilterValue = HealthStateFilter.Error },
};
ServiceHealth serviceHealth = await fabricClient.HealthManager.GetServiceHealthAsync(queryDescription);
PowerShell
The cmdlet to get the service health is Get-ServiceFabricServiceHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet.
The following cmdlet gets the service health by using default health policies:
PS D:\ServiceFabric> Get-ServiceFabricServiceHealth -ServiceName fabric:/WordCount/WordCountService
ServiceName : fabric:/WordCount/WordCountService
AggregatedHealthState : Warning
UnhealthyEvaluations :
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Warning'.
Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false.
PartitionHealthStates :
PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600
AggregatedHealthState : Warning
HealthEvents :
SourceId : System.FM
Property : State
HealthState : Ok
SequenceNumber : 15
SentAt : 7/13/2017 5:57:05 PM
ReceivedAt : 7/13/2017 5:57:18 PM
TTL : Infinite
Description : Service has been created.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM
HealthStatistics :
Replica : 5 Ok, 0 Warning, 0 Error
Partition : 0 Ok, 1 Warning, 0 Error
REST
You can get service health with a GET request or a POST request that includes health policies described in the body.
Get partition health
Returns the health of a partition entity. It contains the replica health states. Input:
- [Required] The partition ID (GUID) that identifies the partition.
- [Optional] The application health policy used to override the application manifest policy.
- [Optional] Filters for events and replicas that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events and replicas are used to evaluate the entity aggregated health, regardless of the filter.
- [Optional] Filter to exclude health statistics. If not specified, the health statistics show how many replicas are in ok, warning, and error states.
API
To get partition health through the API, create a FabricClient and call the GetPartitionHealthAsync method on its HealthManager. To specify optional parameters, create PartitionHealthQueryDescription.
PartitionHealth partitionHealth = await fabricClient.HealthManager.GetPartitionHealthAsync(partitionId);
PowerShell
The cmdlet to get the partition health is Get-ServiceFabricPartitionHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet.
The following cmdlet gets the health for all partitions of the fabric:/WordCount/WordCountService service and filters out replica health states:
PS D:\ServiceFabric> Get-ServiceFabricPartition fabric:/WordCount/WordCountService | Get-ServiceFabricPartitionHealth -ReplicasFilter None
PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600
AggregatedHealthState : Warning
UnhealthyEvaluations :
Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false.
ReplicaHealthStates : None
HealthEvents :
SourceId : System.FM
Property : State
HealthState : Warning
SequenceNumber : 72
SentAt : 7/13/2017 5:57:29 PM
ReceivedAt : 7/13/2017 5:57:48 PM
TTL : Infinite
Description : Partition is below target replica or instance count.
fabric:/WordCount/WordCountService 7 2 af2e3e44-a8f8-45ac-9f31-4093eb897600
N/P RD _Node_2 Up 131444422260002646
N/S RD _Node_4 Up 131444422293113678
N/S RD _Node_3 Up 131444422293113679
N/S RD _Node_1 Up 131444422293118720
N/S RD _Node_0 Up 131444422293118721
(Showing 5 out of 5 replicas. Total available replicas: 5.)
RemoveWhenExpired : False
IsExpired : False
Transitions : Ok->Warning = 7/13/2017 5:57:48 PM, LastError = 1/1/0001 12:00:00 AM
SourceId : System.PLB
Property : ServiceReplicaUnplacedHealth_Secondary_af2e3e44-a8f8-45ac-9f31-4093eb897600
HealthState : Warning
SequenceNumber : 131444445174851664
SentAt : 7/13/2017 6:35:17 PM
ReceivedAt : 7/13/2017 6:35:18 PM
TTL : 00:01:05
Description : The Load Balancer was unable to find a placement for one or more of the Service's Replicas:
Secondary replica could not be placed due to the following constraints and properties:
TargetReplicaSetSize: 7
Placement Constraint: N/A
Parent Service: N/A
Constraint Elimination Sequence:
Existing Secondary Replicas eliminated 4 possible node(s) for placement -- 1/5 node(s) remain.
Existing Primary Replica eliminated 1 possible node(s) for placement -- 0/5 node(s) remain.
Nodes Eliminated By Constraints:
Existing Secondary Replicas -- Nodes with Partition's Existing Secondary Replicas/Instances:
--
FaultDomain:fd:/4 NodeName:_Node_4 NodeType:NodeType4 UpgradeDomain:4 UpgradeDomain: ud:/4 Deactivation Intent/Status: None/None
FaultDomain:fd:/3 NodeName:_Node_3 NodeType:NodeType3 UpgradeDomain:3 UpgradeDomain: ud:/3 Deactivation Intent/Status: None/None
FaultDomain:fd:/1 NodeName:_Node_1 NodeType:NodeType1 UpgradeDomain:1 UpgradeDomain: ud:/1 Deactivation Intent/Status: None/None
FaultDomain:fd:/0 NodeName:_Node_0 NodeType:NodeType0 UpgradeDomain:0 UpgradeDomain: ud:/0 Deactivation Intent/Status: None/None
Existing Primary Replica -- Nodes with Partition's Existing Primary Replica or Secondary Replicas:
--
FaultDomain:fd:/2 NodeName:_Node_2 NodeType:NodeType2 UpgradeDomain:2 UpgradeDomain: ud:/2 Deactivation Intent/Status: None/None
RemoveWhenExpired : True
IsExpired : False
Transitions : Error->Warning = 7/13/2017 5:57:48 PM, LastOk = 1/1/0001 12:00:00 AM
HealthStatistics :
Replica : 5 Ok, 0 Warning, 0 Error
REST
You can get partition health with a GET request or a POST request that includes health policies described in the body.
Get replica health
Returns the health of a stateful service replica or a stateless service instance. Input:
- [Required] The partition ID (GUID) and replica ID that identifies the replica.
- [Optional] The application health policy parameters used to override the application manifest policies.
- [Optional] Filters for events that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events are used to evaluate the entity aggregated health, regardless of the filter.
API
To get the replica health through the API, create a FabricClient and call the GetReplicaHealthAsync method on its HealthManager. To specify advanced parameters, use ReplicaHealthQueryDescription.
ReplicaHealth replicaHealth = await fabricClient.HealthManager.GetReplicaHealthAsync(partitionId, replicaId);
PowerShell
The cmdlet to get the replica health is Get-ServiceFabricReplicaHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet.
The following cmdlet gets the health of the primary replica for all partitions of the service:
PS D:\ServiceFabric> Get-ServiceFabricPartition fabric:/WordCount/WordCountService | Get-ServiceFabricReplica | where {$_.ReplicaRole -eq "Primary"} | Get-ServiceFabricReplicaHealth
PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600
ReplicaId : 131444422260002646
AggregatedHealthState : Ok
HealthEvents :
SourceId : System.RA
Property : State
HealthState : Ok
SequenceNumber : 131444422263668344
SentAt : 7/13/2017 5:57:06 PM
ReceivedAt : 7/13/2017 5:57:18 PM
TTL : Infinite
Description : Replica has been created._Node_2
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM
REST
You can get replica health with a GET request or a POST request that includes health policies described in the body.
Get deployed application health
Returns the health of an application deployed on a node entity. It contains the deployed service package health states. Input:
- [Required] The application name (URI) and node name (string) that identify the deployed application.
- [Optional] The application health policy used to override the application manifest policies.
- [Optional] Filters for events and deployed service packages that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events and deployed service packages are used to evaluate the entity aggregated health, regardless of the filter.
- [Optional] Filter to exclude health statistics. If not specified, the health statistics show the number of deployed service packages in ok, warning, and error health states.
API
To get the health of an application deployed on a node through the API, create a FabricClient and call the GetDeployedApplicationHealthAsync method on its HealthManager. To specify optional parameters, use DeployedApplicationHealthQueryDescription.
DeployedApplicationHealth health = await fabricClient.HealthManager.GetDeployedApplicationHealthAsync(
new DeployedApplicationHealthQueryDescription(applicationName, nodeName));
PowerShell
The cmdlet to get the deployed application health is Get-ServiceFabricDeployedApplicationHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet. To find out where an application is deployed, run Get-ServiceFabricApplicationHealth and look at the deployed application children.
The following cmdlet gets the health of the fabric:/WordCount application deployed on _Node_2.
PS D:\ServiceFabric> Get-ServiceFabricDeployedApplicationHealth -ApplicationName fabric:/WordCount -NodeName _Node_0
ApplicationName : fabric:/WordCount
NodeName : _Node_0
AggregatedHealthState : Ok
DeployedServicePackageHealthStates :
ServiceManifestName : WordCountServicePkg
ServicePackageActivationId :
NodeName : _Node_0
AggregatedHealthState : Ok
ServiceManifestName : WordCountWebServicePkg
ServicePackageActivationId :
NodeName : _Node_0
AggregatedHealthState : Ok
HealthEvents :
SourceId : System.Hosting
Property : Activation
HealthState : Ok
SequenceNumber : 131444422261848308
SentAt : 7/13/2017 5:57:06 PM
ReceivedAt : 7/13/2017 5:57:17 PM
TTL : Infinite
Description : The application was activated successfully.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:17 PM, LastWarning = 1/1/0001 12:00:00 AM
HealthStatistics :
DeployedServicePackage : 2 Ok, 0 Warning, 0 Error
REST
You can get deployed application health with a GET request or a POST request that includes health policies described in the body.
Get deployed service package health
Returns the health of a deployed service package entity. Input:
- [Required] The application name (URI), node name (string), and service manifest name (string) that identify the deployed service package.
- [Optional] The application health policy used to override the application manifest policy.
- [Optional] Filters for events that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events are used to evaluate the entity aggregated health, regardless of the filter.
API
To get the health of a deployed service package through the API, create a FabricClient and call the GetDeployedServicePackageHealthAsync method on its HealthManager. To specify optional parameters, use DeployedServicePackageHealthQueryDescription.
DeployedServicePackageHealth health = await fabricClient.HealthManager.GetDeployedServicePackageHealthAsync(
new DeployedServicePackageHealthQueryDescription(applicationName, nodeName, serviceManifestName));
PowerShell
The cmdlet to get the deployed service package health is Get-ServiceFabricDeployedServicePackageHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet. To see where an application is deployed, run Get-ServiceFabricApplicationHealth and look at the deployed applications. To see which service packages are in an application, look at the deployed service package children in the Get-ServiceFabricDeployedApplicationHealth output.
The following cmdlet gets the health of the WordCountServicePkg service package of the fabric:/WordCount application deployed on _Node_2. The entity has System.Hosting reports for successful service-package and entry-point activation, and successful service-type registration.
PS D:\ServiceFabric> Get-ServiceFabricDeployedApplication -ApplicationName fabric:/WordCount -NodeName _Node_2 | Get-ServiceFabricDeployedServicePackageHealth -ServiceManifestName WordCountServicePkg
ApplicationName : fabric:/WordCount
ServiceManifestName : WordCountServicePkg
ServicePackageActivationId :
NodeName : _Node_2
AggregatedHealthState : Ok
HealthEvents :
SourceId : System.Hosting
Property : Activation
HealthState : Ok
SequenceNumber : 131444422267693359
SentAt : 7/13/2017 5:57:06 PM
ReceivedAt : 7/13/2017 5:57:18 PM
TTL : Infinite
Description : The ServicePackage was activated successfully.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM
SourceId : System.Hosting
Property : CodePackageActivation:Code:EntryPoint
HealthState : Ok
SequenceNumber : 131444422267903345
SentAt : 7/13/2017 5:57:06 PM
ReceivedAt : 7/13/2017 5:57:18 PM
TTL : Infinite
Description : The CodePackage was activated successfully.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM
SourceId : System.Hosting
Property : ServiceTypeRegistration:WordCountServiceType
HealthState : Ok
SequenceNumber : 131444422272458374
SentAt : 7/13/2017 5:57:07 PM
ReceivedAt : 7/13/2017 5:57:18 PM
TTL : Infinite
Description : The ServiceType was registered successfully.
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM
REST
You can get deployed service package health with a GET request or a POST request that includes health policies described in the body.
Health chunk queries
The health chunk queries can return multi-level cluster children (recursively), per input filters. It supports advanced filters that allow a lot of flexibility in choosing the children to be returned. The filters can specify children by the unique identifier or by other group identifiers and/or health states. By default, no children are included, as opposed to health commands that always include first-level children.
The health queries return only first-level children of the specified entity per required filters. To get the children of the children, you must call additional health APIs for each entity of interest. Similarly, to get the health of specific entities, you must call one health API for each desired entity. The chunk query advanced filtering allows you to request multiple items of interest in one query, minimizing the message size and the number of messages.
The value of the chunk query is that you can get health state for more cluster entities (potentially all cluster entities starting at required root) in one call. You can express complex health query such as:
- Return only applications in error, and for those applications include all services in warning or error. For returned services, include all partitions.
- Return only the health of four applications, specified by their names.
- Return only the health of applications of a desired application type.
- Return all deployed entities on a node. Returns all applications, all deployed applications on the specified node and all the deployed service packages on that node.
- Return all replicas in error. Returns all applications, services, partitions, and only replicas in error.
- Return all applications. For a specified service, include all partitions.
Currently, the health chunk query is exposed only for the cluster entity. It returns a cluster health chunk, which contains:
- The cluster aggregated health state.
- The health state chunk list of nodes that respect input filters.
- The health state chunk list of applications that respect input filters. Each application health state chunk contains a chunk list with all services that respect input filters and a chunk list with all deployed applications that respect the filters. Same for the children of services and deployed applications. This way, all entities in the cluster can be potentially returned if requested, in a hierarchical fashion.
Cluster health chunk query
Returns the health of the cluster entity and contains the hierarchical health state chunks of required children. Input:
- [Optional] The cluster health policy used to evaluate the nodes and the cluster events.
- [Optional] The application health policy map, with the health policies used to override the application manifest policies.
- [Optional] Filters for nodes and applications that specify which entries are of interest and should be returned in the result. The filters are specific to an entity/group of entities or are applicable to all entities at that level. The list of filters can contain one general filter and/or filters for specific identifiers to fine-grain entities returned by the query. If empty, the children are not returned by default. Read more about the filters at NodeHealthStateFilter and ApplicationHealthStateFilter. The application filters can recursively specify advanced filters for children.
The chunk result includes the children that respect the filters.
Currently, the chunk query does not return unhealthy evaluations or entity events. That extra information can be obtained using the existing cluster health query.
API
To get cluster health chunk, create a FabricClient and call the GetClusterHealthChunkAsync method on its HealthManager. You can pass in ClusterHealthQueryDescription to describe health policies and advanced filters.
The following code gets cluster health chunk with advanced filters.
var queryDescription = new ClusterHealthChunkQueryDescription();
queryDescription.ApplicationFilters.Add(new ApplicationHealthStateFilter()
{
// Return applications only if they are in error
HealthStateFilter = HealthStateFilter.Error
});
// Return all replicas
var wordCountServiceReplicaFilter = new ReplicaHealthStateFilter()
{
HealthStateFilter = HealthStateFilter.All
};
// Return all replicas and all partitions
var wordCountServicePartitionFilter = new PartitionHealthStateFilter()
{
HealthStateFilter = HealthStateFilter.All
};
wordCountServicePartitionFilter.ReplicaFilters.Add(wordCountServiceReplicaFilter);
// For specific service, return all partitions and all replicas
var wordCountServiceFilter = new ServiceHealthStateFilter()
{
ServiceNameFilter = new Uri("fabric:/WordCount/WordCountService"),
};
wordCountServiceFilter.PartitionFilters.Add(wordCountServicePartitionFilter);
// Application filter: for specific application, return no services except the ones of interest
var wordCountApplicationFilter = new ApplicationHealthStateFilter()
{
// Always return fabric:/WordCount application
ApplicationNameFilter = new Uri("fabric:/WordCount"),
};
wordCountApplicationFilter.ServiceFilters.Add(wordCountServiceFilter);
queryDescription.ApplicationFilters.Add(wordCountApplicationFilter);
var result = await fabricClient.HealthManager.GetClusterHealthChunkAsync(queryDescription);
PowerShell
The cmdlet to get the cluster health is Get-ServiceFabricClusterChunkHealth. First, connect to the cluster by using the Connect-ServiceFabricCluster cmdlet.
The following code gets nodes only if they are in Error except for a specific node, which should always be returned.
PS D:\ServiceFabric> $errorFilter = [System.Fabric.Health.HealthStateFilter]::Error;
$allFilter = [System.Fabric.Health.HealthStateFilter]::All;
$nodeFilter1 = New-Object System.Fabric.Health.NodeHealthStateFilter -Property @{HealthStateFilter=$errorFilter}
$nodeFilter2 = New-Object System.Fabric.Health.NodeHealthStateFilter -Property @{NodeNameFilter="_Node_1";HealthStateFilter=$allFilter}
# Create node filter list that will be passed in the cmdlet
$nodeFilters = New-Object System.Collections.Generic.List[System.Fabric.Health.NodeHealthStateFilter]
$nodeFilters.Add($nodeFilter1)
$nodeFilters.Add($nodeFilter2)
Get-ServiceFabricClusterHealthChunk -NodeFilters $nodeFilters
HealthState : Warning
NodeHealthStateChunks :
TotalCount : 1
NodeName : _Node_1
HealthState : Ok
ApplicationHealthStateChunks : None
The following cmdlet gets cluster chunk with application filters.
PS D:\ServiceFabric> $errorFilter = [System.Fabric.Health.HealthStateFilter]::Error;
$allFilter = [System.Fabric.Health.HealthStateFilter]::All;
# All replicas
$replicaFilter = New-Object System.Fabric.Health.ReplicaHealthStateFilter -Property @{HealthStateFilter=$allFilter}
# All partitions
$partitionFilter = New-Object System.Fabric.Health.PartitionHealthStateFilter -Property @{HealthStateFilter=$allFilter}
$partitionFilter.ReplicaFilters.Add($replicaFilter)
# For WordCountService, return all partitions and all replicas
$svcFilter1 = New-Object System.Fabric.Health.ServiceHealthStateFilter -Property @{ServiceNameFilter="fabric:/WordCount/WordCountService"}
$svcFilter1.PartitionFilters.Add($partitionFilter)
$svcFilter2 = New-Object System.Fabric.Health.ServiceHealthStateFilter -Property @{HealthStateFilter=$errorFilter}
$appFilter = New-Object System.Fabric.Health.ApplicationHealthStateFilter -Property @{ApplicationNameFilter="fabric:/WordCount"}
$appFilter.ServiceFilters.Add($svcFilter1)
$appFilter.ServiceFilters.Add($svcFilter2)
$appFilters = New-Object System.Collections.Generic.List[System.Fabric.Health.ApplicationHealthStateFilter]
$appFilters.Add($appFilter)
Get-ServiceFabricClusterHealthChunk -ApplicationFilters $appFilters
HealthState : Error
NodeHealthStateChunks : None
ApplicationHealthStateChunks :
TotalCount : 1
ApplicationName : fabric:/WordCount
ApplicationTypeName : WordCount
HealthState : Error
ServiceHealthStateChunks :
TotalCount : 1
ServiceName : fabric:/WordCount/WordCountService
HealthState : Error
PartitionHealthStateChunks :
TotalCount : 1
PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600
HealthState : Error
ReplicaHealthStateChunks :
TotalCount : 5
ReplicaOrInstanceId : 131444422293118720
HealthState : Ok
ReplicaOrInstanceId : 131444422293118721
HealthState : Ok
ReplicaOrInstanceId : 131444422293113678
HealthState : Ok
ReplicaOrInstanceId : 131444422293113679
HealthState : Ok
ReplicaOrInstanceId : 131444422260002646
HealthState : Error
The following cmdlet returns all deployed entities on a node.
PS D:\ServiceFabric> $errorFilter = [System.Fabric.Health.HealthStateFilter]::Error;
$allFilter = [System.Fabric.Health.HealthStateFilter]::All;
$dspFilter = New-Object System.Fabric.Health.DeployedServicePackageHealthStateFilter -Property @{HealthStateFilter=$allFilter}
$daFilter = New-Object System.Fabric.Health.DeployedApplicationHealthStateFilter -Property @{HealthStateFilter=$allFilter;NodeNameFilter="_Node_2"}
$daFilter.DeployedServicePackageFilters.Add($dspFilter)
$appFilter = New-Object System.Fabric.Health.ApplicationHealthStateFilter -Property @{HealthStateFilter=$allFilter}
$appFilter.DeployedApplicationFilters.Add($daFilter)
$appFilters = New-Object System.Collections.Generic.List[System.Fabric.Health.ApplicationHealthStateFilter]
$appFilters.Add($appFilter)
Get-ServiceFabricClusterHealthChunk -ApplicationFilters $appFilters
HealthState : Error
NodeHealthStateChunks : None
ApplicationHealthStateChunks :
TotalCount : 2
ApplicationName : fabric:/System
HealthState : Ok
DeployedApplicationHealthStateChunks :
TotalCount : 1
NodeName : _Node_2
HealthState : Ok
DeployedServicePackageHealthStateChunks :
TotalCount : 1
ServiceManifestName : FAS
ServicePackageActivationId :
HealthState : Ok
ApplicationName : fabric:/WordCount
ApplicationTypeName : WordCount
HealthState : Error
DeployedApplicationHealthStateChunks :
TotalCount : 1
NodeName : _Node_2
HealthState : Ok
DeployedServicePackageHealthStateChunks :
TotalCount : 1
ServiceManifestName : WordCountServicePkg
ServicePackageActivationId :
HealthState : Ok
REST
You can get cluster health chunk with a GET request or a POST request that includes health policies and advanced filters described in the body.
General queries
General queries return a list of Service Fabric entities of a specified type. They are exposed through the API (via the methods on FabricClient.QueryManager), PowerShell cmdlets, and REST. These queries aggregate subqueries from multiple components. One of them is the health store, which populates the aggregated health state for each query result.
Note
General queries return the aggregated health state of the entity and do not contain rich health data. If an entity is not healthy, you can follow up with health queries to get all its health information, including events, child health states, and unhealthy evaluations.
If general queries return an unknown health state for an entity, it's possible that the health store doesn't have complete data about the entity. It's also possible that a subquery to the health store wasn't successful (for example, there was a communication error, or the health store was throttled). Follow up with a health query for the entity. If the subquery encountered transient errors, such as network issues, this follow-up query may succeed. It may also give you more details from the health store about why the entity is not exposed.
The queries that contain HealthState for entities are:
- Node list: Returns the list nodes in the cluster (paged).
- API: FabricClient.QueryClient.GetNodeListAsync
- PowerShell: Get-ServiceFabricNode
- Application list: Returns the list of applications in the cluster (paged).
- API: FabricClient.QueryClient.GetApplicationListAsync
- PowerShell: Get-ServiceFabricApplication
- Service list: Returns the list of services in an application (paged).
- API: FabricClient.QueryClient.GetServiceListAsync
- PowerShell: Get-ServiceFabricService
- Partition list: Returns the list of partitions in a service (paged).
- API: FabricClient.QueryClient.GetPartitionListAsync
- PowerShell: Get-ServiceFabricPartition
- Replica list: Returns the list of replicas in a partition (paged).
- API: FabricClient.QueryClient.GetReplicaListAsync
- PowerShell: Get-ServiceFabricReplica
- Deployed application list: Returns the list of deployed applications on a node.
- API: FabricClient.QueryClient.GetDeployedApplicationListAsync
- PowerShell: Get-ServiceFabricDeployedApplication
- Deployed service package list: Returns the list of service packages in a deployed application.
- API: FabricClient.QueryClient.GetDeployedServicePackageListAsync
- PowerShell: Get-ServiceFabricDeployedApplication
Note
Some of the queries return paged results. The return of these queries is a list derived from PagedList<T>. If the results do not fit a message, only a page is returned and a ContinuationToken that tracks where enumeration stopped. Continue to call the same query and pass in the continuation token from the previous query to get next results.
Examples
The following code gets the unhealthy applications in the cluster:
var applications = fabricClient.QueryManager.GetApplicationListAsync().Result.Where(
app => app.HealthState == HealthState.Error);
The following cmdlet gets the application details for the fabric:/WordCount application. Notice that health state is at warning.
PS C:\> Get-ServiceFabricApplication -ApplicationName fabric:/WordCount
ApplicationName : fabric:/WordCount
ApplicationTypeName : WordCount
ApplicationTypeVersion : 1.0.0
ApplicationStatus : Ready
HealthState : Warning
ApplicationParameters : { "WordCountWebService_InstanceCount" = "1";
"_WFDebugParams_" = "[{"ServiceManifestName":"WordCountWebServicePkg","CodePackageName":"Code","EntryPointType":"Main","Debug
ExePath":"C:\\Program Files (x86)\\Microsoft Visual Studio
14.0\\Common7\\Packages\\Debugger\\VsDebugLaunchNotify.exe","DebugArguments":" {74f7e5d5-71a9-47e2-a8cd-1878ec4734f1} -p
[ProcessId] -tid [ThreadId]","EnvironmentBlock":"_NO_DEBUG_HEAP=1\u0000"},{"ServiceManifestName":"WordCountServicePkg","CodeP
ackageName":"Code","EntryPointType":"Main","DebugExePath":"C:\\Program Files (x86)\\Microsoft Visual Studio
14.0\\Common7\\Packages\\Debugger\\VsDebugLaunchNotify.exe","DebugArguments":" {2ab462e6-e0d1-4fda-a844-972f561fe751} -p
[ProcessId] -tid [ThreadId]","EnvironmentBlock":"_NO_DEBUG_HEAP=1\u0000"}]" }
The following cmdlet gets the services with a health state of error:
PS D:\ServiceFabric> Get-ServiceFabricApplication | Get-ServiceFabricService | where {$_.HealthState -eq "Error"}
ServiceName : fabric:/WordCount/WordCountService
ServiceKind : Stateful
ServiceTypeName : WordCountServiceType
IsServiceGroup : False
ServiceManifestVersion : 1.0.0
HasPersistedState : True
ServiceStatus : Active
HealthState : Error
Cluster and application upgrades
During a monitored upgrade of the cluster and application, Service Fabric checks health to ensure that everything remains healthy. If an entity is unhealthy as evaluated by using configured health policies, the upgrade applies upgrade-specific policies to determine the next action. The upgrade may be paused to allow user interaction (such as fixing error conditions or changing policies), or it may automatically roll back to the previous good version.
During a cluster upgrade, you can get the cluster upgrade status. The upgrade status includes unhealthy evaluations, which point to what is unhealthy in the cluster. If the upgrade is rolled back due to health issues, the upgrade status remembers the last unhealthy reasons. This information can help administrators investigate what went wrong after the upgrade rolled back or stopped.
Similarly, during an application upgrade, any unhealthy evaluations are contained in the application upgrade status.
The following shows the application upgrade status for a modified fabric:/WordCount application. A watchdog reported an error on one of its replicas. The upgrade is rolling back because the health checks are not respected.
PS C:\> Get-ServiceFabricApplicationUpgrade fabric:/WordCount
ApplicationName : fabric:/WordCount
ApplicationTypeName : WordCount
TargetApplicationTypeVersion : 1.0.0.0
ApplicationParameters : {}
StartTimestampUtc : 4/21/2017 5:23:26 PM
FailureTimestampUtc : 4/21/2017 5:23:37 PM
FailureReason : HealthCheck
UpgradeState : RollingBackInProgress
UpgradeDuration : 00:00:23
CurrentUpgradeDomainDuration : 00:00:00
CurrentUpgradeDomainProgress : UD1
NodeName : _Node_1
UpgradePhase : Upgrading
NodeName : _Node_2
UpgradePhase : Upgrading
NodeName : _Node_3
UpgradePhase : PreUpgradeSafetyCheck
PendingSafetyChecks :
EnsurePartitionQuorum - PartitionId: 30db5be6-4e20-4698-8185-4bd7ca744020
NextUpgradeDomain : UD2
UpgradeDomainsStatus : { "UD1" = "Completed";
"UD2" = "Pending";
"UD3" = "Pending";
"UD4" = "Pending" }
UnhealthyEvaluations :
Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%.
Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'.
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='a1f83a35-d6bf-4d39-b90d-28d15f39599b', AggregatedHealthState='Error'.
Unhealthy replicas: 20% (1/5), MaxPercentUnhealthyReplicasPerPartition=0%.
Unhealthy replica: PartitionId='a1f83a35-d6bf-4d39-b90d-28d15f39599b',
ReplicaOrInstanceId='131031502346844058', AggregatedHealthState='Error'.
Error event: SourceId='DiskWatcher', Property='Disk'.
UpgradeKind : Rolling
RollingUpgradeMode : UnmonitoredAuto
ForceRestart : False
UpgradeReplicaSetCheckTimeout : 00:15:00
Read more about the Service Fabric application upgrade.
Use health evaluations to troubleshoot
Whenever there is an issue with the cluster or an application, look at the cluster or application health to pinpoint what is wrong. The unhealthy evaluations provide details about what triggered the current unhealthy state. If you need to, you can drill down into unhealthy child entities to identify the root cause.
For example, consider an application unhealthy because there is an error report on one of its replicas. The following PowerShell cmdlet shows the unhealthy evaluations:
PS D:\ServiceFabric> Get-ServiceFabricApplicationHealth fabric:/WordCount -EventsFilter None -ServicesFilter None -DeployedApplicationsFilter None -ExcludeHealthStatistics
ApplicationName : fabric:/WordCount
AggregatedHealthState : Error
UnhealthyEvaluations :
Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%.
Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'.
Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%.
Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Error'.
Unhealthy replicas: 20% (1/5), MaxPercentUnhealthyReplicasPerPartition=0%.
Unhealthy replica: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', ReplicaOrInstanceId='131444422260002646', AggregatedHealthState='Error'.
Error event: SourceId='MyWatchdog', Property='Memory'.
ServiceHealthStates : None
DeployedApplicationHealthStates : None
HealthEvents : None
You can look at the replica to get more information:
PS D:\ServiceFabric> Get-ServiceFabricReplicaHealth -ReplicaOrInstanceId 131444422260002646 -PartitionId af2e3e44-a8f8-45ac-9f31-4093eb897600
PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600
ReplicaId : 131444422260002646
AggregatedHealthState : Error
UnhealthyEvaluations :
Error event: SourceId='MyWatchdog', Property='Memory'.
HealthEvents :
SourceId : System.RA
Property : State
HealthState : Ok
SequenceNumber : 131444422263668344
SentAt : 7/13/2017 5:57:06 PM
ReceivedAt : 7/13/2017 5:57:18 PM
TTL : Infinite
Description : Replica has been created._Node_2
RemoveWhenExpired : False
IsExpired : False
Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM
SourceId : MyWatchdog
Property : Memory
HealthState : Error
SequenceNumber : 131444451657749403
SentAt : 7/13/2017 6:46:05 PM
ReceivedAt : 7/13/2017 6:46:05 PM
TTL : Infinite
Description :
RemoveWhenExpired : False
IsExpired : False
Transitions : Warning->Error = 7/13/2017 6:46:05 PM, LastOk = 1/1/0001 12:00:00 AM
Note
The unhealthy evaluations show the first reason the entity is evaluated to current health state. There may be multiple other events that trigger this state, but they are not be reflected in the evaluations. To get more information, drill down into the health entities to figure out all the unhealthy reports in the cluster.
Next steps
Use system health reports to troubleshoot
Add custom Service Fabric health reports
How to report and check service health