Why ESXi Server Could Not See (Shown as Inactive)iSCSI Datastore Any more?

Problem/Symptom

Why ESXi server could not see (shown as inactive) iSCSI Datastore Anymore?

After successfully setup iSCSI adaptor on VMware ESXi server and connect with QSAN storage system, QSAN datastore icon shows (Inactive) and had to remove iSCSI configuration with target LUN, manually reassign LUN then reboot, but still getting “Inactive” datastore.
(ESXi host loses connectivity to a VMFS5 datastore, while using the VAAI ATS heartbeat.)

Cause

The change VMware made in VMFS(VMFS3/VMFS5/VMFS6) hearbeat update method that leads to “Lost access to datastore”
An official troubleshooting guide was release by VMware August 2017;
ESXi host loses connectivity to a VMFS3 and VMFS5 datastore (2113956)

https://kb.vmware.com/s/article/2113956

Solution

To resolve this matter;
Revert the heartbeat to non-ATS mechanisms by disabling ATS heartbeat on ALL hosts sharing the datastore where these errors are seen.

Notes:

• If you suspect that ATS Heartbeat may be causing an issue with array workload or IO responsiveness, engage with your storage vendor to determine if they recommend disabling this function.
• This change disables or enables using ATS primitive for creating or updating VMFS heartbeat and does not change ATS primitive configuration itself.
• These operations can be safely performed online, while the storage is in use.

Caution:

Disabling ATS heartbeat:

• Does not impact Acquiring a HB slot (starting heartbeat).
• Periodic/Routine heartbeat updates are affected by this change.
• Reverts the heartbeat related activity in the ESXi host using plain SCSI reads and writes to update its heartbeat on VMFS datastores.

 

To revert the heartbeat to non-ATS mechanisms:

For VMFS5 and VMFS6 datastores:

To disable ATS heartbeat, run either the CLI command or the PowerCLI command:

• Command line:

# esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS5

• PowerCLI:

Get-AdvancedSetting -Entity VMHost-Name -Name VMFS3.UseATSForHBOnVMFS5 |
Set-AdvancedSetting -Value 0 -Confirm:$false

To enable ATS heartbeat, run either the CLI command or the PowerCLI command:

• Command line:

# esxcli system settings advanced set -i 1 -o /VMFS3/UseATSForHBOnVMFS5

• PowerCLI: 

Get-AdvancedSetting -Entity VMHost-Name -Name VMFS3.UseATSForHBOnVMFS5 |
Set-AdvancedSetting -Value 1 -Confirm:$false

To review the results of changing options, run this command:

For example:

# esxcli system settings advanced list -o /VMFS3/UseATSForHBonVMFS5 

You see output similar to:

Path: /VMFS3/UseATSForHBOnVMFS5
Type: integer
Int Value: 0 <--- check this value
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Use ATS for HB on ATS supported VMFS5 volumes

For VMFS3 datastores:

To disable ATS heartbeat, run either the CLI command or the PowerCLI command:

• Command line:

# esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS3

• PowerCLI:

Get-AdvancedSetting -Entity VMHost-Name -Name VMFS3.UseATSForHBOnVMFS3 |
Set-AdvancedSetting -Value 0 -Confirm:$false

To enable ATS heartbeat, run either the CLI command or the PowerCLI command:

• Command line:

# esxcli system settings advanced set -i 1 -o /VMFS3/UseATSForHBOnVMFS3

• PowerCLI:

Get-AdvancedSetting -Entity VMHost-Name -Name VMFS3.UseATSForHBOnVMFS3 |
Set-AdvancedSetting -Value 1 -Confirm:$false

Notes:

• This change takes effect immediately without reboot.
• This change does not affect ESXi host OS.
• The root node of these options is /VMFS3 regardless of the VMFS version. The last character of the option matches the corresponding VMFS version.

To review the results of changing options, run this command:

# esxcli system settings advanced list -o /path

For example:

# esxcli system settings advanced list -o /VMFS3/UseATSForHBonVMFS3 

You see output similar to:

Path: /VMFS3/UseATSForHBOnVMFS3 
Type: integer
Int Value: 0 <--- Check this value
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Use ATS for HB on ATS supported VMFS3 volumes

This reversion of VMFS heartbeat activity is preferred instead of globally disabling VAAI or ATS when using applicable storage systems. Although some storage systems require that the heartbeat-related activity be reverted to the legacy methodology, they still handle non-heartbeat-related ATS commands normally and there are dramatic performance and scale benefits to the use of ATS even if ATS should not be used for VMFS heartbeats.

Related Information

You experience these additional symptoms:

• In the /var/run/log/vobd.log file and Virtual Center Events, you see the VOB message:

Lost access to volume due to connectivity issues. Recovery attempt is in progress and the outcome will be reported shortly

• In the /var/run/log/vmkernel.log file, you see the message:

ATS Miscompare detected between test and set HB images at offset XXX on vol YYY

• In the /var/log/vmkernel.log file, you see similar error messages indicating an ATS miscompare:

2015-11-20T22:12:47.194Z cpu13:33467)ScsiDeviceIO: 2645: Cmd(0x439dd0d7c400) 0x89, CmdSN 0x2f3dd6 from world 3937473 to dev "naa.50002ac0049412fa" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

For full detail of this VMware documentation please visit VMware online knowledgebase;

Document id:2113956

https://kb.vmware.com/s/article/2113956

Please update VMware datastore related software to the latest version available when connecting to QSAN products for having best system stability.

Reference

VMware ESXi host loses connectivity to a VMFS3 and VMFS5 datastore (2113956): 
ESXi host loses connectivity to a VMFS3 and VMFS5 datastore (2113956).(n.d.). In VMware,  Inc.,Retrived December 19, 2017, from 

https://kb.vmware.com/s/article/2113956

Apply to

 All QSAN Products