I’ve spent a lot of time in the past 12 months working with the <verb>-CsUserData commandlets in SfB, and it’s been an interesting time.

This is part 3 in a series where I document the traps I’ve fallen into with the Import-CsUserData and Update-CsUserData commands in particular.

Part 1 – Update-CsUserData fails on bad data.
Part 2 – Your Front-End won’t start after Import-CsUserData imports bad data.
Part 3 – Update-CsUserData throws false red and runs slow if your paired pool is offline
Part 4 – You can’t trust Import-CsUserData to tell you if it fails.

Part 3 – Update-CsUserData throws false red and runs slow if your paired pool is offline

In this scenario we have two pools that are paired. For operational reasons one pool has been failed-over to the other, but all our activities are taking place on the ‘surviving’ pool. The user in question is homed to this pool.

Update-CsUSerData with the -TestMode switch reveals no problems, but we saw in Part 1 that it can’t be relied upon:

Full of false hope from the above, the command was repeated without -TestMode, where it paused for a while and “failed”:

Update-CsUserData : Exception from HRESULT: 0xC3EE7A1C
At line:1 char:1
+ Update-CsUserData -FileName .\Jess-2Buddies.zip -UserFilter jessica@gisandbox.co ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (jessica@gisandbox.com:String) [Update-CsUserData], COMException
    + FullyQualifiedErrorId : Microsoft.Rtc.Management.AD.Cmdlets.ImportOcsUserDataCmdlet

0 users were successfully imported. 1 users were failed to import. 0 users were ignored. 0 users were not processed

It didn’t fail. Mostly*

Despite the error message, the command DID actually appear to complete successfully. When the user signed out and back in again, their contacts updated to correctly show only those that were in the update file – as intended.

What actually failed was the attempt to update the backup copy of the user’s data on the paired pool. This is confirmed if you run a CLS Logger trace of the “PowerShell” scenario when the command executes:

The first of those two errors reports a timeout:

TL_ERROR(TF_COMPONENT) [SEFE03\SEFE03]1E14.29CC::02/15/2020-03:17:18.679.00002AC6 (UserServices,CResAttrChangeContext::OnTimeout:ResAttrChangeContext.cpp(32)) 
( 0000008F4B34E920 ) Res attr change http request to server ([SEFE02.gisandbox.com]) for user [jessica@gisandbox.com] timed out.

.. and the second is the clincher:

TL_ERROR(TF_COMPONENT) [SEFE03\SEFE03]1E14.3728::02/15/2020-03:17:18.679.00002AC8 (UserServices,CSprocSetMoveResourceDataXml::SetMoveResourceData:AdminSprocs.cpp(11)) 
Failed to update pool info on at least one FE in target pool ([SEFE03.gisandbox.com]) or its backup pool ([SEFE02.gisandbox.com]) for user [jessica@gisandbox.com].

 
In my customer’s case this timeout added approximately 45s to each iteration, which was a show-stopper for us given the thousands of users for whom we needed to import their contacts. This ultimately forced us to switch to Import-CsUserData, but not before we’d logged a ticket with Microsoft Support.

Microsoft’s Guidance

Microsoft’s support rep was quickly able to replicate our “error” and confirmed it’s a known issue with Update-CsUserData and pool pairing.

They went further to say that another known issue with Update-CsUserData is that if the client registers to a different FE server, they will see a different list of contacts, as apparently the users contacts aren’t always copied correctly from the primary to their secondary replicas.

In production we followed this recommendation and abandoned Update-CsUserData in favour of Import-CsUserData.

My own attempts testing Update-CsUserData in an Azure lab haven’t been able to reproduce the reported issue, but I’m prepared to take Support’s word for it and steer clear. Given the issue is one of replication between the user’s primary and secondary registrars, you’ll be unlikely to ever see the issue in a Standard Edition or single-server EE pool, and this is the only scenario in which I’m comfortable using this commandlet any more.

Reproduction Steps

In this demonstration of the issue, I was trying to update one user’s contacts.

The host in this demo is an SfB 2015 Standard Edition server patched to CU10HF1, however in production this has been confirmed to affect EE pools as well.

1. Invoke-CsPoolFailOver -PoolFqdn <FQDN>…
2. Wait for the dust to settle
3. Export-CsUserData -UserFilter <User> -LegacyFormat …
4. Edit the file as appropriate
5. Convert-CsUserData -UserFilter <User> -TargetVersion Current -InputFile …
6. Update-CsUserData -FileName <Filename.ZIP> -UserFilter <User> -TargetPoolFqdn <FQDN> -TestMode
7. No errors
8. Update-CsUserData -FileName <Filename.ZIP> -UserFilter <User> -TargetPoolFqdn <FQDN>
9. ~14 seconds later, an error indicating the command had failed

Revision History

15th February 2020: This is the initial publication.

– G.