Non-US dates make for Unhealthy OOS Farms

You’d think by now Microsoft would have stopped hard-coding its products to expect the US date format “MMDDYY”, but apparently the team behind Office Online Server 2016 are still holding out – and as a result you might find your OOS farm remains steadfastly “Unhealthy”, despite your best efforts.

OOS-Unhealthy

If you’re not in the US (i.e. are running a different time and date format) and none of Wictor Wilén’s many fixes have borne fruit, try this one.

The telltale sign this time is the presence of one or both of these events in the Applications and Services Logs / Microsoft Office Web Apps event log, spewing forth roughly every 4 minutes:

41268:

OOS-Event41268

<HealthMessage>UlsControllerWatchdog reported status for UlsController in category 'Verify Trace Logging'.
Reported status: Likely caused by shipping osipartner outputs from somewhere other than a devmain checkpoint.
Trace string is found but the format is not correct in C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS.
The following fields has the wrong format or content: [ Timestamp ].</HealthMessage>

21268:

OOS-Event21268

<HealthMessage>UlsControllerWatchdog reported status for UlsController in category 'Verify Trace Logging'.
Reported status: Likely caused by shipping osipartner outputs from somewhere other than a devmain checkpoint.
Trace string is found but the format is not correct in C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS.
The following fields has the wrong format or content: [ Timestamp ].</HealthMessage>

The Fix

The fix involves at least temporarily changing the date format back to the US.

  1. Open the Control Panel and select Region
  2. On the Formats tab, change the format to “English (United States)” & Click Apply:
    OOS-RegionFormatEnglishUS
    Trivia: this change appears to be what kills 21268.
  3. Select the Location tab. Change the “home location” to tell it you’re in the US & Click Apply:
    OOS-RegionLocationUS
    Trivia: this change appears to be for 41268.
  4. Select the Administrative tab. Under “welcome screen and new user accounts” click “Copy Settings”:
    ControlPanel-Region-AdminCopySettings
  5. Check “welcome screen and system accounts” and click OK:
    OOS-RegionAdminWelcomeScreen
  6. Click OK to close the Control Panel.
  7. Bounce the OOS service:
    Restart-Service WACSM
  8. Go make a coffee. By the time you’re back it should be Healthy. If you’re a fast drinker it may take 2 coffees.
  9. At this stage you can repeat steps 1 -3 (only) to revert the Format back to “Match Windows display language” and your Location back to home, restoring your locale’s native display formats.

Reproduction Steps

As demonstrated here, the issue was reproduced on a new machine thus:

  • Build new Server 2016 with Australia as the time and currency format:

    Server2016-TimeAndCurrencyFormat

  • Join to domain
  • Apply Windows updates
  • Apply some more
  • Add OOS pre-req’s using Pat’s script
  • Install OOS media (en_office_online_server_last_updated_november_2017_x64_dvd_100181876.iso)
  • Reboot
  • Patch OOS to April/May 2018 (wacserver2016-kb4011025-fullfile-x64-glb.exe)
  • Reboot
  • Request / generate / apply internal cert
  • New-OfficeWebAppsFarm
  • Get-OfficeWebAppsFarm > Unhealthy
  • Confirm presence of events 21268 & 41268
  • Resolve as above. (Waited 15 minutes after restart-WacSM)
  • Get-OfficeWebAppsFarm > Healthy
  • Confirm the offending events are no longer logged

(NB: Not all reboots shown).

Credits

A big thanks to my colleague Mike S for sharing this fix with me.

Revision History

13th January 2019: This is the initial post.

 

– G.

2 Comments

  1. Hi Greg,

    I had the same issue with OOS+Windows 2012 R2 French and with the statman Web interface on Windows 2012 R2 French server.
    what I’ve done: everything you’ve published without changing the location: stayed “French”
    what I’ve noticed : after 8 minutes, the Get-OfficeWebAppsMachine showed “Healthy”. Error 41268 disappears and 21268 is now informational event
    If you look at the eventviewer, you can notice each 4 minutes the informational or error event 21268 (UlsControllerWatchdog) appends so OOS needs 2 UlsControllerWatchdog information events pool to be Healthy.
    StatisticsManager : if the OS isn’t US, the KHI view doesn’t show anything, with your fix the KHI is operational : same problem about date format…..
    Thanks

  2. I had this in a cluster of OOS servers and as usual your fix worked a treat.

    If you run into an issue where SandboxHost.exe restarts over and over again every 10 seconds with an error message “Service Proofing has exited, sleeping for 10 seconds before restarting it”

    Check you clicked apply in the Region Settings window before clicking “Copy Settings…” otherwise not all the settings are updated correctly.

Leave a Reply

Your email address will not be published.

... and please just confirm for me that you're not a bot first: Time limit is exhausted. Please reload the CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.