ntdll: On macOS x86_64, swap GSBASE between the TEB and macOS TSD when entering/leaving PE code.
x86_64 Windows and macOS both use %gs
to access thread-specific data (Windows TEB, macOS TSD). To date, Wine has worked around this conflict by filling the most important TEB fields (0x30
/Self
, 0x58
/ThreadLocalStorage
) in the macOS TSD structure (Apple reserved the fields for our use). This was sufficient for most Windows apps.
CrossOver's Wine had an additional hack to handle 0x60
/ProcessEnvironmentBlock
, and binary patches for certain CEF binaries which directly accessed 0x8
/StackBase
. Additionally, Apple's libd3dshared could activate a special mode in Rosetta 2 where code executing in certain regions would use the Windows TEB when accessing %gs
.
Now that the PE separation is complete, GSBASE can be swapped when entering/exiting PE code. This is done in the syscall dispatcher, unix-call dispatcher, and for user-mode callbacks. GSBASE also needs to be set to the macOS TSD when entering signal handlers (in init_handler()
), and then restored to the Windows TEB when exiting (in leave_handler()
).
Some changes to the syscall dispatcher were needed to ensure that the TEB is not accessed through %gs
while on the kernel stack (since a SIGUSR1 while on the kernel stack will result in GSBASE being set to the TSD).
I've tested this successfully on macOS 15 (Apple Silicon and Intel) and macOS 10.13 with several apps and games, including the cefclient.exe
CEF sample.
Encouragingly, in some simple tests I didn't see a noticeable performance regression from this MR.
There are drawbacks though:
- libraries which jump directly from PE code into Unix code (expecting that %gs is always pointing to the macOS TSD) will crash. Notable examples are D3DMetal and DXMT. These will need to be changed to use Unix calls.
- If Windows code uses the
syscall
instruction directly, the stack pointer likely needs to be valid (which is probably not true on Windows). This is due to the syscall dispatcher saving registers onto the user stack and having to call_thread_set_tsd_base
. I can't say I've ever seen direct syscalls done with an invalid%rsp
, but it seems like something anticheat code might do.
macOS does not have a public API for setting GSBASE, but the private _thread_set_tsd_base()
works and was added in macOS 10.12.
_thread_set_tsd_base()
is a small thunk that sets %esi
, %eax
, and does the syscall
: https://github.com/apple-oss-distributions/xnu/blob/8d741a5de7ff4191bf97d57b9f54c2f6d4a15585/libsyscall/custom/custom.s#L119.
The syscall instruction itself clobbers %rcx
and %r11
.
I've tried to save as few registers as possible when calling _thread_set_tsd_base()
, but there may be room for improvement there.
I also tested an alternate implementation strategy for this which took advantage of the expanded "full" thread state which is passed to signal handlers when a process has set a user LDT. The full thread state includes GSBASE, so GSBASE is set back to whatever is in the sigcontext on return (like every other field in the context). This would avoid needing to explicitly reset GSBASE in leave_handler()
.
This strategy was simpler, but I'm not using it for 2 reasons:
- the "full" thread state is only available starting with macOS 10.15, and we still support 10.13.
- more crucially, Rosetta 2 doesn't seem to correctly implement the GS.base field of the full thread state. It's set to 0 on entry, and isn't read on exit.