[release/10.0] Fix EH - throw from catch in UnmanagedCallersOnly

[janvorli]

Backport of #131163 to release/10.0

Customer Impact

• Customer reported - Stack overflow when throwing SEH exception in catch block inside UnmanagedCallersOnly callback #123579
• Found internally

There is an issue when an exception is thrown from catch in a method marked as UnmanagedCallersOnly on Windows. When such exception escapes the catch, it is not detected as going into CallEHFunclet, but it is thought to be escaping the UnmanagedCallersOnly method itself, thus going through a reverse pinvoke transition. That results in an incorrect handling of that exception. We don't detect the unwind is colliding with previous exception and we end up calling the same catch again. This results in an infinite loop and a stack overflow after a while.

Regression

• Yes - in .NET 9 when new exception handling was enabled by default
• No

Testing

Directed repro from the customer, regular coreclr / libraries tests.

Risk

Low. The change just fixes a condition that was incorrect for UnmanagedCallersOnly case.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Backport to release/10.0 addressing incorrect unwind classification when an exception escapes a catch inside an [UnmanagedCallersOnly] callback on Windows (leading to re-entering the same catch and eventual stack overflow). The PR adjusts reverse-P/Invoke detection during funclet unwinds and adds regression coverage in the exception interop tests.

Changes:

• Update CoreCLR EH stack-walk logic to ignore the parent method’s reverse-P/Invoke GC info when unwinding a funclet.
• Add new [UnmanagedCallersOnly] callbacks and corresponding regression tests for throwing native/managed exceptions from inside a catch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/tests/baseservices/exceptions/exceptioninterop/ExceptionInterop.cs	Adds new UnmanagedCallersOnly callbacks and intended regression tests for throw-from-catch scenarios (Windows-only).
src/coreclr/vm/exceptionhandling.cpp	Adjusts reverse-P/Invoke propagation detection when unwinding from funclets to avoid misclassifying EH transitions.

Comments suppressed due to low confidence (1)

src/tests/baseservices/exceptions/exceptioninterop/ExceptionInterop.cs:284

• This method looks like a new test, but it is missing a [Fact] attribute, so it won't be discovered/executed by the test runner.

    [PlatformSpecific(TestPlatforms.Windows)]
    [SkipOnMono("Exception interop not supported on Mono.")]
    public static void ThrowManagedExceptionFromCatchInUnmanagedCallersOnlyCallback()
    {

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[JulieLeeMSFT]

LGTM.