When I have two (or more) projects open – multiple tabs in one Scrivener instance – then enter composition mode, the composition mode project is covered by the regular Scrivener app.
It is possible to work with the composition mode project by clicking on it, when the Scrivener app will go into the background.
In addition, when exiting composition mode, the composition mode project always (re)attaches itself (to the main window) as the final (rightmost) tab instead of its original position.
Expected behaviour is for the composition mode project to have focus (and be frontmost) and for it to reattach itself in its original position.
Edit addition:
If you separate the tabs/projects before entering composition mode – with the intention of only using composition mode on an isolated project – this is also problematic. In this case, Scrivener combines all the projects (as tabs) after leaving composition mode even though the projects were separate before entering composition mode.
I’m unable to reproduce the issue you describe where the project window floats over the composition window, though I kind of did if I actually did float the project window (with Window ▸ Float Window). I don’t think that option is compatible with tab window usage. Even though it seems to work, I noticed that the system disables the View ▸ Show Tab Bar command when the active project window is set to float.
I did run into some issues with the Hide main window in composition mode option enabled, in Behaviors: Composition Mode, in conjunction with tabs. I didn’t quite get the same results you describe, but the projects would detach and become separate windows upon closing Composition. I presume whatever is hiding the project window is doing so in such a way that Apple never tested the tab feature to work with. It does hide the window differently than the typical ⌘M minimise. So try turning that off and see if that improves the situation. Tab placement and inclusion was quite stable for me with it disabled.
Opening a document as a quick reference, then switching to Composition Mode with that QR as the active window and the project window behind it, behaves as described. The QR is above the Composition Mode. It is also the window displayed in Composition Mode.