I tried this on my PC last weekend and it was as underwhelming as expected.
The app basically forces fullscreen mode for everything so if you open settings from within the Xbox app then it is fullscreen. And because the Settings window is fullscreen rather than a window, there is no way to close it, meaning you have to use Alt-Tab to bring up the apps browser then click the X in the preview window to close it and return to the Xbox app. Not a very user-friendly experience in my opinion. How would that work with a controller? I was frustrated with this using the mouse and keyboard!
Also, the Xbox app itself displays at 1920x1080 on my 2560x1440 display (surrounded by a black border on all sides, basically) with no option to make it use the fullscreen estate. It is very poorly laid out compared with, say, Steam's Big Picture mode which has nicer icons etc.
The biggest issue I have though is that it actually does not close background tasks at all to maximise resources for the Xbox app itself; it just forces a fullscreen app on top of Windows. When you close the app, you instantly return to the desktop with all previous background apps already running. As such, the entire experience is really no different from enabling Game mode from the Settings app and just maximising the previous Xbox app!!!
Very poor showing from Microsoft. It's like they just ported the app from the handheld with little thought for how it would look and run on an actual PC.