AI has its place in coding (which can be verified to be correct by proper usage of tests), and 2D art since that's subjective, but they simply cannot be trusted for most other tasks.
They constantly spit out nonsense, or "hallucinate" bullshit since they're forced to come to an answer or conclusion.
For 2D art, it's an ethical issue considering a lot of GenAI use LLM that have been trained on stolen artwork. If these companies weren't so greedy & stupid, and actually allowed artists to opt-in & paid them for use of their art, that would at least be some compensation. Some companies actually do that now, but it's only been more recent, and the damage in terms of stigma's been done.
I.e any real 2D artist who voluntarily provides work for an AI LLM will probably be shunned from their peer community and find it extremely difficult to build up an authentic fanbase, given the growing resentment among some pockets (especially in areas of art communities) towards AI. It's unfortunate, but the way these companies went about training their models in the first place is what enabled that environment to fester.
As for AI making stuff up, well yeah that's definitely a thing that happens. But a smart prompter is going to have some idea of what they are asking about and be able to correct the AI if it gives an incorrect response. If you're completely oblivious about what something is, and it's a complex subject, and you're asking the AI to answer for you (or even more than that, generate new solutions for something you know nothing about), to me that's a test on the prompter's level of critical thinking.
If the prompter isn't aware enough to recognize when a mistake's made, can't clarify what they're looking for, can't correct the AI's thinking behavior or set parameters, then they probably shouldn't be using an AI. Not until they get a basic understanding of the topic at least, and have some idea of what they are actually looking for.
What user data?
For game data, Nintendo for example has telemetry up the rectum on Switch. Almost every fart you make in-game is routed to HQ. Everything is already logged by everyone on every platform.
Here's a fun write-up:
https://zeldamods.org/wiki/Telemetry
They know exactly what you do and how you do it.
Things tied to your account I'd suspect, similar to the telemetry stuff with Nintendo you just posted. Disabling these types of things only stops them from actively pestering you and maybe ends some of the services, but the core program service(s) are still resident in memory. If it's a program with kernel-side access (which I'd expect Copilot to be such a program), then it's still getting some kind of telemetry even if you disable it.
You'd have to completely uninstall the program to keep it from doing that, but MS will never allow people to uninstall core Windows utilities like Copilot.
The hallucinations are going to hilarious. As are all the totally incorrect tips copilot is going to give.
I've really not used Copilot outside of maybe 1-2 prompts, and once was by accident. Cant' speak to its accuracy ; I'll say Google's is pretty accurate tho, atlhough it too can make mistakes, and sometimes I feel you have to "refresh" it on details established earlier so that it uses a proper context.
. I don't have to use it for work, thankfully, but for some of the spare time things I've been doing (basically to help flesh out technical aspects of fake/hypothetical consoles), I've made sure to document the shit out of everything. Taking care to phrase prompts as best as possible (i.e if the AI suddenly starts naming something a certain way, I try referencing the name it came up with going forward so that it can link context better) and catching discrepancies.