Later on this year, AMD is going to release big refreshes to both their CPU and GPU lines, with new architectures (codenamed Zen and Polaris respectively) and a jump to a smaller manufacturing node.
Their CPUs are expected to see the biggest jump in years, as they're completely ditching the Bulldozer architecture that they've had since 2011. The Bulldozer architecture (combined with their manufacturing node lag) have resulted in AMD's CPUs dropping significantly behind Intel's over the past few years, particularly in gaming performance. The new Zen architecture, however, promises to bring substantial performance improvements, and although it's hard to gauge without any actual hardware tests, it's estimated to just about match Ivy Bridge (Intel's 2012 architecture) in terms of IPC (instructions per clock).
While matching a four year old processor might not sound like much, it would actually be quite a big jump for AMD. Intel's desktop chips have only been improving by about 5% a year since then, and even our very own "I Need A New PC" thread extols the virtues of the Intel i5 2500K right in its title, a Sandy Bridge CPU which released a year before Ivy Bridge. While a new architecture and a node shrink might bring AMD into the same ballpark as Intel, the advantage that promises to make AMD's CPUs actually competitive in gaming is the report that they'll initially sell the CPUs without integrated graphics. This means smaller, cheaper dies, which means they should be able to offer decent price/performance ratio in the entry to mid-level market. It's unlikely that they'll provide much competition at the high end, but actually having viable CPUs for gaming PCs at all would be a big enough deal as it is.
On the GPU side of the coin, while architectural improvements are expected to give the cards a bit of a boost, it's really the manufacturing improvement (jumping two nodes from 28nm to 16nm/14nm) that will provide the big jump, allowing them to squeeze more transistors onto a die, reach higher clockspeeds and reduce both power consumption and heat. Nvidia will be getting exactly the same benefits at the same time, but the node shift should jump-start the graphics card market, after 4 years of both companies trying to squeeze what they could out of the 28nm process.
With all this, though, AMD will still have a steep journey ahead of them if they want to regain significant marketshare in the gaming sector. Gamers have been (admittedly justifiably) ignoring their CPUs for years now, and even though they've been broadly competitive with nVidia across most price segments in the graphics card market for the past few years, their market share has done almost nothing but drop.
Basically, AMD has a huge branding problem. Even if they knocked it out of the park with both Zen and Polaris, it would still be a long time before system builders and OEMs start thinking about AMD as their first choice, rather than just an alternative to Intel and nVidia. If AMD really wants to show that there's a big market for their parts in gaming PCs, they should do so directly, by building their own line of console form-factor gaming PCs and selling them directly to customers.
This kind of step from building components to building entire systems isn't rare; Microsoft jumped into building PCs with the Surface line, which has turned out to be a big success for them, both in terms of brand building and the profitability of the product line itself. On a smaller scale, Intel now sells both NUC mini-PCs and Compute Sticks, and both seem to be modest successes.
The success of these kinds of endeavours depend on two criteria: the first is the ability to actually build a solid system in the first place, and the second is a lack of OEMs in the market who would be dissatisfied with a supplier stepping on their toes. The first of these shouldn't be a big hurdle to jump for AMD, they obviously already make the CPUs and GPUs, and the only components that would need any R&D would be the motherboard, cooling system and case (for the first two they should have ample expertise in-house, and for the last they would hire an industrial design firm). The second is basically a given. The console-sized gaming PC market is still small, and is dominated by the Alienware Alpha, which doesn't contain a single AMD component. Even the less popular options for compact gaming PCs are overwhelmingly Intel/nVidia affairs.
The question then becomes, can AMD make gaming PCs that offer better performance than the competition (ie Alienware Alpha) at a similar price? So long as Zen manages to be at least mildly competitive as a gaming CPU, then absolutely. OEMs like Dell have to make certain concessions to make a compact gaming PC, the most important one being the GPU. Compared to the X51, which uses full desktop graphics cards, but pushes the limits of being called "console size", the Alpha instead uses a laptop GPU, which can be soldered directly to the motherboard, bringing the size down substantially. The trade-off for this, though, is that laptop GPUs have much lower performance for the same price than their desktop counterparts. AMD could side-step this issue entirely by soldering full-fat desktop GPUs straight to the motherboard. The case would need to be larger than the Alpha's to accommodate the cooling system, but there's no reason they couldn't keep it smaller than the PS4, even with a high-end 16nm GPU.
In addition to this, AMD have the obvious financial benefit of building the most expensive components themselves. Furthermore, if AMD treat the project as principally a brand building exercise, they could push their own profit margins down (for at least the first few years) to much lower than any OEM could afford. If well-managed by AMD there's no reason to believe they couldn't end up competing on performance per dollar terms with any pre-built gaming PC full stop, let alone console-sized.
Building small gaming PCs is pretty much the only area where AMD have a technical competitive advantage over the rest of the industry. By combining their own desktop CPUs and GPUs on a compact motherboard and slotting it inside a decent looking case they'd have a product that no-one else is offering, and there's no reason they shouldn't be able to find a decent degree of success with it if the price is right. Most importantly, though, they need to start improving their brand image, and if they can place themselves as firmly the first choice in a single market (even one that's currently as small as compact gaming PCs are), then that would provide a sizeable boost to their reputation, which should bleed over into other markets.
Their CPUs are expected to see the biggest jump in years, as they're completely ditching the Bulldozer architecture that they've had since 2011. The Bulldozer architecture (combined with their manufacturing node lag) have resulted in AMD's CPUs dropping significantly behind Intel's over the past few years, particularly in gaming performance. The new Zen architecture, however, promises to bring substantial performance improvements, and although it's hard to gauge without any actual hardware tests, it's estimated to just about match Ivy Bridge (Intel's 2012 architecture) in terms of IPC (instructions per clock).
While matching a four year old processor might not sound like much, it would actually be quite a big jump for AMD. Intel's desktop chips have only been improving by about 5% a year since then, and even our very own "I Need A New PC" thread extols the virtues of the Intel i5 2500K right in its title, a Sandy Bridge CPU which released a year before Ivy Bridge. While a new architecture and a node shrink might bring AMD into the same ballpark as Intel, the advantage that promises to make AMD's CPUs actually competitive in gaming is the report that they'll initially sell the CPUs without integrated graphics. This means smaller, cheaper dies, which means they should be able to offer decent price/performance ratio in the entry to mid-level market. It's unlikely that they'll provide much competition at the high end, but actually having viable CPUs for gaming PCs at all would be a big enough deal as it is.
On the GPU side of the coin, while architectural improvements are expected to give the cards a bit of a boost, it's really the manufacturing improvement (jumping two nodes from 28nm to 16nm/14nm) that will provide the big jump, allowing them to squeeze more transistors onto a die, reach higher clockspeeds and reduce both power consumption and heat. Nvidia will be getting exactly the same benefits at the same time, but the node shift should jump-start the graphics card market, after 4 years of both companies trying to squeeze what they could out of the 28nm process.
With all this, though, AMD will still have a steep journey ahead of them if they want to regain significant marketshare in the gaming sector. Gamers have been (admittedly justifiably) ignoring their CPUs for years now, and even though they've been broadly competitive with nVidia across most price segments in the graphics card market for the past few years, their market share has done almost nothing but drop.
Basically, AMD has a huge branding problem. Even if they knocked it out of the park with both Zen and Polaris, it would still be a long time before system builders and OEMs start thinking about AMD as their first choice, rather than just an alternative to Intel and nVidia. If AMD really wants to show that there's a big market for their parts in gaming PCs, they should do so directly, by building their own line of console form-factor gaming PCs and selling them directly to customers.
This kind of step from building components to building entire systems isn't rare; Microsoft jumped into building PCs with the Surface line, which has turned out to be a big success for them, both in terms of brand building and the profitability of the product line itself. On a smaller scale, Intel now sells both NUC mini-PCs and Compute Sticks, and both seem to be modest successes.
The success of these kinds of endeavours depend on two criteria: the first is the ability to actually build a solid system in the first place, and the second is a lack of OEMs in the market who would be dissatisfied with a supplier stepping on their toes. The first of these shouldn't be a big hurdle to jump for AMD, they obviously already make the CPUs and GPUs, and the only components that would need any R&D would be the motherboard, cooling system and case (for the first two they should have ample expertise in-house, and for the last they would hire an industrial design firm). The second is basically a given. The console-sized gaming PC market is still small, and is dominated by the Alienware Alpha, which doesn't contain a single AMD component. Even the less popular options for compact gaming PCs are overwhelmingly Intel/nVidia affairs.
The question then becomes, can AMD make gaming PCs that offer better performance than the competition (ie Alienware Alpha) at a similar price? So long as Zen manages to be at least mildly competitive as a gaming CPU, then absolutely. OEMs like Dell have to make certain concessions to make a compact gaming PC, the most important one being the GPU. Compared to the X51, which uses full desktop graphics cards, but pushes the limits of being called "console size", the Alpha instead uses a laptop GPU, which can be soldered directly to the motherboard, bringing the size down substantially. The trade-off for this, though, is that laptop GPUs have much lower performance for the same price than their desktop counterparts. AMD could side-step this issue entirely by soldering full-fat desktop GPUs straight to the motherboard. The case would need to be larger than the Alpha's to accommodate the cooling system, but there's no reason they couldn't keep it smaller than the PS4, even with a high-end 16nm GPU.
In addition to this, AMD have the obvious financial benefit of building the most expensive components themselves. Furthermore, if AMD treat the project as principally a brand building exercise, they could push their own profit margins down (for at least the first few years) to much lower than any OEM could afford. If well-managed by AMD there's no reason to believe they couldn't end up competing on performance per dollar terms with any pre-built gaming PC full stop, let alone console-sized.
Building small gaming PCs is pretty much the only area where AMD have a technical competitive advantage over the rest of the industry. By combining their own desktop CPUs and GPUs on a compact motherboard and slotting it inside a decent looking case they'd have a product that no-one else is offering, and there's no reason they shouldn't be able to find a decent degree of success with it if the price is right. Most importantly, though, they need to start improving their brand image, and if they can place themselves as firmly the first choice in a single market (even one that's currently as small as compact gaming PCs are), then that would provide a sizeable boost to their reputation, which should bleed over into other markets.