So, you now know the basics of approaching PCIe, and perhaps you have a PCIe-related goal in mind. Maybe you want to equip a single-board computer of yours with a bunch of cheap yet powerful PCIe WiFi cards for wardriving, perhaps add a second NVMe SSD to your laptop instead of that Ethernet controller you never use, or maybe, add a full-size GPU to your Raspberry Pi 4 through a nifty adapter. Whatever you want to do – let’s make sure there isn’t an area of PCIe that you aren’t familiar of.
You might have heard the term “bifurcation” if you’ve been around PCIe, especially in mining or PC tinkering communities. This is splitting a PCIe slot into multiple PCIe links, and as you can imagine, it’s quite tasty of a feature for hackers; you don’t need any extra hardware, really, all you need is to add a buffer for REFCLK. See, it’s still needed by every single extra port you get – but you can’t physically just pull the same clock diffpair to all the slots at once, since that will result in stubs and, consequently, signal reflections; a REFCLK buffer chip takes the clock from the host and produces a number of identical copies of the REFCLK signal that you then pull standalone. You might have seen x16 to four NVMe slot cards online – invariably, somewhere in the corner of the card, you can spot the REFCLK buffer chip. In a perfect scenario, this is all you need to get more PCIe out of your PCIe.
In reality, proper bifurcation support is messy. Bifurcation requires support from the host – and this support is quite situational, with only some chipsets being able to do bifurcation, and quite a few motherboards being able to do it but opting to not make it possible software-wise for whatever reason. Even if you’re making your own PCIe host with an FPGA and the sheer flexibility this implies, your FPGA might straight up not support bifurcation for internal reasons. What’s more, you can’t just get sixteen x1 links out of a single x16 slot – usually, the supported combination is four x4 links out of an x16 slot, maybe two x4 links out of an x8-wired x16 slot.
Now, there’s an alternative, and it’s PCIe switches – these are hefty chips which present a PCIe device interface to your host, and multiple PCIe host interfaces for all your devices; you could compare them to Ethernet switches in terms of how they work, except that lanes of a PCIe switch have either host or device roles and Ethernet switch ports are generally all equal. These switches are sized from a RP2040-style QFN (in case of ASM1182, a 1:2 switch) to the size of a x86 CPU, they will often need external cooling, they also tend to be pricey, and consume a decent amount of power. If you’re looking for bifurcation cards on say, Aliexpress, you will inevitably stumble upon cards with PCIe switches on them.
The benefit is – they don’t require special support from the host, since they look just like any regular PCIe device from a host’s perspective. The drawback – again, they cost a significant amount, and they’re also rarely documented. For instance, I was looking to get some PCIe 1:4 switch chips, and stumbled upon the ASM1184 1:4 controller, which produces four x1 links out of a single x1 link. This was just what I needed, and when googling, I found this chip being used in tons of Chinese PCIe splitter boards of all kinds, desktop motherboards from manufacturers like Gigabyte, with boards even sold in places like Walmart. The ASM118x (1:4 for ASM1184 and 1:2 for ASM1182) chips themselves are easy to find on Aliexpress, too – the main problem is that they lack documentation. Technically, a real-world device schematic is the bare minimum needed, since the chips seem to be meant to work standalone, without external control – I’ve found one for ASM1182, and my friend recently found one for ASM1184. However, things like I2C registers for external control will remain a mystery.
So, even though chips like the ASM1184 is tempting, cheap and is readily available on Aliexpress as both standalone IC and as part of products, debugging might get tricky. Thankfully, there’s one Western manufacturer I know, that will save your day when it comes to PCIe switches – it’s Diodes Incorporated, with their Pericom-acquired PCIe product line; you can openly download full datasheets for their PCIe switches, and there’s a good few on Digikey. You might also stumble upon PCIe switches from Broadcom, but as you might guess, their openness is par for the course for a typical Broadcom product – aka, non-existent.
It’s also quite interesting how some Aliexpress listings paint over the ASMedia marking
If you’re looking to pull PCIe through a cable and you’re okay with an x1 link, you could do worse than an USB3 cable – in fact, it’s a pretty suitable medium. There’s two high-speed USB3 diffpairs, and given that USB3 is a finicky interface when it comes to integrity requirements, the USB3-intended cabling works well for PCIe. USB3 cables also carry a USB2 pair – and the 100MHz REFCLK pair fits it perfectly; all in all, USB3 cables and PCIe x1 links are a match made in heaven. Mining equipment manufacturers have recognized this, flooding the market with all sorts of accessories that pull PCIe links through USB3 cables and connectors – slot extenders, PCIe switches, NVMe adapters and so on. Ever since mining has become less popular, these accessories are there for the taking, available on the cheap.
A typical mining riser board consists of an x1 plug board with a USB3 socket that is meant to plug into a mainboard, a x16 slot board with a corresponding USB3 socket, as well as extra power inputs, sometimes, also with some power regulators; the third component is a USB3 cable with USB-A connectors on both ends. For a hacker designing something unconventional with PCIe, reusing these accessories might be tempting – USB3 sockets and cables are easy to find in many varieties, after all.
Indeed, you will likely succeed – if anything can be proof that it’s absolutely viable to peruse PCIe in such a way, it’s that all these mining extenders have, indeed, been working for all the countless miners, in 24/7/365 conditions. However, watch out for a few hiccups. For instance, there’s one minor pinout hiccup that doesn’t occur often – the high-speed TX and RX diffpairs will stay on the USB3 pins, but which PCIe pair goes to which USB3 pair,isn’t quite guaranteed.
More importantly, though, the kind of signals put on +5V and GND (yes, a separate wire) wires of the USB2 part can differ – and this is known to happen. My friend has built a device with these wires, and the PCIe risers she had, were found to have PEWAKE and PERST, and the design has worked wonders. However, later on, a riser compatibility investigation on a Russian mining forum (translated) has found a few riser and switch boards where the +5V wire carried 3.3V, or even 12V instead – instead of either PERST or PEWAKE that you might expect if you design for the specific riser you own. If you try to design for riser equipment and you rely on either the “in-mainboard x1 plug” or the “x16 slot” boards out of a typical riser extender, you better make double sure that the ones you’re using, do indeed comply to the pinout you have in mind; using a dodgy riser can lead to an explosive situation. Sadly, these risers have almost defined a standard for carrying PCIe over convenient cables – but they’re not quite there.
Also, make sure to avoid the power adapters sometimes included with such miner kits – all those “SATA power to PCIe 6-pin” have a serious mismatch between input and output power; if there’s anything off with those mining boards, it’s the inclusion of these adapters, which can be a genuine fire hazard in practice, if your PCIe card happens to consume a bit more power than the adapter’s cheaply made SATA power connector can handle.
The hacker route of subverting mining accessories for PCIe shenanigan goals isn’t the only one. If you find a tasty chip that connects over PCIe, or you want to extend a PCIe link that you found somewhere through totally legitimate means, go on and design your own PCB! Nowadays, you have the choice between full-size slots for desktop use or M.2 slots for mobile use – and if you’re aiming at older or industrial equipment, mPCIe works too.
For desktop use, there’s an abundance of PCIe sockets to pick from. To plug into such a socket, you need a PCB of 1.6mm thickness – which is, thankfully, the standard PCB thickness that you can get from any fab, and is also the cheapest PCB thickness as a rule, at that! You will want to chamfer the PCB edge that plugs into the PCIe socket – your PCB fab can often do this at a bit of extra cost, but you can also add this chamfer post-factum with a file, or an xacto knife with a spare blade.
As for mobile slots, I’ve talked about M.2 extensively before. The simple M.2 guidelines are – use a 0.8mm PCBs with ENIG for a card, and make sure you have a stencil if you’re adding a M.2 slot to your PCB. Then, pick the keying that you’re comfortable using – A+E for WiFi cards, E sockets for connecting WiFi cards, M-key slots and cards for PCIe up to x4, and B+M cards for PCIe up to x2 and compatibility with both B-key (WWAN) and M-key (SSD) sockets. There’s plenty of M.2 sockets for sale, and templates for both slots and cards, too! For mPCIe, 1mm PCBs will work, and gold (ENIG) plating is, again, a must.
Whether you’re doing a mobile or a desktop PCIe card or socket, the diffpair layout guidelines, the few required signals, and the strong compatibility between link widths and generations stays the same. Whatever you’re designing with PCIe, is not a hard task to accomplish – and thanks to the resillience of PCIe aiding us in our high-speed forays, you are highly likely to achieve your PCIe project goals.
