White Paper

Expansion Connectors

Details on the connectors BittWare uses for expanding FPGA cards

BittWare’s expansion cables help customers address specialized challenges. The most common uses for expansion include:

Expansion Connector History:

BittWare’s first expansion connector used a Samtec SEARAY connector. It became very popular with customers. Our second generation expansion connector used 8x SlimSAS (SFF-8654).  We moved to OCuLink when it became a PCIe standard.  BittWare then migrated its expansion connectors to Amphenol Mini Cool Edge IO (MCIO) for FPGAs with PCIe Gen 4 and above.

Connector Standards Related to OCuLink:

The name “OCuLink” is a PCIe signal mapping standard.  The name used in the mechanical standard is MiniLink or SFF-8612. The Molex implementation is known as Nano-Pitch I/O.  Whatever the name, it is available in two widths.

  • SFF-8612 4x receptacle
  • SFF-8612 8x receptacle

The mechanical specification for the above is here:  https://members.snia.org/document/dl/26705  The Molex product page is https://www.molex.com/molex/products/family/nanopitch_io_interconnect_system

Note that the cable side of the connector system is SFF-8611. From the cable’s perspective,  the connector width is specified as 4i and 8i instead of 4x and 8x.   The 4i cables do not directly plug into 8x receptacles. 

The 8x SlimSAS (SFF-8654) connector that BittWare used for PCIe before OCuLink has a vendor-specific pin mapping into PCIe that is documented in SFF-9402.  That connector remains widely deployed.

Connector Standards Related to MCIO:

Late in 2020, the PCI-SIG issued a Request for Proposal to help them select at least two connector standards for PCIe Gen 5 and Gen 6. The RFP asked for a connector optimized for use inside a chassis and a second optimized for interconnecting chassis. The PCIe-SIG standards work continues.  However, just like last time, BittWare could not wait for the PCI-SIG’s decision. We adopted MCIO based upon conversations with our early Gen 4 customers. 

Just like we did with SlimSAS,  we are using a vendor-specific pin mapping into PCIe that is derived from SFF-9402.  It is possible that the PCIe-SIG will ultimately select a different connector.

The MCIO connectors themselves are formal standards:

  • SNIA SFF-TA-1016 covers the cable and connectors
  • SNIA SFF-TA-1024 is the test specification

PCIe Over Cable Standards:

The PCIe SIG maintains two cabling standards. OCuLink is the name of the standard for cables optimized for use inside a chassis.  It leverages the SFF-8611/8612 connector standards.  Cable lengths inside a chassis are roughly restricted to about 0.5 meter without re-timer chips in the path.

“External Cabling” is the name of the standard for longer cables optimized for use between chassis. The latest revision of that standard leverages HD Mini SAS (SFF-8643/8644) connectors. Its implementation generally requires retiming between an FPGA and the cables, often implemented with a two port configuration of a PCIe switch chip.

Both standards define PCIe Gen 3, 1-wide and 4-wide mappings to a 4i/4x connector system.  They supplement that with “cable aggregation” specifications to create PCIe 8-wide and 16-wide mappings.  BittWare’s example cable projects support this dynamic change in PCIe bus width depending upon the cable used.

It is important to note that one side of a PCIe link will always be a root complex and the other side an end point.  This reality makes it complicated to build an FPGA card where a single connector could be either a “root” or an “end point”, depending upon the application.  For some cards, users must specify either “root” or “end point” when the card is assembled.  For most of our cards we only support the “root” configuration.

Also, the automatic link equalization phase of PCIe power-up is often insufficient when applied to cables.  Thus an FPGA is expected to read a PROM found in OCuLink and “External Cabling” cables. The PROM provides hints that lead to the FPGA inserting better starting values before automatic equalization begins.  This is very similar to what an FPGA is expected to do to support DAC cables plugged into QSFP cages to run Ethernet. The PROM’s content is defined by the PCIe standards but access to the PROM is based upon SFF-8449 here: https://members.snia.org/document/dl/26392

Unfortunately, not every OCuLink cable follows the standard and provides the PROM.  Thus connecting the OCuLink power pin that the PROM requires is a build option on some BittWare cards.

To download any PCI-SIG standard one must join the organization.

Note that non PCI-SIG standards Thunderbolt and USB 4 include a PCIe Tunneling protocol that has not yet impacted FPGA markets.

Power for PCIe Devices:

Cables that carry PCIe signals usually supply just enough 3.3 volt wattage to power active cables.  This means there isn’t any 5 volt power and the 3.3 volt does not pass along the cable to the far end.  Thus PCIe end points at the far end of the cable must supply their own power locally.

System Topology:

Customer systems in production will likely connect their FPGAs to PCIe switch chips. One such example is connecting the FPGA to a switched PCIe backplane with U.2 slots for NVMe drives.

However, we have seen non-production, transitional setups in our customer development labs. All of these options require a desktop computer with the top off and wires hanging out:

  • Connecting to a U.2 drive using a cable between the FPGA and the U.2 or U.3 connector, which also includes a separate power connector
  • Connecting an M.2 drive using a small circuit board with both PCIe and power connectors
  • Connecting to an external JBOD or PCIe expansion chassis using an adapter between the FPGA and MiniSAS HD connectors. This needs to be a single, short cable, not a chain with multiple cables.  In contrast, any customer “production” product with MiniSAS HD will likely require a PCIe switch chip immediately in front of the MiniSAS HD bulkhead connectors.