Supporting Peer-to-Peer Memory Devices

Logan Gunthorpe recently posted a patch to support peer-to-peer PCI memory devices. He explained that this is "a PCI card with a BAR space that points to regular memory. This may be an independent PCI card or part of another completely unrelated device (like an IB card or a NVMe card)." A base access register (BAR) is used for addressing memory on the device. Using PCI memory devices as regular system memory may increase the latency between memory transfers as compared with the primary RAM on the system motherboard, but it could come in very handy in cases where the primary RAM starts to be used up.

Logan, however, acknowledged there were caveats to this approach, particularly when dealing with older PCI devices. As a result, he explained, "The code is designed to only utilize the p2pmem device if all the devices involved in a transfer are behind the same PCI switch. Other cases may still work or be desirable for some end users but it was decided this would be the best course of action to prevent users enabling it and wondering why their performance dropped."

Sinan Kaya had some concerns about the portability of Logan's patch. He understood the desire to avoid unexplained performance hits, but he felt that requiring all devices to be behind the same switch was essentially arbitrary and should be left as a policy decision to be made by the user. He said, "I'd rather see the feature enabled by default without any assumptions. Using it with a switch is just a use case that you happened to test."

[...]