The NetworkDirect architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. The NetworkDirect service provider interface (SPI) defines the interface that NetworkDirect providers implement to expose their hardware capabilities to applications.
NetworkDirect client applications use Component Object Model (COM) interfaces to communicate with providers. The NetworkDirect client applications use only the core COM programming model, not the COM runtime. COM interfaces provide flexible extensibility through the IUnknown::QueryInterface method, allowing providers to return any other interface they desire, to expose hardware-specific features easily. NetworkDirect providers do not register their objects in the system because they are not true COM objects.
There is no support for marshaling in the NetworkDirect SPI architecture, and the COM interfaces that are exposed by a provider are always instantiated in-process. This model is similar to the driver model that is used by the Windows Driver Foundation User Mode Driver Framework (UMDF).
The NetworkDirect SPI framework defines the following interfaces:
- IND2Provider: Represents a service provider. IND2Adapter objects are instantiated through the IND2Provider::OpenAdapter method.
- IND2Overlapped: Base class for objects on which overlapped operations can be performed.
- IND2Adapter: Interface to a service provider’s hardware adapter instance.
- IND2CompletionQueue: Interface to a completion queue instance. Created through the IND2Adapter::CreateCompletionQueue method.
- IND2MemoryRegion - Interface to a memory region, which represents a local buffer, registered with an adapter instance. Created through the IND2Adapter::CreateMemoryRegion method.
- IND2SharedReceiveQueue: Interface to a shared receive queue, used to aggregate receive buffers between queue pairs. Created through the IND2Adapter::CreateSharedReceiveQueue method.
- IND2QueuePair: Interface to a queue pair instance, which is used to perform I/O operations. Created through the IND2Adapter::CreateQueuePair or IND2Adapter::CreateQueuePairWithSrq method.
- IND2Connector: Interface to a connector instance that is used to manage connection establishment for IND2QueuePair objects. Created through the IND2Adapter::CreateConnector method.
- IND2Listener: Interface to a listen request. Created through the IND2Adapter::CreateListener method.
NetworkDirect SPI applications support zero, one, or more service provider libraries. NetworkDirect provider management follows the established mechanisms defined for Windows Sockets Direct providers and traditional layered service providers.
To register your provider, call the WSCInstallProvider function. The following fields of the WSAPROTOCOL_INFO structure identify the provider as a NetworkDirect provider.
dwServiceFlags1:
- XP1_GUARANTEED_DELIVERY
- XP1_GUARANTEED_ORDER
- XP1_MESSAGE_ORIENTED
- XP1_CONNECT_DATA
dwProviderFlags:
- PFL_HIDDEN - Prevents the WSAEnumProtocols function from returning the provider in the enumeration results.
- PFL_NETWORKDIRECT_PROVIDER (0x00000010) - Identifies the provider as a NetworkDirect provider.
iVersion: 2
The current version of the NetworkDirect SPI.
iAddressFamily: AF_INET or AF_INET6
Providers that support IPv4 and IPv6 addresses do not need to register their provider multiple times. A single registration as AF_INET6 is sufficient. The NetworkDirect SPI framework can support IPv4 and IPv6 addresses that are reported simultaneously.
iSocketType: -1 (0xFFFFFFFF)
iProtocol: 0
No protocols are defined by the NetworkDirect SPI architecture.
iProtocolMaxOffset: 0
To get the NetworkDirect providers that are registered on the computer, call the WSCEnumProtocols function. For each protocol that the function returns, compare the members of the WSAPROTOCOL_INFO structure to those that are specified in the above section. If the member values match, the protocol can be a NetworkDirect provider.
To determine if the provider is a NetworkDirect service provider, you must try to instantiate the IND2Provider interface.
The following steps show how to instantiate the provider:
- Load the provider's DLL. To get the path to the DLL, call the WSCGetProviderPath function by using the ProviderId member from the WSAPROTOCOL_INFO structure.
- Call the library's DllGetClassObject entry point to instantiate the IND2Provider interface.
Service provider libraries are expected to implement a DllCanUnloadNow entry point to allow clients to unload service provider libraries when they are no longer in use. Providers must also be able to handle multiple calls for the IND2Provider interface.
Now that you have a provider, you need to determine if the provider supports the IP address that you want to use. To determine the list of IP addresses that the provider supports, call the IND2Provider::QueryAddressList method. IND2Provider interface describes details about getting an interface to the NetworkDirect adapter that you want to use.
If you know the IP address of the NetworkDirect adapter that you want to use, call the IND2Provider::OpenAdapter method. If not, you first need to determine the local IP address of the adapter that will give you the best access to the destination address. For example, you could call the GetBestInterfaceEx function to get the local address.
After getting the local IP address, call the QueryAddressList method and enumerate the IP addresses that the provider supports. If the provider supports your local IP address, call the IND2Provider::OpenAdapter method to get an interface to the adapter.
Many operations in the NetworkDirect SPI take an OVERLAPPED structure as input. These operations are expected to provide the same functionality as traditional Win32 asynchronous I/O. Coupled with exposing the adapter’s underlying file handle on which asynchronous operations are performed, this design provides flexibility for applications to handle operations as best suits them. For example, an application can register the underlying file handle with an IOCP. All notifications, for example CQ notifications, are delivered by completing a request for notification that includes an OVERLAPPED structure. As such, providers never invoke callbacks into client code. All notifications are explicitly requested by the client.
Functions that take a pointer to an OVERLAPPED structure will always receive a valid pointer, never nullptr.
Note that the ND providers internally call SetFileCompletionNotificationModes with both FILE_SKIP_COMPLETION_PORT_ON_SUCCESS and FILE_SKIP_SET_EVENT_ON_HANDLE set. As a result, the only return value that will cause a notification to be reported is if the operations return ND_PENDING. If the operations return ND_SUCCESS, then the request is complete and the overlapped will not be used/notified, or if you use IOCP you will not see it reported there. All error values from the operations indicate immediate failure, so there will not be an async notification since the request failed to submit. It is necessary for the application to handle two distinct code paths where an operation can either return successfully or has to be completed asynchronously.
The current design for asynchronous operations places the burden of handling synchronous calls on the clients. As such, providers will always see asynchronous calls whenever an overlapped operation is possible.
Providers are likely to have synchronous commands (such as CreateCompletionQueue), and they will likely handle the synchronous I/O internally or issue the request on a different file handle. If the provider uses two file handles, one for asynchronous and the other for synchronous calls, it may be simpler for provider libraries to handle the synchronous requests (where pOverlapped == nullptr) rather than having the client do so for them.