TECHNOTE 1059: Open Transport Application Performance, P4

On Improving Open Transport Network Application Performance

Improving Connection Turnaround

Managing the Connection Queue
A high performance server must be able to handle multiple connections simultaneously - and efficiently. How you manage the connection queue is a key component of this. The connection queue determines the number of outstanding connect indications or calls for a server's listening endpoints - i.e., connections that have neither been accepted via OTAccept nor rejected via OTSndDisconnect.
Under the XTI framework, managing the connection queue can be complicated. The following section discusses the problems - as well as solutions - that you might encounter.

Handling kOTLookErr
One consequence of setting up a connection-oriented, listening endpoint to handle multiple outstanding calls is being able to handle the (dreaded) Look Error, or kOTLookErr. The kOTLookErr occurs when another concurrent event has arrived on the same endpoint, and cannot be acted on until the blocking event is consumed.

The Problem
To help illustrate why kOTLookErr occurs, you may want to think of the listening endpoint stream head as an event FIFO. When you bind the listening endpoint, you specify the queue length of this FIFO. If you specify a queue length of greater than one, then multiple T_LISTEN or T_DISCONNECT events will queue up.
Certain rules apply when processing this queue. The rules which regulate your interaction with this queue are codified by the X/Open XTI specification. The following are the relevant rules you need to know in order to write your connection management code (in particular order):

A T_LISTEN event is cleared by OTListen.
A T_DISCONNECT event is cleared by OTRcvDisconnect.

A call is cleared by OTAccept, OTSndDisconnect, OTRcvDisconnect.
An OTListen can get a kOTLookErr because a T_DISCONNECT is on the top of the stream.
An OTAccept can get a kOTLookErr because a T_DISCONNECT or T_LISTEN is on the top of the stream.
An OTSndDisconnect can get a kOTLookErr because a T_DISCONNECT is on the top of the stream.

An OTRcvDisconnect cannot get a kOTLookErr.

An Example
Properly handling these connect requests requires you to be able to simultaneously process the number of calls that you specified in the queue length.
For example:

Assume you specified a qlen of 5 in the OTBind.
The client end sends you a connection request.
Your notifier receives a T_LISTEN event.
You invoke OTListen to get the call.
You do an OTAccept on the call.
OTAccept returns an kOTLookErr, because another T_LISTEN event is pending.
This can continue until you have filled the queue with 5 inbound connection requests, at which point Open Transport will automatically refuse any further connection requests on that endpoint.
We're going to set up a example scenario of accepting connections from our listening endpoint. Let's make the following assumptions:
You are using endpoints in asynchronous/blocking mode.
You have a queue length greater than 1.
You are handing connections off from a listening endpoint to acceptor endpoints.
You do most of the work inside your notifier routine.

The Traditional (OT 1.1) Solution
You can apply the following strategy to handle processing inbound connection requests.
The first thing you want to do is create a list (LIST) or array of call (CALL) instances large enough to handle your queue.

When your notifier gets a T_LISTEN event, call OTListen in a loop until you get a kOTNoDataErr or a kOTLookErr. For each OTListen which returns kOTNoError, queue the CALL to the LIST for later processing. If the result is kOTNoDataErr, you have already gotten all of the CALLs, so go to step #3. If the result is kOTLookErr, it is because of a T_DISCONNECT event. Go to step #2.
Do an OTRcvDisconnect and get the associated CALL's sequence number. Find that CALL in the LIST and delete it. Return to step #1.
If the LIST is not empty, take the first CALL and determine if you want to accept it. If you want to accept it, go to step #4. To reject it, go to step #5.
Call OTAccept on the CALL. If the result is kOTNoError, delete the CALL from the LIST and return out of the notifier. If the result is kOTLookErr, do a OTLook. If the look result is T_LISTEN, go to step #1. If the look result is T_DISCONNECT, go to step #2.
Call OTSndDisconnect on the CALL. If the result is kOTNoError, delete the CALL from the LIST and return out of the notifier. If the result is kOTLookErr, go to step #2, T_DISCONNECT is the only reason OTSndDisconnect can return a look error.
When your notifier gets a T_PASSCON, check to see if the T_ACCEPTCOMPLETE has already happened. If not, return out of the notifier. If so, go back to step #3.
Check this....
When your notifier gets a T_ACCEPTCOMPLETE, check the result. If the result is anything other than kOTNoError, you won't be getting a T_PASSCON event. Do the appropriate error handling, then return to step #3. If the result is kOTNoError, check to see if the T_PASSCON has already happened. If not, return out of the notifier. If so, go back to step #3.
When your notifier gets a T_DISCONNECTCOMPLETE, go back to step #3.
Alternatively, you could make step #3 a loop and handle all of the CALLs in the LIST simultaneously, but that makes handling the T_ACCEPTCOMPLETE / T_PASSCON events and LIST processing more complicated. In general, the added complexity may not be worth it.
If you don't handle everything inside your notifier, there's one gotcha: if you set a flag in your notifier and process the event back in your main thread, you must deal with the following type of synchronization issue:
There is a T_LISTEN on top of the queue hiding a T_DISCONNECT. Your notifier gets a T_LISTEN and you set a flag to handle it back in the main thread. The main thread does an OTListen, which clears the T_LISTEN and brings the T_DISCONNECT up to the top. Before the OTListen completes, your notifier will be interrupted with the T_DISCONNECT notification.
Once you understand how things work, handling kOTLookErr is not as perplexing as it might seem.

A Better (OT 1.1.1) Solution
OT 1.1.1 includes the tilisten module, it can be used to serialize the connection requests. Push module over TCP.. generates forward progress. Although it is still experimental, it has demonstrated much promise...
Make sure Alex knows that if he uses tilisten he either needs to require OT 1.1.1 or he needs to install the streams module himself. It is provided as a sample with the OT 1.1.1 sdk. I personally favor requiring OT 1.1.1 since the upgrade is free.
You might explain to him what tilisten really is (a STREAMS module serializing inbound connection requests so your OTAccept isn't interrupted by another inbound connect or disconnect). It's pretty simple and if he has a clear picture what it does he may be able to simplify code.
config = OTCreateConfiguration("tilisten,tcp");
Negotiate qlen on Bind
As mentioned at the beginning of this section, you specify the length of a listening endpoint's connection queue when you bind a connection-oriented service, such as TCP or ADSP. During the bind process, however, it is possible for the value of queue length to be negotiated by the endpoint. This length may be changed if the endpoint cannot support the requested number of outstanding connection indications. Therefore, it is important for your application not to assume that the value of qlen returned by the OTBind is the same as requested. You should always check it.

Streamlining Endpoint Creation
The time required to create and open an endpoint can directly impact connection set-up time. This is of prime concern for servers, especially HTTP servers, since they must manage high connection turnover rates. Here are three tips that will speed up endpoint creation.

Preallocating Endpoints
One cost of the transport independence provided by Open Transport is that the process of setting up STREAMS plumbing for each endpoint can be time-consuming.
Rather than incur this delay each time a connection is established, a server designed to handle multiple outstanding connections can preallocate a pool of open, unbound endpoints into an endpoint cache. When a connection is requested, you can quickly dequeue a ready-to-use endpoint from this cache, resulting in a decreased connection turnaround delay (e.g., 10 to 30 times faster). For example:

Open a number of endpoints with OTAsyncOpenEndpoint.
When the notifier receives the T_OPENCOMPLETE event, queue the returned EndpointRef to endpoint-cache.
When your accepting notifier receives the T_LISTEN event, you can dequeue an endpoint from the endpoint-cache and pass it to OTAccept. Thus, the only time you have to wait for a endpoint to be created is if the queue is empty, where you can allocate a block of endpoints.

Recycling Endpoints
You can use the endpoint-cache is to recycle endpoints when your connection is closed. Rather than call OTCloseProvider each time a connections terminates, cache the unbound endpoint. This recycles it for a later open request.
On receiving the T_DISCONNECT, unbind the endpoint with OTUnbind.
When your session endpoint receives the T_UNBINDCOMPLETE event, enqueue that endpoint into your endpoint-cache.
Optionally, to save memory, you can deallocate the endpoint when the endpoint-cache reaches some predetermined limit.

Cloning Configurations
Another tactic to consider is to create a prototype OTConfiguration with OTCreateConfiguration, then use OTCloneConfiguration to pass the OTConfiguration to the OTOpenEndpoint. You'll find that OTCloneConfiguration is approximately 5 times faster than OTCreateConfiguration.
Don't forget to free up the prototype OTConfiguration before you quit your application.

Client closes connection
Speed up server turnaround time by having client send FIN using OTSndDisconnect instead of having server do it. beacuse closer holds connections resources
refer to TCP State transition diagram.
-- http implemenation issue..
Handling Dead Clients
A properly designed server should be prepared to handle what happens when a remote client unexpectedly disappears. The problem of a disappearing (or crashed) client is further aggravated when the link has been flow-controlled. This is illustrated in the following scenario :

You are transmitting a large amount of data to a client.
Your transport provider enters a flow-control state.
The client crashes or becomes unreachable.
After a timeout your server decides to force a disconnect from that client and issues a disconnect request.
But under XTI the stream T_DISCON_REQ is a M_PROTO message and is thus also subject to flow control; this causes your link to hang.
Under XTI it is the application's responsibility to flush the stream before issuing a disconnect. The best way to force a flush is by sending the IFLUSH command to the stream head with an OTIoctl. This will result in your notifier receiving all T_MEMORYRELEASED events for outstanding OTSnd() callwith Ack Sends. You should then attempt to send a disconnect. Error checking this point it is possible that the connection will aleady be terminated As illustrated below:
	#include <stropts.h>
	error = OTIoctl(ep, IFLUSH,  (void*) FLUSHRW);
	if (error) OTUnBind(ep)
    .....
    
    NotifyProc( .... void* theParam)    // Notifier Proc    
     
     case kStreamIoctlEvent:             // I_FLUSH Ioctl has complete, so
        (void) OTSndDisconnect(ep,NULL); // attempt to send disconnect now..
        break;    ... 
    
TCP/IP - Avoiding 2 minute delay on bind
In order to prevent stale data from corrupting a new connection, TCP imposes a 2-minute timeout on a binding after a connection has closed, before allowing the same port to bound to again. This can be worked around by setting the IP_REUSEADDR option with an OTOptionManagement call.
This technique is further documented in Macintosh Technical Q&A: NW 28 - TCP Application Acquires Different Port Address After Relaunch.

[Prev] p. 1 2 3 4 5 6 7 8 [Next]