Sockets, Protocols and Sending
This guide will explain some principles behind lNet when it comes to sending big amounts of data and how it comes together with protocols and sockets.
lNet currently consists of 4 parts:
1. The eventer
2. The sockets
3. The protocols
4. The sessions
The eventers and sessions both have their own guide chapter so I’ll skip them here.
The sockets are a basic OOP abstraction above the basic OS level handle which enables us to do basic operations, as well as keep track of states, options and event-readyness (think eventer). Now it is possible to send and receive data directly via sockets, but IT SHOULD NOT BE DONE. The problem is that some higher level protocols need to parse the received data so they can keep track of commands, states etc.
So, first thing to keep in mind is to NOT SEND OR RECEIVE USING SOCKETS DIRECTLY. Always use your protocol’s Send and Get methods instead. Another important thing to understand is that in lNet, sockets but even protocols are, if possible, 0-copy or bufferless. What does this mean then? It means that they do not save the received or sent data to buffers and they do not send or receive automatically for you. There are two implications:
1. You HAVE TO RECEIVE on OnReceive, even if you don’t need the data, otherwise the protocol won’t work
2. Your SENDS ARE NOT GUARANTEED to be 100%.
So what does it mean that I have to receive? For example, say you’re writing an automated FTP program which just uploads one file and ends, no interaction from user, no logging etc. Even so, you have to call .Get or .GetMessage on the FTP, otherwise the FTP protocol won’t know what the server answered! It doesn’t call them for you, it only tells you when it’s possible to do so. So you must call them, even if you’re not interrested in the contents yourself (just do a .GetMessage(local_string)).
What about those sends? Well, this just means that you can’t expect big chunks to be sent in one try. When you do .Send or .SendMessage the return value tells you how much got sent to the other side. If the data is too big, you will only get a small amount successfuly sent. This brings us to the core of this guide:
How do I send big chunks of data like files?
You can send big chunks by using the “ping-pong” principle. There are 2 events you need to understand first namely OnError and OnCanSend.
OnError is the most obvious one, it tells you that some error occured. It is important to watch this one when sending too because sends can fail with network errors (connection reset, timeout etc.).
OnCanSend is usually misunderstood. It’s the event which gets fired whenever you can send data AGAIN. This means that you first need to get to the state, when sending data is no longer possible (due to the OS buffer being full). This can be done by sending too much data too fast. How do you know it happened? If .Send or .SendMessage returns 0, and no OnError got called, you filled the OS send buffers and need to wait a bit. The OnCanSend event will tell you when you can continue sending.
So here’s how to best do sending. I’m using a “function GetNewChunk(): string” for imaginary function/method which returns a string of up to 65535 bytes (the usual max send size defined by OS). This function should either read from file or some other source you want to send from.
procedure TMyClass.OnCanSend(aSocket: TLSocket); var Sent: Integer; // number of bytes sent each try TempBuffer: string = ''; // our local temp. buffer for the filestream, can be done smarter tho begin repeat if Length(TempBuffer) = 0 then TempBuffer := GetNewChunk; // get next chunk if we sent all from the last one Sent := FConnection.SendMessage(TempBuffer, aSocket); // remember, don't use the aSocket directly! Delete(TempBuffer, 1, Sent); // delete all we sent from our temporary buffer! until (Sent = 0) or (AllIsSent); // try to send until you can't send anymore end;
So basically, this handler will try to send in a loop until it can’t send anymore or there’s no more data to send. So how do you start sending then? I mean OnCanSend doesn’t get fired by lNet until you CAN’T send anymore.. well it’s simple. Just call the OnCanSend with the appropriate socket! But which socket you ask? Well if you’re a basic TCP client, then it’s simple, just the Iterator. If you’re a server, you need to know on which socket you want to send anyway.
But how come this sends the whole thing? What happens is this. You first call the OnCanSend handler yourself, ensuring that either everything is sent in one go, or that you sent so much that the OS buffer got filled up. If so then OnCanSend will get automagically called by lNet again when sending is possible. This way you end up with a kind of “ping-pong” style and all data gets sent when possible.