Sockets, Protocols and Sending
This guide will explain some principles behind lNet when it comes to sending big amounts of data and how it comes together with protocols and sockets.
lNet currently consists of 4 parts:
1. The eventer
2. The sockets
3. The protocols
4. The sessions
The eventers and sessions both have their own guide chapter so I’ll skip them here.
The sockets are a basic OOP abstraction above the basic OS level handle which enables us to do basic operations, as well as keep track of states, options and event-readyness (think eventer). Now it is possible to send and receive data directly via sockets, but IT SHOULD NOT BE DONE. The problem is that some higher level protocols need to parse the received data so they can keep track of commands, states etc.
So, first thing to keep in mind is to NOT SEND OR RECEIVE USING SOCKETS DIRECTLY. Always use your protocol’s Send and Get methods instead. Another important thing to understand is that in lNet, sockets but even protocols are, if possible, 0-copy or bufferless. What does this mean then? It means that they do not save the received or sent data to buffers and they do not send or receive automatically for you. There are two implications:
1. You HAVE TO RECEIVE on OnReceive, even if you don’t need the data, otherwise the protocol won’t work
2. Your SENDS ARE NOT GUARANTEED to be 100%.
So what does it mean that I have to receive? For example, say you’re writing an automated FTP program which just uploads one file and ends, no interaction from user, no logging etc. Even so, you have to call .Get or .GetMessage on the FTP, otherwise the FTP protocol won’t know what the server answered! It doesn’t call them for you, it only tells you when it’s possible to do so. So you must call them, even if you’re not interrested in the contents yourself (just do a .GetMessage(local_string)).
What about those sends? Well, this just means that you can’t expect big chunks to be sent in one try. When you do .Send or .SendMessage the return value tells you how much got sent to the other side. If the data is too big, you will only get a small amount successfuly sent. This brings us to the core of this guide:
How do I send big chunks of data like files?
You can send big chunks by using the “ping-pong” principle. There are 2 events you need to understand first namely OnError and OnCanSend.
OnError is the most obvious one, it tells you that some error occured. It is important to watch this one when sending too because sends can fail with network errors (connection reset, timeout etc.).
OnCanSend is usually misunderstood. It’s the event which gets fired whenever you can send data AGAIN. This means that you first need to get to the state, when sending data is no longer possible (due to the OS buffer being full). This can be done by sending too much data too fast. How do you know it happened? If .Send or .SendMessage returns 0, and no OnError got called, you filled the OS send buffers and need to wait a bit. The OnCanSend event will tell you when you can continue sending.
So here’s how to best do sending. I’m using a “function GetNewChunk(): string” for imaginary function/method which returns a string of up to 65535 bytes (the usual max send size defined by OS). This function should either read from file or some other source you want to send from.
procedure TMyClass.OnCanSend(aSocket: TLSocket);
var
Sent: Integer; // number of bytes sent each try
TempBuffer: string = ''; // our local temp. buffer for the filestream, can be done smarter tho
begin
repeat
if Length(TempBuffer) = 0 then
TempBuffer := GetNewChunk; // get next chunk if we sent all from the last one
Sent := FConnection.SendMessage(TempBuffer, aSocket); // remember, don't use the aSocket directly!
Delete(TempBuffer, 1, Sent); // delete all we sent from our temporary buffer!
until (Sent = 0) or (AllIsSent); // try to send until you can't send anymore
end;
So basically, this handler will try to send in a loop until it can’t send anymore or there’s no more data to send. So how do you start sending then? I mean OnCanSend doesn’t get fired by lNet until you CAN’T send anymore.. well it’s simple. Just call the OnCanSend with the appropriate socket! But which socket you ask? Well if you’re a basic TCP client, then it’s simple, just the Iterator. If you’re a server, you need to know on which socket you want to send anyway.
But how come this sends the whole thing? What happens is this. You first call the OnCanSend handler yourself, ensuring that either everything is sent in one go, or that you sent so much that the OS buffer got filled up. If so then OnCanSend will get automagically called by lNet again when sending is possible. This way you end up with a kind of “ping-pong” style and all data gets sent when possible.
vice said,
September 29, 2008 at 16:09
I can´t fill the buffer, SendMessage never returns 0. Why?
My code:
procedure TForm1.Button1Click(Sender: TObject);
var
Buffer: string;
Sent: integer;
begin
Buffer := ‘START—…—END’; // –> 1198469 characters
Sent := LTCPComponent1.SendMessage(Buffer,LTCPComponent1.Iterator);
if Sent = 0 then
ShowMessage(‘Buffer full’)
else
ShowMessage(IntToStr(Sent)); // –> Show 1198469
end;
Best regards!
almindor said,
September 29, 2008 at 16:09
You’re doing three things wrong here.
1. You need to check the return value and move which part of the data you want to send. In your case, if the data is string, you can do Delete(Buffer, 1, Sent) for example. This way you won’t be sending the same thing over and over…
2. You need to put the whole thing into a repeat until cycle like the example here shows. Otherwise you only send a fraction of the buffer each time you click the button.
3. Your sending code should either reside inside OnCanSend event handler directly, or in a procedure called by OnCanSend, and your OnButtonClick should just call this procedure.
vice said,
September 29, 2008 at 16:09
But the SendMessage function never returns the 0 value although the string is very long. Why?
Almindor said,
September 29, 2008 at 17:09
It must be repeated fast enough. If you don’t do it in a cycle, you will only send a small amount of the buffer each time. 0 is returned when the network isn’t capable of sending anymore due to too much data.
vice said,
September 29, 2008 at 18:09
I do this but I get an error when I push the botton:
unit Unit1;
{$mode objfpc}{$H+}
interface
uses
Classes, SysUtils, LResources, Forms, Controls, Graphics, Dialogs, StdCtrls,
lNetComponents, lNet;
type
{ TForm1 }
TForm1 = class(TForm)
Button1: TButton;
LTCPComponent1: TLTCPComponent;
procedure FormCreate(Sender: TObject);
procedure Button1Click(Sender: TObject);
procedure LTCPComponent1Error(const msg: string; aSocket: TLSocket);
procedure LTCPComponent1CanSend(aSocket: TLSocket);
procedure FormClose(Sender: TObject; var CloseAction: TCloseAction);
private
{ private declarations }
function GetNewChunk(pos: integer): string;
data: TextFile;
public
{ public declarations }
end;
var
Form1: TForm1;
implementation
{ TForm1 }
procedure TForm1.FormCreate(Sender: TObject);
begin
LTCPComponent1.Connect(‘127.0.0.1′,9923);
System.Assign(data,’data.txt’);
System.Rewrite(data);
end;
procedure TForm1.Button1Click(Sender: TObject);
begin
LTCPComponent1.OnCanSend(LTCPComponent1.Iterator);
end;
procedure TForm1.LTCPComponent1Error(const msg: string; aSocket: TLSocket);
begin
ShowMessage(‘Error’);
end;
function TForm1.GetNewChunk: string;
begin
Result := ;
end;
procedure TForm1.LTCPComponent1CanSend(aSocket: TLSocket);
var
TempBuffer: string = ”;
Sent: Integer;
begin
repeat
if Length(TempBuffer) = 0 then
TempBuffer := GetNewChunk;
Sent := LTCPComponent1.SendMessage(TempBuffer, aSocket);
Delete(TempBuffer, 1, Sent);
until (Sent = 0);
end;
procedure TForm1.FormClose(Sender: TObject; var CloseAction: TCloseAction);
begin
System.Close(data);
end;
initialization
{$I unit1.lrs}
end.
Alan Wood said,
June 2, 2009 at 18:06
Hi,
Just for my own understanding, are you saying that in the original code, it would only have sent part of the message, the amount of which would have been defined in the return value, Sent.
Is it also true, that with a return value of 0, the original code with a delete and loop added, could have eventually sent the whole thing, but would have been a lot more resource hungry than using a callback that only fires when it is ready?
Also, in your highlevel example, I would assume I really need to release unsent data back to the place that getNewChunk would get it form..
Thanks,
Alan
almindor said,
June 2, 2009 at 20:06
Yes, one .send or .Sendmessage simply sends the given amount of data or less depending on various things (including OS specific settings). The return value gives you the actual amount of data sent.
Yes you could loop until all the stuff is sent but as you said, it’d simply hog the CPU, that’s why you should only loop until result = 0 and then wait for the next possible time to send (indicated by OnCanSend).
If you read your data-to-be-sent from some buffer and you fail to send a chunk in “one go” then yes, you should put it back. Best is to use a position indicator and only delete what you actually sent.
M Dovdovan said,
June 5, 2009 at 02:06
Hi,
We’re moving our research project DBMS from Delphi on M$Win to FPC/Lazarus on Linux. We have the database application working and would like to use LNET to FTP a changing list of small data files to the web server.
We have it working in a loop that correctly reads a TListBox to the end, but the http://FTP.Put sends only 5 files! Very Odd! I shifted the order of the files to see if one file was interrupting, but it still only sends the first 5 files. Is there a ‘buffer’ or some such setting that is limiting the http://FTP.Put?
I tried nesting the FTP code inside the loop “for i := 0 to ListBox.count-1 do”, but that doesn’t quite work at all. Any thoughts? Some detailed template documentation on FTP?
Thanks!
almindor said,
June 5, 2009 at 13:06
I’d have to see your sending code. The 5 file limit makes no sense tho, there’s no internal buffer with FTP or any other size limitation. Internally a TFileStream is created for given file and is sent via the ping-pong oncansend principle described in this article. There’s no actual internal buffer except for the 64kb temporary variable to hold the data read from TFileStream. I have no idea why your sending stops after 5 files. You can send me your code if you wish and I can try to debug it but I have my finals 10.6 so I’m out at least until then.
M Dovdovan said,
June 12, 2009 at 16:06
Thank you, Almindor!
The LNET seems to fit our needs exactly, the only glitch is the odd five file uploat limit.
What have I mangled?
System:
Linux Fedora,
Gnome,
FPC 2.2.2,
Lazarus 9.26.2
INet: //kept the default values
pipeline – false
startpoint – 1025
transfer method – ftpassive
The upload procedure looks at a ListBox with the names of image files and uploads each file. The files are small – usually less than 50K each. There can be several or many at an upload. People send them, and we add them to the database and upload them to the web site.
Meanwhile, as a workaround, I’ve been processing two files and uploading them to stay under the five file upload limit, since each file is uploaded as a small version and a large version, (removed the small version code from code below)
Everything seems to work, except that it stops at five files. It looks at the ListBox and reads the file names, but the upload stops. I tested to see if a file is stopping it, by using a different set of files, and same thing happens…
This code is mostly the same as the LNET FTP example. What can be incorrect?
I tried sending the code with this letter several times, but is there a limit on size of messages?
edwin said,
July 18, 2009 at 15:07
Hi!
I am newbie with Lnet. I used Indy components in delphi for calls to http to post information. I want to replace this component for lnet to use IDHttp but I dont find information how to use Idhttp. It works like idhhtp from indy?
With idhhtp i Can send post/get information to web page and receive the answer in a variable . Can i do this with lnet??
almindor said,
September 22, 2009 at 09:09
Hey, I’m not sure what IdHttp is, but you can GET/POST with lNet. I’ll try to get Micha (the author of HTTP code in lNet) to write a tutorial on these things since people ask about it a lot.
didiergm said,
October 29, 2009 at 19:10
Almindor,
Have you got any news about these tutorials ? Sending form fields and uploading files is what I would like to use lNet for on Mac, Win and Linux; lNet is so far the most promising and light weight suite for Lazarus. Please keep up the good work
Enrico Pergola said,
November 4, 2009 at 13:11
Hi
I made a client app for WINCE by cloning and modifying the sample lclient in the lnet demos. I have 2 major problems (maybe is lnet unstable in Wince?)
1.OnConnect.
procedure TLTCPTest.OnCon(aSocket: TLSocket);
begin
showmessage( ‘Connected’); // inform user of successful connect
FConConnected := True; // this ia a boolean I added
end;
I call connect() like this:
….
if FCon.Connect(Address, p) then begin // if connect went ok
Fquit := False;
p := 1;
repeat
FCon.CallAction; // wait for “OnConnect”
Inc(p);
until FConConnected or Fquit;
Result := FConConnected;//not FQuit;
end;
This ALWAYS connects, even when the the server application is not even running. What am I doing wrong?
2. OnCanSend;
This is another mistery.
I call initialize the event like this (as per the sample on this web page)
FCon.OnCansend(Fcon.iterator);
and this is the method:
procedure TLTCPTest.CanSend(aSocket: TLSocket);
var
Sent: integer =0; // number of bytes sent each try
TempBuffer: string = ”;
begin
repeat
if Length(TempBuffer) = 0 then
TempBuffer := GetNewChunk; // get next chunk if we sent all from the l
Sent := FCon.SendMessage(TempBuffer, aSocket);
showmessage(’sent= ‘+inttostr(sent));
// remember, don’t use the aSocket directly!
Delete(TempBuffer, 1, Sent); // delete all we sent from our temporary buffer!
until (Sent = 0) or (Send_Str = ”); // try to send until you can’t send anymore
And this is the
function GetNewChunk: string;
begin
Result := copy(Send_Str, 1, 16000);
Delete(Send_Str, 1, 16000);
end;
Well, the thing is, if the showmessage stays there, all works fine, BUT when I comment that out, the OnCanSend returns a Sent =0 in the middle, i.e. it successfully sends part of the Send_Str, then it exit without finishing.
What am I doing wrong here?
I would greatly appreciate any help.
Thanks
almindor said,
November 7, 2009 at 14:11
By the looks of things you’re mixing two event manager principles here, let me explain.
In non-visual programs (things without forms and TApplication) you use CallAction directly to integrate lNet event handling (OnXXX calls) into your main loop.
In visual apps (TApplication/TForm etc.) lNet integrates itself automatically. You just have to assign and handle the OnXXX events, and not call CallAction.
John vd Waeter said,
December 24, 2009 at 12:12
Hi,
Using a lNet UDP, created in a thread in a Lazarus/FPC linux console application.
The UDP acts as a server.
It receives a question and puts in a buffer. Another thread handles the buffer, gets the answer and put it back in a buffer. There it is picked up by the UDP’s thread and sent to the client.
I try to send using
ActSent:= Send(Buffer, sizeof(buffer), Peer);
Buffer allways has the same size: 116
Peer has ip:port
Works ok, except occasionally actsent = 114 instead of 116.
This happens abt 3 on 10000 times. Not bad really, but I don’t understand.
Speed is no issue, it happens both at a rate of abt 10 packets/s as on 100 packets/s.
The connection between the clientpc and the serverpc is a 100 MBit lan, that should handle this traffic easy.
No OnErrors are triggered.
If I increase the size from 116 to 216, it reports actsent=214. Again 2 bytes missing… what could be the issue?
tia!
John
John vd Waeter said,
December 24, 2009 at 13:12
Wait, solved!
Nothing wrong with UPD. I just didn’t realize one thread could catch up with another thread and read buffers that where not finished filling by another thread…