Sockets, Protocols and Sending

This guide will explain some principles behind lNet when it comes to sending big amounts of data and how it comes together with protocols and sockets.

lNet currently consists of 4 parts:

1. The eventer

2. The sockets

3. The protocols

4. The sessions

The eventers and sessions both have their own guide chapter so I’ll skip them here.

The sockets are a basic OOP abstraction above the basic OS level handle which enables us to do basic operations, as well as keep track of states, options and event-readyness (think eventer). Now it is possible to send and receive data directly via sockets, but IT SHOULD NOT BE DONE. The problem is that some higher level protocols need to parse the received data so they can keep track of commands, states etc.

So, first thing to keep in mind is to NOT SEND OR RECEIVE USING SOCKETS DIRECTLY. Always use your protocol’s Send and Get methods instead. Another important thing to understand is that in lNet, sockets but even protocols are, if possible, 0-copy or bufferless. What does this mean then? It means that they do not save the received or sent data to buffers and they do not send or receive automatically for you. There are two implications:

1. You HAVE TO RECEIVE on OnReceive, even if you don’t need the data, otherwise the protocol won’t work

2. Your SENDS ARE NOT GUARANTEED to be 100%.

So what does it mean that I have to receive? For example, say you’re writing an automated FTP program which just uploads one file and ends, no interaction from user, no logging etc. Even so, you have to call .Get or .GetMessage on the FTP, otherwise the FTP protocol won’t know what the server answered! It doesn’t call them for you, it only tells you when it’s possible to do so. So you must call them, even if you’re not interrested in the contents yourself (just do a .GetMessage(local_string)).

What about those sends? Well, this just means that you can’t expect big chunks to be sent in one try. When you do .Send or .SendMessage the return value tells you how much got sent to the other side. If the data is too big, you will only get a small amount successfuly sent. This brings us to the core of this guide:

How do I send big chunks of data like files?

You can send big chunks by using the “ping-pong” principle. There are 2 events you need to understand first namely OnError and OnCanSend.

OnError is the most obvious one, it tells you that some error occured. It is important to watch this one when sending too because sends can fail with network errors (connection reset, timeout etc.).

OnCanSend is usually misunderstood. It’s the event which gets fired whenever you can send data AGAIN. This means that you first need to get to the state, when sending data is no longer possible (due to the OS buffer being full). This can be done by sending too much data too fast. How do you know it happened? If .Send or .SendMessage returns 0, and no OnError got called, you filled the OS send buffers and need to wait a bit. The OnCanSend event will tell you when you can continue sending.

So here’s how to best do sending. I’m using a “function GetNewChunk(): string” for imaginary function/method which returns a string of up to 65535 bytes (the usual max send size defined by OS). This function should either read from file or some other source you want to send from.

procedure TMyClass.OnCanSend(aSocket: TLSocket);
var
  Sent: Integer; // number of bytes sent each try
  TempBuffer: string = ''; // our local temp. buffer for the filestream, can be done smarter tho
begin
  repeat
    if Length(TempBuffer) = 0 then
      TempBuffer := GetNewChunk; // get next chunk if we sent all from the last one
    Sent := FConnection.SendMessage(TempBuffer, aSocket); // remember, don't use the aSocket directly!
    Delete(TempBuffer, 1, Sent); // delete all we sent from our temporary buffer!
  until (Sent = 0) or (AllIsSent); // try to send until you can't send anymore
end;

So basically, this handler will try to send in a loop until it can’t send anymore or there’s no more data to send. So how do you start sending then? I mean OnCanSend doesn’t get fired by lNet until you CAN’T send anymore.. well it’s simple. Just call the OnCanSend with the appropriate socket! But which socket you ask? Well if you’re a basic TCP client, then it’s simple, just the Iterator. If you’re a server, you need to know on which socket you want to send anyway.

But how come this sends the whole thing? What happens is this. You first call the OnCanSend handler yourself, ensuring that either everything is sent in one go, or that you sent so much that the OS buffer got filled up. If so then OnCanSend will get automagically called by lNet again when sending is possible. This way you end up with a kind of “ping-pong” style and all data gets sent when possible.

99 Comments »

  1. vice said,

    I can´t fill the buffer, SendMessage never returns 0. Why?

    My code:

    procedure TForm1.Button1Click(Sender: TObject);
    var
    Buffer: string;
    Sent: integer;
    begin
    Buffer := ‘START—…—END’; // –> 1198469 characters
    Sent := LTCPComponent1.SendMessage(Buffer,LTCPComponent1.Iterator);
    if Sent = 0 then
    ShowMessage(‘Buffer full’)
    else
    ShowMessage(IntToStr(Sent)); // –> Show 1198469
    end;

    Best regards!

    • almindor said,

      Because you’re not filling it fast enough. The send buffer is a kernel structure which gets emptied by the kernel when it’s running and handling network sys requests. The only way to fill the send buffer is to do so in a loop with enough data so that the buffer gets filled during one time-slice of the process execution.

  2. almindor said,

    You’re doing three things wrong here.

    1. You need to check the return value and move which part of the data you want to send. In your case, if the data is string, you can do Delete(Buffer, 1, Sent) for example. This way you won’t be sending the same thing over and over…

    2. You need to put the whole thing into a repeat until cycle like the example here shows. Otherwise you only send a fraction of the buffer each time you click the button.

    3. Your sending code should either reside inside OnCanSend event handler directly, or in a procedure called by OnCanSend, and your OnButtonClick should just call this procedure.

  3. vice said,

    But the SendMessage function never returns the 0 value although the string is very long. Why?

  4. Almindor said,

    It must be repeated fast enough. If you don’t do it in a cycle, you will only send a small amount of the buffer each time. 0 is returned when the network isn’t capable of sending anymore due to too much data.

  5. vice said,

    I do this but I get an error when I push the botton:

    unit Unit1;

    {$mode objfpc}{$H+}

    interface

    uses
    Classes, SysUtils, LResources, Forms, Controls, Graphics, Dialogs, StdCtrls,
    lNetComponents, lNet;

    type

    { TForm1 }

    TForm1 = class(TForm)
    Button1: TButton;
    LTCPComponent1: TLTCPComponent;
    procedure FormCreate(Sender: TObject);
    procedure Button1Click(Sender: TObject);
    procedure LTCPComponent1Error(const msg: string; aSocket: TLSocket);
    procedure LTCPComponent1CanSend(aSocket: TLSocket);
    procedure FormClose(Sender: TObject; var CloseAction: TCloseAction);
    private
    { private declarations }
    function GetNewChunk(pos: integer): string;
    data: TextFile;
    public
    { public declarations }
    end;

    var
    Form1: TForm1;

    implementation

    { TForm1 }

    procedure TForm1.FormCreate(Sender: TObject);
    begin
    LTCPComponent1.Connect(‘127.0.0.1′,9923);
    System.Assign(data,’data.txt’);
    System.Rewrite(data);
    end;

    procedure TForm1.Button1Click(Sender: TObject);
    begin
    LTCPComponent1.OnCanSend(LTCPComponent1.Iterator);
    end;

    procedure TForm1.LTCPComponent1Error(const msg: string; aSocket: TLSocket);
    begin
    ShowMessage(‘Error’);
    end;

    function TForm1.GetNewChunk: string;
    begin
    Result := ;
    end;

    procedure TForm1.LTCPComponent1CanSend(aSocket: TLSocket);
    var
    TempBuffer: string = ”;
    Sent: Integer;
    begin
    repeat
    if Length(TempBuffer) = 0 then
    TempBuffer := GetNewChunk;
    Sent := LTCPComponent1.SendMessage(TempBuffer, aSocket);
    Delete(TempBuffer, 1, Sent);
    until (Sent = 0);
    end;

    procedure TForm1.FormClose(Sender: TObject; var CloseAction: TCloseAction);
    begin
    System.Close(data);
    end;

    initialization
    {$I unit1.lrs}

    end.

  6. Alan Wood said,

    Hi,

    Just for my own understanding, are you saying that in the original code, it would only have sent part of the message, the amount of which would have been defined in the return value, Sent.

    Is it also true, that with a return value of 0, the original code with a delete and loop added, could have eventually sent the whole thing, but would have been a lot more resource hungry than using a callback that only fires when it is ready?

    Also, in your highlevel example, I would assume I really need to release unsent data back to the place that getNewChunk would get it form..

    Thanks,
    Alan

    • almindor said,

      Yes, one .send or .Sendmessage simply sends the given amount of data or less depending on various things (including OS specific settings). The return value gives you the actual amount of data sent.

      Yes you could loop until all the stuff is sent but as you said, it’d simply hog the CPU, that’s why you should only loop until result = 0 and then wait for the next possible time to send (indicated by OnCanSend).

      If you read your data-to-be-sent from some buffer and you fail to send a chunk in “one go” then yes, you should put it back. Best is to use a position indicator and only delete what you actually sent.

  7. M Dovdovan said,

    Hi,
    We’re moving our research project DBMS from Delphi on M$Win to FPC/Lazarus on Linux. We have the database application working and would like to use LNET to FTP a changing list of small data files to the web server.
    We have it working in a loop that correctly reads a TListBox to the end, but the http://FTP.Put sends only 5 files! Very Odd! I shifted the order of the files to see if one file was interrupting, but it still only sends the first 5 files. Is there a ‘buffer’ or some such setting that is limiting the http://FTP.Put?
    I tried nesting the FTP code inside the loop “for i := 0 to ListBox.count-1 do”, but that doesn’t quite work at all. Any thoughts? Some detailed template documentation on FTP?
    Thanks!

    • almindor said,

      I’d have to see your sending code. The 5 file limit makes no sense tho, there’s no internal buffer with FTP or any other size limitation. Internally a TFileStream is created for given file and is sent via the ping-pong oncansend principle described in this article. There’s no actual internal buffer except for the 64kb temporary variable to hold the data read from TFileStream. I have no idea why your sending stops after 5 files. You can send me your code if you wish and I can try to debug it but I have my finals 10.6 so I’m out at least until then.

  8. M Dovdovan said,

    Thank you, Almindor!
    The LNET seems to fit our needs exactly, the only glitch is the odd five file uploat limit.
    What have I mangled?
    System:
    Linux Fedora,
    Gnome,
    FPC 2.2.2,
    Lazarus 9.26.2
    INet: //kept the default values
    pipeline – false
    startpoint – 1025
    transfer method – ftpassive

    The upload procedure looks at a ListBox with the names of image files and uploads each file. The files are small – usually less than 50K each. There can be several or many at an upload. People send them, and we add them to the database and upload them to the web site.

    Meanwhile, as a workaround, I’ve been processing two files and uploading them to stay under the five file upload limit, since each file is uploaded as a small version and a large version, (removed the small version code from code below)

    Everything seems to work, except that it stops at five files. It looks at the ListBox and reads the file names, but the upload stops. I tested to see if a file is stopping it, by using a different set of files, and same thing happens…

    This code is mostly the same as the LNET FTP example. What can be incorrect?
    I tried sending the code with this letter several times, but is there a limit on size of messages?

  9. edwin said,

    Hi!
    I am newbie with Lnet. I used Indy components in delphi for calls to http to post information. I want to replace this component for lnet to use IDHttp but I dont find information how to use Idhttp. It works like idhhtp from indy?

    With idhhtp i Can send post/get information to web page and receive the answer in a variable . Can i do this with lnet??

    • almindor said,

      Hey, I’m not sure what IdHttp is, but you can GET/POST with lNet. I’ll try to get Micha (the author of HTTP code in lNet) to write a tutorial on these things since people ask about it a lot.

  10. didiergm said,

    Almindor,

    Have you got any news about these tutorials ? Sending form fields and uploading files is what I would like to use lNet for on Mac, Win and Linux; lNet is so far the most promising and light weight suite for Lazarus. Please keep up the good work 🙂

  11. Enrico Pergola said,

    Hi

    I made a client app for WINCE by cloning and modifying the sample lclient in the lnet demos. I have 2 major problems (maybe is lnet unstable in Wince?)

    1.OnConnect.
    procedure TLTCPTest.OnCon(aSocket: TLSocket);
    begin
    showmessage( ‘Connected’); // inform user of successful connect
    FConConnected := True; // this ia a boolean I added
    end;
    I call connect() like this:
    ….
    if FCon.Connect(Address, p) then begin // if connect went ok
    Fquit := False;
    p := 1;
    repeat
    FCon.CallAction; // wait for “OnConnect”
    Inc(p);
    until FConConnected or Fquit;
    Result := FConConnected;//not FQuit;
    end;
    This ALWAYS connects, even when the the server application is not even running. What am I doing wrong?
    2. OnCanSend;
    This is another mistery.
    I call initialize the event like this (as per the sample on this web page)
    FCon.OnCansend(Fcon.iterator);
    and this is the method:
    procedure TLTCPTest.CanSend(aSocket: TLSocket);
    var
    Sent: integer =0; // number of bytes sent each try
    TempBuffer: string = ”;
    begin
    repeat
    if Length(TempBuffer) = 0 then
    TempBuffer := GetNewChunk; // get next chunk if we sent all from the l
    Sent := FCon.SendMessage(TempBuffer, aSocket);
    showmessage(‘sent= ‘+inttostr(sent));
    // remember, don’t use the aSocket directly!
    Delete(TempBuffer, 1, Sent); // delete all we sent from our temporary buffer!
    until (Sent = 0) or (Send_Str = ”); // try to send until you can’t send anymore

    And this is the
    function GetNewChunk: string;
    begin
    Result := copy(Send_Str, 1, 16000);
    Delete(Send_Str, 1, 16000);
    end;

    Well, the thing is, if the showmessage stays there, all works fine, BUT when I comment that out, the OnCanSend returns a Sent =0 in the middle, i.e. it successfully sends part of the Send_Str, then it exit without finishing.
    What am I doing wrong here?
    I would greatly appreciate any help.
    Thanks

  12. almindor said,

    By the looks of things you’re mixing two event manager principles here, let me explain.

    In non-visual programs (things without forms and TApplication) you use CallAction directly to integrate lNet event handling (OnXXX calls) into your main loop.

    In visual apps (TApplication/TForm etc.) lNet integrates itself automatically. You just have to assign and handle the OnXXX events, and not call CallAction.

  13. John vd Waeter said,

    Hi,

    Using a lNet UDP, created in a thread in a Lazarus/FPC linux console application.

    The UDP acts as a server.
    It receives a question and puts in a buffer. Another thread handles the buffer, gets the answer and put it back in a buffer. There it is picked up by the UDP’s thread and sent to the client.

    I try to send using

    ActSent:= Send(Buffer, sizeof(buffer), Peer);

    Buffer allways has the same size: 116
    Peer has ip:port
    Works ok, except occasionally actsent = 114 instead of 116.
    This happens abt 3 on 10000 times. Not bad really, but I don’t understand.
    Speed is no issue, it happens both at a rate of abt 10 packets/s as on 100 packets/s.

    The connection between the clientpc and the serverpc is a 100 MBit lan, that should handle this traffic easy.

    No OnErrors are triggered.

    If I increase the size from 116 to 216, it reports actsent=214. Again 2 bytes missing… what could be the issue?

    tia!
    John

  14. John vd Waeter said,

    Wait, solved!
    Nothing wrong with UPD. I just didn’t realize one thread could catch up with another thread and read buffers that where not finished filling by another thread…

  15. Jinsong Chen said,

    Hi, almindor,
    thanks for your works about the lnet package. I write a server application use the lnet on FreeBSD serval weeks ago, it runs fine now. currently, I am shifting my work from delphi to lazarus. I need to use HTTP to POST data stream and waiting for repose(sorry, I noticed there are some body ask this question), I hope I can get some sample code or advice.

    Thanks!

  16. Ajay S said,

    Hi Almindor,

    I am trying to create an embedded “http server” and am following the ‘fphttpd.pp’ program in the examples directory.
    While the web server is serving static html file like ‘index.html’, I need to get any ‘get’ parameters from these static files for ex http://127.0.0.1:3880/index.html?name=ajay.
    I need to get the name/value parameter in my program when some one clicks the above link. (I don’t want to use any cgi mode)
    Is this possible?, could you please send me some code as I new to FPC & Lnet but experienced in Delphi

    Thanks

    Ajay S

  17. Dibo said,

    Hi,

    Sorry for my English.
    I’m writing Jabber client using TLTCPComponent. I connect TLSSLSessionComponent (TLS method) to TLTCPComponent.Session. I can connect to serwer. Serwer response with right xml. I am on stage where I must do TLS negotiation (http://www.ietf.org/rfc/rfc3920.txt “5.3 Client-to-Server Example”). On step 5 I get this command from server:
    “”
    I switch SSLActive property to True but nothing happens. If I send next command to server, he disconnect me. Events SSLOnAccept and SSLOnConnect doesn’t fire. What I must do to upgrade connection to TLS? I tested this on Indy and it upgrade SSL connection automatically. I change SocketClass to TLSSLSocket but it doesn’t help.

    Regards

    • almindor said,

      Hello, can you give me (you can email me if it’s not public info) the server URL and possible credentials to test? TLS negotiation after connecting is already tested with gmail for example (if you have gmail you can try lnet/examples/visual/smtp).

      • Dibo said,

        SMTP Component have procedure StartTLS which TCP component doesn’t have. I send you example on e-mail.

  18. Dibo said,

    Hi,

    @Almindor, if you helped me with TCP Client, it might help me also with HTTP Client 😛

    Some times ago I post problem with HTTP client: http://www.lazarus.freepascal.org/index.php/topic,10024.msg49493.html#msg49493

    I respect LNETs for non-blocking mode and use them wherever I can, but lack of this feature disqualify HTTP client in my project :/

    Regards

    • almindor said,

      The problem is, I don’t know 😀 HTTP is done by Micha Nelissen (neli on irc) and he’s a bit inactive these days so getting responses or code updates from him isn’t easy. I plan to make a rewrite of the HTTP part (there are some changes really needed) but I just don’t have the time atm.

  19. Dibo said,

    Ok. I’ll be patiently waiting 🙂

  20. Dibo said,

    Ok, I very much needed this solution, so I analyze procedure TLHTTPClientSocket.SendRequest. By default, http client does not send absolutely no header (only host), so I sniff what sends synapse and do this same by AddExtraHeader procedure. XML is just new text block in document so I add this by AddExtraHeader too. This is solution code which work, maybe someone need this too:

    DecomposeURL(‘http://some.host.com/login.xml’, sHost, sUrl, sPort);
    httpclient.Method := hmPost;
    httpclient.Port := sPort;
    httpclient.Host := sHost;
    httpclient.URI := sUrl;
    httpclient.AddExtraHeader(‘Keep-Alive: 300’);
    httpclient.AddExtraHeader(‘Connection: keep-alive’);
    httpclient.AddExtraHeader(‘Content-Length: 111′); // <-XML length
    httpclient.AddExtraHeader('Content-Type: application/xml; charset=utf-8');
    httpclient.AddExtraHeader(LineEnding);
    httpclient.AddExtraHeader(
    '’ + LineEnding +
    ”+ LineEnding +
    ‘dibo’+ LineEnding +

    );
    httpclient.SendRequest;

    LHTTPClient is a very good socket but lacks some basic features visible for programmer (and PUT, DELETE http methods).
    But this is not the end of my problems 😛 . There is demo how to download file, but there is no how to upload 🙂 . Maybe I’ll find a way …

    Regards

  21. Barry said,

    I’m new to lazarus & lnet, and maybe a little dense, but could you please show an example of adding in the SSL visual component to the included telnet client example? Am I mistaken in thinking this would be a way to create an SSH Client?

    Thanks

    • almindor said,

      Hello,

      to make a SSL enabled connection, you just put both the Telnet client component and the SSL component on the form and connect them by assigning the SSL component in your telnet component’s Session. Once that is done, you work with the telnet client as usual.

      Doing this however does not create a full SSH client, there’s more needed for that (I’m not 100% sure what, but SSH has it’s own specialialities).

      Hope this helps.

      • Barry said,

        OK, got the SSL component in, but fails at runtime with “Error creating SLL CTX: SSLCTXNew”. What is intended to go in the CAFile and KeyFile fields for this component?

        Thanks

      • almindor said,

        Do you have openSSL installed?

  22. Dibo said,

    Hi,

    I have problem with TLHTTPClient in DLL. When I call SendRequest nothing happens. No results, no erros, no events fired. Silent. Anyone know a reason?

    Regards

    • almindor said,

      Please be more specific about the dll. If you think it’s a bug, report at the fpc bug tracker (http://bugs.freepascal.org) under lazarus projects/lNet.

  23. Dibo said,

    I reported this. But I found temporary solution (see comments)
    http://bugs.freepascal.org/view.php?id=18625

  24. Claudio said,

    Hi, I have a big doubt. I’ve just read that never send data directly via sockets, but I need to do it, so how can I solve this.
    My app is a simple game where one is a server and others clients (most cases 2 clients only).
    I put tcp server to listen and the other guys connect to it, so after they’ve connected I have 3 sockets available on Socks property (if 2 clients connected, of course), first socket is server and the rest are clients. So far so good but the problem is that I send and receive messages and sometime I must respond only to the client who are talking to server.
    The client send a message to server, server reads the message on the Receive event and send back a message to client. When I send back the message to client I use the aSocket variable that comes in the Receive event, but this is wrong, so how can I respond only to the client wich are talking with server ?

    I put some code to clarify:

    procedure TfrmMain.tcpReceive(aSocket: TLSocket);
    var
    Msg :TStrings;
    S :String;

    begin
    if aSocket.GetMessage(S) = 0 then exit;
    Msg := TStringList.Create;
    Msg.Text := S;

    if Msg[0] = CMD_DATE then begin
    aSocket.SendMessage ( DateToStr( Date ) )
    end;

    So “aSocket.SendMessage ( DateToStr( Date ) )” is wrong, but I can’t figure out another way to achieve this becouse aSocket is the client I need to comunicate.
    Any help would be much appreciated.

    • almindor said,

      You CAN communicate to a specific socket of course 😉 You just shouldn’t use “socket.sendmessage” but “TCP.SendMessage(string, Socket)”.

      In your example you just need to change “aSocket.SendMessage ( DateToStr( Date ) )” to “tcp.SendMessage (DateToStr( Date ), aSocket)” 🙂

      In case of pure TCP/UDP this isn’t a problem tho, it can be a problem with the higher protocols (like FTP etc.). I just discourage it globally so people don’t get used to it.

      Hope it helped.

      • Claudio said,

        Perfect, I understand now. Thanks a lot for your help.

        I’ve just discovered another issue, the tcp has a “Listen” function to start listening but there’s no “Stop Listen” function, so the only way to stop listening is freeing the component ?

        Thanks in advance.

  25. Claudio said,

    No need to answer me, I’ve found my mistake.
    To help others like me, tcp component has a Disconnect function, with this function you stop listening in server mode.

  26. Claudio said,

    Hi again, is there a way to accept only n connections and then disable listening ?.
    I’ve tried in the onaccept event to close the socket just connected and it works, clients get disconnected, but server tcp component keeps creating sockets, they are not connected but they are grabin memory of course.
    So is there any way to connect, for instance, 3 clients and no more ?

    Thanks.

    • almindor said,

      You’re right about disconnecting on the OnAccept as one solution. The memory hogging shouldn’t happen if the clients disconnect cleanly. I’ll need to check this out.

  27. Giorgio said,

    I have a bug in the TCP socket client: every second I have to connect to a TCP server, send a request string and wait for my response as a string. Once you receive the answer I have to disconnect. Problem is here: after about 670 transactions are no longer generated the event “onConnect” even if the server connects. I use the “OnConnect” event to send the request string, the event OnReceive to read the data received and the disconnect command to disconnect.
    Using Lazarus 09.30, LNET 0.6.5 visual, S.O. is Ubuntu 9.04.

    Thanks

  28. Rajiv Ranjan said,

    I have an issue with LNET client tcp socket.I am using lazarus on MAC.I have an application that runs on startup,and connect to a server.On some computer in which ethernet net card take time to become active and connect to public network, my client application not able to connect to server.The connect get fails obviously with the network card is not up.once up still connect fails.Not able to understand why…

    Any help will be appreciated.

    Rajiv

    • almindor said,

      Hey,

      I need more info. Are you using visual lNet (forms etc.) or “console” version? What error message do you get on OnError?

      • Hi, I’m working in the same team with Rajiv. We are using lNetBase (console) version.

        The Structure:
        We have a class (called TalkHdlr) that manage connection, encoding, decoding of messages and also ping from server. This class is created on FormCreate. When we need to send a message the class check if in the stringlist there is present a connection with same IP:Port and if not it call the init procedure that creates a new one and add it’s IP:Port to the list. In the init we Create a new TLTCP passing it a reference to main form, assigning all needed events, address and make a loop on .callAction until it become .connected or are passing 15 seconds, after this i enable the 100ms timer that calls .callAction.

        The real use:
        We have a timer that every 30 seconds try to send authentication message to server, and if the connection is not present it creates a new one as described above.

        The test use:
        In a test application i have a timer that every 30 seconds try to connect to server. I’v arrived to try to recreate TalkHdlr every time i try to connect and call directly the connection init procedure, but i get always the same error (IP:Port::Error): 0.0.0.0:8014::Error on connect: connection refused. The IP i try is real IP of server or DNS, but i get always that 0.0.0.0 in error.

        As Rajiv said if we manually launch application after connection is up this mechanism works just perfect, it disconnects on network errors, connection down, unplugging of network cable ecc. and then when connection is restored it reconnects.
        If we launch application with no network it gives same error always…

      • almindor said,

        This is odd..

        so to sum things up, you get trouble only if you try to initiate connection on broken network first, and then “fix” the network and re-try it always fails afterwards (but works fine if the network is working from the start).

        I’ll have to do some tests on these situations perhaps there’s some state-lock.

      • Hi, i have digged a bit more inside this thing and i noticed that when i start app with internet connection, lose it and try to reconnect, ResolveHostByName function takes like 1-2 seconds for execute, but if i launch my app with broken internet connection ResolveHostByName always take like nothing!
        Can it be a bug inside netdb.pp? (maybe it never refresh the resolve file with dns servers? i see that at init CheckResolveFileAge is set to False, but i have no clue of how netdb works or how to properly init…)

  29. almindor said,

    Hmm this might be something. NetDB looks only inside /etc/resolv.conf when there’s no connection, so that’s why it’s so fast. Could you please construct a lnet-less netdb only example and post it as a bug to http://bugs.freepascal.org ?

  30. snorkel said,

    Hi,
    What’s the best way to associate user data with a socket?
    For example if I am writing a custom chat server using the echo server as a example and a user logs on and sends a user name, how can I associate that user name with a socket so I can find it again when another user sends a message to that user? Should I maintain a separate hashlist or some type of associative array?

    • almindor said,

      You can use the TLSocket.UserData pointer 🙂

      Using a StringList or some other list (with object holding the socket pointer) is also usable.

  31. robert said,

    Hi,

    I am trying a simple text send and text receive test Between delphi 6.0 seversocket and lazarus Lnet HTTPClient. It works fine when i send text to the delphi server. But i don’t seem to get any text back from the server. I wrote the same procedure in Delphi with clientSocket and that does get text back. Can you point me to where i go wrong in Lazarus?

    ====================================================
    Delphi Server:

    unit server;

    interface

    uses
    Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
    Dialogs, ScktComp, StdCtrls;

    type
    TForm1 = class(TForm)
    ServerSocket1: TServerSocket;
    Memo1: TMemo;
    Button1: TButton;
    procedure FormCreate(Sender: TObject);
    procedure FormClose(Sender: TObject; var Action: TCloseAction);
    procedure ServerSocket1ClientRead(Sender: TObject;
    Socket: TCustomWinSocket);
    procedure Button1Click(Sender: TObject);
    private
    { Private declarations }
    public
    { Public declarations }
    end;

    var
    Form1: TForm1;

    implementation

    {$R *.dfm}

    procedure TForm1.FormCreate(Sender: TObject);
    begin
    ServerSocket1.Port := 7777;
    ServerSocket1.Active := True;
    end;

    procedure TForm1.FormClose(Sender: TObject; var Action: TCloseAction);
    begin
    ServerSocket1.Active := false;
    end;

    procedure TForm1.ServerSocket1ClientRead(Sender: TObject;
    Socket: TCustomWinSocket);
    var
    i:integer;
    sRecr : string;
    begin
    for i := 0 to ServerSocket1.Socket.ActiveConnections-1 do
    begin
    with ServerSocket1.Socket.Connections[i] do
    begin
    sRecr := ReceiveText;
    if sRecr ” then
    begin
    Memo1.Lines.Add(RemoteAddress + ‘ sends :’) ;
    Memo1.Lines.Add(sRecr);
    SendText(datetimetostr(now)+ ‘Received by server’);
    end;
    end;
    end;
    end;

    procedure TForm1.Button1Click(Sender: TObject);
    var
    i:integer;
    sRecr : string;
    begin
    for i := 0 to ServerSocket1.Socket.ActiveConnections-1 do
    begin
    with ServerSocket1.Socket.Connections[i] do
    begin
    begin
    SendText(datetimetostr(now)+ ‘test by server’);
    end;
    end;
    end;
    end;
    end.

    ===================================================

    Lazarus Client
    procedure TForm1.FormCreate(Sender: TObject);
    begin
    LHTTPClient.Port := 7777;
    LHTTPClient.Host := ‘127.0.0.1’;
    LHTTPClient.Connect();
    end;

    procedure TForm1.Button1Click(Sender: TObject);
    begin
    LHTTPClient.Method:=hmPost;
    if not LHTTPClient.Active then
    LHTTPClient.connect();

    LHTTPClient.SendMessage(Edit1.Text);

    end;

    procedure TForm1.Button2Click(Sender: TObject);
    var
    tekst:string;
    tekstNr:integer;
    begin
    LHTTPClient.Method:=hmGet;
    tekstnr:=LHTTPClient.GetMessage(tekst);
    memo1.Append(tekst);
    end;

    end.

    • almindor said,

      You cannot use .GetMessage inside a buttonclick. You must assign the OnReceive event, and do it there.

  32. robert said,

    tnx for your quick reply

    i tried showmessage in the onReceive event, but this event did not get triggerd…

    So no showmessage and no .getmessage, thats why i put it under a buttonclick. so that i could force the reading.

    Later i tried with more luck a different component. i Tried the lTCPcomponent. This worked, until i tried to start the application for the second time. (the first one was still running) the second one gave an 100053 error.

    Right now i can’t access the procedure but tommorow i will post the latest version.

  33. pinoyden said,

    Is it possible to use this library on an android native project?

    • almindor said,

      It might be possible. Someone has to try and compile lNet with native android FPC and see if it works. Since android uses a new Linux kernel, the epoll eventer should be used and everything should work as in linux.

      All this is just the “console” lib. Visual lazarus integration (do they have that for Android??) would require similar work as the MacOSX port.

  34. zandoye said,

    Hi, 🙂 I have several game servers start and quit on demand, so these server processes doesn’t listen to fixed number ports. They(tLTcp) start, listen to (*,0), then register their port number that allocated by OS on a NameServer.
    I’v tried
    > writeln(‘host: ‘, sock.host, ‘ port: ‘, sock.port);
    > writeln(‘host: ‘, sock.RootSock.localAddress, ‘ port: ‘, sock.RootSock.localPort);
    > writeln(‘host: ‘, sock.RootSock.peerAddress, ‘ port: ‘, sock.RootSock.peerPort);
    but the three statements all print out zero.

    Now, I have to invoke
    > fpGetSockName(sock.RootSock.handle, @sockAddr, @sockAddr_size);
    > writeln(‘host: ‘, hostAddrToStr(in_addr(nToHl(sockAddr.sin_addr.s_addr))), ‘ port: ‘, nToHs(sockAddr.sin_port));
    to get the real port number.

    I’m using lnet svn reversion 2591.
    Is there an easy and portabal way to get the real port that is listening to?

    • almindor said,

      Good question. I remember these problems myself but IIRC I also did some call to “wake up” the port assignment so to speak. If there’s no other way, we could add a cross-platform functionality and integrate it to lNet.

  35. Ryan Davis said,

    I have a question. I’m using lnet currently cross-platform on Mac and Windows using the non-visual methods. I currently have a server hosting and then the same application that has the server connects to that server with a client as well as copies of the application from elsewhere connect via their client.

    In practice this works rather well for most things. Indeed, when I want to test things on Mac I run two copies of the program and connect them using 127.0.0.1. It works fine on Mac.

    However, I tried doing this same operation on Windows, where I test the application by connecting two copies to 127.0.0.1 and every time I do it the hosting application freezes with the error “Send buffer full, try again later.” I’ve heard of other people getting this error before too, though inconsistently, when connecting over the internet normally.

    What causes this error and how can it be preempted or prevented? How can I run two copies of the same app for testing on Windows like I do on Mac and also prevent this error from ever troubling my users in general?

    • almindor said,

      So the server has this error? Sending buffer full occurs when you try to send too much data at once. You should only get a send = 0 tho and OnCanSend should fire when the OS is ready to send more. Are you using this event, is the handler assigned? It probably works on Mac OS X because it has bigger default OS send buffers.

      • Ryan Davis said,

        I’ve tried to replicate this behavior. I was experiencing it consistently one day all day, however, I haven’t been able to get it to happen again. It’s consistency may have been an artifact of my virtual machine I use to run Windows, which I recently changed (VMWare to Parallels). If I see it again I’ll let you know. Meanwhile, I have another question, that I’ll ask below.

  36. Andyk said,

    Can’t get callAction to work on version 0.6.5, the eventer code seems to just return true. Is this a bug ?

    function TLEventer.CallAction: Boolean;
    begin
    Result := True;
    // override in ancestor
    end;

    • almindor said,

      TLEventer is just the abstract ancestor to specific eventers like TLSelectEventer or TLEpollEventer. Your eventer is either created automatically by lNet (on first call to Listen/Connect) in which case you can use the component directly (e.g.: TLTcp.CallAction) or you create them yourself. Unless you need your own specific eventer type, you should always create the best eventer for given platform via the levents.BestEventerClass().

      • Andyk said,

        OK!

        I’ve deleted the eventer I created manually and let the connect routine create its own which seems to work. Although i had to add a call callAction to make the connect event fire.

        Originally, I couldn’t get anything to work without the eventer, it just came up with error’s, Seems to work now though.

        I’m using the telnet protocol although in Delphi I created a wrapper around the Tclientsocket. I really need a blocking communication channel that acts like an ASCII RS232 connection for communicating with scientific instrumentation.

        When I tried to set the lnet ssblocking state, it just gave an error though. Is there an easier way to do this ?

  37. Wojtek said,

    How to use telnet SendCommand(const ACommand:Char ; const How: TLHowEnum) ? What is TLHowEnum? I’d like turn off echo from telnet response.

    best regards

  38. KazakOFF said,

    Hi
    Can you give an example of how to listen to the UDP protocol.

    • almindor said,

      You can see it in lnet/examples/console/ludp

  39. Niros Tamos said,

    This component package seem great! I was able to build a Telnet server that can deal with many clients (basicly WinCe RF data collectors running a simple Telnet client) and treat each session separetedly according to what they need to do!
    Congratulations!!!!
    niros

  40. Ricardo said,

    Hello i have this problem. I’ll try to explain.
    Downloaded component, installed and compiled succefully on lazarus 1.2.4.
    Now opened the example in tcpudp folder (testnet) and worked fine.
    Then i’ll make a simple app, drop an ltcpcomponent on it and filled the on connect, ondisconnect and onerror events showing a simple string connected, disconected or the error string.
    I used your example app as server. Started to listen in, say, 44444 port then fire my app and then click on connect and it connects showing it on the label, if i click on disconnect it disconnects and show that in the label. The problem is that, if on your example app (the server) click on disconnect my app doesn’t notice the disconnection. If i click connect again it gives me the “connection refused error” and that’s fine because can’t find the server.
    I’ve checked this behaviour on your example app and the app does notice on disconnections on every way, but i can’t find an event or anything to add to my app to notice this disconnections. I’ll paste the code, thanks

    TForm1 = class(TForm)
    Button1: TButton;
    Button3: TButton;
    Edit1: TEdit;
    Edit2: TEdit;
    Label1: TLabel;
    Label2: TLabel;
    Label3: TLabel;
    LTCPComponent1: TLTCPComponent;
    procedure Button1Click(Sender: TObject);
    procedure Button3Click(Sender: TObject);
    procedure FormCreate(Sender: TObject);
    procedure LTCPComponent1Connect(aSocket: TLSocket);
    procedure LTCPComponent1Disconnect(aSocket: TLSocket);
    procedure LTCPComponent1Error(const msg: string; aSocket: TLSocket);
    private
    { private declarations }
    public
    { public declarations }
    end;

    var
    Form1: TForm1;
    red : TLConnection;

    implementation

    {$R *.lfm}

    { TForm1 }

    procedure TForm1.Button1Click(Sender: TObject);
    begin
    if red.Connect(edit1.Text, strtoint(edit2.Text)) then;
    end;

    procedure TForm1.Button3Click(Sender: TObject);
    begin
    red.Disconnect;
    end;

    procedure TForm1.FormCreate(Sender: TObject);
    begin
    red:=LTCPComponent1;
    end;

    procedure TForm1.LTCPComponent1Connect(aSocket: TLSocket);
    begin
    label3.Caption:=’Conectado’;
    end;

    procedure TForm1.LTCPComponent1Disconnect(aSocket: TLSocket);
    begin
    label3.Caption:=’Desconectado’;
    end;

    procedure TForm1.LTCPComponent1Error(const msg: string; aSocket: TLSocket);
    begin
    label3.Caption:=’Error: ‘+msg;
    end;

    • almindor said,

      I’ll try to reproduce. Since you’re sending long texts, could you please e-mail me your code/project as an attachment? The disconnect should’ve been cought.

      Also was this the latest release or the trunk version from SVN?

      Ales

      • Ricardo said,

        It’s easy to reproduce, just drop ltcp component and write the connect, disconnect and error procedures and used the tcpip example from the package as server. My app doesn’t notice when i stop the host from example app but if i use two instances of the example app works.
        The version i’m using is the latest from the page (not svn), 0.6.5 and lazarus 1.2.4.
        Thanks

  41. Ricardo said,

    Hello again, there is any way to find out my local (LAN) ip with this components?
    I can’t find any method to do it

    • almindor said,

      You need a TLSocket and then go for .LocalAddress and .LocalPort. There were some bugs reported with these tho. You can only use this after you connect however. In any case you’d only get your local IP not the public facing one unless you’re directly connected to the internet.

      • Ricardo said,

        Mmm i think there would be no trouble to work in that way. Do you have an example of code for do that? What kind of bugs were reported?

  42. Mike Maerker said,

    Hi Aleš,
    I’m a little bit confused.
    Do I use PUT or the ping-pong way to upload files?
    I’m using lnet visual in my project (uploading tables as .html
    to remote monitor processes)
    Thx,
    Mike

    • almindor said,

      I take it you mean a http upload?

      If so then I think you just need to use SendRequest of the TLHTTPClient provided you set it up properly as a PUT and give it the data in the buffer object.

      • Mike Maerker said,

        Hi Almindor,
        it was my fault, but I forgot to say I want to upload via FTP.
        And my questtion – properly asked – should have been:
        do I have to use the ping-pong way or does http://FTP.put encapsulate
        the so called ping-pong?
        Sorry and thx a lot!
        Mike

  43. Mike Maerker said,

    Hi Almindor,

    maybe my last post did not com clear but did ask a question.
    It was not an urgent need, bacause my »transfer-project« paused for a while. But now it’s important for me to know how to proceed.
    Please tell me if I have to use the ping-pong way or does the »put method« encapsulate the ping-pong way?

    Thanks in advance and kind regards,
    Mike

    • almindor said,

      For FTP all you should need to upload is call http://FTP.Put (and if using commandline then a regular call to CallAction).

      • Mike Maerker said,

        Thanks Almindor,
        got it now.
        Btw – thanks a (very) lot for helping to desing an upload module for my billing application by providing the lnet package. …and kind regards to Slovakia.
        Mike

  44. Mirek said,

    Hi Almindor,
    Just a quick question: is it possible at all to use server and client sockets in one program at the same time?

    I’m a newbie in networking on PC, learnt the basics mostly from your Testnet demo program. It only uses one socket (LTCP and FNET components) and switches it between server and client role. I need to send data out as a client, while having an open server socket receiving data in the background all the time. When I created a second socket component, it sort of merged with the first one, firing its event handlers and all.

    Yes, I have a lot to learn. I’d just like to know whether I’m heading in the right direction :-).

    Thanks!

    • Mirek said,

      Never mind, problem solved. If I use the non-visual way to do the client stuff (as in the Client example program), it works.

      • almindor said,

        You should be able to place two separate visual components too and then assign separate handlers (or common ones where applicable, e.g. on error).

        You just need to use Listen on one and Connect on the other.

  45. Mirek said,

    Thank you. That means I just did it wrong, not that it’s impossible :-). Good to know.

  46. Mike Märker said,

    Hi Almindor,
    maybe it’s a (minor) bug, but…
    …when you “connect” in your FTP-Example the assigned Path (in the Site Manager) is opened correctly, but the log frame reports: 257 “/” is the current directory and the path in txtPath is set back to “/”.
    Regards,
    Mike

  47. Martin said,

    I have a really weird problem.
    When using a Windows CE scanner and a specific docking station (!), FTP connections do not work.
    I mean the TCP connection is established, but no communication takes place. I’m not sure if the 220 is received and USER doesn’t get sent out or if the 220 is not received.

    What’s even more weird is that the exact same scanner works in an older version of the docking station. AND even works in the new one, when I switch it to USB mode.

    Internet Explorer on that scanner works regardless.
    Something really fishy is going on.

    That application works fine even cross-compiled on Windows 7 btw.

    • Martin said,

      Answering myself:

      It seems it’s a bug inside the dock driver.
      I just added some debug output inside the GUI application. I hooked ready to send, data available to no avail. Then I went further back and noticed that I never get a “connection established” callback. When I then put some GUI code inside http://FTP.Connect() it started to work all of a sudden.

      So I added a Sleep(50) inside http://FTP.Connect() and it keeps working.

      It seems as if the driver has an issue when an application tries to connect to a port and then immediately checks if the connection was established. That somehow makes it go mad and then it will never tell the application that a connection was indeed established.

      Will probably create a small testcase code for the company of the scanner, so that they hopefully fix it. At least I have a workaround now to make it work.

  48. Norberto said,

    Hi, I’m making an IRC client based on a thread/loop which receives messages from the server.
    It works fine and the GUI updates with no problem except when I add a pause in the thread/loop.
    I managed to join an IRC crowded room and deactivate synchronize and I see a delay in the GUI update (like a TEdit getting frozen for moments.
    What am I doing wrong? Could be because a TLConnection is a visual component?
    Thanks in advance.

    constructor connex.Create(CreateSusupended:boolean; createnode: boolean; co: smallint; ip,port: string);
    begin
    inherited Create(true);
    FreeOnTerminate := false; // better code…

    //fEvent := TEventObject.Create(nil, True, False, ”);
    fEvent:= RTLEventCreate;

    //dat.n1:= nick;
    dat.ip:= ip;
    dat.port:= port;

    conn:= TLConnection.Create(fmainc);
    ltcp:= TLTCPComponent.Create(fmainc);
    ssl:= TLSSLSessionComponent.Create(fmainc);

    ltcp.OnConnect:= @OnConnect;
    ltcp.OnReceive:= @OnRecv;
    ltcp.OnError:= @OnErr;

    conn:= ltcp;
    start;
    end;

    // Synchronize
    procedure connex.sync;
    begin
    collector(a); // Process message and output
    end;

    procedure connex.Execute;
    var tmp: string;
    //n: integer = 0;
    begin
    while not (conn.Connected) do conn.CallAction;
    while (Terminated = false) do begin

    if not (fast) then RTLeventWaitFor(fEvent, 2000);

    if (cnt) then begin
    conn.GetMessage(a);
    tmp:= a;

    if a ” then
    Synchronize(@sync);
    a:= ”;

    end; //
    RTLeventResetEvent(fEvent);
    end; // Terminate
    end;

  49. lLama said,

    Hi, Almindor!

    I want to list all IP-addresses of all adapters on the local machine. With WinSock2 one can write something like:

    var
    i: Integer;
    wsDat: TWSAData;
    AHost: PHostEnt;
    PAdrPtr: PaPInAddr;
    Ch: array[0..250] of Char;
    begin
    FillChar(wsDat, SizeOf(TWSAData), 0);
    if WSAStartup(WINSOCK_VERSION, wsDat) SOCKET_ERROR then
    begin
    GetHostName(Ch, Length(Ch));
    AHost := GetHostByName(Ch);
    if AHost = nil then
    raise Exception.CreateFmt(‘Socket error (%d)’, [WSAGetLastError])
    else begin
    IPList.Clear;
    PAdrPtr := PAPInAddr(AHost^.h_addr_list);
    i := 0;
    while PAdrPtr^[i] nil do
    begin
    IPList.Items.Add(Format(‘%d.%d.%d.%d’,
    [Ord(TInAddr(PAdrPtr^[i]^).S_un_b.s_b1),
    Ord(TInAddr(PAdrPtr^[i]^).S_un_b.s_b2),
    Ord(TInAddr(PAdrPtr^[i]^).S_un_b.s_b3),
    Ord(TInAddr(PAdrPtr^[i]^).S_un_b.s_b4)]));
    Inc(i);
    end;
    end;
    WSACleanup;
    end;
    end;

    But I want to get such list in cross-platform manner (and possibly with lNet).
    Some suggestions can be found at (http://free-pascal-general.1045716.n5.nabble.com/lNet-getting-the-local-IP-td3200339.html) but they looks pretty ugly.
    So my question is is there any way to do it?

    Thanks!

    • almindor said,

      This is a bit out of scope for lNet as even the question of “what is a network interface” is not answerable cross-platform. Windows will give a different answer to Linux and embedded is it’s own beast of course. I wouldn’t be opposed if someone pushed a PR with a cross-platform method using OS specifics in the back but I suspect it’ll have to be at least 2-3 different implementations. Windows, Linux and BSDs. Mac might need more specific stuff too.

      Overall I think the best approach is the OS specific one and abstract it behind ifdefs. Sorry I couldn’t help more.

      • lLama said,

        I see…
        Thanks a lot for your reply!

  50. Martin said,

    Hey,

    I left a message on here some time ago.
    I finally figured out the UNDERLYING problem of it.

    The problem is that on these Windows CE scanners the messages are not in the right order. It seems to happen when the network connection is too fast. It’s quite obviously a network driver/Windows CE bug, but the company behind them told me that they don’t really care about fixing it, because the CE Internet Explorer works with it.

    You get a data received message before the connection established messages arrives.
    That makes the data received message get discarded and thus the FTP startup is ignored, which then makes the program wait endlessly.

    I added a boolean, where I remember that data has arrived and I check that boolean right when connection established was received and then act on it.

    This solves the problem 100%. I can send you the code if you want me to.

    Kind Regards

    • almindor said,

      Wow that’s quite a find! Glad it worked out in the end.

      • Martin said,

        You want me to send you the code?
        Please add a workaround for this case either inside the library, or let me send you mine. It was a major hassle to figure it out, especially because Windows CE is really limited in regards of debugging and as I said the company behind it was also uncooperative.
        It was the Datalogic scanner model “Memor”. The revision called “X3” has a few other CE bugs already (GUI related) and this error happens way way way more often on the hardware revision than on the original Memor hardware. We weren’t able to use the X3 model until I finally figured it out, because even the short delay did only sometimes fix it. Depending on the network connection it didn’t really help, which is why I looked into it once again in 2019.

  51. Martin said,

    I’m even surprised that Windows CE allows this to happen in the first place. Originally I never expected that to be the case.

  52. lijumen said,

    Hi,Thx for this great work。
    May i send fix size bytes in a package ,using tcp send function?
    when i call the send function more than onece,it seemed only one package be send,all data bytes merged in it。

    • almindor said,

      TCP does not support “messages”. That is anything you send via TCP is part of the same stream of bytes. You will need to make messaging work yourself by using a higher protocol or implementing something like count bytes at the start or a separator sequence.

      • lijumen said,

        THX for replay. My project is to communicate with an embedded device that is already set to TCP, but whose instruction format is a message, not a stream. So I can only look for a compromise. It is not known whether the “nodelay” parameter is useful.

    • bd4kc said,

      Can be set TCP_NODELAY with tcpserver?

      • almindor said,

        You mean Nadle’s algorithm? That might produce individual packets but it depends on payload size and congestion/scheduling can still mean that the other side gets both sends as one receive. You really should create a messaging protocol if you need to receive individual packets separately.


Leave a reply to almindor Cancel reply