Read Timeout TIdTCPClient

This is the forum for miscellaneous technical/programming questions.

Moderator: 2ffat

Read Timeout TIdTCPClient

Postby d7d1cd » Tue Dec 17, 2019 1:08 am

Good day. I use the TIdTCPClient component to send requests to the server and read the response. I know the size of the response for certain requests, but not for others.

When I know the size of the response, then my data reading code looks like this:
Code: Select all
IdTCPClient1->Socket->Write(requestBuffer);
IdTCPClient1->Socket->ReadBytes(answerBuffer, expectSize);

When the size of the response is not known to me, then I use this code:
Code: Select all
IdTCPClient1->Socket->Write(requestBuffer);
IdTCPClient1->Socket->ReadBytes(answerBuffer, -1);

In both cases, I ran into problems.
In the first case, if the server does not return all the data (less than expectSize), then IdTCPClient1 will wait for ReadTimeout to finish, but there will be no data at all in the answerBuffer (even if the server sent something). Is this the logic behind TIdTCPClient? It is right?

In the second case, ReadTimeout does not work at all. That is, the ReadBytes function ends immediately and nothing is written to the answerBuffer, or several bytes from the server are written. However, I expected that since this function in this case does not know the number of bytes to read, it must wait for ReadTimeout and read the bytes, who came during this time. For the experiment, I inserted Sleep (500) between writing and reading, and then I read all the data that arrived.
May I ask you to answer why this is happening?
d7d1cd
 
Posts: 9
Joined: Tue Dec 17, 2019 12:41 am

Re: Read Timeout TIdTCPClient

Postby rlebeau » Tue Dec 17, 2019 1:19 pm

d7d1cd wrote:Good day. I use the TIdTCPClient component to send requests to the server and read the response. I know the size of the response for certain requests, but not for others.


Why do you not know the size of all of the responses? What does your protocol actually look like? TCP is a byte stream, each message MUST be framed in such a way that a receiver can know where each message begins and ends in order to read the messages correctly and preserve the integrity of the stream. As such, messages MUST either include their size in their payload, or be uniquely delimited between messages. So, which is the case in your situation? It doesn't sound like you are handling either possibility.

d7d1cd wrote:When the size of the response is not known to me, then I use this code:
Code: Select all
IdTCPClient1->Socket->Write(requestBuffer);
IdTCPClient1->Socket->ReadBytes(answerBuffer, -1);



When you set AByteCount to -1, that tells ReadBytes() to return whatever bytes are currently available in the IOHandler's InputBuffer. If the InputBuffer is empty, ReadBytes() waits, up to the ReadTimeout interval, for at least 1 byte to arrive, and then it returns whatever bytes were actually received into the InputBuffer, up to the maximum specified by the IOHandler's RecvBufferSize. So it may still take multiple reads to read an entire message in full.

In general, you should NEVER set AByteCount to -1 when dealing with an actual protocol. -1 is good to use only when proxying/streaming arbitrary data, where you don't care what the bytes actually are. Any other use require knowledge of the protocol's details of how messages are framed.

d7d1cd wrote:In the first case, if the server does not return all the data (less than expectSize), then IdTCPClient1 will wait for ReadTimeout to finish, but there will be no data at all in the answerBuffer (even if the server sent something). Is this the logic behind TIdTCPClient? It is right?


Yes. When AByteCount is > 0, ReadBytes() waits for the specified number of bytes to be available in the InputBuffer before then extracting that many bytes into your output TIdBytes. Your answerBuffer will not be modified unless all of the requested bytes are available. If the ReadTimeout elapses, an EIdReadTimeout exception is raised, and your answerBuffer is left untouched.

If that is not the behavior you want, then consider using ReadStream() instead of ReadBytes(), using a TIdMemoryBufferStream or TBytesStream to read into.

d7d1cd wrote:In the second case, ReadTimeout does not work at all. That is, the ReadBytes function ends immediately and nothing is written to the answerBuffer.


I have never heard of ReadBytes() not waiting for the ReadTimeout. What you describe should only happen if there are no bytes available in the InputBuffer and the ReadTimeout is set to some very small value, like 0 msecs.

d7d1cd wrote:or several bytes from the server are written.


That is a perfectly reasonable outcome given you are asking ReadBytes() to read an arbitrary number of bytes between 1..RecvBufferSize inclusive, or read no bytes if the timeout elapses.

d7d1cd wrote:However, I expected that since this function in this case does not know the number of bytes to read, it must wait for ReadTimeout and read the bytes, who came during this time.


That is how it should be working, yes. And how it has always worked. So I suggest you debug into ReadBytes() at runtime and find out why it is not working the way you are expecting. Also, are you using an up-to-date version of Indy to begin with (or at least a version from the last few years)? And what platform are you running your code on when you are having this trouble?
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1648
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: Read Timeout TIdTCPClient

Postby d7d1cd » Tue Dec 17, 2019 11:57 pm

rlebeau wrote:Why do you not know the size of all of the responses?

Because, in fact, I'm doing a survey of an electronic device. This device has its own network IP address and port. So, the device can respond to the same request in different ways, depending on its status. Strictly speaking, there can be two answers to some queries and they have different lengths. It is in these cases, when reading, I specify AByteCount = -1 to read any device response.

rlebeau wrote:I have never heard of ReadBytes() not waiting for the ReadTimeout.

You're right! I was wrong. When specifying AByteCount = -1, I get one byte. As you said, if at least one byte arrives, it returns and ReadBytes() ends.

rlebeau wrote:Also, are you using an up-to-date version of Indy to begin with (or at least a version from the last few years)? And what platform are you running your code on when you are having this trouble?

I work in C ++ Builder 10.3 Community Edition, Indy version 10.6.2.5366. I create an application for Windows.

rlebeau wrote:If that is not the behavior you want, then consider using ReadStream() instead of ReadBytes(), using a TIdMemoryBufferStream or TBytesStream to read into.

Is the behavior of ReadStream() different from ReadBytes()? I thought these were just different ways of reading information from a socket...
Thank you invaluable help!
d7d1cd
 
Posts: 9
Joined: Tue Dec 17, 2019 12:41 am

Re: Read Timeout TIdTCPClient

Postby rlebeau » Wed Dec 18, 2019 11:07 am

d7d1cd wrote:Because, in fact, I'm doing a survey of an electronic device. This device has its own network IP address and port. So, the device can respond to the same request in different ways, depending on its status.


That does not necessarily mean that there is not a structure to the responses. Again, what does the actual protocol look like?

d7d1cd wrote:Strictly speaking, there can be two answers to some queries and they have different lengths.


And? Just because they have different lengths does not necessarily mean there is no way to know what those lengths actually are every time.

d7d1cd wrote:It is in these cases, when reading, I specify AByteCount = -1 to read any device response.


Without knowing what the protocol actually looks like, I'm not convinced that is the best solution. In fact, it is rarely ever the correct solution.

d7d1cd wrote:You're right! I was wrong. When specifying AByteCount = -1, I get one byte. As you said, if at least one byte arrives, it returns and ReadBytes() ends.


OK.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1648
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: Read Timeout TIdTCPClient

Postby d7d1cd » Thu Dec 19, 2019 12:11 am

rlebeau wrote:Again, what does the actual protocol look like?

A protocol similar to MODBUS. There is no criterion for ending the sequence of answers in it. Except for the criterion of cyclic redundancy code (CRC16) at the end of each response.
Although, probably, I confused you. The protocol of exchange with the device ensures that for each request there will be a response of a specific length. However, there may be the following situation. Before sending requests to the device, you must log in to the device. That is, you need to send a special access request. If this is not done, then the device will respond to any other request with the same sequence with the access error code. Therefore, before sending requests, of course, you need to open the communication channel and then the response from the device will be expected. Thus, I misled you. Of course, knowing the request and observing the order during the exchange (first authorization, then everything else), the length of the answers will be known.
Although there is one situation where the length of the response will not be known. This is if you create an application that allows the user to form queries to the device by simply typing the command in hexadecimal numbers. Here we can say that we need to analyze the request and determine how long the device should return, but what if the application is universal? What if it is designed simply to send requests and read responses?
And? Just because they have different lengths does not necessarily mean there is no way to know what those lengths actually are every time.

Allow another question about the operation of the ReadStream function.
Wrote the following code:
Code: Select all
TIdTCPClient* IdTCPClient1 = new TIdTCPClient();
IdTCPClient1->ReadTimeout = 5000;
TByteDynArray request, answer, tmp;
TBytesStream* bytesStream = new TBytesStream(tmp);
int expect;

// Some code

expect = 8;
try {
  IdTCPClient1->Connect();
  IdTCPClient1->Socket->Write(request);
  IdTCPClient1->Socket->ReadStream(bytesStream, expect);
} catch(...) {}

answer = bytesStream->Bytes;
answer.set_length(bytesStream->Size);

A request is sent to the Write function, to which the device gives 8 bytes of response. If expect = 8, then everything works fine. If expect = -1, then the ReadStream function, as expected, waits 5 seconds, but only 4 bytes are written to bytesStream, not 8. Why is this happening?
And one more thing: why do I need to pass a parameter to the TBytesStream constructor (in my tmp code)? Why can't you create an empty stream?
d7d1cd
 
Posts: 9
Joined: Tue Dec 17, 2019 12:41 am

Re: Read Timeout TIdTCPClient

Postby rlebeau » Fri Dec 20, 2019 3:38 pm

d7d1cd wrote:There is no criterion for ending the sequence of answers in it. Except for the criterion of cyclic redundancy code (CRC16) at the end of each response.


I seriously doubt that. There HAS to be a way to know what each message length is. The actual MODBUS TCP/IP protocol does exactly that. It uses a structure where every message begins with a fixed 7-byte header:

2-byte Transaction ID
2-byte Protocol ID
2-byte Length Field
1-byte Unit ID

The Length Field is the size of the MODBUS data that follows the header (the MODBUS Address and Checksum fields are not transmitted over TCP). So, a receiver would simply read the first 6 bytes, then read the number of bytes specified in the Length Field to finish the message (the length includes the Unit ID).

I strongly suspect you need to do something similar in your situation, but I can't be sure since you have not provides ANY details about the actual protocol you are dealing with. Saying it is "similar to MODBUS" is not the same as "actually being MODBUS". So what is different in this situation? Do you have ANY documentation about your device's actual transmission protocol?

d7d1cd wrote:The protocol of exchange with the device ensures that for each request there will be a response of a specific length. However, there may be the following situation. Before sending requests to the device, you must log in to the device. That is, you need to send a special access request. If this is not done, then the device will respond to any other request with the same sequence with the access error code. Therefore, before sending requests, of course, you need to open the communication channel and then the response from the device will be expected.


That does not change anything I have said so far. EVERY message back and forth has a layout to it. You need to write your code to follow that layout so you know where each message begins and ends.

d7d1cd wrote:Of course, knowing the request and observing the order during the exchange (first authorization, then everything else), the length of the answers will be known.


It is not common for a TCP-based protocol to dictate every request has a specific fixed-length response, like you describe. Most TCP-based protocol are more generalized than that, using a single unified format for every message. So again, I ask you, what are the specific details about the ACTUAL PROTOCOL that your device is really using? I can't help you fix your Indy code without that information.

d7d1cd wrote:Although there is one situation where the length of the response will not be known. This is if you create an application that allows the user to form queries to the device by simply typing the command in hexadecimal numbers. Here we can say that we need to analyze the request and determine how long the device should return, but what if the application is universal? What if it is designed simply to send requests and read responses?


Even if you allow the user to format their own queries, your code is still responsible for reading the responses before presenting them to the user. So your code is responsible for reading the responses correctly.

d7d1cd wrote:Allow another question about the operation of the ReadStream function.
...
A request is sent to the Write function, to which the device gives 8 bytes of response. If expect = 8, then everything works fine. If expect = -1, then the ReadStream function, as expected, waits 5 seconds, but only 4 bytes are written to bytesStream, not 8. Why is this happening?


The only possibility is that only 4 bytes were actually available after the timeout elapsed.

As I said earlier, using AByteCount=-1 returns whatever bytes are available at that very moment. If there are no bytes available yet, then it waits for the ReadTimeout to elapse and then returns whatever bytes are available.

Using AByteCount>0 instead, it actually waits for all of the requested bytes to arrive, however long they may take (subject to the ReadTimeout, of course).

I strongly suspect that using AByteCount=-1 is the WRONG solution to your situation.

d7d1cd wrote:why do I need to pass a parameter to the TBytesStream constructor (in my tmp code)?


Because that is just the way TBytesStream works. It is a wrapper for a byte array, so you have to give it an initial byte array to work with, even if it is just an empty one. TBytesStream does not have a parameter-less constructor. If you want that, use TMemoryStream instead.

d7d1cd wrote:Why can't you create an empty stream?


You can, if you give it an empty byte array. That is just the way TBytesStream was coded to operate.
Last edited by rlebeau on Mon Dec 23, 2019 1:26 pm, edited 1 time in total.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1648
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: Read Timeout TIdTCPClient

Postby d7d1cd » Sun Dec 22, 2019 11:33 pm

rlebeau wrote:Do you have ANY documentation about your device's actual transmission protocol?

Yes, of course, there is a protocol, only it is in Russian. The device has 2 communication interfaces: an infrared transceiver, as well as a wired RS-485, work through which is carried out via the COM port. Access to the device via TCP is carried out through additional equipment that creates a “transparent” communication channel, due to which data exchange with the device via TCP is no different from working with it via RS-485 or infrared channel. I can provide you with a protocol translated through a Google translator.

rlebeau wrote:The only possibility is that only 4 bytes were actually available after the timeout elapsed.
As I said earlier, using AByteCount=-1 returns whatever bytes are available at that very moment. If there are no bytes available yet, then it waits for the ReadTimeout to elapse and then returns whatever bytes are available.

That is, in fact, the behavior of ReadStream is no different from ReadBytes?

rlebeau wrote:That is just the way TBytesStream was coded to operate.

OK.
d7d1cd
 
Posts: 9
Joined: Tue Dec 17, 2019 12:41 am

Re: Read Timeout TIdTCPClient

Postby rlebeau » Mon Dec 23, 2019 1:30 pm

d7d1cd wrote:Yes, of course, there is a protocol, only it is in Russian.


So what? Never heard of online translators, like Google Translate, Bing Translator, etc?

d7d1cd wrote:The device has 2 communication interfaces: an infrared transceiver, as well as a wired RS-485, work through which is carried out via the COM port. Access to the device via TCP is carried out through additional equipment that creates a “transparent” communication channel, due to which data exchange with the device via TCP is no different from working with it via RS-485 or infrared channel.


All of that information is irrelevant to this discussion. The only thing that matters right now is the format of the messages that are being transmitted over TCP. Please only focus on that.

d7d1cd wrote:I can provide you with a protocol translated through a Google translator.


Please do.
Last edited by rlebeau on Tue Dec 24, 2019 2:06 pm, edited 1 time in total.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1648
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: Read Timeout TIdTCPClient

Postby d7d1cd » Mon Dec 23, 2019 11:20 pm

rlebeau wrote:Please do.

I am sending you the exchange protocol in the original. I wanted to send in the translated version, but I could not copy the translated text from the online translator.
Attachments
Protocol.zip
(250.67 KiB) Downloaded 171 times
d7d1cd
 
Posts: 9
Joined: Tue Dec 17, 2019 12:41 am

Re: Read Timeout TIdTCPClient

Postby rlebeau » Tue Dec 24, 2019 2:15 pm

d7d1cd wrote:I am sending you the exchange protocol in the original. I wanted to send in the translated version, but I could not copy the translated text from the online translator.


Here is an English PDF that I translated from the original using this Russian-to-English translator.

I'll dig into the PDF more after Christmas. But from a cursory glance, it appears that you are likely going to have to know ahead of time which exact response you are reading in order to know how many bytes the response contains. I think (I may be wrong) that the first byte is a sequence number that is copied from the request. So, you might have to keep track of the requests you send, and how many bytes each expected response is, then when reading a response read just the 1st byte and match it to your list of requests, and then read the expected number of bytes for that response.

This is not a very well-designed protocol for TCP usage.
Attachments
translated.7z
(254.94 KiB) Downloaded 166 times
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1648
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: Read Timeout TIdTCPClient

Postby d7d1cd » Wed Dec 25, 2019 11:39 pm

rlebeau wrote:I think (I may be wrong) that the first byte is a sequence number that is copied from the request.

The first byte is the so-called network address. You probably already understood that the protocol describes an electric energy meter (counter). So, there can be many of these counters in one network (RS-485). To address a request to a specific one, its network address is indicated, which are programmed different for all counters on the same network.

rlebeau wrote:So, you might have to keep track of the requests you send, and how many bytes each expected response is, then when reading a response read just the 1st byte and match it to your list of requests, and then read the expected number of bytes for that response.

Information exchange with the device is carried out synchronously. That is, if a request is sent, then you must immediately, given the timeouts, read the answer.

rlebeau wrote:This is not a very well-designed protocol for TCP usage.

This protocol was not designed to work through the TCP. It is necessary to work through the TCP here in the same way as through the COM port in synchronous mode.
d7d1cd
 
Posts: 9
Joined: Tue Dec 17, 2019 12:41 am

Re: Read Timeout TIdTCPClient

Postby rlebeau » Sun Dec 29, 2019 5:50 pm

d7d1cd wrote:Information exchange with the device is carried out synchronously. That is, if a request is sent, then you must immediately, given the timeouts, read the answer.


Well, then there shouldn't be any problem, since you know what kind of request you are sending each time, so you should know what kind of response to expect and how many bytes it requires to read. If a timeout occurs, all you can do is close the connection and move on. So what are you actually having trouble with exactly?

d7d1cd wrote:This protocol was not designed to work through the TCP.


TCP wrappers do usually require adequate message framing, though. Which this protocol does not seem to provide.

d7d1cd wrote:It is necessary to work through the TCP here in the same way as through the COM port in synchronous mode.


Even COM port protocols usually use adequate message framing.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1648
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA


Return to Technical

Who is online

Users browsing this forum: Baidu [Spider] and 7 guests

cron