response from GET request is truncated (RAK8211)

[espruino-discuss]

Posted at 2019-02-12 by Nadnerb

I am working on a RAK8211 module with Espruino 2v01.17

I have been able to successfully send data using the http module with a "POST" request.
I am now trying to receive data from a server. To do this I am sending this HTTP request:

GET /api/v1/devices/configuration HTTP/1.1
User-Agent: Espruino 2v01.17
Content-Type: application/json
Host: serverPath.com

This is the data I am expecting, and the data that I am actually receiving:

Expected:

[{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":true,"pressure":true,"temperature":true,"eventTriggers":true},{"type":"Actions","reboot":false,"dfwUpdate":false,"findDevice":false}]

Received:

[{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t

As you can see, the response is truncated.

I have tried this using both the http.get function and using the http.request function, both results in the same output.

What is interesting is that the content length is 300, but only 168 characters are printed.
When I do a print(res.available()) it returns 168

The response from the server has the following headers:

HTTP/1.1 200
Server: nginx/1.12.2
Date: Tue, 12 Feb 2019 06:28:31 GMT
Content-Type: application/json;charset=UTF-8
Content-Length: 300
Connection: close
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY

Why would the available bytes be showing only 168 when the Content-Length is 300 bytes?

Some more detailed information of things I've tried below:

[espruino-discuss]

Posted at 2019-02-12 by Nadnerb

Using http.get():

http.get(options, function(res) {
	print("available: " + res.available());						//returns 168
	print("Content-Length: " + res.headers['Content-Length']);	//returns 300
	print("Manual read: " + res.read())

	res.on('data', function(data) {
		print("\t Response: code: " + res.statusCode);      
		print("--> " + data);
	});

	res.on('close', (data) => print("close: " + data));	//for debugging
	res.on('drain', (data) => print("drain: " + data));	//for debugging
	res.on('error', (e) => print("Error: " + e));			//for debugging
});

req.on('error', function(e) {
	print("ERROR", e);
});

Results:

available: 168
Content-Length: 300
Manual Read: [{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t
code: 200
--> [{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t

Using http.request:

var req = http.request(options, function(res) {
	print("available: " + res.available());						//returns 168
	print("Content-Length: " + res.headers['Content-Length']);	//returns 300
	print("read: " + res.read())

	res.on('data', function(data) {
	  print("\t Response: " + "code: " + res.statusCode);
	  print("--> " + data);
	});
});

req.on('error', function(e) {
    print("error: " + e);
});

Results:

available: 168
Content-Length: 300
Manual Read: [{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t
code: 200
--> [{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t

I also tried concatenating a string with each data that is returned:

tempMsg = "";
var req = http.request(options, function(res) {
	var l = res.headers['Content-Length'];
	print("l = " + l);
	res.on('data', function(data) {
		tempMsg += data;
		if (tempMsg.length >=1) {
		print("tempMsg: " + tempMsg);
		print("End of Message");
		}
	});
	res.on('close', function(data) {
		print("\t Response: " + "code: " + res.statusCode);
	});
	res.on('error', (e)=> print("error: " + e));    //for debugging
});

req.on('error', function(e) {
	if (callback) callback(e.code, e.message);
});

req.end(body);

Results:

l = 300
[{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t
tempMsg: [{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t
End of Message
code: 200

---

Posted at 2019-02-12 by @gfwilliams

Can you try just going with what's done in all the examples? http://www.espruino.com/Internet

Eg. remove print("Manual read: " + res.read()), concat all the data, and ONLY print it on close:

http.get(options, function(res) {
    print("available: " + res.available());                     //returns 168
    print("Content-Length: " + res.headers['Content-Length']);  //returns 300
    var content = "";
    res.on('data', function(data) {  content += data; });
    res.on('close', (data) => print("close: " + content));
    res.on('error', (e) => print("Error: " + e));
});

I believe by doing the read you may be pulling data out that should be going to the data event, and when you are concatenating the data into tempMsg you seem to be printing End of Message after the very first data event.

But basically:

The callback for get/request gets called after the header is received - and not when all data has come in. It's then up to you to add a data handler and handle the data as it comes in.

The reasoning is it's pretty common for HTTP requests to send more data than would fit in the device's RAM, so by leaving the handling of the data up to you, you can decide what is done with it.

---

Posted at 2019-02-13 by Nadnerb

I forgot to mention that I had tried that one too.
Unfortunately I got the same result:

function getHttp(options, callback) {

  http.get(options, function(res) {
    print("available: " + res.available()); //returns 168
    print("Content-Length: " + res.headers['Content-Length']);  //returns 300
    var content = "";

    res.on('data', (data) => content+=data);
    res.on('close', (data) => print("close: " + content))
    res.on('error', (e) => print("Error: " + e));

  }).on('error', function(e) {
    print("ERROR", e);
  });
}

Results:

available: 168
Content-Length : 300
close: [{"type":"Positioning","accelerometer":true,"trilateration":true,"globalPositioning":true},{"type":"EventCadence","smartMode":true},{"type":"Environmental","humidity":t

I only added the "manual read" after it failed the first time, just to see if I could force it to read all 300 characters, which didnt work either because theres only 168 bytes available.

---

Posted at 2019-02-13 by Nadnerb

I turned on debug statements in the lower level code (QuectelBC95.js) and there seems to be an issue there. I'll take a look at that instead.

---

Posted at 2019-02-13 by @gfwilliams

Ok - what's your full code? https://www.espruino.com/RAK8211 says you should be using QuectelM35 for the 8211-G, not QuectelBC95 (in fact there is no BC95 module?)

Are you definitely using firmware 2v01? I seem to recall there was some issue like this with one of the RAK boards in the past but I was pretty sure it got fixed.

---

Posted at 2019-02-13 by Nadnerb

Hi Gordon,
We made our own QuectelBC95.js file, because our device uses a BC95-G module.

The low level code that we've written in this file is what is causing issues, specifically in the receive handler. So its nothing with Espruino that is the issue. I have figured out a solution now, by using the "+NSONMI" receive handler to output the number of bytes received and then retrieve them at a later time using "AT+NSORF", rather than retrieve them in the receive handler itself.

Thanks for your help.