Newbie Espruino Wifi createServer Problem?

[espruino-discuss]

Posted at 2017-08-03 by seanclark

I'm loving my Espruino Wifi. However, I'm currently trying to do something very simple but can't see why it's not working. I'm simply trying to create a server and listen on port 80. I've written a simple script based on the examples:

var WIFI_NAME = "Kraken";
var WIFI_OPTIONS = { password : "xxx" };

var wifi;
var config = {
  'Title': 'Test Piece',
  'Network': WIFI_NAME,
  'macaddress': '',
  'ipaddress': ''
};

function onInit() {
  wifi = require("EspruinoWiFi");
  wifi.connect(WIFI_NAME, WIFI_OPTIONS, function(e) { 
    if (e) {
      console.log("Connection Error: "+e);
      return;
    }
    wifi.getIP(function(f,ip) {
      config.macaddress = ip.mac;
      config.ipaddress = ip.ip;
      console.log("Connected: ",config);
    });
    require("http").createServer(function(req,res) {
      console.log(req);
      res.writeHead(200);
      res.end("Hello World");
    }).listen(80);
  });
}

onInit();

But get this error when I try to connect to the device from Chrome:

>
 _____                 _
|   __|___ ___ ___ _ _|_|___ ___
|   __|_ -| . |  _| | | |   | . |
|_____|___|  _|_| |___|_|_|_|___|
          |_| http://espruino.com
 1v91 Copyright 2016 G.Williams
>
=undefined
Connected:  {
  "Title": "Test Piece",
  "Network": "Kraken",
  "macaddress": "5c:cf:7f:c0:f4:85",
  "ipaddress": "192.168.1.111"
 }
>Uncaught Error: CIPSERVER failed (Timeout)
 at line 1 col 53
throw Error("CIPSERVER failed ("+(a?a:"Timeout")+")");
                                                    ^
in function called from system
 

You'll see it's connecting to wifi OK. I did discover that the examples I were using needed to be modified to work with the Espruino Wifi chip, but what else am I missing?

Thanks,

Sean

[espruino-discuss]

Posted at 2017-08-03 by @gfwilliams

It shouldn't really be an issue, but it might be because you're calling getIP and then createserver right after (rather than using callbacks).

You could try the following and see if that works better:

function onInit() {
  wifi = require("EspruinoWiFi");
  wifi.connect(WIFI_NAME, WIFI_OPTIONS, function(e) { 
    if (e) {
      console.log("Connection Error: "+e);
      return;
    }
    wifi.getIP(function(f,ip) {
      config.macaddress = ip.mac;
      config.ipaddress = ip.ip;
      console.log("Connected: ",config);
      require("http").createServer(function(re­q,res) {
        console.log(req);
        res.writeHead(200);
        res.end("Hello World");
      }).listen(80);
    });    
  });
}

Let me know how you get on - if that's the issue I'll see what I can do to fix it in the driver

---

Posted at 2017-08-03 by seanclark

Mmm. Just tried that, and then commenting out the whole getIP bit. Now I just get a non-loading page that times out in the end. I popped a console.log in the createServer callback function and it's definitely not being called when I make the request.

Sean

---

Posted at 2017-08-03 by @gfwilliams

But you don't get an error now? Can you try updating the firmware to 1v92 just in case?

I just tried that exact code here (the original code and my suggested change) with a Wifi board running 1v93 and both of them are working perfectly for me, so at the moment that's the only difference I can think of.

---

Posted at 2017-08-03 by seanclark

Very strange. I updated one of my boards to 1v93 and it still doesn't respond on port 80. The other board won't go in to flashing mode.

One thing I noticed, I was having problems with the neopixel library too. It seemed to think I was trying to use an SD card and was not loading the library. However with the update neopixel now works.

Here is my latest code:

// Neopixel Matrix Controller
// 3rd August 2016
// Sean Clark
// HTTP Version

var WIFI_NAME = "Kraken";
var WIFI_OPTIONS = { password : "xxx" };
var MATRIXPIN = B5;
var MATRIXSIZE = 256;
var SERVERIP = "192.168.1.101";

var wifi;
var config = {
  'Title': 'Test Piece',
  'Network': WIFI_NAME,
  'macaddress': '',
  'ipaddress': '',
  'type': 'neopixel',
  'width': 16,
  'height': 16
};

var matrix = new Uint8ClampedArray(MATRIXSIZE*3);

function processRequest(req,res) {
  console.log(req);
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end("Hello World");
}

function onInit() {
  neopixel = require("neopixel");
  neopixel.write(MATRIXPIN,[255,255,0]); // On
  wifi = require("EspruinoWiFi");
  wifi.connect(WIFI_NAME, WIFI_OPTIONS, function(e) { 
    if (e) {
      console.log("Connection Error: "+e);
      neopixel.write(MATRIXPIN,[255,0,0]); // Connection Error
      return;
    }
    wifi.getIP(function(f,ip) {
      config.macaddress = ip.mac;
      config.ipaddress = ip.ip;
      console.log("Connected: ",config);
      require("http").createServer(processRequest).listen(80);
      neopixel.write(MATRIXPIN,[0,0,255]); // Connected
    });
  });
}

onInit();

---

Posted at 2017-08-03 by @gfwilliams

The neopixel library only got introduced in 1v92, so that might be why you had problems on one board? (you could do neopixels before, it just wasn't as 'nice').

Just tried your code, and it runs and says hello world over Wifi again (I haven't wired up any neopixels to it though)

Does it say something like the following after you upload?

Connected:  {
  "Title": "Test Piece",
  "Network": "Kraken",
  "macaddress": "5c:cf:7f:c0:f4:85",
  "ipaddress": "192.168.1.111",
  "type": "neopixel",
  "width": 16, "height": 16 }

Is it possible that your PC and the Wifi are on separate subnets, or there's some firewall software or something that might be stopping you accessing it? Is it possible you're trying to access https://192.168.1.111 and not http://192.168.1.111?

What if you try and create a server on a separate port, just in case?

---

Posted at 2017-08-03 by seanclark

A couple more observations. On the other Espruino WiFi when I hold the button down an add power the green light goes on, then the red light flashes, then they both go off. I can't then update it.

Once I've uploaded my software the Espruino should run it without the computer connected I assume? I'm finding that if I plug my device in to power then the script does not run. If I connect it to my computer and connect to it in the IDE it does run. Am I missing something here too?

Sean

---

Posted at 2017-08-03 by @gfwilliams

when I hold the button down an add power the green light goes on, then the red light flashes, then they both go off.

After about 3 seconds? I think you're just holding down the button for too long. Plug it in with the button held down, then release the button as soon as the green LED is lit.

If you hold down the button for > 3 seconds it continues to boot Espruino, but without loading any of the saved code (it's a recovery thing in case you saved code that caused problems).

Once I've uploaded my software the Espruino should run it without the computer connected I assume?

Usually your upload goes into RAM. To save to flash you'll want to just type save() on the left-hand side after uploading (but remove the call to onInit() that you put at the end of your code).

---

Posted at 2017-08-03 by seanclark

Correct! I was holding the button down for too long!

I have progress with my networking problem.

For some reason my Mac and PC refuse to connect to the Espruino. Neither Chrome or curl requests get through on both machines. However, other devices such as my iPhone and iPad connect fine. The Mac and PC are on the same WiFi/subnet as the other devices.

Maybe it's down to my BT router? Perhaps the laptops are on 5GHz and the other devices on 2.4? Would this cause a problem? I've tried http and https. I'll try to work out why, but for now I'm happy to test on my iPad.

Despite this problem I've managed to create a webpage with a canvas on that lets you draw on a 16x16 LED matrix in just a couple of hours. It's very odd (in a good way!) writing JavaScript for both the user interface and microcontroller backend.

I'm using http for now and have noticed that not all requests seem to get through when I'm drawing quickly. How many requests per second should the Espruino be able to handle? I'm only POSTing 20 bytes or so, but there can be 20 per second. I'll explore sockets next,

Cheers,

Sean

---

Posted at 2017-08-04 by @gfwilliams

Great! That's really strange... I guess it could be a firewall issue? It's possible it's some new 'feature' that block connections to other devices on the local network? I imagine if you're connected to the public BTWiFi rather than BTHub then that'd do it?

I think realistically Espruino will struggle at much past 5 requests/sec. It's not just the 20 bytes, but the headers that get sent are usually pretty huge each time (plus there's extra for opening/closing the socket) - and the connection to the Wifi is usually only around 11 kBytes/sec.

I guess you could actually rate-limit your HTTP requests to 2 a second or so, but send the whole frame each time?

Or yes, websockets would be the way to go - you should be able to push packets of data through that pretty fast.

Also, I don't know if you're doing it or not, but you can use 'template strings' that use the backtick character for HTML: `

They can be multi-line, so it gets really nice and easy to write actual HTML alongside the JS code

---

Posted at 2017-08-04 by seanclark

Thanks Gordon,

I've created another WiFi network with an Apple Airport hub and everything is fine now. There is definitely something 'in' my BT hub that is stopping the connections. I'll have a poke about.

WebSockets is not working as expected at the moment. I'm wanting to create a server on the Expruino and then connect to it from a client running in Chrome.

On the Chrome side I connect like this:

	var socket = new WebSocket("ws://"+matrixip);
	socket.onopen = function(event) {
  		console.log("Connected to WebSocket");
	};
	socket.onerror = function(e) {
	  console.log('WebSocket Error: '+e);
	};

And later on simply:

socket.send("Hello World!");

And on the Espruino:

function onInit() {
  neopixel = require("neopixel");
  neopixel.write(MATRIXPIN,[255,0,0]); // On
  wifi = require("EspruinoWiFi");
  wifi.connect(WIFI_NAME, WIFI_OPTIONS, function(e) { 
    if (e) {
      console.log("Connection Error: "+e);
      neopixel.write(MATRIXPIN,[0,255,0]); // Connection Error
      return;
    }
    wifi.getIP(function(f,ip) {
      info.macaddress = ip.mac;
      info.ipaddress = ip.ip;
      console.log("Connected: ",info);
      var server = require("ws").createServer(processRequest);
      server.listen(80);
      server.on("websocket", function(ws) {
        console.log("Connection upgraded");
        ws.on('message',function(msg) {
          console.log("Message received");
          print("[WS] "+JSON.stringify(msg));
        });
        ws.send("Hello from Espruino!");
      });
      neopixel.write(MATRIXPIN,[0,0,255]); // Connected
    });
  });
}

But the connection never completes from Chrome with this error:

'ws://192.168.1.111/' failed: Error during WebSocket handshake: Response must not include 'Sec-WebSocket-Protocol' header if not present in request: undefined

Any thoughts?

Sean

---

Posted at 2017-08-04 by @gfwilliams

Which version of chrome do you have? I just bodged your code into something using the websocket example HTML, and this works for me:

var WIFI_NAME = "";
var WIFI_OPTIONS = { password : "" };
var MATRIXPIN = B5;
var info = {};

var page = `<html><body><script>var ws;setTimeout(function(){
ws = new WebSocket("ws://" + location.host + "/my_websocket", "protocolOne");
ws.onmessage = function (event) { console.log("MSG:"+event.data); };
setTimeout(function() { ws.send("Hello to Espruino!"); }, 1000);
},1000);</script></body></html>`;

function processRequest(req,res) {
  console.log(req);
  res.writeHead(200, {'Content-Type': 'text/html'});
  res.end(page);
}

function onInit() {
  neopixel = require("neopixel");
  neopixel.write(MATRIXPIN,[255,0,0]); // On
  wifi = require("EspruinoWiFi");
  wifi.connect(WIFI_NAME, WIFI_OPTIONS, function(e) { 
    if (e) {
      console.log("Connection Error: "+e);
      neopixel.write(MATRIXPIN,[0,255,0]); // Connection Error
      return;
    }
    wifi.getIP(function(f,ip) {
      info.macaddress = ip.mac;
      info.ipaddress = ip.ip;
      console.log("Connected: ",info);
      var server = require("ws").createServer(processRequest);
      server.listen(80);
      server.on("websocket", function(ws) {
        console.log("Connection upgraded");
        ws.on('message',function(msg) {
          console.log("Message received");
          print("[WS] "+JSON.stringify(msg));
        });
        ws.send("Hello from Espruino!");
      });
      neopixel.write(MATRIXPIN,[0,0,255]); // Connected
    });
  });
}

onInit();

How are you accessing the server? I wonder if something differs when it's cross-origin?

---

Posted at 2017-08-04 by @gfwilliams

Ok, so it looks like a bug. This works:

new WebSocket("ws://" + location.host , "anything");

this doesn't:

new WebSocket("ws://" + location.host);

---

Posted at 2017-08-04 by seanclark

Thanks.

Yes, I am trying to access the server from a page on a web server, viewed through Chrome on my Mac and sending messages to my Espruino. This would be classed as cross origin I guess?

Sean

---

Posted at 2017-08-04 by @gfwilliams

I've filed a bug for it here: espruino/EspruinoDocs#375

But it's pretty easy to work around just by adding a random protocol name as a second argument to the WebSocket constructor

---

Posted at 2017-08-05 by seanclark

Lots of progress made. I am loving Espruino!

Most of my communications problems seem to be down to my BT router. I've created a separate wifi network for running the Espruino set-up on on and everything is working as expected. There must be some 'security' settings in the BT router that are messing things up.

My permission problems when POSTing via XHRs in the webpage to the Espruino WiFi were fixed by including. Access Control info in the response header:

res.writeHead(200, {
    'Access-Control-Allow-Origin': '*',
    'Content-Type': 'application/json',
  });

I'm happily making websocket connections from a page in Chrome (not being served by the Espruino) to the Espruino. It's much faster than using HTTP, but I still get some packet loss. How many messages per second should the Espruino ws server be able to handle? Also, can my client tell when a packet hasn't got through? Would I have to send a message back from the Espruino to do this?

I find that some of my packets are coming through mashed up though. I'm expecting 2 digit hex chars in my message (four pairs, like "0FEFDE99") but sometimes the Espruino ends up with:

Hex Error: ÿM
Hex Error: F�
Hex Error: õ�
Hex Error: ¨T
Hex Error: Qâ
Hex Error: V�
Hex Error: ZD

The chars after "Hex Error:" are what's being received and that my function is trying to turn in to decimal. The packets are definitely being sent correctly.

I'm also getting these error messages every now and then on the Espruino:

WS: Unknown opcode 13
WS: Unknown opcode 15
WS: Unknown opcode 4

I'm pretty close to having a little web drawing program talking to a Espruino controlling a neopixel grid. Not bad for a couple of days work. I'll do some tidying up at some point and share it.

Nice platform.

Sean

---

Posted at 2017-08-07 by @gfwilliams

How fast are you sending packets out? You could check E.getErrorFlags() to see if you're getting any kind of buffer overflow errors that'd be a sign that you're losing data... That would explain the hex errors and unknown opcode errors.

Espruino's connection to the WiFi is only 11kBytes/sec - and there's a bit of overhead, in websocket packets as well as actual send data commands. So if you're sending 100 bytes, realistically I'd say it's 50 packets absolute max, and probably less.

I've actually just looked at this and there is a big gotcha though. Right now, in order to get reliable neopixel output, interrupts are disabled while outputting to the neopixel strings. Normally it's quite quick and wouldn't be a problem, but you are sending quite a lot of data to neopixels, and that means that realistically if the Wifi sends any data to Espruino during that time, it could get lost.

If you try the build here (in a few minutes - it takes time to build): https://www.espruino.com/binaries/travis/6773f83c2fb53f314724c01d3c7bd123c7586927

Then it might solve the problems - it should have re-enabled the IRQs. However, it could potentially make the neopixel update glitchy - so can you let me know what happens please?

---

Posted at 2017-08-07 by seanclark

Thanks, I'll give this a go. My next experiment is to change the topology of my network so that rather than sending data directly from the web browser to the Espruino via Websockets I will use Websockets to send data from the browser to a node.js server running on a Pi and then use standard sockets to send data from the Pi to the Espruino Wifi. Do you think I will see a change in performance? This will solve my Safari problem as well I think without needing to do the HTTP/1.1 hack.

I also need to get DMX light control working (as an alternative to neopixels). Can you foresee any issues here?

Sean

---

Posted at 2017-08-07 by seanclark

Another thought. If a network packet is lost during the update of the neopixels is there a way of the sender knowing? At the moment I'm not getting an error message. Should I implement some sort of "acknowledge" on the Espruino so that the sender knows when to resend a packet?

Sean

---

Posted at 2017-08-07 by @gfwilliams

You should be fine with DMX sending, although the page on it on Espruino.com only covers receive: http://www.espruino.com/DMX

As far as I know you just send a pulse, and then bunch of data down serial. I don't have any DMX receiver stuff to test with though.

I'm not sure changing to having a separate server will help that much - unless you can ensure that you send the update command and then wait a while.

The issue isn't really losing a single packet, but losing part of a packet - because then the websockets get out of sync. I'm afraid there's no way to detect that at the moment.

Did you get a chance to test out my changes?

---

Posted at 2017-08-13 by seanclark

I've just gone to install the new firmware, but I'm not sure which on in the list to use? Is it the specific _wifi one?

Sean

---

Posted at 2017-08-14 by @gfwilliams

Yes - it'll be espruino_1v93_wifi.bin

Just copy the link to the file itself and paste it into the Web IDE

---

Posted at 2017-08-14 by seanclark

I've tried the alternative firmware and - you're right - it makes the neopixels act a little randomly, with flashes and unpredictable colour changes!

I've improved my script quite a bit since my earlier post. I now use a node.js server to batch up my colour changes and only release them to the Espruino via MQTT after a predetermined time. I've currently set this to 250ms to give the Espruino time to update the neopixels.

This mostly works very reliably, however I sometimes get errors in my packets. The basic neopixel packet is 12 hex characters, with multiple packets being grouped together. Here is a run of correct packets:

[0000E1FF0000]
[0000E2FF00000000DCFF00000000DAFF00000000C6FF00000000C8FF00000000B5FF00000000B4FF00000000B2FF00000000ADFF00000000AEFF00000000AFFF0000]
[0000AEFF00000000ADFF00000000ACFF0000000095FF0000]

However, every now and then I get some erroneous data in the packet. It appears to be the MQTT topic, eg. "biosphere/ecology/matrix/color", preceded by text in the form "+IPD,0,70:0". As you can see from the example below, this erroneous data is stuck in the middle of a packet.

[000085FF000000000052FF000000006DFF0000000071FF000000008EFF0000
+IPD,0,70:0D�biosphere/ecology/matrix/color000091FF00000000AEFF00000000ADFF0000
+IPD,0,46:]

Any thoughts about what the problem might be? Like I say, I'm only sending a packet every 250ms now and they are at the very most 100 or so characters.

Sean

---

Posted at 2017-08-15 by @gfwilliams

Can you check E.getErrorFlags() and see if it reports anything? The flags get reset each time you call it so you should be able to see if something's causing a problem.

It sounds to me like data from the Wifi module is getting lost again - IPD is part of the data sent when data is received - usually it should be stripped out but presumably it's got out of sync somehow. Are you using the old firmware when you get that, or the new one?

---

Posted at 2017-08-15 by seanclark

Hi,

I was getting it on both firmwares, here is a log showing bad packets. For some there are no flags, but a few are showing LOW MEMORY:

bad packet [FE3100FFFD3100FFE33100FFE53100FFD931
+IPD,0,90:0X�biosphere/ecology/matrix/color8B3100FF743100]
[  ]
bad packet [ ��biosphere/ecology/matrix/color743100FF753100FF763100FF773100FF783100FF853100]
[  ]
bad packet [F13100FFEE3100FFD13100
+IPD,0,82:0P�biosphere/ecology/matrix/c]
[
  "LOW_MEMORY"
 ]
bad packet [6A3100FF893100FFB73100FFE73100
+IPD,0,42:0(�biosphere/ecology/matrix/color5C3100FF
+IPD,0,114]
[  ]
bad packet [145FFF00155FFF0000]
[  ]
bad packet [745FFF008B5FFF00945FFF00955FFF00AA5FFF00A95FFF00A85FFF00985FFF00865FFF007A5FFF00435FFF003C5FFF00235FFF001C5FFF00035FFF00
+IPD,0,227:0à�]
[
  "LOW_MEMORY"
 ]

Sean

---

Posted at 2017-08-16 by @gfwilliams

LOW_MEMORY implies that it's been forced to garbage collect, which is quite unusual - usually it can do it on idle where it doesn't affect anything. Looking at the code I'm struggling to see how it could lose UART data that way though - unless you're printing a load of stuff to USB at the same time?

Do you think you'd be able to narrow this down to some very small program that I could run on an Espruino WiFi that'd reproduce it? Ideally with no usage of neopixels/etc.

---

Posted at 2017-08-16 by seanclark

Hi Gorden,

When you say "printing to USB" would "console.log" be included? Not that I'm printing much, just error messages.

Is there a maximum length on MQTT topic and message lengths? Again, they don't strike me as too long but I could be over doing it. Is it OK to put stringified JSON in an MQTT message on Espruino? I'm not doing it at the moment, but am thinking of doing so.

Also, I was wondering about using "tinyMQTT". Is there any advantage in using this over the native "mqtt"? I notice in the docs that this does have a "127 byte publishing length limit" though. Is this topic and/or message?

I'll keep exploring. It's not causing me major problems. Currently my Espruino has been up for 21 hours, processed 841 packets and only had 6 bad packets. It would be nice to get to the bottom of it though!

Sean

---

Posted at 2017-08-16 by @gfwilliams

When you say "printing to USB" would "console.log" be included? Not that I'm printing much, just error messages.

Yes, it would - but if you're only printing errors then the error has already happened when it's printing, so that won't be it :)

Is there a maximum length on MQTT topic and message lengths?

I could be wrong, but maybe 126 chars I think? I doubt you'd be hitting it.

Is it OK to put stringified JSON in an MQTT message on Espruino?

Yes, totally - that should be fine.

Also, I was wondering about using "tinyMQTT". Is there any advantage in using this over the native "mqtt"? I notice in the docs that this does have a "127 byte publishing length limit" though. Is this topic and/or message?

I'm not sure about the limit - it could be both. I think tinyMQTT was started because some people wanted something that would fit inside the ESP8266 when the port was very limited, but it could be worth a try. If it does work correctly it'd narrow the problem down a lot!

I'll keep exploring. It's not causing me major problems. Currently my Espruino has been up for 21 hours, processed 841 packets and only had 6 bad packets. It would be nice to get to the bottom of it though!

Ok, that's not so bad - and it recovers ok? I'd assumed it stayed totally broken when you got the error.

The latest travis builds (http://www.espruino.com/binaries/travis/master/) now check specifically for a UART overflow - so if you get a chance that could be interesting I guess, as it'd tell you if data was getting lost at a lower level or not.