Filesystem load to RAM vs piping

[espruino-discuss]

Posted at 2017-05-05 by dave_irvine

Trying to send a 16KB file from the Espruino to a HTTP client. I've hooked up an SDcard reader to the Espruino and configured it, can successfully read and write data to/from it.

I've tried both the methods mentioned here: http://www.espruino.com/Internet to send data to a HTTP client.

The first, reading the file in its entirety to RAM and then sending to the client, is relatively fast , 4.52 seconds to send 16.1KB.

The second, piping the file directly from the SD card is massively slower. I could understand it being a second or two slower, but 16.1KB takes 21.24 seconds.

I don't believe the SD card interface is the bottleneck, because reading the file to RAM first happens on a page request, so if the SD card took 16.72 seconds to read the file and the remaining 4.52 seconds was just sending the data over http, then the first request would take the same amount of time as the piping request.

Have I found another bug, am I doing something wrong, or am I misunderstanding the limitations of the platform?

[espruino-discuss]

Posted at 2017-05-05 by @gfwilliams

Please can you try with ...pipe(res, {chunkSize:512});

The issue will probably be due to the much smaller default chunk size (64 bytes IIRC) when piping data. While it shouldn't be that much slower, it can mean that the WiFi connection chooses to send many small packets rather than just a few large packets - and it can also be less efficient loading smaller amounts of data from the filesystem as well.

---

Posted at 2017-05-05 by dave_irvine

Down to 5.65 seconds, love it :)

Something else for the docs!

---

Posted at 2017-05-05 by dave_irvine

Although 16KB is nothing, I need to stream about 2MB...

---

Posted at 2017-05-08 by @gfwilliams

You could try some bigger chunk sizes and see if it helps, but I'm afraid it's never going to be super-fast. By default the WiFi is connected at 115200 baud, so realistically even with zero overhead you're still looking at only around 12k/second.

By increasing the WiFi baud rate (you need to send a command to the ESP8266 to set it up) and also increasing the SD card bit rate you could improve things, but I think it may still not be as fast as you really want for a 2MB file.

---

Posted at 2017-05-08 by dave_irvine

So while I could set a new baud rate, I can't see a way to then tell the Espruino to communicate with the wifi module at that rate, by the look of http://www.espruino.com/modules/EspruinoWiFi.js its fixed to 115200.

Is there any chance you could alter the module so I can pass configuration information for the Wifi module if I want to?

---

Posted at 2017-05-08 by dave_irvine

For anyone as silly as me who decides to change the baud rate without waiting for a configurable wifi communication module, here is how to rescue yourself.

AT+CIOBAUD seems to carry across reboots, which everyone says it doesn't, but it does!

digitalWrite(A14, 0);
Serial2.setup(9600, { rx: A3, tx : A2 });
at = require('AT').connect(Serial2);
at.cmd("\r\nAT+RST\r\n", 10000, function cb(d) {
  at.cmd(“AT+CIOBAUD=115200\r\n”, 1000, function () {
    at.debug();
    at.cmd(“AT+RST\r\n”, 1000);
  });
});
digitalWrite(A13, 1);
digitalWrite(A14, 1);

---

Posted at 2017-05-08 by @gfwilliams

Argh - sorry - I hadn't realised you were on Espruino WiFi. You can do Serial2.setup after WiFi is initialised I think, but yes, coping with the changed baud rate afterwards is a pain. I thought there was a non-persistent way to change it, but I'm afraid I can't find it now.

Thanks for posting up your code for the fix though - that's great!

---

Posted at 2017-05-08 by dave_irvine

AT+UART_CUR should do the job but I don't think it is supported on the firmware that comes on the Espruino WiFi.

---

Posted at 2017-05-08 by dave_irvine

Oh poop now I killed it by setting an unsupported baud rate.

Gordon could you convert the instructions from http://www.espruino.com/ESP8266 for Flashing the ESP8266 so that they work for the Espruino WiFi? Looks like I need to reflash to get this working again.

---

Posted at 2017-05-08 by dave_irvine

I had hoped I'd managed to convert the instructions myself, but no luck.

Here's what I had

digitalWrite(A14,1);
digitalWrite(A13,0);
digitalWrite(A15,0);
digitalWrite(A15,1);

Serial2.setup(115200, { rx: A3, tx : A2 });
Serial2.on('data', function(d) { USB.write(d); });
USB.on('data', function(d) { Serial2.write(d); });
Serial1.setConsole();

but when I run esptool.py I just get:

esptool.py v1.3
Connecting....
Auto-detected Flash size: 32m
Running Cesanta flasher stub...

A fatal error occurred: Invalid head of packet (b'\x13')

---

Posted at 2017-05-08 by @gfwilliams

Looks promising - but maybe try:

digitalWrite(A14,0); // power off
digitalWrite(A13,0); // boot mode
digitalWrite(A14,1); // power on

Serial2.setup(115200, { rx: A3, tx : A2 });
Serial2.on('data', function(d) { USB.write(d); });
USB.on('data', function(d) { Serial2.write(d); });
Serial1.setConsole();

I think you set the boot mode after powering the device up?

Are you sure you're disconnected with the Web IDE?

---

Posted at 2017-05-08 by dave_irvine

Still no joy. Should I expect a specific output before I disconnect the IDE? At the moment I get different responses depending on the baud but nothing readable.

USB.on('data', function(d) { Serial2.write(d); });
=undefined
Serial1.setConsole();�þ
-> Serial1
Disconnected

or

=undefined
-> Serial1
ÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀ
Disconnected

By doing some mad and barely reproducible set of steps, including a baud rate of 4608000(!), I can occasionally get the flasher to connect

esptool.py v1.3
Connecting....
Running Cesanta flasher stub...
Flash params set to 0x0200
Writing 520192 @ 0x0... 120832 (23 %)
A fatal error occurred: Unexpected packet while writing:

but never to complete a write. I get different percentages of success but 23% is the highest so far.

I've ordered a new board because I'm pretty sure I've fried this one :(

---

Posted at 2017-05-08 by @gfwilliams

Well, if you get to 23% that's a really good sign. I'm sure we'll be able to get this one going.

Did you change the buffer sizes in the flasher program? That can make it a lot more stable.

---

Posted at 2017-05-08 by dave_irvine

Yeah I tried setting

ESP_RAM_BLOCK and ESP_FLASH_BLOCK all the way down to 0x1 as well as ESP_ROM_BAUD (although I'm fairly sure this is overridden by the -b flag) and ESP_FLASH_SECTOR.

No luck with any combo I could come up with.

Regards to your code sample, don't I need to pulse on A15?

---

Posted at 2017-05-08 by @gfwilliams

Don't I need to pulse on A15?

No, Reset isn't connected on the Espruino WiFi, since you can do the same just by powering the device off with CH_PD.

Could you try with a block size of 128 and a baud rate of 57600? It'll be really slow, but the slower it is the more reliable it should be.

Your other option if that fails is to do:

digitalWrite(A14,0); // power off
digitalWrite(A13,0); // boot mode
digitalWrite(A14,1); // power on
digitalRead(A2);
digitalRead(A3);

and to then connect a USB-TTL adaptor directly to the ESP8266. It'll be a bit fiddly to solder, but if you use the high baud rates used for normal programming you could probably connect GND to the Espruino WiFi's pin and then just hold RX and TX on by hand.

---

Posted at 2017-05-08 by dave_irvine

Gordon is that a baud rate in esptool's settings, or in the code I'm running on the Espruino to become the serial passthrough? Or do they both need to match?

---

Posted at 2017-05-09 by @gfwilliams

It's the Espruino WiFi one. The baud rate in esptool shouldn't matter since it's over USB, but it'd be better to make the two match just in case.

---

Posted at 2017-05-09 by dave_irvine

No luck there either. A different language of serial gobbledegook, but the flasher still complains :(

Serial2.setup(57600, { rx: A3, tx : A2 });
=undefined
Serial2.on('data', function(d) { USB.write(d); });
=undefined
=undefined
Serial1.setConsole();F�EÚ��Q�ALiÈVPØ�¡ÑÜ»êÍ��H×¥¡×¥ëL�PËÔTGð
-> Serial1
Disconnected

---

Posted at 2017-05-09 by @gfwilliams

Very odd - not sure what to suggest with trying to upgrade via the WiFi. Do you have a USB-TTL converter handy that you could try the upgrade with?

---

Posted at 2017-05-09 by dave_irvine

I've got one https://www.amazon.co.uk/gp/product/B01CYBHM26 on the way, should arrive today.

Can you confirm which pins of the usb-ttl I'm hooking to which pins on the ESP8266 module on the WiFi?

I'm guessing I can use the + and - pins on the WiFi itself and hook those to the 5v and GND on the usb-ttl. Not sure where to hook RXD and TXD.

Assuming https://github.com/espruino/EspruinoBoard/blob/master/WiFi/pdf/espruino_wifi_sch.pdf is oriented in the real world, I've guessed as per my attachment.

Should I hook usb-ttl RXD to ESP8266 RXD and TXD to TXD or do I want to crossover?

Also, while I think of it, do you have a copy of the AT 0.40 firmware? I was going to flash with 0.25 as it was the newest I could find from your downloads (https://www.espruino.com/files/ESP8266_AT25-SDK112-512k.bin), but I'd rather flash 0.40, I just can't find it.

---

Attachments:

• IMG_20170509_091058183.jpg

---

Posted at 2017-05-09 by @gfwilliams

Yes, looks good.

TXD is out from the ESP8266 towards the PC, and RXD is into the ESP8266 - so depending on your USB-TTL dongle you may need to swap them. I have dongles labelled both ways here to if it doesn't work it's just best to swap the wires - but it won't damage anything if you have them around the wrong way for a minute or two.

I wouldn't connect + at all. Just connect GND, RX and TX, and leave the WiFi board connected by USB so you can set up the pins correctly.

With the firmware, I don't actually have 0.40 here - I'm afraid you'd just have to google and find it.

This might be the most helpful: http://www.electrodragon.com/w/ESP8266_AT-Command_firmware

As you're all set up, I'd try uploading the newest 'espressif firmware' from that page. It should work fine, and I know it does have some new features (the ability to advertise itself via mDNS is cool).

However it looks like the 0.40.0.0 firmware is available at the bottom of http://domoticx.com/esp8266-wifi-at-firmware/

---

Posted at 2017-05-09 by dave_irvine

The 0.40 firmware there looks to have some questionable strings in it, I've managed to find 0.60 from espressif official, will let you know how the flashing goes.

*** Modified by ***
HuangRui
vowstar@gmail.com
JiYaping
jyp1817@gmail.com

---

GET /ota/device/at/rom/v1/?is_format_simple=true HTTP/1.0
Host: service.vowstar.net

---

Posted at 2017-05-09 by dave_irvine

No, I lied, 0.51+ only works on devices with 8Mbit flash and I think the esp8266 we have is 4Mbit. So I have 0.50 official http://bbs.espressif.com/viewtopic.php?f=46&t=1123

I have found some documentation that suggests even the newest firmwares will actually fit on 4Mbit, will give it a try, not sure who to believe! https://syonyk.blogspot.co.uk/2016/11/installing-latest-at-firmware-on-4mbit.html

---

Posted at 2017-05-09 by dave_irvine

AT+GMR
AT version:1.3.0.0(Jul 14 2016 18:54:01)
SDK version:2.0.0(656edbf)
compile time:Jul 19 2016 18:43:55
OK

Wohoo!

---

Posted at 2017-05-09 by dave_irvine

Looks like the Espruino_WiFi module doesn't like the 1.3.0 AT command set, so I've stuck with 0.50.0 for now.

esptool command to get that going:

esptool.py -p /dev/tty.SLAB_USBtoUART -b 115200 write_flash 0x00000 noboot/eagle.flash.bin 0x40000 noboot/eagle.irom0text.bin 0x7c000 esp_init_data_default.bin 0x3e000 blank.bin 0x7e000 blank.bin

AT+UART_CUR now works, and properly resets to default params when the module reboots. Maximum baud I can get is 1843200, so the command is:

AT+UART_CUR=1843200,8,1,0,0

---

Posted at 2017-05-09 by @gfwilliams

Nice - thanks! Glad you got it sorted!

And you're using 0.50 from this link? http://bbs.espressif.com/viewtopic.php?f=46&t=1123

---

Posted at 2017-05-09 by dave_irvine

Yes, however stability has gone completely downhill, getting frozen http connections and all sorts. Going to try my fresh Espruino WiFi to see if its a firmware issue.

---

Posted at 2017-05-09 by dave_irvine

Hmm, interesting.

Sending an index.html file that contains a link to a stylesheet with pipe() without a chunk size, works fine. If the stylesheet is sent using pipe() with a chunk size, all ok.

However, if you send the index.html with a chunk size, the http request for the stylesheet never arrives at the Espruino even though the browser sends it. I think what is happening is that the connection is held open during the pipe, and when another request arrives, the ESP is too busy sending to deal with the arrival.

Also +Gordon, while you're looking, can you give me any more info on https://www.espruino.com/Reference#l_SPI_setup with regards to the "baud" rate? What is the unit there? I see a default of baud:integer=100000. Can you give some wild guesses what I can increase that to in order to speed up the SD card reads?

---

Posted at 2017-05-09 by @gfwilliams

And that's all happening on a fresh Espruino WiFi with 0.40 firmware, at the default baud rates? But it's serving off the SD card?

Can you check E.getErrorFlags() and see if it reports something? Could be Espruino's internal buffer has got full.

I guess another option is that the browser is attempting to send the request for the CSS on the same connection, and that Espruino is closing the connection right after finishing the HTML? It shouldn't do that though as Espruino reports that it only supports HTTP 1.0.

Can you come up with a really basic no-frills example that exhibits the problem? Even better if you can do it without using an SD card.

SPI "baud" rate? What is the unit there?

It's bits per second. You could give 10000000 a try, but it may depend a little on your wiring. Try reading and listing files first - things that don't write to the SD card :)

---

Posted at 2017-05-09 by dave_irvine

I'll try and work up a minimum breaking example tomorrow.

Can you think of any more tips for getting the maximum rates out of the wifi? At the moment I've pushed everything about as far as I think it'll go and I'm getting 16KB in 2.5 seconds, so about 6.4kBps. I must admit I was hoping for something a bit more in the mBps range (although perhaps thats a little over optimistic)... what is likely to be the bottleneck here, the ESP8266 or the Espruino itself?

I've found a thread or two to do with ESP8266 speed, esp8266/Arduino#1853, http://www.esp8266.com/viewtopic.php?f=5&t=245 Espressif quote a slightly higher rate http://bbs.espressif.com/viewtopic.php?t=2187

If I watch the AT debug data, I see 64byte packets being sent more often than not. Is this something I can control?

["AT+CIPSEND=1,131\r\n"
] "\r\nOK\r\n> " <--- "\r\nOK\r\n> "
] "\r\nRecv 131 bytes\r\n" <--- "\r\nRecv 131 bytes\r\n"
] "\r\nSEND OK\r\n" <--- "\r\nSEND OK\r\n"
["AT+CIPSEND=1,64\r\n"
] "\r\nOK\r\n> " <--- "\r\nOK\r\n> "
] "\r\nRecv 64 bytes\r\n\r\nSEND OK\r\n" <--- "\r\nRecv 64 bytes\r\n\r\nSEND OK\r\n"
["AT+CIPSEND=1,64\r\n"
] "\r\nOK\r\n> " <--- "\r\nOK\r\n> "
] "\r\nRecv 64 bytes\r\n" <--- "\r\nRecv 64 bytes\r\n"
] "\r\nSEND OK\r\n" <--- "\r\nSEND OK\r\n"
["AT+CIPSEND=1,64\r\n"
] "\r\nOK\r\n> " <--- "\r\nOK\r\n> "

According to the docs for AT+CIPSEND I think it will take up to 2048, so 64 is a bit tiny! http://www.ctr-electronics.com/downloads/pdf/4A-ESP8266__AT_Instruction_Set__EN_v0.40.pdf

---

Posted at 2017-05-10 by @gfwilliams

I must admit I was hoping for something a bit more in the mBps range

Yes, you're not going to get close to that I'm afraid! If you want that kind of thing IMO you really need to be looking at much more heavyweight, linux-based systems.

I believe that page could be quoting bits per second, so at 300kbits you're still only transmitting 30k bytes per second. Given Espruino's having to read from the SD card as well, 8kbytes per second isn't that unreasonable.

what is likely to be the bottleneck here?

The serial comms to the ESP8266 won't be helping at all, but the throughput from the SD card really isn't amazing either. Unfortunately you can't wind the serial baud rate right up because then it's possible that Espruino can miss characters that come in, and it gets unstable.

If I watch the AT debug data, I see 64byte packets being sent

That is strange - did you remove the chunkSize from Pipe again? It should send around 530 byte chunks each time if left to its own accord, so with the chunkSize set to 512 the FAT sector size and the amount of data sent at once should match up pretty well.

It's worth noting that with at.debug on you might well notice some instability/slowdowns - since Espruino is having to relay everything that happens on the serial link over USB, buffers could get full and it might miss something.

---

Posted at 2017-05-10 by dave_irvine

Yes at.debug definitely causes some issues, its only switched on when I'm trying to debug something :D

chunkSize was definitely still set. I'm trying to figure out a way to sniff the traffic without at.debug. I guess wireshark is my best bet but I'm running short on time, need to get this thing working.

Going to try serving gzipped files and see if the size reduction will make this viable. I definitely don't want to switch away from the Espruino... maybe one day you will release an Espruino-Mega ;)

---

Posted at 2017-05-10 by @gfwilliams

I guess you could modify the EspruinoWiFi library send and recv handlers to dump simple information like sent X bytes, recv X bytes - that might help without spamming too much information?