0

I've based a sketch off the vanilla WiFi101 server examples.

If I define a const char array like this:

const char HTML_index[] = "<html><head></head><body>test</body></html>";

The server works well. However if I increase the char[] size to about 1,300 bytes the server fails to send a response. The device (Feather M0 WiFi) doesn't lock up, I can continue to make HTTP requests and the serial console functions correctly, just no HTTP response. The device and laptop (client) are both on the same WiFi AP so nothing weird there.

First I thought this was a memory issues, but as I'm using newer architecture and IDE using const should ensure the string is in FLASH.

So I then read some threads about buffer limits per response using other libraries: https://github.com/esp8266/Arduino/issues/3205

But again not sure how applicable this is to an ARM based system/ WiFi101 library.

Is there a limiting buffer size etc when sending HTTP data in my setup? Do I need to split the string up into smaller pieces and send multiple responses? How could I go about debugging this situation?

Edit: This is the example sketch my code is based off: https://www.arduino.cc/en/Tutorial/Wifi101SimpleWebServerWiFi

Edit2: ok I think I know what's going on but can't test right now, will post an answer when I do. I found this thread https://github.com/arduino-libraries/WiFi101/issues/163 which talks about SOCKET_BUFFER_UDP_SIZE. So I looked through the WiFi101 src and found this:

#ifdef LIMITED_RAM_DEVICE
#define SOCKET_BUFFER_SIZE 64
#else
#define SOCKET_BUFFER_SIZE 1472
#endif

https://github.com/arduino-libraries/WiFi101/blob/master/src/utility/WiFiSocket.cpp

It's likely that my string was actually over 1472 bytes so this makes sense.

Edit3: Found more references:

https://github.com/arduino-libraries/ArduinoHttpClient/issues/14

https://github.com/arduino-libraries/WiFi101/pull/116

So I take it the 1500 in WINC1500 could mean the buffer size. Either way looks like there's a hardware buffer limit and responses larger than ~1400 bytes need to be chunked.

Improve this question
4
  • the examples don't use const char HTML_index[]. show the sketch you use. do you set content-legth header?
    – Juraj ♦
    Jan 30, 2019 at 7:59
  • I'll add a stripped down example sketch soon. But it follows this example exactly, except I define a const char HTML_index[] = "..."; in global scope and call client.print(HTML_index); in the client connected loop. arduino.cc/en/Tutorial/Wifi101SimpleWebServerWiFi
    – Geordie
    Jan 30, 2019 at 19:34
  • try to add a delay before client.stop()
    – Juraj ♦
    Jan 30, 2019 at 20:11
  • I'll give that a try, but since my HTTP header doesn't specify Connection: close execution stays in the while (client.connected()) loop and doesn't hit client.stop(). But maybe that's a problem
    – Geordie
    Jan 30, 2019 at 21:21

1 Answer 1

Reset to default
1

As per my notes the WINC1500 has a limit of 1500 bytes per transmission. But some of this is always taken up by TCP overhead so the payload buffer must be smaller, that's why you see numbers like 1460.

From: http://ww1.microchip.com/downloads/en/appnotes/atmel-42739-tcp-client-and-server-operation-using-atwinc1500_at14596_applicationnote.pdf

2.2.2 TCP Packet Size

A packet is sized as per the lowest MTU (Maximum Transmission Unit) on the path, which is typically 1500 bytes (for Ethernet) of data including TCP (20 bytes) and IP headers (20 bytes). The packet size is defined in main.h as :

#define MAIN_WIFI_M2M_BUFFER_SIZE 1460

The application then defines a socket buffer of this size in main.c as: 

static uint8_t gau8SocketTestBuffer[MAIN_WIFI_M2M_BUFFER_SIZE];

This means that HTTP responses over ~1400 bytes need to be chunked which means split up into multiple calls.

FYI chunked Transfer-Encoding is superseded in HTTP/2: http://undertow.io/blog/2015/04/27/An-in-depth-overview-of-HTTP2.html

Info on HTTP chunked encoding: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Transfer-Encoding: chunked

Data is sent in a series of chunks. The Content-Length header is omitted in this case and at the beginning of each chunk you need to add the length of the current chunk in hexadecimal format, followed by '\r\n' and then the chunk itself, followed by another '\r\n'. The terminating chunk is a regular chunk, with the exception that its length is zero. It is followed by the trailer, which consists of a (possibly empty) sequence of entity header fields.

https://www.httpwatch.com/httpgallery/chunked/

https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88

Transfer-Encoding

The use of Transfer-Encoding: chunked is what allows streaming within a single request or response. This means that the data is transmitted in a chunked manner, and does not impact the representation of the content.

Officially an HTTP client is meant to send a request with a TE header field that specifies what kinds of transfer encodings the client is willing to accept. This is not always sent, however most servers assume that clients can process chunked encodings.

The chunked transfer encoding makes better use of persistent TCP connections, which HTTP 1.1 assumes to be true by default.

Chunked data is represented in this manner:

4\r\n
Wiki\r\n
5\r\n
pedia\r\n
e\r\n
 in\r\n\r\nchunks.\r\n
0\r\n
\r\n

Each chunk starts with its byte length expressed as a hexadecimal number followed by optional parameters (chunk extension) and a terminating CRLF sequence, followed by the chunk data. The final chunk is terminated by a CRLF sequence.

Chunk extensions can be used to indicate a message digest or an estimated progress. They are just custom metadata that your layer 7 receiver needs to parse. There's no standardised format for it. Because of this, it's probably better to just add your metadata (if any) into the chunk itself for your layer 7.5 application to parse.

For your application to send out chunked data, you must first send out the Transfer-Encoding header, and then you must flush content in chunks according to the chunk format. If you don't have an appropriate HTTP server that handles this, then you need to implement the syntax generator yourself. Sometimes you can use a library to provide an abstract interface.

Possible fix on Arduino platforms is to use StreamLib library or roll-your-own chunking system.

Update: I hacked together a sketch and confirmed this works. Next issue was the buffer must be null terminated, BUT this is stripped out before transmitted over TCP so the HTTP chunk size is the buffer size -1.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.