Arduino HTTP Request Fails Slow Cellular Connection

Question

Setup:

• Arduino Mega 2560 w/ Wiznet W5500 Ethernet Sheild

• Using TP-Link TL-MR6400 LTE router to connect to internet over cellular connection in Uganda.

---

Using an HTTP Post with the following,

client.println("POST /hook HTTP/1.1");
client.print("Host: "); client.println(server);
client.println("Accept: */*");
client.print("Content-Type: text/html;");
client.println("User-Agent: Arduino/1.0");
client.println("Connection: close");
client.print("Content-Length: ");
client.println(PostData.length());
client.println();
client.println(PostData);

where PostData is a String and Ethernet2 is the library used.

This works just fine from the US to the same AWS server, or a https://requestbin.com/ instance.

---

When testing from Uganda using a Raspberry Pi and curl it works fine.

$ curl -vs -d 'ABCDEFG' -H "User-Agent: Arduino/1.0"  -H "Content-Type: text/html" -H "Connection: close"   http://<server here>/hook
*   Trying <ip here>...
* TCP_NODELAY set
* Connected to <server here> (<ip here>) port 80 (#0)
> POST /tg HTTP/1.1
> Host: <server here>
> Accept: */*
> User-Agent: Arduino/1.0
> Content-Type: text/html
> Connection: close
> Content-Length: 7
> 
* upload completely sent off: 7 out of 7 bytes
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Date: Tue, 06 Aug 2019 21:06:49 GMT
< x-pd-status: sent to primary
< X-Powered-By: Express
< Content-Length: 16
< Connection: Close
< 
* Curl_http_done: called premature == 0
* Closing connection 0
{"success":true}

---

I even tried an HTTP GET example from the Ethernet2 library,

https://www.arduino.cc/en/Tutorial/WebClientRepeating

It worked just fine in the US but in Uganda I get,

My IP address: 192.168.1.177

connecting...

connecting...

connecting...

I increased the time between requests here to 100 seconds where it is listening for the whole time inbetween.

It seems that client.connect(server, 80) is successful and the client.print messages go through but the message never reaches the server so there is no response. My AWS server does not recieve the request and this is the same for the requestbin instance. This behavior is mirrored in all my Arduino tests.

I thought the ISP might be blocking but the RPi can send nearly the same message. I thought maybe a timeout somewhere but I can not find where with my tests. I thought my message was too long but I reduced the size to only 10 characters with the same results.

---

Trying to figure out why the Arduino HTTP request cannot reach any webhooks from Uganda and how to fix it if possible. Different ISP, slower connection, cellular connection?

Any help would be appreciated.

Answer 1

Buffering and flush()

Many arduino libraries suffer of slow communication because the authors didn't use buffering. To not to send every small piece of data over a connection with overhead of protocol bytes use buffering. Data are collected until the buffer is full and then the collected data are send and the buffer is ready for next bytes. The flush() functions sends the buffer even if it is not full, but you do not have more data. (The flush() 'dilemma' was resolved and in Arduino core classes the flush() method is declared in Print class which is the Arduino standard interface for output streams.)

My StreamLib library offers a BufferedPrint class which wraps the target stream and buffers the data to be send over the target stream. Of course, do not forget to use flush() after last data printed.

---

HTTP data length

It is common practice in Arduino sketches to not to send the Content-length header with POST requests. In most cases it is because when the HTTP headers are send, the size of data is not known. The WebServer then doesn't know if it received all the data and waits until the connection is closed or timeout is reached. It leads to long time until a Web Server sends the response to requesting Arduino sketch.

If the data can be prepared to a string in memory, fill the Content-length HTTP header.

If the data to be send as response or in POST request wouldn't fit into available memory, ChunkedPrint from my StreamLib can be used with HTTP chunked Transfer-Encoding. And ChunkedPrint is an extension of BufferedPrint so the Web Server handles your request without hesitating.