Arduino ESP8266 HTTPS Request returning empty response

Question

I want to get some JSON data from a HTTPS secured Webserver using a ESP8266. For some reason I'm getting an empty response, I believe there's an error parsing the GET request. In Chrome I get the JSON response without problems.

GET Request:

httpsClient.print(String("GET ") + path + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +               
               "Connection: close\r\n\r\n");

Host and Path:

const char *host = "www.adler-mannheim.de";
const char *path = "/jsonapi/game/current";

UPDATE:

I tried fetching some JSON from another Server and it worked finde, so i can rule out my code. Fiddling around with the host and path I observe this:

• Hostname without "www" results in a 301

• Hostname with "https://www" results in no HTTPS connection at all

So it seems that a "www" is required although it unfortunately leads into a empty response.

The output is as follows:

headers received
reply was:
==========
==========
closing connection

In the first case, without using "www" in ahead of host there's some 301 HTML in between the two lines of "=".

Whole function:

void connect() {
  WiFiClientSecure httpsClient;    //Declare object of class WiFiClient

  Serial.println(host);
  httpsClient.setInsecure();
  
  Serial.print("HTTPS Connecting");
  int r=0; //retry counter
  while((!httpsClient.connect(host, httpsPort)) && (r < 30)){
      delay(100);
      Serial.print(".");
      r++;
  }
  if(r==30) {
    Serial.println("Connection failed");
  }
  else {
    Serial.println("Connected to web");
  }
  
  String getData, Link;

  //GET Data

  Serial.print("requesting URL: ");
  Serial.println(host+Link);

  httpsClient.print(String("GET ") + path + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +               
               "Connection: close\r\n\r\n");

  Serial.println("request sent: ");
  Serial.println(String("GET ") + path + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +               
               "Connection: close\r\n\r\n");
  while (httpsClient.connected()) {
    String line = httpsClient.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }

  Serial.println("reply was:");
  Serial.println("==========");
  String line;
  while(httpsClient.available()){        
    line = httpsClient.readStringUntil('\n');  //Read Line by Line
    Serial.println(line); //Print response
  }
  Serial.println("==========");
  Serial.println("closing connection");
    
  delay(2000);  //GET Data at every 2 seconds
}

UPDATE 2:

I tried the following code from the ESP8266HTTPClient Library examples:

void connect2() {

  WiFiClientSecure client;

  //client.setFingerprint(fingerprint);
  client.setInsecure();
    
  HTTPClient https;

  Serial.print("[HTTPS] begin...\n");
  if (https.begin(client, "https://github.com/esp8266/Arduino/issues/3417")) {  // HTTPS

    Serial.print("[HTTPS] GET...\n");
    // start connection and send HTTP header
    int httpCode = https.GET();

    // httpCode will be negative on error
    if (httpCode > 0) {
      // HTTP header has been send and Server response header has been handled
      Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

      // file found at server
      if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
        String payload = https.getString();
        Serial.println(payload);
      }
    } else {
      Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
    }

    https.end();
  } else {
    Serial.printf("[HTTPS] Unable to connect\n");
  }

  Serial.println("Wait 10s before next round...");
  delay(10000);
}

This one gives me a 200 but no payload. Other sites work

UPDATE 3:

I had a look on the headers.

https.headers()

This gives me 0 headers on every page, even when there's a payload.

I also had a closer look at the payload that returns from my desired api and its filled with NULL. In a weird way this is only the case with my desired page and only my desired subpages. Requesting something more at top level e.g "https://www.adler-mannheim.de/jsonapi" which is basically a SilverStripe Welcome page, I get a fully fetched beautiful HTML payload.

I got literally no Idea why. Do I have to set a specific User Agent or something so that the Server responds?

Answer 1

Okay I have no Idea what I changed apart from deleting some imports which might have collided, but now it works. Regular call with "https://www" does the thing.

Thanks anyways

Answer 2

I had this same problem and I realized it was because I was using an WiFiClientSecure to try to an endpoint over HTTP instead of HTTPS. If anyone else is having this same problem, I would check that.