2

Folks, I want to use my arduino with ethernet shield to access the following webpage "http://www.met.ie/forecasts/sea-area.asp" and take some information.

On the website it has a line "Small Craft Warning"- This is always either "In Operation" or "Nil".

Basically i want to hook up a RGB LED to the duino and if "Small Craft Warning" is "in Operation" i want LED to be Red, and if "Nil" LED can be green.

My issue is i cant figure out hot to go about getting the information from the webpage.- Can the arduino read this directly?

I am confident with arduino but this is my first web based project!

Any Tips or advice would be savage helpful Thanks

  • I suspect that you may get a better answer on Stack Overflow. – Steve G Aug 27 '16 at 11:19
2

As I commented on deltaray's answer, a regular expression library is overkill for finding a fixed string. There is a standard avr-libc function for that called strstr():

char * strstr(const char *haystack, const char *needle);

where haystack is the Web page and needle is the fixed string you are searching for.

There is an issue with this function though: in order to use it, you need to have the whole haystack in RAM. For a non-trivial Web page, this may not be doable on a small Arduino.

Below is a custom solution based on a finite state machine: the parser can be in a number of different states:

  • state 0 = waiting for the 'S' of "Small Craft"
  • state 1 = waiting for the 'm' of "Small Craft"
  • etc...

The state is actually an index in a character array representing the string we are looking for. Each time the parser finds the expected character, it moves to the next state. Otherwise it moves back to state 0. There is a small kludge though: when in the state waiting for the 'N' of "Nil", if it doesn't get an 'N', it moves to the state waiting for the 'I' of "In Operation" and tries the match again. This way it can look for both the possible strings at once. Also, once it got the full preamble "Small Craft Warning: </b>", if it gets something invalid, it moves to an extra state called INVALID instead of state 0.

Here is the code:

// Possible return values of parse_forecast().
// The last three are sticky.
enum parse_state {
    NO_PREAMBLE,        // no complete preamble seen yet
    PREAMBLE_COMPLETE,  // preamble just completed
    PARTIAL_NIL,        // received preamble and start of "Nil"
    PARTIAL_IN_OP,      // received preamble and start of "In Operation"
    COMPLETE_NIL,       // result = Nil
    COMPLETE_IN_OP,     // result = In Operation
    INVALID             // valid preamble followed by invalid data
};

parse_state parse_forecast(char c)
{
    // position:      0                        25   29            42
    const char T[] = "Small Craft Warning: </b>Nil\0In Operation\0";
    static size_t i;  // T[i] is the character we expect next

    if (T[i]) {  // non-sticky state
        if (i == 25 && c != T[i])
            i = 29;      // switch to "In Op" branch
        if (c == T[i])   // found expected char
            i++;         // wait for the next one
        else {
            if (i < 25)
                i = 0;   // reset to start of the string
            else
                i = 42;  // invalid state
        }
    }
    return i <  25 ? NO_PREAMBLE :
           i == 25 ? PREAMBLE_COMPLETE :
           i <  28 ? PARTIAL_NIL :
           i == 28 ? COMPLETE_NIL :
           i <  41 ? PARTIAL_IN_OP :
           i == 41 ? COMPLETE_IN_OP :
           INVALID;
}

With this you do not need to store the Web page in RAM. Instead, you feed the parser one character at a time and, at the end, check the return value. You expect it to be either COMPLETE_NIL or COMPLETE_IN_OP.

Implementation detail: there are three NUL characters in the template string, at positions 28, 41 and 42 (the last one is implicit). They correspond to the three possible "sticky" states.

| improve this answer | |
0

Unfortunately that text is delivered as part of the complete web page and not, for example, an AJAX request.

Do you supply an RSS feed of your forecasts?

This is not something we provide at present, although we hope to do so in the future. Source.

Your only option is to scan through the whole page looking for the text strings.

This is a little dangerous as the page could, for example, contain the phrase as part of a help file and give you an incorrect status reading. I wouldn't set to sea on the basis of your LED.

Bí cúramach!

| improve this answer | |
0

To add to Transitor's answer, while not guaranteed to work perfectly, you can always try to narrow down the page's source by only searching the component the text is in which should hopefully work. (unless they redo their website)

| improve this answer | |
  • Thanks...I have no clue how to code that. And cant find examples. Any hints? – Omasín Aug 28 '16 at 15:11
  • Simply use if (String.substring(length of element data here) == "element name here") {} to find it on the webpage and then execute whatever code you need inside – dalearn Aug 28 '16 at 20:34
  • what is lenght of element data representing? – Omasín Aug 28 '16 at 20:50
  • Go into the source code of the website you wish to get (you know how to in chrome right?) and copy the html element which contains said text into the "element name here" part of the code and put the length (in chars) in place of the "length of element data here". – dalearn Aug 28 '16 at 22:44
0

I think this can be accomplished with the use of a few libraries. Also, I'd recommend the use of a third color to let you know when your program has encountered an unexpected condition, such as couldn't retrieve the webpage or couldn't find "Small Craft Warning" in the page or if the value for that attribute is something unexpected.

So first you'd want to use the Ethernet library that comes with the Arduino IDE. There is even an example sketch that shows you how to use it to download a webpage. Make sure you parse the status returned from the webserver to ensure its giving you a 200 HTTP status code and produce an error if it does not or if the connection fails.

As for parsing the webpage itself, you might try this regular expression library that Nick Gammon wrote. Just have it search for the line with Small Craft Warning and then parse that line as a string as test for matches against the strings you are expecting to find or erroring out. Or you can use additional regular expressions to match your expected values once you find the right line. Remember to make the LED show an error color if it doesn't find the line or an expected value.

Just talking about this makes me want to try it. ;-)

| improve this answer | |
  • Using a regular expression parser to find a fixed string is kind of overkill. – Edgar Bonet Aug 28 '16 at 17:04
  • Just trying to make it easier for them. – deltaray Aug 28 '16 at 17:25
  • Thanks. This is awsome help. Just read the readme and its brilliant but im just not getting it. I dont know where to start – Omasín Aug 28 '16 at 18:03
  • If you're trying to go from 0 to parsing a webpage with an arduino, you're pressing too hard on the gas. You need to work your way through some other examples and tutorials first, then it will all make more sense. And you'll have a lot of fun along the way. – deltaray Aug 28 '16 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.