How To Pull Data Counter From A Website To Use In Another Html Project As A Js Variable

Hi I am trying to find a way to be able to pull a variable that is displayed on a website that is not my own onto one that is my own so I can use it on mine as a JavaScript variabl

Solution 1:

Pasting previous comment as an answer to have more space:

A possible way would be to do some web page scraping.

Every X amount of time you grab a copy of the page you are interested in, and then you can just scan the page source for the value you want, with a regular expression for example. Then you can return that value after scanning and put that into a variable.

It's not the most efficient route (ideally they would provide an API, but I think this would be overkill for their use case) but it can work.

For example, a quick Google search for "web page scraper" gives:

• https://import.io/
• http://webscraper.io/
• http://scrapy.org/

You can either use one of those (or similar, I really haven't used those particular ones) or you can build your own, but the concept is the same:

Get the web page source code, discard anything that you don't need or alternatively extract only what you want and that's it.

Solution 2:

In your particular case, you could use it, but you don't need web scraping. As mentioned in the comments by JasonK You can use the same API call that the page is using:

https://www.jmu.edu/cgi-bin/parking_get_sign_data.cgi?date=1441292695108

Now, you cannot use that API from your website because of same-origin policy, but you can create a small service to get your data from. In node.js it could look like this, but you can easily implement the same function in php:

var request = require("request");
var http    = require('http');

var server  = http.createServer(onRequest);

server.listen(3000);


//----------------------------------------------------functiononRequest(req, res){

    var parkingUrl = 'https://www.jmu.edu/cgi-bin/parking_get_sign_data.cgi?date=' + (newDate()).getTime();

    request(parkingUrl, function (error, response, body) {

        var data   = error;
        var status = 404;

        if(!error){
            status = 200;
            data = {
                championStatus : getStatus(body, '2'), 
                warsawStatus   : getStatus(body, '10')
            };
        }

        res.writeHead(status, { 'Content-Type': 'application/json', "Access-Control-Allow-Origin":"*" });
        res.write(JSON.stringify(data));
        res.end();
    });
}


//----------------------------------------------------functiongetStatus(ss, si){
    var status = ss;

    status = status.split("<SignId>"+si+"</SignId>"); 
    status = status[1];
    status = status.split("<Display>"); 
    status = status[1];
    status = status.split("</Display>"); 
    status = status[0];
    status = status.replace(' ','');
    if(isNaN(status)){
        // do nothing 
    } else {
        status = parseInt(status);
    }

    if( status == 'Errors'){status = '';}
    elseif(status != 'FULL' && isNaN(status)){status = 'Unavailable';}
    elseif(status != '' && status != 'FULL'  && status != 'OPEN'){
        if(status == '   1'){status = status + ' space available'; }
        else{status = status + ' spaces available'; }
    } 
    return status;
}

The getStatus function is taken straight from the https://www.jmu.edu/parking/ website, i'd rather use xml2js or a similar module to parse the response and the data.

From your website you can now get the status like this:

functionhttpGetAsync(url, callback)
{
    var xmlHttp = newXMLHttpRequest();
    xmlHttp.onreadystatechange = function() { 
        if (xmlHttp.readyState == 4 && xmlHttp.status == 200){
            callback(xmlHttp.responseText);
        }  
    }
    xmlHttp.open("GET", url, true); // true for asynchronous
    xmlHttp.send(null);
}

httpGetAsync("http://localhost:3000/", function(res){
    var data = JSON.parse(res);
    console.log(data);
});

Don't forget to change localhost:3000 to your server address, adjust the Access-Control-Allow-Origin header to limit who can use your service and add some error handling.

Solution 3:

Unless you have a way to communicate with the school server and get that data, you're probably stuck with scraping. If you look at the code of the school website, you see that the amount of free spaces is generated by calling a cgi script and parsing it.

If you have access to this cgi script, you can just use that script to get your values, and parse it as described in the function getStatus from the source code of the school site.

If you don't have access to the cgi, you can try doing an ajax call to this website and check if the node containing the numbers is available for you to select from the DOM.

If you can't access the DOM of the website and/or if accessing it is too slow, load the site with ajax, but instead of text/html, ask for text/plain so you just get a long string containing the website. Then you can scrape this string with a regular expression to get your value.

If all of this fails, load the site into a hidden iframe, to ensure that the script that inserts the parking lot numbers, is run. Then continue as normal, by selecting the correct node from out of this iframe.

These are all options for clientside. There are probably more options serverside, (like easier interaction with the schools cgi) but the general principles remain. Either use their own API (the cgi script), use the website itsself to scrape, or use a text representation of the fully loaded website to regex.