Introduction

Xill IDE has a pretty extensive web browsing kit included in the Web package. You can load pages with it and interact with buttons and other page elements, like form fields. Xillio uses these functions mostly for website crawling and data extraction, but you could also develop things like test tooling with it.

Web browsing is closely related to handling html (and xml). Since that subject has its own tutorial, this one will focus on types of web navigation that you normally need a mouse cursor for.

Loading pages

The function where web navigation starts, is Web.loadPage([url]). Of course you also need to include the Web package first. Loading a page goes like this:

use System;
use Web;

// loading Xill support home page
var support_page = Web.loadPage("http://support.xillio.com");
System.print(support_page );

Try this out. Copy the code to a new robot to be able to see what happens. If you're not familiar with stepping through Xill code yet, first read about the debugging options. The next examples will add to this first code snippet.

Using a form

As an example, we could start a search from this robot, on this website here. The search bar on top is a so-called form, which means you can enter text in it with the Web.input(element,text) function. First, you need to select the form node using an XPath. The easiest way to decide what XPath you need, is probably by using the developer tools in your favorite browser. There's usually a button called 'inspect element' in the context menu that pops up when you right-click the element you need, which takes you to (or near) the html node in the source of the page. Then you can input the text 'Web browsing'. Add this code to your robot and walk through it:

// entering query into search bar
var search_input = Web.xPath(support_page,"//form[@id='hc-search-form']//input[@name='term']");
System.print(search_input);
Web.input(search_input,"Web browsing");

Dynamic content

By typing text into the search input, the website changes because of the dynamic content. By typing text into the search field, a suggestions list is popping up. To view the suggestions that were given when the search input was typed in, we need to do something smart. Again, by using XPath, we could read the suggestions from inside the auto-complete suggestion list, that was not visible earlier. This suggestion list needs a little time to load, so we need to give it a little time. Let's give the list a second to load. But how do we know that the suggestion list is visible now? Well, the Xill code gives us the opportunity to make screenshot of the current Web browser status. So let's do this, to actually see the suggestions:   

// Give the browser some time to load the suggestion box
System.wait(1000);

// Saving a screenshot to prove that the suggestion box is showing
Web.screenshot(support_page, "C:/tmp/screenshot-suggestions.jpg");

Open up the saved image at 'C:/tmp/screenshot-suggestions.jpg' and you will see the screenshot of the website with the words "Web browsing" typed into the search input field and that the suggestion list has popped up.

To fetch the suggestions from this box, we again use XPath. This way we can output every suggestion's title and url. To do this, you add the following code to the robot:   

// get text from suggestion box
var suggestions = Web.xPath(support_page, "//ul[contains(@class,'ui-autocomplete')]//li");
foreach(suggestion in suggestions) {
   var text = Web.xPath(suggestion, "./span/text()");
   var url = Web.xPath(suggestion, "./a/@href");
   System.print("Suggestion=> " :: text :: "(" :: url :: ")");
}

Clicking a button

Although these suggestions might be coming in handy some time later, we won't be using them now. Let's just push the search button. Use an XPath to select the right node again. After putting the submit node in a variable, use the click(element) function on it. Add the following lines to your robot to enable it to do that:

// clicking submit button to start the search
var submit_button = Web.xPath(support_page,"//button[@type='submit']");
System.print(submit_button);
Web.click(submit_button);
var breakpoint = true; // this is only a statement to enable a paused state after the click statement

With the output you can verify if the button was actually selected. You can also see in the debug panel that the value of support_page has been changed.

Extracting information

Say you wish to conveniently view all titles and urls of the pages that resulted from your search. For that you will need some html extraction that is also covered in the html/xml tutorial, but this overlap is deliberate.

First, do the same search query ('web browsing') in your browser on this support site. Then, look at the html of the search results. That way, you should come up with an XPath like //ul[@class='search-results-list']/li[@class='article-item']. Since the title and url are both contained in the <a tag, it's a good idea to add '/a' to the XPath as well.

Add the following lines to your robot:

var search_results = Web.xPath(support_page,"//ul[@class='search-results-list']/li[@class='article-item']/a");
System.print(search_results);

So now we are close to what we want to achieve, but the title and url are still isolated. If you looked at the html well enough, you have noticed that the found keywords in the titles are enclosed in highlighting tags, which means that you will not be able to extract full titles with an XPath only. 

Extracting text from elements

Now that we have selected the search results, we should extract the text from these results. To do this, we use the Web.getText() function. This function will be the solution of the problem that arises when the text you want is spread over multiple html tags. Using that for the titles, the XPath /@href for the urls and a foreach loop to walk over the results, this code is what should do what we were trying:

System.print("SEARCH RESULTS:\r\n");
foreach(result in search_results) {
    var text = Web.getText(result);
    var url = Web.xPath(result,"./@href");
    System.print(text :: " (" :: url :: ")");
}

Download

Let's assume we want to download a file or an image that is used on a website. We could do this using the REST-library, but the Web kit has a function build in for this. The Web.download() function is used to download the given url to a file on the local hard drive. We could use this to e.g. download the logo of this web site.

// download the logo of the website
var url = Web.xPath(support_page, "//img[@data-type='logo']");
url = Web.xPath(url, "./@src");
Web.download(url, "C:/tmp/logo.jpg", null, 6000);