Get the Source of Webpages in AppleScript

Sometimes you need data from the internet to set as a variable in an AppleScript. Fortunately, there's a shell program that allows you to get the source of a page: curl.

To use curl in a script, just use:

set the_source to do shell script "curl http://www.theurl.com"

Obviously, replace the URL in the above sample with the URL you want to get the data from. Unfortunately, this doesn't work with every site, such as Google search results.

Once you have the source, you then need to use AppleScript's text item delimiters to parse the text.