Do you need Python to fetch a URL for you? Depending on your needs, there are various ways python can request a URL:

• urllib2 - built in library
• httplib - built in library
• cURL - 3rd party library
• open a socket with asyncore - built in library

For this article, we'll be taking a look at urllib2 for it's ease of use and flexibility.

Basics: A Simple URL Request

To keep it simple we'll focus on http requests even though urllib2 can work on http, https, ftp, ftps, and local files.

With just 2 lines of code, you just requested python to grab the contents of Yahoo!'s homepage.

import urllib2
response = urllib2.urlopen('http://yahoo.com') # fetch Yahoo!'s homepage

In that simple request we also received the server status / response code, the http response headers, and the contents of the HTML page.

code = response.code # the server response code (e.g. 200, 301, 404, 503, etc.)
headers = response.headers # header object that contains all the server response headers
contents = response.read() # contents of the URL (HTML, javascript, css, img, etc.)
    

This is a simple example of requesting a URL with Python. Check out the next post where we show you how to change the http request headers so you can change your user agent.