Source Of Summary:

handout 1 - visualization.pdf

Data Collection and Visualization with Python

1. Data Collection 🚀💾

Overview

• First Step in Data Science:
  • The very first step is to get some data!
  • Data can be obtained in four ways:
    1. Direct Download: Manually download a data file (or files). 📥
    2. Database Query: Query data from a database (covered in later lectures). 🗄️
    3. Query an API: Use web-based APIs to obtain data. (Very common today) 🌐
    4. Web Scraping: Extract data from a webpage (covered today). 🕸️

2. Issuing HTTP Queries & REST APIs 🌐🔍

HTTP Request Basics

• Automated Data Queries:

  • Most automated queries use HTTP requests (the dominant protocol for web queries).

  • Python's Requests Library:

    import requests
    response = requests.get("<http://www.datasciencecourse.org>")
    # Check status and content
    response.status_code
    response.content  # or response.text
    response.headers
    response.headers['Content-Type']
    

  • Reference: Requests Documentation

Using URL Parameters

• Example:

  • When you see a URL with extra parameters (e.g., from Google), you can provide them with the params argument:

    params = {"sa": "t", "rct": "j", "q": "", "esrc": "s", "source": "web", "cd": "9", "cad": "rja", "uact": "8"}
    response = requests.get("<http://www.google.com/url>", params=params)
    

  • HTTP GET is most common; methods like PUT, POST, and DELETE also exist.

RESTful APIs

• Key Points:

  1. Use standard HTTP methods (GET, PUT, POST, DELETE).
  2. Stateless: The server does not retain client state.
  3. Access Tokens: Often, each call includes an access token (e.g., for GitHub API).

• Example Query with REST API:

  token = ""  # Your access token here
  response = requests.get("<https://api.github.com/user>", params={"access_token": token})
  print(response.content)
  

Authentication

• Basic Authentication:

  • Traditionally used, but many APIs now use OAuth.

  • Example:

    response = requests.get("<https://api.github.com/user>", auth=('username', 'passwd'))
    

  • Note: Basic auth is largely being replaced by OAuth.

3. Data Formats 📁🔤