Stefan Laser

2.2. Daten mit Jupyter-Notebooks scrapen

Tool #2

Goals

  • Learn how to run a Jupyter notebook from Jupyter. See https://ruhr-uni-bochum.jupyterhub.nrw/
  • Call an API to get data
  • Make an annotated visualization

Der Fall

Wir bleiben bei KI, nun aber mit 5k Einträgen! Hierfür muss man automatisieren.

Daten

CSV:


wikipedia-articles_AI.csv

Hier ist eine Liste aller KI-gerankten Artikel enthalten.

Open the notebook in Google Colab

A bit of context:

  • A Jupyter notebook is an online document that contains executable code (in Python), text, and images.
  • Executing the code requires a computer. Und wir verbinden uns mit JupyterLab, einer virtuellen Maschine, die das für uns ausführt.
  • The script calls the API of Wikipedia to ask for all the edits on each of the pages on the list. These edits are called reivions and are available through this endpoint.

We will use the following script. It opens directly in Colab (although it is actually stored on GitHub).

🍹 Wikipedia articles to edits list

Note: the emoji of each notebook is just there to help you memorize which notebook does what.

But check this tutorial first. It uses the same file and notebook.

Run the notebook to harvest data

  • Upload the CSV data into the virtual machine. Click on the File icon on the left, then on the icon on top the reads Upload to session storage and upload the CSV from above.
  • Read the notebook text itself, SETTINGS included, stopping at SCRIPT. You are not expected to understand the code itself. Just what it does.
  • Edit the settings. In particular, the name of the input file is probably not that indicated in the settings. You can change the variable in the settings, or rename the file in the virtual machine.
  • Run the notebook. Each cell can be executed individually, but we recommend to run everything at once from the menu: Runtime > Run all. It is done when the last cell outputs “Done”.
  • Download the output file from the virtual machine. First you to refresh the files of the virtual machine, in the side bar, by clicking on the Refresh icon. Then look for the output file, whose name was specified in the settings of the script, and download it via its drop-down menu, on the three dots on the right of the file.

You should obtain a file like this one.

Make an annotated visualization

The tableau visualization may look like this…

Timeline

…or maybe like that…

Stacked timeline

…and there are many other possibilities.

The notebook allows dealing with bigger data, which unlocks new research questions. You may pick one of those for your annotations:

  • Are the articles synchronized or not? Are they edited at the same time?
  • Are some articles older, more recent? Note: would you look at the date of first edit, or something else?
  • Are there different moments where different articles get the most revisions?
  • Are certain articles only edited in certain parts of the world?

Tip: you can use the Article dimension in Tableau’s Pages shelf (see help) to quickly compare across Articles.

Stacked timeline

Documents produced

Keep somewhere, for sharing, the following document:

  • The annotated visualization (JPEG or PNG)

Next tutorial

Last break before lunch! Then head for the last activity of the morning:

 1.7. Activate your knowledge about Tableau (30 min)

Additional resources

Relation to the course readings

  • The process of getting data through APIs is covered in Chapter 6: Collecting and curating digital records of Venturini, T. & Munk, A.K. (2021). Controversy Mapping: A Field Guide.
  • The intricacies of Wikipedia and the different ways in which the platform may be reappropriated for controversy analysis are covered in Weltevrede, E., & Borra, E. (2016). Platform affordances and data practices: The value of dispute on Wikipedia Big Data & Society, 3(1).

Tools for getting similar data (CSV format with timestamps) from other sources: