MyMethodsNET
MyCommunities

D04: Collecting Data from Large Online Platforms with APIs, Webscraping and DSA applications

Overview

This intensive statistical approaches course introduces advanced techniques for collecting and managing data from VLOPs, combining three major access regimes: API-based data retrieval, web scraping, and the emerging legal framework for researcher access under the EU Digital Services Act (DSA).

Applications are closed

About the Course

This course introduces advanced techniques for collecting and managing data from VLOPs, combining three major access regimes: API-based data retrieval, web scraping, and the emerging legal framework for researcher access under the EU Digital Services Act (DSA).

Digital researchers increasingly rely on data from Very Large Online Platforms (VLOPs) to study, for example, political communication, mis- and disinformation, recommender systems, and other online phenomena. 

This course introduces advanced techniques for collecting and managing data from VLOPs, combining three major access regimes: API-based data retrieval, web scraping, and the emerging legal framework for researcher access under the EU Digital Services Act (DSA).

Participants will learn to design and execute reproducible data collection workflows using open APIs (e.g., Reddit, YouTube, Wikipedia), build ethical and robust web scrapers, and understand how the DSA enables regulated access to platform data for studying systemic risks. The course also discusses AI-augmented data collection, and how large language models can and cannot support scraping and API-based workflows.

Participants should have basic familiarity with Python (e.g., variables, loops, functions, working with packages, simple data handling). No prior experience with APIs, web scraping, or EU regulatory frameworks is required. All coding exercises will be supported with template notebooks and example scripts.

Learning Objectives

By the end of the course, participants will be able to combine technical, legal, and methodological perspectives to collect platform data responsibly and effectively, and to understand how AI can (and cannot) be used in modern data collection pipelines.

Participants will learn to:

  • Collect data using open APIs
  • Build web scrapers for semi-structured webpage content
  • Understand and navigate DSA researcher-access provisions
  • Use AI tools to enhance scraping and API querying
  • Evaluate legal, ethical, and methodological implications of all access regimes
Day 1 APIs: Immediate Hands-On Start

09:00 – 12:00

  • Brief overview of access regimes (web scraping, API’s, and DSA)
  • API fundamentals and live demos (e.g., YouTube, Wikipedia)

12:00 – 13:30 Lunch Break

13:30 – 15:15

  • Pagination, errors, data storage
  • Hands-on exercise
Day 2    Web Scraping I: Foundations

09:00 – 12:00

  • HTML/DOM basics
  • CSS selectors, BeautifulSoup
  • Legal/ethical boundaries

12:00 – 13:30 Lunch Break

13:30 – 15:15

  • Paginated scraping, data storage
  • Hands-on exercise
Day 3 Web Scraping II: Harder Cases 

09:00 – 12:00

  • JS-rendered pages
  • Selenium
  • Infinite scroll and dynamic content
  • Anti-fragile scrapers: patterns, anti-patterns, and bot compliance

12:00 – 13:30 Lunch Break

13:30 – 15:15

  • Debugging
  • Hands-on exercise 
Day 4 DSA + AI-Augmented Collection

09:00 – 12:00 — DSA-Based Researcher Access by Prof. Dr. Hajo Boomgaarden

  • What the DSA offers researchers
  • Art. 40 and access mechanism
  • Available datasets and transparency tools
  • Tips on drafting access requests
  • Governance and methodological implications

12:00 – 13:30 Lunch Break

13:30 – 15:15 

  • AI-Augmented Data Collection
  • Hands-on exercise
Day 5  09:00 – 12:00 — Practical Collection Workflows
  • Scheduling scrapers
  • Monitoring breakage
  • Logging/versioning
  • Making your collectors reproducible

 12:00 – 13:30 Lunch Break

13:30 – 15:15 — Multi-Source Integration

  • Combining API + scraping + DSA-based data collection
  • Matching via IDs, URLs, timestamps
  • Hands-on exercise

This description is subject to change at the discretion of the Instructor

2 Credits

For completion of all work before and during the course, as outlined by the Instructor, and 90% participation and attendance of the course.

2 Additional Credits

Course specific extra assignments. These can include submitting assignments before the course, daily assignments, and/or a final assignment to be completed after the course as decided by the Instructor.

Instructor

Noëlle Sophie Lebernegg

noelle.sophie.lebernegg@univie.ac.at

Dr. Noëlle Sophie Lebernegg is a postdoctoral researcher in the Strategic Communication Science Group at Wageningen University & Research. Her work focuses on political communication, computational text analysis, and large-scale data collection, with a particular interest in transparent, reproducible, and bias-aware research practices. She earned her PhD from the University of Vienna, where she tested automated and transparent approaches to analyzing news content using linguistic and stylistic features. She brings extensive experience in building web-scraping tools, working with APIs, and coordinating large-scale data collection pipelines. In her teaching, she covers quantitative methods, data science, and computational research methods.

Noëlle Sophie Lebernegg

Pricing

15% off During Early Bird!

  • Student Member

    €699.99

  • Student Non-Member

    €999.99

  • Other Member

    €849.99

  • Other Non-Member

    €999.99

Secure Your Place!

Please complete this webform for your registration.

Registration

Important Information

  1. Complete in English only 
  2. Do not complete in capital letters
  3. Course fees are reduced for MethodsNET members
  4. If you are not a MethodsNET Member at the time of completing this form you will pay the non-member fee
  5. If your institution or organization is paying for your course, complete the correct invoice information 
  6. Please note that your seat in a course is only reserved and guaranteed after you submit full payment of the registration fee.
Course Selection

You can select one Online + one All Day in each week or one Online + one AM & PM in each week

Address
Invoice Address (if different from above)

By submitting the form, you are agreeing with our Terms & Conditions, Code of conduct, and Privacy Policy.