Leveraging Columnar Formats for Analyzing Large Web Archive Collections

The Parquet format is an open source, column-oriented data file format designed for efficient data storage and retrieval. To demonstrate the potential for Parquet files to be used in analyzing web archives, we will use the End of Term (EOT) Web Archive as a data testbed for this tutorial. This half-day in-person tutorial will introduce participants to columnar formats and how they can be used in the analysis and exploration of large web archive collections. Specifically, it will work with the Parquet file format to demonstrate the features and opportunities that the file format offers. The subject area used to demonstrate these features will be analyzing and exploring large web archives like the End of Term (EOT) Datasets. The tutorial will include both lecture components and live-coding examples of how these file formats can be used to solve challenges presented by large web archives.