Benjamin W. Bohl bwbohl

Web Scraping with XQuery

Overview

Learning how to web scrape empowers you to apply your XQuery skills to any data residing on the web. You can fetch data from remote sites and services—for example, entire web pages or just the pieces of a page that matter to you. Once fetched, you can perform further analysis on the data, clean it up, mash it up with other data, transform it into different formats, etc.

Built-in functions for making HTTP requests

XPath-based languages like XQuery offer an standard function for accessing remote documents, the fn:doc() function. However, a limitation of this function is that it only works if the URI returns a well-formed XML document.

Project Title

One Paragraph of project description goes here

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

-<!--
-  XSLT for removing unused namespaces from an XML file.
-  Author: Dimitre Novatchev
-  Source: https://stackoverflow.com/a/4594626
-  License: CC BY-SA, https://creativecommons.org/licenses/by-sa/2.5/
-  Usage:
-      xmlstarlet tr remove-unused-namespaces.xslt -

	<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:math="http://exslt.org/math" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:uuid="http://www.uuid.org" version="2.0">
	<xsl:output method="xml" encoding="UTF-8"/>
	<xsl:template match="/">
	<Payment xmlns="http://ws.apache.org/ns/synapse">
	<xsl:for-each select="//Order/lunch">
	<discount>
	<li>
	<xsl:attribute name="uid" select="uuid:get-uuid(drinkName)"/>
	<xsl:value-of select="drinkName"/>
	<span>

	$ curl --help
	Usage: curl [options...] <url>
	--abstract-unix-socket <path> Connect via abstract Unix domain socket
	--alt-svc <file name> Enable alt-svc with this cache file
	--anyauth Pick any authentication method
	-a, --append Append to target file when uploading
	--basic Use HTTP Basic Authentication
	--cacert <file> CA certificate to verify peer against
	--capath <dir> CA directory to verify peer against
	-E, --cert <certificate[:password]> Client certificate file and password

	#!/usr/bin/env sh
	set -e

	echo "Pulling latest code..."
	git pull
	echo "Deleting local branches that were removed in remote..."
	git fetch -p
	git branch -vv \| awk '/: gone]/{print $1}' \| xargs git branch -D
	echo "Remaining local branches:"
	git branch -vv

	xquery version "3.0";

	module namespace dtp-utils = 'http://46.28.111.241:8081/exist/db/apps/karolinum-x/modules/dtp-utils';
	import module namespace cust-utils = 'http://46.28.111.241:8081/exist/db/apps/karolinum-x/modules/cust-utils' at 'cust-utils.xqm';
	declare namespace tei = 'http://www.tei-c.org/ns/1.0';

	(:~ This module is useful for in-memory converting of TEI Simpe XML into XML suitable
	: for importing into InDesign. It is not a silver bullet, it was tested and developed
	: for a very specific scenario. However, it should be useful for anyone who uses TEI
	: Simple XML (possibly with minor modifications of replacing strings and so on) and

	1. git rebase --interactive HEAD~10
	2. pick commit and replace `pick` for "child" commits to `s` (it means squash)
	3. save & force push

	xquery version "3.0";
	(:~
	This module contains helper functions for dealing with URLs.

	When receiving webmention the post parameter 'source' is a URL pointing to the
	origin of the mention.

	Before we get and store a sanitized version of an HTML document the url links to
	we want to able to