User:Multichill/Guide to data importing
Data importing or data ingestion is the creation or expansion of multiple items in an automated manner. This guide aims to explain how to do this.
Before you start: licensing and permissions[edit]
Before you even think about data importing, be sure 100% sure that the metadata under a Creative Commons Zero (CC0).
But my data is published under a CC-BY-SA license! We can't use it through auto mass import because it is incompatible with the license of Wikidata. It's a restrictive license for data and goes against our mission of providing free, reusable data.
For data in databases in the USA we have a similar copyright situation as reproductions of 2D artworks: It's (probably) in the public domain. See the Wikimedia Foundation’s preliminary perspective on the legal issue of database rights. So from a legal standpoint we're probably save to copy data from a museum in the US, just like we can copy the reproductions of old paintings. Just like with with copying reproductions, we rather do it together with the GLAM than without their knowledge and consent. See also this overview of cases studies of working together with GLAMs.
Examples[edit]
Throughout this guide I'll use two examples: One Rijksmonument and one painting in the Rijksmuseum. Both have data available in json format. This could be any other format like cvs, tsv, xml, etc as long as it is record based. Some example code is available in git
Rijksmonument[edit]
Grote Kerk (Q1545193) 
from https://tools.wmflabs.org/heritage/api/api.php?action=search&format=json&srlang=nl&srcountry=nl&srid=19264 :
Extended content |
|---|
{
"country": "nl",
"lang": "nl",
"id": "19264",
"adm0": "nl",
"adm1": "nl-nh",
"adm2": "Haarlem",
"adm3": null,
"adm4": null,
"name": "[[Grote of Sint-Bavokerk]]",
"address": "Oude Groenmarkt 23",
"municipality": "Haarlem",
"lat": 52.38107,
"lon": 4.63729,
"image": "Grote-Kerk-Haarlem.jpg",
"commonscat": "Grote Kerk, Haarlem",
"source": "https://nl.wikipedia.org/w/index.php?title=Lijst_van_rijksmonumenten_in_Haarlem_Centrum&redirect=no&useskin=monobook&oldid=43017369",
"monument_article": "Grote_of_Sint-Bavokerk",
"registrant_url": "http://monumentenregister.cultureelerfgoed.nl/php/main.php?cAction=search&sCompMonNr=19264",
"changed": "2015-02-14 04:28:33"
}
|
Painting[edit]
The Continence of Scipio (Q17340637) from https://www.rijksmuseum.nl/api/en/collection/SK-A-4690?key=secret&format=json :
Extended content |
|---|
{
"elapsedMilliseconds": 58,
"artObject":
{
"links":
{
"search": "https://www.rijksmuseum.nl/api/nl/collection"
},
"id": "en-SK-A-4690",
"priref": "8966",
"objectNumber": "SK-A-4690",
"language": "en",
"title": "The Continence of Scipio",
"copyrightHolder": null,
"webImage":
{
"guid": "dac1084a-db3a-4ba9-9030-7ecbe44f349c",
"offsetPercentageX": 50,
"offsetPercentageY": 50,
"width": 2896,
"height": 1592,
"url": "http://lh4.ggpht.com/M0r69XZnc2N7XyJL8UFiVZVlQJSblSkQPRomkJr3HajYE3KBD51hKfPf8PXH5DuRm7qOZNb7NhsgKjFymPgR581042w=s0"
},
"colors":
[
"#42281F",
" #64796A",
" #5C503D",
" #ACA080",
" #9C7A56",
" #122455",
" #254880"
],
"colorsWithNormalization":
[
{
"originalHex": "#42281F",
"normalizedHex": "#000000"
},
{
"originalHex": " #64796A",
"normalizedHex": "#2F4F4F"
},
{
"originalHex": " #5C503D",
"normalizedHex": "#737C84"
},
{
"originalHex": " #ACA080",
"normalizedHex": "#E0CC91"
},
{
"originalHex": " #9C7A56",
"normalizedHex": "#B35A1F"
},
{
"originalHex": " #122455",
"normalizedHex": "#4019B1"
},
{
"originalHex": " #254880",
"normalizedHex": "#4279DB"
}
],
"normalizedColors":
[
"#8B4513",
" #696969",
" #D2B48C",
" #CD853F",
" #191970",
" #483D8B"
],
"normalized32Colors": [ ],
"titles": [ ],
"description": "De Grootmoedigheid van Scipio. Links herenigt Scipio, omringd door soldaten, de twee verloofden. Rechts en in het midden zien mannen en vrouwen toe. Op de achtergrond een landschap aan de rand van een bos met antieke ruïnes en gevechten. Op de achterzijde staat een allegorie op de natuur: een naakte man wordt onderwezen door de vrouwelijke personificaties van Natuur (met stedenkroon en vele borsten) en Wet (met de wetstafelen).",
"labelText": null,
"objectTypes":
[
"painting"
],
"objectCollection":
[
"paintings"
],
"makers":
[
{
"name": "Karel van Mander (I)",
"unFixedName": "Mander, Karel van (I)",
"placeOfBirth": "Meulebeke",
"dateOfBirth": "1548-05",
"dateOfBirthPrecision": null,
"dateOfDeath": "1606-09-11",
"dateOfDeathPrecision": null,
"placeOfDeath": "Amsterdam",
"occupation":
[
"print maker",
"draughtsman",
"painter"
],
"roles":
[
"painter"
],
"nationality": "Zuid-Nederlands",
"biography": null,
"productionPlaces": [ ],
"schoolStyles": [ ],
"qualification": null
}
],
"principalMakers":
[
{
"name": "Karel van Mander (I)",
"unFixedName": "Mander, Karel van (I)",
"placeOfBirth": "Meulebeke",
"dateOfBirth": "1548-05",
"dateOfBirthPrecision": null,
"dateOfDeath": "1606-09-11",
"dateOfDeathPrecision": null,
"placeOfDeath": "Amsterdam",
"occupation":
[
"print maker",
"draughtsman",
"painter"
],
"roles":
[
"painter"
],
"nationality": "Zuid-Nederlands",
"biography": null,
"productionPlaces": [ ],
"schoolStyles": [ ],
"qualification": null
}
],
"plaqueDescriptionDutch": null,
"plaqueDescriptionEnglish": null,
"principalMaker": "Karel van Mander (I)",
"artistRole": null,
"associations": [ ],
"acquisition":
{
"method": null,
"date": "1977-03-01T00:00:00Z",
"creditLine": "Purchased with the support of the Stichting tot Bevordering van de Belangen van het Rijksmuseum"
},
"exhibitions":
[
{
"title": "Het Rijksmuseum aan de Schelde",
"organiser": null,
"place": "Antwerpen",
"start": "2004-10-02T00:00:00Z",
"end": "2008-01-01T00:00:00Z",
"startYear": 2004,
"endYear": 2008
}
],
"materials":
[
"oil paint"
],
"techniques": [ ],
"productionPlaces": [ ],
"dating":
{
"early": null,
"earlyPrecision": null,
"late": null,
"latePrecision": null,
"year": 1600,
"yearEarly": 1600,
"yearLate": 1600,
"period": 16
},
"classification":
{
"iconClassIdentifier":
[
"98B(SCIPIO AFRICANUS MAJOR)51",
"20",
"25L11"
],
"iconClassDescription":
[
"the continence of Scipio: he restores a captive woman to her bridegroom Allucius, who kneels before him; the ransom offered by the bride's parents is given to Allucius as a wedding-gift",
"'Natura' (allegorical figure or scene; or as Diana of Ephesus, with many breasts); 'Natura' (Ripa)",
"city-crown, mural crown"
],
"motifs": [ ],
"events": [ ],
"periods": [ ],
"places": [ ],
"people": [ ],
"objectNumbers":
[
"SK-A-4690"
]
},
"hasImage": true,
"historicalPersons": [ ],
"inscriptions":
[
"signature and date",
"16 KM 00"
],
"documentation":
[
"A. Jensen Adams, Public Faces and Private Identities in Seventeenth-Century Holland, Portraiture and the Production of Community, New York 2009, p. 191, fig. 56"
],
"catRefRPK": [ ],
"principalOrFirstMaker": "Karel van Mander (I)",
"dimensions":
[
{
"unit": "cm",
"type": "height",
"part": null,
"value": "44"
},
{
"unit": "cm",
"type": "width",
"part": null,
"value": "79"
}
],
"physicalProperties": [ ],
"physicalMedium": "painting",
"longTitle": "The Continence of Scipio, Karel van Mander (I), 1600",
"subTitle": "h 44cm × w 79cm",
"scLabelLine": "Karel van Mander i (1548–1606), oil on copper, 1600",
"label":
{
"title": "The Continence of Scipio",
"makerLine": "Karel van Mander i (1548–1606), oil on copper, 1600",
"description": "Depicted here is a famous example of clemency and restraint from Classical antiquity. After conquering Carthage, the Roman general Scipio gave a beautiful young woman he had received as war booty back to her fiancé. He even returned the gifts her parents offered him. Van Mander favoured bold colours, which are here set off to good advantage against the landscape.",
"notes": null,
"date": "2013-03-28"
},
"showImage": true
},
"artObjectPage":
{
"id": "en-SK-A-4690",
"similarPages":
[
{
"name": "Karel van Mander I",
"url": "/en/explore-the-collection/overview/karel-van-mander-i"
},
{
"name": "Mannerism",
"url": "/en/explore-the-collection/styles/mannerism"
},
{
"name": "History scenes",
"url": "/en/explore-the-collection/works-of-art/history-scenes"
}
],
"lang": "en",
"objectNumber": "SK-A-4690",
"tags": [ ],
"plaqueDescription": null,
"audioFile1": null,
"audioFileLabel1": null,
"audioFileLabel2": null,
"createdOn": "0001-01-01T00:00:00Z",
"updatedOn": "0001-01-01T00:00:00Z",
"adlibOverrides":
{
"titel": null,
"maker": null,
"etiketText": null
}
}
}
|
Analyze the data[edit]
Take on or more records and see if you can see what all the fields are about. Look for a field that can function as a primary key and a field that can be used for the label. You don't have to use all fields in the original record and you don't have to understand all of them from the start
Rijksmonument[edit]
- "country" - Every Rijksmonument is in the Netherlands so we can just add country (P17) -> Netherlands (Q55) to every item
- "lang" - We can use this to set the label in the right language
- "id" - Can be used in Rijksmonument ID (P359). Dedicated property for unique key
- "adm0" - Same as country
- "adm1" - On of the provinces of the Netherlands
- "adm2" - One of the municipalities of the Netherlands
- "adm3" - Probably empty
- "adm4" - Probably empty
- "name" - Could be used for label
- "address" - Can be used in P969 (P969)
- "municipality" - Same as adm2
- "lat" & "lon" - Can be used in coordinate location (P625)
- "image" - Can be used in image (P18)
- "commonscat" - Can be used in Commons category (P373)
- "source" - Permalink to the source, not used
- "monument_article" - Wikipedia article. Mostly empty
- "registrant_url" - Url to the source, can be used as reference
- "changed" - Not used
Painting[edit]
The metadata from the Rijksmuseum is a bit more extensive. Some selected fields:
- "objectNumber" - Can be used in inventory number (P217). It's not a dedicated property so to make it unique we add collection (P195) -> Rijksmuseum (Q190804) to the item and as qualifier
- "language" - We can use this to set the label in the right language. Rijksmuseum api can serve in Dutch (nl) and English (en)
- "title" - Could be used for label
- "physicalMedium" - Can be used in instance of (P31)
Find unique key[edit]
For every data import we need to find an unique key. This way we can update existing items instead of creating duplicate items on each run. We have two types of unique keys:
- Dedicated properties: Properties that are unique in itself
- Combined properties: A property that is unique in a certain domain. This property needs to be qualified by it's domain
Wikidata Query can be used to make a lookup tabel with <key> -> <wikidata item>
Rijksmonument[edit]
Each Rijksmonument (Q916333) has an unique identifier. This identifier is stored in the dedicated property Rijksmonument ID (P359). For our example Rijksmonument ID (P359) is "19264". Each Rijksmonument should have exactly one identifier and each identifier should only be used on one item. Each night a constraint violation report is run to report on any items that don't confirm to these and other constraints. Wikidata Query output for lookup table
Painting[edit]
Each painting in the Rijksmuseum has an unique object identifier. This identifier is only unique within the Rijksmuseum collection. This identifier is stored in the combined property inventory number (P217). It's not a dedicated property so to make it unique we add collection (P195) -> Rijksmuseum (Q190804) as qualifier. For our example inventory number (P217) is "SK-A-4690". For these combined properties we don't have constraint violation reports (yet). Wikidata Query output for lookup table
Clean up existing items[edit]
You figured out the data, you have some sort of unique identifier, you want to start importing. Hold on. Before you do that you should first clean up existing items so that you don't import duplicates or increase a messy subset. This generally consists of several steps:
- Try to find all in scope items and add statements to put them in your view
- Add identifiers to all in scope items
- Do other clean up on existing items (optional)
Rijksmonument[edit]
Every Rijksmonument should have heritage designation (P1435) -> Rijksmonument (Q916333). I had a bot loop over nl:Categorie:Rijksmonument and subcategories to add this statement to all articles (and thus items) about Rijksmonumenten. When this was done I asked Autolist which items are a Rijksmonument, but don't have an identifier. I worked on all the items to find the missing identifiers so the list should be empty.
Paintings[edit]
Every painting in the Rijksmuseum should have instance of (P31) -> painting (Q3305213) and collection (P195) -> Rijksmuseum (Q190804). I had a bot loop over en:Categorie:Schilderij in het Rijksmuseum and nl:Category:Collections of the Rijksmuseum Amsterdam to add this. When this was done I asked Autolist which items are in the Rijksmuseum collection, but don't have an identifier. I worked on all the items to find the missing identifiers so the list should be empty.
Basic import[edit]
Add more statements[edit]
See also[edit]
- This guide is (loosely) based on the Guide to batch uploading.