Wikidata:WikiProject Authority control/Archive

From Wikidata
Jump to navigation Jump to search

Wikidata pays a lot of tribute to authority control, linking to all kinds of datasets and databases with various IDs. The holy grail of every GLAM worker Sum of All People, with links to their Works is coming about!

But we’re just at the start of a lot of work in that direction. The purpose of this project is to try and coordinate such work.

I know a few things (ViafBot, Mix-n-Match) and I'd like to help with some things, but I don't know what others are doing. --Vladimir Alexiev (talk) 08:08, 27 January 2015 (UTC)[reply]

Data Sources[edit]

The report Name Data Sources for Semantic Enrichment shows that when it comes to name data sources, maybe the two that matter are VIAF and Wikidata.

  • Their name coverage is fairly orthogonal: VIAF has more name variations and permutations, Wikidata has more translations (Venn diagram of names for Cranach).
  • VIAF is much bigger: 35M persons/orgs. Wikidata has 2.7M persons and maybe 1M orgs.
  • Only 0.5M of Wikidata persons/orgs are coreferenced to VIAF, with maybe another 0.5M coreferenced to other datasets, either VIAF-constituent (eg GND) or non-constituent (eg RKDartists). So coreferenced part between the two is still quite small (30%) and a lot of work remains!
  • A lot can be gained by leveraging coreferencing across VIAF and Wikidata: finding errors in Authority files, finding merge candidates in Wikidata, promulgating identifiers...
  • Wikidata has great tools for crowd-sourced coreferencing.

Please comment!

RKDArtists Coreferencing[edit]

RKDartists is an important Authority that does not yet participate in VIAF. There are already 21760 RKDartist id's on Wikidata. These could be imported to VIAF for free!

British Museum Coreferencing[edit]

The BM has several thesauri that are not co-referenced to anything in the world. I think they'd see it as a major win if the community helps them to co-reference.

This could be followed by importing the 2.5M cultural objects of the BM.

ULAN Coref Relations[edit]

ULAN does record possible matches and mismatches in their editorial system: ULAN Artists Whose Identity May be Associated or Confused With Another (608 pairs).

Looks like this:

x x_name x_bio rel y y_name y_bio
ulan:500071106 Master of 1515 Portuguese painter, active 1515 gvp:ulan1005_possibly_identified_with ulan:500025279 Afonso, Jorge Portuguese painter and court artist, born ca. 1470-1475, died before 1540
ulan:500042027 Master of the Madre de Deus Retablo Portuguese painter, active 16th century gvp:ulan1005_possibly_identified_with ulan:500025279 Afonso, Jorge Portuguese painter and court artist, born ca. 1470-1475, died before 1540
ulan:500032055 Monogrammist A. M. Spanish artist, active 19th century gvp:ulan1005_possibly_identified_with ulan:500038287 Aguirre, Marcial Spanish sculptor, 1841-1900

Here's to proper coreferencing! --Vladimir Alexiev (talk) 18:07, 12 March 2015 (UTC)[reply]

Match Persons not Disambiguation Pages[edit]

We should match persons to persons, not disambiguation pages to persons or other disambiguation pages.

Wikipedias, GND and RKD all have disambiguation pages (in GND they are called "undifferentiated names"). 13 Feb 2015:

Do you agree with my reasoning:

  • Jane said "any match is better than none"
  • I countered "A correct match is better than none"
  • the only way to make sure it's correct is to examine more data about the person, which will necessarily lead you to a real person page.
  • Look at the ULAN data above: that's good data that gives you some basis for decision. A name alone does not.

--Vladimir Alexiev (talk) 19:34, 12 March 2015 (UTC)[reply]

 Support @Vladimir Alexiev, Randykitty, Ghuron: A GND Tn (Thesaurus name = undifferentiated) is not a stable disambiguation page. A Tn is a placeholder. It can be deleted, it can be upgraded into a Tp (Thesaurus person), or changed into a redirect. Works connected with a Tn will be checked by the library or archive who owns them and afterwards might be delinked. The database Online GND (OGND) includes only Tp numbers. --Kolja21 (talk) 00:04, 28 March 2015 (UTC)[reply]

Coreference AAT[edit]

AAT is a crucial thesaurus in cultural heritage. 

    select (count(*) as ?c) {
      ?x a skos:Concept; skos:inScheme aat: }

I think that's BAD. I'm sure going to need that coref for the Europeana Food and Drink Classification Scheme that will be based on Wikidata and AAT:

Update: the AAT-Wordnet coreference described below is brought into Wikidata. AAT is actively coreferenced on Mix-n-Match: 12985 (32%) matched, 3293 (8%) awaiting confirmation, 1553 (3.8%) confirmed no-matches, and 22543 (55.7%) awaiting matching. So it's way better than 2 years ago. Help coreference this pivot thesaurus that is of immense importance for Cultural Heritage! --Vladimir Alexiev (talk) 15:04, 19 September 2017 (UTC)[reply]

Coreference AAT with Mix-n-Match[edit]

The Wikidata coref tool Mix-n-Match has mostly been used for people until now. But I hope it can be used for concepts as well.

I made an export that includes AAT URL, preferred English label (without qualifier), parents (ascendants to root) and scope note (description). Could also add alternative labels, and labels in other languages (Dutch, Spanish, Chinese). 

select ?id (str(?lab) as ?label) ?parents (str(?scopeNote) as ?note) {
  ?x a gvp:Concept; dc:identifier ?id; gvp:prefLabelGVP/gvp:term ?lab;
     gvp:parentString ?parents.
  optional {?x skos:scopeNote [dct:language gvp_lang:en; rdf:value ?scopeNote]}
}

I saved as XML then converted to TDV: aat.rar. 

    rset --results tsv aat.xml > aat.tdv

Also see https://meta.wikimedia.org/wiki/Talk:Mix%27n%27match#Coreference_AAT !!!

Coreference AAT through BabelNet[edit]

Mix-n-Match has good automatic matching, but that works for people.

So let's check what other vocabs that are coref to AAT may be coref to Wikidata: According to Michiel Hildebrand's famous CH LOD diagram:

CH LOD, cultural heritage linked open data (thesauri only)
  • Wordnet. No such prop in Wikidata
  • I'd guess Wiktionary is coref to Wordnet, but Wikidata got no site links to Wiktionary
  • RKD Concepts. There's prop "RKDartists" and "RKDimages" but none for concepts
  • Rijksmuseum Concepts. There's "Rijksmonument" but none for concepts
  • Joconde: aha! There's Joconde work ID (P347), and it has 2275 instances, so that's better. Joconde is 18% coref to Wikidata but I don't know how much to AAT, maybe I can gain 1k here.
    • Looked at the results: nope, Joconde are all paintings, not concepts
  • Bibliopolis: never heard of it, and nope
  • SVCN: never heard of it, and nope

Then it dawns on me.

AAT-Wordnet coref[edit]

Ok, so off to look for that AAT-Wordnet coref. - Why yes, it's part of http://semanticweb.cs.vu.nl/europeana/skos/browse/ - I got a file from somewhere that says 

<aat_wordnet20_mappings>
  a void:Linkset;
    dcterms:title "AAT-Wordnet 2.0 mappings by Anna Tordai (baseline)" ;
    lib:source <http://semanticweb.cs.vu.nl/lod/getty/aat/> ;
    void:dataDump <bl_aat_wn.rdf> , <bl_norm_aat_wn.rdf> , <bl_sing_aat_wn.rdf> .

(Note: you can get those files from URLs like: http://semanticweb.cs.vu.nl/europeana/api/export_graph?graph=http://semanticweb.cs.vu.nl/lod/getty/aat/bl_sing_aat_wn.rdf&mimetype=text/plain&format=turtle)

These are called "baseline" (i.e. mostly literal matches). A quick conversion to Turtle and a line count: 

$ wc -l bl*
   2300 bl_aat_wn.ttl
   4369 bl_norm_aat_wn.ttl
   4303 bl_sing_aat_wn.ttl
  10972 total

Run a query at http://semanticweb.cs.vu.nl/europeana/user/query (specify entailment=None or else!): 

prefix getty:  <http://purl.org/vocabularies/getty/> 
prefix aat:  <http://purl.org/vocabularies/getty/aat/> 
select * {?x skos:inScheme getty:aat; skos:closeMatch ?y}

It returns 4592 (see below why).

AAT-Wordnet Overlaps[edit]

There is significant overlap between the files: 

$ cat bl* | sort| uniq | wc -l
4596
$ cat bl* |cut -d " " -f 1 | sort| uniq | wc -l
4581

The following AAT concepts have 2 matches: 

aat:bleachers
aat:boxcars
aat:cleavers
aat:feudalism
aat:groats
aat:jackstraws
aat:lats
aat:leotards
aat:morocco
aat:ninepins
aat:quoits
aat:shekels
aat:stairs

We need to reconcile them manually, eg 

aat:bleachers  skos:closeMatch  <http://www.w3.org/2006/03/wn/wn20/instances/synset-bleacher-noun-1> .
aat:bleachers  skos:closeMatch  <http://www.w3.org/2006/03/wn/wn20/instances/synset-bleachers-noun-1> .

The AAT definition is:

  • aat:bleachers vp:descriptiveNote "Use for benchlike tiered seating for spectators at, for example, outdoor sporting events, usually without weather or sun protection, affording less advantageous views than grandstands; may also be used for similarly constructed, often telescoping, indoor seating."@en .
  • Inspection at Wordnet 3.1 shows that the second one is right.

That's 4.6k matches, or 11% of AAT.

AAT-Wordnet2 Representation[edit]

The coref looks like this: 

    aat:wrought_iron skos:closeMatch <http://www.w3.org/2006/03/wn/wn20/instances/synset-wrought_iron-noun-1> .

And there's another file aat.ttl with rep like: 

aat:wrought_iron aat:parentPreferred aat:iron_alloy .
aat:wrought_iron vp:id "300011012" .
aat:wrought_iron vp:labelPreferred "wrought iron"@en .
aat:wrought_iron vp:labelNonPreferred "iron, wrought"@en .
aat:wrought_iron vp:labelNonPreferred "wrought-iron"@en .

This is quite old rep. The new rep uses numeric URL: http://vocab.getty.edu/aat/300011012 (and a bunch more data). So we need to construct a numeric URL.

AATNED-Cornetto Mapping[edit]

Cornetto is NL Wordnet and AATNED is NL AAT. I got another file saying: 

<aatned_cornetto_mappings>
  a void:Linkset ;
    dcterms:title "AATNED-Cornetto mappings by Anna Tordai (baseline)";
    lib:source <http://semanticweb.cs.vu.nl/lod/rkd/aatned/> ;
    void:dataDump <bl_aatned_cn.rdf.gz> , <bl_norm_aatned_cn.rdf.gz> , <bl_sing_aatned_cn.rdf.gz> .

Eg we have this for AAT 300191645 "salinity": 

bl_aatned_cn.ttl: aatned:zoutheid  skos:closeMatch  cornetto:synset-zoutheid-1-noun .
cornetto-wn20.ttl: cornetto:synset-zoutheid-1-noun cornetto:eqNearSynonym instances:synset-brininess-noun-1 .
cornetto-wn30.ttl: cornetto:synset-zoutheid-1-noun cornetto:eqNearSynonym wn30:synset-brininess-noun-1 .
aatned.ttl: aatned:zoutheid core:notation "300191645" .

The number of AATNED-Cornetto matches is as follows: 

> cat bl*|sort|uniq> bl_aatned_all.ttl
> wc -l bl_aatned_all.ttl
6917 bl_aatned_all.ttl
> cat bl_aatned_all.ttl|cut -d " " -f 1 | sort| uniq | wc -l
6857

There are more matches than AAT-Wordnet. There are also overlaps: 60 AATNED concepts (0.9%) have two Cornetto matches.

We need to merge AATNED-Cornetto with AAT-Wordnet. The correlation is simply by id, eg 

aatned.ttl: aatned:zwerfkeien core:notation "300011671"
aat.ttl: aat:boulder vp:id "300011671"

I guess the overlaps between them are quite big, eg for wrought_iron: 

aatned.ttl: aatned:smeedijzer core:notation "300011012"
bl_aatned_all.ttl: aatned:smeedijzer  skos:closeMatch  cornetto:synset-smeedijzer-1-noun .
cornetto-wn20.ttl:cornetto:synset-smeedijzer-1-noun cornetto:eqNearSynonym instances:synset-wrought_iron-noun-1 .
cornetto-wn30.ttl:cornetto:synset-smeedijzer-1-noun cornetto:eqNearSynonym wn30:synset-wrought_iron-noun-1 .
DBpedia-Wordnet3 coref[edit]

The other problem is bigger: 

    bn:s00081730n skos:exactMatch dbpedia:Wrought_iron, lemon-WordNet:wn30-14802262-n

It doesn't look like Wordnet3 and Wordnet2 share any IDs; we'll deal with that in next section.

Lets first do some queries at http://babelnet.org/sparql/ to see what we can see. Look for DBpedia-Wordnet matches: 

SELECT * WHERE {
  ?x skos:exactMatch ?y, ?z 
  filter(strstarts(str(?y),"http://dbpedia.org/resource/"))
  filter(strstarts(str(?z),"http://lemon-model.net/lexica/pwn/"))
} LIMIT 30

Download: https://www.dropbox.com/s/92gq5r1qm3yytkp/WN3toDBP.csv?dl=1.

It has 47607 rows like this (there's a decent chance this will cover the 6k AAT matches): 

"http://babelnet.org/rdf/s00075206n","http://dbpedia.org/resource/Sundowner_(drink)","http://lemon-model.net/lexica/pwn/wn30-07913081-n"
"http://babelnet.org/rdf/s00039711n","http://dbpedia.org/resource/Sonora_(genus)","http://lemon-model.net/lexica/pwn/wn30-01736256-n"
"http://babelnet.org/rdf/s00070026n","http://dbpedia.org/resource/Sealskin","http://lemon-model.net/lexica/pwn/wn30-04160261-n"
Wordnet3-Wordnet2 coref[edit]

Since Wordnet3 and Wordnet2 don't share any IDs, we can try to use Wordnet2-Wordnet3 coref made by Jacco van Ossenbruggen and Marc van Assem (VU University Amsterdam) in May 2010 with this VOID (manifest): 

<wn30-wn20-mappings-jacco>
	a void:Linkset ;
	dcterms:title "synset-level mappings from Wordnet 3.0 to 2.0, created by jacco's code" ;
	lib:source <http://purl.org/vocabularies/princeton/wn30/> ;
	void:dataDump
		<label-child-matches.ttl.gz> ,
		<label-childparent-matches.ttl.gz> ,
		<label-instance-matches.ttl.gz> ,
		<label-meronym-matches.ttl.gz>,
		<label-neargloss-matches.ttl.gz> ,
		<label-parent-matches.ttl.gz> ,
		<label-unique-matches.ttl.gz> ,
		<nearlabel-matches.ttl.gz> ,
		<glossmatches-m.ttl.gz> .

<wn30-wn20-mappings-sense>
	a void:Linkset ;
	dcterms:title "synset-level mappings from Wordnet 3.0 to 2.0, created by Mark using the Princeton WordSense mappings" ;
	lib:source <http://purl.org/vocabularies/princeton/wn30/> ;
	void:dataDump
		<synset-matches-based-on-multiple-sense-mappings-princeton.ttl.gz> ,
		<synset-matches-based-on-single-sense-mappings-princeton.ttl.gz> .

It's a complex affair consisting of many steps, but the major step (contributing 87% of all matches) is glossmatches-m.ttl that looks like 

    wn30:synset-wrought_iron-noun-1 terms:replaces instances:synset-wrought_iron-noun-1 .

And looking at wordnet-synset.ttl, we find the required wn30 ID: 

    wn30:synset-wrought_iron-noun-1 wn20schema:synsetId 114802262 .
AAT-Wikidata Sheets[edit]

After much querying and manual cleaning (over a day of effort), I made some sheets in this google folder:

  • AAT-DBpedia-Babelnet.xlsx: 3324 potential matches, fairly clean, but need checking by more people
  • AAT-DBpedia-Babelnet-80-judged.xlsx: example of correct & incorrect matches
  • AAT-Wikidata-25-judged.xlsx: example of correct & incorrect matches on Mix-n-Match
Vladimir Alexiev Jonathan Groß Andy Mabbett Jneubert Sic19 Wikidelo ArthurPSmith PKM Ettorerizza Fuzheado Daniel Mietchen Iwan.Aucamp Epìdosis Sotho Tal Ker Bargioni Carlobia Pablo Busatto Matlin Msuicat Uomovariabile Silva Selva 1-Byte Alessandra.Moi CamelCaseNick Songceci moz AhavaCohen Kolja21 RShigapov Jason.nlw MasterRus21thCentury Newt713 Pierre Tribhou Powerek38 Ahatd JordanTimothyJames Silviafanti Back ache AfricanLibrarian M.roszkowski Rhagfyr 沈澄心 MrBenjo S.v.Mering

Notified participants of WikiProject Authority control: I need your help!

User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)[reply]
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)[reply]
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)[reply]
Multichill (talk) 19:13, 8 July 2014 (UTC)[reply]
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative[reply]
Mushroom (talk) 00:10, 24 August 2014 (UTC)[reply]
Jheald (talk) 17:09, 9 September 2014 (UTC)[reply]
Spinster (talk) 15:16, 12 September 2014 (UTC)[reply]
PKM (talk) 21:16, 8 October 2014 (UTC)[reply]
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)[reply]
Sic19 (talk) 21:12, 19 February 2016 (UTC)[reply]
Wittylama (talk) 13:13, 22 February 2017 (UTC)[reply]
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)[reply]
Musedata102 (talk) 20:27, 26 November 2019 (UTC)[reply]
Hannolans (talk) 18:36, 16 April 2017 (UTC)[reply]
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)[reply]
User:7samurais
User:mrtngrsbch
User:Buccalon
Infopetal (talk) 17:54, 9 August 2019 (UTC)[reply]
Karinanw (talk) 16:38, 24 March 2020‎ (UTC)[reply]
Ahc84 (talk) 17:38, 26 August 2020 (UTC)[reply]
User:BeatrixBelibaste
Valeriummaximum
Bitofdust (talk) 22:52, 26 March 2021 (UTC)[reply]
Mathieu Kappler
Zblace (talk) 07:22, 24 December 2021 (UTC)[reply]
Oursana (talk) 13:16, 17 May 2022 (UTC)[reply]
Ham II (talk) 08:30, 25 January 2024 (UTC)[reply]
DaxServer (talk) 16:00, 24 May 2024 (UTC)[reply]

Notified participants of WikiProject Visual arts: Yours too!

  • Do some checks (add your initials in column "check")
  • Add Q numbers to the sheet
  • Merge WD items that already have Art & Architecture Thesaurus ID (P1014) (there are 8477) to the sheet to compare the matches (or remove them from the sheet if you're quite confident)

I could post the sheet as QuickStatements, but I think there are still 10% incorrect matches, especially for Styles and Periods (see Wikidata talk:WikiProject Visual arts/Item structure/Art movements. --Vladimir Alexiev (talk) 16:02, 7 March 2017 (UTC)[reply]

AAT-LCSH coreferencing[edit]

445 AAT-LCSH coreferences made by Getty editors.

400 of them are on the Getty LOD site (see query below), 45 are newly extracted 

select * {
  ?x skos:exactMatch|skos:closeMatch ?y.
  ?x skos:inScheme aat:
  filter not exists {?y skos:inScheme aat:}}

Geonames Feature Code[edit]

Vladimir Alexiev Jonathan Groß Andy Mabbett Jneubert Sic19 Wikidelo ArthurPSmith PKM Ettorerizza Fuzheado Daniel Mietchen Iwan.Aucamp Epìdosis Sotho Tal Ker Bargioni Carlobia Pablo Busatto Matlin Msuicat Uomovariabile Silva Selva 1-Byte Alessandra.Moi CamelCaseNick Songceci moz AhavaCohen Kolja21 RShigapov Jason.nlw MasterRus21thCentury Newt713 Pierre Tribhou Powerek38 Ahatd JordanTimothyJames Silviafanti Back ache AfricanLibrarian M.roszkowski Rhagfyr 沈澄心 MrBenjo S.v.Mering

Notified participants of WikiProject Authority control

Kopiersperre Jklamo ArthurPSmith S.K. Givegivetake fnielsen rjlabs ChristianKl Vladimir Alexiev Parikan User:Cardinha00 MB-one User:Simonmarch User:Jneubert Mathieudu68 User:Kippelboy User:Datawiki30 User:PKM User:RollTide882071 Andber08 Sidpark SilentSpike Susanna Ånäs (Susannaanas) User:Johanricher User:Celead User:Finnusertop cdo256 Mathieu Kappler RShigapov User:So9q User:1-Byte pmt Rtnf econterms Dollarsign8 User:Izolight maiki c960657 User:Automotom applsdev Bubalina Fordaemdur

Notified participants of WikiProject Companies WikiProject Cultural heritage has more than 50 participants and couldn't be pinged. Please post on the WikiProject's talk page instead.

User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)[reply]
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)[reply]
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)[reply]
Multichill (talk) 19:13, 8 July 2014 (UTC)[reply]
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative[reply]
Mushroom (talk) 00:10, 24 August 2014 (UTC)[reply]
Jheald (talk) 17:09, 9 September 2014 (UTC)[reply]
Spinster (talk) 15:16, 12 September 2014 (UTC)[reply]
PKM (talk) 21:16, 8 October 2014 (UTC)[reply]
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)[reply]
Sic19 (talk) 21:12, 19 February 2016 (UTC)[reply]
Wittylama (talk) 13:13, 22 February 2017 (UTC)[reply]
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)[reply]
Musedata102 (talk) 20:27, 26 November 2019 (UTC)[reply]
Hannolans (talk) 18:36, 16 April 2017 (UTC)[reply]
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)[reply]
User:7samurais
User:mrtngrsbch
User:Buccalon
Infopetal (talk) 17:54, 9 August 2019 (UTC)[reply]
Karinanw (talk) 16:38, 24 March 2020‎ (UTC)[reply]
Ahc84 (talk) 17:38, 26 August 2020 (UTC)[reply]
User:BeatrixBelibaste
Valeriummaximum
Bitofdust (talk) 22:52, 26 March 2021 (UTC)[reply]
Mathieu Kappler
Zblace (talk) 07:22, 24 December 2021 (UTC)[reply]
Oursana (talk) 13:16, 17 May 2022 (UTC)[reply]
Ham II (talk) 08:30, 25 January 2024 (UTC)[reply]
DaxServer (talk) 16:00, 24 May 2024 (UTC)[reply]

Notified participants of WikiProject Visual arts


GeoNames feature code (P2452) is applied only 33 times (see disscussion and archive, while there are 669 codes on Geonames. I'll ask Magnus to add the Geonames list to Mix-n-Match.

Update Jul 2020: Things are much better at https://www.wikidata.org/wiki/Property:P2452:

http://www.geonames.org/ontology/mappings_v3.01.rdf has the following mappings: 

     32 dbo    http://dbpedia.org/ontology/
      5 frgeo  http://rdf.insee.fr/geo/
     79 lgdo   http://linkedgeodata.org/ontology/
     31 schema http://schema.org/

Can we use them somehow to push this coreferencing further?

I'm a little wary about importing these wholescale, because geonames in not a CC0 database. It's one thing to be providing external links to GeoNames, it's another to be importing data.
I did look at these values recently for English places with geonames links that are marked as both village (Q532) and civil parish (Q1115575) (see eg Abberton (Q3137539) for an example), to identify which Geonames link corresponded to which role; but I purposely decided not to add a GeoNames feature code (P2452) statement.
It might be useful to be able to map the codes to Q-numbers here, to facilitate sanity checking of exisiting or proposed co-references. However, even then there are difficulties -- for example, I found that PPLA3 or PPLA4 at Geonames didn't necessarily match to distinctions we would want to make in a instance of (P31) here. Jheald (talk) 14:02, 13 March 2017 (UTC)[reply]

Supplement Wikidata items with properties from authorities (GND in particular)[edit]

Data from DifferentiatedPersons of GND can be used to fill missing properties of according items, e.g.,

  • date of birth/death (directly from gnd:dateOfBirth and gndo:dateOfDeath, for entries following YYYY or YYYY-MM-DD - everything else to be skipped)
  • affiliation (P1416) can be obtained by a join of gndo:affiliation to wd organizations (may be sparse currently, but can be repeated later on)
  • country (country (P17) or country of citizenship (P27)??) requires translation from gndo:geographicAreaCode, which refers to a customized code table derived from ISO 3166 (not part of GND) (table (pdf), rules)
  • aliases - require filtering of gndo:variantNameForThePerson, which carry no language tag, re. script and presumed language (would Lingua::Identify work here?)

For appropriate source statements see project chat -- Jneubert (talk) 06:35, 21 May 2017 (UTC) (with thanks to User:MisterSynergy and User:ChristianKl)[reply]

I am a big fan of standards but the ISO 3166 is used for modern countries, it does not as a consequence give the "nationality" of people who did precede a country. Thanks, GerardM (talk) 07:03, 21 May 2017 (UTC)[reply]
NB yes there are some, but at Wikidata we know about many more former countries. GerardM (talk) 07:13, 21 May 2017 (UTC)[reply]

A "sibling" of Mix-n-Match now imports birth/death dates from authority files: https://www.wikidata.org/wiki/User:Magnus_Manske/Mix%27n%27match_date_import. Also see discussion about this in relation to Getty ULAN: https://groups.google.com/forum/#!topic/gettyvocablod/TkdelW9RP1g --Vladimir Alexiev (talk) 09:38, 2 October 2017 (UTC)[reply]

Property proposal for applying SKOS mapping relations to "external identifiers"[edit]

In order be able to map a thesaurus more completely, and - more general - to make Wikidata fit as a linking hub for knowledge organiziation systems, I've proposed a new property which allows to qualify individual links by properties of type "external identifier" as in-exact (close/broad/narrow/related) match.

Please feel free to comment at https://www.wikidata.org/wiki/Wikidata:Property_proposal/mapping_relation_type.

Cheers, Jneubert (talk) 12:27, 28 August 2017 (UTC)[reply]

Grant proposal soweego[edit]

Vladimir Alexiev Jonathan Groß Andy Mabbett Jneubert Sic19 Wikidelo ArthurPSmith PKM Ettorerizza Fuzheado Daniel Mietchen Iwan.Aucamp Epìdosis Sotho Tal Ker Bargioni Carlobia Pablo Busatto Matlin Msuicat Uomovariabile Silva Selva 1-Byte Alessandra.Moi CamelCaseNick Songceci moz AhavaCohen Kolja21 RShigapov Jason.nlw MasterRus21thCentury Newt713 Pierre Tribhou Powerek38 Ahatd JordanTimothyJames Silviafanti Back ache AfricanLibrarian M.roszkowski Rhagfyr 沈澄心 MrBenjo S.v.Mering

Notified participants of WikiProject Authority control

There is a new grant proposal soweego for authority control. See discussion at https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego#Endorsements.

I've considered it seriously but I think it doesn't address the main problem, see a list of 11 considerations (which can also be read as a sort of programme for next important steps for WD authority control). Please express your opinion there. --Vladimir Alexiev (talk) 10:19, 2 October 2017 (UTC)[reply]

Where is this discussion with 11 considerations? Thanks, GerardM (talk) 12:22, 2 October 2017 (UTC)[reply]

Remove Obsolete Getty Vocabularies IDs[edit]

Getty Vocabs have 8k obsolete subjects (identifiers). You can find them at http://vocab.getty.edu/sparql with a query like 

select * {
  ?old a gvp:ObsoleteSubject; skos:inScheme aat: .
  optional {?old dct:isReplacedBy ?new}
}

Here are the numbers: I can put up the files somewhere: obsolete-AAT-2106.tsv, obsolete-TGN-1016.tsv, obsolete-ULAN-5574.tsv.

Vladimir Alexiev Jonathan Groß Andy Mabbett Jneubert Sic19 Wikidelo ArthurPSmith PKM Ettorerizza Fuzheado Daniel Mietchen Iwan.Aucamp Epìdosis Sotho Tal Ker Bargioni Carlobia Pablo Busatto Matlin Msuicat Uomovariabile Silva Selva 1-Byte Alessandra.Moi CamelCaseNick Songceci moz AhavaCohen Kolja21 RShigapov Jason.nlw MasterRus21thCentury Newt713 Pierre Tribhou Powerek38 Ahatd JordanTimothyJames Silviafanti Back ache AfricanLibrarian M.roszkowski Rhagfyr 沈澄心 MrBenjo S.v.Mering

Notified participants of WikiProject Authority control

Any takers to replace any old values of AAT ID Search, TGN ID Search, ULAN ID Search with the new values from the respective files?

@Magnus Manske: rdf:type gvp:ObsoleteSubject should be removed from Mix-n-Match consideration.

Done, but only changed 5 entries for the ~2100 obsolete ones, the others were either already N/A or matched with an item. --Magnus Manske (talk) 15:22, 21 February 2018 (UTC)[reply]
Data should not be removed, but - if necessary - marked as deprecated, with a qualifier giving the reason for deprecation. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 11:02, 26 February 2018 (UTC)[reply]

VIAF Games[edit]

Vladimir Alexiev Jonathan Groß Andy Mabbett Jneubert Sic19 Wikidelo ArthurPSmith PKM Ettorerizza Fuzheado Daniel Mietchen Iwan.Aucamp Epìdosis Sotho Tal Ker Bargioni Carlobia Pablo Busatto Matlin Msuicat Uomovariabile Silva Selva 1-Byte Alessandra.Moi CamelCaseNick Songceci moz AhavaCohen Kolja21 RShigapov Jason.nlw MasterRus21thCentury Newt713 Pierre Tribhou Powerek38 Ahatd JordanTimothyJames Silviafanti Back ache AfricanLibrarian M.roszkowski Rhagfyr 沈澄心 MrBenjo S.v.Mering

Notified participants of WikiProject Authority control

Ash Crow
Dereckson
Harmonia Amanda
Hsarrazin
Jura
Чаховіч Уладзіслаў
Joxemai
Place Clichy
Branthecan
Azertus
Jon Harald Søby
PKM
Pmt
Sight Contamination
MaksOttoVonStirlitz
BeatrixBelibaste
Moebeus
Dcflyer
Looniverse
Aya Reyad
Infovarius
Tris T7
Klaas 'Z4us' van B. V
Deborahjay
Bruno Biondi
ZI Jony
Laddo
Da Dapper Don
Data Gamer
Luca favorido
The Sir of Data Analytics
Skim
E4024
JhowieNitnek
Envlh
Susanna Giaccai
Epìdosis
Aluxosm
Dnshitobu
Ruky Wunpini
Balû
★Trekker

Notified participants of WikiProject Names

User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)[reply]
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)[reply]
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)[reply]
Multichill (talk) 19:13, 8 July 2014 (UTC)[reply]
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative[reply]
Mushroom (talk) 00:10, 24 August 2014 (UTC)[reply]
Jheald (talk) 17:09, 9 September 2014 (UTC)[reply]
Spinster (talk) 15:16, 12 September 2014 (UTC)[reply]
PKM (talk) 21:16, 8 October 2014 (UTC)[reply]
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)[reply]
Sic19 (talk) 21:12, 19 February 2016 (UTC)[reply]
Wittylama (talk) 13:13, 22 February 2017 (UTC)[reply]
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)[reply]
Musedata102 (talk) 20:27, 26 November 2019 (UTC)[reply]
Hannolans (talk) 18:36, 16 April 2017 (UTC)[reply]
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)[reply]
User:7samurais
User:mrtngrsbch
User:Buccalon
Infopetal (talk) 17:54, 9 August 2019 (UTC)[reply]
Karinanw (talk) 16:38, 24 March 2020‎ (UTC)[reply]
Ahc84 (talk) 17:38, 26 August 2020 (UTC)[reply]
User:BeatrixBelibaste
Valeriummaximum
Bitofdust (talk) 22:52, 26 March 2021 (UTC)[reply]
Mathieu Kappler
Zblace (talk) 07:22, 24 December 2021 (UTC)[reply]
Oursana (talk) 13:16, 17 May 2022 (UTC)[reply]
Ham II (talk) 08:30, 25 January 2024 (UTC)[reply]
DaxServer (talk) 16:00, 24 May 2024 (UTC)[reply]

Notified participants of WikiProject Visual arts WikiProject Cultural heritage has more than 50 participants and couldn't be pinged. Please post on the WikiProject's talk page instead.

Anyone interested in Authority Control, please read this section. It provides info about VIAF volumetrics, WD-VIAF volumetrics (very low!), correlation VIAF code - WD entity&prop, and an idea how to proliferate identifiers.

VIAF volumetrics[edit]

We at Ontotext are developing a VIAF reconciliation service, and here are some statistics about their dump 2018-07 (note: this dump is missing links, see below, so it may be missing entities too):

Type Count Comment
skos:Concept 54,741,718 Contributor (per-source) node, 1.7x per entity (cluster)
foaf:Document & ont:InformationResource 32,254,690 Entities (clusters), or RDF docs describing each cluster
schema:Person 21,034,181 persons
schema:Organization 4,359,408 organizations. Pure orgs (not places): 4,249,192
schema:Place 972,703 places
schema:Place & schema:Organization 110,216 Eg viaf:233665742 Chile: country and its government
schema:CreativeWork 4,110,185 creative work (uniform title)
bib:Agent 2,163,020 "placeholder URI in need of further matching"
wd:Q1387388 2,148,663 "chimera: mythological hybrid creature combining body parts of more than one real species. VIAF URI corrupted by an 'undifferentiate name' and should be treated as unusable"
pto:Pseudonym 4,639 A person with a pseudonym

Wikidata to VIAF Coreferencing[edit]

Number of WD entities with VIAF id, per class (2019-11).

  • 9.8k entity types have VIAF ids, the top 500 are shown below
  • WD has 6,215,688 humans, of which 1,571,460 are coreferenced. So about 5.2% (?) of VIAF persons are coreferenced, and 25.28% of WD persons are coreferenced. This is the same percentage as 2018-04, but the absolute number has increased about 50%
  • WD has 2,339,103 organizations, of which 327,694 are coreferenced, so 5% (?) of VIAF orgs are co-referenced, and 14% of WD orgs are coreferenced. This is a significant increase of the percentage in 2018-04 (8.4%), and the absolute number has increased 110%
  • I believe a similar percent of places are coreferenced: between 15 and 25%

--Vladimir Alexiev (talk) 08:23, 27 November 2019 (UTC)[reply]

Counts:

VIAF Source Codes[edit]

To do coreferencing work, we need to correlate VIAF source codes (eg JPG) to WD props (eg P245). We can find some of them with this query, and I found the others at the list of sources on http://viaf.org/. 

select ?entity ?entityLabel ?prop ?propLabel ?code ?countryLabel {
  ?prop wdt:P1552 wd:Q26921380.
  optional {
    ?prop wdt:P1629 ?entity
    optional {?entity wdt:P17 ?country}
  }
  optional {
    ?prop wdt:P3303 ?formatter 
    filter(regex(?formatter,"https://viaf.org/processed/(.*)\\|\\$1"))
    bind(replace(?formatter,"https://viaf.org/processed/(.*)\\|\\$1","$1") as ?code)
  }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
} order by ?propLabel

The sources are listed in the next section.

--Vladimir Alexiev (talk) 16:15, 22 August 2018 (UTC)[reply]

VIAF Links per Source[edit]

Counted with viaf-links-count.pl, see this gist. I also tried counting WD external-ids with SPARQL (.rq) but those queries timed out. HELP NEEDED.

2016-12 2018-07 2018-08 code entity property country
359497 366280 369712 B2Q Bibliothèque et Archives nationales du Québec (Q39628) P3280 Canada>Quebec
369985 369985 369987 BAV Vatican Library (Q213678) P1017 Vatican
286192 300088 BLBNB National Library of Brazil (Q948882) P4619 Brazil
104455 1795284 1797032 BIBSYS BIBSYS (Q4584301) P1015 Norway
365936 185314 370676 BNC Name and Title Authority File of Catalonia (Q8342938) P1273 Spain>Catalonia
412032 205976 412034 BNCHL Collective Catalog of Bibliographic Authorities of Chile (Q19896851) P7369 Chile (not P1890)
620739 642770 646521 BNE Biblioteca Nacional de España (Q750403) P950 Spain
4562204 2395213 4806862 BNF Bibliothèque nationale de France (Q193563) P268 France
29007 29007 29009 BNL National Library of Luxembourg (Q856651) P7028 Luxembourg
54935 79128 79991 CYT National Central Library (Q618340) P1048 Taiwan
111686 111686 DE663 Répertoire International des Sources Musicales (Q2178828) P5504 intl (RISM)
282952 162634 327462 DBC Danish Bibliographic Centre (Q12307383) P3846 Denmark
26490921 8866855 17785894 DNB Integrated Authority File (Q36578) P227 DE, AT, CH
46201 52350 52350 EGAXA Bibliotheca Alexandrina (Q501851) P1309 Egypt
57791 57794 ERRR National Library of Estonia (Q609471) P6394 Estonia
178835 179726 180164 FAST Faceted Application of Subject Terminology (Q3294867) P2163 intl (OCLC)
152003 165845 GeoNames GeoNames (Q830106) P1566 intl (places)
343166 173713 347426 ICCU National Library Service of Italy (Q576951) P396 Italy
9941 9964 IMAGINE !!?
7741093 8187394 8187405 ISNI International Standard Name Identifier (Q423048) P213 intl (authors)
241631 264833 264834 JPG Union List of Artist Names (Q2494649) P245 US Getty
240857 331791 331797 KRNLK National Library of Korea (Q495005) P5034 Korea
718263 726993 730007 LAC Library and Archives Canada (Q913250) P1670 Canada
16256543 10156417 16912675 LC Library of Congress Authorities (Q13219454) P244 US, UK, MX, ZA, NZ
186566 205260 205260 LNB National Library of Latvia (Q1133733) P1368 Latvia
14768 15876 15879 LNL Lebanese National Library (Q2901488) P7026 Lebanon
6299 8567 8569 MRBNR National Library of the Kingdom of Morocco (Q2901478) P7058 Morocco
227699 227651 227707 N6I National Library of Ireland (Q1672830) P1946 Ireland
1079274 1114415 1116232 NDL Web NDL Authorities (Q2553334) P349 Japan
1617979 1712963 1712980 NII CiNii (Q10726338) P271 Japan
817783 872953 879014 NKC Czech National Authority Database (Q13550863) P691 Czech Rep
1036908 1111676 1114335 NLA National Library of Australia (Q623578) P409 Australia
608 3940 3940 NLB National Library Singapore (Q890364) P3988 Singapore
850304 1788698 1810067 NLI National Library of Israel (Q188915) P949 Israel
1219045 1482467 1494282 NLP National Library of Poland (Q856423) P1695 Poland
245488 147268 294546 NLR Russian State Library (Q1048694) P7029 Russia (not P947)
508256 537830 539669 NSK National and University Library in Zagreb (Q631375) P1375 Croatia>Zagreb
67454 33727 67454 NSZL National Széchényi Library (Q1063819) P951 Hungary
2650671 2700697 2703554 NTA Koninklijke Bibliotheek (Q1526131) P1006 Netherlands
3090182 1731205 3482894 NUKAT NUKAT (Q11789729) P1207 Poland>WarsawU
7389 17359 ORCID ORCID iD (Q51044) P496 intl (researchers)
1228 1228 1228 PERSEUS Perseus Digital Library (Q639661) P7041 intl (ancient places)
404594 423917 423920 PTBNP National Library of Portugal (Q245966) P1005 Portugal
237240 253180 253914 RERO Library Network of Western Switzerland (Q3456783) P3065 Switzerland
205630 215367 215798 SELIBR LIBRIS (Q1798125) P906 Sweden
209 209 209 SRP syriaca.org (Q64866015) P6934 Syriac Reference Portal
2927879 3281884 3300486 SUDOC IdRef (Q47757534) P269 France academic
75989 124465 124466 SWNL Integrated Authority File (Q36578) P227 Switzerland (was SZ; part of GND)
85956 85958 UIY National and University Library of Iceland (Q627423) P7039 Iceland (NULI)
8768 10452 10452 VLACC OpenVlacc (Q24247813) P7024 Flemish Public Libraries
125635 126826 W2Z National Library of Norway (Q924551) P1015 Norway
1286035 1570365 WKP Wikidata entities
6272821 7105991 Wikipedia Wikipedia links
528 1102 1140 XA Extended Authorities n/a OCLC manual
2069996 2026405 2036656 XR Extended Relationships n/a OCLC manual

--Vladimir Alexiev (talk) 10:13, 24 August 2018 (UTC)[reply]

VIAF Help Needed[edit]

Vladimir Alexiev Jonathan Groß Andy Mabbett Jneubert Sic19 Wikidelo ArthurPSmith PKM Ettorerizza Fuzheado Daniel Mietchen Iwan.Aucamp Epìdosis Sotho Tal Ker Bargioni Carlobia Pablo Busatto Matlin Msuicat Uomovariabile Silva Selva 1-Byte Alessandra.Moi CamelCaseNick Songceci moz AhavaCohen Kolja21 RShigapov Jason.nlw MasterRus21thCentury Newt713 Pierre Tribhou Powerek38 Ahatd JordanTimothyJames Silviafanti Back ache AfricanLibrarian M.roszkowski Rhagfyr 沈澄心 MrBenjo S.v.Mering

Notified participants of WikiProject Authority control

Ash Crow
Dereckson
Harmonia Amanda
Hsarrazin
Jura
Чаховіч Уладзіслаў
Joxemai
Place Clichy
Branthecan
Azertus
Jon Harald Søby
PKM
Pmt
Sight Contamination
MaksOttoVonStirlitz
BeatrixBelibaste
Moebeus
Dcflyer
Looniverse
Aya Reyad
Infovarius
Tris T7
Klaas 'Z4us' van B. V
Deborahjay
Bruno Biondi
ZI Jony
Laddo
Da Dapper Don
Data Gamer
Luca favorido
The Sir of Data Analytics
Skim
E4024
JhowieNitnek
Envlh
Susanna Giaccai
Epìdosis
Aluxosm
Dnshitobu
Ruky Wunpini
Balû
★Trekker

Notified participants of WikiProject Names

User:Zolo
Jane023 (talk) 08:50, 30 May 2013 (UTC)[reply]
User:Vincent Steenberg
User:Kippelboy
User:Shonagon
Marsupium (talk) 13:46, 18 October 2013 (UTC)[reply]
GautierPoupeau (talk) 16:55, 9 January 2014 (UTC)[reply]
Multichill (talk) 19:13, 8 July 2014 (UTC)[reply]
Susannaanas (talk) 11:32, 12 August 2014 (UTC) I want to synchronize the handling of maps with this initiative[reply]
Mushroom (talk) 00:10, 24 August 2014 (UTC)[reply]
Jheald (talk) 17:09, 9 September 2014 (UTC)[reply]
Spinster (talk) 15:16, 12 September 2014 (UTC)[reply]
PKM (talk) 21:16, 8 October 2014 (UTC)[reply]
Vladimir Alexiev (talk) 17:12, 7 January 2015‎ (UTC)[reply]
Sic19 (talk) 21:12, 19 February 2016 (UTC)[reply]
Wittylama (talk) 13:13, 22 February 2017 (UTC)[reply]
Armineaghayan (talk) 08:40, 10 March 2017 (UTC)[reply]
Musedata102 (talk) 20:27, 26 November 2019 (UTC)[reply]
Hannolans (talk) 18:36, 16 April 2017 (UTC)[reply]
User:Martingggg
Zeroth (talk) 02:21, 4 June 2018 (UTC)[reply]
User:7samurais
User:mrtngrsbch
User:Buccalon
Infopetal (talk) 17:54, 9 August 2019 (UTC)[reply]
Karinanw (talk) 16:38, 24 March 2020‎ (UTC)[reply]
Ahc84 (talk) 17:38, 26 August 2020 (UTC)[reply]
User:BeatrixBelibaste
Valeriummaximum
Bitofdust (talk) 22:52, 26 March 2021 (UTC)[reply]
Mathieu Kappler
Zblace (talk) 07:22, 24 December 2021 (UTC)[reply]
Oursana (talk) 13:16, 17 May 2022 (UTC)[reply]
Ham II (talk) 08:30, 25 January 2024 (UTC)[reply]
DaxServer (talk) 16:00, 24 May 2024 (UTC)[reply]

Notified participants of WikiProject Visual arts WikiProject Cultural heritage has more than 50 participants and couldn't be pinged. Please post on the WikiProject's talk page instead.


  • If you sort by col "property" and look for "!!!", this represents VIAF sources that don't have corresponding WD properties (which should be marked as VIAF component (Q26921380). Wikidata:Property_proposal/Authority_control#missing_VIAF_components suggests to create them.
  • Please help me verify props in the table above marked "?" or cases where multiple props are listed (although VIAF has both persons and works, maybe from a certain source it takes only persons)
  • Do you think we should expand the Qnnn and Pnnn above using the respective macros, so they show up with their names?
  • Do you think we should record col "code" in WD in some way? A few of these are available as third-party formatter URL (P3303), but most aren't. It took me a few hours to correlate all VIAF-component sources against WD, and it wasn't fun. Guess we can add it as qualifier? Eg
IdRef ID (P269) has characteristic (P1552) VIAF component (Q26921380)
code (P3295) "SUDOC"
  • And most importantly, help me with "id proliferation" as described below. Please comment on that section!

--Vladimir Alexiev (talk) 08:25, 23 August 2018 (UTC)[reply]

Resolve Against VIAF Links[edit]

Wikidata authority IDs are obtained from various places, and hand-edited. But http://viaf.org/viaf/data/viaf-20150115-links.txt.gz has links to IDs of VIAF participants (constituent national libraries) and Wikipedia.

Stats:

  • 2Gb unzipped size
  • 27684634 subjects (27.7M)
  • 1.67 links per subject
  • 46248396 links, of which 377650 enwiki links.
  • Link breakdown:
  27684634 VIAF    total subjects
  10531522 DNB     Germany
   9154093 LC      LC (NACO)
   7655649 ISNI    ISNI
   2555033 NTA     Netherlands
   2508374 SUDOC   France (Sudoc)
   2036493 BNF     France (BnF)
   2018647 XR      xR OCLC file
   1351105 NUKAT   Poland (NUKAT)
   1032862 NDL     Japan (NDL)
   1016708 NLA     Australia
    844024 NLP     Poland (Nat lib)
    743215 NKC     Czech
    689827 LAC     Canada
    570840 NLI     Israel
    562244 BNE     Spain
    473518 NSK     Croatia
    377650 WKP     Wikipedia
    373078 PTBNP   Portugal
    320898 BAV     Vatican
    232327 JPG     Getty (ULAN)
    220304 RERO    Swiss (RERO)
    187073 SELIBR  Sweden
    169028 ICCU    Italy
    158515 LNB     Latvia
    144299 BNC     Catalunya
    101500 DBC     Denmark (DBC)
     73421 BIBSYS  Norway
     45633 SWNL    Swiss (Nat lib)
     37004 EGAXA   Egypt
     33727 NSZL    Hungary
     11000 LNL     Lebanon
      9953 IMAGINE
      5723 VLACC   Belgium (Flemish)
      1228 PERSEUS Perseus
       997 RSL     Russia
       408 NLB     Singapore
       267 XA      xA OCLC file
       209 SRP     Syriac

Now let's count some IDs in Wikidata, using WDQ API and working down the list:

  • VIAF: 504736
    • would be nice to cross-check this against the 377650 enwiki links
    • Wikidata has 13M items and VIAF has 27.7M subjects, so I would expect at least 3-4M common subjects. This means that we have co-referencing for only 15% of the possible items! A lot of work remains
  • GND: 335883
    • 37% of all VIAF items have GND id, while in Wikidata the ratio is 66%. This means that GND co-referencing is more advanced than VIAF co-referencing
  • VIAF or GND: 567240
    • Everything in GND has a VIAF ID, but this shows that in Wikidata, 62504 with GND id don't have a VIAF id. We can assign VIAF id to these easily!
  • LCNAF: 210845
    • VIAF or LCNAF: 506136
    • Again, we can leverage LCNAF ids to assign 1400 VIAF ids easily
  • RKDartists: 21760: most of these can be added to VIAF for free!
  • NTA: 335883
  • SUDOC: 103120

More importantly: we could cross-check all 36 ID's in VIAF against Wikidata to:

  • add the missing ones,
  • flag different ones with a qualifier "questionable".
unsigned comment by user:Vladimir Alexiev 16:33, 23 January 2015‎
I imported the NTA stuff some time ago. Could easily do the same for other properties, but I don't know how good the data is and if I need to do some transformations. Please start a wikiproject. Would love to comment over there and get everything imported. Multichill (talk) 21:20, 26 January 2015 (UTC)[reply]
User:Multichill Thank you sir/lady, will do. Chill :-)
Some remarks:
  • VIAF contains "name authorities" (historically persons first, later then corporate bodies, finally "geographic" items, works and "expressions" too. However "Subject Headings" ("topical terms" as in LCSH against LCNAF) are not part of VIAF, although some of the authority files (like GND) include them. Thus "X completely included in VIAF" does not hold for all constituent files of VIAF.
  • Please be careful with respect to the ODC-BY license which reigns the VIAF dumps: Whereas the constituent authority files (GND for sure) are CC-0, VIAF definitely is not. (My interpretation: Import on a case-by-case basis is o.k., everything more must make certain that the license is met. Thus it fr.wikipedia would present SUDOC numbers fetched from Wikidata which here would have been collected by bulk-matching existing GND numbers against VIAF, french wikipedia would be obliged to present VIAF attribution or VIAF identifiers next to the SUDOC numbers.
  • OCLC some years ago made a matching between VIAF and en.wikipedia.org and afterwards donated the data such that VIAF numbers could be imported into English Wikipedia (afterwards some cross-check against the VIAF numbers in corresponding articles of de.wikipedia was performed) from where they moved on to wikidata. I doubt that they ever repeated the matching and believe that "wikipedia" linking in VIAF still reflects the original matching. Therefore in a sense there is no need to compare wikidata to VIAF again. Unless of course this can magically restricted to VIAF numbers which never appeared in en.wikipedia. -- Gymel (talk) 12:58, 27 January 2015 (UTC)[reply]

New VIAF Situation[edit]

Have you read this?

Yes, this is a very nice announcement since it means that VIAF will source Wikidata actively, which hopefully will close the gap between the two. I think this makes it even more important to leverage VIAF and other authority IDs in Wikidata. --Vladimir Alexiev (talk) 09:57, 29 May 2015 (UTC)[reply]

Update Mar 2017: 

select (count(*) as ?c) {
  ?x wdt:P214 ?viaf
  filter exists {?x wdt:P31 wd:Q5}
}
Try it!

From Wikipedia to Wikidata Link[edit]

The 2019-01-07 dump of Viaf links contains 1,519,171 unique Viaf IDs that have at least one link to a Wikipedia page (information extracted with gunzip -kc viaf-20190107-links.txt.gz | awk -F '\t' '/wikipedia/ {print $1}' | sort | uniq -c | wc -l ). However, Wikidata has currently only 1,352,478 items with a VIAF property. We could therefore add 166,693 Viaf IDs relatively easily - unless they are largely related to Wikipedia pages without equivalent in Wikidata, which would be interesting. Of course, it is still necessary to figure out which ones are already in Wikidata. What's the best workflow for this kind of verification? Ettorerizza (talk) 01:39, 14 January 2019 (UTC)[reply]

ID Proliferation with VIAF[edit]

See the low WD coreference percentages in #VIAF Volumetrics. I believe the situation is not much different for all VIAF-contributing sources.

What can we do to increase them? There is a simple way: proliferate IDs between all of these authorities.

--Vladimir Alexiev (talk) 16:15, 22 August 2018 (UTC)[reply]

@Vladimir Alexiev: I get a "query is malformed" error on your ULAN without VIAF... - PKM (talk) 21:01, 22 August 2018 (UTC)[reply]
@PKM: fixed. And somehow the number now is lower?? --Vladimir Alexiev (talk) 08:03, 23 August 2018 (UTC)[reply]
@Vladimir Alexiev: Thanks. Maybe someone jumped in to fix some? It wasn’t me (yet). - PKM (talk) 19:15, 23 August 2018 (UTC)[reply]

VIAF members[edit]

#All properties representing VIAF members
SELECT ?id ?idLabel ?cod
WHERE {
  ?id wdt:P31 wd:Q55586529 ;
      p:P1552 [ ps:P1552 wd:Q26921380; pq:P3295 ?cod ] .
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
ORDER BY ?cod
Try it!

Swedish National Library moves direction Linked data[edit]

Monday june 11 the Swedish National Library is moving in direction Linked data and release a new system called LIBRIS XL ==>

  1. We have a new identifier LIBRIS-URI Libris-URI (P5587) in Wikidata

Open issues are if we need to "upgraded" to support the new identifier in other systems. I have found

  1. User:Magnus_Manske/authority_control.js
    1. tool to read from VIAF.org and populate WD
  2. User:Magnus_Manske/Mix'n'match_date_import
  3. wdmapper

See also:

- Salgo60 (talk) 14:07, 10 June 2018 (UTC)[reply]

Status update 16 September 2018
A project wmse-library-data-2018 done by Alicia_Fagerving_(WMSE) is converting and adding about 60 000 identifiers to Wikidata....
I sent in today a question to VIAF asking about plans they have to start supporting Libris-URI (P5587) - Salgo60 (talk) 06:54, 16 September 2018 (UTC)[reply]
Status update 28 September 2019
  • Still no one has specified how LIBRIS <-> VIAF should work together see Phabricator T223259
  • We lack a change process for errors in VIAF and LIBRIS see example of error Talk:Q21522286#Summary
- Salgo60 (talk) 18:32, 28 September 2019 (UTC)[reply]

Introducing mapping relation type (P4390) for full SKOS mappings[edit]

For knowledge organization systems which do not deal with clearly defined entities (such as human (Q5)), you often find in-exact relations between the external entity and matching Wikidata items. E.g., the STW concept Yugoslavia (until 1990) is not an exact match to Yugoslavia (Q36704), which is described as "1918–1992 country in Southeastern and Central Europe". For STW's Executive selection, Assessment center (Q265558) is closely related, but categorically different (process vs. instrument). While in the latter case it might be useful to create an exactly matching item, in the former it clearly would be not.

Within the domain of traditional knowledge organization system, different mapping relations (such as "close match") have been used to cover such situations. The "external id" properties of Wikidata lacked such expressiveness, till the mapping relation type (P4390) property was introduced. The property, to be used exclusively as a qualifier on external-id properties, allows to more precisely define relations as exact match (Q39893449), close match (Q39893184), broad match (Q39894595), narrow match (Q39893967) or related match (Q39894604). These relation types reflect the according SKOS mapping properties.

Since its introduction in October last year, the property has seen some uptake, particularly in the biomedial field. The ongoing mapping of STW Thesaurus for Economics (Q26903352) to Wikidata is based on qualified relations (state of the mapping, shown as SKOS). The effort aims at creating a finally complete mapping of the STW descriptors.

Workflow for the mapping process[edit]

Our - still experimental - workflow for the STW mapping is as follows:

  1. Use Mix-n-match catalogs for each sub-thesaurus (#507, #1259/#1260) to assign STW descriptor IDs one-by-one to Wikidata items.
    This sounds simple, yet often reveals qualitiy issues. Some of these, such as obvious duplicates in Wikidata, can be resolved immediately. Sometimes an ugly mess shows up, which can only be reported to be solved by the community later on. Of course, quality flaws in STW might be revealed also. During this step, it may also be advisable to take notes about other items, which are not the closest ones and not linked via external-id entry in Wikidata, but may be worth linking from the side of the external vocabulary.
    • For non-exact relationships, immediately open the newly linked item and manually qualify STW Thesaurus for Economics ID (P3911).
    • For STW descriptors which are lacking a counterpart in Wikidata and would make sense there, add an item semi-automatically, with exact match (Q39893449) to the STW ID and all avaialable information from the thesaurus (more on that later).
  2. Assign exact match (Q39893449) to the remaining unqualified STW Thesaurus for Economics ID (P3911) via Quickstatements, with the input produced by a script executing a SPARQL query. This step can be executed multiple times during the mapping process for one sub-thesaurus, in order to keep the list of unqualified entries short.

The sequence within the Mix-n-match input file turned out to be crucial for a smooth one-by-one workflow. We sorted the generated M-n-m input by the minimal notation of attached subject categories for the descriptors, and within that alphabetically.

Quality control for mappings using mapping relation types[edit]

Maintenance and qualitiy control on a mapping have to take into account that multiple external-id values for a Wikidata item, or one external-id linked to multiple Wikidata items are possible and may perfectly make sense with in-exact mapping relations (e.g., STWs Appenzell is a "broad match" to Appenzell Ausserrhoden (Q12079) and Appenzell Innerrhoden (Q12094)). This is not reflected in the "single value" and "distinct values" constraints. Therefore, we defined a number of QS reports for the STW mapping to catch anomalies specifically in qualified mappings.

Feedback welcome[edit]

We are interested to exchange experiences with others who are mapping KOS to Wikidata, possibly with different workflows or other tools used. The reports and scripts linked above are meant to be customizable, we'd be happy to receive suggestions for improvment or github pull requests. -- Jneubert (talk) 15:09, 29 August 2018 (UTC)[reply]

Akuckartz (talk) 07:29, 9 August 2020 (UTC) JakobVoss (talk) Vladimir Alexiev (talk) 09:27, 5 July 2016 (UTC) TomT0m author  TomT0m / talk page 20:15, 21 September 2016 (UTC) Jneubert (talk) 09:22, 29 January 2018 (UTC) Ettorerizza (talk) 11:00, 26 September 2018 (UTC) ElanHR (talk) 10:01, 17 March 2019 (UTC) Dcflyer (talk) 19:46, 29 March 2019 (UTC) ArthurPSmith (talk) 16:32, 1 September 2022 (UTC) Melmakko (talk) 17:34, 4 November 2022 (UTC) Maxime[reply]
Notified participants of WikiProject KOS

Item creation from a thesaurus concept via Quickstatements[edit]

During the above mentioned mapping process from STW Thesaurus for Economics (Q26903352), we sometimes want to create new items in Wikidata. With the "New item" button in Mix-n-match, only rudimentary information can be transferred to the new item. Therefore we generated a list of all not-yet-mapped STW descriptors, formatted for Quickstatements input. It includes labels and aliases (skos:preLabel/skos:altLabel) in all available languages, as well an instance-of economic concept (Q29028649) statement, sourced from the STW descriptor, optionally a link to the according GND concept, derived from the STW/GND mapping, rarely a description (skos:scopeNote), but always of course a STW Thesaurus for Economics ID (P3911) link.

The workflow is - during working through a mix-n-match list - for any missing concept simply to copy & paste the complete set of QS statements into the QS input window, removing aliases which are not appropriate for Wikidata (such as "oil platform" for "offshore industry"), and running the statements. The list is sorted exactly like the mix-n-match list and recreated every hour, so the same items are on top of both lists, and every case solved by either linking or creating an item disappears automatically from the list. In our experiences, this works quite smoothly.

If others want to adapt such a workflow for other vocabularies - thanks to SKOS and LOD standards that shouldn't be too difficult -, here is the script for generating the list, and the query called with it. -- Feedback, as always, welcome. Jneubert (talk) 12:11, 5 September 2018 (UTC)[reply]

History[edit]

Please add here references, blogs etc on the topic.

https://twitter.com/hashtag/coreferencing: tweet using tag #coreferencing. Tweets on involving Getty, British Museum thesauri, some fancy shots...

Retrieved from "https://www.wikidata.org/w/index.php?title=Wikidata:WikiProject_Authority_control/Archive&oldid=1276350131"