Wikidata:Requests for comment/Sourcing requirements for bots

From Wikidata
Jump to navigation Jump to search
An editor has requested the community to provide input on "Sourcing requirements for bots" via the Requests for comment (RFC) process. This is the discussion page regarding the issue.

If you have an opinion regarding this issue, feel free to comment below. Thank you!


THIS RFC IS CLOSED. Please do NOT vote nor add comments.

With the newly adopted guideline Help:Sources, it is time to clarify the sourcing requirements for bots and in particular for bots that are importing data from Wikipedia. I propose a very simple rule based on the guideline's list of criteria for statements that don't require a reference. Bots should only import unreferenced statements from Wikipedia if these statements match one of these exceptions. I believe these restrictions are important for the following reasons.

Main proposal[edit]

The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.

As a whole, the idea of requiring sources is rejected. Sources are still recommended though, and bots should try to use better/reliable sources when possible. Legoktm (talk) 06:04, 23 August 2013 (UTC)[reply]
  1. Statements should be referenced This is what the new guideline Help:Sources recommends and Wikidata:Introduction explicitly states that Wikidata will "support the notion of verifiability". There's no reason to have an exemption for bots.
  2. On Wikidata, bots set the example Bots have the capability of adding millions of statements in a matter of weeks. Anyone who's sifted through the current Wikidata knows that at least 90% (and probably much more) of the current statements have been added by bots. If we allow bots to add millions of unreferenced statements, it will be impossible to ask human users to add sources to their statements when all they see are unreferenced statements. It will also be impossible to argue that Wikidata is serious about sourcing.
  3. Wikipedias will turn away from Wikidata if Wikidata turns away from sourcing. All Wikipedias have increasingly strict sourcing policies. If Wikidata is careless about sourcing, its data can't reasonably be included automatically in infoboxes.
  4. Blind data imports from Wikipedias can create an echo chamber of dubious data. en.wiki, de.wiki and fr.wiki (to cite only those three) specifically warn against using other Wikipedias or Wikipedia mirrors as sources. Blind imports from Wikipedia do exactly that and create a mechanism for disseminating dubious data and mistakes from one Wikipedia to the next.
  5. Sourcing requirements make life more difficult for bots but it does not make their life impossible. Many basic statements do not require a source and can be imported by bots. These form the backbone of Wikidata and there's still a vast amount to import. Moreover, bots can import statements from reliable sources, including specialized databases. This is more complicated because specialized databases have a limited scope and because information might be more complicated to harvest. You won't be able to get 50 thousand statements from 50 lines of code but you will get properly referenced data. There are already bots planning to do this.

Pichpich (talk) 19:38, 4 July 2013 (UTC)[reply]

  • I'm not sure I understand the point. Do you really think there should be no guidelines whatsoever on what bots are allowed to do? For instance, would you support the complete elimination of Wikidata:Requests for permissions/Bot? Perhaps I misunderstand your comment and what you mean is that verifiability policies on Wikipedia are a bad idea. I guess you're free to think that but they are widely accepted policies on every Wikipedia I know and rejecting sourcing on Wikidata will quickly make interaction with Wikipedia impossible. In any case, I'd appreciate if you can expand on your rationale. Thanks. Pichpich (talk) 21:59, 4 July 2013 (UTC)[reply]
  •  Support makes sense. --Stryn (talk) 21:03, 4 July 2013 (UTC)[reply]
  •  Support everthing which is needed to identify the concept of an item can be imported as this is a prerequisite for referencing. Everthing which is not needed to identify an item must be omitted. We should search for reliable sources instead of squeezing "in most cases true" claims out of wikipedias.  — Felix Reimann (talk) 21:14, 4 July 2013 (UTC)[reply]
  •  Comment – Identify the threshold. I think that once Wikipedias start sucking infobox information from Wikidata, this will be a gamechanger for us: Wikipedians will come here and add/challenge the sources, which they would otherwise add to the Wikipedia page in question. With no data, this will never happen.
Wikipedia is a good description of source – it is honestly admitting being a weak source. Surely "is a person" or "is a female" can be reffed by Wikipedia? What about date of birth?
"Unsourced (or weakly sourced) claims me be challenged and removed" seems to provide a sensible rule, by which Wikipedias seem to work, although "no untrivial unsourced (or weakly sourced) claims" would be ideal. (We can also start looking for sourced info for bots to add from Wikipedias, like country populations.) Littledogboy (talk) 12:31, 5 July 2013 (UTC)[reply]
  •  Oppose. Three cases :
    • There is information on Wikipedia, with source : it will be possible to import the source ; if it is not, a human could transfer the source later, it will not disapear because we put it in Wikidata.
    • There is information in Wikipedia, unsourced : if we do not import it, the information will remain, unsourced, on the infobox. On the other hand, we import it : it will be shown on any Wikipedia with the infobox, and it will be shown in such a way that it is clear it is unsourced, thus making an incentive to add manually a source. Other good thing : this incentive will be shown in any language ... so it will touch all users, whereas if it stays in one wikipedia, other will simply show nothing.
    • There is no information in Wikipedias : we can do nothing automatically
     Comment The problem is not to import or not to import data, the data will be there wether or not we import them. The problem is to find a good incentive for people to source them, whether they are on Wikipedia or Wikidata. For datas that are not supposed to be shown in any Wikipedia this is another problem though TomT0m (talk) 14:48, 5 July 2013 (UTC)[reply]
    The problem with this argument is that I seriously doubt that Wikipedias will accept to display data which Wikidata has imported from unreliable sources. In fact, we have already seen these concerns expressed explicitly on local wikis and a few Wikidata newbies have questioned why Wikidata includes data from unreliable sources. It's also misleading to claim that the unsourced data (and potentially bad data, vandalized data and so on) will remain in the infobox whether we import or not because Wikidata has the potential of taking bad data from one wiki and turning it into bad data on 286 wikis which doesn't sound like progress to me. Pichpich (talk) 17:46, 6 July 2013 (UTC)[reply]
    One thing to do : with lua, we can display informations that are just imported of Wikidata with something like a reference needed added, just as any sentence in Wikipedias. This will also leave the choice for those individual wikipedia to not display data from Wikidata in those cases. Reference needed is a strong incentive to add a source without losing the information that it has currently no source. In those cases, they could display the information that might? have or not a source in the Wikipedia, just as before, or generate a list infobox that have sources here not imported in wikipedia. It's not a question of wether or not import as I see things, It's a question on how we encourage people to add source. A reference needed displayed in 286 wikis is better than no information at all. And a question for infoboxes coders. TomT0m (talk) 15:22, 16 July 2013 (UTC)[reply]
    This is not realistic. There is absolutely no way that you will convince any Wikipedia where there are strong sourcing policies (including at least the three largest: en.wiki, de.wiki and fr.wiki) to include unsourced data from Wikidata with a "citation needed" template. Just try and start a thread on that topic at fr:Discussion Wikipédia:Vérifiabilité or en:Wikipedia talk:Verifiability if you don't believe me. I'm certain that the reaction will be a) "if your data is no good, then keep your data to yourself" and b) "don't pollute our wiki with citation needed templates to solve your problems". Pichpich (talk) 20:58, 18 July 2013 (UTC)[reply]
    Then they can as well choose not to show those informations. It's just one or two lines of code if the infoboxes code is done well. And two more lines (in the whole template codebase) to filter the imported from wikipedia statements before checking if there is references left. Plus there is more beautiful and less polluting ways to do the same thing (the reference needed template), like a different color and a link to point to the Wikidata statement. Am I the only one to understand that this is perfectly realistic ? In the meantime, those Wikipedias who will choose this solution will synergise to find a source. TomT0m (talk) 21:38, 18 July 2013 (UTC)[reply]
    The problem is not that this would be hard to implement technically but that it would be soundly rejected on principle. There are explicit guidelines, even policies, against using de.wiki to reference a claim on fr.wiki so it's very hard to imagine that people on fr.wiki would find it more acceptable to include unreferenced claims copied blindly by a bot from de.wiki to Wikidata. You may think this sort of resistance is a mistake but that resistance is there and I don't see how you're going to change it. And if Wikipedias refuse to display poorly referenced data from Wikidata, your plan disintegrates. Pichpich (talk) 02:36, 19 July 2013 (UTC)[reply]
    Do you take only one example (dewiki) which is as far as I know one of the most restrictive, there is a technical solution easy to present and to implement that entails absolutely no changes in their guidelines while keeping advantages of Wikidata anyway, a solution that is easy to explain, and you conclude that we should not even try to explain it in local Wikipedias ? My experience is slightky different. Wikipedians can be very reasonable people who listen to arguments even if they don't agree, and sometimes change their minds. And don't forget the smaller wikipedia who probably struggle for datas. TomT0m (talk) 08:58, 19 July 2013 (UTC)[reply]
    I'm using en.wiki, fr.wiki and de.wiki because they are the three I'm familiar with and the three whose policies and guidelines I can read on my own. Over the last few years, all three have seen strong culture shifts towards stronger sourcing policies and practices but my understanding is that this shift has occurred across all the main wikis and it would be a mistake to ignore that trend. For instance, you can see that concerns with sourcing and bot imports of unsourced data were cited pretty quickly at Wikipedia:Requests for comment/Wikidata Phase 2. Pichpich (talk) 04:42, 20 July 2013 (UTC)[reply]
    So to say, just Wikipedia which are already really rich of datas. But actually most of the infoboxes I see in french Wikipedia (for example) do not have a source, partly because it's a very low granularity sourcing policy, every statement. Your policy implies that we simply cannot use these datas to put them into a not that data rich smaller Wikipedia through Wikidata. I also not that you are talking of mass imports of external sources to Wikipedia by bots, which already have high sourcing standards by alternate policy proposal. I really don't see where your problem is :) TomT0m (talk) 10:25, 20 July 2013 (UTC)[reply]
    Obviously, I only speak a limited number of languages but GoogleTranslate is still able to confirm that there are explicit warnings against using other Wikipedias as sources in smaller Wikipedias as well: I checked af.wiki, gl.wiki, fi.wiki for instance. My problem is that your proposal not only chooses to ignore these warnings but also unrealistically expects local Wikipedias to be ok with that. On the other hand, my proposal respects these local policies (and the constraints of Help:Sources) and seeks to maximize the reliability and verifiability of the data passed on to local Wikipedias big and small. Just because someone is hungry doesn't mean you should feed him shit. Pichpich (talk) 19:36, 20 July 2013 (UTC)[reply]
    Nobody talked about seeing Wikipedia as valid sources. TomT0m (talk) 19:55, 20 July 2013 (UTC)[reply]
  •  Support. Three cases:
    • There is information on Wikipedia, with a properly formatted source: a bot can and should import with source.
    • There is information on Wikipedia without source, or with a source the bot can't parse: the bot should refrain from import, and leave it to non-artificial intelligence.
    • There is no information in Wikipedias : we can do nothing automatically.

In no case is there reason to import without source by bot. Lsj (talk) 17:04, 5 July 2013 (UTC)[reply]

  •  Support, per Lsj, although I think bots should preferably consult reliable databases not Wikipedia (and especially refrain from using categories on Wikipedia). --HHill (talk) 07:23, 6 August 2013 (UTC)[reply]
  •  Oppose There are many different applications: <imported from Wikimedia project (P143)> English Wikipedia (Q328) is enough for some of them and does not enough for others (for example for Wikipedia`s infobox). Every application can decide that source set is enough for it. For example infobox can check "if (there is at least one non-p143 source) then <use data> else <ignore data>". — Ivan A. Krestinin (talk) 19:53, 6 August 2013 (UTC)[reply]
  •  Oppose This proposal and the next; see below for my explanation.
  •  Oppose Wikidata is greate for bots but not with this. In the future there will be WD:Rank there I would say a claim needs a source to be in the top rank but these could be added later by another bot or by a user. I think Wikipedias especially de.wiki, en.wiki and fr.wiki will only allow those rank# Oppose ed claims for automated use of data. We could have a RFC later for this to state that only sourced Claims are allowed to have this high rank. And maybe even exclude <imported from Wikimedia project (P143)>: "some Wikipedia" for this high rank. --Sk!d (talk) 20:24, 20 August 2013 (UTC)[reply]
  •  Strong support We need a new kind of intelligent bots. --Succu (talk) 20:35, 20 August 2013 (UTC)[reply]
  •  Oppose when a bot operator is stupid, all his work will be reverted.. however, we do need many more factoids before Wikidata has enough weight. Sources are great and many sources are at fault. What we need is the ability to compare sources for facts.. GerardM (talk) 12:35, 21 August 2013 (UTC)[reply]
I blogged about this subject and explain why it is a bad idea .. Wikidata needs more data to be useful
  •  Oppose. Bots should be allowed to import birth and death dates from Wikipedias. Ayack (talk) 13:50, 21 August 2013 (UTC) be useful][reply]
  •  Oppose: totally unwiki and a way to reduce fact checking. Wikidata is here to manage the data and make it better, not to keep the data out of it and manage it somewhere else. If we want better sourcing for the statements we're making on our projects, Wikidata is the place where to move them and improve them in a coordinated way. [1] [2] --Nemo 14:54, 21 August 2013 (UTC)[reply]
  •  Oppose: As an aspiration for the future, it's fine, but for now there is no way for a bot to manage under these conditions for the Wikimedia projects. A specialised bot harvesting data from a specialised database could do so but most data is not held in easily machine-readable format. That's part of the point of creating Wikidata in the first place. As with every other project in the family, we will have to have some element of trust balanced by prudent scepticism (at least for now). - AdamBMorgan (talk) 17:27, 21 August 2013 (UTC)[reply]
  •  Oppose: (as community member) I would prefer not to have too hard rules too early. Also, an unreferenced statement is obviously unreferenced, and a consumer - e.g. a Wikipedia - can choose to discard unreferenced statements in Lua. I think we will tend toward stronger rules for sourcing over time, but we first need to understand what works and what doesn't. The project is not even a year old yet, let us be careful not to stifle it. This is no carte blanche, though: for BLPs I would have a different opinion. But in the general case... --Denny (talk) 13:03, 22 August 2013 (UTC)[reply]
The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.

Alternative Proposal[edit]

The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.

There is no consensus to implement this (leaning against). Legoktm (talk) 06:09, 23 August 2013 (UTC)[reply]
Alternative Proposal
  • Bots should add information about from where the data comes to every statement they add.
  • Bots should add information about when they imported data from online ressources which may change.
  • Bots should add sources always, when it is possible.
  • Imports from Wikipedia are allowed to create basic data to be extended by humans.
  •  Oppose per TomT0m. I proposed a different solution (above) which will help Wikidata to become interresting for Wikipedians because they can set easily sources to existing unsourced statements.
    @Pippich It would be great if you would add sources to the statements you add. You fight the 'evil' bots, but you don't add sources yourself, there are sources which can be already set using statements. --Pyfisch (talk) 10:58, 6 July 2013 (UTC)[reply]
    Show me a non-trivial statement that I added without a source and I'll gladly reference it. Let me stress once again that the sourcing requirements only apply to statements that are not common knowledge or supported by the item itself. Pichpich (talk) 19:16, 6 July 2013 (UTC)[reply]
    Also, I would appreciate if you can refrain saying that I'm "fighting 'evil' bots". That's completely dishonest as I'm sure you know. I think bots have done a great job and am all in favor of their continued use, I've made constructive suggestions to bot proposals, I've proposed tasks for new bots. Pichpich (talk) 21:27, 6 July 2013 (UTC)[reply]
    It is nice to know that you appreciate the work of the bots, but I don't understand is that if you think in this way, why you always want to stop the bots which import statements from Wikipedia, without references. Because this group of bots is the majority of bots, as far I know and you tried multiple times said that you want to stop them and started discussions about this topic(Wikidata talk:Bots#More precise requirements for statement-adding bots, Wikidata:Requests for comment/References and sources#Pyfischs comments end of section), I think it is correct that you fight the statement adding bots. We all know that these bots are not evil, but for me it looks like as you think that they are evil. So I don't see a need to refrain it. --Pyfisch (talk) 16:23, 7 July 2013 (UTC)[reply]
    To John Hartford (Q704700) you added the statement occupation (P106) → singer-songwriter (Q488205) is no common knowledge and you also can't say that it is stated by the item itself, this only applies for works as far I understand. If this would be wrong you can't oppose with your argumentation against adding birth and death dates by bot (Wikidata:Requests for permissions/Bot/SamoaBot 26) --Pyfisch (talk) 16:23, 7 July 2013 (UTC)[reply]
    Please don't distort my record. I do not oppose bots adding statements: I oppose bots adding unreferenced statements when these statements require a reference. I have been supportive of bots that add statements that don't require a source (such as Wikidata:Requests for permissions/Bot/SamoaBot 33) and I've been supportive of bots that add non-trivial statements when they do so based on a reliable source (such as Wikidata:Requests for permissions/Bot/SamoaBot 32). My proposal above explicitly makes room for bots in both of these categories. As for the case of John Hartford, this is a perfect example of "common knowledge" or as Felix Reimann puts it above statements "needed to identify the concept of an item". Anyone with a passing knowledge of Hartford knows that he's a singer-songwriter and in fact that's the only reason why he has a Wikipedia article. Common knowledge is not what is literally known to everybody as it is not hard to find people who don't know that the sun is a star, people who don't know that John Lennon is a member of The Beatles and people who don't know that Brumaire is a month of the French Republican Calendar. But all these statements are still common knowledge as nobody in their right mind would challenge them or tag the corresponding sentence with a citation needed tag on Wikipedia. On the other hand only a ridiculously small percentage of people familiar with the Sun and John Lennon know that the Sun's absolute magnitude is 4.83 or that John Lennon was born on 9 October 1940. These statements should be sourced properly on Wikipedia and they should be sourced properly here. Pichpich (talk) 19:40, 7 July 2013 (UTC)[reply]
    I really like it if people create terms for Wikidata and after I have understood them using translators and the internet these people say no, you understood the soures on the internet not wrong, we only use our own very special definition for Wikidata, which we have not explained before but now I explain it extra for you they say. Thanks for the explanations. I agree with you that after your now introduced definition it is common knowledge that Hartford is a singer-songwriter, before it was no common knowledge. --Pyfisch (talk) 19:33, 9 July 2013 (UTC)[reply]
  •  Oppose (to the requirement that bots needs external sources). Property imported from Wikimedia project (P143) is in my opinion valid for wikipedia as a source, which can be modified later if an external source is found. Bots should add sources always, external or Wikipedia. HenkvD (talk) 11:16, 6 July 2013 (UTC)[reply]
    1) Wikipedia is not a source, 2) by separating data value and data sources, if something changes in wikipedia you lose completly the possibility to retrieve tha good source. Snipre (talk) 10:04, 15 July 2013 (UTC)[reply]
    Because Wikipedia can change I added to the alternative proposal that the bots should state when the retrieved information so that it can be retrieved over the version history of the page. --Pyfisch (talk) 10:37, 15 July 2013 (UTC)[reply]
  •  Support I support the Alternative Proposal except from point two which I believe is superfluous. I assume bots import data live, which implies that the source was accessed only seconds before the data was imported. It also makes sens to have a distiction between imported from Wikimedia project (P143) which means data was imported from a Wikipeda article where the statement (hopefully) is verifiable by sources, and stated in (P248) which means that the claim not only is verifiable, but also verified. Indirect vs. direct source references. /Esquilo (talk) 14:43, 6 July 2013 (UTC)[reply]
    • The import time of statements can be important because Wikipedia changes and some people said that the data can't be validated after Wikpedia removed Information or uses Wikidata. For this I propose to add a timestamp, so that the information can be found always over the version history. --Pyfisch (talk) 16:23, 7 July 2013 (UTC)[reply]
  •  Support – To me the essence is that bots should follow the same guidelines for sourcing as human editors. Byrial (talk) 15:10, 6 July 2013 (UTC)[reply]
    • The structure of the page might be confusing with the alternative proposal in the middle. To avoid misunderstandings: It is the original proposal I support. Byrial (talk) 20:02, 9 July 2013 (UTC)[reply]
  •  Support alternative proposal (obvious :) ) TomT0m (talk) 15:46, 6 July 2013 (UTC)[reply]
  •  Support When bots import thousands of statements they need to import the references as well. I believe it should be acceptable for bots to copy the source references from Wikipedia without checking back to the wording of the original source provided the bot labels the source "imported from fooish wikipedia' in addition to including all of the source reference info. Filceolaire (talk) 16:22, 6 July 2013 (UTC)[reply]
  • I oppose Pichpich's comment and support the one Pyfisch has made. We only transfere the data from Wikipedia to Wikidata. Bots have mainly the task to get the data and structure it at Wikidata. Addings sources is another nice feature but not the basic one. In addition Wikipedias could decide not to use data which is imported from Wikipedia itself. I am sure there would be the technical possibilities. I don't understand why there is such a huge difference between unsourced data in Wikipedia and Wikidata. -- Bene* talk 21:36, 6 July 2013 (UTC)[reply]
    Wikipedia is not a source and you separate data value from its source information: if something changes in wikipedia the change won't be propagated in wikidata leading to discrepancy. Snipre (talk) 12:02, 16 July 2013 (UTC)[reply]
    However, if something changes in one Wikipeida the change won't be propagated to all other Wikipedias, too. So it would be even better if all the data is at Wikidata because here all Wikipedias have the same data which is the most recent actually. Apart from sources, this is the advantage and sense of Wikidata: Centralizing, not sourcing. -- Bene* talk 17:14, 16 July 2013 (UTC)[reply]
    Partly valid point by Snipre – already Wikipedians have less control over Interwikis, as their changes do not show in page history and are not clearly enough marked in Wikipedia's watchlists. This issue needs sorting first. It's too easy to make a mess here without anyone noticing. But how? Littledogboy (talk) 19:29, 16 July 2013 (UTC)[reply]
    @Bene: Don't mix no comparable things, Wikipedia have different policies about sourcing and if one wikipedia doesn't require that statements have to be sourced, this is this problem. But as data from wikidata will be shared between wikipedia and among them some have strong policies about sourcing wikidata has to respect the strongest policy in order to offer the same sercive to all wikipedias. You can't select the policy you want because it's the easiest: the goal of wikidata is to support all wikipedias even the ones which require source for most of the statements. Snipre (talk) 09:07, 17 July 2013 (UTC)[reply]
    Except this in no way make it impossible to have stronger policies in Wikipedias that in Wikidata as the logic to show or not an information will be in the infoboxes source code. So to be useful for all Wikipedias, we should do the opposite of your comment's suggestion. TomT0m (talk) 09:41, 17 July 2013 (UTC)[reply]
    Dear Bene*: Sourcing is not „another nice feature” as you claimed it. On Sourcing depends the whole project. --Succu (talk) 20:53, 16 July 2013 (UTC)[reply]
    "another nice feature" is of course greatly exaggerated, but I want to note that with this proposal we won't have even a percent of the statements we could have from Wikipedia. Why can't we say "this statement is perhaps correct, but we don't have sources for it so don't trust it"? Wikipedias could filter this statements. -- Bene* talk 20:59, 16 July 2013 (UTC)[reply]
    We will not have to do this, when the API will be full we probably will (if not already) be able to access Wikidata sources, so we will be able to so this without adding a statement. TomT0m (talk) 21:12, 16 July 2013 (UTC)[reply]
    Speaking of grossly exaggerated, the "one percent of statements we could have from Wikipedia" is a pure 100% product of your imagination and if you think about it for just a few minutes you'll see that the number is ridiculous. Let me stress again that a lot of statements don't require a reference and would not be affected by this guideline. That includes the vast majority of statements that have been added by bots thus far. Pichpich (talk) 01:02, 17 July 2013 (UTC)[reply]
  •  Oppose alternative Proposal. Bots have to verify imports against third party vendors. Wikipedias are not a valid sources. IUCN conservation status (P141) imported from enWP is a very sad example. --Succu (talk) 19:55, 9 July 2013 (UTC)[reply]
  •  Support original proposal,  Oppose alternative. Wikidata should be for sourced data (with the exceptions listed at Help:Sources). --Avenue (talk) 01:16, 10 July 2013 (UTC)[reply]
  •  Oppose because of the following reasons :
    • Obvious statements have no need to be sourced : Ex : Barack O'bama is a person
    • Wikipedia should not be a source ( in my opinion )
    • It should be the responsibility of the bot owner with the opinion of the community to decide the sources to implement ( reliable sources ) per category of information
    • Do not forget that wikipedias are not obliged to use a specific information. It can only take the information that is the most relevant for it ( That was one of the first specifications )
    • And last, in my opinion, the goal of wikidata is bot to conform to wikipedias. It has to have its own policies. Wikipedia does not serve wikidata but wikidata has to serve wikipedias and not only wikipedias.
In a nutshell : we have to have our own guidelines and policies and focus on reliability of our sources through a deeper analysis. Let us time improve the sources rules. – The preceding unsigned comment was added by 109.12.251.93 (talk • contribs).
  •  Comment Sourcing is necessary if Wikidata wants to be reliable. Wikipedia is no reliable source, collecting data from Wikipedia should be stopped. Wikipedia cannot use data with "Wikipedia" as a source. NNW (talk) 08:17, 18 July 2013 (UTC)[reply]
  • strong support: This is the core of Wikidata. Without sources for each and every entry, the idea of using entries at Wikidata as content in Wkipedias is dead. If there is already a community of Wikidatians, it is your responsibility to enforce this rule. And I mean enforce in a militant way. Ban every single user who speaks out against it. This is a matter of dead or life for Wikidata. Act accordingly. rgds --H-stt (talk) 14:11, 18 July 2013 (UTC)[reply]
  •  Supportif statements have no source they wont be usefull for dewp --Wetterwolke (talk) (where ist the button for signature?)
    Let us get that straight and take a concrete example : de:Anna_Freud. There is no source directly linked with her birth and death dates. Is'nt it a bit extreme to say we will nether import that? And this example is directly taken from dewp, I find that a little ironic :). TomT0m (talk) 16:06, 20 July 2013 (UTC)[reply]
dewp isn't perfect jet, maybe there should be an ref. wikidata is for the hole wikimedia comunity not only the german one, so in wikidata they can do what they wan't. i only prmise you, that dewp will not accept data from wikidata with no ref. maybe that is in other wps ok...
  •  Oppose, I strongly disagree with the last point. Where do those humans magically come from, if there are millions of unsourced properties on Wikidata? --HHill (talk) 07:23, 6 August 2013 (UTC)[reply]
    Hi newcomer, they are the same humans that put and read the datas on Wikipedias, it's the same scale, no magic, nothing changed, the situation is as before. Except they're pushed to add a proper source. TomT0m (talk) 08:14, 6 August 2013 (UTC)[reply]
    I have been visiting Wikidata occasionally now since March. As I do not operate a bot I have rather few edits here, does this make me a newcomer? Humans adding sourced content themselves might be working, but I seriously doubt many humans will like adding sources to thousands upon thousands of unsourced bot-edits. And there are very problematic bot-edits, like e. g. the adding of Property:P27 using categories on Wikipedia (mostly because these categories are a mess on Wikipedia itself). --HHill (talk) 09:19, 6 August 2013 (UTC)[reply]
    I agree that categories are a mess (I could have said that), but it's off topic. If you operate a bot on Wikidata, did you go through the bot approval process ? (I learned about it recently myself.) You could be blocked if you do not do that. For the rest, we do not meet a lot of reference needed template in Wikipedia's article nowdays, this mean that they have been filled now, or that globally the information is not that problematic. TomT0m (talk) 09:32, 6 August 2013 (UTC)[reply]
    As long as they are used to find statements on Wikipedia that can be easily added by bots as properties on Wikidata, categories are clearly not off topic. The lack of a reference needed template does not necessarily mean the information given is unproblematic or even correct. Some Wikipedia versions do not have a citation needed tag for specific statements and most have very few experts who could note (widespread) errors in articles on more or less fringe subjects. But maybe our difference of opinion lies just in the long term vision of Wikidata. I personally think it should be as a whole more reliable than Wikipedia and rather say nothing about a specific property of an item than something that is unsourced and possibly plain wrong or misleading. --HHill (talk) 11:01, 6 August 2013 (UTC)[reply]
    My long term vision is that anything that is possible to do if the datas are not imported for Wikidata from Wikipedia is also possible to do if they are. In the meantime those datas are already on the (at least local) Wikipedias so trying to import them on a single point will help to find inconsistencies between Wikipedia and put the focus on unrealiable datas that are already in Wikipedias. So in the long run the import solution is not wronger than the non import solution, if not better because it as incentive to work on the datas, the mass imports with sources will be exactly the same. The alternative proposal does not advocate for unreliable mass imports. So in my opinion it's better to find good incentive for Wikipedians to work on the datas we already have in Wikipedias to improve their quality, not to make tabula rasa from them, as local WIkipedia with strong requirements can filter them to avoid WIkipedia imported datas without proper sources. I don't see any bad points. TomT0m (talk) 11:58, 6 August 2013 (UTC)[reply]
  •  Support If strict sourcing is not implemented as a policy by WD and specifically for bot edits, WD will be dead as a project sooner or later, i.e. it will survive as a replacement for iw-link maintainance. --WolfgangRieger (talk) 00:14, 7 August 2013 (UTC)[reply]
  •  Oppose It wouldn't "make life more difficult" for bots, but impossible in all but the most trivial cases. Having a statement on Wikidata with "imported from en.wikipedia" is infinitely more valuable than having no statement. Importing such statements benefits Wikidata users, other Wikipedias, and the mentioned "non-bot intelligences" who like to add proper sources; namely, they can (soon) easily find such statements, and add proper sources, without having to do the data import as well. Having a "placeholder source" in no way impedes adding of "proper" sources! --Magnus Manske (talk) 12:09, 20 August 2013 (UTC)[reply]
  •  Oppose See my statement above to use WD:Rank. --Sk!d (talk) 20:25, 20 August 2013 (UTC)[reply]
  •  Oppose Premature. Don't put barriers to mass uploads of data. Or you will kill Wikidata. Pyb (talk) 11:34, 21 August 2013 (UTC)[reply]
  •  Oppose as Pyb. Bots should be allowed to import birth and death dates from Wikipedias! Ayack (talk) 11:52, 21 August 2013 (UTC)[reply]
  •  Oppose Some points are redundant, "information about from where the data comes" doesn't mean much (the original source? secondary? tertiary? Wikipedia? other DBs?). Can't see any benefit from this. --Nemo 14:58, 21 August 2013 (UTC)[reply]
  • It depends. If by "information about from where the data comes" you mean the current "imported from Wikipedia" source then:  Support. If you mean the original source that Wikipedia used then:  Oppose. In the latter case, there is no realistic way for a bot to get that information from, for example, a Wikipedia article. Wikipedia citations might make sense to a human but they are gibberish to a computer. Take for example a simple birth year on English Wikipedia: this could be listed in the lead, in an infobox, in a category and in the body of the biographical article. A bot could read the category and could possibly read the infobox. However, categories are not referenced and it is very rare for an infobox item to be referenced either (usually relying on a citation within the prose). Of the prose, the lead and the body, either may have a citation but these are usually placed at the end of sentences or paragraphs. First, it would be hard for a bot to identify the birth year within the rest of the prose, especially within the body (the semi-standard form of the opening sentence of the lead might help it there). Secondly, even if it could identify the year, a bot could not comprehend the meaning of the citation. Possibly, if the citation was given directly after the year, a bot could make the direct connection, but this is not done on Wikipedia. Further, Wikivoyage does not use citations; most other Wikimedia projects do not use citations like this either. I do agree that this can be improved later, by humans, and that specialised bots could provide a better source in some, very specific circumstances. - AdamBMorgan (talk) 17:47, 21 August 2013 (UTC)[reply]
The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.

What kind of statement doesn't need a source/needs a source ?[edit]

I think this question becomes more crucial in order to keep a good collaboration among the different contributors. But first I think that we have to consider imported from Wikimedia project (P143) as deprecated and its use has to be prohibited for new additions. The question is then which statements need a source and which don't need a source.

imported from Wikimedia project (P143) should be considered as deprecated[edit]

The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.

imported from Wikimedia project (P143) is not deprecated. Legoktm (talk) 06:15, 23 August 2013 (UTC)[reply]
  1.  Support This property was created before the guidelines about sources in order to allow some data imports. But since June we have a guideline (see Help:Sources) and no mention of imported from Wikimedia project (P143) is made so this property should be avoided in future additions. Then we can't source wikipedia with another wikipedia even if one wikipedia contains the original source: because we are creating links between wikipedia without any way to check the validity of the links after some times (and we all know that wikipedia articles are not static documents). Snipre (talk) 09:30, 17 July 2013 (UTC)[reply]
  2.  Oppose It is a valuable information. The fact is that it should not be considered as a valid source does not make it deprecated. TomT0m (talk) 09:38, 17 July 2013 (UTC)[reply]
    By allowing the use of imported from Wikimedia project (P143) you mess the structure of wikidata: under the source section we put normally the source data and not the place where you can find the source data. Then you speak about valuable information now but after 2,3,5 years when article are updated and sources are changed in wikipedia your valuable information loose all value because you creat a link between 2 informations located at different places without a system to propagate changes. You prepare the credibility lost of wikidata in the future by accepting this property. Just think about broken web links in articles and you will understand what is the future of most data in wikidata. Snipre (talk) 10:10, 17 July 2013 (UTC)[reply]
  3.  Oppose I think you asked a good question – the previous one. Way to go. Imported form does not indicate source, but provenance of the piece of data. Littledogboy (talk) 10:18, 17 July 2013 (UTC)[reply]
  4.  Oppose We need this property to source bot additions. Also properties from the list below, which do not need a source, should state from where they were imported. This helps to find bugs in bots and fixing bad informations in local Wikipedia. We all now that Wikipedia articles are not static, and I thougth always that also everybody knows that articles have a version history page. With the version history of the Wikidata page and the Version history of the Wikipedia you can verify the provenance of data also when Wikipedia was changed. The sources guidelines and the alternative proposal state that bots should add the date retrieved so that the old version can be easier checked for validity. --Pyfisch (talk) 11:17, 17 July 2013 (UTC)[reply]
    Sorry, you don't need this property: we have Help:Sources for sourcing. You want this property because you want to import data without sources or to avoid to extract the sources. And for the properties for which no sources are necessary, no sources are needed or if some sources are needed use Help:Sources to correctly source them. And again, it is better to import data without any references instead of trying to add some informations which will be outdated: keep the things simple, we have the source, we add the source, we have no source or we don't want to add it, we let the source section empty. Don't cheat on the quality: we have the source or not but don't say the source is perhaps there and if you are lucky you will find it. Snipre (talk) 11:47, 17 July 2013 (UTC)[reply]
    Did you read and understood my comment? If you didn't understood it, please ask. I don't want to speak against a wall. --Pyfisch (talk) 11:58, 17 July 2013 (UTC)~[reply]
    I understand one thing: if according to your comment it is so easy to find the source in the article or in the history why don't you extract the source when you add the information in wikidata (or at least the version of the articles corresponding to the data addition) ? Be coherent: either it is easy to find the data in the wikipedia article/history and in that case we can extract its source when we import the data in wikidata or it's something which can be complex and in that case don't ask the data user to do the job you can't do it. Snipre (talk) 12:23, 17 July 2013 (UTC)[reply]
    I said that you can find easily the page version on Wikipedia from which the bot imported the information. I proposed to add add the version over a date retrieved property, alternativly the version id could be added. This is nothing complex in. To find a source in the Wikipedia article is difficult in the most cases. --Pyfisch (talk) 16:44, 17 July 2013 (UTC)[reply]
  5.  Comment It is useful to be able to see where the statement was imported from without checking the history so as long as bot are only adding statements that don't require a reliable source, I'm ok with this property. But we have to find a way of distinguishing this from ordinary sources. Pichpich (talk) 17:10, 17 July 2013 (UTC)[reply]
  6.  Oppose, it is useful for debug purpose and it is used with non-Wikipedia items too, for example sex or gender (P21) <imported from Wikimedia project (P143)> patronymic (Q110874). — Ivan A. Krestinin (talk) 21:19, 5 August 2013 (UTC)[reply]
  7.  Oppose --Magnus Manske (talk) 12:09, 20 August 2013 (UTC)[reply]
  8.  Strong support - See my comment above. --Succu (talk) 20:38, 20 August 2013 (UTC)[reply]
  9.  Oppose Premature. Pyb (talk) 11:29, 21 August 2013 (UTC)[reply]
  10. -1 --Nemo 14:55, 21 August 2013 (UTC)[reply]

List of properties without a source[edit]

Authority control[edit]

TO be continued...

P31 and P279 might require a source in some non obvious case. There exists different possibility of classification, the question is whether or not it is a POV to arbitrarily choose one of them in Wikidata without source. TomT0m (talk) 09:36, 17 July 2013 (UTC)[reply]
We can spend ages to discuss every case but here we are discussing the general trend and specific cases has to be treated later by the task forces according on their own policies. Snipre (talk) 09:58, 17 July 2013 (UTC)[reply]
Then general trend will be endless discussions. TomT0m (talk) 10:18, 17 July 2013 (UTC)[reply]
Why ? General discussion take more time than analyzing every use of each property ? We don't have the same experience of discussion. Snipre (talk) 12:02, 17 July 2013 (UTC)[reply]
Well, if you set restrictive rules, and discuss for any use case, there is plenty of usecases. If you set restrictive rules and a user has a relevant exception, you will have to fight against him. Then a second, then a third, ... too much restrictions will lead to discuss not only classes of exceptions but also individual ones. TomT0m (talk) 15:21, 17 July 2013 (UTC)[reply]
I added all authority control properties because they help to find more and better informations about the item and they do not need sources per the guidlines point #2 (can be checked using the property itself or other authority control properties). I maybe missed some similar properties for the list. --Pyfisch (talk) 12:12, 17 July 2013 (UTC)[reply]
Ok, so we can add all external database identifiers too. Snipre (talk) 12:26, 17 July 2013 (UTC)[reply]

Help:Sources makes an exception when "the item itself is a source for a statement " so that would include properties like author (P50), illustrator (P110), performer (P175), director (P57) and so on as well as more technical stuff like ISBN-13 (P212). The properties part of the series (P179), occupation (P106) and part of (P361) typically lead to statements that are both common knowledge and likely to fall under the guideline's #2 exemption so I would be ready to add them to the list. It's important to remember that proposed bot tasks still have to go through an approval process and that provides some flexibility. Pichpich (talk) 16:01, 17 July 2013 (UTC)[reply]

So, if I understand well, there may be a consensus on this:

Only certain properties (listed above) may be added unsourced (by both bots and humans), but where the piece of data comes from must always be indicated (ie imported from: English Wikipedia + date or revision number).

  •  Comment <text mode="supposed to be funny">I'm waiting to see how this list will weaken the strong positions emitted in the first part of this discussion :) </text> TomT0m (talk) 17:52, 18 July 2013 (UTC)[reply]

Guidelines for imported from Wikimedia project (P143)-based sources in conjunction with stated in (P248)-based sources[edit]

There are already some bots and humans which add claims and references according to external, non-Wikipedia sources. As I found no specific rule how to handle imported from Wikimedia project (P143)-based sources in conjunction with stated in (P248)-based sources following Help:Sources, I want to propose:

  1. All Wikipedia-based imported from Wikimedia project (P143) references of a claim which is sourced with stated in (P248) should be deleted. See for example Ecchlorolestes nylephtha (Q307144): As taxon name (P225): Ecchlorolestes nylephtha is now referenced by a valid third-party source, the source "imported from en-wp" is now redundant as the third-party source is both, a validation of the claim and a valid source for it. Of course, a claim which has already a source based on valid reference should not be amended by additional Wikipedia-based imported from Wikimedia project (P143) sources, e.g. [3]. A bot should search for these redundant sources and remove them.
  2. A bot must not change a claim or even add an alternative value if the claim exists and has a stated in (P248)-based source if the only source the bot has is a Wikipedia-based imported from Wikimedia project (P143).

I hope, this is something like the lowest common denominator where we all can agree upon.  — Felix Reimann (talk) 11:03, 19 July 2013 (UTC)[reply]

It may be difficult for a bot to see if a reference with imported from Wikimedia project (P143) is Wikipedia based or not. There is currently 161 different values to imported from Wikimedia project (P143) in references claims, and many or these is not Wikipedias. Byrial (talk) 11:31, 19 July 2013 (UTC)[reply]
For example: instance of Wikipedia ?  — Felix Reimann (talk) 11:39, 19 July 2013 (UTC)[reply]
If that's a difficult thing for you I understand that the discussions can sometime be difficult with person with a slightly different background around here :) TomT0m (talk) 18:33, 19 July 2013 (UTC)[reply]
I did not say it is difficult for me. I said it may be difficult for a bot, and I am definitely not a bot. Please do not use that form of personal attack. Byrial (talk) 15:23, 22 July 2013 (UTC)[reply]
Sorry, but it is not difficult for a bot: a bot is doing what the bot owner asks, so it the task of the bot owner to define what is the correct source and it's its responsability to do it correctly. Snipre (talk) 15:44, 22 July 2013 (UTC)[reply]
It was not a personal attack. It's just so obvious to me that's it not difficult for a bot that I think anybody with the background to discuss here should understand that two. That's partly why discussions can be difficult here. TomT0m (talk) 08:37, 23 July 2013 (UTC)[reply]
Well, it is not obvious to me. The bot needs to see to if any item used as value for imported from Wikimedia project (P143) has the claim instance of (P31) Wikipedia (Q52). If the bot framework is not set up to do this, it may indeed be complicated for the individual bot operator to program this extra task. I am not saying that it cannot be done, but bot operators who just use a bot which they did not program themselves may mave trouble fulfilling this requirement. Byrial (talk) 09:46, 23 July 2013 (UTC)[reply]
There is libraries and framework like pywikibot for that. These frameworks should implements Wikidata policies for users, and maybe predifined models for well defined application domains like sources. TomT0m (talk) 09:52, 23 July 2013 (UTC)[reply]
I agree. Byrial (talk) 10:36, 23 July 2013 (UTC)[reply]
I thought imported from was only used with values for Wikipedias? Littledogboy (talk) 23:58, 19 July 2013 (UTC)[reply]
See for example P21 in Kylie Minogue (Q11998). I hope bots citing third party databases switched to stated in (P248) in the meantime but this is not sure. Nonetheless, the discussing here is about P143 for Wikipedia input.  — Felix Reimann (talk) 15:01, 22 July 2013 (UTC)[reply]
The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.
Retrieved from "https://www.wikidata.org/w/index.php?title=Wikidata:Requests_for_comment/Sourcing_requirements_for_bots&oldid=74029041"