1 00:00:00,000 --> 00:00:05,910 Read, make sure you grab some stickers that are floating around. My name is Kevin karate I go by

2 00:00:05,910 --> 00:00:12,750 user super hamster. I chose my username when I was 12 years old, but a comedian since 2007. And

3 00:00:12,750 --> 00:00:17,820 I'm joined with Jennifer eight Lee. Yes, her monologue is actually eight. Well, you want to

4 00:00:17,820 --> 00:00:19,290 say by yourself, I'm

5 00:00:19,290 --> 00:00:23,880 gonna say that people forgot the dot after the eighth, which is a decimal point, actually not

6 00:00:23,880 --> 00:00:33,630 correct in an exam. Serious like I, what do I do? I am, I raised money. That's it. I'm the

7 00:00:33,630 --> 00:00:38,910 money person with a lot of stuff to do with credibility. And I'm on the board of, of hacks,

8 00:00:38,910 --> 00:00:44,010 hackers, which actually runs a project called or helps run a project called Deke red in

9 00:00:44,280 --> 00:00:52,650 conjunction with Wikimedia DC mostly. And I'm a fun fact. Fun fact. I'm on the Unicode emoji

10 00:00:52,650 --> 00:00:59,820 subcommittee, so I help regulate the emoji that are on your phone. So those are that's my, my

11 00:00:59,820 --> 00:01:00,870 like, fun fact. But anyway.

12 00:01:02,070 --> 00:01:06,300 Yeah. So we're just going to begin with a quick overview of like verifiability and reliable

13 00:01:06,300 --> 00:01:10,710 sources on Wikipedia. If you're a Wikipedia editor, this should be pretty familiar to you.

14 00:01:10,710 --> 00:01:14,520 If you're not a Wikipedia editor. Hopefully, this is kind of a quick good overview for you.

15 00:01:15,090 --> 00:01:20,100 Wikipedia is based on reliable published sources, people always ask how do you know he is

16 00:01:20,100 --> 00:01:25,770 reliable? The answer is you check the sources where the information is coming from. And we

17 00:01:25,770 --> 00:01:31,050 have a bunch of policy pages related to this. So if you go to Wikipedia and type in WP colon Rs,

18 00:01:31,380 --> 00:01:37,860 that's a good starting point to kind of learn about reliable sources on Wikipedia. And one of

19 00:01:37,860 --> 00:01:41,820 the favorite ways I like to introduce reliable sources is Wikipedia is a perennial sources

20 00:01:41,820 --> 00:01:47,370 list. And this is basically a huge table of hundreds of entries. That describes how the

21 00:01:47,370 --> 00:01:53,790 English Wikipedia community views sources. Wikipedia editors are always discussing sources

22 00:01:53,790 --> 00:01:58,230 whether or not they're reliable, what context certain sources not be reliable. And based on

23 00:01:58,230 --> 00:02:03,300 these discussions, we kind of develop consensus, and from this consensus, we are able to produce

24 00:02:03,300 --> 00:02:09,120 this table. So in this image, we see here you can see for example, as TVs considered to be

25 00:02:09,150 --> 00:02:14,580 unreliable, being like an Iranian government propaganda outlet right.com is considered

26 00:02:14,580 --> 00:02:21,000 marginally reliable. Project Veritas has been deprecated and on a spam blacklist. While pro

27 00:02:21,000 --> 00:02:27,480 publica is generally considered reliable, really fascinating page to kind of dig through. And

28 00:02:27,480 --> 00:02:33,510 English Wikipedia is not the only version of Wikipedia with this list, about a dozen other

29 00:02:33,540 --> 00:02:37,680 language, Wikipedia versions also have their own perennial sources lists. And then

30 00:02:38,040 --> 00:02:42,360 I just want to say as someone who came into this sort of on the later side, the name is very

31 00:02:42,360 --> 00:02:45,780 confusing because perennial sources does not mean they are perennially reliable means that

32 00:02:45,840 --> 00:02:50,280 they're generally perennially debated. And that's when you're trying to explain to funders

33 00:02:50,280 --> 00:02:51,900 is very is very confusing.

34 00:02:55,620 --> 00:03:00,660 And something I like to highlight is our sources change over time. There's kind of a sad story of

35 00:03:00,660 --> 00:03:05,550 CNET seen, it's been like a side that I always use as a kid growing up learning about like PC

36 00:03:05,550 --> 00:03:12,960 news and PC gaming news. Prior to 2000 Plenty seen it was considered to be generally reliable

37 00:03:12,960 --> 00:03:19,260 related to technology, until red ventures acquired them in October 2021. Spread ventures

38 00:03:19,290 --> 00:03:24,600 undelete acquired them they deleted 1000s of old CNET articles, and the only way we can access

39 00:03:24,600 --> 00:03:30,960 them is through the Internet Archive through the Wayback Machine. From October 2020 to November

40 00:03:30,960 --> 00:03:37,950 2022. Wikipedia community considers CNET to be marginally reliable, because red ventures

41 00:03:37,950 --> 00:03:44,250 acquisition resulted in a kind of a deterioration of editorial standards. And then

42 00:03:44,250 --> 00:03:50,220 what's interesting is starting in November 2022, with like the rise of Chachi Beatty and MLMs.

43 00:03:52,260 --> 00:03:57,990 CNET started publishing articles using AI. These articles had very limited or no fact checking.

44 00:03:58,020 --> 00:04:03,810 They were riddled with factual errors. And they didn't disclose they were using AI. They just

45 00:04:03,810 --> 00:04:09,780 kind of attributed the article to quote CNET money staff. So once this happened, and it was

46 00:04:09,780 --> 00:04:16,140 exposed, there will be a community now considers CNET to be unreliable research on Wikipedia. So

47 00:04:16,140 --> 00:04:21,150 it's kind of a sad tale of the downfall of Wikipedia or have seen it through Wikipedia is

48 00:04:21,150 --> 00:04:28,290 eyes. And then reliability is also context dependent. So there are several sources that we

49 00:04:28,290 --> 00:04:33,780 consider reliable or unreliable on certain topic areas. So the Huffington Post, for example, we

50 00:04:33,780 --> 00:04:39,480 consider reliable for topics not related to politics, but politics are considered marginally

51 00:04:39,480 --> 00:04:44,700 reliable. And in the Huff Post also has a contributor section of kind of like, the public

52 00:04:44,700 --> 00:04:50,190 can kind of create articles and we consider those to be generally unreliable. Some similar

53 00:04:50,190 --> 00:04:55,260 vein Fox News news, excluding politics and sciences is considered marginally reliable or

54 00:04:55,260 --> 00:05:00,780 there's no consensus there, but it is considered generally unreliable related to Paul optics in

55 00:05:00,780 --> 00:05:09,360 science, and of course their talk shows as well. So, context is important. Lots of wiki projects

56 00:05:09,360 --> 00:05:15,360 also have their own list of sources. Mr. Project Video Games has a list of generally reliable

57 00:05:15,360 --> 00:05:21,030 sources for gaming. Film has one Korea has one. Very interestingly, Christian music has their

58 00:05:21,030 --> 00:05:24,900 own list of reliable and reliable sources. So there's a lot of interesting lists that you can

59 00:05:24,900 --> 00:05:28,710 kind of trawl through to see topics specific source listings.

60 00:05:30,960 --> 00:05:41,190 We've talked about what, yes. So there, you see how they say. So I've been through parts of my

61 00:05:41,190 --> 00:05:46,410 life and more working on the issues of credibility and misinformation since basically

62 00:05:46,410 --> 00:05:54,900 after the 2016 elections, and starting in 2018. And haven't showed up at a hackathon that we

63 00:05:54,900 --> 00:05:59,550 read that this is fun, actually, we used to have these conferences called Miss info cons, and we

64 00:05:59,550 --> 00:06:05,460 still do, we still do, then Facebook funded one. And the Miss info con logo is actually one of

65 00:06:05,460 --> 00:06:11,460 the most amazing logos I've ever worked on it is it is half in English. And the mirror half of it

66 00:06:11,460 --> 00:06:20,130 isn't Cyrillic. So it says mid fitness info con in Cyrillic. So Facebook now meta funded one

67 00:06:20,130 --> 00:06:27,060 have wanted to find one of these conferences. They're like, we can't find a conference where

68 00:06:27,060 --> 00:06:30,450 the logo is half in Cyrillic, right? Because this is, after all, like the Russian

69 00:06:30,450 --> 00:06:35,370 misinformation and all of that sleek audit cred con, which is sort of like, which is very

70 00:06:35,370 --> 00:06:38,880 Silicon Valley, because everything in Silicon Valley is like named the opposite of what you

71 00:06:38,880 --> 00:06:43,560 deal with, like, the Trust and Safety Team is dealing with everything that is not trustworthy

72 00:06:43,560 --> 00:06:47,520 or safe. And then things like customer success is always dealing with people who are not

73 00:06:47,520 --> 00:06:54,690 successful. Right. So con was I was the hackathon that we did in Austin, and Andrew Lee,

74 00:06:54,690 --> 00:06:59,070 who many of you guys know, I guess suggested to Kevin that he drove up from Dallas, and he shows

75 00:06:59,070 --> 00:07:07,080 up. And at that point, we I think the foundation had decided not to fight and he cried out had

76 00:07:07,080 --> 00:07:14,520 not decided not to fun, wiki conference. 2019 I think it was, and I think it was really sad, or

77 00:07:14,520 --> 00:07:17,400 he was really sad. He was really sad. And I was like, why are you sad? And he's like, because we

78 00:07:17,400 --> 00:07:21,780 just found out we're not gonna get money from the foundation. And I was like, what's, what's

79 00:07:21,780 --> 00:07:25,950 the budget for your hombre? And he's like, 50,000, and I was like, I can you make this

80 00:07:25,950 --> 00:07:31,110 conference about credibility, I can find you 50,000. So we found 135,000. And we did a great

81 00:07:31,110 --> 00:07:38,850 conference around credibility in 2019 at MIT, and from that entire sort of a larger initiative

82 00:07:38,880 --> 00:07:46,590 of the role that Wikipedia can play on in credibility on the Internet as a whole was born

83 00:07:46,620 --> 00:07:52,770 because I do sort of the thesis there was that Wikipedia is one of the very few places where

84 00:07:52,770 --> 00:07:58,500 you can scale information reliability in a sort of systematic infrastructure way, right? There's

85 00:07:58,500 --> 00:08:03,150 a lot of fact checking loved fact checkers, whatever, but they they're, you know, on a good

86 00:08:03,150 --> 00:08:07,320 day, you can do maybe like 10 back checks, you know, as a as an individual. So that so you need

87 00:08:07,320 --> 00:08:13,800 some kind of more systemic reliability. And because of the way that the new players on the

88 00:08:13,800 --> 00:08:20,880 Internet, either Apple or Google, depend on Wikipedia, if you make things they're more

89 00:08:20,910 --> 00:08:26,280 reliable, then we can we can have an impact on the Internet scale. So among the different

90 00:08:26,280 --> 00:08:30,120 things that wiki cred, which is our initiative have done now, obviously, we kind of helped with

91 00:08:30,120 --> 00:08:36,120 it 2019 conference, we've given out credibility grants, sort of initiatives, up to 10,000. We've

92 00:08:36,510 --> 00:08:40,110 gotten funding from the foundation itself, Craig Newmark, you know, of which many things in

93 00:08:40,110 --> 00:08:44,760 journalism are named after. And we've worked very closely, obviously with hacks, hackers,

94 00:08:44,940 --> 00:08:49,290 Wikimedia, DC, and apparently, Wikimedia New England, which I did not know, until I saw this

95 00:08:49,290 --> 00:08:51,990 slide. Yeah.

96 00:08:54,150 --> 00:08:58,680 So I'll just be going over kind of the some of the projects that the Credit Initiative has kind

97 00:08:58,680 --> 00:09:04,080 of helped fund and support over the last few years. First one I'll touch on is called the

98 00:09:04,080 --> 00:09:10,410 vaccine safety project, which exists on Wikipedia to go to WP colon V safe. This was a

99 00:09:10,410 --> 00:09:17,160 project which was ran by Netta, which many of you know, she basically compiled, kind of like

100 00:09:17,160 --> 00:09:22,950 the perennial sources list, a list of sources that are considered reliable for vaccine vaccine

101 00:09:22,950 --> 00:09:27,240 safety. And this was all like during the pandemic, right, so vaccine safety was like a

102 00:09:27,240 --> 00:09:30,720 very hot topic. And there's a lot of misinformation on the Internet. So kind of a

103 00:09:30,720 --> 00:09:36,630 list of what's considered reliable and unreliable again in the context. So very cool

104 00:09:36,630 --> 00:09:42,660 project. It's worth checking out. Another one is actually a tool I developed called sight unseen

105 00:09:42,690 --> 00:09:49,200 hackathon. You showed up at the hackathon I showed up but yeah, in 2018 this user scripted

106 00:09:49,200 --> 00:09:53,520 ads, little icons and citations to kind of denote their nature. So whether something is an

107 00:09:53,520 --> 00:10:00,210 opinion piece and from an advocacy group, a new source, a blog, social media, etc. The idea As

108 00:10:00,210 --> 00:10:05,820 to kind of give you a quick glance of the nature of the sources being cited, our context does

109 00:10:05,820 --> 00:10:09,450 matter. So like, you should never look at an icon and be like, Oh, this is a bad source or

110 00:10:09,450 --> 00:10:14,400 good source, but it's just there to kind of provide extra context. My favorite example is

111 00:10:14,400 --> 00:10:20,370 the first one, which is where a article from the San Antonio Express News is marked as an opinion

112 00:10:20,370 --> 00:10:25,410 piece. Oftentimes, you might see a article come from New York Times or Washington Post, and you

113 00:10:25,410 --> 00:10:28,980 go, Oh, that must be a news source. But it actually is an opinion piece. It's kind of good

114 00:10:28,980 --> 00:10:35,700 to get that context upfront. And there's also a bunch of other citation entertaining scripts out

115 00:10:35,700 --> 00:10:40,350 there head bomb has his own unreliable, unreliable source script. That's really good.

116 00:10:41,430 --> 00:10:46,500 Yeah, his script is does good at marketing, predatory journals. So that's also worth

117 00:10:46,500 --> 00:10:54,660 checking out. When you get bought, do you ever talk about this? Okay. Another project that's

118 00:10:54,660 --> 00:11:00,570 very recent that we have decred's working on is called wiki credit bot. So one kind of perennial

119 00:11:00,570 --> 00:11:07,680 issue on Wikipedia is there's no good structured database of sources, even the perennial sources

120 00:11:07,680 --> 00:11:08,220 list being looked

121 00:11:08,220 --> 00:11:12,210 at, there's no, there's no structured database sources, except for spam list. Even

122 00:11:12,210 --> 00:11:16,320 the perennial sources list we looked at, that's just the big HTML table, right. It's not like a

123 00:11:16,320 --> 00:11:22,200 JSON on a POC page. Yeah, there's no good structuring of that. So what we're working on is

124 00:11:22,200 --> 00:11:31,440 a bot called wiki cred bot. And the idea is, for each wiki project will kind of come up with the

125 00:11:31,440 --> 00:11:35,520 list of unknown sources that are good known sources that are bad, based on the perennial

126 00:11:35,520 --> 00:11:39,930 sources list, as well as sources that that project considers to be reliable or unreliable.

127 00:11:40,170 --> 00:11:44,700 And once we kind of have that structuring of data in place, we can run this bot, which will

128 00:11:44,700 --> 00:11:50,490 generate reports, it'll say when articles underneath that wiki project are citing our

129 00:11:50,700 --> 00:11:54,840 sources that are considered not good, it'll kind of flag them so that editors can go in and

130 00:11:54,840 --> 00:12:02,310 review their sources, those sources. We can also see articles that are citing good sources. So so

131 00:12:02,310 --> 00:12:06,240 far, we've just enabled this bot for the vaccine safety project, which we just kind of covered.

132 00:12:06,990 --> 00:12:11,700 But we're hoping to get this going for other wiki projects as well, you can kind of see the

133 00:12:11,700 --> 00:12:17,370 stats that this bot is able to generate. Still very new, right, only 3% of sources identified

134 00:12:17,370 --> 00:12:21,990 are known reliable sources. But over time, we hope this kind of data set can grow. And this

135 00:12:21,990 --> 00:12:30,120 tool becomes more and more useful. Thanks. wiki site is also a project worth knowing about this

136 00:12:30,120 --> 00:12:37,410 is a very long term and ongoing effort. Finally, in the same line to kind of structure citations

137 00:12:37,410 --> 00:12:43,230 and the data behind citations. So for example, what wiki site has led to is, data for millions

138 00:12:43,230 --> 00:12:48,600 of academic works in journals has been imported into wiki data, you can now on Wikipedia

139 00:12:48,600 --> 00:12:54,450 actually use a template to cite a journal that will pull data from wiki data. So kind of having

140 00:12:54,450 --> 00:12:58,770 a structured data that can be reused across all the Wikipedia and remain consistent is pretty

141 00:12:58,770 --> 00:13:03,360 valuable. And there's a bunch of tooling they've also developed for helping improve wiki data

142 00:13:03,360 --> 00:13:07,620 entries. So it's a project worth knowing about and following. If you want to learn more, you

143 00:13:07,620 --> 00:13:16,170 can go to the meta wiki look up wiki site. And other cool project called ft dot news. It's not

144 00:13:16,500 --> 00:13:21,540 directly underneath Wikipedia. It's like an external project. But they're kind of working on

145 00:13:21,540 --> 00:13:29,640 this idea of structured citation data as well by listing unreliable and fake news websites. And

146 00:13:29,640 --> 00:13:33,630 wiki credit provided them funding to actually kind of import Wikipedia data into their

147 00:13:33,630 --> 00:13:39,030 database. And it's just a really cool tool to kind of see hundreds, if not 1000s, of

148 00:13:39,030 --> 00:13:44,790 classifications of what's considered, you know, fake news, unreliable, reliable, and so on. So

149 00:13:44,790 --> 00:13:46,740 that's another cool project worth checking out. Yeah.

150 00:13:46,740 --> 00:13:53,610 And also, this is sort of interesting. So before, he imported data from Wikipedia, he was

151 00:13:53,610 --> 00:14:01,170 just some dude basically doing his own kind of evaluations of how reliable some sources were in

152 00:14:01,170 --> 00:14:07,050 and you know, there's some that are like, maybe they've gotten publicity that there is

153 00:14:07,380 --> 00:14:15,180 consensus, like, you know, Sputnik or RT or whatnot, but it was just one dude. So there is

154 00:14:15,180 --> 00:14:21,060 some value because of the consensus driven processes in judging on Wikipedia that is

155 00:14:21,060 --> 00:14:28,320 valuable to the Internet at large. They'll Google whenever we have conversations with them

156 00:14:28,320 --> 00:14:34,200 about some of the structured search when you're structuring the databases of the domains always

157 00:14:34,200 --> 00:14:38,490 pushes back. I mean, it's it's very fascinating, like how many people judge like what you know,

158 00:14:38,490 --> 00:14:42,300 what is the universe of people, you know, who are participating, the judges, you know, is

159 00:14:42,300 --> 00:14:46,980 there a bias so, there's sort of a it's very interesting, because on some level, it's also

160 00:14:46,980 --> 00:14:52,350 the I think it's kind of like the best we we got that there's some commercial services,

161 00:14:52,350 --> 00:14:58,290 obviously, that do something similar, but I do think that it is, it is better when it's an open

162 00:14:58,290 --> 00:15:05,280 discussion for how Different things are, are are assessed by the Wikipedia community.

163 00:15:07,680 --> 00:15:11,280 It's always funny linking to these discussions about reliable sources. And you've got comments

164 00:15:11,280 --> 00:15:14,850 by people named like super hamster and head bomb and like all these silly user names, but

165 00:15:14,850 --> 00:15:18,150 they're, you know, having very serious discussions about what is and is not a good

166 00:15:18,150 --> 00:15:24,450 source. Last thing we'll touch on is just AI because why not? That's what everyone's talking

167 00:15:24,450 --> 00:15:25,290 about these days.

168 00:15:25,740 --> 00:15:29,310 We didn't know that much about it. But yes, we follow without we won't, we're remiss not to

169 00:15:29,310 --> 00:15:32,550 have a slide it said AI. Wikipedia conference.

170 00:15:32,880 --> 00:15:37,920 So yeah, of course, it's been a hot topic in the Wikipedia universe, like when should we use AI?

171 00:15:37,980 --> 00:15:43,560 Is AI a threat to the community and so on? And it's quite interesting because, of course, a lot

172 00:15:43,560 --> 00:15:48,360 of these algorithms like chat, GBT have been trained on Wikipedia, a very significant portion

173 00:15:48,360 --> 00:15:48,750 of them.

174 00:15:49,890 --> 00:15:58,290 I will also say, the two largest corpuses of text that the GPT universe has been trained on

175 00:15:58,320 --> 00:16:06,120 one is, one is Wikipedia. Do you guys know what the other one is? It's Reddit. So it is. So

176 00:16:06,120 --> 00:16:12,780 given the choice between LLM is being shaped by Reddit or Wikipedia. I do I do prefer Wikipedia.

177 00:16:12,810 --> 00:16:18,960 They sort of just interesting to know that those those were the largest sort of inbound in the

178 00:16:18,990 --> 00:16:26,520 original source obviously now Reddit has sort of quasi blog quasi negotiating with the large lol

179 00:16:26,550 --> 00:16:33,990 M providers to licensor data so we can media so remind remains much more open. That is something

180 00:16:33,990 --> 00:16:41,370 to keep in mind when you hear like the the kind of tone GPT GPT, returning something back to you

181 00:16:41,370 --> 00:16:44,760 because it is often Wikipedia or Reddit.

182 00:16:49,650 --> 00:16:55,020 Yeah, just kind of the whole thing about IS LM a threat to media as more and more people turn

183 00:16:55,020 --> 00:16:58,680 into MLMs. And like, you know, Google is experimenting with when you do a Google search,

184 00:16:58,740 --> 00:17:03,480 they'll try to auto generate an alum response at the top. And that will of course, kind of drive

185 00:17:03,480 --> 00:17:07,680 traffic away from Wikipedia, which kind of reminds me of the fear we had maybe 10 years ago

186 00:17:07,680 --> 00:17:11,940 or so when or when Google introduced the Knowledge Graph, which is that little box in the

187 00:17:11,940 --> 00:17:16,710 top right of Google searches that kind of has a little blurb from Wikipedia has images has basic

188 00:17:16,710 --> 00:17:21,300 information. The fear then was would that drive traffic away from Wikipedia and lead to its

189 00:17:21,300 --> 00:17:25,770 downfall, which it did not move? We're kind of looking at that question again, in the context

190 00:17:25,770 --> 00:17:30,270 of MLMs. What I remember is, of course, that we've we've as we've all seen hallucinate

191 00:17:30,270 --> 00:17:36,180 information. They don't provide sources. There has been some experimentation. I know Google's

192 00:17:36,180 --> 00:17:42,450 LLM, they kind of generate their text. And then they'll kind of go back and try to find sources

193 00:17:42,450 --> 00:17:48,090 that match the text they generated and link to those as sources. How much actual fact checking

194 00:17:48,090 --> 00:17:53,580 that is probably not too much. So there's like some experimentation in that realm. But

195 00:17:53,580 --> 00:17:57,810 ultimately, the data you're getting from an LLM is not coming from a specific source, you know,

196 00:17:57,810 --> 00:18:03,750 it's just generated text. While Wikipedia on the other hand, of course, is sourced every

197 00:18:03,750 --> 00:18:10,410 statement on Wikipedia should have a source. And another interesting point is the context of LLM

198 00:18:10,410 --> 00:18:17,190 is collapsing. If an LLM is trained on its own data, so as more and more LLM text is added to

199 00:18:17,190 --> 00:18:23,040 the web, if LLM is like catchy beats, he started grabbing that information that it's self

200 00:18:23,040 --> 00:18:28,620 generated, it leads to what's kind of known as a collapse where the quality of the generic text

201 00:18:28,620 --> 00:18:33,420 just takes a huge nosedive, which is something that open AI and other institutions have to kind

202 00:18:33,420 --> 00:18:38,160 of watch out for and figure out how to solve. Wikipedia doesn't have that problem. Of course,

203 00:18:38,190 --> 00:18:44,340 Wikipedia might sometimes accidentally site itself, accidentally, inadvertently, but not to

204 00:18:44,340 --> 00:18:53,160 that same scale. So yeah, that's pretty much it for our presentation. We kind of want to open up

205 00:18:53,160 --> 00:18:58,110 the floor. I think now to just kind of hear your thoughts or people working on very interesting

206 00:18:58,110 --> 00:19:02,730 projects related to reliable sources or if you have questions Yeah. Yeah, go ahead.

207 00:19:03,480 --> 00:19:05,940 Can you introduce yourself before as you speak

208 00:19:09,000 --> 00:19:16,590 there's a new company called add Ponce's media that we face new sources according to learn the

209 00:19:16,590 --> 00:19:24,600 right bias and reliability. If anybody would care to take a look at that article, there

210 00:19:24,600 --> 00:19:31,170 appears to be some what can be what your feelings do not want me to update the news

211 00:19:31,170 --> 00:19:32,820 somehow so

212 00:19:32,820 --> 00:19:37,800 interesting, that they don't they don't want you to update the article about five your

213 00:19:37,800 --> 00:19:44,910 article on that content. You hear things. They're saying I bought this is not reliable

214 00:19:44,910 --> 00:19:55,980 source. But conversations about Armitage was I could see that so that needs a little attention.

215 00:19:56,730 --> 00:20:01,050 So I am familiar with the site. This has been kind of very popular Then in terms of like,

216 00:20:01,800 --> 00:20:08,040 ranking sites on the reliability and bias for bias, I would say then Yeah, we like as you

217 00:20:08,040 --> 00:20:11,850 said, the perennial sources list currently list as Fontys, at least from the community's

218 00:20:11,850 --> 00:20:22,110 perspective as a generally unreliable source. So well, I feel like God opened up another

219 00:20:22,110 --> 00:20:31,800 discussion. Yeah. Yeah, looks like the last discussion was in 2021. Roy?

220 00:20:41,520 --> 00:20:46,410 It's my understanding is it's not exclusively English line. My gosh, this is a question is

221 00:20:46,410 --> 00:20:50,910 about whether or not the large language models are only training on English language, Wikipedia

222 00:20:50,910 --> 00:20:56,190 or other languages. My understanding is it's not just, it is not just English language,

223 00:20:56,190 --> 00:20:59,850 Wikipedia, but obviously, the English language Wikipedia is the most robust and yatta yatta

224 00:20:59,850 --> 00:21:08,370 yatta yatta. So I don't know, like, you know, whose back and even like is, you know, how

225 00:21:08,370 --> 00:21:14,790 robust, you know, obviously, that is. And then I also, I also have other thoughts. I mean, that

226 00:21:14,790 --> 00:21:18,960 we can go for more questions, but I was I can also like, stand here and pontificate on all the

227 00:21:18,960 --> 00:21:27,750 things that I think that the foundation can do to, to help us with some of these issues. But I

228 00:21:27,750 --> 00:21:29,820 will, but let's do some more questions also.

229 00:21:32,880 --> 00:21:36,480 Thinking with sight unseen, free tool,

230 00:21:36,540 --> 00:21:38,340 it's a great tool. Yeah. I feel like more of

231 00:21:40,800 --> 00:21:46,890 something in the community was was somebody you know, now knowing you're typing into that into

232 00:21:46,890 --> 00:21:56,910 giving something that in Billings, what is the future? We can have a little pop up box a US

233 00:21:56,910 --> 00:21:58,560 excellent Navy and underline?

234 00:21:58,590 --> 00:22:05,100 Yes, yeah, yes. Okay, can I get I can get into okay. So this is my Johnny, Johnny's little fist

235 00:22:05,100 --> 00:22:09,510 on the whole situation. So that exactly what you're talking about, which is there the

236 00:22:09,630 --> 00:22:13,470 foundation, I think I've found the foundation currently is development well, but she's about

237 00:22:13,470 --> 00:22:19,530 to pull up on, on whatever that site is, whenever they talk about whatever tools are

238 00:22:19,560 --> 00:22:27,060 developing. So what's interesting is they're, they're developing something, something like

239 00:22:27,060 --> 00:22:33,930 that, which is, oh, you're, you know, you're citing you're citing this thing. Maybe it's a

240 00:22:34,080 --> 00:22:39,000 unreliable source. And it's based on my understanding the same kind of infrastructure.

241 00:22:40,200 --> 00:22:47,670 That is that kind of looks at whether or not a site is on a spam blacklist, both universal or

242 00:22:47,670 --> 00:22:56,160 in the English language thing. And we are looking at this means basically, an API database

243 00:22:56,160 --> 00:23:01,320 of the sources that are real, considered reliable, non reliable, marginally reliable. And

244 00:23:01,320 --> 00:23:06,840 I like really record on getting this fanbase, because I think it's always true. I couldn't

245 00:23:06,840 --> 00:23:13,080 believe this in the year like 2023. But I checked I was there is no structure database. Of

246 00:23:13,800 --> 00:23:19,650 the liable sources, there is the perennial sources list. But that literally is a talk page.

247 00:23:19,860 --> 00:23:23,940 That is, you know, there's some effort to be like, well, let's make it into a JSON file,

248 00:23:23,940 --> 00:23:29,880 which is at least like semi sort of structured, and that has not existed. So the foundation is

249 00:23:29,880 --> 00:23:36,510 kind of building something you did. When did you make the JSON? Sources? Why did you do not

250 00:23:36,540 --> 00:23:47,880 recently, last year, I feel like yes, so you wouldn't, I think we would like more of that

251 00:23:47,880 --> 00:23:52,260 first, like, first of all, so that's wanted at least our JSON file, and then at least it should

252 00:23:52,260 --> 00:23:57,390 like, go into some database world so that this tool that the foundation is building, so it can

253 00:23:57,390 --> 00:24:03,630 at least talk to something. And the other thing that is happening is, at least with the

254 00:24:03,630 --> 00:24:08,430 Enterprise product, which I know has all their all kinds of opinions about the foundation

255 00:24:08,430 --> 00:24:16,830 charging for, you know, better, more robust access, the, you know, the kind of clients that

256 00:24:16,830 --> 00:24:21,840 care, so like Google, Apple, Microsoft, whatever. Microsoft doesn't pay, by the way.

257 00:24:22,290 --> 00:24:30,390 This is actually when I say, I don't know if I'm allowed to say this, but something I'll talk

258 00:24:30,390 --> 00:24:36,900 whenever it's kind of being streamed. What's kind of interesting is there are some companies

259 00:24:37,140 --> 00:24:43,020 where the entertainer because there's a lot of API access, and then the idea is like, Well, how

260 00:24:43,020 --> 00:24:47,730 do you make some of the bigger companies pay? So one of the interesting things is you you can set

261 00:24:47,730 --> 00:24:53,310 a threshold, but above a certain kind of rate access, you should say, right? And so there are

262 00:24:53,310 --> 00:24:57,870 some companies who are like, Okay, we want to be good as citizens, we will pay and there are

263 00:24:57,870 --> 00:25:03,540 other companies that are like, Oh, this is your Rate kind of like access will do that. I was

264 00:25:03,540 --> 00:25:09,300 one. And so they get, they get just underneath, and then they don't pay. So there's a variety

265 00:25:09,330 --> 00:25:18,270 of, of activities in terms of the large big players on in the tech space and how they

266 00:25:18,330 --> 00:25:25,290 interact with the Enterprise product. And what's what's nice about enterprise is because a lot of

267 00:25:25,290 --> 00:25:32,970 the larger players care about the quality of the information, they would also like to see kind of

268 00:25:32,970 --> 00:25:39,630 more structured access with regard to the citations, and also to the domain reliability.

269 00:25:39,870 --> 00:25:46,830 And so my other our other big wish list that is floating around the community is actually

270 00:25:46,830 --> 00:25:53,430 creating a giant master database of all the citations, right? Because right now, one of the

271 00:25:53,460 --> 00:26:00,540 problems is if you have a debate on a site on us on a sort of source on one article page, and

272 00:26:00,540 --> 00:26:05,310 like you just you decide it's not generally reliable, there isn't really a good systematic

273 00:26:05,370 --> 00:26:12,210 way to find like, whatever the hypothetical 20 other articles that cite that, that cite that

274 00:26:12,210 --> 00:26:18,870 source. And there is sort of like a good reason why you would want to systematically kind of

275 00:26:18,870 --> 00:26:24,840 look at the body of citations in the weaver it's not as a very large software product that is

276 00:26:24,840 --> 00:26:28,380 beyond the scope of like, you know, I think the volunteer community, but as much as like, we

277 00:26:28,380 --> 00:26:33,960 want to make noise about it as the community to the foundation. I think I think I personally

278 00:26:33,960 --> 00:26:39,330 think that's a good thing. I know, this is also like a big debate of whether or not wiki wiki

279 00:26:39,330 --> 00:26:47,700 data is scalable enough to handle a lot of the citation data. So I think so it could be or as

280 00:26:48,150 --> 00:26:56,010 or could not be, I think, now. Yeah, and there are perennials versus there's like, one for

281 00:26:56,010 --> 00:27:00,690 Krishna music, there is one for video games, there's one. So communities actually keep their

282 00:27:00,690 --> 00:27:05,310 own perennial sources. Listen, we think that we think that is a good thing, right. And, and so

283 00:27:05,310 --> 00:27:12,030 whatever kind of structured data format, you go, and you develop, there's in my, in my view of

284 00:27:12,030 --> 00:27:16,890 the world, there is not like, one list to rule them all. It is very, very context dependent.

285 00:27:17,040 --> 00:27:23,700 And you have to develop a data structure that allows you to say like this sore, this domain,

286 00:27:23,730 --> 00:27:27,690 the other problem is like we only can right now evaluate things on a domain level, which is,

287 00:27:27,690 --> 00:27:30,810 which is problematic when when you're thinking like the domain that you're looking at is

288 00:27:30,810 --> 00:27:36,090 YouTube, right? So is this video reliable, or if it's come from the BBC, is that reliable,

289 00:27:36,090 --> 00:27:42,450 obviously, it can vary significantly, depending on like, the sort of account within a within a

290 00:27:42,780 --> 00:27:50,010 platform. So this is something that we're very well. And then also, as we know, that even

291 00:27:50,280 --> 00:27:56,190 domains within across time and across topics vary in reliability, according to the Wikipedia

292 00:27:56,190 --> 00:27:56,880 consensus.

293 00:27:58,410 --> 00:28:01,620 I just want, I do want to go back to what you were talking about Daniel, so there was a

294 00:28:01,620 --> 00:28:06,300 request last year, kind of along the lines, what you were saying, which is to kind of warn when

295 00:28:06,300 --> 00:28:11,760 people are adding a link on a spam blacklist. So instead of the user using that link, and then

296 00:28:11,760 --> 00:28:16,320 going to save the age and then getting blocked, have a more proactive warning as you're doing

297 00:28:16,320 --> 00:28:21,480 it. So that actually is currently in progress by the Foundation, I believe they'll start with

298 00:28:21,480 --> 00:28:26,010 spam blacklist, but there's a lot of potential for that to kind of grow to support other lists

299 00:28:26,010 --> 00:28:30,630 of links that aren't just the spam blacklist. So the good news, is there is some progress being

300 00:28:30,630 --> 00:28:31,920 made there and that space.

301 00:28:33,210 --> 00:28:39,570 Yes. Yeah. More more noises good along this front, obviously.

302 00:28:41,310 --> 00:28:44,730 Those are very good thoughts. And I wish I knew the answer you're gonna go for

303 00:28:44,760 --> 00:28:49,980 I mean, I can say some that the perennial source of let's just say, the idea of like, looking at

304 00:28:49,980 --> 00:28:54,510 the pattern and perennial sources, what can we tell over time? You know, based on what happened

305 00:28:54,510 --> 00:28:58,140 in the past, and what's going forward. So the first part is the perennial sources list, by

306 00:28:58,140 --> 00:29:06,750 some standards is like tiny, I think it's less than 500 domains. And so our take on it, or my,

307 00:29:06,780 --> 00:29:15,900 my personal take, this is not a sort of a wiki wide take is that we actually need to go through

308 00:29:16,020 --> 00:29:22,290 and kind of force at least tentative evaluations for a lot more media sources, the reason that

309 00:29:22,290 --> 00:29:26,520 these come up, it's because they're repeatedly cited, or they're repeatedly in dispute, you

310 00:29:26,520 --> 00:29:31,350 know, you know, within the community, and that's how they kind of get on to list because there

311 00:29:31,350 --> 00:29:36,090 was some amount of discussion, but it was interesting, if you look at historical

312 00:29:36,090 --> 00:29:42,300 newspapers, let's say going back No, maybe they're maybe they're reliable in certain ways.

313 00:29:42,300 --> 00:29:47,070 Maybe not because, you know, obviously even institutional news bias in over the last half of

314 00:29:47,070 --> 00:29:57,780 the 20th century is debatable, but one thing that is notable is many things on our only side

315 00:29:57,780 --> 00:30:03,360 of on Wikipedia, if they're digitally available. And so you're really only looking at new sites

316 00:30:03,360 --> 00:30:11,790 that are, you know, post 1996, when the web kind of became have started growing and becoming of

317 00:30:11,790 --> 00:30:19,920 age. So that's one chance, you don't have archival articles that are available. And if you

318 00:30:19,920 --> 00:30:24,750 are Wikipedia editor, I'm gonna make a plug for the wiki library. If you are an active editor,

319 00:30:24,750 --> 00:30:28,860 you have access to I think newspapers.com and a couple of other sources. ProQuest, I think is

320 00:30:28,860 --> 00:30:34,770 another thing. But in general, your average kind of Wikipedia editor does not necessarily have

321 00:30:34,770 --> 00:30:46,140 access to pre 1996. Sources. So that's a second thought. And then the third thought would be the

322 00:30:46,140 --> 00:30:53,400 source, the sources that are reliable, are, again, sort of a consensus driven and are our,

323 00:30:53,730 --> 00:30:59,640 you know, related to the context of our times. So I'm, like, a very simple example is in New

324 00:30:59,640 --> 00:31:04,260 York, the Los Angeles Times, for example, published an article a couple of years ago, my

325 00:31:04,260 --> 00:31:09,930 friend was like an editor that apologized for how they covered rates, you know, LA Times

326 00:31:09,930 --> 00:31:16,830 historically, kind of, you know, reliable. But even now, and you know, it was like a post,

327 00:31:17,100 --> 00:31:22,050 George Floyd thing, acknowledged the failings that they've had in their past. And I think, as

328 00:31:22,140 --> 00:31:25,770 with the mirror of history, looking back, you're always going to have something like that in

329 00:31:25,770 --> 00:31:35,310 flux. So I will say this, at least going forward. The, you know, as we all know, local

330 00:31:35,310 --> 00:31:40,320 media is in a crisis, you know, like, whatever I think something like 20 to 30% of Americans are

331 00:31:40,320 --> 00:31:45,150 going to be living in news deserts within the next five years. And that there is there is sort

332 00:31:45,150 --> 00:31:49,500 of efforts from all kinds of philanthropic and maybe government partners to at least try to

333 00:31:49,500 --> 00:31:56,550 shore up information quality on a local level, because they feel like our democracy depends on

334 00:31:56,550 --> 00:32:03,150 it. And I do feel like media plays a small role, because it's one of the places where you have

335 00:32:03,150 --> 00:32:09,780 civilized debates on the Internet still. So I know that doesn't completely answer your

336 00:32:09,780 --> 00:32:12,870 question, but it kind of hit it in like tiny, little ways. The

337 00:32:12,870 --> 00:32:16,320 trends will be very interesting to look at, like, I don't know if we have enough, big enough

338 00:32:16,350 --> 00:32:20,490 data set to like extract anything from there. But we have the history of RSP. Right. So we can

339 00:32:20,490 --> 00:32:25,500 kind of look back and do some sort of analysis analysis, which I think we'd pretty interesting.

340 00:32:27,780 --> 00:32:33,180 Any other thoughts? Questions, comments? Anyone working on any cool projects related to reliable

341 00:32:33,180 --> 00:32:33,930 sources?

342 00:32:35,400 --> 00:32:36,870 Okay, good. Yeah,

343 00:32:36,870 --> 00:32:41,280 definitely. Thank you. I think the comment was about the need for analyzing foreign language

344 00:32:41,280 --> 00:32:45,750 sources. And English, it could be, for example, the resigning foreign language sources, other

345 00:32:45,750 --> 00:32:49,620 editors might not be able to form a good opinion or have thoughts on that source. So having more

346 00:32:49,620 --> 00:32:53,280 kind of like in our language efforts might be helpful there. So the question was, how do we

347 00:32:53,280 --> 00:33:02,820 judge the reliability of a source for systemic list? So generally, Wikipedia editors, we have

348 00:33:02,820 --> 00:33:09,450 this massive guideline page, this policy page describing how we kind of look at sources, and

349 00:33:09,450 --> 00:33:13,650 there's a lot of costs that go into it. But it can be things like, is there an editorial

350 00:33:13,650 --> 00:33:18,660 process for that source? how reputable are the authors at a source you have provenance eat for

351 00:33:18,660 --> 00:33:23,490 the information? Are there known biases about the creation of that source, and so on and so

352 00:33:23,490 --> 00:33:28,830 forth. For the perennial sources, this specifically, that's just a summary of all the

353 00:33:28,830 --> 00:33:35,070 discussions that wikipedians have had related to that source. And those editors presumably would

354 00:33:35,070 --> 00:33:40,890 kind of follow these guidelines we have in place for my tool in particular that I covered sight

355 00:33:40,890 --> 00:33:45,600 unseen. This is kind of a downside of sight unseen is pretty much just me, right? So I'm

356 00:33:45,600 --> 00:33:50,700 just kind of like looking at sources and kind of trying to gauge and as, as well as I can, which

357 00:33:50,700 --> 00:33:55,470 categorizations as far as falls under, but that's not perfect, that's just me. So kind of

358 00:33:55,500 --> 00:34:00,210 being able to crowdsource this sort of these lists will be a lot more valuable instead of

359 00:34:00,210 --> 00:34:04,260 just relying on you know, a few people to do it. Ryan

360 00:34:08,909 --> 00:34:18,029 wonder what? That labeling for people or the lack of resolve ambiguity, when in fact, most

361 00:34:18,029 --> 00:34:26,309 things are gray or whenever the color is misleading? Which a lot of them are broken down

362 00:34:26,309 --> 00:34:34,649 in separate section like boxes. But there's a lot of nuance that is lost, and the existence of

363 00:34:34,649 --> 00:34:43,769 the RSP increased. And this led to discussions. Label and titles are good or bad.

364 00:34:43,800 --> 00:34:49,800 Yeah, I don't. Yeah, definitely. I do think it's a problem that be sort of categorizations

365 00:34:49,800 --> 00:34:55,020 committed people kind of a source completely in one bucket when it shouldn't always be and as

366 00:34:55,020 --> 00:34:59,550 you saw, like, the Reliable Sources page we don't say a source is reliable. We say a sources

367 00:34:59,700 --> 00:35:03,510 J Interview, they're liable. So there's some nuance there and that there's nothing is 100%.

368 00:35:03,870 --> 00:35:09,300 And I have seen users of sight unseen. I've seen some editors remove a source and say, Oh, sight

369 00:35:09,300 --> 00:35:13,650 unseen, said this is unreliable, they have to be like no sight unseen, didn't say that. It's just

370 00:35:13,650 --> 00:35:18,990 providing you categorizations context matters. You have to like do your own homework, none of

371 00:35:18,990 --> 00:35:23,400 these categorizations should be taken as like, you know, 100% truth, they should be viewed as

372 00:35:23,400 --> 00:35:27,300 guides to kind of push you in the right direction. But do you need need to do your own

373 00:35:27,300 --> 00:35:32,880 homework, you need to look at the context of how sources are used. Nothing is ever 100% unless

374 00:35:32,880 --> 00:35:36,480 something is like, you know, really deprecated and blacklisted, then, you know.

375 00:35:38,700 --> 00:35:45,750 I mean, I agree, I don't love it. I don't think in our ideal world, we would have like nuanced

376 00:35:45,780 --> 00:35:53,040 ways of evaluating things. I, we, in our credibility, world credit, con, credibility

377 00:35:53,040 --> 00:36:01,170 coalition we, we've spent a lot of time on, on thinking about this. And in general, do you

378 00:36:01,170 --> 00:36:05,700 think one of the most terrible things you ever want is a score, like a numeric score, like a

379 00:36:05,700 --> 00:36:10,560 single numeric score is is a terrible thing. And that has been an idea that was tossed around by

380 00:36:11,640 --> 00:36:18,450 some folks in the, in a, in some folks in the found Foundation, like in sort of thinking more

381 00:36:18,450 --> 00:36:25,800 from a scalable product perspective? And we're like, Absolutely not. There are lots of

382 00:36:25,800 --> 00:36:33,210 different things that you can, like, our ideal situation, and I think this may be still very

383 00:36:33,210 --> 00:36:39,330 far off from a mathematical perspective is, is things like a nutrition label, right? So it's

384 00:36:39,330 --> 00:36:42,840 not like this food is good for your dispute, it is bad for you, but you know, like

385 00:36:42,960 --> 00:36:48,870 carbohydrates, the number of grams of fat, the number of you know, weight, fat, sugary fat,

386 00:36:48,900 --> 00:36:54,780 carbohydrates and protein and to certain extent, it's the collective information from that

387 00:36:54,780 --> 00:37:01,110 nutrition name label that can give you the information that you need to make a general

388 00:37:01,110 --> 00:37:07,890 assessment about whether or not this is good for you in the context that you eat it in. And if we

389 00:37:07,890 --> 00:37:14,520 can get I mean that you know, the nutrition label took like 25 years to develop. So we given

390 00:37:14,520 --> 00:37:22,110 sort of the crisis of information and quality is still I mean, I would say a decade or so old, we

391 00:37:22,110 --> 00:37:28,410 have we have a long we have a long way to go. But it it is definitely different schools of

392 00:37:28,410 --> 00:37:34,410 thought right, some places just want a score on a scale from one to 100 something personally

393 00:37:34,410 --> 00:37:39,990 where I come from is I think that is it that that causes all kinds of unintended consequences

394 00:37:39,990 --> 00:37:45,750 and all kinds of claps that are have second order consequences that we probably will come to

395 00:37:45,750 --> 00:37:46,080 regret.

396 00:37:50,730 --> 00:37:56,040 Great, yeah, we are at time we have a break now. closing plenary. Hi, thank you. Thank you.

397 00:37:56,730 --> 00:38:03,930 That's one of the things that is we are going out we are doing a presentation when we looked

398 00:38:03,930 --> 00:38:08,220 at the schedule my look, Did someone tell you that we're doing Adnan instead of fact this

399 00:38:08,220 --> 00:38:13,140 morning based on Jake orlo it says slides I want to give credit to Jake and him