[Wikidata] Slow response and incomplete result for RecentChange API in wikidata.org

[Wenjie Song]

Dear WikiMedia,

https://www.wikidata.org/w/api.php?action=query&format=json&continue=&list=recentchanges&rclimit=500&rcnamespace=0%7C2%7C4%7C6%7C10%7C14%7C120%7C122%7C828&rcprop=comment%7Cflags%7Cids%7Cloginfo%7Coresscores%7Csizes%7Ctags%7Ctimestamp%7Ctitle%7Cuser
<https://www.wikidata.org/w/api.php?action=query&format=jsonfm&continue=&list=recentchanges&rclimit=500&rcnamespace=0%7C2%7C4%7C6%7C10%7C14%7C120%7C122%7C828&rcprop=comment%7Cflags%7Cids%7Cloginfo%7Coresscores%7Csizes%7Ctags%7Ctimestamp%7Ctitle%7Cuser>

But we've found that this API become very slow recently, the response time
of one API request is between 15s ~ 25s.

Sometimes the API returns an empty recent change list:
[image: SHgNOsVRJPz.png]
Other times the length of the response recent change list is always between
35~37, even if we set the rclimit parameter to 500.

Is there any changes related to www.wikidata.org recently?

Sincerely,
Wenjie

[Thomas Arrow]

Hi,

Thanks for the report!

We've just opened a ticket about this:
https://phabricator.wikimedia.org/T209459#4745495 where you can follow
along for updates.

You'll see that the problem appears to be related to the `oresscores`
parameter. If you omit this then you get a timely response.

Best,
Tom

[Lydia Pintscher]

Post by Wenjie Song
Dear WikiMedia,
We are using Action API to get recent change updates from www.wikidata.org

https://www.wikidata.org/w/api.php?action=query&format=json&continue=&list=recentchanges&rclimit=500&rcnamespace=0%7C2%7C4%7C6%7C10%7C14%7C120%7C122%7C828&rcprop=comment%7Cflags%7Cids%7Cloginfo%7Coresscores%7Csizes%7Ctags%7Ctimestamp%7Ctitle%7Cuser
<https://www.wikidata.org/w/api.php?action=query&format=jsonfm&continue=&list=recentchanges&rclimit=500&rcnamespace=0%7C2%7C4%7C6%7C10%7C14%7C120%7C122%7C828&rcprop=comment%7Cflags%7Cids%7Cloginfo%7Coresscores%7Csizes%7Ctags%7Ctimestamp%7Ctitle%7Cuser>

But we've found that this API become very slow recently, the response time
of one API request is between 15s ~ 25s.
[image: SHgNOsVRJPz.png]
Other times the length of the response recent change list is always
between 35~37, even if we set the rclimit parameter to 500.
Is there any changes related to www.wikidata.org recently?
Sincerely,
Wenjie

Hi Wenjie,

Can you please check and let us know if you are still experiencing the
problem?


Cheers
Lydia
--
Lydia Pintscher - http://about.me/lydia.pintscher
Product Manager for Wikidata

Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnÃŒtzig anerkannt durch das Finanzamt fÃŒr
Körperschaften I Berlin, Steuernummer 27/029/42207.

[Stas Malyshev]

Hi!

Post by Lydia Pintscher
Can you please check and let us know if you are still experiencing the
problem? 

We have a task https://phabricator.wikimedia.org/T202764 which I suspect
describes the same issue. It is still open, and even though WDQS is
running on Kafka in production and thus is not affected by it, I see it
every time I run it on Labs (where Kafka stream is not available). So I
think the issues with RC API on wikidata are still alive.

There's also a parallel issue of
https://phabricator.wikimedia.org/T207718 with RDF fetching, which also
still happens.

[Amir Sarabadani]

Also given that it uses oresscores, we recently fixed some performance
issues caused by it. Do you still have issues with it?

Best

Post by Stas Malyshev
Hi!

Post by Lydia Pintscher
Can you please check and let us know if you are still experiencing the
problem?

We have a task https://phabricator.wikimedia.org/T202764 which I suspect
describes the same issue. It is still open, and even though WDQS is
running on Kafka in production and thus is not affected by it, I see it
every time I run it on Labs (where Kafka stream is not available). So I
think the issues with RC API on wikidata are still alive.
There's also a parallel issue of
https://phabricator.wikimedia.org/T207718 with RDF fetching, which also
still happens.
--
Stas Malyshev
_______________________________________________
Wikidata mailing list
https://lists.wikimedia.org/mailman/listinfo/wikidata

--
Amir Sarabadani
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. [phone]
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/

Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnÃŒtzig anerkannt durch das Finanzamt fÃŒr
Körperschaften I Berlin, Steuernummer 27/029/42207.

[Stas Malyshev]

Hi!

Post by Amir Sarabadani
Also given that it uses oresscores, we recently fixed some performance
issues caused by it. Do you still have issues with it?

Yes, the issues I have listed still happen. My API calls do not use
ORES. E.g. see:
https://logstash.wikimedia.org/goto/63db4ce68fb5da3cdc7828150de10c59