Aggregate API calls in cr_works with multiple DOIs?

[LukasWallrich]

Currently, it seems that cr_works loops over DOIs - rather than requesting all/many DOIs at once ... unless I am missing something? If the API calls are combined, that leads to drastic speed-up and obviously less load on crossref servers.

ChatGPT generated code for me to do that directly (https://gist.github.com/LukasWallrich/9413261130a882a0d9995981fee50743) which is 50x faster on large sets of DOIs. Please let me know if you would appreciate a PR that allows to use cr_works with similar efficiency ... or if I am doing something wrong with rcrossref.

[njahn82]

Hi @LukasWallrich Thank you for getting in touch. Yes, it can take a couple of time to request such many dois. If you want to speed up queries, you might find the async option useful (see the reprex snippet below for 50 DOIs: 8 sec vs >1 min).

Another option would be to use parallel processing. For example, the purrr package has just introduced this feature.

If you try it, please be gentle with the API and don't overload it. In general, I think it would be better to leave it to the user to speed things up outside of this package at their own risk.

library(rcrossref) 
# Get random number of DOIs
my_dois <- cr_r(50)
# Async
system.time(cr_works(query = my_dois, async = TRUE))
#>    user  system elapsed 
#>   2.564   0.080   8.219
# Loop
system.time(lapply(my_dois, function(z) cr_works(query = z)))
#>    user  system elapsed 
#>   4.824   0.064  64.087

^{Created on 2025-08-01 with reprex v2.1.1}

[njahn82]

If you need specific metadata for a very large set of DOIs, you could try using my university library's Google BigQuery copy of Crossref monthly metadata. Access is available to everyone with a GC account.
https://subugoe.github.io/scholcomm_analytics/data.html#current-snapshot-cr_instant
Let me know in case you need some help.

[LukasWallrich]

Excellent, thank you! Will have a look at the library, as we indeed need details on many DOIs Re the original suggestion, isn't it still kinder to send 1 API query for all dois, rather than multiple queries (async or not)?

…

On Fri, 1 Aug 2025, 17:27 Najko Jahn, ***@***.***> wrote: *njahn82* left a comment (ropensci/rcrossref#249) <#249 (comment)> If you need specific metadata for a very large set of DOIs, you could try using my library's Google BigQuery instance. Access is available to everyone with a GC account. https://subugoe.github.io/scholcomm_analytics/data.html#current-snapshot-cr_instant Let me know in case you need some help. — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOK6NGIS7VSSGQTTOLJGGIL3LOIOHAVCNFSM6AAAAACC5C7R7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNBVGEZDIMJYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[njahn82]

Great, I didn't realise that the question was about the filter option. You can supply a named vector of DOIs to cr_work. This drastically reduces the query time with just one API call, very nice!

library(rcrossref) 
# Get 100 random DOIs
my_dois <- cr_r(100)

# Loop cr_works
system.time(lapply(my_dois, function(z) cr_works(query = z)))
#>    user  system elapsed 
#>   9.491   0.110  43.620

# Use async cr_works
system.time(cr_works(query = my_dois, async = TRUE))
#>    user  system elapsed 
#>   5.561   0.082   8.857



# Use filter

## Prepare named vector for filtering
names(my_dois) <- rep("doi", length(my_dois))

system.time(cr_works(filter = my_dois, limit = 100))
#>    user  system elapsed 
#>   0.421   0.004   0.878

^{Created on 2025-08-02 with reprex v2.1.1}

[LukasWallrich]

Thanks for adding that example - very helpful!

[LukasWallrich]

With more DOIs (e.g. 250), the URL becomes too long, eventually failing with
Warning: 414 (client error): /works - <h1>Bad Message 414</h1><pre>reason: URI Too Long</pre>

Maybe worth checking this explicitly and throwing an error specifying that the filter it too long (or adding a note to the example) - but maybe you deem this API response clear enough for the user.