Notices

1. jartigag (jartigag@mastodon.social)'s status on Sunday, 30-May-2021 21:00:02 CEST jartigag

   recently i started an analysis project, using #lichess data. these screenshots were just downloading this year so far and uncompressing a month 🥵

   now i have to preprocess such a BIG data 😅

   • jartigag (jartigag@mastodon.social)'s status on Sunday, 30-May-2021 21:00:11 CEST jartigag

     in reply to

     6h 😶 fortunately it was done in one night

   • jartigag (jartigag@mastodon.social)'s status on Sunday, 30-May-2021 21:00:11 CEST jartigag

     in reply to

     such a simple "tweak" (much simpler than multiprocessing.Pool and anything else i've tried these days) make a decisive improvement in performance ✌️

   • jartigag (jartigag@mastodon.social)'s status on Sunday, 30-May-2021 21:00:11 CEST jartigag

     in reply to

     from this (1% each ~3h):

   • jartigag (jartigag@mastodon.social)'s status on Sunday, 30-May-2021 21:00:12 CEST jartigag

     in reply to

     found a solution. #grep to the rescue! 🦸

     https://github.com/jartigag/chess-blunders/blob/master/data/raw/pre_preprocess.sh

   • Ekaitz Zárraga 👹 (ekaitz_zarraga@mastodon.social)'s status on Sunday, 30-May-2021 21:00:40 CEST Ekaitz Zárraga 👹

     in reply to

     @jartigag let me tell you something: one of the best ways to reduce the processing time is to remove the prints :

   • jartigag (jartigag@mastodon.social)'s status on Sunday, 30-May-2021 21:13:51 CEST jartigag

     • Ekaitz Zárraga 👹

     @ekaitz_zarraga yeah, i could remove them now that i have an acceptable preprocessing method 👌

     but look at the times in that case: i need at least one print each 3 hours to know that it keeps working! 😂