Required features when developing new PFR workflows

[mrcaseb]

• Automatically re-scrape current week on Wednesday or Thursday
• Make it as easy as possible to re-scrape a game/week/season. Currently we have to download the released games csv, modify it and reupload it. A csv could be fine but then it should live in git and not in a release. Also the list should be sorted by newest to oldest.
• Fix variable naming or selection related to [BUG] 2024 data - some games missing, some mis-mapped columns #36 (comment)
• Spread datatypes across multiple runners? This could avoid the variable naming and selection problem above

Unrelated to the actual workflow but the repo needs to be cleaned up. There's a lot of code that isn't in use.

[tanho63]

Make it as easy as possible to re-scrape a game/week/season. Currently we have to download the released games csv, modify it and reupload it. A csv could be fine but then it should live in git and not in a release. Also the list should be sorted by newest to oldest.

We should really be able to pass in a game ID via gha workflow input rather than that dance we do right now, I don't think we knew how to do that when we wrote PFR stuff

[tanho63]

Spread datatypes across multiple runners? This could avoid the variable naming and selection problem above

SEB WE DON'T NEED TO DO MULTIPLE RUNNERS FOR EVERYTHING,,,,,,,,,,,,,,,,,,,,,,,,,,
actually it could be interesting to do one runner per game if you wanted to

[mrcaseb]

Spread datatypes across multiple runners? This could avoid the variable naming and selection problem above

SEB WE DON'T NEED TO DO MULTIPLE RUNNERS FOR EVERYTHING,,,,,,,,,,,,,,,,,,,,,,,,,, actually it could be interesting to do one runner per game if you wanted to

I am actually not sure about that anyways. I wonder if concurrent calls to undercover could cause problems

[tanho63]

SEB WE DON'T NEED TO DO MULTIPLE RUNNERS FOR EVERYTHING,,,,,,,,,,,,,,,,,,,,,,,,,, actually it could be interesting to do one runner per game if you wanted to

I am actually not sure about that anyways. I wonder if concurrent calls to undercover could cause problems

Good question, undercover seems to be limited on our current plan to just 1 concurrent call so we probably should stick with what we have for now

[mrcaseb]

So I guess the most important thing is the ability to trigger rebuild of a week and specific game through workflow dispatch as well as automatic rescrape of current weeks

[john-b-edwards]

We do need to ensure that we are following PFR's scraping rules correctly: https://www.sports-reference.com/bot-traffic.html

Based on this, I think a 4-second sleep is required between each call, which is not currently implemented in our code.

I wonder if this has something to do with some of the missing PFR data issues we've had...