pokemon tcg market analysis fueled by microservices

date: 2024-05-02, tags = ['projects', 'gaming']

the repo for this project can be found here

a millennium problem: i am broke

i wanna play the BDIF for PTCG Standard at time of writing, but i have no money... it's time to collect market pricing data!!! tcgplayer however does not provide do i get the data i need?.. by some combination of web scraping or finding undocumented apis probably

approach 1 (FAILED)

use undetected-chromedriver, seleniumrequests, etc

pain points:

  • unable to use seleniumgrid + docker + kubernetes for at scale scraping b/c seleniumgrid doesnt support custom drivers
  • computationally expensive, realized there was no need to even use chrome drivers (see nodriver as an alternative)
  • unnecessary to use, tcgplayer does not seem to care about the api im hitting
  • too many dependencies for what should be a simple solution

approach 2 (FAILED)

use undetected-chromedriver's cooler, younger cousin (nodriver)

pain points:

  • still not really necessary
  • because it doesn't use selenium, i cant use seleniumrequests to hit their api and pretend im a real human

approach 3 (current)

fall back on just using httpx and asyncio

pain points:

  • easily detected if they wanted to detect me

good points:

  • easier to create a flask app around and deploy
  • faster & simpler

miffy zone dedicated to my loving gf, jasmine e. tran