We introduce AdCraft, a novel benchmark environment for the Reinforcement
Learning (RL) community distinguished by its stochastic and non-stationary
properties. The environment simulates bidding and budgeting dynamics within
Search Engine Marketing (SEM), a digital marketing technique utilizing paid
advertising to enhance the visibility of websites on search engine results
pages (SERPs). The performance of SEM advertisement campaigns depends on
several factors, including keyword selection, ad design, bid management, budget
adjustments, and performance monitoring. Deep RL recently emerged as a
potential strategy to optimize campaign profitability within the complex and
dynamic landscape of SEM but it requires substantial data, which may be costly
or infeasible to acquire in practice. Our customizable environment enables
practitioners to assess and enhance the robustness of RL algorithms pertinent
to SEM bid and budget management without such costs. Through a series of
experiments within the environment, we demonstrate the challenges imposed by
sparsity and non-stationarity on agent convergence and performance. We hope
these challenges further encourage discourse and development around effective
strategies for managing real-world uncertainties.