The problem with the epsilon greedy method
I decided I wanted to roll my own AB testing app for Django (https://github.com/crobertsbmw/RobertsAB) when I was finished, I came across this: http://stevehanov.ca/blog/index.php?id=132 Which is a very convincing article on why AB testing sucks and with a few extra lines, you can improve your algorithm to select the best test so you never go back and update your code (yeah right.) I then thought, how many tests does this thing have to run to truly figure out which is best? I made 4 tests with probability of success equalling 1/2, 1/4,…
What it does
In the maker’s words, at launch
I decided I wanted to roll my own AB testing app for Django (https://github.com/crobertsbmw/RobertsAB) when I was finished, I came across this: http://stevehanov.ca/blog/index.php?id=132 Which is a very convincing article on why AB testing sucks and with a few extra lines, you can improve your algorithm to select the best test so you never go back and update your code (yeah right.) I then thought, how many tests does this thing have to run to truly figure out which is best? I made 4 tests with probability of success equalling 1/2, 1/4, 1/5, 1/6 and found that for this algorithm to settle on the best success rate (1/2), it took 91 hits on average with a max of 876 tests. I ran the same test using a standard AB algorithm. Picking whichever test has been tested the least and run that test. It took on average 32 tests to figure out which performed the best with a maximum of 363. On average 3 times better than the greedy epsilon method. I then tried tweaking my success ratios to something a little less dramatic. 1/10, 1/11, 1/12, 1/13. Which just made everything take a LOT longer. The only problem is that in reality you don't know what the best solution is, so you can never know if you have gotten to the "actual" solution. The epsilon greedy method will eventually get there (although you will never know when). And if you are using the standard AB method you will never know if you have arrived at the best option either, especially when we are talking about the difference between 1/20 clicks versus 1/21 clicks. Moral of the story -- AB testing is probably a waste of time. Here is a link to all the tests I ran (python3): https://github.com/crobertsbmw/EpsilonGreedy
Does the same job
all alternatives →
- MFMy first Genetic Algorithm program2015 · ▲19
Wrote a program in FreeBASIC to try and find out how Genetic Algorithms work. It's a quick hack that I wrote to try some theories with because the Wikipedia Article on GAs reads like greek to me. So I built it from the general principles i knew at the time. The "genome" per individual is an array of 9000 integers but I used only monochrome pictures with either black or white; An individual also has a fitness score. There are 1000 individuals in the population and only two are selected to reproduce. So that makes 9 million (yes, MEELION as El Reg would write) data points and the scoring…
- GBGAE/Bingo, A/Bingo split testing for App Engine & Khan Academy2011 · ▲11
We just open sourced GAE/Bingo, a re-imagining of patio11's A/Bingo for use in App Engine (w/ his approval). The lib was built for Khan Academy, where it's currently serving up A/B tests at a few hundred requests/second w/ minimal page load drag. It should get anyone on App Engine (python) up and running and A/B testing in minutes -- if you have trouble please let me know. Does its best to maintain identities as users transition from anonymous to logged-in, persists experiment results to the datastore in the background, etc.…
- ITI tried to reverse engineer Claude Code usage limitsMar 2026 · github.com · ▲5
I hit my Claude Code usage limit yesterday in the middle of regular work. Not doing anything unusual - just a couple of coding sessions. It felt like it came out of nowhere. Then today Anthropic announced they're tightening 5-hour limits during peak hours. It feels like a rug pull is coming and we're all just supposed to accept "you've used too much" with no explanation of what "too much" means. Usage limits have always been opaque. A vague percentage bar. No dollar figure. No formula. No way to plan your work around it. So I built a proxy that captures the rate-limit headers Anthropic sends…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
