- Updated Friday, January 15th 2016 @ 23:11:18
Hey everyone, I thought I would make this thread for anyone who wanted to share their strategy. I am curious how you guys approached the problem.
My bot consisted of two parts: A memory and a decision maker.
For every hand that is played, I counted how many raises occurred in the hand between both me and my opponent. With this information, I can calculate the proportion of games in which at least N (N = 0, 1, 2, 3, 4) raises occurred in the game. (So the proportion of games in which at least 0 raises occur is always going to be 100%). I capped N at 4 since most games do not progress beyond 4 raises.
I also kept track of my opponent's actions and kept counts of what he did, given that N raises occurred in the game. With these counts, I come up with the probability that my opponent will call, raise, or fold given that N raises already occurred in the hand.
My bot's memory needs to be initialized with some data. Over the course of testing, I came up with some settings which worked okay. My bot tends to play pretty loose at the beginning of the game and tightens up with more information being observed.
My bot is initialized with two memories, one for playing hands when it is on the button and one for off the button.
My bot doesn't store information about the sizes of my opponent's bets.
When it comes time to make a move, my bot will count how many raises occurred in the hand preflop and postflop. Let's say X raises occurred preflop and Y raises occurred postflop. I will compare these values with my memory and get the proportion of games in which X and (X + Y) raises occurred. (We use X + Y raises because I want to know how many total raises happened) These two proportions give me a sense of how rare my opponent's betting pattern is.
Then I look at all the possible hands that my opponent could have and eliminate them based on these values. Based on the preflop proportion, I'll eliminate the worst starting hands my opponent could have. Then, based on the postflop proportion, I'll eliminate the worst hands that do not match the table cards. The lower the proportions are, the more hands from the total set I will eliminate.
Finally, with this narrower range of hands, I calculate my probability of winning using a monte-carlo simulation. Then my bot follows a set of rules I have made and plays the game by these rules. The set of rules are basically if-statements which raise, call, or fold based on my probability of winning and the probability of my opponent's actions. Some of these rules include conditions for calling, check-raising, value betting, and bluffing.
For simplicity, my bot raises half the pot almost 100% of the time, It will raise 1/4 of the pot at the last round to value bet, and it goes all-in/fold at the end of the game to end it.
- Created Friday, January 15th 2016 @ 01:54:54
Thanks for sharing. Your bot plays very well.
What I did: Inspiration: Will Tipton's 2 volume set: Expert Heads Up No Limit Hold'em. He has GTO solutions for some simple representative cases on various streets and much more. Developed some heuristics based on data in the book. Heuristics are divided by street and position (SB or BB) .
Testing: using omaha engine ran 100 game matches versus a variety of previous versions and some simple strategic bots I generated (examples all-in, all-in flop, always raise, starter bot etc). At the end I had 16 test versions hence would generate 1600 games vs latest version and had some tools to look at results in various details. Would take almost 2 hrs to run the 1600 games.
Generated hand values: just did brute force - took less time than I anticipated. Pre-flop was of course precalculated and saved in a table. Tracked wins, ties and losses with special handling for two cases: no losses and A high straight w/no pair or 3 to suit (otherwise scores too low due to high number of ties).
Kept track of both global folds/check/calls/raises as well as generally on each street per position (SB or BB). Also rough track of 'huge raise' percentages. I had specific code for starter bots as initially it seemed they were a somewhat common occurrence (score about 87% vs them). The most important seemed to be % donk raised and % continuation bet. Besides the 'huge raise', I only had some fixed buckets for various raise amounts (in terms of BBs) without tracking their percentage.
Tried to narrow opponent's range as well based on their betting.
For the all-in preflop players I tried to model the hands I would play based on chart in Tipton's book which plots hand percentage vs % of total stacks remaining - a 'u' shaped graph, get looser as you move away from 50% of the starting total chips. Even so, only score around 63% vs all-in preflop players (such as ApocolypticAces and Carducci) in my testing.
For endgame play I used Tipton's charts for push/fold, but switched in final version to the simpler 'power-index' given in previous link in one of the discussions here - don't know why, just tinkering....
The bot's overall play approximates raise/call/fold percentages given in Tipton's work. It does appear that the very aggressive bots in this tournament such as yours and PBX have been quite successful.
- Created Friday, January 15th 2016 @ 19:19:15
Interesting reading of both of you.
@cowzow: you must be a good poker player? In my heuristic bot versions I also kept basic statistics about the opponent, but that did not help me much; even if we a decent idea how our opponent plays: how to convert that in how we should act?
@Likes2Golf: and you must also be a good poker player to have books on heads up strategy? Or did you buy Tipton's book especially for this competition?
- Created Friday, January 15th 2016 @ 20:42:12
Got the books for this project, although I had some poker books from quite some time ago. It was an interesting challenge to tackle this. Tipton's books are really good if you want to understand heads-up poker. I would guess the very aggressive bots are very exploitable by a good human player, but quite the challenge to defend against with a bot.
- Updated Friday, January 15th 2016 @ 23:08:52
@DaFish: Ah I see. I developed my strategy based on how I try to play poker, but I'm by no means an expert. Most of my experience of poker has been from watching it on TV. I used opponent statistics mostly when calculating rough expected values for actions I take, such as when I raise or call. I guess these concepts roughly translate into the ideas of raising for fold-equity and calling with good pot-odds.
Also one small strategy I implemented is to go all-in on an opponent if he has been reraising me too much. From data I collected about other aggressive bots, I basically set a rule that if my opponent reraises me more than 55% of the time on any given round, I will classify that as "aggressive" and all-in them if I have at least an average hand.
But I think the main thing is that if you have statistics on how your opponent acts, you can translate these into a rough calculation of the expected value of raising, folding, or calling. And I used that to make decisions for my bot.
- Created Sunday, January 17th 2016 @ 07:15:18
I just have a lot of if-statements :C
- Created Sunday, January 17th 2016 @ 15:02:43
I don't know a lot about poker, except for the rules. I never played it and I don't even really like the game. However, I did like to program my bot and to find ways to improve it, based on statistics and simulations.
As most of your bots, the base strategy of my bot QuispelT is based on statistics of my opponent's behavior.
For folding I have some simple hardcoded fold thresholds, which are going up or down based on my oponent's raise behavior. I use exponential moving averages to cater for changes in behavior of my opponent during the game.
For raising, my basic strategy is to raise a random amount (relative to the big blind) with probability equal to my winning probability (so for example if my win probability is 60%, then I raise in 60% of the cases, else I check). Except when my win probability is very high: then in flop and turn I raise up to 6 resp. 10 times the big blind, and I go all in in the river.
There are some exceptions: if my opponent raises aggressively, then my fold threshold is fixed at 77% win percentage. The 77% was calculated by analyzing about 1,000 games. I guess this improved my bot against moderate opponents, but it might make it perform worse against the top bots, because it made my bot predictable.
For preflop all-in opponents, like ApocalypticAces, I have a dedicated strategy, which theoretically gives me close to 65% (calculated mathematically) win rate. Theoretically because it takes a few turns before I really know that my opponent is a preflop all-in opponent. It always folds or calls, depending on my win probability, and my stack and the height of the big blind. I use this strategy with a small veriation also against bots, who go all in a lot in the preflop, but not always. Against those bots, I do raise someimtes to bluff them away.
I also tried the Sit And Go strategy for a while, but that did not really work out, not sure why.
In the last few days before lockdown, I worked on a very cunning plan to detect who my opponent is, based on the time he needs to play his turns, against based on analyzing more than 1,000 games. The detection part was quite successful: After 15 subrounds, 50% was detected correctly, and this increased to about 70% as the game progressed. Especially AKaiAK was easy to recognize, because he spends a lot of time in flop and turn. The hard part however, like DaFish mentions, is what to do with this information. I did not really have a strategy that worked better against AKaiAK or any other relevant opponent than the default strategy. Also it was too late to develop one.
Despite all these great ideas, every new version of my bot seemed to perform worse than the previous one. It went down from top 10 to position 31 in the leaderboard. So last Sunday, after programming like hell the whole weekend, with only 15 minutes to go until lockdown, I decided to revert to a version of a week ago, which was my most successful one until then. It did not have the opponent recognition and not the Sit And Go strategy. I'm afraid it won't make it to the last eight this time, but alas, I enjoyed working on it. Maybe I'll retry some ideas after the finals are over.
- Created Sunday, January 17th 2016 @ 18:04:59
Interesting reading, AdsRiskbot, a pity that your hard work did not result in a better version. Haha, which bot was fastest and which was slowest?
I also played with the idea of sampling individual opponents, try to emulate it using a random forest or similar, and calculate a counter strategy offline using CFRM. And then, in order to fit all the counter strategies in 2 MB, create a random forest that emulates the counter strategy. And then at runtime try to detect the opponent. Using spent time is just genious, I would never have thought of that:) But it was just an idea; way and way too much work to implement, and doubtful if it really would pay off.
- Created Sunday, January 17th 2016 @ 18:58:55
Fastest by far is ApocalypticAces, with 19 ms average in the preflop, and 0 for all other subrounds, but that is to be expected: it should not be very hard to calculate how much to raise in order to play all in.
Slowest is AKaiAK, with 34 ms in preflop, 1,032 ms in flop, 708 in turn and 78 ms in river.
Apparently quite some bots use the type of algorithms, because my detection always confused DaFish, Likes2Golf, BetterCallSoul and ThunderBluff: they are all quite fast, and they spend more time in preflop and river than in flop and turn.
- Created Sunday, January 17th 2016 @ 19:41:27
Interesting...note to self - random delay timer needed in future iteration ;-)
The variance is so large in the game that using the ladder results or position has little value for determining if a change improves play or not. Just for grins I just calculated the winning percentage for the top few programs and it varies from 62% (DaFish and ThunderBluff) to 51% (CArducci - which I think is just an 'allin preflop' player). The 62% bots are recent versions with <500 games played. Of the top 17 bots, all with >1000 games have <60% win rate, and consider that many of those games were probably played against weaker bots.
Curious about 65% theoretical expectation vs allin preflop like ApocolypticAces. If that is just a pokerstove kind of metric comparing a selected range vs ATCs, then I don't think that can be realized due to increasing blinds.
BTW - what is a 'sit and go' strategy?
- Updated Sunday, January 17th 2016 @ 20:57:04
@AdsRiskbot: Dang, that is too bad that your future versions did not do well. Profiling your opponent that way is really clever! I wonder if your technique will prove useful in future competitions. Perhaps it might have been useful in the Warlight 2 competition haha.
@Likes2Golf: Yea I agree, the high variance of the competition made testing very difficult. It's difficult to tell whether or not a new version is better than previous ones. I used local testing to make most of my decisions for changes, but that may be unreliable. I wonder if there is a good technique out there for testing this kind of stuff.
- Updated Tuesday, January 19th 2016 @ 23:37:44
@cowzow: I think the profiling technique is useful in any competition, where a strategy works for most opponents, but not for some specific opponents. I think it's a bit more useful for the poker competitions than for example for Warlight or BlockBattle. By the way: apart from durations, there may be other properties of an opponent's game play, which may be useful for detecting who your opponent is, like % of folds in preflop, or % of raises in preflop.
@Likes2Golf: Sit and Go was mentioned in the "Poker Strategy for Heads-Up?" thread. It is an end game strategy, when big blinds are high relative to the lowest stack. It is explained quite well here: http://www.tournamentterminator.com/tournament-strategy/sng-strategy/sitn-go-heads-up-strategy-sage/.
@Likes2Golf: About 65% expectation vs allin preflop: I did not use a PokerStove tool or anything like that. I calculated it myself, as follows:
Let P(mystack, round) denote the probability that I win a game against an allin preflop player, given my stack and the round number. Stack and round number together determine the optimum fold threshold. The higher my stack and the lower the round number (so the lower the big blind), the higher the fold threshold should be.
If mystack <= 2000, then
P(mystack, round) = pCall * pWin * p(2 * mystack, round + 1) + pCall * (1 - pWin) * 0 + (1 - pCall) * p(stack - lossOnFold, round + 1)
where pWin = the fold threshold = win percentage, below which we fold, and above which we call
and pCall = the corresponding percentage of hands, of which the win percentage >= pWin
The first term corresponds to winning the pot (=doubling my stack) when we call and win with a good hand.
The second term corresponds to loosing the pot (= loosing the game) when we call and loose.
The third term corresponds to loosing the small/big blind when we have a bad hand.
Similarly, if mystack > 2000, P(mystack, round) can be expressed as follows:
p(mystack, round) = pCall * pWin + pCall * (1 - pWin) * p(4000 - 2 * hisStack, round + 1) + (1 - pCall) * p(stack - lossOnFold, round + 1);
pWin can be optimized for each combination of (mystack, round) individually. After doing that, p(2000, 1) is the overall win rate against this kind of opponents. By the way, I did not optimize each pWin individually. Instead, I optimized pWin(n) for each n < mystack / big blind < n + 1. To make a long story short: in the end I had P(2000, 1) = 64.98%.
- Created Wednesday, January 20th 2016 @ 00:30:47
@AdsRiskbot: thanks, I didn't catch the 'sit-n-go' reference. The 'power index' I switched to is exactly the same thing for endgame.
Thanks for description of your calc vs all-in preflop. I tried to model some curves seen in Tipton's book, and since it worked a little better than my other tries, I didn't explore much further. Since last time I changed the all-in preflop code my bot averaged 63.21% over 14000 matches. Of course, there is also some small loss of blinds before you determine you are playing an 'all-in preflop'.