oraymw's picture
By: oraymw, Matthew Watkins
Mar 13 2013 4:50am
5
Login or register to post comments
3726 views


Welcome back to Ars Arcanum, the MTGO stats-based column. A few weeks ago, I put out my in-depth Gatecrash draft study. The response was tremendously positive; however, one of the biggest questions that people had about that article was regarding the impact of overdrafting and underdrafting on the win rates of Boros and Dimir respectively. Over the past few months, as I’ve been putting together these articles, I’ve learned not to overestimate the effects of over and under drafting. However, I recognize that it seems like a logical conclusion to assume that the popularity of these decks affects their win rates. I decided that it was important for me to put together an article discussing the topics of overdrafting and underdrafting.

Those of you who are regular readers of Ars Arcanum know that I generally put together large studies about particular formats. It has also always been my goal to eventually take all of these studies and begin using them to piece data in order to put together articles on general limited topics. I couldn’t do this until I had enough studies built up, but I feel like that time has arrived. In this article, I’ll be taking the information from all of the articles I’ve written over the past year, and comparing decks according to their popularity and win rates. My goal is to use this data to help us figure out how over/underdrafting affect the win rates of decks. While I’ll be using data from several different formats, I’ll end up by looking at how we can apply that data directly to GTC.

The Effect of Popularity on Win Ratio

The first question we need to answer is “what effect, if any, does popularity have on a deck’s win ratio?” Several of the comments I received on my GTC article were that it was obvious that the most heavily drafted deck would have a lower win rate than the least heavily drafted deck. But it turns out that that assumption is not as correct as you might think.

In order to test this, I took all the decks that I have studied over the past year, and I calculated their population ratios and their win percentages. The population ratio is a comparison of an archetype’s popularity against the average popularity of decks within its format. I divide the deck’s popularity by the average popularity in the format, and then get a number between 0 and 2. I’ll use the numbers from Gatecrash for an example. In Gatecrash, Boros had a popularity of 25.10%. This means that about 1/4th of all the decks in the field were Boros decks. The average popularity in the format was 19.9%; since there are five guilds, and all but a very few decks belong to those five guilds, their average popularity is very close to 20%. By dividing 25.10% by 19.9%, we get the popularity ratio of Boros which is 1.26. In other words, Boros shows up about 26% more often than the average deck in GTC. On the flip side, we have Dimir, which had a popularity of 13.7%. This means that the popularity ratio for Dimir was 0.69. The reason why this ratio is important is because every format has a different number of statistically relevant archetypes. For example, a format with ten archetypes is going to have an average popularity of close to 10%. By using a popularity ratio, we put all of the archetypes on equal footing, since we are comparing their popularity against their own format.

But what does that mean for a typical draft environment? It depends on how many decks are valid within the format. For example, the typical draft format, like AVR, you’ll have about 10 valid decks, and your average popularity will be just under 10%.. However, if you have a deck like UG Tempo in AVR, which had a population rate of 1.46, then it is showing up at 14%, or 46% more often than a typical deck. Meanwhile, you have WB which had a population rate of 0.5, which meant that it showed up at about 4.8%. In that format, AVR was being drafted about three times as often as WB. Let’s compare this to a format with five viable decks, like GTC. If UG Tempo was drafted as heavily in GTC as in AVR, we would see it have a 30% popularity rate. This means that it would show up about 2.5 times in every draft. If WB were drafted at the same rate, it would have a 10% popularity rate, which means that you wouldn’t necessarily always see that deck in a given draft.

Since we’ll also be comparing the current draft format against all of this data, I should point out the numbers for each of the guilds.

GTC

Boros

Simic

Orzhov

Gruul

Dimir

Popularity

25.10%

22.10%

20.50%

18.10%

13.70%

Win Rate

48%

40%

62%

44%

62%

Pop Ratio

1.26

1.11

1.03

0.91

0.69

 

Once I had had determined the population rate, I then organized all of the decks according to this number. I then determined the average win rates for decks within a certain population category. The next chart shows us one of those categories.

Average Win Rates of the Top and Bottom 50% of Decks According to Popularity Ratio

In this chart, I’ve compared the decks according to their popularity ratios. We see the top half of the list bringing in about a 51% win rate, compared with the 47.5% win rate of the bottom half of the list. This means that the decks that are drafted with a population rate of at least 1 are winning about 51% of the time, while the decks that were drafted at a population rate of less than one are winning about 47.5% of the time.  In lay terms, decks with a higher than average popularity have a higher than average win rate, while decks with a lower than average popularity have a lower than average win rate.

I should note that this is directly contrary to what people assume when they say that Boros has a lower win rate than Dimir because of overdrafting and underdrafting respectively. In fact, this data shows us that decks that are drafted more heavily tend to have a higher win rate, despite the fact that this goes directly against the common wisdom. Most people assume that overdrafting depresses a deck’s win rate, and we would expect to see these numbers reversed if that were true.

Now, one problem with comparing the numbers with the above chart is that it doesn’t distinguish between decks that were drafted at a 1.86 population rate and a 1.1 population rate, or between a .98 population rate and a .37 population rate. In order to get a better idea of the relationship between popularity and win rate, we would want to see how those win rates change at the extremes. In other words, we want to see what happens for decks that are very heavily drafted compared against the win rates for those that are the most underdrafted. The second problem with the above chart is that we don’t see a huge difference between the two numbers. The best conclusion that we could draw is that a deck’s popularity has a negligible impact on its win rate. Luckily, I have another chart that solves both problems. In the next chart, I’ll be comparing decks according to the standard deviation of their population rates. This filters out the extreme edges on both sides of the over/under drafting spectrum.

Average Win Rates of Decks by Popularity Ratio Arranged by Standard Deviation

This gives us a little bit different of a perspective of the relationship between win rates and population rates. The first column shows all the entries that are greater than one positive standard deviation away from the mean, the second column shows all entries that are within one positive standard deviation away from the mean, and so on. In lay terms, the column on the left refers to the decks that are the most heavily overdrafted, while the ones on the right are the most extremely underdrafted.

The thing that ought to jump out at you immediately is that the column that contains the most heavily drafted decks is also the column with the highest win rate. This group only contains decks that were drafted at 36% to 86% more than the average for their format; it means that these decks had to have been fighting for playables with other people that were in their archetype, in virtually every draft. And yet, this is the only column that manages to score higher than fifty percent. And even though it is hard for you to see, the next winningest column is the second one, though its win rate is only higher than column four by the slimmest of margins. In any case, this really does puncture a huge whole in the argument that overdrafting will cause a deck’s win rate to fall. In fact, we are seeing the exact opposite; it would almost lead us to believe that the high win rate was being caused by the high popularity. However, the readers of this column should be astute enough to realize that this is a case where correlation is not causation. It doesn’t take much thought to arrive at a logical conclusion. Magic players are drawn to decks that are more likely to win. Therefore, the decks with high win rates will often be the most popular.

Later on in the article, we’ll touch on the factors that do make a deck have a higher win rate, but it is definitely worth repeating that these charts seem to contain evidence that overdrafting has a negligible effect on a deck’s win rate.

The next point I want to bring up has to do with column three. In this column, we see decks with population rates between 1 and about .64. These are the decks that are being drafted a little bit less than the average deck in their format, but not significantly less. This is also the section with the lowest win rate, coming in at about 47%. Again, we see that Magic players tend to shy away from decks that don’t put up a good amount of wins. That isn’t to say that people will avoid those decks completely, but they will try to get into a deck that has a better shot of taking down some matches.

However, the second most fascinating thing about this chart is column four. In that column, we see decks that are drafted at a ratio of .63 to .37. On average, their population rate is .51, meaning that these decks tend to get played about half as often as the other decks in their format. While this is still the third lowest win rate of all the decks, it is still a bump up from the third column. It follows that when a deck is seriously underdrafted, that can cause a deck to get a little bit of a bounce in the win column. Not much, but enough to make it a much more valid choice. Keep in mind, though, that this is not a huge increase. This data continues to show us that underdrafting has a minimal impact on win rate.

Finally, I want to orient how Gatecrash fits within this chart. Boros, with a population rate of 1.26 is firmly in the second column, while Dimir, with a population rate of 0.69, is definitely in the third column. While Boros seems to be a textbook case demonstrating the success of a deck that is being drafted slightly more than the average deck in the format, Dimir is definitely defying what our numbers would predict for its win rate. The argument that these decks are achieving their win rates as a result of their popularity is, frankly, silly.

The Effect of Win Rate on Popularity

Although we can see that popularity doesn’t have much of an effect on win rate, it is worth questioning how much of an effect win rate has on popularity. In order to communicate that, I have two more graphs. The first one is similar to the first chart from the last section. It separates all of the decks according to their win rates. The 50% most winningest decks are in the left column, while the 50% most losingest decks are in the right column. The numbers in these graphs will reflect the average population ratios of the categories in question.

Average Population Ratios of Decks by Win Rate

We see the same correlation showing up again. The decks that win more than average tend to be drafted a little more highly than those that don’t. This chart really doesn’t give us any new information; it really just confirms what we have already seen.

However, the next chart is a little bit more interesting. In the next chart, we’ll see the decks arranged according to win rate once again, but they will be grouped into categories according to the standard deviation of their win rates. The first column contains decks with win rates between 72% and 60.3%, the second column has decks with 60% through 50%, the third column has decks with 50% through 41%, and the last column has decks with win rates from 40.5% down to 25.5%.

Average Population Ratios of Decks by Win Rate According to Standard Deviation

Again, the second chart paints a different picture of the phenomenons that are shaping a limited environment. The first column is the only real anomaly, so I’ll talk about it last. The other three columns follow exactly what we saw before; when a deck has a high win rate, players tend to gravitate towards playing it more often, while they tend to avoid decks with low win rates. The question, then, is why the highest extreme in winning decks does not have the highest popularity.

This is the first time that we see evidence of overdrafting affecting a deck’s win rate. While we have demonstrated that heavy drafting doesn’t turn a good deck into a deck with a lower than 50% win rate, this chart is showing us that overdrafting does have some effect in suppressing the win rates of good decks. Specifically, we see that it is hard to obtain the highest win rates, between 60% and 72% unless your deck is being drafted a little bit less than average. The lesson that we learn here is that while overdrafting certainly doesn’t turn good decks into bad ones, it can certainly act as a balancing factor to bring the best decks in line with the rest of the format. However, we also see that one of the keys to having an extremely high win rate is to find a powerful deck that is being underdrafted. This is what happened in the first few weeks of AVR when people didn’t realize how amazing UG really was. It was the same sort of thing that happened in OLS when people were underdrafting Black, even though it was clearly the best color in the format. And it is also exactly what is happening in Gatecrash. People are drafting Boros heavily, even though it really isn’t the best deck. Meanwhile, the most powerful decks in the format, Orzhov and Dimir, are being underdrafted, which makes their win rates skyrocket up to 62%.

Change in Archetype Statistics

Now that we’ve compared all of the decks together, I’m going to go back in time to some of my other articles. The format we’ll take a look at is Return to Ravnica. This is a format that was recently on our minds, and it is worth going back and taking a look at how the win rates of the decks in the format changed from the beginning of the format until later on in the format. I’m going to show you both the popularity charts and win rate charts for both articles, and then move on to the discussion. The articles that these charts come from are the Return to Ravnica Draft Overview and the Return to Ravnica Follow up. The first two charts will show popularity, while the second two charts show win rate.

Popularity of Guilds in Return to Ravnica #1

 

Popularity of Guilds in Return to Ravnica #2

Guild Win Rate in Return to Ravnica #1

Guild Win Rate in Return to Ravnica #2

In addition to these charts, I’ve put in a table with the numbers for each of these guilds in each of those studies. Specifically, I’ve included the deck’s popularity, but also its popularity ratio and its win rate.

RTR One

Selesnya

Rakdos

Azorius

Golgari

Izzet

Popularity

26.80%

24.70%

18.20%

15.60%

13.20%

Pop Ratio

1.36

1.25

0.92

0.79

0.67

Win Rate

56.90%

48.99%

43.96%

46.47%

53.41%

 

RTR Two

Selesnya

Rakdos

Azorius

Golgari

Izzet

Popularity

26.00%

18.80%

18.30%

17.30%

16.50%

Pop Ratio

1.34

0.97

0.94

0.89

0.85

Win Rate

51.92%

45.21%

50.55%

54.91%

50.00%

 

In the first study, we see that Selesnya was definitely the most drafted deck. However, it was followed very closely by Rakdos. Azorius was only a little bit behind average on population ratio, Golgari was behind by a little bit more, and Izzet was at the bottom. The feedback for that study was very similar to the feedback for GTC. I was told that the only reason why Izzet was performing higher than Rakdos was because Rakdos was being overdrafted, while Izzet was being underdrafted.

By the second study, we see a major shift in popularity. Selesnya stayed mostly the same, but its win rate dropped by several percentage points. Meanwhile, the popularity for Rakdos went down significantly, while the win rates for every other guild went up. In the case of Izzet, that increase was by a wide margin. Again, if we were to believe the conventional wisdom about over drafting, we would expect the win rate for Rakdos to go up, while seeing an overall loss for Azorius, Golgari, and Izzet. In fact, we see just about the opposite happening. While Izzet did lose a few points, it does make sense considering that it wasn’t that far above 50% in the first place, and its popularity rate increased by 0.18. But for Azorius and Golgari, we saw a substantial increase in win rate. Meanwhile, even though Rakdos fell substantially in popularity, it also saw a three point loss in win rate, which put it at the lowest performing deck in the format.

Rakdos should have seen an increase, but it didn’t. Golgari and Azorius should have fallen, but they didn’t. Obviously, over and underdrafting were not the causes of their win rates in the first place. The question, then, is why those decks did see such changes. For Rakdos, we see that people began to abandon the guild in order to be in either Golgari or Izzet. The problem is that the decks in the format adjusted their strategies in order to beat Rakdos. You couldn’t just rely on curving out and hoping that your opponent wouldn’t be able to deal with your creatures. People drafted things like Dramatic Rescue a little more highly, which was a significant problem for Rakdos whenever they tried to play an aura to get their guys through the stalled boards. For Golgari and Azorius, we see a much more interesting reason for their jump in win rate. These decks both struggled against very aggressive Rakdos decks. They were a little bit slow, and just couldn’t block a steady stream of three power creatures early on in the game. For example, a 2/2 flyer for 3 mana looks really bad against a 3/3 first striker for 3 mana. However, since the Rakdos decks fell by a substantial portion, the Azorius and Golgari decks did not face as much pressure from fast decks. This meant that they could focus a little bit more on Selesnya, which was the most popular deck by a very wide margin at this point.

The key is that both of these decks were good against Selesnya. So, while they were not as good against Rakdos, that was fine because Rakdos had fallen so much in popularity. Meanwhile, they were very good at beating the most popular deck. At that point, Selesnya was often having trouble picking up a steady stream of tokens and populators. This meant that Selesnya wasn’t able to put together the same kind of continuous card advantage that it had early on in the format. Meanwhile, Azorius had the bounce spells to deal with early token creatures. They could keep the Selesnya deck off of a huge populate advantage, and then win by closing out the game with flyers. On the other side, Golgari was picking up a few removal spells, but mostly, they were just very well equipped to win the long game against a Selesnya deck that was light on populate. Golgari could play the long game and then get some scavenge tokens on a significant threat, and then just win the game. Both of these strategies were particularly well suited to their current metagame, because that metagame had shifted as the popularity of Rakdos fell.

This is the key point that I want to make. I have shown that in most cases, over and under drafting do not play a significant role in affecting the win rates of the decks in a format. However, the thing that does make a huge impact is the metagame.

 Let’s compare RTR with GTC for a moment. In my last study, I showed that Boros was being drafted at a popularity ratio of 1.26. That is significantly less than the popularity of Selesnya in RTR, which had a popularity ratio of 1.36. And yet, many players claim that the cause of Boros’s problems is overdrafting. The question, then, is “Why didn’t Selesnya have the same problem?” The deck that Boros matches the most is Rakdos, which fell dramatically in both popularity and win rate by the next study, because it was poorly suited to beat the environment. On the flip side, Dimir put up a very good win rate in Gatecrash. The question is, where will those decks go in the future? Our answer comes from looking at the metagame.

Orzhov is easily one of the best decks in the format. One reason why Orzhov is performing so well is because it is so good against Boros and Gruul. Those decks just have a hard time beating a deck that can interact well in the early game, gain a lot of life in the mid game, and put the nail in the coffin with extort in the late game. As Orzhov picks up more popularity, Boros and Gruul are going to continue to pick up losses instead of wins. However, the best deck against Orzhov is Dimir, which is able to compete with them on extort, since they can link up extort with cipher. While they do this, Dimir is able to build up card advantage and put Orzhov out of the game. Furthermore, there are many games where Dimir can simply ignore Orzhov’s life total and mill them out of the game. If Orzhov picks up more popularity, then Dimir will likely pick up more wins.

The wild card is Simic. A good Simic deck is the best deck against Dimir, and it is one of the best decks against Orzhov. The problem with Simic is that it often has a hard time against Boros. It is just hard for most Simic decks to beat a concentrated in the early game. However, if the popularity of Boros falls while the popularity of Orzhov and Dimir rises, then Simic will be well positioned to take over the metagame.

I should note that all of these predictions are mere possibilities. Predicting the metagame of a format is often like predicting the weather; there are a lot of moving pieces, and the way they interact depends on the actions of so many different people. If there is one thing that I have learned about Magic players over the past few years, it is that while they are sometimes as regular as clockwork, sometimes they completely surprise you.

Conclusion

Thanks for sticking through this article. I do realize that it is a little bit more abstract even than my format overview articles, but I really hope that this helps people put things into a little bit better context when they take a look at limited environments. Here are the conclusions that I came to in the article:

1.       Over and under drafting has a very small impact on the win rates of decks in a draft format.

2.       In general, popularity tends to be dependent on a deck’s win rate.

3.       On average, the very underdrafted decks, those that have a population ratio of under 0.65, tend to pick up a few more wins than they would otherwise.

4.       Compared to averages across many formats, Boros is not being heavily overdrafted, and Dimir is not being massively underdrafted. Their respective win rates are likely a function of their overall strength as guilds.

5.       Decks with the highest win rates, those above 60%, usually only achieve that kind of success if they are a little bit underdrafted with respect to the format.

6.       There is not data to support the idea that overdrafting does not change good decks into bad decks or that underdrafting does not change bad decks into good decks.

7.       The factor that does seem to have a dramatic effect on win rates is the way that a metagame evolves over time. Choosing a deck that is well-positioned against the most popular decks in the metagame seems to be the best way to gain a high win rate.

As always, you can follow me on twitter @oraymw for updates about articles. I’ve also put up a Tumblr account at http://oraymw.tumblr.com/ where I post links to my articles. You can go there and subscribe to the RSS feed, and then you’ll be able to get updates whenever a new article goes live.

Finally, I encourage you to check out the podcast that I do with my buddy Zach Orts, which is called All in the Telling. In it, we look at stories from a professional standpoint in order to get a better understanding of why they are important to the human experience. But mostly, we just talk about what makes awesome stories awesome. You can also follow Zach on twitter at @zvazda.

 

Ars Arcanum Archive


 

10 Comments

Another great article! by Lobster667 at Wed, 03/13/2013 - 11:06
Lobster667's picture
5

Nice work, Matthew! I really enjoy each article you put out, keep up the good work! And looking forward to that new "All in the telling"-episode, too!

While I remain impressed with the usefulness of raw data like this, I think you're a little too fast with your conclusions/impressions this time around; I agree with the basic premise that people probably put too much weight on over- and underdrafting and that it can sometimes serve as a convenient scapegoat for your deck not putting in the results despite drafting the "best deck" (or, what you think is the best deck, at least).

I really liked the population-ratio factor, but I feel like it would have been a much more powerful tool along with what I consider an absolutely crucial factor if we are to discuss over- and underdrafting; color (or guild) depth. One of the reasons I think this is important is that I think that the popularity ratio might actually differ significantly from the "depth ratio", at least in new formats or formats that are opaque or hard to figure out. I feel like Gatecrash is a set that confuses a lot of drafters; both blue guilds, per example, are not completely intuitive to draft properly (I spent some time figuring out the format myself, and I can only imagine there are less experienced drafters out there who spent more time - or haven't "solved" the format yet!).

For simplification purposes, let's say that the "depth ratio", just like the "popularity ratio", must end up totalling exactly 5 (that is, "there are no non-guild decks and there are exactly 8 decks' worth of card depth in an 8-man draft"). This might be a gross simplification but I don't think it makes it a lot less applicable. Your stats tell us that, at an 8-man table, the numbers would look like this:
Boros: Pop-ratio 1.26 - ~2 drafters.
Simic: Pop-ratio 1.11 - ~1,75 drafters.
Orzhov: Pop-ratio 1.03 - ~1,67 drafters.
Gruul: Pop-ratio 0.91 - ~1,4 drafters.
Dimir: Pop-ratio 0.69 - ~1 drafter.

If we go by the (quite faulty) assumption that every guild/color is equally deep, this would clearly be an easy explanation for Dimir's success (and Boros' somewhat lackluster results): This is the simple argument of over- and under-drafting that I think we agree is a gross oversimplification.

However, let us introduce some (purely thought up, by me, right now) color depth ratios. These are guesses off my sleeve of how the color depths of Gatecrash actually look, but aren't meant to be accurate, merely to illustrate my point:
Boros: CD-Ratio 1.1 - ~1,75 drafters.
Simic: CD-Ratio 0.7 - ~1 drafter.
Orzhov: CD-Ratio 1.4 - ~2,25 drafters.
Gruul: CD-Ratio 0.8 - ~1,4 drafters.
Dimir: CD-Ratio 1 - ~1,6 drafters.
If the numbers even roughly match up like this, it becomes apparent that the over-/underdrafted explanation might have at least some merit. Even if more people start drafting Dimir, there's room for quite an increase before the color becomes "overdrafted" though it will certainly get a little diluted.

I don't mean to (and certainly isn't!) disregarding your analysis, I just feel like there are lots of other factors to consider that the framing of these stats could perhaps help illuminate if done a little differently. Another consideration to color depth is the single colors; as an example, it is my impression that both Red and Green have cards that are valuable in both of their guilds; I think Gruul suffers in your stats by the high popularity of Simic and Boros both. On the other hand, the white cards are more firmly seperated by guild (Smite, Dutiful Thrull and Basilica Guards are a lot better in Orzhov, Denizen, Maneuver and Elite in Boros), which means that Orzhov is a little less affected by Boros' popularity (in my opinion, at least).

One of the concern with "color depth" is, of course, how good a deck needs to be to be "good enough": How much depth is needed to support one drafter? I think this problem might be one of the reasons that people keep liking Boros: A "decent" Boros deck (ie one which got eight ninths of a "deck", depth-wise) can still "get there" on speed and good draws some of the time, especially in a draft which is only three rounds. People are slow to learn if their feedback is not consistent. Decks with a more fragile game-plan, like Simic (not getting enough evolve/curve) or Orzhov (not being able to stay alive early on) might be scaring people off a bit faster since their flaws are more obvious even in a short three-round tournament.

I hope you found this comment helpful/insightful and would like to discuss it further, and once more thanks for a really great article!

I don't disagree with you. I by oraymw at Wed, 03/13/2013 - 15:37
oraymw's picture

I don't disagree with you. I realize that there is a lot more complexity to an individual draft format than just looking at population ratio compared with win rate. There are many small factors that influence how different decks perform. The problem is that it is incredibly difficult to quantify those small factors. For example, you brought up the point of depth compared with popularity. The problem is that there is no unbiased way for me to determine that; the only thing I can think of is assigning playability numbers to the cards that can be drafted by certain decks, and then try to determine how a deck's popularity matches up against its depth. This introduces a tremendous amount of bias. There is already enough inherent bias in any kind of study I do, and so one of my biggest goals is always to eliminate as much bias as possible. Doing what you suggest automatically takes it out of the realm of what I try to do in my articles and puts it more into the realm of what I want people to do in the comments of my articles.

In any case, I agree with a lot of your analysis; Gruul suffers because its cards can be more easily poached by neighboring guilds. I would argue that there is also the problem that Gruul is just not all that good anyway, but the fact that its cards often go into Boros and Simic decks certainly hurts. Of course, neither of those two decks are performing all that well, so maybe there is a different problem going on.

The point I was trying to make with this article is that people always try to make reductive arguments about why decks are performing a certain way. They do this, not to understand a format better, but in order to confirm what they were already thinking about a format. This is confirmation bias. People look at my data, and they only look at the things that confirm what they were already thinking. I would challenge people to be a little bit more open-minded and try to actually understand the things I am demonstrating.

Finally, I think that the biggest problem with these studies is that they are outdated very quickly. The metagame of a format changes pretty quickly, and it is incredibly difficult to try to stay right on top of its pulse. I think that the principles in the articles are always applicable, so I recommend that people understand the principles, and then apply them to what they actually see. Again, this is the blueprint vs. map argument. Treat this like a map to help you understand where things are generally happening, but be aware of the immediate events that are happening in a specific draft. Don't treat this like a blueprint that you need to follow precisely.

In any case, thanks for the comments. I do appreciate what you're saying, and even agree with you, but it kind of falls outside the realm of what I'm trying to do with my articles.

Gotcha by Lobster667 at Wed, 03/13/2013 - 16:31
Lobster667's picture

I see where you're going with the "challenge people to think critically about their presumptions about a format" - that's definitely a worthy goal and something a lot of us could learn to do better/more often!

I'm also glad you mention the transitional character of the data (unlike the principles derived from it), as I've run into a couple of people on the WOTC forums who will use that exact approach to the stats you present ("Stats say Orzhov is best so we should pick the Orzhov card here" in community drafts... Eugh).

Keep up the good work!

Great article as always. Just by Doctor Anime at Wed, 03/13/2013 - 23:57
Doctor Anime's picture
5

Great article as always. Just curious, how big of a sample size do you use for something like GTC data?

About 600 decks. Which is by oraymw at Thu, 03/14/2013 - 00:16
oraymw's picture

About 600 decks. Which is about 700 games. It takes about that much in order to get enough statistically relevant data. I mean, I could technically find out a lot of useful data off of closer to 400 decks, but I'd be missing out on data for the less drafted archetypes. Specifically, I need 600 decks to get enough data points to cover the splashing data.

I generally agree with and by xger at Thu, 03/14/2013 - 17:21
xger's picture

I generally agree with and understand everything you are getting at here but there is one thing bothering me which has some small implications.

In the first chart, as far as the explanation goes, it would seem every deck is in consideration { one side being (0,1) the other [1,2) } but the win rates do not add to 1. There are no draws in MTGO so that wouldn't be the distortion, so the chart seems to imply ~1.5% of the wins just up and vanished.

Actually, the explanation is by oraymw at Thu, 03/14/2013 - 22:39
oraymw's picture

Actually, the explanation is pretty easy. You made the incorrect assumption that both sides have an equal number of total decks.

The columns represent an equal number of total archetypes. Those archetypes have different popularity ratios. There are more decks in column one than in column two. This accounts for the discrepancy.

I'm not sure that matters. In by xger at Fri, 03/15/2013 - 14:16
xger's picture

I'm not sure that matters. In the end you had a finite number of decks between the two columns, and in MTGO the average win rate of a closed set of decks (in the sense they only play against other decks within the set) is 50%, so the two columns should still average to 50%. The only thing I can think of to explain the difference if you are including wins for an archetype when it played against a deck that is not in the archetypes (non-guild decks, which would make the chart not a closed set).

How heavily drafted? by Felorin at Fri, 03/15/2013 - 14:47
Felorin's picture
5

Another possible explanation for popular decks continuing to do well could be to theorize "the community is mostly fairly good about jumping out of an archetype if it isn't open enough from their seat". The data can't distinguish between a draft where five people pursued a guild in pack one but three of them switched out, versus a draft where two people went for it and stayed there.

In some sense the guild that gets tried but abandoned a lot is more popular than one where this doesn't happen often. But the correct winnowing down to a number of drafters the carpool can support keeps us from seeing the dilution of win rate we would expect if 4-5 drafters stuck it out to the bitter end with any frequency.

It might be telling to compare results between 8-4s, 4-3-2-2s, and Swiss drafts to see if overdrafting has more impact in some queue types. I will say anecdotally I have the impression that core set Swiss drafts will more often see a color severely over or under drafted than I see In other types of queues. This is just an impressing, though, I haven't applied any statistical rigor to the question.

This is exactly the question by oraymw at Fri, 03/15/2013 - 16:06
oraymw's picture

This is exactly the question that I have about the data. I didn't really address it, because I simply don't have an answer for this, and it is much more complicated than what I can predict statistically.

I should mention, this kind of approach doesn't actually invalidate the data. In many ways, I suspect that it actually reinforces the idea that popularity is very bad at predicting how much a deck is over or under drafted.

The problem is that I haven't figured a way to figure out how this works statistically. Looking at the three different queues is an interesting proposition, but I worry that there may be more variables changing between the different queues. Do some queues favor decks that are easy to play? Does that change the way the results play out? I'll have to think about a way to run these kinds of numbers, but this is exactly the kind of question that I appreciate so much on my columns; the kind that take a thoughtful and multi-faceted look at the statistics.