Can we approximate the upper bound score for the 2015 recsys challenge?

I was one of the first to tweet the opening of the 2015 RecSys Challenge.

After just looking at it and not really doing anything with it for 2 months. I started to compete and see what this competition has to offer. (I’m aiming for a top3 place this year as I was travelling last year and missed all the good stuff)

So let’s start:

Can we approximate the upper bound for the score?
As said in

Selection_015

Selection_016

First we need to look at the training set and extract some informations:

Nb of session with zero item bought: 8,740,033
Nb of session with at least one item bought: 509,696
Ratio: 0.0551038846652

Now let’s take a look at the number of session in the test set : 2,312,431 sessions

Now let’s look at the distribution of the test set compared to the training set.

For each matrix, first column is the distribution of buying event for training. Second column is click event for training. Third column is click event for testing. (All columns are normalised)

Selection_019 Selection_020 Selection_021

With little knowledge, we can say (might always be wrong), that the test set represents fairly the training set.

With that it’s easy to make an estimate of the upper and lower bound

def getBound():
   S = 2312431
   Sb = S * 0.0551038846652 #ratio
   upper_bound = Sb + (Sb**2)/S
   lower_bound = -Sb + (Sb**2)/S
   print [lower_bound, upper_bound] # [-120402.37, 134445.48]
   print "Sb:", Sb # 127423.93112

As we can see here the score is completely dominated by the computation of the Jaccard score. Predicting if a session is a buying session is way less important.

We can conclude that unfortunately the competition will be dominated by how well we can predict the set of item bought during one session. Prediction in a wrong session is not really penalizing (here each wrong session penalize us)

Edit: In fact, it’s not true since there is a file size limit of 25Mb to avoid the approach above (Would have been better to pick a better metric than to limit the file size !)

So better at least predict one or two items per session than predicting nothing. And that is what is happening with having a set of the 10% most popular item and every time they appear in a session, you predict them. http://playwithnlp.blogspot.jp/2014/12/how-to-build-naive-very-naive-system.html

I’m saying that because I created an histogram representing the distribution of the number of product bought during buying session

Selection_013

Data: Counter({1: 259579, 2: 119093, 3: 61040, 4: 32045, 5: 16960, 6: 9262, 7: 4953, 8: 2680, 9: 1542, 10: 976, 11: 559, 12: 315, 13: 222, 14: 145, 15: 96, 16: 66, 17: 53, 18: 39, 19: 16, 20: 13, 21: 4, 22: 5, 23: 6, 24: 6, 25: 4, 26: 4, 27: 1, 28: 1, 29: 3, 30: 2, 31: 1, 36: 1, 39: 1, 41: 2, 53: 1})

And for the fun, the ratio of unique products bought and number of unique items seen during a same session

Selection_014

This graph is quiet funny because you can see different peaks. A huge one at 1, meaning that the person came knowing what he wanted.

And then peak at 0.5, 0.33 and 0.66 (most likely a person looking 2 items, buying 1 – looking 3 items, buying 1 – looking 3 items, buying 2)

I have more awesome stuff that I keep under wraps for now but try to take a look from time to time.

 

PS: I was keen to start this blog after seeing this blog post analysing the competition too.

 

Advertisements

One thought on “Can we approximate the upper bound score for the 2015 recsys challenge?

  1. totó! February 11, 2015 / 06:36

    Interesting article!
    I think you could have a better approximation for the lower bound if you just tested submitting an item and calculated S and Sb from there. I think ratio is (I think) 0.0532914

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s