Categories
pandas python

Convert columns into rows with Pandas

218

So my dataset has some information by location for n dates. The problem is each date is actually a different column header. For example the CSV looks like

location    name    Jan-2010    Feb-2010    March-2010
A           "test"  12          20          30
B           "foo"   18          20          25

What I would like is for it to look like

location    name    Date        Value
A           "test"  Jan-2010    12       
A           "test"  Feb-2010    20
A           "test"  March-2010  30
B           "foo"   Jan-2010    18       
B           "foo"   Feb-2010    20
B           "foo"   March-2010  25

My problem is I don’t know how many dates are in the column (though I know they will always start after name)

1

348

UPDATE
From v0.20, melt is a first order function, you can now use

df.melt(id_vars=["location", "name"], 
        var_name="Date", 
        value_name="Value")

  location    name        Date  Value
0        A  "test"    Jan-2010     12
1        B   "foo"    Jan-2010     18
2        A  "test"    Feb-2010     20
3        B   "foo"    Feb-2010     20
4        A  "test"  March-2010     30
5        B   "foo"  March-2010     25

OLD(ER) VERSIONS: <0.20

You can use pd.melt to get most of the way there, and then sort:

>>> df
  location  name  Jan-2010  Feb-2010  March-2010
0        A  test        12        20          30
1        B   foo        18        20          25
>>> df2 = pd.melt(df, id_vars=["location", "name"], 
                  var_name="Date", value_name="Value")
>>> df2
  location  name        Date  Value
0        A  test    Jan-2010     12
1        B   foo    Jan-2010     18
2        A  test    Feb-2010     20
3        B   foo    Feb-2010     20
4        A  test  March-2010     30
5        B   foo  March-2010     25
>>> df2 = df2.sort(["location", "name"])
>>> df2
  location  name        Date  Value
0        A  test    Jan-2010     12
2        A  test    Feb-2010     20
4        A  test  March-2010     30
1        B   foo    Jan-2010     18
3        B   foo    Feb-2010     20
5        B   foo  March-2010     25

(Might want to throw in a .reset_index(drop=True), just to keep the output clean.)

Note: pd.DataFrame.sort has been deprecated in favour of pd.DataFrame.sort_values.

4

  • @DSM what would be the inverse of this function. i.e. how would one convert df2 [back] to df

    – 3kstc

    Mar 8, 2018 at 23:02


  • 3

    @3kstc Try here or here. You’re wanting to look into pivots. Possibly pandas.pivot_table(df2,values='Value',index=['location','name'],columns='Date').reset_index().

    – Teepeemm

    Mar 9, 2018 at 18:59

  • 1

    @DSM is there any way to go backwards? Meaning that I have a lot of rows with the same name and I would want all the dates to be on different columns

    – Adrian

    Jan 23, 2019 at 16:44

  • 1

    @Adrian you can unmelt / reverse melt (a.k.a pivoting) on df operations. For more details check this stackoverflow.com/questions/28337117/…

    Mar 12, 2021 at 11:42

25

Use set_index with stack for MultiIndex Series, then for DataFrame add reset_index with rename:

df1 = (df.set_index(["location", "name"])
         .stack()
         .reset_index(name="Value")
         .rename(columns={'level_2':'Date'}))
print (df1)
  location  name        Date  Value
0        A  test    Jan-2010     12
1        A  test    Feb-2010     20
2        A  test  March-2010     30
3        B   foo    Jan-2010     18
4        B   foo    Feb-2010     20
5        B   foo  March-2010     25

    9

    pd.wide_to_long

    You can add a prefix to your year columns and then feed directly to pd.wide_to_long. I won’t pretend this is efficient, but it may in certain situations be more convenient than pd.melt, e.g. when your columns already have an appropriate prefix.

    df.columns = np.hstack((df.columns[:2], df.columns[2:].map(lambda x: f'Value{x}')))
    
    res = pd.wide_to_long(df, stubnames=['Value'], i='name', j='Date').reset_index()\
            .sort_values(['location', 'name'])
    
    print(res)
    
       name        Date location  Value
    0  test    Jan-2010        A     12
    2  test    Feb-2010        A     20
    4  test  March-2010        A     30
    1   foo    Jan-2010        B     18
    3   foo    Feb-2010        B     20
    5   foo  March-2010        B     25
    

    3

    • I know this is few years old now, but while learning the differences on how to use pd.stack() pd.melt() and pd.wide_to_long() i came across to this answer, tested it myself and it just didn’t want to get me the same result (I just got an empty df for res). In the end I figured out I need to add suffix=r".+" to come to the same result. Was it different back then or did it never worked but nobody noticed or cared? Or did I miss something ? It is not that I want to correct here, I just want to be sure I understand what is going on with these commands.

      – Rabinzel

      Apr 12 at 9:09


    • @Rabinzel, I’m not sure what has changed in the functionality. But what I can say is that I tested the code and it worked at the time I wrote this answer. It would be interesting, if it’s true, to know why the suffix parameter is required.

      – jpp

      Apr 12 at 17:23

    • thanks for the reply. Just wanted to verify if the problem is on my side or I am missunderstanding something. After googling a bit, I read several times that wide_to_long expects numerical suffix or it will fail but in the documentation all it says is that suffix="\d+" is the default.

      – Rabinzel

      Apr 12 at 17:34