python - shift by partition using groupby -

i have dataframe 1 column shift, on partition rather whole dataframe.

for example, go dataframe:

state, year, gdp ny,2011,100 ny,2012,110 ny,2013,120 ca,2011,70 ca,2012,80 ca,2013,90 

to dataframe:

state, year, gdp, gdp y-1 ny,2011,100,nan ny,2012,110,100 ny,2013,120,110 ca,2011,70,nan ca,2012,80,70 ca,2013,90,80 

so far i've used code this:

grouped = df.groupby("state") state, state_df in grouped:     state_df["gdp y-1"]=state_df["gdp"].shift(1) 

i believe gives me want each state, don't know how combine (basically appending each dataframe below 1 another). how can that?

you can store intermediate dataframes in list , use pd.concat join them together:

grouped = df.groupby("state") l = [] state, state_df in grouped:     state_df["gdp y-1"]=state_df["gdp"].shift(1)     l.append(state_df)  pd.concat(l) out[149]:    state  year  gdp  gdp y-1 3    ca  2011   70      nan 4    ca  2012   80       70 5    ca  2013   90       80 0    ny  2011  100      nan 1    ny  2012  110      100 2    ny  2013  120      110