Celica 5 speed shifter help please

Hi all Im doing a Toyota 5 speed conversion, I have installed the gearbox and found the stick too far forward.  No problem, I bought a cast gear stick extension to relocate it.  The housing cam with a lengthened sleeve which attaches to the selector shaft by  roll pin.  I need to remove the existing sleeve to replace it with the extended one to suit the longer housing.
 
Problem is I punched out the roll pin and move the sleeve back, it hits the back of the fixed extension housing.  Selecting gear and pusing the shaft forward still wont allow the sleeve to slide off the end of the shaft, im about 1 to 2mm short .  
 
Has anyone else ever had this problem and if so how on earth do I get the sleeve off?  All help appreciated

Yep just got the word that is how you need to do it.  As the box is in the car and the floor is already cut looks like its staying that way....