South Norfolk Council has defended plans for a £6m leisure centre in Diss following backlash from Diss Heritage Trust.