With the continuous improvement of spatial and spectral resolution, the application of multispectral images has greatly increased in remote sensing. However, the amount of image data also increases sharply, which brings great pressure to data storage and transmission. To solve this issue, we propose an end-to-end image compression scheme according to spatial–spectral feature extraction, which can be implemented by a spatial–spectral memory unit (SSMU). Furthermore, to improve the feature extraction capability of this deep network, the multichannel grouping fusion module is adopted to reconstruct and fuse the image features. In the encoder of the proposed compression scheme, the SSMU first extracts spatial–spectral features along the spatial direction and the spectral direction, and the multichannel grouping fusion module extracts the spatial and spectral features of different levels by recombination and fusion of band features of multispectral images. Then, the extracted deep spatial and spectral features are compressed by downsampling. Next, the quantizer and entropy coding convert the data into a compressed bitstream. In the decoder, a reverse process is used to restore the original images. The experiments take the multispectral images of Landsat 8 and WorldView3 as the datasets to verify the superiority of our method and compare it with JPEG2000, 3D-SPIHT, and the CNN-based methods. The results show that the proposed method outperforms the JPEG2000, 3D-SPIHT, and CNN-based methods in PSNR, spectral similarity, and spectral angle mapping metrics at different bit rates.