Quercetin is a potent cancer therapeutic agent and dietary antioxidant present in fruit and vegetables. Quercetin prevents tumor proliferation by inducing cell cycle arrest and is a well known cancer therapeutic agent and autophagy mediator. Recent studies showed that drug delivery by nanoparticles have enhanced efficacy with reduced side effects. In this regard, gold-quercetin into poly(DL-lactide-co-glycolide) nanoparticles was examined. In this study, we explored the role and possible underlying mechanisms of quercetin nanoparticle in regulation of antitumor activity in liver cancer cells. Treatment with quercetin nanoparticle effectively inhibited the liver cancer cell proliferation, cell migration and colony formation, thus suppressing liver cancer progression. Quercetin nanoparticle also upregulated apoptosis markedly. Further study suggested that quercetin nanoparticle accelerated the cleavage of caspase-9, caspase-3, and induced the up-releasing of cytochrome c (Cyto-c), contributing to apoptosis in liver cancer cells. Quercetin nanoparticles also promoted telomerase reverse transcriptase (hTERT) inhibition through reducing AP-2β expression and decreasing its binding to hTERT promoter. In addition, quercetin nanoparticle had an inhibitory role in cyclooxygenase 2 (COX-2) via suppressing the NF-κB nuclear translocation and its binding to COX-2 promoter. Quercetin nanoparticle also inactivated Akt and ERK1/2 signaling pathway. Taken together, our results suggested that quercetin nanoparticle had an antitumor effect by inactivating caspase/Cyto-c pathway, suppressing AP-2β/hTERT, inhibiting NF-κB/COX-2 and impeding Akt/ERK1/2 signaling pathways. Our results provided new mechanistic basis for further investigation of quercetin nanoparticles to find potential therapeutic strategies and possible targets for liver cancer inhibition.