Objectives: Prostate cancer is becoming increasingly common, and aggressive and metastatic growth continues to be the main factor in patient fatalities. Zeylenone (Zey) is a natural plant extract with anti-cancer effects, but its mechanism is unclear. This study aims to decipher the effect and mechanism of Zey in prostate cancer. Methods: After using Zey to inhibit androgen non-dependent prostate cancer cells PC3 and DU145, the survival rate of cells was detected by 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyl tetrazolium bromide (MTT) to select the best dose. The capacity of cells to proliferate, migrate, and invade was then determined through colony formation test and Transwell® migration/invasion assay, respectively. Next, nuclear morphology of apoptotic cells and cell cycle were detected. In addition, the signaling pathways of Wnt/β-catenin (Wnt5a, β-catenin, cylin), phosphatidylinositol 3-kinase (PI3K)/protein kinase B (AKT) [PI3K, AKT, mechanistic target of rapamycin (mTOR)], glycogen synthase kinase-3beta (GSK-3β), and epithelial-mesenchymal transition (EMT) signaling [Vimentin, N-cadherin, E-cadherin, alpha-smooth muscle actin (α-SMA)] were explored using quantitative reverse transcriptase polymerase chain reaction (qRT-PCR). We also conducted a tumor-forming model of nude mice and the tumor volume and weight were measured. Then, hematoxylin and eosin (HE) staining of the tumor tissues was performed and mRNA levels of the above signaling protein genes were also detected in the tumor tissues. Results: Cellular experiments revealed that Zey inhibited both PC3 cells and DU145 cells growth, colonization, invasion, and migration in a dose-dependent manner (p < 0.01). Medium and high doses of Zey significantly promoted PC3 cell apoptosis after 12, 24, and 48 h (p < 0.01), while low doses of Zey significantly promoted PC3 cell apoptosis only after 48 h (p < 0.01). All Zey treatment groups promoted DU145 cell apoptosis after 12, 24, and 48 h (p < 0.05 and p < 0.01), with a stronger effect observed with increasing time and dose. Besides, all Zey treatment groups boosted the ratio of G0/G1 stage cells (p < 0.01), reduced G2/M stage (p < 0.05 and p < 0.01) and S stage cells (p < 0.05 and p < 0.01) in both PC3 and DU145 cells. Mechanism research showed that all Zey treatment groups decreased the mRNA expression of key genes related to the PI3K/AKT signaling pathway (p < 0.05 and p < 0.01), Wnt/β-catenin signaling pathway (p < 0.05 and p < 0.01), EMT-related pathway (p < 0.05 and p < 0.01), and GSK-3β mRNA expression (p < 0.05 and p < 0.01) in both PC3 and DU145 cells. Animal experiments revealed that Zey could reduce the tumor dimensions of prostate cancer mouse tumor-forming model (p < 0.01), promote tumor cell necrosis (p < 0.01), and reduce the key genes mRNA expression of EMT (p < 0.01) and AKT/GSK-3β/β-catenin signaling pathway (p < 0.01). Conclusions: Zey can inhibit the proliferation, migration, and invasion of prostate cancer cells. Its mechanism of action may be linked to the synergistic inhibition of the AKT/GSK-3β and Wnt5/GSK-3β signaling pathways, which further inhibits EMT in prostate cancer cells.