Solid electrolytes (SEs) play an essential role in the development of all-solid-state batteries (ASSBs) due to their exceptional ionic conductivity. However, conventional pelletized SEs with hundreds of micrometers result in highly reduced energy density and suffer from mechanical brittleness, which delay the commercialization of ASSBs. In this study, we present an innovative approach to fabricate SE membranes with perforated Al2O3 nanolayer-coated polyethylene (PE) separator as a mechanical supporter. Al2O3-coated separators exhibit better adhesion with LPSCl particles and better thermal stability. As a result, the SE membrane exhibits thickness of 35 μm while maintaining a superior mechanical tensile strength. Furthermore, when the SE membrane is applied to Li||LiNi0.7Co0.15Mn0.15O2 full-cells, stable cycling performance over 250 cycles, which is comparable to thick LPSCl pellet cell, can be achieved even with higher conductance (>150 mS). Our results highlight the potential of thin and durable SE membranes in contributing to the commercialization of ASSBs.