A thin but robust solid electrolyte layer is crucial for realizing the theoretical energy density of all-solid-state batteries (ASSBs) beyond state-of-the-art Li-ion batteries (LIBs). This study proposes a simple but practical strategy for fabricating thin solid electrolyte membranes using 5-µm perforated polyethylene separators with 35% open areas as the supporting component, which ensures mechanical robustness for commercial-level cell assembly. The thickness of this frame-based solid electrolyte (f-SE) membrane can be reduced to ≈45 µm, even after coating the Li6PS5Cl (LPSCl) solid electrolyte composite. Despite a slightly lower ionic conductivity compared to that of thick LPSCl pellets, the f-SE membranes show high conductance and low overpotential in Li||Li symmetric cells. Their incorporation into LiNi0.7Co0.15Mn0.15O2 full cells increases the reversible capacity and rate capability compared to those of cells with conventional LPSCl pellets. The f-SE membrane cells exhibit excellent cycling stability over 250 cycles, while maintaining high-capacity retention and Coulombic efficiency. Notably, the f-SE membranes significantly increase the energy density of ASSBs (314 Wh kg−1), exceeding the values reported for sulfide-based cells. These results highlight the crucial role of f-SE membranes in improving the mechanical properties and energy density of ASSBs, thereby contributing to the development of next-generation Li battery technologies.