Frequency-domain mixed-potential integral-equations are commonly used for analyzing scattering from structures residing in layered backgrounds (K. A. Michalski and D. Zheng, IEEE TAP, 38(3), 335–344, 1990). Their traditional direct method-of-moments solution based on LU-decomposition is limited to small problems, however, because it requires O(N2) operations to fill the N × N dense impedance matrix (with a large constant in front of the complexity estimate because of the high cost of computing Sommerfeld integrals), O(N3) operations to factorize the impedance matrix, and O(NrhsN2) operations to find the solution for Nrhs different excitations. Moreover, modern fast iterative algorithms that enabled the solution of extreme problems for homogeneous backgrounds are often inapplicable, inefficient, or must be modified significantly (K. Yang and A. E. Yılmaz, CEM Int. Workshop, 2013) for layered backgrounds because of the more complicated Green functions. In this article, a fast direct solver based on the hierarchical matrix (ℋ-matrix) framework (W. Hackbusch, Computing, 62, 89–108, 1999) is used to accelerate the analysis of scattering from structures residing in layered media.