This paper presents two size-miniaturized quarter mode (QM) and eighth mode (EM) substrate integrated waveguide (SIW) bandpass filters (BPFs), which are embedded with a novel frequency-dependent coupling (FDC) structure. The proposed FDC is implemented as a composition of balanced folding lines and inductive iris. One additional transmission zero (TZ) introduced by FDC between two cavities leads to higher frequency selectivity and better out-of-band rejection. Higher order modes suppression appears by combining the loaded paired open stubs on feeder lines with FDC technique, achieving a wide stopband. Meanwhile, the circuit dimension is further reduced by symmetrically cutting SIW. To validate the novel approach, the frequency-dependent coupling matrix (CM) is implemented to determine characteristics of the proposed structure in theory, QM- and EM-SIW BPFs loaded with FDC have been designed, fabricated and measured. Experimental results illustrate the characteristics of miniaturization and good performance. All results are in good agreement.