Working through Josh Lynch's Masterclass "Creating Stone Wall Materials in Substance Designer" gave me incredible insights into the levels of complexity that make up a realistic material derived from real-world reference. I can highly suggest looking into his stuff. I also feel even more motivated to use my color calibrated 3D scans as reference for future materials.