Fermi Paradox - All the aliens went dark !

Suppose the cosmologists have it wrong and dark matter really is composed of baryons.  It's not like cosmologists have a great track record.  Most of their theories have been proven wrong over the past few centuries.  Comparing the track record of mathematicians with cosmologists is pretty much night and day.  When a mathematician says they have a new discovery it's right, they have cred.  A new cosmology theory and $1.50 will buy you a cup of coffee at Starbucks.  So why should a computation on the density of baryonic matter in the universe be one of the precious few ideas they actually got right? If dark matter is baryonic then why is it dark?  Could it be that it's dark because it's actually dominated by Matrioshka brains?  Matrioshka brains are shells which absorb all the light from a star, and all the light which radiates from that shell, and so on until the outer shell is ice cold, let's say for the sake of argument that a typical Matrioshka brain radiates at 2.73 Kelvin so it's in equilibrium with the cosmic background (or maybe the only relevant source of the cosmic background !!)

What if virtually all sentient life evolves in the far outer reaches of a galaxy and galaxies start out as 100% visible matter.  As a few Matrioshka brains are born they begin to send out ships and to convert neighboring stars.  Within a few hundred million years all stars outside of the core of a galaxy are dark.  As galaxies age they "shrink" because the number of visible stars drops, they are being absorbed by the Matrioshka civilizations.  This offers an intriguing observational program, watch for vanishing stars at the very edges of the Milky Way.

The interesting implications of this theory are:  there is no such thing as time travel, and there is no such thing as FTL travel (faster than light).   Either (or both) would mean much faster (essentially instantaneous) conversion of the entire galaxy.

Would there be any hope for us if the first survey ship arrived tomorrow to begin "Matriforming" our solar system?  It would certainly take centuries to convert our solar system, but it might only be a few decades before Earth became uninhabitable.  Could we fight back, would they absorb us into the collective before we had a chance to fight back?  Makes for an intriguing plot.