What you’re saying sounds like you want to replicate the Daisy Seed in your own board.
In order to use libDaisy you’ll need to have all of the peripherical devices like audio codec, SDRAM, etc, to be the same (not to mention the bootloader which is most likely not open).
Is there a specific reason you want to use the libDaisy in case you want to design your own hardware from scratch? Many external devices have C libraries ready.
For example, I’m currently learning the embedded design, and having the platform with many abstractions already implemented makes it easier to focus on the actual application logic, which also makes it easier to use with my own hardware designs where Daisy Seed will be a core.
To make all of this from scratch would be more reasonable in case some peripherical device should be different, but it also means that libDaisy would not support this.
I’m also curious - would the own hardware cost much less considering the PCB manufacturing and non-manual SMT parts placing?