my opinion and practice is that using sounds i did not record myself is acceptable when:
- the frame of the composition is explicitly that, writing with sounds made by others (ie. collabs, remixes, competitions).
- the project requires (as in, required by a client) specific sounds there is no time or budget to record, or that have been already recorded before i get onboard.
Other than that, making your own sounds from the start is a unavoidable in acousmatic music. Recording is such an important and defining stage !
To me “making music” [^1], came to be because of listening and recording, not really the other way around. At one point i was like, i want to share the listening state i get into when recording. How do i give people access to the lovely details [^2] , the embedding textures etc. The answer was, well, by organizing sound (aka “make music”).
I am probably excessive in that regard, but to me instrument libraries, sample packs, etc are useful only to gain time/money in an industrial process. If working for an industry (or wanting to make industry-calibrated products) then fine, use them. Other than that, i’d think really hard at why i need these ready-to-use things, and if they impart too much on the main discourse of your piece (the subversion/over-use of ready-to-use may be part of the discourse!).
If you feel the extraneous sounds you use are “still there”, that’s because they are, even if only you know it, their original properties are still the basis of what’s to be heard in the end. There’s no escaping a “hidden psychological relation” with your works. (side note: I think it’s a common occurrence to secretly put things that no one will ever know of into a work of art, be it a un-hearable sound, an intimate intention, a secret memory, and so on).
Anyways, if it is a concern, you should try recording. It’s a rewarding practice imho, if only by the fact that it connects you to the world. Microphone techniques, places, displaces, will be the pigments of your subsequent music. It will add a layer of experimentation, trial/error, do←→listen, that is what musique concrète is at heart: composing in a feedback loop with listening (as opposed to writing notes on a paper or “using sounds from real life” which i always found a very reductionist (let’s be polite) definition of that music).
Also you might end with a cabinet of microphones and an obsession for the qualities of sound rather than the message it eventually carries 
[^1]: i mean, “real music” that i can stand behind and assert is “mine” without second thoughts.
Before that (and i don’t repudiate that as it was a main part of my path after all) i mainly tried to emulate genres of pop music that i liked to listen to, but finding a voice within that was way too hard, encumbered by concerns which had nothing to do with music and everything to do with how to fit codes and find a place in the world (which was not there, ahah. Growing up etc.).
[^2]: i’m still wondering when the ASMR crowd will find out about experimental electronics and will flow en-masse towards a field that could use refreshed audiences ^^
ps. do footnotes don’t work? i could swear i saw a post using them recently.