#21

Well thanks for getting started on it, sounds like it’s going to take a minute… might be worth waiting until Friday when (I assume) the bandwidth limit gets reset for the host

#22

That’s a good idea. It would be really nice to just grab the real thing with wget instead of trying to get around archive.org weirdness.

#23

Just put this snapshot up: https://web.archive.org/web/20181215121947/http://www.musicdsp.org/

http://musicdsp.radio.af/

And a zip of it: http://musicdsp.radio.af/musicdsp2.zip

The weird incomplete snapshot is over here: http://musicdsp.radio.af/partial-snapshot/

Edit: ah, I see there are just still some absolute URLs in here. Maybe this is complete – for example: http://musicdsp.radio.af/showmany.php

Nope, still seeing lots of broken links… http://musicdsp.radio.af/showone.php?id=98

Ok, I’m done for the moment. Tag, you’re it. (Anyone) :slight_smile:

3 Likes
#24

I could help with front-end, if need be.

Another approach would be to turn scraped content into markdown files, which should be fairly easy since the entries are not too complicated and even done manually, recreate the site using static-site generator (I personally know and have used gatsby, but there are other options) and host it on netlify or now.sh. They both have quite generous free plans and it would lessen the burden of maintaining a server, securing it etc.
Both also offer github integrations, so contributions could be done via pull requests to public repository (given the nature of the site I’d assume that it wouldn’t be a hurdle for potential contributors) - current admin or admins would just need to approve pull request via github interface and new version of the static site would be built and served immediately. If I remember correctly both netlify and now.sh also offer CDN and https for free.

3 Likes
#25

I think putting this up on github really makes the most sense. Generating a site with the data would be easy afterwards but not even really neccesary; github offers all the functionality needed to search for and discuss the code.

A question would be how to format the entries, which on top of the code includes some meta data and comments. I think having each entry be a directory, with the each code snippet in its own file with the correct file type, and another file (yaml or json?) with the metadata, so that it can be easily consumed by other scripts.

3 Likes
#26

I’d like to use github, by the by is their markdown renderer that they use for github pages open-source? Looking far ahead in case github becomes a less attractive option in the future.

Also, we should probably add another directory level to loosely group code by type of signal generator/processor.

#27

I believe this is what they use: https://github.com/gjtorikian/commonmarker

Given that a piece of code could potentially fall into multiple categories, I’m not sure this won’t cause problems down the road?

#28

Respectfully disagree–I think that loose categories (like those that exist on musicdsp.org’s archive today) are rather useful for naively sorting.

#29

There’s also the wiki content to consider: http://musicdsp.radio.af/phpWiki/

(Minus all the spam links of course :P)

This page seems like it has the most worth preserving: http://musicdsp.radio.af/phpWiki/index.php/MusicDspFaqs/?PHPSESSID=a2be73ef0bf0f597274f86e4b80f837a

(Edit: I’m taking these weird mirrors down since we have a different plan now!)

#30

Maybe the new site should just be a wiki?

#31

I like that… wiki with pull requests sounds rather nice to avoid bot-posting and implementation of captcha’s. Still agree with @mateo’s sentiment that it should be a git repo

#32

Well, github supports wikis, so that content could be supported there.

1 Like
#33

Does lock us in specifically to github though. (Unless the github wikis are portable too?)

Edit, yeah looks like the github wikis are git-backed too. That’s cool. So we could maybe do editing / updates directly in github and have some skinned mirror of it running at music.dsp.systems?

#34

Yeah, Github wikis are just repositories themselves.

Yeah, it would be pretty simple to setup a static site generator that pulls the data from github and builds html files whenever there’s a merge to master. It could be hosted in Github itself (via pages) or another service like Netlify

1 Like
#35

I suppose since there seems to be enough interest to make this happen, and plans are starting to form, maybe we should let Bram know?

A database dump would be easier to work with to bootstrap this than a scraped clone of the site…

2 Likes
#36

20 characters of yes definitely!

1 Like
#37

I’ll send him a message today, but keep in mind this current discussion was spawned as a direct result of my initial conversation with Bram.

1 Like
#38

Right on, I’d be happy to get in touch too, just seemed like better to be up front even if he doesn’t want to participate or doesn’t have time to make a dump. Maybe just posting about it on the music-dsp list would be better. Wasn’t trying to drop this on you. :slight_smile:

1 Like
#39

No problem at all, he got back to be rather quickly and I’ll direct him to this topic

1 Like
#40

Gentlemen,

Let’s not scrape that makes no sense. I’m trying to get access again to the database, that should be a lot easier :wink: The server was moved from one machine to another and I get a sneaky suspicion that something scraping or spamming musicdsp.org is killing that machine.

I want to replace to replace the whole site with something static.

I do like the github idea, but contrary to what someone here said that, the wikis in github are NOT part of the repo. So, if musicdsp would be replaced with a wiki it would mean maintaining and cleaning up the wiki from abuse - blergh. So it would just have to be a plan repository with maybe markdown snippets?? Let’s see.

Maybe just plain old HTML with a bootstrap CSS should already be rather nice.

greetings,

Bram

6 Likes