ASCII by Jason Scott

Jason Scott's Weblog

4 Comments

  1. Dragan says:

    Urrm, since when does the Archive Team care about robots.txt??

  2. Jason Scott says:

    Archive.org is not Archive Team. Archive Team doesn’t care a whit about robots.txt, but we’re an EMT group coming in and saving things by any means possible, meaning we’ll end up with a certain level of completeness but with no thought to long-term storage and standards.

    By removing robots.txt, Archive.org can aim their crawlers at it and add the data to the Wayback Machine, ensuring much more permanent storage in a non-profit, official library and archive. There is no reason for this not to be the case.

  3. Dragan says:

    This was a bit confusing, maybe because this is your personal blog. You already appear in as many different roles as Davd Bowie. Anyway, I am agree, me too, etc!

  4. V says:

    I think a big part of the problem is that, besides being driven by shortsighted profit or convenience motives, people administering these kind of sites in general don’t know what historians do, why seemingly inconsequential data is important for future historians, or the scope of industrial society’s digitization of information and what the loss of that information could mean for humanity’s future.

    You make appeals to pathos (for example, in the case of memorial sites on Geocities), to ethos (in terms of treating users as people rather than customers), and I don’t know how much more an appeal to logos (in terms of explaining what historians do) will do.

    But I can say that I certainly didn’t have this perspective on materials in our era before learning of your efforts.