Heritrix Wiki

News

This wiki has been upgraded to [UseMod 1.0]. One new feature is an [RSS feed] of recent changes.

A new editor password has been established to deter spamming. Email any IA team member for details. Additionally, you will now enter the editor password via the ‘Preferences‘ page, into the ‘Administrator/Editor? Password‘ box, rather than being challenged for the password on edit-submits. Only after entering the editor password will you have access to the ‘Edit this page‘ links.

Project Overview

Updated intermittently; may only loosely reflect current priorities

• HeritrixRoadmapFor2005
• ArchiveAccessRoadmapFor2005

Brainstorming

• SettingsFrameworkRefactoring
• CrawlAndQueueDynamics
• FutureWayback
• StreamCrawling
• ExtractingLinkObjectsInsteadOfStrings
• PerpetualCrawlerNode
• SurtOrdering A version has been implemented and added to 1.2.
• AccessTools (future wayback utilities, etc.)
• TrapDetectionIdeas
• MultimachineCrawl
• JmxControlStatus
• NwaToolSet -- Notes from an NWA Toolset 1.0.0 Review.
• HigherThroughputArcWriter -- Ruminations on alternate ARCWriterProcessor?.
• WhyJdk150 -- Reasons for moving to 1.5.0.

Design Notes

• ArcRevision, ArcRevisionCopenhagenDiscussion, ArcRevisionProposal, ArcRevisionProposalStartConversation
• CharsetsAndCrawling
• NewLibraries
• ClassicScopingModel
• NewScopingModel
• CrawlSettings -- Suggested settings. Most have been implemented.
• ChaffControl
• CheckPointing

Crawl Reports

• CrawlerOverview -- Original Project Outline
• CalendarOfEvents -- Lists key crawls.

Miscellaneous

• FrontierInternals -- Description of Default Frontier Operation.
• ArcFileFormat
• ArcReaders
• BloomUriUniqFilter
• ArchivePassProject -- Project to pass through the total archive to find answers to a designated set of questions.
• HeritrixWebapp -- Notes on Heritrix as a WAR.
• CrawlingPerformance -- Includes profiling notes.
• CrawlerBenchmarks
• CrawlTestPlan 1.0.0 Release. TestPlanExecution120, TestPlanExecution140.
• ExpertSettings
• ListOfFeatures (1.0.0)
• JavaFreeCaches -- A look at a few of the Caching implementations.
• EffectOfXss -- Effect of the ‘-Xss‘ java command-line argument.
• WhySourceforge
• Heritrix Workshops
  • HeritrixCanberraWorkshop?, 15-16 Nov, 2004, Canberra/National? Library of Australia
  • HeritrixCopenhagenWorkshop 9-11 June, 2004
  • HeritrixEarlyAccessWorkshop -- The Feb 12-14, 2004, San Francisco Heritrix Workshop

Old Pages

The below pages are no longer current, mostly of historic interest only, or the ideas described have been implemented, or are pages that were started and not kept up.

• ProjectPlan -- No longer maintained. See plans posted to mailing list for high-level goals and sourceforge bug and RFE priorities for short-term priorities.
• WeeklyCrawlNumbers -- No longer maintained.
• CodingStandards -- Superceded by [Developer Manual]
• TeamMembers -- Superceded by [The Team]
• FetchStatusCodes -- Superceded by [FAQ].
• XmlConfig -- Superceded by New JMX-derived Settings System.
• UsageFeedback
• HeritrixDocumentation
• NutchSearchingArcs -- These notes have been made redundant by [nutchwax] work.

Key Links

• [Project Website]
• [Sourceforge Project]
• [Discussion List @ Yahoo Groups]
• [Browse CVS]

Wiki Notes

• [TextFormattingRules] (at the UseMod site)