crestapps-core-web-crawlers

CrestApps.Core Web Crawlers

A Web AI data source (AIDataSourceSourceTypes.Web) is a target bucket with no site configuration and no field mapping. A web crawler (WebCrawler record) chooses a strategy (its Source, for example Sitemap), stores that strategy's settings, and points at the data source. Many crawlers can fill one knowledge base, each re-crawled on its own interval. Folders and file servers use file sources instead (crestapps-core-file-sources); the data source model is in crestapps-core-data-sources.

Registration

using CrestApps.Core.AI.WebCrawlers;
using CrestApps.Core.Data.YesSql;

builder.Services.AddCrestAppsCore(crestApps => crestApps
    .AddAISuite(ai => ai.AddOpenAI())
    .AddIndexingServices(indexing => indexing
        .AddYesSqlStores()
        .AddElasticsearch(
            builder.Configuration.GetSection("CrestApps:Elasticsearch"),
            elasticsearch => elasticsearch.AddAIDataSources())));

builder.Services.AddCoreWebCrawlers();
builder.Services.AddCoreWebCrawlerStoresYesSql(); // or AddCoreWebCrawlerStoresEntityCore()

builder.Services.Configure<WebCrawlerOptions>(options =>
{
    options.DefaultMaxPages = 500;
    options.DefaultMaxConcurrentRequests = 4;
    options.DefaultRequestTimeoutSeconds = 30;
    options.DefaultReindexIntervalMinutes = 1440;
    options.ReindexCheckIntervalMinutes = 15;
});

AddCoreWebCrawlers() registers the named HttpClient (WebCrawlerConstants.HttpClientName, gzip, deflate and brotli, redirects followed), ISitemapCrawler, the keyed Sitemap strategy and its descriptor in WebCrawlerStrategyOptions, IWebCrawlerReindexPlanner, IWebCrawlerReindexService, the Web source handler and keyed IAIReferenceLinkResolver, the crawler catalog handlers, WebCrawlerReindexBackgroundService, the shared ingestion runtime, and the Web entry in AIDataSourceSourceOptions.

AddCoreWebCrawlerStoresYesSql() registers IWebCrawlerStore, IWebCrawlStateStore, IIngestionItemStateStore and the knowledge object store. YesSql hosts create CreateWebCrawlerIndexSchemaAsync, CreateWebCrawlStateIndexSchemaAsync and CreateIngestionItemStateIndexSchemaAsync tables (the last one is required even without file sources).