crestapps-core-file-sources

CrestApps.Core File Sources

A File AI data source (AIDataSourceSourceTypes.File) is an inert bucket with no folder, credential or field mapping. A file source (FileSource record) picks a connector, configures it, and points at the File data source it fills. Several file sources can feed one knowledge base, each on its own schedule. Web crawlers (crestapps-core-web-crawlers) are the same shape for websites.

FileSource (FileSystem) -+
FileSource (Ftp)        -+->  File AI data source  ->  knowledge-base index (typed objects)
FileSource (Sftp)       -+

Each file becomes separately retrievable objects (document, article, text, figure, chart, table) with a page number and a link to its article and document. Data source concepts are in crestapps-core-data-sources; readers and figure processing are in crestapps-core-document-processing.

Registration

using CrestApps.Core.AI.Documents;
using CrestApps.Core.AI.Documents.Endpoints;
using CrestApps.Core.AI.FileSources;
using CrestApps.Core.AI.Ftp;
using CrestApps.Core.AI.Sftp;
using CrestApps.Core.Data.YesSql;

builder.Services.AddCrestAppsCore(crestApps => crestApps
    .AddAISuite(ai => ai
        .AddOpenAI()
        .AddDocumentProcessing(documentProcessing => documentProcessing   // optional: chat uploads, Office readers
            .AddYesSqlStores()
            .AddPdf()
            .AddReferenceDownloads()))
    .AddIndexingServices(indexing => indexing
        .AddYesSqlStores()
        .AddPostgreSQL(
            builder.Configuration.GetSection("CrestApps:PostgreSQL"),
            postgreSQL => postgreSQL.AddAIDataSources())));

// After AddDocumentProcessing, so the HTML reader registered here wins .html.
builder.Services
    .AddCoreFileSources(builder.Configuration)
    .AddCoreFileSystemConnector()
    .AddCoreFtpIngestionConnector()    // CrestApps.Core.AI.Ftp (FluentFTP)
    .AddCoreSftpIngestionConnector();  // CrestApps.Core.AI.Sftp (SSH.NET)

builder.Services.AddCoreFileSourceStoresYesSql(); // or AddCoreFileSourceStoresEntityCore()

var app = builder.Build();

app.AddDownloadKnowledgeFigureEndpoint();