Data Sources

A data source is the knowledge base that powers your AI agents. By uploading various types of content, you train your agents to understand and respond to user queries based on your specific information.

Data Source Formats

When creating a data source, you choose a format that determines how your data will be processed and which agent types can use it. Each format is optimized for specific use cases:

Knowledge Base

The default format for general-purpose data sources. Upload documents, articles, FAQs, product documentation, or any text-based content. This format is used by:

AI Assistant — For conversational support and Q&A
Quiz — For product recommendation quizzes

Knowledge Base data sources support text files, CSV, JSON, and PDF uploads, as well as website scraping for Pro and Business plans.

Order Data

A structured format for order history and purchase data. This format is exclusively used by Recommendation Strip agents to analyze buying patterns and generate intelligent product recommendations.

Order data can be uploaded in multiple ways: standard file upload (for pre-formatted data), AI-assisted upload (for automatic field mapping), or direct import from connected integrations like Shopify, BigCommerce, and WooCommerce. See the Order Data documentation for schema details and upload options.

A structured format for product catalogs with promotion information. Each record represents one product with a variants array (all SKUs for that product). This format is used by agents that display targeted promotional content:

Nudge — For on-page promotional nudges and notifications
Exit-Offer — For exit-intent promotional offers

Like order data, product & promo data supports standard upload, AI-assisted upload, and direct integration import. See the Product & Promotions Data documentation for the complete schema and upload options.

Upload Options for Structured Data

Order Data and Product & Promotions data sources offer multiple ways to add data:

Standard Upload: Upload CSV or JSON files formatted to match the exact schema. Best for accuracy and custom metadata support.
AI-Assisted Upload: Upload your data as-is and let our AI automatically map your fields to the required schema. Convenient but may require verification.
Integration Import: Pull data directly from Shopify, BigCommerce, or WooCommerce. See the Integrations documentation for setup instructions.

File Uploads

You can upload several common file types to create a data source. When a file is uploaded, it's converted into a vector format, which allows our AI to efficiently search and retrieve relevant information. The original file is retained for approximately 24 hours for processing, and the data is then stored as text in our secure database.

Supported File Types

Plain Text (.txt)
Comma Separated Values (.csv)
JSON (.json)
Portable Document Format (.pdf)

📋 Format-Specific File Types

Order Data and Product & Promotions data sources only accept CSV and JSON files, as these formats require structured data for AI field mapping. Knowledge Base data sources accept all file types listed above.

Data Source Storage Limits

Each subscription plan has different total storage limits. These limits apply to your total data storage across all data source files in your account:

Basic Plan: 50MB maximum total storage
Pro Plan: 100MB maximum total storage
Business Plan: 500MB maximum total storage

💡 Need More Storage? Business plan offers up to 500MB of total storage with 100MB included. Overage pricing applies at $1/MB for usage beyond the included amount.

Important Notes

Included Storage: Each subscription plan includes a specific amount of data storage (Basic: 10MB, Pro: 25MB, Business: 100MB). Overage fees apply if your total storage exceeds your plan's included amount.
Account-Wide Storage: Plan storage limits apply to the entire organization account, not per project or per data source.
Overage Pricing: Storage overage is billed monthly: Basic at $2.50/MB, Pro at $2/MB, and Business at $1/MB.
Overage Calculation: Data storage overage is calculated and billed monthly, coinciding with your plan's billing cycle.

Website Scrape Jobs

📋 Plan Requirement

Website scraping is available on Pro and Business plans only. Basic plan subscribers can upload files directly or upgrade to access website scraping.

In addition to direct file uploads, you can provide a sitemap URL to automatically scrape content from your website. This is an efficient way to index a large amount of web-based information for your agents.

Scrape Limitations

URL Limit: We limit the number of URLs scraped to 500 per sitemap to ensure efficient processing and prevent abuse.
Content Accessibility: Our scraper cannot access data behind authentication walls or client-rendered content in single-page applications (SPAs) that require JavaScript execution to display content.
Excluded Paths: Certain common site paths, such as privacy policies, legal terms, and typical logged-in user sections, are automatically excluded as they generally do not provide useful content for AI agents.
Supplemental Data: You are always free to upload additional data as data source files to supplement content gathered from website scrapes.
API Scrapes: We do not currently offer a way to programmatically scrape websites via our Consumer API.