Overview
About This Document
This document is the product reference for Flipkart Commerce Cloud (FCC) Competitive Intelligence. It covers the crawler and scraping capabilities, the product matching engine, CI reporting and alerting, and integration guidance for publishers and their commercial and technical teams.
Audience: Network publishers, commercial and strategy teams, pricing managers, integration engineers, and e-commerce product managers.
Prerequisites: Familiarity with basic e-commerce pricing concepts is helpful but not required.
Table of Contents
- Introduction
- How the Platform Works
- CI Crawl
- CI Match — Product Matching Engine
- CI Reporting & Alerting
- Onboarding Checklist
- Glossary
1. Introduction
Overview
In e-commerce, your price is only as good as your awareness of the market. The moment a key competitor drops their price on a bestselling SKU — even by 2% — you can lose price-conscious buyers in seconds. Most retailers discover this hours or days later, when conversion data starts to slip. By then, the window to respond has already closed.
FCC Competitive Intelligence (CI) is the market awareness engine built into the FCC Pricing Manager. It continuously monitors competitor websites — scraping product listings, prices, availability, and promotions — and matches competitor products to the client's own catalogue with high precision. The output is a continuous stream of actionable, SKU-level competitive data that feeds directly into the Dynamic Pricer for automated rule-based repricing, and surfaces through a comprehensive set of dashboards, reports, and real-time price alerts.
CI is built on Flipkart's 15+ years of large-scale e-commerce intelligence capabilities. It handles the full complexity of modern web scraping — IP rotation, browser fingerprinting, captcha bypass, login simulation, site definition management — so pricing teams get clean, reliable competitor data without needing to manage a scraping infrastructure of their own.
Expected Business Impact
| Outcome | Expected Lift |
|---|---|
| Revenue uplift from closing price gaps on Key Value Items | +3–8% |
| Reduction in price-driven conversion loss | -15–25% |
| Time-to-reprice on competitor price changes | From hours/days to hours/same-day |
| Reduction in manual competitive analysis effort | 60–80% |
Who This Platform Is For
| Persona | Description |
|---|---|
| Network Publishers | Retailers and marketplaces needing continuous, accurate, and automated competitive price monitoring across their catalogue |
| Commercial & Pricing Teams | Teams who need SKU-level competitor prices to make informed daily pricing decisions — fed directly into Dynamic Pricer rules |
| Strategy & Insights Teams | Teams who want to understand competitive positioning at category, brand, and Business Unit level through structured dashboards and reports |
| Integration Engineers | Technical teams connecting CI data to the Dynamic Pricer, the client's analytics platform, and downstream alerting systems |
2. How the Platform Works
FCC Competitive Intelligence operates in three sequential stages: Crawl, Match, and Reports. The Crawl stage continuously retrieves competitor product data from the web. The Match stage identifies which competitor products correspond to the client's own catalogue items. The Reports stage converts matched data into decision-ready insights, dashboards, and alerts — and feeds matched competitor prices directly into the Dynamic Pricer as a live rule input.
Key functional concepts:
Two crawl modes. Search/Ripper mode discovers competitor listings for a given product or category — fast and suited to high-priority SKU monitoring and assortment coverage. Product Crawl mode directly accesses specific competitor product pages for deeper data including geo-specific availability, promotions, and variant details.
Multi-modal matching. FCC CI uses three complementary matching algorithms — Text, Attribute, and Image — to maximise both precision and recall across the client's catalogue. Image matching, powered by a deep learning model, captures matches where product titles differ across platforms, improving coverage significantly on private label and variant-heavy categories. LLMs are also employed in the matching pipeline to interpret product context holistically and produce high-precision matches at catalogue scale.
Human-in-the-Loop (HITL) Feedback. Algorithm-generated matches are validated through a human review interface. Feedback is logged and used to continuously refine match thresholds and improve precision over time.
CI as a live pricing input. Matched competitor prices are not just surfaced in dashboards — they are a live data input to the Dynamic Pricer's rule engine. Rules like "If competitor price < our price, reprice to 2% below" run directly on the CI feed, closing the loop from market intelligence to automated pricing action.
3. CI Crawl
Status: ✅ Live
Overview
The CI Crawl engine is the data acquisition layer of the Competitive Intelligence platform. It is responsible for accessing competitor websites, extracting structured product data (prices, availability, promotions, titles, images), and storing that data for downstream matching and reporting. The crawl engine is purpose-built to run continuously against websites that actively detect and block automated access — using dual-tier proxy management, headful Chrome-based scraping with randomised fingerprints, and site-specific extraction scripts to maintain reliable data delivery at scale.
How It Works
Every crawl setup follows an initial site feasibility check before any data collection begins. The feasibility test validates that the competitor site does not require human verification or login to navigate, and flags any lazy-load behaviours or pop-ups that need handling. Sites passing feasibility are then configured with a proxy group assignment and a site definition file.
Crawl operates in three sequential functions:
1. Search / Ripper — Discovery
Before individual product pages can be scraped, the system first needs to discover the relevant competitor URLs.
- Search looks up the client's products on competitor sites using configured search terms (typically product name and brand). Searches can be executed on the competitor's own search engine or via Google search. The resulting competitor product URLs are scraped and stored alongside the mapping to the originating client product.
- Ripper takes a category-first approach — given a set of competitor category URLs, it scrapes all product URLs within those categories. This is used to build broad assortment coverage rather than targeted product lookups.
2. Crawl — Data Extraction
Using the product URLs discovered through Search or Ripper, the Crawl function accesses each competitor product page and extracts the configured data fields. Extraction logic for each site is defined in a Site Definition (Sitedef) file — a script that encodes the precise selectors and processing logic for that competitor's page structure.
Two sitedef types are supported:
| Sitedef Type | Description | When Used |
|---|---|---|
| Browser Sitedef | Uses JavaScript/CSS selectors to identify and extract data from the competitor page's HTML DOM | Standard web pages where data is rendered in the browser |
| API Sitedef | Makes direct API calls to endpoints discovered on the competitor site and processes the JSON/XML response | Competitor sites that expose internal APIs with full product data |
Data fields extracted fall into two categories:
- Regular fields — standard attributes common across all sites: product name, URL, image URL, price, availability
- Custom fields — site-specific attributes configured per competitor (e.g., Prime eligibility on Amazon, seller rating, pincode-level stock status)
3. Price Refresh — Ongoing Monitoring
Once the initial crawl is complete and products have been matched to the client's catalogue, a Price Refresh schedule is configured. Price Refresh periodically re-crawls matched competitor product pages to retrieve the latest price data, ensuring the client always has a current competitive price signal for pricing decisions.
Scraping Infrastructure
The crawl engine uses headful Chrome (Chrome Dev Tools) for browser-based extraction, maintaining a pool of multiple Chrome versions and user agent strings that are randomly assigned per crawl session. This randomisation, combined with proxy assignment, minimises detection by anti-bot systems.
Proxy Management uses two tiers:
| Proxy Type | Description | Primary Use |
|---|---|---|
| Datacenter Proxy | IP addresses from data centre infrastructure — fast and high-volume | Default for most competitor sites; initial feasibility testing |
| Residential Proxy | IP addresses assigned to real physical devices via ISPs — appear as genuine residential users | Sites with stricter bot detection that blocks datacenter IPs; geo-specific pricing verification |
Proxy pools are grouped into Proxy Groups and assigned to specific competitor sites based on each site's blocking sensitivity. This separation allows independent management and rotation of proxy resources per competitor.
Crawler scalability. The crawl infrastructure is built for horizontal scale — crawl jobs are distributed across a pool of parallel workers via a cloud message queue, making it straightforward to increase capacity as crawl volume grows.
Event Sale Support
For competitor sale events — where prices and promotions change multiple times per day — FCC CI supports an Events pipeline that runs an accelerated crawl cycle for the duration of the sale:
- Extraction — crawls are run against the competitor's sale-specific URLs to discover sale products and their live prices
- Active Monitoring — the crawl pipeline is monitored in real time to ensure completion within the event's time window
- Report Generation — sale-specific competitive reports are generated after each crawl cycle
The Events pipeline supports running sale scrapes for multiple competitors in parallel, enabling simultaneous monitoring across a broad competitor set throughout the sale period.
Anti-Blocking Capabilities
| Capability | Description |
|---|---|
| Dual-tier Proxy Management | Datacenter proxies for standard sites; residential proxies for detection-sensitive sites. Proxy Groups assigned per competitor. |
| Chrome Version & User Agent Randomisation | Multiple Chrome versions and user agent strings are randomly assigned per session to mimic genuine human browsing patterns |
| Headful Browser Execution | Chrome runs in headful (non-headless) mode with randomised flags to reduce detectability by anti-bot systems |
| Sitedef Abstraction | Site-specific extraction logic is encapsulated in versioned sitedef files — when a competitor changes their page structure, only the sitedef needs updating, not the core crawl engine |
| Crawl Failure Alerting | Automated alerts notify the operations team when crawl success rates drop below configured thresholds, with per-site status codes to diagnose failure type |
Configuration Options
| Parameter | Description | Configurable |
|---|---|---|
| Competitor sites | Which sites to crawl, each configured with a site ID, URL, proxy group, and sitedef | Yes |
| Search terms | Keywords used to search for client products on competitor sites (product name, brand, or custom terms) | Yes |
| Data fields | Which product attributes to extract — regular fields (price, title, URL, image) plus custom site-specific fields | Yes |
| Crawl mode | Search, Ripper (category-based), or direct Product Crawl — configurable per site and category | Yes |
| Crawl schedule & frequency | Daily, twice-daily, or event-triggered; KVI products can be assigned higher-frequency refresh cycles | Yes |
| Proxy group assignment | Which proxy pool (datacenter vs. residential, which region) is used per competitor site | Yes |
Business Use Cases
| Use Case | Example |
|---|---|
| Daily competitor price monitoring | Track competitor prices on the Top 1,000 electronics SKUs every morning, with results available in the Dynamic Pricer before business hours |
| Geo-specific competitive analysis | Use residential proxies to retrieve location-specific pricing and availability for the client's top delivery pincodes |
| Competitor sale event monitoring | Run the Events pipeline with 4–6 crawl cycles per day during a competitor's sale, generating hourly price reports |
| New category launch preparation | Run Ripper across a competitor's full category tree ahead of a client's entry into a new vertical |
| Assortment gap analysis | Use Ripper to map a competitor's full product assortment and identify SKUs the client does not carry |
4. CI Match — Product Matching Engine
Status: ✅ Live
Overview
Raw competitor data is only useful when you can reliably map it to your own catalogue. FCC CI Match is the product matching engine that identifies exact and similar competitor products for every SKU in the client's catalogue. It combines three complementary matching algorithms — Image, Text, and Attribute — with a human-in-the-loop feedback mechanism and an automatic refresh cycle to maintain match quality and coverage as both the client's and competitor's catalogues evolve.
How It Works
Matching can proceed in two directions:
Client-first matching — for each client product, the engine searches the competitor catalogue for matching products. This is the default approach for all client catalogues.
Competitor-first matching — the entire competitor catalogue is crawled and each competitor product is matched to the closest client product. Used for smaller vertical competitors where full catalogue overlap analysis is beneficial.
Multi-modal matching combining images, attributes, and LLMs:
At the core of FCC CI Match is a multi-modal matching pipeline that analyses products simultaneously across visual and structured dimensions — combining image signals with attribute data — rather than relying on any single signal. This multi-modal approach is what enables the matcher to find accurate matches even when product titles differ significantly across platforms, which is common for private label items, variant-heavy SKUs, and categories where sellers use non-standard naming conventions.
To generate matches at speed and scale, FCC CI Match employs Large Language Models (LLMs) in the matching pipeline. LLMs interpret product context holistically — understanding that "16GB RAM, M3 chip, Space Grey" and "Apple MacBook Pro M3 / 16 RAM / Space Gray" refer to the same product even when the structured attribute representation differs. This enables high-precision matching to be produced at catalogue scale without the manual threshold tuning that traditional similarity models require.
The three matching signals used are:
Text Match — product titles and descriptions are tokenised and compared between client and competitor products. Fast and effective where naming conventions are reasonably consistent across platforms.
Attribute Match — non-standardised product attributes (colour, storage size, model number, specifications) are extracted and compared. Catches matches that text matching misses due to title variation or abbreviation differences.
Image Match — a deep learning model encodes both client and competitor product images into a shared embedding space. Client images are queried for nearest-neighbour similarity against the competitor image index. Significantly improves recall for private label products, variants, and categories where product images are more distinctive than titles.
Human-in-the-Loop (HITL) Feedback
Algorithm-generated matches are not assumed to be final. The HITL feedback interface allows ops reviewers to:
- Validate matches — confirm that a system-generated match is correct
- Correct matches — override an incorrect match and assign the right competitor product
- Reject matches — flag a proposed match as incorrect and exclude it
Feedback is logged per match and fed back into the matching algorithms to continuously improve precision thresholds and recall over time.
Match Quality Classification
| Match Type | Description | Threshold |
|---|---|---|
| Exact Match | Competitor product is identified as the same product (same model, same specification) as the client's SKU | Configurable per category — high precision threshold |
| Similar Match | Competitor product is identified as functionally comparable but not identical (e.g., same model, different colour) | Configurable — lower threshold with HITL validation recommended |
Auto-Refresh
Matches are not static. As competitor catalogues change — new products launch, URLs change, listings are discontinued — the matching engine runs automated refresh cycles to update the match index without requiring full re-matching. This ensures coverage is maintained without manual intervention.
Business Use Cases
| Use Case | Example |
|---|---|
| Exact product competitive pricing | Match Apple iPhone 15 Pro 128GB Black on the client's site to the same SKU on Amazon for a direct price comparison |
| Similar product benchmarking | Find the nearest competitor equivalent for a private label product with no identical match on any competitor site |
| New product onboarding | Automatically generate competitor matches for new catalogue additions within 24 hours of product launch |
| Match coverage reporting | Ensure 90%+ of electronics Top-1000 SKUs have at least one matched competitor price before a major sale event |
5. CI Reporting & Alerting
Status: ✅ Live
Overview
FCC CI's reporting layer converts matched competitor data into decision-ready insights. It provides a suite of CI-specific reports covering competitiveness trends, price index analysis, product match coverage, and price freshness — all filterable by hierarchy level, competitor, brand, KVI flag, and event. The alerting system monitors for significant competitor price events and dispatches notifications in near-real-time, ensuring pricing teams can respond before the market opportunity closes.
Competitiveness & Price Index Reports
These reports answer the core strategic question: how is the client priced relative to the competition, across every level of the catalogue hierarchy?
| Report | Description |
|---|---|
| Competitiveness Trend | Client pricing position vs. competitors, tracked over time |
| Competitiveness by BU | Competitive analysis at Business Unit level |
| Competitiveness by Supercategory | Competitive breakdown at L1 category level |
| Competitiveness by Category | Competitive breakdown at L2 category level |
| Price Index Trend | Client's price index (ratio of client price to competitor price) over time — values below 1 indicate client is cheaper |
| Price Index by Category | Price index breakdown at L2 category level |
| Brand Level Competitiveness | Competitive pricing position aggregated by brand |
| Product Level Competitiveness | SKU-level competitive analysis with direct price and availability comparison |
Match Coverage & Freshness Reports
These reports track the health of the CI data itself — ensuring the matched competitor dataset is comprehensive and current.
| Report | Description |
|---|---|
| Product Match % Trend | % of active catalogue with at least one matched competitor product, over time |
| Product Match Absolute Trend | Absolute count of matched products over time |
| Product Match % by BU/Category | Match coverage breakdown by product hierarchy level |
| Product Match Absolute by Category | Absolute match count breakdown by category |
| Price Freshness Trend | % of matched products with competitor price data refreshed within a configured recency threshold |
All CI reports support filtering by: Date, Competitor, Business Unit (L0), Supercategory (L1), Category (L2), Vertical (L3), KVI flag, Event (BAU or named sale events), Brand, and Retail/Marketplace channel.
Alerting & Notifications
The alerting framework monitors competitor price events and dispatches near-real-time notifications:
| Alert Type | Trigger | Example |
|---|---|---|
| Price Drop Alert | Competitor drops price on a matched SKU by more than a configured threshold | "Amazon dropped price by 5%+ on 18 matched electronics SKUs in the last 2 hours" |
| Price Parity Alert | Competitor's price crosses below the client's current price | "Competitor is now cheaper than client on 12 matched SKUs in Mobile Accessories" |
| KVI-Specific Alert | Price event on a Key Value Item — elevated priority notification | "KVI alert: Samsung Galaxy S25 now cheaper on a competitor platform than on client platform" |
| Crawl Failure Alert | Operational alert when crawl success rate drops below configured threshold | "Crawl failure on competitor site X — success rate dropped to 62% in last run" |
6. Onboarding Checklist
Overview
The following are some of the high-level milestones of onboarding FCC Competitive Intelligence:
- Competitor and crawl configuration — define the competitor set to monitor, agree on data fields to extract per site, configure crawl schedules, and establish site definitions for each competitor
- KVI and geo setup — label Key Value Items for priority crawl treatment and configure any geo-specific crawl requirements for location-aware pricing data
- Initial matching and validation — run the first text, attribute, and image matching pass across the client catalogue, and validate a sample of matches through the HITL feedback interface
- Match coverage baseline — establish match coverage for priority categories before enabling CI-driven Dynamic Pricer rules; configure match auto-refresh to maintain coverage as catalogues evolve
- Reporting and alerting go live — configure CI dashboards, define price alert thresholds and notification channels, and enable the CI data feed as a live input to the Dynamic Pricer
- Ongoing competitive review — establish a regular competitive review cadence using CI dashboards and plan coverage expansion to additional categories and competitor sites
Glossary
| Term | Definition |
|---|---|
| FCC | Flipkart Commerce Cloud |
| CI | Competitive Intelligence — the systematic collection, matching, and analysis of competitor pricing and product data |
| CSS Selector | A configuration that identifies specific HTML elements on a competitor webpage to extract data from. Used in site definitions to locate prices, titles, and availability information |
| Exact Match | A competitor product identified as the same product as the client's SKU with high algorithmic confidence — same model, same specification |
| Fingerprinting | Generating unique combinations of browser, device, user agent, and session parameters to simulate legitimate user sessions during web crawling |
| HITL | Human-In-The-Loop — a feedback mechanism where human reviewers validate, correct, or reject algorithm-generated match outputs to improve model quality |
| KVI | Key Value Item — hero or price-sensitive products flagged for priority crawl frequency, elevated alerting, and focused competitive monitoring |
| Price Index | The ratio of the client's price to the matched competitor's price for a given SKU. A value below 1 means the client is cheaper; above 1 means the client is more expensive |
| Proxy | An intermediary server used to route web crawl requests, masking the origin IP to avoid detection and blocking by competitor websites |
| Ripper Module | A component of the crawl engine that extracts structured product data (price, title, availability) from unstructured web pages using pre-configured CSS selectors |
| Similar Match | A competitor product identified as functionally comparable but not identical to the client's SKU — e.g., same product model in a different colour or storage tier |
| Site Definition | The configuration that maps the structure of a competitor website — defining which CSS selectors capture each required data point on a given page type |
| Unblocking | Automated techniques used to bypass competitor website security measures, including captcha solvers, headless browser simulation, and session rotation |
© 2026 Flipkart Commerce Cloud — Confidential. For Publisher and Partner Use Only.