Whenever an import job is performed, only new records will be updated to the main database.

The following are the important points to consider before starting with import.

*The mentioned time period is in 
- IST for APAC and APAC2 cluster 
- CST for China cluster
- GMT for EU cluster

Duration for data import in the day time (Peak hours)

* 6 AM - 10 PM
Duration for data import in night time (FTP hours)    * 10 PM - 6 AM
No of records allowed in the last 24 hours by direct import 200,000
No of records allowed in the last 24 hours in the day time (Peak hours) day FTP import 200,000
No of records allowed in last 24 hours in night time (FTP hours) by FTP import     1,000,000
Maximum size of a single file (in MB)   200 MB

The configuration of the Data Import job includes the following general steps:

  1. Identify the data that you want to import from the system 
  2. Clean up the data in the system to reduce the idle processing time while importing. This involves deleting previous records and removing blank/invalid/duplicate records
  3. Create and configure templates for each profile that you want to import 
  4. Create different folders for each profile type in the FTP server
  5. Copy data files of each profile to the respective folder of the FTP server. For example, registration data in Registration folder, points redemption data in the Points Redeem folder and so on
  6. Map the fields in the database to the respective column names of the CSV file
  7. Perform a trial import for each profile type
  8. Perform the actual import job

The following sections guide you on how to configure an import job for large data.

Step 1: Select the job type as FTP

  1. On the Settings page, click Master Data Management > Data Import.
  2. Click Configure Data Upload via. FTP

Step 2: Configure the Source FTP Server

Provide read/write access to the FTP server from where the data needs to be imported. Select all profiles that you want to import through the current import job.

  1. Check Enable SFTP if your FTP server is a secure FTP server. Skip this step for a normal FTP Server
  2. In FTP Server Name, enter the URL of the source FTP server  
  3. In the Username and Password boxes, type the login credentials of the FTP server
  4. In Folder Name, type the directory in the FTP server from where the data needs to be uploaded. For example, /import
  5. In Proceeded Folder Name, specify the name of the processed data folder 
  6. Click Submit to save the configuration.
  7. It is recommended to save the data of each type (profile) in different folders.
  8. To create folders on the FTP server for each profile, check the respective checkboxes in Configure Folders for FTP Server field.

You will see the following screen.

Step 3: Set up for data processing

  1. In the No. of rows to be ignored from top, set the number of rows from the top that you want to exclude from importing. For example, if number 2 is set in this field then while uploading data to the server the top two rows will not be considered. This can be used when there are field names, duplicate data or blank data in the top rows
  2. In No. of rows to be ignored from bottom, set the number of rows from the bottom that you want to exclude from importing. For example, if 15 is set, then while importing the last 15 rows will be ignored. You can set this option in case of duplicate/blank/invalid entries in the last rows of the CSV file.
  3. In File Delimiter,  type in the character used to separate each value in the file. For example, a file can have either ',' separated or 'tab' separated or ';' separated values. If ','  is mentioned then values will be separated after every ',' found in the file.
  4. In Enclosure, type the special character used in the file for 'string' data types
  5. In Escape Character, type in the special character that needs to be ignored from the file
  6. Click Submit

Step 4: Choose templates for each data model(Profile)

In the Configure Templates for Data Upload field, choose the desired template for each profile and click Submit.

Step 5: Schedule the import job

As the data size is large, it is recommended to schedule an import job in between 10 pm to 6 am to avoid inconvenience to your stores' operations.

  1. In Status, choose Running
  2. In Chek for Files, choose the frequency of import - Daily, Weekly, Monthly or on a Fixed Date and set the respective time interval. For a fixed date schedule, set the date on which you want to schedule the import 
  3. In Email ids to be notified, enter email ids separating each with a comma to notify about the status updates the import job. The selected recipients are notified through email when files are queued up for importing, data is imported into the temporary database, data is successfully imported to the main database, and when import failed due to some error.