Synchronization - load and transfer data

Summary

Introduction

In Adobe Commerce, the seamless exchange of data between your e-commerce store is essential for maintaining a dynamic and efficient online presence. Our comprehensive synchronization and data loading capabilities empower you to connect your Akeneo PIM with your store, ensuring that your product information is consistently up-to-date. This section provides you with a deep dive into the intricacies of synchronization and data loading, enabling you to harness the full potential of Adobe Commerce's robust connector functionalities.

Connector Performance and scalability

Understanding Connector Performance

Customers frequently inquire about the metrics and performance of Akeneo Connectors. Unfortunately, gauging performance is a multifaceted endeavor. In general, it's important to note that three primary criteria significantly influence the performance of the connector:

1. Catalog Volume: While catalog size is an obvious consideration, it's not the sole factor at play.

2. Catalog Complexity: Importing 500,000 products can be straightforward when the product data is minimal (comprising just a name, description, and image). However, if the products are intricate, featuring numerous attributes, reference entities, variations, and the like, importing even 2,000 products can become a laborious and time-consuming process.

3. Adobe Commerce Limitations: Additionally, Adobe Commerce has its own constraints, as previously discussed in the introduction.

Optimizing Connector Performance

To enhance your connector's performance, we recommend the following strategies:

1. Direct Data Mapping: Always prioritize direct data mapping between Akeneo PIM and Adobe Commerce. This approach eliminates the need to rely on Adobe Commerce for data conversion, thereby boosting system efficiency.

2. Asset Management: Consider leveraging a Digital Asset Management (DAM) system and the Asset Manager (utilizing media links and file links) instead of uploading image binaries directly into the PIM. By saving URLs rather than high-definition images, you can significantly reduce processing time.

3. Data Filtering: Scrutinize the amount of data to be transferred to Adobe Commerce. You may have attributes in Akeneo PIM that are unnecessary for your storefront. Eliminating irrelevant data during synchronization can expedite the import process. Check the synchronization scope paragraph below for more information. 

Basic Technical information on Akeneo connector data transfer

  • Data Transfer: Data transfer is facilitated through the use of an API to XML format, ensuring seamless communication between systems. Please refer to the synchronizing data section for more information. 
  • Temporary tables transfer:
    • Akeneo Connector for Adobe Commerce inserts all PIM data into a temporary table. Then, data is manipulated (mapping,...) within this temporary table in SQL. Finally, the modified content is directly inserted into SQL in Adobe Commerce tables.
    • Even if raw SQL insertion is not the way you usually import data into a system, it is still faster than anything else, especially with the volume of data within a full Akeneo catalog. This results in a significant time saving for your imports.
    • Table Engine MyISAM: In the development of the connector, the choice was made to utilize the MyISAM Engine, as opposed to InnoDB, for improved performance in data reading and writing. With MyISAM, the connector can store Akeneo data in temporary tables, limited to 1250 columns. In contrast, InnoDB restricts this to only 160 columns. Each column is dedicated to a scopable or localizable attribute. For example, an attribute available in three languages would have three corresponding columns in the table.

Akeneo connector synchronization feature set

Synchronization panel 

Each job is responsible for importing PIM architecture and data into Adobe Commerce. The jobs order above should be observed to build your catalog properly. You could skip steps, but please be careful if you do. For example, if you want to import attribute options and you have created new attributes in your PIM, you will first need to import attributes or it will result in an error. Please check your data before importing it!

Akeneo Connector for Adobe Commerce is composed by 5 main jobs (5 main processes):

  1. Category
  2. Family
  3. Attribute
  4. Option
  5. Product

The Product Model and the Family Variant import processes has been removed since the 101.0.0 version of the Akeneo Connector for Adobe Commerce, and are handled by the Product job

 

As Akeneo Connector for Adobe Commerce is using SQL insertion, it will not trigger any of the events related to the entities imported. For example, plugins or observers located at product save will not be triggered during the import process.

 

Each job can be triggered manually or automatically on its own.You will find below the technical details for each process:

Category import process

Create a temporary table

  1. Insert data into the temporary table
  2. Match the PIM code with the entity
  3. Create a URL key
  4. Exclude category trees from connector configuration
  5. Create a position
  6. Create some category entities
  7. Set a value for the attribute
  8. Count of child categories
  9. Update URL keys
  10. Drop the temporary table
  11. Clean the cache
 
 

Family import process

  1. Create temporary table
  2. Insert data into the temporary table
  3. Match the PIM code with the entity
  4. Create some families
  5. Create family attribute relations
  6. Init default groups
  7. Drop the temporary table
  8. Clean the cache
 

Attribute import process

  1. Create a temporary table
  2. Insert data into the temporary table
  3. Match the PIM code with the entity
  4. Match attribute types
  5. Match the family
  6. Add some attributes
  7. Drop the temporary table
  8. Clean the cache
 

Option import process

  1. Create temporary table
  2. Insert data into the temporary table
  3. Match the PIM code with the entity
  4. Insert options
  5. Insert options label
  6. Drop the temporary table
  7. Clean the cache
 
 

Family Variant import process 

The Family Variant import process has been removed since the 101.0.0 version of the Akeneo Connector for Adobe Commerce, and is handled by the Product job

  1. Create a temporary table
  2. Insert data into the temporary table
  3. Update axis column in the product model table
  4. Update product model data from variants data
  5. Clean the cache
 
 

Product import process

Product import process is importing the products family after family (starting from the 100.5.4) and import both the products and the product models (starting from 101.0.0).

  1. Create a temporary table
  2. Create temporary reference attribute options table
  3. Create temporary reference entity records table
  4. Insert data into the temporary table
  5. Product Model Import
  6. Family Variant Import
  7. Add product required data
  8. Create configurable products
  9. Create Options for Variant Metrics Attributes
  10. Match the PIM code with the entity
  11. Update the family
  12. Update column values for options
  13. Create product entities
  14. Import file attributes
  15. Import asset file attribute
  16. Set values to attributes
  17. Update configurable products relation
  18. Update simple products relation
  19. Set products to websites
  20. Set products to categories
  21. Init stock
  22. Update related, up-sell and cross-sell products
  23. Set URL rewrites
  24. Import image attributes
  25. Import asset attributes
  26. Drop the temporary table
  27. Clean the media folder
  28. Clean the cache
 
 

Before 101.0.0: Product Model import process

The Product Model import process has been removed since the 101.0.0 version of the Akeneo Connector for Adobe Commerce, and is handled by the Product job

 
  1. Create a temporary table
  2. Create temporary reference attribute options table
  3. Create temporary reference entity records table
  4. Insert data into the temporary table
  5. Remove columns from product model table
  6. Add new columns to product model table
  7. Insert product models data into product model table
 
 

Synchronization scope

Synchronization scope for our connector defines the extent of data exchange and coordination between your Akeneo PIM and your Adobe Commerce. It specifies what data are included in the synchronization process. By customizing the synchronization scope, you have the flexibility to tailor your integration to match your business needs precisely.

Full or differential / Delta

When importing product data, you have two options to choose from:

1. Full Import: This is the most time-consuming import process because it synchronizes everything, including media, between Akeneo PIM and Adobe Commerce. However, the full import typically occurs mainly once during the initial synchronization for building your Adobe Commerce catalog. Subsequently, it is often automated for nightly resynchronization. The three performance criteria mentioned earlier significantly influence this process.

2. Partial/Differential/Delta Import: The system can synchronize only new products when necessary. However, for certain data reconstruction on the Adobe Commerce side, some tasks must be run as a full import (e.g., fully importing categories even in a differential import). Nevertheless, the differential import process reduces the required import time and is employed daily to resynchronize data that has changed within the day. This type of import is usually initiated manually.

To make an informed choice between a full import and a differential import, you should define your data update policy and set the threshold accordingly.

The choice of full or differential import within Adobe Commerce connector is to be configured within the connector configuration page and not in the jobs. To select the import mode in the connector configuration, whenever applicable, here are the steps to reproduce:

Apply filters to your imports

In order to select and narrow the data you want to import, you can filter the entities within the connector configuration. Go to the setup and use the connector section for more details on filtering possibilities.

Synchronization type

Manually trigger jobs 

The connector allows you to launch your synchronizations manually. To launch a synchronization manually, follow the steps below:

Since 102.0.0

Jobs are now run asynchronously and will be scheduled before being triggered.

To ensure that jobs are run as fast as possible, a new crontask ("akeneo_connector_launch_scheduled_job") has been added to the connector, and will check every minute if a job is ready to be triggered.

When a job is manually scheduled, the job status is changed to "Scheduled", and the Akeneo Connector crontask will trigger it in background under a minute.

In order to manually run the jobs, you can follow these steps:

  1. Make sure that Adobe Commerce CRON are correctly setup and running (CRON Documentation)
  2. Click on the "Schedule Job" link to schedule one job or select all the jobs you want to schedule and use the action "Schedule Job"
  3. Wait for the Akeneo Connector Cron task to run the job(s) in background

When a job is currently running, its status will be switched to "Processing".

If a job execution fails, its status will be switched to "Error", otherwise it will be set to "Success".

When manually scheduling multiple jobs together, the crontask will manage the correct order of the jobs, and trigger them accordingly:

  1. Category
  2. Family
  3. Attribute
  4. Option
  5. Product

You can only schedule a job if its status is: "Pending", "Success" or "Error"

 

A new Adobe Commerce cron group has been created for the connector cron tasks.

You can find the configurations under "Stores > Configuration > Advanced > System > Cron configuration options for group:akeneo_connector"

 

You can reset a job status by using the grid action: "Reset Status". The job status will be set to "Pending".

This action can be used if an independant bug or incident happens during a job execution and that the job is stuck in "Processing" status.

 
 
 

Before 102.0.0

You can follow this procedure:

  1. Select the job you want to run (import type),
  2. Then click on the Import button to trigger the selected job,
  3. Wait until the process ends.

The Job import process is completed when its status changes to Import complete in the console window.

If it fails, you could see a red error in the console window. Please take a screenshot of the console window: you will probably need it to communicate with your technical team or to open a ticket for our Support team via our Helpdesk platform (Akeneo Connector for Adobe Commerce Enterprise Edition only).

 
 
 

Scheduled or asynchronous synchronization

With the connector, you enjoy the convenience of scheduling your imports to occur on a recurring basis. This feature provides you with the flexibility to automate and streamline the process of bringing data into your system at specified intervals, ensuring that your product information remains up-to-date and synchronized with external sources. Whether you need daily updates, weekly refreshes, or any other custom schedule, our connector empowers you to tailor your import routines to suit your specific business needs and operational efficiency. The scheduled synchronization need to be expressed in CRON.

There are two ways to schedule import tasks in the connector : creating a Server Crontab or, for the Enterprise Edition, using the Magento Crontab within the UI of the command line. 

Adobe Commerce Crontab

Available since 103.5.0 version of the Akeneo connector for Adobe Commerce.

You can use the native Magento Cron system within the UI of the command line. 

This feature is only available for the Akeneo connector for Adobe Commerce Enterprise Edition.

 

Requirements

The Adobe Commerce Cron system must be enabled: Configure cron jobs

Configuration

You need to add your Cron Expression to the jobs page in the Adobe Commerce admin: 

System > Akeneo Connector > Jobs

In the Cron column, set the desired Cron Expression. After the update, the Schedule At value will be automatically calculated from the Cron Expression and will be refreshed after each import.

The task will be always launched except if the job has the “Processing” status. Indeed, the job is already running in that case so the import can not be launched again as it is already processing.

 

Cron Expression

If you need help writing a Cron expression, you can also use the “Cron Guru” online tool: crontab.guru

 
 

Server Crontab

Both versions of the Akeneo Connector (Community and Enterprise) provides a CLI command to launch the imports:

bin/magento akeneo_connector:import --code=category
bin/magento akeneo_connector:import --code=family
bin/magento akeneo_connector:import --code=attribute
bin/magento akeneo_connector:import --code=option
bin/magento akeneo_connector:import --code=product

Configure your server Crontab to launch imports with custom Cron expression:

crontab -e

Example:

0 1 * * * /var/www/magento/bin/magento akeneo_connector:import --code=product

 

Automatically trigger jobs

You can set cron tasks for your Akeneo Connector for Adobe Commerce for each import jobs.

Set your cron expression in your crontab according to this example:

0 22 * * * /usr/bin/php /path/to/magento/current/bin/magento akeneo_connector:import --code=import-type >> /path/to/magento/current/var/log/akeneo_connector_import_type.cron.log`
 

You can also launch an import job directly from command line:

php bin/magento akeneo_connector:import --code=import-type
 

Replace import-type with any of these jobs:

  • category
  • family
  • attribute
  • option
  • product

Product model and Family variant jobs has been removed since the 101.0.0 version of the Akeneo Connector for Adobe Commerce and merged into the Product job

 

If you are using a version of the connector older than 101.0.0, you can trigger the Product Model and the Family Variant job using these codes : product_model / family_variant

 

You can also launch multiple imports with command line:

php bin/magento akeneo_connector:import --code=import-type1,import-type2,import-type3
 

Since the 102.0.0 version of the Akeneo Connector for Adobe Commerce, if a job is already scheduled or processing, the command line will return an error and won't process the job execution.

 

Synchronization start times are automatic:

  • Products: 10:00 AM (GTM +2)
  • Products with variants: 4:30 PM (GTM +2)

 

 
 

Synchronization status

The following synchronization statuses are available in the Adobe Commerce connector.

 
Status Definition

PENDING

The job has been launched but not running yet.   
This status should last no more than a few seconds.

PROCESSING

The import is proceeding. The length will vary depending on the number of products imported. 

SUCCESS

The import is finished with no warnings or errors. It means that all your Products have been imported entirely with the information entered in your PIM instance.

SCHEDULED

The job is scheduled and will run

ERROR

The import has crashed with an error. It usually means that there is an overall problem with either PIM or Adobe Commerce instance.  
Please consult the import logs.

Starting from the version 103.3.0, The job grid will be automatically reloaded when a job is "Scheduled" or "Processing" in order to update in real time the job status. The automatic reload can be disabled from the connector configuration page, in Advanced > Enable Job grid auto reload.

You can reset a job status by using the grid action: "Reset Status". The job status will be set to "Pending".

This action can be used if an independant bug or incident happens during a job execution and that the job is stuck in "Processing" status.

Product deletion

Deleting synchronized products in Adobe Commerce is not handled by the connector. If a product is deleted on the Akeneo side, it must be manually deleted in Adobe Commerce. 
If a product is deleted in Adobe Commerce, it must be manually deleted in Akeneo. Otherwise, it will be recreated by the app during the next synchronization.

The recommended steps are:

  1. set the product status to disable (not enabled)
  2. synchronize your data with the command line of the connector. Product are deactivated - not enabled anymore. 
  3. if needed, delete the product in the PIM and/Or delete the products in Adobe Commerce.

If we remove a category from Akeneo PIM, will it be removed from Adobe Commerce after synchronization?

No, you need to delete it manually as stated above.

 

 

Empty attributes

You can reset the value of an attribute in Adobe Commerce after synchronization from Akeneo PIM. Since 104.3.10, this feature is optional and handled by a boolean.

Akeneo Adobe Commerce connector parameter Parameter
Create Empty Attributes Columns step Boolean Yes/No

Example 1: There is a text attribute: Promotional Message

  1. Today, we want to set it to in Akeneo PIM: "Discount 50%""
  2. Tomorrow, we want to set it to [empty] in Akeneo PIM.
 

If you clear the content of a text attribute (make it empty) in Akeneo PIM it will be synchronized by Akeneo Connector for Adobe Commerce.

Example 2: There is a simple select attribute: Color

  1. Today, we want to set it to in Akeneo PIM: "Red"
  2. Tomorrow, we want to set it to empty in Akeneo PIM.
 

For simple select attribute type, you need to have a “none” option for this attribute to be able to reset it (because you can’t « unselect » a simple select). It will then be synchronized by Akeneo Connector for Adobe Commerce.

Example 3: There is a multi select attribute: Promotion Campaign

  1. Today, we want to set it to “Black Friday” in Akeneo
  2. Tomorrow, we want to set it to blank in Akeneo
 

For multi-select, you can clean all option content in Akeneo PIM, it will be synchronized by Akeneo Connector for Adobe Commerce.

Where to find import jobs in Adobe Commerce?

Since 102.0.0

You can find the jobs under:

  1. Adobe Commerce SYSTEM menu
  2. Then click on Jobs in Akeneo Connector:

Statuses displays as below

 
 

Before 102.0.0

You can find the jobs under:

  1. Adobe Commerce SYSTEM menu
  2. Then click on Import in Akeneo Connector: