Product Import for VirtueMart

Product documentation

XML Import is a Joomla 2.5.x/3.x component. It allows you to import data from an XML or CSV file to VirtueMart.

Key Features

• Import from XML and CSV.
• Import to Virtuemart - products, categories, prices, custom fields, images (any media), child products, parent products, manufacturer.
• Supports Joomla! plugins.
• XML templates export and import.
• Automated CRON jobs.
• Language support. Parsing available in all languages. The component itself is localized to English and Czech language.
• Handling large XMLs. The only limitation is execution time configuration of your server.
• XPath support. Uses xPath to translate the tag name

What is XPath?

XPath is a major element in the XSLT standard.
XPath can be used to navigate through elements and attributes in an XML document.

• XPath stands for XML Path Language
• XPath uses "path like" syntax to identify and navigate nodes in an XML document
• XPath contains over 200 built-in functions (not all is implementated - see on bellow)
• XPath is a major element in the XSLT standard
• XPath is a W3C recommendation

XPath Syntax: See there

XML Import – Component Overview

List of XML Templates

After installing the XML Import Component, you will see the list of predefined XML templates. Just go to Components > XML Import. To create a new XML template, just click the "New" icon in the upper right corner. To edit an existing XML template, click on it's title. If you do not need the other templates, you can delete them.

Creating a New XML Template

After clicking the "New" icon, you will see a form where you define the new template. You have to fill in: Title, URL of XML and Language. The XML URL must be provided by an existing online store from where you want to transfer the products. You can also upload an XML file. If you want to upload a CSV file, the XML Import Component will first convert it to an XML file and the work with it further. You have to set the language every time as Joomla! creates individual table for every language. Once everything is set, click the "Save" icon.

XML Template Settings

After saving the XML URL, further parameters will show up. In the XML Preview tab on the left, you will see the XML tag overview. You can copy/paste the tags into the Product tab on the right and thus map your XML to the Joomla/Virtuemart database. In the picture on the left, you can see a ready XML template. Once your tags are mapped, just set how many products you want to import by setting the "Start On" and "End On" fields in the "General Settings" and the click the "Import" icon in the upper right corner.

Debug Log

The XML Import Component logs almost all operations. If there is an error or something is not parsed correctly, you can find in in the "Debug Log". The "Debug Log" can be configured to show "Errors only" or "Nothing". It helps parse the product quicker and saves your database capacity.

CSV Settings

The XML Import Component can operate with XML files only but it can process a CSV file as well. It converts the CSV file to XML automatically so it can parse it. You can do a CSV export in all common spreadsheet applications such as Microsoft Excel or Open Office Calc. Therefore, you can edit your products in, say, Microsoft Excel and then import them into VirtueMart.

Automated Jobs (CRON)

The XML can be updated by automated CRON jobs. CRON is a server feature where you enter the URL which you want to update and set the time, i.e. every day at 3pm. You can set your custom URL by adding a Secret (a custom alfa-numeric code) to prevent attacks. In the template settings you can define whether the XML should be updated from the given public address.

Since version 1.1.0, there is option to import XML template with selected ID. To do so, add to the end of import URL this parameter:

&id=[ID]

Where [ID] is number of the XML template you want to import. If you want to import more different XML templates, create multiple cron instances and insert particular template ID for every instance.

Example URLs for cron job:

Cron URL1: http://[yourdomain.com]/index.php?option=com_productimport&task=runcron&secret=[secret]&id=1

Cron URL2: http://[yourdomain.com]/index.php?option=com_productimport&task=runcron&secret=[secret]&id=2

Cron URL3: http://[yourdomain.com]/index.php?option=com_productimport&task=runcron&secret=[secret]&id=5

Secret is the same for all URLs. It is configurable from Product Import component Options.

To configure cron job, go to your hosting configuration (CPanel) and create cron job or ask your webhosting provider how to do it.

Example full Cron:

00 06 * * * wget --no-check-certificate -O /dev/null "https://copypartner.easy.minion.cz/index.php?option=com_productimport&task=runcron&secret=[pasword from administration]&id=5" &>/dev/null

Since version 1.3.0, there is option to import XML template with CLI Cron with selected ID. To do so, add to the end of import URL this parameter:

--id:[ID]

Where [ID] is number of the XML template you want to import. If you want to import more different XML templates, create multiple cron instances and insert particular template ID for every instance.

Example URLs for cron job:

Cron URL1: [server_path]/administrator/components/com_productimport/cli/import.php --id:1

Cron URL2: [server_path]/administrator/components/com_productimport/cli/import.php --id:2

Cron URL3: [server_path]/administrator/components/com_productimport/cli/import.php --id:5

To configure cron job, go to your hosting configuration (CPanel) and create cron job or ask your webhosting provider how to do it.

Example full CLI Cron:

00 05 * * * php/data/www/copypartner.easy.minion.cz/public_html/administrator/components/com_productimport/cli/import.php --id:4

General Settings

Every XML template must have a title. It is recommended to set the title according to the type of imported items or source > simply to know what you are importing in the future (in case of multiple templates).

As mentioned before, you can upload an XML or CSV file or set a public URL. When uploading a file form your computer, the URL is defined automatically.

Then you have to set the start item and end item by filling in the "Start On" and "End On" fields. This is useful when testing an upload of one item. It is also possible that the parsing process will take longer and will exceed the server's maximum execution time. In this case, you can divide the XML into two parts by parsing the first half and the the second half separately.

It is necessary to select the template type, or Handler, which will parse the XML. So far, there are two handlers available: VirtueMart and Form2Content. (this option is no longer available - you use a separate application to import data into the Form2Content application).

Also, it is possible to hide old items by setting the "Hide older then [h]" option. The value is enter in hours. After parsing, the time of the last modification is changed at each item and a script un-publishes all items that are older then the value you entered. This way, you can exclude old items from the import.

When parsing VirtueMart products, it it essential to set the language because the text is stored in individual table for each language. When parsing to Form2Content, select the Language option just in case of a multilingual website.

The last option in the General Settings is setting an XML URL updated by CRON jobs.

The XML Import Component will create a simple XML Overview where you can see the structure and what XML tags are used including their count. In the picture on the left you can see that there are 4 097 products in the XML.

Using this overview, you can copy/paste the tags into particular fields in the Product tab on the right. Be aware that this is just an overview and check the tags with your XML file.

Importing into VirtueMart

Product Identification

The first link, as the name implies, will delete all VirtueMart tables that are used by the XML Import Component. This feature was developed mainly for testing purposes. If something goes wrong, you don't have to delete thousands of products in the VirtueMart administration – all you have to do is use the link. Watch out! The data will be removed permanently. Therefore, it is recommended to backup the database continuously.

In order to process large XML files, the component does not load the file to a memory but parses it step by step instead. It parses a short section, then drops it and moves on. This section is defined by the XML object field. Enter the name of the XML item - in our case, a product.

It is very important to choose the identifier which is represented by the Identificator field. Thanks to the "ID", the component decides whether the product already exists and therefore will be updated or does not exist and will be created.

The next field "ID is" defines which column in the database is used to represent the ID. In most cases, it is the SKU (stock-keeping unit), however it is possible to enter any custom field. This way, you are able to assemble a product from various sources which do not have the same identifier.

Category Specification

There are two ways how online stores define categories: 1) category is a tag within a product, 2) product is a tag within a category.

1. Category Is a Tag Within a Product

<products>
    <product>
        <sku>hola001</sku>
        <name>Audi A3</name>
        <category>Automobily|Audi</category>
    </product>
</products>

See that an "Audi A3" product is in the "Cars" category and the "Audi" sub-category. The XML Import Component creates these categories automatically, all you need to do is enter a separator which is used in the XML and separates the sub-categories. In this case, it is a "|" character.

2. Product Is a Tag Within a Category

<categories>
    <category>
        <category_name>Audi</category_name>
        <product>
            <sku>hola001</sku>
            <name>Audi A3</name>
        </product>
    </category>
</categories> 

It is also possible to specify to which existing category should all products and sub-categories be parsed. This option is useful when you are importing from various sources – every product can have its parent category.

Product Specification

There are several fields: Product Name, Short Description, Product Description, SKU, Weight, Product URL and Product Price. There you have to enter the name of the appropriate XML tag. If the imported XML does not contain any of the fields (i.e. a short description), just leave the field empty.

There are several fields where you can choose whether the information is supposed to be updated or not. Its purpose is simple: in some cases you do not want to update all fields and you want to update/edit them in the VirtueMart administration instead.

Fields with a Specific Function:

Product Parent ID / SKU

Some XML files contain information about parent products. VirtueMart can work with such products and the XML Import Component can store them. The only question is whether to map parent product according to IDs or SKUs.

In Stock

If there is information about product's stock, you can paste it to this field. It must be a number.

Un/Published

If there is information about the stock amount in the XML, you can put the particular XML tag here. Note, that the XML tag must contain a valid numeric value only.

Created on, Modified on

If these fields are empty, the XML Import Component will set the "Created on" date according to the date of the first XML import and it will change the "Modified on" date after every update. If you want to parse this information form the XML, enter the particular XML tags. Preferably, it should be a DateTime MySQL format.

Image URL

If there is a tag with an image name in the XML, enter the tag into this field. The component will download the image, save it to a particular folder and assign it to a product. The thumbnail is created by VirtueMart itself. The size of the thumbnail is therefore set directly in the VirtueMart administration.

Price Specification

Enter the price XML tag into the "Price" field. The price must exclude VAT because VirtueMart calculates the VAT automatically.

You can choose if the price should be updated or not.

If there is a tax rate in the XML file, enter the tag into the TAX [%] field. If the tax rate you entered is already defined in VirtueMart, the products will be assigned to this existing tax rate. If the tax rate does not exist in VirtueMart, the XML Import Component will create a new one and assign the products to it.

You can also choose a default tax rate and currency. If you use the default values, all products will have VirtueMart default values. 

Price Modifier

Price can be modified during import. You can select operator (+, -, /, *), value which will modify the product price and if the value is percent or just number. You can see what exactly is happening during price modification at debug log. Example of debug log output:

Product Parameters Specification   

The product parameters can be imported in two ways. It depends in how they are stored in the XML file.

Product parameters - array

The first option: product custom fields are stored in the XML. Below is an example of how an array of custom fields can look like.

To give you an example of what a product custom field is, imagine you have a pair of shoes which is a product and the product custom fields are shoe sizes (9, 10, 11, etc.).

First, enter the parameter path (custom field path) into the first text field (Product parameters - array). Then enter the parameter definition in the following fields: Parameter Name, Parameter Detail, Parameter Value and Parameter Price.

The parameter (custom field) price is stored in VirtueMart as the subtraction from the product price. Example: "We have imported a XML file where the product price was empty and all the prices were given in the parameter price (custom field price)". For such cases, there is the "Count Price" checkbox which takes the lowest parameter price and sets it as the product price. Then it subtracts this value from the price of the parameters (custom fields). Example: suppose you have a pair of shoes where the shoe sizes differ in price. Size 9 is for 100 €, size 10 is for 110 € and size 11 is for 120 €. The XML import Component take the lowest price (which is 100 €) and sets it as the product price. Therefore the product costs 100 €. Then it subtracts the product price from the parameter price so the shoe size 10 costs +10€ (110 – 100). The result is +10 € which is then added to the parameter price (custom field price) and makes it 110 € again. 

An Example of an XML Array Called

<products>
    <product>
        <sku>hola001</sku>
        <name>Audi A3</name>
        <variants>
            <variant>
                <variant_name>Barva</variant_name>
                <variant_desc>Barva karoserie</variant_desc>
                <variant_value>Červená</variant_value>
                <variant_price>10000</variant_price>
            </variant>
            <variant>
                <variant_name>Barva</variant_name>
                <variant_desc>Barva karoserie</variant_desc>
                <variant_value>Zelená</variant_value>
                <variant_price>20000</variant_price>
            </variant>
            <variant>
                <variant_name>Převodovka</variant_name>
                <variant_value>Automatická</variant_value>
                <variant_price>200000</variant_price>
            </variant>
        </variants>
    </product>
</products>

Product parameters - single

This method is more difficult to setup but allows you to insert any information as a product parameter (custom field), regardless on where the information is in the tag tree structure. It is best if the information is stored at the highest level in the tag tree structure. If the XML structure is more complex, we can make use of the fact that the XML Import Component uses xPath to translate the tag name.

Sample XML:

<products>
    <product>
        <sku>hola001</sku>
        <name>Audi A3</name>
        <gear>automatická</gear>
        <wheels>16",17",18"</wheels>
        <variants>
            <variant>
                <variant_name>Barva</variant_name>
                <variant_desc>Barva karoserie</variant_desc>
                <variant_value>Červená</variant_value>
                <variant_price>10000</variant_price>
            </variant>
        </variants>
    </product>
</products>

Custom fields parameters:

• name
• value
• list separator (if values are list)
• type of custom fields (see the legend below)
• cart attribute (0/1)
• list (0/1)
• parent custom field id (if not exist, then put 0)
• layout position (if not exist, then don't use)

The parameters are defined with a special syntax code, for example:

{Převodovka|gear}
{variants/variant[0]/variant_name|variants/variant[0]/variant_value}
{Velikost kol|wheels|,|S|1|1|0|addtocart}

Line 1

{Převodovka|gear}

This is the easiest way. We want to import a parameter which represents the gear (Automatic/Manual) from the first level of the XML object. The first value is the name of the parameter and the second is the value from the XML file.

Line 2

{variants/variant[0]/variant_name|variants/variant[0]/variant_value}

There we make use of the xPath feature. We take the first element (variant[0]) from the parent element. You can also use you custom field name (variants/variant[0]/variant_name) and assign the (variants/variant[0]/variant_value) value from the same element.

Line 3

{Velikost kol|wheels|,|S|1|1|0|addtocart}

This is an extended option of the first line (Line 1). The XML Import Component will divide the value (in our case "wheels") according to the position of the comma and will assign the values to the product. Other custom fields (see the legend below) can have values 0, 1 a they define whether the field is a "cart attribute" | "list" | "parent custom field" | "layout position". The "cart attribute" | "list" | "parent custom field" | "layout position" options can be set in the VirtueMart administration at each custom field.

Legend of available types of custom fields in VirtueMart

S ... STRING
B ... BOOLEN
D ... DATE
T ... TIME
Y ... TEXTAREA
X ... EDITOR
M ... MEDIA
G ... GROUP
C ... MULTI_VARIANT
A ... GENERIC_CHILD_VARIANT
R ... RELATED_PRODUCTS
Z ... RELATED_CATEGORIES
E ... PLUGIN

Special symbols

Component can work with these special symbols:

• [@] before text means that text is a constant and that component even shouldn't try to find it in XML file.
• [!@] before text means that text is a value and if the value does not exist, it will return empty string.
• [+] can merge two variables or constants.

Example how to use these symbols

[@]http://www.some-domain.com/images/[+]pictureName

Explenation of example

Sometimes at XML file there is only name of the image, not the whole URL. So at XML can be image defined like this:

some-image.jpg

XML import component needs to download the image so it needs whole URL. Not just a name. So we put a URL before the image name as a constant and then we merge this constant with a variable using [+] symbol.

Component then translate example as this:

http://www.some-domain.com/images/some-image.jpg