Item Import Process Flow
The Item Import is EASYProcess Process 287. The Item Import starts when this Process is called either by a batch job or a button.
When the item import first starts, it collects all the variables to be used for its run. This information can answer questions like “is the item import running for all items or just a single item refresh?” or “is the item import running to refresh all the items or just the ones that have changed in JDE recently?”. These sort of variables are defined in the Configuration Variables in EASYProcess for the environment.
The Item Import then calls EASYProcess Process 574. This is responsible for retrieving the list of changed items. If the Item Import is running a full import, PRC574 tells PRC287 that the query to JDE should not be filtered by this criteria. If a Net Change Import is running, PRC574 then gets the list of items that have changed. It retrieves this list by looking in the date updated column in either the Item Master (F4101) or the Item Branch File (F4102). Whether it looks in one, or both of these tables for their corresponding date updated column to check for a change is determined by a configuration variable.
It is base functionality to run a full Item Import once, then each subsequent Item Import run a Net Change Import. This is because each Net Change Import looks for any changes since the last Import was run. For this reason, the first import must be a full one. Also, if for any reason a full import is needed, a configuration variable can be set to force a full Item Import, but during this run, the configuration variable is updated so that the next Item Import will be a Net Change Import.
At this point, PRC287 looks at PRC574’s output and determines if the output should be used as an additional filter for the query to JDE. It then uses that filter and an additional where clause from the configuration variables, if it is not blank, and builds the query it will use to for the Item Import. So far, this is just stored in an evaluate service. The query is written, but has not been run against the JDE database.
Next, PRC287 runs the query against JDE, but places it in a subquery and performs a COUNT(*) on it so that an estimate of records returned can be recorded. This value is recorded in the Item Import Batch Job’s history.
Before beginning, if a full Item Import is about to begin, all the records that could be imported are updated with an Imported=”No” flag. This is done for all records in all three tables updated by the import: EC_Item_Master, EC_Item_Branch, EC_Item_Branch_Price. This is done because it is risky to delete the records and assume the Item Import is successful. Instead, we mark all the “old” records as Imported=”No”. Then, as an item is imported and records in these three tables are updated, the flag is changed to imported=”Yes”. At the end, all of the records still with a “No” flag are deleted. The exception to this is if the number of records to be deleted is too great. For example, if the Item Import ran for a few minutes, but then the connection to JDE was lost, that might result in the Item Import ending. At the end, it might try to delete thousands of item records. This protects against that. Also, since all the “old” records were never deleted them to begin with, the K-Rise Item tables are still intact with records.
Now, the query to begin the Item Import is run. This is a ReadQuery. These query JDE, but only return one result at a time. This way, the process can retrieve the one result, perform an action on it (in our case, update our Items tables with it), then loop back around and when ready ask JDE for the next result. This is done until there are no more records returned by the query.
Immediately after running this query, we check if the query returned an exception. This would happen if the connection to JDE were down or the query was in an incorrect syntax. If an exception were returned, the Item Import would stop here. The connection to the JDE database would be closed and the status would be updated in the Item Import Batch Job’s history.
If there was no exception returned by the query to JDE, we next check if we have reached the end of our returned results. Since we are requesting one record at a time from JDE, in a loop, eventually we will reach the end of the results. At this point, the connection to the JDE database would be closed and the status would be updated in the Item Import Batch Job’s history.
If the query had not reached the end of the results (or end of file), three evaluate services would run. These would define the association between the JDE columns and the K-Rise columns. For instance, if only three category codes are used in JDE, but they were not IMSRP1, IMSRP2, IMSRP3, the mapping could be changed so that the ones that are used map to the K-Rise Category Codes 1, 2, 3 anyway, just for ease.
This data is then fed into PRC357 which takes it and either updates the existing records in the K-Rise tables, or creates new ones if they do not exist. It does this for all three of the K-Rise tables used for the Item Import. It is during this process the configuration variable is used to ask if the Item Import uses JDE to populate the Category Codes. If so, the mapping from JDE to the K-Rise tables that was defined back in PRC287’s evaluates is used. If not, the Category Codes are not included in the Upsert service. This configuration variable might be used if the EASYCommerce site were configured to maintain the Category Codes manually in the K-Rise portal. This way an Item Import does not accidentally override these values. Next, PRC357 saves the price. However, if the JDE to K-Rise mapping in PRC287’s evaluate was left blank or after that mapping the price field is blank because a price record did not exist in JDE, then the K-Rise record is not upserted.
After PRC357 is finished, PRC287 records the output of it in the Item Import Batch Job history. At this point, if more records have been processed than the original estimate, the Item Import ends, the connection to the JDE database would be closed and the status would be updated in the Item Import Batch Job’s history.
Now PRC287 begins the loop and asks JDE for the next record returned by the query and does this same action again. This continues until either an error occurs or all the records are processed. As some finishing actions, if the Item Import was not a Net Change Import, the records still set to Imported=”No” would be deleted and the Solr catalog would be reindexed.