APEX Data Load Wizard Customizations ~ Revisited

Posted on by
Reply

dataLoadWizard
I was revisiting our APEX Data Load application, which uses a highly customized version of the APEX Data Load Wizard – several of them. Key words here: Highly Customized.

Customizing the Data Load Wizard – or modifying any set feature of APEX – leaves one open to the possibility that things might not work in the following versions. This has been the case with every major APEX upgrade since we made the Data Load Wizard customization in APEX 4.2.  Each time we have had to plan time/resources to revisit the customizations and make corrections – all on our own sleuthing.  No big deal, right?

This year an upgrade to APEX 5.1,  left the last step, the load into our defined Data Load table, not working. I had two choices, rebuild the Data Load Wizard pages ( 3 sets), and re-apply the customizations.  Or, write my own load into the final table.

This time I chose the latter.
As a quick reminder, the Data Load Wizard loads uploaded data into a series of collections.  To learn more about the Data Load Wizard and why we customized, read the previous Data Load Wizard posts. The data from those collections – after transformations are applied – gets loaded into the Data Load target table.  In our case, we needed complex validations and transformations, and logging of each change. The logging of each change is what really drove us to customizing the process. So we pulled data from the uploaded collection (SPREADSHEET CONTENT) into a temporary table, performed all our validations, transformations and logging, then moved the data back into the load collection (LOAD_CONTENT).  And continued with the Data Load process.  That’s a simplification, there are a lot of other pieces involved, but that’s enough for here. (Read previous blog posts for more details).

By choosing to write my own load into the final table, and bypass the APEX processes, I lose some other features that I now have to code on my own.  What I lose:

  • Automatic insert/updates
  • Automatics insert/update failure counts and logging.

I am OK with handling these in my own bulk process insert with the LOG ERRORS clause. We only get inserts, never updates, which simplifies things.  Your mileage may vary. Note that if you need to process updates and inserts, and report which records are inserted vs updated, the solution will be more complicated.

Next time we upgrade, I expect another set of minor Data Load Wizard changes … to which I will adjust somehow.  Now, IF the Data Load Wizard were to allow greater flexibility in logging transformations for each row – then I may be able to abandon the customized code altogether and revert to the mainstream Data Load Wizard.  Until then I am resigned to maintenance with each update.

P.S.
I just had someone ask about customizing the Data Load Wizard – breaking into the collections as I did. My recommendation was, use the Data Load Wizard to load into a staging table, then write PL/SQL processes to perform the validations needed – in his case to only do inserts no updates – filter out existing rows. Such a solution will be in-the-sandbox (no customizations) and will readily upgrade.
I strongly advise against breaking into the Data Load Wizard unless there is a solid business requirement to do so. Unless …

    • You have plenty of time to go back and fix things upon upgrade, and
    • You have a finding Fairy to pay for all those upgrades.

Happy Coding, whichever approach you choose.

Bulk Bind Reminder: ORA-06502: PL/SQL: numeric or value error: Bulk Bind: Truncated Bind

Posted on by
Reply

Bulk Bind: Truncated bind

I recently had a good reminder on one of the subtleties of PL/SQL bulk fetches and inserts. I kept getting this error on the FETCH:

ORA-06502: PL/SQL: numeric or value error: Bulk Bind: Truncated Bind  

The procedure purpose is to bulk insert data from a temporary staging table to the real, properly-sized column Load table.  Simple, right?  Except when I use the wrong table types.

My problem?

The elements I was selecting in my cursor did not logically fit into the table type target of my cursor fetch.

I defined my cursor using a query on the temporary staging table (larger VARCHAR2 columns), not the real table (exact-fit VARCHAR2, NUMBER and DATE columns).
But, I defined my table type to receive the cursor results on my real table:


CURSOR c_dl IS
SELECT ...
...
...
FROM my_staging_table stg
WHERE action IN ( 'INSERT','UPDATE');
TYPE my_load_table_tt IS TABLE OF my_load_table%ROWTYPE
INDEX BY PLS_INTEGER;
load_data_arr my_load_table_tt;

BEGIN
OPEN c_dl;
LOOP
FETCH c_dl BULK COLLECT INTO load_data_arr LIMIT 1000; ----- Error  happens here!
FORALL ii IN 1 .. load_data_arr.COUNT
INSERT INTO my_load_table
...
...

The Bulk Bind: Truncated Bind error is because the cursor – with bigger elements, cannot logically fit in the defined array/table type – the columns are too big.

To correct this, I changed the definition of the table type to be a table of the staging table. Then the FETCH proceeds fine, as does the INSERT.

But wait, why didn’t the INSERT fail, since I am Inserting from a (potentially) large-column cursor into the exact-fit table?

The INSERT proceeds because the elements being inserted fit  – the transforms and validations (processed previous to this point in the code) have ensured that all elements are indeed the size they should be in the exact-fit table.  So the INSERT does not complain.

LOG ERRORS REJECT LIMIT UNLIMITED

Now, IF one column in one row was too large, or the wrong format for a date, the entire INSERT would have failed. Unless I used the LOG ERRORS clause, which I did.

IF one or more records is/are bad, I want to load all the good records, and single out the bad. To do this, I used the LOG ERRORS REJECT LIMIT UNLIMITED clause at the end of the INSERT statement:

INSERT INTO my_load_table
VALUES ( ..load_data_arr(ii).col1,
load_data_arr(ii).col2,
load_data_arr(ii).col3,
...
load_data_arr(ii).coln)
LOG ERRORS REJECT LIMIT UNLIMITED;

This LOG ERRORS REJECT LIMIT UNLIMITED clause ensures that all clean rows will insert, and all “bad” rows get logged to an error table, in my case called ERR$MY_LOAD_TABLE.

I created the error log table previously, using the command:
BEGIN
DBMS_ERRLOG.create_error_log (dml_table_name => 'MY_LOAD_TABLE');
END;

We define the staging tables with generous VARCHAR2s on purpose. We want to allow leeway in the incoming data, knowing we can strip punctuation, perform lookups, format dates, and perform other transformations in the validation stage. That means doing all that validation is on us, to make sure the data is in shape for the final insert. Just one way of doing things – there are many other approaches.

My reminder is – especially when writing things quickly – to pay attention to the nuances of the features in use.  I won’t make that mistake for a while now :).

For a full treatment of bulk processing with BULK COLLECT and FORALL, see Steve Feuerstein’s article PL/SQL 101: Bulk Processing with BULK COLLECT and FORALL.

 

Submit to ODTUG Kscope18 – Last Days!

Posted on by
Reply

Hurry up! Abstract submissions for ODTUG Kscope18 are due January 5th.

No more procrastinating, no more wavering, Just Do It! Submit your abstract here.

Presenting at ODTUG KScope is an awesome experience – share your knowledge, share your technical success stories, share you case study of geez-it-was-hard-but-we-figured-it-out.  We are looking for presenters in these areas:

Application Express
Database
Big Data & Data Warehousing
EPM Infrastructure
EPM Data Integration
EPM Reporting, BI Analytics, and Visualization
Essbase
Financial Close
Multi-Product EPM Solutions
Planning
Real World EPM

Not sure how to write an abstract, or what to share?  Check out these blogs for more information and ideas:

http://spendolini.blogspot.com/2013/11/presentation-advice.html
http://www.odtug.com/p/bl/et/blogid=1&blogaid=497
https://www.linkedin.com/pulse/20140630141618-4832037-how-to-get-your-conference-abstract-accepted
http://www.pythian.com/blog/concrete-advice-for-abstract-writers/
https://chadthompson.me/2012/12/how-to-submit-winning-presentations/

We need topics at all levels of expertise, beginner to advanced.  No excuses – write that abstract now  Join us in Orlando, June 1-14, 2018!

 

Submit to ODTUG Kscope18 – Last Days!

Posted on by
Reply

Hurry up! Abstract submissions for ODTUG Kscope18 are due January 5th.

No more procrastinating, no more wavering, Just Do It! Submit your abstract here.

Presenting at ODTUG KScope is an awesome experience – share your knowledge, share your technical success stories, share you case study of geez-it-was-hard-but-we-figured-it-out.  We are looking for presenters in these areas:

Application Express
Database
Big Data & Data Warehousing
EPM Infrastructure
EPM Data Integration
EPM Reporting, BI Analytics, and Visualization
Essbase
Financial Close
Multi-Product EPM Solutions
Planning
Real World EPM

Not sure how to write an abstract, or what to share?  Check out these blogs for more information and ideas:

http://spendolini.blogspot.com/2013/11/presentation-advice.html
http://www.odtug.com/p/bl/et/blogid=1&blogaid=497
https://www.linkedin.com/pulse/20140630141618-4832037-how-to-get-your-conference-abstract-accepted
http://www.pythian.com/blog/concrete-advice-for-abstract-writers/
https://chadthompson.me/2012/12/how-to-submit-winning-presentations/

We need topics at all levels of expertise, beginner to advanced.  No excuses – write that abstract now  Join us in Orlando, June 1-14, 2018!

ODTUG 2018 Elections ~ Vote Now!

Posted on by
Reply

Elections for the 2018 ODTUG Board are underway – Vote Now!

Calling all ODTUG Members:  Exercise your right as an ODTUG member and vote for the Board of Directors.  This may be the most important thing you can do for ODTUG.

(Not an ODTUG Member? You are missing out – see why to join ODTUG and become a member here)

Vote for ODTUG Board

Why vote for the Board? Why is it important?

ODTUG Board members determine the current and future direction of ODTUG.  Board members set strategic goals for the organization at the beginning of the year, and execute on that plan throughout that year and into the next. We strive to stay on budget while launching webinars and other educational sessions, marketing campaigns to spread the word and attract new members, and of course there is our fabulous Kscope conference. We are also your voice to Oracle, bringing members in touch with key Product Managers at ODTUG events, online and in person.

For the Board to do our job, we need to hear from you, our members.  Starting with who you want as Directors determining ODTUG goals and direction.

Board members face tough questions in the coming years:

  • Do we need more (or fewer) Meetups? On what topics?
  • What is the direction of the Database Community?
  • Where do tools like Node.js and Docker belong within ODTUG, or do they belong at all?
  • How can ODTUG reach and engage more millennial members?
  • How do we provide our members with the latest and best in Oracle tools technology, as that technology is changing under out feet, or rather, above our heads into the Cloud.

It is your right – and your responsibility – as an ODTUG member to vote in the candidates you feel can best form our future in the direction you feel it should go.

By now you have received your voting email from Association Voting providing you with an email address and Member ID. Please use that information to log in,  learn about our candidates and vote.

( Can’t locate the email? Click here and then click on the help button to request the information be resent.)

Find that email, and please do two things:

  1. Read all the candidate’s statements carefully.  Then
  2. Vote for the candidates you feel are best able to guide ODTUG for the next few years.

Let’s keep ODTUG serving our members in the best way possible.

Thank you, and Good luck to all 2018 candidates!

APEX Interactive Grid: Customize Toolbar

Posted on by
Reply

APEX Interactive Grids allow customization of the Toolbar, the Grid Menu and the Column Heading options using JavaScript in the appropriate Grid or Column Attributes Advanced –> JavaScript Code element. This post discusses how to build a function to make simple changes to the Interactive Grid Toolbar.

widget.toolbar.js

The toolbar is implemented by a toolbar widget ,which is created by the interactive grid widget. To learn more about the toolbar widget, read library file widget.toolbar.js.  That file documents the properties of the toolbar widget,  the toolbarData array, control types and their properties.

toolbarData

toolbarData is an array that describes the toolbar – the buttons, fields and menus visible to the user, and their actions. Toolbar objects are grouped into control groups.  Each control group has one or more controls, of type Static, Text, Select, Button, Menu, Radio Group or Toggle.

By default, the toolbarData is based on the Grid default options and declarative settings.

Developers (you) can override the toolbarData property by adding a function to the Advanced–> JavaScript Code section of the Interactive Grid Attributes. Basically, you write a function that resets the toolbarData array to the configuration you want.

There are two general approaches to writing an override function:

  1. Totally replace
  2. Copy and adjust

That is, your custom configuration function can completely (re)define the toolbar, or, it can make adjustments to the default or current toolbarData.    Which approach is best depends on your requirements – the degree of customization required.

If you are totally replacing, you can fill the array as you want. You might want to examine the default toolbarData to see what is possible.

If you are adjusting, you need to understand the default toolbarData array to know where to add your customization.

Default toolbarData Array

By default, for an editable grid,  there are 7 control groups in the toolbar, as shown below:

toolbarDataArrayLabeled

Note that the reports, views, actions2 and actions3 toolbar groups are conditionally visible: the reports group becomes visible when there are multiple reports, the views group becomes visible when there are multiple views; the actions2 and actions3 groups become visible when edit mode is on.

To examine all of the elements in the toolbarData object (as well as all other objects of the interactiveGrid widget) use this command in the Console:

apex.region("emp_igrid").widget().interactiveGrid("option","config")

using the static id of your interactive Grid for “emp_igrid”. This command says, Show me the config options for the emp_igrid Interactive Grid region.

Examine the returned contents by clicking through all the objects.  Look at the toolbarData object, expanding the elements beneath it.  After a few clicks you will no doubt recognize menu and button elements of the interactive grid Actions menu and other toolbar features.

The default toolbarData array has 7 elements, for 7 control groups, shown below:

toolbarDataArray

If we look at just “actions1” object, you will see that it is a menu containing  the familiar elements of the Actions menu. This image shows the download dialog menu element expanded.

actions1Download

Let’s say our users do not like doing two clicks (Actions –> Download) to get to Download – they want a Download button on their toolbar.  In fact, they want it all the way over to the right.

To do this, we need to construct a JavaScript function to adjust the toolbarData to add a Download button in the actions4 (the rightmost) control group.

This function does the trick:

function(config) {
 var $ = apex.jQuery,
 toolbarData = $.apex.interactiveGrid.copyDefaultToolbar(),
 toolbarGroup = toolbarData.toolbarFind("actions4"); // this IS the rightmost GROUP WITH the RESET button

// add a Download button next to the Reset button
 toolbarGroup.controls.push( {
 type: "BUTTON",
 action: "show-download-dialog",
 iconBeforeLabel: true,
 hide: false,
 disabled: false
 });
 
 config.toolbarData = toolbarData;
 return config;
}

The toolbar stuff adds the button, with the action show-download-dialog.  The show-download-dialog action is the same definition found while examining the actions menu.

If you are a PL/SQL developer, and relatively new to JavaScript, the above JavaScript is very readable:

  • Copy the default toolbarData – we will make changes to a copy
  • Get the actions4 toolbar group
  • Add (Push) the download dialog to the actions4 toolbar group
  • Copy the modified toolbar to the return config

Very easy to follow.  But this is a very basic example.  If you will be doing a lot of such customization, pursue additional JavaScript training.  Hint: Remember that JavaScript is case sensitive, where PL./SQL is not.  That alone will save some grief.

Note that Download is a declarative option –

DeclarativeDownload

So, if/when a (the next?) developer sets Download Off … your newly-placed Download button disappears.  The declarative nature of the show-download-dialog action is contained within the action – our moving the button around does not overwrite that declarative toggle.

One can also turn off Download (and other features) programmatically. To do so, we use this command:

config.features.download = false;

as part of you Advanced –> JavaScript Code function, or in a dynamic action.  This command will also turn off your newly-placed Download button.

This simple example show how to add one button – you can of course add more, or redefine the entire toolbarData area.  For additional, more complex toolbar customization examples, see the examples in the Sample Interactive Grid packaged application.  And read the Interactive Grid posts in the HardLikeSoftware blog by John Snyders of the Oracle APEX team.

Interactive Grid is a great new region type – and it is still evolving. This writing is for APEX 5.1.2 – APEX 5.2 may bring more declarative options, and/or may bring documented APIs for customizing the toolbar – who knows.  Stay tuned!

 

 

 

APEX Interactive Grid: JavaScript Basics Cheat Sheet

Posted on by
3

APEX Interactive Grid can be customized by JavaScript in the Advanced –> JavaScript Code attribute of the Grid or a Grid column.  Yup – JavaScript.

JavaScript may be out of the comfort zone for PL/SQL developers, even those who implemented extensive tabular form customization working with PL/SQL collections.  Time to say Goodbye to those collections … Welcome JavaScript!

The following is a collection of simple JavaScript lines most likely to be needed by a developer wanting to customize an Interactive Grid, or access data elements in the Grid.

This is not a comprehensive list or a complete function – just a simple collection of lines to give you an idea of the process and examples of  – a reference for – the syntax.

JSRefLines

Taken line by line:

var $te = $(this.triggeringElement);

This line gets the triggering element – the element that caused the dynamic action (DA) to fire. Usually we want to do this in a Grid to get the value of a particular cell – the one clicked on in this case. To get the value of the cell, we need to know which row id,  then we can narrow things down to a column using the column static id.

To get the row id, we start with the triggering element.

Next, we find the closest row – a ‘tr’ – and get the “id” data from it.  It helps to know that Interactive Grid has an “id” data element on each row:

closestTrDataId

var rowId = $te.closest('tr').data('id');

If our data element was data-thingamajig, we would access it via

 ... .data('thingamajig');

The next line gets the grid widget. The apex.region function is the preferred way to access region widgets.  “grid_static_id” is the Static ID of the Interactive Grid, set by the developer in the Advanced –> Static Id attribute of the Grid.  If you do not set one, a static id gets assigned, but it will be a long difficult-to-read identifier – it is much easier and better practice to set a meaningful static id, then use that meaningful static id in your code.

var ig$ = apex.region("grid_static_id").widget();

Given the Grid, we can now get the data model.  The data set is referred to as a data model.  There is a data model for every view:  grid,  chart, group by, icon, detail.  The following line gets the grid data model.

var model = ig$.interactiveGrid("getViews", "grid").model;

Given the grid data model, which we know is a table, we can get the record of the model, using our rowId which we identified via properties of the triggering element.

var record = model.getRecord(rowId);

Once we have the record,  we can access properties of any column in our Grid – any column in that record, using the column name – or the aliased name we assigned to the column.  Here my column name is COMM for commission.

var comm = model.getValue( record, "COMM");

model.getValue gives us the value of a cell in a record.  The corresponding model.setValue sets the value of a cell in a record.

if (comm < 100) { 
  model.setValue(record,"RIFF",'Y');
}

The above examples are easy on purpose. In fact, I bet any PL/SQL developer could follow these lines without headache.

Now you know how to access a row id, access the grid widget, access the data model of a grid, access rows – records – in that data model, and how to get and set values of the columns in that data model record.  That covers most of the basics!

You will need to learn more if you plan on complex customization or perhaps on building plugins.  For now, I recommend examining the grid in the Console, and reading the APEX widget js files.  Looking forward to APEX 5.2 (Oracle Safe Harbor), there may be documentation for all the Interactive Grid widget APIs.  Won’t that be nice!

Happy coding!

I highly recommend reviewing all of the examples in the Sample Interactive Grid packaged application.  And, read John Snyders’ hardlikesoftware.com blog posts on How to Hack the Interactive Grid.

 

APEX Interactive Grid: Column Widths and noStretch

Posted on by
2

Setting column widths in Interactive Grid is totally different than how you may have set column widths in Interactive Reports.  While it is nice to blindly let APEX and the browser handle it, there are times when you need to explicitly set a column width, or at least set the width of one column relative to another, and for that you need to know how Interactive Grid handles column widths.

Recall that Interactive Reports let the browser figure out the column width, using table-layout:auto;  We could do little tricks like making the column heading longer or shorter to trick the browser into making the column wider or narrower.  That won’t work with Interactive Grids.  Interactive Grids are Fixed Table Width – table-layout:fixed;.  APEX will, by default, try to fill the full table width with the displayed columns, adjusting  – stretching or shrinking – the width of columns to  fit the full width of the table.

So how to set Interactive Grid column widths?

First,  the Width property (visible in the Appearance property group) for an Interactive Grid column does not set the width of the column.

ColAppearWidth

Instead, run the Interactive Grid, then set the column widths interactively using Drag and Drop on the columns themselves.

Select the || divider to dynamically adjust column width:

ChangeWidth

Select the 4-way divider between columns to activate Drag and Drop column order:

DragDrop

Hover over the divider between column headings to grab either the column width adjustment icon or the drag-and-drop column order icon.

Adjust column widths and column order as desired.  (Notice that APEX will adjust column widths so the full width of the table is filled.  This auto-adjustment may not be what you want – more on “noStretch” later on in this post).  Then save the report: Action –> Report –> Save.   The column order and the relative column widths are saved with the report.   When a user adjusts column widths, or does any other actions, a Reset action will restore the column order and column widths to the default set by the Developer’s Save Report.  Much easier that messing with numbers!

reportSave

You can, in the Column action, set a minimum column width. This sets just that – sets a minimum column width for the column that APEX tries to honor.

ColMinWidth

What about Max width?  You may have a Yes/No column, or a single-digit column, and do not want that column to get auto-stretched to some absurd-looking width.

SillyWidth

 

This is where the noStretch option comes in.

Setting noStretch on a column will hopefully (Oracle Safe Harbor) be declarative as of  APEX 5.2.0.

Until then, to set noStretch, as of APEX 5.1.1, in the Advanced –> JavaScript for the column you do not want stretched, set the noStretch grid column option to true:

function(config) {
   config.defaultGridColumnOptions = {
       noStretch: true
   };
   return config;
}

If you want noStretch true for all columns, the solution is similar, but this time we adjust the configuration in the Advanced–> JavaScript region for the Grid:

function(config) {
   var cfg = cfg;
  cfg.stretchColumns = false;
  return config;
}]

With noStretch on, APEX will not resize your noStretch column(s) to fill the full table width:

noStretchOn

 

While this works perfectly if the current view is your Grid view, remember that there are also Chart, Group By, Icon and Detail views – the above code will fail if the current view is not Grid.  So one should make that code that code a bit smarter, so it only acts on – changes the configuration for – a Grid view:

function(config) {
   var cfg = cfg;
  "views.grid.features".split(".").forEach( function (p) {
      if ( !cfg[p] ) {           cfg[p] = {};      cfg = cfg[p];  });
  cfg.stretch.noStretch = true;
Columns = false;
  return config;
}

In version APEX 5.1.0, setting noStretch is a bit different.  If you can, upgrade to APEX 5.1.1 or higher.  There are significant updates to Interactive Grid widgets and APIs in APEX 5.1.1.  IF you cannot upgrade and need noStretch, it can be implemented per column, by adding JavaScript to the Page Execute When Page Loads element:

var igrid = apex.region ("my igrid").widget().interactiveGrid("getViews", "grid");igrid.modelColumns.COLUMN1_NAME.noStretch = true;igrid.modelColumns.COLUMN2_NAME.noStretch = true;
igrid.view$.grid("refreshColumns");igrid.view$.grid("refresh");

That’s it!

I highly recommend checking out the APEX Sample Interactive Grid packaged application. Your best source of Interactive Grid documented examples.

Good luck!
  

 

Going to the Grid: What the APEX Interactive Grid Means to You and Your End Users

Posted on by
Reply

I will be speaking at Kscope 17 in San Antonio, TX on APEX 5.1 Interactive Grid:

Going to the Grid: What the APEX 5.1 Interactive Grid Means for You and Your End Users

Karen Cannell , TH Technology
When: Jun 26, 2017, Monday Session 2 , 11:45 am – 12:45 pm
Room: Cibolo Canyon 5
KScope_Pic
Topic: Application Express – Subtopic: APEX New Release

The long-awaited APEX 5.1 Interactive Grid region is here – but what does it mean to you and your end users? Need some practical guidance on when, why, and how to adopt the interactive grid region type? This session is for those who have seen the introductory demonstrations, are looking to upgrade to APEX 5.1, and need some advice. Does the interactive grid replace interactive reports? No more tabular forms? This session answers practical functional and technical questions raised by this new region type:

• When and why should I upgrade existing regions to interactive grid?
• Is there an upgrade wizard?
• What features will I gain; what features will I lose?
• What about customizations – will they upgrade?
• How does an editable interactive grid compare to my existing tabular form?
• Which features must still be manually written? Do I still need to write all that collection logic?
• How can I customize appearance?
• How can I add dynamic actions?
• Can I extend the interactive grid functionality?
• Will my end users like it? What about data entry users?

This session compares and contrasts interactive grids with the interactive reports and tabular forms we are familiar with. We will pay particular attention to maintaining or replacing features and common customizations in existing applications: checkboxes, 32K limits, cascading select lists, row-level validations, interactions between columns, and navigation between fields. How does the interactive grid region measure up? Should you upgrade all your interactive reports and tabular forms now?

How we do settings of which features are on or off for a particular Grid and for a particular column is now controlled by javascript. Sound scary? It is really not, you just need to learn where to put what piece of code when.  How to access data in the Grid data model?  We will show you that too.

If you cannot attend, don’t worry, I will be posting bits of that presentation here over the next few weeks.  IF you can attend, I will see you there!

Rest up ~ our Kscope 17 schedule is packed, so don’t expect much sleep or downtime.

Read XLXS Files from APEX 5.0 and Higher – One (or Two) Ways

Posted on by
2

In my work as a consultant I often have the task of getting data from MS Excel spreadsheets into tables in the database.

If this is a one-time load, the process is simple – use either SQL Developer or, easier yet, use the Data Load Workshop in APEX:  SQL Workshop –> Utilities –> Data Workshop.

If the requirement is for a repeated load or many files, I need to set up some automated process for reading in the data.  I do not want to ask my users to open the XLSX files, save the data to CSV, then upload the CSV files.  That makes for unhappy users, at best.

I am sure there are several options for load XSLX files into Oracle data tables.  I am going to focus on a two I have used (and continue to use) successfully across several clients, across many projects.  And another option that used to be a very good option, that goes away with APEX 5.0 and higher.

Let’s do Bad News First.

The Option That Goes Away:

With APEX Listener, now Oracle REST Data Services, ORDS, we gained a wonderful utility for loading XLS or XLSX files into the database.

I will not elaborate on that here, for reasons which will be obvious in a bit.  I did cover that option, and several other Data Load options, in this presentation, How Do I Load Data, Let Me Count the Ways..  Most of these options still apply.  Yes, it is a PowerPoint, but gives you enough detail to get going.

This ORDS-based utility works fabulous for APEX versions less than APEX 5.0.  It loaded data into a collection. You can process your data however you wish from there.  However, as of APEX 5.0, APEX uses JSON to move data, and the ORDS Excel file upload function no longer works.  Joel Kallman explains this clearly in his blog, Let’s Wreck This Together, here.  It will still work for APEX versions < 5.0 – that is, APEX 4.2, APEX 4.1.  But beware, if you are using this Excel-Upload option of ORDS in APEX 4.1 or APEX 4.2 applications, you will need to make some changes when you upgrade those applications to APEX 5.0 and above.

The XLSX Option That Still Works – READ_XLSX

Thanks to Anton Scheffer from AMIS, we have a PL/SQL package that reads .XSLX files from a file in a file server folder that has been mapped to an Oracle directory.  That sounds like a mouthful, but it is simple.

You have files in a folder.  You or your system administrators arrange for that file system folder to be available to the database – usually a mapping of the physical folder to a virtual folder on the database file server.  Then your DBA creates an Oracle directory that points to the folder, physical or virtual, on the database file system. And grants appropriate grants so you can see and read and/or write from /to that Oracle directory.

Now Anton’s package, READ_XLSX comes in.  download the package, read Anton’s blog post.  I am only going to summarize here.

READ_XLSX consists of two main functions, one to read the contents of an XLSX file into a BLOB (named file2blob), and another to read the contents of that BLOB into records (this one is named “read“).  Cast the results of the read function as a TABLE, and voila, you have the results of your XLSX file.

This statement is the guts of it:

SELECT * FROM TABLE( as_read_xlsx.read( as_read_xlsx.file2blob('MY_ORA_DIRECTORY',
 'MyExcelWorkbook.xlsx' ) ) );

Now if you are like me, you do not want ot have to keep typing that SELECT statement.  So I turn it into a view:

CREATE VIEW MY_SPREADSHEET_V AS SELECT * FROM TABLE( as_read_xlsx.read( as_read_xlsx.file2blob('MY_ORA_DIRECTORY',
 'MyExcelWorkbook.xlsx' ) ) );

Now I can query from the view the same as I query from any other table or view I have access.  And incorporate queries from that view into any packages, procedures and functions that I build to meet the requirements of the task at hand.

When I have a series of files in a folder, I use a version of GET_DIR_LIST to read files from the Oracle directory.  (GET_DIR_LIST is a java utility, wrapped in PL/SQL, that returns the list of file names in a folder in a one-column table). You probably have your own method of doing the same.  Then, in a loop, I use dynamic SQL to point my view to the next file in the loop, call my loading procedure to process the contents of that file, then move on to the next file.  OF course I have all kinds of validations and error checking in there, as much is needed depending on the task at hand.  Once configured, the process is clean, reusable, easily customizable and – best of all – works in APEX 5.0 and above.

I did not use Anton’s READ_XLSX directly.  Why?  At the time I downloaded it (years ago now), Anton’s package handled cells of up to 4000 characters.  The spreadsheets I had to load contains some cells with > 4000 characters.  So I made a slight adjustment to handle CLOBs.  I described those customizations here.  I suspect most persons will not need such a customization.

Another (Better?) Option – EXCEL2COLLECTION Plugin:

Is READ_XLSX the only way? No. Seems like too much work? Yup, it can be.  Anton apparently thought so too, because he came up with EXCEL2COLECTION, a utility APEX Plugin that, with less coding than the READ_XLSX package, will read data from an XLSX, XSL, XML 2003 or CSV file into a PL/SQL collection.  I have also used this plugin successfully across several customers and projects.

I have used the EXCEL2COLLECTION plugin in conjunction with a customized instance of the APEX Data Load Wizard to enable simpler (from a user perspective) upload of XLSX files.  I describe more of that process here.  The APEX Data Load Wizard does not out-of-the-box allow upload of XLSX files – one needs to cut/paste.  Integrating the EXCEL2COLLECTION plugin into a customized Data Load Wizard series of pages makes use of other features of the Data Load Wizard – like the column mapping page – possible.  Beware, that customizing the Data Load Wizard pages and process flow means that your work is subject to break upon any and every APEX upgrade.  This customization of the Data Load Wizard process is unsupported territory.  Note that the EXCEL2COLLECTION plugin itself is supported (by Anton). My use of it in customizing the Data Load Wizard is not.

Bottom line:

Anton’s READ_XLSX remains one of the best methods of reading XLSX data into an Oracle database, when the requirement is to read many XLSX files.  When using APEX, investigate using the EXCEL2COLLECTION plugin; that will make your XLSX load tasks easier.  For one-of data loads, I still use SQL Developer or the APEX SQL Workshop Data Workshop utility.  Need to script the load for Production?  Use SQL Developer in a DEV environment, then use SQL Developer to generate a load script, there are several options for that.

Happy Data Loading!