In the past few weeks I've had the opportunity to study the Generic Sensor Format (GSF) version of swath bathymetry data files and pick appart how it works. GSF was created by [SAIC] (www.saic.com) for the Navy’s hydrographic programs and is supported by most major software suites for the processing and display of swath bathymetry data including Caris and Fledermaus, among many others. My experience is limited but I thought it might be generally helpful to provide a primer on how to use the library.

Our goal here will be to show how to create a GSF file using the library and write data to it, followed by how one reads data from a GSF file that you or someone else has created. The code will be in C. I won’t handle all cases, nor will I cover every detail or gotchya. But it should be enough to get most knowledgeable c programmers started.

Getting GSF

Source code for the GSF library can be downloaded at the SAIC GSF web site. I'm leaving it up to you to sort out the proper method for compiling it and where it will live. If you're on a unix-like system, you might want to write your own Makefile for it and install it as a library in /usr/local/lib with the header file in /usr/local/include. But for testing purposes maybe you just want it all in the local directory.

I will mention that GSF was not written with common gcc compiler flags such as -Wall -ansi -pedantic -pedantic-errors -Werror turned on. Therefore turning these on in your own code will cause a whole host of compile errors.

Writing a GSF file.

1. Include the gsf header file.

   
   
   #include "gsf.h"
   
   
   

2. Declare some standard GSF variables.

   To understand this step it is helpful to understand how GSF works. GSF defines a parent data structure (gsfRecords) that is a container for every possible kind of data record you can write to a file. It is a structure of structures, if you will. Data records in this structure are populated with the data you wish to write, and the whole thing is the passed to gsfWrite which will in turn encode the data and write it to your file. Similarly, when reading from a file, the data read from each record is decoded and populates the appropriate data record structure within gsfRecords which was passed to gsfRead. Here is the definition of the gsfRecords structure showing the parent data records that one can write to or read from a gsf data file.:

   typedef struct t_gsfRecords
   {
   gsfHeader               header;
   gsfSwathBathySummary    summary;
   gsfSwathBathyPing       mb_ping;
   gsfSingleBeamPing       sb_ping;
   gsfSVP                  svp;
   gsfProcessingParameters process_parameters;
   gsfSensorParameters     sensor_parameters;
   gsfComment              comment;
   gsfHistory              history;
   gsfNavigationError      nav_error;
   gsfHVNavigationError    hv_nav_error;
   gsfAttitude             attitude;
   } gsfRecords;
   

   You'll note a few things here. You can encode both single beam and swath bathymetry data. You can encode sensor parameters, which includes things like sonar system installation survey offsets and roll and pitch biases. You can encode comments, sound velocity profiles (this should be sound speed profiles, as sound speed is a scalar value not a vector quantity), and although GSF is generally a synchronous data format, meaning that all ancillary data streams such as navigation, vessel attitude are interpolated to the ping times and recorded within ping records, one can also encode attitude asynchronously in their own records.

   As an aside, GSF suffers, in my opinion, from a maintenance model that is limited by what SAIC is paid to do. It also suffers from having too many features that are specific to SAIC’s own data processing suite (SABER) and they are therefore sometimes of little use or generally confusing to other folks. The fact that attitude can be encoded asynchronously, but no other data stream can, seems to me to be a break and inconsistency in the previously established philosophy of GSF which makes GSF confusing. I digress.

   To start we need a name for our file. We need a container of the gsfRecords type to hold the data we want to write to the file. We also need a gsfDataID record which holds information about which of the possible records we will actually write in a call to gsfWrite. Also, gsfWrite has the ability to return the unencoded byte stream read from the file. We won’t use this, but we need a place for this to go. So we create a pointer to a buffer for it. Finally, the gsf library keeps track of the file names, their FILE* file handle and other metadata in a “file table” for the files you are reading or writing from. This table is indexed with an integer “file handle” (you can have up to four files open at once). We'll define that too.

   char            filename[30] = "test.gsf";
   gsfRecords      record;
   gsfDataID       id;
   unsigned char*  buf;
   int             handle;
   

3. Open a GSF file for writing.

   The next step is to call the function gsfOpen to open a GSF file for writing.

   if ( gsfOpen(filename, GSF_CREATE, &handle) != 0 ) {
               printf("Failed to open new GSF file!\n");
               return(-1);
       }
   

   Here we see that gsfOpen takes a the name of the file, a flag for how the file will be accessed and the address of the file handle so the handle can be returned. Here are the modes in which you can open a file (from gsf.h):

   
   
   /* Define the GSF data file access flags */
   #define GSF_CREATE             1
   #define GSF_READONLY           2
   #define GSF_UPDATE             3
   #define GSF_READONLY_INDEX     4
   #define GSF_UPDATE_INDEX       5
   #define GSF_APPEND             6
   
   
   

   One detail I'm overlooking in the code block above is gsfError. gsfError is a global variable defined in gsf.h for the entire library. It is used to convey many type of errors (which are defined in #define statements within gsf.h) through the code, including the following ones pertaining to gsfOpen.

   
   
   * Error Conditions :
   *     GSF_BAD_ACCESS_MODE
   *     GSF_TOO_MANY_OPEN_FILES
   *     GSF_FOPEN_ERROR
   *     GSF_SETVBUF_ERROR
   *     GSF_UNRECOGNIZED_FILE
   
   
   

   I should be checking gsfError on failure of gsfOpen and providing more information, but I'm being lazy for now. You'll do better.

   Note that the gsfHeader record is written to the file automatically when a new file is opened, and read from it when an existing file is opened.

4. Populate your data record with data to write.

   In this step, we'll fill in a data record with the data we want encoded in our GSF data file. As an example, we'll consider writing a gsfSwathBathyPing record. It is a big one and understanding it makes most of the others easy. Here is the record definition:

   typedef struct t_gsfSwathBathyPing
   {
   
   
   struct timespec    ping_time;          /* seconds and nanoseconds */
   double             latitude;           /* in degrees, positive going north */
   double             longitude;          /* in degrees, positive going east */
   double             height;             /* height above ellipsoid, positive value defines a point above ellipsoid */
   double             sep;                /* distance from ellipsoid to vertical datum, positive value indicates datum above ellipsoid */
   short              number_beams;       /* in this ping */
   short              center_beam;        /* offset into array (0 = portmost outer) */
   unsigned short     ping_flags;         /* flags to mark status of this ping */
   short              reserved;           /* for future use */
   double             tide_corrector;     /* in meters */
   double             gps_tide_corrector; /* in meters */
   double             depth_corrector;    /* in meters */
   double             heading;            /* in degrees */
   double             pitch;              /* in degrees */
   double             roll;               /* in degrees */
   double             heave;              /* in meters    */
   double             course;             /* in degrees */
   double             speed;              /* in knots */
   gsfScaleFactors    scaleFactors;       /* The array scale factors for this data */
   double            *depth;              /* depth array (meters) */
   double            *nominal_depth;      /* Array of depth relative to 1500 m/s */
   double            *across_track;       /* across track array (meters) */
   double            *along_track;        /* along track array (meters) */
   double            *travel_time;        /* roundtrip travel time array (seconds) */
   double            *beam_angle;         /* beam angle array (degrees from vertical) */
   double            *mc_amplitude;       /* mean, calibrated beam amplitude array (dB re 1V/micro pascal at 1 meter) */
   double            *mr_amplitude;       /* mean, relative beam amplitude array (dB re 1V/micro pascal at 1 meter) */
   double            *echo_width;         /* echo width array (seconds) */
   double            *quality_factor;     /* quality factor array (dimensionless) */
   double            *receive_heave;      /* Array of heave data (meters) */
   double            *depth_error;        /* Array of estimated vertical error (meters) */
   double            *across_track_error; /* Array of estimated across track error (meters) */
   double            *along_track_error;  /* Array of estimated along track error (meters) */
   unsigned char     *quality_flags;      /* Two bit beam detection flags provided by Reson sonar */
   unsigned char     *beam_flags;         /* Array of beam status flags */
   double            *signal_to_noise;    /* signal to noise ratio (dB) */
   double            *beam_angle_forward; /* beam angle forward array (degrees counterclockwise from stbd.) */
   double            *vertical_error;     /* Array of estimated vertical error (meters, at 95% confidence) */
   double            *horizontal_error;   /* Array of estimated horizontal error (meters, at 95% confidence) */
   unsigned short    *sector_number;      /* Array of values that specify the transit sector for this beam */
   unsigned short    *detection_info;     /* Array of values that specify the method of bottom detection */
   double            *incident_beam_adj;  /* Array of values that specify incident beam angle adjustment from beam_angle */
   unsigned short    *system_cleaning;    /* Array of values that specify data cleaning information from the sensor system */
   double            *doppler_corr;       /* Array of values used to correct the travel times for Doppler when transmission is FM */
   int                sensor_id;          /* a definition which specifies the sensor */
   gsfSensorSpecific  sensor_data;        /* union of known sensor specific data */
   gsfBRBIntensity   *brb_inten;          /* Structure containing bathymetric receive beam time series intensities */
   
   
   }
   gsfSwathBathyPing;
   

   Most of this is an exercise in transcription, however understanding how the data gets encoded is essential to getting out what you put in.

   First, don’t be afraid, there is no requirement to fill all of this in. In fact, GSF places few if any requirements on what fields are filled in, other than by common practice. The variation that results in GSF files created by different sonar vendors and data processors causes all kinds of compatibility issues. If you are doing research you can get around this. If you are producing commercial production code, it requires that you do lots of checking of fields on your own.

   Second, you should no that although many of the fields are defined as doubles, they are not encoded into the files as doubles. In general, most values are multiplied by a scaling factor to capture the number of significant digits thought to be needed, and then the result is converted to signed or unsigned integer (short or long depending). This multiplier is hard-coded into the GSF library and most of the time it is set to a reasonable value. Occasionally you will specify a value with more significant digits than the library supports for that field and it will be truncated. But this is admittedly rare.

   This multiplier should not be confused with the scaleFactors table, which holds a multiplier and offset for each of the subrecords. We'll get to that in a minute.

   You assign each of the variables in the record to values from your data. For example:

   record.mb_ping.latitude = 43.1234567;
   record.mb_ping.longitude = 70.001234;
   ...
   

   For fields that are not defined, GSF defines some standard values to indicate that they have purposely been omitted. These are defined in gsf.h:

   
   
   /* Define null values to be used for missing data */
   #define GSF_NULL_LATITUDE               91.0
   #define GSF_NULL_LONGITUDE             181.0
   #define GSF_NULL_HEADING               361.0
   #define GSF_NULL_COURSE                361.0
   #define GSF_NULL_SPEED                  99.0
   #define GSF_NULL_PITCH                  99.0
   #define GSF_NULL_ROLL                   99.0
   #define GSF_NULL_HEAVE                  99.0
   #define GSF_NULL_DRAFT                   0.0
   #define GSF_NULL_DEPTH_CORRECTOR        99.99
   #define GSF_NULL_TIDE_CORRECTOR         99.99
   #define GSF_NULL_SOUND_SPEED_CORRECTION 99.99
   #define GSF_NULL_HORIZONTAL_ERROR       -1.00
   #define GSF_NULL_VERTICAL_ERROR         -1.00
   #define GSF_NULL_HEIGHT               9999.99
   #define GSF_NULL_SEP                  9999.99
   
   
   
   
   #define GSF_NULL_SS                    0.0
   
   /* Define macros used to indicate that a certain parameter is not known */
   #define GSF_BEAM_WIDTH_UNKNOWN        -1.0
   
   
   

   At the bottom of the gsfSwathBathyPing record definition above you will see numerous pointers defined to various arrays. For example…

   double            *depth;              /* depth array (meters) */
   double            *nominal_depth;      /* Array of depth relative to 1500 m/s */
   double            *across_track;       /* across track array (meters) */
   

   In the lingo of GSF, these are “subrecords”. Any of these may be assigned values in a given ping record. Those that you don’t assign are not are set to NULL like this.

   record.mp_ping.depth = (double*) NULL;
   

   Because none of these are required, it is possible to have bathymetric ping records with no depth values in it. I think Geoacoustics GS+ program creates GSF files of this type (they populate amplitude only. (Why is a long story.)

   It is good to know that the length of each of these array subrecords must be equal to the number of beams in the ping (as assigned to record.mb_ping.number_beams). If the array is larger, some of the data will not be encoded. If the array is smaller I'm not sure what happens – probably a segmentation fault. As an example, assignment of the depth array from “mydepths” might look something like this:

   
   
   /* Assigne memory to the array pointer */
   if( (record.mb_ping.depth = malloc(record.mb_ping.number_beams * sizeof(double) )) == NULL) 
   {
       printf("Memory Allocation Failed.\n";
       return(-1);
   }
   
   for( i=0;i<record.mb_ping.number_beams; i++) {
   
       record.mb_ping.depths[i] = mydepths[i];
   }
   
   
   

   It is possible that an array will have an undefined value. For example, some sonars do not report depth soundings for a beam when no depth was found. In that case GSF specifies a default value to stick in the field. This might be a bit confusing because the value itself is insufficient to determine if the field might be valid. Rather the beam_flags specifies which beams are to have valid data.

   
   
   /* Define null values for the swath bathymetry ping array types. Note that
    * these zero values do not necessarily indicate a non-valid value.  The
    * beam flags array should be used to determine data validity.
    */
   #define GSF_NULL_DEPTH                 0.0
   #define GSF_NULL_ACROSS_TRACK          0.0
   #define GSF_NULL_ALONG_TRACK           0.0
   #define GSF_NULL_TRAVEL_TIME           0.0
   #define GSF_NULL_BEAM_ANGLE            0.0
   #define GSF_NULL_MC_AMPLITUDE          0.0
   #define GSF_NULL_MR_AMPLITUDE          0.0
   #define GSF_NULL_ECHO_WIDTH            0.0
   #define GSF_NULL_QUALITY_FACTOR        0.0
   #define GSF_NULL_RECEIVE_HEAVE         0.0
   #define GSF_NULL_DEPTH_ERROR           0.0
   #define GSF_NULL_ACROSS_TRACK_ERROR    0.0
   #define GSF_NULL_ALONG_TRACK_ERROR     0.0
   #define GSF_NULL_NAV_POS_ERROR         0.0
   
   
   

   Repeat the process for each subrecord you would like to fill in.

   The beam_flags array is important one worth mentioning. This array defines a set of flags that specify which beams contain valid data and which should be ignored. The values to set are defined in gsf.h:

   
   
   /* Define the GSF bit flags flags for the beam status array.
    * The GSF_IGNORE_BEAM flag may be set to indicate that this beam should
    *  not be used by any processing/display software.  The flags
    *  GSF_BEAM_USER_FLAG_01-07 may be set/read by application specific software
    */
   #define GSF_IGNORE_BEAM       (unsigned)0x01
   #define GSF_BEAM_USER_FLAG_01 (unsigned)0x02
   
   
   

   and can be set, cleared and checked easily with these macros (which are for ping status flags, but work the same).:

   
   
   /* Define a set of macros to set, clear, and test the state of the
    *  ping status flags.
    *  Where:
    *     ping_flags: The ping flags field of the gsfSwathBathyPing structure.
    *     usflag:     The definition of the flag to test, set, or clear.
    */
   #define gsfTestPingStatus(ping_flags, usflag)    (((ping_flags) & (usflag)) ? 1 : 0)
   #define gsfSetPingStatus(ping_flags, usflag)      ((ping_flags) |= (usflag))
   #define gsfClearPingStatus(ping_flags, usflag)    ((ping_flags) &= (~(usflag)))
   
   
   

   As I mentioned above, GSF has a scaleFactors table which contains, for each subrecord, a multiplier, an offset and a compression flag which are used by the library to encode each of the array subrecords into the file in an efficient way. So before we can write the ping record to the file, we need to set those values.

   The scaleFactors table is not held in a global variable, but it is passed around through functions with a similar result. It has the same number of rows as there are possible subrecords and it is indexed by the subrecord ID. Subrecord IDs are defined as #define statements in gsf.h. Maybe it makes more sense to see how it is defined.

   
   
   /* Define the internal form of the array subrecord scale factor information,
   * which is used to scale the swath bathymetry ping record to/from
   * internal/external form. The subrecord id is specified by the index into
   * the scaleTable array.
   */
   typedef struct t_gsfScaleInfo
   {
       unsigned char   compressionFlag;    /* Specifies bytes of storage in high order nibble and type of compression in low order nibble */
       double          multiplier;         /* the scale factor (millionths)for the array */    double          offset;             /* dc offset to scale data by */
   } gsfScaleInfo;
   
   typedef struct t_gsfScaleFactors
   {    int             numArraySubrecords; /* the number of scaling factors we actually have */
   gsfScaleInfo    scaleTable[GSF_MAX_PING_ARRAY_SUBRECORDS];
   } gsfScaleFactors;
   
   
   

   So the scaleTable has a number of rows equal to the maximun number of possible subrecords (not the number of records in this ping, but the number that are defined in the GSF library). And for each one, is defined a compressionflag, offset and multiplier.

   Honestly, I haven’t enough experience with GSF to provide much guidance on how to set the multipler and offset values. However we can look to MB-System, the open source swath mapping software and see how they do it. From MB-System/src/mbio/mbsys_gsf.c:

   
   
   /* get scale factor for bathymetry */                bathmax = 0.0;
   bathacrosstrackmax = 0.0;
   bathalongtrackmax = 0.0;
   for (i=0;i<nbath;i++)                        
   {
       if (beamflag[i] != MB_FLAG_NULL)
       {
           bathmax = MAX(bathmax, fabs(bath[i]));                            bathacrosstrackmax = MAX(bathacrosstrackmax, fabs(bathacrosstrack[i]));
           bathalongtrackmax = MAX(bathalongtrackmax, fabs(bathalongtrack[i]));                  }
   }
   if (bathmax < 10.0)
      mb_ping->scaleFactors.scaleTable[GSF_SWATH_BATHY_SUBRECORD_DEPTH_ARRAY-1].multiplier = 1000.0;
   
   
      else if (bathmax < 100.0)
   
   mb_ping->scaleFactors.scaleTable[GSF_SWATH_BATHY_SUBRECORD_DEPTH_ARRAY-1].multiplier = 100.0;
   
   else if (bathmax < 1000.0)
   
   mb_ping->scaleFactors.scaleTable[GSF_SWATH_BATHY_SUBRECORD_DEPTH_ARRAY-1].multiplier = 10.0;
   
   else
   
   mb_ping->scaleFactors.scaleTable[GSF_SWATH_BATHY_SUBRECORD_DEPTH_ARRAY-1].multiplier = 1;
   
   
   

   In the code snippet above the maximum depth value is found (the for loop) and then multipliers are set based on the maximum value’s order of magnitude. I can’t see that the offset value is set anywhere in the mbsystem code, so I assume it to be zero. The compression flag is set with #define variables which can be found within gsf.h. As of version 3.1, I do not think compression of these records is actually supported.

   The only other part we have to define are the sensor specific subrecords. For most every sonar that GSF supports there is a sensor specific records. To see these you'll have to take a look at gsf.h. These are optional records in that they may or may not be found in any given ping record. How optional they are to other software packages reading the GSF file is unknown to me. As an example, here is the sensor specific record definition for the EM121A:

   
   
   /* Define the EM121A specific data structure */
   typedef struct t_gsfEM121ASpecific
   {
       int             ping_number;
       int             mode;
       int             valid_beams;
       int             pulse_length;
       int             beam_width;
       int             tx_power;
       int             tx_status;
       int             rx_status;
       double          surface_velocity;
   }
   t_gsfEM121ASpecific;
   
   
   

   A field in this record would be assigned with a statement like this:

   record.mb_ping.sensor_specific.surface_velocity = 1500.0;
   

   Be sure to also set the sensor_id field in the ping record as this is used to identify the appropriate sensor specific record to encode/decode from the file.

5. Assign the “gsfDataID” variable to the record you want to write.

   Although gsfWrite accepts a pointer to a structure containing to all the possible records you can write (e.g. gsfRecords) it only actualy writes one of them on each call. Which one is determined by the gsfDataID which is also passed to gsfWrite. We defined id as type DataID above, so we define it now.

   
   
   id.checksumFlag = 0;
   id.recordID = GSF_RECORD_SWATH_BATHYMETRY_PING;
   id.record_number = 1;
   
   
   

   The checksumFlag is a boolean value. The record ID is defined for each type of record in gsf.h #define statements. For record_number the following guidance is provided in gsf.h:

   
   
   /* specifies the nth occurance of */
   /* record type specified by recordID */
   /* relavent only for direct access */
   /* the record_number counts from 1 */
   
   
   

   As an aside, I do not think that MB-System sets or increments the data record at all.

6. Write your data record to the file.

   You've done all the hard already. The next step requires only calling gsfWrite to write the record to the file.

   if (gsfWrite(handle, &id, &record) <= 0) {
       printf("Failed to write ping data.\n");
   }
   

   gsfWrite returns the number of bytes written which you may want to capture if you're keeping score.

7. Repeat

   You repeat the process for subsequent records you wish to write, first populating the records.xxx placeholder with the data, then setting the appropriate fields in the DataID and calling gsfWrite. Note that if the number of beams change (which is possible) you'll have to free your memory and reallocate it. To quickly free your memory you can use the GSF utility function:

   gsfFree(record);
   

8. Close the file and free up our memory.

   gsfFree(record);
   gsfClose(handle);
   

   Pretty straight forward.

Reading a GSF file.

Now that we've done the writing, now lets see how to do the reading. The process is similar.

1. Define variables.

   We begin by defining the same variables to hold the data and to specify the type of record we've read, and of course the name of the file we want to read.

   
   
   char            filename[30] = "test.gsf";
   gsfRecords      record;
   gsfDataID       id;
   unsigned char*  buf;
   int             handle;
   
   
   

2. Open the file for reading.

   
   
   if (gsfOpen(filename, GSF_READONLY, &handle) != 0) {
       printf("Failed to open GSF file for reading.");
       return(-1);
   }
   
   
   

3. Read the next record.

    
   
   
   
   /* read the test record from the file */
   if (gsfRead(handle, GSF_NEXT_RECORD, &id, &record, buf, 0) < 0) {
       printf("Failed to read GSF record\n");
   }
   
   
   

   Here GSFNEXTRECORD tells gsfRead that we want whatever is the next record in the file. We can alternatively specify a particular record type, in which case gsfRead will search through the file until that record type is found and return it. The record data is placed in our record structure and the type of record that was read is specified in the gsfDataID variable (id).

4. See and check what we've got.

   Because there is little to no error or sanity checking on what is written to a GSF file, the onus is on the reader to check the fields himself. One should not, for example, get latitude values greater than 90, although nothing prohibits one from writing such a file.

   One will want to check for the standard values to be assigned for missing data and which subrecords (arrays) have null pointers so one knows within what subrecords to look for data.

   It is clear to me that each call to gsfRead clears pointers in the caller’s array (by assigned them to NULL) and then reassigns them to new memory locations when the next record is read. But it is not clear to me that the original memory is freed back to the system and one must not call gsfFree(record) between subsequent reads to prevent a memory leak. If the number of beams doesn’t change, it would not make a difference. If the number of beams does change, it might.

5. Close the file and free memory.

   When you're done reading data, then we free up the memory and close the file.

   gsfFree(record);
   gsfClose(handle);
   

I think that’s it. I'm sure I have left out all kinds of details, but this should get you started in reading and writing GSF data files.

I like this post because when I worked at Qwest Communications, long ago, when they were still a wily startup and much less of a lumbering utility company I tried to convince my superiors to do the same thing. I wanted to create a video and audio link between two conference rooms – one in our local office and one in Qwest’s headquarters in Denver. I wanted to place the cameras and projection screen to create the virtual effect of one room and conference table extending into the other room and conference table. If done correctly, I think one could have created a pretty seamless link between the two rooms that would allow teams of people on both ends to more effectively work together. It would have been great for us, it would have been a fantastic thing to sell. Qwest was/is sitting on loads of unlit fiber, but at the time, there was no demand for it because software and peripheral hardware could not keep up with the increase in capacity created by multiple frequency multiplexing (among other technologies). IMO, Qwest had neither the vision nor the guts to invest heavily in the software development that would have driven demand for their networks. There were too many people looking out for number one. Now they are a huge local phone company whose share price has been above $10/share for almost a decade.

map = [[0 0 0]; colormap];
colormap(map);

Today’s Quote of the Day:

Lela: "I thought you knew this person."

Han: "Well, that was a long time ago. I'm sure he's forgotten about that." 

What I should really do is read up on how to use svn from inside subversion. Perhaps issues like this would not crop up.

Here I execute a double loop to do the matrix math on the 4D matrics, M, xyz, Cxyz and J. These calculations rotate xyz triplets in xyz, through a set of rotations specified in the 3x3 matrics in M and propagate the uncertainties Cxyz through the calculation using the Jacobian matrices in J.

 for idx=1:Xrows
    for jdx=1:Xcols
        CXYZ(:,:,idx,jdx) = J(:,:,idx,jdx)*Cxyz(:,:,idx,jdx)*J(:,:,idx,jdx)';

        XYZ(:,:,idx,jdx) = M(:,:,idx,jdx)*xyz(:,:,idx,jdx)';
    end
end

Time to execute when the number of rows is 2118 and the number of columns is 3962:

Elapsed time is 1622.861264 seconds.

Now, using mtimesx on the same input data:

CXYZ = mtimesx(J,mtimesx(Cxyz,J,'t'));
XYZ = mtimesx(M,xyz,'t');

Time to execute:

Elapsed time is 74.202723 seconds.

That comes to a whopping 2087.1% increase in processing speed!

First turn your X, Y and Z matrices into a single 4D matrix of xyz triplets like this.

xyz = shiftdim(cat(4,x,y,z),2);

The resulting matrix will be 3x1xRowsxCols and you can access a single xyz triplet vector (the j,kth one) like this:

xyz(:,:,j,k)

You can do the same thing with your rotation matrix with M by repeating the rotation matrix, R, for each xyz triplet you've got like this:

M = repmat(R,[1,1,rows,cols])

Now you can do one of two things. You can use MATLAB’s native code to double loop over the rows and columns like this:

for idx=1:rows
    for jdx=1:cols
    XYZ(:,:,idx,jdx) = M(:,:,idx,jdx)*xyz(:,:,idx,jdx);
    end
end

Or you can download and compile the routine mtimesx, which will do this math for you in one shot:

XYZ = mtimesx(M,xyz);

You may have to transpose the xyz vector on the fly. This can be done simply with:

XYZ = mtimesx(M,xyz,'t');

And this is FAST!

profile -memory on

setpref('profiler','showJitLines',1)