Thread: Weird byte swapping-new MacPro + SAC102

Started: 2021-10-07 21:27:34
Last activity: 2021-11-18 06:00:36
Topics: SAC Help
Craig Jones
2021-10-07 21:27:34
All,

We’ve run into something we don’t fully understand. We have a new MacPro running the current OS with SAC 102 installed, and then we have an ancient MacPro (c2011) running 101.6 under High Sierra. Getting SAC files with SOD from either machine results in the same files on both machines. SAC on both machines will read these files. Writing them back out on the old machine basically reproduces the file. Writing it out on the new machine results in the bytes being seriously reordered, and not just with a trivial byte swap. IN the original file, say, bytes are in order 1-2-3-4 5-6-7-8. In the output version on the new Mac we see 4-3-2-1 8-7-6-5 ordering. Other codes that use SAC are unable to read these. We’re not too sure why this would be the case or what i/o option might correct it (sacswap under 101.6 does not do anything to these files). I’ve been looking at this with the octal dump command (od).

Thanks for any insight,

Craig
--
Craig H. Jones
Professor, Dept. Geological Sciences, CB 399
and Fellow, CIRES, CB 216
University of Colorado, Boulder, CO 80309-0399
http://cires1.colorado.edu/people/jones.craig/



  • George Helffrich
    2021-10-25 16:44:01
    Dear Craig -

    It sounds like you are running on a big-endian machine - is that possible for your new Mac? All of SAC’s fields are 4 bytes long. What you describe seems to be a word-for-word byte-swapped file. I think that IRIS SAC will write native byte order always, so if the machine is big endian and the original file is little endian, then that’s what you would see.

    The Unix “file” command might also tell you if your SAC runs in some byte-swapped environment.

    Here’s a Fortran test program to display your machine's endianness.




    On 25 Oct 2021, at 09:49, Craig Jones (via IRIS) <sac-help-bounce<at>lists.ds.iris.edu> wrote:

    All,

    We’ve run into something we don’t fully understand. We have a new MacPro running the current OS with SAC 102 installed, and then we have an ancient MacPro (c2011) running 101.6 under High Sierra. Getting SAC files with SOD from either machine results in the same files on both machines. SAC on both machines will read these files. Writing them back out on the old machine basically reproduces the file. Writing it out on the new machine results in the bytes being seriously reordered, and not just with a trivial byte swap. IN the original file, say, bytes are in order 1-2-3-4 5-6-7-8. In the output version on the new Mac we see 4-3-2-1 8-7-6-5 ordering. Other codes that use SAC are unable to read these. We’re not too sure why this would be the case or what i/o option might correct it (sacswap under 101.6 does not do anything to these files). I’ve been looking at this with the octal dump command (od).

    Thanks for any insight,

    Craig
    --
    Craig H. Jones
    Professor, Dept. Geological Sciences, CB 399
    and Fellow, CIRES, CB 216
    University of Colorado, Boulder, CO 80309-0399
    http://cires1.colorado.edu/people/jones.craig/



    ----------------------
    SAC Help
    Topic home: http://ds.iris.edu/message-center/topic/sac-help/ | Unsubscribe: sac-help-unsubscribe<at>lists.ds.iris.edu

    Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
    Update subscription preferences at http://ds.iris.edu/account/profile/



    George Helffrich
    george<at>elsi.jp


    Attachments
    • Craig Jones
      2021-11-18 06:00:36
      All,

      Sorry for the silence. A solution was found locally and as I was passing on from the person grappling with this, it wasn’t entirely clear what was up.

      Anyways, all the machines involved still have Intel processors so there was no change in endianness here. But apparently there was something changed at IRIS. The incarnation of this problem for me was that files we got through Wilbur would read OK in SAC but a matlab script I had for reading files was choking on the novel ordering. I was pointed to a C code (saccvt) that seems to get things all in the right order, which came from Bob Herrmann long ago (2003!) which I am pasting below (since sometimes the mail servers aren’t happy with attachments). This seems to do the trick with the -I option.

      Craig

      /* convert SAC binary SUN to SAC binary Intel */
      /* Robert B. Herrmann
      May 8, 2001
      Seoul National University

      Changes:

      27 APRIL 2003 - build in intelligence to determine if byte order
      should be reversed. the criteria is that
      o at least one integer header = -12345
      o and NO integer header values are < -99999
      note we could also try to look at the year doy field
      for time series (but may not work for spectra other stuff
      */

      #include <stdio.h>
      #include <stdlib.h>
      #ifdef MSDOS
      #include <fcntl.h>
      #endif
      #include <strings.h>

      #define BUFFLOAT 280
      #define BUFINTEGER 160
      #define BUFCHAR 192
      #define BUF 8192

      #define DO_FLOAT 1
      #define DO_INTEGER 2

      char arr[BUF];
      int iarr[BUFINTEGER/4];

      void ieee2intel( char *arr, int nbytes, int type);
      void gcmdln(int argc, char **argv, int *do_intelligence);
      void usage();
      int do_sac_swap(void);

      int aargc;
      char **aargv;



      main(int argc, char**argv)
      {
      int nread;
      int do_intelligence;
      int do_convert;
      /* parse fommand line arguments */
      aargc = argc;
      aargv = argv;
      gcmdln(aargc, aargv, &do_intelligence);

      /* apply test that tries to guess the order */
      if(do_intelligence != 0){
      rewind(stdin);
      do_convert = do_sac_swap();
      rewind(stdin);
      } else {
      do_convert = 1;
      }
      fprintf(stderr,"do_intelligence %d do_convert %d\n",do_intelligence, do_convert);


      #ifdef MSDOS
      setmode(fileno(stdin),O_BINARY);
      setmode(fileno(stdout),O_BINARY);
      #endif
      if ((nread=fread(arr, sizeof(char), BUFFLOAT, stdin)) == BUFFLOAT){
      if(do_convert==1)ieee2intel( arr, BUFFLOAT, DO_FLOAT );
      fwrite(arr,sizeof(char), BUFFLOAT, stdout);
      } else {
      exit (1);
      }

      if ((nread=fread(arr, sizeof(char), BUFINTEGER, stdin)) == BUFINTEGER){
      if(do_convert==1)ieee2intel( arr, BUFINTEGER, DO_FLOAT );
      fwrite(arr,sizeof(char), BUFINTEGER, stdout);
      } else {
      exit (1);
      }

      if ((nread=fread(arr, sizeof(char), BUFCHAR, stdin)) == BUFCHAR){
      fwrite(arr,sizeof(char), BUFCHAR, stdout);
      } else {
      exit (1);
      }

      while((nread=fread(arr, sizeof(char), BUF, stdin))> 0) {
      if(do_convert==1)ieee2intel( arr, nread/4, DO_FLOAT );
      fwrite(arr, sizeof(char), nread, stdout) ;
      }
      }

      void ieee2intel( char *arr, int nbytes, int type)
      /* arr array of bytes to be converted in place
      nbytes number of bytes to be converted
      type 0 float
      */
      {
      char a, b, c, d;
      char *cp;
      int i;
      for (i=0; i < nbytes*4 ; i+=4 ){
      a= arr[i] ;
      b= arr[i+1];
      c= arr[i+2];
      d= arr[i+3];
      arr[i] = d;
      arr[i+1] = c;
      arr[i+2] = b;
      arr[i+3] = a;
      }
      }

      void gcmdln(int argc, char **argv, int *do_intelligence){

      /* parse the command line flags */
      *do_intelligence = 0;
      while(argc-- > 1){
      if(*argv[1] == '-'){
      switch(argv[1][1]){
      case 'I':
      *do_intelligence = 1;
      break;
      case '?':
      usage();
      break;
      case 'h':
      usage();
      break;
      default:
      break; }
      }
      argv++;
      }
      }

      void usage()
      {
      fprintf(stderr, "Usage: saccvt [-I] [-h] [-?]\n");
      fprintf(stderr, " Convert SAC binary IEEE to INTEL format \n");
      fprintf(stderr, " Convert SAC binary INTEL to IEEE format \n");
      fprintf(stderr, " All 4 byte integers and floats (a,b,c,d) are\n");
      fprintf(stderr, " transposed to (d,c,b,a)\n");
      fprintf(stderr, " Example: saccvt < SAC_BINARY > tmp ; mv tmp SAC_BINARY\n");
      fprintf(stderr, " -I (default none) intelligently guess whether to convert\n");
      fprintf(stderr, " -h (default none) this help message\n");
      fprintf(stderr, " -? (default none) this help message\n");
      exit(0);
      }

      int do_sac_swap(void)
      {
      /* the SAC file consists of a
      * FLOAT HEADER 280 bytes
      * INTEGER HEADER 160 bytes
      * CHARACTER HEADER 192 bytes
      * FLOAT DATA
      */
      /* skip to integer header */
      int neg12345 = 0;
      int havebigneg = 0;
      int i;
      int nread;

      fseek(stdin, (long) BUFFLOAT, SEEK_SET);
      /* read integer header */
      if ((nread=fread(iarr,sizeof(int),BUFINTEGER/4,stdin))==BUFINTEGER/4){
      /* now apply the rules */
      for(i=0;i < BUFINTEGER/4;i++){
      if(iarr[i] == -12345)neg12345++;
      if(iarr[i] < -99999)havebigneg++;
      }
      } else {
      exit (1);
      }
      if(neg12345 ==0 && havebigneg > 0 )
      return(1);
      else
      return(0);

      }


      On Oct 25, 2021, at 5:44 PM, George Helffrich (via IRIS) <sac-help-bounce<at>lists.ds.iris.edu<sac-help-bounce<at>lists.ds.iris.edu>> wrote:

      Dear Craig -

      It sounds like you are running on a big-endian machine - is that possible for your new Mac? All of SAC’s fields are 4 bytes long. What you describe seems to be a word-for-word byte-swapped file. I think that IRIS SAC will write native byte order always, so if the machine is big endian and the original file is little endian, then that’s what you would see.

      The Unix “file” command might also tell you if your SAC runs in some byte-swapped environment.

      Here’s a Fortran test program to display your machine's endianness.

      <check.f>


      On 25 Oct 2021, at 09:49, Craig Jones (via IRIS) <sac-help-bounce<at>lists.ds.iris.edu<sac-help-bounce<at>lists.ds.iris.edu>> wrote:

      All,

      We’ve run into something we don’t fully understand. We have a new MacPro running the current OS with SAC 102 installed, and then we have an ancient MacPro (c2011) running 101.6 under High Sierra. Getting SAC files with SOD from either machine results in the same files on both machines. SAC on both machines will read these files. Writing them back out on the old machine basically reproduces the file. Writing it out on the new machine results in the bytes being seriously reordered, and not just with a trivial byte swap. IN the original file, say, bytes are in order 1-2-3-4 5-6-7-8. In the output version on the new Mac we see 4-3-2-1 8-7-6-5 ordering. Other codes that use SAC are unable to read these. We’re not too sure why this would be the case or what i/o option might correct it (sacswap under 101.6 does not do anything to these files). I’ve been looking at this with the octal dump command (od).

      Thanks for any insight,

      Craig
      --
      Craig H. Jones
      Professor, Dept. Geological Sciences, CB 399
      and Fellow, CIRES, CB 216
      University of Colorado, Boulder, CO 80309-0399
      http://cires1.colorado.edu/people/jones.craig/



      ----------------------
      SAC Help
      Topic home: http://ds.iris.edu/message-center/topic/sac-help/ | Unsubscribe: sac-help-unsubscribe<at>lists.ds.iris.edu<sac-help-unsubscribe<at>lists.ds.iris.edu>

      Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
      Update subscription preferences at http://ds.iris.edu/account/profile/



      George Helffrich
      george<at>elsi.jp<george<at>elsi.jp>


      ----------------------
      SAC Help
      Topic home: http://ds.iris.edu/message-center/topic/sac-help/ | Unsubscribe: sac-help-unsubscribe<at>lists.ds.iris.edu<sac-help-unsubscribe<at>lists.ds.iris.edu>

      Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
      Update subscription preferences at http://ds.iris.edu/account/profile/

      --
      Craig H. Jones
      Professor, Dept. Geological Sciences, CB 399
      and Fellow, CIRES, CB 216
      University of Colorado, Boulder, CO 80309-0399
      http://cires1.colorado.edu/people/jones.craig/


  • Brian Savage
    2021-10-25 17:29:49
    Craig,

    A couple of questions:

    Did you compile or are you running a distributed binary?

    Does the following set of commands work properly on the new MacPro?

    fg impulse
    write tmp.sac
    read tmp.sac

    I am working on finding a kind soul who has a new mac to test SAC 102 on.

    Thanks,
    Brian

    On Oct 25, 2021, at 12:49 PM, Craig Jones (via IRIS) <sac-help-bounce<at>lists.ds.iris.edu> wrote:

    All,

    We’ve run into something we don’t fully understand. We have a new MacPro running the current OS with SAC 102 installed, and then we have an ancient MacPro (c2011) running 101.6 under High Sierra. Getting SAC files with SOD from either machine results in the same files on both machines. SAC on both machines will read these files. Writing them back out on the old machine basically reproduces the file. Writing it out on the new machine results in the bytes being seriously reordered, and not just with a trivial byte swap. IN the original file, say, bytes are in order 1-2-3-4 5-6-7-8. In the output version on the new Mac we see 4-3-2-1 8-7-6-5 ordering. Other codes that use SAC are unable to read these. We’re not too sure why this would be the case or what i/o option might correct it (sacswap under 101.6 does not do anything to these files). I’ve been looking at this with the octal dump command (od).

    Thanks for any insight,

    Craig
    --
    Craig H. Jones
    Professor, Dept. Geological Sciences, CB 399
    and Fellow, CIRES, CB 216
    University of Colorado, Boulder, CO 80309-0399
    https://urldefense.proofpoint.com/v2/url?u=http-3A__cires1.colorado.edu_people_jones.craig_&d=DwIBaQ&c=dWz0sRZOjEnYSN4E4J0dug&r=xlrnQuY_RWoRIvnCDJ5AfQ&m=W44lK3c0VdDpDHBLBUAf_wFjpbFc5J2LtsFR1FHPzDEOs0Tb-S-_KeLYGaxKPmoD&s=cpdB9PvI2AfR3mBoFOg5ePfjITZw3aG8Idkc4idXI_g&e=



    ----------------------
    SAC Help
    Topic home: https://urldefense.proofpoint.com/v2/url?u=http-3A__ds.iris.edu_message-2Dcenter_topic_sac-2Dhelp_&d=DwICAg&c=dWz0sRZOjEnYSN4E4J0dug&r=xlrnQuY_RWoRIvnCDJ5AfQ&m=W44lK3c0VdDpDHBLBUAf_wFjpbFc5J2LtsFR1FHPzDEOs0Tb-S-_KeLYGaxKPmoD&s=zIHMN9ej7pl-MBfLkYKMD8_Ml0W5LRhOoosmzGoGy5M&e= | Unsubscribe: sac-help-unsubscribe<at>lists.ds.iris.edu

    Sent from the IRIS Message Center (https://urldefense.proofpoint.com/v2/url?u=http-3A__ds.iris.edu_message-2Dcenter_&d=DwICAg&c=dWz0sRZOjEnYSN4E4J0dug&r=xlrnQuY_RWoRIvnCDJ5AfQ&m=W44lK3c0VdDpDHBLBUAf_wFjpbFc5J2LtsFR1FHPzDEOs0Tb-S-_KeLYGaxKPmoD&s=nHuX3eD5U8pKlhPK_hO8mGrkCXWh3PyH5noZORVsURM&e= )
    Update subscription preferences at https://urldefense.proofpoint.com/v2/url?u=http-3A__ds.iris.edu_account_profile_&d=DwICAg&c=dWz0sRZOjEnYSN4E4J0dug&r=xlrnQuY_RWoRIvnCDJ5AfQ&m=W44lK3c0VdDpDHBLBUAf_wFjpbFc5J2LtsFR1FHPzDEOs0Tb-S-_KeLYGaxKPmoD&s=n7K1Zzs_10RW6fa2WeJhOHbdoKuU7dDfhTYeGscAQw4&e=



16:27:12 v.01697673