NNG Coders

Discussion in 'Off Topic' started by Torqameda, Aug 28, 2014.

NNG Coders

Discussion in 'Off Topic' started by Torqameda, Aug 28, 2014.

  1. Torqameda

    Torqameda .

    Joined:
    Apr 3, 2014
    Messages:
    97
    Likes:
    115
    Any other computer programmers out there in NNG? If so, I seriously need some help (specifically with Python). :<
     
  2. Canvera

    Canvera NNG Data Analyst

    Joined:
    Oct 22, 2012
    Messages:
    341
    Likes:
    254
    if(dateDue >= new Date() -1)
    {
    throw RuntimeException("I don't give help on the last day");
    }

    In all seriousness, I can help out but if it's a rush project, I myself won't have any time to help out, since I'd need to actually look up python since it has been ages since I've last used it.
     
  3. Torqameda

    Torqameda .

    Joined:
    Apr 3, 2014
    Messages:
    97
    Likes:
    115
    I'm having two primary issues: 1) concatenating and subsequently compressing six binary files (half .idx and the other half a proprietary .son format) into a single .dat file (which is needed in order to convert the data from a .dat to a .xtf format since, unfortunately, the data within is 100% proprietary until it's converted via a son2xtf converter software) and 2) I'm having issues running a .py script in IPython (purpose of it is to convert the data contained in the .xtf file and spit out a .mat... but, for whatever reason, whenever I call the script via command line, it can't be found).
     
  4. Canvera

    Canvera NNG Data Analyst

    Joined:
    Oct 22, 2012
    Messages:
    341
    Likes:
    254
    As I understand it
    You're going from .idx + .son -> .dat -> .xtf -> .mat
    Sounds a bit roundabout but anyhow.

    I stole this off of another place for concatenation in python. Not a language I'm that familiar with but oh well.

    fout = file('C','wb')
    for n in ['A','B']:
    fin = file(n,'rb')
    while True:
    data = fin.read(65536)
    if not data:
    break
    fout.write(data)
    fin.close()
    fout.close()

    This is the groundwork for at least part of it. I don't know the format that your proprietary file would use but hey. At least this would concat all the .idx and .son seperately so you'd have two giant .idx and .son files.

    Anyhow, it then becomes a question of how would you merge an .idx and a .son. Is it straight concatenation?
    If you're doing trickier stuff then it gets more complicated.

    As to running the .py script, I'm going to have to ask how you call it, where you are in the directory tree and any arguments, as well as the spit out error message. Your description is generic enough that it's possible a few errors or configuration mishaps are happening. :V
     
  5. Torqameda

    Torqameda .

    Joined:
    Apr 3, 2014
    Messages:
    97
    Likes:
    115
    I've tried a similar concatenation; the concatenation isn't necessarily the issue, but rather the compression that somehow needs to occur. The converter only reads a specifically formatted .dat that is 64 bytes, so I have no idea how the code is formatted. What's worse is that it's impossible to really know since the file format is protected by its respective company (in this case Humminbird). I'm sure it's possible to crack it but that's beyond the scope of what I'm interested in (and, to be honest, I don't want to spend time doing that either). Likewise, I have to use that roundabout conversion path because of those proprietary issues; basically:

    Humminbird Device ---record---> .idx and .son per recording ---auto-package upon recording termination/save---> .dat ---son2xtf converter---> .xtf format (non-proprietary, wahoo!) ---.xtf to .mat python converter---> course plot which allows for additional statistical and mathematical analysis

    As for the .py script: it's a matter of using the script to convert the .xtf file to that .mat format. Off the top of my head, the first way I tried calling it was simply through the MS command prompt:

    C:\Users\Name> python
    C:\Users\Name> C:\python27\scripts\xtf.py -o R00028.mat -i C:\users\desktop\R00028\R00028.xtf

    That generated at "not a valid Win32 application" error if I recall correctly. I'd have to check my log to see how I called it specifically on IPython.

    EDIT:

    ........

    rofl The computer I was running it on is a 32-bit python, not 64. God damn it.
     
  6. Canvera

    Canvera NNG Data Analyst

    Joined:
    Oct 22, 2012
    Messages:
    341
    Likes:
    254
    When it comes to the first issue, I'm starting to think I have zero ability to help out without some serious reverse engineering. Like, you can automate some of the conversion process, sure, but it seems you need to go through hoops to get the data into the format you want.
    If the converter takes only a specifically formatted .dat, well, I don't know exactly what to tell you.


    For the .py script, I see why you're having that problem. Command line should actually be on one line, instead of two.

    Try python C:\python27\scripts\xtf.py -o R00028.mat -i C:\users\desktop\R00028\R00028.xtf

    Calling python itself doesn't quite do it, and the second line tries to invoke xtf.py without python, and since windows doesn't recognize the file, kebam.

    That hopefully should rectify the whole calling from command line issue.
     
  7. Torqameda

    Torqameda .

    Joined:
    Apr 3, 2014
    Messages:
    97
    Likes:
    115
    I invoke python prior to the command because it simply opens up the interpreter in the command prompt. For instance, upon opening cmd.exe:

    Microsoft Windows [Version 6.1.7600]
    Copry (C) 2009 Microsoft Corporation. All rights reserved.

    C:\Users\Name> python
    Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v. 1500 32 bit (Intel)] on win 32
    Type "help", "copyright", "credits" or "license" for more information
    >>>

    But, in this case, it was unnecessary and was just out of habit. Now it's just a matter of running it on my personal computer (64 bit) as opposed to the 32-bit due to memory issues.
     
  8. Torqameda

    Torqameda .

    Joined:
    Apr 3, 2014
    Messages:
    97
    Likes:
    115
    Figured out the .dat issue. Fuck metadata.