Reading PTCOP data, or converting to MIDI. [ G&J say hi ]

Hey, it's me, George.

I feel guilty that my introduction has to be accompanied by a request, but I'm really scared of forums.

I'm in a band called George & Jonathan, and we use pxtone exclusively to make our music. There is no other instrument that I have more virtuosity or proficiency with. It's my favorite way to express musical ideas. I plan on using it to write music for a long time.

Jonathan and I have about 8 or 9 years worth of PTCOPs. We like writing really "dense" songs, with lots of little details. I'm probably a better visual artist than I am a musician, and I've always wanted to create animated visualisations of our music. The simplest of which might be something like a stylized version of Synthesia's animated piano roll.

I need some programmatic way of accessing the note event data stored in our ptcop files. I always thought I'd need PTCOP => MIDI conversion to do this. While that would certainly do the trick, any solution that would allow me to programmatically access the note event data in a PTCOP file would also get the job done. For example, maybe I could write a program that reads the note events in a PTCOP file and outputs the data to a JSON file. It would just be a lot easier to verify that the output of a MIDI file is correct, and reading a MIDI file is a task I know I can solve.

I've tried looking at the documentation of the DLL's to no avail. The only programming languages I know well are Python, JavaScript and Java ... so headers and makefiles are sort of a mystery to me. On top of the fact that I can't read Japanese ... Pixel and I have exchanged some emails on the subject (at a painfully slow rate while I tried to find friends to translate between Japanese and English), but as far as I could glean, he's (understandably) too busy to look into the matter.

If anyone can ...
a. point me to a tool that would help me convert PTCOP to MIDI
b. point me to a resource that would help me create said tool myself

... then George & Jonathan would be eternally grateful. I'm most keen to be able to accurately convey the data in the "portamento" table. We're very much into pitch bends, and the ideal solution faithfully translates those portamento commands into MIDI pitch bend events.

I'll briefly list my failed attempts to achieve the aforementioned:

1. PTCOP => (something else?) => MIDI
ORG to MIDI exists, but can't find much about PTCOP => ORG. I saw a Java converter posted in these forums that does the opposite. Having the source to that project would definitely teach me a lot about reading / writing ptcops.

2. Hex code hell.
Make blank ptcop. Save. Stare at hex editor. Add one track. Stare at diff in hex. Add one note. Stare at diff in hex. This fails because I'm not smart enough.

3. Exporting stems, using pitch detection.
Changed all the samples in a given ptcop to a sine wave and tried to run pitch detection code. It sort of works, but it is so labor intensive (no good way to export "stems" from pxtone) and I get frequency over time data as opposed to "note" data.


If I get access to this data, it will unlock a universe of art that I've been dying to make for years. We have a new album coming out very, very soon, and as much as I would love for you to hear it, I would kill for you to be able to see it too.

Nice to meet you all, and thank you if you made it this far!!

- g&j

 

Hey guys, nice to see ya here! Love your jams.
I'm the guy that wrote the ORG=>PTCOP converter, so I know a thing or two about the format and I might be able to help you out. I only know a little bit about the format of MIDI but to my knowledge Pixel's PTCOP format actually borrows some things from midi format so a translation doesn't seem too far off.

Here are the notes I took on the format when I reverse-engineered it.

PTCOP FILE FORMAT

Generally speaking, the format seems to follow the pattern of
Block Name [0x8]
Block Size [0x4]
- Block data

Blocks marked with <!> should never be missing, 
but all other blocks are optional
It is currently unknown if the ordering of the blocks matters,
but blocks here are listed according to order of appearance in sample files.

//***********************************************************

0x00 : File header "PTCOLLAGE-071119"
0x10 : Unknown int (0x00000392)

"MasterV5" block <!>
0x0[0x2] Unknown data
0x2[0x1] Beat
0x3[0x2] Unknown data
0x5[0x2] Beat Tempo
0x7[0x4] "Repeat" location (meas * beat * 0x1E0?)
0xB[0x4] "Last" location

"Event V5" block
-This block contains both note signal events and effect events
-If there are no events, field contains a single null int
-If there are events, block data contains the following:
0x0[0x4] number of events
0x4[0x?] event data

Events have the following format
	Position indicator: a variable-length number 

	98  02 is
	(02-01)x80 +98

	80 8E 02 is
	(((02-01)*80)+8e-01)*80+80

	0x6000 = 80 C0 01

	Unit ID: A char indicating which unit this event applies to

	Event ID: a byte indicating which type of event it is
		0x1 - Play Instrument (duration)
		0x2 - Note pitch change(new note)
		0x3 - Pan(volume) (new pan)
		0x4 - Velocity
		0x5 - Volume (new volume)
		0x6 - Key Porta (length)
		0x7 - 
		0x8 - 
		0x9 - 
		0xA - 
		0xB - 
		0xC - Voice No (new voice)
		0xD - Group No (new group)
		0xE - Key Correct (key correct value)
		0xF - Pan(Time) (new pan?)
	
	Event Value: A variable-length number of the same format used
	for the position indicator that specifies the value being
	changed by the event. for example, the event value
	of a Volume event is the new volume of the unit.

"textNAME" block <!>
ASCII string containing the name of the song (not null-terminated)

"textCOMM" block
ASCII string containing song comments (not null-terminated)

"effeDELA" block
Unknown data

"effeOVER" block
Unknown data

"matePCM\20" block
0x00[0x3] Unknown data (padding?)
0x03[0x1] Basic Key field
0x04[0x4] Voice flags
	0x01 - Loop
	0x02 - Smooth
	0x04 - Beat Fit
0x08[0x2] number of channels (mono or stereo)
0x0A[0x2] bitrate (8, 16 or 32 bits)
0x0C[0x4] sample rate (Hz)
0x10[0x4] key correct (32Bit float?)
0x14[0x4] number of samples
0x18[0x?] sample frames - size of (bitrate/8 * nSample)

"matePTV\20"
0x00[0x0C] Unknown data
0x0C[0x08] "PTVOICE-"
0x14[0x0A] Unknown data (doesn't match .ptvoice file)
0x18[0x1A] Unknown data (matches .ptvoice file)
	--note: .ptvoice file is 2 bytes larger (?)

"matePTN\20"
Used for PTN samples, unresearched

"mateOGGV"
Used for OGG voices, unresearched

"assiWOIC" block - comes after every "mateXXXX" block
0x0[0x04] Voice number
0x4[0x10] Voice name. If less than 0x10 in len, will be null-terminated

"num UNIT" block <!>
-A single integer denoting the number of units in use by this song

"assiUNIT" block
-one for each unit
0x0[0x04] Unit number
0x4[0x10] Unit name. If less than 0x10 in len, will be null-terminated

"pxtoneND" block <!>
-zero size, denotes end of file

As you can see I only really put a lot of effort into the parts of the format that I'd need to translate ORG files, since those don't need some of pxtone's advanced features. But I might be able to fill out missing information.

http://noxid.ca/files/OrgPtcopConverter_src.zip

Also, here's my source. It's kind of a mess but maybe you'll be able to glean something from it.
I'm not sure if I have the time right now to fully commit to another converter but I'd love to be able to help out and share my knowledge.

 

Thanks so much -- this is spectacularly helpful. I've got a playdate soon with a friend of mine who's got of a bit of experience reverse engineering file formats, so I'm going to study this before then.

Everything here makes perfect sense to me in terms of the schema you describe -- with one exception: "position indicator: a variable-length number." Not sure what you're describing with these numbers:


98 02 is
(02-01)x80 +98

80 8E 02 is
(((02-01)*80)+8e-01)*80+80

0x6000 = 80 C0 01


Thankfully the parts that were irrelevant to you are also irrelevant to me

I basically only need the tempo and the event list. Maybe if there are >16 units, the converter outputs more than one MIDI file. Thanks so much though. You are shedding light on a centuries-old quest, I'm terribly excited.

 

Ah yeah, that bit was confusing to me as well but basically it is a special way of encoding numbers that based on my research is what's used in MIDI files. I guess Pixel used it to save space / store long numbers. Perhaps wikipedia explains it better:
http://en.wikipedia.org/wiki/Variable-length_quantity
The numbers in my 'example' are hex bytes as they appear in the file

The bit that decodes and encodes it is as follows

public int decodePxInt(byte[] array, int pos)
{
	int v = array[pos];
	if (v > 0x7f)
		return v + 80*(decodePxInt(array, ++pos) - 1);
	else
		return v;
}

public int int2pxInt(long val, int pos, byte[] array)
{
	if (val <= 0x7F) {
		array[pos] = (byte) val;
		return ++pos;
	} else {
		int fPos = int2pxInt(val/0x80, pos+1, array);
		array[pos] = (byte) (val - ((array[pos+1] & 0xFF)-1)*0x80);
		return fPos;
	}
}

 

Hey there again, starting coding a bit and I have two questions:



1. Are the event "position indicator" values relative? My guess is that they are given Unit ID's will jump around so much from event to event. All of the numbers generally seem to be [0, 40, 80, 120, 240] and rarely much bigger. And what unit are they in?



2. For the "note pitch change" directives, the event values seem really big (assuming I've ported the VLQ function correctly), things like 15408 or 17968 or 13808. I have no idea what unit that's supposed to be ... Here's my python so far:

import re, mmap

# map of bytes to event types
EVENTS = dict()
EVENTS[0x1] = 'On'
EVENTS[0x2] = 'Note'
EVENTS[0x3] = 'Pan (Volume)'
EVENTS[0x4] = 'Velocity'
EVENTS[0x5] = 'Volume'
EVENTS[0x6] = 'Key Porta'
EVENTS[0xC] = 'Voice No'
EVENTS[0xD] = 'Group No'
EVENTS[0xE] = 'Key Correct'
EVENTS[0xF] = 'Pan (Time)'

# function for reading variable-length quantities from a byte stream
def vlq(buf):
    v = ord(buf.read(1))
    if v > 0x7f:
        return v + 80*(vlq(buf) - 1)
    else:
        return v

# open my ptcop
f = open('inoj8.ptcop', 'r+b')

# use a memory mapped file, because ptcops are big.
mf = mmap.mmap(f.fileno(), 0)
mf.seek(0)

# look for the event block
m = re.search('Event V5', mf)
mf.seek(m.end())

# i don't know how to read this as an int ...
mf.read(4)

# look for the textNAME block (assuming this terminates the event block)
m = re.search('textNAME', mf)

# until you've hit the textNAME block ...
while mf.tell() < m.start():

    position = vlq(mf)

    unit_id = ord(mf.read(1))

    event_id = ord(mf.read(1))

    event_name = EVENTS[event_id] if event_id in EVENTS else '????????????????????????'
    event_value = vlq(mf)

    print position, unit_id, event_id, event_name, event_value

mf.close()
f.close()

But I'm delighted, the output looks totally sane to me with the exception of the first event:



(truncated)

122 51 0 ???????????????????????? 0
0 0 12 Voice No 24
0 1 12 Voice No 22
0 2 12 Voice No 12
0 4 12 Voice No 1
0 5 12 Voice No 2
0 7 12 Voice No 8
0 8 12 Voice No 25
0 10 12 Voice No 11
0 9 12 Voice No 3
0 11 12 Voice No 9
0 12 12 Voice No 20
0 18 12 Voice No 10
0 15 12 Voice No 14
0 16 12 Voice No 16
0 19 12 Voice No 13
0 20 12 Voice No 4
0 21 12 Voice No 4
0 22 12 Voice No 4
0 23 12 Voice No 4
0 24 12 Voice No 6
0 25 12 Voice No 5
0 26 12 Voice No 5
0 27 12 Voice No 5
0 28 12 Voice No 5
0 33 12 Voice No 15
0 34 12 Voice No 7
0 35 12 Voice No 18
0 29 12 Voice No 0
0 30 12 Voice No 0
0 31 12 Voice No 0
0 32 12 Voice No 0
0 14 12 Voice No 19
0 13 12 Voice No 21
0 36 12 Voice No 27
0 6 12 Voice No 30
0 3 12 Voice No 31
0 17 12 Voice No 32
0 38 12 Voice No 35
0 39 12 Voice No 36
0 40 12 Voice No 37
0 41 12 Voice No 38
0 42 12 Voice No 39
0 43 12 Voice No 40
0 37 12 Voice No 34
0 44 12 Voice No 42
0 45 12 Voice No 45
0 19 13 Group No 1
0 24 13 Group No 0
0 34 13 Group No 1
0 19 2 Note 11088
0 20 2 Note 16368
0 21 2 Note 17008
0 22 2 Note 17488
0 23 2 Note 16208
0 37 2 Note 13328
0 19 1 On 1368
0 20 1 On 1248
0 21 1 On 1248

I can almost taste it. You rule!


As far as I got tonight. Was able to convert the "on" and "note pitch" events for each unit into its own MIDI track. However, I'm pretty sure the note pitch is being interpreted incorrectly (sometimes?) and all the notes have the same duration. But oh my god I think it's gonna work. Baby steps.

Here's a link to the little python library I started for reading PTCOP files. At tonight's commit for posterity's sake.

https://github.com/georgealways/ptcop.py/commit/9dfcde8a500f3a096c026adb8d65e3cbdf8d96dc

 

1) Yeah, the note values are relative.

Here I have a note size 1 and a note size 4. The big one (1) has a length of 480, and the small one (4) has a note length of 120. The smallest placeable note length is a 48th, which has length 10.

2) The pitch values should be pretty large. A440 has a value of 0x6000, or 24576. Each halfnote goes up or down approximately 0x100 or 256. So, C4 is 0x6300 (25344) and C5 is 0x6F00 (28416). I don't know why it's like this, but this is the conversion that I use on org files notes that seems to give consistent results.
Your values seem a bit small to me but I'm not familiar with the intricacies of python so I couldn't say whether it's something subtle in your code causing an error or my calculations are just all wack.

 

Okay, so where I netted out on conversion to MIDI notes is as follows:

note = 60 + (value - 15888) / 160

What's giving me the most problems now turns out to be the "beat." MIDI wants a float position in "beats," (4 beats per measure in 4/4) so I'm converting as follows:

beat = value / 480.0

... and the MIDI events just don't wind up on the grid. If you play all the tracks together, it sounds musically correct, but if you turn on the metronome it's all wrong. I made a test PTCOP that was just a series of quarter notes (pxtone division size "1") and the values it reported back as the duration were 384 .... ?

So changing to 384 worked for that one particular PTCOP ... but not for another? Any reason this duration constant would change between PTCOP's ?


Upon further digging it turns out there's some events in this file with "Position Indicators" that aren't divisible by 10?I have a PTCOP where the first non-0 position indicator is "672" ... thereby putting a pea way down at the bottom of my stack of mattresses. It's for a Portamento command with length 528.

Seems like it should be impossible if the smallest division (48) is 10.

updated the repo a bunch too, check it out if you get a chance: https://github.com/georgealways/pxtone.py

UPDATE.

figured out why my note numbers looked small to you, and why the number 384 ever showed up in the first place.

I copied the VLQ function in your post too faithfully.

You wrote 80* where you meant 0x80*



we're on the grid.