[Release] [C#] [LIBRARY] SABSFile library (open sound files)

[JariZ]

Hi everybody, Today I bring you a small project I've been working on (and struggling with).
The idea was to create a opensource dotZip-similar type of extracting and packaging library for the black ops soundfiles.
Sadly I was unable to finish the packaging (replacing) part.
I hope I can get people excited about this and you are free to improve it (please do)
I do not have that much experience involving algorithms and hex'n'bytes'n'stuff so excuse me if I didn't use the right methods (criticism is welcome)
I do believe that the information and the offsets of each flac file is defined in the header, because there's a lot of stuff in the header (that I don't understand, sadly)

Pro's:

• Scan for FLAC files inside of sabs files
  
• Get raw bytes from each flac file in there
  
• Extract file directly from class
  
• Replace (doesn't work properly, works when extracting again but crashes game.)
  
• Very low memory usage
  

Cons:

• Keep in mind that when calling the constructor (new SABSFile("d.sabs")) the class starts looking for offsets which takes a few seconds. Do NOT call it from the main thread in a form application.
  
• Replacing fails ingame (soundbank failed to initialize)
  

master131-flac-extractor-style code example:
Code:

CSHARP Code

Console.WriteLine("SABSFile library example extractor");
            string a = GetSteamPath()+"\\steamapps\\common\\Call of Duty Black Ops II";
            foreach(string file in Directory.GetFiles(GetSteamPath()+"\\steamapps\\common\\Call of Duty Black Ops II\\sound", "*.sabs"))
            {
                Console.WriteLine("Extracting " + Path.GetFileName(file));
                SABSFile sabs = new SABSFile(file);
                string path = Path.GetDirectoryName(Application.ExecutablePath);
                int i = 0;
                string dir = path+"\\extracted\\"+Path.GetFileNameWithoutExtension(file);
                    if (!Directory.Exists(dir)) Directory.CreateDirectory(dir);
                    else foreach (string filef in Directory.GetFiles(dir)) File.Delete(filef);
                foreach (SABSEntry entry in sabs.Entries)
                {
                    i++;
                    Console.Write("\r      Progress: "+Convert.ToInt32((double)i / (double)sabs.Entries.Length * 100.0) + "%");
                    entry.Extract(dir + "\\" + entry.Offset.ToString("X").ToLower() + ".flac");
                }
                Console.WriteLine();
            }

Result:



All that with 20 lines of code?
Yes, that's the power of SABSFile

Download (library, no source, just example source)

  SABSFile.zip (Size: 4.4 KB / Downloads: 129)
Source

CSHARP Code

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Collections;
 
namespace JariZ
{
    public class SABSFile
    {
        public SABSFile(string filename) {
            if (string.IsNullOrEmpty(filename)) throw new ArgumentException("Invalid argument");
            if (Path.GetExtension(filename) != ".sabs") throw new FileLoadException("Only SABS files supported");
            if (!File.Exists(filename)) throw new FileNotFoundException("SAB File not found");
            /*
            try
            {*/
                _filename = filename;
                byte[] header = new byte[8] { 0x66, 0x4C, 0x61, 0x43, 0x00, 0x00, 0x00, 0x22};
                HeaderSearch hs = new HeaderSearch(header);
                using (FileStream reader = new FileStream(filename, FileMode.Open))
                {
                    int read;
                    int index = 0x800;
                    reader.Position = 0x800;
 
                    List<int> offsets = new List<int>();
                    while ((read = reader.ReadByte()) >= 0)
                    {
                        index++;
                        if (hs.Search((byte)read))
                        {
                            int offset = index - header.Length;
                            offsets.Add(offset);
                        }
                    }
                    offsets.Add(Convert.ToInt32(reader.Position-1));
 
                    int indexx = 0;
                    foreach(int offset in offsets)
                    {
                        indexx++;
                        if (indexx != 1)
                            _entries.Add(new SABSEntry(this, offsets[indexx-2], offset));
                    }
 
                }
               /* 
            }
            catch (Exception z)
            {
                throw new Exception("Unable to read file", z);
            }*/
        }
 
        string _filename = null;
        public string Filename
        {
            get 
            {
                return _filename;
            }
        }
 
 
        List<SABSEntry> _entries = new List<SABSEntry>();
        public SABSEntry[] Entries
        {
            get
            {
                return _entries.ToArray();
            }
        }
    }
 
    public class SABSEntry
    {
        public SABSEntry(SABSFile parent, int offsetstart, int offsetend)
        {
            _offsetend = offsetend;
            _offsetstart = offsetstart;
            _parent = parent;
        }
 
        public int Offset
        {
            get
            {
                return _offsetstart;
            }
        }
 
        public void Extract(out byte[] buffer)
        {
            List<byte> lbuffer = new List<byte>();
            using (FileStream fs = new FileStream(_parent.Filename, FileMode.Open))
            {
                fs.Position = _offsetstart;
                while (fs.Position <= _offsetend) lbuffer.Add((byte)fs.ReadByte());
            }
            buffer = lbuffer.ToArray();
        }
 
        SABSFile _parent;
        public void Extract(string filename)
        {
            using (FileStream fs = new FileStream(_parent.Filename, FileMode.Open, FileAccess.Read, FileShare.None))
            {
                if (!Directory.Exists(Path.GetDirectoryName(filename))) Directory.CreateDirectory(Path.GetDirectoryName(filename));
                if (!File.Exists(filename)) File.Create(filename).Dispose();
                using(FileStream gs = new FileStream(filename, FileMode.Truncate, FileAccess.Write, FileShare.None))
                {
                    fs.Position = _offsetstart;
                    while (fs.Position <= _offsetend) gs.WriteByte((byte)fs.ReadByte());
                    gs.Flush();
                    gs.Close();
                    fs.Close();
                    gs.Dispose();
                    fs.Dispose();
                }
            }
        }
 
        public void Replace(byte[] content)
        {
            byte[] b = File.ReadAllBytes(_parent.Filename);
            List<byte> lb = new List<byte>();
            lb.AddRange(b);
            lb.RemoveRange(_offsetstart, _offsetend - _offsetstart);
            lb.InsertRange(_offsetstart, content);
            File.WriteAllBytes(_parent.Filename, lb.ToArray());
 
            /*using (FileStream fs = new FileStream(_parent.Filename, FileMode.Open, FileAccess.ReadWrite, FileShare.None))
            {
                fs.Position = _offsetstart;
                byte[] otherdata = null;
                bool crossed_offset = false;
                string tempfile = string.Empty;
                foreach(byte bite in content)
                {
                    if (fs.Position <= _offsetend)
                    {
                        fs.WriteByte(bite);
                    }
                    else
                    {
                        //we've crossed our offset, no worries, but we need to start inserting data from now on.
                        //this is simple. we read the rest of the data, write it to a temporary file (to prevent memleaks)
                        if (!crossed_offset)
                        {
                            int b4 = Convert.ToInt32(fs.Position);
                            FileStream gs = new FileStream((tempfile = Path.GetRandomFileName()), FileMode.OpenOrCreate, FileAccess.Write, FileShare.None);
                            while (fs.Position < Convert.ToInt32(fs.Length)) gs.WriteByte((byte)fs.ReadByte());
                            gs.Flush();
                            gs.Close();
                            gs.Dispose();
                            crossed_offset = true;
                            fs.Position = b4;
                        }
                        fs.WriteByte(bite);
                    }
                }
                //now that we have written our new file, update the offset
                _offsetend = Convert.ToInt32(fs.Position);
                if (crossed_offset)
                {
                    fs.Position = _offsetend;
                    FileStream gs = new FileStream(tempfile, FileMode.Open, FileAccess.Read, FileShare.None);
                    while (gs.Position < Convert.ToInt32(gs.Length)) fs.WriteByte((byte)gs.ReadByte());
                    gs.Flush();
                    gs.Close();
                    gs.Dispose();
                }
                fs.Flush();
                fs.Close();
                fs.Dispose();
            }*/
        }
 
        public void Replace(string filename)
        {
            Replace(File.ReadAllBytes(filename));
        }
 
        int _offsetstart = 0;
        int _offsetend = 0;
    }
 
    public class HeaderSearch
    {
        public HeaderSearch(byte[] header)
        {
            _bufferlength = header.Length;
            _header = header;
            buffer = new List<byte>(_bufferlength);
            _headerend = _header[_header.Length-1];
 
            //fill buffer
            while (buffer.Count != _bufferlength) buffer.Add(0);
        }
 
        byte[] _header;
        int _bufferlength = 0;
        List<byte> buffer;
        int _headerend = 0;
        public bool Search(byte newbyte)
        {
            buffer.RemoveAt(0);
            buffer.Add(newbyte);
            if (newbyte == _headerend)
            {
                byte[] bbuffer = buffer.ToArray();
                if (bbuffer.SequenceEqual(_header))
                    return true;
                else return false;
            }
            else return false;
        }
    }
}

[master131]

Nice job however....
1. The way you are using to search for the FLAC header seems extremely slow. You are using a list and removing the 0th item, forcing a resize and array shift and therefore slowing down performance.
2. You have not taken the hash data at the end of the SABS file which will be appended to the end of the FLAC file into account (most players will probably ignore this but it can cause some problems).
3. You have hardcoded the position 0x800, but this could easily change if necessary. You should be relying on the header's start offset value or starting at 0. It's probably just a coincidence that all the data start at the same position in all the files.
4. Similar to point #1, using Lists for byte intensive operations like this can seriously slow down everything.
5.

Code:

gs.Flush();
gs.Close();
fs.Close();
gs.Dispose();
fs.Dispose();

None of that is necessary, the using statement automatically does that because FileStream implements IDisposable which will automatically call Dispose -> Close and Dispose -> Flush.
6. Try to avoid using Convert.ToInt32 and use the cast operator, aka (int). It just creates extra overhead for something simple.
7. Obviously the replace won't work if the original and replacement data size are different. The SABS file stores the positions of the data, it doesn't just check them in the order they appear in, otherwise it'd be super slow for the game to find a sound. The best way to solve this would be to pad the data with zeros if it is too small or throw an exception if the replacement data is too large. The problem with this though is if you are replacing music that is supposed to loop, it may create a major silence and I haven't found a solution around that yet.

Perhaps I might be too critical, but you wanted criticism so I gave it.

[JariZ]

1. I know, I could load the whole thing into the memory with readallbytes or something but that would max memory usage trough the roof so this is what I was trying to avoid. I noticed it's slow but I simply could not think of a better way.
2. I didn't knew there was any data that wasn't part of the FLAC
3. I don't think that's a coincidence although I agree that it's a bad way
4. See 1.
5. Didn't knew that, I'll remember that, I just wanted to make sure the file was closed
6. I thought that was more likely to throw errors, but okay.
7. As I said in the introduction of the post, I already thought of that. There's a lot of data in the header (everything until 0x800) and I'm sure the positions are defined there as well. It would be pretty cool if we could change them so we could use larger files than the originals.
I will try to fuck around a bit more with replacing.
Also, Do you have any idea what the sabl files are and what they are used for?

PS: This was my method of approaching it, and I know, it's not perfect (at all). I'm still learning while I'm working on this and that's why I asked for criticism.

---

It should also be noted that the reason why I decided to release it anyway was because I got stuck and just didn't want to give up because it's a pretty cool idea.

[master131]

No, I have no idea what the SABL files do or what they're used for.

> I noticed it's slow but I simply could not think of a better way.

Oh, there is. Also there is a better method than pattern scanning for the FLAC header/signature. The SABS file actually stores an array of structs containing file size and file position.

[JariZ]

Yes. that's what I was asking.
Where and how do I read it?

[master131]

Code:

struct sabs_header_t
{
    char _0x0[20];
    int entry_count; // 0x14
    int start_offset; // 0x18
    char _0x1C[4];
    hash_block_t** hashes; // 0x20, relative. (not sure about this one)
    char _0x24[4];
    audio_entry_t** entries; // 0x28, relative.
};

struct hash_block_t
{
    char md5[16];
    char 0x10[16];
};

struct audio_entry_t
{
    char _0x00[4];
    int size; // 0x4
    int offset; // 0x8
    char _0x0C[8];
}; // size = 0x14

You can figure it out yourself.

[JariZ]

Thanks, this is awesome!

[JariZ]

Well, I failed, again. http://easycaptu.re/bEbW4
I'm not sure what I'm doing wrong but the header doesn't seem to match up with what you said either.
The only things that seem to be succeeding is the entry count and the offset (even though it says 0x8 instead of 0x800)
Idk, I've been trying for 4 hours now.

[master131]

Don't use the MD5 hashes thing, it's incorrect, I'm still figuring that one out. As for it not working, it is a pointer to an array of hash_block_t and audio_entry_t. I forgot to add an extra * to show that, my bad.

Code:

fs.ReadBytes(20);
entrycount = fs.ReadInt32();
fs.ReadBytes(4); // start offset
fs.ReadBytes(4); // unknown
fs.ReadBytes(4); // hashes?
fs.ReadBytes(4); // unknown
int entryoffset = fs.ReadInt32();
fs.BaseStream.Seek(entryoffset, SeekOrigin.Begin);
for (int i = 0; i < entrycount; i++)
{
    fs.ReadBytes(4);
    SABSEntry sabs = new SABSEntry { Offset = fs.ReadInt32(), Size = fs.ReadInt32() };
    fs.ReadBytes(8);
    entries.Add(sabs);
}

[master131]

Just came with a major breakthrough, and realised that the SABL files contain headerless WAV files, however they do not use the header format seen in Black Ops, I don't think they even have headers....

Anyway, I used mpl_nightclub.all.sabl as an example and found a water fountain sound stored as raw data encoded using 16-bit PCM signed, little endian, stereo and a sample rate of 44100KHz or 48000KHz (not sure which one, both sound the same).