Feat/fat decoding (#14)

* First step in implementing FAT file decoding : brute-force extraction without FNT lookup.

The FAT alone contains enough information to extract every file in a
folder chosen by the user. However, reference to the file name table
(FNT) is needed to restore the files' names and the directory structure
(and avoid dumping useless data that may be left over in the data
section).

This commit only implements the dumping of all data by incrementing
through the FAT and writing every section marked by a FAT range into a
separate file, giving them incremental names.

The next commit will use the FNT to name them properly.

* Proper FAT decoding : FNT lookup and directory structure

Code comments explain the algorithm (which was understood thanks to the 
wonderfully simple "FNT-Tool" script at : 
https://github.com/RoadrunnerWMC/FNT-Tool) ; the implementation is more 
or less a C++ port of the Python code.

Everything may still be a little dirty, but it works !

* Updated README!

* Updated README!

* MINOR compliance modifications 

- Removed extra entry in .gitignore
- Removed superfluous debug inclusions 
- Replaced magic values by preprocessor macros

Co-authored-by: rblard <rblard@enseirb-matmeca.fr>

.gitignore

@@ -44,3 +44,5 @@ CMakeLists.txt.user*
 
 # OS specific
 .DS_Store
+
+build/

README.md

@@ -3,8 +3,7 @@
 [![GPL Licence](https://badges.frapsoft.com/os/gpl/gpl.png?v=103)](https://opensource.org/licenses/GPL-3.0/)
 
 
-A tool to unpack & repack Nintendo DS roms (.nds)
-
+A tool to unpack & repack Nintendo DS ROMs (.nds)
 
 If you find this software useful, please [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/Z8Z511SOI)
 
@@ -27,8 +26,8 @@ Basically an NDS software is composed of the following sections:
 * Icon/Title Logo
 * FAT Files (The actual files used by the software, like Graphics, Music etc.)
 
-With NDSFactory you can extract these sections, modify them using your prefered way and the rebuild the rom with your edited sections.
-If the modified sections are bigger than the original ones, you can specify their new physical adddress and size in the header: if so, **make
+With NDSFactory you can extract these sections, modify them using your preferred way and the rebuild the rom with your edited sections.
+If the modified sections are bigger than the original ones, you can specify their new physical address and size in the header: if so, **make
 sure that they DON'T OVERLAP, and remember to PATCH THE FAT.BIN** (more on this later).
 
 **This software will be particularly useful if you want to mod your games or write a trainer for them.**
@@ -36,24 +35,22 @@ sure that they DON'T OVERLAP, and remember to PATCH THE FAT.BIN** (more on this
 # How to use it
 
 ## Unpacker Tab
-In the upacker tab you can load your Nintendo DS software (.nds) and then you can extract the rom sections.
+In the unpacker tab you can load your Nintendo DS software (.nds) and then you can extract the ROM sections, including the individual FAT files.
 Please note the Original Address of the FAT Files, you will need this value later if you are going to alter the addresses and size of the sections.
 
 **You can then do what you want with these sections (inject code, apply patches etc.)**
 
 ## Packer Tab
-In the packer tab you can re-create an .nds file using your edited sections. If your sections are bigger than the originals, you have to edit their addresses and size (in the header). **Make sure that the addresses don't overlap, or the final rom will be broken**. If you are repacking edited sections, and the FAT Files Address is different than the original one, **make sure to patch the FAT (fat.bin)**: the FAT is a list of absolute addresses (representing each file start adddress and end andress), so you need to update them (you can easily do this using the FAT Patching Tab).
+In the packer tab you can re-create an .nds file using your edited sections. If your sections are bigger than the originals, you have to edit their addresses and size (in the header). **Make sure that the addresses don't overlap, or the final rom will be broken**. If you are repacking edited sections, and the FAT Files Address is different than the original one, **make sure to patch the FAT (fat.bin)**: the FAT is a list of absolute addresses (representing each file start address and end address), so you need to update them (you can easily do this using the FAT Patching Tab).
 
 ## Fat Patching Tab
 In this tab you can easily patch the FAT section (fat.bin). You have to do this only if the FAT Files (fat_data.bin) final address is different than the original one.
 Patching the FAT is easy, all you have to do is load your fat.bin, and fill the original address and the new address of fat_data.bin. This will produce a patched fat.bin that
 you can use in the packing process.
 
-
 # Known Limitations/Possible Future Features/Bugs
 
 * Add support for roms with OVERLAY
-* Add support to decode FAT Files (extract single files one by one)
 * Add support to rebuild a new fat_data.bin and fat.bin from a set of files inside a directory
 * Design a nice logo/icon
 
@@ -61,4 +58,3 @@ If you found a bug, feel free to open an issue or send a PR :)
 
 ### Developed with ❤ by Luca D'Amico
 ### Special thanks to Antonio Barba & Davide Trogu
-

ndsfactory/ndsfactory.cpp

@@ -80,6 +80,12 @@ bool NDSFactory::writeSectionToFile(const std::string& sectionPath, const std::s
     return false;
 }
 
+bool NDSFactory::writeFatSectionToFile(const std::string& romPath, FatRange* pfatrange, const std::string& savePath){
+    uint32_t size=pfatrange->endAddr-pfatrange->startAddr;
+    if(!dumpDataFromFile(romPath, savePath, pfatrange->startAddr, size)) return false;
+    return true;
+}
+
 bool NDSFactory::writeBytesToFile(std::vector<char>& byteBuffer, const std::string& savePath, uint32_t startAddr, uint32_t size)
 {
     std::ofstream savedFile (savePath, std::ios::out|std::ios::binary|std::ios::app);

ndsfactory/ndsfactory.h

@@ -5,6 +5,7 @@
 #include <vector>
 #include <cstdint>
 #include "ndsheader.h"
+#include "fatstruct.h"
 
 
 
@@ -16,6 +17,7 @@ public:
     bool dumpDataFromFile(const std::string& romPath, const std::string& savePath, uint32_t startAddr, uint32_t size);
     bool readBytesFromFile(std::vector<char>& byteBuffer, const std::string& romPath, uint32_t startAddr, uint32_t size);
     bool writeSectionToFile(const std::string& sectionPath, const std::string& savePath, uint32_t startAddr, uint32_t size);
+    bool writeFatSectionToFile(const std::string& romPath, FatRange* pfatrange, const std::string& savePath);
     bool writeBytesToFile(std::vector<char>& byteBuffer, const std::string& savePath, uint32_t startAddr, uint32_t size);
     bool writePaddingToFile(char paddingChar, const std::string& savePath, uint32_t startAddr, uint32_t size);
     int getCardSizeInBytes(int cardType);

ui/dialogs/about/revision.h

@@ -1,6 +1,6 @@
 #ifndef REVISION_H
 #define REVISION_H
 
-#define GIT_COMMIT_HASH "319d0e2"
+#define GIT_COMMIT_HASH "30425b6"
 
 #endif

ui/mainwindow.h

@@ -108,7 +108,7 @@ private:
     //QString extractUnpackerHeaderTableData(int index);
     QString extractPackerHeaderTableData(int index);
 
-    bool decodeFatFiles();
+    bool decodeFatFiles(QString dirPath);
 
     bool patchFat(const std::string& loadPath, uint32_t shiftSize, const std::string& savePath);
 };

ui/tabs/unpacker/unpackertabfunctions.cpp

@@ -1,13 +1,26 @@
 #include <QDir>
 #include <stdlib.h>
-#include <cstring>
 #include <sstream>
 #include <iomanip>
 #include "../../mainwindow.h"
 #include "../../ui_mainwindow.h"
 #include "../commons/headernames.h"
 #include "../../models/ndsheadermodel.h"
+#include "../../../ndsfactory/fatstruct.h"
 
+// Byte offsets for interpreting memory
+
+#define SECOND_BYTE_SHIFT 8
+#define THIRD_BYTE_SHIFT 16
+#define FOURTH_BYTE_SHIFT 24
+
+// Magic values for FAT extraction
+
+#define CONTROL_BYTE_LENGTH_MASK 0x7F
+#define CONTROL_BYTE_DIR_MASK 0x80
+#define DUMMY_CONTROL_VALUE 0xFF
+#define FNT_HEADER_OFFSET_MASK 0XFFF
+#define ROOT_DIRECTORY_ADDRESS 0xF000
 
 void MainWindow::populateHeader(NDSHeader* ndsHeader)
 {
@@ -167,8 +180,149 @@ bool MainWindow::dumpEverything(QString dirPath)
     return true;
 }
 
-bool MainWindow::decodeFatFiles()
+bool MainWindow::decodeFatFiles(QString dirPath)
 {
-    // TODO: implement me!
+    // Prepare necessary info from ROM
+
+    std::string romPath = ui->loadedRomPath->text().toStdString(); // ROM itself
+
+    // Addresses of the file allocation table and file name table
+
+    uint32_t fatAddr = ui->unpackerHeaderDataTable->model()->index(NDSHeaderNames::FATAddress, 1).data().toString().toUInt(nullptr,16);
+    uint32_t fatSize = ui->unpackerHeaderDataTable->model()->index(NDSHeaderNames::FATSize, 1).data().toString().toUInt(nullptr,16);
+
+    // Sizes of these tables
+
+    uint32_t fntAddr = ui->unpackerHeaderDataTable->model()->index(NDSHeaderNames::FilenameTableAddress, 1).data().toString().toUInt(nullptr,16);
+    uint32_t fntSize = ui->unpackerHeaderDataTable->model()->index(NDSHeaderNames::FilenameTableSize, 1).data().toString().toUInt(nullptr,16);
+
+    // Buffers to receive the contents of the FAT and FNT
+
+    std::vector<char> fatBytes(static_cast<unsigned long>(fatSize));
+    std::vector<char> fntBytes(static_cast<unsigned long>(fntSize));
+
+    // Fill them
+
+    if(!ndsFactory.readBytesFromFile(fatBytes, romPath, fatAddr, fatSize)) return false;
+    if(!ndsFactory.readBytesFromFile(fntBytes, romPath, fntAddr, fntSize)) return false;
+
+    // Use the available FAT range struct and read the FAT bytes as such
+
+    FatRange* pfatrange = reinterpret_cast<FatRange*>(fatBytes.data());
+
+    // Recursive function that looks up FNT info to find file names and directory structures,
+    // And writes the ROM data in the ranges indicated by the FAT simultaneously.
+
+    auto parseFolder = [this, fntBytes, pfatrange, romPath](uint32_t folderId, std::string curPath, auto& parseFolder){
+
+        if(false) return false; // this is stupid, but it's C++
+        // If we take it out, the compiler will complain because it doesn't known the return type of the lambda
+        // But we can't make the lambda bool either...so this is the best option...
+
+        QDir curDir(QString::fromStdString(curPath)); // useful a bit later
+
+        uint32_t currentOffset = 8 * (folderId & FNT_HEADER_OFFSET_MASK); // offset for the current directory's info in the FNT header
+        // Only the lower 12 bit of the given offset are relevant
+
+        // ---------------------------------------------------------------------
+        // About how the FAT and FNT work :
+
+        // The FNT has two sections :
+        // a "header" where every entry contains :
+          // - a 4-byte address where the corresponding directory's data starts in the body
+          // - a 2-byte offset that is the index of the first file of the directory in the FAT
+          // (e.g. : if the offset is 42, the first file in the directory is situated at the ROM addresses stored in the 42nd FAT entry)
+          // (and its second will be 43, etc.)
+        // a "body" where every entry contains :
+         // - a length+status/control byte : lower 7 bits (control byte & 0x7F) are a length, highest bit (control byte & 0x80) is set if entry is a directory, and not set if it's a file
+         // - a name which length is the length portion of the previous control byte (e.g. : if the control byte was 0x83, the name is three bytes long)
+         // - if the entry is a directory, a 2-byte address (where only the lower 12 bit are relevant for some reason) at which this directory's info is located in the FNT header
+
+        // Thus, the FNT reading operation will consist in bouncing back and forth between body and header every time we must process a subdirectory
+        // Thank Heavens for random-access containers !
+
+        // ---------------------------------------------------------------------
+
+        // Get the 4-byte address for the folder data
+
+        uint32_t fntBodyOffset =
+          (uint32_t)((unsigned char) fntBytes[currentOffset+3] << (uint32_t) FOURTH_BYTE_SHIFT |
+          (unsigned char) fntBytes[currentOffset+2] << (uint32_t) THIRD_BYTE_SHIFT |
+          (unsigned char) fntBytes[currentOffset + 1] << (uint32_t) SECOND_BYTE_SHIFT |
+          (unsigned char) fntBytes[currentOffset]);
+        currentOffset+=4;
+
+        // Get the 2-byte offset for the folder's first file in the FAT
+
+        uint16_t fatOffset =
+          (uint16_t)((unsigned char) fntBytes[currentOffset+1] << SECOND_BYTE_SHIFT |
+          (unsigned char) fntBytes[currentOffset]);
+
+        // Jump to FNT body a specified address
+
+        currentOffset = fntBodyOffset;
+
+        uint8_t controlByte = DUMMY_CONTROL_VALUE;
+
+        while(true){
+
+            controlByte = fntBytes[currentOffset]; // Entry's control byte
+            if(controlByte==0) break; // A control byte of 0 terminates the directory's contents
+            currentOffset++;
+
+            uint8_t nameLength = controlByte & CONTROL_BYTE_LENGTH_MASK; // length of entry name
+            bool isDir = controlByte & CONTROL_BYTE_DIR_MASK; // set if entry is a directory
+
+            // Reconstitute name from bytes
+            // Btw I wish I could use the actual byte type but I have to comply with the software's choice of using char
+
+            std::vector<char> nameString;
+            for(size_t i = 0 ; i<nameLength ; i++) nameString.push_back(fntBytes[currentOffset++]);
+            std::string name(&nameString[0], (size_t)nameLength);
+
+            // We'll need this either way
+
+            QString newPath(QDir::toNativeSeparators(QString::fromStdString(curPath+"/"+name)));
+
+            if(isDir){
+
+                // Get the 2-byte address for this folder's info in the FNT header
+                uint16_t subFolderId = ((unsigned char) fntBytes[currentOffset+1] << SECOND_BYTE_SHIFT |
+                (unsigned char) fntBytes[currentOffset]);
+                currentOffset+=2;
+
+                // Now the QDir we created earlier comes into play :
+                // C++ doesn't automatically create directories (!!!) so we have to rely on QT to do that manually.
+                // Otherwise, the ofstream will not open,
+                // And even if we force it open, it will just write into nothingness !!!
+
+                if(!curDir.exists(newPath)) curDir.mkdir(newPath); // I don't think the check is even necessary, actually
+
+                // Jump back to the FNT header and repeat the process for subdirectory !
+
+                if(!parseFolder(subFolderId,newPath.toStdString(),parseFolder)) return false;
+
+            }
+            else{
+
+                // Remember we have the offset for the directory's first file in the FAT.
+                // From then, every file is just the next entry.
+                // So we just have to use that offset and increment it every time.
+
+                if(!ndsFactory.writeFatSectionToFile(
+                  romPath,
+                  pfatrange+fatOffset,
+                  newPath.toStdString()))
                   return false;
+                fatOffset++;
+
+            }
+        }
+
+        return true;
+    };
+
+    // The root folder's ID is, obviously, 0 (only the lower 12-bit count!)
+
+    return parseFolder(ROOT_DIRECTORY_ADDRESS,dirPath.toStdString(),parseFolder);
 }

ui/tabs/unpacker/unpackertabsignals.cpp

@@ -202,7 +202,14 @@ void MainWindow::on_unpackerDumpEverythingBtn_clicked()
 
 void MainWindow::on_unpackerDecodeFatFilesBtn_clicked()
 {
-    QMessageBox::warning(this, tr("NDS Factory"), tr("This function is currently not implemented!"));
-    decodeFatFiles();
-}
+    QString dirPath = QFileDialog::getExistingDirectory(
+                this, tr("Select Directory"),
+                "",
+                QFileDialog::ShowDirsOnly | QFileDialog::DontResolveSymlinks);
 
+    if (!dirPath.isNull())
+    {
+        decodeFatFiles(dirPath) ? QMessageBox::information(this, tr("NDS Factory"), tr("FAT files successfully decoded!"))
+                         : QMessageBox::critical(this, tr("NDS Factory"), tr("Error during FAT file decoding!"));
+    }
+}